Eeg-to-text translation method based on decoupled eeg representation learning and related device

Abstract

This application provides a method and related equipment for EEG-to-text translation based on decoupled EEG representation learning, belonging to the interdisciplinary field of brain-computer interface technology and natural language processing technology. The method includes: a decoupled pre-training stage where static subject features and dynamic semantic features are extracted from EEG signals by an individual encoder and a semantic encoder, respectively, and temporal shuffling and mutual information constraints are introduced to force the two to be independent, training a pure semantic feature extractor. In the text generation stage, the EEG signal to be translated is input into the trained semantic encoder to obtain pure EEG semantic features. Then, through sentence-level and word-level multi-granularity cross-modal contrastive learning, these features are hierarchically aligned with the natural language text feature space, and finally input into a pre-trained language model for decoding to generate natural language text. This application can achieve universal translation across subjects without subject identity labels, and significantly improves the accuracy and coherence of the generated text through multi-granularity semantic alignment.

Description

Technical Field

[0001] This application relates to the intersection of brain-computer interface technology and natural language processing technology, and in particular to a brain-to-text translation method and related equipment based on decoupled brain-computer representation learning. Background Technology

[0002] The development of brain-computer interface (BCI) technology has made the direct translation of brain activity signals into natural language text a hot research topic. For individuals with severe language disorders (such as locked-in syndrome), this technology has revolutionary communication assistance potential. Compared to invasive methods requiring implanted electrodes (such as ECoG) or expensive non-invasive methods (such as fMRI and MEG), electroencephalography (EEG) has become the most promising brain-text decoding method in real-world scenarios due to its portability, non-invasiveness, and cost-effectiveness.

[0003] Existing EEG-to-text decoding methods typically utilize deep learning to map raw EEG signals into text. To bridge the representational gap between EEG signals and text, some methods introduce contrastive learning strategies, constructing positive and negative sample pairs to narrow the distance between EEG representations and corresponding text features. Furthermore, to address the differences in EEG data among individuals, existing methods often employ adaptive strategies tailored to specific subjects or utilize contrastive learning with subject-specific labels to construct a unified representation space.

[0004] However, existing technologies have the following drawbacks: First, EEG signals are essentially an aggregation of multiple neural activity sources, interspersed with a large number of non-semantic fluctuations (such as the individual physiological characteristics of the subject, environmental noise, etc.). Due to significant individual differences, different individuals will produce drastically different neural activity patterns when processing the same language stimuli. Existing methods cannot reliably decouple semantic content from subject-specific noise. The models are highly dependent on explicit subject identity labels for supervised training, resulting in extremely weak generalization ability and a sharp decline in performance when facing new individuals. Second, existing EEG-to-text generation methods are often limited to fine-grained semantic alignment at the word level, lacking macro-level semantic structure alignment guidance at the sentence level. As a result, although the final generated text may translate some specific words, it has serious deviations in grasping the overall meaning and contextual coherence of the sentence. Summary of the Invention

[0005] The main objective of this application is to propose an EEG-to-text translation method, electronic device, storage medium, and program product based on decoupled EEG representation learning, which can achieve EEG decoding that is universal across subjects and significantly improve the accuracy and coherence of the translated text.

[0006] To achieve the above objectives, one aspect of this application proposes a brainwave-to-text translation method based on decoupled brainwave representation learning, the method comprising: Decoupling the pre-training phase: The raw EEG signals are input into an individual encoder and a semantic encoder respectively. By constraining the features extracted by the two to be independent, a semantic encoder for extracting pure semantic features from EEG signals is trained. The individual encoder is used to extract static subject features that are not semantically independent, and the semantic encoder is used to extract dynamic semantic features that are related to time. Text generation stage: The target EEG signal to be translated is input into the trained semantic encoder to obtain pure EEG semantic features; The EEG semantic features are aligned with the feature space of natural language text in a multi-granularity cross-modal manner to obtain the aligned EEG semantic features. The aligned EEG semantic features are input into a pre-trained language model, which is then decoded to generate the corresponding natural language text.

[0007] In some embodiments, the decoupling pre-training phase specifically includes: The original EEG feature sequence was shuffled to obtain a shuffled sequence. The shuffled sequence is input into the individual encoder to extract the static subject features; The EEG feature sequence, which maintains its original temporal order, is input into the semantic encoder to extract the dynamic semantic feature sequence; The static subject features are fused with the dynamic semantic feature sequence to obtain fused features; The fused features are input into the decoder to reconstruct the original EEG signal; Based on reconstruction error and mutual information constraints, a first objective function is constructed, and the individual encoder, semantic encoder and decoder are jointly trained to force the static subject features and the dynamic semantic features to be independent of each other by minimizing the first objective function.

[0008] In some embodiments, fusing the static subject features with the dynamic semantic feature sequence to obtain fused features specifically includes: The static subject features are copied and extended in the time dimension so that their length is consistent with the dynamic semantic feature sequence; The expanded static subject features are concatenated with the dynamic semantic feature sequence along the feature channel dimension to obtain the fused features.

[0009] In some embodiments, the mutual information constraint is implemented through a variational information bottleneck, and the first objective function is expressed as:

[0010] The first term is the reconstruction loss, and the last two terms are KL divergence constraint terms, which are used to limit the channel capacity of latent variables.

[0011] In some embodiments, the multi-granularity cross-modal alignment includes: Extract word-level and sentence-level text features from the target text; Using sentences as units, the global EEG semantic features are compared and aligned with the sentence-level text features in a coarse-grained manner; Using words as units, local EEG semantic features are compared and aligned with the word-level text features in a fine-grained manner.

[0012] In some embodiments, the training process of the text generation stage is jointly optimized by a second objective function, which is expressed as:

[0013] in, Cross-entropy reconstruction loss for the generated text, For sentence-level contrast loss, For word-level contrast loss, , , For weight hyperparameters.

[0014] In some embodiments, the individual encoder includes a sequentially connected residual network, a global average pooling layer, and a multilayer perceptron; the semantic encoder includes a sequentially connected one-dimensional convolutional layer and a multi-head self-attention network.

[0015] In some embodiments, the pre-trained language model is a large language model based on an encoder-decoder architecture or a decoder-only architecture.

[0016] To achieve the above objectives, another aspect of this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the method described above.

[0017] To achieve the above objectives, another aspect of the embodiments of this application proposes a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described above.

[0018] To achieve the above objectives, another aspect of the embodiments of this application proposes a computer program product, including a computer program that, when executed by a processor, implements the method described above.

[0019] Compared with the prior art, this application includes at least the following beneficial effects: 1) This application separates EEG signals through independent individual encoders and semantic encoders, and incorporates a randomized shuffling sequence design in the individual branch. Combined with mutual information regularization, it achieves complete decoupling of identity information and semantic information in EEG signals, ensuring that the EEG decoding to text generation process is unaffected by individual information. This solves the problem of existing technologies relying on subject identity labels. Because the features extracted by the semantic encoder are filtered out from interference caused by individual differences in the subjects, the system can still maintain extremely high generalization ability and accurate translation performance when facing completely new and unseen individuals.

[0020] 2) This application innovatively introduces a multi-granularity cross-modal contrastive learning module that combines sentence-level and word-level approaches in cross-modal translation. Compared to existing techniques that only perform local word-level constraints, this approach not only accurately anchors local details through word-level alignment but also captures the overall meaning of the sentence. This significantly narrows the huge modal gap between EEG neural representations and natural language text representations, and substantially improves the objective metric scores of the generated text as well as the accuracy of vocabulary translation. Attached Figure Description

[0021] Figure 1 This is a flowchart of the EEG-to-text translation method based on decoupled EEG representation learning provided in the embodiments of this application; Figure 2 This is a schematic diagram of the EEG-to-text translation system according to an embodiment of this application; Figure 3 This is a schematic diagram of the structure of the subject-semantic decoupling module in an embodiment of this application; Figure 4 This is a schematic diagram of the structure of the EEG-to-text generation module according to an embodiment of this application; Figure 5 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit it. In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with those of this application; they are merely examples of apparatuses and methods consistent with some aspects of the embodiments of this application as detailed in the appended claims.

[0023] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0024] Before providing a detailed description of the embodiments of this application, some of the nouns and terms involved in the embodiments of this application will be explained first. The nouns and terms involved in the embodiments of this application are subject to the following interpretations.

[0025] 1) Decoding EEG to Text The development of brain-computer interface (BCI) technology has made the direct translation of brain activity signals into natural language text a hot research topic. For individuals with severe language disorders (such as locked-in syndrome), this technology has revolutionary communication assistance potential. Compared to invasive methods requiring implanted electrodes (such as ECoG) or expensive non-invasive methods (such as fMRI and MEG), electroencephalography (EEG) has become the most promising brain-text decoding method in real-world scenarios due to its portability, non-invasiveness, and cost-effectiveness.

[0026] 2) Existing EEG feature extraction and contrastive learning methods Existing EEG-to-text decoding methods typically utilize deep learning (such as end-to-end models or discrete coding mechanisms) to map raw EEG signals into text. To bridge the representational gap between EEG signals and text, some methods introduce contrastive learning strategies, constructing positive and negative sample pairs to narrow the distance between EEG representations and corresponding text features. Furthermore, to address the differences in EEG patterns among individuals, existing methods often employ adaptive strategies tailored to specific subjects or utilize contrastive learning with subject-specific labels to construct a unified representation space.

[0027] Existing EEG-to-text translation systems and methods primarily collect subjects' EEG signals, extract signal features, and then use deep learning or contrastive learning models to map them into natural language text for brain-computer interface (BCI) decoding. However, while EEG signals can reflect language stimuli, they are not limited to semantic information; they are essentially an aggregation of multiple neural activity sources, interspersed with a large number of non-semantic fluctuations (such as the subject's individual physiological characteristics and environmental noise). Due to significant individual differences, different individuals will produce drastically different neural activity patterns when processing the same language stimuli. Therefore, existing methods for feature extraction lack specificity, cannot reliably decouple semantic content from subject-specific noise, and the models heavily rely on explicit subject identity labels for supervised training, resulting in extremely weak generalization ability and a sharp decline in performance when encountering new individuals, thus failing to achieve universal cross-subject EEG translation.

[0028] Furthermore, existing EEG-to-text generation methods are often limited to fine-grained semantic alignment at the lexical level, lacking guidance on macro-level semantic structure alignment at the sentence level. When performing cross-modal translation, most models can only capture scattered lexical information through microscopic local feature mapping, failing to provide multi-granular cross-modal guidance. This results in the generated text, while potentially translating some specific words, exhibiting significant deviations in grasping the overall meaning and contextual coherence of the sentence. It is prone to logical breaks or inaccurate wording, failing to accurately and effectively recreate the subject's true, complete, macro-level intentions.

[0029] In view of this, this application provides an EEG-to-text translation method, electronic device, storage medium, and program product based on decoupled EEG representation learning. It integrates subject-semantic decoupling pre-training, multi-granularity cross-modal contrastive learning alignment, and text generation decoding into a unified framework, facilitating seamless deployment and generalized application. Upon receiving the subject's EEG signal, the system can automatically decouple the original signal into independent subject physiological features and pure semantic content features in real-time, without labels. After extracting the pure semantic features, the decoding system performs multi-granularity semantic constraints, hierarchically aligning the EEG semantic features with the target text at the sentence and lexical levels to establish a high-precision cross-modal mapping relationship. When text generation is required, the system directly decodes the corresponding natural language text from the pre-trained language model based on the finely aligned, physiologically noise-free pure semantic EEG features. This approach avoids signal aliasing interference caused by individual differences and physiological noise, and the use of pure semantic features for text translation is highly targeted, enabling generalized decoding of brain intent across subjects. Meanwhile, multi-level semantic alignment guidance bridges the modal gap, greatly improving the accuracy of translated texts in terms of specific words and logic.

[0030] The EEG-to-text translation method provided in this application relates to the intersection of brain-computer interface technology and natural language processing technology. This EEG-to-text translation method can be applied to a terminal, a server, or software running on either a terminal or a server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, or in-vehicle terminal, but is not limited to these. The server can be configured as an independent physical server, a server cluster consisting of multiple physical servers, or a distributed system. It can also be configured as a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network. The software can be an application implementing the EEG-to-text translation method, but is not limited to the above forms.

[0031] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0032] like Figure 1 As shown, this embodiment provides a brainwave-to-text translation method based on decoupled brainwave representation learning, including a decoupled pre-training stage and a text generation stage: Decoupling the pre-training phase: Step S1: Input the raw EEG signal into the individual encoder and the semantic encoder respectively, and train a semantic encoder to extract pure semantic features from the EEG signal by constraining the features extracted by the two to be independent of each other; wherein, the individual encoder is used to extract static subject features that are not related to semantics, and the semantic encoder is used to extract dynamic semantic features that are related to time. Text generation stage: Step S2: Input the target EEG signal to be translated into the trained semantic encoder to obtain pure EEG semantic features; Step S3: Align the EEG semantic features with the feature space of the natural language text at multiple granularities across modalities to obtain the aligned EEG semantic features; Step S4: Input the aligned EEG semantic features into the pre-trained language model and decode to generate the corresponding natural language text.

[0033] The solutions of the embodiments of this application will be described in detail below with reference to the accompanying drawings and specific application examples.

[0034] like Figure 2 As shown, Figure 2 This is a schematic diagram of an EEG-to-text translation system. The technical solution mainly consists of two core modules: 1) Self-supervised subject-semantic decoupling pre-training module. This module aims to address the problem of individual physiological noise aliasing. Without relying on any subject identity labels, the system constructs a dual-branch independent encoder, decoupling the original complex EEG signal into two independent latent subspaces: one for capturing individual subject features, and the other specifically for extracting sentence content features. The main objective of this stage is to train an encoder capable of accurately extracting semantic information from EEG signals by removing individual interference. This encoder will serve as the core feature extractor for the subsequent text generation module.

[0035] 2) Multi-granularity aligned EEG-to-text generation module. This module aims to address the problem of inaccurate cross-modal semantic alignment. The system directly reuses the pre-trained semantic encoder from the first stage to extract high-fidelity pure semantic EEG features and innovatively introduces a multi-granularity (global sentence-level and local word-level) contrastive learning module to perform hierarchical and precise alignment of EEG semantic features with the feature space of natural language text. Finally, the aligned and calibrated EEG features are decoded through a pre-trained language model to generate context-coherent natural language text end-to-end.

[0036] The EEG-to-text translation method based on decoupled EEG representation learning proposed in this embodiment has the following main workflow: The decoupling of the pre-training steps specifically includes: Step 1: The preprocessed word-level EEG feature sequences are randomly shuffled along the time dimension, completely disrupting their original grammatical and semantic order, but preserving the overall physiological EEG distribution characteristics of the subject. The shuffled sequences are then input into an individual encoder to extract static subject feature vectors that represent the subject's individual identity and do not change over time. Step 2: The word-level EEG feature sequence, maintaining the original word order, is input into the semantic encoder. This encoder extracts a dynamic semantic content feature sequence that changes over time during the reading process by capturing the complex temporal dynamics and contextual dependencies of the EEG signals. Step 3: In order to reconstruct the complete EEG signal in subsequent steps, the extracted static subject feature vector is copied and expanded in the time dimension to make its length consistent with the dynamic semantic content feature sequence; then the two features are spliced ​​in the channel dimension to form a complete fusion feature map that includes the subject's physiology and text semantics. Step 4: Input the spliced ​​and fused feature map into the EEG signal decoder to reconstruct the original EEG input signal. During the training process, which continuously reduces the reconstruction error, the system introduces a mutual information constraint mechanism to forcibly cut off the information overlap between the main features and semantic features, so that the two achieve strict orthogonal separation. After pre-training, a semantic encoder that is immune to individual differences among subjects and dedicated to extracting pure semantics is obtained.

[0037] The steps for generating text from an electroencephalogram (EEG) include: After obtaining a semantic encoder capable of removing individual noise from the subjects, the final translation of EEG signals into natural language text is achieved through cross-modal alignment and generative models.

[0038] Step 1: Process the target natural language text read by the participants and extract the corresponding text feature representations. On the one hand, obtain word-level features corresponding to each specific word in the sentence; on the other hand, extract sentence-level features that cover the overall meaning of the entire sentence through operations such as pooling and aggregation. Step 2: Input the original ordered EEG feature sequence to be translated into the pre-trained semantic encoder. At this point, the encoder can directly filter out physiological and environmental noise in the EEG signal and output a high-fidelity, pure EEG semantic feature sequence; Step 3: To bridge the modal differences between EEG and text, the system performs feature alignment training at two different granularities. At the coarse-grained level, global EEG features are compared and matched with sentence-level features of the text to ensure consistency in general meaning. At the fine-grained level, local EEG segment features are bound and constrained one by one with word-level features of the text to achieve accurate word mapping. This multi-granularity alignment operation acts as a bridge, precisely guiding EEG semantic features to be embedded into the feature space of natural language. Step 4: Input the purified EEG semantic features, guided by multi-granularity alignment, into the pre-trained sequence-to-sequence large language model. The large model uses the received EEG features as cue input, decodes them end-to-end, and generates the corresponding natural language sentences. In actual inference deployment, the EEG signals of new subjects only need to be sequentially processed through acquisition, preprocessing, semantic feature extraction, and decoder generation to output high-precision translated text in real time.

[0039] (1) Subject-Semantic Decoupling Pre-training Module See Figure 3 The subject-semantic decoupling pre-training module, primarily in an unsupervised state, removes physiological noise caused by individual differences from EEG signals, extracting pure textual semantic information, thus laying the foundation for subsequent cross-subject generalized decoding. Assuming the input word-level EEG feature sequence is... ,in Represents the sequence length. This represents the feature dimension. This module specifically includes the following parts: 1.1) Shuffling Module: This module is primarily used to disrupt the semantic coherence of EEG signals, providing the model with training data containing only the subjects' physiological characteristics. It does this by introducing a strong inductive bias in the time dimension. The above is the input EEG feature sequence Perform a random shuffle operation to generate a shuffle sequence. This operation completely disrupted the original temporal dependencies and syntactic and semantic structures, making... Only the statistical distribution characteristics of the subject's neural activity in terms of overall amplitude and energy were preserved, thus physically isolating the interference of semantic information.

[0040] 1.2) Individual Encoder: This module is specifically designed to capture and characterize the subject's unique, time-invariant (reading process) physiological and identity characteristics. It receives EEG sequences scrambled by the shuffling module. As input, firstly, a residual network consisting of one-dimensional convolutional layers and the ReLU activation function is used to extract hierarchical local physiological features; then, global average pooling is applied to reduce the temporal dimension. Compression and elimination are employed to highlight its static properties; finally, a multilayer perceptron is used to map it to the latent variable space, outputting a static subject representation vector. The formula for its encoding process is expressed as follows:

[0041] 1.3) Semantic Encoder: This module is responsible for capturing the semantic content in the EEG signals that changes dynamically over time and is directly related to the read text. It receives the original, unscrambled EEG sequences. To ensure the integrity of the sequence structure, a one-dimensional convolutional layer is first used to extract short-range temporal patterns and local transition features. Then, a multi-head self-attention mechanism is incorporated, assigning different attention weights to features at different locations in the sequence. This effectively overcomes distance limitations and captures long-range contextual language dependencies during the reading process. The module ultimately outputs a dynamic semantic content representation sequence with the same length as the input. Its formula is expressed as:

[0042] As an alternative implementation, the individual encoder and semantic content encoder described above can be implemented using a combination of 1D CNN and Self-Attention, or other sequence processing models, such as Long Short-Term Memory (LSTM), Gated Recurrent Unit (GRU), or pure Transformer architecture to extract latent variable representations.

[0043] 1.4) Decoder: This module represents the decoupled main body. and semantic representation The original EEG signals are reassembled to reconstruct them, ensuring that no crucial original information is lost during feature extraction. Because... It is a static vector, and It is a time series; the decoder first represents the main body. Copy along the time dimension Next, make its shape and Alignment is then performed; subsequently, the two are concatenated along the feature channel dimension. The fused feature map is then passed through a decoding network consisting of self-attention blocks and convolutional layers. Mapping back to the original feature space yields the reconstructed signal. Its formula is expressed as:

[0044] The system calculates and minimizes Compared to real input The reconstruction error between the two drives the front-end encoder to retain sufficient signal reconstruction information.

[0045] 1.5) Variational Mutual Information Regularization: Simple architectural separation cannot guarantee complete orthogonality of information; the network may still seek shortcuts, leading to information leakage (i.e., physiological noise mixed into semantic features, or vice versa). To force the subject representation and semantic representation to be independent, the system introduces a variational upper bound on mutual information as a regularization constraint. Based on the variational information bottleneck theory, KL divergence is used as a penalty term to limit the channel capacity of latent variables. The final decoupling optimization objective function of the system is... as follows:

[0046] The first term is the reconstruction loss to ensure information integrity; the latter two terms are KL divergence constraint terms. and The prior distribution is set; and These are Lagrange multipliers. Through fine-tuning of the weights... and This is equivalent to setting up two information "pressure valves" in the network, strictly regulating and limiting the information allocation between the two encoders, forcing the model to allocate all purely semantic information to... Distribute the subject's physiological information to This achieves complete orthogonality and decoupling of features.

[0047] (2) EEG to text generation module See Figure 4 The EEG-to-text generation module aims to accurately map the pre-decoupled and purified semantic features of EEG signals into the feature space of natural language, and finally decode and generate highly readable target text. Due to the significant modal gap between EEG signals and natural language text, direct mapping can lead to misaligned feature distributions. Therefore, this module guides the embedding of EEG semantic features into the target text space through hierarchical feature extraction and multi-granularity alignment. Assuming a batch contains N samples, each sequence contains M words, and the feature dimension is D, the module specifically includes the following parts: 2.1) Text Sentence Preprocessing: This module extracts feature representations of the target natural language text at different levels, providing cross-modal anchors for EEG features. First, the system inputs the target sentence into the text encoder to extract fine-grained word-level text features corresponding to each word. Subsequently, in order to capture macro-level semantics, the system performs average pooling on word-level features along the sequence dimension, aggregating and extracting coarse-grained sentence-level text features that encompass the global semantics of the entire sentence. This allows for the construction of a complete multi-level text representation space.

[0048] 2.2) Multi-granularity semantic alignment module: This module bridges the modal gap between EEG physiological signals and natural language text, guiding EEG semantic features to accurately align to the correct semantic position in the text to complete text generation. It employs cross-modal contrastive learning technology; at the sentence level, the system integrates global EEG features and global text features belonging to the same sentence. Positive sample pairs are constructed from the features of different sentences, while negative sample pairs are constructed from combinations of features from different sentences. Cosine similarity is calculated, and a sentence-level contrastive loss is built based on the InfoNCE framework. This brings positive samples closer together and pushes negative samples further apart. The similarity calculation formula is as follows:

[0049] At the word level, the system associates local EEG segment features with their precise corresponding text word features. Treating words as positive sample pairs and misplaced words within the same sentence as negative sample pairs, a fine-grained word-level contrastive loss is constructed. This approach strongly binds EEG features to text words at a local level. The purified EEG semantic features, after double-granularity alignment and calibration, are treated as continuous cue signals and input into a pre-trained sequence language model with full parameter fine-tuning capabilities. The large model decodes and generates the final natural language sentence based on the aligned EEG features in an autoregressive manner. Throughout the joint training of the entire generative network, the system's ultimate optimization objective is... Cross-entropy reconstruction loss generated from text It is composed of the multi-granularity contrast loss and the loss function. The joint loss formula is expressed as:

[0050] in, , , These are the hyperparameters used to adjust the weights of each loss component. Through this multi-objective joint optimization, the system ensures both the fluency of the generated text and the accurate semantic reproduction.

[0051] As an alternative implementation, a pre-trained T5 model can be used as the generator, or other pre-trained large language models with encoder-decoder or decoder-only architectures such as BART, GPT series, and LLaMA can be used.

[0052] (3) Advantages and beneficial effects In summary, the method of this application has the following advantages and beneficial effects compared with the prior art: 3.1) This application separates EEG signals through independent individual encoders and semantic encoders, and incorporates a random shuffling sequence design in the individual branch. Combined with mutual information regularization, it achieves complete decoupling of identity information and semantic information in EEG signals, so that the process of decoding EEG signals into text is not affected by individual information.

[0053] Advantages: It solves the problem of existing technologies relying on subject identity labels. Because the features extracted by the semantic encoder are filtered out from the interference caused by individual differences in subjects (non-stationary noise in EEG signals), the system can still maintain extremely high generalization ability and accurate translation performance when facing completely new and unseen individuals (zero-sample cross-subject scenario).

[0054] 3.2) This application innovatively introduces a multi-granularity cross-modal contrastive learning module that combines sentence-level and word-level approaches in cross-modal translation.

[0055] Advantages: Compared to existing techniques that only perform local word-level constraints, this solution not only accurately anchors local details (such as nouns and prepositions) through word-level alignment, but also captures the overall meaning of the sentence. This significantly narrows the huge modal gap between EEG neural representations and natural language text representations, and substantially improves the objective metric scores of the generated text and the accuracy of vocabulary translation.

[0056] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described method. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0057] It is understood that the content of the above method embodiments is applicable to this device embodiment. The specific functions implemented by this device embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0058] Please see Figure 5 , Figure 5 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes: The processor 501 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application. The memory 502 can be implemented as a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 502 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 502 and is called and executed by the processor 501 using the methods described in the embodiments of this application. The input / output interface 503 is used to implement information input and output; The communication interface 504 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.). Bus 505 transmits information between various components of the device (e.g., processor 501, memory 502, input / output interface 503, and communication interface 504); The processor 501, memory 502, input / output interface 503, and communication interface 504 are connected to each other within the device via bus 505.

[0059] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method.

[0060] It is understood that the content of the above method embodiments is applicable to this storage medium embodiment. The specific functions implemented in this storage medium embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0061] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0062] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0063] It is understood that the content of the above method embodiments is applicable to the embodiments of this program product. The specific functions implemented in the embodiments of this program product are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments. The executable computer program code or "code" used to perform the various embodiments can be written in high-level programming languages ​​such as C, C++, Python, Smalltalk, Java, JavaScript, Visual Basic, Structured Query Language (e.g., Transact-SQL), Perl, or in various other programming languages.

[0064] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0065] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0066] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0067] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0068] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0069] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0070] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0071] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0072] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0073] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0074] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A brainwave-to-text translation method based on decoupled brainwave representation learning, characterized in that, The method includes the following steps: Decoupling the pre-training phase: The raw EEG signals are input into an individual encoder and a semantic encoder respectively. By constraining the features extracted by the two to be independent, a semantic encoder for extracting pure semantic features from EEG signals is trained. The individual encoder is used to extract static subject features that are not semantically independent, and the semantic encoder is used to extract dynamic semantic features that are related to time. Text generation stage: The target EEG signal to be translated is input into the trained semantic encoder to obtain pure EEG semantic features; The EEG semantic features are aligned with the feature space of natural language text in a multi-granularity cross-modal manner to obtain the aligned EEG semantic features. The aligned EEG semantic features are input into a pre-trained language model, which is then decoded to generate the corresponding natural language text.

2. The method according to claim 1, characterized in that, The decoupling pre-training phase specifically includes: The original EEG feature sequence was shuffled to obtain a shuffled sequence. The shuffled sequence is input into the individual encoder to extract the static subject features; The EEG feature sequence, which maintains its original temporal order, is input into the semantic encoder to extract the dynamic semantic feature sequence; The static subject features are fused with the dynamic semantic feature sequence to obtain fused features; The fused features are input into the decoder to reconstruct the original EEG signal; Based on reconstruction error and mutual information constraints, a first objective function is constructed, and the individual encoder, semantic encoder and decoder are jointly trained to force the static subject features and the dynamic semantic features to be independent of each other by minimizing the first objective function.

3. The method according to claim 2, characterized in that, The process of fusing the static subject features with the dynamic semantic feature sequence to obtain fused features specifically includes: The static subject features are copied and extended in the time dimension so that their length is consistent with the dynamic semantic feature sequence; The expanded static subject features are concatenated with the dynamic semantic feature sequence along the feature channel dimension to obtain the fused features.

4. The method according to claim 2, characterized in that, The mutual information constraint is achieved through a variational information bottleneck, and the first objective function is expressed as: The first term is the reconstruction loss, and the last two terms are KL divergence constraint terms, which are used to limit the channel capacity of latent variables.

5. The method according to claim 1, characterized in that, The multi-granularity cross-modal alignment includes: Extract word-level and sentence-level text features from the target text; Using sentences as units, the global EEG semantic features are compared and aligned with the sentence-level text features in a coarse-grained manner; Using words as units, local EEG semantic features are compared and aligned with the word-level text features in a fine-grained manner.

6. The method according to claim 5, characterized in that, The training process of the text generation stage is jointly optimized by a second objective function, which is expressed as: in, Cross-entropy reconstruction loss for the generated text, For sentence-level contrast loss, For word-level contrast loss, , , For weight hyperparameters.

7. The method according to any one of claims 1 to 6, characterized in that, The individual encoder comprises a sequentially connected residual network, a global average pooling layer, and a multilayer perceptron; the semantic encoder comprises a sequentially connected one-dimensional convolutional layer and a multi-head self-attention network.

8. The method according to any one of claims 1 to 6, characterized in that, The pre-trained language model is a large language model based on an encoder-decoder architecture or a decoder-only architecture.

9. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method according to any one of claims 1 to 8.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 8.

Images