Model training method and device based on electroencephalogram signals and electroencephalogram signal decoding method and device

Abstract

The invention provides a model training method and device based on electroencephalogram signals and an electroencephalogram signal decoding method and device, and relates to the technical field of artificial intelligence. According to the method, a unified shared potential representation space is constructed, multi-granularity alignment targets are utilized in the space to conduct joint training on electroencephalogram signals and multi-modal data such as vision, audio and text, and therefore the multi-modal data of the electroencephalogram signals and the multi-modal data of the vision, the audio and the text are obtained. The electroencephalogram signals with low signal-to-noise ratio and high individual difference are semantized and standardized successfully, efficient and semantic-level fusion of an electroencephalogram mode and a general multi-mode large model system is achieved, and the model obtained through training can directly map the electroencephalogram signals of any user into semantic-rich vector representation.

Description

EEG signal-based model training methods, EEG signal decoding methods, and devices Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a model training method, a method and apparatus for decoding electroencephalogram (EEG) signals. Background Technology

[0002] Brain-computer interface (BCI) technology, especially non-invasive EEG interfaces, aims to identify user intentions and states by interpreting electroencephalograms (EEGs) generated by brain activity. However, its decoding capabilities are limited by the low signal-to-noise ratio, high individual variability, and non-stationary characteristics of the signals, resulting in limited semantic representation. Meanwhile, large-scale pre-trained multimodal models have demonstrated powerful cross-modal understanding and generalization capabilities in visual and language modalities. However, existing large models are primarily designed for high signal-to-noise ratio stable modalities such as images and text, lacking specialized modeling of EEG signal characteristics and effective mechanisms to align EEG with multimodal semantic spaces. This prevents EEG from being integrated into general multimodal intelligence frameworks, thus hindering the development of BCI towards natural and intelligent interaction. Summary of the Invention

[0003] This invention provides a model training method, a method and apparatus for decoding EEG signals based on EEG signals, to address the shortcomings of existing technologies where EEG signals cannot be effectively integrated into general multimodal large-scale intelligent systems due to modal characteristic mismatch and lack of alignment mechanisms. It achieves efficient, semantic-level fusion of EEG modalities with general multimodal large-scale intelligent systems, enabling the trained model to directly map the EEG signals of any user into semantically rich vector representations.

[0004] This invention provides a model training method based on electroencephalogram (EEG) signals, comprising the following steps: acquiring synchronously recorded EEG signal data and external modality data to form multimodal alignment samples; the external modality data includes at least one of visual data, audio data, and text data; encoding EEG segments in the multimodal alignment samples based on an EEG coding network to obtain EEG feature representations, and mapping the EEG feature representations to a shared latent representation space; encoding external modality segments in the multimodal alignment samples based on an external modality coding network to obtain external modality feature representations, and mapping the external modality feature representations to the shared latent representation space; and training the EEG coding network and the external modality coding network based on the multimodal alignment samples, jointly using at least one alignment target in the shared latent representation space, to obtain a multimodal alignment model.

[0005] According to the present invention, a model training method based on electroencephalogram (EEG) signals includes at least one of the following alignment objectives: using EEG segments and external modal segments within the same time window as positive sample pairs, and using multimodal segments from different time windows as negative sample pairs, and performing comparative learning training based on the positive sample pairs and the negative sample pairs; aligning the time-slice-level features of the EEG signals with the corresponding time-time frame-level features of the external modal; extracting semantic prototype vectors based on the text data or task labels, and aligning the EEG segment representations with semantic prototype vectors of the same semantic category.

[0006] According to the present invention, a model training method based on electroencephalogram (EEG) signals is provided, wherein the EEG coding network and the external modality coding network are trained based on the multimodal alignment samples and in the shared latent representation space by jointly applying at least one alignment target to obtain a multimodal alignment model, the method includes: training the EEG coding network and the external modality coding network on multimodal alignment samples without task labels by applying at least one alignment target in the shared latent representation space to obtain an initialized network model; and on labeled data containing the task labels, jointly optimizing the alignment loss corresponding to at least one alignment target with at least one downstream task loss function to fine-tune the initialized network model to obtain the multimodal alignment model.

[0007] According to the present invention, a model training method based on electroencephalogram (EEG) signals is provided, wherein the EEG coding network includes a time-domain-frequency domain joint feature extraction module and a temporal modeling module; the method of encoding EEG segments in the multimodal aligned samples based on the EEG coding network to obtain EEG feature representations includes: extracting spatial-spectral joint features from the EEG segments based on the time-domain-frequency domain joint feature extraction module; and performing time-dimensional modeling on the spatial-spectral joint features based on the temporal modeling module to obtain the EEG feature representations; wherein the EEG feature representations include segment-level representations and multiple time-segment-level representations.

[0008] According to a model training method based on electroencephalogram (EEG) signals provided by the present invention, the external modality coding network includes a visual coding network, an audio coding network, and a text coding network. The method of encoding external modality segments in the multimodal aligned samples based on the external modality coding network to obtain external modality feature representations includes: encoding visual segments in the external modality segments based on the visual coding network to obtain frame-level visual features and segment-level visual representations; performing time-frequency feature modeling and temporal sequence modeling on audio segments in the external modality segments based on the audio coding network to obtain time-slice-level audio representations and segment-level audio representations; and encoding text segments in the external modality segments based on the text coding network to obtain word-level text representations and sentence-level text representations.

[0009] According to the present invention, a model training method based on electroencephalogram (EEG) signals is provided, which further includes: performing cross-subject alignment and individual adaptation processing on the multimodal alignment model; wherein the cross-subject alignment and individual adaptation processing includes at least one of the following methods: assigning learnable subject embedding vectors to different subjects and fusing the subject embedding vectors with the intermediate features of the EEG coding network; aligning the EEG feature representations of different subjects using domain adversarial training or statistical alignment methods to suppress task-irrelevant individual-specific information; and fine-tuning the parameter-efficient adaptation module for new subjects while freezing the backbone parameters of the model.

[0010] The present invention also provides a method for decoding electroencephalogram (EEG) signals, comprising the following steps: acquiring a target EEG signal to be decoded; inputting the target EEG signal into a multimodal alignment model to obtain a feature representation of the target EEG signal in a shared latent representation space output by the multimodal alignment model; and determining external modal data or control instructions associated with the semantic content of the target EEG signal based on the feature representation of the target EEG signal in the shared latent representation space.

[0011] The present invention also provides a model training device based on electroencephalogram (EEG) signals, comprising the following modules: a data acquisition module, used to acquire synchronously recorded EEG signal data and external modality data to form a multimodal alignment sample; the external modality data includes at least one of visual data, audio data, and text data; a first encoding module, used to encode EEG segments in the multimodal alignment sample based on an EEG encoding network to obtain EEG feature representations, and map the EEG feature representations to a shared latent representation space; a second encoding module, used to encode external modality segments in the multimodal alignment sample based on an external modality encoding network to obtain external modality feature representations, and map the external modality feature representations to the shared latent representation space; and a model training module, used to train the EEG encoding network and the external modality encoding network based on the multimodal alignment sample, in conjunction with at least one alignment target in the shared latent representation space, to obtain a multimodal alignment model.

[0012] The present invention also provides an EEG signal decoding device, comprising the following modules: a target EEG signal acquisition module, used to acquire a target EEG signal to be decoded; a model inference module, used to input the target EEG signal into a multimodal alignment model to obtain the feature representation of the target EEG signal in a shared latent representation space output by the multimodal alignment model; and a decoding module, used to determine external modal data or control instructions associated with the semantic content of the target EEG signal based on the feature representation of the target EEG signal in the shared latent representation space.

[0013] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the model training method based on electroencephalogram signals as described above.

[0014] The present invention provides a model training method, a method and apparatus for decoding EEG signals, and a multimodal alignment sample by acquiring synchronously recorded EEG signal data and external modal data. The external modal data includes at least one of visual data, audio data and text data. Based on an EEG coding network, EEG segments in the multimodal alignment sample are encoded to obtain EEG feature representations, and the EEG feature representations are mapped to a shared latent representation space. Based on an external modal coding network, external modal segments in the multimodal alignment sample are encoded to obtain external modal feature representations, and the external modal feature representations are mapped to the shared latent representation space. Based on the multimodal alignment sample, at least one alignment target is jointly used in the shared latent representation space to train the EEG coding network and the external modal coding network to obtain a multimodal alignment model. This invention constructs a unified shared latent representation space and uses multi-granularity alignment targets within this space to jointly train EEG signals with multimodal data such as vision, audio, and text. This successfully semanticizes and standardizes EEG signals with low signal-to-noise ratio and high individual variability, achieving efficient, semantic-level fusion of EEG modalities with a general multimodal large model system. The trained model can directly map the EEG signals of any user into semantically rich vector representations. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0016] Figure 1 is a flowchart illustrating the model training method based on electroencephalogram (EEG) signals provided by this invention.

[0017] Figure 2 is a flowchart illustrating the EEG signal decoding method provided by the present invention.

[0018] Figure 3 is a flowchart illustrating the large model training and inference method based on EEG multimodal alignment provided by this invention.

[0019] Figure 4 is a schematic diagram of the structure of the model training device based on electroencephalogram (EEG) signals provided by the present invention.

[0020] Figure 5 is a schematic diagram of the structure of the EEG signal decoding device provided by the present invention.

[0021] Figure 6 is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0023] The following describes the model training method, EEG signal decoding method, and apparatus based on EEG signals of the present invention with reference to Figures 1-6.

[0024] Figure 1 is a flowchart of the model training method based on EEG signals provided by the present invention. As shown in Figure 1, the method includes: step 101, acquiring synchronously recorded EEG signal data and external modal data to form a multimodal aligned sample.

[0025] It should be understood that electroencephalogram (EEG) signal data refers to time-series signal data that reflects spontaneous or induced potential changes in the electrophysiological activity of brain neuronal groups, recorded from the scalp surface using non-invasive electrodes.

[0026] External modal data includes at least one of visual data, audio data, and text data, wherein visual data includes image data or video data.

[0027] The study collects EEG signals from subjects when they receive external stimuli, which are at least one of visual, auditory, or textual stimuli that are time-synchronized with the EEG signals. If the collected EEG signal data is not synchronized with the external modality data, a synchronization trigger signal can be used to time-stamp the presentation events of the collected EEG signal data and the external modality data to ensure alignment of the modality data in terms of timestamps. Continuous EEG signals undergo at least one preprocessing step, including filtering, artifact removal, and rereference, and the preprocessed EEG signals are then segmented into multiple EEG segments according to experimental events or preset time windows. Simultaneously, the external modality data is preprocessed to generate external modality segments corresponding to the EEG segments. Each EEG segment is associated with an image frame sequence, audio segment, and text annotation within a corresponding time range to form a multimodal alignment sample.

[0028] Step 102: Based on the EEG coding network, encode the EEG segments in the multimodal aligned samples to obtain EEG feature representations, and map the EEG feature representations to a shared latent representation space.

[0029] It should be understood that EEG coding networks refer to deep learning models (or neural network modules) used to process and understand EEG signals. Shared latent representation space refers to a high-dimensional, continuous vector space learned through multimodal alignment training.

[0030] EEG segments from multimodal aligned samples are input into an EEG coding network to encode the EEG segments, resulting in an EEG feature representation output by the network. Simultaneously, the EEG feature representation is mapped to a shared latent representation space via a projection head at the end of the EEG coding network. This projection head can be a fully connected neural network layer with an output dimension consistent with the dimension of the shared latent representation space.

[0031] Step 103: Based on the external modality coding network, encode the external modality segments in the multimodal aligned samples to obtain external modality feature representations, and map the external modality feature representations to the shared latent representation space.

[0032] It should be understood that external modality coding networks are a series of mature, pre-trained deep neural network models.

[0033] External modal fragments from multimodal aligned samples are input into an external modal encoding network (EMC) to encode the external modal fragments, resulting in an external modal feature representation output by the EMC. Simultaneously, the external modal feature representation is mapped to a shared latent representation space via a projection head at the end of the EMC. This projection head can be a lightweight neural network whose input dimension matches the output dimension of the corresponding EMC, and whose output dimension matches the dimension of the shared latent representation space.

[0034] The external modality feature representation obtained through the external modality coding network can include at least one of the following granularities: a segment-level global feature representation for alignment with the EEG segment-level representation; or a time-slice-level or frame-level local feature representation for alignment with the EEG time-slice-level representation.

[0035] Step 104: Based on the multimodal alignment samples, at least one alignment target is jointly used in the shared latent representation space to train the EEG coding network and the external modality coding network to obtain a multimodal alignment model.

[0036] It should be understood that the multimodal alignment model includes an EEG encoding network for encoding input EEG signals into EEG feature representations; at least one external modality encoding network for encoding corresponding input external modality data into external modality feature representations; and a projection head connected to the EEG encoding network and the external modality encoding network respectively for mapping the EEG feature representations and the external modality feature representations to the same shared latent representation space. After processing by the multimodal alignment model, the similarity between feature representations of data with the same semantic content but different modalities in the shared latent representation space is higher than that of data with different semantic content.

[0037] Alignment objectives refer to mathematical functions or constraints defined and optimized during model training to promote specific semantic relationships between different modal data in a shared latent representation space.

[0038] In one embodiment, the alignment objective may include at least one of a fragment-level cross-modal contrastive learning objective, a time-slice-level alignment objective, and a semantic-level prototype alignment objective. The fragment-level cross-modal contrastive learning objective can be achieved using the InfoNCE loss function, the time-slice-level alignment objective can be achieved using mean squared error loss or cosine similarity loss, and the semantic-level prototype alignment objective can be achieved using prototype contrast loss or center loss.

[0039] Based on multimodal alignment samples, at least one alignment objective is constructed and jointly optimized in a shared latent representation space to train EEG coding networks and external modality coding networks, so that the representations of the same semantic content between different modalities are close to each other in the shared latent representation space, that is, to narrow the distance between EEG feature representations and external modality feature representations that occur synchronously in the same multimodal alignment sample.

[0040] The model training method based on EEG signals provided in this invention involves acquiring synchronously recorded EEG signal data and external modality data to form multimodal alignment samples. The external modality data includes at least one of visual data, audio data, and text data. Based on an EEG coding network, EEG segments in the multimodal alignment samples are encoded to obtain EEG feature representations, which are then mapped to a shared latent representation space. Based on an external modality coding network, external modality segments in the multimodal alignment samples are encoded to obtain external modality feature representations, which are then mapped to the shared latent representation space. Based on the multimodal alignment samples, at least one alignment target is jointly used in the shared latent representation space to train the EEG coding network and the external modality coding network, resulting in a multimodal alignment model. This invention constructs a unified shared latent representation space and uses multi-granularity alignment targets within this space to jointly train EEG signals with multimodal data such as vision, audio, and text. This successfully semanticizes and standardizes EEG signals with low signal-to-noise ratio and high individual variability, achieving efficient, semantic-level fusion of EEG modalities with a general multimodal large model system. The trained model can directly map the EEG signals of any user into semantically rich vector representations.

[0041] Based on the above embodiments, the alignment target includes at least one of the following: taking EEG segments and external modal segments within the same time window as positive sample pairs, taking multimodal segments from different time windows as negative sample pairs, and performing comparative learning training based on the positive sample pairs and the negative sample pairs; aligning the time-slice-level features of the EEG signal with the corresponding time-level features of the external modal; extracting semantic prototype vectors based on the text data or task labels, and aligning the EEG segment representation with semantic prototype vectors of the same semantic category.

[0042] It should be understood that an EEG segment is a multi-channel temporal data block segmented from continuously recorded EEG signals at fixed time lengths (e.g., 2 seconds) or specific experimental events (e.g., image presentation). An external modal segment is a segment of external stimulus data that is temporally synchronized with an EEG segment, such as an image, a 2-second audio clip, or a word / sentence presented on a screen. A multimodal segment is a data unit composed of an EEG segment and one or more corresponding external modal segments.

[0043] Temporal slice features of EEG signals are feature vectors extracted by an EEG coding network after a segment of EEG is divided into finer-grained segments (e.g., every 100 milliseconds), representing the neural state of that short-term window. External modality frame-level features are feature vectors extracted by an external modality coding network after frame sampling of an external modality segment (e.g., video or audio), representing the content of a single frame (e.g., image frame, audio short-time Fourier transform frame).

[0044] Task labels are discrete or continuous identifiers used to indicate their category or target value. Semantic prototype vectors are generalized central feature vectors that represent a semantic category in a shared latent representation space.

[0045] When the alignment objective is fragment-level cross-modal contrastive learning, in the shared latent representation space, EEG fragments and external modal fragments within the same time window are considered positive sample pairs, while multimodal fragments from different time windows are considered negative sample pairs. Alternatively, EEG feature representations and external modal feature representations within the same multimodal alignment sample are defined as positive sample pairs, and feature representations from different multimodal alignment samples are defined as negative sample pairs. A contrastive loss function is employed to narrow the distance between positive sample pairs and widen the distance between negative sample pairs.

[0046] When the alignment target includes a time-slice level alignment target, the time-slice level EEG feature representation output by the EEG coding network is aligned with the visual frame feature representation or audio frame feature representation output by the external modality coding network at the synchronization time point, and optimized using local contrast loss or alignment regression loss.

[0047] When the alignment target includes a semantic-level prototype alignment target, a semantic prototype vector is constructed based on the text annotation or task label of the multimodal alignment sample. During training, the EEG feature representation is aligned with the prototype vector of its corresponding semantic category, and prototype contrast loss is used to enhance semantic discriminability.

[0048] In one embodiment, the alignment objective further includes a cross-modal reconstruction objective, which is achieved by reconstructing the corresponding external modal features using EEG feature representations, or by reconstructing the corresponding EEG features using external modal features, and optimizing using a reconstruction loss function.

[0049] By simultaneously aligning at the fragment level, time slice level, and semantic level, this invention enables a more refined capture of the correspondence between brain activity and external stimuli, thereby improving the modeling ability of complex cognitive processes and the downstream decoding accuracy.

[0050] Based on the above embodiments, the step of training the EEG coding network and the external modality coding network based on the multimodal alignment samples, in conjunction with at least one alignment target in the shared latent representation space, to obtain a multimodal alignment model includes: training the EEG coding network and the external modality coding network on multimodal alignment samples without task labels by applying at least one alignment target in the shared latent representation space to obtain an initialized network model; and on labeled data containing the task labels, jointly optimizing the alignment loss corresponding to at least one alignment target with at least one downstream task loss function to fine-tune the initialized network model to obtain the multimodal alignment model.

[0051] It should be understood that multimodal aligned samples without task labels refer to data samples in which only EEG signals and corresponding raw external stimulus data (such as images, sounds, and text) are synchronously recorded during data acquisition, without pre-assigning specific task-related classification labels or regression values ​​to these data.

[0052] Labeled data with task labels refers to datasets that, in addition to containing multimodal aligned samples, also include manually labeled or explicitly defined labels related to a specific application task.

[0053] Downstream task loss function refers to a mathematical function used during the model fine-tuning stage to measure the model's performance on a specific application task and drive the optimization of model parameters accordingly. The downstream task loss function can be at least one of EEG classification loss, regression loss, retrieval loss, or generation loss.

[0054] The model training process includes the following two stages: (1) Self-supervised pre-training stage, which is carried out on large-scale multimodal alignment samples without explicit task labels, and trains the EEG coding network and the external modality coding network with at least one alignment target. Specifically, on large-scale multimodal alignment samples without task labels, the EEG coding network and the external modality coding network are trained by applying at least one alignment target in a shared latent representation space to obtain an initialized network model.

[0055] (2) Multi-task joint training stage: This stage is carried out on labeled datasets containing task labels, where the alignment loss corresponding to at least one alignment target is jointly optimized with at least one downstream task loss function. Specifically, on labeled data containing task labels, the initialized network model is fine-tuned by jointly optimizing the alignment loss corresponding to at least one alignment target and one or more preset downstream task loss functions in a shared latent representation space, and finally a multimodal alignment model is obtained.

[0056] When training with multiple loss functions, optimization can be achieved by using a task weight adaptive adjustment strategy or a network parameter gradual unfreezing strategy.

[0057] This invention employs a self-supervised comparison and reconstruction task on large-scale unlabeled EEG-multimodal data, combined with a multi-task training strategy involving multiple downstream EEG tasks on small-scale labeled data, to obtain a large model with general alignment capabilities and task performance. Self-supervised pre-training on large-scale unlabeled EEG-multimodal synchronous data enhances the model's representation and alignment capabilities, providing a stronger generalization foundation for downstream tasks.

[0058] Based on the above embodiments, the step of encoding EEG segments in the multimodal aligned samples based on the EEG coding network to obtain EEG feature representations includes: extracting spatial-spectral joint features from the EEG segments based on the time-domain-frequency domain joint feature extraction module; and performing time-dimensional modeling on the spatial-spectral joint features based on the temporal modeling module to obtain the EEG feature representations; wherein, the EEG feature representations include segment-level representations and multiple time-time-level representations.

[0059] It should be understood that spatial-spectral joint features refer to a high-dimensional representation that can simultaneously encode the spatial distribution (inter-channel relationship) and frequency energy distribution (inter-band relationship) of EEG signals on the scalp.

[0060] The EEG coding network includes: a time-frequency joint feature extraction module, used to extract spatial-spectral joint features from EEG segments; and a temporal modeling module, used to model the spatial-spectral joint features in the time dimension to output EEG feature representations.

[0061] In one embodiment, the time-domain-frequency domain joint feature extraction module processes multi-channel EEG segments in at least one of the following ways: multi-scale convolution, for example, using convolution kernels of different sizes to capture EEG patterns under different temporal receptive fields; channel attention modeling, for example, adaptively assigning weights to different EEG channels to enhance the contribution of signals from key brain regions; and band attention modeling, for example, adaptively assigning weights to different neural oscillation bands.

[0062] In one embodiment, the temporal modeling module employs at least one of the following architectures: a Transformer architecture that utilizes a self-attention mechanism to capture long-range dependencies in EEG time series; a temporal convolutional network architecture that utilizes causal convolution and dilated convolution to efficiently model temporal information; or a recurrent neural network architecture or a variant thereof for progressive processing and memorization of EEG sequences.

[0063] The EEG feature representation output by the EEG coding network includes a segment-level representation, which is a global feature aggregation of the entire EEG segment; and multiple time-slice-level representations, which are local features at various time points within the EEG segment.

[0064] The EEG coding network also includes a projection head, which is one or more fully connected layers, used to linearly or non-linearly map the EEG feature representations output by the temporal modeling module to a shared latent representation space.

[0065] The embodiments of the present invention achieve multi-level deep representation of EEG signals from local to global and from static to dynamic by cascading the time-domain-frequency domain joint feature extraction module and the time-series modeling module.

[0066] Based on the above embodiments, the step of encoding external modal segments in the multimodal aligned samples using an external modal coding network to obtain external modal feature representations includes: encoding visual segments in the external modal segments using the visual coding network to obtain frame-level visual features and segment-level visual representations; performing time-frequency feature modeling and temporal sequence modeling on audio segments in the external modal segments using the audio coding network to obtain time-slice-level audio representations and segment-level audio representations; and encoding text segments in the external modal segments using the text coding network to obtain word-level text representations and sentence-level text representations.

[0067] It should be understood that external modality coding networks include at least one of the following modality-specific coding networks: visual coding networks for processing image / video data, audio coding networks for processing audio data, and text coding networks for processing text data.

[0068] When the external modal data is image / video data, a visual coding network is used to encode the image or video frames to generate an external modal feature representation that includes frame-level features and / or fragment-level visual representations. The visual coding network can be a pre-trained model built on convolutional neural networks, visual Transformers, or a combination thereof.

[0069] When the external modal data is audio data, an audio coding network is used to perform time-frequency feature modeling and temporal modeling on the audio segments to generate external modal feature representations that include time-slice-level representations and / or segment-level audio representations. The time-frequency feature modeling is based on spectrograms or Mel-frequency cepstral coefficients extracted from the audio data.

[0070] When the external modal data is text data, a text encoding network is used to encode the tokenized text description to generate an external modal feature representation that includes word / sub-word level representations and / or sentence-level text representations. The text encoding network can be a pre-trained language model.

[0071] This invention provides precise anchor points for the alignment of EEG signals by designing and outputting multi-level granular (frame-level / segment-level, time-slice-level / segment-level, word-level / sentence-level) feature representations for three external modalities: visual, audio, and text. The segment-level global representation can be used to align the user's overall intention or stable cognitive state, while the fine-grained frame-level, time-slice-level, and word-level representations can accurately correspond to the instantaneous changes in the user's perception or cognition process, thereby enhancing the accuracy and robustness of cross-modal alignment.

[0072] Based on the above embodiments, the model training method based on EEG signals further includes: performing cross-subject alignment and individual adaptation processing on the multimodal alignment model; wherein the cross-subject alignment and individual adaptation processing includes at least one of the following methods: assigning learnable subject embedding vectors to different subjects and fusing the subject embedding vectors with the intermediate features of the EEG coding network; using domain adversarial training or statistical alignment methods to align the EEG feature representations of different subjects to suppress task-irrelevant individual-specific information; and for new subjects, fine-tuning the parameter-efficient adaptation module under the condition of freezing the model backbone parameters.

[0073] It should be understood that cross-subject alignment and individual adaptation processing refers to a series of technical means used when training and / or applying multimodal alignment models, which aim to reduce non-task-related differences in EEG signals between different users (subjects) and achieve rapid personalized adaptation for new users.

[0074] Subject embedding vectors are learnable low-dimensional real vectors that are tied to a specific subject (user). For example, a subject embedding vector is a learnable low-dimensional real vector whose dimension is independent of the data dimension of the EEG signal; the subject embedding vector is randomly initialized at the start of training and is optimized along with the parameters of the EEG coding network during training.

[0075] Domain adversarial training involves introducing a domain discriminator network and employing gradient inversion to force the main feature extraction network (EEG coding network) to generate feature representations that are difficult to distinguish from which user they originate.

[0076] Statistical alignment is a method that uses mathematical transformations to directly standardize the statistical distributions (such as mean and variance) of EEG characteristics of different users, making their distributions converge.

[0077] A parameter-efficient adaptation module is a fine-tuning technique component that introduces only a very small number of additional trainable parameters and inserts them into specific locations in a pre-trained large model, thereby enabling rapid and lightweight adjustments to the model's behavior.

[0078] In one embodiment, fusing the subject embedding vector with the intermediate features of the EEG coding network refers to performing the fusion operation at at least one of the following locations: concatenating or adding it with preprocessed EEG segments at the input layer of the EEG coding network; fusing it in an intermediate layer after the time-frequency joint feature extraction module and before the temporal modeling module; or fusing it between multiple sub-layers of the temporal modeling module. The fusion operation may include at least one of the following methods: copying and expanding the subject embedding vector and concatenating it with the intermediate features along the channel dimension; projecting the subject embedding vector through a fully connected layer to the same dimension as the intermediate features and then adding it element-wise; or performing an affine transformation on the intermediate features using a scaling factor and bias term generated conditioned on the subject embedding vector.

[0079] In one embodiment, adversarial training in the domain includes: introducing a domain discriminator network whose input is an EEG feature representation or a vector in a shared latent representation space, and whose task is to determine which subject the input features originate from; during training, jointly optimizing the EEG encoding network and the domain discriminator, wherein the goal of the EEG encoding network is to generate features that can deceive the domain discriminator, making it unable to accurately determine the subject's origin. It should be understood that joint optimization can be achieved through a gradient inversion layer. For example, during forward propagation, the gradient inversion layer acts as an identity mapping, passing the features to the domain discriminator; during backpropagation, the gradient inversion layer multiplies the loss gradient from the domain discriminator by a negative coefficient before backpropagating it to the EEG encoding network, thereby achieving adversarial optimization of the EEG encoding network.

[0080] In one embodiment, the statistical alignment method includes at least one of the following algorithms: maximum mean difference minimization, used to reduce the MMD distance between the EEG feature distributions of different subjects; correlation alignment, aligning the covariance matrices of the EEG features of different subjects through whitening transformation; and instance normalization, performing individual normalization on the EEG features of each subject to eliminate the differences in the first-order statistics of the distribution. The statistical alignment method can be implemented in at least one of the following locations: within the EEG coding network, aligning intermediate layer features; and in the shared latent representation space, aligning the mapped EEG feature representations.

[0081] This application's embodiments achieve feature alignment between different subjects through subject embedding, domain adversarial alignment, and efficient parameter adaptation layers. It also supports training and inference methods for rapid individualized adaptation of new subjects with limited data, effectively mitigating distribution shifts caused by individual differences and enabling rapid adaptation to new users and tasks.

[0082] Figure 2 is a flowchart of the EEG signal decoding method provided by the present invention. As shown in Figure 2, the method includes: step 201, acquiring the target EEG signal to be decoded.

[0083] The target EEG signal to be decoded refers to the raw EEG signal newly collected from a specific target user during the reasoning (application) stage, which has not yet been analyzed by the model. The semantic information contained within it is unknown and needs to be interpreted by the system.

[0084] EEG signal streams are acquired in real time from the target subjects for online decoding.

[0085] Step 202: Input the target EEG signal into the multimodal alignment model to obtain the feature representation of the target EEG signal in the shared latent representation space output by the multimodal alignment model.

[0086] It should be understood that the feature representation output by the multimodal alignment model refers to the vector obtained in the shared latent representation space after being processed by the EEG coding network and mapped via its projection head.

[0087] The target EEG signal is input into the EEG coding network in the multimodal alignment model. In the time-frequency joint feature extraction module of the EEG coding network, the signal segments are processed through multi-scale convolution and attention mechanisms to extract spatial-spectral joint features. These spatial-spectral joint features are then input into the temporal modeling module of the EEG coding network, where self-attention mechanisms or temporal convolution capture their temporal dependencies, resulting in a high-level temporal feature representation. This high-level temporal feature representation is then input into the projection head of the EEG coding network, mapping it to feature representations in a shared latent representation space. After obtaining the feature representations, vector normalization is performed, and the feature representations are associated with and stored with the corresponding target EEG signal segments for subsequent model updates or data analysis.

[0088] In one embodiment, during the processing of the target EEG signal by the EEG coding network, an individual adaptation mechanism corresponding to the target subject is activated. The individual adaptation mechanism includes at least one of the following: fusing the loaded subject embedding vector with the intermediate features of the EEG coding network; and enabling the parameter efficient adaptation module in the EEG coding network to take effect.

[0089] Step 203: Based on the feature representation of the target EEG signal in the shared latent representation space, determine the external modal data or control instructions associated with the semantic content of the target EEG signal.

[0090] Control commands refer to machine-readable commands or command sequences generated or triggered based on decoded EEG semantic features, which can directly drive external devices, software applications, or virtual environments to perform specific actions.

[0091] Calculate the similarity between the feature representation of the target EEG signal and each feature representation in a pre-stored external modality feature representation library; identify the external modality data corresponding to the feature representation with the highest similarity as the associated data and output it.

[0092] The feature representation of the target EEG signal is used as a conditional input to a pre-trained conditional generation model. The model then receives control command sequences, text descriptions, image data, or audio data generated by the conditional generation model based on these conditions, and outputs these as well. The conditional generation model can be at least one of the following: a large language model for generating text descriptions or command sequences; a text-to-image model for generating images; or a symbolic instruction planner for generating machine-executable control code.

[0093] The EEG signal decoding method provided in this invention involves: acquiring the target EEG signal to be decoded; inputting the target EEG signal into a multimodal alignment model to obtain the feature representation of the target EEG signal in a shared latent representation space output by the multimodal alignment model; and determining external modal data or control commands associated with the semantic content of the target EEG signal based on the feature representation of the target EEG signal in the shared latent representation space. This invention, through a trained multimodal alignment model, achieves semantic decoding of EEG signals from any user, upgrading traditional EEG decoding, which requires complex calibration and is limited to a finite set of commands, into a universal, flexible, and highly semantic neural interface. Users do not need to predefine psychological tasks; the system can map their EEG signals into semantically rich universal feature vectors in real time, and directly retrieve related images, sounds, and texts, or generate precise control commands based on these vectors. This eliminates dependence on predefined paradigms and enhances the naturalness and richness of brain-computer interaction.

[0094] To further explain the model training method and EEG signal decoding method proposed in this invention, please refer to the following embodiments.

[0095] Referring to Figure 3, which is a flowchart illustrating the large-scale model training and inference method based on EEG multimodal alignment provided by this invention, the method includes: Step 1, Multimodal Data Acquisition and Preprocessing: Acquiring synchronous records containing multimodal data of EEG signals and external stimuli, wherein the external stimuli multimodal data includes at least image / video modality, audio modality, and text description modality. Preprocessing of the EEG signals includes filtering, artifact removal, rereference, and time-frequency transformation. The continuous EEG sequence is segmented into several EEG segments according to experimental events or time windows and normalized. Preprocessing of image / video data includes size normalization and frame sampling. Preprocessing of audio data includes sampling rate unification and feature extraction (such as spectrograms, Mel-Frequency Cepstral Coefficients (MFCC)). Preprocessing of text data includes word segmentation, sub-wording, or tokenization. Each EEG segment is associated with the corresponding time range of image frame sequences, audio segments, and text annotations to form multimodal alignment samples.

[0096] Step 2, EEG Coding Network Construction: Construct an encoding network for EEG signals, including a time-domain-frequency domain joint feature extraction module and a temporal modeling module; the time-domain-frequency domain joint feature extraction module performs multi-scale convolution, channel attention and frequency band attention modeling on multi-channel EEG segments to extract spatial-spectral joint features; the temporal modeling module, based on Transformer, temporal convolution or a combination thereof, models EEG segments in the time dimension and outputs a set of EEG temporal feature vectors, including segment-level representations and several time-slice-level representations; a projection head is set at the end of the EEG coding network to map the EEG features to a multimodal shared latent representation space.

[0097] Step 3: Construction of External Modality Encoding Networks: A pre-trained or de novo trained visual encoding network is used to encode image / video frames to obtain frame-level features and overall segment-level visual representations; a pre-trained or de novo trained audio encoding network is used to perform time-frequency feature modeling and temporal modeling on audio segments to obtain audio segment-level representations and time-slice-level representations; a pre-trained text encoding network is used to encode text descriptions to obtain sentence-level text representations and optional word / sub-word-level representations; modality-specific projection heads are set at the ends of each external modality encoding network to map visual, audio, and text features to the same shared latent representation space as the EEG modality.

[0098] Step 4, Multi-granularity alignment target construction: Constructing a segment-level cross-modal contrastive learning target: EEG segments within the same time window are used as positive samples along with corresponding visual segments, audio segments, and text descriptions, while multimodal segments from other time windows or other experiments are used as negative samples. Contrast loss is used to narrow the distance between positive samples and widen the distance between negative samples. Constructing a time-slice-level alignment target: EEG time slices are aligned with the features of corresponding visual or audio frames. Fine-grained temporal alignment is achieved through local contrast loss or alignment regression loss. Constructing a semantic-level prototype alignment target: Semantic prototype vectors are extracted based on text descriptions or task labels. EEG segment representations are aligned with prototype vectors of the same semantic category. Semantic discriminativeness is enhanced through prototype contrast loss. Optionally, a cross-modal reconstruction target is introduced: Visual / audio / text features are reconstructed from EEG representations, or EEG features are predicted from visual / audio / text representations. Reconstruction loss is used to strengthen the capture of mutual information between different modalities.

[0099] Step 5, Cross-Subject Alignment and Individual Adaptation Mechanism: Assign a learnable subject embedding vector to each subject and fuse this embedding with the intermediate features of the EEG coding network to explicitly model individual differences; use domain adversarial training or statistical alignment methods to align the EEG features of different subjects to a shared latent space, suppressing task-independent individual distribution differences; for new subjects, introduce parameter-efficient adaptation layers or cue vectors, and with the main multimodal large model frozen, only fine-tune a small number of adaptation parameters to reduce individual calibration costs.

[0100] Step 6: Self-supervised pre-training and multi-task joint training: On large-scale unlabeled EEG and its synchronous multimodal data, pre-training is performed using the aforementioned self-supervised tasks such as multi-granularity comparison and reconstruction to obtain a large multimodal model with general alignment capabilities; On small-scale labeled data with task labels, downstream task losses such as EEG classification, regression, retrieval, and generation are introduced and combined with the self-supervised alignment loss for multi-task joint training; Through adaptive adjustment of task weights or a gradual unfreezing strategy, the performance and generalization ability of downstream tasks are improved while ensuring alignment capabilities.

[0101] Step 7, EEG Multimodal Decoding and Application in the Reasoning Stage: In the reasoning stage, only online-acquired EEG signals are received and mapped to a multimodal shared latent space via an EEG coding network and a projection head. Based on similarity metrics in the shared space, the most similar text, visual, or audio representations are retrieved to achieve semantic decoding without explicit labels. Using pre-trained multimodal generation or planning modules, the decoded semantic representations are transformed into control commands, virtual environment interaction behaviors, content generation requests, etc., to realize various brain-computer interaction applications. Optionally, the individual adaptation layer is continuously updated in small steps based on online-acquired data to further improve individual robustness during long-term use.

[0102] This invention constructs a multimodal alignment large model training and inference method for EEG modalities, realizing multi-granular alignment of EEG signals with external modalities such as vision, audio, and text in a unified latent space, thereby improving the semantic expression ability and cross-scene generalization ability of EEG decoding.

[0103] The model training device based on EEG signals provided by the present invention will be described below. The model training device based on EEG signals described below can be referred to in correspondence with the model training method based on EEG signals described above.

[0104] Referring to Figure 4, the model training device based on EEG signals provided by the present invention includes: a data acquisition module 401, used to acquire synchronously recorded EEG signal data and external modality data to form a multimodal alignment sample; the external modality data includes at least one of visual data, audio data, and text data; a first encoding module 402, used to encode EEG segments in the multimodal alignment sample based on an EEG encoding network to obtain EEG feature representations, and map the EEG feature representations to a shared latent representation space; a second encoding module 403, used to encode external modality segments in the multimodal alignment sample based on an external modality encoding network to obtain external modality feature representations, and map the external modality feature representations to the shared latent representation space; and a model training module 404, used to train the EEG encoding network and the external modality encoding network based on the multimodal alignment sample, in conjunction with at least one alignment target in the shared latent representation space, to obtain a multimodal alignment model.

[0105] The model training device based on EEG signals provided in this invention constructs a unified shared latent representation space and uses multi-granularity alignment targets within this space to jointly train EEG signals with multimodal data such as vision, audio, and text. This successfully semanticizes and standardizes EEG signals with low signal-to-noise ratio and high individual variability, achieving efficient and semantic-level fusion of EEG modalities with a general multimodal large model system. This enables the trained model to directly map the EEG signals of any user into semantically rich vector representations.

[0106] In one embodiment, the alignment target includes at least one of the following: using EEG segments and external modal segments within the same time window as positive sample pairs, and multimodal segments from different time windows as negative sample pairs, and performing comparative learning training based on the positive sample pairs and the negative sample pairs; aligning the time-slice-level features of the EEG signal with the corresponding time-time external modal frame-level features; extracting semantic prototype vectors based on the text data or task labels, and aligning the EEG segment representation with semantic prototype vectors of the same semantic category.

[0107] In one embodiment, the model training module 404 is further configured to: train the EEG coding network and the external modality coding network on multimodal alignment samples that do not contain task labels by applying at least one alignment target in the shared latent representation space to obtain an initialized network model; and fine-tune the initialized network model on labeled data that contains the task labels by jointly optimizing the alignment loss corresponding to at least one alignment target with at least one downstream task loss function to fine-tune the initial network model to obtain the multimodal alignment model.

[0108] In one embodiment, the first encoding module 402 is further configured to: extract spatial-spectral joint features from the EEG segment based on the time-domain-frequency domain joint feature extraction module; and perform time-dimensional modeling on the spatial-spectral joint features based on the time-series modeling module to obtain the EEG feature representation; wherein the EEG feature representation includes segment-level representation and multiple time-segment-level representations.

[0109] In one embodiment, the second encoding module 403 is further configured to: encode visual segments in the external modal segment based on the visual encoding network to obtain frame-level visual features and segment-level visual representations; perform time-frequency feature modeling and temporal sequence modeling on audio segments in the external modal segment based on the audio encoding network to obtain time-slice-level audio representations and segment-level audio representations; and encode text segments in the external modal segment based on the text encoding network to obtain word-level text representations and sentence-level text representations.

[0110] In one embodiment, the model training module 404 is further configured to: perform cross-subject alignment and individual adaptation processing on the multimodal alignment model; wherein the cross-subject alignment and individual adaptation processing includes at least one of the following methods: assigning learnable subject embedding vectors to different subjects and fusing the subject embedding vectors with the intermediate features of the EEG coding network; aligning the EEG feature representations of different subjects using domain adversarial training or statistical alignment methods to suppress task-irrelevant individual-specific information; and fine-tuning the parameter-efficient adaptation module for new subjects while freezing the backbone parameters of the model.

[0111] The EEG signal decoding device provided by the present invention is described below. The EEG signal decoding device described below can be referred to in correspondence with the EEG signal decoding method described above.

[0112] Referring to Figure 5, the EEG signal decoding device provided by the present invention includes: a target EEG signal acquisition module 501, used to acquire a target EEG signal to be decoded; a model inference module 502, used to input the target EEG signal into a multimodal alignment model to obtain the feature representation of the target EEG signal in a shared latent representation space output by the multimodal alignment model; and a decoding module 503, used to determine external modal data or control instructions associated with the semantic content of the target EEG signal based on the feature representation of the target EEG signal in the shared latent representation space.

[0113] The EEG signal decoding device provided in this invention acquires the target EEG signal to be decoded; inputs the target EEG signal into a multimodal alignment model to obtain the feature representation of the target EEG signal in a shared latent representation space output by the multimodal alignment model; and determines external modal data or control commands associated with the semantic content of the target EEG signal based on the feature representation of the target EEG signal in the shared latent representation space. This invention, through a trained multimodal alignment model, achieves semantic decoding of EEG signals from any user, upgrading traditional EEG decoding, which requires complex calibration and is limited to a finite set of commands, into a universal, flexible, and highly semantic neural interface. Users do not need to predefine psychological tasks; the system can map their EEG signals into semantically rich universal feature vectors in real time, and directly retrieve related images, sounds, and texts, or generate precise control commands based on these vectors. This eliminates dependence on predefined paradigms and improves the naturalness and richness of brain-computer interaction.

[0114] Figure 6 illustrates a schematic diagram of the physical structure of an electronic device. As shown in Figure 6, the electronic device may include: a processor 610, a communication interface 620, a memory 630, and a communication bus 640. The processor 610, the communication interface 620, and the memory 630 communicate with each other through the communication bus 640. The processor 610 can call logic instructions in the memory 630 to execute a model training method based on EEG signals. This method includes: acquiring synchronously recorded EEG signal data and external modality data to form multimodal alignment samples; the external modality data includes at least one of visual data, audio data, and text data; encoding EEG segments in the multimodal alignment samples based on an EEG coding network to obtain EEG feature representations, and mapping the EEG feature representations to a shared latent representation space; encoding external modality segments in the multimodal alignment samples based on an external modality coding network to obtain external modality feature representations, and mapping the external modality feature representations to the shared latent representation space; and training the EEG coding network and the external modality coding network based on the multimodal alignment samples, jointly using at least one alignment target in the shared latent representation space, to obtain a multimodal alignment model.

[0115] Alternatively, the processor 610 may invoke logical instructions in the memory 630 to execute an EEG signal decoding method, the method comprising: acquiring a target EEG signal to be decoded; inputting the target EEG signal into a multimodal alignment model to obtain a feature representation of the target EEG signal in a shared latent representation space output by the multimodal alignment model; and determining external modal data or control instructions associated with the semantic content of the target EEG signal based on the feature representation of the target EEG signal in the shared latent representation space.

[0116] Furthermore, the logical instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0117] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute the model training method based on EEG signals provided by the above methods. The method includes: acquiring synchronously recorded EEG signal data and external modality data to form a multimodal alignment sample; the external modality data includes at least one of visual data, audio data, and text data; encoding EEG segments in the multimodal alignment sample based on an EEG coding network to obtain EEG feature representations, and mapping the EEG feature representations to a shared latent representation space; encoding external modality segments in the multimodal alignment sample based on an external modality coding network to obtain external modality feature representations, and mapping the external modality feature representations to the shared latent representation space; and training the EEG coding network and the external modality coding network based on the multimodal alignment sample, in conjunction with at least one alignment target in the shared latent representation space, to obtain a multimodal alignment model.

[0118] Alternatively, when the computer program is executed by a processor, the computer can execute the EEG signal decoding method provided by the above methods, the method comprising: acquiring a target EEG signal to be decoded; inputting the target EEG signal into a multimodal alignment model to obtain a feature representation of the target EEG signal in a shared latent representation space output by the multimodal alignment model; and determining external modal data or control instructions associated with the semantic content of the target EEG signal based on the feature representation of the target EEG signal in the shared latent representation space.

[0119] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the model training method based on EEG signals provided by the above methods. This method includes: acquiring synchronously recorded EEG signal data and external modality data to form a multimodal alignment sample; the external modality data includes at least one of visual data, audio data, and text data; encoding EEG segments in the multimodal alignment sample based on an EEG coding network to obtain EEG feature representations, and mapping the EEG feature representations to a shared latent representation space; encoding external modality segments in the multimodal alignment sample based on an external modality coding network to obtain external modality feature representations, and mapping the external modality feature representations to the shared latent representation space; and training the EEG coding network and the external modality coding network based on the multimodal alignment sample, jointly using at least one alignment target in the shared latent representation space, to obtain a multimodal alignment model.

[0120] Alternatively, when the computer program is executed by a processor, it implements an EEG signal decoding method provided by the methods described above, the method comprising: acquiring a target EEG signal to be decoded; inputting the target EEG signal into a multimodal alignment model to obtain a feature representation of the target EEG signal in a shared latent representation space output by the multimodal alignment model; and determining external modal data or control instructions associated with the semantic content of the target EEG signal based on the feature representation of the target EEG signal in the shared latent representation space.

[0121] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0122] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0123] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A model training method based on electroencephalogram (EEG) signals, characterized in that, include: Acquire synchronously recorded EEG signal data and external modal data to form a multimodal aligned sample; the external modal data includes at least one of visual data, audio data and text data; based on the EEG coding network, encode the EEG segments in the multimodal aligned sample to obtain EEG feature representations, and map the EEG feature representations to a shared latent representation space; Based on the external modality coding network, the external modality segments in the multimodal aligned samples are encoded to obtain external modality feature representations, and the external modality feature representations are mapped to the shared latent representation space; Based on the multimodal alignment samples, at least one alignment target is jointly used in the shared latent representation space to train the EEG coding network and the external modality coding network to obtain a multimodal alignment model.

2. The model training method based on electroencephalogram (EEG) signals according to claim 1, characterized in that, The alignment target includes at least one of the following: using EEG segments and external modal segments within the same time window as positive sample pairs, using multimodal segments from different time windows as negative sample pairs, and performing comparative learning training based on the positive sample pairs and the negative sample pairs; Align the time-slice-level features of the EEG signal with the corresponding external modal frame-level features; extract semantic prototype vectors based on the text data or task labels, and align the EEG segment representations with semantic prototype vectors of the same semantic category.

3. The model training method based on electroencephalogram (EEG) signals according to claim 1, characterized in that, The step of training the EEG coding network and the external modality coding network based on the multimodal alignment samples, in conjunction with at least one alignment target in the shared latent representation space, to obtain a multimodal alignment model includes: training the EEG coding network and the external modality coding network on multimodal alignment samples without task labels by applying at least one alignment target in the shared latent representation space to obtain an initialized network model; and on labeled data containing the task labels, jointly optimizing the alignment loss corresponding to at least one alignment target with at least one downstream task loss function to fine-tune the initialized network model to obtain the multimodal alignment model.

4. The model training method based on electroencephalogram (EEG) signals according to claim 1, characterized in that, The EEG coding network includes a time-domain-frequency domain joint feature extraction module and a temporal modeling module. The process of encoding EEG segments in the multimodal aligned samples based on the EEG coding network to obtain EEG feature representations includes: extracting spatial-spectral joint features from the EEG segments based on the time-domain-frequency domain joint feature extraction module; and performing temporal dimension modeling on the spatial-spectral joint features based on the temporal modeling module to obtain the EEG feature representations. The EEG feature representations include segment-level representations and multiple time-segment-level representations.

5. The model training method based on electroencephalogram (EEG) signals according to claim 1, characterized in that, The external modality coding network includes a visual coding network, an audio coding network, and a text coding network. The process of encoding external modality segments in the multimodal aligned samples based on the external modality coding network to obtain external modality feature representations includes: encoding visual segments in the external modality segments based on the visual coding network to obtain frame-level visual features and segment-level visual representations; performing time-frequency feature modeling and temporal sequence modeling on audio segments in the external modality segments based on the audio coding network to obtain time-slice-level audio representations and segment-level audio representations; and encoding text segments in the external modality segments based on the text coding network to obtain word-level text representations and sentence-level text representations.

6. The model training method based on electroencephalogram (EEG) signals according to claim 1, characterized in that, The model training method based on EEG signals further includes: performing cross-subject alignment and individual adaptation processing on the multimodal alignment model; wherein the cross-subject alignment and individual adaptation processing includes at least one of the following methods: assigning learnable subject embedding vectors to different subjects and fusing the subject embedding vectors with the intermediate features of the EEG coding network; using domain adversarial training or statistical alignment methods to align the EEG feature representations of different subjects to suppress task-irrelevant individual-specific information; and for new subjects, fine-tuning the parameter-efficient adaptation module under the condition of freezing the model backbone parameters.

7. A method for decoding electroencephalogram (EEG) signals, characterized in that, The EEG signal decoding method includes: acquiring a target EEG signal to be decoded; inputting the target EEG signal into a multimodal alignment model to obtain a feature representation of the target EEG signal in a shared latent representation space output by the multimodal alignment model; the multimodal alignment model is obtained based on the model training method based on EEG signals according to any one of claims 1 to 6; and determining external modal data or control instructions associated with the semantic content of the target EEG signal based on the feature representation of the target EEG signal in the shared latent representation space.

8. A model training device based on electroencephalogram (EEG) signals, characterized in that, include: The data acquisition module is used to acquire synchronously recorded EEG signal data and external modal data to form multimodal aligned samples; The external modal data includes at least one of visual data, audio data, and text data; the first encoding module is used to encode the EEG segments in the multimodal aligned samples based on the EEG encoding network to obtain EEG feature representations, and to map the EEG feature representations to a shared latent representation space; The second encoding module is used to encode the external modality segments in the multimodal aligned samples based on the external modality encoding network to obtain external modality feature representations, and to map the external modality feature representations to the shared latent representation space; The model training module is used to train the EEG coding network and the external modality coding network based on the multimodal alignment samples, in conjunction with at least one alignment target in the shared latent representation space, to obtain a multimodal alignment model.

9. A brainwave signal decoding device, characterized in that, The EEG signal decoding method includes: a target EEG signal acquisition module for acquiring a target EEG signal to be decoded; a model inference module for inputting the target EEG signal into a multimodal alignment model to obtain a feature representation of the target EEG signal in a shared latent representation space output by the multimodal alignment model; the multimodal alignment model is obtained based on the model training method based on EEG signals according to any one of claims 1 to 6; and a decoding module for determining external modal data or control instructions associated with the semantic content of the target EEG signal based on the feature representation of the target EEG signal in the shared latent representation space.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the model training method based on EEG signals as described in any one of claims 1 to 6, or when the processor executes the computer program, it implements the EEG signal decoding method as described in claim 7.

Images