Cognitive state classification method and apparatus based on multi-modal neural signals, device and medium

Abstract

This invention relates to a method, apparatus, device, and medium for classifying cognitive states based on multimodal neural signals. The classification method includes: acquiring electroencephalogram (EEG) information from a subject and preprocessing it to obtain a standardized EEG feature matrix; mapping the matrix using an EEG encoder from a pre-trained variational autoencoder to generate an enhanced feature representation containing high-resolution neural image information; inputting the matrix into a multi-task expert hybrid model decoder, dynamically allocating weights through a routing network, and calling multiple functionally specialized expert networks for processing to obtain a weighted integrated expert output; mapping the matrix to a classification space to output the classification result of the subject's cognitive state. This method achieves endogeneous feature enhancement of EEG signals through cross-modal deep fusion and simultaneously constructs an expert hybrid decoding architecture that conforms to the principle of brain functional partitioning, significantly improving the accuracy of cognitive state classification.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a method, apparatus, device, and medium for classifying cognitive states based on multimodal neural signals. Background Technology

[0002] Cognitive state classification quantifies and differentiates the neurophysiological signals generated by complex cognitive processes such as memory, attention, evaluation, reasoning, and problem-solving, enabling an objective assessment of human cognitive function. In the field of clinical medicine, this technology is of vital importance for the early screening, dynamic monitoring of disease progression, and rehabilitation assessment of neurodegenerative diseases such as mild cognitive impairment (MCI) and Alzheimer's disease (AD).

[0003] Currently, electroencephalography (EEG) has become the most promising neuroimaging modality for cognitive state decoding due to its high temporal resolution at the millisecond level, low cost, and excellent portability. However, existing technologies face the following fundamental bottlenecks when dealing with complex clinical environments and multi-domain cognitive tasks: First, there is an inherent deficiency in feature quality. The inherently low spatial resolution and low signal-to-noise ratio (SNR) of EEG signals lead to severe signal ambiguity. This makes it almost impossible for traditional methods to accurately capture the activity of deep brain structures such as the hippocampus, on which higher cognitive functions depend.

[0004] Second, there are architectural limitations to decoding accuracy. This stems from the reliance of traditional methods on single, general-purpose models. These models struggle to cope with the high heterogeneity between different cognitive tasks and individuals. As a result, the models exhibit extremely poor generalization ability in cross-task or multi-domain applications, and even suffer catastrophic forgetting when continuously learning multiple tasks.

[0005] There is currently no effective solution to the above problems. Summary of the Invention

[0006] Therefore, it is necessary to provide a method, apparatus, device, and medium for cognitive state classification based on multimodal neural signals to address the aforementioned technical problems. This invention enhances the endogeneous features of EEG signals through cross-modal deep fusion, overcoming the limitations of low spatial resolution and poor signal-to-noise ratio in single-modal scenarios, and obtaining high spatiotemporal resolution feature representations. Simultaneously, it constructs an expert hybrid (MoE) decoding architecture that conforms to the principle of brain functional partitioning, efficiently and adaptively handling heterogeneous multi-domain cognitive decoding tasks, significantly improving the accuracy of cognitive state classification.

[0007] According to a first aspect of the present invention, a cognitive state classification method based on multimodal neural signals is provided, comprising: Electroencephalogram (EEG) data of the subjects were collected and preprocessed to obtain a standardized EEG feature matrix. Using the EEG encoder in a pre-trained variational autoencoder, the above... Mapping is performed to generate enhanced feature representations that include high-resolution neural image information. ; will the The input is fed into a multi-task expert hybrid model (MoE) decoder, where weights are dynamically allocated through a routing network, and multiple function-specific expert networks are invoked for processing to obtain a weighted ensemble of expert outputs. ;Will Mapped to the classification space, the classification results of the subject's cognitive state are output.

[0008] In some embodiments, the variational autoencoder includes an EEG encoder and a high-resolution neuroimaging information encoder; the training method of the variational autoencoder includes: using the EEG encoder and the high-resolution neuroimaging information encoder to map EEG information and corresponding high-resolution neuroimaging information to the same latent feature space and obtain corresponding latent variables. During the training loop, paired EEG and high-resolution neuroimaging data are input, and a masking operation is performed on the EEG and / or high-resolution neuroimaging data. The decoder then uses latent variables... Using information from the unmasked modalities, EEG information and / or high-resolution neuroimaging information are reconstructed; based on the difference between the reconstructed results and the real information, a first loss function is calculated. And perform minimum training; The enhanced feature representation Based on latent variables It is obtained by performing feature mapping.

[0009] In some embodiments, the high-resolution neuroimaging information is functional magnetic resonance imaging (fMRI) information; The process involves mapping EEG information and corresponding high-resolution neuroimaging information to the same latent feature space and obtaining the corresponding latent variables. ,include: The first encoder is used to extract the preprocessed EEG feature matrix through convolution operators. Obtain instantaneous spatial features ; Will Input a time series model and obtain a hidden state vector containing time series dependency information. ; Through two independent mapping networks Transform into a first latent distribution parameter, the first latent distribution parameter including the mean. With log variance ; Nuclear magnetic resonance feature matrix obtained by using a second encoder on preprocessed functional magnetic resonance imaging (fMRI) information Feature extraction is performed using a graph neural network structure. Based on the adjacency matrix A and degree matrix D of brain regions, graph convolution operation is performed on the active features of each brain region to capture the dynamic functional connectivity features between brain regions. The second latent distribution parameters are obtained by mapping pooling and fully connected layers. These parameters include the mean. Sum of logarithmic variance ; The latent variable z is obtained using the first and second latent distribution parameters. The formula for calculating the latent variable z is as follows:

[0010] in, and These are the mean and standard deviation of the corresponding modes, respectively. To obtain from the standard normal distribution Random noise in the sampled data.

[0011] In some embodiments, the standardized EEG feature matrix The methods for obtaining it include: The original EEG signal was bandpass filtered, and the independent component analysis algorithm was used to remove artifacts from the filtered signal to obtain the denoised EEG signal. The EEG signals were segmented according to the preset task test time window. The potential intensity of C electrode channels at T time sampling points in each test segment was extracted, and an EEG feature matrix of dimension C×T was constructed. ; The nuclear magnetic resonance feature matrix The methods for obtaining it include: The raw functional magnetic resonance imaging (fMRI) information was standardized to obtain processed whole-brain imaging data; Based on the functional brain region atlas, the whole brain imaging data is divided into R brain regions (ROIs). Calculate the mean blood oxygenation level-dependent signal of all voxels within each brain region R at time point V to obtain the signal time series of the corresponding brain region. ; Integrating signal sequences from R brain regions across V time volumes, constructing a dimensionless array... nuclear magnetic resonance characteristic matrix .

[0012] In some embodiments, the first loss function The calculation formula is:

[0013] in, Let the mean square error function be used. and These are the EEG feature matrix and MRI feature matrix reconstructed by the decoder, respectively. This is the KL divergence constraint term; These are the weighting coefficients.

[0014] In some embodiments, the weights are dynamically allocated through a routing network, and multiple function-specific expert networks are invoked for processing to obtain a weighted integrated expert output. ,include: Calculate the weight vector:

[0015] in, This is the weight matrix. This is the weight vector; Expert network processing:

[0016] in, Let N be the i-th functionally specialized expert network, and N be the total number of expert networks. Let be the learnable parameters of the i-th expert network. Weight vector The corresponding weights in the middle, This is the expert output after weighted integration.

[0017] In some embodiments, Mapped to a classification space, the classification results of the subject's cognitive state are output, including: Calculate the class probability P:

[0018] in, For classification fully connected layers, Softmax is a normalized exponential function.

[0019] In some embodiments, the training of the task expert hybrid (MoE) decoder is achieved by minimizing a second loss function. The formula for calculating the second loss function is:

[0020] in, It is the one-hot encoding of the real label; M is the total number of cognitive state categories; This represents the c-th component of the class probability P; The balance coefficient is N; the total number of expert networks is N. The percentage of samples in which expert i is selected as the optimal processing path by the routing network; The average confidence level of the routing network for expert i; For classification loss; For load balancing losses; Second loss function.

[0021] According to a second aspect of the present invention, a cognitive state classification device based on multimodal neural signals is provided, comprising: The acquisition module is used to collect the subject's electroencephalogram (EEG) information and preprocess it to obtain a standardized EEG feature matrix. ; The enhancement module is used to utilize the EEG encoder in a pre-trained variational autoencoder architecture to perform the following... Mapping is performed to generate enhanced feature representations that implicitly contain high-resolution neural image information. ; Decoding module, used to decode the The input is fed into a multi-task expert hybrid model (MoE) decoder, where weights are dynamically allocated through a routing network, and multiple function-specific expert networks are invoked for processing to obtain a weighted ensemble of expert outputs. ; Output module, used to convert Mapped to the classification space, the current cognitive state of the subject is judged.

[0022] According to a third aspect of the present invention, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor executes the computer program to implement the steps of any of the methods described above.

[0023] According to a fourth aspect of the present invention, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of the methods of any of the above embodiments.

[0024] This invention introduces a multimodal neural signal feature enhancement mechanism and utilizes a deep learning model to explore the intrinsic correlation between EEG signals and high-resolution neural image signals. This significantly improves the spatial resolution and signal-to-noise ratio of features without sacrificing the advantages of EEG temporal resolution. At the same time, it combines an expert-based cognitive state decoding architecture, which enables neural signals from different cognitive domains to be specifically modeled by corresponding expert networks. This avoids learning conflicts and improves the accuracy and robustness of cognitive state discrimination, thereby solving the dual bottlenecks of existing technologies in feature quality and decoding accuracy. Attached Figure Description

[0025] Figure 1 This is a flowchart of some embodiments of the cognitive state classification method based on multimodal neural signals of the present invention; Figure 2 This is a schematic diagram of the data processing flow of the variational autoencoder of the present invention; Figure 3 These are schematic diagrams illustrating the feature mapping process of some embodiments of the present invention; Figure 4 These are schematic diagrams illustrating the structure of some embodiments of the cognitive state classification device based on multimodal neural signals of the present invention. Figure 5 This is an internal structural diagram of a computer device used to implement some embodiments of the present invention. Detailed Implementation

[0026] Embodiments of the invention will now be described more fully with reference to the accompanying drawings, in which embodiments of the invention are illustrated. However, the invention may be practiced in many different forms, and should not be construed as limited to the embodiments set forth herein.

[0027] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. As used herein, the singular forms “a” and “this” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should also be understood that, when used herein, the term “comprising” specifies the presence of the stated features, integrals, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or groups thereof.

[0028] Unless otherwise defined, the terms used herein (including technical and scientific terms) shall have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains. Terms used herein shall be interpreted as having the same meaning as they have in the context of this specification and in the relevant field, and shall not be interpreted in an idealized or overly formal sense, unless specifically defined herein.

[0029] Electroencephalography (EEG) is a non-invasive neural signal acquisition technique that uses multiple electrodes placed on the scalp of a subject to record in real time the potential changes generated by the synchronous activity of a large number of neurons, typically with a time resolution in the millisecond range.

[0030] High-resolution neuroimaging information refers to signals from neuroimaging techniques with high spatial resolution, capable of precisely locating functional areas of brain activity. Examples include functional magnetic resonance imaging (fMRI) and magnetoencephalography (MEG) signals.

[0031] Mixture-of-Experts (MoE) is a neural network architecture consisting of a routing network and multiple expert networks. The routing network dynamically allocates input data to the most suitable expert for processing, and the final result is a weighted combination of the outputs of each expert. It is suitable for handling scenarios with high heterogeneity and complex tasks.

[0032] Foundation Model: A large-scale model pre-trained on massive, diverse data, capable of learning general, transferable feature representations. In this invention, it specifically refers to a large-scale EEG model pre-trained on massive amounts of EEG data.

[0033] Joint Autoencoder: An autoencoder model whose encoder and decoder are designed to process input data from multiple modalities by learning a consistent representation across modalities by mapping different modal data to a shared hidden space.

[0034] This invention provides an end-to-end cognitive state classification framework that follows the brain's cognitive principles of "functional specialization and on-demand activation." By combining multimodal neural signal feature enhancement with brain-like expert decoding, it achieves accurate identification of a subject's cognitive state. The framework supports the construction of high-precision cross-modal fusion features under multimodal neural signal conditions and, in practical applications, can complete reasoning relying solely on a single EEG signal, thus balancing model performance and application feasibility.

[0035] During the model building (training) phase, simultaneous acquisition devices were used to acquire both electroencephalography (EEG) and functional magnetic resonance imaging (fMRI) signals from the subjects. This ensured the temporal alignment of the two modalities, thereby establishing a stable multimodal cognitive feature mapping relationship. Specifically, the EEG signals underwent preprocessing operations such as bandpass filtering, independent component analysis artifact removal, and reference electrode repositioning to remove noise from electrooculography (EOG), electromyography (EMG), and power frequency interference. The fMRI signals underwent preprocessing operations such as head movement correction, temporal layer correction, spatial normalization, and high-pass filtering to eliminate scanning artifacts and physiological noise. Subsequently, cognitive-related features such as power spectral density, temporal statistical characteristics (mean, variance, peak value), time-frequency domain features (wavelet transform coefficients), and brain network connectivity features were extracted from the preprocessed EEG signals. Features from the fMRI signals were extracted using methods such as functional connectivity strength analysis (FC), local consistency (ReHo), low-frequency amplitude (ALFF), and voxel-based morphological features (VBM). The aforementioned multimodal features are input into the multimodal precision self-enhancing model. The cross-modal feature modeling mechanism suppresses redundant noise and strengthens features related to cognitive states, outputting enhanced multimodal fusion features. The enhanced fusion features are then input into the multi-task expert hybrid model decoder to jointly train multiple expert networks, enabling each expert to adapt to different cognitive state sub-tasks, and collaboratively complete model construction through a gating mechanism.

[0036] During the model usage (testing or inference) phase, only the subject's EEG signal needs to be collected; fMRI acquisition equipment is not required. First, noise suppression is performed on the EEG signal using the same preprocessing procedure as in the training phase. Then, based on the feature extraction method determined in the training phase, unimodal cognitive features are extracted from the preprocessed EEG signal. Subsequently, these unimodal features are input into the trained multimodal precision self-enhancing model. Based on the cross-modal mapping relationship learned in the training phase, enhanced features containing high-resolution neuroimaging information are generated. Finally, the enhanced features are input into the trained multi-task expert hybrid model decoder, which invokes the corresponding expert network through a brain-like sparse activation mechanism to output the subject's current cognitive state classification result.

[0037] Figure 1 This is a flowchart of some embodiments of the cognitive state classification method based on multimodal neural signals of the present invention.

[0038] like Figure 1 As shown, the method includes: Step S102: Collect the subject's EEG information and preprocess it to obtain a standardized EEG feature matrix. ; In some embodiments, the original EEG signal is denoted as It first uses a bandpass filter operator. Frequency band limiting was performed to remove low-frequency drift and high-frequency noise. Then, independent component analysis (ICA) was used to separate and remove artifacts such as electrooculography (EOG), electromyography (EMG), and power line interference from the EEG signal, resulting in the artifact-free EEG signal. The calculation process can be expressed as follows:

[0039] After denoising, the data is segmented according to the experiment, and the data for each experiment can be represented as a matrix.

[0040] in, This represents the number of electrode channels. This represents the number of time sampling points.

[0041] Step S104: Using the EEG encoder in the pre-trained variational autoencoder, the data is processed... Mapping is performed to generate enhanced feature representations that include high-resolution neural image information. ; In some embodiments, the variational autoencoder includes an electroencephalogram (EEG) encoder. and high-resolution neural imaging encoder .

[0042] The EEG encoder is used to extract spatiotemporal joint features from EEG signals, and its specific implementation process includes the following sub-steps: Spatial feature extraction: EEG feature matrix Input a two-dimensional convolutional network (Conv2D) and perform convolution operations in the channel and time dimensions to fuse the spatial correlations between different electrode channels and obtain spatial feature representations. :

[0043] in, This represents the learnable parameters of a convolutional network.

[0044] Capturing temporal dependencies: Representing spatial features The signals are input sequentially into a gated recurrent unit (GRU) network to model the dynamic changes of the EEG signal over time, resulting in the final hidden state vector. :

[0045] in, These are the parameters for the GRU network.

[0046] Calculation of potential distribution parameters: Hidden state vector Two independent parameter mapping networks (e.g., fully connected layers) are input separately to compute the mean and log-variance of the latent variable distribution:

[0048] Ultimately, the final hidden state is obtained through two independent mapping networks. Calculate the mean of the latent variable distribution. Sum of logarithmic variance This mapping network can be implemented using fully connected layers, convolutional layers, or other parameterized functions.

[0049]

[0050] During the model training phase, Used for feature modeling of synchronously acquired functional magnetic resonance imaging (fMRI) signals.

[0051] In some embodiments, the raw fMRI data, after standardization, is divided into brain regions according to a preset brain region atlas. Regions of Interest (ROIs). For each brain region... At the point in time The mean of all voxel signals within this brain region is calculated to obtain the brain region-level BOLD signal:

[0052] This constitutes the fMRI feature matrix:

[0053] in, This represents the time volume.

[0054] Subsequently, the aforementioned The input is a Graph Neural Network (GNN), which uses the adjacency matrix A and degree matrix D of brain regions to perform graph convolution operations on the activity features of each brain region, capturing the dynamic functional connectivity features between brain regions.

[0055] in D is the adjacency matrix, and D is the degree matrix. These are features of layer l. These are learnable parameters. Finally, through pooling and fully connected mapping, the latent distribution parameters of fMRI are obtained. and .

[0056] Based on the aforementioned latent distribution parameters, latent variables z are sampled from the distribution obtained by the encoder.

[0057]

[0058] For latent variables After feature mapping, an enhanced feature representation is obtained. .

[0059] The specific method for obtaining this information can be: based on the mean. With log variance Obtaining latent variables using reparameterization operations The obtained latent variable z is input into the feature mapping network for processing. Latent variable It is a highly compressed latent representation that, after training with a variational autoencoder, has implicitly learned the shared information between electroencephalography (EEG) and high-resolution neuroimaging (such as fMRI). The feature mapping network is used to map the latent variable z to a feature space of a preset dimension, decompressing or projecting the latent representation, thereby recovering a feature representation containing spatial structural details from the latent space to compensate for the low spatial resolution of the original EEG signal. After processing by the feature mapping network, an enhanced feature representation is output. .

[0060] Figure 2 A schematic diagram of the data processing flow of a variational autoencoder is shown. During the training phase, a modality masking strategy is introduced to randomly mask some channels, time segments, or the entire modality of the EEG or fMRI modality, so that the decoder relies only on the unmasked modalities and latent variables. Reconstructing the target modality forces the model to learn shared representations between modalities in the latent space, thereby enabling it to complete cross-modal information and generalize.

[0061] The final loss function:

[0062] The first term is the reconstruction loss, which is calculated using mean squared error (MSE) in practice. The second term is the KL divergence, used as a regularization term. These are weighting coefficients. Under the modality masking strategy (taking EEG input as an example), the loss function is specified as:

[0063] Step S106, the... The input is fed into a multi-task expert hybrid model decoder, where weights are dynamically allocated through a routing network, and multiple function-specific expert networks are invoked for processing to obtain a weighted ensemble of expert outputs. ; In some embodiments, the weights are first calculated by the routing network:

[0064] Subsequently, various expert networks Parallel processing of input features:

[0065] The final output is the weighted fusion result:

[0066] The introduced expert hybrid model employs a "divide and conquer" strategy, designed in accordance with the principles of brain functional partitioning. This architecture comprises a routing network and multiple parallel expert networks, each specifically trained to decode neural signal patterns for a particular cognitive domain (e.g., visual, auditory, or motor tasks). This design avoids conflicts between different task patterns during the learning process, significantly improving decoding accuracy and robustness.

[0067] Specifically, this is implemented by receiving the enhanced features. Calculate the route logical values ​​(logits) and obtain the route weights. (N is the number of experts).

[0068]

[0069] In the multi-task expert hybrid model (MoE) decoder, each expert network These are function-specific independent neural networks that process input feature representations in parallel. Their core design focuses on enabling each expert to concentrate on decoding a specific cognitive functional domain (such as vision, hearing, or motor imagery). This strategy aligns with the principles of brain functional partitioning, aiming to avoid conflicts between different task modes during learning. For example, a visual processing expert can decode visual evoked potentials in the occipital cortex using a convolutional neural network (CNN), an auditory processing expert can analyze the P300 component of the temporal lobe using a temporal convolutional network (TCN), and a motor imagery expert can decode μ / β rhythmic event-related desynchronization in the sensorimotor cortex using common spatial pattern (CSP) features. The construction of these function-specific models typically follows a data-driven process: first, high-quality labeled neural signal data for a specific cognitive task is collected; then, an appropriate deep learning architecture (such as CNN, RNN, or Transformer) is selected or designed for end-to-end training to learn the neural pattern representations specific to that cognitive domain. A routing network acts as a scheduling center, dynamically evaluating input features and assigning them to the most relevant expert networks. Finally, a weighted integration of the expert outputs achieves accurate cognitive state classification results. See the formula below for a specific implementation example:

[0070] The final expert output after weighted integration is obtained:

[0071] Step S108, will Mapped to the classification space, the classification results of the subject's cognitive state are output.

[0072] In some embodiments, Input the classification layer and obtain the class probabilities using the Softmax function:

[0073] The task expert hybrid (MoE) decoder is trained by minimizing the second loss function. The formula for calculating the second loss function is:

[0074] in, This indicates that the device was routed to the specified number in the current training batch. The sample proportion of each expert network is calculated as follows:

[0075] in, For training batch size, Indicates the first The routing logical value vector obtained by the routing network for each sample This is an indicator function that takes the value 1 when the condition is true and 0 otherwise.

[0076] This indicates that the routing network is for the first [number] in the current batch. The average confidence level of an expert network is calculated as follows:

[0077] in, For the first The nth sample is assigned to the nth sample after Softmax normalization. The weight values ​​of each expert network. It is the one-hot encoding of the real label; M is the total number of cognitive state categories; This represents the c-th component of the class probability P; The balance coefficient is N; the total number of expert networks is N. For classification loss; For load balancing losses; Second loss function.

[0078] Figure 3 A schematic diagram of the feature mapping process according to some embodiments of the present invention is shown. For example... Figure 3 As shown, the feature mapping process includes the following steps: Step 1: Receive EEG signals as input, including high-lead EEG signals and low-lead EEG signals. The high-lead EEG signals provide more complete spatial distribution information, while the low-lead EEG signals simulate EEG acquisition conditions in real-world application scenarios.

[0079] Step 2: Input the high-lead and low-lead EEG signals into a pre-trained EEG baseline model for feature extraction. The EEG baseline model is a neural network model (e.g., the LaBraM model) pre-trained on large-scale EEG data. It is used to uniformly model EEG signals under different lead configurations and output standardized initial EEG feature representations, thereby reducing the impact of differences in the number and layout of leads on feature distribution.

[0080] Step 3: The initial EEG feature representation is input into a feature mapping network for feature transformation. The feature mapping network is preferably a multilayer perceptron (MLP) structure. Simultaneously, pre-acquired high-resolution neural image features are introduced as auxiliary input into the feature mapping network. These high-resolution neural image features are, for example, functional magnetic resonance imaging (fMRI) features, and can be pre-extracted offline. Based on the high-resolution neural image features, the feature mapping network performs spatial feature enhancement and alignment on the initial EEG features.

[0081] Step 4: Output the enhanced EEG feature representation. This enhanced feature maintains the original high temporal resolution advantage of EEG while aligning with high-resolution neural image information in the feature space, thereby implicitly encoding richer spatial structural information.

[0082] This invention employs a multimodal precision self-enhancing model, introducing high spatial resolution neuroimaging signals to calibrate and enhance EEG signals. This allows the generated EEG features to implicitly contain high spatial resolution information while maintaining high temporal resolution, effectively improving the signal-to-noise ratio and feature discriminative power. Simultaneously, it combines a multi-task expert hybrid decoding architecture, enabling different expert networks to learn neural patterns for specific cognitive tasks, avoiding learning conflicts inherent in single models and significantly improving the accuracy and robustness of complex cognitive state classification. Furthermore, by introducing a pre-trained EEG baseline model, this invention can unify the modeling of high-lead and low-lead EEG data, achieving effective mapping from low-lead EEG to the high-lead feature space, enhancing the method's scalability and practical application value. Moreover, the expert hybrid model adopted is based on a brain-like modular and on-demand activation mechanism, activating only expert networks relevant to the current task. This significantly reduces computational overhead while maintaining high model expressive power, making the overall design more aligned with the brain's natural cognitive patterns.

[0083] Figure 4 The diagram shows structural schematics of some embodiments of the cognitive state classification device based on multimodal neural signals of the present invention.

[0084] like Figure 4 As shown, it includes: The acquisition module 100 is used to acquire the subject's electroencephalogram (EEG) information and preprocess it to obtain a standardized EEG feature matrix. ; Enhancement module 200 is used to utilize the EEG encoder in a pre-trained variational autoencoder architecture to... Mapping is performed to generate enhanced feature representations that implicitly contain high-resolution neural image information. ; Decoding module 300, used to decode the The input is fed into a multi-task expert hybrid model (MoE) decoder, where weights are dynamically allocated through a routing network, and multiple function-specific expert networks are invoked for processing to obtain a weighted ensemble of expert outputs. ; Output module 400 is used to... Mapped to the classification space, the current cognitive state of the subject is judged.

[0085] For specific limitations regarding a cognitive state classification device based on multimodal neural signals, please refer to the limitations of a cognitive state classification method based on multimodal neural signals mentioned above, which will not be repeated here. Each module in the aforementioned cognitive state classification device based on multimodal neural signals can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0086] The present invention also provides a computer device, which may be a terminal, and its internal structure diagram may be as follows: Figure 4 As shown. The computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements the aforementioned cognitive state classification method based on multimodal neural signals. The display screen can be a liquid crystal display (LCD) or an e-ink display. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the computer device casing, or an external keyboard, touchpad, or mouse. Those skilled in the art will understand that… Figure 4 The structure shown is merely a block diagram of a portion of the structure related to the present invention and does not constitute a limitation on the computer device to which the present invention is applied. A specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0087] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described cognitive state classification method based on multimodal neural signals.

[0088] Those skilled in the art will understand that implementing all or part of the processes in the above method embodiments can be accomplished by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided by this invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), memory bus (RAMbus), direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0089] The embodiments of the present invention have now been described in detail. To avoid obscuring the concept of the invention, some details known in the art have not been described. Those skilled in the art will fully understand how to implement the technical solutions disclosed herein based on the above description.

[0090] While specific embodiments of the present invention have been described in detail by way of examples, those skilled in the art should understand that the above examples are for illustrative purposes only and are not intended to limit the scope of the invention. Those skilled in the art should understand that modifications can be made to the above embodiments or equivalent substitutions can be made to some technical features without departing from the scope and spirit of the invention. The scope of the invention is defined by the appended claims.

Claims

1. A cognitive state classification method based on multimodal neural signals, characterized in that, include: Electroencephalogram (EEG) data of the subjects were collected and preprocessed to obtain a standardized EEG feature matrix. ; Using the EEG encoder in a pre-trained variational autoencoder, the above... Mapping is performed to generate enhanced feature representations that include high-resolution neural image information. ; The The input is fed into a multi-task expert hybrid model (MoE) decoder, where weights are dynamically allocated through a routing network, and multiple function-specific expert networks are invoked for processing to obtain a weighted ensemble of expert outputs. ; Will Mapped to the classification space, the classification results of the subject's cognitive state are output.

2. The cognitive state classification method based on multimodal neural signals according to claim 1, characterized in that, The variational autoencoder includes an EEG encoder and a high-resolution neuroimaging information encoder. The training method for the variational autoencoder includes: Using an EEG encoder and a high-resolution neuroimaging information encoder, EEG information and corresponding high-resolution neuroimaging information are mapped to the same latent feature space, respectively, and the corresponding latent variables are obtained. ; During the training loop, pairs of EEG information and high-resolution neuroimaging information are input, and a masking operation is performed on the EEG information and / or the high-resolution neuroimaging information. Based on the latent variables by the decoder From information on unmasked modalities, EEG information and / or high-resolution neuroimaging information are reconstructed; The first loss function is calculated based on the difference between the reconstruction result and the real information. And perform minimum training; The enhanced feature representation Based on latent variables It is obtained by performing feature mapping.

3. The cognitive state classification method based on multimodal neural signals according to claim 2, characterized in that, The high-resolution neuroimaging information is functional magnetic resonance imaging (fMRI) information; The process involves mapping EEG information and corresponding high-resolution neuroimaging information to the same latent feature space and obtaining the corresponding latent variables. ,include: The first encoder is used to extract the preprocessed EEG feature matrix through convolution operators. Obtain instantaneous spatial features ; Will Input a time series model and obtain a hidden state vector containing time series dependency information. ; Through two independent mapping networks Transform into a first latent distribution parameter, the first latent distribution parameter including the mean. With log variance ; Nuclear magnetic resonance feature matrix obtained by using a second encoder on preprocessed functional magnetic resonance imaging (fMRI) information Feature extraction is performed using a graph neural network structure. Based on the adjacency matrix A and degree matrix D of brain regions, graph convolution operation is performed on the active features of each brain region to capture the dynamic functional connectivity features between brain regions. The second latent distribution parameters are obtained by mapping pooling and fully connected layers. These parameters include the mean. Sum of logarithmic variance ; The latent variable z is obtained using the first and second latent distribution parameters. The formula for calculating the latent variable z is as follows: in, and These are the mean and standard deviation of the corresponding modes, respectively. To obtain from the standard normal distribution Random noise in the sampled data.

4. The cognitive state classification method based on multimodal neural signals according to claim 3, characterized in that, The standardized EEG feature matrix The methods for obtaining it include: The original EEG signal was bandpass filtered, and the independent component analysis algorithm was used to remove artifacts from the filtered signal to obtain the denoised EEG signal. The EEG signals were segmented according to the preset task test time window. The potential intensity of C electrode channels at T time sampling points in each test segment was extracted, and an EEG feature matrix of dimension C×T was constructed. ; The nuclear magnetic resonance feature matrix The methods for obtaining it include: The raw functional magnetic resonance imaging (fMRI) information was standardized to obtain processed whole-brain imaging data; Based on the functional brain region atlas, the whole brain imaging data is divided into R brain regions (ROIs). Calculate the mean blood oxygenation level-dependent signal of all voxels within each brain region R at time point V to obtain the signal time series of the corresponding brain region. ; Integrating signal sequences from R brain regions across V time volumes, constructing a dimensionless array... nuclear magnetic resonance characteristic matrix .

5. The cognitive state classification method based on multimodal neural signals according to claim 3, characterized in that, The first loss function The calculation formula is: in, Let the mean square error function be used. and These are the EEG feature matrix and MRI feature matrix reconstructed by the decoder, respectively. This is the KL divergence constraint term; These are the weighting coefficients.

6. The cognitive state classification method based on multimodal neural signals according to claim 1, characterized in that, The process involves dynamically allocating weights through a routing network and invoking multiple specialized expert networks for processing to obtain a weighted, integrated expert output. ,include: Calculate the weight vector: in, This is the weight matrix. This is the weight vector; Expert network processing: in, Let N be the i-th functionally specialized expert network, and N be the total number of expert networks. Let be the learnable parameters of the i-th expert network. Weight vector The corresponding weights in the middle, This is the expert output after weighted integration.

7. The cognitive state classification method based on multimodal neural signals according to claim 6, characterized in that, Will Mapped to a classification space, the classification results of the subject's cognitive state are output, including: Calculate the class probability P: in, For classification fully connected layers, Softmax is a normalized exponential function.

8. The cognitive state classification method based on multimodal neural signals according to claim 7, characterized in that, The task expert hybrid (MoE) decoder is trained by minimizing the second loss function. The formula for calculating the second loss function is: in, It is the one-hot encoding of the real label; M is the total number of cognitive state categories; This represents the c-th component of the class probability P; The balance coefficient is N; the total number of expert networks is N. The percentage of samples in which expert i is selected as the optimal processing path by the routing network; The average confidence level of the routing network for expert i; For classification loss; For load balancing losses; Second loss function.

9. A cognitive state classification device based on multimodal neural signals, characterized in that, include: The acquisition module is used to collect the subject's electroencephalogram (EEG) information and preprocess it to obtain a standardized EEG feature matrix. ; The enhancement module is used to utilize the EEG encoder in a pre-trained variational autoencoder architecture to perform the following... Mapping is performed to generate enhanced feature representations that implicitly contain high-resolution neural image information. ; Decoding module, used to decode the The input is fed into a multi-task expert hybrid model (MoE) decoder, where weights are dynamically allocated through a routing network, and multiple function-specific expert networks are invoked for processing to obtain a weighted ensemble of expert outputs. ; Output module, used to convert Mapped to the classification space, the current cognitive state of the subject is judged.

10. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 8.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 8.

Images