A cognitive state interpretability approach based on the brain-language-vision grand model

Abstract

The present invention belongs to the technical field of cognitive state interpretation of learners, and specifically discloses a cognitive state interpretable method based on a brain-language-vision large model, including the following steps: multimodal data collection for learners' cognitive states; characterization of learners' cognitive states based on a multi-scale graph convolutional network; brain-vision-language hybrid multimodal temporal feature fusion based on hierarchical mutual information maximization; knowledge graph node embedding to enhance the multimodal large model's ability to understand complex tasks; knowledge-guided brain-vision-language multimodal large model training; learners' cognitive state identification based on a multimodal large model; interpretable analysis of learners' cognitive states supported by multimodal embodied reasoning. The present invention adopts the above-mentioned cognitive state interpretable method based on a brain-language-vision large model to convert the input multimodal fusion features into high-quality prompts to guide the large language model to extract implicit knowledge, providing important support for reasoning generation.

Description

Technical Field

[0001] The present invention relates to the technical field of learner cognitive state interpretation, and in particular to a cognitive state interpretable method based on a brain-language-vision macro model. Background Art

[0002] As the core place for learners to process information, the brain encodes information through vision, language and other means, shaping learners' cognitive representation of the outside world. Decoding brain states related to learning tasks from neural signals can explain the brain's information processing process. In particular, interpretable processing of learners' cognitive processing can deepen learners' understanding of cognitive processing, promote the construction of more diverse cognitive network models, and enhance their ability to autonomously regulate learning strategies, thereby promoting deep and meaningful learning. The dual coding theory of cognitive neuroscience shows that specific concepts are encoded in the brain from both visual and linguistic aspects, among which language, as an effective prior experience, helps to shape visually derived representations. Therefore, when dealing with specific concepts, learners can not only perceive the external world through visual information, but also connect this information with existing knowledge structures through language. This dual coding process of language and vision helps learners establish richer cognitive representations. However, the brain's cognitive processing and knowledge construction is an implicit process. Learners often find it difficult to autonomously realize their own cognitive state, resulting in the inability to identify and adjust learning blind spots in a timely manner, hindering the deep understanding and application of knowledge. Therefore, developing a multimodal deep learning decoding method based on the visual-linguistic features of the learner's brain can help to deeply monitor and analyze the learner's cognitive state, promote the effective integration of multimodal information of brain activity data, and reveal the cognitive mechanism of the human brain, so as to correctly interpret the learner's cognitive state and provide personalized learning feedback, thereby promoting more effective learning strategy regulation.

[0003] Current issues:

[0004] At present, the recognition of learners' cognitive states only relies on teachers' subjective perception and traditional evaluation methods such as questionnaires, behavioral observations, interviews, and physiological measurements, which cannot be objectively and efficiently identified. The analysis of cognitive states through brain data still mostly relies on fMRI and EEG data, lacking multimodal evidence, resulting in low accuracy.

[0005] Specifically:

[0006] 1. Limitations of visual neural representation decoding methods. Most current visual neural representation decoding methods rely on visual semantic knowledge extracted from images. For example, brain activity is analyzed using Gabor wavelet filters or convolutional neural networks (CNNs), or generative models are trained through generative adversarial networks (GANs) to reconstruct visual stimuli. Although these techniques can capture key features in images, they often ignore the impact of individual prior knowledge on the processing of visual stimuli. Relying solely on visual features cannot fully reflect the brain's complex processing of specific visual stimuli. In addition, there is currently a lack of embodied language models that combine visual features with information from other modalities such as language, which makes it difficult for us to more accurately simulate the human brain's cognitive mechanism of perceived objects.

[0007] 2. Insufficient research on language feature decoding. Although some studies have explored the use of language features alone to decode natural visual experiences, these studies mainly rely on standard noun-like vectors automatically extracted from large corpora such as Common Crawl. However, the word vectors of these noun-like words are poorly aligned with the visual information, resulting in the accuracy of neural decoding not meeting practical standards. In addition, existing cognitive state detection methods can usually only decode brain activity of predefined categories, lack the ability to generalize to new categories, and are slow when processing multimodal information sources, thus still facing challenges in accuracy, real-time performance, and interpretability of results.

[0008] 3. The potential of brain visual representation in cognitive state has not been fully utilized. First, existing educational technologies and cognitive state assessment tools mainly rely on behavioral performance and emotional feedback, and pay less attention to the in-depth exploration of brain visual representation. For example, eye tracking technology can capture learners' visual attention, but it cannot fully reveal how visual information is decoded in the brain and how this information affects learners' understanding, reasoning and memory formation. Secondly, the potential of brain visual representation in multimodal data integration has not been fully utilized. Existing multimodal data analysis mainly combines voice, text and behavioral data, but does not deeply explore the potential connection between brain visual representation and these data.

[0009] Finally, although recent advances in neuroscience have enabled us to monitor the brain's visual processing through brain imaging techniques (such as fMRI or EEG), the application of these technologies in the field of education is often limited to laboratory research and has not yet been translated into practical teaching tools or learning analysis methods on a large scale. Summary of the invention

[0010] The purpose of the present invention is to provide an interpretable method for cognitive states based on a brain-language-vision large model, extract multimodal features and combine them with brain visual representation to construct a brain map network architecture, use a spatiotemporal pyramid graph convolution module to decode brain data, fuse and regularize multimodal information such as brain representation data, visual image perception data and language text data, inject continuous and embodied fused data into the embedding space of a pre-trained language model, and infer the learner's cognitive state by combining knowledge graphs and thinking chains, thereby generating a hybrid multimodal embodied reasoning method to accurately interpret the learner's cognitive state.

[0011] To achieve the above object, the present invention provides a cognitive state interpretable method based on a brain-language-vision macro model, comprising the following steps:

[0012] S1. Collect multimodal data on learners’ cognitive status;

[0013] S2, cognitive state representation based on multi-scale graph convolutional network;

[0014] S3, Brain-vision-language hybrid multimodal temporal feature fusion based on hierarchical mutual information maximization;

[0015] S4, knowledge graph node embedding to enhance the ability of multimodal large models to understand complex tasks;

[0016] S5. Knowledge-guided brain-vision-language multimodal large model training;

[0017] S6. Identification of learner cognitive states based on multimodal large models;

[0018] S7. Interpretable analysis of learners’ cognitive states supported by multimodal embodied reasoning.

[0019] Preferably, in S1, the learner's cognitive state includes memory, understanding, application, reasoning, analysis, evaluation, creation and automation, and the multimodal data collection includes brain data collection, visual data collection and language data collection;

[0020] The specific process of brain data collection is as follows:

[0021] High-resolution imaging technology fMRI is used to collect learners' resting-state brain imaging data, and the collected fMRI resting-state brain imaging data are preprocessed into The input matrix of

[0022] The visual data collection process is as follows:

[0023] Record videos based on learners’ classroom learning scenarios, extract video frames at fixed intervals, and process each static image;

[0024] The language data collection process is as follows:

[0025] Collect learners' learning materials and text data generated during the learning process, as well as teacher-student and student-student classroom dialogue data. Represents a specific word or subword produced by the learner, using a finite set Represents a vocabulary of all possible words or subwords, maps discrete words to a continuous vector space, and preprocesses them into a vector space of size The large embedding matrix of .

[0026] Preferably, in S2, the learner's current cognitive state is determined by fMRI resting-state brain imaging during each learning task period:

[0027] The fMRI resting-state brain imaging data is labeled as ,in Represents the feature matrix of fMRI resting-state brain imaging data related to cognitive state, including ROIs and time samples, From the tag set The cognitive state label is used to indicate the type of cognitive state corresponding to the fMRI resting-state brain imaging data. is the number of samples in the dataset;

[0028] Learning an optimal graph network from a training dataset , and based on each fMRI resting-state brain imaging data in the test dataset Prediction task label:

[0029] (1).

[0030] Preferably, the cognitive state representation based on the multi-scale graph convolutional network in S2 includes the following two modules:

[0031] S21, brain map construction;

[0032] The brain is divided into The functional connectivity FC of each individual in each brain region was calculated by fMRI resting-state brain imaging data, and the average functional connectivity matrix was binarized and then used Graph building strategy, building brain graph ;

[0033] Among them, the node ROIs are regions of interest, Indicates the connection relationship between nodes;

[0034] but The brain activity of a specific ROI at a time point is calculated by averaging all voxels within the ROI, assuming that there are learner, The brain functional connectivity matrix of each learner The Pearson correlation of fMRI resting-state brain imaging data between ROIs was calculated, and then the adjacency matrix of all learners was averaged to obtain the group-level adjacency matrix as follows:

[0035] (2);

[0036] Construct a brain map with undirected edges for a specific ROI, connecting it to the top correlation rankings in other ROIs The area, from the group-level adjacency matrix Prune the connections in the middle to obtain an undirected group-level adjacency matrix with binary Brain map with features :

[0037] (3);

[0038] in, Indicates ROI and The connection strength between ROIs, Indicates The first order of connection strength in the ROI Large value, is the binary adjacency matrix, indicating the ROI and Whether there is a connection between the ROIs;

[0039] S22, Brain Decoding;

[0040] Collect blood oxygen level dependent signal data BOLD from functional magnetic resonance imaging. Each signal data BOLD is used to reflect the dynamic response of multiple brain region ROIs at different time points. Record the activity changes of N ROIs within m frame time. The specific process is as follows:

[0041] The BOLD signal is converted into a feature matrix after preprocessing, and the feature matrix information is used as the input data of the model to analyze the activity patterns of ROIs in different cognitive states of learners;

[0042] In the feature matrix Based on this, a feature graph is generated, where the nodes of the graph represent the brain regions of interest, and the edges between the nodes represent the associations between these brain regions, which is used to reflect the interconnection requirements of different cognitive tasks in the classroom on brain regions.

[0043] The feature map is input into the spatiotemporal pyramid graph convolution module for multi-layer spatiotemporal convolution processing to capture the dynamic correlation between brain regions.

[0044] Preferably, the spatiotemporal pyramid graph convolution module in S22 includes three spatiotemporal pathways with time convolutions of different scales and a plurality of bottom-up pathways:

[0045] The spatiotemporal pathway includes a time-gated convolution module and a spatial convolution module. is the input vector With filter The product of:

[0046] (4);

[0047] in, and They are the normalized graph Laplacian matrices The eigenvector matrix and the diagonal matrix of eigenvalues ​​are ;

[0048] Graph Laplacian Matrix Through the adjacency matrix The conversion is obtained, expressed as:

[0049] ,in is the degree matrix, whose diagonal elements are , is the identity matrix, and the adjacency matrix is ​​obtained from fMRI resting-state brain imaging data ;

[0050] Using Chebyshev polynomials Approximate Filter , reformulating the spatial graph convolution module as Chebyshev polynomials A linear function of:

[0051] (5);

[0052] in, , express The maximum eigenvalue of and ,Will and Substituting into formula (5), the spatial graph convolution is approximately expressed as:

[0053]

[0054] (6);

[0055]

[0056] The time-gated convolution module is a network with a width of A one-dimensional convolution kernel followed by a gated linear unit GLU, where Indicates j The temporal convolution kernel in the spatiotemporal path, the input of each node in the figure is represented as ,in Indicates the input length, Indicates the number of input channels, time convolution kernel ,Will Mapped to an output element , and Divided into two parts with the same number of channels, where Indicates the number of output channels;

[0057] Then the time-gated convolution module is expressed as:

[0058] (7);

[0059] in, represents temporal convolution; and They are the inputs of the gated unit in GLU respectively; ⊙ represents the element-wise Hadamard product; Represents sigmoid gating, which is used to determine the input The importance of

[0060] In the bottom-up pathway, each path connects the fine-grained and coarse-grained features at different spatiotemporal scales layer by layer and transmits them upward. The spatial features of the bottom path are transmitted to the top path through the gradual integration of multi-scale information, forming a feature representation with global and local perspectives, thereby generating multi-level semantic information.

[0061] Preferably, before S3, the step also includes predicting a text sequence by only decoding the large language model LLMs. Probability , where the text sequence Represents a sequence of tokens , M represents the total length of the text sequence;

[0062] By factorizing the neural network architecture we get:

[0063] (8);

[0064] in, It is a Transformer network;

[0065] The pre-trained model is prefixed without changing the architecture. To make a conditional forecast:

[0066] (9);

[0067] Prefix Used to provide LLM continuation prediction follow-up marks the context in which it is based;

[0068] Convert words in natural language into vector representations processed by the large language model LLM and perform the following mapping:

[0069] definition To indicate a specific word or subword, use Represents a finite set of discrete words. Internally, LLM maps Will Embedded into word token embedding space In ,in , mapping Usually expressed as a size of A large embedding matrix.

[0070] Preferably, the text embedding based on continuous observation of multimodal data of learner brain data and visual image data is as follows:

[0071] For encoder Different architectural choices are proposed to directly map continuously observed multimodal data to the language embedding space of a large language model instead of discretizing it. middle:

[0072] (10);

[0073] The continuous observation embedding vectors are alternately arranged with the normal text token embedding to form a prefix, that is, each vector in the prefix All from word token embedders or encoder :

[0074] (11);

[0075] Observation Encoded into multiple embedding vectors, using different encoders alternately at different positions of the prefix , to combine information from different observation spaces;

[0076] The self-attention mechanism and visual transformer ViT are used to extract tags from brain decoding data and 2D images and embed them into the large language model LLM respectively;

[0077] Among them, the self-attention mechanism is used to input brain ROIs and spatiotemporal sequence feature dimensions into the large language model. , N represents the number of regions of interest ROIs, d represents the feature dimension of a specific ROI, then the number of features extracted from fMRI resting-state brain imaging data is:

[0078] (12);

[0079] (13);

[0080] (14);

[0081] in, is the brain feature vector obtained by convolutional decoding of the spatiotemporal pyramid graph, , , is a weight matrix mapping to query, key, and value, is the dimension of the key, used for the scaling factor;

[0082] Introducing affine transformation The embeddings generated by ViT are adjusted to match the dimensions of the language model, and for each image the embedding is labeled ,pass Transformed into a vector representation suitable for language model .

[0083] Preferably, in S3, a multimodal autoencoder variational Bayesian model is constructed for multimodal feature fusion, using three modality-specific encoding networks , , the unimodal feature , Transformed into a joint latent variable , then passes through three modality-specific decoding networks , Perform feature reconstruction respectively;

[0084] The multimodal joint modeling process is as follows: a variational autoencoder (VAE) latent variable model is used for scalable unsupervised representation learning. For unimodal data, VAE combines latent variables Define the marginal log-likelihood:

[0085] ;

[0086] VAE maximizes the evidence lower bound ELBO to obtain:

[0087] (15);

[0088] in, is the approximate posterior distribution represented by the encoder network, and its parameters are , is the conditional distribution represented by the decoder network, which is parameterized by ;

[0089] The joint marginal log-likelihood and its evidence lower bound ELBO are expressed as:

[0090] (16);

[0091] Parameterized by The decoder is composed of , ;

[0092] Using the variational joint posterior To approximate the joint posterior of the model ,maximize , further optimize ;

[0093] By applying the formula with joint latent variable learning and cross-modal generation capabilities in the neural decoding task, the above formula can be further written as:

[0094] (17);

[0095] in, yes is a possible subset of Is A new set consisting of all possible subsets of ;

[0096] Two mutual information regularization terms are introduced into the original multimodal ELBO objective, namely, maximizing the intra-modal MI and maximizing the cross-modal mutual information MI.

[0097] Preferably, the specific process of maximizing intra-modal MI is:

[0098] First, we will bias towards the joint latent variable The intra-modal MI regularization with high mutual information between individual modes is introduced into the original ELBO objective. , joint latent variable The posterior distribution sample and the observed data The MI between is defined as:

[0099] (18);

[0100] An MI variational lower bound is derived as follows:

[0101] (19);

[0102] in, is parameterized by Auxiliary distribution of deep neural networks;

[0103] The intra-modal MI maximization regularizer is defined as :

[0104] (20);

[0105] in, In the known category dataset On, In the new category dataset superior;

[0106] The cross-modal mutual information MI maximization process is:

[0107] In the known category dataset superior, Defined as:

[0108] (twenty one);

[0109] in, , , Both are used to construct negative samples;

[0110] Will With mutual information regularizer , Combined, the following goals are formed:

[0111] (twenty two);

[0112] in, is a hyperparameter that controls the influence of maximizing mutual information.

[0113] Preferably, in S4, the knowledge graph is embedded into the large language model through GCN, and the entity and relationship embeddings are converted into vectors as additional features of the model input, and are fused with the brain-vision-language multimodal data. Assume that the embedding representation of each entity and relationship in the knowledge graph is , then the expression of knowledge embedding is:

[0114] (twenty three);

[0115] in, It is the first The original representation of an entity, is the knowledge embedding function, output is the embedding vector of the corresponding entity;

[0116] The features of each modality are integrated with knowledge embedding to obtain a comprehensive feature representation. :

[0117] (twenty four).

[0118] Therefore, the present invention adopts the above-mentioned cognitive state interpretable method based on the brain-language-vision large model, and the beneficial effects are as follows:

[0119] (1) Inspired by the physical and functional hierarchical characteristics of the brain, this paper constructs a brain-like graph network architecture, defines regions of interest (ROIs) as nodes in the graph, and establishes edges between nodes based on the strength of functional connectivity to capture the functional relationships between brain regions; and designs a bottom-up space-time path to extract spatial features between ROIs through a graph neural network, and combines multi-scale temporal information to better reflect the dynamic changes of the brain in different cognitive tasks, thereby learning an optimal graph network model in the fMRI training dataset. , to achieve accurate classification of learners’ different cognitive states (such as memory, understanding, application, reasoning, analysis, evaluation, creation, and automation).

[0120] (2) The present invention jointly learns brain-vision-language features, combines visual semantic knowledge with language semantic knowledge, and improves the generalization performance of the neural decoding model. (including brain activity features, visual features, and text features) as well as data of unknown categories (Only visual features and text features) are included to build a dataset. Feature preprocessing is performed. First, the stable selection technique is used to screen out brain region voxels with high consistency from the fMRI data. Then, principal component analysis (PCA) is performed on the training data of known categories to reduce the dimension. At the same time, visual features are extracted and reduced in dimension through a pre-trained convolutional network, and the text is averaged and pooled to generate sentence embeddings.

[0121] (3) The present invention adopts a multimodal autoencoder variational Bayesian learning framework, which converts single modality features into joint latent variables through three modality-specific encoding networks, and then reconstructs the features through a decoding network; and introduces intra-modality and inter-modality mutual information regularization, while maximizing the mutual information of the two levels, enhancing the learning efficiency of the model, maintaining a strong correlation between the joint representation and the observed data, and achieving more efficient and accurate decoding of cognitive states, which helps to identify unseen cognitive categories during testing.

[0122] (4) The present invention uses only the decoder LLM as a generative model and uses the autoregressive mechanism to predict the text sequence, which can predict the subsequent content under the condition of a given prefix. , through the mapping function Tokenizing discrete text into vectors enables the model to understand natural language input.

[0123] In order to achieve the integration of multimodal information, the present invention trains an encoder , the continuously observed brain data and visual image data ( ) is converted into a multi-vector sequence of the same dimension as the text embedding vector; then, these observation vectors are alternately combined with the text embedding vector in the prefix, thereby realizing the injection of multimodal information without changing the original structure of LLM; this method dynamically embeds the observation results into the context, enabling the model to perform embodied reasoning and prediction based on the multimodal context.

[0124] (5) The present invention defines cognitive state categories and processes different models through cyclic iterative processing to extract information from each category; with the support of multimodal data, ExtractMultiModal(TempWds, WdMultiModalSegment, modality) is used to extract feature information from different modalities, including text, images and fMRI resting-state brain imaging data; at the same time, a variety of large model languages ​​are used to extract different cognitive states with the help of two prompting methods: natural prompting and forced prompting. When the model output is obviously inconsistent with the actual state, the model is forced to adjust the output to ensure the accuracy and reliability of the recognition result.

[0125] (6) This paper proposes an innovative multimodal knowledge reasoning framework MultiCoT. The knowledge-driven chain reasoning generated by this framework supports reasoning tasks of visual, language and brain data without the need for explicit knowledge sources to explain the reasoning process. This reasoning mechanism not only improves the ability to predict the learner's cognitive state, but also achieves a deeper knowledge integration and understanding. In the reasoning step, the input multimodal fusion features are converted into high-quality prompts to guide the large language model to extract implicit knowledge, providing important support for reasoning generation.

[0126] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0127] Figure 1 It is an overall flow chart of an embodiment of a cognitive state interpretable method based on a brain-language-vision macro model of the present invention;

[0128] Figure 2It is a schematic diagram of a learner's cognitive brain data representation process of a multi-scale graph convolutional network in an embodiment of a cognitive state interpretable method based on a brain-language-vision large model of the present invention;

[0129] Figure 3 It is a schematic diagram of brain-image-text joint multimodal temporal feature fusion in an embodiment of a cognitive state interpretable method based on a brain-language-vision large model of the present invention;

[0130] Figure 4 It is a schematic diagram of the architecture of an embodiment of a cognitive state interpretable method based on a brain-language-vision macro model of the present invention;

[0131] Figure 5 It is a thought chain reasoning instruction of an embodiment of a cognitive state interpretable method based on a brain-language-vision large model of the present invention;

[0132] Figure 6 It is an example of thought chain reasoning of an embodiment of a cognitive state explainable method based on the brain-language-vision large model of the present invention. DETAILED DESCRIPTION

[0133] The technical solution of the present invention is further described below through the accompanying drawings and embodiments.

[0134] Unless otherwise defined, technical or scientific terms used in the present invention shall have the common meanings understood by one having ordinary skills in the field to which the present invention belongs.

[0135] like Figure 1 As shown, a cognitive state interpretable method based on the brain-language-vision big model includes the following steps:

[0136] S1. Collect multimodal data on learners’ cognitive status;

[0137] In a classroom task, most learners' learning tasks can induce eight cognitive states of learners, including memory, understanding, application, reasoning, analysis, evaluation, creation and automation. Multimodal data collection includes brain data collection, visual data collection and language data collection;

[0138] 1) The specific process of brain data collection is as follows:

[0139] High-resolution imaging technology (fMRI) was used to collect resting-state brain imaging data of learners, and the collected fMRI resting-state brain imaging data were preprocessed into Input matrix; input a new spatiotemporal pyramid graph convolution module to output the feature representation of brain data , and then through the trained encoder Will continue to observe Mapping (brain features) to word vector space in the Large Language Model (LLM) , and insert the formed word vector into the input sequence of the language model.

[0140] 2) The specific process of visual data collection is as follows:

[0141] Record videos according to the learner's classroom learning scenario, extract video frames at fixed intervals, process each static image, and perform basic image processing, including image preprocessing and feature enhancement. Perform learner behavior recognition, detect and extract learner facial features for expression analysis, estimate the learner's body and head state through a posture estimation algorithm, and perform gesture detection. The present invention trains an encoder Will continue to observe (Visual features) are mapped to the word vector space in the Large Language Model (LLM) , and insert these word vectors into the input sequence of the language model.

[0142] 3) The language data collection process is as follows:

[0143] Collect learners' learning materials and text data generated during the learning process, as well as teacher-student and student-student classroom dialogue data. Represents a specific word or subword produced by the learner, using a finite set Represents a vocabulary of all possible words or subwords, maps discrete words to a continuous vector space, and preprocesses them into a vector space of size The large embedding matrix is ​​trained in an end-to-end manner.

[0144] S2, cognitive state representation based on multi-scale graph convolutional network;

[0145] Decoding the learner's brain state is a typical classification problem, which is based on the fMRI resting-state brain imaging data of learners during different learning tasks in the classroom (such as recitation, reading comprehension, knowledge application, etc.) to decode cognitive states, that is, to judge the learner's current cognitive state through fMRI resting-state brain imaging during each learning task period. For example, in the recitation task stage, it may correspond to the cognitive state of "memory"; in the reading comprehension task, it may correspond to the "understanding" or "analysis" state.

[0146] Specifically, the fMRI resting-state brain imaging data is labeled and expressed as ,in Represents the feature matrix of fMRI resting-state brain imaging data related to cognitive state (used to represent the activities of different brain regions), including ROIs and time samples, From the tag set The cognitive state label is used to indicate the type of cognitive state corresponding to the fMRI resting-state brain imaging data. is the number of samples in the dataset;

[0147] The purpose of this invention is to learn an optimal graph network from a training data set , and based on each fMRI resting-state brain imaging data in the test dataset Prediction task label:

[0148] (1).

[0149] The overall structure of the cognitive state representation brain decoding method based on multi-scale graph convolutional network proposed in this invention is as follows: Figure 2 Specifically, the framework includes the following two modules:

[0150] S21, brain map construction;

[0151] like Figure 2 As shown in the upper part (a), in order to construct the brain map , using a standard brain atlas to divide the brain into The functional connectivity FC of each individual between brain regions is calculated by fMRI resting-state brain imaging data. The functional connectivity of the resting state can be used as a good indicator of the human brain structural connection group to a certain extent. On this basis, the present invention first binarizes the average functional connectivity matrix and then uses Graph building strategy, building brain graph ;

[0152] Among them, the node are regions of interest ROIs, and edges Represents the connection relationship between nodes.

[0153] but The brain activity of a specific ROI at a time point is calculated by averaging all voxels within the ROI, assuming that there are learner, The brain functional connectivity matrix of each learner The Pearson correlation of fMRI resting-state brain imaging data between ROIs was calculated, and then the adjacency matrix of all learners was averaged to obtain the group-level adjacency matrix as follows:

[0154] (2);

[0155] The present invention constructs a brain map with undirected edges for a specific ROI, connecting it with the top correlation rankings in other ROIs. The area is called Brain map construction strategy". That is, from the group level adjacency matrix Prune the connections in the middle to obtain an undirected group-level adjacency matrix with binary Brain map with features :

[0156] (3);

[0157] in, Indicates ROI and The connection strength between ROIs, Indicates The first order of connection strength in the ROI is the binary adjacency matrix, indicating the ROI and Check whether there is a connection between the ROIs.

[0158] S22, Brain Decoding;

[0159] Collect blood oxygen level dependent signal data BOLD from functional magnetic resonance imaging. These signals represent the brain activity patterns when learners complete various cognitive tasks (such as memory, understanding, application, reasoning, etc.) in class. Each signal data BOLD is used to reflect the dynamic response of multiple brain ROIs at different time points. These data record the activity changes of N ROIs within m frame time. The specific process is as follows:

[0160] The BOLD signal is preprocessed and converted into a feature matrix. Each row of the matrix corresponds to the activity of a brain region, and each column represents the signal of a time frame. These feature matrix information are used as the input data of the model to help analyze the activity patterns of ROIs in different cognitive states of learners.

[0161] In the feature matrix Based on this, a feature map is generated, in which the nodes of the map represent the brain regions of interest, and the edges between the nodes represent the associations between these brain regions, which are used to reflect the interconnection requirements of different cognitive tasks in the classroom for brain regions. Afterwards, the feature map is input into the spatiotemporal pyramid graph convolution module for multi-layer spatiotemporal convolution processing to capture the dynamic associations between brain regions.

[0162] like Figure 2As shown in the lower part (b), the blue line in the lower layer represents a shorter feature extraction path, which can process higher-level, faster temporal features and is suitable for processing tasks such as memory and understanding in classroom learning, which require the brain to react quickly and integrate information in a short time. The green line includes convolution and normalization layers, which can balance the capture of spatiotemporal features at different time scales and is suitable for processing data with medium-term dependencies.

[0163] For example, application and reasoning require learners to apply knowledge to new scenarios or perform logical reasoning in a short period of time, which involves a long time dependency. The brown line contains eight convolutional layers, which can process information over a longer time span and have higher expressive power, so it performs particularly well in complex tasks such as analysis, evaluation, and creation. These tasks usually involve learners thinking and integrating information for a long time, and require a higher comprehensive processing ability of spatiotemporal features.

[0164] Based on this, the novel spatiotemporal pyramid graph convolution module constructed by the present invention takes a multi-channel time series (size greater than 4) as input and outputs a spatiotemporal pyramid feature. The architecture of the spatiotemporal pyramid graph convolution module includes three spatiotemporal pathways with different scales of temporal convolution and several bottom-up pathways, which are described as follows:

[0165] ① Space-time pathway:

[0166] The spatiotemporal pathway aims to pursue unique representations along different temporal dimensions while considering the correlations between spatial features. Figure 2 As shown in (b), the new spatiotemporal pyramid graph convolution module contains three spatiotemporal pathways, each of which includes a time-gated convolution module and a spatial convolution module:

[0167] Spatial convolution module:

[0168] Spatial Convolution Module Defined as the input vector With filter The product of:

[0169] (4);

[0170] in, and They are the normalized graph Laplacian matrices The eigenvector matrix and the diagonal matrix of eigenvalues ​​are ;

[0171] Graph Laplacian Matrix Through the adjacency matrix The conversion is obtained, expressed as:

[0172] ,in is the degree matrix, whose diagonal elements are , is the identity matrix, and the adjacency matrix is ​​obtained from fMRI resting-state brain imaging data .

[0173] In order to reduce the computational complexity, the Chebyshev polynomial is used Approximate Filter , therefore, the spatial graph convolution module can be reformulated as a Chebyshev polynomial A linear function of:

[0174] (5);

[0175] in, , express The maximum eigenvalue of the present invention is set by introducing a first-order approximation and ,Will and Substituting into formula (5), the spatial graph convolution is approximately expressed as:

[0176] (6).

[0177] Time-gated convolution module:

[0178] The time-gated convolution module designed in the present invention is a convolution module with a width of A one-dimensional convolution kernel is followed by a gated linear unit GLU to increase nonlinearity, where Indicates j The temporal convolution kernel in the spatiotemporal path, the input of each node in the graph can be expressed as ,in Indicates the input length, Indicates the number of input channels, time convolution kernel ,Will Mapped to an output element , and Divided into two parts with the same number of channels, where Indicates the number of output channels;

[0179] Then the time-gated convolution module is expressed as:

[0180] (7);

[0181] in, represents temporal convolution; and They are the inputs of the gated unit in GLU respectively; ⊙ represents the element-wise Hadamard product; Represents sigmoid gating, which is used to determine the input The importance of.

[0182] ② Bottom-up pathway

[0183] In the "bottom-up pathway", each path connects the fine-grained and coarse-grained features at different spatiotemporal scales layer by layer and transmits them upward. The spatial features of the bottom path are transmitted to the top path through the gradual integration of multi-scale information, forming a feature representation with global and local perspectives, thereby generating multi-level semantic information.

[0184] Inspired by this mechanism, the "bottom-up pathway" is used to enhance the adaptability of the model in complex cognitive tasks. Through this mechanism, the model can effectively capture the multi-level information required for different cognitive tasks, such as focusing on fine-grained instantaneous activities in tasks such as memory and understanding, and focusing more on integrating the long-term linkage of multiple brain regions in tasks such as analysis and creation. In addition, the one-dimensional convolution operation after each concatenation is specifically used to process the concatenated features of each node, so that the fused features remain consistent in the node dimension, thereby ensuring spatial fidelity. This processing method ensures the integrity of spatial information during feature transmission, helping the model to more accurately decode the brain activity patterns of learners when performing different tasks in a classroom environment.

[0185] The present invention also includes a method for predicting a text sequence by only decoding large language models LLMs. Probability , where the text sequence Represents a sequence of tokens , M represents the total length of the text sequence. This process can help analyze text data and dialogue data generated in different learning tasks to support cognitive state analysis.

[0186] By factorizing a typical neural network architecture we get:

[0187] (8);

[0188] in, It is a large Transformer network.

[0189] Since LLM is autoregressive, a pre-trained model can be obtained by prefixing it without changing the architecture. To make a conditional forecast:

[0190] (9);

[0191] Prefix Used to provide LLM continuation prediction follow-up marks The context on which the LLM is based, which is typically used for reasoning to guide the model's predictions. For example, the prompt can contain a description of the task that the LLM should solve, or examples of text completion required for similar tasks.

[0192] In order to convert words in natural language into vector representations that can be processed by the large language model LLM, the following mapping is performed:

[0193] definition To indicate a specific word or subword, use Represents a finite set of discrete words. Internally, LLM maps Will Embedded into word token embedding space In ,in , mapping Usually expressed as a size of , and train it end-to-end.

[0194] Text embedding based on continuous observation of multimodal data including learner brain data and visual image data. The specific process is as follows:

[0195] Since the recognition of learner cognitive state requires input of data of different modalities for accurate recognition, the present invention is an encoder Different architecture choices are proposed to directly map continuously observed multimodal data (brain features, visual image features) to the language embedding space of the large language model instead of discretizing them. middle:

[0196] (10);

[0197] The continuous observation embedding vectors (brain features, visual image features) are alternately arranged with the normal text tag embedding to form a prefix (i.e., the initial information input to the LLM), which means that each vector in the prefix All from word token embedders or encoder :

[0198] (11);

[0199] An observation Encoded into multiple embedding vectors, different encoders can be used alternately at different positions of the prefix , to combine information from different observation spaces (e.g. brain imaging, vision, etc.).

[0200] Since the tag embedding generated by brain decoding data and image information is inconsistent with the text dimension of the large language model, the present invention adopts a self-attention mechanism and a visual transformer ViT to extract tags from brain decoding data and 2D images respectively and embed them into the large language model LLM.

[0201] Among them, the self-attention mechanism is used to input brain ROIs and spatiotemporal sequence feature dimensions into the large language model. , N represents the number of regions of interest ROIs, d represents the feature dimension of a specific ROI, then the number of features extracted from fMRI resting-state brain imaging data is:

[0202] (12);

[0203] (13);

[0204] (14);

[0205] in, is the brain feature vector obtained by convolutional decoding of the spatiotemporal pyramid graph, , , is a weight matrix mapping to query, key, and value, is the dimension of the key, used for the scaling factor.

[0206] It is a method that can transform images The invention introduces affine transformation The embeddings generated by ViT are adjusted to match the dimensions of the language model. Specifically, for each image tag embedding ,pass Transformed into a vector representation suitable for language model .

[0207] This method of multimodal information injection can achieve more natural information fusion in the model, which is particularly suitable for multimodal tasks. It enables the model to understand brain and visual information at the same time and use the existing structure of the language model for information processing.

[0208] S3. Brain-vision-language hybrid multimodal temporal feature fusion based on hierarchical mutual information maximization.

[0209] In order to integrate brain data, vision, and language data, this paper constructs a multimodal autoencoder variational Bayesian model that can jointly utilize visual and text features to enhance brain activity representation learning and new category neural decoding performance. Specifically, three modality-specific encoding networks are used: , , the unimodal feature , Transformed into a joint latent variable , then passes through three modality-specific decoding networks , In the classroom, the model can effectively integrate information from different modalities and provide learners with a more comprehensive understanding of the brain, vision, and language text for cognitive state recognition.

[0210] The multimodal joint modeling process is:

[0211] The variational autoencoder VAE latent variable model is used for scalable unsupervised representation learning. For unimodal data, VAE combines latent variables Define the marginal log-likelihood:

[0212] ;

[0213] Due to the precise estimation This is usually not feasible, so VAE instead maximizes a feasible evidence lower bound ELBO to obtain:

[0214] (15);

[0215] in, is the approximate posterior distribution represented by the encoder network, and its parameters are is the conditional distribution represented by the decoder network, which is parameterized by .

[0216] In the neural decoding task, VAE is conditioned on brain activity, visual features, and corresponding text features to form a multimodal VAE. Formally, the joint marginal log-likelihood and its evidence lower bound ELBO can be similarly expressed as:

[0217] (16);

[0218] In the seen class dataset superior, Parameterized by The decoder is composed of , ; Using variational joint posterior To approximate the joint posterior of the model , to maximize , which can be further optimized .

[0219] By applying the formula with joint latent variable learning and cross-modal generation capabilities in the neural decoding task, the above formula can be further written as:

[0220] (17);

[0221] in, yes is a possible subset of Is A new set consisting of all possible subsets of .

[0222] In multimodal VAE, the best This forces the variational joint posterior to be close to the prior distribution, regardless of How strong is the parameterization ability of . This usually leads to the famous posterior collapse problem: 1) The joint latent variable With observational data Not relevant; 2) The reconstruction of cannot benefit from the encoder, which means that the decoder can ignore the ; 3) In multimodal scenarios, the joint generation consistency between the modalities is weak. In order to alleviate the above problems, the present invention introduces two mutual information regularization terms into the original multimodal ELBO objective, namely, maximizing the intra-modal MI and maximizing the cross-modal mutual information MI.

[0223] The specific process of maximizing intra-modal MI is as follows:

[0224] First, we will bias towards the joint latent variable The intra-modal MI regularization with high mutual information between individual modes is introduced into the original ELBO objective. , joint latent variable The posterior distribution sample and the observed data The MI between is defined as:

[0225] (18);

[0226] Here, the present invention derives an accurate and concise MI variational lower bound as follows:

[0227] (19);

[0228] in, is parameterized by Auxiliary distribution of deep neural networks;

[0229] like Figure 3 As shown, Decoder The output of is used as input. = When , the lower bound is closely matched and can be optimized For simplicity, the figure omits , .

[0230] Finally, the intra-modal MI maximization regularizer is defined as :

[0231] (20);

[0232] in, In the known category dataset On, In the new category dataset superior.

[0233] The cross-modal mutual information MI maximization process is:

[0234] The above intra-modal mutual information MI regularization focuses on alleviating the problem of learning consistent and useful joint representations in multimodal VAEs. However, due to the random noise and potential overfitting problems of specific modalities, it cannot be guaranteed that the joint generation between all modalities is highly consistent. Therefore, the present invention further introduces inter-modal point-by-point MI regularization. , the regularization favors high mutual information between multiple modalities. This can suppress specific modality noise that is irrelevant to the cognitive state recognition task, improve data efficiency in the case of limited brain activity data, and thus maintain the consistency of generation as much as possible.

[0235] In the known category dataset superior, Defined as:

[0236] (twenty one);

[0237] in, , , Both are used to construct negative samples.

[0238] Overall goals and training objectives:

[0239] The present invention will With mutual information regularizer , Combined, the following goals are formed:

[0240] (twenty two);

[0241] in, is a hyperparameter that controls the influence of maximizing mutual information.

[0242] For each training mini-batch, the training process is divided into two stages. In the first stage, Maximize the objective so that the variational lower bound on the intrinsic modal mutual information fits more closely (i.e., minimize and In the second stage, we target the parameter and Maximize the objective for training with intramodal and cross-modal mutual information regularizers.

[0243] S4. Knowledge graph node embedding that enhances the ability of multimodal large models to understand complex tasks.

[0244] Using external resources (such as knowledge graphs, expert systems, etc.) to enhance or guide the data during model training can improve the model's ability to understand and generate complex tasks. The knowledge guidance mechanism can help the model better understand the semantic relationship of the input data, thereby reducing the model's dependence on context and improving its reasoning ability. In multimodal learning, it improves the model's performance in multimodal tasks by combining brain signals, visual data, and language information with domain knowledge.

[0245] In this paper, the knowledge graph is embedded into the large language model through GCN, and the entity and relationship embeddings are converted into vectors as additional features of the model input, and are fused with brain-vision-language multimodal data to obtain the joint latent variable Assume that the embedding representation of each entity and relationship in the knowledge graph is , then the expression of knowledge embedding is:

[0246] (twenty three);

[0247] in, It is the first The original representation of an entity, is the knowledge embedding function, output is the embedding vector of the corresponding entity.

[0248] The features of each modality are integrated with knowledge embedding to obtain a comprehensive feature representation. :

[0249] (twenty four).

[0250] In this way, the large language model formed by combining the knowledge graph not only judges the cognitive state based on the text description formed by the vector features, but also judges based on these texts, images, brain data and the relevant knowledge they involve, which improves the interpretability of the model while enhancing the recognition performance of the model.

[0251] S5. Knowledge-guided brain-vision-language multimodal large model training Figure 4 As shown, the specific process is as follows:

[0252] 1) Multimodal encoder

[0253] Multimodal encoder Apply the above data collection and data fusion method to input information Combined into joint feature representation , the formula is as follows:

[0254] (25).

[0255] 2) Input projector

[0256] Through the input projector Combine multimodal encoding features with text feature space Alignment, aligned features as hints , and then with the text features Input into the LLM backbone together, given a text dataset The goal is to minimize the text generation loss under the condition of fusion feature alignment of brain data, visual data, and language data. :

[0257] (26);

[0258] in .

[0259] The input projector can be implemented directly through linear projection or multi-layer perceptron (MLP), or through more complex ways such as Cross-attention, Q-former, MQ-former, etc.

[0260] 3) LLM large model backbone

[0261] The multimodal large language model with the large language model as the core agent can perform zero-sample generalization, few-sample self-learning, chain thinking and command following. The knowledge-guided brain-vision-language multimodal large model proposed in this invention can process representations from multimodalities, perform semantic understanding, reasoning and decision-making, and can output direct text output. , signal markers for other modes These tags serve as instructions to guide the generator whether it needs to generate multimodal content.

[0262] (27);

[0263] Among them, the alignment representation of other modalities It can be considered as a soft hint tuning for LLM.

[0264] 4) Output to projector

[0265] Output Projector The signal generated by the LLM backbone network is marked Mapping to Features , so that the subsequent modal generator Understandable. Given mixed features Text dataset In the Input into LLM to generate the corresponding , which is then mapped to In order to make the mapped features Alignment, the goal is to minimize and with The conditional text represents the distance between:

[0266] (28);

[0267] in, ,and yes The output projector is implemented as a tiny Transformer or MLP with a learnable decoder feature sequence.

[0268] 5) Multimodal Generator

[0269] Modal Generator Responsible for generating outputs of different modalities. Output projector mapping features As a conditional input in the denoising process to generate multimodal content. During the training process, the real content is first converted into latent variables through the pre-trained VAE , then Add noise Obtaining noise latent features . Pre-trained Unet Used to calculate the conditional LDM loss As shown below:

[0270] (29);

[0271] This loss is minimized by To optimize the parameters and .

[0272] S6. Identification of learner cognitive states based on multimodal large models;

[0273] In order to identify the learner's cognitive state and extract relevant information from the multimodal data features, the present invention first defines a variety of cognitive state categories such as 'MEMORY', 'UNDERSTANDING', 'APPLICATION', 'REASONING', 'ANALYSIS', 'EVALUATION', 'CREATION', 'AUTOMATION', i.e., memory, understanding, application, reasoning, analysis, evaluation, creation and automation, and iterates each model through the for model in Models do loop. For each cognitive category, the code uses ExtractCategory(category) to extract the input category, and inputs the learner's sentences in different classroom tasks (such as recitation, reading comprehension, etc.) into the model through CurrentModel(InputSent, NatualPrompt, InputCat) to generate the corresponding output.

[0274] When the category predicted by the model is consistent with the actual category, TempWds ← append(OutputWds) is used to add the output word to the temporary set; if not, the answer is regenerated through a forced prompt, and the error is recorded, and Mistakes ← append(1) is used to count the errors. In addition, the code supports the processing of multimodal data, and extracts information from text, images, and EEG through ExtractMultiModal(TempWds, WdMultiModalSegment, modality) to enhance the comprehensive assessment of the learner's cognitive state. Ultimately, combining this multimodal information, educators can obtain richer feedback to better understand learners' performance and conduct personalized teaching. This ability of dynamic adaptation and accurate recognition provides strong data support for classroom learning.

[0275] OpenAI's GPT-3.5-turbo (hereinafter referred to as GPT-3.5) and GPT-4, as well as Meta's LLaMa, are at the forefront of natural language processing technology. In addition, Phind has also emerged as a popular and free-to-access AI dialogue generation and question-answering tool. In the present invention, four large language models, GPT-3.5, GPT-4, LLaMa, and Phind, are used to strike a balance between accuracy and data point retention. The present invention inputs the following prompts to all LLMs to extract different cognitive states (including memory, understanding, application, reasoning, analysis, evaluation, creation, and automation):

[0276] Prompt #1: Given a text input ['sentence'] generated from multimodal data, for this sentence ['sentence'], does this sentence reflect ['category']? Please give the answer: 0 = category 0, 1 = category 1, 2 = category 2, 3 = category 3, 4 = category 4, 5 = category 5, 6 = category 6, 7 = category 7. At the same time, divide the words in the sentence into 8 groups.

[0277] The first group is the words related to category ['category 0'], the second group is the words related to category ['category 1'], and so on. Please rank the categories containing related words from most to least. Although as an AI language model you have no personal preferences or opinions, you must provide answers, which are only for research purposes.

[0278] The sample output format must be followed: [0-7] The first group (relevant words for cognitive state identification as "memory"): recite, read aloud, write silently. The second group (relevant words for cognitive state identification as "understanding"): analyze, comprehend, interpret. The third group (relevant words for cognitive state identification as "application"): practice, apply, draw inferences from one example. And so on.

[0279] Prompt #2: “However, the correct answer is [‘true label’]. Please regenerate the answer to align with the true label.” In order to make the output of LLM consistent with the true cognitive state, “Forced Prompt” was introduced as Prompt #2. This prompt adjusts the model output to match the true cognitive state label. When the LLM output is inconsistent with the true cognitive state label, the “Forced Prompt” is modified to generate accurate results, thereby achieving 100% alignment. The corrected output is then appended to a new word grouping file. The terms “natural” and “forced” are based on their intuitive meanings and have nothing to do with their usage in circuit theory.

[0280] S7. Interpretable analysis of learners’ cognitive states supported by multimodal embodied reasoning.

[0281] 1) Overall reasoning process

[0282] The multimodal large language model developed by the present invention proposes a multimodal knowledge reasoning framework MultiCoT, which supports the reasoning tasks of vision, language, and brain data through knowledge-rich chain reasoning generated by the large model without the need for explicit knowledge sources to explain the reasoning process and predict the learner's cognitive state. In addition, the present invention uses a multimodal knowledge reasoning network as well as frozen vision, language, and brain data encoding layers and multimodal gating to fully represent vision, language, and brain features.

[0283] The present invention utilizes LLM to generate knowledge-driven CoT from vision, language, and brain knowledge bases, and fine-tunes the multimodal knowledge reasoning model with parameters, thereby providing high-quality human-interpretable insights into the knowledge reasoning ability of the language model.

[0284] The reasoning process is formalized as follows: given an input query And multimodal fusion features , the reasoner constructs multimodal instruction samples. Given the input multimodal fusion features and query , the reasoner constructs multimodal instruction samples. A sample contains instructions , multimodal input , and the “real” reasoning generated by the large model and PromptCap module , MultiCoT generates reasoning reasons And predict the answer based on multimodal input and instructions:

[0285] (30);

[0286] in To predict the answer, the question and the true answer are represented as and , Represents the trainable parameters of the model.

[0287] 2) Reasoning steps

[0288] Multimodal input Transformed into a prompt command, command like Figure 5 As shown in the figure, "Please give me some inferences based on the above content", guiding the large language model to extract implicit knowledge. The present invention uses the PromptCap model to convert each image into an image description and converts the fMRI resting state brain imaging data scan data into brain data description, such as Figure 6 This can provide important support for generating content through chain reasoning.

[0289] Considering that each input case contains fused features, the decoder layer Operate on these features. This transform decoder layer helps generate knowledge-rich inferences and ensures that the predicted answers are more representative. The inferences and predicted answers are shown below:

[0290] (31);

[0291] in, It is the fusion feature representation.

[0292] Therefore, the present invention adopts the above-mentioned cognitive state interpretable method based on the brain-language-vision large model, which not only improves the predictive ability of learners' cognitive state, but also achieves a deeper knowledge integration and understanding. In the reasoning step, the input multimodal fusion features are converted into high-quality prompts to guide the large language model to extract implicit knowledge, providing important support for reasoning generation.

[0293] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solution of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solution to deviate from the spirit and scope of the technical solution of the present invention.

Claims

1. A cognitive state interpretable method based on the brain-language-vision macro model, characterized in that: The following steps are involved: S1. Collect multimodal data on learners’ cognitive status; S2, cognitive state representation based on multi-scale graph convolutional network; S3, Brain-vision-language hybrid multimodal temporal feature fusion based on hierarchical mutual information maximization; S4, knowledge graph node embedding to enhance the ability of multimodal large models to understand complex tasks; S5. Knowledge-guided brain-vision-language multimodal large model training; S6. Identification of learner cognitive states based on multimodal large models; S7. Interpretable analysis of learners’ cognitive states supported by multimodal embodied reasoning; In S3, a multimodal autoencoder variational Bayesian model is constructed for multimodal feature fusion, using three modality-specific encoding networks , the unimodal feature Transformed into a joint latent variable , then passes through three modality-specific decoding networks Perform feature reconstruction respectively; The multimodal joint modeling process is as follows: a variational autoencoder (VAE) latent variable model is used for scalable unsupervised representation learning. For unimodal data, VAE combines latent variables Define the marginal log-likelihood: ； VAE maximizes the evidence lower bound ELBO to obtain: ； （1）； in, is the approximate posterior distribution represented by the encoder network, and its parameters are , is the conditional distribution represented by the decoder network, which is parameterized by ; The joint marginal log-likelihood and its evidence lower bound ELBO are expressed as: （2）； Parameterized by The decoder is composed of , ; Using the variational joint posterior To approximate the joint posterior of the model ,maximize , further optimize ; By applying the formula with joint latent variable learning and cross-modal generation capabilities in the neural decoding task, the above formula can be further written as: （3）； in, yes is a possible subset of Is A new set consisting of all possible subsets of ; Two mutual information regularization terms are introduced into the original multimodal ELBO objective, namely, maximizing the intra-modal MI and maximizing the cross-modal mutual information MI.

2. A cognitive state interpretable method based on the brain-language-vision large model according to claim 1, characterized in that: In S1, learners’ cognitive states include memory, understanding, application, reasoning, analysis, evaluation, creation, and automation, and multimodal data collection includes brain data collection, visual data collection, and language data collection; The specific process of brain data collection is as follows: High-resolution imaging technology fMRI is used to collect learners' resting-state brain imaging data, and the collected fMRI resting-state brain imaging data are preprocessed into The input matrix is represents the number of ROIs in the learner's brain, Indicates the number of time samples; The visual data collection process is as follows: Record videos based on learners’ classroom learning scenarios, extract video frames at fixed intervals, and process each static image; The language data collection process is as follows: Collect learners' learning materials and text data generated during the learning process, as well as teacher-student and student-student classroom dialogue data. Represents a specific word or subword produced by the learner, using a finite set Represents a vocabulary of all possible words or subwords, maps discrete words to a continuous vector space, and preprocesses them into a vector space of size The large embedding matrix of Represents the embedding dimension.

3. A cognitive state interpretable method based on the brain-language-vision large model according to claim 2, characterized in that: In S2, the learner's current cognitive state is determined by fMRI resting-state brain imaging during each learning task period: The fMRI resting-state brain imaging data is labeled as ,in Represents the feature matrix of fMRI resting-state brain imaging data related to cognitive state, including ROIs and time samples, From the tag set The cognitive state label is used to indicate the type of cognitive state corresponding to the fMRI resting-state brain imaging data. is the number of samples in the dataset; Learning an optimal graph network from a training dataset , and based on each fMRI resting-state brain imaging data in the test dataset Prediction task label: （4）。 4. The cognitive state interpretable method based on the brain-language-vision large model according to claim 3 is characterized in that: The cognitive state representation based on multi-scale graph convolutional network in S2 includes the following two modules: S21, brain map construction; The brain is divided into The functional connectivity FC of each individual in each brain region was calculated by fMRI resting-state brain imaging data, and the average functional connectivity matrix was binarized and then used Graph building strategy, building brain graph ; Among them, the node ROIs are regions of interest, Indicates the connection relationship between nodes; but The brain activity of a specific ROI at a time point is calculated by averaging all voxels within the ROI, assuming that there are learner, The brain functional connectivity matrix of each learner The Pearson correlation of fMRI resting-state brain imaging data between ROIs was calculated, and then the adjacency matrix of all learners was averaged to obtain the group-level adjacency matrix as follows: （5）； Construct a brain map with undirected edges for a specific ROI, connecting it to the top correlation rankings in other ROIs The area, from the group-level adjacency matrix Prune the connections in the middle to obtain an undirected group-level adjacency matrix with binary Brain map with features : （6）； in, Indicates ROI and The connection strength between ROIs, Indicates The first Large value, is the binary adjacency matrix, indicating the ROI and Whether there is a connection between the ROIs; S22, Brain Decoding; Collect blood oxygen level dependent signal data BOLD from functional magnetic resonance imaging. Each signal data BOLD is used to reflect the dynamic response of multiple brain region ROIs at different time points. Record the activity changes of N ROIs within m frame time. The specific process is as follows: The BOLD signal is converted into a feature matrix after preprocessing, and the feature matrix information is used as the input data of the model to analyze the activity patterns of ROIs in different cognitive states of learners; In the feature matrix Based on this, a feature graph is generated, where the nodes of the graph represent the brain regions of interest, and the edges between the nodes represent the associations between these brain regions, which is used to reflect the interconnection requirements of different cognitive tasks in the classroom on brain regions. The feature map is input into the spatiotemporal pyramid graph convolution module for multi-layer spatiotemporal convolution processing to capture the dynamic correlation between brain regions.

5. The cognitive state interpretable method based on the brain-language-vision large model according to claim 4 is characterized in that: The spatiotemporal pyramid graph convolution module in S22 includes three spatiotemporal pathways with temporal convolutions of different scales and several bottom-up pathways: The spatiotemporal pathway includes a time-gated convolution module and a spatial convolution module. is the input vector With filter The product of: （7）； in, and They are the normalized graph Laplacian matrices is the diagonal matrix of eigenvectors and eigenvalues, and ; Graph Laplacian Matrix Through the adjacency matrix The conversion is obtained, expressed as: ,in is the degree matrix, whose diagonal elements are , is the identity matrix, and the adjacency matrix is ​​obtained from fMRI resting-state brain imaging data ; Using Chebyshev polynomials Approximate Filter , reformulating the spatial graph convolution module as Chebyshev polynomials A linear function of: （8）； in, , express The maximum eigenvalue of and ,Will and Substituting into formula (8), the spatial graph convolution is approximately expressed as: ； （9）； ； The time-gated convolution module is a network with a width of A one-dimensional convolution kernel followed by a gated linear unit GLU, where Indicates The temporal convolution kernel in the spatiotemporal path, the input of each node in the figure is represented as ,in Indicates the input length, Indicates the number of input channels, time convolution kernel ,Will Mapped to an output element , and Divided into two parts with the same number of channels, where Indicates the number of output channels; Then the time-gated convolution module is expressed as: （10）； in, represents temporal convolution; and They are the inputs of the gate control unit in GLU; represents the element-wise Hadamard product; Represents sigmoid gating, which is used to determine the input The importance of In the bottom-up pathway, each path connects the fine-grained and coarse-grained features at different spatiotemporal scales layer by layer and transmits them upward. The spatial features of the bottom path are transmitted to the top path through the gradual integration of multi-scale information, forming a feature representation with global and local perspectives, thereby generating multi-level semantic information.

6. The cognitive state interpretable method based on the brain-language-vision large model according to claim 5, characterized in that: Before S3, it also included predicting a text sequence by only decoding large language models LLMs. Probability , where the text sequence Represents a sequence of tokens , M represents the total length of the text sequence; By factorizing the neural network architecture we get: （11）； in, It is a Transformer network; The pre-trained model is prefixed without changing the architecture. To make a conditional forecast: （12）； Prefix Used to provide LLM continuation prediction follow-up marks the context in which it is based; Convert words in natural language into vector representations processed by the large language model LLM and perform the following mapping: definition To indicate a specific word or subword, use Represents a finite set of discrete words. Internally, LLM maps Will Embedded into word token embedding space In ,in , mapping Usually expressed as a size of A large embedding matrix.

7. The cognitive state interpretable method based on the brain-language-vision large model according to claim 6 is characterized in that: Text embedding based on continuous observation of multimodal data including learner brain data and visual image data. The specific process is as follows: For encoder Different architectural choices are proposed to directly map continuously observed multimodal data to the language embedding space of a large language model instead of discretizing it. middle: （13）； The continuous observation embedding vectors are alternately arranged with the normal text token embedding to form a prefix, that is, each vector in the prefix All from word token embedders or encoder : （14）； Observation Encoded into multiple embedding vectors, using different encoders alternately at different positions of the prefix , to combine information from different observation spaces; The self-attention mechanism and visual transformer ViT are used to extract tags from brain decoding data and 2D images and embed them into the large language model LLM respectively; Among them, the self-attention mechanism is used to input brain ROIs and spatiotemporal sequence feature dimensions into the large language model. , N represents the number of regions of interest ROIs, d represents the feature dimension of a specific ROI, then the number of features extracted from fMRI resting-state brain imaging data is: （15）； （16）； （17）； in, is the brain feature vector obtained by convolutional decoding of the spatiotemporal pyramid graph, , , is a weight matrix mapping to query, key, and value, is the dimension of the key, used for the scaling factor; Introducing affine transformation The embeddings generated by ViT are adjusted to match the dimensions of the language model, and for each image, the embedding is labeled ,pass Transformed into a vector representation suitable for language model .

8. The cognitive state interpretable method based on the brain-language-vision large model according to claim 7, characterized in that: The specific process of maximizing intra-modal MI is as follows: First, we will bias towards the joint latent variable The intra-modal MI regularization with high mutual information between individual modes is introduced into the original ELBO objective. , joint latent variable The posterior distribution sample and the observed data The MI between is defined as: （18）； An MI variational lower bound is derived as follows: （19）； in, is parameterized by Auxiliary distribution of deep neural networks; The intra-modal MI maximization regularizer is defined as : （20）； in, In the known category dataset On, In the new category dataset superior; The cross-modal mutual information MI maximization process is: In the known category dataset superior, Defined as: ； ； ； （21）； in, , , Both are used to construct negative samples; Will With mutual information regularizer , Combined, the following goals are formed: （22）； in, is a hyperparameter that controls the influence of maximizing mutual information.

9. The cognitive state interpretable method based on the brain-language-vision large model according to claim 8, characterized in that: In S4, the knowledge graph is embedded into the large language model through GCN, and the entity and relationship embeddings are converted into vectors as additional features of the model input, and fused with brain-vision-language multimodal data. Assume that the embedding representation of each entity and relationship in the knowledge graph is , then the expression of knowledge embedding is: （23）； in, It is the first The original representation of an entity, is the knowledge embedding function, output is the embedding vector of the corresponding entity; The features of each modality are integrated with knowledge embedding to obtain a comprehensive feature representation. : （24）。

Images