Construction method and system of multi-modal mixed teaching resources based on cognitive neuroscience

Abstract

The application discloses a multi-modal mixed teaching resource construction method and system based on cognitive neuroscience, and the method comprises the following steps: acquiring a hidden vector of static and dynamic data in a teaching environment; constructing consistent semantic information with brain data based on the brain data of students and the hidden vector of static and dynamic data; performing denoising consistency loss processing on a backbone network Unet to construct a teaching resource generation model embedded with brain science; constructing a teaching resource generation model for an actual teaching scene based on the actual teaching environment style; inputting the static data in the teaching environment and the brain data of students into the teaching resource generation model to perform load prediction, and obtaining a teaching resource-cognitive load prediction model. The application can generate teaching resources according to learning and teaching targets and teaching contents, and reduce the cognitive load of students. The application can be widely applied to the technical field of intelligent auxiliary teaching as a multi-modal mixed teaching resource construction method and system based on cognitive neuroscience.

Description

Technical Field

[0001] This invention relates to the field of intelligent assisted teaching technology, and in particular to a method and system for constructing multimodal hybrid teaching resources based on cognitive neuroscience. Background Technology

[0002] Teaching resources that align with students' cognitive development can reduce cognitive load, promote teaching innovation, enrich students' learning experience, and improve the quality of education. Currently, most teaching resources used in actual classrooms are pre-designed and do not consider students' cognitive load or other quantitative information during actual learning. This results in a lack of interactivity, flexibility, and personalization, failing to be student-centered and leading to low efficiency in the classroom. Existing teaching methods often incorporate deep neural network technology, which has made breakthroughs in multimodal data generation modeling. Compared to previous shallow learning methods, these methods can support the embedding of more complex supervised information, integrate deeper semantic information, and are more stable and realistic. However, relying solely on deep neural networks to generate teaching resources lacks effective supervised information; it cannot adapt to different learning objectives and content, and therefore cannot effectively and stably reduce students' cognitive load. Summary of the Invention

[0003] To address the aforementioned technical problems, the present invention aims to provide a method and system for constructing multimodal hybrid teaching resources based on cognitive neuroscience. This method can utilize multimodal teaching data embedding from actual teaching scenarios, and use exogenous EEG signals as modality alignment centers and instrumental variables to explore causal essence, thereby obtaining the debiased encoding of each modality encoder. This enables a generative model led by brain science and assisted by other modalities, thereby effectively improving the accuracy, interpretability, and robustness of personalized teaching resource generation.

[0004] The first technical solution adopted in this invention is: a method for constructing multimodal hybrid teaching resources based on cognitive neuroscience, comprising the following steps:

[0005] The dynamic and static data in the teaching environment are acquired and encoded to obtain the latent vectors of the static and dynamic data.

[0006] The student's brain data is acquired and semantically fused with the latent vectors of the static and dynamic data to obtain consistent semantic information with the brain data.

[0007] Based on the consistent semantic information of the brain data, the backbone network Unet is subjected to denoising consistency loss processing to construct a teaching resource generation model embedded in brain science.

[0008] The actual teaching environment style is embedded into the brain science embedded teaching resource generation model in the form of incremental pluggable components to construct a teaching resource generation model oriented towards actual teaching scenarios.

[0009] The static data in the teaching environment and the brain data of the students are input into the teaching resource generation model for actual teaching scenarios to perform load prediction, and the teaching resource generation model for actual teaching scenarios is updated according to the load prediction results to obtain the teaching resource-cognitive load prediction model.

[0010] Multimodal hybrid teaching resources are generated based on the teaching resource-cognitive load prediction model.

[0011] Furthermore, the step of encoding and processing the dynamic and static data in the teaching environment to obtain the latent vectors of the static and dynamic data specifically includes:

[0012] Acquire static data from the teaching environment, including student background information, teacher background information, and teaching material metadata.

[0013] The static data is parsed and processed using a multi-level semantic role parser to obtain semi-structured tree-like semantic data.

[0014] The semi-structured tree semantic data is positionally encoded by a bidirectional pre-trained encoder distilled from a Chinese dataset to obtain the latent vector of the static data.

[0015] Acquire dynamic data in the teaching environment, including student feedback data and teacher-student interaction data;

[0016] The dynamic data is time-series captured by QKV adaptive coding to obtain the timing information of the dynamic data;

[0017] The temporal information of the dynamic data is positionally encoded by a debiased knowledge distillation modal encoder to obtain the latent vector of the dynamic data.

[0018] The implicit vectors of the static data and the implicit vectors of the dynamic data are integrated to obtain the implicit vectors of the static and dynamic data.

[0019] Furthermore, the step of acquiring students' brain data and performing semantic fusion processing with the latent vectors of the static and dynamic data to obtain semantic information consistent with the brain data specifically includes:

[0020] Collect students' electroencephalogram (EEG) data and functional magnetic resonance imaging (fMRI) signals.

[0021] The latent vector of the static data is introduced, and the temporal semantic fusion processing of the student's EEG data signal and the student's functional magnetic resonance imaging signal is performed on the same time dimension based on the encoder to obtain the fused latent vector.

[0022] Obtain the cross-modal mutual information entropy between the fused latent vector and the latent vector of the dynamic data;

[0023] Based on the cross-modal mutual information entropy, the latent vectors of the dynamic data are debiased and aligned using the fused latent vectors to obtain consistent semantic information with brain data.

[0024] Furthermore, based on the consistent semantic information of the brain data, the backbone network Unet is subjected to denoising and consistency loss processing to construct a teaching resource generation model embedded in brain science:

[0025] Based on the dimensions of the static and dynamic data, the noise signal of the latent vector of the static and dynamic data is obtained in the standard normal distribution;

[0026] The noise signal of the latent vector of the static and dynamic data is added to the consistent semantic information with brain data to obtain the noise-added consistent semantic information.

[0027] The backbone network Unet is subjected to denoising consistency iterative training using the noisy consistency semantic information to obtain the preliminary backbone network Unet.

[0028] The preliminary backbone network Unet is trained in a supervised manner by a pre-set supervised signal embedding mechanism to construct a teaching resource generation model for brain science embedding.

[0029] Furthermore, the step of performing denoising and consistency iterative training on the backbone network Unet using the noisy consistency semantic information to obtain the preliminary backbone network Unet specifically includes:

[0030] A backbone network Unet is introduced, which includes a low-dimensional semantic feature extraction part and a high-dimensional semantic feature extraction part;

[0031] Based on the low-dimensional semantic feature extraction part, the noisy consistency semantic information is subjected to low-dimensional semantic layer-by-layer downsampling processing to obtain low-dimensional semantic feature information.

[0032] Based on the high-dimensional semantic feature extraction part, the low-dimensional semantic feature information is subjected to high-dimensional semantic layer-by-layer upsampling processing to obtain high-dimensional semantic feature information;

[0033] The low-dimensional semantic feature information and the high-dimensional semantic feature information are fused together to obtain the data of the previous state before adding noise to the consistent semantic information;

[0034] The data of the previous state before adding noise to the consistency semantic information is input into the backbone network Unet for cyclic training until the consistency semantic information of state 0 is output, then the training stops, and the preliminary backbone network Unet is obtained.

[0035] Furthermore, the pre-defined supervised signal embedding mechanism includes a context-additive embedding mechanism, a gradient-based guidance mechanism, and a cross-modal interaction mechanism between contextual information and latent variables. The step of supervising the preliminary backbone network Unet through the pre-defined supervised signal embedding mechanism to construct a brain science embedded teaching resource generation model specifically includes:

[0036] Based on the aforementioned contextual additive embedding mechanism, consistent semantic information with brain data is used as a supervision signal for supervised training of the initial backbone network Unet.

[0037] Based on the gradient-based guidance mechanism, modal encoders are added to the downsampling and upsampling phases of the initial backbone network Unet.

[0038] Based on the cross-modal interaction mechanism of the aforementioned contextual information and latent variables, a cross-modal attention mechanism is designed to introduce contextual data information into the initial backbone network Unet sampling stage.

[0039] By integrating the processing results of the aforementioned pre-set supervisory signal embedding mechanism, a teaching resource generation model for brain science embedding is constructed.

[0040] Furthermore, the step of embedding the actual teaching environment style into the brain science-embedded teaching resource generation model in the form of incrementally pluggable components to construct a teaching resource generation model oriented towards actual teaching scenarios specifically includes:

[0041] Based on the actual teaching environment style, the preferences of teachers and students in the teaching environment are obtained through a self-rating network;

[0042] Construct a bypass matrix based on the preferences of teachers and students in the teaching environment;

[0043] The bypass matrix is ​​embedded as an incremental side-tuning component into the convolutional bypass of the brain science embedded teaching resource generation model to construct a teaching resource generation model oriented towards actual teaching scenarios.

[0044] Furthermore, the specific expression for updating the teaching resource generation model oriented towards actual teaching scenarios based on the load prediction results is as follows:

[0045] brain t+1 =g(f(brain) t +Node2veci +enc(resource t ))

[0046] In the above formula, brain t+1 This represents the brain data of students at time t+1. t Let f(·) represent the brain data of the student at time t, and let f(·) represent the aforementioned cross-modal brain semantic signal encoder, Node2vec. i This indicates the embedding of teaching content and student static information, resource t Let represent the teaching resources designed by the student at time t, enc(·) represent the modal encoder of the corresponding teaching resources, and g(·) represent the β-weighted adaptive general temporal feedforward neural network.

[0047] Furthermore, it also includes editing the multimodal mixed teaching resources through text-based interaction and click-and-drag interaction, wherein:

[0048] The text-based interactive method involves using the multimodal mixed teaching resources as input data and adjusting the multimodal mixed teaching resources according to the text data input by the student.

[0049] The click-and-drag-based interactive method uses the multimodal hybrid teaching resources as input data and collects students' keyboard and mouse click data in real time to adjust and process the multimodal hybrid teaching resources.

[0050] The second technical solution adopted in this invention is: a multimodal hybrid teaching resource construction system based on cognitive neuroscience, comprising:

[0051] The acquisition module is used to acquire dynamic and static data in the teaching environment, encode and process them, and obtain the latent vectors of the static and dynamic data.

[0052] The fusion module is used to acquire students’ brain data and perform semantic fusion processing with the latent vectors of the static and dynamic data to obtain consistent semantic information with the brain data.

[0053] The initial construction module performs denoising and consistency loss processing on the backbone network Unet based on the consistent semantic information of the brain data, and constructs a teaching resource generation model embedded in brain science.

[0054] The secondary construction module is used to embed the actual teaching environment style into the brain science embedded teaching resource generation model in the form of incremental pluggable components, and to build a teaching resource generation model oriented towards actual teaching scenarios.

[0055] The final construction module is used to input static data in the teaching environment and the students' brain data into the teaching resource generation model for actual teaching scenarios for load prediction, and update the teaching resource generation model for actual teaching scenarios based on the load prediction results to obtain the teaching resource-cognitive load prediction model.

[0056] The generation module generates multimodal hybrid teaching resources based on the teaching resource-cognitive load prediction model.

[0057] The beneficial effects of the method and system of this invention are as follows: This invention acquires and encodes dynamic and static data from the teaching environment to obtain latent vectors for both static and dynamic data. Multi-source heterogeneous student data is pre-extracted using different encoders according to different dynamic and static characteristics to obtain latent vector representations while preserving the data's inherent features as much as possible. Furthermore, it combines this with student brain data for fusion processing, aligning the outputs of different modal encoders to the same space. The semantic vector obtained after semantic fusion of brain signals is used as an instrumental variable to perform debiasing and alignment of other variables through contrastive learning, enabling the original single-modal encoder to possess information from the brain fusion modality. Finally, the consistent semantic information from the brain data is used to denoise the backbone network Unet. To address consistency loss, a neuroscience-embedded teaching resource generation model is constructed. This model uses denoising consistency loss to create a unified system capable of generating various teaching resources, including text, images, audio, and video. Through multimodal data embedding, it comprehensively considers factors influencing cognitive load, such as teachers' teaching objectives and students' learning levels. The neuroscience-embedded teaching resource generation model is designed to reduce students' cognitive load. Furthermore, style estimation of the pre-generated content is performed using EEG data and actual physiological selection tendencies of students when exposed to different teaching resources. Finally, by analyzing students' original data and the teaching resources generated in subsequent actual teaching scenarios, it is possible not only to measure the actual cognitive load generated by students but also to reveal the extent to which teaching resources reduce students' cognitive load. Attached Figure Description

[0058] Figure 1 This is a flowchart illustrating the steps of a multimodal hybrid teaching resource construction method based on cognitive neuroscience according to an embodiment of the present invention.

[0059] Figure 2 This is a structural block diagram of the multimodal hybrid teaching resource construction system based on cognitive neuroscience according to an embodiment of the present invention;

[0060] Figure 3 This is a schematic diagram of the backbone network Unet in a specific embodiment of the present invention. Detailed Implementation

[0061] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments. The step numbers in the following embodiments are only for ease of explanation and do not limit the order of the steps. The execution order of each step in the embodiments can be adapted according to the understanding of those skilled in the art.

[0062] This invention proposes a cognitive neuroscience-inspired method for generating teaching resources. The model utilizes multimodal teaching data (including brain signals such as fMRI and EEG) embedded in actual teaching scenarios, and uses exogenous EEG signals as modality alignment centers and instrumental variables to explore causal nature, thereby obtaining the debiased encoding of each modality encoder. This achieves a generative model led by brain science and assisted by other modalities, thus effectively improving the accuracy, interpretability, and robustness of personalized generation of teaching resources.

[0063] Reference Figure 1 This invention provides a method for constructing multimodal hybrid teaching resources based on cognitive neuroscience, which includes the following steps:

[0064] S1. Obtain dynamic and static data in the teaching environment, encode and process them to obtain the implicit vectors of static and dynamic data;

[0065] Specifically, the data generated in actual teaching is characterized by multiple sources and heterogeneity, with dynamic and static data coexisting, and the context is constantly changing and difficult to perceive. This invention integrates static and dynamic data separately and designs different backbone networks to represent the data based on different data types.

[0066] S11. Obtain the implicit vector of static data;

[0067] Specifically, static data from the teaching environment is acquired, including student background information, teacher background information, and teaching material metadata. The static data is parsed and processed based on a multi-level semantic role parser to obtain semi-structured tree-like semantic data. The semi-structured tree-like semantic data is then positionally encoded using a bidirectional pre-trained encoder distilled on a Chinese dataset to obtain the latent vector of the static data.

[0068] In this embodiment, for data that remains unchanged in the teaching environment, such as student background information, teacher background information, and teaching material metadata, a multi-level semantic role parser is used to construct semi-structured tree-like semantic data. This is combined with positional encoding of tree nodes. Subsequently, a bidirectional pre-trained encoder distilled on a Chinese dataset is used to represent the tree nodes as vectors. The vector representation formula for each tree node can be described as follows:

[0069] Node2Vec i =DistilledBLM(emb(Node i)+emb(depth i )+emb(pos i ))

[0070] In the above formula, Node i Let represent the i-th node in the tree, emb represent the weighted learnable embedding function, and depth represent the depth. i pos represents the depth of node i in the tree. i This represents the learnable positional encoding of node i in the weighted tree. By modeling the weighted tree, the latent vector Node2Vec of each static data point can be obtained. i .

[0071] S12, Obtain the implicit vector of dynamic data.

[0072] Specifically, dynamic data in the teaching environment is acquired, including student feedback data and teacher-student interaction data; the dynamic data is captured temporally using QKV adaptive coding to obtain the temporal information of the dynamic data; and the temporal information of the dynamic data is processed by a biased knowledge distillation modal encoder to obtain the latent vector of the dynamic data.

[0073] In this embodiment, for dynamically changing data in the teaching environment, such as student feedback data, and real-time captured text, images, videos, EEG, eye-tracking, and interaction data, this invention employs a method of first capturing data temporally and then robust knowledge distillation to design unsupervised pre-trained encoders for each modality. In the temporal capture stage, this invention uses QKV adaptive encoding. For Q, its expression can be described as:

[0074]

[0075] Where m represents the position and d represents the dimension of the Q vector;

[0076] The K formula can be described as follows:

[0077]

[0078] In the above formula, n represents the position, and d represents the dimension of the K vector;

[0079] Combining the traditional attention calculation formula, this invention can achieve relative position encoding using absolute position encoding without modifying V. It injects prior knowledge of temporal distance and long-range decay patterns into the traditional attention module, and, in conjunction with SequenceFormer, captures temporal information. Subsequently, it uses debiased knowledge distillation to obtain the modal encoder, whose loss function is shown below:

[0080]

[0081] In the above formula, CE represents the binary cross-entropy calculation rule, α represents the hyperparameter used to adjust the loss ratio, KL represents the probability distribution distance metric function, and y cls y kd Let represent the outputs of the classifier head and distillation head of the teacher network in knowledge distillation, respectively, and let y represent the output of the student network. Indicates manually labeled sample labels;

[0082] By considering the probability distribution weight allocation of the classifier head and distillation head of the teacher network, the bias problem caused by the transfer from different domains is overcome, and debiased distillation is achieved.

[0083] In summary, by using different encoders to pre-extract multi-source heterogeneous student data according to their dynamic and static characteristics, latent vector representations of the data can be obtained while preserving the data's own features as much as possible.

[0084] S2. Obtain the student's brain data and perform semantic fusion processing with the latent vectors of the static and dynamic data to obtain consistent semantic information with the brain data.

[0085] Specifically, in order to learn a unified vector representation for different modalities and align the outputs of different modal encoders to the same space, this invention uses the fusion modality of EEG (electroencephalography) and fMRI (functional magnetic resonance imaging) collected from students as the core modality, and aligns other modalities (including text modality, image modality, audio modality, video modality, etc.) to a vector space. Furthermore, considering that there are many biases and errors in multimodal data, and that the EEG and fMRI modalities have exogenous properties and the advantage of explaining implicit variables in causal networks, other modalities are decontaminated.

[0086] S21, EEG and fMRI signal temporal semantic fusion;

[0087] Specifically, the system collects students' electroencephalogram (EEG) data signals and students' functional magnetic resonance imaging (fMRI) signals; it introduces the latent vector of static data, and performs temporal semantic fusion processing on the students' EEG data signals and students' fMRI signals in the same time dimension based on the encoder to obtain the fused latent vector.

[0088] In this embodiment, the signal-to-noise ratio of EEG and fMRI data is low. After semantic representation by the encoder, a high-dimensional representation is obtained. These two signals have a high degree of synchronization. Based on the temporal patterns captured by the aforementioned SequenceFormer and the static information that has been encoded, this invention first models this process as follows, in conjunction with the feature interactions on the same time dimension:

[0089] h=σ(CMA(Q fMRI +emb t +vec,KEEG +emb t +vec, V EEG +emb t +uec))

[0090] In the above formula, σ represents the activation function, CMA(·) represents the cross-modal attention operation between fMRI and EEG signals, and emb t Let Q represent the embedding vector at each time step, vec represent the weighted embedding vector of the aforementioned static information, and Q represent the embedding vector at each time step. fMRI K represents the latent vector of brain magnetic resonance imaging as the query vector. EEG and V EEG The EEG latent vector is used as the key and value vector. By embedding time steps and static information, cross-modal semantic fusion of brain signals is achieved. h represents the fused latent vector, which contains both EEG (electroencephalography) and fMRI (functional magnetic resonance imaging) information. The anchor point for subsequent alignment of other modalities is h.

[0091] S22, Brain signal fusion semantic alignment with other modalities.

[0092] Specifically, the cross-modal mutual information entropy between the fused latent vector and the latent vector of the dynamic data is obtained; based on the cross-modal mutual information entropy, the latent vector of the dynamic data is debiased and aligned using the fused latent vector to obtain consistent semantic information with brain data.

[0093] In this embodiment, other modalities such as video, image, and text modalities, after being encoded into latent vectors, contain a large number of spurious correlations, i.e., non-causal relationships. When only two modalities are considered, brain signals can influence the final output (such as learning performance and cognitive load) by affecting the other modality. Furthermore, brain signals are often more exogenous and more capable of interpreting the student's implicit state than the other modality. Therefore, brain signals meet the criteria for being used as instrumental variables, eliminating the need for pairwise comparative learning between any two modalities. This invention uses the semantic vectors obtained after semantic fusion of brain signals as instrumental variables to perform debiasing and alignment of other variables through comparative learning. This process can be described using cross-modal mutual information entropy, the expression of which is:

[0094]

[0095] In the above formula, X represents all modalities to be aligned (including text modality, image modality, audio modality, video modality, etc.), f function represents the linear dimension projection of the brain latent vector, g function represents the linear dimension projection of the latent vectors of other modalities, τ represents the controllable alignment hyperparameter, which essentially means that two modal sample pairs with index i form a positive sample pair, and two modal sample pairs with index j form a negative sample pair. Modal alignment is achieved by reducing the distance between positive sample pairs and increasing the distance between negative sample pairs. L brain,X Let f(brain) represent the cross-modal alignment function between the brain mode and the X mode. i ) T G(X) represents a linear mapping of the brain's latent vectors. i ), g(X) j ) respectively represent the linear mappings of X modality data belonging to the same sample as EEG data and X modality data from different samples in the same batch of data;

[0096] By removing bias and aligning, in the vector space, any modality of any sample is closest to the corresponding EEG fusion modality of this sample, so that the original single-modal encoder (including text encoder, image encoder, audio encoder, video encoder) has the information given by the brain fusion modality. Combined with the unbiasedness generated by the brain fusion modality itself, the encoding method of the encoder itself also tends to be unbiased.

[0097] S3. Based on the consistent semantic information of the brain data, perform noise reduction and consistency loss processing on the backbone network Unet to construct a teaching resource generation model embedded in brain science.

[0098] Specifically, by embedding multimodal data and comprehensively considering factors affecting cognitive load such as teachers' teaching objectives and students' learning levels, a brain science-embedded teaching resource generation model is designed to reduce students' cognitive load. This invention creates a cross-modal latent state EEG hybrid model, Latent Mind Mixer, to generate various teaching resources (text, images, audio, video, slides, etc.).

[0099] S31. A general multimodal generative model based on denoising consistency pre-training;

[0100] Specifically, based on the dimensions of static and dynamic data, noise signals of the latent vectors of the static and dynamic data are obtained in a standard normal distribution; the noise signals of the latent vectors of the static and dynamic data are added to the consistency semantic information of brain data to obtain the noise-added consistency semantic information; the backbone network Unet is subjected to denoising consistency iterative training through the noise-added consistency semantic information to obtain the preliminary backbone network Unet.

[0101] The process involves introducing a backbone network, Unet, which comprises a low-dimensional semantic feature extraction part and a high-dimensional semantic feature extraction part. Based on the low-dimensional semantic feature extraction part, the consistent semantic information after adding noise is subjected to low-dimensional semantic downsampling layer by layer to obtain low-dimensional semantic feature information. Based on the high-dimensional semantic feature extraction part, the low-dimensional semantic feature information is subjected to high-dimensional semantic upsampling layer by layer to obtain high-dimensional semantic feature information. The low-dimensional semantic feature information and the high-dimensional semantic feature information are fused to obtain the data of the previous state of the consistent semantic information before adding noise. The data of the previous state of the consistent semantic information before adding noise is input into the backbone network Unet for cyclic training until the consistent semantic information of state 0 is output, at which point training stops, and the preliminary backbone network Unet is obtained.

[0102] In this embodiment, the process of the human brain constructing things is similar to denoising. It stores the fuzzy concept of things through short-term memory, and then refines the fuzzy concept of things through elaboration and semantic processing to complete the reconstruction and storage of things. The short-term memory blocks that may be allocated for any modality may differ, but the information processing flow remains unchanged. Inspired by this, this invention constructs a multimodal generation model through denoising consistency. For any modality, this invention performs feature learning on the latent representation of the original modal data (text modality, image modality, audio modality, video modality, etc.) in a manner of first adding noise and then denoising. For any modal data, it is first encoded into data of (sequence length * latent vector dimension) by the corresponding modal encoder in step S22. Noise is added by adding noise of the same dimension as the latent vector. A denoising neural network is trained to obtain the data representation before noise addition. To improve the network's modeling ability at various scales, the entire process is divided into t-step iterations. The network input is (state t, data X at state t). t Output X t-1 The loss function can be obtained by minimizing the variational lower bound:

[0103]

[0104] In the above formula, x0 represents the implicit representation of the original modal data distribution, x t q(x) represents the data after adding noise in state t. t |x t-1 ) is x t-1 The data distribution after adding noise x t Distribution, p θ (x t-1 |x t ) is a deep neural network used to fit the q distribution corresponding to state t in the z-axis. This invention uses a multi-scale Unet as the backbone network of p, X T The probability distribution obtained at the final state step T can be viewed as a quasi-normal distribution, Eq p(x) represents the expected value of the variational lower bound. T ) represents the probability distribution of the standard normal distribution;

[0105] This invention can obtain data X of an input mode in any state t through supervised training using the above-mentioned loss function calculation formula. t The input state t returns the data distribution X of state t-1. t-1 For multi-scale backbone Unet networks, such as Figure 3 As shown;

[0106] The multi-scale Unet possesses the ability to model any modality at multiple scales. Shallow layers learn low-dimensional semantics through convolution and attention mechanisms and downsample layer by layer. Deeper layers further process the low-dimensional semantic information to obtain high-dimensional semantics, followed by upsampling to fuse semantic information at different levels. Whether in shallow or deep layers, downsampling or upsampling, attention modules are used to extract and integrate important information from modalities such as text, video, image, and audio. The network's input and output dimensions are equal. By training a denoising network and combining it with reverse sampling, this invention can reconstruct any modality (text, video, audio, image, etc.). The specific steps are as follows:

[0107] (1) Based on the dimension of the original mode, noise with the same dimension as the original mode is extracted from a specific standard normal distribution, i.e., the above x T x is obtained by feeding it into the pre-trained Unet. t-1 .

[0108] (2) The probabilities of states T-1 up to 0 are iteratively fed into Unet to finally obtain the reconstructed X0 of the original mode.

[0109] Based on the different dimensions of the initial noise (i.e., the input data) and the prediction of the early state of reconstruction, the model introduces time dimension attention for the simulation of time-series data such as audio and video to ensure the continuity of data at different times, improve the fidelity of teaching resources generation, and reduce the inconsistency between frames.

[0110] S32, the core supervisory signal embedding mechanism in brain science.

[0111] Specifically, a brain science embedded teaching resource generation model is constructed by supervising the preliminary backbone network Unet through a pre-defined supervised signal embedding mechanism. This pre-defined mechanism includes a context-additive embedding mechanism, a gradient-based guidance mechanism, and a cross-modal interaction mechanism between contextual information and latent variables. According to the context-additive embedding mechanism, consistent semantic information with brain data is used as the supervision signal for the supervised training of the preliminary backbone network Unet. According to the gradient-based guidance mechanism, modal encoders are added to the downsampling and upsampling stages of the preliminary backbone network Unet. According to the cross-modal interaction mechanism between contextual information and latent variables, a cross-modal attention mechanism is designed to introduce contextual data information into the post-sampling stage of the preliminary backbone network Unet. Finally, the results of the above pre-defined supervised signal embedding mechanism are integrated to construct the brain science embedded teaching resource generation model.

[0112] In this embodiment, the pre-training task trained the ability to reconstruct the corresponding original modality from a piece of noise, but the embedding of supervisory signals has not yet been achieved, such as controllable text-to-image, text-to-audio, text-to-video, or image-to-image generation, image-to-video generation, image captioning, etc. That is, it not only achieves single-modality reconstruction, but also uses guidance information from other modalities to provide supervisory information for the generation of this modality. To achieve controllable and stable control of the generated teaching resources by each modality, this invention simultaneously uses three mechanisms to improve the Unet's joint regulation of supervisory signals.

[0113] First mechanism: Context-additive embedding;

[0114] For each layer of Unet, the supervision signal is added in the form of context embedding. This process can be modeled as follows:

[0115] h = layer(x + f(context) i )))

[0116] In the above formula, f is used to map the cross-modal context to the dimension of x, where x represents the latent vector of the modality to be generated (such as text modality, image modality, audio modality, video modality), and context... i This represents the supervision signal after alignment with the aforementioned neuroscience. For example, in the textural image case, x represents the noisy image vector at time t, and context... i This refers to the latent vector of the original text after it has been aligned by the aforementioned brain science text encoder, and layer represents the network layer in Unet.

[0117] This invention supports the joint guidance of multiple modalities in the generation of a specified modality. For example, by assigning a reference image after the given text, more stable and controllable image generation can be achieved.

[0118] The second mechanism: gradient-based guidance;

[0119] Its expression is:

[0120]

[0121] In the above formula, This represents a modal encoder fine-tuned on a certain noisy dataset. The gradient-guided mechanism expression shows that we do not need to adjust the original Unet; we only need to add the modal encoder's input to X during the sampling phase. t The gradient can be used to embed the supervisory signal. For example, in the Wensheng graph case, the θ network represents the Unet network. The network represents the EEG-aligned text encoder in step S2, and the second term in the gradient-based guidance mechanism expression represents the gradient of the original text after passing through the text encoder.

[0122] The third mechanism: cross-modal interaction of contextual information and latent variables;

[0123] This invention introduces contextual information by designing a cross-modal attention mechanism after the Unet downsampling and upsampling layers. This process can be modeled as follows:

[0124]

[0125] In the above formula, the context information f(context) is set as the key and value, and the original latent vector h is set as the query, designing cross-modal attention. For example, in the text-to-image case, h represents the latent vector of the image with added noise at state t, context represents the latent vector of the original text after passing through the text encoder aligned with the EEG in step S2, and d m The dimension of the f(context) vector is represented, and CMA(·) represents the cross-modal attention between Unet latent variables and context.

[0126] Throughout the training process described above, all aligned modal encoder parameters are kept unchanged. Furthermore, during each training iteration, no supervision signal is input with probability p, requiring the model to simulate the output under supervised conditions, thus preserving the contextual metadata in UNet.

[0127] With the steps outlined above, the basic architecture of Latent Mind Mixer is complete. It can accept input text descriptions, images, audio, video, and other guiding signals, and output any category of teaching resources (text, image, audio, video).

[0128] S4. Embed the actual teaching environment style into the brain science embedded teaching resource generation model in the form of incremental pluggable components to construct a teaching resource generation model oriented towards actual teaching scenarios.

[0129] Specifically, however, in actual teaching, teaching resources not only need to be closely aligned with the teaching content, teachers, and students, but ideally, they should also be customizable to better meet the needs of personalization, creativity, students' knowledge construction habits, information processing methods, and metacognitive strategies. This information cannot be obtained solely from static student attribute data and large-scale datasets; it needs to be captured in actual dynamic teaching scenarios. Style estimation of pre-generated content is performed using EEG data and actual physiological selection tendencies of students when exposed to different teaching resources, and finally, it is embedded into Unet as incrementally pluggable components.

[0130] S41, supports incremental adjustment components for small-batch data style capture;

[0131] Specifically, based on the actual teaching environment style, the preference of teachers and students in the teaching environment is obtained through a self-scoring network; a bypass matrix is ​​constructed according to the preference of teachers and students in the teaching environment; the bypass matrix is ​​embedded into the convolutional bypass in the brain science embedded teaching resource generation model in the form of incremental side-tuning components to construct a teaching resource generation model oriented towards actual teaching scenarios.

[0132] In this embodiment, learning styles from mini-batch data can lead to catastrophic forgetting of the model. This invention overcomes this problem by freezing the pre-trained network and designing incrementally pluggable components. First, a self-scoring network is designed to estimate student and teacher preferences; the loss function can be expressed as:

[0133]

[0134] In the above formula, K represents the K teaching resources presented to the students, and r θ This indicates that the reward model is used to fit students' learning preferences and consists of a lightweight UNet with scalar (score) outputs. σ represents the activation function, x represents the student's input including text and other modalities, and y represents the input. i Indicates superior resources for students, y j D represents the student's non-optimal resources, and D represents the database storing the scoring information.

[0135] This invention uses user interface priority selection and brain signal capture methods to determine students' preferences for teaching resource styles. θ The learning process is supervised by students' ratings of specific learning resources, gradually understanding their preferences and assigning higher scores to better resources and lower scores to less desirable ones. This rating model will then be used as a supervisory signal in subsequent steps to fit the students' preferences to the Latent Mind Mixer. θ The reward function can automate and expand the previously limited student scoring data to an unlimited extent, thus meeting the need for subsequent alignment of student preferences. Once the function is optimized, each student will have an independent reward score for quickly training the incremental side-tuning component. This process can be modeled as follows:

[0136]

[0137] In the above formula, r θ It is the aforementioned scoring function. It is a copy of the aforementioned pre-trained Unet network. This is achieved by designing bypass matrices alongside each convolutional layer of the original Unet. During optimization, the Unet weights remain unchanged, and only the parameters of the bypass matrices are optimized. The parameters of the bypass matrices are much smaller than those of Unet, allowing for efficient customization of styles for each student and providing a plug-and-play module compatible with Unet. Through parameter optimization, each bypass matrix becomes a pluggable component directly reflecting the current student's preference for the teaching resources. Even when modalities are missing in actual teaching scenarios, the generation effect can still be controlled through the bypass matrices. When group collaboration is involved, multiple bypass matrices can be embedded into Unet, and the weights of each bypass matrix can be adjusted according to the overall group preference to ensure that the final teaching resources match the preferences of all students in the group.

[0138] In this stage, the present invention does not change the pre-trained parameters of Unet in the Latent Mind Mixer that has been trained in step S3. Instead, it designs a reward model that fits the student's preferences and many bypass matrices, so that the Latent Mind Mixer, which originally conforms to the general style, has a teaching resource generation model that is aligned with the specific student's preferences.

[0139] S5. Input the static data in the teaching environment and the brain data of the students into the teaching resource generation model for actual teaching scenarios to perform load prediction, and update the teaching resource generation model for actual teaching scenarios according to the load prediction results to obtain the teaching resource-cognitive load prediction model.

[0140] Specifically, through Latent Mind Mixer, a teaching resource generation model based on cognitive neuroscience, the level of cognitive load is determined by the teaching resources designed by students. By analyzing the students' original data and the teaching resources generated in subsequent actual teaching scenarios, it is possible not only to measure the actual cognitive load generated by students, but also to reveal the extent to which teaching resources reduce students' cognitive load. Based on the above considerations, this invention establishes an integrated feedback mechanism for teaching resources and cognitive load. The model input is students' static information and teaching resources generated by students themselves during the teaching process, and the output is the cognitive load change factor. The monitoring signal is the brain science signal before and after the generation of teaching resources.

[0141] S51, Teaching Resources-Cognitive Load Prediction Model;

[0142] Specifically, the brain science matrix is ​​predicted by analyzing all the teaching resource logs created by students during the learning process. The model input includes brain science signals before the creation of teaching resources and static information such as teaching content and students in the actual created teaching resources. The output is the brain science signal after the use of teaching resources. The mean squared error between the matrices is used to measure the difference between the predicted matrix and the actual brain science matrix, and the model parameters are updated accordingly. This process can be modeled as follows:

[0143] brain t+1 =g(f(brain) t +Node2vec i +enc(resource t ))

[0144] In the above formula, brain t+1 This represents the brain data of students at time t+1. t Let f(·) represent the brain data of the student at time t, and let f(·) represent the aforementioned cross-modal brain semantic signal encoder, Node2vec. i This indicates the embedding of teaching content and student static information, resource t Let represent the teaching resources designed by the student at time t, enc(·) represent the modal encoder of the corresponding teaching resources, and g(·) represent the β-weighted adaptive general temporal feedforward neural network.

[0145] After training, g(·) can predict the degree of change in brain signals after students receive specific teaching resources, based on student information and teaching content. Combined with existing brain science visualization tools, it can conduct a more in-depth exploration of the cognitive load mechanism for different knowledge points and different generated teaching resources.

[0146] S6. Generate multimodal hybrid teaching resources based on the teaching resource-cognitive load prediction model.

[0147] Specifically, teaching resources should be interactive. This not only aligns with the development trend of human-computer collaboration but is also an essential step in making teaching resources truly dynamic, and a characteristic that student-centered teaching resources should possess. This invention allows students to interactively edit generated images through text (keyboard input, etc.) and click-and-drag (mouse, touch controls, etc.) to partially redraw them.

[0148] S61, Text-based interactive editing;

[0149] Specifically, text-based interactive learning involves using multimodal blended learning resources as input data and adjusting these resources based on the text data input by students.

[0150] In this embodiment, text-based interactive editing uses the generated resources and text as input to the model, and the output is the learning resource after secondary adjustment guided by text. To prevent excessive differences from the original resource, this invention uses the cross-modal attention mapping between the original resource and the new text as a mask matrix. Users can manually refine the mask (in image and video modalities). Regions with higher weights are redrawn more heavily, and regions with lower weights are redrawn less heavily. Redrawing involves adding noise and resampling. Regions with high redrawing intensity are added with higher noise intensity, and vice versa. By adaptively adding noise to the original resource according to the mask matrix, the image retains certain semantics of the original resource and regenerates new resources according to the aforementioned text-supervised teaching resource generation method. To avoid generating a large number of identical resources, special markers such as {} are supported to increase the model's attention to specific text descriptions. For content with special markers, this invention uses an inversion augmented mechanism to force the enhanced text attention to replace the unenhanced text attention at a specified sampling step t and redistribute the weights so that the text changes provide more complete semantics in a specific sampling step.

[0151] S62, Interactive editing based on click and drag.

[0152] Specifically, the click-and-drag interactive approach uses multimodal mixed teaching resources as input data and collects students' keyboard and mouse click data in real time to adjust and process the multimodal mixed teaching resources.

[0153] In this embodiment, unlike text-based interactive editing, click-and-drag interactive editing is only applied to image and video resources. Users can draw a mask (optional for precise control) on the intelligent guidance of Manipulation Everything and set a drag path for several points. This invention achieves this function through motion supervision of neighboring image blocks and nearest-neighbor point tracking. By inputting the point sequence of the dragging action and existing teaching resources, the system generates the dragged image or video.

[0154] The movement of neighboring image patches is achieved by making n tiny movements to the latent vector at step t of the Latent Mind Mixer model. Simultaneously, this invention strives to maintain minimal semantic differences across the entire resource. This process can be modeled as follows:

[0155]

[0156] In the above formula, i represents the i-th pair of source and target points, and p i That is, the source point, χ1(p i r1) indicates that p is used on the original image. i Centered on r1, a neighborhood block with radius r1, sg represents the gradient stopping operation, and m represents the aforementioned mask matrix. Let d represent the latent vector after the k-th minute movement in the t-th sampling step. i It points to the source point p. i to target point g i Let L1 be a unit vector, and L1 represent the L1 loss function. That is, when sampling from t to t-1, the previous term uses bilinear interpolation to make p... i The first term moves the neighboring block with radius r1 to the target domain, while the second term tries to keep the unmasked area unaffected by the small movements.

[0157] Although the aforementioned process defines the source point p i The k-th minute movement and the improved Latent Mind Mixer sampling process. However, the p-th minute movement is not given. i The precise position after the k-th minute movement, so that p i To track corresponding points on an object as closely as possible, this invention proposes nearest neighbor tracking to achieve the above objective, the expression of which is:

[0158]

[0159] In the above formula, Indicated to A point space with an absolute value of r², z t This represents the latent vector of the original image at the t-th sampling step. Represents the source point p iThe position after the k-th operation, i.e. Update the node to its nearest neighbor in the original graph before any operation as the value after the (k+1)th tiny move.

[0160] The combination of motion supervision for neighboring image patch movement and nearest neighbor tracking forms an iterative closed loop, where the movement of neighboring image patches requires movement... The neighborhood of the nearest neighbor, and the nearest neighbor tracking specification The update scheme not only allows image patches in the neighborhood to move according to user input, but also ensures that specific points after the k-th step can still correspond to the original image and be used as their update method, thus maintaining stability and controllability.

[0161] Text-based guidance is achieved by fine-tuning the redrawing of the mask matrix, while drag-and-drop guidance is achieved by learning latent vectors during the reverse sampling process. Thus, the original Latent Mind Mixer can not only be customized to students' preferred styles, but can also be "reworked" based on students' interaction information—text / drag and drop—for even more refined adjustments.

[0162] Reference Figure 2 A multimodal blended learning resource construction system based on cognitive neuroscience includes:

[0163] The acquisition module is used to acquire dynamic and static data in the teaching environment, encode and process them, and obtain the latent vectors of the static and dynamic data.

[0164] The fusion module is used to acquire students’ brain data and perform semantic fusion processing with the latent vectors of the static and dynamic data to obtain consistent semantic information with the brain data.

[0165] The initial construction module performs denoising and consistency loss processing on the backbone network Unet based on the consistent semantic information of the brain data, and constructs a teaching resource generation model embedded in brain science.

[0166] The secondary construction module is used to embed the actual teaching environment style into the brain science embedded teaching resource generation model in the form of incremental pluggable components, and to build a teaching resource generation model oriented towards actual teaching scenarios.

[0167] The final construction module is used to input static data in the teaching environment and the students' brain data into the teaching resource generation model for actual teaching scenarios for load prediction, and update the teaching resource generation model for actual teaching scenarios based on the load prediction results to obtain the teaching resource-cognitive load prediction model.

[0168] The generation module generates multimodal hybrid teaching resources based on the teaching resource-cognitive load prediction model.

[0169] The content of the above method embodiments is applicable to this system embodiment. The specific functions implemented in this system embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0170] The above is a detailed description of the preferred embodiments of the present invention. However, the present invention is not limited to the embodiments described. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention. All such equivalent modifications or substitutions are included within the scope defined by the claims of this application.

Claims

1. A method for constructing multimodal blended teaching resources based on cognitive neuroscience, characterized in that: Includes the following steps: Acquire and encode dynamic and static data in the teaching environment to obtain the latent vectors of static and dynamic data; The student's brain data is acquired and semantically fused with the latent vectors of the static and dynamic data to obtain consistent semantic information with the brain data. Based on the consistent semantic information of the brain data, the backbone network Unet is subjected to denoising consistency loss processing to obtain a teaching resource generation model embedded in brain science. The actual teaching environment style is embedded into the brain science embedded teaching resource generation model in the form of incremental pluggable components to obtain a teaching resource generation model oriented towards actual teaching scenarios. The static data in the teaching environment and the brain data of the students are input into the teaching resource generation model for actual teaching scenarios to perform load prediction, and the teaching resource generation model for actual teaching scenarios is updated according to the load prediction results to obtain the teaching resource-cognitive load prediction model. Multimodal hybrid teaching resources are generated based on the aforementioned teaching resource-cognitive load prediction model; The step of acquiring and encoding dynamic and static data from the teaching environment to obtain the latent vectors of the dynamic and static data specifically includes: Acquire static data from the teaching environment, including student background information, teacher background information, and teaching material metadata. The static data is parsed and processed using a multi-level semantic role parser to obtain semi-structured tree-like semantic data. The semi-structured tree semantic data is positionally encoded by a bidirectional pre-trained encoder distilled from a Chinese dataset to obtain the latent vector of the static data. Acquire dynamic data in the teaching environment, including student feedback data and teacher-student interaction data; The dynamic data is time-series captured by QKV adaptive coding to obtain the timing information of the dynamic data; The temporal information of the dynamic data is positionally encoded by a debiased knowledge distillation modal encoder to obtain the latent vector of the dynamic data. By integrating the latent vectors of the static data and the latent vectors of the dynamic data, the latent vectors of the static and dynamic data are obtained. The step of acquiring students' brain data and performing semantic fusion processing with the latent vectors of the static and dynamic data to obtain semantic information consistent with the brain data specifically includes: Collect students' brain data, which includes students' electroencephalogram (EEG) data signals and students' functional magnetic resonance imaging (fMRI) signals. Based on the latent vector of the static data, the student's EEG data signal and the student's functional magnetic resonance imaging signal are subjected to temporal semantic fusion processing in the same time dimension based on the encoder to obtain the fused latent vector. Obtain the cross-modal mutual information entropy between the fused latent vector and the latent vector of the dynamic data; Based on the cross-modal mutual information entropy, the latent vectors of the dynamic data are debiased and aligned using the fused latent vectors to obtain consistent semantic information with brain data.

2. The method for constructing multimodal hybrid teaching resources based on cognitive neuroscience according to claim 1, characterized in that, The step of performing denoising and consistency loss processing on the backbone network Unet based on the consistent semantic information of the brain data to obtain the brain science embedded teaching resource generation model specifically includes: Based on the dimensions of the static and dynamic data, the noise signal of the latent vector of the static and dynamic data is obtained in a standard normal distribution; The noise signal of the latent vector of the static and dynamic data is added to the consistent semantic information with brain data to obtain the noise-added consistent semantic information. The backbone network Unet is subjected to denoising consistency iterative training using the noisy consistency semantic information to obtain the preliminary backbone network Unet. The preliminary backbone network Unet is trained in a supervised manner by a pre-set supervised signal embedding mechanism to obtain a teaching resource generation model for brain science embedding.

3. The method for constructing multimodal hybrid teaching resources based on cognitive neuroscience according to claim 2, characterized in that, The step of performing denoising and consistency iterative training on the backbone network Unet using the noisy consistency semantic information to obtain the preliminary backbone network Unet specifically includes: A backbone network Unet is introduced, which includes a low-dimensional semantic feature extraction part and a high-dimensional semantic feature extraction part; Based on the low-dimensional semantic feature extraction part, the noisy consistency semantic information is subjected to low-dimensional semantic layer-by-layer downsampling processing to obtain low-dimensional semantic feature information. Based on the high-dimensional semantic feature extraction part, the low-dimensional semantic feature information is subjected to high-dimensional semantic layer-by-layer upsampling processing to obtain high-dimensional semantic feature information; The low-dimensional semantic feature information and the high-dimensional semantic feature information are fused together to obtain the data of the previous state before adding noise to the consistent semantic information; The data of the previous state before adding noise to the consistency semantic information is input into the backbone network Unet for cyclic training until the consistency semantic information of state 0 is output, then the training stops, and the preliminary backbone network Unet is obtained.

4. The method for constructing multimodal blended teaching resources based on cognitive neuroscience according to claim 3, characterized in that, The pre-defined supervised signal embedding mechanism includes a context-additive embedding mechanism, a gradient-based guidance mechanism, and a cross-modal interaction mechanism between contextual information and latent variables. The step of supervising the preliminary backbone network Unet through the pre-defined supervised signal embedding mechanism to obtain a brain science embedded teaching resource generation model specifically includes: Based on the aforementioned contextual additive embedding mechanism, consistent semantic information with brain data is used as a supervision signal for supervised training of the initial backbone network Unet. Based on the gradient-based guidance mechanism, modal encoders are added to the downsampling and upsampling phases of the initial backbone network Unet. Based on the cross-modal interaction mechanism of the aforementioned contextual information and latent variables, a cross-modal attention mechanism is designed to introduce contextual data information into the initial backbone network Unet sampling stage. By integrating the processing results of the preset supervisory signal embedding mechanism, a teaching resource generation model for brain science embedding is obtained.

5. The method for constructing multimodal blended teaching resources based on cognitive neuroscience according to claim 1, characterized in that, The step of embedding the style of the actual teaching environment into the brain science-embedded teaching resource generation model through the design of incremental pluggable components to obtain a teaching resource generation model oriented towards actual teaching scenarios specifically includes: Based on the actual teaching environment style, the preferences of teachers and students in the teaching environment are obtained through a self-rating network; Construct a bypass matrix based on the preferences of teachers and students in the teaching environment; The bypass matrix is ​​embedded as an incremental side-tuning component into the convolutional bypass of the brain science embedded teaching resource generation model to construct a teaching resource generation model oriented towards actual teaching scenarios.

6. The method for constructing multimodal blended teaching resources based on cognitive neuroscience according to claim 1, characterized in that, The specific expression for updating the teaching resource generation model for actual teaching scenarios based on the load prediction results is as follows: In the above formula, express Students' brain data at all times express Students' brain data at all times This refers to the aforementioned cross-modal brain semantic signal encoder. This indicates the embedding of teaching content and students' static information. Indicates that students are Teaching resources for design that is constantly being created. This represents the modal encoder corresponding to the teaching resources. express Weight-adaptive general-purpose temporal feedforward neural network.

7. The method for constructing multimodal hybrid teaching resources based on cognitive neuroscience according to claim 1, characterized in that, The method further includes editing the multimodal mixed teaching resources through text-based interaction and click-and-drag interaction, wherein: The text-based interactive method involves using the multimodal mixed teaching resources as input data and adjusting the multimodal mixed teaching resources according to the text data input by the student. The click-and-drag-based interactive method uses the multimodal hybrid teaching resources as input data and collects students' keyboard and mouse click data in real time to adjust and process the multimodal hybrid teaching resources.

8. A multimodal blended teaching resource construction system based on cognitive neuroscience, characterized in that, The system is applied to the method according to any one of claims 1-7, and the system comprises the following modules: The acquisition module is used to acquire and encode dynamic and static data in the teaching environment to obtain the implicit vectors of the static and dynamic data. The fusion module is used to acquire students’ brain data and perform semantic fusion processing with the latent vectors of the static and dynamic data to obtain consistent semantic information with the brain data. A preliminary module is constructed, and the backbone network Unet is subjected to denoising consistency loss processing based on the consistent semantic information of the brain data to obtain a teaching resource generation model embedded in brain science. The secondary construction module is used to embed the actual teaching environment style into the brain science embedded teaching resource generation model by designing incremental pluggable components, so as to obtain a teaching resource generation model oriented towards actual teaching scenarios. The final construction module is used to input static data in the teaching environment and the students' brain data into the teaching resource generation model for actual teaching scenarios for load prediction, and update the teaching resource generation model for actual teaching scenarios based on the load prediction results to obtain the teaching resource-cognitive load prediction model. The generation module generates multimodal hybrid teaching resources based on the teaching resource-cognitive load prediction model.

Images