A learner cooperation state classification method and system for conversation electroencephalogram combined learning

Abstract

The application provides a learner cooperation state classification method and system for conversation electroencephalogram combined learning, comprising: a conversation text and electroencephalogram signal combined learning mode is adopted, different modal data is fused through a multi-modal variational autoencoder, information can flow bidirectionally between different modes, cognitive states and behavior characteristics of learners can be captured more comprehensively; feature extraction and selection of appropriate representation modes are carried out according to the data characteristics between different modes, the collaborative activity information index between learners is considered, and the shared feature representation of different modal data is optimized; the loss function optimization model parameter is improved, the joint modeling and generation capability of different modal data is enhanced, and the accuracy of the learner cooperation state classification is improved.

Description

Technical Field

[0001] This invention relates to the field of brain-computer interface technology, and in particular to a method and system for classifying learner collaborative states in conversational EEG co-learning. Background Technology

[0002] As digital technology's impact on education deepens and society's demands for talent increase, collaborative learning has become a widely adopted learning method in the education field. Collaborative state refers to the cognitive, emotional, and behavioral states exhibited by learners in collaborative learning, such as task comprehension, planning, goal setting, monitoring and evaluation, strategy usage, and task execution. With the continuous development of smart devices, collecting and analyzing multimodal data of learners during collaborative learning has become feasible. This data mainly includes learners' behavioral and physiological data. Behavioral data is obtained by observing and recording learners' external behaviors, speech, and performances, primarily in the form of conversational text. Physiological data consists of learners' physiological signals measured through wearable devices or physiological sensors, such as electroencephalogram (EEG) signals, skin conductance, and heart rate. Physiological data can be used to reflect learners' cognitive states and emotional changes. Analyzing and identifying learners' collaborative state has always been a major challenge in the field of learning science. Traditional learner collaborative state analysis relies mainly on manual observation, which suffers from high subjectivity, low efficiency, and a single analytical dimension. Therefore, it is necessary to combine multimodal data and use intelligent methods to improve the accuracy and timeliness of learner collaborative state analysis.

[0003] Joint learning refers to the fusion of data from multiple different modalities for learning and decision-making. By processing different types of data, joint learning establishes connections between data, uncovers potential shared information, and ultimately achieves semantic unification across different modalities, resulting in better learning performance than traditional methods. A joint learning model refers to a neural network model capable of performing multimodal learning tasks, thereby complementing information from different modalities. Compared to single-modal learning models, joint learning models acquire richer and more comprehensive information. However, due to the different sources of information in multimodal data and the varying degrees of contribution from different modalities in different application scenarios, joint learning models still face challenges in representation and fusion. The most challenging aspects are how to transform multimodal data into compact and modality-invariant representations, and how to integrate unique and shared information from different modalities during the fusion process.

[0004] Therefore, providing an effective method for analyzing learner collaborative states by integrating information from multiple modalities is a problem that needs to be solved. Summary of the Invention

[0005] This invention provides a method and system for classifying learner collaborative states in conversational EEG co-learning, in order to overcome the deficiencies in the existing technology.

[0006] In a first aspect, the present invention provides a learner collaborative state classification method for conversational EEG co-learning, comprising:

[0007] Collect conversational text data and EEG signal data of learners in collaborative learning;

[0008] The conversation text data and the EEG signal data are preprocessed respectively to obtain preprocessed conversation text data and preprocessed EEG signal data.

[0009] The preprocessed conversation text data and the preprocessed EEG signal data are aligned, and the conversation text features in the aligned preprocessed conversation text data and the EEG signal features in the aligned preprocessed EEG signal data are extracted.

[0010] The conversational text features and the EEG signal features are input into the initial model for classifying learner collaborative states through conversational EEG joint learning for training. A loss function is constructed to optimize the model, resulting in the learner collaborative state classification model.

[0011] The learner collaborative state classification model is input into the conversation text data and EEG signal data to be detected, and the learner collaborative state classification result is output.

[0012] According to the present invention, a learner collaboration state classification method for conversational EEG co-learning is provided, which collects conversational text data and EEG signal data of learners in collaborative learning, including:

[0013] The conversation text data is determined to be in CSV format, and the EEG signal data is in SET format.

[0014] According to the learner collaboration state classification method for conversational EEG co-learning provided by the present invention, the conversational text data and the EEG signal data are preprocessed respectively to obtain preprocessed conversational text data and preprocessed EEG signal data, including:

[0015] The conversation text data is divided into multiple text segments of unequal length, and the multiple text segments are encoded based on the collaboration status category to obtain encoded conversation text. The encoded conversation text is recorded in a CSV file according to a preset record format to obtain the preprocessed conversation text data. The preset record format includes text content, start time, end time and encoding category.

[0016] The EEG signal data was processed using the EEGLAB toolbox to perform channel localization, removal of useless electrodes, filtering, downsampling, removal of bad segments and artifact removal. The EEG signal data was divided into multiple signal segments according to the start and end times in the conversation text data to obtain the preprocessed EEG signal data.

[0017] According to the present invention, a learner collaborative state classification method for conversational EEG co-learning aligns the preprocessed conversational text data and the preprocessed EEG signal data, and extracts conversational text features from the aligned preprocessed conversational text data and EEG signal features from the aligned preprocessed EEG signal data, including:

[0018] The preprocessed conversation text data and the preprocessed EEG signal data are aligned in chronological order to obtain the aligned preprocessed conversation text data and the aligned preprocessed EEG signal data.

[0019] The aligned and preprocessed conversation text data is input into the Chinese pre-trained model BERT to obtain word vector features, which are used as the conversation text features.

[0020] The fusion feature of the phase lock value (PLV) of every two EEG signals in the aligned and preprocessed EEG signal data is extracted as the EEG signal feature.

[0021] According to the present invention, a learner collaboration state classification method for conversational EEG co-learning is provided, wherein the conversational text features and the EEG signal features are input into an initial model for learner collaboration state classification of conversational EEG co-learning for training, a loss function is constructed to optimize the model, and a learner collaboration state classification model is obtained, including:

[0022] The initial model for classifying learner collaborative states includes a conversational text and EEG signal joint modeling sub-model and a learner collaborative state classification sub-model. The conversational text and EEG signal joint modeling sub-model includes a multimodal variational autoencoder consisting of two encoders and two decoders. The learner collaborative state classification sub-model includes a support vector machine classifier.

[0023] The conversational text features and the EEG signal features are input into the joint modeling sub-model of conversational text and EEG signal. The latent vector of conversational text is obtained through Text-Brain inference, and the latent vector of EEG signal is obtained through Brain-Text inference.

[0024] The latent vector of the conversation text and the latent vector of the EEG signal are concatenated to obtain a joint representation of the conversation text and the EEG signal. The joint representation of the conversation text and the EEG signal is then input into the learner collaboration state classification sub-model to obtain the learner collaboration state category.

[0025] The loss functions are defined as follows: Text-Brain variational autoencoder loss function, cross-modal EEG signal data reconstruction loss function, Brain-Text variational autoencoder loss function, and cross-modal conversational text data reconstruction loss function.

[0026] According to the present invention, a learner collaborative state classification method for conversational EEG co-learning is provided, which obtains latent vectors of conversational texts through Text-Brain inference, including:

[0027] The conversational text features are input into the encoder TextEncoder to obtain the conversational text feature latent vector, and the EEG signal features are input into the encoder BrainEncoder to obtain the EEG signal feature latent vector;

[0028] Using the latent vector of the conversational text features as the approximate posterior distribution parameter and the latent vector of the EEG signal features as the prior distribution parameter, the latent vector of the conversational text is obtained by minimizing the relative entropy KL divergence between the approximate posterior distribution parameter and the prior distribution parameter.

[0029] The latent vectors of the conversational text are input into the decoder TextDecoder to obtain the reconstructed conversational text data, and the latent vectors of the conversational text are input into the decoder BrainDecoder to obtain the reconstructed cross-modal EEG signal data.

[0030] Correspondingly, the loss function of the Text-Brain variational autoencoder is obtained by subtracting the KL divergence constraint between the approximate posterior distribution generated by the conversational text encoder and the prior distribution generated by the EEG signal encoder from the maximum likelihood estimate of the feature vector reconstructed from the conversational text;

[0031] The cross-modal EEG signal data reconstruction loss function is obtained by subtracting the L1 norm of the difference between the cross-modal EEG signal reconstruction data and the feature vector of the EEG signal data.

[0032] According to the present invention, a learner collaborative state classification method for conversational EEG co-learning is provided, which obtains latent vectors of EEG signals through Brain-Text inference, including:

[0033] The EEG signal features are input into the BrainEncoder to obtain the latent vector of EEG signal features, and the conversation text features are input into the TextEncoder to obtain the latent vector of conversation text features.

[0034] Using the latent vector of the EEG signal features as an approximate posterior distribution parameter and the latent vector of the conversation text features as a prior distribution parameter, the latent vector of the EEG signal is obtained by minimizing the KL divergence between the approximate posterior distribution parameter and the prior distribution parameter.

[0035] The latent vector of the EEG signal is input into the decoder BrainDecoder to obtain the reconstructed EEG signal data, and the latent vector of the EEG signal is input into the decoder TextDecoder to obtain the reconstructed cross-modal conversation text data;

[0036] Correspondingly, the Brain-Text variational autoencoder loss function is obtained by subtracting the KL divergence constraint between the approximate posterior distribution generated by the EEG encoder inference and the prior distribution generated by the conversational text encoder inference from the maximum likelihood estimate of the reconstructed feature vector of the EEG signal.

[0037] The cross-modal conversational text data reconstruction loss function is obtained by subtracting the L1 norm of the difference between the conversational text data feature vector and the cross-modal conversational text reconstruction data.

[0038] Secondly, the present invention also provides a learner collaborative state classification system for conversational EEG co-learning, comprising:

[0039] The acquisition module is used to collect conversational text data and EEG signal data of learners in collaborative learning.

[0040] The preprocessing module is used to preprocess the conversation text data and the EEG signal data respectively to obtain preprocessed conversation text data and preprocessed EEG signal data.

[0041] The alignment and extraction module is used to align the preprocessed conversation text data and the preprocessed EEG signal data, and extract the conversation text features in the aligned preprocessed conversation text data and the EEG signal features in the aligned preprocessed EEG signal data.

[0042] The training module is used to input the conversational text features and the EEG signal features into the initial model for classifying learner collaborative states through conversational EEG joint learning, and to construct a loss function to optimize the model, thereby obtaining the learner collaborative state classification model.

[0043] The processing module is used to input the conversation text data to be detected and the EEG signal data to be detected into the learner collaborative state classification model, and output the learner collaborative state classification result.

[0044] Thirdly, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the learner collaborative state classification method for conversational EEG co-learning as described above.

[0045] Fourthly, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the learner collaborative state classification method for conversational EEG co-learning as described above.

[0046] The present invention provides a learner collaborative state classification method and system based on conversational EEG co-learning. By employing a co-learning approach using conversational text and EEG signals, and fusing data from different modalities through a multimodal variational autoencoder, information can flow bidirectionally between different modalities, enabling a more comprehensive capture of learners' cognitive states and behavioral characteristics. The method extracts features and selects appropriate representation methods based on the characteristics of data from different modalities, considering collaborative activity information indicators among learners and optimizing the shared feature representation of data from different modalities. Furthermore, the method improves the loss function, optimizes model parameters, enhances the co-modeling and generation capabilities of data from different modalities, and improves the accuracy of learner collaborative state classification. Attached Figure Description

[0047] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0048] Figure 1 This is one of the flowcharts of the learner collaborative state classification method for conversational EEG co-learning provided by the present invention;

[0049] Figure 2 This is the second flowchart of the learner collaborative state classification method for conversational EEG co-learning provided by the present invention;

[0050] Figure 3 This is a structural diagram of the learner collaboration state classification model provided by the present invention;

[0051] Figure 4 This is a flowchart of the Text-Brain inference process in the model structure provided by this invention;

[0052] Figure 5 This is a flowchart of the Brain-Text inference process in the model structure provided by this invention;

[0053] Figure 6 This is a schematic diagram of the learner collaborative state classification system for conversational EEG co-learning provided by the present invention;

[0054] Figure 7 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0055] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0056] To address the shortcomings of existing technologies, this invention utilizes the semantic features of conversational texts and the cognitive features of EEG signals to construct a joint learning learner collaborative state classification model, thereby improving the accuracy of learner collaborative state classification.

[0057] Figure 1 This is one of the flowcharts illustrating the learner collaborative state classification method for conversational EEG co-learning provided in this embodiment of the invention, such as... Figure 1 As shown, it includes:

[0058] Step 100: Collect conversational text data and EEG signal data of learners during collaborative learning;

[0059] Step 200: Preprocess the conversation text data and the EEG signal data respectively to obtain preprocessed conversation text data and preprocessed EEG signal data;

[0060] Step 300: Align the preprocessed conversation text data and the preprocessed EEG signal data, and extract the conversation text features in the aligned preprocessed conversation text data and the EEG signal features in the aligned preprocessed EEG signal data.

[0061] Step 400: Input the conversation text features and the EEG signal features into the initial model for classifying learner collaborative states of conversational EEG joint learning for training, construct a loss function to optimize the model, and obtain the learner collaborative state classification model;

[0062] Step 500: Input the conversation text data to be detected and the EEG signal data to be detected into the learner collaboration state classification model, and output the learner collaboration state classification result.

[0063] Specifically, such as Figure 2As shown, in this embodiment of the invention, the conversational text data and corresponding EEG signal data generated during collaborative learning by learners are first acquired. The conversational text data and EEG signal data are preprocessed respectively, and the preprocessed conversational text data and EEG signal data are aligned. Then, conversational text features and EEG signal features are extracted from the conversational text data and EEG signal data. The obtained conversational text features and EEG signal features are input into the learner collaborative state classification model constructed based on joint learning of conversational text and EEG signals. A loss function is constructed to optimize the classification result, and the trained learner collaborative state classification model is output. The conversational text data to be detected and the EEG signal data to be detected are input into the model to obtain the learner collaborative state classification result.

[0064] This invention employs a joint learning approach using conversational text and EEG signals, fusing data from different modalities through a multimodal variational autoencoder. This allows information to flow bidirectionally between different modalities, enabling a more comprehensive capture of learners' cognitive states and behavioral characteristics. It extracts features and selects appropriate representation methods based on the characteristics of data across different modalities, considering collaborative activity indicators among learners and optimizing the shared feature representation of data from different modalities. The improved loss function optimizes model parameters, enhancing the joint modeling and generation capabilities of data from different modalities and improving the accuracy of learner collaborative state classification. It demonstrates significant effectiveness in learner collaborative state analysis, outperforming methods that use only single-modal conversational text or EEG signals for classification, and improving analytical efficiency compared to traditional methods.

[0065] Based on the above embodiments, conversational text data and EEG signal data of learners collaborative learning are collected, including:

[0066] The conversation text data is determined to be in CSV format, and the EEG signal data is in SET format.

[0067] Specifically, the conversation text data and the EEG signal data are preprocessed to obtain preprocessed conversation text data and preprocessed EEG signal data, including:

[0068] The conversation text data is divided into multiple text segments of unequal length, and the multiple text segments are encoded based on the collaboration status category to obtain encoded conversation text. The encoded conversation text is recorded in a CSV file according to a preset record format to obtain the preprocessed conversation text data. The preset record format includes text content, start time, end time and encoding category.

[0069] The EEG signal data was processed using the EEGLAB toolbox to perform channel localization, removal of useless electrodes, filtering, downsampling, removal of bad segments and artifact removal. The EEG signal data was divided into multiple signal segments according to the start and end times in the conversation text data to obtain the preprocessed EEG signal data.

[0070] Specifically, in this embodiment of the invention, the conversation text data is saved as a CSV file, i.e., a comma-separated values ​​(CSV) file, and the EEG signals are saved as a SET file. The SET file usually refers to a file with the filename extension .set, which is mainly associated with the Universal Settings File (SET) file type. This extension is usually assigned to files that hold various settings and preferences for operating systems, applications, games, hardware devices, etc.

[0071] Furthermore, the conversation text is segmented into text segments of unequal length, and the segmented text segments are encoded, for example, into six categories: task understanding, planning, goal setting, monitoring and evaluation, strategy usage, and task execution. The segmented conversation text is then saved in a CSV file, with the record format being text content, start time, end time, and encoding category.

[0072] For the preprocessing of EEG signals, the EEGLAB toolbox in MATLAB was used to preprocess the EEG signals. The preprocessing operations included channel localization, removal of useless electrodes, filtering, downsampling, removal of bad segments, and removal of artifacts. Then, the EEG data was divided into segments of unequal length according to the start and end times of the conversation text recording.

[0073] Based on the above embodiments, the preprocessed conversation text data and the preprocessed EEG signal data are aligned, and conversation text features in the aligned preprocessed conversation text data and EEG signal features in the aligned preprocessed EEG signal data are extracted, including:

[0074] The preprocessed conversation text data and the preprocessed EEG signal data are aligned in chronological order to obtain the aligned preprocessed conversation text data and the aligned preprocessed EEG signal data.

[0075] The aligned and preprocessed conversation text data is input into the Chinese pre-trained model BERT to obtain word vector features, which are used as the conversation text features.

[0076] The fusion feature of the phase lock value (PLV) of every two EEG signals in the aligned and preprocessed EEG signal data is extracted as the EEG signal feature.

[0077] Specifically, in this embodiment of the invention, the conversational text data and EEG signal data are first aligned in chronological order.

[0078] The aligned conversation text content data is input into the Chinese pre-trained model BERT to obtain word vector features. The BERT model is used as the word embedding layer to learn the feature representation of the conversation text. BERT-specific markers are added to both ends of the conversation text content. Then, the conversation text is processed by word embedding, sentence embedding, and position embedding. The three are added and fused to obtain the input information of the BERT encoder. Finally, the vector representation of the conversation text is obtained by encoding through multiple stacked Transformer encoders.

[0079] The aligned EEG signal data is then automatically used to calculate the fusion characteristics of pairwise phase-locking values ​​(PLV) of multiple EEG signals. PLV is a statistical indicator used to analyze EEG signals. The phase-locking value measures the degree of phase synchronization between different individuals and brain regions, that is, whether the phase of the neural oscillation waves between different brain regions is synchronized at a specific frequency. The formula for calculating PLV is as follows:

[0080]

[0081] Where N represents the length of the signal segment, and Σ represents the phase difference between different signals, Σ represents the summation over all m and n, and i represents the imaginary unit.

[0082] This invention extracts features based on the characteristics of modal data and selects appropriate representation methods. In particular, in the extraction of EEG data features, it considers the characteristics of multi-person collaboration in collaborative learning and calculates the phase-locked value of the group data index in the EEG signal to obtain information about the collaborative activities between different learners.

[0083] Based on the above embodiments, the conversational text features and the EEG signal features are input into the initial model for learner collaborative state classification of conversational EEG joint learning for training. A loss function is constructed to optimize the model, resulting in a learner collaborative state classification model, including:

[0084] The initial model for classifying learner collaborative states includes a conversational text and EEG signal joint modeling sub-model and a learner collaborative state classification sub-model. The conversational text and EEG signal joint modeling sub-model includes a multimodal variational autoencoder consisting of two encoders and two decoders. The learner collaborative state classification sub-model includes a support vector machine classifier.

[0085] The conversational text features and the EEG signal features are input into the joint modeling sub-model of conversational text and EEG signal. The latent vector of conversational text is obtained through Text-Brain inference, and the latent vector of EEG signal is obtained through Brain-Text inference.

[0086] The latent vector of the conversation text and the latent vector of the EEG signal are concatenated to obtain a joint representation of the conversation text and the EEG signal. The joint representation of the conversation text and the EEG signal is then input into the learner collaboration state classification sub-model to obtain the learner collaboration state category.

[0087] The loss functions are defined as follows: Text-Brain variational autoencoder loss function, cross-modal EEG signal data reconstruction loss function, Brain-Text variational autoencoder loss function, and cross-modal conversational text data reconstruction loss function.

[0088] The latent vectors of the conversational text obtained through Text-Brain inference include:

[0089] The conversational text features are input into the encoder TextEncoder to obtain the conversational text feature latent vector, and the EEG signal features are input into the encoder BrainEncoder to obtain the EEG signal feature latent vector;

[0090] Using the latent vector of the conversational text features as the approximate posterior distribution parameter and the latent vector of the EEG signal features as the prior distribution parameter, the latent vector of the conversational text is obtained by minimizing the relative entropy KL divergence between the approximate posterior distribution parameter and the prior distribution parameter.

[0091] The latent vectors of the conversational text are input into the decoder TextDecoder to obtain the reconstructed conversational text data, and the latent vectors of the conversational text are input into the decoder BrainDecoder to obtain the reconstructed cross-modal EEG signal data.

[0092] Correspondingly, the loss function of the Text-Brain variational autoencoder is obtained by subtracting the KL divergence constraint between the approximate posterior distribution generated by the conversational text encoder and the prior distribution generated by the EEG signal encoder from the maximum likelihood estimate of the feature vector reconstructed from the conversational text;

[0093] The cross-modal EEG signal data reconstruction loss function is obtained by subtracting the L1 norm of the difference between the cross-modal EEG signal reconstruction data and the feature vector of the EEG signal data.

[0094] Among them, the latent vectors of EEG signals obtained through Brain-Text inference include:

[0095] The EEG signal features are input into the BrainEncoder to obtain the latent vector of EEG signal features, and the conversation text features are input into the TextEncoder to obtain the latent vector of conversation text features.

[0096] Using the latent vector of the EEG signal features as an approximate posterior distribution parameter and the latent vector of the conversation text features as a prior distribution parameter, the latent vector of the EEG signal is obtained by minimizing the KL divergence between the approximate posterior distribution parameter and the prior distribution parameter.

[0097] The latent vectors of the EEG signal features are input into the decoder BrainDecoder to obtain EEG signal reconstruction data, and the latent vectors of the EEG signal features are input into the decoder TextDecoder to obtain cross-modal conversation text reconstruction data;

[0098] Correspondingly, the Brain-Text variational autoencoder loss function is obtained by subtracting the KL divergence constraint between the approximate posterior distribution generated by the EEG encoder inference and the prior distribution generated by the conversational text encoder inference from the maximum likelihood estimate of the reconstructed feature vector of the EEG signal.

[0099] The cross-modal conversational text data reconstruction loss function is obtained by subtracting the L1 norm of the difference between the conversational text data feature vector and the cross-modal conversational text reconstruction data.

[0100] Specifically, the learner collaboration state classification model structure constructed in this embodiment of the invention is as follows: Figure 3 As shown, the extracted conversation text features X t and EEG signal characteristics X b As input to the model, data X t X b Moving on to the first part of the model: the joint modeling sub-model of conversational text and EEG signals, the first part consists of, for example... Figure 4 The text-brain reasoning shown, and as shown Figure 5 The Brain-Text inference components are shown.

[0101] First is Text-Brain reasoning, such as Figure 4 As shown:

[0102] X of the conversation text features t and EEG signal characteristics X b Input the respective encoders, TextEncoder and BrainEncoder, to obtain the latent vector μ of the conversational text features. t , σ t and the latent vector μ of EEG signal characteristics b , σ b .

[0103] TextEncoder and BrainEncoder have the same structure, both using two fully connected layers to generate μ. m , σ m Its formula is:

[0104] μ m =w1X m +b1

[0105] σ m =w2X m +b2

[0106] Where w1 and w2 are the weight matrices of the encoder network, b1 and b2 are the bias vectors, and m∈{t, b}.

[0107] The latent vector of the conversation text is used as the approximate posterior distribution parameter (μ). t , σ t The latent vector of the EEG signal is used as the prior distribution parameter (μ). b , σ b By minimizing the approximate posterior distribution With prior distribution p θ (Z b |X b The KL divergence of the conversational text data makes the latent vector generated from the conversational text data... Constrained by EEG signal data.

[0108] Generate Z t The process can be represented by the following formula:

[0109] Z t =μ t +σ t ⊙∈

[0110] Where ∈~N(0,1) is a noise vector sampled from the standard normal distribution, and ⊙ represents element-wise product.

[0111] The KL divergence is expressed by the following formula:

[0112]

[0113] The obtained Z t Simultaneously inputting the decoders TextDecoder and BrainDecoder, we obtain the reconstructed data T of the conversation text. t And cross-modal EEG signal reconstruction data T b ;

[0114] Its generation formula is as follows:

[0115] T t =TextDecoder(Z t )

[0116] TextDecoder(Z t ) = ReLU(W t2·ReLU(W t1 ·Z t +b t1 )+b t2 )

[0117] T b =BrainDecoder(Z) t )

[0118] BrainDecoder(Z t ) = ReLU(W b2 ·ReLU(W b1 ·Z t +b b1 )+b b2 )

[0119] Both TextDecoder and BrainDecoder are decoder networks with corresponding parameters, each consisting of two fully connected layers and a ReLU activation function. t1 W t2 b t1 b t2 W b1 W b2 b b1 b b2 These are the corresponding parameter matrices and bias terms, used to measure the latent variable Z. t Mapping back to the data space generates the corresponding session text reconstruction data T t and cross-modal EEG signal data T b

[0120] Correspondingly, the loss function of Text-Brain inference It consists of two parts: the loss L of the Text-Brain variational autoencoder. VAE1 Cross-modal EEG data reconstruction loss L CAB .

[0121]

[0122] L CAB =||X b -T b ||1

[0123]

[0124] Where E(·) represents the expectation, q Φ (Z t |X t ) and q θ (Z b |X b) represent the conditional distribution functions of the encoder under the corresponding modes, p Φ (Z t |X t ) and p θ (Z b |X b ) represent the conditional distribution functions of the decoder in the corresponding modalities. D represents the maximum likelihood estimate of the feature vector reconstructed from the conversation text. KL D represents the KL divergence distance between functions. KL (q Φ (Z t |X t )||p θ (Z b |X b )) represents the KL divergence constraint between the approximate posterior distribution generated by the conversational text encoder and the prior distribution generated by the EEG encoder, and ||·||1 represents the L1 norm.

[0125] Then comes Brain-Text reasoning, such as... Figure 5 As shown:

[0126] X-ray of brainwave signal characteristics b and conversation text features X t Input the respective encoders BrainEncoder and TextEncoder. Obtain the latent vector μ of the EEG signal features. b , σ b And the latent vector μ of the conversation text features t , σ t .

[0127] Understandably, BrainEncoder and TextEncoder have the same structure, both using two fully connected layers to generate μ. m , σ m Its formula is:

[0128] μ m =w1X m +b1

[0129] σ m =w2X m +b2

[0130] Where w1 and w2 are the weight matrices of the encoder network, b1 and b2 are the bias vectors, and m∈{t, b}.

[0131] The latent vector of the EEG signal data is used as the approximate posterior distribution parameter (μ). b , σ b The latent vector of the conversation text is used as the prior distribution parameter (μ).t , σ t By minimizing the approximate posterior distribution p θ (Z b |X b ) and prior distribution The KL divergence makes the latent vectors of the conversational text data... Constrained by EEG signal data.

[0132] Generate Z b The process can be represented by the following formula:

[0133] Z b =μ b +σ b ⊙∈

[0134] Where ∈~N(0,1) is a noise vector sampled from the standard normal distribution, and ⊙ represents element-wise product.

[0135] The KL divergence is expressed by the following formula:

[0136]

[0137] The obtained Z b Simultaneously inputting into the decoders BrainDecoder and TextDecoder, the reconstructed EEG signal data B is obtained. b And reconstructed data B of cross-modal conversation text t ;

[0138] Its generation formula is as follows:

[0139] B b =BrainDecoder(Z) b )

[0140] BrainDecoder(Z b ) = ReLU(W b2 ·ReLU(W b1 ·Z b +b b1 )+b b2 )

[0141] B t =TextDecoder(Z b )

[0142] TextDecoder(Z b ) = ReLU(W t2 ·ReLU(W t1 ·Z b +b t1 )+b t2 )

[0143] Both BrainDecoder and TextDecoder are decoder networks with corresponding parameters, each consisting of two fully connected layers and a ReLU activation function. b1 W b2 b b1 b b2 W t1 W t2 b t1 b t2 These are the corresponding parameter matrices and bias terms, used to measure the latent variable Z. b Mapping back to the data space generates corresponding EEG signal reconstruction data B b and cross-modal conversation text data B t .

[0144] Correspondingly, the loss function for Brain-Text inference It consists of two parts: the loss L of the Brain-Text variational autoencoder. VAE2 Cross-modal EEG data reconstruction loss L CAT .

[0145]

[0146] L CAT =||X t -B t ||1

[0147]

[0148] Where E(·) represents the expectation, q θ (Z b |X b ) and q Φ (Z t |X t ) represent the conditional distribution functions of the encoder under the corresponding modes, p θ (Z b |X b ) and p Φ (Z t |X t ) represent the conditional distribution functions of the decoder in the corresponding modalities. D represents the maximum likelihood estimate of the reconstructed feature vector of the EEG signal. KL D represents the KL divergence distance between functions. KL (q Φ (Z b |X b )||p θ (Z t |Xt )) represents the KL divergence constraint between the approximate posterior distribution generated by the EEG signal encoder inference and the prior distribution generated by the conversational text encoder inference, ||·||1 represents the L1 norm, X t B represents the feature vector of the conversational text data. t This represents cross-modal conversational text reconstruction data.

[0149] Combining all loss functions, the loss function for the first part of the model It consists of four parts, including the loss L of the Text-Brain variational autoencoder. VAE1 Cross-modal EEG signal data reconstruction loss L CAB The loss L of the Brain-Text variational autoencoder VAE2 Cross-modal conversation text data reconstruction loss L CAT .

[0150]

[0151]

[0152]

[0153]

[0154] Furthermore, the latent vector Z of the conversational text is obtained by jointly modeling the conversational text and EEG signals. t and the latent vector Z of EEG signals b The data is the second part of the input model, namely the classifier prediction.

[0155] latent vector Z of the conversation text t and the latent vector Z of EEG signals b The data are concatenated to obtain the joint representation Z of the conversation text and EEG signals, as shown in the following formula:

[0156]

[0157] in, This indicates that two vectors are concatenated.

[0158] Z is input into a support vector machine classifier to predict the category of the learner's collaborative state.

[0159] Specifically, Support Vector Machine (SVM) is a supervised machine learning algorithm primarily used for classification and regression problems.

[0160] The classification decision function of a support vector machine can be expressed as:

[0161] f(Z) = sign(w) T X+b)

[0162] Where X is the feature vector of the input sample, w is the weight vector of the decision function, b is the bias term (threshold), and sign() is the sign function, which maps the function value greater than 0 to +1 and the value less than 0 to -1.

[0163] The optimization objective of a Support Vector Machine (SVM) is to find an optimal hyperplane that separates the two classes of samples. For the linearly separable case, the optimization objective of the SVM can be expressed as:

[0164]

[0165] sty i (w T x i +b)≥1, i=1,2,...,N

[0166] Where, x i y is the feature vector of the i-th sample. i Let be the label (+1 or -1) of the i-th sample, w be the weight vector, b be the bias term (threshold), and N be the number of samples. In this optimization problem, we aim to maximize the margin. At the same time, ensure that all sample points meet the classification constraints.

[0167] This invention achieves multimodal joint modeling through joint learning of conversational text and EEG signals. It innovatively uses Text-Brain inference and Brain-Text inference to enable bidirectional information flow between different modalities, which can more comprehensively model the relationships between different modalities, promote the integration of unique and shared information from conversational text and EEG signals, and realize cross-modal inference and generation. In addition, by optimizing the loss function, the performance of the classification model is improved, especially when the sample size is small, the improvement in classification performance is more significant.

[0168] The learner collaboration state classification system for conversational EEG co-learning provided by this invention is described below. The learner collaboration state classification system for conversational EEG co-learning described below can be referred to in correspondence with the learner collaboration state classification method for conversational EEG co-learning described above.

[0169] Figure 6 This is a schematic diagram of the learner collaborative state classification system for conversational EEG co-learning provided in an embodiment of the present invention, as shown below. Figure 6 As shown, it includes: a data acquisition module 61, a preprocessing module 62, an alignment extraction module 63, a training module 64, and a processing module 65, wherein:

[0170] The acquisition module 61 is used to acquire conversational text data and EEG signal data of learners collaborative learning; the preprocessing module 62 is used to preprocess the conversational text data and the EEG signal data respectively to obtain preprocessed conversational text data and preprocessed EEG signal data; the alignment and extraction module 63 is used to align the preprocessed conversational text data and the preprocessed EEG signal data, and extract the conversational text features in the aligned preprocessed conversational text data and the EEG signal features in the aligned preprocessed EEG signal data; the training module 64 is used to input the conversational text features and the EEG signal features into the learner collaborative state classification initial model of conversational EEG joint learning for training, construct a loss function to optimize the model, and obtain the learner collaborative state classification model; the processing module 65 is used to input the conversational text data to be detected and the EEG signal data to be detected into the learner collaborative state classification model, and output the learner collaborative state classification result.

[0171] Figure 7 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 7 As shown, the electronic device may include: a processor 710, a communication interface 720, a memory 730, and a communication bus 740, wherein the processor 710, the communication interface 720, and the memory 730 communicate with each other through the communication bus 740. The processor 710 can call logic instructions in the memory 730 to execute a learner collaborative state classification method for conversational EEG co-learning. This method includes: collecting conversational text data and EEG signal data for learner collaborative learning; preprocessing the conversational text data and the EEG signal data respectively to obtain preprocessed conversational text data and preprocessed EEG signal data; aligning the preprocessed conversational text data and the preprocessed EEG signal data; extracting conversational text features from the aligned preprocessed conversational text data and EEG signal features from the aligned preprocessed EEG signal data; inputting the conversational text features and the EEG signal features into an initial model for learner collaborative state classification of conversational EEG co-learning for training; constructing a loss function to optimize the model to obtain a learner collaborative state classification model; inputting the conversational text data to be detected and the EEG signal data to be detected into the learner collaborative state classification model, and outputting the learner collaborative state classification result.

[0172] Furthermore, the logical instructions in the aforementioned memory 730 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0173] On the other hand, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements a learner collaborative state classification method for conversational EEG co-learning provided by the above methods. This method includes: collecting conversational text data and EEG signal data for learner collaborative learning; preprocessing the conversational text data and the EEG signal data respectively to obtain preprocessed conversational text data and preprocessed EEG signal data; aligning the preprocessed conversational text data and the preprocessed EEG signal data; extracting conversational text features from the aligned preprocessed conversational text data and EEG signal features from the aligned preprocessed EEG signal data; inputting the conversational text features and the EEG signal features into an initial model for learner collaborative state classification of conversational EEG co-learning for training; constructing a loss function to optimize the model to obtain a learner collaborative state classification model; inputting the conversational text data to be detected and the EEG signal data to be detected into the learner collaborative state classification model, and outputting the learner collaborative state classification result.

[0174] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0175] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0176] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A learner collaborative state classification method for conversational EEG co-learning, characterized in that, include: Collect conversational text data and EEG signal data of learners in collaborative learning; The conversation text data and the EEG signal data are preprocessed respectively to obtain preprocessed conversation text data and preprocessed EEG signal data. The preprocessed conversation text data and the preprocessed EEG signal data are aligned, and the conversation text features in the aligned preprocessed conversation text data and the EEG signal features in the aligned preprocessed EEG signal data are extracted. The conversational text features and the EEG signal features are input into the initial model for learner collaborative state classification based on conversational EEG joint learning for training. A loss function is constructed to optimize the model, resulting in the learner collaborative state classification model, including: The initial model for classifying learner collaborative states includes a conversational text and EEG signal joint modeling sub-model and a learner collaborative state classification sub-model. The conversational text and EEG signal joint modeling sub-model includes a multimodal variational autoencoder consisting of two encoders and two decoders. The learner collaborative state classification sub-model includes a support vector machine classifier. The conversational text features and the EEG signal features are input into the joint modeling sub-model of conversational text and EEG signal. The latent vector of conversational text is obtained through Text-Brain inference, and the latent vector of EEG signal is obtained through Brain-Text inference. The latent vector of the conversation text and the latent vector of the EEG signal are concatenated to obtain a joint representation of the conversation text and the EEG signal. The joint representation of the conversation text and the EEG signal is then input into the learner collaboration state classification sub-model to obtain the learner collaboration state category. The loss functions are defined as follows: Text-Brain variational autoencoder loss function, cross-modal EEG signal data reconstruction loss function, Brain-Text variational autoencoder loss function, and cross-modal conversational text data reconstruction loss function. The learner collaborative state classification model is input into the conversation text data and EEG signal data to be detected, and the learner collaborative state classification result is output.

2. The learner collaborative state classification method for conversational EEG co-learning according to claim 1, characterized in that, Collect conversational text data and EEG signal data of learners during collaborative learning, including: The conversation text data is determined to be in CSV format, and the EEG signal data is in SET format.

3. The learner collaborative state classification method for conversational EEG co-learning according to claim 1, characterized in that, The conversation text data and the EEG signal data are preprocessed separately to obtain preprocessed conversation text data and preprocessed EEG signal data, including: The conversation text data is divided into multiple text segments of unequal length, and the multiple text segments are encoded based on the collaboration status category to obtain encoded conversation text. The encoded conversation text is recorded in a CSV file according to a preset record format to obtain the preprocessed conversation text data. The preset record format includes text content, start time, end time and encoding category. The EEG signal data was processed using the EEGLAB toolbox to perform channel localization, removal of useless electrodes, filtering, downsampling, removal of bad segments and artifact removal. The EEG signal data was divided into multiple signal segments according to the start and end times in the conversation text data to obtain the preprocessed EEG signal data.

4. The learner collaboration state classification method for conversational EEG co-learning according to claim 1, characterized in that, Align the preprocessed conversation text data and the preprocessed EEG signal data, and extract the conversation text features from the aligned preprocessed conversation text data and the EEG signal features from the aligned preprocessed EEG signal data, including: The preprocessed conversation text data and the preprocessed EEG signal data are aligned in chronological order to obtain the aligned preprocessed conversation text data and the aligned preprocessed EEG signal data. The aligned and preprocessed conversation text data is input into the Chinese pre-trained model BERT to obtain word vector features, which are used as the conversation text features. The fusion feature of the phase lock value (PLV) of every two EEG signals in the aligned and preprocessed EEG signal data is extracted as the EEG signal feature.

5. The learner collaboration state classification method for conversational EEG co-learning according to claim 1, characterized in that, The latent vectors of the conversational text are obtained through Text-Brain inference, including: The conversational text features are input into the encoder TextEncoder to obtain the conversational text feature latent vector, and the EEG signal features are input into the encoder BrainEncoder to obtain the EEG signal feature latent vector; Using the latent vector of the conversational text features as the approximate posterior distribution parameter and the latent vector of the EEG signal features as the prior distribution parameter, the latent vector of the conversational text is obtained by minimizing the relative entropy KL divergence between the approximate posterior distribution parameter and the prior distribution parameter. The latent vectors of the conversational text are input into the decoder TextDecoder to obtain the reconstructed conversational text data, and the latent vectors of the conversational text are input into the decoder BrainDecoder to obtain the reconstructed cross-modal EEG signal data; Correspondingly, the loss function of the Text-Brain variational autoencoder is obtained by subtracting the KL divergence constraint between the approximate posterior distribution generated by the conversational text encoder and the prior distribution generated by the EEG signal encoder from the maximum likelihood estimate of the feature vector reconstructed from the conversational text; The cross-modal EEG signal data reconstruction loss function is obtained by subtracting the L1 norm of the difference between the cross-modal EEG signal reconstruction data and the feature vector of the EEG signal data.

6. The learner collaborative state classification method for conversational EEG co-learning according to claim 1, characterized in that, Brain-Text inference yields latent vectors of EEG signals, including: The EEG signal features are input into the BrainEncoder to obtain the latent vector of EEG signal features, and the conversation text features are input into the TextEncoder to obtain the latent variables of conversation text features; Using the latent vector of the EEG signal features as an approximate posterior distribution parameter and the latent vector of the conversation text features as a prior distribution parameter, the latent vector of the EEG signal is obtained by minimizing the KL divergence between the approximate posterior distribution parameter and the prior distribution parameter. The latent vector of the EEG signal is input into the decoder BrainDecoder to obtain the reconstructed EEG signal data, and the latent vector of the EEG signal is input into the decoder TextDecoder to obtain the reconstructed cross-modal conversation text data; Correspondingly, the Brain-Text variational autoencoder loss function is obtained by subtracting the KL divergence constraint between the approximate posterior distribution generated by the EEG encoder inference and the prior distribution generated by the conversational text encoder inference from the maximum likelihood estimate of the reconstructed feature vector of the EEG signal. The cross-modal conversational text data reconstruction loss function is obtained by subtracting the L1 norm of the difference between the conversational text data feature vector and the cross-modal conversational text reconstruction data.

7. A learner collaboration state classification system for conversational EEG co-learning, based on the learner collaboration state classification method for conversational EEG co-learning according to any one of claims 1 to 6, characterized in that, include: The acquisition module is used to collect conversational text data and EEG signal data of learners in collaborative learning. The preprocessing module is used to preprocess the conversation text data and the EEG signal data respectively to obtain preprocessed conversation text data and preprocessed EEG signal data. The alignment and extraction module is used to align the preprocessed conversation text data and the preprocessed EEG signal data, and extract the conversation text features in the aligned preprocessed conversation text data and the EEG signal features in the aligned preprocessed EEG signal data. The training module is used to input the conversational text features and the EEG signal features into the initial model for classifying learner collaborative states through conversational EEG joint learning, and to construct a loss function to optimize the model, thereby obtaining the learner collaborative state classification model. The processing module is used to input the conversational text data to be detected and the EEG signal data to be detected into the learner collaborative state classification model, and output the learner collaborative state classification result.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the learner collaborative state classification method for conversational EEG co-learning as described in any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the learner collaborative state classification method for conversational EEG co-learning as described in any one of claims 1 to 6.

Images