Online learning evaluation method and system based on time attention fused multi-modal data

Abstract

The invention discloses an online learning evaluation method and system based on time attention fused multi-modal data. The learning state of a student is evaluated in two dimensions of emotion and behavior on the whole. In an emotion dimension, a video acquisition device, a heart rate acquisition device and an audio acquisition device are used for acquiring facial expression data, electrocardiosignal data and audio data generated in an online learning process of a student to perform multi-modal data fusion; and carrying out fusion calculation on the multi-modal data by combining an online learning feature fusion method of time attention to analyze sentiment classification of students in an online learning process, and inputting priori knowledge in a behavior dimension to assist in learning state evaluation. And finally, fusing analysis of emotion dimensions and reference of behavior end data, and finally generating a relatively accurate evaluation result. The system facilitates the supervision of teachers, improves the learning efficiency of students, and assists the teachers in the design of subsequent teaching schemes.

Description

Technical Field

[0001] The present invention relates to the fields of multimodal data fusion, artificial intelligence, online teaching, and in particular to an online learning evaluation method and system based on temporal attention fusion of multimodal data. Background Art

[0002] In the current social environment, online education and teaching have gradually become the norm and are gradually accepted by teachers and students. Distance education and teaching also reflect the advantages of convenience and low teaching cost. However, with the popularization of distance education and teaching, compared with traditional classroom teaching, teachers can easily grasp the students' class status and accurately remind them. In online teaching, due to the limitations of online classrooms, teachers cannot grasp the students' listening status very well. For example, it is difficult to make correct judgments on the narrow video window and the students' audio response information, so they cannot grasp the students' listening status very well. Students will also have a fluke mentality due to the network media, triggering phenomena such as truancy, distraction and even cheating. It seriously affects the students' listening efficiency and teaching quality. Therefore, it is particularly important to design a learning process evaluation system under the background of online teaching. It can help teachers to have a more accurate grasp of the students' correct psychological state and the degree of knowledge absorption immediately, which is convenient for maintaining a healthy classroom order and the development of subsequent teaching plans. There have been a number of studies on using single-modal data to analyze students' learning status at home and abroad. Compared with using only one type of data, the results obtained from multiple modal data are more convincing, so there have been studies based on multimodal data fusion. Using artificial intelligence algorithms to analyze image data generated in online learning is the current mainstream method. In the current online learning research at home and abroad, researchers have also used EEG, heart rate, audio and other data for analysis. How to fuse these different modal data to generate a computational dimension explosion during the fusion process and improve the fusion performance of different modal data. In data fusion, for feature-level fusion, the attention mechanism is the most popular method in the current deep learning field. Therefore, based on the universality and effectiveness of the time attention mechanism, the present invention combines the background discussion of specific problems to propose an online learning evaluation method based on time attention fusion multimodal data, and applies it to the field of learning process evaluation. Summary of the invention

[0003] The purpose of the present invention is to provide an online learning evaluation method based on time-attention fusion of multimodal data, so as to solve the above-mentioned problems. The overall evaluation of the online learning process is divided into two dimensions: emotion and behavior to evaluate the learning status of students. In the emotional dimension, video acquisition equipment, heart rate acquisition equipment, and audio acquisition equipment are used to collect facial expression data, electrocardiogram data, and audio data generated during the online learning process of students for multimodal data fusion. Combined with the online learning feature fusion method of time attention, these multimodal data are fused and calculated to analyze the emotional classification of the students' online learning process. In the behavioral dimension, prior knowledge is input, such as the number of times students switch backstages during the learning process, student quality evaluation, academic performance and other text information, the number of active speeches; the number of correct answers to questions and other prior knowledge or behavioral information generated during the learning process, to assist in the evaluation of the learning status. Finally, the analysis of the emotional dimension and the reference of the behavioral end data are integrated to finally generate a more accurate evaluation result. It is convenient for teachers to supervise and improve students' learning efficiency, and assist teachers in the design of subsequent teaching plans.

[0004] To achieve the above purpose, the technical solution adopted by the present invention is: an online learning evaluation method based on time attention fusion multimodal data, including the following steps:

[0005] Step S1, after the teacher starts the class, the emotional dimension data and the behavioral dimension data are obtained respectively; the emotional dimension data includes the student video data, electrocardiogram signal data, and audio data;

[0006] Step S2, using the pre-trained GTCN neural network to extract features from the video data, the ECG signal data, and the audio data to obtain single-modal features;

[0007] Step S3, mapping the features of different modalities to the same common subspace, taking the video modality as the main modality, and the audio and ECG modalities as auxiliary modalities, using a unidirectional weighted attention module to improve the expression ability of the video modality in the audio and ECG modalities, and after several unidirectional weighted attention modules, obtaining the final unimodal feature information;

[0008] Step S4, concatenating the final single-modal feature information to obtain multi-modal fusion features, and inputting them into the prediction layer to obtain the sentiment classification label;

[0009] Step S5, construct a mapping dictionary K with the sentiment classification labels obtained in step S4, K contains the classification labels and their corresponding scores; input the mapping matrix corresponding to the sentiment labels and the input of the behavior dimension into the multivariate linear regression model together to obtain the final learning evaluation score output.

[0010] Further, in step S1, online learning data of students is collected through a visual acquisition device, an electrocardiogram acquisition device, and an audio acquisition device to obtain student video data, electrocardiogram signal data, and audio data. The number of times students answer questions, the number of correct answers, and the online learning duration are recorded in real time through the background of the online learning software as real-time behavior data. At the same time, text information such as students' historical test scores and teachers' comments is recorded as behavior dimension data.

[0011] Further, in step S2, according to a preset time interval, the student video data collected by the visual acquisition device is constructed into a video tensor. The p-TRPCA low-rank and sparse constraint method is used to process the video tensor to achieve data compression and dimensionality reduction, and then it is input into a pre-trained GTCN neural network for video feature extraction.

[0012] Further, the specific implementation method of the p-TRPCA low-rank and sparse constraint method is as follows;

[0013] First, the low-rank component L and the sparse component S are separated from the observed tensor X. The tensor nuclear norm is used as the low-rank constraint, and the weighted l p -norm is used as the sparse constraint of the singular value, and the l p -norm is used as the constraint of the sparse component. The objective function of p-TRPCA is:

[0014]

[0015] where 0 < p < 1, w is the weight vector, s p represents the weighted Schatten p-norm, and λ is a positive balance parameter. Actually, it is the weighted l p -norm of the singular value, represents the l p -norm of S.

[0016] Further, the GTCN neural network is stacked by k hidden layers. The specific structure of each hidden layer is as follows:

[0017] Input data X t-1 is processed in three branches. In the left branch, the input first passes through 2 dilated causal convolution units and then passes through the gating mechanism G(·); each dilated causal convolution unit includes a dilated causal convolution operation, a weight normalization layer, an activation layer, and a Dropout inactivation layer. The specific formula is expressed as:

[0018]

[0019] where DDC represents the dilated causal convolution operation, Relu represents the activation function, and Norm represents the normalization layer;

[0020] In the right branch, the input passes through the gating mechanism G(·), which uses the weights generated by the Sigmoid activation function to control the proportion of different information. Its output is then added to the random downsampling result of the input x to obtain the output. The gating rule formula is expressed as follows:

[0021] G(x)=Sigmoid(Linear(x))⊙x

[0022] Among them, Linear represents the standard linear layer;

[0023] The middle branch is a one-dimensional convolution operation Conv, and only when X t With X t-1 The subscript t represents the time series dimension, which means that the model is updated from time t-1 to time t. The final output is expressed as follows:

[0024]

[0025] Furthermore, the unidirectional weighted attention module is composed of two unidirectional cross-modal attention UCAs, wherein the unidirectional cross-modal attention includes a standard linear layer Linear, a multi-head cross-modal attention operation MHCA, a layer normalization operation Norm, and a feed-forward neural network FFN;

[0026] First, define the data X from two modalities a , X b , then the UCA processing process is as follows:

[0027] Q a =Linear(X a )

[0028] K b , V b =Linear(X b )

[0029]

[0030]

[0031]

[0032] X a ′=FFN(Norm(X a ))

[0033] Setting W q , W k , W v represents the corresponding weight matrix, then Q a =Xa W q , K b =X b W k , V b =X b W v , where Q comes from mode X a , K, V come from mode X b ; The multi-head cross-modal attention operation MHCA is implemented through multiple unidirectional attention mechanisms. The calculation of a single attention mechanism is as follows:

[0034]

[0035] in K b The transpose of k represents the dimension of the key vector;

[0036] Audio modality and ECG modality as X of UCA a Input, video modality as X b Input, calculate the UCA output of audio mode and video mode and UCA output for ECG and video modalities and Where a represents the audio mode, h represents the ECG mode, and v represents the video mode. and Perform stitching to obtain enhanced video modality features The specific formula is as follows:

[0037]

[0038]

[0039]

[0040] Among them, Project represents the projection operation, and i represents the number of layers; finally, after being processed by L layers of unidirectional weighted attention modules, the output of the last unidirectional weighted module is linearly projected and reduced in dimension as the final unimodal information feature Z v , Z a , Z h , Z v represents the final video modality feature, Z a Represents the final audio modal features, Z h Represents the final ECG modality features.

[0041] Furthermore, in step S4, the final unimodal feature information is concatenated to obtain the final multimodal fusion feature output, and then the softmax activation function is used to obtain the final classification label. The specific formula is as follows:

[0042]

[0043] Among them, Project represents the projection operation, Z v represents the final video modality feature, Z a Represents the final audio modal features, Z h Represents the final ECG modality features; the loss function used in the prediction layer is cross entropy CrossEntropy, and the calculation process is as follows:

[0044] L SC =CrossEntropy(ylabel,ytrue).

[0045] Among them, ytrue is the true label.

[0046] Furthermore, in step S5, the multivariate linear regression model uses mean square error MSE as the loss function, and the specific formula is as follows:

[0047]

[0048] Where n represents the number of samples, represents the predicted value, and yi represents the true value;

[0049] The final total loss function is expressed as:

[0050] L LoSS =L SC +L MSE

[0051] Among them, L SC is the loss function of the prediction layer in step S4.

[0052] Furthermore, the specific implementation method of obtaining the pre-trained GTCN is as follows:

[0053] The first step is to build a face sample library for students when they are learning, including face images of people with head-mounted EEG devices and public face libraries. The audio data generated during the learning process is used to build an audio database. The electrocardiogram database of students when they are learning is also built, including heart rate and blood pressure data uploaded by wearable devices.

[0054] The second step is to perform data enhancement and labeling preprocessing on the collected sample data;

[0055] The third step is to use the GTCN neural network to train and test the video data, audio data, and ECG data in the constructed sample library. By adjusting parameters and using different image processing methods, we can continuously optimize the model with the best accuracy, namely the pre-trained GTCN neural network.

[0056] The present invention also provides an online learning evaluation system based on time attention fusion multimodal data, comprising:

[0057] one or more processors;

[0058] A storage device for storing one or more programs, which, when executed by the one or more processors, enables the one or more processors to implement an online learning evaluation method based on temporal attention fusion of multimodal data as described in the above scheme.

[0059] Compared with the prior art, the advantages and beneficial effects of the present invention are as follows:

[0060] 1. Considering the feature explosion problem caused by the multimodal feature fusion process, the tensor-based p-TRPCA low-rank sparse constraint method is used to compress and reduce the dimension of the video modality data, reduce the computational complexity, and better match the audio and ECG modality features with less information, thereby improving the multimodal fusion performance.

[0061] 2. Considering the temporal nature of video, audio, and ECG data, a temporal convolutional network (GTCN) with filter gating is used to complete the feature extraction of unimodal data and enhance the expression ability of unimodal feature information.

[0062] 3. The unidirectional weighted attention module is used to realize the interaction of unimodal information, extract the enhanced video modality features and retain the independence of audio and ECG modalities, while taking into account the extraction of multimodal common information and unimodal unique information.

[0063] 4. The modeling combines the data from the two dimensions of emotion and behavior. Compared with traditional multimodal neural networks, it has stronger interpretability and the model output is more persuasive. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 It is a flow chart of the technical solution of the present invention.

[0065] Figure 2 This is the hidden layer structure diagram of GTCN.

[0066] Figure 3 This is the structural diagram of the unidirectional cross-modal attention module. DETAILED DESCRIPTION

[0067] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments.

[0068] like Figure 1 As shown in FIG. 1 , the overall input of the online learning evaluation of the present invention is divided into two aspects: the emotional dimension and the behavioral dimension to comprehensively evaluate the student's online learning process. In the emotional dimension, a multimodal fusion mechanism is used to identify and classify emotions. The emotional segment input is divided into three modal data: U v Indicates video mode input, U a Represents the audio dimension input, U h Represents the ECG modality input; considering that the dimension and information richness of the video modality data in the online learning process are much higher than those of the ECG and audio modal data, which makes it difficult to align the data and perform later fusion, we consider using the video modality as the main modality and the audio and ECG modalities as auxiliary modalities to give full play to the sharing and independence of multimodal data. The video modality data is constructed in the form of a tensor, and then the dimensionality reduction and compression of the video modality data are achieved based on the improved low-rank sparse constraint algorithm p-TRPCA algorithm to solve the dimensionality explosion problem caused by the multimodal feature fusion process. Considering the temporal characteristics of the data, the GTCN gated temporal convolutional neural network is used to extract single-modal features. The network has a strong performance in time series prediction tasks. For the extracted single-modal features: X, video modality features, X a Audio modal characteristics, X h ECG modality features; a one-way weighted attention module is used for fusion learning of multimodal features. The one-way weighted module not only uses the attention mechanism for cross-modal interaction, but also maintains the independence of single-modal features to a certain extent. In the specific process, we perform one-way weighted attention calculation on the video modality and the audio modality to obtain the video modality feature X under the guidance of audio. a→v , and the video modality feature X under the guidance of ECG modality h→v , and then concatenate the two video modality features into a new enhanced video modality feature X v ; The ECG mode and audio mode retain the unique information of their own modes as much as possible. After iterative learning of L layers of unidirectional weighted modules, our Z is finally generated v Video modality features, Z a Audio modal characteristics, Z hECG modal features; then the three single modal features are spliced ​​and input into the sentiment classification prediction layer to obtain the final output at the sentiment dimension end; some behavioral data generated during the online learning process are input into the behavioral dimension end, such as the number of active speeches, the correct rate of answering questions in class, the time of leaving the online class, and other real-time data; and some prior information such as historical data such as student historical quality assessment and student grades can also be input; these contents usually play a certain supporting role in student assessment work. These and the inputs predicted in the sentiment dimension are used as independent variables of the multivariate linear regression problem, and input into the final multivariate linear regression layer to obtain the final learning process evaluation score. It is convenient for teachers to supervise students' learning status and arrange subsequent teaching work.

[0069] The embodiment of the present invention provides an online learning evaluation method based on time attention fusion multimodal data, such as Figure 1 As shown, the implementation of the method includes the following steps:

[0070] S 1. After the teacher starts the class, the visual acquisition device, electrocardiogram acquisition device, and audio acquisition device begin to collect students' online learning data as the data set for the subsequent emotion dimension recognition end; at the same time, the online learning software background records the real-time behavioral data such as the number of times students answer questions, the number of correct speeches, and the length of online learning, and also records text information such as students' historical test scores and teachers' comments, and inputs them into the behavioral dimension end.

[0071] S2, according to the preset time interval, construct the student image data collected by the visual acquisition device into a tensor, use the p-TRPCA low-rank sparse constraint method to process the video tensor to achieve data compression and dimensionality reduction. Then input it into the pre-trained GTCN neural network to extract video features; the audio data collected by the audio acquisition device and the ECG data collected by the ECG acquisition device are directly input into the GTCN network model to extract the features of the ECG modality and audio modality, and use filter gating to effectively filter redundant information, so as to increase the proportion of effective information in the single modality feature.

[0072] S3 maps the features of different modalities to the same common subspace. Considering the richness of information contained in video modality data, the video modality is used as the main modality, and the audio and ECG modalities are used as auxiliary modalities. The unidirectional weighted attention module is considered to improve the expressiveness of the video modality.

[0073] S4, the final single-modal feature information is spliced ​​to obtain multi-modal fusion features, which are input into the prediction layer to obtain the sentiment classification label.

[0074] S5, constructs a mapping dictionary K with the sentiment classification labels obtained in S4, K contains the classification labels and their corresponding scores; inputs the mapping matrix corresponding to the sentiment labels and the input of the behavior dimension into the multivariate linear regression model together to obtain the final learning evaluation score output.

[0075] There is also preparation work before S1, which is to construct the extraction model for each mode. The steps are:

[0076] S11, build a face sample library for students when they are learning, including face images of people with head-mounted EEG devices and public face libraries. The audio data generated during the learning process, such as the audio data when answering questions, build an audio database, and build an ECG database for students when they are learning, usually from heart rate, blood pressure and other data uploaded by wearable devices.

[0077] S12, perform preprocessing such as data enhancement and labeling on the collected sample data.

[0078] S13, using the GTCN neural network to train and test video data, audio data, and ECG data in the constructed sample library, continuously optimized through parameter adjustment and different image processing methods, and obtained a high-precision single-modal feature extraction model, namely the pre-trained GTCN neural network.

[0079] The p-TRPCA low-rank sparse constraint method in S2 is specifically described as follows:

[0080] S21, p-TRPCA is a groundbreaking tensor robust principal component analysis model that can effectively separate the low-rank component L and the sparse component S from the observation tensor X. It uses the tensor nuclear norm as a low-rank constraint. The p-TRPCA model takes the weighted l p -norm as a sparse constraint for singular values, l p -norm as a constraint for sparse components. The objective function of p-TRPCA is:

[0081]

[0082] Among them, 0<p<1, w satisfies The weight vector, s p represents the weighted Schatten p-norm WTSpN. Therefore, the shrinkage operator obtained by minimizing the WTSpN constraint term shrinks less for larger singular values ​​and shrinks more for smaller singular values, which can effectively retain reuse information and eliminate irrelevant information. λ is a positive balancing parameter. In fact, the weighted singular values ​​are p -norm. l represents S p -norm.

[0083] The GTCN neural network in S2 is specifically described as follows:

[0084] S22, GTCN is composed of dilated causal convolution and filter gating. Dilated causal convolution introduces causality in the convolution operation, so that the output of the current convolution operation depends only on the current and past input data, and is not affected by future information; at the same time, the receptive field of the convolution kernel is increased while keeping the number of parameters unchanged, so that each convolution output contains a larger range of information. The added information filter gating can select to retain valid information according to the training weights, thereby reducing the extraction of redundant information. The network structure of GTCN is composed of k hidden layers stacked together, and each specific hidden layer structure is as follows Figure 2 As shown:

[0085] Input data X t-1 First, input the two layers of dilated causal convolution units on the left. Each unit includes a dilated causal convolution operation (Dilated Causal Conv), a weight normalization layer (Weight Norm), an activation layer (Relu), and a Dropout deactivation layer. The specific formula is expressed as:

[0086]

[0087] The right side of the hidden layer is the gating mechanism G(·), which uses the weights generated by the Sigmoid activation function to control the proportion of different information. Its output is then compared with X t-1 The final output of each hidden layer is obtained by adding the random downsampling results. The gating rule formula is as follows:

[0088] G(x)=Sigmoid(Linear(x))⊙x

[0089] The middle conv represents a one-dimensional convolution operation, and only when X t With X t-1 The subscript t represents the time series dimension, which means that the model is updated from time t-1 to time t. The final output of the model is expressed as follows:

[0090]

[0091] The data collected and calculated by each device in S2 include:

[0092] S23, the visual acquisition device collects the online learning video data of the students over a period of time, and obtains the image of each frame through background processing, thereby obtaining the facial expression data of the students during the online learning process to infer the psychological state at that time, such as the dull eyes and daze caused by distraction; the confused expression due to insufficient understanding of the knowledge points; and even the facial state of dozing off.

[0093] S24, the ECG equipment collects the students’ ECG signal data over a period of time, and the heart rate increases when they are absent-minded or lying. As the body’s physical and mental fatigue becomes more and more serious, the heart rate will gradually slow down, and the heart rate will also decrease accordingly. The ECG data is input into the GTCN model to extract the feature input of the ECG modality.

[0094] S25, the audio acquisition device collects audio data during the learning process, such as data when answering questions, or active audio data generated in the process of cooperating with the teacher in class. According to information such as pitch and tone, it can help us understand the psychological state of students, and input the audio data into the GTCN model to extract the input of speech mode.

[0095] The S3 unidirectional weighted attention module is specifically described as follows:

[0096] S31, single-item weighting is to calculate the attention weight based on the features of two sequence data, and then use the weight to update the feature information of one of them, while the feature information of the other will not be affected by the attention weight. The key to achieve this is to rely on the basic unit unidirectional cross-modal attention (UCA), such as Figure 3 As shown, the specific structure is as follows:

[0097] Linear is a standard linear layer, MHCA is a multi-head cross-modal attention operation, Norm is a layer normalization operation, and FFN is a feed-forward neural network. Define the data X from two modalities a , X b , whose dimensions are all R t×d , W=(W q , W k , W v ) represents the corresponding weight matrix, then Q a =X a W q , K b =X b W k , V b =X b W v . where Q comes from mode X a , K, V come from mode X bThe multi-head cross-modal attention operation MHCA is implemented through multiple unidirectional attention mechanisms. The calculation of a single attention mechanism is as follows:

[0098]

[0099] in K b The transpose of k represents the dimension of the key vector. The dot product is divided by This can make the variance of the attention weight more stable, thereby improving the robustness and accuracy of the model. b The feature information with higher correlation between the two sequences is enhanced. At the same time, in order to further extract effective information and filter redundant information, the input data X is first a , X b Perform standard linearization, and then add Norm and FFN feedforward neural network layers. The specific formula is as follows: Formally, unidirectional cross-modal attention requires two input sequences X a ∈R t*d , X b ∈R t*d Get an output sequence X a ′∈R t*d , X b ′∈R t*d The formula is as follows:

[0100] Q a =Linear(X a )

[0101] K b , V b =Linear(X b )

[0102]

[0103]

[0104]

[0105] X a ′=FFN(Norm(X a ))

[0106] Final X b Modal input and modal X a Perform attention calculation to obtain modality X a Unique information, mode X aSince attention calculation is not performed in the unidirectional attention mode, only the normalization layer and the feedforward layer are passed through to retain the unique information of the unidirectional mode. The unidirectional weighted attention module consists of two UCAs, the audio mode and the ECG mode as the X a Input, video modality as X b Input, calculate the UCA output of audio mode and video mode and UCA output for ECG and video modalities and (where a represents the audio mode, h represents the ECG mode, and v represents the video mode), then and Splicing to obtain enhanced video modality features The specific formula is as follows:

[0107]

[0108]

[0109]

[0110] Among them, Project represents the projection operation, and i represents the number of layers; finally, after being processed by L layers of unidirectional weighted attention modules, the output of the last unidirectional weighted module is linearly projected and reduced in dimension as the final unimodal information feature Z v , Z a , Z h (Z v represents the video modality feature, Z a represents the audio modal feature, Z h represents the ECG modality characteristics).

[0111] In S4, all outputs are concatenated to obtain the final multimodal output, including the unimodal feature Z generated in the S3 process. v , Z a , Z h The obtained output linear concatenation is passed through the softmax activation function to obtain the final classification label. The specific formula is as follows:

[0112]

[0113] The loss function used in the prediction layer is cross entropy CrossEntropy, and the calculation process is as follows:

[0114] L SC =CrossEntropy(ylabel,ytrue).

[0115] Among them, ytrue is the true label.

[0116] The final output of the S5 sentiment classification layer is the neural network's judgment of the student's emotions during the learning process. The labels may include adjectives of some emotional directions such as excitement, confusion, dullness, and fatigue. Our sentiment classification model mainly makes a comprehensive judgment based on the collected image information, combined with the student's audio and ECG data, and then classifies our results into the corresponding labels above. This emotion can be used as a factor in our judgment of the student's learning status. If he remains in a relatively excited state, it may mean that he is more attentive. If he is confused, it may mean that he does not understand much. Then we construct a mapping matrix K with the discretized sentiment labels, which contains a certain sentiment category and its corresponding mapping score.

[0117] The data on the behavioral dimension not only include text information such as student test scores and quality evaluation scores, but also include some behavioral data generated during the learning process, such as the number of active speeches, the correct rate of answering questions, the online time, the number of background switching, etc. These behavioral data actually have an impact on the learning evaluation to a certain extent and show a linear correlation. For example, if a student has a high score on the in-class test, it means that he should have a good absorption of this class. If he frequently switches the background, it means that he may be absent-minded. All of these have an impact on our learning evaluation. At the same time, combined with the previous emotional dimension mapping matrix K, we can make a more comprehensive judgment. Then we construct a multivariate linear regression function, in which the input of the behavioral end is used as an independent variable, and the emotional modality mapping matrix is ​​used as a bias term b. We construct a similar multivariate function: learning score Y = w1 in-class test score + w2 number of active speeches + ... wp independent variable n + b (emotional mapping matrix). Through model iterative learning, we retain certain factors that are highly correlated with the learning score and continuously update the weight w of each behavioral end parameter and the bias term emotional mapping matrix b. Finally, our multivariate linear regression model outputs our learning evaluation score. The multivariate linear regression model uses mean square error MSE as the loss function. The specific formula is as follows:

[0118]

[0119] Where n represents the number of samples, Represents the predicted value of the model, y i represents the true value. Therefore, the total loss function of the present invention is expressed as:

[0120] L LoSS =L SC +L MSE .

[0121] On the other hand, an embodiment of the present invention further provides an online learning evaluation system based on time attention fusion multimodal data, comprising:

[0122] one or more processors;

[0123] A storage device for storing one or more programs, which, when executed by the one or more processors, enables the one or more processors to implement an online learning evaluation method based on temporal attention fusion of multimodal data as described in the above scheme.

[0124] The specific embodiments described herein are merely examples of the spirit of the present invention. Those skilled in the art may make various modifications or additions to the specific embodiments described or replace them in similar ways, but they will not deviate from the spirit of the present invention or exceed the scope defined by the appended claims.

Claims

1. An online learning evaluation method based on temporal attention fusion multimodal data, characterized in that: The steps include: Step S1, after the teacher starts the class, the emotional dimension data and the behavioral dimension data are obtained respectively; the emotional dimension data includes the student video data, electrocardiogram signal data, and audio data; Step S2, using the pre-trained GTCN neural network to extract features from the video data, the ECG signal data, and the audio data to obtain single-modal features; Step S3, mapping the features of different modalities to the same common subspace, taking the video modality as the main modality, and the audio and ECG modalities as auxiliary modalities, using a unidirectional weighted attention module to improve the expression ability of the video modality in the audio and ECG modalities, and after several unidirectional weighted attention modules, obtaining the final unimodal feature information; Step S4, concatenating the final single-modal feature information to obtain multi-modal fusion features, and inputting them into the prediction layer to obtain the sentiment classification label; Step S5, construct a mapping dictionary K with the sentiment classification labels obtained in step S4, K contains the classification labels and their corresponding scores; input the mapping matrix corresponding to the sentiment labels and the input of the behavior dimension into the multivariate linear regression model together to obtain the final learning evaluation score output.

2. The online learning evaluation method based on temporal attention fusion multimodal data as claimed in claim 1, characterized in that: In step S1, the students' online learning data are collected through visual acquisition equipment, ECG acquisition equipment, and audio acquisition equipment to obtain student video data, ECG signal data, and audio data. The number of times students answer questions, the number of times they speak correctly, and these real-time behavioral data during online learning are recorded in real time through the background of the online learning software. At the same time, text information such as students' historical test scores and teachers' comments are recorded as behavioral dimension data.

3. The online learning evaluation method based on temporal attention fusion multimodal data as claimed in claim 1, characterized in that: In step S2, according to the preset time interval, the student video data collected by the visual acquisition device is constructed into a video tensor, and the video tensor is processed using the p-TRPCA low-rank sparse constraint method to achieve data compression and dimensionality reduction, and then input into the pre-trained GTCN neural network to extract video features.

4. The online learning evaluation method based on temporal attention fusion multimodal data as claimed in claim 3, characterized in that: The specific implementation of the p-TRPCA low-rank sparse constraint method is as follows; First, separate the low-rank component L and the sparse component S from the observation tensor X, use the tensor nuclear norm as the low-rank constraint, and weight l p -norm as a sparse constraint for singular values, l p -norm is used as a constraint for sparse components, and the objective function of p-TRPCA is: Among them, 0<p<1, w is the weight vector, s p represents the weighted Schatten p-norm and λ is a positive balancing parameter. In fact, the weighted singular values ​​are p -norm, l represents S p -norm.

5. The online learning evaluation method based on temporal attention fusion multimodal data as claimed in claim 1, characterized in that: The GTCN neural network is composed of k hidden layers stacked together. The specific structure of each hidden layer is as follows: Input data X t-1 The processing is divided into three branches. In the left branch, the input first passes through two layers of dilated causal convolution units and then passes through the gating mechanism G(·). Each dilated causal convolution unit includes a dilated causal convolution operation, a weight normalization layer, an activation layer, and a Dropout inactivation layer. The specific formula is expressed as: Among them, DDC represents the dilated causal convolution operation, Relu represents the activation function, and Norm represents the normalization layer; In the right branch, the input passes through the gating mechanism G(·), which uses the weights generated by the Sigmoid activation function to control the proportion of different information. Its output is then added to the random downsampling result of the input x to obtain the output. The gating rule formula is expressed as follows: G(x)=Sigmoid(Linear(x))⊙x Among them, Linear represents the standard linear layer; The middle branch is a one-dimensional convolution operation Conv, and only when X t With X t-1 The function is executed only when the feature dimensions of are inconsistent, where the subscript t represents the time series dimension; the final output is expressed by the following formula:

6. The online learning evaluation method based on temporal attention fusion multimodal data as claimed in claim 1, characterized in that: The unidirectional weighted attention module is composed of two UCAs of unidirectional cross-modal attention, wherein the unidirectional cross-modal attention includes a standard linear layer Linear, a multi-head cross-modal attention operation MHCA, a layer normalization operation Norm, and a feed-forward neural network FFN; First, define the data X from two modalities a , X b , then the UCA processing process is as follows: Q a =Linear(X a ) K b ,V b =Linear(X b ) X a ′=FFN(Norm(X a )) Setting W q , W k , W v represents the corresponding weight matrix, then Q a =X a W q , K b =X b W k , V b =X b W v , where Q comes from mode X a , K, V come from mode X b ; The multi-head cross-modal attention operation MHCA is implemented through multiple unidirectional attention mechanisms. The calculation of a single attention mechanism is as follows: in K b The transpose of k represents the dimension of the key vector; Audio modality and ECG modality as X of UCA a Input, video modality as X b Input, calculate the UCA output of audio mode and video mode and UCA output for ECG and video modalities and Where a represents the audio mode, h represents the ECG mode, and v represents the video mode. and Perform stitching to obtain enhanced video modality features The specific formula is as follows: Among them, Project represents the projection operation, and i represents the number of layers; finally, after being processed by L layers of unidirectional weighted attention modules, the output of the last unidirectional weighted module is linearly projected and reduced in dimension as the final unimodal information feature Z v , Z a , Z h , Z v represents the final video modality feature, Z a Represents the final audio modal features, Z h Represents the final ECG modality features.

7. The online learning evaluation method based on temporal attention fusion multimodal data as claimed in claim 1, characterized in that: In step S4, the final unimodal feature information is concatenated to obtain the final multimodal fusion feature output, and then the softmax activation function is used to obtain the final classification label. The specific formula is as follows: Among them, Project represents the projection operation, Z v represents the final video modality feature, Z a Represents the final audio modal features, Z h Represents the final ECG modality features; the loss function used in the prediction layer is cross entropy CrossEntropy, and the calculation process is as follows: L SC =CrossEntropy(ylabel,ytrue)。 Among them, ytrue is the true label.

8. The online learning evaluation method based on temporal attention fusion multimodal data as claimed in claim 1, characterized in that: In step S5, the multivariate linear regression model uses mean square error MSE as the loss function, and the specific formula is as follows: Where n represents the number of samples, represents the predicted value, and yi represents the true value; The final total loss function is expressed as: L LOSS =L SC +L MSE Among them, L SC is the loss function of the prediction layer in step S4.

9. The online learning evaluation method based on temporal attention fusion multimodal data as claimed in claim 1, characterized in that: The specific implementation method of obtaining the pre-trained GTCN is as follows: The first step is to build a face sample library for students when they are learning, including face images of people with head-mounted EEG devices and public face libraries. The audio data generated during the learning process is used to build an audio database. The electrocardiogram database of students when they are learning is also built, including heart rate and blood pressure data uploaded by wearable devices. The second step is to perform data enhancement and labeling preprocessing on the collected sample data; The third step is to use the GTCN neural network to train and test the video data, audio data, and ECG data in the constructed sample library. By adjusting parameters and using different image processing methods, we can continuously optimize the model with the best accuracy, namely the pre-trained GTCN neural network.

10. An online learning evaluation system based on temporal attention fusion multimodal data, characterized in that: include: one or more processors; A storage device for storing one or more programs, which, when executed by the one or more processors, enables the one or more processors to implement an online learning evaluation method based on temporal attention fusion of multimodal data as described in any one of claims 1 to 9.

Images