Method for evaluating participation degree of gated hybrid expert network based on space-time Transform
Patent Information
- Application Number
- CN202410988732.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-23
- Publication Date
- 2024-11-08
Smart Images

Figure CN118918510A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of emotion computing, and in particular relates to a method for evaluating engagement of a gated hybrid expert network based on a spatiotemporal Transformer. Background Art
[0002] Online classes have become one of the main learning methods. However, in this learning environment, students are easily affected by a variety of uncontrollable factors, such as their own factors and the surrounding learning environment, which may lead to a decrease in students' learning engagement. Engagement is usually defined as the degree of learner involvement in a task or activity. Specifically, engagement reflects the interest, attention, and emotional involvement shown by learners in the task. According to the degree of these involvements, engagement can be divided into four levels: very low, low, high, and very high. This concept has been widely used in many fields such as education, psychology, and human-computer interaction, among which the field of education is the main battlefield for research and application. Compared with traditional education, teachers face greater challenges in maintaining student engagement in online courses, mainly because the "virtual wall" of online classes separates students and teachers. Especially when facing a large number of students, the difficulty of evaluating engagement will also increase significantly. Therefore, the research on automatic evaluation of student engagement is particularly important in the field of online courses.
[0003] Given the universal applicability, high cost-effectiveness and non-intrusiveness of computer vision, it has shown great potential in engagement estimation. Traditional methods mainly rely on manual features such as eye gaze and head posture, and train classifiers to detect engagement levels; recent studies tend to adopt an end-to-end approach to process raw video frames through a convolutional neural network-based model to evaluate engagement. For example, Chinese invention patent CN112801042B discloses a method for identifying student class behavior based on deep learning and video trajectory. The features of each video trajectory moment are extracted through a two-stream convolutional network, and then the features of the video trajectory features in the time dimension are extracted using a recurrent neural network. Finally, the features of the entire video are obtained by spatiotemporal pooling and splicing, and a linear SVM classifier is trained to classify the video. However, due to the limited size of the convolution kernel of the convolutional neural network, the neural network-based model can only learn local spatial and temporal information from the video, and cannot fully capture the long-range spatial and temporal dependencies in the video, which limits the estimation performance. On the other hand, the accuracy of engagement assessment results is not the only goal. In online classes, teachers not only want to know the assessment results of student engagement, but also the credibility of the assessment results. However, many existing models for engagement assessment do not have the ability to perceive uncertainty, and are often overconfident or lack confidence when making predictions, that is, the confidence of the prediction is higher or lower than the expected accuracy. This phenomenon may reduce the reliability of the model, especially when the test sample is vague or uncertain. Therefore, it is necessary to redesign an engagement assessment method to further improve the accuracy and reliability of engagement assessment results and assist teachers in improving classroom efficiency and quality. Summary of the invention
[0004] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and provide a participation evaluation method based on a gated hybrid expert network of spatiotemporal Transformer, further improve the accuracy and reliability of participation evaluation results, and assist teachers in improving classroom efficiency and quality.
[0005] The purpose of the present invention can be achieved by the following technical solutions:
[0006] The present invention provides a method for evaluating participation of a gated hybrid expert network based on a spatiotemporal Transformer, comprising the following steps:
[0007] Obtain video data of students taking online classes, input it into the trained engagement evaluation model of the gated hybrid expert network based on spatiotemporal Transformer, and obtain the evaluation results of students' online class engagement, including the predicted value of engagement level and its uncertainty;
[0008] The engagement evaluation model of the spatiotemporal Transformer-based gated hybrid expert network is iteratively trained by minimizing an objective function constructed based on evidence theory and Dirichlet distribution, and includes a linear mapping module, an expert network module and a Tower network connected in sequence, and also includes a gated network module that generates weights for the expert network module, the expert network module includes M parallel expert networks, and the Tower network includes a temporal Transformer encoder and a classifier;
[0009] The process of input data processing by the participation evaluation model of the spatiotemporal Transformer-based gated hybrid expert network is as follows:
[0010] S1, uniformly extract T frames from the input video data of students taking online classes and perform preprocessing, then map them to a D-dimensional feature space through the linear mapping module, generate a video frame block mapping sequence, and input it into the expert network module;
[0011] S2, each expert network extracts spatial representations of the video frame block mapping sequence respectively, the gated network generates weights of each expert network according to the input video data of the student's online class, and the weighted aggregation of the spatial representations extracted by all expert networks is input into the Tower network;
[0012] S3. The temporal Transformer encoder extracts spatiotemporal features from the weighted aggregated spatial representation and inputs them into the classifier to obtain a predicted value of the engagement level, and quantifies the uncertainty of the predicted value of the engagement level according to the Dirichlet distribution.
[0013] Furthermore, in step S1, the preprocessing includes sample labeling, video frame cropping and normalization.
[0014] Furthermore, in step S1, the generation process of the video frame block mapping sequence is specifically as follows:
[0015] From video data Uniformly extract T frames from the video, where C is the number of channels and H×W is the resolution of a single video frame in the video;
[0016] A single video frame is cropped into small blocks with a resolution of P×P. The video frame block sequence generated by the frame at time point t is Indicates the length of the video frame block sequence;
[0017] Through a trainable linear transformation Map the video frame block sequence to the D-dimensional feature space to obtain the video frame block mapping sequence in, is a learnable position embedding, is a learnable spatial position embedding added to the tokens.
[0018] Furthermore, each spatial Transformer encoder includes multiple alternately connected multi-head self-attention and feedforward networks, and each multi-head self-attention and feedforward network is followed by residual connection and layer normalization.
[0019] Furthermore, the gated network performs linear transformation on the video data of the students' online classes through a Softmax function to generate weights for each expert network.
[0020] Furthermore, the specific expression of the objective function is:
[0021]
[0022] in, is the objective function, λ is the balance weight, To optimize the classification terms, specifically the cross entropy loss based on evidence theory and Dirichlet distribution, This is a regularization term that introduces the correction parameters of the Dirichlet distribution.
[0023] Further, optimize the classification items The specific expression is:
[0024]
[0025] Among them, K is the number of categories, N is the length of the video frame block sequence, Dir(p i |α i ) is the Dirichlet distribution, which means that under the given condition α i The probability density function is p i ,y ik is the true label of sample i belonging to the kth category, is the predicted value of sample i The kth component of , ψ(·) is the digamma function, S i is the Dirichlet intensity, α ik is the Dirichlet distribution parameter,
[0026] Furthermore, the regularization term The specific expression is:
[0027]
[0028] Among them, K is the number of categories, N is the length of the video frame block sequence, KL is the KL divergence, is the Dirichlet distribution, is the modified parameter of the Dirichlet distribution, is the evidence vector, i.e., the predicted value of sample i, ψ(·) is the digamma function, and Γ(·) is the gamma function.
[0029] Furthermore, the modified parameter of the Dirichlet distribution is α i is the Dirichlet distribution parameter,
[0030] Furthermore, the specific calculation formula for the uncertainty of the participation level prediction value is:
[0031]
[0032] Among them, u i is the uncertainty, K is the number of categories, S i is the Dirichlet intensity, α ik is the Dirichlet distribution parameter.
[0033] Compared with the prior art, the present invention has the following beneficial effects:
[0034] 1. The present invention constructs and trains an engagement assessment model based on a gated hybrid expert network of a spatiotemporal Transformer, which can output a predicted engagement level and its uncertainty based on the video data of students taking online classes. The engagement assessment model based on a gated hybrid expert network of a spatiotemporal Transformer is iteratively trained through an objective function constructed based on evidence theory and Dirichlet distribution. Evidence theory can regard the predicted engagement level as a series of subjective opinions without relying on strict prior knowledge. When dealing with classification problems, it can more flexibly deal with uncertainty. Dirichlet distribution can model the probability distribution of the predicted engagement level, making the classification results more interpretable, and can effectively deal with the sparsity and uncertainty of the data, maintain a high classification accuracy, and further improve the accuracy of the engagement assessment results. In addition, the above method also quantifies the uncertainty of the predicted engagement level according to the Dirichlet distribution, which can further improve the reliability of the engagement assessment results and assist teachers in improving classroom efficiency and quality.
[0035] 2. In the engagement evaluation model of the spatiotemporal Transformer-based gated hybrid expert network constructed by the present invention, the expert network includes multiple spatial Transformer encoders, each of which includes multiple alternatingly connected multi-head self-attention and feedforward networks, wherein the multi-head self-attention can divide the input sequence into multiple subsequences, and perform self-attention calculation on each subsequence respectively, and the self-attention calculation can directly calculate the dependency between any two positions in the sequence, thereby effectively capturing long-range spatiotemporal dependencies, and the feedforward network can perform nonlinear transformation and feature extraction on the output of the multi-head attention mechanism. The combination of the two enables the expert network to capture information in the sequence from different angles and subspaces, thereby improving the representation and generalization capabilities of the expert network; residual connections and layer normalization are set after each multi-head self-attention and feedforward network, and layer normalization can accelerate the convergence of the training process, while residual connections can enhance information flow.
[0036] 3. The objective function constructed by the present invention includes a classification optimization term and a regularization term, wherein the classification optimization term is a cross entropy loss constructed based on evidence theory and Dirichlet distribution, which can ensure that the true label of each sample provides stronger evidence, and the regularization term introduces a Dirichlet distribution correction parameter, which can ensure that the evidence of the true label will not be weakened due to errors in model prediction. BRIEF DESCRIPTION OF THE DRAWINGS
[0037] Figure 1 The schematic diagram of the structure of the engagement evaluation model of the gated hybrid expert network based on spatiotemporal Transformer;
[0038] Figure 2 It is a schematic diagram of the structure of the expert network;
[0039] in, represents the video frame block sequence generated by the frame at time point t, Represents the video frame block mapping sequence generated by the frame at time point t, N is the length of the video frame block sequence, is the spatial representation, L is the number of groups of multi-head self-attention and feedforward networks, and CLS represents the classification task. DETAILED DESCRIPTION
[0040] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.
[0041] Example:
[0042] Since the convolution kernel size of the convolutional neural network is limited and cannot fully capture the long-range spatial and temporal dependencies in the video, the existing engagement assessment model based on the convolutional neural network can only learn local spatial and temporal information from the video, which limits the performance of the model. In addition, many existing models tend to be overconfident or lack confidence in their predictions, that is, the confidence of the prediction is higher or lower than the expected accuracy. This phenomenon may reduce the reliability of the model, especially when the test sample is vague or uncertain. To solve the above problems, this embodiment provides an engagement assessment method based on a gated hybrid expert network of a spatiotemporal Transformer, comprising the following steps:
[0043] The video data of students taking online classes is obtained and input into the trained engagement assessment model based on the spatiotemporal Transformer-based gated hybrid expert network to obtain the engagement assessment results of students taking online classes. The engagement assessment results include the predicted value of the engagement level and its uncertainty.
[0044] The structure of the participation evaluation model based on the spatiotemporal Transformer-based gated hybrid expert network is as follows Figure 1 As shown, it includes a linear mapping module, an expert network module and a Tower network connected in sequence, and also includes a gating network module that generates weights for the expert network module. The expert network module includes M parallel expert networks, and the Tower network includes a temporal Transformer encoder and a classifier. The participation evaluation model based on the spatiotemporal Transformer gated hybrid expert network processes the input data as follows:
[0045] S1. Uniformly extract T frames from the input video data of students taking online classes and preprocess them. Then, they are mapped to the D-dimensional feature space through a linear mapping module to generate a video frame block mapping sequence, which is input into the expert network module.
[0046] The generation process of the video frame block mapping sequence is as follows:
[0047] From video data T frames are uniformly extracted from the video, where C is the number of channels and H×W is the resolution of a single video frame in the video. The preprocessing includes sample annotation, video frame cropping and normalization.
[0048] A single video frame is cropped into small blocks with a resolution of P×P. The video frame block sequence generated by the frame at time point t is Indicates the number of video frame blocks, that is, the length of the video frame block sequence.
[0049] Through a trainable linear transformation Block sequence of video frames Map to D-dimensional feature space to obtain video frame block mapping sequence in, is a learnable position embedding that preserves the spatial information of each video frame. A learnable spatial position embedding added to the tokens to preserve spatial information.
[0050] S2. Each expert network extracts spatial representations of the video frame block mapping sequence respectively. The gated network generates weights for each expert network based on the input video data of students taking online classes. The spatial representations extracted by all expert networks are weighted and aggregated and then input into the Tower network.
[0051] like Figure 2 As shown in the figure, each expert network includes T spatial Transformer encoders, each of which includes L groups of alternatingly connected multi-head self-attention (MSA) and feedforward networks (FFN). Multi-head attention can divide the input sequence into multiple subsequences and perform self-attention calculation on each subsequence respectively; self-attention calculation can directly calculate the dependency between any two positions in the sequence, so as to effectively capture long-range dependencies, and finally splice the attention results of each subsequence together. This mechanism enables the expert network to capture information in the sequence from different angles and subspaces, improving the representation and generalization capabilities of the expert network. Residual connections and layer normalization are set after each multi-head self-attention and feedforward network. Layer normalization can accelerate the convergence of the training process, while residual connections can be used to strengthen information flow to obtain better performance. Figure 2 middle, The video frame block embedding sequence is fed into the t-th spatial encoder in the expert network, which generates a sequence of spatial embeddings Where m = 1, ..., M, connecting the outputs of M expert networks can be obtained
[0052] The gating network is based on the input video frame blocks Generate the weight distribution of each expert, and the final output is the weighted sum of all expert outputs, that is, Among them, g(·) is the gating network, is the mth logical value output by g(·), f m (· is the mth expert network. The gated network performs a linear transformation of the input through the Softmax function, i.e. g(X) = softmax(W g X), where Is a trainable matrix, X is the input video frame block sequence.
[0053] S3. The temporal Transformer encoder extracts spatiotemporal features from the weighted aggregated spatial representation and inputs them into the classifier to obtain the predicted value of the engagement level. The uncertainty of the predicted value of the engagement level (0, 1, 2, 3) is quantified according to the Dirichlet distribution.
[0054] The engagement assessment task is transformed into a classification problem (CLS), and the objective function constructed based on Dempster-Shafer evidence theory (DST) and Dirichlet distribution is used to iteratively train the engagement assessment model of the gated hybrid expert network based on spatiotemporal Transformer by minimizing the objective function. The specific expression of the objective function is:
[0055]
[0056] in, is the objective function, λ is the balance weight, To optimize the classification items, a cross entropy loss is constructed based on evidence theory and Dirichlet distribution. This is a regularization term that introduces the correction parameters of the Dirichlet distribution.
[0057] The i-th sample outputs the predicted value after the Softplus activation function Will As an evidence vector, the Dirichlet distribution parameter α can be derived i =[α i1 ,α i2 ,…,α iK ], Then its Dirichlet distribution is Among them, Dir(p i |α i ) means that at a given α i Under the condition of i The probability density function of K is the number of categories, B(α i ) is a K-dimensional polynomial function, is a K-dimensional unit vector.
[0058] Optimize category items The specific expression is:
[0059]
[0060] Among them, K is the number of categories, N is the length of the video frame block sequence, and y ik is the true label of sample i belonging to the kth category, is the predicted value of sample i The kth component of , ψ(·) is the digamma function, is the Dirichlet intensity, α ik is the Dirichlet distribution parameter,
[0061] In building optimization categories When , the model is derived based on the Bayesian risk of the sample to ensure that the true label of each sample provides stronger evidence. However, this construction method does not guarantee wrong labels, which will lead to reduced evidence. To make up for this deficiency, the regularization term is introduced in this embodiment. The specific expression is:
[0062]
[0063] Among them, KL is KL divergence, is the correction parameter of the Dirichlet distribution, which aims to prevent the evidence of the true class from falling to zero, α i is the Dirichlet distribution parameter, This correction ensures that the evidence for the true label is not weakened by errors in model predictions. Γ(·) is the gamma function, 1 is a vector with all elements equal to 1.
[0064] In the Dirichlet distribution Dir(p i |α i ), the confidence b of the i-th sample ik and uncertainty u i Quantification through a trust function allows for a more precise and reliable assessment of engagement levels. S i is the Dirichlet intensity.
[0065] Compared with the convolutional neural network-based method, the above method can effectively capture the long-range spatiotemporal dependencies in the video, use Dirichlet distribution to model the probability distribution of the model prediction value, and regard the model prediction as a series of subjective opinions based on Dempster-Shafer evidence theory (DST). The evidence learned by the model from the video data is used to aggregate these subjective opinions, and the uncertainty of the prediction value is quantified through the trust function, so as to make a more accurate and reliable assessment of the level of participation. The above method can further improve the reliability of the participation evaluation results, better assist teachers in participation evaluation, and improve classroom efficiency and quality.
[0066] To verify the effectiveness of the above method, this example was experimented on a server equipped with an NVIDIA GeForce RTX 3090 24GB graphics card, Python 3.10, and PyTorch 2.1. The number of video frames input to the model is 16, the crop size of each frame is 224×224, the Adam W optimization algorithm is used, the training batch size is 4, and the initial learning rate is set to 1×10 -3 .
[0067] Using the public dataset DAISEE data, Table 1 shows the details of the DAISEE dataset:
[0068] Table 1 DAiSEE data details
[0069]
[0070]
[0071] Table 2 shows the experimental results. The three evaluation indicators of the model classification are F1-macro, F1-weighted and AUC. F1-macro, F1-weighted and AUC reflect the performance of the classification model from different perspectives. Among them, F1-macro focuses on the average performance of all categories, F1-weighted considers the difference in the number of category samples, and AUC is a comprehensive evaluation of the overall performance of the model. It can be found from Table 2 that the method of the present invention (EGMoE) has achieved the best performance in all three indicators. It can be seen that the method of the present invention can effectively evaluate the student participation.
[0072] Table 2 Comparison of the results of this model with other classic models (%)
[0073]
[0074] If the above method is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.
[0075] The above description of the embodiments is to facilitate the understanding and use of the invention by those skilled in the art. It is obvious that those skilled in the art can easily make various modifications to these embodiments and apply the general principles described herein to other embodiments without creative work. Therefore, the present invention is not limited to the above embodiments, and improvements and modifications made by those skilled in the art based on the disclosure of the present invention without departing from the scope of the present invention should be within the scope of protection of the present invention.
Claims
1. A method for evaluating participation of a gated hybrid expert network based on spatiotemporal Transformer, characterized in that: The following steps are involved: Obtain video data of students taking online classes, input it into the trained engagement evaluation model of the gated hybrid expert network based on spatiotemporal Transformer, and obtain the evaluation results of students' online class engagement, including the predicted value of engagement level and its uncertainty; The engagement evaluation model of the spatiotemporal Transformer-based gated hybrid expert network is iteratively trained by minimizing an objective function constructed based on evidence theory and Dirichlet distribution, and includes a linear mapping module, an expert network module and a Tower network connected in sequence, and also includes a gated network module that generates weights for the expert network module, the expert network module includes M parallel expert networks, and the Tower network includes a temporal Transformer encoder and a classifier; The process of input data processing by the participation evaluation model of the spatiotemporal Transformer-based gated hybrid expert network is as follows: S1, uniformly extract T frames from the input video data of students taking online classes and perform preprocessing, then map them to a D-dimensional feature space through the linear mapping module, generate a video frame block mapping sequence, and input it into the expert network module; S2, each expert network extracts spatial representations of the video frame block mapping sequence respectively, and the gated network generates weights of each expert network according to the input video data of the student's online class, and the weighted aggregation of the spatial representations extracted by all expert networks is input into the Tower network; S3. The temporal Transformer encoder extracts spatiotemporal features from the weighted aggregated spatial representation and inputs them into the classifier to obtain a predicted value of the engagement level, and quantifies the uncertainty of the predicted value of the engagement level according to the Dirichlet distribution.
2. According to claim 1, a method for evaluating participation of a gated hybrid expert network based on spatiotemporal Transformer, characterized in that: In step S1, the preprocessing includes sample labeling, video frame cropping and normalization.
3. According to claim 1, a method for evaluating participation of a gated hybrid expert network based on spatiotemporal Transformer, characterized in that: In step S1, the generation process of the video frame block mapping sequence is specifically as follows: From video data Uniformly extract T frames from the video, where C is the number of channels and H×W is the resolution of a single video frame in the video; A single video frame is cropped into small blocks with a resolution of P×P. The video frame block sequence generated by the frame at time point t is Indicates the length of the video frame block sequence; Through a trainable linear transformation Map the video frame block sequence to the D-dimensional feature space to obtain the video frame block mapping sequence in, is a learnable position embedding, is a learnable spatial position embedding added to the tokens.
4. According to claim 1, a method for evaluating participation of a gated hybrid expert network based on spatiotemporal Transformer, characterized in that: Each expert network consists of T spatial Transformer encoders, each of which consists of multiple alternately connected multi-head self-attention and feedforward networks, and each multi-head self-attention and feedforward network is followed by residual connections and layer normalization.
5. According to claim 1, the participation evaluation method of the gated hybrid expert network based on spatiotemporal Transformer is characterized in that: The gated network performs linear transformation on the video data of students' online classes through the Softmax function to generate the weights of each expert network.
6. According to claim 1, a method for evaluating participation of a gated hybrid expert network based on spatiotemporal Transformer, characterized in that: The specific expression of the objective function is: in, is the objective function, λ is the balance weight, To optimize the classification terms, specifically the cross entropy loss based on evidence theory and Dirichlet distribution, This is a regularization term that introduces the correction parameters of the Dirichlet distribution.
7. The participation evaluation method of the spatiotemporal Transformer-based gated hybrid expert network according to claim 6 is characterized in that: Optimize category items The specific expression is: Among them, K is the number of categories, N is the length of the video frame block sequence, Dir(p i |α i ) is the Dirichlet distribution, which means that under the given condition α i The probability density function is p i ,y ik is the true label of sample i belonging to the kth category, is the predicted value of sample i The kth component of , ψ(·) is the digamma function, S i is the Dirichlet intensity, α ik is the Dirichlet distribution parameter, 8. The participation evaluation method of the spatiotemporal Transformer-based gated hybrid expert network according to claim 6 is characterized in that: Regularization term The specific expression is: Among them, K is the number of categories, N is the length of the video frame block sequence, KL is the KL divergence, is the Dirichlet distribution, is the modified parameter of the Dirichlet distribution, is the evidence vector, i.e., the predicted value of sample i, ψ(·) is the digamma function, and Γ(·) is the gamma function.
9. The participation evaluation method of the spatiotemporal Transformer-based gated hybrid expert network according to claim 8, characterized in that: Modified parameter of Dirichlet distribution α i is the Dirichlet distribution parameter, 10. The participation evaluation method of the spatiotemporal Transformer-based gated hybrid expert network according to claim 1, characterized in that: The specific calculation formula for the uncertainty of the predicted value of participation level is: Among them, u i is the uncertainty, K is the number of categories, S i is the Dirichlet intensity, α ik is the Dirichlet distribution parameter.
