Multi-level facial feature fusion method, system and medium for evaluating student engagement

Abstract

The application discloses a multi-level facial feature fusion method and system for evaluating student participation and a medium, relates to the technical field of artificial intelligence, and inputs a learning video of a student into a trained fusion prediction model to obtain a prediction result of student participation; the training process of the fusion prediction model is as follows: a plurality of learning videos of a plurality of students are acquired to construct a training data set; a facial global feature vector, a facial local feature vector and a physiological feature vector are extracted frame by frame from the learning video of the student; the facial global feature vector and the facial local feature vector are fused by using a time sequence pyramid structure to obtain a two-dimensional global feature vector; and the two-dimensional global feature vector and the physiological feature vector are weighted and summed to obtain the prediction result of student participation; the multi-level facial feature fusion method and system and the medium improve the prediction accuracy and robustness of the model.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a multi-level facial feature fusion method, system, and medium for evaluating student engagement. Background Art

[0002] In recent years, with the rapid development of information technology, digital education, as an emerging educational model, has attracted widespread attention from all sectors of society. However, there are also concerns about its teaching effectiveness and student engagement, especially when compared to traditional offline teaching. Research shows that students' attention and engagement in online classes are generally lower than in traditional classrooms. In light of this, there is a growing trend to develop an accurate student engagement analysis model that is adaptable to various video-based teaching environments. This model aims to provide educators with timely and actionable feedback through precise analysis of student performance, helping teachers optimize teaching strategies, improve teaching quality, and enhance course content. Furthermore, it provides students with a deeper understanding of their own performance during the learning process, motivating them to take proactive measures to improve their learning outcomes.

[0003] In traditional classroom settings, educators can judge whether students are engaged based on their classroom behavior, such as vocal activity, head movements, gaze direction, and facial expressions. However, this assessment method is not very effective in online classes because teachers cannot monitor the actual learning situation of each online student. With the advancement of computer vision and facial recognition technology, deep learning techniques can now be used to analyze students' emotions and their contributions to classroom discussions. Initially, people used static images of students in class as input to provide artificial intelligence models to obtain a level of student engagement assessment and compared it with expert evaluation results to determine the accuracy of the model's prediction. For example, the ResNet model is used for student engagement prediction tasks. However, static image-based methods are unable to analyze the key temporal information of the face, resulting in suboptimal performance.

[0004] In recent years, researchers have begun to use student learning videos to fully integrate temporal logic to analyze student engagement. Consequently, researchers have explored dynamic analysis methods for student engagement identification tasks. These methods include 3D convolutional neural networks (CNNs), multi-rate gating networks, multi-layer neural networks, Transformer-based networks with enhanced attention mechanisms, networks based on MAE (Masked Autoencoders) techniques, and networks that improve performance by modifying loss functions. Furthermore, some research aims to integrate other modalities such as sound, physiological signals, and body movements. In the field of engagement analysis, several models have been developed to calculate physiological features such as heart rate, respiratory rate, and peak heart rate based on facial landmarks. These features are estimated using remote photoplethysmography (rPPG), which extracts heart rate features from videos. While these models have significantly improved the ability to extract student features from videos, they inevitably face challenges. Most student engagement models follow a single-feature model architecture, relying primarily on visual features for engagement prediction. However, these models inevitably face challenges such as noise in the data and the omission of important features during the extraction process. Some models adopt a multi-feature model structure that aims to address these limitations by integrating additional modalities. These models use decision fusion strategies, but the results are not satisfactory.

[0005] Therefore, further enhancements are needed to extract features from data and fuse features at different levels. However, several challenges currently exist. The first is the issue of feature selection. Given the large number of features available in videos, some features may be ineffective for model judgment or even have a negative impact due to redundancy. The second issue is how to model spatiotemporal dependencies. It is necessary to identify the spatiotemporal relationships corresponding to different features to enhance the connection between features at each level. Finally, there is the challenge of feature fusion, which requires the development of an effective framework to manage and integrate the spatiotemporal relationships established at different levels. These challenges limit the prediction accuracy and robustness of the model. Summary of the Invention

[0006] Based on the technical problems existing in the background technology, the present invention proposes a multi-level facial feature fusion method, system and medium for evaluating student participation, which improves the prediction accuracy and robustness of the model.

[0007] The multi-level facial feature fusion method proposed in this paper for evaluating student engagement inputs students' learning videos into a trained fusion prediction model to obtain the prediction results of student engagement;

[0008] The training process of the fusion prediction model is as follows:

[0009] Obtain multiple learning videos of multiple students to build a training dataset;

[0010] Extract the global facial feature vector, local facial feature vector and physiological feature vector from the students' learning videos frame by frame;

[0011] Using a temporal pyramid structure, the global facial feature vector and the local facial feature vector are concatenated in the feature dimension to obtain a concatenated feature 1. A temporary position embedding and a learnable embedding vector are generated for the time dimension. The concatenated feature 1 is added to the temporary position embedding and then concatenated with the learnable embedding vector to obtain a concatenated feature 2. Without changing the time dimension, the concatenated feature 2 is used as the input of the decoder and downsampled. The downsampled results obtained from each decoding layer are concatenated and fused at multiple scales to obtain a two-dimensional global feature vector.

[0012] The prediction result of student engagement is obtained by weighted summing the two-dimensional global feature vector and the physiological feature vector;

[0013] Construct a loss function to adjust the trainable parameters in the ensemble prediction model.

[0014] Furthermore, in extracting the global facial feature vector, local facial feature vector, and physiological feature vector from the student's learning video frame by frame, the generation process of the global facial feature vector is as follows:

[0015] Use face recognition tools to extract faces from students’ learning videos and obtain a global facial image;

[0016] Perform convolution extraction on the global facial image to obtain a preliminary feature image, which is then input into the position embedding module of the Transformer.

[0017] Divide each frame of preliminary feature image into image blocks of the same size, generate position embedding for each image block, concatenate each image block with the corresponding learnable embedding vector, and concatenate all image blocks included in each frame of preliminary feature image in batches and input them into the Transformer encoder as the input feature sequence;

[0018] In each layer of the encoder, the learnable embedding vector interacts with all image patches in the input feature sequence using a multi-head attention mechanism, and the output of the learnable embedding vector is used as the global facial feature vector.

[0019] Furthermore, in the process of performing convolution extraction on the global facial image to obtain a preliminary feature image, specifically:

[0020] Use convolutional neural network to perform preliminary feature extraction on each global facial image to obtain feature image 1;

[0021] The feature image 1 is sent to the residual network for convolution processing, and the output feature image 2 is residually connected with the feature image 1 to obtain the preliminary feature image.

[0022] Furthermore, in extracting the global facial feature vector, local facial feature vector, and physiological feature vector from the student's learning video frame by frame, the generation process of the local facial feature vector is as follows:

[0023] Use face recognition tools to extract faces from students' learning videos to obtain facial position feature sequences;

[0024] The facial position feature sequence is divided by temporal convolution with a set step size, and then spatial feature vectors are obtained by spatial feature extraction through a spatial convolution layer.

[0025] The multi-head self-attention mechanism and feedforward neural network are used in turn to extract the spatial feature vector to obtain the local facial feature vector.

[0026] Furthermore, in extracting the global facial feature vector, the local facial feature vector, and the physiological feature vector frame by frame from the student's learning video, the process of generating the physiological feature vector is as follows:

[0027] Obtain pixel values ​​of the cheeks and forehead regions on both sides of the face from the student's learning video and calculate the normalized RGB signal based on them;

[0028] The normalized RGB signal is mapped to the green channel to obtain the matrix S. Two adjacent rows of the matrix S are taken as a group. The rows of the matrix S are divided into groups in sequence, and the weighted average result of each group is calculated to obtain the group matrix.

[0029] After removing the mean value from all group matrices, the processed signals are accumulated to obtain the matrix H;

[0030] After detrending the accumulated matrix H, a band-pass filter is used to separate the heart rate frequency from the trended matrix to obtain a filtered matrix;

[0031] Use the scipy library to obtain the peak value of the heart rate corresponding to the filtered matrix, calculate the maximum and minimum values ​​of the heart rate peak, and thus obtain the interval time between peaks;

[0032] The student's heart rate and the interval time between peaks are used as physiological features and then passed through the fully connected layer, residual block and fully connected layer in sequence to obtain the physiological feature vector.

[0033] Furthermore, the two-dimensional global feature vector and the physiological feature vector are weighted and summed to obtain the prediction result of student participation. , the specific formula is as follows:

[0034] ;

[0035] in, and are the learnable parameters, is the two-dimensional global eigenvector, is the physiological feature vector.

[0036] Furthermore, the loss function The construction formula is as follows:

[0037] ;

[0038] in, is the sample size, is the number of categories, is the sample index, is the category index, is the true label value, is the predicted value output by the fusion prediction model, is the regularization parameter, Represents the squared L2 norm of the trainable parameters.

[0039] A multi-level facial feature fusion system for evaluating student engagement, which inputs student learning videos into a trained fusion prediction model to obtain prediction results of student engagement;

[0040] The training process of the fusion prediction model includes a training set construction module, a feature extraction module, a first fusion module, a second fusion module and a loss construction module;

[0041] The training set construction module is used to obtain students' learning videos to construct a training dataset;

[0042] The feature extraction module is used to extract the global facial feature vector, local facial feature vector and physiological feature vector from the student's learning video frame by frame;

[0043] The first fusion module is used to use the temporal pyramid to concatenate the global facial feature vector and the local facial feature vector in the feature dimension to obtain a concatenated feature 1, and generate a temporary position embedding and a learnable embedding vector for the time dimension. The concatenated feature 1 is added to the temporary position embedding and then concatenated with the learnable embedding vector to obtain a concatenated feature 2. Without changing the time dimension, the concatenated feature 2 is sequentially downsampled as the number of decoding layers decreases. After each downsampling, multi-scale fusion is performed to obtain a two-dimensional global feature vector.

[0044] The second fusion module is used to perform weighted summation of the two-dimensional global feature vector and the physiological feature vector to obtain the prediction result of student engagement;

[0045] The loss building module is used to build a loss function to adjust the trainable parameters in the ensemble prediction model.

[0046] Furthermore, the loss function The construction formula is as follows:

[0047] ;

[0048] in, is the sample size, is the number of categories, is the sample index, is the category index, is the true label value, is the predicted value output by the fusion prediction model, is the regularization parameter, Represents the squared L2 norm of the weight vector.

[0049] A computer-readable storage medium stores a plurality of classification programs, which are used to be called by a processor and execute the multi-level facial feature fusion method described above.

[0050] Those skilled in the art will understand that all or part of the steps of implementing the above-mentioned method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above-mentioned method embodiment; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk, etc. Various media that can store program codes.

[0051] The advantages of the multi-level facial feature fusion method, system and medium for evaluating student engagement provided by the present invention are: by selecting global feature vectors, local facial feature vectors and physiological feature vectors, while ensuring that the model can be effectively determined, the negative impact caused by the redundancy of selected features is avoided; features are modeled at three different visual levels, taking into account time changes and monitoring the trend of students' heart rate, providing a comprehensive assessment of student engagement; the omission of important features is prevented, and the number of redundant features included is minimized; middle-level fusion (fusion of global facial feature vectors and local facial feature vectors) and decision-level fusion (fusion of two-dimensional global feature vectors and physiological feature vectors) are used to integrate features at different levels, and a temporal pyramid structure is used to simulate the temporal relationship of cross-level features; and finally, the student's engagement level is judged through the student's learning video, thereby improving the prediction accuracy and robustness of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 It is a structural schematic diagram of the present invention. DETAILED DESCRIPTION

[0053] The technical solutions of the present invention are described in detail below through specific embodiments. Numerous specific details are set forth in the following description to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways than those described herein, and those skilled in the art may make similar modifications without departing from the scope of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0054] like Figure 1 As shown, the multi-level facial feature fusion method for evaluating student engagement proposed in the present invention inputs the student's learning video into the trained fusion prediction model to obtain the prediction result of the student's engagement;

[0055] The training process of the fusion prediction model is as follows:

[0056] Obtain multiple learning videos of multiple students to construct a training dataset;

[0057] Step 1: Extract the global facial feature vector, local facial feature vector, and physiological feature vector from the learning video frame by frame;

[0058] Step 2: Use the global feature vector of the temporal pyramid structure and the local feature vector of the face to splice in the feature dimension to obtain spliced ​​feature 1, and generate a temporary position embedding and a learnable embedding vector for the time dimension. Add the spliced ​​feature 1 and the temporary position embedding and then splice them with the learnable embedding vector to obtain spliced ​​feature 2. Without changing the time dimension, use the spliced ​​feature 2 as the input of the decoder and downsample it. The downsampling results obtained from each decoding layer are spliced ​​and multi-scale fused to obtain a two-dimensional global feature vector.

[0059] Step 3: The weighted sum of the two-dimensional global feature vector and the physiological feature vector is used to obtain the prediction result of the student participation;

[0060] Step 4: Construct a loss function to adjust the trainable parameters in the fusion prediction model.

[0061] Given the large number of features available in the video, some features may be invalid for fusion prediction and judgment; the embodiment avoids the negative impact of redundancy of selected features by selecting global feature vectors, local facial feature vectors, and physiological feature vectors while ensuring that the model can be effectively judged; this embodiment models features at three different visual levels, considers time changes and monitors the trend of students' heart rate, provides a comprehensive assessment of student participation, and complements multi-dimensional features to improve the comprehensiveness of the analysis, enhance the robustness of the model, reduce the impact of interference, model spatiotemporal correlations, capture dynamic participation patterns, and improve the generalization ability of the model.

[0062] This implementation prevents the omission of important features while minimizing the number of redundant features. Finally, it uses mid-level fusion (fusion of global facial feature vectors and local facial feature vectors) and decision-level fusion (fusion of two-dimensional global feature vectors and physiological feature vectors) to integrate features from different levels. A temporal pyramid structure is used to model the temporal relationships between features across these levels. Finally, student engagement levels are assessed using learning videos, improving the model's prediction accuracy and robustness.

[0063] In one embodiment, step 2 is to extract the global facial feature vector, local facial feature vector, and physiological feature vector from the student's learning video frame by frame, specifically:

[0064] This embodiment selects multi-level representative features, including global facial feature vectors, local facial feature vectors, and physiological feature vectors. Specifically, the global facial feature vector focuses on the student's overall state and captures general facial information. The local facial feature vector focuses on specific areas of the face, capturing more subtle expressions and changes. The physiological feature vector reflects the student's internal state, which is closely related to their physiological state.

[0065] In the initial data processing stage, the face recognition tool (OpenFace) is used to process the learning video frame by frame to extract the facial image in the video and obtain the local key features of the facial image; remote photoplethysmography (rPPG) technology is used to calculate the physiological characteristics of the entire learning video.

[0066] Among the many key local features, we selected key features that represent facial activity as local facial features for this experiment. These key features include changes in eye focus and facial position. While keeping other conditions unchanged, this example discusses the results of engagement analysis based on these two key features (eye focus and facial position).

[0067] Based on previous research, we understand how to use heart rate to determine student engagement and more accurately extract physiological information from learning videos. Using rPPG technology, we can extract student facial information from videos without touching them. Based on this facial information, we can calculate physiological characteristics such as heart rate and the interval between peaks. Using these physiological characteristics, we can observe students' learning from a global perspective, minimizing information leakage and increasing the robustness of the video.

[0068] This embodiment combines the spatiotemporal convolutional VIT network to process the global features of the face; secondly, it combines the spatiotemporal transformation network to process the local facial features in the video; thirdly, it combines the rPPG technology to calculate the physiological characteristics of the entire video.

[0069] Specifically, the spatial correlation of pixels within a video frame can lead to feature omissions when extracting information. Therefore, when extracting a global facial feature vector, it is important not only to ensure that the temporal relationship between video frames is not disrupted, but also to consider the spatial relationship between feature elements within the frames. Therefore, the process for generating a global facial feature vector in this embodiment is as follows (a1) to (a4):

[0070] (a1) Use face recognition tools to extract faces from students’ learning videos and obtain a global facial image;

[0071] First, we build a student engagement dataset based on students' learning videos , using the face recognition (OpenFace) tool to extract the face of the student participation dataset. Get the global face image collection , represents the number of samples, Representative A global face image extracted from a video.

[0072] (a2) Convolution is performed on the global facial image to obtain a preliminary feature image, which is then input into the position embedding module of the Transformer.

[0073] Use one The convolutional neural network performs preliminary feature extraction on each facial global image set to obtain feature images of equal size. The feature image 1 after the convolutional neural network is sent to the residual network for convolution processing, and the output feature image 2 is residually connected with the feature image 1 to obtain the preliminary feature image , (B, T, C, H, and W represent the batch, image sequence length, number of channels, image height, and image width, respectively);

[0074] ;

[0075] in, is the global facial image, is a convolutional neural network, is the residual network.

[0076] (a3) Divide each preliminary feature image into blocks of equal size, generate a positional embedding for each block, concatenate each block with the corresponding learnable embedding vector, and concatenate all blocks included in each preliminary feature image along the first dimension as the input feature sequence to the Transformer encoder.

[0077] Input each frame of preliminary feature image into the position embedding module (PositionEmbedding) of Transformer, and use the size of P (P=16) to divide each frame of preliminary feature image into image blocks , the number of image blocks is . Generate the corresponding position embedding for each image block ( ), (D= Represents the feature dimension). Each image block is concatenated with the learnable embedding vector (CLS Token) to obtain the image block ,in is the number of image blocks.

[0078] (a4) In each layer of the encoder, the learnable embedding vector interacts with all image patches in the input feature sequence using a multi-head attention mechanism, and the output of the learnable embedding vector is used as the global facial feature vector.

[0079] In order to further process the features, we will include the image blocks corresponding to the preliminary feature images of each frame They are stitched together in batches, focusing on the spatial relationships within the images. The image is fed into the Transformer encoder, where a multi-head self-attention module establishes connections between different parts of the image sequence. The linearly transformed query matrix Q, key matrix K, and value matrix V are split into multiple heads, and each head independently calculates the attention score:

[0080] ;

[0081] ;

[0082] ;

[0083] in, is the attention mechanism, are query, key and value respectively, Respectively An attention head, Respectively The query, key, and value corresponding to each attention head, for dimension.

[0084] In the multi-head attention mechanism, the dot product operation of Q and K of each head is performed to obtain the Z matrix. The Z matrix is ​​scaled and the SoftMax function is applied to normalize the attention scores. The normalized attention weights are weighted summed with the V matrix to obtain the output matrix of each head. Finally, the output matrices of multiple heads are spliced ​​to output a complete output matrix R. The matrix R is combined with the input image Perform a residual connection operation and send it to the feedforward neural network layer, and then perform another residual connection operation to maintain the stability of the model. Perform a linear operation on the features output after the Transformer encoder, and obtain the corresponding learnable embedding vector (CLS Token) for the feature sequence of each image block as the global facial feature vector of each frame image. , are batch, time, Represents the feature dimension.

[0085] ;

[0086] in, is the batch normalization layer, is a linear layer, is the residual connection.

[0087] Specifically, extracting student engagement from learning videos often encounters unique challenges. For various reasons (e.g., glare from glasses, hand obstruction), extracting information solely from facial features can miss many details. Therefore, extracting facial feature points is crucial. This allows us to determine whether students are paying attention based on how their facial features change over time. Leveraging the sequential nature of temporal convolutional networks (TCNs), we designed a module combining TCNs and Transformers. This module analyzes the impact of changes in key facial features across time and space.

[0088] The generation process of facial local feature vectors is as follows (b1) to (b3):

[0089] (b1) Using face recognition tools to extract faces from students’ learning videos to obtain facial feature sequences;

[0090] Use the face recognition (OpenFace) tool to extract facial feature point information, and calculate the facial position features of two adjacent frames And unify their lengths, N represents the total number of video datasets, (T, W represent the video length and the number of important facial features respectively) are the important facial features extracted from a video.

[0091] (b2) the face feature sequence is divided by time convolution with a set step, and then spatial feature extraction is performed by a spatial convolution layer to obtain a spatial feature vector;

[0092] A 64-channel time convolution network is constructed to analyze the time sequence relationship of the face position feature. The face position feature sequence is divided by time convolution with a step of , that is, the time interval corresponding to the face image extraction. Then it is sent to a spatial convolution layer with a channel number of and a step of to extract spatial feature information. The extracted features are normalized and sent to an average pooling layer using a RELU function as the activation function to obtain a spatial feature vector ;

[0093] ;

[0094] wherein, is an average pooling layer, is an activation layer, is a batch normalization layer, is a spatial convolution layer, is a time convolution network.

[0095] (b3) the spatial feature vector is sequentially subjected to feature extraction by a multi-head self-attention mechanism and a feedforward neural network to obtain a face local feature vector.

[0096] The spatial feature vector is sent to the multi-head self-attention mechanism for calculation, and the calculation mechanism is similar to the processing process of the multi-head self-attention mechanism in the generation process of the face local feature vector.

[0097] Finally, the feedforward neural network is output to the linear transformation layer, and the feature extraction is completed to obtain the face local feature vector :

[0098] ;

[0099] wherein, is a feedforward neural network, is a multi-head self-attention mechanism.

[0100] Specifically, the generation process of the physiological feature vector is as follows:

[0101] (c1) the pixel values of the two cheek and forehead regions in the face are obtained from the student's learning video, based on which the normalized RGB signal is calculated;

[0102] At the beginning of the physiological feature extraction calculation, it is necessary to obtain the blood volume pulse signal (BVP) of the face from the learning video. The blood volume pulse signal of each video frame reflects the cardiovascular activity of the student. The rPPG technology uses the plane orthogonal skin method (POS) to use the subtle skin color changes caused by blood flow to infer the blood volume pulse signal. Physiologically, the best place to observe the blood volume pulse of the face is on both sides of the cheeks and forehead, because these two parts have rich arterial branches and dense capillary networks. Calculate the normalized RGB signals of these two sub-regions (cheeks and forehead) :

[0103] ;

[0104] in, Indicates calculating the average pixel value of the RGB sub-region. is the pixel matrix of the RGB sub-region, They are the pixel matrix row indices, i.e. Row and The normalized RGB signal is obtained by dividing the pixel matrix of the sub-region by the average pixel value.

[0105] (c2) Map the normalized RGB signal to the green channel to obtain the matrix S. Take two adjacent rows of the matrix S as a group, divide the rows of the matrix S into groups in sequence, and calculate the weighted average of each group to obtain the group matrix;

[0106] Physiologically, the color change of the green channel can best reflect the blood flow under the skin of the face. The normalized RGB signal is mapped to the green channel to obtain the matrix S. W is an orthogonal matrix used to project the RGB values ​​onto the green channel. This can minimize noise and improve the accuracy of the BVP signal. The first group group matrix is ​​obtained by calculating the weighted average of the first and second rows of the matrix S. :

[0107] ;

[0108] ;

[0109] in, is the first row of the matrix S, is the second row of matrix S, is the variance of the first row of matrix S, is the variance of the first row of matrix S, for The transpose of is the normalized RGB signal.

[0110] (c3) After removing the mean value of all group matrices, the processed signals are accumulated to obtain the matrix H;

[0111] In order to reduce the offset of the normalized RGB signal and make the change of the normalized RGB signal more prominent, all group matrices are subjected to the mean value removal operation to combine them to obtain the matrix H.

[0112] (c4) Detrending the accumulated matrix H, and then using a bandpass filter to separate the heart rate frequency from the trended matrix to obtain a filtered matrix;

[0113] Finally, detrending the matrix H yields the matrix The detrending matrix is ​​used to eliminate or weaken the long-term systematic changes in the data, and the heart rate frequency is separated from the Separate and get the filtered matrix .in, is the detrending function, is the bandpass filter function:

[0114] ;

[0115] ;

[0116] ;

[0117] in, Indicates the Go to Between columns The data value of the row, Indicates the The data sequence of the row sensor at all time points, Represents all matrices The average value of is the detrending function, is a bandpass filter function.

[0118] This embodiment is achieved by For the subset of sensors in row 0 Add an offset , this offset is The average value of the entire series Physically, this is equivalent to adjusting the baseline of the sensor during a specific time period to make it closer to the initial value (for example, correcting the system deviation during a certain time period).

[0119] pass , detrending the entire dataset. Transposing ensures that the data from each sensor (original row) is processed independently, removing any linear trend or mean. Physically, this eliminates any long-term drift or DC component from all sensor signals, highlighting short-term variations (such as those detected during event detection).

[0120] (c5) Use the scipy library to obtain the peak value of the heart rate corresponding to the filtered matrix, calculate the maximum and minimum values ​​of the heart rate peak, and thus obtain the interval time between peaks;

[0121] Using the Scipy library, we can obtain the peaks of the heart rate and calculate the maximum and minimum heart rate peaks. Since heart rate and the number of peaks within a certain period are positively correlated, we can calculate the heart rate and the average time difference between peaks. This provides the student's physiological characteristics.

[0122] (c6) The student's heart rate and the interval time between peaks are used as physiological features and then pass through the fully connected layer, residual block and fully connected layer to obtain the physiological feature vector , Indicates Batch batch, Represents the feature dimension, where the student’s heart rate is obtained by capturing the RGB change rate of a specific position on the student’s face.

[0123] Extracting student engagement in learning videos can also be done based on the entire video. Using rPPG technology, students' facial information can be extracted from the video without touching the students. Based on this facial information, characteristics such as the student's heart rate and the interval between peaks can be calculated. Based on these physiological characteristics, students' classroom performance can be observed from a global perspective, minimizing information leakage and increasing the robustness of the video.

[0124] In one embodiment, step three is specifically as follows:

[0125] In step 2, three different levels of feature vectors are obtained, namely facial feature vectors , local key feature vector and physiological feature vectors .in, and The images contain temporal information and have a one-to-one correspondence in time. We aim to identify the temporal relationship between facial feature vectors and local key feature vectors and observe their impact on the prediction accuracy of the fusion prediction model. This embodiment utilizes the downsampling method in the PVT network (Pyramid Visual Transformer) by processing based on the obtained temporal features rather than directly processing the image. This shifts image-based processing to time series processing while ensuring the length of the time series, achieving pyramid-structured feature fusion of the temporal sequence.

[0126] In this embodiment, the multi-level feature fusion process uses mid-level fusion (fusion of global facial feature vectors and local facial feature vectors) and decision-level fusion (fusion of two-dimensional global feature vectors and physiological feature vectors) to integrate features at different levels. Specifically, the mid-level fusion process includes:

[0127] (d1) First, the facial feature vector , local key feature vector Splicing is performed on the feature dimension to obtain splicing feature 1 ,in, is the time dimension, are the dimensions of the global facial feature vector and the local facial feature vector, respectively, and a temporary position embedding is also generated for the time dimension and learnable embedding vector (CLS Token). and After adding, concatenate with CLS Token to get concatenated feature 2 Without changing the time dimension In the case of , each time the transformer decoding layer is passed, the spliced ​​feature 2 is downsampled once, and each downsampling is finally spliced ​​together and sent to the cross-scale embedding layer for multi-scale fusion, and finally the two-dimensional global feature vector after time series processing is returned. , Represents the feature dimension.

[0128] ;

[0129] ;

[0130] ;

[0131] ;

[0132] in, is the learnable embedding vector, For low-level features, For intermediate features, For advanced features, is the cross-scale embedding layer, represents the dimension reduction MLP layer, represents the Transformer decoding layer, and It constitutes a dimensionality reduction function, Represents the features before being fed into the dimensionality reduction function, Represents the feature output by the dimensionality reduction function, which is the feature of the time series feature after different dimensionality reduction.

[0133] In one embodiment, step four is specifically as follows:

[0134] For the obtained two-dimensional global feature vector and physiological feature vectors The weighted summation process is performed to use the psychological characteristics of students obtained through the video to supplement the global features that may be missed in the time series processing. and Realize and Weighted processing, finally get the prediction result :

[0135] .

[0136] In one embodiment, step five is specifically as follows:

[0137] During the training process of the fusion prediction model, cross entropy loss and L2 regularization loss are used to optimize the model. The gradient of the cross entropy loss function is relatively large, which is conducive to the rapid convergence of the model in the early stage. In addition, the cross entropy loss function has a greater penalty for misclassification, which helps the model pay more attention to those difficult-to-classify samples during the training process. Regularization loss is used to penalize model complexity and prevent overfitting. It is given by:

[0138] ;

[0139] in, is the sample size, is the number of categories, that is, the labels of the training data set, such as the classification of the levels of student participation, is the sample index, is the category index, is the true label value, is the predicted value output by the fusion prediction model, It is a regularization parameter that controls the complexity loss of the model by affecting the size of the weight. Represents the squared L2 norm of the trainable parameters.

[0140] To verify that this embodiment can improve the prediction accuracy and robustness of the model, thereby obtaining more accurate student engagement prediction results, this embodiment performs an ablation analysis on the local facial feature vectors. The ablation experiment is as follows:

[0141] This embodiment uses the changes in eye focus and facial position as local facial feature vectors to conduct a student engagement prediction experiment;

[0142] Comparative Example 1 is an experiment to predict student engagement using only the focus change of the eyes as the local facial feature vector;

[0143] Comparative Example 2 is an experiment to predict student engagement by using only the focus change of facial position as the facial local feature vector;

[0144] Table 1 Key feature selection experiments on the DAISEE dataset

[0145]

[0146] Table 2 Key feature selection experiments on the EmotiW2023 dataset

[0147]

[0148] This embodiment uses some commonly used evaluation indicators in machine learning to evaluate model prediction results, such as accuracy, precision, recall, F1 score, and mean absolute error (MAE).

[0149] Extracting students' classroom participation in videos often encounters some special situations, that is, the students' facial details may be missed due to various reasons (such as reflection from glasses, hand covering), resulting in many details being missed when extracting information solely from facial features.

[0150] Among them, the main features are the focus change of the eyes and the position change of the face. In the case of keeping other conditions unchanged, the results of the participation analysis based on the two main features are discussed. It can be found from the observation results that the eye focus and the face position also have different effects on the model in different data sets. On DAiSEE and EmotiW, the accuracy of the face position is higher than that of the eye focus by 0.12% and 0.83% respectively, but the average loss on DAiSEE is lower than that on EmotiW. This depends on the influence of factors such as illumination and resolution of different data sets on the sensitivity of input information. It can be seen that the accuracy of the embodiment combining the eye focus and the face position change is the highest, which is 0.46% and 0.19% higher than the face position accuracy respectively. This can prove that the two features have a complementary effect on improving the prediction accuracy of the model, and can make up for some differences caused by data sensitivity.

[0151] The above merely describes the preferred embodiments of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can make equivalent replacements or changes to the technical solutions and the inventive concept of the present application within the technical scope disclosed by the present application, which should be covered within the protection scope of the present application.

Claims

1. A multi-level facial feature fusion method for evaluating student engagement, characterized by: Input the student's learning video into the trained fusion prediction model to obtain the prediction results of student engagement; The training process of the fusion prediction model is as follows: Obtain multiple learning videos of multiple students to build a training dataset; Extract the global facial feature vector, local facial feature vector and physiological feature vector from the students' learning videos frame by frame; Using a temporal pyramid structure, the global facial feature vector and the local facial feature vector are concatenated in the feature dimension to obtain a concatenated feature 1. A temporary position embedding and a first learnable embedding vector are generated for the time dimension. The concatenated feature 1 is added to the temporary position embedding and then concatenated with the first learnable embedding vector to obtain a concatenated feature 2. Without changing the time dimension, the concatenated feature 2 is used as the input of the decoder and downsampled. The downsampling results obtained at each decoding layer are concatenated and multi-scale fused to obtain a two-dimensional global feature vector. The prediction result of student engagement is obtained by weighted summing the two-dimensional global feature vector and the physiological feature vector; Construct a loss function to adjust the trainable parameters in the ensemble prediction model.

2. The multi-level facial feature fusion method according to claim 1, characterized in that: In extracting the facial global feature vector, facial local feature vector, and physiological feature vector frame by frame from the student's learning video, the generation process of the facial global feature vector is as follows: Use face recognition tools to extract faces from students’ learning videos and obtain a global facial image; Perform convolution extraction on the global facial image to obtain a preliminary feature image, which is then input into the position embedding module of the Transformer. Divide each frame of preliminary feature image into image blocks of the same size, generate position embedding for each image block, concatenate each image block with the corresponding second learnable embedding vector, and concatenate all image blocks included in each frame of preliminary feature image in batches and input them into the Transformer encoder as the input feature sequence; In each layer of the encoder, the second learnable embedding vector interacts with all image patches in the input feature sequence using a multi-head attention mechanism, and the output of the second learnable embedding vector is used as the global facial feature vector.

3. The multi-level facial feature fusion method according to claim 2, characterized in that: In the initial feature image obtained by convolution extraction of the global facial image, specifically: Use convolutional neural network to perform preliminary feature extraction on each global facial image to obtain feature image 1; The feature image 1 is sent to the residual network for convolution processing, and the output feature image 2 is residually connected with the feature image 1 to obtain the preliminary feature image.

4. The multi-level facial feature fusion method according to claim 1, characterized in that: In extracting the global facial feature vector, local facial feature vector, and physiological feature vector frame by frame from the student's learning video, the generation process of the local facial feature vector is as follows: Use face recognition tools to extract faces from students' learning videos to obtain facial position feature sequences; The facial position feature sequence is divided by temporal convolution with a set step size, and then spatial feature vectors are obtained by spatial feature extraction through a spatial convolution layer. The multi-head self-attention mechanism and feedforward neural network are used in turn to extract the spatial feature vector to obtain the local facial feature vector.

5. The multi-level facial feature fusion method according to claim 1, characterized in that: In extracting the global facial feature vector, local facial feature vector, and physiological feature vector from the student's learning video frame by frame, the process of generating the physiological feature vector is as follows: Obtain pixel values ​​of the cheeks and forehead regions on both sides of the face from the student's learning video and calculate the normalized RGB signal based on them; The normalized RGB signal is mapped to the green channel to obtain the matrix S. Two adjacent rows of the matrix S are taken as a group. The rows of the matrix S are divided into groups in sequence, and the weighted average result of each group is calculated to obtain the group matrix. After removing the mean value from all group matrices, the processed signals are accumulated to obtain the matrix H; After detrending the accumulated matrix H, a band-pass filter is used to separate the heart rate frequency from the trended matrix to obtain a filtered matrix; Use the scipy library to obtain the peak value of the heart rate corresponding to the filtered matrix, calculate the maximum and minimum values ​​of the heart rate peak, and thus obtain the interval time between peaks; The student's heart rate and the interval time between peaks are used as physiological features and then passed through the fully connected layer, residual block and fully connected layer in sequence to obtain the physiological feature vector.

6. The multi-level facial feature fusion method according to claim 1, characterized in that: The predicted result of student participation is obtained by weighted summing the two-dimensional global feature vector and the physiological feature vector , the specific formula is as follows: ； in, and are the learnable parameters, is the two-dimensional global eigenvector, is the physiological feature vector.

7. The multi-level facial feature fusion method according to claim 1, characterized in that: The loss function The construction formula is as follows: in, is the sample size, is the number of categories, is the sample index, is the category index, is the true label value, is the predicted value output by the fusion prediction model, is the regularization parameter, Represents the squared L2 norm of the trainable parameters.

8. A multi-level facial feature fusion system for evaluating student engagement, characterized by: Input the student's learning video into the trained fusion prediction model to obtain the prediction results of student engagement; The training process of the fusion prediction model includes a training set construction module, a feature extraction module, a first fusion module, a second fusion module and a loss construction module; The training set construction module is used to obtain students' learning videos to construct a training dataset; The feature extraction module is used to extract the global facial feature vector, local facial feature vector and physiological feature vector from the student's learning video frame by frame; The first fusion module is used to use the temporal pyramid to concatenate the global facial feature vector and the local facial feature vector in the feature dimension to obtain a concatenated feature 1, and generate a temporary position embedding and a first learnable embedding vector for the time dimension. The concatenated feature 1 is added to the temporary position embedding and then concatenated with the first learnable embedding vector to obtain a concatenated feature 2. Without changing the time dimension, the concatenated feature 2 is sequentially downsampled as the number of decoding layers decreases. After each downsampling, multi-scale fusion is performed to obtain a two-dimensional global feature vector. The second fusion module is used to perform weighted summation of the two-dimensional global feature vector and the physiological feature vector to obtain the prediction result of student engagement; The loss building module is used to build a loss function to adjust the trainable parameters in the ensemble prediction model.

9. The multi-level facial feature fusion system according to claim 8, characterized in that: The loss function The construction formula is as follows: in, is the sample size, is the number of categories, is the sample index, is the category index, is the true label value, is the predicted value output by the fusion prediction model, is the regularization parameter, Represents the squared L2 norm of the weight vector.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a plurality of classification programs, which are used to be called by a processor and execute the multi-level facial feature fusion method according to any one of claims 1 to 7.

Images