An online learning concentration evaluation method based on multi-mode feature fusion

Abstract

The application discloses an online learning concentration evaluation method based on multi-mode feature fusion, which comprises the following steps: obtaining a target video with a concentration label, sampling the target video to obtain target frame images, and extracting multi-mode features one by one; fusing the multi-mode features with different weights, and then splicing a time sequence feature matrix through convolution operation to perform time dimension coding and generate a fusion feature matrix of the video through a multi-head self-attention module; obtaining a prediction probability vector through a linear layer mapping and an activation function; obtaining a trained concentration evaluation model through training parameters; inputting a video to be evaluated into the model to obtain a prediction probability vector of the target video to be evaluated, and selecting a concentration label corresponding to the maximum probability value as an evaluation result; the method can well fuse different visual features and realize time sequence alignment, effectively extract fine-grained features of learners during online learning, and has high accuracy and robustness.

Description

Technical Field

[0001] This invention relates to the field of attention assessment technology in online learning, and in particular to a method for assessing attention in online learning based on multimodal feature fusion. Background Technology

[0002] In recent years, the contactless lifestyle has sparked a nationwide wave of online learning, online meetings, online work, and online interviews. Online education, AI-powered tutoring, and educational robot tutoring, which rely on remote networks, offer advantages such as a wide variety of courses and flexible learning methods, making them popular. Simultaneously, learners' learning status has become a major concern. Besides the quality of teaching, students' learning outcomes and quality are closely related to their concentration. Concentration refers to the degree to which students focus their attention during learning. Concentration recognition technology can monitor students' concentration levels in real time during class, reminding them to return to a focused state and assisting teachers in adjusting course content and teaching pace based on students' concentration levels.

[0003] There are various methods for identifying student attention. Traditional teaching practices mainly rely on teacher observation. Early researchers also designed psychological questionnaires for students and teachers to conduct self-evaluation or two-way assessments of student attention. However, these methods are highly subjective, resulting in unsatisfactory accuracy and reliability. Some experts have also used specific medical devices to monitor students' brainwaves and physiological signals during class to assess their attention. However, this method requires sophisticated equipment, and the wearing of invasive devices can interfere with students' concentration levels. To address these issues, more and more researchers are focusing on computer vision-based student attention detection technologies. These technologies can be broadly categorized into three types: facial feature-based attention recognition; behavioral feature-based attention recognition; and a combination of facial and behavioral features. The implementation processes of these technologies are similar: first, a camera is used to acquire images of the upper body, including the face; then, different types of facial or body features are extracted as needed, and these features are fused; finally, attention classification or regression training is performed.

[0004] These solutions often utilize intelligent technologies such as facial recognition, expression recognition, and posture estimation to identify focus levels. However, most studies only use a single type of visual information or combine it with human information, such as survey results and pre-class test results, to identify focus levels. The accuracy, objectivity, scalability, and intelligence of automatic focus level identification need to be improved. Summary of the Invention

[0005] The technical problem to be solved by this invention is to provide an online learning attention assessment method based on multi-modal feature fusion. This method can effectively solve the recognition defects of single feature discrimination models, can effectively fuse different visual features and achieve temporal alignment, effectively extract fine-grained features of learners during online learning, and has high accuracy and robustness in attention assessment.

[0006] The technical solution adopted by the present invention to solve the above-mentioned technical problems is as follows: A method for evaluating online learning focus based on multimodal feature fusion includes the following steps: Step S1, Multimodal Feature Extraction: Obtain the training set of the target videos. Each target video corresponds to a focus label. The focus label defines the expected probability vector of the target video. Sample each target video in the training set to obtain the target frame image. Extract multimodal features from each target frame image one by one. Step S2, Multimodal Feature Fusion: The multimodal features of the extracted target frame image are fused with different weights to obtain the fused features corresponding to the target frame image. The fused features corresponding to each target frame image are then concatenated to form the fused feature matrix of the target frame image. Step S3, Temporal Feature Integration: The fusion feature matrix of the target frame image is generated into multiple hidden layer features through convolution operation. All hidden layer features are concatenated sequentially to form the temporal feature matrix of the frame image. The temporal feature matrix of the frame image is encoded with a time dimension and the fusion feature matrix of the video is generated by multi-head self-attention module. The fusion feature matrix of the video is mapped to a preliminary prediction matrix through a linear layer. The preliminary prediction matrix is ​​input into the activation function to obtain the prediction probability vector of the target video. Step S4, Model Construction: Define a loss function. Input the predicted probability vector of the target video in Step S3 and the expected probability vector of the target video in Step S1 into the loss function. Calculate the loss value through the loss function to train and iterate the parameters. Stop training when the number of iterations reaches the preset maximum number of iterations. Use the parameters obtained when the loss value is minimized during training as the parameters after training. Optimize the parameters in multi-modal feature extraction, multi-modal feature fusion, and temporal feature integration using the trained parameters. Construct an attention evaluation model based on the optimized multi-modal feature extraction, multi-modal feature fusion, and temporal feature integration using the trained parameters. Step S5, Attention Assessment: Input the video to be assessed into the attention assessment model to obtain the predicted probability vector of the target video to be assessed. Select the attention label corresponding to the largest probability value in the predicted probability vector of the target video to be assessed as the attention assessment result of the learner of the video to be assessed.

[0007] Step S1 specifically includes: Step S1-1: Obtain the contents A training set C of target videos, where each target video corresponds to a focus label m, is defined, and the expected probability vector for each target video is defined. , ,when hour, =1, otherwise =0; Uniformly sample each target video in C to obtain multiple target frame images, and these multiple target frame images constitute a target frame image set. ; Step S1-2: Let Indicates that the number in training set C is The target video, for Perform uniform sampling to obtain L frames of images and use them as target frame image set , ∈[0, N); Let the number be Target video target frame image set The target frame image numbered j is G. ij , where j∈[0, L), for the target frame image G ij Extract multimodal features one by one, and repeat this operation until the target frame image set has been processed. ,in .

[0008] The multi-modal features include global image features, eyelid opening features, head orientation features, gaze direction features, facial AU unit features, pose key point group features, and hand key point group features.

[0009] Step S2 specifically includes: Step S2-1: Transfer the target frame image G obtained in step S1-2 to... ij The multimodal features are concatenated to form a feature vector S with a size of 32+2+3+6+35+16+32=126; Step S2-2: Map the feature vector S calculated in step S2-1 to the fused feature S through a linear layer. This allows for the fusion of different features with different weights; where feature S has a dimension of 126, the linear layer has a size of 126×32, and the fused feature S... The dimension is 32; Step S2-3: Repeat steps S2-1 and S2-2 until the target frame image set is reached. The fusion features of the L frames in the image have been calculated individually; the fusion features of the L frames are then concatenated to form the fusion feature matrix M of the target frame image. ={S j }, where S j Let G be the target frame image numbered j. ij The fusion features, j∈[0, L), M The dimensions are L×32.

[0010] Step S3 specifically includes: Step S3-1: Let T be the number of hidden layers in the temporal convolutional network, and T=L. Then, take the fusion feature matrix M of the frame image calculated in step S2-3. Input is fed into a temporal convolutional network with T hidden layers, and for the input M Perform 1D convolution operations to generate the first hidden layer feature h1 and for h1 Continue with 1D convolution operations to generate the second hidden layer feature h2. Repeat this 1D convolution operation for each newly generated hidden layer feature until the T-th hidden layer feature h is obtained. T h1 h2 , ..., h T The temporal feature matrix H of the frame images is formed by sequentially splicing them together. ; Step S3-2: Calculate the temporal feature matrix H of the frame image obtained in step S3-1. With temporal encoding added, the fusion feature matrix of the video is calculated using a multi-head self-attention module. ,in The dimensions are L×d v d v For 64, 128, or 256; then for By flattening, its dimension is changed to L•d. v ; Step S3-3: Calculate the fusion feature matrix of the video obtained in step S3-2. A preliminary prediction matrix is ​​obtained through linear layer mapping. The size of the linear layer is L•d v ×t, the preliminary prediction matrix The dimension is t. Let be the prediction matrix for the target video corresponding to the a-th category; Step S3-4: Use the Softmax activation function to transform the preliminary prediction matrix obtained in step S3-3. Transform into a prediction probability vector of the target video with dimension t. ,in .

[0011] In step S3-1, the temporal feature matrix H of the frame image is obtained. The process specifically includes: Let the kernel size of the first 1D convolution operation be 7, and the initial dilation value be 2; let the kernel size of the second 1D convolution operation be 7, and the initial dilation value be 4; and so on, with the kernel size of the T-th 1D convolution operation being 7, and the initial dilation value being 2. T The fusion feature matrix M of the frame images The size is transposed from L×32 to 32×L, and then the first 1D convolution operation is performed to generate the first hidden layer feature h1 with a hidden layer feature dimension of 128. The first hidden layer feature h1 Perform a second 1D convolution operation to generate a second hidden layer feature h2 with a hidden layer feature dimension of 128. And so on, the Tth hidden layer feature h with dimension 128 is calculated. T h1 h2 , ..., h T The temporal feature matrix H of the frame images with a size of L×128 is formed by sequentially stitching them together. .

[0012] The process of obtaining the fusion feature matrix O of the video in step S3-2 specifically includes: Set an initial value of 0 for a learnable coding matrix of size L×128, and then combine the learnable coding matrix with the temporal feature matrix H of the frame image. The result obtained by adding elements together is denoted as P. , where P The dimensions are L×128; P Input a multi-head self-attention module to extract features and obtain the fused feature matrix of the video. , , Let Q, K, and V be the activation function, where Q, K, and V are three learnable matrices in the multi-head self-attention module. , , ,in , , It is a fully connected layer. The dimensions are all 128×d k , The dimensions are 128×d v Video fusion feature matrix The dimensions are L×d v .

[0013] Step S4 specifically includes: Define the loss function FocalLoss, which is the predicted probability vector of the target video in step S3-4. and the expected probability vector of the target video in step S1-1 The formula is as follows: ,in The loss value is calculated using a loss function to train and iterate the parameters. Training stops when the number of iterations reaches the preset maximum number of iterations. The parameters obtained when the loss value is minimized during training are used as the post-trained parameters. The post-trained parameters are used to optimize the parameters in multi-modal feature extraction, multi-modal feature fusion, and temporal feature integration. Based on the optimized multi-modal feature extraction, multi-modal feature fusion, and temporal feature integration using the post-trained parameters, a focus assessment model is constructed.

[0014] Compared with the prior art, the advantages of the present invention are as follows: (1) The present invention follows a hierarchical processing step of multimodal feature extraction, multimodal feature fusion and temporal feature integration of frame images. It can effectively solve the recognition defects of single feature discrimination model, can effectively fuse different visual features and achieve temporal alignment, effectively extract fine-grained features of learners during online learning, and still have good classification effect when facing difficult datasets. On the public attention dataset DAISEE, the evaluation method based on multimodal feature fusion of the present invention has an accuracy of 64.08% on the four-class classification task and an accuracy of 95.62% on the two-class classification task. It has greatly improved the accuracy of both the four-class classification task and the two-class classification task. At the same time, the accuracy of the four-class classification task is higher than that of the previous evaluation method.

[0015] (2) This invention fuses the multi-modal features of frame images from different frames and calculates the fusion feature matrix of the video through a multi-head self-attention module. Compared with the method of recognizing attention by using a single type of information, this invention fuses multiple types of visual information to recognize attention, which has higher reliability and robustness. There are both connections and differences between different types of information. Information fusion can make up for the shortcomings of each type of single information, so that the strengths of each type of information can be better utilized, and the shortcomings can be compensated by other information. This can greatly improve the spatiotemporal modeling ability of the model, realize spatiotemporal feature alignment, and thus improve the accuracy of attention recognition. Attached Figure Description

[0016] Figure 1 This is a flowchart of the present invention. Detailed Implementation

[0017] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0018] Example 1: A method for evaluating online learning focus based on multimodal feature fusion includes the following steps: (1) Step S1, Multimodal feature extraction: Obtain the training set of the target video. Each target video corresponds to a focus label. The focus label defines the expected probability vector of the target video. Sample each target video in the training set to obtain the target frame image. Extract multimodal features from each target frame image one by one. The specific steps for S1 are as follows: Step S1-1: Obtain the contents A training set C of target videos, where each target video corresponds to a focus label m, is defined, and the expected probability vector for each target video is defined. , ,when hour, =1, otherwise =0; Specifically, if the focus label m of the target video is 1, then the expected probability vector is {0, 1, 0, 0}; For each target video in C, uniform sampling is performed to obtain multiple target frame images, and these multiple target frame images constitute a target frame image set. ; Step S1-2: Let Indicates that the number in training set C is The target video, for Perform uniform sampling to obtain L frames of images and use them as target frame image set , ∈[0, N); Let the number be Target video target frame image set The target frame image numbered j is G. ij , where j∈[0, L), for the target frame image G ij Extract multimodal features one by one, and repeat this operation until the target frame image set has been processed. ,in .

[0019] Among them, the multimodal features include global image features, eyelid opening features, head orientation features, gaze direction features, facial AU unit features, pose key point group features, and hand key point group features; Specifically, in steps S1-2, the target frame image G... ij Extracting global image features includes: Extracting G using the ResNet18 model ijFrame-level features R are mapped to R through a linear layer. To achieve feature dimensionality reduction; target frame image G ij Before being input into the ResNet18 model, the width is converted to W, the height is converted to H, and the feature dimension of R is 512. The feature dimension is 32, and the linear layer size is 512×32; R For target frame image set Image G of target frame number j ij Global image features; for For the remaining target frame images, features are extracted frame by frame following the above process.

[0020] By adopting a method of first extracting features and then reducing the dimensionality of the features, the problems of missing global fine-grained features and excessive feature dimensionality are solved, which has a beneficial effect on supplementing fine-grained information on focus.

[0021] Specifically, in steps S1-2, from frame image G ij Extracting eyelid opening features includes: Extract the target video C using the FeatureExtraction script provided by OpenFace 2.0. i The frame-level features are extracted and a feature file is generated, denoted as OF. i.csv Let frame image G ij The frame features are located in OF i.csv The row containing the cell whose column name is frame and column value is j is denoted as OF. ij ; Frame image G ij The learner eyelid opening feature EAR was calculated from 12 key eye points; among them, , , , , , The six key points of the learner's left eye are represented by the horizontal axis corresponding to OF. ij eye_lmk_x_8, eye_lmk_x_10, eye_lmk_x_12, eye_lmk_x_14, eye_lmk_x_16, eye_lmk_x_18, the ordinate corresponds to OF ij eye_lmk_y_8, eye_lmk_y_10, eye_lmk_y_12, eye_lmk_y_14, eye_lmk_y_16, eye_lmk_y_18 in; , , , , , The six key points of the learner's right eye are represented by their horizontal coordinates corresponding to the OF coordinates. ij eye_lmk_x_36, eye_lmk_x_38, eye_lmk_x_40, eye_lmk_x_42, eye_lmk_x_44, eye_lmk_x_46, the ordinate corresponds to OF ij Let eye_lmk_y_36, eye_lmk_y_38, eye_lmk_y_40, eye_lmk_y_42, eye_lmk_y_44, and eye_lmk_y_46 be the values ​​in the equation; let Eucliden(P, Q) be the Euclid distance between points P and Q. Then the eyelid opening of the learner's left eye is... right eyelid opening of the learner Learner eyelid opening characteristics Depend on and Composed of spliced ​​elements, with a feature dimension of 2; For target frame image set Image G of target frame number j ij The characteristics of eyelid opening; for For the remaining target frame images, features are extracted frame by frame following the above process.

[0022] By calculating the eye keypoints generated by OpenFace, the eyelid opening of the left and right eyes is generated, which solves the problem of measuring eye opening and closing features and has a beneficial effect on supplementing fine-grained information on focus.

[0023] Specifically, in steps S1-2, the target frame image G... ij The head orientation features extracted include: Target frame image G ij The learner's head orientation feature H' is composed of the yaw angle, pitch angle, and roll angle of the head in three-dimensional space, where the yaw angle, pitch angle, and roll angle can be derived from OF. ij The column names are obtained as pose_Rx, pose_Ry, and pose_Rz, in radians; the feature dimension of the learner head orientation feature H' is 3; H' is the target frame image set. The head orientation of the target frame image Gij with target number j; for For the remaining target frame images, features are extracted frame by frame following the above process.

[0024] By calculating the pitch angle generated by OpenFace, the problem of head orientation measurement was solved, which had a beneficial effect on supplementing fine-grained information on attention.

[0025] Specifically, in steps S1-2, the target frame image G... ijThe extraction of gaze direction features includes: Target frame image G ij The learner's gaze direction feature D is described by the eye's direction vector in three-dimensional space. The gaze vector of each eye is composed of three coordinate axes: X, Y, and Z. Among them, the normalized gaze vector D of the learner's left eye is... L Available by OF ij The coordinates gaze_0_x, gaze_0_y, and gaze_0_z are obtained; the normalized gaze vector D of the learner's right eye is obtained. R Available by OF ij The coordinates gaze_1_x, gaze_1_y, and gaze_1_z are obtained from D; the learner's gaze direction feature D is obtained from D. L and D R The image is composed of multiple spliced ​​images with a feature dimension of 6; D represents the target frame image set. Image G of target frame number j ij The characteristics of the line of sight direction; for For the remaining frames, features are extracted frame by frame following the same process.

[0026] By calculating the gaze coordinates generated by OpenFace, the problem of measuring gaze direction was solved, which had a beneficial effect on supplementing fine-grained information on focus.

[0027] Specifically, in steps S1-2, the target frame image G... ij The facial AU unit features extracted include: Target frame image G ij The learner's facial AU unit feature A is described in two forms: "feature intensity" and "feature presence". Feature intensity is described by the feature intensity vector I, which is defined by OF. ij The feature vector E consists of 17 values ​​between [0, 5]: AU01_r, AU02_r, AU04_r, AU05_r, AU06_r, AU07_r, AU09_r, AU10_r, AU12_r, AU14_r, AU15_r, AU17_r, AU20_r, AU23_r, AU25_r, AU26_r, and AU45_r. The feature dimension is 17. The feature existence is represented by the feature existence vector E. ij Description, E ij By OF ijThe feature A consists of 18 values, each consisting of either 0 or 1, concatenated from AU01_c, AU02_c, AU04_c, AU05_c, AU06_c, AU07_c, AU09_c, AU10_c, AU12_c, AU14_c, AU15_c, AU17_c, AU20_c, AU23_c, AU25_c, AU26_c, AU28_c, and AU45_c, with a feature dimension of 18. The learner's facial AU unit feature A is composed of I... ij and E ij It is formed by concatenating two feature vectors, with a feature dimension of 35; A is the target frame image set. Image G of target frame number j ij Facial AU unit features; for For the remaining frames, features are extracted frame by frame following the same process.

[0028] By calculating the facial AU features generated by OpenFace, the problem of measuring fine-grained facial muscle states was solved, which had a beneficial effect on supplementing fine-grained information on attention.

[0029] Specifically, in steps S1-2, the target frame image G... ij The features extracted from the pose keypoint group include: Using the MediaPipe toolkit to process video editing C i For each frame in the dataset, the coordinates of the learner's upper body pose key points are output, forming a set of pose key point coordinate features; using the target frame image G... ij For example, the pose keypoint coordinate group P' consists of the coordinates of the left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, and right hip. These coordinates can be obtained by calling the `process` method of the `MediaPipe.Solutions.Holistic` class, corresponding to the x, y coordinate pairs numbered 11, 12, 13, 14, 15, 16, 23, and 24 in the `pose_landmark.landmark` return value of the function call result. The feature dimension of P' is 8 × 2 = 16. The detection configuration parameters `min_detection_confidence`, `min_tracking_confidence`, and `model_complexity` used by the `MediaPipe.Solutions.Holistic` class are 0.5, 0.5, and 1, respectively. P' is the target frame image set. Image G of target frame number j ij The pose keypoint group features; for For the remaining frames, features are extracted frame by frame following the same process.

[0030] By calculating the pose key points generated by MediaPipe, the problem of human pose measurement was solved, which had a beneficial effect on supplementing fine-grained information on attention.

[0031] Specifically, in steps S1-2, the target frame image G... ij The key point features extracted from the hand include: Process the target video C using the MediaPipe toolkit i For each frame in the dataset, the coordinates of the learner's hand key points are output, forming a set of hand key point coordinate features; the target frame image G... ij For example, the hand keypoint coordinate group feature F is composed of the coordinates of the left and right hand keypoints. Taking the left hand keypoints as an example, it specifically includes: wrist coordinates, thumb metacarpophalangeal joint coordinates, thumb proximal interphalangeal joint coordinates, thumb distal interphalangeal joint coordinates, index finger metacarpophalangeal joint coordinates, index finger proximal interphalangeal joint coordinates, index finger distal interphalangeal joint coordinates, middle finger metacarpophalangeal joint coordinates, middle finger proximal interphalangeal joint coordinates, middle finger distal interphalangeal joint coordinates, ring finger proximal interphalangeal joint coordinates, ring finger metacarpophalangeal joint coordinates, ring finger distal interphalangeal joint coordinates, little finger metacarpophalangeal joint coordinates, little finger proximal interphalangeal joint coordinates, and little finger distal interphalangeal joint coordinates. The coordinates can be obtained from MediaPipe.Sol The `process` method of the `Solutions.Holistic` class obtains the x, y coordinate pairs numbered 0, 1, 2, 3, 5, 6, 7, 9, 10, 11, 13, 14, 15, 17, 18, and 19 from the `left_hand_landmarks.landmarks` return value of the function call result. The feature dimension of `F` is 16 × 2 = 32. The detection configuration parameters used by the `MediaPipe.Solutions.Holistic` class are consistent with the detection parameters used when processing the pose keypoint coordinate group features. `F` represents the target frame image set. Image G of target frame number j ij Key features of the hand; for For the remaining frames, features are extracted frame by frame following the same process.

[0032] By calculating the hand key points generated by MediaPipe, the problem of measuring hand state was solved, which had a beneficial effect on supplementing fine-grained information on focus.

[0033] (2) Step S2, Multimodal feature fusion: The multimodal features of the extracted target frame image are fused with different weights to obtain the fused features corresponding to the target frame image. The fused features corresponding to each target frame image are then concatenated to form the fused feature matrix of the target frame image. The specific steps for S2 are as follows: Step S2-1: Concatenate the 7 multi-modal features obtained in step S1-2 into a feature vector S with a size of 32+2+3+6+35+16+32=126; Step S2-2: Map the feature vector S calculated in step S2-1 to the fused feature S through a linear layer. This allows for the fusion of different features with different weights; where feature S has a dimension of 126, the linear layer has a size of 126×32, and the fused feature S... The dimension is 32; Step S2-3: Repeat steps S2-1 and S2-2 until the target frame image set is reached. The fusion features of the L frames in the image have been calculated individually; the fusion features of the L frames are then concatenated to form the fusion feature matrix M of the target frame image. ={S j }, where S j Let G be the target frame image numbered j. ij The fusion features, j∈[0, L), M The dimensions are L×32.

[0034] (3) Step S3, Temporal Feature Integration: The fusion feature matrix of the target frame image is generated into multiple hidden layer features through convolution operation. All hidden layer features are concatenated in sequence to form the temporal feature matrix of the frame image. The temporal feature matrix of the frame image is encoded with a time dimension. The fusion feature matrix of the video is generated by multi-head self-attention module. The fusion feature matrix of the video is mapped through a linear layer to obtain a preliminary prediction matrix. The preliminary prediction matrix is ​​input into the activation function to obtain the prediction probability vector of the target video. The specific steps include: Step S3-1: Let T be the number of hidden layers in the temporal convolutional network, and T=L. Then, take the fusion feature matrix M of the frame image calculated in step S2-3. Input is fed into a temporal convolutional network with T hidden layers, and for the input M Perform 1D convolution operations to generate the first hidden layer feature h1 and for h1 Continue with 1D convolution operations to generate the second hidden layer feature h2. Repeat this 1D convolution operation for each newly generated hidden layer feature until the T-th hidden layer feature h is obtained. T h1 h2 , ..., h T The temporal feature matrix H of the frame images is formed by sequentially splicing them together. ; In step S3-1, the temporal feature matrix H of the frame image is obtained. The process specifically includes: Let the kernel size of the first 1D convolution operation be 7, and the initial dilation value be 2; let the kernel size of the second 1D convolution operation be 7, and the initial dilation value be 4; and so on, with the kernel size of the T-th 1D convolution operation being 7, and the initial dilation value being 2. T The fusion feature matrix M of the frame images The size is transposed from L×32 to 32×L, and then the first 1D convolution operation is performed to generate the first hidden layer feature h1 with a hidden layer feature dimension of 128. The first hidden layer feature h1 Perform a second 1D convolution operation to generate a second hidden layer feature h2 with a hidden layer feature dimension of 128. And so on, the Tth hidden layer feature h with dimension 128 is calculated. T h1 h2 , ..., h T The temporal feature matrix H of the frame images with a size of L×128 is formed by sequentially stitching them together. .

[0035] Starting from the second 1D convolution operation, the dilation value is doubled compared to the previous one, and the dimension of the hidden layer features is 128. After feature concatenation, the temporal feature matrix H of the frame image... The dimensions are L×128.

[0036] Step S3-2: Calculate the temporal feature matrix H of the frame image obtained in step S3-1. With temporal encoding added, the fusion feature matrix of the video is calculated using a multi-head self-attention module. ,in The dimensions are L×d v d v For 64, 128, or 256; then for By flattening, its dimension is changed to L•d. v ; The fusion feature matrix of the generated video in step S3-2 The process specifically includes: Set an initial value of 0 for a learnable coding matrix of size L×128, and then combine the learnable coding matrix with the temporal feature matrix H of the frame image. The result obtained by adding elements together is denoted as P. , where P The dimensions are L×128; P Input a multi-head self-attention module to extract features and obtain the fused feature matrix of the video. , , Let Q, K, and V be the activation function, where Q, K, and V are three learnable matrices in the multi-head self-attention module. , , ,in , , It is a fully connected layer. The dimensions are all 128×d k , The dimensions are 128×d v fusion feature matrix of video The dimensions are L×d v .

[0037] Step S3-3: Calculate the fusion feature matrix of the video obtained in step S3-2. A preliminary prediction matrix is ​​obtained through linear layer mapping. The size of the linear layer is L•d v ×t, the preliminary prediction matrix The dimension is t. Let be the prediction matrix for the target video corresponding to the a-th category; Step S3-4: Use the Softmax activation function to transform the preliminary prediction matrix obtained in step S3-3. Transform into a prediction probability vector of the target video with dimension t. ,in .

[0038] (4) Step S4, Model Construction: Define a loss function, input the predicted probability vector of the target video in step S3 and the expected probability vector of the target video in step S1 into the loss function, calculate the loss value through the loss function to train and iterate the parameters, stop training when the number of iterations reaches the preset maximum number of iterations, take the parameters obtained when the loss value obtained during training is the minimum value as the parameters after training, optimize the parameters in multi-modal feature extraction, multi-modal feature fusion and temporal feature integration based on the parameters after training, and construct the attention evaluation model based on the optimized multi-modal feature extraction, multi-modal feature fusion and temporal feature integration. Step S4 specifically includes: Define the loss function FocalLoss, which is the predicted probability vector of the target video in step S3-4. and the expected probability vector of the target video in step S1-1 The formula is as follows: ,in The loss value is calculated using a loss function to train and iterate the parameters. Training stops when the number of iterations reaches the preset maximum number of iterations, such as setting the maximum number of iterations to 50. The parameters obtained when the loss value is minimized during training are used as the parameters after training. The parameters in multi-modal feature extraction, multi-modal feature fusion, and temporal feature integration are optimized using the parameters after training. Based on the optimized multi-modal feature extraction, multi-modal feature fusion, and temporal feature integration using the parameters after training, an attention evaluation model is constructed.

[0039] Step S5, Attention Assessment: Input the video to be assessed into the attention assessment model to obtain the predicted probability vector of the target video to be assessed. Select the attention label corresponding to the largest probability value in the predicted probability vector of the target video to be assessed as the attention assessment result of the learner of the video to be assessed.

[0040] Example 2: A comparison of the accuracy of the assessment method of this invention with previous attention assessment methods. The evaluation method of this invention achieves a significant improvement in accuracy compared to previous attention evaluation methods (end-to-end and feature fusion-based methods). Several evaluation methods with good accuracy were selected for comparison on the publicly available attention dataset DAiSEE: I3D, C3D+LSTM, LRCN, DFSTN, C3D+TCN, DRN, ResNet+LSTM, DenseAttNet, and ResNet+TCN. The accuracy comparison is shown in Table 1.

[0041] Table 1. Accuracy Comparison of the Evaluation Method of this Application with Previous Methods

[0042] Results Analysis: As shown in Table 1, the multi-modal feature fusion-based model proposed in this application achieves an accuracy of 64.08% on the four-class classification task and 95.62% on the two-class classification task. It shows a significant improvement in accuracy on both four-class and two-class classification tasks, and its accuracy on the four-class classification task is higher than that of previous evaluation methods.

[0043] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Various modifications and improvements made by those skilled in the art to the technical solutions of the invention without departing from the spirit of the invention should fall within the protection scope of the present invention. The scope of the present invention is defined by the claims.

Claims

1. A method for evaluating online learning focus based on multimodal feature fusion, characterized in that... Includes the following steps: Step S1, Multimodal Feature Extraction: Obtain the training set of target videos. Each target video corresponds to an attention label, which defines the expected probability vector of the target video. Sample each target video in the training set to obtain target frame images. Extract multimodal features from each target frame image. The multimodal features include global image features, eyelid opening features, head orientation features, gaze direction features, facial AU unit features, pose keypoint group features, and hand keypoint group features, specifically including: Step S1-1: Obtain the contents A training set C of target videos, where each target video corresponds to a focus label m, is defined, and the expected probability vector for each target video is defined. , ,when hour, =1, otherwise =0; Uniformly sample each target video in C to obtain multiple target frame images, and these multiple target frame images constitute a target frame image set. ; Step S1-2: Let Indicates that the number in training set C is The target video, for Perform uniform sampling to obtain L frames of images and use them as target frame image set , ∈[0, N); Let the number be Target video target frame image set The target frame image numbered j is G. ij , where j∈[0, L), for the target frame image G ij Extract multimodal features one by one, and repeat this operation until the target frame image set has been processed. ,in ; Step S2, Multimodal Feature Fusion: The extracted multimodal features of the target frame image are fused with different weights to obtain the fused features corresponding to the target frame image. The fused features corresponding to each target frame image are then concatenated to form the fused feature matrix of the target frame image. Specifically, this includes: Step S2-1: Transfer the target frame image G obtained in step S1-2 to... ij The multimodal features are concatenated to form a feature vector S with a size of 32+2+3+6+35+16+32=126; Step S2-2: Map the feature vector S calculated in step S2-1 to the fused feature S through a linear layer. This allows for the fusion of different features with different weights; where feature S has a dimension of 126, the linear layer has a size of 126×32, and the fused feature S... The dimension is 32; Step S2-3: Repeat steps S2-1 and S2-2 until the target frame image set is reached. The fusion features of the L frames in the image have been calculated individually; the fusion features of the L frames are then concatenated to form the fusion feature matrix M of the target frame image. ={S j }, where S j Let G be the target frame image numbered j. ij The fusion features, j∈[0, L), M The dimensions are L×32; Step S3, Temporal Feature Integration: The fusion feature matrix of the target frame image is used to generate multiple hidden layer features through convolution. All hidden layer features are concatenated sequentially to form the temporal feature matrix of the frame image. The temporal feature matrix of the frame image is encoded with a time dimension, and the fusion feature matrix of the video is calculated using a multi-head self-attention module. The fusion feature matrix of the video is mapped through a linear layer to obtain a preliminary prediction matrix. The preliminary prediction matrix is ​​input into an activation function to obtain the prediction probability vector of the target video, specifically including: Step S3-1: Let T be the number of hidden layers in the temporal convolutional network, and T=L. Then, take the fusion feature matrix M of the frame image calculated in step S2-3. Input is fed into a temporal convolutional network with T hidden layers, and for the input M Perform 1D convolution operations to generate the first hidden layer feature h1 and for h1 Continue with 1D convolution operations to generate the second hidden layer feature h2. Repeat this 1D convolution operation for each newly generated hidden layer feature until the T-th hidden layer feature h is obtained. T h1 h2 , ..., h T The temporal feature matrix H of the frame images is formed by sequentially splicing them together. Specifically: Let the kernel size of the first 1D convolution operation be 7, and the initial dilation value be 2; let the kernel size of the second 1D convolution operation be 7, and the initial dilation value be 4; and so on, with the kernel size of the T-th 1D convolution operation being 7, and the initial dilation value being 2. T The fusion feature matrix M of the frame images The size is transposed from L×32 to 32×L, and then the first 1D convolution operation is performed to generate the first hidden layer feature h1 with a hidden layer feature dimension of 128. The first hidden layer feature h1 Perform a second 1D convolution operation to generate a second hidden layer feature h2 with a hidden layer feature dimension of 128. And so on, the Tth hidden layer feature h with dimension 128 is calculated. T h1 h2 , ..., h T The temporal feature matrix H of the frame images with a size of L×128 is formed by sequentially stitching them together. ; Step S3-2: Calculate the temporal feature matrix H of the frame image obtained in step S3-1. With temporal encoding added, the fusion feature matrix of the video is calculated using a multi-head self-attention module. Specifically: Set an initial value of 0 for a learnable coding matrix of size L×128, and then combine the learnable coding matrix with the temporal feature matrix H of the frame image. The result obtained by adding elements together is denoted as P. , where P The dimensions are L×128; P Input a multi-head self-attention module to extract features and obtain the fused feature matrix of the video. , , Let Q, K, and V be the activation function, where Q, K, and V are three learnable matrices in the multi-head self-attention module. , , ,in , , It is a fully connected layer. The dimensions are all 128×d k , The dimensions are 128×d v fusion feature matrix of video The dimensions are L×d v ,in The dimensions are L×d v d v For 64, 128, or 256; then for By flattening, its dimension is changed to L•d. v ; Step S3-3: Calculate the fusion feature matrix of the video obtained in step S3-2. A preliminary prediction matrix is ​​obtained through linear layer mapping. The size of the linear layer is L•d v ×t, the preliminary prediction matrix The dimension is t. Let be the prediction matrix for the target video corresponding to the a-th category; Step S3-4: Use the Softmax activation function to transform the preliminary prediction matrix obtained in step S3-3. Transform into a prediction probability vector of the target video with dimension t. ,in ; Step S4, Model Construction: Define a loss function. Input the predicted probability vector of the target video in Step S3 and the expected probability vector of the target video in Step S1 into the loss function. Calculate the loss value through the loss function to train and iterate the parameters. Stop training when the number of iterations reaches the preset maximum number of iterations. Use the parameters obtained when the loss value is minimized during training as the parameters after training. Optimize the parameters in multi-modal feature extraction, multi-modal feature fusion, and temporal feature integration using the trained parameters. Construct an attention evaluation model based on the optimized multi-modal feature extraction, multi-modal feature fusion, and temporal feature integration using the trained parameters. Step S5, Attention Assessment: Input the video to be assessed into the attention assessment model to obtain the predicted probability vector of the target video to be assessed. Select the attention label corresponding to the largest probability value in the predicted probability vector of the target video to be assessed as the attention assessment result of the learner of the video to be assessed.

2. The method for evaluating online learning focus based on multimodal feature fusion according to claim 1, characterized in that... Step S4 specifically includes: Define the loss function FocalLoss, which is the predicted probability vector of the target video in step S3-4. and the expected probability vector of the target video in step S1-1 The formula is as follows: ,in .

Images