Classroom learning investment automatic evaluation and evolution analysis system and method based on video

Abstract

The invention discloses a video-based classroom learning investment automatic evaluation and evolution analysis system and method, and relates to the technical field of image recognition and image classification, and the system comprises a video data collection module which is used for obtaining original classroom video data; the education domain knowledge definition module is used for dividing a behavior input level and an emotion input level; the data set construction module is used for constructing a behavior data set and collecting an emotion data set; the AI model module is used for target detection, behavior recognition and expression recognition; the classroom learning input identification module is used for identifying behavior input and emotion input levels; and the classroom learning input analysis and visual feedback module is used for integrating the recognition result to obtain the evolution process of learning input. Therefore, by the adoption of the method, the learning process can be evaluated more comprehensively, in real time and accurately, classroom learning input evolution trend analysis and visualization are achieved, and a reference is provided for adjusting a teaching strategy.

Description

Technical Field

[0001] The present invention relates to the technical field of image recognition and image classification, and in particular to a video-based classroom learning input automatic evaluation and evolution analysis system and method. Background Art

[0002] Classroom learning engagement is an important indicator to measure students' enthusiasm, initiative and participation in the learning process, and plays a vital role in improving teaching quality and students' learning outcomes. Accurately measuring and analyzing the level of classroom learning engagement is one of the breakthroughs in solving problems such as low student retention rate and high dropout rate. Traditional learning engagement assessment methods mainly rely on teachers' manual observations or students' self-evaluation reports, which often lack objectivity and are difficult to fully and accurately reflect students' learning status. They also lack the basis for large-scale automated assessments, resulting in high analysis costs and a single form of result presentation.

[0003] With the rapid development of computer vision and deep learning technology, learning engagement assessment methods based on video analysis have received widespread attention. The construction of smart classrooms has facilitated the acquisition of video data containing non-verbal behaviors such as student expressions and gestures, further promoting smarter and more efficient classroom learning engagement measurement. Due to the advantages of unbiased, low-cost and low learner perception, this method has been adopted by more and more researchers.

[0004] However, existing studies on evaluating classroom learning engagement based on video data still have some limitations: (1) Existing studies regard learning engagement as a holistic concept, or only focus on one dimension of learning engagement (most studies focus on achieving student behavior recognition), and do not comprehensively analyze the different dimensions of learning engagement; (2) There is no unified definition of learning engagement, and there is a lack of a classroom learning engagement evaluation indicator system based on educational theory; (3) There is a lack of public datasets for student target detection and behavior recognition in classroom scenarios. The behavior recognition datasets constructed in existing studies have problems such as small data volume and single scenario, which leads to insufficient adaptability to complex classroom environments and makes it difficult to comprehensively and accurately evaluate classroom learning engagement; (4) Existing systems focus on achieving automatic recognition of classroom learning engagement, and lack long-term tracking and evolution analysis of student learning engagement at the individual and group levels.

[0005] Therefore, there is an urgent need for a video-based automatic evaluation and evolution analysis method and system for classroom learning engagement, so as to achieve a comprehensive, real-time and accurate evaluation of the classroom learning process, analyze the evolution trend of students' learning engagement, and be able to feed back the analysis results to teachers in a visual form, so as to help teachers understand students' learning status, and then adjust teaching strategies in time according to actual conditions and provide personalized learning support services. Summary of the invention

[0006] The purpose of the present invention is to provide a video-based classroom learning engagement automatic evaluation and evolution analysis system and method, which can accurately identify student behavior and realize the evolution trend analysis of student learning engagement.

[0007] To achieve the above objectives, the present invention provides a video-based classroom learning input automatic evaluation and evolution analysis system, comprising:

[0008] Video data acquisition module, used to collect video information of learning activities in classroom scenes;

[0009] The education field knowledge definition module is used to define the classroom learning input evaluation indicator system;

[0010] The dataset construction module is used to construct behavioral datasets and collect emotional datasets;

[0011] AI model modules, including student target detection model, student behavior recognition model and expression recognition model;

[0012] Classroom learning engagement identification module, used to identify behavioral engagement and emotional engagement levels;

[0013] The classroom learning engagement analysis and visual feedback module is used to analyze the evolution of learning engagement at the individual and group levels.

[0014] The video-based automatic evaluation and evolution analysis method of classroom learning engagement includes the following steps:

[0015] S1, through the video data acquisition module, using video crawling and video recording to obtain original video data;

[0016] S2, collect the emotion dataset through the dataset construction module, and process the original video data to obtain the behavior dataset;

[0017] S3. Train the AI ​​model module based on the emotion data set and behavior data set;

[0018] S4. Through the classroom learning input recognition module, collect classroom videos of students during class, and perform frame processing to obtain image sequences. Use the trained AI model module to detect targets in the image sequences and identify students' behaviors and emotional states.

[0019] S5. Based on the knowledge definition modules in the education field, the identified student behaviors are divided to obtain the behavioral and emotional engagement levels of the students;

[0020] S6. Use the classroom learning engagement analysis and visual feedback module to quantify students’ behavioral engagement and emotional engagement levels and obtain the evolution of students’ or classes’ learning engagement.

[0021] Preferably, in step S2, obtaining the behavior data set includes:

[0022] Preprocessing the original video data mainly includes video frame extraction and frame reduction to obtain student images from square perspective, oblique upper perspective and student close-up perspective;

[0023] According to the student behavior classification indicators in the education field knowledge definition module, the student images are labeled and a label file is generated;

[0024] Perform data augmentation on student images to expand the original dataset.

[0025] Preferably, in step S4, the trained AI model module identifies student behavioral engagement and emotional engagement including:

[0026] The SSD pre-trained model in the OpenCV library and the Yolov5 model trained by the AI ​​model module are used to detect objects in the image sequence to obtain the position information of the students' faces and bodies;

[0027] The DeepSORT model is used to track students at specific locations, extract continuous video frames, and obtain image sequences of students’ faces and body regions;

[0028] Based on the student face and body region image sequences, the Resnet50 model and VGGNet16 model are used to obtain the student expression sequence and learning behavior sequence respectively;

[0029] Based on the classroom learning engagement indicator system determined in the education field knowledge definition module, a comprehensive quantitative evaluation is conducted on the recognized expression sequences and behavior sequences, that is, the emotional engagement and behavioral engagement status of each learner in the classroom is obtained.

[0030] Preferably, in step S5, the definitions of the student's behavioral engagement and emotional engagement levels are as follows:

[0031] According to the students' movements and behavioral purposes, we identified 9 typical classroom learning behaviors, including lowering heads to write, lowering heads to read, raising heads to listen to lectures, turning heads to look at others, raising hands to answer questions, standing to answer, leaning sideways to communicate, sleeping on the desk, and others. Based on the ICAP framework, we divided classroom learning behaviors into four different levels of behavioral engagement: passive learning engagement, active learning engagement, interactive learning engagement, and non-learning engagement.

[0032] Regarding the emotion categories, the seven basic emotions are divided into three levels: positive, negative, and neutral according to different emotional valences, representing different levels of emotional investment.

[0033] Therefore, the present invention adopts the above-mentioned video-based classroom learning investment automatic evaluation and evolution analysis system and method, which has the following technical effects:

[0034] (1) Analyzing students’ behavioral and emotional involvement can provide a more comprehensive, real-time, and accurate assessment of the learning process;

[0035] (2) Analyzing and visualizing the evolution trend of students’ classroom learning input helps teachers understand students’ learning status and provide reference for adjusting teaching strategies;

[0036] (3) Construct a behavior dataset to solve the problems of small data volume and single scenario in behavior recognition datasets. Use this dataset for model training to increase the stability and generalization ability of the model.

[0037] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 It is the overall framework diagram of the video-based classroom learning input automatic evaluation and evolution analysis system;

[0039] Figure 2 It is a video acquisition equipment installation scheme in an embodiment of a video-based classroom learning input automatic evaluation and evolution analysis system and method;

[0040] Figure 3 These are five perspectives of video collected in the classroom in the embodiment of the video-based classroom learning investment automatic evaluation and evolution analysis system and method, among which: Figure 3 (a) to (e) are the front view, the upper oblique view, the rear view, the teacher's view, and the student's close-up view, respectively;

[0041] Figure 4 This is an image example of data enhancement in an embodiment of a video-based classroom learning engagement automatic evaluation and evolution analysis system and method, wherein: Figure 3 (a) to (g) are original images, flipped, rotated, brightened, Gaussian noise added, salt and pepper noise added, and blurred image samples respectively;

[0042] Figure 5 It is a diagram showing the effect of applying the target detection model in the video-based classroom learning investment automatic evaluation and evolution analysis system and method embodiment to the classroom video;

[0043] Figure 6 It is a structural diagram of a Resnet5 model in an embodiment of a video-based classroom learning engagement automatic evaluation and evolution analysis system and method;

[0044] Figure 7It is a visualization diagram of a behavior recognition confusion matrix in an embodiment of a video-based classroom learning engagement automatic evaluation and evolution analysis system and method;

[0045] Figure 8 It is a flow chart of classroom learning engagement identification in an embodiment of a video-based classroom learning engagement automatic evaluation and evolution analysis system and method;

[0046] Fig. 9 It is a visual diagram of classroom learning engagement status in an embodiment of a video-based classroom learning engagement automatic evaluation and evolution analysis system and method, wherein: Fig. 9 (a) is student behavior identification. Fig. 9 (b) is the expression recognition result. Fig. 9 (c) is the evolution process of behavioral input level. Fig. 9 (d) The evolution of emotional investment level. DETAILED DESCRIPTION

[0047] The present invention can be explained in more detail by the following examples. The purpose of disclosing the present invention is to protect all changes and improvements within the scope of the present invention. The present invention is not limited to the following examples.

[0048] like Figure 1 As shown, the video-based classroom learning investment automatic evaluation and evolution analysis system provided by the present invention includes a video data acquisition module, an education field knowledge definition module, a data set construction module, an AI model training module, a classroom learning investment recognition module, and a classroom learning investment analysis and visualization feedback module. The video data acquisition module is used to record video information of learners performing learning activities in different classroom scenes; the education field knowledge definition module is adjacent to the other four feedback modules, and defines the classroom learning investment evaluation index system according to the educational theory; the data set construction module annotates the image data set by determining the student behavior type and the corresponding definition, and collects the emotional data set that can be used for emotion recognition at the same time, so the final data set contains two sub-data sets for behavior recognition and emotion recognition; the AI ​​model training module is mainly used to train a series of AI models required for classroom learning investment evaluation, including a student target detection model, a student behavior recognition model, and an expression recognition model; the classroom learning investment recognition module uses the trained AI model to first perform target detection, and then send the detection area to the behavior and expression recognition model to obtain the recognition results of the behavior and emotional investment level; the classroom learning investment analysis and visualization feedback module further aggregates and analyzes the recognition results to obtain the learning investment evolution process at the individual level and the group level.

[0049] The method of using the classroom learning engagement automatic evaluation and evolution analysis system in the above video to identify student behavior and analyze the evolution of learning engagement is as follows:

[0050] S1. In order to obtain as many classroom videos from different scenes as possible, the video data acquisition module of this embodiment obtains the original video data, which mainly includes two parts: one is web crawling, and the other is recording and saving from the real classroom, as follows:

[0051] (1) Video crawling

[0052] In this embodiment, the ministerial classroom videos were obtained from the National Resource Public Service Platform (website: http: / / www.eduyun.cn) through crawler tools. Through the observation of classroom videos, it was found that some videos had problems such as abnormal student seat layout (sitting in a circle, sitting opposite each other), abnormal shooting angle (too high, too low), abnormal desks (simple desks, no desks), blurred images, and dark video images. Therefore, after obtaining the original video, data screening is required, and finally 230 classroom videos were obtained.

[0053] (2) Video Recording

[0054] In addition to crawling videos, this embodiment recorded some classroom videos from primary and secondary schools to further expand the amount of original video data. In order to clearly capture the appearance, expressions, behaviors and other information of the student target, it is necessary to install two cameras facing the students in the front of the classroom to capture the student behavior from different angles, such as Figure 2 As shown. In addition, the configuration of the video information acquisition equipment needs to meet the following requirements: (a) support video data and image data output, the image sensor effective pixels are higher than 1920×1080; (b) the horizontal field of view of the equipment needs to fully cover the students in the classroom; (c) the equipment has automatic white balance and brightness adjustment functions; (d) the equipment frame rate is not less than 1080P@25 frames / second; (e) the output image is clear and stable, the color reproduction is good, and there is no obvious color cast, noise and other abnormal phenomena that interfere with the algorithm judgment. Finally, a total of 30 classroom videos were obtained, and each class contained two videos from the left and right perspectives.

[0055] S2. In order to effectively identify students' facial expressions and behaviors, it is first necessary to build a dataset that adapts to different task requirements. For the expression recognition task, this embodiment integrates two open source datasets to form a facial emotion dataset with a larger data volume. As for the student behavior recognition task, given the current lack of public datasets, this embodiment collects videos of students in different classroom situations and develops a dataset specifically for student behavior recognition research, as follows:

[0056] (1) Sentiment Dataset

[0057] The two public datasets used in this embodiment are Facial Expression Recognition 2013 (FER2013) and The Extended Cohn-Kanade Dataset (CK+). The FER2013 dataset consists of 35,887 facial images, all of which are grayscale images of size 48*48. The CK+ dataset contains 327 expression picture sequences of 123 objects. The last three frames of each picture sequence are extracted and saved, and uniformly processed into 48*48 grayscale images, and then these images are added to FER2013 to increase the data volume. According to statistics, the merged new dataset has a total of 36,868 facial expression images.

[0058] (2) Behavioral Dataset

[0059] (a) Data preprocessing

[0060] In order to obtain an image sequence that can be used for annotation, the original video data needs to be preprocessed. The data preprocessing stage mainly includes two steps: video frame extraction and frame reduction. First, use Python to extract a frame every 30 seconds from the collected classroom video and save it locally. Because the staff will switch the camera during the recording process for the sake of viewing experience, the classroom images obtained from some videos (mainly videos crawled from the Internet) contain different shooting perspectives, including square perspective, oblique upper perspective, rear perspective, teacher perspective, and student close-up perspective (such as Figure 3 As shown in the figure). This embodiment mainly focuses on student behavior, so after the frame extraction is completed, the image frames from the rear view and the teacher's view need to be deleted, and only the student images from the square view, the upper oblique view, and the student close-up view are retained. After the preprocessing is completed, a total of 7496 classroom images for annotation are finally obtained.

[0061] (b) Image annotation

[0062] Data labeling refers to the manual classification of pre-processed images, which is an indispensable part of image classification research. First, 8 undergraduate and graduate students from the Department of Education were recruited. According to the specific description of the student behavior classification indicators in the knowledge definition module of the education field, they were provided with an instruction document for image labeling. They were required to use the image labeling tool LabelImg to frame the position of the students in each picture and select appropriate labels. During the labeling process, the entire image dataset directory was imported into the LabelImg labeling tool, and then the images in the directory were labeled in turn according to the image name. After each image is labeled, a txt format file is automatically generated and saved. The label file consists of the coordinate information of the label box and the corresponding behavior category label.

[0063] After all images are annotated, the label files need to be further processed in order to obtain a dataset that can be directly used to train the student target detection model and the learning behavior recognition model. For the target detection model, since its purpose is to detect all students in the classroom video frame, the 9 learning behavior categories in each label file need to be uniformly modified to the "student" category. The target detection dataset finally constructed contains 7496 classroom images and corresponding labels. For the behavior recognition model, the student targets in the classroom images are batch-cropped according to the position information in the txt label file, and saved in different folders according to the behavior category, thereby obtaining a behavior dataset containing 152,823 student images.

[0064] (c) Dataset enhancement

[0065] In the behavior dataset constructed by the above steps, the number of images corresponding to different student behavior categories varies greatly. For example, there are 51,681 images in the "raising head to listen to class" subset, while there are only 984 images in the "sleeping on the desk" category. If an unbalanced dataset is used directly to train the model, the overall accuracy of the model may be good, but for categories with few samples, the accuracy will be very poor because they are ignored during the training process. Therefore, this embodiment adopts some data enhancement methods, including flipping, rotating, brightening, adding noise and blurring (such as Figure 4 As shown in the figure, data augmentation was performed on subsets with significantly fewer images to greatly expand the original dataset and balance the number of images of different behavior types. After data augmentation, the dataset used to train the behavior recognition model contained 312,459 student images. Among them, 41,155 images were of students “bowed their heads to write”, 43,805 images were of students “bowed their heads to read”, 51,681 images were of students “looking up to listen to the class”, 42,896 images were of students “turning their heads to look at others”, 42,617 images were of students “raising their hands to answer questions”, 29,718 images were of students “standing to answer questions”, 41,390 images were of students “leaning sideways to communicate”, 8,856 images were of students “lying over the table to sleep”, and 10,341 images were of students “other”.

[0066] S3. Based on the emotion data set and behavior data set, the AI ​​model module is trained as follows:

[0067] (1) Student object detection model training

[0068] To achieve the recognition and analysis of student behavior, detecting student targets is a crucial step. Target detection has always been a challenging task. To solve this problem, researchers have proposed many methods. Traditional target detection algorithms based on manual features usually use a sliding window strategy to traverse the detection image and select the region of interest. This method often has problems such as unreasonable feature design and incomplete extraction, resulting in low accuracy of the target detection algorithm. At the same time, it is accompanied by a large amount of calculation and is not suitable for real classroom learning scenarios.

[0069] In the era of deep learning, some scholars have proposed the You only look once (Yolo) target detection algorithm, which is a typical single-stage target detection method that simplifies the cumbersome training process and transforms the target detection task into a regression task of identifying the target subject bounding box category and position information. Since its proposal, the algorithm has evolved into multiple versions. Among them, Yolov5 adopts a lightweight network structure and achieves a balance between speed and accuracy by adjusting the depth and width of the network. Therefore, this embodiment adopts the Yolov5 method to train the target detection model based on the student coordinate labels in the self-built data set, and the overall accuracy of the model exceeds 95%. Figure 5 The effect of applying the target detection model to classroom videos is shown. It can be seen that the model can accurately detect student targets in classroom images under different angles, different backgrounds, and different lighting conditions. This work provides a basis for behavior recognition and analysis based on student targets.

[0070] (2) Facial expression recognition model training

[0071] The convolutional neural network was proposed under the inspiration of the biological receptive field mechanism, that is, a neuron only receives signals within the stimulus area it controls. It has been widely used due to its powerful self-learning ability. Generally speaking, as the depth of the network increases, the extracted features are richer and the feature representation ability is stronger. But in fact, because deeper network models are accompanied by the gradient vanishing / explosion problem, when the number of network layers increases to a certain extent, the accuracy of the model will be greatly reduced, that is, the model performance is degraded. In order to solve this problem, He Kaiming and others proposed a deep neural network model-Residual Neural Network (ResNet), which innovatively introduced a residual learning mechanism and jump connection design, so that the stacking of convolutional layers does not affect the performance of the network model. ResNet has multiple versions. The basic structure of these networks is the same, and the main difference lies in the number of layers and the type of residual blocks. In this embodiment, ResNet50 is selected to extract facial features and realize expression classification. The network structure of Resnet50 is as follows Figure 6 shown.

[0072] Before model training, 80% of the data set was randomly selected as training samples, 10% as validation set, and 10% as test set. The learning rate in the algorithm was set to 0.01, the "Batch Size" and "Epoch" parameters were set to 32 and 200 respectively, and the loss function used the cross entropy loss function. After training, the model with better effect on the validation set was saved as an hdf5 file, and then further evaluated on the test set. The evaluation indicators involve calculating macro precision (macro-P), macro recall (macro-R), macro F1 value (macro-F1) and accuracy. The specific calculation formula is as follows:

[0073]

[0074] Where n is the number of categories, m is the number of samples; TP i , FP i and FN i Respectively represent the number of true positives, false positives and false negatives of the i-th category; f(x i ) is the prediction result, y i is the true label. The average recognition accuracy of the training model reached 0.74, and the calculated indicators are shown in Table 1. It can be seen that when recognizing the emotion "happy", the model performed best, with its precision, recall, F1 value and accuracy reaching 0.92, 0.95, 0.93 and 0.93 respectively; when the model recognized the three emotions of "sadness", "fear" and "anger", it had a lower accuracy, which may be because the labels of the dataset were manually annotated through crowdsourcing, which is subjective, and some emotion types are easily confused, resulting in image label errors, and the corresponding accuracy cannot be very high.

[0075] Table 1 Model performance evaluation results

[0076] Emotional Type macro-P macro-R macro-F1 Accuracy Happy 0.92 0.95 0.93 0.93 surprise 0.84 0.82 0.83 0.82 sad 0.62 0.65 0.64 0.65 Fear 0.67 0.62 0.64 0.63 disgust 0.90 0.74 0.81 0.74 anger 0.60 0.67 0.63 0.65 neutral 0.82 0.81 0.81 0.79

[0077] (3) Student behavior recognition model training

[0078] VGGNet16 is a classic deep learning network model with the advantages of simple structure, easy expansion, stable performance, etc. The present invention selects VGGNet16 as the basic model and realizes student behavior recognition through transfer learning, so that the feature extraction ability learned on large-scale data sets can be utilized.

[0079] The network structure of VGGNet16 consists of 13 convolutional layers, 5 pooling layers, and 3 fully connected layers. Its input accepts images of size 3x224x224, so the student images need to be normalized first to meet the requirements. Then freeze the first few layers of VGGNet16, and only fine-tune and train the high layers. Through the convolutional layers and pooling layers, effective feature maps are extracted from the preprocessed student images, which can capture the subtle differences in student behavior. In addition, fully connected layers and softmax layers are added to the back end of VGGNet16 for final behavior classification. The fully connected layer maps high-dimensional features to low-dimensional space, and the softmax layer outputs the probability distribution of each behavior category.

[0080] Before model training, 80% of the behavior data set is randomly selected as training samples, and the rest is used as the test set. There is no overlap between the training set and the test set. When training the model, the Adam optimizer is used to optimize the network parameters. Finally, the trained model is evaluated on the test set, and the confusion matrix visualization result is as follows: Figure 7 As shown in the figure, the horizontal axis represents the predicted label, that is, the model prediction result, and the vertical axis represents the actual label. The values ​​on the diagonal of the matrix represent the proportion of correct predictions, and the non-diagonal elements represent the proportion of incorrect predictions. Figure 7 As can be seen from the figure, due to the similarity in behavior performance between "lowering head to read" and "lowering head to write", there is a certain probability of misprediction between these two behaviors, while the accuracy of the other types of behaviors has reached above 0.90, especially the recognition accuracy of "sleeping on the table", "raising hand to answer questions" and "standing to answer" has reached 0.99, which is mainly because these behaviors have significant characteristics that are different from other behaviors. Overall, the accuracy of the behavior recognition model has reached 0.94.

[0081] S4, such as Figure 8 As shown in the figure, firstly, classroom videos of students in class are collected through cameras, and the acquired videos are frame-processed to obtain image sequences; then, the SSD pre-trained model in the OpenCV library and the Yolov5 model obtained in the AI ​​model training module are used to locate the student faces and body positions in the image, and the DeepSORT model is used to track students at specific positions and extract them separately as continuous video frames; subsequently, the detected student face and body area image sequences are sent to two typical deep learning network models, Resnet50 and VGGNet16, respectively, so as to realize the recognition of each student's expression and learning behavior.

[0082] S5, Fredricks et al. believe that classroom learning engagement includes three dimensions: cognitive engagement, behavioral engagement, and emotional engagement. This is a theoretical model generally accepted by researchers. Among them, cognitive engagement is relatively implicit and difficult to observe directly. In comparison, behavioral engagement and emotional engagement can be directly inferred through explicit behaviors such as facial expressions and body postures. Therefore, the present invention focuses on the recognition and analysis of the two dimensions of emotional engagement and behavioral engagement through non-verbal clues.

[0083] With respect to the emotional dimension, the present invention realizes the recognition of seven basic emotions based on a public data set, namely, happy, surprised, sad, fear, disgust, anger and neutral.

[0084] Regarding the behavioral dimension, this embodiment proposed a classification index of classroom learning behaviors based on the idea of ​​"student actions + behavioral purposes" (see Table 2), and finally determined nine typical classroom learning behaviors, including lowering the head to write, lowering the head to read, raising the head to listen to the class, turning the head to look at others, raising the hand to answer questions, standing to answer, communicating sideways, sleeping on the table, and others.

[0085] Table 2. Classification indicators of student behavior

[0086]

[0087] In order to facilitate further analysis, different behavior and emotion categories need to be fitted to characterize different levels of input. Among them, the classroom learning input indicator system is shown in Table 3.

[0088] Table 3 Classroom learning input indicator system

[0089]

[0090]

[0091] Regarding the emotional categories, the seven basic emotions can be divided into three levels: positive, negative, and neutral according to different emotional valences, representing different levels of emotional investment.

[0092] Regarding behavior categories, this embodiment divides students' behavioral performance in the classroom into different levels of engagement based on the "active learning theory" proposed by Professor Michelene T. H. Chi, a famous American educational psychologist, namely the ICAP framework, including four levels: passive learning engagement, active learning engagement, interactive learning engagement, and non-learning engagement.

[0093] Among them, at the passive learning engagement level, students only store and memorize knowledge, and the corresponding behaviors are looking up to listen to lectures or looking down to read books; at the active learning engagement level, students can actively integrate and apply knowledge, and the corresponding behaviors are looking down to write or raising hands to answer questions; at the interactive learning engagement level, students can communicate with peers and teachers and collaborate to create together, and the corresponding behaviors are standing to answer or communicating sideways; at the non-learning engagement level, students are in a state of free learning, deviating from learning goals, and their behaviors may be turning their heads to look at others, sleeping on the table, or other non-engagement behaviors.

[0094] S6. Apply methods such as target detection and tracking, facial expression recognition, and student behavior recognition to the classroom learning engagement recognition and analysis system, and visualize the evolution of learning engagement at the individual and class levels. This helps teachers understand students' classroom performance and objectively evaluate classroom teaching effectiveness. At the same time, teachers can adjust teaching strategies in a timely manner based on the analysis results and provide timely support and feedback.

[0095] This embodiment intercepts a 20-minute classroom video for testing, as follows:

[0096] First, the video frames are extracted from the video at a certain frequency to form an image sequence.

[0097] Then, the facial expression recognition model and behavior recognition model are used to evaluate the behavior and expression of each student in each frame of the image, and two result matrices Behavior = {B ij :1≤i≤M;1≤j≤N} and Emotion={E ij :1≤i≤M;1≤j≤N}, where B ij and E ij They represent the behavior and emotional state of the jth student in the ith frame image respectively. The value range of i is 1 to M, M represents the total number of frames extracted from the classroom video, and the value range of j is 1 to N, N represents the number of students in the classroom video.

[0098] According to the classroom learning input indicator system determined by the knowledge definition module in the education field, the student behavior and expression recognition results are quantified.

[0099] Afterwards, for a single student, his recognition results S in all image sequences can be extracted beha =[B i :1≤i≤M] and S emo =[E i :1≤i≤M], the frequency of different input states of these two arrays is counted at certain time intervals to obtain the learning input evolution process of a single student.

[0100] Finally, the engagement levels of all students are averaged to obtain the overall class-level engagement process, e.g. Fig. 9 shown.

[0101] Therefore, the present invention adopts the above-mentioned video-based classroom learning engagement automatic evaluation and evolution analysis system and method, which can conduct a more comprehensive, real-time and accurate evaluation of the learning process, realize classroom learning engagement evolution trend analysis and visualization, and provide a reference for adjusting teaching strategies.

[0102] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solution of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solution to deviate from the spirit and scope of the technical solution of the present invention.

Claims

1. Video-based classroom learning input automatic evaluation and evolution analysis system, characterized by: include: Video data acquisition module, used to collect video information of learning activities in classroom scenes; The education field knowledge definition module is used to define the classroom learning input evaluation indicator system; The dataset construction module is used to construct behavioral datasets and collect emotional datasets; AI model modules, including student target detection model, student behavior recognition model and expression recognition model; Classroom learning engagement identification module, used to identify behavioral engagement and emotional engagement levels; The classroom learning engagement analysis and visual feedback module is used to analyze the evolution of learning engagement at the individual and group levels.

2. A video-based automatic evaluation and evolution analysis method for classroom learning engagement, characterized in that: The following steps are involved: S1, through the video data acquisition module, using video crawling and video recording to obtain original video data; S2, collect the emotion dataset through the dataset construction module, and process the original video data to obtain the behavior dataset; S3. Train the AI ​​model module based on the emotion data set and behavior data set; S4. Through the classroom learning input recognition module, collect classroom videos of students during class, and perform frame processing to obtain image sequences. Use the trained AI model module to detect targets in the image sequences and identify students' behaviors and emotional states. S5. Based on the knowledge definition module in the education field, the identified student behaviors and emotional states are divided to obtain the students’ behavioral engagement and emotional engagement levels; S6. Use the classroom learning engagement analysis and visual feedback module to quantify students’ behavioral engagement and emotional engagement levels and obtain the evolution of students’ or classes’ learning engagement.

3. The video-based automatic evaluation and evolution analysis method for classroom learning engagement according to claim 2 is characterized in that: In step S2, obtaining the behavior data set includes: Preprocessing the original video data mainly includes video frame extraction and frame reduction to obtain student images from square perspective, oblique upper perspective and student close-up perspective; According to the student behavior classification indicators in the education field knowledge definition module, the student images are labeled and a label file is generated; Perform data augmentation on student images to expand the original dataset.

4. The video-based automatic evaluation and evolution analysis method for classroom learning engagement according to claim 2 is characterized in that: In step S4, the trained AI model module identifies student behavioral input and emotional input including: The SSD pre-trained model in the OpenCV library and the Yolov5 model trained by the AI ​​model module are used to detect objects in the image sequence to obtain the position information of the students' faces and bodies; The DeepSORT model is used to track students at specific locations, extract continuous video frames, and obtain image sequences of students’ faces and body regions; Based on the student face and body region image sequences, the Resnet50 model and VGGNet16 model are used to obtain the student expression sequence and learning behavior sequence respectively; Based on the classroom learning engagement indicator system determined in the education field knowledge definition module, a comprehensive quantitative evaluation is conducted on the recognized expression sequences and behavior sequences, that is, the emotional engagement and behavioral engagement status of each learner in the classroom is obtained.

5. The video-based automatic evaluation and evolution analysis method for classroom learning engagement according to claim 2 is characterized in that: In step S5, the definition of students’ behavioral engagement and emotional engagement levels is as follows: According to the students' movements and behavioral purposes, we identified 9 typical classroom learning behaviors, including lowering heads to write, lowering heads to read, raising heads to listen to lectures, turning heads to look at others, raising hands to answer questions, standing to answer, leaning sideways to communicate, sleeping on the desk, and others. Based on the ICAP framework, we divided classroom learning behaviors into four different levels of behavioral engagement: passive learning engagement, active learning engagement, interactive learning engagement, and non-learning engagement. Regarding the emotion categories, the seven basic emotions are divided into three levels: positive, negative, and neutral according to different emotional valences, representing different levels of emotional investment.

Images