Student behavior identification method, system and device and medium

Abstract

The invention discloses a student behavior identification method, system and device and a medium, and relates to the field of video image processing, and the method comprises the steps: taking the classification correct rate of a prediction box as a threshold value, and carrying out the positive and negative sample judgment, so as to construct a classification loss function ClsSlideLoss used for judging the classification accuracy; using hole convolution in a spatial feature enhancement module SEAM so as to construct an LW-SEAM module with enhanced shielding detection capability; an LW-SEAM module is added to the neck of a YOLOv5 model, classification loss of the YOLOv5 model is replaced by a classification loss function ClsSlideLoss, confidence loss is replaced by a loss function SlideLoss, an optimized YOLOv5 model is constructed and trained, a YOLO-Act model is obtained, and a to-be-detected video data set is input into the YOLO-Act model to classify student behaviors. According to the invention, the behavior of each student can be identified and classified under the condition that the students in the classroom are shielded to a certain extent.

Description

Technical Field

[0001] The present invention relates to the field of video image processing, and in particular to a method, system, device and medium for identifying student behavior. Background Art

[0002] Classroom teaching is the main form of teaching activities and an important research object of educational research. Students' classroom behavior is the external manifestation of students' concentration and contains complex and valuable information. With the improvement of educational facilities, many campuses have built a regular classroom recording system, which can comprehensively record students' classroom behavior by installing cameras in the classroom. Real-time analysis and understanding of students' classroom behavior has a positive impact on teachers' timely adjustment of teaching strategies and thus improving teaching effectiveness.

[0003] There are three common methods for detecting students' classroom behaviors in the existing technology: Method 1 is student behavior recognition based on target detection method. Method 2 proposes student behavior recognition based on video behavior recognition methods such as 3DCNN. Method 3 is student behavior recognition based on human posture estimation method, which combines YOLOv8, multi-task cascaded convolutional network MTCNN (Multi-task Cascaded Convolutional Networks), Cov-PoolFER (Coverage Pooling for Fine-grained Facial Expression Recognition) and OpenPifPaf (OpenPose with Part-based Iterative Filtering and Association File Format) to jointly detect five behaviors of raising hands, standing, sleeping, yawning and smiling, and designs a personalized classroom observation automation system.

[0004] However, when students are occluded, the above technology cannot accurately identify the different behaviors of students. Summary of the invention

[0005] The embodiments of the present invention provide a method, system, device and medium for identifying student behavior, which can solve the problem in the prior art that it is difficult to accurately identify student behavior when the student is blocked.

[0006] The embodiment of the present invention provides a method for identifying student behavior, comprising the following steps: obtaining a video data set of student behavior in a classroom scene; constructing an optimized YOLOv5 model, wherein the optimized YOLOv5 model comprises: a backbone network, a neck and a head; adding a lightweight spatial feature enhancement module LW-SEAM to the neck of the YOLOv5 model, and replacing the classification loss of the YOLOv5 model with a classification loss function ClsSlideLoss, and replacing the confidence loss with a loss function SlideLoss; the LW-SEAM comprises: using a dilated convolution in the spatial feature enhancement module SEAM; the classification loss function ClsSlideLoss comprises: In the Loss loss function, the classification accuracy of the prediction box is used as the threshold for positive and negative sample judgment; the optimized YOLOv5 model is trained using the video dataset, and the prediction error is adjusted through the classification loss function ClsSlideLoss and the loss function SlideLoss to obtain the YOLO-Act model; the video dataset to be detected is input into the YOLO-Act model, and the student behavior characteristics are extracted through the backbone network; the student behavior characteristics are input into the neck, multi-scale features are obtained through LW-SEAM, and occlusion detection and feature fusion are performed on the multi-scale features through the neck to obtain the processed student behavior characteristics; the processed student behavior characteristics are input into the head to identify different student behaviors.

[0007] Furthermore, the classification loss function ClsSlideLoss specifically includes: in the classification loss function ClsSlideLoss, the classification accuracy of all prediction boxes is used as a threshold, negative samples are taken when the classification accuracy is less than the threshold, and positive samples are taken when the classification accuracy is greater than the threshold.

[0008] Furthermore, the optimized YOLOv5 model is trained using the video dataset, and the specific steps include: labeling the behavior of each student in the video dataset, wherein the student's behavior is divided into reading and writing behavior, mobile phone use behavior, head-down behavior, lying on the table behavior, head-up listening behavior, head-turning behavior and standing up to answer questions behavior; and training the optimized YOLOv5 model according to the labeled video dataset, so that the model can classify the behavior of each student.

[0009] Furthermore, the LW-SEAM module specifically includes: replacing the large convolution kernel of the block embedding module Patch-Embedding in SEAM with dilated convolutions with dilated rates of 1, 3 and 5 respectively, and the convolution kernel size is 3×3, so as to construct a LW-SEAM module capable of acquiring multi-scale features.

[0010] The embodiment of the present invention provides a student behavior recognition system, including:

[0011] The data acquisition module is used to obtain video datasets of student behaviors in classroom scenarios;

[0012] A model building module is used to build an optimized YOLOv5 model, wherein the optimized YOLOv5 model includes: a backbone network, a neck and a head; a lightweight spatial feature enhancement module LW-SEAM is added to the neck of the YOLOv5 model, and the classification loss of the YOLOv5 model is replaced by a classification loss function ClsSlideLoss, and the confidence loss is replaced by a loss function SlideLoss; the LW-SEAM includes: using a dilated convolution in the spatial feature enhancement module SEAM; the classification loss function ClsSlideLoss includes: in the SlideLoss loss function, the classification accuracy of the prediction box is used as a threshold for positive and negative sample judgment;

[0013] The model training module is used to train the optimized YOLOv5 model using the video dataset, adjust the prediction error through the classification loss function ClsSlideLoss and the loss function SlideLoss, and obtain the YOLO-Act model;

[0014] The behavior classification module is used to input the video dataset to be detected into the YOLO-Act model, extract the student behavior features through the backbone network; input the student behavior features into the neck, obtain multi-scale features through LW-SEAM, and perform occlusion detection and feature fusion on the multi-scale features through the neck to obtain the processed student behavior features; input the processed student behavior features into the head to identify different student behaviors.

[0015] An embodiment of the present invention provides a computer device, comprising: a memory and a processor; the memory stores a computer program, and the processor implements the above-mentioned method for identifying student behavior when executing the computer program.

[0016] An embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method for identifying student behavior described above is implemented.

[0017] The embodiments of the present invention provide a method, system, device and medium for identifying student behavior. Compared with the prior art, the beneficial effects thereof are as follows:

[0018] Construct an optimized YOLOv5 model, including: backbone network, neck and head; add a lightweight spatial feature enhancement module LW-SEAM to the neck of the YOLOv5 model, and replace the classification loss of the YOLOv5 model with the classification loss function ClsSlideLoss, and replace the confidence loss with the loss function SlideLoss. Use the video dataset to train the optimized YOLOv5 model. The replacement of the loss function can improve the training accuracy and obtain a YOLO-Act model with high prediction accuracy. When using the YOLO-Act model, input the video dataset to be detected into the YOLO-Act model, extract the student behavior features through the backbone network; input the student behavior features into the neck, obtain multi-scale features through LW-SEAM, and perform occlusion detection and feature fusion on the multi-scale features through the neck to obtain the processed student behavior features; input the processed student behavior features into the head to identify different student behaviors, and finally achieve the purpose of accurately identifying different student behaviors when the students are occluded. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 Sample images from the StuAct and SCB-DATASET3-S datasets provided for embodiments of the present invention, wherein (a) shows a sample image from the StuAct dataset, and (b) shows a sample image from the SCB-DATASET3-S dataset;

[0020] Figure 2 A YOLO-Act framework diagram provided by an embodiment of the present invention;

[0021] Figure 3 The Double SlideLoss function image provided by the embodiment of the present invention, wherein (a) shows the SlideLoss function image, and (b) shows the ClsSlideLoss function image;

[0022] Figure 4 A structural diagram of LW-SEAM provided by an embodiment of the present invention;

[0023] Figure 5 The sample size of the student classroom behavior provided by the embodiment of the present invention;

[0024] Figure 6 A diagram of the data processing process provided by an embodiment of the present invention;

[0025] Figure 7 mAP curve graph provided for an embodiment of the present invention, wherein (a) shows the mAP (mean average precision) value of different algorithms on the StuAct dataset, and (b) shows the AP (mean average precision) value of each behavior type;

[0026] Figure 8 Result diagrams of ACC and Loss in R3D and ViVit algorithms respectively provided in the embodiments of the present invention;

[0027] Fig. 9 Visualization of student behavior and classroom status portraits provided by embodiments of the present invention;

[0028] Fig.10 An overall proportion diagram of the portraits of each grade and school class provided by an embodiment of the present invention, wherein (a) shows the distribution of class characteristics of different grades, and (b) shows the distribution of class characteristics of each school;

[0029] Fig.11 A dynamic class portrait chart provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0030] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific embodiments of the present invention are described in detail below in conjunction with the accompanying drawings. In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without violating the connotation of the present invention, so the present invention is not limited by the specific embodiments disclosed below.

[0031] The embodiment of the present invention provides a method for identifying student behavior, comprising the following steps:

[0032] Step 1: Obtain a video dataset of student behavior in classroom scenarios.

[0033] Step 2: Build an optimized YOLOv5 model. The optimized YOLOv5 model includes: backbone network, neck and head; add a lightweight spatial feature enhancement module LW-SEAM (Lightweight SEAM) to the neck of the YOLOv5 model, and replace the classification loss of the YOLOv5 model with the classification loss function ClsSlideLoss, and replace the confidence loss with the loss function SlideLoss; the LW-SEAM module includes: using dilated convolution in the spatial feature enhancement module SEAM (Spatial Enhanced Attention Module); the classification loss function ClsSlideLoss includes: in the SlideLoss loss function, the classification accuracy of the prediction box is used as a threshold for positive and negative sample judgment. Use the video dataset to train the optimized YOLOv5 model, adjust the prediction error through the classification loss function ClsSlideLoss and the loss function SlideLoss, and obtain the YOLO-Act model.

[0034] Step 3: Input the video dataset to be detected into the YOLO-Act model, extract the student behavior features through the backbone network; input the student behavior features into the neck, obtain multi-scale features through LW-SEAM, and perform occlusion detection and feature fusion on the multi-scale features through the neck to obtain the processed student behavior features; input the processed student behavior features into the head to identify different student behaviors.

[0035] 1. Introduction

[0036] Classroom teaching is the main form of teaching activities and an important research object of educational research. Students' classroom behavior is the external manifestation of students' concentration and contains complex and valuable information. Real-time analysis and understanding of students' classroom behavior has a positive impact on teachers' timely adjustment of teaching strategies and thus improving teaching effectiveness. With the improvement of educational facilities, many campuses have built a regular classroom recording system, which can comprehensively record students' classroom behavior by installing cameras in the classroom. In colleges and universities, a course usually lasts more than ten weeks, and classes are held at fixed times every week. Video data can record every learning process video, providing rich information for data analysis. At the same time, since the camera is only used to record the overall situation of the classroom, it is usually not clear, and some classrooms are not well lit, resulting in problems such as low quality of these data.

[0037] Deep learning has achieved a series of results in the field of images and videos. Many researchers have applied deep learning and computer vision technology to the analysis of student classroom participation. For example: [1] YOLOv7 and multi-head self-attention are combined to enhance feature extraction to achieve student and teacher behavior detection in the classroom. [2] Attention-based relational reasoning and relational feature fusion algorithms are proposed to simulate the relationship between people and objects in the classroom. [3] YOLOv8, MTCNN, Cov-PoolFER, and OpenPifPaf are combined to jointly detect five behaviors: raising hands, standing, sleeping, yawning, and smiling, and a personalized classroom observation automation system is designed. Existing methods have achieved remarkable results in student classroom behavior recognition, but student behavior recognition in a classroom environment faces some challenges:

[0038] There are many blurred images and small targets. Classroom cameras are often installed on the top, which can cover the entire classroom area and provide a macro perspective. However, the image quality of regular course recordings is often blurred due to the influence of the equipment. In addition, many students are far away from the camera and have few visual features, which may lead to missed detection or false detection.

[0039] Mutual occlusion. High occlusion is the most unavoidable problem in student classroom behavior recognition. High occlusion leads to a large number of missing features, which in turn affects the accuracy of the algorithm in detecting the occluded students.

[0040] The samples are uneven. Some behaviors of students in class, such as looking up to listen to the class, reading and writing, are very common, while some behaviors, such as standing up to answer questions and lying on the table, rarely occur. Uneven behavior samples mean that the algorithm pays more attention to the categories with high frequency, which affects the accuracy of the model for abnormal behaviors that need to be focused on.

[0041] Due to the above challenges, many studies currently focus only on student behavior recognition algorithms rather than on methods to build student behavior profiles. Some public datasets, such as the Student Classroom Behavior dataset (SCB dataset), are mainly pictures of students in the classroom collected from the Internet. The student population is mainly primary and secondary school students, such as Figure 1 As shown in (a), the dataset divides student behaviors into three categories: hand-raising, reading, and writing.

[0042] Firstly, a student behavior dataset for real classroom scenarios of college students was constructed. Figure 1 As shown in (b), the dataset has labeled each student and contains 7 behaviors. Secondly, the YOLO-Act algorithm is used to solve the problem of student behavior recognition. The algorithm combines the SEAM module in YOLO-FaceV2 to solve the occlusion problem, and makes some improvements on the original SEAM to reduce model parameters. Then the SlideLoss loss function is used to enhance the model's recognition ability for a small number of samples by weighting difficult samples. Finally, the R3D model is used to construct a dynamic learning state portrait of the classroom based on the results of behavior recognition and an intelligent student classroom behavior portrait system is proposed.

[0043] (1) A student behavior dataset consisting of classroom surveillance was created. The classroom live video of 33 courses in the second half of the 2023 academic year was downloaded from the academic affairs system of University C, totaling 771 hours. 20,540 key frames of the class time period were obtained, 4,108 (i.e. 1 / 5) were randomly selected, and all students were labeled.

[0044] (2) The SEAM module is introduced and improved to solve the occlusion problem. SEAM enhances context perception by recommending attention and compensates for information loss in occluded areas. However, using SEAM directly will increase many parameters. We designed a lighter LW-SEAM to reduce the model size.

[0045] (3) Apply the SlideLoss loss function to the confidence loss. The SlideLoss loss function is improved and added to the classification loss. The problem of low recognition accuracy of some behaviors caused by sample imbalance and difficult samples is alleviated through adaptive weighting.

[0046] (4) A novel offline Dynamic Class Portraits construction method is proposed. Simply identifying students’ behaviors in class is not enough for teachers to quickly understand students’ overall learning situation. After obtaining students’ behaviors in chronological order in class, a behavior color-coded chart is drawn for each class, and a video classification algorithm is used to construct a dynamic class portrait based on the behavior color-coded chart.

[0047] 2. YOLO-Act

[0048] Student classroom behavior recognition is an important part of correctly constructing classroom portraits. In addition to detecting the location of objects, object detection can also identify object categories, so object detection algorithms can be used to solve student classroom behavior recognition tasks. Compared with mainstream detection algorithms such as SSD and Faster R-CNN. The YOLO algorithm achieves accurate detection of objects of different sizes through multi-scale feature fusion, and has excellent real-time performance. Based on this, we apply YOLO to the task of classroom student behavior detection. In order to improve the accuracy and robustness of the model for difficult examples such as occluded objects and a few samples, this paper modifies YOLOv7 to the YOLO-Act model. YOLO-Act consists of a backbone, neck, and head. The backbone network of YOLOv5 is used to extract image features, and the P2 layer is also added to the feature fusion. This is done to more comprehensively understand objects of different sizes and improve the detection performance of multi-scale objects. In the classroom environment, students gather together, the density of people is high, and there is serious occlusion between each other. LW-SEAM is added to the neck to enhance the occlusion detection ability of the model. In the head, this paper uses a decoupled head to predict the location and category of the target respectively. Although the decoupling head increases the number of parameters, it shows that the content of classification and positioning is inconsistent. Using different branch operations is conducive to improving the effect, such as Figure 2 shown.

[0049] 3. Double Slide Loss

[0050] Since students tend to look up in class and look down in class, while lying on the table and playing with mobile phones rarely occur, there is a serious sample imbalance problem in the classroom student behavior data. In this case, the algorithm has low recognition accuracy for a few cases. Secondly, there is a high similarity between reading and writing, playing with mobile phones, and looking down, and the algorithm is prone to misclassification. For these difficult behaviors, this article uses the SlideLoss loss function to add weights to the difficult samples. This article made some modifications when applying it to classification losses. SlideLoss regards the difference between simple samples and difficult samples as the IoU size, see Figure 3 (a). However, considering the situation that some predicted boxes have large IoU but are classified incorrectly, we design a SlideLoss specifically for classification loss based on SlideLoss and name it ClsSlideLoss. Figure 3 (b). In order to reduce hyperparameters, SlideLoss takes the average IoU value of all bounding boxes as the threshold μ, and takes negative samples if it is less than μ, and takes positive samples if it is greater than μ. Referring to the design of SlideLoss, ClsSlideLoss takes the classification accuracy of all predicted boxes as the threshold β, and takes negative samples if it is less than β, and takes positive samples if it is greater than β. SlideLoss is added to the confidence loss, and ClsSlideLoss is added to the classification loss to adaptively assign higher weights to difficult samples.

[0051]

[0052]

[0053] 4. LW-SEAM

[0054] Occlusion between students is an issue that must be considered in the student behavior recognition task. Occlusion may lead to feature alignment errors, local aliasing, and loss of key information, which seriously affects the accuracy. To overcome this challenge, a spatial attention and multi-scale modeling enhancement module (SEAM) is used and improved on its basis to make it lighter. The improved SEAM is named LW-SEAM.

[0055] First, the channel and spatial mixing module CSMM (channel and spatial mixing module) captures information of different scales from the input features to enhance context perception, and uses the spatial attention mechanism to focus on the occluded areas in the image, so that the network can pay more attention to the contextual information around the occluded areas when processing occluded features, to assist in accurate feature extraction. Then, the output of the CSMM module is average pooled to achieve global context modeling in the spatial dimension. Secondly, in the channel dimension, LW-SEAM uses a two-layer fully connected network to fuse the information of each channel. Further extract and enhance features related to occlusion problems, and improve detection accuracy in the presence of occlusion. Finally, channel expansion Channel exp is used to enhance channel feature representation, increase the network's feature representation capability and robustness to occlusion features. The structure of LW-SEAM is as follows: Figure 4 shown.

[0056] Figure 4 In the figure, the upper left side is the overall architecture of SEAM. The SEAM module first includes three CSMMs of different sizes. The upper right side is the detailed structure of the CSMM module. Patch Embedding (PE) in CSMM implements multi-scale feature extraction through three sizes of convolution, but large convolution kernels increase a large number of parameters. Therefore, PE first uses three dilated convolutions (dilated convolutions) with dilation rates (dilated rates) of 1, 3, and 5 to obtain multi-scale features, and splices the features together along the channel.

[0057] PE is calculated as follows:

[0058]

[0059] Among them, F∈R C×H×W represents the input feature map, Represents a convolution operation with a kernel size of 3×3 and a dilation rate of rate.

[0060] After obtaining feature maps of different scales, deep separable convolution is used to extract features. Although deep separable convolution can learn the importance of different channels and reduce the number of parameters, it ignores the information relationship between channels. In order to make up for this loss, point-by-point convolution is then used to combine. The above constitutes the entire CMSS module. Figure 4 Shown in the upper right portion.

[0061] The CMSS calculation method is expressed as follows:

[0062]

[0063] It indicates that the convolution kernel is 3×3 depth-separable convolution. represents point-by-point convolution, σ represents GELU (Gaussian Error Linear Unit) function and BN.

[0064] Then, average pooling is used to capture the global context information of the spatial dimension, and the two-layer fully connected network fuses the information of each channel, so that the network strengthens the connection between channels. Then the range is expanded from [0,1] to [1,e] through an exponential function. This exponential normalization provides a monotonic mapping relationship, making the result more tolerant to position errors. Finally, the output is multiplied with the original feature as the attention coefficient.

[0065] 5. Experimental Setup

[0066] In order to verify and compare the performance of our model, we use Precision (P), Recall (R), Mean Average Precision (mAP), Giga Floating-point Operations Per Second (GFLOPs), and Frames Per Second (FPS) as evaluation indicators. P is the proportion of positive samples detected that are correctly classified, and R is the proportion of positive samples detected. The definitions of Precision, Recall, and mAP are as follows:

[0067]

[0068] Where TP represents the number of correctly classified positive samples, FP represents the number of negative samples that are misclassified as positive samples, and FN represents the number of positive samples that are misclassified as negative samples. The formula for calculating mAP (mean average precision) is as follows:

[0069]

[0070] where mAP_0.5denotes the mean average precision when IoU threshold isset to 0.5.

[0071] GFLOPs is a derivative unit of FLOPs, which means one billion floating point operations per second. It is used to measure the computing speed and capability of the model.

[0072] FPS is used to measure the inference speed of the model. The calculation method of FPS is as follows:

[0073]

[0074] Among them, t preprocessrepresents the preprocessing time, t inference represents the inference time, t postprocess Indicates the post-processing time in ms.

[0075] The embodiment of the present invention provides a student behavior recognition system, including:

[0076] The data acquisition module is used to obtain video datasets of student behaviors in classroom scenarios.

[0077] A model building module is used to build an optimized YOLOv5 model. The optimized YOLOv5 model includes: a backbone network, a neck and a head; a lightweight spatial feature enhancement module LW-SEAM is added to the neck of the YOLOv5 model, and the classification loss of the YOLOv5 model is replaced by a classification loss function ClsSlideLoss, and the confidence loss is replaced by a loss function SlideLoss; LW-SEAM includes: using a dilated convolution in the spatial feature enhancement module SEAM; the classification loss function ClsSlideLoss includes: in the SlideLoss loss function, the classification accuracy of the prediction box is used as a threshold for positive and negative sample judgment.

[0078] The model training module is used to train the optimized YOLOv5 model using the video dataset, adjust the prediction error through the classification loss function ClsSlideLoss and the loss function SlideLoss, and obtain the YOLO-Act model.

[0079] The behavior classification module is used to input the video dataset to be detected into the YOLO-Act model, extract the student behavior features through the backbone network; input the student behavior features into the neck, obtain multi-scale features through LW-SEAM, and perform occlusion detection and feature fusion on the multi-scale features through the neck to obtain the processed student behavior features; input the processed student behavior features into the head to identify different student behaviors.

[0080] An embodiment of the present invention provides a computer device, including: a memory and a processor; the memory stores a computer program, and the processor implements the steps of a student behavior recognition method when executing the computer program.

[0081] An embodiment of the present invention provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps of a method for identifying student behavior are implemented.

[0082] A specific embodiment is as follows:

[0083] 1. StuAct Datasets

[0084] 1. Data processing

[0085] We collected monitoring data from 33 courses at University C, with the number of students in each course ranging from 40 to 120. The data covers four grades and eight colleges of the university. The total duration is about 771 hours. The original monitoring videos downloaded from the academic affairs system are mixed with the teacher's class time and the time before and after class. In order to (1) efficiently process the original data and obtain differentiated frames and (2) filter out data outside the class time. We use the local maximum frame difference method to extract the video key frames. The specific steps are as follows:

[0086] (1) For each video, the inter-frame difference is calculated every 24 frames, and an image of the inter-frame difference changing over time is plotted.

[0087] (2) Obtain the range of inter-frame differences during the class period and calculate the average value.

[0088] (3) Filter out frames with values ​​below the average. Through the above steps, we obtained a total of 20,540 images, each with a size of 1920×1080. We randomly selected 4,108 frames. In addition, in order to enrich and diversify the data, we collected 1,370 additional images of students listening to lectures from various angles from the Internet, for a total of 5,478 images. After the data collection was completed, the students’ classroom behaviors were divided into seven categories: reading and writing, playing with mobile phones, looking down, lying on the table, looking up to listen to lectures, turning heads, and standing up to answer questions. The number of samples in each category of the StuAct dataset is as follows: Figure 5 shown.

[0089] 2. Data Annotation

[0090] In order to save manpower and time costs, we use the EasyData intelligent data service platform to semi-automatically annotate the StuAct dataset. After the annotation is completed, we use yolov5 to train and fine-tune it manually, and obtain the final annotated data after repeated iterations. Figure 6 shown.

[0091] 2. Experimental results on the StuAct dataset

[0092] In order to verify the contribution of the proposed Double-SlideLoss and LW-SEAM to improving model performance, YOLOv5s is used as the baseline model. Double-SlideLoss and LW-SEAM are applied to their network structures in turn, and experiments are carried out on the StuAct dataset. Table 1 shows that the SCB-DATASET3-S dataset does not have a serious data imbalance problem, so the use of Double-SlideLoss is almost consistent with the baseline effect. After adding LW-SEAM, the mAP value increased by 0.8, indicating that the LW-SEAM module has a certain effect on SCB-DATASET3-S. After using Double-SlideLoss on the StuAct dataset, P decreased by 3.3, but R increased by 5.6, and mAP increased by 1.3. LW-SEAM increased the R value by 5.1 and the mAP value by 1.2 under the condition that the model lost P by 3.6.

[0093] Table 1 Comparison before and after adding different modules

[0094]

[0095] Table 2 shows the comparison of AP values ​​before and after adding different modules to the StuAct database. The mAP value of each behavior category is given, as well as the effect after using Double-SlideLoss and LW-SEAM. As can be seen from the table, the recognition accuracy of Reading & Writing (88.4%), Bow head (87.1%), Head up to listen to the class (84.4%) and Stand (82%) categories is relatively high. However, the recognition accuracy of Using a phone (20.8%), Lying on the table (49.2%) and Turn the head (27%) is relatively low. This is because these three categories are abnormal behaviors in the classroom, and the number of samples is less than that of other categories. On the other hand, the behavior similarity between Using a phone, Lying on the table and Bow head categories is very high, and the behavior similarity between Turn the head and Head up to listen to the class categories is also very high, resulting in poor recognition accuracy of the model for these samples. After using Double-SlideLoss, the mAP of Using a phone and Lying on the table increased significantly, except for Turn the head, which shows that Double-SlideLoss does help improve the accuracy of some abnormal behaviors. Adding LW-SEAM also improves the detection of "uncommon" behaviors in the classroom.

[0096] Table 2 Comparison of AP values ​​before and after adding different modules in the StuAct database

[0097]

[0098] 3. Comparison with existing object detectors

[0099] We conducted a comparative analysis between YOLO-Act and several target detection methods, including RT-DETR, YOLOv8s, Faster R-CNN, and SSD. The comparison results on the StuAct dataset are shown in Table 3. In addition to comparing the model mAP, we compared Parameters, FPS, and GFLOPs to measure the model size and inference speed. As can be seen from the table, the Transformer-based RT-DETR and the two-stage Faster R-CNN are significantly inferior in speed and cannot meet real-time requirements. Compared with the YOLO series of algorithms, the accuracy is lower. The SSD algorithm has a high FPS, but the algorithm has the lowest accuracy. YOLO-Act only adds a small number of parameters compared to YOLOv8s, but the FPS has not decreased. And the mAP of YOLO-Act is 6% higher than that of YOLOv8s. The mAP curves of these algorithms are shown in Figure 7 (a). Figure 7 As shown in (b), YOLO-Act has the highest AP value in all five categories. Secondly, we observe that Fast R-CNN’s recognition effect in the head turning category is much higher than other algorithms.

[0100] Table 3 Comparative study on StuAct dataset

[0101]

[0102] IV. Experimental Results on Category Performance

[0103] The purpose of identifying students' behaviors in class is to help teachers quickly understand the state of the class. Therefore, this paper represents students' behaviors in class with 8×8 pixel patches, and the color of each patch corresponds to a behavior. We plot the behaviors of all students in a certain frame on a 112×112 pixel image, namely the Behavior Color-Coded Chart. Similarly, we can get a Behavior Color-Coded Chart of the entire class with a time series, and use them to construct a classroom portrait.

[0104] In order to utilize time series information, we innovatively applied video classification algorithms to the task of classroom portrait construction. We used the R3D and ViVit algorithms to construct classroom portraits. The results are shown in Figure 2. Figure 8 shown.

[0105] Depend on Figure 8 It can be seen that the accuracy of the R3D algorithm exceeds 85%, which is much higher than ViVit, and the loss is smaller. Therefore, we use the R3D algorithm to construct classroom portraits for 33 courses (a total of 342 classes). Fig. 9The output of MSC, namely student behavior visualization and class portrait, is shown. Fig. 9 In the second row of Figure 1, teachers can quickly obtain information about student behavior and class status (e.g., student behavior is visualized in the lower left corner and class portrait is displayed in the lower right corner).

[0106] We also counted and analyzed the results of classroom portraits. First, we counted the class portraits of different grades and different colleges and plotted the results on Fig.10 (a) and Fig.10 (b). From Fig.10 It can be seen that among all types of class status diagrams, the most common one is the focused listening class, which reflects that students are more interested in the course content, have a good learning attitude, and also reflects that the teacher's teaching method is attractive. Secondly, the number of Interactive Diversity Class ranks second. Fig.10 (a), we found that freshmen had the highest rate of looking up during class, while juniors had the highest rate of looking down during class.

[0107] Secondly, we found that the classroom portraits of most courses remained unchanged overall. That is, for a course, the behavior coding diagrams of students from the beginning to the end of the semester are likely to be similar. We selected 6 courses and plotted the changes in class portraits from week 1 to week 12, as shown in the figure below: Fig.11 shown.

[0108] Finally, in this paper, a dataset StuAct containing 7 categories of student classroom behaviors was created. Classroom student behavior recognition faces some challenges. Solutions for difficult behavior recognition were explored. First, the SlideLoss loss function was used and improved to solve the detection and recognition of a small number of samples and difficult samples. Secondly, for the high occlusion problem between students, SEAM was used to compensate for the occlusion feature extraction, and some improvements were made to make it lightweight. Finally, the network YOLO-Act proposed for classroom behavior recognition achieved good results on StuAct and SCB-DATASET3-S data. A solution for building offline class portraits was also proposed.

[0109] The above-mentioned embodiments only express several implementation methods of the present invention, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

Claims

1. A method for identifying student behavior, characterized in that: The following steps are involved: Obtain a video dataset of student behavior in classroom scenarios; An optimized YOLOv5 model is constructed, wherein the optimized YOLOv5 model includes: a backbone network, a neck and a head; a lightweight spatial feature enhancement module LW-SEAM is added to the neck of the YOLOv5 model, and the classification loss of the YOLOv5 model is replaced by a classification loss function ClsSlideLoss, and the confidence loss is replaced by a loss function Sl ideLoss; the LW-SEAM includes: using a dilated convolution in the spatial feature enhancement module SEAM; the classification loss function ClsSlideLoss includes: in the SlideLoss loss function, the classification accuracy of the prediction box is used as a threshold for positive and negative sample judgment; The optimized YOLOv5 model is trained using the video dataset, and the prediction error is adjusted through the classification loss function ClsSlideLoss and the loss function SlideLoss to obtain the YOLO-Act model. The video dataset to be detected is input into the YOLO-Act model, and the student behavior features are extracted through the backbone network; the student behavior features are input into the neck, multi-scale features are obtained through LW-SEAM, and occlusion detection and feature fusion are performed on the multi-scale features through the neck to obtain the processed student behavior features; the processed student behavior features are input into the head to identify different student behaviors.

2. A method for identifying student behavior as claimed in claim 1, characterized in that: The classification loss function ClsSlideLoss specifically includes: In the classification loss function ClsSlideLoss, the classification accuracy of all predicted boxes is used as the threshold, negative samples are taken when the accuracy is less than the threshold, and positive samples are taken when the accuracy is greater than the threshold.

3. A method for identifying student behavior as claimed in claim 1, characterized in that: The optimized YOLOv5 model is trained using the video dataset, and the specific steps include: The behavior of each student in the video dataset is labeled. The student behaviors are divided into reading and writing behaviors, mobile phone use behaviors, head-down behaviors, lying on the table behaviors, head-up behaviors, head-turning behaviors, and standing up to answer questions behaviors; The optimized YOLOv5 model is trained based on the annotated video dataset so that the model can classify the behavior of each student.

4. A method for identifying student behavior as claimed in claim 1, characterized in that: The LW-SEAM module specifically includes: The large convolution kernel of the patch embedding module Patch-Embedding in SEAM is replaced with dilated convolutions with dilation rates of 1, 3, and 5, respectively. The convolution kernel size is 3×3, and an LW-SEAM module that can obtain multi-scale features is constructed.

5. A student behavior recognition system, characterized in that: include: The data acquisition module is used to obtain video datasets of student behaviors in classroom scenarios; A model building module is used to build an optimized YOLOv5 model, wherein the optimized YOLOv5 model includes: a backbone network, a neck and a head; a lightweight spatial feature enhancement module LW-SEAM is added to the neck of the YOLOv5 model, and the classification loss of the YOLOv5 model is replaced by a classification loss function ClsSl ideLoss, and the confidence loss is replaced by a loss function SlideLoss; the LW-SEAM includes: using a dilated convolution in the spatial feature enhancement module SEAM; the classification loss function ClsSlideLoss includes: in the SlideLoss loss function, the classification accuracy of the prediction box is used as a threshold for positive and negative sample judgment; The model training module is used to train the optimized YOLOv5 model using the video dataset, adjust the prediction error through the classification loss function ClsSlideLoss and the loss function SlideLoss, and obtain the YOLO-Act model; The behavior classification module is used to input the video dataset to be detected into the YOLO-Act model, extract the student behavior features through the backbone network; input the student behavior features into the neck, obtain multi-scale features through LW-SEAM, and perform occlusion detection and feature fusion on the multi-scale features through the neck to obtain the processed student behavior features; input the processed student behavior features into the head to identify different student behaviors.

6. A computer device comprising: Memory and processor; The memory stores a computer program, wherein the processor implements a student behavior recognition method according to any one of claims 1 to 4 when executing the computer program.

7. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, a method for identifying student behavior as described in any one of claims 1 to 4 is implemented.

Images