A student classroom behavior detection method based on improved YOLOv7

Abstract

The application discloses a student classroom behavior detection method based on an improved YOLOv7, and belongs to the technical field of classroom behavior detection. Firstly, the Detect prediction head is changed into an ASFFDetect structure, so that the YOLOv7 network model performs feature fusion on different feature levels, so as to capture target information of different scales and improve the target positioning capability. Secondly, a WDLoss is used to replace a CIoU loss function in the original YOLOv7 network model, so as to adapt to unbalanced data and improve the model generalization capability. Finally, an attention mechanism ACmix module is added, so that the network pays more attention to the target to be detected, and the feature processing capability of the network is enhanced. The improved YOLOv7 model can effectively detect the classroom behavior of students under the conditions of low image resolution, different scale targets and occlusion.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of classroom behavior detection, more specifically, the present application relates to a student classroom behavior detection method based on improved YOLOv7. BACKGROUND

[0002] With the development of education, the importance of classroom teaching in the field of education and teaching is increasingly valued, especially the reaction and behavior change of students in the classroom. The proposal of "new curriculum reform" puts forward higher requirements for teaching evaluation. At the same time, in recent years, China has orderly promoted the construction of smart school, and gradually constructed a school mode characterized by smart teaching, smart management, smart life and so on. Student classroom is one of the key links to build a smart school, and the quality of student classroom is influenced by many factors, including teaching design, classroom practice and teaching evaluation. Among them, observing the classroom behavior of students to evaluate teaching is an effective and commonly used method.

[0003] In traditional teaching evaluation, there will be an evaluation teacher sitting in the back row to evaluate the students' class state and the teacher's teaching situation. However, due to the position limitation of the evaluation teacher, it is difficult to observe the specific class state of the students comprehensively. The evaluation teacher can only evaluate the class state of a few students, resulting in the incompleteness of the evaluation data. In addition, there are differences in evaluation standards, observation methods and thinking angles of different evaluation teachers, which also leads to the difference of teaching evaluation results. At different periods of the same class, the mental state of the evaluation teacher is different, it is difficult for the evaluation teacher to observe the classroom behavior of the students for a long time, which further increases the difference of teaching evaluation. Therefore, it is of great significance to evaluate teachers, teachers, school leaders and parents to detect and analyze the behavior of students in the classroom from an objective point of view. If computer technology can be used to automatically identify and detect the classroom behavior of students, it will be able to provide comprehensive and objective data reference for teaching evaluation, and help to improve the teaching quality.

[0004] With the development of video analysis and computer vision technology, analyzing student behavior in classroom videos or pictures for teaching evaluation can provide more accurate and objective feedback. In the field of classroom behavior detection, common algorithms include video-based action recognition, pose estimation, and object detection methods. Video action recognition faces the problem of large-scale and high-dimensional video data processing, which requires a large amount of computing resources and storage space. Behavior in videos has long-term dependence, which needs to capture and model the temporal relationship. Pose estimation is challenging in multi-person scenarios, as the accuracy of pose estimation decreases when body parts are occluded or pose changes dramatically. Time series analysis needs to establish long-term dependencies to adapt to different behavior patterns and contexts. Using object detection for behavior recognition can accurately locate target objects and simultaneously detect and recognize multiple targets in complex multi-person interactions and group behavior scenarios. Object detection technology has made significant progress in real-time applications, providing strong support for behavior recognition tasks.

[0005] There are many student targets and severe occlusion in classroom teaching videos, which poses great research challenges for student behavior recognition in classroom scenarios. To automatically identify the classroom behavior of all students, a more robust multi-person behavior recognition model needs to be developed. Traditional student classroom behavior detection methods based on object detection are affected by factors such as the number of student targets, inconsistent target sizes, target occlusion, and low video or image resolution, making it difficult to accurately and efficiently identify the behavior status of students in the classroom. SUMMARY

[0006] To address the deficiencies in the prior art, the present patent proposes a student classroom behavior detection method based on improved YOLOv7. The student classroom behavior detection method based on improved YOLOv7 of the present invention mainly improves the backbone network, prediction head, IOU calculation loss, and other modules of YOLOv7. The improved model pays more attention to the target to be detected, thereby improving the behavior detection capability in the student classroom scenario. The problems mentioned in the above background technology are solved. Experimental results prove that the method of the present invention is more superior than the prior art.

[0007] To achieve the above purpose, the technical solution adopted by the present invention is: a student classroom behavior detection method based on improved YOLOv7, comprising the following steps:

[0008] Step 1, obtain a student classroom behavior video, and frame the obtained video to obtain student classroom behavior pictures;

[0009] Step 2: Preprocess the images obtained in Step 1, label the student classroom behavior dataset using the labelImg image labeling tool, and divide the dataset to obtain the student classroom behavior dataset.

[0010] Step 3: Construct a student classroom behavior detection network based on the improved YOLOv7, add an ACmix attention mechanism in the main network of YOLOv7 algorithm, improve the prediction head part in YOLOv7 algorithm, replace the original Detect in YOLOv7 algorithm with ASFFDetect structure, and introduce NWD-based Regression Loss as the loss function;

[0011] Step 4: Input the image data in the dataset as input into the improved YOLOv7 model for training to obtain the trained student classroom behavior detection model;

[0012] Step 5: Send the student classroom scene image to be detected into the trained model to obtain the behavior category and confidence of the student;

[0013] The image preprocessing and image labeling obtained in Step 2 include the following steps:

[0014] Step 2.1: Use the OpenCV library to preprocess the obtained student classroom behavior images, such as changing brightness and contrast, removing background, local image, smoothing, noise reduction, and fusing pictures to obtain student classroom behavior images;

[0015] Step 2.2: Use the labelImg image labeling tool to label the student actions in the obtained student classroom behavior images, and save the label information in a txt file with the same name as the picture to obtain the student classroom behavior dataset;

[0016] Step 2.3: Divide the student classroom behavior image dataset into training dataset and test dataset, and divide all pictures and labeled labels into training set and test set according to the ratio of 8:2.

[0017] The student classroom behavior detection network based on improved YOLOv7 mainly includes four parts of input (Input), backbone network (Backbone), neck (Neck) and head (Head). The ACmix attention convolution module is introduced in the Neck part of the basic YOLOv7, which highlights the key target features contained in the shallow network and weakens irrelevant information, improves the detection performance of small targets, and makes the network pay more attention to the target to be detected. In the Head part, the original Detect prediction head in the network is replaced by the ASFFDetect prediction head. By learning the best fusion method of different levels of features during the training process, other layer features carrying contradictory information are filtered out, thereby solving the problem of inconsistent learning goals. The NWD-based Regression Loss is introduced to replace the CIoU in the original YOLOv7 network model to optimize the loss function, adapt to unbalanced data, and improve the generalization ability of the model;

[0018] The ACmix attention convolution module introduced in the Neck part can be roughly divided into three steps. The first stage: project the input features through 3 1x1 convolutions, and then reorganize them into N blocks. Therefore, a feature map containing 3xN intermediate features is obtained. The second stage: use different norms, for the self-attention path, collect the intermediate features into N groups, where each group contains three features corresponding to q, k, and v. For the convolution path with kernel size k, a lightweight fully connected layer is used to generate K2 feature maps, and features are generated by shifting and aggregating. The third stage: add the outputs of the two paths, and the strength is controlled by two learnable scalars as shown in the following formula:

[0019] F out =αF att +βF conv (1)

[0020] Where F out represents the final output of the path, F att represents the output of the self-attention branch, F conv represents the output of the convolution attention branch, and the values of parameters α and β are both 1. After the output results of the two branches are combined, global features and local features are considered, thereby improving the detection effect of the network on small targets.

[0021] The Detect prediction head in the Head part is replaced by the ASFF Detect prediction head, and the ASFF module includes two steps: same size transformation and adaptive feature fusion. Feature same size transformation: the feature maps of different layers are inconsistent in size, so any fusion method needs to be reshaped to the same size. Small size to large size needs to be up-sampled, and large size to small size needs to be down-sampled. Adaptive fusion: taking ASFF-3 as an example, features X1, X2 and X3 from level, level2 and level3 are multiplied by weight parameters α3, β3 and γ3 respectively and added to obtain new fusion features ASFF-3:

[0022] (2)

[0023] wherein, means that the (i, j) vector of the output feature is mapped between channels y l , refers to the spatial importance weight of the feature map from three different levels to level L. Because the addition method is adopted, the size of the output features of level1~3 layers needs to be the same when added, and the number of channels also needs to be the same. The features of different layers need to be up-sampled or down-sampled and the number of channels needs to be adjusted. For weight parameters α, β and γ, the resized level1~level3 feature maps are obtained through 1×1 convolution. And parameters α, β and γ pass through the concat layer and then pass through the softmax function, so that their range is in [0, 1] and the sum is 1:

[0024] (3)

[0025] wherein, means that the (i, j) vector of the output feature is mapped between channels y l , refers to the spatial importance weight of the feature map from three different levels to level L. The loss function of the original model is replaced by the loss function designed based on NWD measurement:

[0026] (4)

[0027] wherein N p is a Gaussian distribution model of the prediction box P, and N g is a Gaussian distribution model of the GT box G; the loss based on NWD provides gradients |P∩G|=0 and |P∩G|=P or G.

[0028] The technical scheme of the present application can achieve the following technical effects: the student classroom behavior detection method based on improved YOLOv7 can highlight the key target features contained in the shallow network by adding the ACmix attention convolution module, weaken irrelevant information, and make the network pay more attention to the target to be detected, solving the problem of multiple target occlusion in the above-mentioned classroom scene. The Detect prediction head of the Head part of the original YOLOv7 model is replaced by the ASFFDetect prediction head, which learns the best fusion method of different levels of features during the training process, filters out other layer features carrying contradictory information, thereby solving the learning target inconsistency problem, and solving the problem of large size difference of the target in the above-mentioned classroom scene. In addition, the NWD-based Regression Loss is introduced to replace the CIoU in the original YOLOv7 network model to optimize the loss function, adapt to unbalanced data, improve the model generalization ability, and solve the detection problem in the case of low image resolution in the classroom scene. BRIEF DESCRIPTION OF DRAWINGS

[0029] Figure 1 A flowchart of a student classroom behavior detection method based on improved YOLOv7.

[0030] Figure 2 A network model structure of a student classroom behavior detection method based on improved YOLOv7.

[0031] Figure 3 A generated effect diagram of a student classroom behavior detection method based on improved YOLOv7. DETAILED DESCRIPTION

[0032] The present application will be further described in detail below in conjunction with the drawings and specific embodiments: the present application is further described and explained by way of example. Obviously, the described examples are only part of the embodiments of the present application, not all embodiments.

[0033] Figure 1 A flowchart of a student classroom behavior detection method based on improved YOLOv7 is shown. The student classroom behavior detection method based on improved YOLOv7 specifically includes the following steps:

[0034] Step 1, acquire a student classroom behavior video, and frame the acquired video to obtain student classroom behavior pictures;

[0035] Acquire a student classroom behavior video, download a student classroom behavior dataset, the data is from Github, download a classroom behavior video, read the video, set the resolution of the output image, output each frame in order in the image format, and obtain a student classroom behavior image.

[0036] Step 2, pre-process the image obtained in step 1, label the student classroom behavior dataset using the labelImg image labeling tool, and divide the dataset to obtain the student classroom behavior dataset;

[0037] Step 2.1, pre-process the student classroom behavior image using the OpenCV library, change the brightness and contrast, remove the background, local image, smooth processing, denoising, and fuse the pictures to obtain the student classroom behavior image.

[0038] Step 2.2, use the labelImg image labeling tool to label the student classroom behavior image obtained, and save the label information in the txt file with the same name as the picture to obtain the student classroom behavior dataset;

[0039] Step 2.3, divide the student classroom behavior image dataset into training dataset and testing dataset, divide all pictures and labeled labels into training set and testing set according to the ratio of 8:2;

[0040] Step 3, construct a student classroom behavior detection network based on improved YOLOv7, add ACmix attention mechanism in the main network of YOLOv7 algorithm, improve the prediction head part in YOLOv7 algorithm, replace the original Detect in YOLOv7 algorithm with ASFFDetect structure, and introduce NWD-based Regression Loss as the loss function;

[0041] Construct a student classroom behavior detection network based on improved YOLOv7, which specifically includes adding attention convolution module, changing prediction head and replacing loss function:

[0042] The student classroom behavior detection network based on improved YOLOv7 mainly includes input (Input), backbone network (Backbone), neck (Neck), and head (Head) 4 parts. The ACmix attention convolution module is introduced in the Neck part of the basic YOLOv7 to highlight the key target features contained in the shallow network, weaken irrelevant information, improve the detection performance of small targets, and make the network pay more attention to the target to be detected. In the Head part, the original Detect prediction head in the network is replaced by the ASFFDetect prediction head. By learning the best fusion method of different levels of features during training, other layer features carrying contradictory information are filtered out, thereby solving the problem of inconsistent learning goals. NWD-based Regression Loss is introduced to replace CIoU in the original YOLOv7 network model to optimize the loss function, adapt to unbalanced data, and improve the model generalization ability;

[0043] The ACmix attention convolution module introduced in the Neck section can be roughly divided into three steps: First, the input features are projected using three 1×1 convolutions and then reorganized into N blocks. This yields a feature map containing 3×N intermediate features. Second, depending on the paradigm, for self-attention paths, the intermediate features are collected into N groups, each containing three features corresponding to q, k, and v. For convolutional paths with a kernel size of k, a lightweight fully connected layer is used to generate K² feature maps, which are then generated through shifting and aggregation. Third, the outputs of the two paths are summed, and their strength is determined by two learnable scalar expressions.

[0044] F out =αF att +βF conv (1)

[0045] Where F out F represents the final output of the path. att F represents the output of the self-attention branch. conv This represents the output of the convolutional attention branch, with parameters α and β both set to 1. The combined outputs of the two branches take into account both global and local features, thus improving the network's ability to detect small objects.

[0046] The Head section replaces the original Detect prediction head with the ASFFDetect prediction head. The ASFF module consists of two steps: same-size transformation and adaptive feature fusion. Same-size transformation: feature maps from different layers have inconsistent sizes, so regardless of the fusion method, they need to be resized to the same size. Upsampling is required when reducing a small size to a large size, and downsampling is required when reducing a large size to a small size. Adaptive fusion: Taking ASFF-3 as an example, features X1, X2, and X3 from level 1, level 2, and level 3 respectively are multiplied by weight parameters α3, β3, and γ3 and added together to obtain the new fused feature ASFF-3.

[0047] (2)

[0048] in, This means that the (i, j) vector of the output feature is mapped to y between channels. l , is the spatial importance weight of the feature map of three different levels to level L. Because the addition method is adopted, the feature sizes of the level 1-3 layer outputs need to be added, and the channel numbers also need to be the same, so the features of different layers need to be up-sampled or down-sampled and the channel numbers need to be adjusted. For the weight parameters a, b and g, the resized level 1-level 3 feature maps are obtained through 1x1 convolution. And after the parameters a, b and g pass through the concat layer, they are made to be in the range of [0, 1] and the sum is 1 through the softmax function:

[0049] (3)

[0050] wherein, means that the (i, j) vector of the output feature is mapped between channels y l , is the spatial importance weight of the feature map of three different levels to level L. The loss function of the original model is replaced with the loss function designed based on the NWD metric:

[0051] (4)

[0052] wherein N p is the Gaussian distribution model of the prediction frame P, and N g is the Gaussian distribution model of the GT frame G; the loss based on NWD provides gradients |P∩G|=0 and |P∩G|=P or G.

[0053] Step 4, input the image data in the data set as input into the improved YOLOv7 model for training to obtain a trained student classroom behavior detection model;

[0054] The image data in the student classroom behavior data set is input into the improved YOLOv7 model for training, the training parameters are set, the learning rate is set to 0.001, the confidence threshold is set to 0.5, all the pictures in the training set are input into the improved YOLOv7 model for training, and the training operation is repeated to obtain the model with the best training effect.

[0055] Step 5, input the student classroom scene image to be detected into the trained model to obtain the behavior category and confidence of the student;

[0056] The trained student classroom behavior detection network based on the improved YOLOv7 is used for student classroom behavior detection.

[0057] Figure 2A network model structure of a student classroom behavior detection method based on improved YOLOv7 is shown. As shown in the figure, the method designs a novel network structure for student classroom behavior detection, adds an ACmix attention convolution module on the basis of the YOLOv7 network, changes the ASFFDetect into a prediction head, and replaces the original loss function with an NWD-based Regression Loss. The experimental results prove that the method has superiority in accuracy and real-time compared with the prior art.

[0058] A group of student classroom behavior images are taken as input, and different input images are detected through step 5 to obtain student classroom behavior detection images. Figure 3 The detection effect diagram of the group of pictures is shown, from Figure 3 It can be seen that the method can accurately detect the behavior of students in the multi-target occlusion classroom scene, proving the feasibility and effectiveness of the method.

Claims

1. A method for detecting student classroom behavior based on an improved YOLOv7, characterized in that, Includes the following steps: Step 1: Obtain video of students' classroom behavior, and split the video into frames to obtain images of students' classroom behavior; Step 2: Preprocess the images obtained in Step 1, use the labelImg image annotation tool to annotate the student classroom behavior dataset, and divide the dataset to obtain the student classroom behavior dataset. Step 3: Construct a student classroom behavior detection network based on the improved YOLOv7 algorithm. Add an ACmix attention convolution module to the YOLOv7 algorithm backbone network, improve the prediction head part of the YOLOv7 algorithm, and replace the Detect in the original YOLOv7 algorithm with an ASFFDetect structure. By learning the best fusion method of features at different levels during training, other layer features carrying contradictory information are filtered out. At the same time, NWD-based Regression Loss is introduced as the loss function. The improved YOLOv7-based student classroom behavior detection network mainly consists of four parts: Input, Backbone, Neck, and Head. The ACmix attention convolution module introduced in the Neck part specifically includes the following steps: The first stage involves projecting the input features through three 1×1 convolutions and then reorganizing them into N blocks to obtain a feature map containing 3×N intermediate features. The second stage: According to different paradigms, for the self-attention path, the intermediate features are collected into N groups, each group containing three features, corresponding to q, k, and v; for the convolutional path with a kernel size of k, a lightweight fully connected layer is used to generate K2 feature maps, and features are generated through shifting and aggregation. In the third stage, the outputs of the two paths are added together, and their strength is represented by two learnable scalar control equations: F out =αF att +βF conv (1) Where F out F represents the final output of the path. att F represents the output of the self-attention branch. conv This represents the output of the convolutional attention branch, with parameters α and β both having a value of 1. In the Head section, the original Detect prediction head in the network is replaced with the ASFFDetect prediction head. The ASFFDetect module consists of two steps: same-size transformation and adaptive feature fusion. (1) Feature map size transformation: The feature maps of different layers are not the same size, so they are reshaped to the same size; upsampling is required when changing from small size to large size, and downsampling is required when changing from large size to small size. (2) Adaptive fusion: The features X1, X2, and X3 from level 1, level 2, and level 3, respectively, are multiplied by the weight parameters α, β, and γ and added together to obtain the new fused feature ASFF-3: （2） in, This means that the (i, j) vector of the output feature is mapped to y between channels. l , This refers to the spatial importance weights of feature maps at three different levels up to level L. Using an additive approach, it's necessary to upsample or downsample the features at different levels and adjust the number of channels to ensure that the output feature sizes and channel numbers of levels 1 through 3 are the same. The weight parameters α, β, and γ are obtained by convolving the resized feature maps of levels 1 to 3 with a 1×1 convolution; and after passing through a concat layer, the weight parameters α, β, and γ are processed by a softmax function so that their ranges are all within [0,1] and their sum is 1. （3） in, This means that the (i, j) vector of the output feature is mapped to y between channels. l , This refers to the spatial importance weights of feature maps at three different levels up to level L, replacing the original model's loss function with a loss function designed using NWD (Non-Wide Difference of Interest) metric. （4） Where N p For the Gaussian distribution model of the prediction box P, N g The Gaussian distribution model is given for the GT box G; the gradients |P∩G|=0 and |P∩G|=P or G are provided based on the NWD loss. Step 4: Use the image data in the dataset as input to train the improved YOLOv7 model and obtain the trained student classroom behavior detection model. Step 5: Input the classroom scene images of the students to be detected into the trained model to obtain the students' behavior categories and confidence levels.

2. The student classroom behavior detection method based on improved YOLOv7 according to claim 1, characterized in that, Step 2 involves the following steps for image preprocessing and annotation: Step 2.1: Using the OpenCV library, preprocess the obtained student classroom behavior images by changing the brightness and contrast, removing the background, local images, smoothing, denoising, and merging the images to obtain student classroom behavior images. Step 2.2: Use the labelImg image annotation tool to annotate the actions of students in the obtained student classroom behavior images, and save the label information in a txt file with the same name as the image to obtain the student classroom behavior dataset; Step 2.3: Divide the student classroom behavior image dataset into a training dataset and a test dataset. All images and labeled labels are divided into the training set and the test set in an 8:2 ratio.

Images