Student behavior identification method, device and system based on improved YOLOv7 and readable storage medium

By improving the YOLOv7 model, combining the channel attention mechanism and switchable hollow convolution module, the accuracy and efficiency problems in student classroom behavior recognition are solved, and more efficient student behavior recognition is achieved, and teachers can better monitor students' learning status are supported. .

CN118942161APending Publication Date: 2024-11-12YUNNAN NORMAL UNIV

Patent Information

Application Number
CN202411287373.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-09-13
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

The existing technology has problems such as inaccurate identification, high calculation volume and mutual obstruction of students in the recognition of students' classroom behavior, which is difficult to effectively improve classroom efficiency.

Method used

The student behavior recognition method based on improved YOLOv7 is adopted, and the channel feature extraction is performed through the efficient layer aggregation network module integrating the channel attention mechanism, and combined with the switchable hollow convolution output head module to improve the feature representation, and the bounding box loss using the dynamic non-monotonic focusing mechanism Function, improve model generalization ability and recognition accuracy.

Benefits of technology

It improves the accuracy and accuracy of student behavior recognition, enhances the model's representation and robustness of the characteristics of the region of interest, improves the overall effect of object detection, and helps teachers monitor students' learning status more effectively.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118942161A_ABST
    Figure CN118942161A_ABST
Patent Text Reader

Abstract

The invention relates to a student behavior recognition method, device and system based on improved YOLOv7 and a readable storage medium, and belongs to the technical field of computer vision student behavior recognition. The method comprises the following steps: firstly, preprocessing a student classroom behavior data set to obtain preprocessed image data; secondly, inputting the preprocessed image data into a network optimized based on YOLOv7, performing channel feature extraction on the input data by fusing a channel attention mechanism into an efficient layer aggregation network module, and performing feature fusion on effective channel features and other multi-scale features to obtain three feature maps of different scales; thirdly, inputting the feature maps of different scales into an output head module based on switchable cavity convolution so as to effectively improve feature representation; and finally, inputting the improved feature representation into a student behavior output network detection head to obtain a student behavior identification result. The problem that a computer identifies classroom behaviors of students in a real teaching scene is solved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to a student behavior recognition method, device, system and readable storage medium based on improved YOLOv7, belonging to the technical field of computer vision student behavior recognition. Background Art

[0002] With the rapid development of computer technology, deep learning technology has been widely used in various fields. These technologies mainly use convolutional networks to classify or detect target features, especially in the task of student behavior recognition. At present, in the field of education, China has rapidly developed smart classrooms by citing deep learning technology. Student classroom behavior recognition can help teachers better understand the quality of student learning. As an important part of learning input, student classroom behavior recognition research has always attracted the attention of researchers. At present, research in this field is mainly divided into traditional classroom behavior recognition methods and deep learning-based classroom behavior recognition methods.

[0003] Traditional analysis of student classroom behavior is achieved through manual observation and recording, for example: ST classroom teaching analysis method, which evaluates students' learning effects by manually observing and recording students' classroom behaviors; Flanders interactive analysis system, which captures classroom dynamics and interactive patterns by coding and classifying the language communication behaviors between teachers and students in the classroom; in addition, there are interactive analysis systems based on information technology, which use improved quantitative analysis tools to conduct quantitative evaluation of classroom teaching and reflect the effectiveness of classroom teaching. These methods rely on human observation to analyze classroom teaching, and do not take advantage of technologies such as artificial intelligence and computer vision. They are time-consuming, labor-intensive and inefficient.

[0004] With the booming development of artificial intelligence, some scholars are trying to improve this situation with the help of artificial intelligence technology. Xie Wei et al. identified classic student classroom behaviors based on the improved YOWO algorithm, and optimized the network structure of the model to improve the recognition accuracy; Su Chao et al. added SENet to extract student targets based on Yolov3, and used lightweight Openpose to extract key points of human skeletons, and used ST-SVM to classify the extracted skeleton joint coordinates; Yan Xingya et al. used YOLOv5 with improved loss function and MobileNetV3 model to improve the backbone network, and proposed BetaPose lightweight posture recognition model, which achieved good results in crowded scenes.

[0005] At present, in the field of education, China has rapidly developed smart classrooms by citing deep learning technology. Although target detection technology is becoming more and more mature, there are still many challenges in the research of student classroom behavior recognition. Therefore, a student behavior recognition method based on improved YOLOv7 is needed to identify and analyze the main manifestations of student classroom behavior and improve classroom efficiency. Summary of the invention

[0006] The technical problem to be solved by the present invention is to provide a student behavior recognition method based on improved YOLOv7, aiming to solve the problem of computer recognition of student classroom behavior in real teaching scenarios.

[0007] The technical solution of the present invention is: a student behavior recognition method based on improved YOLOv7. First, a student classroom behavior data set is constructed, and the student classroom behavior data set is subjected to data division, data enhancement and other operations to obtain preprocessed image data; secondly, the preprocessed image data is input into a network optimized based on YOLOv7, and channel features are extracted from the preprocessed image data by integrating a channel attention mechanism into an efficient layer aggregation network module to obtain channel features, and the channel features are feature fused with other multi-scale features to obtain feature maps of three different scales; then, the feature maps of different scales are input into an output head module based on switchable dilated convolution to improve the feature representation before the classification result is generated; finally, the improved feature representation is input into a student behavior output network detection head to obtain a student behavior recognition result.

[0008] The specific steps are:

[0009] Step 1: Preprocessing stage;

[0010] Step 1.1: Preprocess the classroom teaching videos from multiple real-life scenarios in different schools and grades, and construct a student classroom behavior dataset through manual annotation.

[0011] Step 1.2: Crop the real classroom teaching video collected in Step 1.1 at preset intervals, such as every 1 second, and then extract single frame data of the video within these intervals.

[0012] Step 1.3: Label the single frame data obtained in Step 1.2 using the manual labeling tool LabelImg, mark the student's position and boundary with a rectangular box around the target, and obtain the student target boundary box.

[0013] Step 1.4: Classify and label the student behaviors in the target boundary box marked in Step 1.3 by presetting student behavior categories, such as writing, looking down, playing with mobile phones, standing, reading, and sleepiness, to obtain a data set composed of student behavior categories. The data set composed of student behavior categories is determined as the student classroom behavior data set.

[0014] Step 1.5: The student classroom behavior dataset obtained in Step 1.4 is subjected to preprocessing operations such as data segmentation, data enhancement, image standardization, and data format conversion to obtain preprocessed image data.

[0015] Step 2: Feature extraction and fusion stage;

[0016] Step 2.1: Input the preprocessed image data obtained in Step 1.5 into the network optimized based on YOLOv7, and integrate the channel attention mechanism into the high-efficiency layer aggregation network module, which can more effectively extract channel features from the preprocessed image data, thereby obtaining effective channel feature representation for subsequent student behavior recognition tasks.

[0017] Step 2.2: Fuse the channel features obtained in Step 2.1 with other multi-scale features to obtain a richer and more comprehensive feature representation and obtain feature maps of three different scales.

[0018] Step 3: Feature representation improvement stage;

[0019] Step 3.1: Input the feature maps of different scales obtained in Step 2.2 into the output head module based on switchable dilated convolution. This module is designed to effectively improve the feature representation and further optimize and integrate the features before the final classification result is generated.

[0020] Step 4: Student behavior identification stage;

[0021] Step 4.1: Input the feature representation improved by Step 3.1 into the student behavior output network detection head. This detection head is used for student behavior recognition. Its purpose is to accurately identify and classify different behaviors of students from the input feature representation.

[0022] In the Step 2.1, there are two efficient layer aggregation network modules that incorporate the channel attention mechanism, and the efficient layer aggregation network modules include multiple convolutional layers, batch normalization layers, and sliu activation function layers. By introducing this module, the network can more effectively integrate and utilize feature information of different scales and levels, thereby improving the performance and efficiency of the network in the student behavior recognition task.

[0023] In the Step 3.1, there are three output head modules of the switchable dilated convolution, which include a two-dimensional convolution layer with a self-attention mechanism and a detection head module. By introducing this module, the network can not only process feature information at different scales and spatial levels more flexibly, but also capture the key features of student behavior more accurately, thereby improving the performance and efficiency of the student behavior recognition task.

[0024] In the student behavior recognition method based on improved YOLOv7, in terms of loss function, the error of the center position of the student target bounding box is calculated by adopting a bounding box loss function based on a dynamic non-monotonic focusing mechanism, and the penalty of geometric factors is weakened when the anchor box and the target box coincide well, so that the model can obtain better generalization ability.

[0025] To achieve the above object, the present invention also provides a device for student behavior recognition method based on improved YOLOv7, the device comprising:

[0026] The feature map extraction module is used to extract channel features from the preprocessed image data obtained in Step 1.5 by integrating the channel attention mechanism into the efficient layer aggregation network module, and to fuse the channel features with other multi-scale features to obtain feature maps of three different scales.

[0027] Improve the feature representation module, which is used to input the feature maps of different scales obtained in Step 2.2 into the output head module based on switchable dilated convolution, and further optimize and integrate the features before the final classification result is generated;

[0028] The bounding box loss processing module calculates the error of the center position of the student target bounding box by adopting the bounding box loss function based on the dynamic non-monotonic focusing mechanism, weakens the penalty of geometric factors when the anchor box and the target box overlap well, so that the model can obtain better generalization ability;

[0029] The behavior recognition module is used to input the feature representation improved by Step 3.1 into the student behavior output network detection head to accurately identify and classify different behaviors of students.

[0030] In addition, to achieve the above-mentioned purpose, the present invention also provides a student behavior identification system, which includes a memory, a processor, and a student behavior identification program based on improved YOLOv7 stored in the memory and executable on the processor, and the student behavior identification program, when executed by the processor, implements the steps of the student behavior identification method described above.

[0031] In addition, to achieve the above-mentioned purpose, the present invention also provides a computer-readable storage medium, on which is stored a student behavior recognition program based on the improved YOLOv7, and when the student behavior recognition program is executed by a processor, the steps of the student behavior recognition method described above are implemented.

[0032] The beneficial effects of the present invention are as follows: by extracting channel features from preprocessed image data based on an efficient layer aggregation network module that incorporates a channel attention mechanism, the purpose of improving the model's attention to feature channels can be achieved; feature maps of different scales are input into an output head module based on a switchable dilated convolution, and feature representation is improved before classification results are generated, so as to enhance the model's feature representation and robustness for regions of interest and improve the overall effect of target detection; the error of the center position of the student target bounding box is calculated by adopting a bounding box loss function based on a dynamic non-monotonic focusing mechanism, and the penalty of geometric factors is weakened when the anchor box and the target box overlap well, so that the model obtains better generalization ability. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 It is a schematic diagram of the process of the present invention;

[0034] Figure 2 This is a schematic diagram of the structure of the efficient layer aggregation network module of the channel attention mechanism of the present invention;

[0035] Figure 3 A schematic diagram of the output head structure based on switchable dilated convolution according to the present invention;

[0036] Figure 4 It is a schematic diagram of the structure of the student behavior recognition device according to the present invention;

[0037] Figure 5 The present invention is a schematic diagram of the hardware environment of the student behavior recognition system. DETAILED DESCRIPTION

[0038] Although target detection technology is becoming more and more mature, there are still many challenges in the recognition of students' classroom behavior, such as the inaccurate recognition of students' classroom behavior caused by the high complexity of the target detection model, the high amount of model calculation, and the mutual occlusion between students. In order to identify and analyze the main manifestations of students' classroom behavior, so as to improve classroom efficiency. Therefore, the present invention mainly focuses on improving the recognition precision and accuracy of the student behavior recognition method.

[0039] Example 1: Figure 1 As shown in FIG. 1 , a student behavior recognition method based on improved YOLOv7 is shown in FIG. 1 . The specific steps are as follows:

[0040] Step 1: Preprocessing stage;

[0041] Step 1.1: Preprocess the classroom teaching videos from multiple real-life scenarios in different schools, subjects, and grades, and construct a student classroom behavior dataset through manual annotation.

[0042] Step 1.2: Set fixed time intervals in the real classroom teaching video collected in Step 1.1, such as every 1 second or every few seconds, and then extract single-frame data of the video within these intervals, and save the extracted single-frame data as image files.

[0043] Step 1.3: Label the single frame data obtained in Step 1.2 using the manual labeling tool LabelImg, mark the student's position and boundary with a rectangular box around the target, and obtain the student target boundary box to provide accurate location information for subsequent student behavior recognition.

[0044] Step 1.4: The student behaviors in the target boundary box marked in Step 1.3 are marked by preset student behavior classifications, such as writing, looking down, playing with mobile phones, standing, reading, and sleepiness, and the data set composed of the student behavior categories is determined as the student classroom behavior data set, which is the main data set of the present invention.

[0045] Step 1.5: The student classroom behavior dataset obtained in Step 1.4 is subjected to preprocessing operations such as data segmentation, data enhancement, image standardization, and data format conversion to obtain preprocessed image data, which provides more training data for subsequent student behavior recognition tasks.

[0046] Step 2: Feature extraction and fusion stage;

[0047] Step 2.1: Input the preprocessed image data obtained in Step 1.5 into the network optimized based on YOLOv7. By integrating the channel attention mechanism into the efficient layer aggregation network module, the channel features of the preprocessed image data can be extracted more effectively, thereby obtaining effective channel feature representation for subsequent student behavior recognition tasks.

[0048] Step 2.2: The channel features obtained in Step 2.1 are fused with other multi-scale features to obtain a richer and more comprehensive feature representation and obtain feature maps of three different scales. Among them, other multi-scale features include subtle textures, local structures, overall structures, and semantic information of the image.

[0049] Step 3: Feature representation improvement stage;

[0050] Step 3.1: Input the feature maps of different scales obtained in Step 2.2 into the output head module based on switchable dilated convolution. This module is designed to effectively improve the feature representation and further optimize and integrate the features before the final classification result is generated.

[0051] Step 4: Student behavior identification stage;

[0052] Step 4.1: Input the feature representation improved by Step 3.1 into the student behavior output network detection head. This detection head is used for student behavior recognition. Its purpose is to accurately identify and classify different behaviors of students from the input feature representation, so as to obtain the student behavior recognition result.

[0053] Furthermore, in Step 2.1, there are two efficient layer aggregation network modules that incorporate the channel attention mechanism, and the efficient layer aggregation network modules include multiple convolutional layers, batch normalization layers, and sliu activation function layers. By introducing this module, the network can more effectively integrate and utilize feature information of different scales and levels, thereby improving the performance and efficiency of the network in the student behavior recognition task. Among them, the formula of the attention mechanism mentioned in Step 2.1 is:

[0054] (1)

[0055] (2)

[0056] (3)

[0057] (4)

[0058] In the above formula (1), for an input feature map X, its size is usually H*W*C, where H and W are the height and width of the spatial dimension, respectively, and C is the number of channels. In formulas (2) and (3), two fully connected layers are applied to the newly generated feature vector Z, and nonlinear activation functions ReLU and Sigmoid are incorporated respectively. and are the weight matrices of the two fully connected layers, and is the corresponding bias term. In formula (4), The output of the network module is aggregated for the efficient layer;

[0059] Furthermore, in Step 3.1, there are three output head modules of the switchable dilated convolution, which include a two-dimensional convolution layer with a self-attention mechanism and a detection head module. By introducing this module, the network can not only process feature information at different scales and spatial levels more flexibly, but also capture the key features of student behavior more accurately, thereby improving the performance and efficiency of the student behavior recognition task.

[0060] Among them, the formula of the switchable hole convolution mentioned in Step 3.1 is:

[0061] (5)

[0062] (6)

[0063] In the above formula (5), x is the input feature map, w is the weight, r is the hyperparameter of the switchable dilated convolution, and y is the output feature map. In formula (6), are trainable weights, It is a switch function, which consists of an average pooling layer and a convolutional layer. The symbol indicates substitution.

[0064] Based on the above method, a comparative test was conducted, and the experimental results are shown in Table 1:

[0065] Table 1: Comparison results of different models

[0066]

[0067] As can be seen from Table 1, both the accuracy and average precision have been greatly improved, providing data support for the effectiveness of the present invention. The present invention can help schools and teachers understand the listening status and efficiency of students and provide technical support for education quality assessment.

[0068] Figure 1The overall architecture of the classroom student behavior recognition method based on improved YOLOv7 is shown. The input is the student classroom behavior dataset and the output is the student behavior category. The student classroom behavior dataset is preprocessed by data partitioning, data enhancement, image standardization and data format conversion to obtain preprocessed image data. The preprocessed image data is input into the network optimized based on YOLOv7. By integrating the channel attention mechanism into the efficient layer aggregation network module, the channel features are extracted to obtain the channel features of the image. The channel features of the image are fused with other multi-scale features to obtain feature maps of three different scales. The feature maps of different scales are input into the output head module based on switchable dilated convolution, and the features are further optimized and integrated before the final classification result is generated. The improved feature representation is input into the student behavior output network detection head to obtain the student behavior recognition result.

[0069] Figure 2 The schematic diagram of the structure of the efficient layer aggregation network module of the channel attention mechanism of the present invention is shown. Channel feature extraction is performed on preprocessed image data to achieve the purpose of improving the model's attention to feature channels and improving the feature expression ability. The channel attention mechanism is integrated into the ELAN module, and the channels are weighted by the attention mechanism module to strengthen the expression of target features, helping the YOLOv7 model to flexibly adjust the focus of features and enhance the robustness of the model. The attention mechanism module can also fine-tune the feature hierarchy to improve the model's attention to feature channels.

[0070] Figure 3 The present invention shows a schematic diagram of the output head structure based on switchable dilated convolution. The switchable dilated convolution can flexibly capture target features at different scales through switchable dilated rates, which helps to improve the detection capability of targets of various sizes. A switchable dilated convolution module is introduced into the output head module of the network to improve the feature representation before the final result is generated, so as to enhance the feature representation and robustness of the model for the region of interest. The switchable dilated convolution can also expand the receptive field without increasing the amount of calculation, which helps to improve the positioning accuracy of the bounding box, especially when dealing with dense scenes such as student classrooms.

[0071] Figure 4The present invention shows a schematic diagram of the architecture of a student behavior recognition device, which includes: a feature map extraction module, an improved feature representation module, a bounding box loss processing module and a behavior recognition module; wherein the feature map extraction module is used to extract channel features from a preprocessed classroom student behavior data set based on an efficient layer aggregation network module that incorporates a channel attention mechanism, and obtain feature maps of three different scales; the improved feature representation module is used to input each feature map into an output head based on a switchable dilated convolution, and improve the feature representation before the final result is generated; the bounding box loss processing module is used to process the error of the center position of the student behavior bounding box using a bounding box loss function, and improve the generalization ability and overall performance of the network model; the behavior recognition module is used to input the feature representation into a student behavior classification output network, so as to determine the student behavior recognition result through the classification output network.

[0072] Figure 5 The schematic diagram of the hardware environment of the student behavior recognition system of the present invention is shown, and the student behavior recognition system includes: a processor 1001, a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. The processor 1001 can be used to call the student behavior recognition program based on the improved YOLOv7 stored in the memory 1005; the communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 is mainly used to connect to the terminal and communicate data with the terminal; the network interface 1004 is mainly used for the background server and communicates data with the background server; the memory 1005 may include an operating system, a network communication module, a user interface module, and a student behavior recognition program based on the improved YOLOv7. Figure 5 The architecture of the student behavior recognition system shown in the figure does not constitute a limitation on the student behavior recognition system, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0073] In addition, the present invention also provides a computer-readable storage medium, which stores a student behavior identification program based on an improved YOLOv7. When the student behavior identification program based on the improved YOLOv7 is executed by a processor, the various steps of the student behavior identification method described in the above embodiment are implemented.

[0074] The specific implementation modes of the present invention are described in detail above in conjunction with the accompanying drawings, but the present invention is not limited to the above implementation modes, and various changes can be made within the knowledge scope of ordinary technicians in this field without departing from the purpose of the present invention.

Claims

1. A student behavior recognition method based on improved YOLOv7, characterized in that: The method comprises: Step 1: Construct a student classroom behavior dataset, preprocess the student classroom behavior dataset, and obtain preprocessed image data; Step 2: Input the preprocessed image data into the network module optimized based on YOLOv7, and extract the channel features of the preprocessed image data by integrating the channel attention mechanism into the high-efficiency layer aggregation network module to obtain the channel features; The channel features are fused with other multi-scale features to obtain feature maps of three different scales; Step 3: Input feature maps of different scales into the output head module based on switchable dilated convolution to improve feature representation before classification results are generated; Step 4: Input the improved feature representation into the student behavior output network detection head to obtain the student behavior recognition result.

2. The student behavior recognition method based on improved YOLOv7 according to claim 1, characterized in that: The Step 1 is specifically as follows: Step 1.1: Preprocess the collected classroom teaching videos in multiple real scenarios and construct a student classroom behavior dataset through manual annotation; Step 1.2: The collected classroom teaching videos are cropped at preset intervals to extract single-frame data of the video; Step 1.3: Use the manual labeling tool LabelImg to label the extracted single frame data and obtain the student target bounding box; Step 1.4: Classify and label the student behaviors in the student target boundary box by presetting the behavior categories, obtain a data set consisting of the student behavior categories, and determine the data set as the student classroom behavior data set; Step 1.5: Preprocess the student classroom behavior dataset to obtain preprocessed image data.

3. The student behavior recognition method based on improved YOLOv7 according to claim 2 is characterized in that: The preprocessing includes data segmentation, data enhancement, image normalization and data format conversion.

4. The student behavior recognition method based on improved YOLOv7 according to claim 1, characterized in that: The efficient layer aggregation network module includes multiple convolutional layers, batch normalization layers and sliu activation function layers.

5. The student behavior recognition method based on improved YOLOv7 according to claim 2, characterized in that: In the student target bounding box, a bounding box loss function based on a dynamic non-monotonic focusing mechanism is used to calculate the error of the center position of the student target bounding box.

6. A device for implementing the student behavior recognition method based on improved YOLOv7 as claimed in claim 1, characterized in that: The device comprises: The feature map extraction module is used to extract channel features from preprocessed image data by integrating the channel attention mechanism into the efficient layer aggregation network module, and to fuse the channel features with other multi-scale features to obtain feature maps of three different scales. Improved feature representation module, used to input feature maps of different scales into the output head module based on switchable dilated convolution to improve feature representation; The bounding box loss processing module is used to calculate the error of the center position of the student target bounding box using a bounding box loss function based on a dynamic non-monotonic focusing mechanism to improve the generalization ability of the model; The behavior recognition module is used to input the improved feature representation into the student behavior output network detection head to obtain the student behavior recognition result.

7. A student behavior recognition system, characterized in that: The system includes: a memory, a processor, and a student behavior recognition program based on improved YOLOv7 stored in the memory and executable on the processor. When the student behavior recognition program is executed by the processor, the steps of the student behavior recognition method as described in any one of claims 1 to 5 are implemented.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a student behavior recognition program based on improved YOLOv7, and when the student behavior recognition program is executed by the processor, the steps of the student behavior recognition method as described in any one of claims 1 to 5 are implemented.

Citation Information

Patent Citations

  • Student behavior identification method, device and system and computer readable storage medium

    CN117152846A

  • Light-weight hot-rolled strip steel surface defect detection method based on improved YOLOv7

    CN117274233A

  • Remote sensing satellite image target detection method based on deep learning

    CN117456376A

  • High-speed rail overhead line system bolt defect state detection method

    CN118071976A

  • Insulator defect detection method

    CN118154565A

Cited By

  • Classroom scene student behavior intelligent identification method and system based on computer vision

    CN120071061A