Multi-scale feature fusion classroom behavior detection method and system

Abstract

The present invention discloses a multi-scale feature fusion classroom behavior detection method and system, which realizes accurate detection of students' classroom behavior based on an improved YOLOv5 detection model, including: 1) constructing a student classroom behavior data set, dividing a training set, a validation set and a test set, and annotating and labeling the divided data; 2) performing data enhancement on the training set; 3) clustering the training set with a priori frames; 4) using the training set to train the improved YOLOv5 detection model; 5) decoding the prediction results obtained in the forward propagation of the improved YOLOv5 detection model; 6) using CIOU's non-maximum suppression to screen the overlapping parts of the prediction frame, retaining the prediction frame with the highest confidence; 7) drawing the obtained prediction frame on the original picture, and marking the prediction category information in the upper left corner area of ​​the prediction frame. The present invention can help teachers better understand students' classroom learning, thereby improving the teaching quality of the classroom, improving students' learning efficiency, and helping to promote the development of smart education.

Description

Technical Field

[0001] The present invention relates to the technical field of smart education and intelligent monitoring, and in particular to a multi-scale feature fusion classroom behavior detection method and system based on an improved YOLOv5 detection model, which can be applied to schools and tutoring institutions for real-time monitoring, detection and analysis of students' classroom behavior. Background Art

[0002] With the development of deep learning and the advancement of science and technology, artificial intelligence has penetrated into research in various fields. Under the influence of the artificial intelligence craze, all walks of life are gradually moving towards the research and development of artificial intelligence. How to combine artificial intelligence technology with classroom teaching tasks, improve the quality and efficiency of classroom teaching, and promote the development of smart education is the main theme of the future education industry. It is particularly important to improve the quality of students' performance in the classroom. Through the participation of artificial intelligence, students' movements and behaviors can be intelligently identified and their concentration can be evaluated and scored, so as to improve the quality of students' performance in the classroom in a targeted manner. It is very meaningful to observe and evaluate the quality of students' performance in the classroom. How to more efficiently identify and analyze students' classroom behavior has become a research hotspot in smart education.

[0003] Classroom behavior analysis aims to study the internal mechanism of teachers' teaching activities and students' academic development in the classroom, help teachers and students reflect on their own classroom performance, and thus promote the improvement of classroom teaching quality and effectiveness. Traditional classroom teaching behavior analysis mostly collects and analyzes data through self-evaluation, manual supervision, classroom inspections and other methods. It has the disadvantages of strong subjectivity of personal evaluation, small sample size, time-consuming and labor-intensive, resulting in low interpretability and scalability. The popularization of artificial intelligence technology has brought opportunities to improve these shortcomings.

[0004] Using artificial intelligence technology to collect and analyze data can identify classroom behaviors more comprehensively and timely, understand the teaching status of classroom teachers and students, and provide strong support for improving teaching quality. Summary of the invention

[0005] The first purpose of the present invention is to overcome the shortcomings and deficiencies of the prior art and provide a multi-scale feature fusion classroom behavior detection method based on an improved YOLOv5 detection model. The method mainly utilizes the attention mechanism and feature enhancement fusion to extract local features of students' classroom behavior, thereby realizing the recognition of students' performance in the classroom in the video sequence. It can help teachers better understand students' classroom learning situation, thereby improving teachers' teaching quality, improving students' learning efficiency, and helping to promote the development of smart classrooms.

[0006] The second object of the present invention is to provide a multi-scale feature fusion classroom behavior detection system based on an improved YOLOv5 detection model.

[0007] The first purpose of the present invention is achieved through the following technical solutions: a multi-scale feature fusion classroom behavior detection method, which is based on an improved YOLOv5 detection model to achieve accurate detection of student classroom behavior. The improved YOLOv5 detection model is an improvement on the backbone network module, the pooling pyramid module and the prediction module of the original YOLOv5 detection model. Among them, the improvement of the backbone network module is: using the CBAM attention module to enhance the features extracted by the backbone network; the improvement of the pooling pyramid module is: introducing the CSPC structure into the original pooling pyramid to obtain speed improvement while keeping the receptive field unchanged; the improvement of the prediction module is: adding a multi-scale generalization detection module SCAM before each prediction module outputs the feature to improve the accuracy of small target detection;

[0008] The specific implementation of the multi-scale feature fusion classroom behavior detection method includes the following steps:

[0009] 1) Preliminary data collection and construction: by collecting multiple campus classroom surveillance, the obtained pictures are used as sample data sets of students' classroom behavior. The sample data sets are divided into training sets, validation sets, and test sets, and the divided samples are annotated and labeled;

[0010] 2) Perform data enhancement on the images in the training set according to the behavioral characteristics of students in the classroom, randomly adjust the brightness, contrast and saturation of the images in the training set, randomly crop, randomly flip and mirror, and perform Mosaic enhancement on the images in the training set to expand the data volume and improve the generalization ability of the model;

[0011] 3) According to the characteristics of students’ actions and position targets in the processed training set pictures, the k-means algorithm based on IOU is used to cluster the prior frames of students’ classroom behavior targets, and 9 clustering prior frames of real student behavior targets are obtained;

[0012] 4) Inputting the images in the training set processed by step 2) into the improved YOLOv5 detection model for model training, using the value of the prior box obtained in step 3) as the prior value of the improved YOLOv5 detection model, training according to a preset number of training iterations epochs, saving the model parameters of the improved YOLOv5 detection model during training, and using the model parameters with the smallest loss in the validation set as the best model parameters of the improved YOLOv5 detection model;

[0013] 5) Improve the YOLOv5 detection model, load the best model parameters obtained in step 4), input the pictures in the test set, decode the prediction results obtained in the forward propagation of the model, obtain the confidence of detecting students' classroom behaviors in all prediction boxes, the probability of detection categories, and the position adjustment parameters of the prediction boxes, and select the prediction boxes with confidence greater than the preset threshold for retention;

[0014] 6) Further screening the retained prediction boxes, using CIOU-based non-maximum suppression to screen the overlapping boxes in the prediction boxes. When the CIOU value in the overlapping box is greater than the preset threshold, only the prediction box with a large confidence is retained;

[0015] 7) Draw the final prediction box on the original image, and mark the prediction category information in the upper left corner of the prediction box to complete the detection of students' classroom behavior.

[0016] Furthermore, in step 1), data is collected by means of camera shooting and monitoring recording to collect students' different classroom behaviors in the classroom. The action video of each behavior is collected continuously for 10 to 20 seconds, and multiple frames of pictures are randomly selected for each behavior from different shooting angles. The pictures finally obtained are used as sample data sets of students' classroom behaviors. The obtained pictures with students' classroom actions are annotated by the labelimg annotation tool, and rectangular boxes are used to mark the students in the pictures and the positions of the actions made by the students, and the annotated information is classified and integrated into training label data.

[0017] Furthermore, in step 2), in view of the complex classroom scene with dense crowds, the images in the training set are data enhanced, and the brightness, contrast and saturation of the images in the training set are changed to increase the diversity of the training set; in view of the diverse characteristics of students' classroom behaviors, the images in the training set are randomly cropped and Mosaic data enhanced; in view of the changeable environment and noise in the classroom, the images in the training set are expanded by using random flipping and mirroring.

[0018] Further, in step 3), the training label data is input into the k-means algorithm, n cluster centers are set, n prior boxes are randomly selected from the training label data as initial cluster centers, the Euclidean distance from each training label data to the current existing cluster center is calculated, the training label data with the shortest distance is classified into a cluster cluster, and the next cluster center is selected; the above steps are repeated until the values ​​of all cluster centers no longer change. When the number of cluster centers reaches n, the maximum number of iterations is obtained, and each training label data and the nearest cluster center are classified into one category, and finally 9 prior boxes of different sizes are obtained.

[0019] Further, in step 4), the backbone network module of the improved YOLOv5 detection model is CSPDarknet53, and CSPDarknet53 is used to extract features. The backbone network module reads the input image information, performs 5 downsampling, obtains feature information of targets of different sizes, and adds a CBAM attention module after BottleNeckCSP of the backbone network. The CBAM attention module is composed of a channel attention module CAM and a spatial attention module SAM connected in series;

[0020] CAM processes the input feature map through global maximum pooling and global average pooling, and then passes through two fully connected layers. The obtained features are element-wise added and then activated by sigmoid to obtain the final channel attention feature map. The input feature map and the channel attention feature map are element-wise multiplied to obtain the feature map output by CAM. The process is shown in the following formula (1):

[0021] CAM(F)=σ(MLP(AvgPool(F))+MLP(MaxPool(F))) (1)

[0022] In the formula, CAM(F) represents the feature map output by CAM, F represents the input feature map, MLP represents the fully connected layer, AvgPool represents the global average pooling, MaxPool represents the global maximum pooling, and σ represents the sigmoid activation function;

[0023] SAM concatenates the input feature map after channel-based global maximum pooling and global average pooling, and then generates a spatial attention feature map through a 7×7 convolution layer and a sigmoid activation function. The final output feature map is obtained by multiplying it with the input feature map. The process is shown in the following formula (2):

[0024] SAM(F)=σ(k 7 ([AvgPool(F)⊕MaxPool(F)])) (2)

[0025] In the formula, SAM(F) represents the feature map output by SAM, F represents the input feature map, AvgPool represents the global average pooling, MaxPool represents the global maximum pooling, ⊕ represents the concat operation, and k 7 represents a 7×7 convolution operation, and σ represents a sigmoid activation function;

[0026] The pooling pyramid module of the improved YOLOv5 detection model is to pass the feature map output by the backbone network module into the CSPC-SPPF pooling pyramid. The CSPC-SPPF pooling pyramid is divided into a four-layer architecture, and its specific situation is as follows:

[0027] The first layer is a series of 1×1 convolution operations, 3×3 convolution operations, and 1×1 convolution operations, which are used to extract the size of the input feature map and enhance the image pixels;

[0028] The second layer is the SPPF spatial pooling pyramid. The SPPF spatial pooling pyramid is composed of a multi-level pooling sliding window. The sliding window consists of three global maximum pooling convolutions with a kernel size of 5×5.

[0029] The third layer is a series of 1×1 convolution operations, 3×3 convolution operations, and 1×1 convolution operations, which are used to reduce the dimension of the feature map extracted by the SPPF spatial pooling pyramid;

[0030] The fourth layer is CSPC feature enhancement, which jumps the feature map output by the backbone network module and fuses it with the feature map output by the third layer through a 1×1 convolution operation. Using the CPSC structure to fuse features can not only reduce the amount of calculation by half, but also speed up the detection and improve the accuracy.

[0031] The prediction module of the improved YOLOv5 detection model adds a multi-scale generalization detection module SCAM before outputting the features of the detection target to improve the accuracy of the detection target. The multi-scale generalization detection module SCAM is divided into a three-layer architecture, the details of which are as follows:

[0032] The first layer is composed of 3×3 dilated convolutions with a dilation rate of 1, 3×3 dilated convolutions with a dilation rate of 2, and 3×3 dilated convolutions with a dilation rate of 3, which are fused in parallel to form a new feature map, thus forming a multi-scale feature pyramid structure to increase the receptive field of the output feature map;

[0033] The second layer uses the SE attention mechanism to enhance the features of the feature map of the fusion output of the first layer. The SE attention mechanism consists of a global average pooling, a fully connected layer and a Sigmoid activation function. The feature map of the fusion output of the first layer is aggregated using the SE attention mechanism to form a local context feature enhancement.

[0034] The third layer performs a dot multiplication operation on the feature map output by the second layer and the original feature map to obtain the final output global context feature map;

[0035] The head scales of the detected targets output by the multi-scale generalized detection module SCAM are three YOLO detection heads with 13×13, 26×26, and 52×52 scales respectively.

[0036] Furthermore, the SE attention mechanism is divided into a three-layer architecture, the details of which are as follows:

[0037] The first layer performs a global average pooling operation on the feature map output by the parallel fusion, compressing the global spatial information into the channel local factor, thereby enhancing the aggregation of spatial information. The process is shown in the following formula (3):

[0038]

[0039] In the formula, i represents the pixel of the feature map height, j represents the pixel of the feature map width, H represents the height of the feature map, W represents the width of the feature map, and x c Represents the feature map of parallel fusion output, Z c Represents the feature map x c The feature map after performing H×W global average pooling in the spatial dimension;

[0040] The second layer is for Z c Perform standard normalization operation, perform ReLU activation on the normalized linear features, then perform full connection operation on the feature map, and finally obtain normalized weights through Sigmoid activation function. Its function is to fully capture the channel dependency, learn the nonlinear relationship between channels, and enable the fully connected layer and nonlinear activation function to establish the connection between channels. The process is shown in the following formula (4):

[0041] S c =σ(W 2 δ(W 1 Z c )) (4)

[0042] Where W 1 Represents Z c The weight parameter of the feature map, σ represents the sigmoid activation function, δ represents the relu activation function, W 2 represents the weight parameter of nonlinear mapping, S c The normalized weights obtained for the activation function;

[0043] The third layer is Z c The global average coefficient and the normalized weight S c Multiply them, and get the spatial local feature factor coefficients by multiplication, and weight them channel by channel to each space of the original feature map, so as to complete the recalibration of the original features by channel attention, thereby enhancing the global receptive field of the feature map. The process is shown in the following formula (5):

[0044] F c =Z c *S c (5)

[0045] In the formula, F c is the spatial local characteristic factor coefficient.

[0046] Further, in step 4), the images in the training set processed by step 2) are input into the improved YOLOv5 detection model for training, the training parameters are set, and the coco pre-trained weights are loaded for training. During the training process, the number of training batch samples batchsize is set to 64, the initial learning rate is set to 0.001, and the training is performed for 300 epochs; the training optimizer uses the Adam optimizer; after the training is completed, the model parameters with the smallest loss in the validation set are selected from the saved training parameters as the optimal model parameters of the detection model;

[0047] In step 5), the improved YOLOv5 detection model is loaded with the best model parameters and the pictures in the test set are input to obtain the prediction results of the three YOLO detection heads. The predicted results are decoded to obtain the confidence of the predicted targets contained in all prediction boxes, the probability of the type of the target contained in the prediction box, and the position adjustment parameters of the prediction box; only the prediction boxes with confidence greater than the preset threshold are retained.

[0048] Furthermore, in step 6), the prediction boxes are further screened to prevent multiple prediction boxes from repeatedly detecting the same target; the prediction results of the three YOLO detection heads are sorted according to the scores of all student classroom behavior prediction boxes, the highest score and its corresponding prediction box are selected, and the overlapping boxes are removed using CIOU-based non-maximum suppression. The CIOU is calculated between all prediction boxes of the same category. When the CIOU between two boxes is greater than the preset threshold, the box with a larger confidence is retained as the final prediction box, and the box with a smaller confidence is discarded.

[0049] Further, in step 7), the result obtained by the final prediction box detection is drawn on the original picture, the coordinate position information of the student is located on the prediction box, and the prediction category information of the prediction box is marked in the upper left corner area, thereby completing the detection of the student's classroom behavior.

[0050] The second object of the present invention is achieved by the following technical solution: a multi-scale feature fusion classroom behavior detection system, which is used to implement the above-mentioned multi-scale feature fusion classroom behavior detection method, and comprises:

[0051] The data acquisition and processing module collects multiple campus classroom surveillance images as sample data sets of students' classroom behavior, divides the sample data sets into training sets, validation sets and test sets, and annotates and labels the divided samples; performs data enhancement processing on the images in the training set according to the behavioral characteristics of students in the classroom, randomly adjusts the brightness, contrast and saturation of the images in the training set, randomly crops, randomly flips and mirrors, and performs Mosaic enhancement on the images in the training set to expand the data volume and improve the generalization ability of the model; according to the characteristics of the students' actions and position targets in the processed training set images, the k-means clustering algorithm based on IOU is used to cluster the prior frames of the students' classroom behavior targets, and 9 clustering prior frames of the real student behavior targets are obtained;

[0052] Improve the YOLOv5 detection model, use the attention mechanism and feature enhancement fusion to extract local features of students' classroom behavior, and recognize students' performance in class in the video sequence;

[0053] A training module is used to input the images in the training set processed by the data acquisition and processing module into the improved YOLOv5 detection model for model training, use the value of the prior box obtained in the data acquisition and processing module as the prior value of the improved YOLOv5 detection model, perform training according to a preset number of training iterations epoch, save the model parameters of the improved YOLOv5 detection model during training, and use the model parameters with the smallest loss in the validation set as the best model parameters of the improved YOLOv5 detection model;

[0054] The student classroom behavior detection module is used to load the improved YOLOv5 detection model with the best model parameters obtained by the training module and input the pictures in the test set, decode the prediction results obtained in the forward propagation of the model, obtain the confidence, detection category probability, and position adjustment parameters of the prediction boxes for detecting student classroom behaviors in all prediction boxes, and screen out the prediction boxes with confidence greater than a preset threshold for retention; further screen the retained prediction boxes, and use CIOU-based non-maximum suppression to screen the overlapping boxes in the prediction boxes. When the CIOU value in the overlapping box is greater than the preset threshold, only the prediction box with a large confidence is retained; the final prediction box is drawn on the original picture, and the prediction category information is marked in the upper left corner of the prediction box, thereby completing the detection of student classroom behavior.

[0055] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0056] 1. The core of the present invention is to analyze the classroom behavior of students transmitted by the camera and analyze the students' class situation. The core is to improve the multi-scale feature fusion method of the YOLOv5 detection model, enhance the extraction of action features, and use multi-scale features to improve the spatial features and fusion efficiency of the fused model. The model has the expressiveness of multi-scale contextual information.

[0057] 2. The present invention adds a CBAM attention module to the original backbone network. The channel space attention mechanism (channel attention module CAM and spatial attention module SAM) of the CBAM attention module can effectively enhance the feature diversity of the input feature map and improve the training speed and accuracy of the model. At the same time, using attention enhancement on channels and spaces can improve the feature expression ability of the network. Among them, the channel attention module CAM can determine the weight increase in the feature map of multiple channels, and the spatial attention module SAM can determine the increase of the eigenvalue weight in the spatial domain of the feature map so that the result of model training is more accurate.

[0058] 3. The CSPC-SPPF pooling pyramid proposed in the present invention uses a multi-level pooling sliding window. The multi-layer pooling sliding window has a strong robustness feature for image enhancement; it improves the flexibility of the image input scale and extracts pooling features under variable scales. At the same time, using CSPC feature enhancement can reduce the amount of calculation by half, making the speed faster and the accuracy improved.

[0059] 4. The multi-scale generalization detection module SCAM proposed in the present invention can increase the receptive field of features and effectively improve the effect of the algorithm on student action detection. It has a simple structure and a small number of parameters. The multi-scale generalization detection module SCAM is set before the network detection head and can perform multi-scale processing on features after feature fusion. It is a feature enhancement method. The advantages of introducing the multi-scale generalization detection module SCAM are: using dilated convolution to increase the receptive field; using a multi-scale feature pyramid structure to increase the receptive field.

[0060] The present invention extracts local features of students' actions and behaviors based on attention mechanism and feature enhancement fusion, and semantically annotates students' actions and behaviors, thereby analyzing and evaluating students' performance in class in video time sequence. Through the present invention, real-time target detection is combined with video behavior understanding, which can help teachers better understand students' classroom learning situation, thereby improving teachers' teaching quality, improving students' learning efficiency, and promoting the development of smart classrooms.

[0061] Based on the above discussion, the invention of a classroom behavior detection method and system that meets high precision and real-time requirements has high practical application value. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 Flow chart of the method of the present invention.

[0063] Figure 2 Structural diagram for improving the YOLOv5 detection model.

[0064] Figure 3 This is the structural diagram of the CBAM attention module.

[0065] Figure 4 This is the structural diagram of the CSPC-SPPF pooling pyramid.

[0066] Figure 5 This is the structural diagram of the multi-scale generalization detection module SCAM.

[0067] Figure 6 Schematic diagram of the system of the present invention. DETAILED DESCRIPTION

[0068] The present invention is further described in detail below in conjunction with embodiments and drawings, but the embodiments of the present invention are not limited thereto.

[0069] Example 1

[0070] This embodiment discloses a multi-scale feature fusion classroom behavior detection method based on an improved YOLOv5 detection model, and the specific situation is as follows:

[0071] 1) Establish an efficient dataset of student classroom behavior in real scenarios. Through the participation of multiple volunteers, volunteers will perform corresponding actions according to the behaviors that occur in real classroom scenarios, such as playing with mobile phones, listening to lectures, sleeping, etc. Data collection is carried out by camera shooting and monitoring recording, and students perform different classroom behaviors in the classroom. Each action video is collected for 10 to 20 seconds, and the video stream data is converted into image frames. 25 frames of images are randomly selected for each behavior from different shooting angles, and a total of 300 images for each student are finally obtained as a sample dataset of student classroom behavior. The obtained pictures with student classroom actions are annotated by the labelimg annotation tool, and the students in the picture and the positions of the actions issued by the students are annotated with rectangular boxes, and the annotated information is classified and integrated into training label data. The dataset is divided into training set, validation set and test set.

[0072] 2) In view of the complex classroom scenes with dense crowds, the images in the training set are enhanced, and the brightness, contrast and saturation of the images in the training set are changed to increase the diversity of the training set. In view of the diverse characteristics of students' classroom behaviors, the images in the training set are randomly cropped and Mosaic data enhanced. In view of the changeable environment and noise in the classroom, the images in the training set are expanded by randomly flipping the mirror.

[0073] 3) Input the training label data into the k-means algorithm, set n cluster centers, randomly select n prior boxes from the training label data as the initial cluster centers, calculate the Euclidean distance from each training label data to the current existing cluster center, classify the training label data with the shortest distance into a cluster cluster, and select the next cluster center; repeat the above steps until the values ​​of all cluster centers no longer change. When the number of cluster centers reaches n, obtain the maximum number of iterations, classify each training label data and the nearest cluster center into one category, and finally obtain 9 prior (anchor) boxes of different sizes.

[0074] 4) Construct an improved YOLOv5 detection model, which is used to efficiently detect student classroom behavior. The improved YOLOv5 detection model improves the backbone network module, pooling pyramid module and prediction module of the original YOLOv5 detection model. The improvement to the backbone network module is: using the CBAM attention module to enhance the features extracted by the backbone network; the improvement to the pooling pyramid module is: introducing the CSPC structure to the original pooling pyramid to increase the speed while keeping the receptive field unchanged; the improvement to the prediction module is: adding a multi-scale generalization detection module SCAM before each prediction module outputs features to improve the accuracy of small target detection.

[0075] The improved YOLOv5 detection model structure is as follows Figure 2 As shown in Figure 1, the backbone network module is CSPDarknet53, which is used to extract features, and the CBAM attention module is added after BottleNeckCSP of the backbone network. The structure of the CBAM attention module is as follows: Figure 3 As shown, the CBAM attention module is composed of a channel attention module CAM and a spatial attention module SAM connected in series.

[0076] CAM processes the input feature map through global maximum pooling and global average pooling, and then passes through two fully connected layers. The obtained features are element-wise added and then activated by sigmoid to obtain the final channel attention feature map. The input feature map and the channel attention feature map are element-wise multiplied to obtain the feature map output by CAM. The process is shown in the following formula (1):

[0077] CAM(F)=σ(MLP(AvgPool(F))+MLP(MaxPool(F))) (1)

[0078] In the formula, CAM(F) represents the feature map output by CAM, F represents the input feature map, MLP represents the fully connected layer, AvgPool represents the global average pooling, MaxPool represents the global maximum pooling, and σ represents the sigmoid activation function.

[0079] SAM concatenates the input feature map after channel-based global maximum pooling and global average pooling, and then generates a spatial attention feature map through a 7×7 convolution layer and a sigmoid activation function, and then multiplies it with the input feature map to obtain the final output feature map. The process is shown in the following formula (2):

[0080] SAM(F)=σ(k 7 ([AvgPool(F)⊕MaxPool(F)])) (2)

[0081] In the formula, SAM(F) represents the feature map output by SAM, F represents the input feature map, AvgPool represents the global average pooling, MaxPool represents the global maximum pooling, ⊕ represents the concat operation, and k 7 represents a 7×7 convolution operation, and σ represents the sigmoid activation function.

[0082] The pooling pyramid module passes the feature map output by the backbone network module into the CSPC-SPPF pooling pyramid. Figure 4 As shown, the CSPC-SPPF pooling pyramid is divided into four layers of architecture. The specific details of the CSPC-SPPF pooling pyramid are as follows:

[0083] The first layer is a series of 1×1 convolution operations, 3×3 convolution operations, and 1×1 convolution operations, which are used to extract the size of the input feature map and enhance the image pixels.

[0084] The second layer is the SPPF spatial pooling pyramid. The SPPF spatial pooling pyramid is composed of a multi-level pooling sliding window. The sliding window consists of 3 global maximum pooling convolutions with a kernel size of 5×5.

[0085] The third layer is a series of 1×1 convolution operations, 3×3 convolution operations, and 1×1 convolution operations, which are used to reduce the dimension of the feature map extracted by the SPPF spatial pooling pyramid.

[0086] The fourth layer is CSPC feature enhancement, which jumps the feature map output by the backbone network module and then fuses it with the feature map output by the third layer through a 1×1 convolution operation. Using the CPSC structure to fuse features can not only reduce the amount of calculation by half, making the detection speed faster, but also improve the accuracy.

[0087] The prediction module adds a multi-scale generalization detection module SCAM before outputting the features of the detection target to improve the accuracy of the detection target; the structure of the multi-scale generalization detection module SCAM is as follows Figure 5 As shown, the multi-scale generalization detection module SCAM is divided into a three-layer architecture. The specific details of the multi-scale generalization detection module SCAM are as follows:

[0088] The first layer is composed of 3×3 dilated convolutions with a dilation rate of 1, 3×3 dilated convolutions with a dilation rate of 2, and 3×3 dilated convolutions with a dilation rate of 3, which are fused in parallel into a new feature map, thus forming a multi-scale feature pyramid structure to increase the receptive field of the output feature map.

[0089] The second layer uses the SE attention mechanism to enhance the features of the feature map of the first layer fusion output. The SE attention mechanism consists of global average pooling, a fully connected layer and a Sigmoid activation function; the SE attention mechanism is used to aggregate the feature map of the first layer fusion output to form a local context feature enhancement. The details of the SE attention mechanism are as follows:

[0090] The first layer performs a global average pooling operation on the feature map output by the parallel fusion, compressing the global spatial information into the channel local factor, thereby enhancing the aggregation of spatial information. The process is shown in the following formula (3):

[0091]

[0092] In the formula, i represents the pixel of the feature map height, j represents the pixel of the feature map width, H represents the height of the feature map, W represents the width of the feature map, and x c Represents the feature map of parallel fusion output, Z c Represents the feature x c The feature map after performing H×W global average pooling in the spatial dimension.

[0093] The second layer is for Z c Perform standard normalization, perform Relu activation on the normalized linear features, and then perform full connection on the feature map. Finally, obtain the normalized weights through the Sigmoid activation function. Its function is to fully capture the channel dependency and learn the nonlinear relationship between channels. The fully connected layer and the nonlinear activation function establish connections between channels. The process is shown in the following formula (4):

[0094] S c =σ(W 2 δ(W 1 Z c )) (4)

[0095] Where W 1 Represents Zc The weight parameter of the feature map, σ represents the sigmoid activation function, δ represents the relu activation function, W 2 represents the weight parameter of nonlinear mapping, S c Get normalized weights for the activation function.

[0096] The third layer is Z c The global average coefficient and the normalized weight S c Multiply them, and get the spatial local feature factor coefficients by multiplication, and weight them channel by channel to each space of the original feature map, so as to complete the recalibration of the original features by channel attention. This will enhance the global receptive field of the feature map. The process is shown in the following formula (5):

[0097] F c =Z c *S c (5)

[0098] In the formula, F c is the spatial local characteristic factor coefficient.

[0099] The third layer of the multi-scale generalized detection module SCAM performs a dot multiplication operation on the feature map output by the second layer and the original feature map to obtain the final output global context feature map. The head scales of the detection targets output by the multi-scale generalized detection module SCAM are three YOLO detection heads with 13×13, 26×26, and 52×52 scales respectively.

[0100] 5) Input the images in the training set into the improved YOLOv5 detection model for training, set the training parameters, load the coco pre-trained weights for training. During the training process, the number of training batch samples (batchsize) is set to 64, the initial learning rate is set to 0.001, and the training is performed for 300 iterations (epochs); the training optimizer uses the Adam optimizer; observe the changes in the loss curve during the training process to prevent the model from overfitting, save the training parameters every epoch, and perform model inference verification on the training parameters every 5 epochs. From the saved training parameters, the model parameters with the smallest verification set loss are selected as the best model parameters for the detection model.

[0101] 6) After loading the best model parameters obtained by improving the YOLOv5 detection model, input the pictures in the test set to obtain the prediction results of the three YOLO detection heads, and decode the predicted results to obtain the confidence of the predicted targets contained in all prediction boxes, the probability of the type of the target contained in the prediction box, and the position adjustment parameters of the prediction box; the confidence threshold is set to 0.6, and only when the confidence of the prediction box is greater than 0.6 will it be retained, and the prediction box with a confidence less than 0.6 will be eliminated.

[0102] 7) Further screening is performed on the prediction boxes to prevent the situation where multiple prediction boxes repeatedly detect the same target; overlapping boxes are eliminated using non-maximum suppression based on CIOU, and CIOU is calculated between all prediction boxes of the same class. When the CIOU between two boxes is greater than the preset threshold, the box with a larger confidence is retained as the final prediction box, and the box with a smaller confidence is discarded.

[0103] 8) Draw the result of the final prediction box detection on the original picture, use the rectangle function of opencv to draw a rectangular box, locate the student's coordinate position information on the rectangular box, and use the putText function of opencv to mark the predicted category information of the prediction box in the upper left corner area, thereby completing the detection of students' classroom behavior.

[0104] Example 2

[0105] This embodiment discloses a multi-scale feature fusion classroom behavior detection system based on an improved YOLOv5 detection model, which is used to implement the multi-scale feature fusion classroom behavior detection method described in Example 1. Figure 6 As shown, the system includes the following functional modules:

[0106] The data acquisition and processing module collects multiple campus classroom surveillance images as sample data sets of students' classroom behavior, divides the sample data sets into training sets, validation sets and test sets, and annotates and labels the divided samples; performs data enhancement processing on the images in the training set according to the behavioral characteristics of students in the classroom, randomly adjusts the brightness, contrast and saturation of the images in the training set, randomly crops, randomly flips and mirrors, and performs Mosaic enhancement on the images in the training set to expand the data volume and improve the generalization ability of the model; according to the characteristics of the students' actions and position targets in the processed training set images, the k-means clustering algorithm based on IOU is used to cluster the prior frames of the students' classroom behavior targets, and 9 clustering prior frames of the real student behavior targets are obtained;

[0107] Improve the YOLOv5 detection model, use the attention mechanism and feature enhancement fusion to extract local features of students' classroom behaviors, and recognize students' classroom performance in video time sequence;

[0108] A training module is used to input the images in the training set processed by the data acquisition and processing module into the improved YOLOv5 detection model for model training, use the value of the prior box obtained in the data acquisition and processing module as the prior value of the improved YOLOv5 detection model, perform training according to a preset number of training iterations (epochs), save the model parameters of the improved YOLOv5 detection model during training, and use the model parameters with the smallest loss in the validation set as the best model parameters of the improved YOLOv5 detection model;

[0109] The student classroom behavior detection module is used to load the improved YOLOv5 detection model with the best model parameters obtained by the training module and input the pictures in the test set, decode the prediction results obtained in the forward propagation of the model, obtain the confidence, detection category probability, and position adjustment parameters of the prediction boxes for detecting student classroom behaviors in all prediction boxes, and screen out the prediction boxes with confidence greater than a preset threshold for retention; further screen the retained prediction boxes, and use CIOU-based non-maximum suppression to screen the overlapping boxes in the prediction boxes. When the CIOU value in the overlapping box is greater than the preset threshold, only the prediction box with a large confidence is retained; the final prediction box is drawn on the original picture, and the prediction category information is marked in the upper left corner of the prediction box, thereby completing the detection of student classroom behavior.

[0110] Example 3

[0111] This embodiment discloses a storage medium storing a program. When the program is executed by a processor, the multi-scale feature fusion classroom behavior detection method described in Example 1 is implemented.

[0112] The storage medium in this embodiment can be a disk, an optical disk, a computer memory, a read-only memory (ROM), a random access memory (RAM), a USB flash drive, a mobile hard disk, or other media.

[0113] Example 4

[0114] This embodiment discloses a computing device, including a processor and a memory for storing programs executable by the processor. When the processor executes the program stored in the memory, the multi-scale feature fusion classroom behavior detection method described in Example 1 is implemented.

[0115] The computing device described in this embodiment may be a desktop computer, a laptop computer, a smart phone, a PDA handheld terminal, a tablet computer, a programmable logic controller (PLC), or other terminal devices with a processor function.

[0116] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be equivalent replacement methods and are included in the protection scope of the present invention.

Claims

1. A multi-scale feature fusion classroom behavior detection method, characterized in that: The method is based on the improved YOLOv5 detection model to achieve accurate detection of students' classroom behavior. The improved YOLOv5 detection model improves the backbone network module, pooling pyramid module and prediction module of the original YOLOv5 detection model. Among them, the improvement of the backbone network module is: using the CBAM attention module to enhance the features extracted by the backbone network; the improvement of the pooling pyramid module is: introducing the CSPC structure to the original pooling pyramid to obtain speed improvement while keeping the receptive field unchanged; the improvement of the prediction module is: adding a multi-scale generalization detection module SCAM before each prediction module outputs features to improve the accuracy of small target detection. The multi-scale generalization detection module SCAM is divided into a three-layer architecture, and the specific situation is as follows: The first layer is composed of 3×3 dilated convolutions with a dilation rate of 1, 3×3 dilated convolutions with a dilation rate of 2, and 3×3 dilated convolutions with a dilation rate of 3, which are fused in parallel to form a new feature map, thus forming a multi-scale feature pyramid structure to increase the receptive field of the output feature map; The second layer uses the SE attention mechanism to enhance the features of the feature map of the fusion output of the first layer. The SE attention mechanism consists of a global average pooling, a fully connected layer and a Sigmoid activation function. The feature map of the fusion output of the first layer is aggregated using the SE attention mechanism to form a local context feature enhancement. The third layer performs a dot multiplication operation on the feature map output by the second layer and the original feature map to obtain the final output global context feature map; The specific implementation of the multi-scale feature fusion classroom behavior detection method includes the following steps: 1) Preliminary data collection and construction: by collecting multiple campus classroom surveillance, the obtained pictures are used as sample data sets of students' classroom behavior. The sample data sets are divided into training sets, validation sets, and test sets, and the divided samples are annotated and labeled; 2) Perform data enhancement on the images in the training set according to the behavioral characteristics of students in the classroom, randomly adjust the brightness, contrast and saturation of the images in the training set, randomly crop, randomly flip and mirror, and perform Mosaic enhancement on the images in the training set to expand the data volume and improve the generalization ability of the model; 3) According to the characteristics of students’ actions and position targets in the processed training set pictures, the k-means algorithm based on IOU is used to cluster the prior frames of students’ classroom behavior targets, and 9 clustering prior frames of real student behavior targets are obtained; 4) Inputting the images in the training set processed by step 2) into the improved YOLOv5 detection model for model training, using the value of the prior box obtained in step 3) as the prior value of the improved YOLOv5 detection model, training according to a preset number of training iterations epoch, saving the model parameters of the improved YOLOv5 detection model during training, and using the model parameters with the smallest loss in the validation set as the best model parameters of the improved YOLOv5 detection model; 5) Improve the YOLOv5 detection model, load the best model parameters obtained in step 4), input the pictures in the test set, decode the prediction results obtained in the forward propagation of the model, obtain the confidence of detecting students' classroom behaviors in all prediction boxes, the probability of detection categories, and the position adjustment parameters of the prediction boxes, and select the prediction boxes with confidence greater than the preset threshold for retention; 6) Further screening the retained prediction boxes, using CIOU-based non-maximum suppression to screen the overlapping boxes in the prediction boxes. When the CIOU value in the overlapping box is greater than the preset threshold, only the prediction box with a large confidence is retained; 7) Draw the final prediction box on the original image, and mark the prediction category information in the upper left corner of the prediction box to complete the detection of students' classroom behavior.

2. The multi-scale feature fusion classroom behavior detection method according to claim 1 is characterized in that: In step 1), data is collected by means of camera shooting and monitoring recording to collect students' different classroom behaviors in the classroom. The action video of each behavior is collected continuously for 10 to 20 seconds. Multiple frames of pictures are randomly selected for each behavior from different shooting angles. The pictures finally obtained are used as sample data sets of students' classroom behaviors. The pictures with students' classroom actions are annotated with the labelimg annotation tool. Rectangular boxes are used to mark the students in the pictures and the positions of the actions made by the students. The annotated information is classified and integrated into training label data.

3. The multi-scale feature fusion classroom behavior detection method according to claim 2 is characterized in that: In step 2), in view of the complex classroom scene with dense crowds, the images in the training set are data enhanced, and the brightness, contrast and saturation of the images in the training set are changed to increase the diversity of the training set; in view of the diverse characteristics of students' classroom behaviors, the images in the training set are randomly cropped and Mosaic data enhanced; in view of the changeable environment and noise in the classroom, the images in the training set are expanded by randomly flipping the mirror.

4. The multi-scale feature fusion classroom behavior detection method according to claim 3 is characterized in that: In step 3), input the training label data into the k-means algorithm, set n cluster centers, randomly select n prior boxes from the training label data as the initial cluster centers, calculate the Euclidean distance from each training label data to the current existing cluster center, classify the training label data with the shortest distance into a cluster cluster, and select the next cluster center; repeat the above steps until the values ​​of all cluster centers no longer change. When the number of cluster centers reaches n, obtain the maximum number of iterations, classify each training label data and the nearest cluster center into one category, and finally obtain 9 prior boxes of different sizes.

5. The multi-scale feature fusion classroom behavior detection method according to claim 4 is characterized in that: In step 4), the backbone network module of the improved YOLOv5 detection model is CSPDarknet53, and CSPDarknet53 is used to extract features. The backbone network module reads the input image information, performs 5 downsamplings, obtains feature information of targets of different sizes, and adds a CBAM attention module after BottleNeckCSP of the backbone network. The CBAM attention module is composed of a channel attention module CAM and a spatial attention module SAM connected in series; CAM processes the input feature map through global maximum pooling and global average pooling, and then passes through two fully connected layers. The obtained features are element-wise added and then activated by sigmoid to obtain the final channel attention feature map. The input feature map and the channel attention feature map are element-wise multiplied to obtain the feature map output by CAM. The process is shown in the following formula (1): CAM(F)=σ(MLP(AvgPool(F))+MLP(MaxPool(F))) (1) In the formula, CAM(F) represents the feature map output by CAM, F represents the input feature map, MLP represents the fully connected layer, AvgPool represents the global average pooling, MaxPool represents the global maximum pooling, and σ represents the sigmoid activation function; SAM concatenates the input feature map after channel-based global maximum pooling and global average pooling, and then generates a spatial attention feature map through a 7×7 convolution layer and a sigmoid activation function. The final output feature map is obtained by multiplying it with the input feature map. The process is shown in the following formula (2): In the formula, SAM(F) represents the feature map output by SAM, F represents the input feature map, AvgPool represents the global average pooling, and MaxPool represents the global maximum pooling. represents the concat operation, k7 represents the 7×7 convolution operation, and σ represents the sigmoid activation function; The pooling pyramid module of the improved YOLOv5 detection model is to pass the feature map output by the backbone network module into the CSPC-SPPF pooling pyramid. The CSPC-SPPF pooling pyramid is divided into a four-layer architecture, and its specific situation is as follows: The first layer is a series of 1×1 convolution operations, 3×3 convolution operations, and 1×1 convolution operations, which are used to extract the size of the input feature map and enhance the image pixels; The second layer is the SPPF spatial pooling pyramid. The SPPF spatial pooling pyramid is composed of a multi-level pooling sliding window. The sliding window consists of three global maximum pooling convolutions with a kernel size of 5×5. The third layer is a series of 1×1 convolution operations, 3×3 convolution operations, and 1×1 convolution operations, which are used to reduce the dimension of the feature map extracted by the SPPF spatial pooling pyramid; The fourth layer is CSPC feature enhancement, which jumps the feature map output by the backbone network module and fuses it with the feature map output by the third layer through a 1×1 convolution operation. Using the CPSC structure to fuse features can not only reduce the amount of calculation by half, but also speed up the detection and improve the accuracy. The head scales of the detected targets output by the multi-scale generalized detection module SCAM are three YOLO detection heads with 13×13, 26×26, and 52×52 scales respectively.

6. The multi-scale feature fusion classroom behavior detection method according to claim 5 is characterized in that: The SE attention mechanism is divided into three layers, the details of which are as follows: The first layer performs a global average pooling operation on the feature map output by the parallel fusion, compressing the global spatial information into the channel local factor, thereby enhancing the aggregation of spatial information. The process is shown in the following formula (3): In the formula, i represents the pixel of the feature map height, j represents the pixel of the feature map width, H represents the height of the feature map, W represents the width of the feature map, and x c Represents the feature map of parallel fusion output, Z c Represents the feature map x c The feature map after performing H×W global average pooling in the spatial dimension; The second layer is for Z c Perform standard normalization operation, perform ReLU activation on the normalized linear features, then perform full connection operation on the feature map, and finally obtain normalized weights through Sigmoid activation function. Its function is to fully capture the channel dependency, learn the nonlinear relationship between channels, and enable the fully connected layer and nonlinear activation function to establish the connection between channels. The process is shown in the following formula (4): S c =σ(W2δ(W1Z c )) (4) Where W1 represents Z c The weight parameter of the feature map, σ represents the sigmoid activation function, δ represents the relu activation function, W2 represents the weight parameter of the nonlinear mapping, S c The normalized weights obtained for the activation function; The third layer is Z c The global average coefficient and the normalized weight S c Multiply them, and get the spatial local feature factor coefficients by multiplication, and weight them channel by channel to each space of the original feature map, so as to complete the recalibration of the original features by channel attention, thereby enhancing the global receptive field of the feature map. The process is shown in the following formula (5): F c =Z c *S c (5) In the formula, F c is the spatial local characteristic factor coefficient.

7. The multi-scale feature fusion classroom behavior detection method according to claim 6 is characterized in that: In step 4), the images in the training set processed by step 2) are input into the improved YOLOv5 detection model for training, the training parameters are set, and the coco pre-trained weights are loaded for training. During the training process, the number of training batch samples batchsize is set to 64, the initial learning rate is set to 0.001, and the training is performed for 300 epochs; the training optimizer uses the Adam optimizer; after the training is completed, the model parameters with the smallest loss in the validation set are selected from the saved training parameters as the optimal model parameters of the detection model; In step 5), the improved YOLOv5 detection model is loaded with the best model parameters and the pictures in the test set are input to obtain the prediction results of the three YOLO detection heads. The predicted results are decoded to obtain the confidence of the predicted targets contained in all prediction boxes, the probability of the type of the target contained in the prediction box, and the position adjustment parameters of the prediction box; only the prediction boxes with confidence greater than the preset threshold are retained.

8. The multi-scale feature fusion classroom behavior detection method according to claim 7 is characterized in that: In step 6), the prediction boxes are further screened to prevent multiple prediction boxes from repeatedly detecting the same target. The prediction results of the three YOLO detection heads are sorted according to the scores of all student classroom behavior prediction boxes, and the highest score and its corresponding prediction box are selected. The overlapping boxes are removed using CIOU-based non-maximum suppression. The CIOU is calculated between all prediction boxes of the same category. When the CIOU between two boxes is greater than the preset threshold, the box with a larger confidence is retained as the final prediction box, and the box with a smaller confidence is discarded.

9. The multi-scale feature fusion classroom behavior detection method according to claim 8 is characterized in that: In step 7), the result of the final prediction box detection is drawn on the original image, the student's coordinate position information is located on the prediction box, and the prediction category information of the prediction box is marked in the upper left corner area, thereby completing the detection of the student's classroom behavior.

10. The multi-scale feature fusion classroom behavior detection system is characterized by: A multi-scale feature fusion classroom behavior detection method for implementing any one of claims 1 to 9, comprising: The data acquisition and processing module collects multiple campus classroom surveillance images as sample data sets of students' classroom behavior, divides the sample data sets into training sets, validation sets and test sets, and annotates and labels the divided samples; performs data enhancement processing on the images in the training set according to the behavioral characteristics of students in the classroom, randomly adjusts the brightness, contrast and saturation of the images in the training set, randomly crops, randomly flips and mirrors, and performs Mosaic enhancement on the images in the training set to expand the data volume and improve the generalization ability of the model; according to the characteristics of the students' actions and position targets in the processed training set images, the k-means clustering algorithm based on IOU is used to cluster the prior frames of the students' classroom behavior targets, and 9 clustering prior frames of the real student behavior targets are obtained; Improve the YOLOv5 detection model, use the attention mechanism and feature enhancement fusion to extract local features of students' classroom behaviors, and recognize students' classroom performance in video time sequence; A training module is used to input the images in the training set processed by the data acquisition and processing module into the improved YOLOv5 detection model for model training, use the value of the prior box obtained in the data acquisition and processing module as the prior value of the improved YOLOv5 detection model, perform training according to a preset number of training iterations epoch, save the model parameters of the improved YOLOv5 detection model during training, and use the model parameters with the smallest loss in the validation set as the best model parameters of the improved YOLOv5 detection model; The student classroom behavior detection module is used to load the improved YOLOv5 detection model with the best model parameters obtained by the training module and input the pictures in the test set, decode the prediction results obtained in the forward propagation of the model, obtain the confidence, detection category probability, and position adjustment parameters of the prediction boxes for detecting student classroom behaviors in all prediction boxes, and screen out the prediction boxes with confidence greater than a preset threshold for retention; further screen the retained prediction boxes, and use CIOU-based non-maximum suppression to screen the overlapping boxes in the prediction boxes. When the CIOU value in the overlapping box is greater than the preset threshold, only the prediction box with a large confidence is retained; the final prediction box is drawn on the original picture, and the prediction category information is marked in the upper left corner of the prediction box, thereby completing the detection of student classroom behavior.

Images