Student classroom behavior detection and evaluation method based on deep learning

Abstract

The invention discloses a student classroom behavior detection and evaluation method based on deep learning, and the method comprises the steps: constructing a student target detection sub-model based on improved YOLOv8, and constructing a classroom behavior classification sub-model through employing an MBConv structure; combining the two sets to obtain a two-stage target detection model; training and verifying the two-stage model to obtain a student classroom behavior detection model; performing image acquisition on a to-be-detected classroom; inputting the acquired image into a student classroom behavior detection model to obtain the position and behavior category of the student; and based on the positions and behavior categories of the students, using a target tracking algorithm to realize behavior analysis and evaluation of the whole individual students in the classroom. According to the invention, a three-stage scheme for student classroom behavior detection and evaluation is constructed, and the accuracy and efficiency of student behavior detection can be effectively improved so as to objectively evaluate student learning behaviors.

Description

Technical Field

[0001] The present invention relates to the technical field of student classroom behavior detection, and in particular to a student classroom behavior detection and evaluation method based on deep learning. Background Art

[0002] With the rapid development of artificial intelligence in the field of education, visual behavior detection technology has been applied to the field of education. By accurately analyzing the behavior of students in surveillance videos, it can evaluate their performance at different learning stages and provide practical references for teachers, managers, students and parents. When using visual behavior detection technology to detect students' classroom behavior, it is often faced with the problem that the distance between the front and back rows of students is large, making it difficult to detect small target students in the back row, students blocking each other, and the detection is not real-time. When evaluating student behavior, it tends to be subjective and the evaluation method is single.

[0003] At present, in order to solve the problem of low accuracy and speed of detecting students' classroom behaviors, the commonly used methods include using high-performance single-stage detectors to detect students' behaviors; enhancing data sets to strengthen model training; and increasing the depth of the network to achieve better detection results. However, there are still many pain points: general target detection algorithms tend to identify large targets and ignore small targets, which is more obvious in scenes with large spans of seats in front and behind the classroom; on the other hand, the large number of students and the dense density of students cause some students to be blocked, and the amount of information received by the model is reduced and disturbed, which increases the difficulty of detection; in addition, in order to ensure that the model can run on devices with limited computing power, it is necessary to lightweight the model and ensure the availability of detection accuracy. For example, a classroom behavior recognition method based on deep learning disclosed in Publication No. CN117523677A and a classroom behavior recognition method based on lightweight networks disclosed in Publication No. CN118135669A still have the problem of not being able to balance detection accuracy and speed well due to hardware equipment. Summary of the invention

[0004] 1. Technical problems to be solved:

[0005] In view of the above technical problems, the present invention provides a student classroom behavior detection and evaluation method based on deep learning. A two-stage student classroom behavior detection model is constructed using the single-stage target detection method YOLOv8 and the image classification algorithm EfficientNet. Then, a subjective and objective student classroom behavior evaluation method is designed based on the behavior detection data. Finally, a three-stage scheme for evaluating student groups and individual students is implemented in combination with visualization. This method can not only improve the accuracy and efficiency of student behavior detection, but also realize objective evaluation of student classroom behavior.

[0006] 2. Technical solution:

[0007] A method for detecting and evaluating student classroom behavior based on deep learning, characterized in that it includes the following steps:

[0008] Step 1: Improve the YOLOv8 network structure to obtain the student target detection sub-model; the improvement of the backbone network is to replace the C2F modules of the 4th, 6th and 8th layers with Ghost convolution modules; replace the SPPF pooling module of the 9th layer with a context enhancement module and a feature refinement module connected in sequence; the output of the feature refinement feature module is sent to the neck network; the improvement of the neck is to replace the splicing modules of the neck with BiFPN bidirectional feature fusion modules, and replace the ordinary convolution with the decomposed large-core convolution S-Kerrnel;

[0009] Step 2: Construct a classroom behavior classification sub-model; the classroom behavior classification sub-model includes a sequentially connected convolution kernel CONV module, a depth-separable convolution MBConv module, an ordinary 1*1 convolution layer, an average pooling layer and a fully connected layer;

[0010] Step 3: Combine the student target detection sub-model with the classroom behavior classification sub-model to construct a two-stage target detection model; the first stage of the model detects the location of the students through the student target detection sub-model, and the second stage identifies the student behavior characteristics through the classroom behavior classification sub-model;

[0011] Step 4: Obtain classroom images, preprocess the images to obtain a student classroom image dataset for training a student classroom behavior detection model, and train and verify the two-stage target detection model to obtain a student classroom behavior detection model;

[0012] Step 5: Capture images of the classroom to be inspected; input the captured images into the student classroom behavior detection model to obtain the student's location and behavior category; use the target tracking algorithm based on the student's location and behavior category to realize the overall individual behavior analysis and evaluation of the students in the classroom.

[0013] Furthermore, the student target detection sub-model includes an input end, a backbone network, and output ends of a neck and a head; wherein the input end is used to input a classroom image to be identified;

[0014] The backbone network is used to extract features from classroom pictures input at the input end; specifically, it includes the 0th layer of ordinary convolution layer, the 1st layer of ordinary convolution layer, the 2nd layer of C2F module, the 3rd layer of ordinary convolution layer, the 4th layer of Ghost convolution layer, the 5th layer of ordinary convolution layer, the 6th layer of Ghost convolution layer, the 7th layer of ordinary convolution layer, the 8th layer of Ghost convolution layer, the 9th layer of context enhancement module, and the 10th layer of feature refinement module; the image sent to the backbone network passes through an ordinary convolution layer, an ordinary convolution layer + C2F module and 3 stacked ordinary convolution layers + Ghost convolution layers to extract feature information of different scales, and finally sent to the context enhancement module and the feature refinement module to improve the context information of small targets;

[0015] The neck is used to classify the features extracted by the backbone network; specifically, it includes two sequentially connected upsampling modules at the 11th layer, the BIFPN module at the 12th layer, the C2F module at the 13th layer, the upsampling module at the 14th layer, the BiFPN bidirectional feature fusion module at the 15th layer, and the C2F module at the 16th layer: the large-core deep convolution decomposition structure at the 17th layer, namely the S-Kernel module, the BIFPN module at the 18th layer, the C2F module at the 19th layer, the S-Kernel module at the 20th layer, the BiFPN bidirectional feature fusion module at the 21st layer, and the C2F module at the 22nd layer; the neck receives features of different scales extracted by the backbone network, and realizes the detection of small and large targets through the fusion of multiple features of different scales by the BiFPN bidirectional feature fusion module;

[0016] The output end of the head is used to generate the final prediction result, specifically including three output ends connected to the C2F modules of the 16th, 19th, and 22nd layers respectively.

[0017] Furthermore, the student behavior classification sub-model adopts an MBConv structure, which sends the input to a 1*1 convolution layer and a k*k depth-separable convolution module for dimensionality increase and parameter reduction, and then inputs the SE module and the Dropout module for output; the feature refinement module includes a channel purification module and a spatial purification module, wherein the channel purification module is used to compress the feature map and extract the global spatial information of the image to obtain the key features of the image, and the spatial purification module is based on the feature refinement of the spatial dimension and focuses on the key areas in the image.

[0018] Furthermore, the BiFPN bidirectional feature fusion module fuses the features of different scales extracted by the backbone network through a feature pyramid network to output features of different scales.

[0019] Furthermore, in step 5, the target tracking algorithm is used based on the position and behavior category of the students to realize the behavior analysis and evaluation of the students in the class as a whole, that is, weight calculation is performed on each behavior of the detected target behavior, and the target behavior is analyzed according to the detected behavior data and the result of the weight calculation; the specific process of calculating the weight is:

[0020] S51: Define m classroom behaviors, and score them according to their importance as follows, and use the analytic hierarchy process and entropy weight method to construct the behavior feature judgment matrix A:

[0021]

[0022] In the above formula, a ij is the importance of the preset i-th and j-th classroom behaviors;

[0023] but

[0024] S52: According to the above-mentioned behavior feature judgment matrix A, the weight vector is calculated using the square root method, and the m-th power of the product of each row is calculated to obtain an m-dimensional vector

[0025]

[0026] S53: As shown in the following formula, the vector Standardization, the standardized vector ω i The elements of correspond to the weight vector of the behavior, that is, the corresponding subjective weights of m classroom behaviors are obtained:

[0027]

[0028] S54: Input the collected images into the student classroom behavior detection model to obtain the positions of the students and their behavior categories, which are represented by a matrix B. Each row of the matrix includes m elements, and the number of rows is the total number of students monitored as targets. The element b in the matrix ij is the category corresponding to the behavioral characteristics of the i-th student, b ij ∈{0,1}, if b ij =1, it means that the student is in the classroom behavior corresponding to column j; the n*m ​​matrix B and the m*m matrix diag(ω i ) can be multiplied to obtain the student behavior matrix X at the detection time, where n is the total number of student targets detected: The matrix X is normalized by the following formula to obtain the standardized student behavior matrix Y: where diag(ω i ) is as follows:

[0029]

[0030] Then the element yi in the student behavior matrix Y is j As follows:

[0031]

[0032] In the above formula, x ij represents the element in the i-th row and j-th column of the student behavior matrix; x i represents all elements in the jth column;

[0033] S55: Calculate the weight of each classroom behavior for each student goal in the score as follows:

[0034]

[0035] In the above formula, y ij represents the element in the i-th row and j-th column of matrix Y; p ij is the proportion of the i-th student sample in the j-th classroom behavior;

[0036] S56: For the obtained p ij , calculate the information entropy of each classroom behavior as:

[0037]

[0038] In the above formula, information entropy E j ≥0, indicating the information entropy of the jth classroom behavior; if p ij =0, then define E j =0;

[0039] Then calculate the information redundancy of each information entropy to get the weight of each classroom behavior at that moment:

[0040]

[0041] where d j =1-e j ;v j represents the information entropy weight corresponding to the jth classroom behavior;

[0042] S57: Finally, the comprehensive classroom behavior weight is calculated:

[0043]

[0044] In the above formula, s i Represents the comprehensive weight of the i-th classroom behavior.

[0045] Furthermore, the method further comprises step 6: presenting the student's classroom behavior by means of visualization based on the classroom behavior weight obtained in step 5; the visualization means comprises:

[0046] S61: Based on the behavior weight calculated at a certain moment, score the behavior of all students at the current moment and the preset time before the moment according to the weight, take the average of the scores and normalize them as the engagement score at the current moment; draw a student group engagement curve with time as the horizontal axis and engagement score as the vertical axis;

[0047] S62: Based on the behaviors detected at each moment, the number of behaviors and the time series records of each student are stored and calculated; the proportion of classroom behaviors of all students is counted using a table, and the classroom behavior status of any student is continuously monitored and analyzed according to the time sequence, with the horizontal axis being time and the vertical axis being behavior category, so as to realize the classroom behavior analysis of a single student;

[0048] S63: Aggregate the classroom behavior characteristics of all students during the class according to categories to obtain the frequency distribution of each behavior during the class; use a heat map to represent the frequency of classroom behavior categories in different time periods during the class.

[0049] 3. Beneficial effects:

[0050] (1) The student target detection sub-model obtained by improving the YOLOv8 network structure in this method is to optimize the small target detection by adding image decomposition method, improving FPN module and constructing bidirectional feature fusion network to solve the problems of large classroom with large seat span, small targets in the back row, occlusion of students in front and behind, and low real-time detection. Manhattan distance is introduced into NMS algorithm to optimize the occlusion problem. Large kernel deep convolution is decomposed and GhostBottleNeck module is introduced to replace C2F1 structure to improve model performance. Experimental results show that the mean average precision (mAP@0.5) of the improved scheme reaches 90.6%, and the frame rate (FPS) reaches 44.4, which can well meet the needs of learning behavior detection compared with the original algorithm.

[0051] (2) The student classroom behavior detection model in this method includes two sub-models, which effectively solves the accuracy problem of multi-target classification detection.

[0052] (3) In this method, the learning behavior data identified by the student classroom behavior detection model is used to score the learning behavior by combining the entropy weight method with the hierarchical analysis method. A classroom behavior evaluation method is designed and represented in a visual way. The evaluation methods include student group engagement curve, individual student classroom behavior analysis, and overall classroom behavior frequency analysis of students in the classroom, so as to display the students' learning engagement in classroom teaching from two dimensions: student group and individual student.

[0053] In summary, this method uses the target detection method YOLOv8 and the image classification algorithm EfficientNet to construct a two-stage classroom behavior detection algorithm, and then designs a subjective and objective student learning behavior evaluation model based on the behavior detection data. Finally, a three-stage scheme for student classroom behavior detection and evaluation is constructed by combining the student behavior detection algorithm with the classroom behavior evaluation model to improve the accuracy and efficiency of student behavior detection and objectively evaluate student learning behavior. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 A schematic diagram of the three stages of the method for detecting and evaluating students' classroom behavior;

[0055] Figure 2 This is the structural diagram of the student target detection sub-model obtained by improving the YOLOv8 network structure in this method;

[0056] Figure 3 This is the structural diagram of the student behavior classification sub-model in this method;

[0057] Figure 4 Schematic diagram of the BiFPN bidirectional feature fusion module involved in this method;

[0058] Figure 5 is a student group engagement curve obtained in a specific embodiment;

[0059] Figure 6 It is a time sequence diagram of individual student behavior states obtained in a specific embodiment;

[0060] Figure 7 It is a student group detection record diagram obtained in a specific embodiment;

[0061] Figure 8 It is a frequency diagram of classroom behavior of a student group obtained in a specific embodiment;

[0062] Fig. 9 It is a heat map of behavior frequency in different time periods of the classroom obtained in a specific embodiment. DETAILED DESCRIPTION

[0063] The present invention will be described in detail below with reference to the accompanying drawings.

[0064] As attached Figure 1 As shown, a method for detecting and evaluating student classroom behavior based on deep learning is characterized by comprising the following steps:

[0065] Step 1: Improve the YOLOv8 network structure to obtain the student target detection sub-model; the improvement of the backbone network is to replace the C2F modules of the 4th, 6th and 8th layers with Ghost convolution modules; replace the SPPF pooling module of the 9th layer with a context enhancement module and a feature refinement module connected in sequence; the output of the feature refinement feature module is sent to the neck network; the improvement of the neck is to replace the splicing modules of the neck with BiFPN bidirectional feature fusion modules, and replace the ordinary convolution with the decomposed large-core convolution S-Kerrnel;

[0066] Step 2: Construct a classroom behavior classification sub-model; the classroom behavior classification sub-model includes a sequentially connected convolution kernel CONV module, a depth-separable convolution MBConv module, an ordinary 1*1 convolution layer, an average pooling layer and a fully connected layer;

[0067] Step 3: Combine the student target detection sub-model with the classroom behavior classification sub-model to construct a two-stage target detection model; the first stage of the model detects the location of the students through the student target detection sub-model, and the second stage identifies the student behavior characteristics through the classroom behavior classification sub-model;

[0068] Step 4: Obtain classroom images, preprocess the images to obtain a student classroom image dataset for training a student classroom behavior detection model, and train and verify the two-stage target detection model to obtain a student classroom behavior detection model;

[0069] Step 5: Capture images of the classroom to be inspected; input the captured images into the student classroom behavior detection model to obtain the student's location and behavior category; use the target tracking algorithm based on the student's location and behavior category to realize the overall individual behavior analysis and evaluation of the students in the classroom.

[0070] As attached Figure 2 As shown, further, the student target detection sub-model includes an input end, a backbone network, a neck, and an output end of a head;

[0071] The input end is used to input the classroom image to be recognized;

[0072] The backbone network is used to extract features from classroom pictures input at the input end; specifically, it includes the 0th layer of ordinary convolution layer, the 1st layer of ordinary convolution layer, the 2nd layer of C2F module, the 3rd layer of ordinary convolution layer, the 4th layer of Ghost convolution layer, the 5th layer of ordinary convolution layer, the 6th layer of Ghost convolution layer, the 7th layer of ordinary convolution layer, the 8th layer of Ghost convolution layer, the 9th layer of context enhancement module, and the 10th layer of feature refinement module; the image sent to the backbone network passes through an ordinary convolution layer, an ordinary convolution layer + C2F module and 3 stacked ordinary convolution layers + Ghost convolution layers to extract feature information of different scales, and finally sent to the context enhancement module and the feature refinement module to improve the context information of small targets;

[0073] The neck is used to classify the features extracted by the backbone network; specifically, it includes two sequentially connected upsampling modules at the 11th layer, the BIFPN module at the 12th layer, the C2F module at the 13th layer, the upsampling module at the 14th layer, the BiFPN bidirectional feature fusion module at the 15th layer, and the C2F module at the 16th layer: the large-core deep convolution decomposition structure at the 17th layer, namely the S-Kernel module, the BIFPN module at the 18th layer, the C2F module at the 19th layer, the S-Kernel module at the 20th layer, the BiFPN bidirectional feature fusion module at the 21st layer, and the C2F module at the 22nd layer; the neck receives features of different scales extracted by the backbone network, and realizes the detection of small and large targets through the fusion of multiple features of different scales by the BiFPN bidirectional feature fusion module;

[0074] The output end of the head is used to generate the final prediction result, specifically including three output ends connected to the C2F modules of the 16th, 19th, and 22nd layers respectively.

[0075] As attached Figure 3 As shown, further, the student behavior classification sub-model adopts an MBConv structure, which sends the input to a 1*1 convolution layer and a k*k depth-separable convolution module for dimensionality increase and parameter reduction, and then inputs the SE module and the Dropout module for output; the feature refinement module includes a channel purification module and a spatial purification module, wherein the channel purification module is used to compress the feature map and extract the global spatial information of the image to obtain the key features of the image, and the spatial purification module is based on the feature refinement of the spatial dimension and focuses on the key areas in the image.

[0076] As attached Figure 4 As shown, the BiFPN bidirectional feature fusion module fuses the features of different scales extracted by the backbone network through the feature pyramid network to output features of different scales.

[0077] Furthermore, in step 5, the target tracking algorithm is used based on the position and behavior category of the students to realize the behavior analysis and evaluation of the students in the class as a whole, that is, weight calculation is performed on each behavior of the detected target behavior, and the target behavior is analyzed according to the detected behavior data and the result of the weight calculation; the specific process of calculating the weight is:

[0078] S51: Define m classroom behaviors, and score them according to their importance as follows, and use the analytic hierarchy process and entropy weight method to construct the behavior feature judgment matrix A:

[0079]

[0080] In the above formula, a ij is the importance of the preset i-th and j-th classroom behaviors;

[0081] but

[0082] S52: According to the above-mentioned behavior feature judgment matrix A, the weight vector is calculated using the square root method, and the m-th power of the product of each row is calculated to obtain an m-dimensional vector

[0083]

[0084] S53: As shown in the following formula, the vector Standardization, the standardized vector ω i The elements of correspond to the weight vector of the behavior, that is, the corresponding subjective weights of m classroom behaviors are obtained:

[0085]

[0086] S54: Input the collected images into the student classroom behavior detection model to obtain the positions of the students and their behavior categories, which are represented by a matrix B. Each row of the matrix includes m elements, and the number of rows is the total number of students monitored as targets. The element b in the matrix ij is the category corresponding to the behavioral characteristics of the i-th student, b ij ∈{0,1}, if b ij =1, it means that the student is in the classroom behavior corresponding to column j; the n*m ​​matrix B and the m*m matrix diag(ω i ) can be multiplied to obtain the student behavior matrix X at the detection time, where n is the total number of student targets detected: The matrix X is normalized by the following formula to obtain the standardized student behavior matrix Y: where diag(ω i ) is as follows:

[0087]

[0088] Then the element y in the student behavior matrix Y ij As follows:

[0089]

[0090] In the above formula, x ij represents the element in the i-th row and j-th column of the student behavior matrix; x i represents all elements in the jth column;

[0091] S55: Calculate the weight of each classroom behavior for each student goal in the score as follows:

[0092]

[0093] In the above formula, y ij represents the element in the i-th row and j-th column of matrix Y; p ij is the proportion of the i-th student sample in the j-th classroom behavior;

[0094] S56: For the obtained p ij , calculate the information entropy of each classroom behavior as:

[0095]

[0096] In the above formula, information entropy E j ≥0, indicating the information entropy of the jth classroom behavior; if p ij =0, then define E j =0;

[0097] Then calculate the information redundancy of each information entropy to get the weight of each classroom behavior at that moment:

[0098]

[0099] where d j =1-e j ;v j represents the information entropy weight corresponding to the jth classroom behavior;

[0100] S57: Finally, the comprehensive classroom behavior weight is calculated:

[0101]

[0102] In the above formula, s i Represents the comprehensive weight of the i-th classroom behavior.

[0103] Furthermore, the method further comprises step 6: presenting the student's classroom behavior by means of visualization based on the classroom behavior weight obtained in step 5; the visualization means comprises:

[0104] S61: Based on the behavior weight calculated at a certain moment, score the behavior of all students at the current moment and the preset time before the moment according to the weight, take the average of the scores and normalize them as the engagement score at the current moment; draw a student group engagement curve with time as the horizontal axis and engagement score as the vertical axis;

[0105] S62: Based on the behaviors detected at each moment, the number of behaviors and the time series records of each student are stored and calculated; the proportion of classroom behaviors of all students is counted using a table, and the classroom behavior status of any student is continuously monitored and analyzed according to the time sequence, with the horizontal axis being time and the vertical axis being behavior category, so as to realize the classroom behavior analysis of a single student;

[0106] S63: Aggregate the classroom behavior characteristics of all students during the class according to categories to obtain the frequency distribution of each behavior during the class; use a heat map to represent the frequency of classroom behavior categories in different time periods during the class. Specific embodiment:

[0108] The classroom behaviors used in this embodiment include raising the head, reading and writing, using electronic devices, and lying on the table. The mutual importance of these four behaviors is judged to obtain the following table, and a 4*4 judgment matrix A containing specific values ​​is obtained according to the following table:

[0109]

[0110] The pictures in this embodiment are as shown in the attached Figure 7 As shown, the boxed part in the figure is the student target identified by the student classroom behavior detection model, and different colors represent different student classroom behaviors corresponding to the target. According to the process of steps S52 to S56, the weight corresponding to each of the four classroom behaviors included in the picture can be obtained.

[0111] Based on the obtained classroom behavior weights, as shown in the attached Figures 5 to 8 As shown, in this embodiment, the following contents are presented through visualization:

[0112] After calculating the behavior weight at a certain moment, we can score the behavior of all students at the current moment and the previous minute according to the weight, take the average as the current moment engagement score, and draw a student group engagement curve with time as the horizontal axis and score as the vertical axis, as shown in the attached figure. Figure 5 shown.

[0113] According to the behaviors detected at each moment, the storage calculation can obtain the number and time series records of each student's behaviors. Taking 5 students as an example, according to the behavior category and quantity statistics, as shown in the following table, the proportion and score of students' learning behaviors in this class are displayed.

[0114] look up Read and Write Using electronic devices Lying on the table Engagement Score Student 1 79.4％ 12.8％ 2.2％ 0％ 0.76 Student 2 70.2％ 17.3％ 5.6％ 1％ 0.72 Student 3 63.4％ 14.3％ 6.5％ 0.9％ 0.64 Student 4 53.6％ 29.8％ 12.0％ 0.02％ 0.71 Student 5 65.8％ 28.6％ 13.6％ 0％ 0.80

[0115] Choose a student in the class as Figure 6 The left figure in the figure shows a continuous monitoring and analysis of its behavior status according to the time sequence, where the horizontal axis of the coordinate is time and the vertical axis is the behavior category, such as Figure 6 Right picture in .

[0116] All test records of the student group are as follows: Figure 7 As shown in the figure, by aggregating all students' behavior data and extracting overall features, it can be used to study the relationship between behavior frequency and class time period and individual students. For example, a behavior frequency graph can be used to reflect the behavior frequency distribution of the student group, such as Figure 8 shown.

[0117] At the same time, the class can also be divided into four time periods, the behavior frequency in each time period can be calculated, and the relationship between the frequency of different learning behaviors and different time periods can be studied. The behavior frequency heat map of different time periods in the class can be used to represent it, such as Fig. 9 shown.

[0118] Although the present invention has been disclosed as above in terms of preferred embodiments, they are not intended to limit the present invention. Anyone skilled in the art can make various changes or modifications without departing from the spirit and scope of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection defined by the claims of this application.

Claims

1. A method for detecting and evaluating student classroom behavior based on deep learning, characterized by: The following steps are involved: Step 1: Improve the YOLOv8 network structure to obtain the student target detection sub-model; the improvement of the backbone network is to replace the C2F modules of the 4th, 6th and 8th layers with Ghost convolution modules; replace the SPPF pooling module of the 9th layer with a context enhancement module and a feature refinement module connected in sequence; the output of the feature refinement feature module is sent to the neck network; the improvement of the neck is to replace the splicing modules of the neck with BiFPN bidirectional feature fusion modules, and replace the ordinary convolution with the decomposed large-core convolution S-Kerrnel; Step 2: Construct a classroom behavior classification sub-model; the classroom behavior classification sub-model includes a sequentially connected convolution kernel CONV module, a depth-separable convolution MBConv module, an ordinary 1*1 convolution layer, an average pooling layer and a fully connected layer; Step 3: Combine the student target detection sub-model with the classroom behavior classification sub-model to construct a two-stage target detection model; the first stage of the model detects the location of the students through the student target detection sub-model, and the second stage identifies the student behavior characteristics through the classroom behavior classification sub-model; Step 4: Obtain classroom images, preprocess the images to obtain a student classroom image dataset for training a student classroom behavior detection model, and train and verify the two-stage target detection model to obtain a student classroom behavior detection model; Step 5: Capture images of the classroom to be inspected; input the captured images into the student classroom behavior detection model to obtain the student's location and behavior category; use the target tracking algorithm based on the student's location and behavior category to realize the overall individual behavior analysis and evaluation of the students in the classroom.

2. According to the method for detecting and evaluating student classroom behavior based on deep learning in claim 1, it is characterized by: The student target detection sub-model includes an input end, a backbone network, a neck, and an output end of a head; The input end is used to input the classroom image to be recognized; The backbone network is used to extract features from classroom pictures input at the input end; specifically, it includes the 0th layer of ordinary convolution layer, the 1st layer of ordinary convolution layer, the 2nd layer of C2F module, the 3rd layer of ordinary convolution layer, the 4th layer of Ghost convolution layer, the 5th layer of ordinary convolution layer, the 6th layer of Ghost convolution layer, the 7th layer of ordinary convolution layer, the 8th layer of Ghost convolution layer, the 9th layer of context enhancement module, and the 10th layer of feature refinement module; the image sent to the backbone network passes through an ordinary convolution layer, an ordinary convolution layer + C2F module and 3 stacked ordinary convolution layers + Ghost convolution layers to extract feature information of different scales, and finally sent to the context enhancement module and the feature refinement module to improve the context information of small targets; The neck is used to classify the features extracted by the backbone network; specifically, it includes two sequentially connected upsampling modules at the 11th layer, the BIFPN module at the 12th layer, the C2F module at the 13th layer, the upsampling module at the 14th layer, the BiFPN bidirectional feature fusion module at the 15th layer, and the C2F module at the 16th layer: the large-core deep convolution decomposition structure at the 17th layer, namely the S-Kernel module, the BIFPN module at the 18th layer, the C2F module at the 19th layer, the S-Kernel module at the 20th layer, the BiFPN bidirectional feature fusion module at the 21st layer, and the C2F module at the 22nd layer; the neck receives features of different scales extracted by the backbone network, and realizes the detection of small and large targets through the fusion of multiple features of different scales by the BiFPN bidirectional feature fusion module; The output end of the head is used to generate the final prediction result, specifically including three output ends connected to the C2F modules of the 16th, 19th, and 22nd layers respectively.

3. The method for detecting and evaluating student classroom behavior based on deep learning according to claim 2 is characterized in that: The student behavior classification sub-model adopts the MBConv structure, which sends the input to a 1*1 convolution layer and a k*k depth-separable convolution module for dimensionality increase and parameter reduction, and then inputs the SE module and the Dropout module for output; the feature refinement module includes a channel purification module and a spatial purification module, wherein the channel purification module is used to compress the feature map and extract the global spatial information of the image to obtain the key features of the image, and the spatial purification module is based on the feature refinement of the spatial dimension and focuses on the key areas in the image.

4. The method for detecting and evaluating student classroom behavior based on deep learning according to claim 3 is characterized by: The BiFPN bidirectional feature fusion module fuses the features of different scales extracted by the backbone network through the feature pyramid network to output features of different scales.

5. The method for detecting and evaluating student classroom behavior based on deep learning according to claim 4 is characterized in that: In step 5, the target tracking algorithm is used based on the student's position and behavior category to realize the behavior analysis and evaluation of the overall individual students in the class. The weight of each behavior is calculated for the detected target behavior, and the target behavior is analyzed according to the detected behavior data and the result of weight calculation. The specific process of calculating the weight is: S51: Define m classroom behaviors, and score them according to their importance as follows, and use the analytic hierarchy process and entropy weight method to construct the behavior feature judgment matrix A: In the above formula, a ij is the importance of the preset i-th and j-th classroom behaviors; but S52: According to the above-mentioned behavior feature judgment matrix A, the weight vector is calculated using the square root method, and the m-th power of the product of each row is calculated to obtain an m-dimensional vector S53: As shown in the following formula, the vector Standardization, the standardized vector ω i The elements of correspond to the weight vector of the behavior, that is, the corresponding subjective weights of m classroom behaviors are obtained: S54: Input the collected images into the student classroom behavior detection model to obtain the positions of the students and their behavior categories, which are represented by a matrix B. Each row of the matrix includes m elements, and the number of rows is the total number of students monitored as targets. The element b in the matrix ij is the category corresponding to the behavioral characteristics of the i-th student, b ij ∈{0,1}, if b ij =1, it means that the student is in the classroom behavior corresponding to column j; the n*m ​​matrix B and the m*m matrix diag(ω i ) can be multiplied to obtain the student behavior matrix X at the detection time, where n is the total number of student targets detected: The matrix X is normalized by the following formula to obtain the standardized student behavior matrix Y: where diag(ω i ) is as follows: Then the element yi in the student behavior matrix Y is j As follows: In the above formula, x ij represents the element in the i-th row and j-th column of the student behavior matrix; x i represents all elements in the jth column; S55: Calculate the weight of each classroom behavior for each student goal in the score as follows: In the above formula, y ij represents the element in the i-th row and j-th column of matrix Y; p ij is the proportion of the i-th student sample in the j-th classroom behavior; S56: for the obtained pi j , calculate the information entropy of each classroom behavior as: In the above formula, information entropy E j ≥0, indicating the information entropy of the jth classroom behavior; if p ij =0, then define E j =0; Then calculate the information redundancy of each information entropy to get the weight of each classroom behavior at that moment: where d j =1-e j ;v j represents the information entropy weight corresponding to the jth classroom behavior; S57: Finally, the comprehensive classroom behavior weight is calculated: In the above formula, s i Represents the comprehensive weight of the i-th classroom behavior.

6. A method for detecting and evaluating student classroom behavior based on deep learning according to claim 5, characterized in that: The step also includes step 6: presenting the students' classroom behaviors by means of visualization based on the classroom behavior weights obtained in step 5; the visualization means includes: S61: Based on the behavior weight calculated at a certain moment, score the behavior of all students at the current moment and the preset time before the moment according to the weight, take the average of the scores and normalize them as the engagement score at the current moment; draw a student group engagement curve with time as the horizontal axis and engagement score as the vertical axis; S62: storing and calculating the number and time series records of each student's behaviors according to the behaviors detected at each moment; Use a table to count the proportion of classroom behaviors of all students, and continuously monitor and analyze the classroom behavior status of any student in chronological order, with the horizontal axis being time and the vertical axis being behavior category, to achieve classroom behavior analysis of a single student; S63: Aggregate the classroom behavior characteristics of all students during the class according to categories to obtain the frequency distribution of each behavior during the class; use a heat map to represent the frequency of classroom behavior categories in different time periods during the class.

Images