Classroom discipline detection method and system

Abstract

The invention discloses a classroom discipline detection method and system, and the method comprises the steps: obtaining classroom monitoring which comprises a plurality of monitoring pictures, carrying out the head detection of each student in the first monitoring picture, generating a detection frame for the head of each student, and numbering the detection frame; detecting and tracking the head of each student frame by frame, recording the change of each detection frame which is numbered together in the current frame and the previous frame of monitoring picture, when an unnumbered detection frame appears in the monitoring picture, marking the unnumbered detection frame as a newly added detection frame, adding a number for the newly added detection frame, and recording the change of each detection frame which is numbered together; integrating the change of each detection frame with the same serial number and the newly added detection frame to obtain change data of each frame; and calculating the classroom discipline of each frame according to the change data of each frame, and outputting the classroom discipline of each frame. By detecting the head of each student in the classroom picture, the discipline change in the classroom is judged according to the change of inter-frame student head detection, and the problems that an existing discipline detection method is high in cost and low in efficiency are solved.

Description

Technical Field

[0001] The present invention relates to the technical field of teaching management, and in particular to a classroom discipline detection method and system. Background Art

[0002] Currently, methods for monitoring classroom discipline are limited, relying primarily on manual inspection and management by teachers. However, in scenarios without on-site teacher supervision, such as self-study classes, student discipline is often lax, disrupting classroom order.

[0003] To address this issue, patent CN113591678B proposes a classroom discipline detection method based on artificial intelligence technologies such as speech processing, image processing, and deep learning. This patent analyzes sound source distribution and volume changes for speech processing, and in image processing, analyzes key points on students' bodies and facial orientation to comprehensively determine classroom discipline.

[0004] Although this method can effectively achieve intelligent detection of classroom discipline, several problems still exist in practical applications. First, the system requires the deployment of a large number of devices, including cameras and sound collectors, which increases hardware costs. Second, the system uses multiple deep learning models, which places high demands on computing resources and further increases costs. In addition, in specific scenarios such as morning reading or reading aloud, the system may fail due to the limitations of the voice processing part.

[0005] In summary, although the scheme has certain effects in classroom discipline detection, its high cost and limitations of speech processing failure remain challenges to be solved in practical applications. Summary of the Invention

[0006] To solve the above problems, the present invention provides a classroom discipline detection method and system. By performing head detection on each student in the classroom image, the changes in discipline in the classroom are judged based on the changes in student head detection between frames, thereby solving the problems of high cost and low efficiency of existing discipline detection methods.

[0007] To achieve the above objectives, the present invention provides the following technical solutions:

[0008] A classroom discipline detection method comprises the following steps:

[0009] S1. Obtain classroom monitoring, which includes several frames of monitoring screen. Perform head detection on each student in the first frame of the monitoring screen. Generate a detection frame for each student's head and number it. The detection frame of the same student has the same number.

[0010] S2. Detect and track each student's head frame by frame, recording the changes in each numbered detection frame between the current frame and the previous frame. When an unnumbered detection frame appears in the monitoring image, mark it as a newly added detection frame and assign a number to the newly added detection frame. Combine the changes in each numbered detection frame and the newly added detection frame to generate per-frame change data.

[0011] S3. Calculate the classroom discipline for each frame based on the change data for each frame and output the classroom discipline for each frame.

[0012] Furthermore, in step S1, the head detection is performed on each student in the first frame of the monitoring image, and the specific implementation method is as follows:

[0013] A head recognition model is constructed, which includes a convolution module, a sampling module and a feature fusion module. The head recognition model is trained, and the first frame of monitoring image is input into the trained head recognition model. The head recognition model detects the head in the first frame of monitoring image.

[0014] Furthermore, the head recognition model is trained in the following manner:

[0015] Acquire a head data set, the head data set including a plurality of image data and a plurality of label data, each image data corresponds to a piece of label data, and the label data includes the position of the head detection frame in the image data;

[0016] During each training, any image data in the head data set is input into the head recognition model. The head recognition model recognizes the head in the image data and generates a head detection frame based on the head position in the image data. The head detection frame is the head recognition result. The detection loss between the head recognition result and the label data corresponding to the image data is calculated. The parameters of the head recognition model are optimized according to the detection loss until the number of training times reaches the preset number or the detection loss is less than the preset threshold, and the training ends.

[0017] Furthermore, the head recognition model detects the head in the first frame of the monitoring image, and its specific implementation method is as follows:

[0018] The convolution module of the head recognition model includes five convolution layers, and the convolution scales of the five convolution layers are different. The monitoring images are input into the five convolution layers respectively to obtain five feature maps P1, P2, P3, P4 and P5 with sizes from large to small. The sampling module is a multi-scale sampling structure. The sampling module upsamples P5 twice with P3 and P4 respectively to obtain the feature map P 4t and P 3t , the sampling layer is P 3t Downsample twice with P4 and P5 respectively to obtain the feature map P 4o and P 5oThe feature fusion module performs feature fusion on all feature maps to obtain a detection feature map, and outputs the detection feature map through a fully connected layer to obtain a detection result.

[0019] Furthermore, the feature fusion module performs feature fusion on all feature maps, and its specific implementation is as follows:

[0020] Adaptive weighted feature fusion is performed on all feature maps. The specific formula is:

[0021]

[0022] Among them, P f To detect the feature map, P i represents the i-th feature map, σ(w i ) represents the weight of the normalized i-th feature map, σ(w i ) is:

[0023]

[0024] Among them, w i represents the weight of the i-th feature map, and n is the number of feature maps.

[0025] Furthermore, in step S2, the head of each student is detected and tracked frame by frame, and the specific implementation method is as follows:

[0026] Using the deepport algorithm, the head of each student in the current frame is tracked based on the detection results of the previous frame of monitoring images.

[0027] Furthermore, in step S2, the changes of each detection frame with the same number in the current frame and the previous frame of the monitoring screen are recorded, specifically including: for each detection frame with the same number, calculating the intersection-and-union ratio of the detection frame with the same number and the area change of the detection frame with the same number; when the intersection-and-union ratio is not 1, calculating the change in the center position of the detection frame with the same number.

[0028] Furthermore, in step S2, the changes of each detection frame with the same number and the newly added detection frame are integrated to obtain the change data of each frame. The specific implementation method is: assigning the intersection and union ratio of the newly added detection frame, the change of the center position of the detection frame and the change of the area of ​​the detection frame to 1, integrating the intersection and union ratios of all detection frames into the intersection and union ratio set a, integrating the center position changes of all detection frames into the set m, and integrating the area changes of all detection frames into the set s.

[0029] Furthermore, in step S3, the classroom discipline of each frame is calculated based on the change data of each frame, and the specific implementation method is as follows:

[0030]

[0031] Wherein, C represents the degree of discipline, ∑a represents the sum of all elements in set a, ∑m represents the sum of all elements in set m, ∑s represents the sum of all elements in set s, and N represents the number of detection boxes.

[0032] Through the above technical solution, the present invention has the following beneficial effects: by combining the head detection model with the tracking algorithm, the changes in the head detection frame of each student are recorded, and the changes in the detection frame are used to calculate the discipline in the classroom. Only one camera is needed for monitoring, which reduces the hardware cost, and the three indicators are combined to make judgments, which makes the judgments more accurate and more efficient. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 The figure is a schematic diagram of the overall flow of a classroom discipline detection method of the present invention.

[0034] Figure 2 Schematic diagram of the structure of a classroom discipline detection system in an embodiment of the present invention. DETAILED DESCRIPTION

[0035] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0036] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0037] Example 1

[0038] See also Figure 1 , a classroom discipline detection method, comprising the following steps:

[0039] S1. Obtain classroom monitoring, which includes several frames of monitoring screen. Perform head detection on each student in the first frame of the monitoring screen. Generate a detection frame for each student's head and number it. The detection frame of the same student has the same number.

[0040] S2. Detect and track each student's head frame by frame, recording the changes in each numbered detection frame between the current frame and the previous frame. When an unnumbered detection frame appears in the monitoring image, mark it as a newly added detection frame and assign a number to the newly added detection frame. Combine the changes in each numbered detection frame and the newly added detection frame to generate per-frame change data.

[0041] S3. Calculate the classroom discipline for each frame based on the change data for each frame and output the classroom discipline for each frame.

[0042] In an optional embodiment, in step S1, the head detection is performed on each student in the first frame of the monitoring image, and the specific implementation method is as follows:

[0043] A head recognition model is constructed, which includes a convolution module, a sampling module and a feature fusion module. The head recognition model is trained, and the first frame of monitoring image is input into the trained head recognition model. The head recognition model detects the head in the first frame of monitoring image.

[0044] The purpose of detection is to obtain the detection frame information of all students' heads in the classroom through the self-trained head recognition model, which includes the coordinates of the upper left corner (x1, y1) and the lower right corner (x2, y2). Since the face of each student in classroom monitoring accounts for a small proportion, this application improves the conventional head recognition model. It performs feature fusion through up and down sampling to ensure the model's learning rate of features, and uses multiple convolutional layers to extract feature maps of different sizes to achieve multi-scale feature expression. Shallow feature maps retain detailed information and are suitable for small target detection. Deep feature maps extract high-level semantic information and enhance the ability to recognize large targets. At the same time, the feature map size is gradually reduced to reduce the amount of calculation while retaining key information and improving reasoning efficiency.

[0045] In an optional embodiment, the training of the head recognition model is specifically implemented as follows:

[0046] Acquire a head data set, the head data set including a plurality of image data and a plurality of label data, each image data corresponds to a piece of label data, and the label data includes the position of the head detection frame in the image data;

[0047] During each training, any image data in the head data set is input into the head recognition model. The head recognition model recognizes the head in the image data and generates a head detection frame based on the head position in the image data. The head detection frame is the head recognition result. The detection loss between the head recognition result and the label data corresponding to the image data is calculated. The parameters of the head recognition model are optimized according to the detection loss until the number of training times reaches the preset number or the detection loss is less than the preset threshold, and the training ends.

[0048] In an optional embodiment, the head recognition model detects the head in the first frame of the monitoring image, and its specific implementation is as follows:

[0049] The convolution module of the head recognition model includes five convolution layers, and the convolution scales of the five convolution layers are different. The monitoring images are input into the five convolution layers respectively to obtain five feature maps P1, P2, P3, P4 and P5 with sizes from large to small. The sampling module is a multi-scale sampling structure. The sampling module upsamples P5 twice with P3 and P4 respectively to obtain the feature map P 4t and P 3t , the sampling layer is P 3t Downsample twice with P4 and P5 respectively to obtain the feature map P 4o and P 5o The feature fusion module performs feature fusion on all feature maps to obtain a detection feature map, and outputs the detection feature map through a fully connected layer to obtain a detection result.

[0050] The conventional feature fusion method is direct addition, which does not consider the importance of different features. The head detection model proposed in this application not only trains the convolution layer parameters during the training process, but also performs weighted fusion of different features during the training process, and adjusts the feature weights according to the results of each training, thereby realizing adaptive weighted feature fusion.

[0051] In an optional embodiment, the feature fusion module performs feature fusion on all feature maps, and its specific implementation is as follows:

[0052] Adaptive weighted feature fusion is performed on all feature maps. The specific formula is:

[0053]

[0054] Among them, P f is the detection feature map, n is the number of feature maps, P i represents the i-th feature map, w i Represents the weight of the i-th feature map, and ε is a non-zero constant.

[0055] The adaptive weighted feature fusion method can automatically learn the importance of features at each scale based on the data. For small targets that are difficult to detect or with large background interference, weight adjustment can obtain more effective features. However, it also has disadvantages. i It is self-learning, so it may lead to a small denominator, which ultimately leads to slow gradient descent, affecting the training effect and training speed. Therefore, a normalization function is used to normalize the weights to make feature fusion more stable. Specifically, the normalization function expression is as follows:

[0056]

[0057] Among them, w irepresents the weight of the i-th feature map, n is the number of feature maps, and the adaptive weighted feature fusion is performed according to the formula as follows:

[0058]

[0059] Among them, P f To detect the feature map, P i represents the i-th feature map, σ(w i ) represents the weight of the normalized i-th feature map.

[0060] In an optional embodiment, in step S2, the head of each student is detected and tracked frame by frame, and the specific implementation method is as follows:

[0061] Using the deepport algorithm, the head of each student in the current frame is tracked based on the detection results of the previous frame of monitoring images.

[0062] DeepSort (Deep Simple Online and Realtime Tracking) is an efficient multi-target tracking algorithm based on target detection, motion prediction, and appearance feature matching. Its core process is as follows: after obtaining the detection frame, the Kalman filter is used to predict the next frame position of the tracked target and estimate its motion state. The Hungarian algorithm and Mahalanobis distance are then combined to perform a preliminary match between the detection frame and the prediction frame to screen out candidate pairs with high likelihood. Finally, a deep learning model is introduced to extract the target's appearance features, and the match is further optimized using cosine distance to address ID switching issues caused by target occlusion, deformation, or temporary disappearance. The DeepSort algorithm uses multi-target tracking technology to continuously and accurately identify and track the motion trajectories of multiple objects in continuous images, while maintaining the unique identity of each target, allowing for further calculation of classroom discipline based on the tracking results.

[0063] In an optional embodiment, in step S2, the changes in each detection frame with the same number in the current frame and the previous frame of the monitoring screen are recorded, specifically including: for each detection frame with the same number, calculating the intersection-and-union ratio of the detection frame with the same number and the area change of the detection frame with the same number, and when the intersection-and-union ratio is not 1, calculating the change in the center position of the detection frame with the same number.

[0064] The intersection-over-union ratio is used to calculate the movement of the student's head and the change in the size of the head target frame. For the newly appeared detection frame, it can be considered that its disciplinary impact is very large, so the newly appeared detection frame is directly assigned a value of 1.

[0065] When the detection frames of the upper and lower frames move too much, there is no intersection between them, so the calculated IOU is 0, and this indicator has no reference value. Therefore, to make up for the defects caused by this problem, for the detection frames with an IOU of 0, we use the change in the center position of the detection frame to evaluate the student's change. In addition, when calculating, we need to normalize the calculated position change. The specific formula is:

[0066]

[0067] Among them, F p is the detection frame of the previous frame, F c Indicates the current frame detection box, x c Indicates the horizontal coordinate of the center point of the current frame detection frame, y c Indicates the vertical coordinate of the center point of the current frame detection box, x p Indicates the horizontal coordinate of the center point of the detection frame of the previous frame, y p Indicates the vertical coordinate of the center point of the detection frame of the previous frame. x represents the length of the monitoring image, and y represents the width of the monitoring image.

[0068] For the change in the detection box area, normalization is also required. The specific calculation formula is:

[0069]

[0070] s id Indicates the change in the detection box area, S(C id ) represents the area of ​​the current frame detection box, S(P id ) represents the area of ​​the detection frame of the previous frame, max(S(C id ),S(P id )) means taking the maximum value between the current frame detection frame and the previous frame detection frame.

[0071] In an optional embodiment, in step S2, the changes of each detection frame with the same number and the newly added detection frame are integrated to obtain the change data of each frame, and the specific implementation method is: assigning the intersection and union ratio of the newly added detection frame, the change of the center position of the detection frame and the change of the area of ​​the detection frame to 1, integrating the intersection and union ratios of all detection frames into the intersection and union ratio set a, integrating the center position changes of all detection frames into the set m, and integrating the area changes of all detection frames into the set s.

[0072] In an optional embodiment, in step S3, the classroom discipline of each frame is calculated based on the change data of each frame, and the specific implementation method is as follows:

[0073]

[0074] Where C represents the level of discipline, ∑a represents the sum of all elements in set a, ∑m represents the sum of all elements in set m, ∑s represents the sum of all elements in set s, and N represents the number of detection boxes. By weighting these three indicators, we can comprehensively evaluate student activity and determine whether there is any abnormal disturbance or disorder.

[0075] Example 2

[0076] See also Figure 2 , a classroom discipline detection system, comprising:

[0077] A head detection module is used to obtain classroom monitoring, which includes several frames of monitoring images. The module performs head detection on each student in the first frame of monitoring images, generates a detection frame for each student's head and numbers it. The detection frames of the same student have the same number.

[0078] The detection and tracking module is used to detect and track each student's head frame by frame, recording the changes of each detection frame with the same number in the current frame and the previous frame of the monitoring screen. When an unnumbered detection frame appears in the monitoring screen, it is marked as a new detection frame and a number is added to the new detection frame. The changes of each detection frame with the same number and the new detection frame are integrated to obtain the change data of each frame;

[0079] The discipline calculation module is used to calculate the classroom discipline of each frame based on the change data of each frame and output the classroom discipline of each frame.

[0080] The embodiment disclosed in this specification is merely an illustration of one aspect of the present invention. The scope of protection of the present invention is not limited to this embodiment. Any other functionally equivalent embodiments fall within the scope of protection of the present invention. Those skilled in the art can make various other corresponding changes and modifications based on the technical solutions and concepts described above, and all such changes and modifications should fall within the scope of protection of the claims of the present invention.

Claims

1. A classroom discipline detection method, characterized in that: The following steps are involved: S1. Obtain classroom monitoring, which includes several frames of monitoring screen. Perform head detection on each student in the first frame of the monitoring screen. Generate a detection frame for each student's head and number it. The detection frame of the same student has the same number. S2. Detect and track each student's head frame by frame, recording the changes in each numbered detection frame between the current frame and the previous frame. When an unnumbered detection frame appears in the monitoring image, mark it as a newly added detection frame and assign a number to the newly added detection frame. Combine the changes in each numbered detection frame and the newly added detection frame to generate per-frame change data. S3. Calculate the classroom discipline for each frame based on the change data for each frame and output the classroom discipline for each frame.

2. A classroom discipline detection method according to claim 1, characterized in that: In step S1, the head detection is performed on each student in the first frame of the monitoring image, and the specific implementation method is as follows: A head recognition model is constructed, which includes a convolution module, a sampling module and a feature fusion module. The head recognition model is trained, and the first frame of monitoring image is input into the trained head recognition model. The head recognition model detects the head in the first frame of monitoring image.

3. A classroom discipline detection method according to claim 2, characterized in that: The head recognition model is trained in the following manner: Acquire a head data set, the head data set including a plurality of image data and a plurality of label data, each image data corresponds to a piece of label data, and the label data includes the position of the head detection frame in the image data; During each training, any image data in the head data set is input into the head recognition model. The head recognition model recognizes the head in the image data and generates a head detection frame based on the head position in the image data. The head detection frame is the head recognition result. The detection loss between the head recognition result and the label data corresponding to the image data is calculated. The parameters of the head recognition model are optimized according to the detection loss until the number of training times reaches the preset number or the detection loss is less than the preset threshold, and the training ends.

4. A classroom discipline detection method according to claim 2, characterized in that: The head recognition model detects the head in the first frame of the monitoring image, and its specific implementation method is as follows: The convolution module of the head recognition model includes five convolution layers, and the convolution scales of the five convolution layers are different. The monitoring images are input into the five convolution layers respectively to obtain five feature maps P1, P2, P3, P4 and P5 with sizes from large to small. The sampling module is a multi-scale sampling structure. The sampling module upsamples P5 twice with P3 and P4 respectively to obtain the feature map P 4t and P 3t , the sampling layer is P 3t Downsample twice with P4 and P5 respectively to obtain the feature map P 4o and P 5o The feature fusion module performs feature fusion on all feature maps to obtain a detection feature map, and outputs the detection feature map through a fully connected layer to obtain a detection result.

5. A classroom discipline detection method according to claim 4, characterized in that: The feature fusion module performs feature fusion on all feature maps, and its specific implementation method is as follows: Adaptive weighted feature fusion is performed on all feature maps. The specific formula is: Among them, P f To detect the feature map, P i represents the i-th feature map, σ(w i ) represents the weight of the normalized i-th feature map, σ(w i ) is: Among them, w i represents the weight of the i-th feature map, and n is the number of feature maps.

6. A classroom discipline detection method according to claim 1, characterized in that: In step S2, the head of each student is detected and tracked frame by frame, and the specific implementation method is as follows: Using the deepport algorithm, the head of each student in the current frame is tracked based on the detection results of the previous frame of monitoring images.

7. A classroom discipline detection method according to claim 1, characterized in that: In step S2, the changes of each detection frame with the same number in the current frame and the previous frame of the monitoring screen are recorded, specifically including: for each detection frame with the same number, calculating the intersection-and-union ratio of the detection frame with the same number and the area change of the detection frame with the same number; when the intersection-and-union ratio is not 1, calculating the change in the center position of the detection frame with the same number.

8. A classroom discipline detection method according to claim 7, characterized in that: In step S2, the changes of each detection frame with the same number and the newly added detection frame are integrated to obtain the change data of each frame. The specific implementation method is: assigning the intersection of union ratio, the change of the center position of the detection frame and the change of the detection frame area to 1, integrating the intersection of union ratios of all detection frames into the intersection of union ratio set a, integrating the center position changes of all detection frames into the set m, and integrating the area changes of all detection frames into the set s.

9. A classroom discipline detection method according to claim 8, characterized in that: In step S3, the classroom discipline of each frame is calculated based on the change data of each frame, and the specific implementation method is as follows: Wherein, C represents the degree of discipline, ∑a represents the sum of all elements in set a, ∑m represents the sum of all elements in set m, ∑s represents the sum of all elements in set s, and N represents the number of detection boxes.

10. A classroom discipline detection system, characterized in that: include: A head detection module is used to obtain classroom monitoring, which includes several frames of monitoring images. The module performs head detection on each student in the first frame of monitoring images, generates a detection frame for each student's head and numbers it. The detection frames of the same student have the same number. The detection and tracking module is used to detect and track each student's head frame by frame, recording the changes of each detection frame with the same number in the current frame and the previous frame of the monitoring screen. When an unnumbered detection frame appears in the monitoring screen, it is marked as a new detection frame and a number is added to the new detection frame. The changes of each detection frame with the same number and the new detection frame are integrated to obtain the change data of each frame; The discipline calculation module is used to calculate the classroom discipline of each frame based on the change data of each frame and output the classroom discipline of each frame.

Images