A primary and secondary school classroom student face turning detection method based on deep learning

Abstract

The application provides a primary and secondary school classroom student face turning detection method based on deep learning, comprising the following steps: step 1, training a deep learning target detection model and detecting a bounding box; step 2, creating a background image; step 3, calculating and drawing a bounding box center point; step 4, straight line detection on the background image; step 5, filtering and extracting straight lines, judging that the straight lines cannot intersect and the slope range; step 6, calculating the remaining bounding box center points; step 7, center point fitting straight line; step 8, dividing the bounding box onto the straight line; step 9, dividing the bounding box into rows and the straight line into columns; step 10, obtaining a student head posture; step 11, calculating a student head yaw angle and an included angle between the corresponding straight line of the bounding box, judging whether the student turns the face; and step 12, drawing the straight line, the bounding box, the row and column and the included angle on the image. The application realizes automatic positioning of student row and column information and whether the face is turned, and digitalizes the listening situation in the classroom learning.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of artificial intelligence deep learning, and relates to a primary and secondary school classroom student face turning detection method based on deep learning. BACKGROUND

[0002] Analyzing the behavior of students in class in primary and secondary schools is of great significance for improving students' learning habits and improving teaching quality. At present, some data of primary and secondary school classrooms are mostly manually counted, such as classroom attendance rate, students' attentive listening, whether the teacher is focused on teaching, etc. According to these data, the teaching quality can be improved, and the students' learning situation can be improved. However, relying on manpower is not only time-consuming and laborious, but also has a greater possibility of error. If the classroom teaching situation can be automatically and intelligently analyzed, it will be very beneficial.

[0003] With the development of machine learning, especially the development of deep learning, some problems that cannot be solved by traditional computer vision have been solved, and some aspects of traditional visual effects have been greatly improved. SUMMARY

[0004] In order to solve the above problems, the application uses machine learning, deep learning and computer vision methods to propose an automatic and intelligent detection method for student face turning in primary and secondary school classrooms in an educational scene. Given the input primary and secondary school classroom image or video, the student face turning action in the class can be automatically detected.

[0005] The application discloses a primary and secondary school classroom student face turning detection method based on deep learning, comprising the following steps:

[0006] Step 1: training a deep learning target detection model for detecting students, and detecting the bounding box of the students in the image;

[0007] Step 2: creating a background image with a black background and the same size as the image;

[0008] Step 3: calculating the center point of the bounding box of the students, and drawing the center point on the background image in white;

[0009] Step 4: performing straight line detection on the background image obtained in step 3;

[0010] Step 5: filtering and extracting three straight lines away from the camera, and the judgment principle is that the straight lines in the student area direction cannot intersect and the slope is within a certain range;

[0011] If the classroom is taken as an example, four straight lines are extracted in five columns.

[0012] Step 6: removing the bounding box in step 1 intersected by the three straight lines, and calculating the center point of the remaining bounding box;

[0013] Step 7: Linear regression is performed on the center points obtained in step 6 to obtain a straight line;

[0014] Step 8: Each bounding box in step 1 is divided onto each straight line;

[0015] Step 9: The bounding boxes corresponding to each straight line are divided into rows according to the x-axis coordinate of the center point of the bounding box, and into columns for each straight line;

[0016] Step 10: The screenshot corresponding to each student's bounding box detected in step 1 is put into the head pose estimation network to obtain the head pose of each student;

[0017] Step 11: The yaw angle of each student's head pose and the included angle between each student's bounding box and the corresponding straight line are calculated, and whether the student turns his face is determined according to the included angle;

[0018] Step 12: Draw the straight line, bounding box, row number, column number, and included angle on the image.

[0019] Further, step 1 includes the following steps:

[0020] Step 1-1: Record a sufficient amount of classroom video and take screenshots discretely;

[0021] Step 1-2: Use the image of step 1-1 to label the student target model learning training set;

[0022] Step 1-3: Construct a yolo v5 target detection network, the backbone network of yolo v5 uses CSP and Focus, the neck network uses PAN and FPN, and the detection head uses the yolo method. The backbone is used to extract the basic features of the image through continuous convolution and downsampling. The neck part is used to solve the problem of different sizes of targets in the image, and multiple scale feature maps are constructed for detection, while the low-level strong details and high-level strong semantic information are fused, which is more conducive to the detection of multiple size targets. The detection head is the part of the network that generates all the bounding boxes and performs non-maximum suppression algorithm to remove redundant bounding boxes of a single target, and finally obtains the detection result;

[0023] Step 1-4: Initialize the backbone network of yolo v5 using pre-trained weights, and initialize other parts as needed;

[0024] Step 1-5: Use the dataset of step 1-2 to train the target detection model of step 1-4 to obtain a generalization model that can detect desk targets;

[0025] Step 1-6: Use the model in step 1-5 to detect students in the image and obtain the bounding box.

[0026] Further, step 2 comprises the following steps:

[0027] A new black background image with the same size as the original image is created.

[0028] Further, step 3 comprises the following steps:

[0029] Draw the center point of the bounding box in white on the background image.

[0030] Further, step 4 comprises the following steps:

[0031] Step 4-2: Use the Hough transform line detection algorithm to detect lines in the background image.

[0032] Further, step 5 comprises the following steps:

[0033] Extract lines with slopes within a certain range from the detected lines.

[0034] Further, step 6 method is as follows:

[0035] Use the lines obtained in step 5 to filter the bounding boxes obtained in step 1-6, remove the bounding boxes that are crossed by the lines and the distance from the center point to the line is less than a certain threshold.

[0036] Further, step 7 method is as follows:

[0037] Step 7-1: Use the remaining bounding boxes obtained in step 6 to find the center point;

[0038] Step 7-2: Use linear regression to fit a straight line through the center points.

[0039] Further, step 8 method comprises:

[0040] For the straight line obtained, divide the bounding boxes obtained in step 1-6 into one of the straight lines.

[0041] Further, step 9 method comprises the following steps:

[0042] Determine the number of columns according to the relationship of the straight lines, and determine the number of rows according to the x-coordinate values of the center points of the bounding boxes corresponding to each straight line.

[0043] Further, step 10 method comprises:

[0044] The head pose estimation network estimates the head pose of each student.

[0045] Further, step 11 method comprises:

[0046] Calculate the angle between the head pose of each student and the straight line.

[0047] Further, the step 12 method comprises:

[0048] The bounding box, straight line, row number, column number and included angle are drawn on the image to obtain a final result image.

[0049] The present application has the following advantages:

[0050] 1) The deep learning and computer vision algorithm is used to automatically locate the position of students, determine the row and column information, and determine whether to turn the face, thereby digitalizing the classroom learning and listening situation, for example, counting the face turning action of each student and the serious listening situation in a class.

[0051] 2) The student position is automatically positioned and the row and column information is calculated in an intelligent manner, and the algorithm speed is very fast, which can be applied to a video to detect and locate the student position and calculate the row and column information in real time, and the algorithm speed is fast. BRIEF DESCRIPTION OF DRAWINGS

[0052] The above or other aspects of the present application will become more apparent from the following detailed description, taken in conjunction with the accompanying drawings.

[0053] Figure 1 It is a flow chart for face turning detection method;

[0054] Figure 2 It is a schematic diagram of step 1;

[0055] Figure 3 It is a schematic diagram of step 2;

[0056] Figure 4 It is a schematic diagram of step 3;

[0057] Figure 5 It is a schematic diagram of step 4;

[0058] Figure 6 It is a schematic diagram of step 7;

[0059] Figure 7 It is a schematic diagram of step 9;

[0060] Figure 8 It is a schematic diagram of step 11;

[0061] Figure 9 It is a schematic diagram of step 12;

[0062] Figure 10 It is a network overall structure diagram of deep learning target detection yolo v5;

[0063] Figure 11 It is a sub-module structure diagram of deep learning target detection yolo v5.

[0064] Reference signs: B, background image; box, bounding box; sdt, student; pt, white center point; J0, J1, J2, detected straight line; N1, fitted straight line; β, yaw angle; α, included angle. DETAILED DESCRIPTION

[0065] The present application will be described in detail below with reference to the accompanying drawings and examples.

[0066] As Figure 1 The flow chart of the face turning detection method of the present embodiment is shown. The present application discloses a primary and secondary school classroom student sdt face turning detection algorithm based on deep learning, comprising the following steps:

[0067] Step 1: a deep learning target detection model for detecting student sdt is trained, and the bounding box box of the student sdt in the image is detected; as Figure 2 The box represents the bounding box box of the student sdt in the image;

[0068] Step 2: as Figure 3 shown, a background image B with black background and same size as the image is created;

[0069] Step 3: the center point of the bounding box box of the student sdt is calculated, and the center point is drawn in white on the background image B, as Figure 4 shown, the background image B of the bounding box box with white center point pt;

[0070] Step 4: as Figure 5 shown, straight line detection is performed on the background image B obtained in step 3;

[0071] Step 5: three straight lines far away from the camera are filtered and extracted, and the judgment principle is that the straight lines in the student sdt region direction cannot intersect and the slope is within a certain range;

[0072] In the present embodiment, the image has four rows of desks, and each row of desks is different in distance from the camera.

[0073] Step 6: as Figure 5 shown, the bounding box box in step 1 intersected by the three straight lines is removed, and the center point of the remaining bounding box box is calculated;

[0074] Step 7: as Figure 6 shown, a straight line N1 is fitted by linear regression for the center point obtained in step 6;

[0075] Step 8: each bounding box box in step 1 is divided onto each straight line;

[0076] Step 9: as Figure 7As shown, the bounding box box corresponding to each straight line is divided into rows according to the x-axis coordinate size of the center point of the bounding box box, and each straight line is divided into columns.

[0077] The length direction of the image is the y-axis, and the width direction is the x-axis.

[0078] Step 10: Put the screenshot corresponding to the bounding box box of each student sdt detected in step 1 into the head pose estimation network to obtain the head pose of each student sdt;

[0079] Step 11: As shown, calculate the yaw angle β of the head pose of each student sdt and the included angle α between the straight line corresponding to the bounding box box of each student sdt, and determine whether the student sdt turns his face according to the included angle α; Figure 8

[0080] When the included angle α is greater than the set angle, it is determined that the student sdt turns his face, and when the included angle α is less than the set angle, it is not determined that the student sdt turns his face, wherein the set angle is set according to actual experience, and the set angle is between 20-50 degrees.

[0081] Step 12: As shown, draw the straight line, the bounding box box, the row number, the column number, and the included angle α on the image. Figure 9

[0082] Step 1 includes the following steps:

[0083] Step 1-1: Record a sufficient amount of classroom video and take screenshots discretely;

[0084] Step 1-2: Use the image of step 1-1 to learn and train the target detection model of the student sdt target;

[0085] Step 1-3: Construct a yolo v5 target detection network, as shown, the backbone network of yolo v5 uses CSP and Focus, the neck network uses PAN and FPN, and the detection head uses the method of yolo. The backbone is used to extract the basic features of the image by continuously convolving and downsampling. The neck network is used to solve the problem of different sizes of targets in the image, and multiple scale feature maps are constructed for detection, while the low-level strong details, positioning information and high-level strong semantic information are fused, which is more conducive to the detection of multiple size targets. The detection head is the part of the network that generates all the bounding boxes box and performs non-maximum suppression algorithm to remove redundant bounding boxes box of a single target to obtain the final detection result; Figure 10-11

[0086] Step 1-4: Initialize the backbone network of yolo v5 using pre-trained weights, and initialize other parts as needed;

[0087] ​​​Step 1-5: Using the data set of step 1-2, the target detection model of step 1-4 is trained to obtain a generalization model that can detect desk targets.

[0088] Step 1-6: Using the model in step 1-5 to detect the student sdt in the image, a bounding box box is obtained.

[0089] Step 2 includes the following steps: a new black background image B with the same size as the original image is created.

[0090] Step 3 includes the following steps: draw the center point of the bounding box box in white on the background image B.

[0091] Step 4 includes the following steps: Step 4-2: Use the Hough transform line detection algorithm to detect straight lines J0, J1, J2 in the background image B.

[0092] Step 5 includes the following steps: Extract the straight lines with a slope within a certain range from the detected straight lines.

[0093] Step 6 method as follows: using the straight lines obtained in step 5, the bounding box box obtained in step 1-6 is filtered to remove the bounding box box that is crossed by the straight line and the distance from the center point to the straight line is less than a certain threshold.

[0094] Step 7 method as follows: Step 7-1: using the remaining bounding box box obtained in step 6, the center point of the bounding box box is calculated;

[0095] Step 7-2: Use linear regression to fit a straight line N1 through the center points.

[0096] Step 8 method includes: for the straight line calculated, the bounding box box obtained in step 1-6 is divided into one of the straight lines.

[0097] Step 9 method includes the following steps: determine the number of columns according to the relationship of the straight lines, determine whether the straight line belongs to the 0th column or the 1st column, or the 2nd column, or the 3rd column according to the order of the straight line from right to left in the image, so as to locate the column position of the student sdt in the image.

[0098] Determine the number of rows according to the x-coordinate of the center point of the bounding box box corresponding to each straight line, and there are multiple bounding boxes on each straight line. According to the x-coordinate sorting of the bounding box box on the straight line, the order of the bounding box box on the straight line is determined, and the row position of the student sdt on the straight line in the image is determined.

[0099] Step 10 method includes: using a head pose estimation network to estimate the head pose of each student sdt, wherein the head pose includes pitch angle, yaw angle β, and roll angle.

[0100] The step 11 method comprises: calculating the yaw angle β of the head posture of each student sdt and the included angle α between the straight line corresponding to the bounding box box of each student sdt.

[0101] The step 12 method comprises: drawing the bounding box box, the straight line, the row number, the column number and the included angle α on the image to obtain a final result image, and through a deep learning algorithm, the face turning situation of each student sdt can be directly displayed on the image, and the position of each student sdt in the image can be quickly located.

[0102] The specific embodiments described herein are merely illustrative of the spirit of the present application. Those skilled in the art of the present application can make various modifications or supplements to the described specific embodiments or replace them with similar ways, but will not deviate from the spirit of the present application or exceed the scope defined by the appended claims.

Claims

1. A deep learning-based primary and secondary school classroom student face turning detection method, characterized in that, The method comprises the following steps: Step 1: train a deep learning target detection model for detecting students, and detect the bounding box of students in the image; Step 2: create a background image with a black background and the same size as the image; Step 3: calculate the center point of the bounding box of the student, and draw the center point on the background image in white; Step 4: perform straight line detection on the background image obtained in step 3; Step 5: filter and extract three straight lines away from the camera, and the judgment principle is that the straight lines in the student area direction cannot intersect and the slope is within the set range; Step 6: remove the bounding box in step 1 that is crossed by the three straight lines, and calculate the center point of the remaining bounding box; Step 7: perform linear regression fitting on the center point obtained in step 6 to obtain a straight line; Step 8: divide each bounding box in step 1 into each straight line; Step 9: divide the bounding boxes corresponding to each straight line into rows according to the x-axis coordinate of the center point of the bounding box, and divide each straight line into columns; Step 10: put the screenshot corresponding to each student's bounding box detected in step 1 into the head pose estimation network to obtain the head pose of each student; Step 11: calculate the yaw angle of each student's head pose and the included angle between each student's bounding box corresponding to the straight line, and determine whether the student turns his face according to the included angle; Step 12: draw the straight line, bounding box, row and column, and included angle on the image.

2. The deep learning-based primary and secondary school classroom student face turning detection method according to claim 1, characterized in that, Step 1 comprises the following steps: Step 1-1: record a sufficient amount of classroom video and discretely take screenshots; Step 1-2: use the image in step 1-1 to label the student target detection model training set; Step 1-3: construct a yolo v5 target detection network, the backbone network of yolo v5 uses CSP and Focus, the neck network uses PAN and FPN, and the detection head uses the yolo method, the backbone network is used to extract the basic features of the image through continuous convolution and downsampling, the neck network constructs multiple scale feature maps for detection, while fusing low-level details, positioning information and high-level semantic information, which is more conducive to the detection of multiple size targets, the detection head is the part of the network that generates all the bounding boxes, and performs non-maximum suppression algorithm to remove redundant bounding boxes of a single target, and obtains the final detection result; Step 1-4: initialize the backbone network of yolo v5 using pre-trained weights, and initialize other parts as needed; Step 1-5: use the data set in step 1-2 to train the target detection model in step 1-4 to obtain a general model that can detect desk targets; Step 1-6: use the target detection model in step 1-5 to detect students in the image to obtain the bounding box. 3.The method of claim 2, wherein, Step 4 comprises the following steps: Step 4-1: use the Hough transform straight line detection algorithm to detect straight lines in the background image.

4. The deep learning-based elementary and middle school classroom student face turning detection method according to claim 3, characterized in that, Step 5 comprises the following steps: Extract straight lines with a slope within a set range from the detected straight lines.

5. The deep learning-based elementary and middle school classroom student face turning detection method according to claim 4, characterized in that, Step 6 method is as follows: Use the straight lines obtained in step 5 to filter the bounding boxes obtained in step 1-6, and remove the bounding boxes that are crossed by the straight lines and whose center point distance to the straight line is less than a set threshold.

6. The deep learning-based elementary and middle school classroom student face turning detection method according to claim 1, characterized in that, Step 8 method is as follows: For the straight line, the bounding box obtained in step 1-6 is divided into one of the straight lines.

7. The deep learning-based elementary and middle school classroom student face turning detection method according to claim 6, characterized in that, The method of step 9 is as follows: According to the relationship of the straight line, the column is determined, and the row is determined according to the x coordinate value of the center point of the bounding box corresponding to each straight line.

Images