A method for monitoring panoramic attendance in a smart classroom based on deep learning

Abstract

The application provides a kind of wisdom classroom monitoring panoramic attendance method based on deep learning, to solve the problems such as high cost, limited accuracy and monitoring blind area of existing classroom automatic roll call system. The method is to integrate panoramic camera and deep learning technology to improve attendance accuracy and efficiency while significantly reducing system cost. The core of this method is to use panoramic camera to capture the panorama of the classroom, and combine face feature extraction model to accurately identify and verify the identity of students. Compared with existing technology, the application has significant cost-effectiveness, can effectively reduce the monitoring blind area, and improve the efficiency of classroom monitoring attendance through automation. In addition, the flexibility and adaptability of this method make it widely applicable to different teaching environments, which is of great significance to improve the quality of teaching management. These advantages show that the application has broad application prospects in the field of smart education.

Description

Technical fields:

[0001] This invention belongs to the field of smart classrooms, and in particular relates to a method for panoramic attendance monitoring in smart classrooms based on deep learning. Background technology:

[0002] In recent years, with the rapid development of information technology and the Internet of Things (IoT), intelligent teaching management systems have received increasing attention. Especially in attendance management, how to improve the accuracy and efficiency of attendance through technology has become a hot topic. Currently, various automated attendance systems based on different technologies have been proposed and implemented.

[0003] For example, patent CN202121513074 describes an automatic classroom attendance system. This system automatically records student attendance by installing human body capture devices on each seat and connecting these devices to an information display device using IoT technology. While this method achieves automated attendance, it suffers from high costs and low accuracy in identity recognition. CN207965910U describes a teaching management system based on facial recognition. This system uses facial recognition technology to collect students' facial information through a camera device to monitor their attendance and class participation. Although this system improves the intelligence of attendance tracking, it still faces problems such as students avoiding the camera and inaccurate evaluation of classroom participation.

[0004] While existing automatic attendance systems and facial recognition-based teaching management systems each have their advantages, they also have some obvious drawbacks: installing a data collection device for each seat is costly; relying on simple signature methods makes them easy to forge, and the recognition process is time-consuming; students may evade attendance by avoiding the camera; and relying solely on facial information to assess class attendance may lead to misjudgments. Summary of the Invention:

[0005] The technical problem to be solved by this invention is to provide a method for panoramic attendance monitoring in smart classrooms based on deep learning, which effectively overcomes the limitations of existing classroom attendance systems and brings new technical solutions to the field of smart education.

[0006] To address the aforementioned technical issues, this invention provides a method for panoramic attendance monitoring in a smart classroom based on deep learning, specifically comprising the following steps: S1: Using a facial feature extraction model, construct a collection of student facial information from multiple angles; S2: A camera captures panoramic photos in real time, and then these panoramic photos are processed by a facial detection model and the same facial feature extraction model used in S1 to obtain a collection of panoramic facial feature vectors. Finally, a check-in record database and a check-in photo database are established using a cosine distance matrix; S3: Based on the information from the check-in record database and the check-in photo database, the attendance status of each student is interactively output.

[0007] As a preferred technical solution of the present invention, the step S1 of constructing the student multi-angle facial information collection specifically includes the following steps:

[0008] S1-1: First, organize students to collect photos from multiple angles and enter them into the database. The multiple angles include frontal face angle, left side face angle, and right side face angle.

[0009] S1-2: Next, Gaussian blur and affine transformation methods are used to perform data augmentation on the student photos in the database;

[0010] S1-3: Finally, the student photos are fed into the face feature extraction model M1 to extract feature vectors and construct a multi-angle face information set H for each student. This set H contains each student's student ID number H. d Facial photos m and multi-angle feature vectors {H fu ,Η fl ,Η fr}

[0011] As a preferred technical solution of the present invention, the step S2 of establishing the sign-in record database and the sign-in photo database specifically includes the following steps:

[0012] S2-1: Determine if the current time is between 10 minutes before the start of class and the end of get out of class. If yes, execute S2-2; otherwise, sleep for 1 minute.

[0013] S2-2: Collect panoramic photos of the classroom using a camera, and construct an attendance list Q with "class name-date-time" as the attribute. The length of the attendance list is m, where m represents the length of the collection of students' multi-angle facial information. Execute S2-3.

[0014] S2-3: Input the collected panoramic photos into the face detection model M2. The face detection model M2 outputs a collection of face images. Then execute S2-4.

[0015] S2-4: Use the set of face images as input to the face feature extraction model M1 to extract the panoramic face feature vector set K. f Execute S2-5;

[0016] S2-5: Based on the panoramic face feature vector set K f For each feature vector in the dataset, calculate the minimum cosine distance with each multi-angle feature vector in the student multi-angle face information set H to obtain the cosine distance matrix. Then, execute S2-6, the calculation formula of which is:

[0017]

[0018]

[0019]

[0020] Where d represents the calculation of cosine distance, and md represents the calculation of the minimum cosine distance. This represents the i-th feature vector set in the panoramic face feature vector set. Let represent the j-th feature vector set in the information set, D represent the cosine distance matrix of size n*m, the length of the panoramic face feature vector set is n, and the length of the student multi-angle face information set is m.

[0021] S2-6: Calculate the minimum value for each column index in the cosine distance matrix, compare the minimum value with a preset threshold. If it is less than the preset threshold, execute S2-7; otherwise, execute S2-8. The preset threshold is 0.1.

[0022] S2-7: For each column index, the index position of the record check-in list is 1, and execute S2-9;

[0023] S2-8: For each column index, the index position of the record check-in list is 0, and execute S2-9;

[0024] S2-9: Store the sign-in list in the sign-in record library, store the panoramic photo in the sign-in photo library, and repeat S2-1.

[0025] As a preferred technical solution of the present invention, the interactive output of each student's attendance status in step S3 specifically includes the following steps:

[0026] S3-1: Extract the status of each student's index in the sign-in record database for each sign-in list and construct a sign-in status list Z. index ,in c represents the number of check-in records in the check-in record database. Execute S3-2.

[0027] S3-2: Sum the sign-in status list to get the sign-in count for student index, then execute S3-3;

[0028] S3-3: Determine whether the number of sign-in attempts is greater than the sign-in threshold. If yes, execute S3-4; otherwise, execute S3-5. The sign-in threshold is set to 10.

[0029] S3-4: Output that the student did not arrive late or leave early, and attendance is successful;

[0030] S3-5: Extract the first and last panoramic images from the sign-in photo library and send them to the teacher, then execute S-6;

[0031] S3-6: The teacher makes a second judgment based on the panoramic image in S3-5. If the student is indeed late or leaves early, then S3-7 is executed; otherwise, S3-4 is executed.

[0032] S3_7: Output that the student was late or left early, and the attendance record is not passed.

[0033] As a preferred technical solution of the present invention, the network structure of the face feature extraction model M1 is as follows: the input is a 224×224 image, the output is a 512-dimensional vector, and from its input end to its output end, it includes a first-stage convolutional layer, a first-stage pooling layer, a second-stage three residual blocks, a third-stage four residual blocks, a fourth-stage six residual blocks, a fifth-stage three residual blocks, a sixth-stage normalization layer, and a seventh-stage fully connected layer.

[0034] As a preferred technical solution of the present invention, the network structure of the face detection model M2 is as follows: the input is a panoramic photo, and the output is three feature maps, representing classification prediction, face bounding box position, and face key point position, respectively; from its input end to its output end, it includes the following in sequence: the first stage image size reshaping; the second stage multiple convolutional layers; the third stage convolutional layer output feature map F1; the fourth stage convolutional layer output feature map F2; the fifth stage convolutional layer output feature map F3; the sixth stage convolves F1, F2, and F3 respectively to obtain F1*, F2*, and F3*; the seventh stage convolves F3* to obtain S1; the eighth stage samples F3* and adds it to F2* to obtain F2** and convolves it to obtain S2; the ninth stage samples F2** and adds it to F1* and convolves it to obtain S3; the tenth stage fuses the features of S1, S2, and S3 and performs non-maximum suppression to obtain three feature maps.

[0035] As a preferred technical solution of the present invention, the three feature maps respectively have the following features: the size of the classification prediction is 16800×2, and the category is 0 or 1; the size of the face box position is 16800×4, and the position includes the coordinates of the upper left and lower right corners; the size of the face key point position is 16800×10, and includes the coordinates of the upper left and lower right corners of the five face key points.

[0036] Compared with the relevant prior art, the present application has the following main technical advantages:

[0037] Significant cost-effectiveness: Compared to the high-cost solution of installing human body collection devices at each seat, this invention greatly reduces the overall cost of the system by using panoramic cameras to achieve full coverage of the classroom.

[0038] Improved accuracy and efficiency: By leveraging deep learning technology and facial recognition algorithms, the accuracy and automation of student identity verification have been enhanced. Automated processing and analysis of panoramic image data have reduced reliance on manual intervention, improving the efficiency of the attendance process.

[0039] Comprehensive monitoring and assessment capabilities: The use of panoramic cameras reduces blind spots and increases the comprehensiveness and coverage of the attendance system.

[0040] Improving the quality of teaching management: Accurate attendance data can help schools better manage student attendance, improve teaching quality and student participation. Attached image description:

[0041] Figure 1 This is a flowchart illustrating a method for panoramic attendance monitoring in a smart classroom based on deep learning, provided by the present invention.

[0042] Figure 2 This is a schematic diagram illustrating the process of constructing a collection of multi-angle facial information of students according to an embodiment of the present invention;

[0043] Figure 3 This is a schematic diagram illustrating the process of establishing a sign-in record library and a sign-in photo library according to an embodiment of the present invention;

[0044] Figure 4 This is a flowchart illustrating the interactive output of each student's attendance status according to an embodiment of the present invention;

[0045] Figure 5 This is the network structure of the face feature extraction model M1 provided in this embodiment of the invention;

[0046] Figure 6 This is the network structure of the face detection model M2 provided in this embodiment of the invention. Detailed implementation method:

[0047] The present invention will be further described below with reference to the accompanying drawings and embodiments. However, the present invention can be implemented in many different ways and should not be construed as limited to the embodiments shown; rather, these embodiments provide those skilled in the art with implementation methods that meet applicable legal requirements.

[0048] Example 1: As Figure 1 As shown, this invention provides a method for panoramic attendance monitoring in smart classrooms based on deep learning, which completes panoramic monitoring of smart classroom attendance through the following steps:

[0049] S1: Use a facial feature extraction model to construct a collection of students' multi-angle facial information;

[0050] S2: The camera captures panoramic photos in real time. Then, the panoramic photos are processed by a face detection model and the same face feature extraction model as in S1 to obtain a set of panoramic face feature vectors. Finally, a check-in record library and a check-in photo library are established through the cosine distance matrix.

[0051] S3: Interactively output each student's attendance status using information from the sign-in record library and sign-in photo library.

[0052] like Figure 2 As shown, this invention provides a method for panoramic attendance monitoring in smart classrooms based on deep learning, which constructs a collection of students' multi-angle facial information through the following steps:

[0053] S1-1: First, organize students to collect photos from multiple angles and enter them into the database. The multiple angles include frontal face angle, left side face angle, and right side face angle.

[0054] S1-2: Next, Gaussian blur and affine transformation methods are used to perform data augmentation on the student photos in the database;

[0055] S1-3: Finally, the student photos are fed into the face feature extraction model M1 to extract feature vectors and construct a multi-angle face information set H for each student. This set H contains each student's student ID number H. d Facial photos m and multi-angle feature vectors {H fu ,Η fl ,Η fr}

[0056] like Figure 3 As shown, this invention provides a method for panoramic attendance monitoring in smart classrooms based on deep learning, which establishes an attendance record library and an attendance photo library through the following steps:

[0057] S2-1: Determine if the current time is between 10 minutes before the start of class and the end of get out of class. If yes, execute S2-2; otherwise, sleep for 1 minute.

[0058] S2-2: Collect panoramic photos of the classroom using a camera, and construct an attendance list Q with "class name-date-time" as the attribute. The length of the attendance list is m, where m represents the length of the collection of students' multi-angle facial information. Execute S2-3.

[0059] S2-3: Input the collected panoramic photos into the face detection model M2. The face detection model M2 outputs a collection of face images. Then execute S2-4.

[0060] S2-4: Use the set of face images as input to the face feature extraction model M1 to extract the panoramic face feature vector set K. f Execute S2-5;

[0061] S2-5: Based on the panoramic face feature vector set K f For each feature vector in the dataset, calculate the minimum cosine distance with each multi-angle feature vector in the student multi-angle face information set H to obtain the cosine distance matrix. Then, execute S2-6, the calculation formula of which is:

[0062]

[0063]

[0064]

[0065] Where d represents the calculation of cosine distance, and md represents the calculation of the minimum cosine distance. This represents the i-th feature vector set in the panoramic face feature vector set. Let represent the j-th feature vector set in the information set, D represent the cosine distance matrix of size n*m, the length of the panoramic face feature vector set is n, and the length of the student multi-angle face information set is m.

[0066] S2-6: Calculate the minimum value for each column index in the cosine distance matrix, compare the minimum value with a preset threshold. If it is less than the preset threshold, execute S2-7; otherwise, execute S2-8. The preset threshold is 0.1.

[0067] S2-7: For each column index, the index position of the record check-in list is 1, and execute S2-9;

[0068] S2-8: For each column index, the index position of the record check-in list is 0, and execute S2-9;

[0069] S2-9: Store the sign-in list in the sign-in record library, store the panoramic photo in the sign-in photo library, and repeat S2-1.

[0070] like Figure 4 As shown, this invention provides a method for panoramic attendance monitoring in a smart classroom based on deep learning, which completes the interactive output of each student's attendance status through the following steps:

[0071] S3-1: Extract the status of each student's index in the sign-in record database for each sign-in list and construct a sign-in status list Z. index ,in c represents the number of check-in records in the check-in record database. Execute S3-2.

[0072] S3-2: Sum the sign-in status list to get the sign-in count for student index, then execute S3-3;

[0073] S3-3: Determine whether the number of sign-in attempts is greater than the sign-in threshold. If yes, execute S3-4; otherwise, execute S3-5. The sign-in threshold is set to 10.

[0074] S3-4: Output that the student did not arrive late or leave early, and attendance is successful;

[0075] S3-5: Extract the first and last panoramic images from the sign-in photo library and send them to the teacher. Then execute S3-6.

[0076] S3-6: The teacher makes a second judgment based on the panoramic image in S3-5. If the student is indeed late or leaves early, then S3-7 is executed; otherwise, S3-4 is executed.

[0077] S3-7: Output that the student was late or left early, and the attendance record is not passed.

[0078] like Figure 5 As shown, the network structure of the face feature extraction model M1 is as follows: the input is a 224×224 image, and the output is a 512-dimensional vector. From its input end to its output end, it includes the following layers in sequence: first stage convolutional layer, first stage pooling layer, second stage three residual blocks, third stage four residual blocks, fourth stage six residual blocks, fifth stage three residual blocks, sixth stage normalization layer, and seventh stage fully connected layer.

[0079] like Figure 6 As shown, the network structure of the face detection model M2 is as follows: the input is a panoramic photo, and the output is three feature maps, representing classification prediction, face bounding box location, and face key point location, respectively. From its input to its output, the network structure includes: the first stage image resizing; the second stage multiple convolutional layers; the third stage convolutional layer (output feature map F1); the fourth stage convolutional layer (output feature map F2); the fifth stage convolutional layer (output feature map F3); the sixth stage convolutions F1, F2, and F3 respectively to obtain F1*, F2*, and F3*; the seventh stage convolutions F3* to obtain S1; the eighth stage samples F3* and adds it to F2* to obtain F2** and convolutions it to obtain S2; the ninth stage samples F2** and adds it to F1* and convolutions it to obtain S3; the tenth stage fuses the features of S1, S2, and S3 and performs non-maximum suppression to obtain three feature maps (16800×2, 16800×4, and 16800×10). In the tenth stage, 2 in the feature map represents the classification prediction, yes or no; 4 represents the coordinates of the top left and bottom right corners of the face bounding box; and 10 represents the coordinates of the top left and bottom right corners of the five key points of the face.

[0080] In summary, this patent, through two innovative deep learning network algorithms and a panoramic monitoring and attendance solution, not only improves the efficiency and accuracy of classroom attendance but also brings a more advanced, efficient, and secure technical solution to the field of smart education.

[0081] The above embodiments merely illustrate several implementation methods of the present invention, and their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention.

Claims

1. A method for panoramic attendance monitoring in a smart classroom based on deep learning, characterized in that, The method includes: S1: Use a facial feature extraction model to construct a collection of students' multi-angle facial information; S2: The camera captures panoramic photos in real time. Then, the panoramic photos are processed by a face detection model and the same face feature extraction model as in S1 to obtain a set of panoramic face feature vectors. Finally, a check-in record library and a check-in photo library are established through the cosine distance matrix. S3: Interactively output each student's attendance status using information from the sign-in record library and sign-in photo library.

2. The method for panoramic attendance monitoring in a smart classroom based on deep learning according to claim 1, characterized in that, Step S1, which involves constructing a collection of students' multi-angle facial information, specifically includes the following steps: S1-1: First, organize students to collect photos from multiple angles and enter them into the database. The multiple angles include frontal face angle, left side face angle, and right side face angle. S1-2: Next, Gaussian blur and affine transformation methods are used to perform data augmentation on the student photos in the database; S1-3: Finally, the student photos are fed into the face feature extraction model M1 to extract feature vectors and construct a multi-angle face information set H for each student. This set H contains each student's student ID number H. d Facial photos m And multi-angle feature vectors {Hfu,Hfl,Hfr}.

3. The method for panoramic attendance monitoring in a smart classroom based on deep learning according to claim 1, characterized in that, Step S2, which involves establishing the sign-in record library and the sign-in photo library, specifically includes the following steps: S2-1: Determine if the current time is between 10 minutes before the start of class and the end of get out of class. If yes, execute S2-2; otherwise, sleep for 1 minute. S2-2: Collect panoramic photos of the classroom using a camera, and construct an attendance list Q with "class name-date-time" as the attribute. The length of the attendance list is m, where m represents the length of the collection of students' multi-angle facial information. Execute S2-3. S2-3: Input the collected panoramic photos into the face detection model M2. The face detection model M2 outputs a collection of face images. Then execute S2-4. S2-4: Use the set of face images as input to the face feature extraction model M1 to extract the panoramic face feature vector set K. f Execute S2-5; S2-5: Based on the panoramic face feature vector set K f For each feature vector in the dataset, calculate the minimum cosine distance with each multi-angle feature vector in the student multi-angle face information set H to obtain the cosine distance matrix. Then, execute S2-6, the calculation formula of which is: Where d represents the calculation of cosine distance, and md represents the calculation of the minimum cosine distance. This represents the i-th feature vector set in the panoramic face feature vector set. Let represent the j-th feature vector set in the information set, D represent the cosine distance matrix of size n*m, the length of the panoramic face feature vector set is n, and the length of the student multi-angle face information set is m. S2-6: Calculate the minimum value for each column index in the cosine distance matrix, compare the minimum value with a preset threshold. If it is less than the preset threshold, execute S2-7; otherwise, execute S2-8. The preset threshold is 0.

1. S2-7: For each column index, the index position of the record check-in list is 1, and execute S2-9; S2-8: For each column index, the index position of the record check-in list is 0, and execute S2-9; S2-9: Store the sign-in list in the sign-in record library, store the panoramic photo in the sign-in photo library, and repeat S2-1.

4. The method for panoramic attendance monitoring in a smart classroom based on deep learning according to claim 1, characterized in that, Step S3, which interactively outputs the attendance status of each student, specifically includes the following steps: S3-1: Extract the status of each student's index in the sign-in record database for each sign-in list and construct a sign-in status list Z. index ,in c represents the number of check-in records in the check-in record database. Execute S3-2. S3-2: Sum the sign-in status list to get the sign-in count for student index, then execute S3-3; S3-3: Determine whether the number of sign-in attempts is greater than the sign-in threshold. If yes, execute S3-4; otherwise, execute S3-5. The sign-in threshold is set to 10. S3-4: Output that the student did not arrive late or leave early, and attendance is successful; S3-5: Extract the first and last panoramic images from the sign-in photo library and send them to the teacher. Then execute S3-6. S3-6: The teacher makes a second judgment based on the panoramic image in S3-5. If the student is indeed late or leaves early, then S3-7 is executed; otherwise, S3-4 is executed. S3-7: Output that the student was late or left early, and the attendance record is not passed.

5. A method for panoramic attendance monitoring in a smart classroom based on deep learning, as described in claims 1 to 3, characterized in that, The network structure of the face feature extraction model M1 is as follows: the input is a 224×224 image, and the output is a 512-dimensional vector. From its input end to its output end, it includes the following layers in sequence: first stage convolutional layer, first stage pooling layer, second stage three residual blocks, third stage four residual blocks, fourth stage six residual blocks, fifth stage three residual blocks, sixth stage normalization layer, and seventh stage fully connected layer.

6. A method for panoramic attendance monitoring in a smart classroom based on deep learning, as described in claims 1 to 3, characterized in that, The network structure of the face detection model M2 is as follows: the input is the photo captured by the original camera device, and the output is three feature maps, which represent the classification prediction, the face bounding box position, and the face key point position, respectively. From its input to its output, the process includes the following steps: First stage: image resizing; Second stage: multiple convolutional layers; Third stage: convolutional layer output feature map F1; Fourth stage: convolutional layer output feature map F2; Fifth stage: convolutional layer output feature map F3; Sixth stage: convolution of F1, F2, and F3 respectively to obtain F1*, F2*, and F3*; Seventh stage: convolution of F3* to obtain S1; Eighth stage: sampling of F3* and adding it to F2* to obtain F2** and convolution to obtain S2; Ninth stage: sampling of F2** and adding it to F1* and convolution to obtain S3; Tenth stage: fusing the features of S1, S2, and S3 and performing non-maximum suppression to obtain three feature maps.

7. The method for panoramic attendance monitoring in a smart classroom based on deep learning as described in claim 6, characterized in that, The three feature maps have the following features: the size of the classification prediction is 16800×2, and the category is 0 or 1; the size of the face bounding box is 16800×4, and the position includes the coordinates of the upper left and lower right corners; the size of the face key point position is 16800×10, and it includes the coordinates of the upper left and lower right corners of the five face key points.

Images