College classroom behavior detection system based on computer vision

Abstract

The invention discloses a college classroom behavior detection system based on computer vision, relates to the field of computer vision and college intelligent teaching, and solves the problem that it is difficult to create an API in a college intelligent platform to gather non-listening behavior data of each class. The technical problems that Pconv is used for replacing part of convolution in an original algorithm for a possibly-shielded or damaged area of a college classroom picture, and detection of data transmission safety is lacked are solved, and the method comprises the following modules: a system deployment and configuration module used for creating an API in an existing college intelligent platform; the data set making and model training module is used for constructing a computer vision model for the preprocessed data; the real-time detection and data processing module is used for identifying classroom behaviors of students; the data uploading and summarizing module is used for summarizing and counting non-listening behaviors according to the course information; and the privacy protection and security measure module is used for performing encryption processing in the data transmission process.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision and university smart teaching, and specifically relates to a university classroom behavior detection system based on computer vision. Background Art

[0002] With the rapid development of information technology and the continuous advancement of educational informatization, the teaching methods of colleges and universities are undergoing profound changes. Traditional classroom teaching evaluation mainly relies on teachers' subjective observations and students' self-reports. This method has problems such as lack of objectivity and low evaluation efficiency. In recent years, the rapid development of computer vision technology has provided new ideas and methods for solving this problem.

[0003] Computer vision is a science that studies how to make machines "see". It uses cameras and computers to simulate the functions of the human eye to identify, track and measure targets, and further processes graphics so that the computer processes them into images that are more suitable for human observation or transmission to instruments for detection. In computer vision technology, the breakthrough progress of deep learning, especially convolutional neural networks, has brought revolutionary changes to the fields of face detection, posture estimation, and behavior recognition. These technologies are not only highly accurate, but can also process large amounts of data in real time, making it possible to detect behavior in college classrooms.

[0004] In the field of education, classroom student behavior detection systems based on computer vision have gradually become a research hotspot. This type of system collects classroom video information through cameras and uses deep learning algorithms to analyze and identify student behavior in the video, thereby achieving real-time monitoring and evaluation of classroom status. Compared with traditional evaluation methods, this system has the advantages of high objectivity, strong real-time performance, and large data volume. It can provide teachers with more comprehensive and accurate feedback on students' classroom performance, which helps improve teaching quality and efficiency.

[0005] However, existing computer vision-based classroom behavior detection systems still face some challenges in practical applications. For example, complex and changing scenarios such as environmental conditions in different classrooms and dynamic changes in student seats place higher demands on the robustness and accuracy of the system. In addition, how to effectively protect student privacy and data security is also an important issue that must be considered when designing the system.

[0006] Therefore, the present invention aims to propose a computer vision-based college classroom behavior detection system, which improves the adaptability and detection accuracy of complex and changing scenarios by optimizing algorithm design and system architecture, while focusing on privacy protection and data security, and provides strong technical support for smart teaching in colleges and universities. The successful implementation of the present invention is expected to promote the process of educational informatization and promote the overall improvement of the teaching quality of colleges and universities. Summary of the invention

[0007] The present invention aims to solve at least one of the technical problems existing in the prior art; to this end, the present invention proposes a college classroom behavior detection system based on computer vision, which is used to solve the following technical problems: Existing classroom behavior detection systems based on computer vision still face some challenges in practical applications. For example, complex and changing scenarios such as environmental conditions in different classrooms and dynamic changes in student seats place higher demands on the robustness and accuracy of the system. In addition, how to effectively protect student privacy and data security is also an important issue that must be considered when designing the system.

[0008] To solve the above problems, the first aspect of the present invention provides a college classroom behavior detection system based on computer vision, comprising the following modules: System deployment and configuration module: collect student pictures in the classroom through the camera, connect the main controller to the camera, and create an API interface in the university smart platform; Dataset creation and model training module: Use classroom cameras to collect pictures and video data of student behavior in the classroom, clean and pre-process the collected data, and use the labeled training set to train the computer vision model; Real-time detection and data processing module: During class, the camera collects student images in real time and transmits them to the main controller. The main controller runs the trained computer vision model to process the received images and identify students' classroom behaviors. The main controller organizes the detection results into structured data. Data upload and summary module: The main controller uploads the test results to the server through the API interface of the university smart platform. The server receives and stores the uploaded data and summarizes and counts the behavior of not attending classes based on the course information; Privacy protection and security measures module: Encryption during data transmission, control of data access rights, and regular security checks and updates of the system.

[0009] As a further solution of the present invention, the method of collecting student pictures in the classroom by using a camera, connecting the main controller with the camera, and creating an API interface in the university smart platform includes the following steps: Install or confirm that there is an existing camera at the back of the university classroom, use the camera to collect pictures of students in the classroom, connect the main controller to the camera, configure relevant parameters, including image resolution and acquisition frequency, and create an API interface in the university smart platform to receive and process the detection data uploaded by the main controller.

[0010] As a further solution of the present invention, the method of collecting classroom student behavior pictures and video data using classroom cameras, cleaning and preprocessing the collected data, and using the labeled training set to train the computer vision model includes the following steps: Use classroom cameras to collect pictures and video data of student behavior in the classroom, clean and pre-process the collected data, remove invalid parts from the recorded video, including break time and non-teaching time, and automatically take pictures of the video at 1 second intervals to obtain static pictures of students under different times and lighting conditions; pre-processing includes denoising and contrast enhancement, and using annotation software to draw bounding boxes and add labels on the cleaned and pre-processed images to classify student behavior, including those who listen carefully and those who do not listen carefully, and draw a bounding box for each student; Draw green bounding boxes for students who are paying attention to the class; For students who do not listen to the class carefully, we draw different colored boundary boxes according to the classified behaviors. For students who look down at their phones, we draw red boundary boxes. For students who turn their heads to talk, we draw blue boundary boxes. For students who sleep, we draw yellow boundary boxes. Label each bounding box in the annotation software. The label describes the type of student behavior. Use the labeled data set to train the computer vision model. Among them, listening carefully to the class includes the student's head facing the blackboard or the teacher, and the student's eyes facing the blackboard or the teacher; Not paying attention in class includes students looking down at their phones, students turning their heads to talk, and students sleeping.

[0011] As a further solution of the present invention, the method of training a computer vision model using a labeled data set comprises the following steps: Find the implementation part of the C3 module in the YoloV5s detection algorithm, replace the ordinary convolution layer in the C3 module with the Pconv layer, use the replaced YoloV5s model framework to build a computer vision model, and use the labeled data set to train the computer vision model.

[0012] As a further solution of the present invention, the main controller runs a trained computer vision model to process the received pictures and recognize the students' classroom behaviors, including the following steps: Deploy the trained computer vision model on the main controller, use the deep learning framework to load the model, and process the received images. Preprocess each received image, including denoising, enhancement, and image resizing. Use the loaded computer vision model to identify the preprocessed images and identify students' classroom behaviors. The computer vision model classifies the student behaviors in the images and outputs the predicted labels for each student. Organize the output results of the model into a structured data format, including timestamp, student ID, student behavior label, classroom number, and course name, and store the structured detection results in the database.

[0013] As a further solution of the present invention, the summary statistics of the behavior of not attending classes according to the course information include the following steps: An API interface is configured on the university smart platform to receive the detection data uploaded by the main controller. The uploaded data is in ISON format and contains timestamp, student ID, student behavior label, classroom number and course name. It also has an identity authentication mechanism. The data is read from the database and the behaviors of not attending classes are summarized and counted according to the course information. The behaviors are grouped according to the course ID and the aggregation function is used to count the number of students corresponding to different category labels of not attending classes in each course to obtain the total number of students not attending classes in each course. The calculation formula for the total number of students not attending class in each course is:

[0014] in, is the total number of students not attending class in each course, The number of students who looked down at their phones in each course, The number of students who turned their heads to talk in each course, is the number of students who engage in sleeping behavior in each course.

[0015] As a further solution of the present invention, the encryption process is performed during data transmission to control data access rights, including the following steps: A random symmetric key is generated on the client side, including the teacher side and the student account side of this class. The public key symmetric key of the asymmetric encryption algorithm is used to encrypt the key. The encrypted key is sent to the server side. After the server side uses its own private key to decrypt the symmetric key, both the client and the server side use the symmetric key to encrypt and decrypt the data.

[0016] Beneficial effects of the present invention: The present invention provides objective feedback to teachers by detecting students' classroom behaviors in real time, helping teachers to understand classroom conditions in a timely manner and adjust teaching methods, thereby improving teaching quality and efficiency. By creating an API in the university smart platform, the present invention not only saves teaching videos, but also summarizes the data of non-attending behavior in each class, providing strong support for teachers' long-term teaching improvement and educational research; The present invention utilizes existing cameras in university classrooms to collect data, without the need to invest a large amount of money to purchase equipment, thus reducing the implementation cost of the system; The present invention effectively reduces the impact of ambient light changes on detection results by creating a data set containing data under different ambient light conditions, thereby improving the robustness and accuracy of the system; Aiming at the problem that the size of university classroom pictures is large and there may be occlusion or damaged areas, the present invention optimizes the detection algorithm and uses Pconv to replace part of the convolution in the original algorithm, thereby reducing the influence of invalid pixels on feature extraction and improving the detection performance and reasoning speed of the model. The present invention effectively avoids data leakage and protects students' privacy rights and interests by transmitting the detection data only to the teacher's end and the student account end of the current class. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 It is a flowchart of the implementation process of the present invention; Figure 2 It is a schematic diagram of the university classroom space of the present invention. DETAILED DESCRIPTION

[0019] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0020] See also Figure 1-2 As shown, the present invention is a college classroom behavior detection system based on computer vision, including the following modules: System deployment and configuration module: collect student pictures in the classroom through the camera, connect the main controller to the camera, and create an API interface in the university smart platform; Dataset creation and model training module: Use classroom cameras to collect pictures and video data of student behavior in the classroom, clean and pre-process the collected data, and use the labeled training set to train the computer vision model; Real-time detection and data processing module: During class, the camera collects student images in real time and transmits them to the main controller. The main controller runs the trained computer vision model to process the received images and identify students' classroom behaviors. The main controller organizes the detection results into structured data. Data upload and summary module: The main controller uploads the test results to the server through the API interface of the university smart platform. The server receives and stores the uploaded data and summarizes and counts the behavior of not attending classes based on the course information; Privacy protection and security measures module: Encryption during data transmission, control of data access rights, and regular security checks and updates of the system.

[0021] Specifically, the camera is used to collect pictures of students in the classroom, the main controller is used to process the pictures collected by the camera and upload the detection results to the API of the university's smart platform, and the smart platform is used to analyze and summarize the data and send it to the corresponding client; an API is created in the existing university platform to save the teaching video of each class and also summarize the number of people who do not listen to the class, so that the instructor can adjust the teaching method; cameras have been installed in the back of most university classrooms, so that a large amount of real data can be collected for the production of data sets, so as to avoid the impact of different ambient light on the detection results; because the picture size in university classrooms is too large, for the occluded and damaged areas in the pictures, Pconv is used to replace the ordinary convolution part of the C3 module in the original algorithm to reduce the impact of invalid pixels on feature extraction, thereby improving the detection performance of the model; the detection data is only transmitted to the teacher's end and the account end of the students in this class, which avoids data leakage and protects the students' privacy to a relative extent.

[0022] In one embodiment of the present invention, the method of collecting student pictures in the classroom by using a camera, connecting the main controller to the camera, and creating an API interface in the university smart platform includes the following steps: Install or confirm that there is an existing camera at the back of the university classroom, use the camera to collect pictures of students in the classroom, connect the main controller to the camera, configure relevant parameters, including image resolution and acquisition frequency, and create an API interface in the university smart platform to receive and process the detection data uploaded by the main controller.

[0023] Specifically, install or confirm that there is an existing camera at the back of the university classroom to ensure that the images of students in the classroom can be clearly captured, connect the main controller to the camera, configure relevant parameters such as image resolution, acquisition frequency, etc., and create an API interface in the university smart platform to receive and process the detection data uploaded by the main controller.

[0024] In one embodiment of the present invention, the method of collecting classroom student behavior pictures and video data using classroom cameras, cleaning and preprocessing the collected data, and using the labeled training set to train the computer vision model includes the following steps: Use classroom cameras to collect pictures and video data of student behavior in the classroom, clean and pre-process the collected data, remove invalid parts from the recorded video, including break time and non-teaching time, and automatically take pictures of the video at 1 second intervals to obtain static pictures of students under different times and lighting conditions; pre-processing includes denoising and contrast enhancement, and using annotation software to draw bounding boxes and add labels on the cleaned and pre-processed images to classify student behavior, including those who listen carefully and those who do not listen carefully, and draw a bounding box for each student; Draw green bounding boxes for students who are paying attention to the class; For students who do not listen to the class carefully, we draw different colored boundary boxes according to the classified behaviors. For students who look down at their phones, we draw red boundary boxes. For students who turn their heads to talk, we draw blue boundary boxes. For students who sleep, we draw yellow boundary boxes. Label each bounding box in the annotation software. The label describes the type of student behavior. Use the labeled data set to train the computer vision model. Among them, listening carefully to the class includes the student's head facing the blackboard or the teacher, and the student's eyes facing the blackboard or the teacher; Not paying attention in class includes students looking down at their phones, students turning their heads to talk, and students sleeping.

[0025] Specifically, we use classroom cameras to collect a large amount of real images and video data of student behaviors in the classroom, clean and preprocess the collected data, remove cluttered and invalid data, annotate the cleaned data, divide student behaviors into those who listen carefully and those who do not, and then divide the behaviors of listening carefully and not listening carefully. We draw different color bounding boxes for each student, and use the annotated data set to train the computer vision model.

[0026] In one embodiment of the present invention, the method of training a computer vision model using a labeled data set includes the following steps: Find the implementation part of the C3 module in the YoloV5s detection algorithm, replace the ordinary convolution layer in the C3 module with the Pconv layer, use the replaced YoloV5s model framework to build a computer vision model, and use the labeled data set to train the computer vision model.

[0027] Specifically, the computer vision model is trained using the labeled dataset, and the optimized YOLOv5s model is used, in which PConv is used to replace the ordinary convolution part in the C3 module.

[0028] In one embodiment of the present invention, the main controller runs a trained computer vision model to process received images and recognize students' classroom behaviors, including the following steps: Deploy the trained computer vision model on the main controller, use the deep learning framework to load the model, and process the received images. Preprocess each received image, including denoising, enhancement, and image resizing. Use the loaded computer vision model to identify the preprocessed images and identify students' classroom behaviors. The computer vision model classifies the student behaviors in the images and outputs the predicted labels for each student. Organize the output results of the model into a structured data format, including timestamp, student ID, student behavior label, classroom number, and course name, and store the structured detection results in the database.

[0029] Specifically, during class, the camera collects student images in real time and transmits them to the main controller. The main controller runs the trained computer vision model, processes the received images in real time, and identifies students' classroom behaviors. The main controller organizes the detection results into structured data, including timestamp, student ID, student behavior label, classroom number and course name.

[0030] In one embodiment of the present invention, the process of collecting statistics on the behavior of not attending classes according to the course information includes the following steps: An API interface is configured on the university smart platform to receive the detection data uploaded by the main controller. The uploaded data is in ISON format and contains timestamp, student ID, student behavior label, classroom number and course name. It also has an identity authentication mechanism. The data is read from the database and the behaviors of not attending classes are summarized and counted according to the course information. The behaviors are grouped according to the course ID and the aggregation function is used to count the number of students corresponding to different category labels of not attending classes in each course to obtain the total number of students not attending classes in each course. The calculation formula for the total number of students not attending class in each course is:

[0031] in, is the total number of students not attending class in each course, The number of students who looked down at their phones in each course, The number of students who turned their heads to talk in each course, is the number of students who engage in sleeping behavior in each course.

[0032] Specifically, the main controller uploads the detection results to the server through the API interface of the university smart platform. The server receives and stores the uploaded data, and summarizes and counts the behaviors of not paying attention to the class based on the course information. Bounding boxes of corresponding colors are drawn for each type of student not paying attention to the class behavior. The number of boundary boxes of different colors corresponds to the number of students with different not paying attention behaviors. The total number of students who do not pay attention in each class is obtained by adding them up.

[0033] In one embodiment of the present invention, the encryption process during data transmission and the control of data access rights include the following steps: A random symmetric key is generated on the client side, including the teacher side and the student account side of this class. The public key symmetric key of the asymmetric encryption algorithm is used to encrypt the key. The encrypted key is sent to the server side. After the server side uses its own private key to decrypt the symmetric key, both the client and the server side use the symmetric key to encrypt and decrypt the data.

[0034] Specifically, provide access rights to the teacher and the student account of this class, allowing relevant users to view test results and statistical data, ensure encryption during data transmission, prevent data leakage, and strictly control data access rights, allowing only authorized users to access test results and statistical data.

[0035] The above embodiments are only used to illustrate the technical method of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical method of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical method of the present invention.

Claims

1. A computer vision-based college classroom behavior detection system, characterized in that: Includes the following modules: System deployment and configuration module: collect student pictures in the classroom through the camera, connect the main controller to the camera, and create an API interface in the university smart platform; Dataset creation and model training module: Use classroom cameras to collect pictures and video data of student behavior in the classroom, clean and pre-process the collected data, and use the labeled training set to train the computer vision model; Real-time detection and data processing module: During class, the camera collects student images in real time and transmits them to the main controller. The main controller runs the trained computer vision model to process the received images and identify students' classroom behaviors. The main controller organizes the detection results into structured data. Data upload and summary module: The main controller uploads the test results to the server through the API interface of the university smart platform. The server receives and stores the uploaded data and summarizes and counts the behavior of not attending classes based on the course information; Privacy protection and security measures module: Encryption during data transmission, control of data access rights, and regular security checks and updates of the system.

2. According to the computer vision-based college classroom behavior detection system of claim 1, it is characterized in that: The method of collecting student pictures in the classroom through a camera, connecting the main controller to the camera, and creating an API interface in the university smart platform includes the following steps: Install or confirm that there is an existing camera at the back of the university classroom, use the camera to collect pictures of students in the classroom, connect the main controller to the camera, configure relevant parameters, including image resolution and acquisition frequency, and create an API interface in the university smart platform to receive and process the detection data uploaded by the main controller.

3. According to the computer vision-based college classroom behavior detection system of claim 1, it is characterized in that: The method of using classroom cameras to collect classroom student behavior pictures and video data, cleaning and preprocessing the collected data, and using the labeled training set to train the computer vision model includes the following steps: Use classroom cameras to collect pictures and video data of student behavior in the classroom, clean and pre-process the collected data, remove invalid parts from the recorded video, including break time and non-teaching time, and automatically take pictures of the video at 1 second intervals to obtain static pictures of students under different times and lighting conditions; pre-processing includes denoising and contrast enhancement, and using annotation software to draw bounding boxes and add labels on the cleaned and pre-processed images to classify student behavior, including those who listen carefully and those who do not, and draw a bounding box for each student; Draw green bounding boxes for students who are paying attention to the class; For students who do not listen to the class carefully, we draw different colored boundary boxes according to the classified behaviors. For students who look down at their phones, we draw red boundary boxes. For students who turn their heads to talk, we draw blue boundary boxes. For students who sleep, we draw yellow boundary boxes. Label each bounding box in the annotation software. The label describes the type of student behavior. Use the labeled data set to train the computer vision model. Among them, listening carefully to the class includes the student's head facing the blackboard or the teacher, and the student's eyes facing the blackboard or the teacher; Not paying attention in class includes students looking down at their phones, students turning their heads to talk, and students sleeping.

4. A computer vision-based college classroom behavior detection system according to claim 3, characterized in that: The method of training a computer vision model using a labeled data set includes the following steps: Find the implementation part of the C3 module in the YoloV5s detection algorithm, replace the ordinary convolution layer in the C3 module with the Pconv layer, use the replaced YoloV5s model framework to build a computer vision model, and use the labeled data set to train the computer vision model.

5. The computer vision-based college classroom behavior detection system according to claim 1 is characterized in that: The main controller runs the trained computer vision model to process the received pictures and recognize the students' classroom behaviors, including the following steps: Deploy the trained computer vision model on the main controller, use the deep learning framework to load the model, and process the received images. Preprocess each received image, including denoising, enhancement, and image resizing. Use the loaded computer vision model to identify the preprocessed images and identify students' classroom behaviors. The computer vision model classifies the student behaviors in the images and outputs the predicted labels for each student. Organize the output results of the model into a structured data format, including timestamp, student ID, student behavior label, classroom number, and course name, and store the structured detection results in the database.

6. The computer vision-based college classroom behavior detection system according to claim 1 is characterized in that: The method of collecting and counting the behaviors of not attending classes according to the course information includes the following steps: An API interface is configured on the university smart platform to receive the detection data uploaded by the main controller. The uploaded data is in ISON format and contains timestamp, student ID, student behavior label, classroom number and course name. It also has an identity authentication mechanism. The data is read from the database and the behaviors of not attending classes are summarized and counted according to the course information. The behaviors are grouped according to the course ID and the aggregation function is used to count the number of students corresponding to different category labels of not attending classes in each course to obtain the total number of students not attending classes in each course. The calculation formula for the total number of students not attending class in each course is: in, is the total number of students not attending class in each course, The number of students who looked down at their phones in each course, The number of students who turned their heads to talk in each course, is the number of students who engage in sleeping behavior in each course.

7. The computer vision-based college classroom behavior detection system according to claim 1 is characterized in that: The encryption process is performed during the data transmission process to control the data access rights, including the following steps: A random symmetric key is generated on the client side, including the teacher side and the student account side of this class. The public key symmetric key of the asymmetric encryption algorithm is used to encrypt the key. The encrypted key is sent to the server side. After the server side uses its own private key to decrypt the symmetric key, both the client and the server side use the symmetric key to encrypt and decrypt the data.

Images