Classroom interaction quality evaluation method based on image recognition

Abstract

The invention relates to the technical field of image recognition, and particularly discloses a classroom interaction quality evaluation method based on image recognition, which comprises the following steps: acquiring a monitoring video, determining a sub-video, and recognizing the total number of times of raising hands of students in the sub-video; sub-videos are framed to obtain sub-video frames, m sub-video frames are selected as target frames, m is a preset number, and whether the class attending states of the students are abnormal or not is determined based on the target frames; marking abnormal students as to-be-determined students, counting the proportion of the to-be-determined students in a single target frame, and if the proportion is less than or equal to 0.2, marking the to-be-determined students as abnormal students; according to the total number of times that the students are marked as abnormal students, correcting the total number of times of hand raising to obtain a correction number of times, and performing classroom interaction quality evaluation according to the correction number of times. According to the invention, the computing resource consumption of classroom interaction quality evaluation based on image recognition is reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of image recognition, and particularly to a method for evaluating the quality of classroom interaction based on image recognition. Background Art

[0002] Image Recognition refers to the use of a computer to process, analyze, and understand images. It is a core technology in the field of Computer Vision, aiming to identify various different patterns of targets and objects through algorithms.

[0003] By using computer vision and image recognition technologies to analyze image or video data in the classroom scenario, teaching quality indicators such as teacher-student interaction and student engagement can be automatically evaluated. The core is to quantify classroom dynamics through visual data, replacing traditional manual observation or questionnaires to achieve more objective, real-time, and traceable teaching evaluation.

[0004] In large-class teaching with a large number of students, when using image recognition technology for interaction quality evaluation, face detection and expression recognition algorithms need to independently analyze the faces of each student. The algorithm complexity usually increases linearly or even exponentially with the number of targets. For example, when using a Convolutional Neural Network (CNN), for each additional detection object, the video memory and computing power required for model inference will increase synchronously. Secondly, high-precision pose estimation (such as OpenPose) needs to track the skeletal key points of dozens of people, and the calculation time of a single-frame image may extend from milliseconds in a single-person scenario to several seconds, resulting in a serious decline in real-time performance. In addition, to capture detailed actions (such as micro-expressions, small hand raises), the system often needs to increase the camera resolution or frame rate, further exacerbating the data volume expansion. Ordinary educational institutions are difficult to afford the deployment cost of high-performance computing clusters. These factors together lead to a bottleneck in the operation efficiency and cost control of large-scale classroom evaluation systems. Summary of the Invention

[0005] The purpose of the present invention is to provide a method for evaluating the quality of classroom interaction based on image recognition, and solve the following technical problems:

[0006] In large-class teaching with a large number of students, when using image recognition technology for interactive quality assessment, face detection and expression recognition algorithms need to independently analyze the faces of each student. Generally, the algorithm complexity increases linearly or even exponentially with the number of targets. For example, when using a convolutional neural network (CNN), with each additional detection object, the video memory and computing power required for model inference will increase synchronously. Secondly, high-precision pose estimation (such as OpenPose) needs to track the skeletal key points of dozens of people, and the calculation time for a single-frame image may extend from milliseconds in a single-person scenario to several seconds, resulting in a serious decline in real-time performance. In addition, to capture detailed actions (such as micro-expressions and slight hand-raising), the system often needs to increase the camera resolution or frame rate, further exacerbating the data volume expansion. Ordinary educational institutions can hardly afford the deployment cost of high-performance computing clusters. These factors together lead to bottlenecks in the operation efficiency and cost control of large-scale classroom assessment systems.

[0007] The object of the present invention can be achieved by the following technical solutions:

[0008] A method for evaluating the quality of classroom interaction based on image recognition, comprising the following steps:

[0009] Collect a monitoring video, determine a sub-video, and identify the total number of times a single student raises their hand in the sub-video;

[0010] Divide the sub-video into frames to obtain sub-video frames, select m sub-video frames as target frames, where m is a preset number, and determine whether there is an abnormality in the listening state of the students based on the target frames; mark the students with abnormalities as pending students, and count the proportion of pending students in a single target frame. If the proportion is less than or equal to 0.2, mark the pending students as abnormal students;

[0011] Correct the total number of times of raising hands according to the total number of times a student is marked as an abnormal student to obtain a corrected number, and evaluate the quality of classroom interaction according to the corrected number.

[0012] As a further scheme of the present invention: The process of obtaining the total number of times of raising hands specifically includes:

[0013] Collect the monitoring video in the classroom during the class, use the monitoring video when the course of the class belongs to subject i as the sub-video, and identify the total number of times C that the students in the sub-video raise their hands based on the Faster R-CNN model.

[0014] As a further scheme of the present invention: The process of determining the abnormal students specifically includes:

[0015] Perform frame segmentation on the sub-video to obtain sub-video frames, randomly select m sub-video frames as target frames, where m is a preset number, determine the skeletal points of the students in the target frames based on the OpenPose algorithm, mark the skeletal points at the preset positions on the shoulders as key points, connect the key points to obtain key lines, extend the key lines, and use the intersection points of the extended key lines and the boundaries of the target frames as target lines. Obtain the target lines by connecting the target points. The target lines divide the key frames into two parts, which are respectively denoted as the main part and the sub-part, and the part where the direction of the student's face is located is used as the main part;

[0016] Obtain the midpoint A of the key line, draw two rays starting from the midpoint A, the angles between the rays and the target line are both preset angles, and the rays point to the sub-part. Denote the intersection points of the two rays and the boundary of the key frame as ray points, connect the midpoint A and the ray points to obtain ray segments, determine the region enclosed by the ray segments and the boundary of the target frame, mark the region belonging to the sub-part as the target region. When the teacher is in the target region, determine the corresponding student as a pending student, count the proportion of pending students. If the proportion is less than or equal to 0.2, mark the pending students as abnormal students.

[0017] As a further solution of the present invention: The process of evaluating the quality of classroom interaction according to the number of corrections specifically includes:

[0018] Count the proportion b of the total number of times a single student is marked as an abnormal student in the number of target frames, and calculate the number of corrections C' = b * C;

[0019] When the number of corrections is less than or equal to the preset quantity threshold, mark the student as the target student for subject i. When the class belongs to subject i, evaluate the quality of classroom interaction of the target students for subject i.

[0020] As a further solution of the present invention: The process of evaluating the quality of classroom interaction of the target students for subject i specifically includes:

[0021] Mark the students other than the target students as comparison students, calculate the average value P1 of the total number of times the comparison students raise their hands, and calculate the number difference P = P1 - P2, where P2 represents the total number of times a single target student raises their hand;

[0022] If the number difference P ≥ Pys, it is judged that the interaction quality of the corresponding target student is poor, where Pys represents the preset number difference threshold;

[0023] If the number difference P < Pys, it is judged that the interaction quality of the corresponding target student is good.

[0024] As a further solution of the present invention: in the process of determining the pending students in the target frame, when the teacher is not in the classroom in the target frame, the target frame is discarded, and a new sub-video frame is selected as the target frame again.

[0025] As a further solution of the present invention: in the process of evaluating the classroom interaction quality of the target students of subject i, the following steps are further included:

[0026] Set a holding period T, determine the time point t1 when the student is marked as the target student of subject i, and obtain the current time point t2. When the time difference t = t2 - t1 ≥ T, re-determine whether the student is still the target student of subject i.

[0027] As a further solution of the present invention: in the process of identifying the total number of times the students in the sub-video raise their hands, when the total length of the sub-video is less than the preset total length threshold, the subsequent steps are not executed until the total length of the sub-video is greater than or equal to the preset total length threshold.

[0028] Advantages of the present invention: In this solution, by collecting surveillance videos during class and taking the surveillance videos when the course belongs to subject i as sub-videos separately, the student behavior data within the required course range can be accurately focused on, reducing the interference of irrelevant information at the source and contributing to the accuracy and efficiency of subsequent recognition algorithms; based on the Faster R-CNN model, the total number of times C that students raise their hands in the sub-video is recognized, enabling the system to quickly and accurately capture the classroom interaction situation of each student. This step helps in preliminarily screening out relatively active or inactive students, laying a foundation for further skeleton point analysis; the sub-video is framed and m sub-video frames are randomly selected as target frames, which can not only ensure the representativeness of data sampling but also avoid the problem of excessive computational complexity caused by processing all frames. This is particularly important for the real-time requirements of large-scale classroom scenarios; based on the OpenPose algorithm, the skeleton points of students in the target frames are determined, and the shoulder key points are connected as key lines, which can more accurately capture the pose information of students, thereby judging the facial orientation of students without the need for complex recognition of the whole body or face, providing a simplified but effective technical support for subsequent recognition of the relationship and interaction status between students and teachers; the key line is extended to intersect with the boundary of the target frame to obtain the target line, and the key frame is divided into a main part and a sub-part according to the facial orientation. Through this geometric method, the area where the student is facing can be further accurately located, avoiding the interference of redundant background or irrelevant people on recognition; two rays are drawn using the midpoint A of the key line and the target area is obtained. When the teacher is located in the target area, the student can be determined as a pending student. This ray-based determination method greatly simplifies the pose tracking logic and reduces the computational amount in large-scale scenarios; then, the abnormal students are marked according to the proportion of the number of pending students, and the correction times C' are calculated by combining the total number of times of being marked as abnormal and the total number of times C of raising hands, which helps to distinguish between occasional anomalies and persistent anomalies and ensures that the evaluation results are more objective; when the correction times C' are lower than the threshold, the students are marked as target students of subject i and the interaction quality is evaluated. This can concentrate computational resources and analysis efforts to conduct more in-depth research on key objects, achieving a balance between pertinence and feasibility in a large class, making the whole system more practical and scalable; finally, by comparing the difference P between the average number of times P1 that students raise their hands and the number of times P2 that target students raise their hands, the interaction quality of target students is judged, avoiding high-cost face and pose recognition for all students, and thus achieving both high efficiency and accuracy in evaluating key groups in the large-class teaching scenario, making the whole system more practical and scalable. Description of the Drawings

[0029] The present invention will be further described below with reference to the accompanying drawings.

[0030] Figure 1 It is a schematic flowchart of a method for evaluating classroom interaction quality based on image recognition according to the present invention. Detailed implementation manners

[0031] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present invention.

[0032] Please refer to Figure 1 As shown, the present invention is a method for evaluating the quality of classroom interaction based on image recognition, including the following steps:

[0033] Collect surveillance videos, determine sub-videos, and identify the total number of times a single student raises their hand in the sub-videos;

[0034] In a preferred embodiment of the present invention, the process of obtaining the total number of times of raising the hand specifically includes: collecting surveillance videos in the classroom during class, taking the surveillance videos when the course of the class belongs to subject i as sub-videos, and identifying the total number of times C that students raise their hands in the sub-videos based on the Faster R-CNN model;

[0035] It can be understood that a surveillance camera that can panoramically cover the classroom is installed in a classroom. With the user's authorization, surveillance video data in the classroom during class is collected. The video content covers the behaviors of students during the entire class. Combining with the school class schedule, when the course belongs to subject i (for example, when the course is mathematics class, course i can be any subject and can be selected according to the actual situation), the surveillance video segment marked as a mathematics class from 9:00 to 9:45 in the morning is extracted as a sub-video. Subsequently, the system analyzes frame by frame whether students raise their hands in this sub-video: The specific method is to input each frame into a pre-trained Faster R-CNN model, and the model will output all detected areas where students raise their hands (such as the image features where students raise their arms above the shoulders) in the picture, and give the corresponding confidence and bounding boxes. The system integrates the continuously detected hand-raising information on the time axis (to avoid double counting the same hand-raising in multiple frames), and finally obtains the total number of times C that a single student raises their hand during the entire mathematics class period, helping us understand which students participated in the interaction in the classroom and providing a data basis for subsequent evaluation of the quality of classroom interaction;

[0036] Divide the sub-video into sub-video frames, select m sub-video frames as target frames, where m is a preset number, and determine whether there is an abnormality in the listening state of the students based on the target frames; Mark the students with abnormalities as pending students, and count the proportion of pending students in a single target frame. If the proportion is less than or equal to 0.2, mark the pending students as abnormal students;

[0037] In a preferred case of this embodiment, the process of determining the abnormal student specifically includes:

[0038] Perform frame splitting on the sub-video to obtain sub-video frames, randomly select m sub-video frames as target frames, where m is a preset quantity. Based on the OpenPose algorithm, determine the skeletal points of the students in the target frames, mark the skeletal points at the preset positions on the shoulders as key points, connect the key points to obtain key lines, extend the key lines, and use the intersection points of the extended key lines and the boundaries of the target frames as target lines. Obtain the target lines by connecting the target points. The target lines divide the key frames into two parts, which are respectively denoted as the main part and the sub part, and the part where the direction of the student's face is located is used as the main part;

[0039] Obtain the midpoint A of the key line, draw two rays starting from the midpoint A, the angles between the rays and the target line are both preset angles, and the rays point to the sub part. Denote the intersection points of the two rays and the boundary of the key frame as shooting points, connect the midpoint A and the shooting points to obtain shooting segments, determine the regions enclosed by the shooting segments and the boundaries of the target frames, mark the regions belonging to the sub part as target regions. When the teacher is in the target regions, determine the corresponding students as pending students, count the proportion of the pending students. If the proportion is less than or equal to 0.2, then mark the pending students as abnormal students;

[0040] It should be noted that in a large-class teaching classroom with 50 students, the system first processes the monitoring video corresponding to this class into frame-by-frame sub-video frames; then randomly selects a preset number m (such as 100 frames) from all the sub-video frames as target frames, which can not only take into account the representativeness of sampling but also avoid the high computational cost of processing all frames. Subsequently, the system calls the OpenPose algorithm to locate the skeletal points of each student in the target frame, especially focusing on the skeletal points at the shoulder position, and marks them as key points respectively; then, a key line is connected by the key points of the left and right shoulders, and this key line is extended outward until it intersects with the boundary of the target frame and is recorded as the target line. This target line roughly divides the image of the target frame into two parts. The part containing the student's face orientation is recorded as the main part, and the other side is recorded as the sub-part. At this time, the midpoint A of this key line is obtained, and two rays are drawn from point A in the direction of the sub-part, so that these two rays form set angles (such as 30 degrees each) with the target line respectively, and then these two rays are extended until they intersect with the edge of the target frame, and the intersection points are the shooting points; the shooting segment is connected from point A to these two shooting points, and the area enclosed by this shooting segment and part of the boundary of the target frame is marked as the target area. In this state, if the system detects that the teacher (which can be detected by additional human detection or registered teacher feature recognition) is within this target area, it can be determined that the student faces the teacher and there is an abnormality in the listening state in this frame (because normally students should face the main part where the teacher is located); for each target frame, any student determined to have an abnormality is marked as a pending student. The system will count the proportion of pending students in this frame. If this proportion is less than or equal to 20%, it means that only a few students have the abnormality of turning their backs to the teacher. Therefore, these pending students will be further marked as abnormal students to provide more accurate and efficient data support for subsequent learning behavior evaluation;

[0041] In a preferred case of this embodiment, during the process of determining the pending students in the target frame, when the teacher is not in the classroom in the target frame, then this target frame is discarded, and a new sub-video frame is reselected as the target frame.

[0042] It should be noted that when selecting the target frame, it will first be determined whether there is a teacher in the frame through human detection or the registered teacher feature model. If in a certain frame, the teacher just walks out of the camera coverage range or is blocked and cannot be recognized, then this frame loses the meaning of analyzing the relationship between the student's face orientation and the teacher because it is impossible to determine whether the student is paying attention to the teacher. At this time, the system will directly discard this frame and randomly or sequentially select another frame from the list of sub-video frames to replace it to ensure that each target frame has the basic conditions for determining the relative position relationship of "student - teacher";

[0043] The total number of times of raising hands is corrected according to the total number of times a student is marked as an abnormal student to obtain a corrected number of times, and the quality of classroom interaction is evaluated according to the corrected number of times;

[0044] It should be noted that the process of evaluating the quality of classroom interaction for the target students of subject i specifically includes:

[0045] Students other than the above-mentioned target students are marked as comparison students. The average value P1 of the total number of times of raising hands of the comparison students is calculated, and the number difference P = P1 - P2 is calculated, where P2 represents the total number of times of raising hands of a single target student;

[0046] If the number difference P ≥ Pys, it is determined that the interaction quality of the corresponding target student is poor, where Pys represents a preset number difference threshold;

[0047] If the number difference P < Pys, it is determined that the interaction quality of the corresponding target student is good

[0048] It can be understood that in the process of evaluating the quality of classroom interaction for the target students of subject i, the following steps are also included:

[0049] A holding period T is set, the time point t1 when a student is marked as a target student of subject i is determined, and the current time point t2 is obtained. When the time difference t = t2 - t1 ≥ T, it is re-determined whether the student is still a target student of subject i;

[0050] It should be noted that the system will mark classmate Wang as a target student of subject i (mathematics) in a math class and record the time point t1 of this marking behavior; at the same time, the system sets a holding period T (such as one week or one month) to track the status of target students in a longer time dimension. When entering the math class monitoring again in the future, the system will obtain the new time t2. If the calculated time difference t2 - t1 is greater than or equal to T, the re-determination logic will be triggered: the system will comprehensively consider Wang's interaction situation in recent courses and then determine whether he still meets the criteria of a target student; if not, Wang's target student status in the math class will be automatically cancelled and relevant monitoring and computing resources will be released to avoid long-term and meaningless monitoring of students who are no longer applicable;

[0051] In a preferred case of this embodiment, in the process of identifying the total number of times a student raises hands in the sub-video, when the total length of the sub-video is less than a preset total length threshold, the subsequent steps are not executed until the total length of the sub-video is greater than or equal to the preset total length threshold;

[0052] If the total length threshold set in the system is 5 hours, then when collecting classroom surveillance videos, if a video segment (sub-video) of a certain subject (such as a math class) is only 20 minutes, the student raising hand detection algorithm based on Faster R-CNN will not be immediately launched to identify the total number of times students raise their hands. Instead, the video segment will be temporarily stored in the cache and wait for continuous recording or splicing of new video segments. When the system detects that the total length of the current sub-video has accumulated to 5 hours or more, the subsequent recognition algorithm will be called to count the number of times students raise their hands in this sub-video. This can avoid frequently starting the highly time-consuming recognition process in videos with too short durations, thereby saving system resources, improving the overall processing efficiency, and making the hand-raising detection results more statistically significant and valuable for reference;

[0053] It should be noted that different students may have different levels of interest in different subjects. Therefore, by distinguishing the target students of different subjects and only monitoring the target students of the subjects during class, the computing resources and hardware costs required for monitoring all students can be reduced.

[0054] The above has described a detailed implementation example of the present invention, but the content described is only the preferred implementation example of the present invention and cannot be considered as used to limit the implementation scope of the present invention. Any equivalent changes and improvements made according to the scope of the present invention application should still fall within the patent coverage scope of the present invention.

Claims

1. A method for evaluating the quality of classroom interaction based on image recognition, characterized in that It includes the following steps: Collect surveillance videos, determine sub-videos, and identify the total number of times a single student raises their hand in the sub-videos; Frame the sub-videos to obtain sub-video frames, select m sub-video frames as target frames, where m is a preset number, and determine whether there is an abnormality in the listening state of the students based on the target frames; Mark the students with abnormalities as pending students, and count the proportion of pending students in a single target frame. If the proportion is less than or equal to 0.2, mark the pending students as abnormal students; Correct the total number of times of raising hands according to the total number of times a student is marked as an abnormal student to obtain a corrected number, and conduct a classroom interaction quality assessment based on the corrected number.

2. The method for evaluating the quality of classroom interaction based on image recognition according to claim 1, characterized in that, The process of obtaining the total number of times of raising hands specifically includes: Collect the surveillance video in the classroom during class, use the surveillance video when the course of the class belongs to subject i as the sub-video, and identify the total number of times C that students raise their hands in the sub-video based on the Faster R-CNN model.

3. The method for evaluating the quality of classroom interaction based on image recognition according to claim 2, characterized in that, The process of determining the abnormal students specifically includes: Perform frame processing on the sub-video to obtain sub-video frames, randomly select m sub-video frames as target frames, where m is a preset number, determine the skeletal points of the students in the target frames based on the OpenPose algorithm, mark the skeletal points at the preset position of the shoulders as key points, connect the key points to obtain a key line, extend the key line, and use the intersection point of the extended key line and the boundary of the target frame as the target line. Connect the target points through a straight line to obtain the target line. The target line divides the key frame into two parts, which are respectively recorded as the main part and the sub-part, and the part where the direction of the student's face is located is used as the main part; Obtain the midpoint A of the key line, draw two rays starting from the midpoint A, the angles between the rays and the target line are both preset angles, and the rays point to the sub-part. Mark the intersection points of the two rays and the boundary of the key frame as the shooting points, connect the midpoint A and the shooting points to obtain the shooting segments, determine the area enclosed by the shooting segments and the boundary of the target frame, mark the area belonging to the sub-part as the target area. When the teacher is in the target area, determine the corresponding student as a pending student, count the proportion of pending students. If the proportion is less than or equal to 0.2, mark the pending students as abnormal students.

4. The method for evaluating the quality of classroom interaction based on image recognition according to claim 3, wherein The process of conducting a classroom interaction quality assessment based on the corrected number specifically includes: Count the proportion b of the total number of times a single student is marked as an abnormal student in the number of target frames, and calculate the corrected number C' = b * C; When the corrected number is less than or equal to the preset quantity threshold, mark the student as the target student of subject i, and conduct a classroom interaction quality assessment on the target students of subject i when the course of the class belongs to subject i.

5. The method for evaluating the quality of classroom interaction based on image recognition according to claim 4, characterized in that The process of conducting a classroom interaction quality assessment on the target students of subject i specifically includes: Mark the students other than the target students as comparison students, calculate the mean P1 of the total number of times the comparison students raise their hands, and calculate the number difference P = P1 - P2, where P2 represents the total number of times a single target student raises their hand; If the number difference P ≥ Pys, judge that the interaction quality of the corresponding target student is poor, where Pys represents the preset number difference threshold; If the difference in the number of times P < Pys, it is determined that the interaction quality of the corresponding target student is good.

6. The method for evaluating the quality of classroom interaction based on image recognition according to claim 3, characterized in that In the process of determining the pending student in the target frame, when the teacher is not in the classroom in the target frame, the target frame is discarded and a new sub-video frame is selected as the target frame.

7. The method for evaluating the quality of classroom interaction based on image recognition according to claim 5, wherein In the process of evaluating the classroom interaction quality of the target student for subject i, the following steps are also included: Set the holding period T, determine the time point t1 when the student is marked as the target student for subject i, and obtain the current time point t2. When the time difference t = t2 - t1 ≥ T, re-judge whether the student is still the target student for subject i.

8. The method for evaluating the quality of classroom interaction based on image recognition according to claim 1, characterized in that In the process of identifying the total number of times a student raises their hand in the sub-video, when the total length of the sub-video is less than the preset total length threshold, the subsequent steps are not executed until the total length of the sub-video is greater than or equal to the preset total length threshold.

Images