Interactive classroom teaching effect enhancement method, system and device based on expression and behavior recognition

Abstract

The invention relates to an interactive classroom teaching effect enhancement method based on expression and behavior recognition, and the method comprises the steps: enabling a camera to collect a video containing the facial expression information of a student at the beginning of classroom teaching; the teacher sets a starting point and an ending point of a time period; extracting face image data of students in the classroom from the collected video according to a set time period; identifying student expressions and behaviors in each face image of each student through a student expression and behavior identification algorithm, and classifying the student expressions and behaviors in each face image; calculating a classroom teaching effect according to the expression classification result and the behavior classification result of the student; and transmitting the classroom teaching effect data corresponding to the set time period to a display, and displaying the feedback classroom teaching effect data in real time. According to the method, an interaction mechanism is introduced, so that a teacher can autonomously set a classroom evaluation feedback mechanism according to actual requirements of a classroom, and the teaching method is flexibly adjusted in real time.

Description

Technical Field

[0001] The present invention relates to the technical field of Internet data analysis, and in particular to a method, system and device for enhancing interactive classroom teaching effects based on expression and behavior recognition. Background Art

[0002] In order to ensure the accuracy and reliability of teaching quality evaluation, intelligent teaching evaluation has attracted much attention. In the existing technology, expression and behavior recognition technology has been used to obtain intelligent analysis and evaluation results of classroom teaching. This technology is mainly used to grasp the classroom teaching effect after class. Usually, different teaching contents are taught in each class in a semester according to the teaching plan. Teachers use the teaching evaluation results to improve teaching methods after the current class, which can only prepare for classroom teaching in the next semester. Therefore, existing research work ignores the students' learning status and students' understanding level in the current semester.

[0003] In order to improve the effectiveness of classroom teaching in real time, teachers need to grasp students' learning status and understanding in real time, and adjust teaching methods in time to improve students' learning status and understanding in real time. However, it is not easy for teachers to observe the behavior and expressions of each student in the classroom, especially in classrooms with a large number of students. Different knowledge points have different levels of difficulty, and teachers have different teaching times. Therefore, teachers need to set up a classroom evaluation feedback mechanism according to the actual needs of the classroom, so as to grasp students' learning status and understanding, and adjust teaching methods to improve classroom teaching effectiveness.

[0004] Patent publication number CN116778543A discloses a classroom teaching effect detection system, which uses a human posture device and a facial recognition device to sense and identify students' bad classroom behaviors, and compares them with thresholds to determine students' bad classroom behaviors. When there are many violations, it will remind teachers to further control the classroom order and atmosphere. However, it is only used as a detection method for classroom discipline. On the one hand, students' bad classroom behaviors belong to the students' learning status and cannot reflect the students' level of understanding; on the other hand, the alarm output of the alarm signal may affect the students' learning status and the teaching atmosphere of the classroom.

[0005] The prior art also discloses a technical solution in which a teacher terminal is set on the teacher's podium, and the teacher uses two-dimensional and three-dimensional charts to understand the learning status of the entire class and timely grasp the teaching effect of the class. Such complex and diverse charts require teachers to spend time observing data in front of the terminal, which takes up teaching time. In addition, the above research work ignores the analysis and feedback of students' learning status and students' understanding level for different knowledge points in a class, resulting in poor teaching effect. Summary of the invention

[0006] The purpose of the present invention is to solve the above-mentioned problems existing in the prior art. The present invention provides an interactive classroom teaching effect enhancement system based on expression and behavior recognition.

[0007] The present invention is a method for enhancing the interactive classroom teaching effect based on expression and behavior recognition, which is realized by the following technical scheme and comprises the following steps:

[0008] S1. When classroom teaching begins, the camera starts to collect videos containing students’ facial expression information;

[0009] S2, the teacher sets the starting and ending points of a time period;

[0010] S3, extracting facial image data of students in the classroom from the collected video according to a set time period;

[0011] S4, using a student expression and behavior recognition algorithm, identifying the student expression and behavior in each face image of each student, and classifying the student expression and behavior of each face image;

[0012] S5. Calculate the classroom teaching effect based on the student expression classification results and behavior classification results;

[0013] S6. Transmit the classroom teaching effect data corresponding to the set time period to the display, and display the feedback classroom teaching effect data in real time; repeat steps S2 to S6 until the classroom teaching is finished.

[0014] In a further technical solution, the student expression categories include understanding, confusion, frustration, pain, and discomfort expressed towards the teaching content; the student behavior categories include raising the head, turning the head, and lowering the head.

[0015] In a further technical solution, the step S3 further includes: for the extracted facial image data of the students in the classroom, according to the pre-stored student image sequence table of the corresponding class, performing portrait recognition on the extracted facial image data of the students in the classroom to obtain the identity information of the students corresponding to the facial image;

[0016] In a further technical solution, the classroom teaching effect calculation formula in step S5 is:

[0017]

[0018]

[0019] When the percentage of students not attending the class is greater than a given threshold, the number of head turning images and the number of head lowering images at the corresponding time point are recorded as invalid data; the value of the given threshold is not less than 50%.

[0020] The step S5 also includes: calculating, based on the expression classification results and behavior classification results of each student

[0021] Personal learning effect; the calculation formula of the personal learning effect is:

[0022]

[0023] Based on the personal learning effect data, the personal learning effect is calculated according to the formula: personal learning effect = personal listening concentration percentage * A + personal listening understanding percentage * B; when the personal learning effect is less than the set threshold, the student will be included in the list of students for teaching attention; A is the weight of the personal listening concentration, and B is the weight of the personal listening understanding.

[0024] The second aspect of the present invention provides an interactive classroom teaching effect enhancement system based on expression and behavior recognition, the system comprising

[0025] A time setting unit, configured to collect a start signal and an end signal of a time period set by the control terminal;

[0026] A video acquisition unit is configured to acquire student videos through a camera device, wherein the student videos include student facial expression information;

[0027] An image processing unit is configured to extract face image data from the student video taken according to the received data within a set time period, and send the face image data to the expression behavior recognition unit;

[0028] an expression and behavior recognition unit, configured to recognize the student expression and behavior in each face image of each student through a student expression and behavior recognition algorithm, and classify the student expression and behavior in each face image;

[0029] A classroom teaching effect calculation unit is configured to calculate the classroom teaching effect according to the student expression classification data and behavior classification data, wherein the classroom teaching effect data includes the percentage of students who have not attended the class and the percentage of students who have attended the class to understand the class;

[0030] The classroom teaching effect display unit is configured to display the feedback classroom teaching effect data and the feedback teaching focus student list in real time at the end of a set time period.

[0031] In a further technical solution, the system further includes

[0032] The student identification unit is configured to extract facial image data of students in the classroom, and perform portrait recognition on the extracted facial image data of students in the classroom according to a pre-stored student image sequence table of the corresponding class to obtain identity information of the students corresponding to the facial images;

[0033] A personal learning effect calculation unit, configured to calculate the personal learning effect according to the expression classification results and behavior classification results of each student;

[0034] The teaching attention student list generating unit is configured to determine whether the individual needs to be included in the teaching attention student list according to the individual learning effect score, and if so, the individual needs to be included in the teaching attention student list.

[0035] The third aspect of the present invention also provides an interactive classroom teaching effect enhancement device based on expression and behavior recognition, the device includes a device applying the above-mentioned interactive classroom teaching effect enhancement system. Compared with the prior art, the beneficial effects of the present invention include:

[0036] The interactive classroom teaching effect enhancement method based on expression and behavior recognition of the present invention introduces an interactive mechanism, so that teachers can independently set up a classroom evaluation feedback mechanism according to the actual needs of the classroom, flexibly and in real time adjust the teaching method, and improve the classroom teaching effect. The method of the present invention includes both the analysis of the overall classroom teaching effect and the accurate analysis of the individual learning effect. It provides teachers with more accurate classroom effect feedback through the combination of overall and individual effect analysis, and the system self-detection method further accurately feedbacks the classroom effect; the classroom can carry out targeted teaching and guidance in a timely manner according to the learning effect of each student, thereby better improving the classroom teaching effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 A flowchart of a specific implementation of the interactive classroom teaching effect enhancement method based on expression and behavior recognition of the present invention;

[0038] Figure 2 The flowchart is another specific implementation of the interactive classroom teaching effect enhancement method based on expression and behavior recognition of the present invention. DETAILED DESCRIPTION

[0039] The interactive classroom teaching effect enhancement method based on expression and behavior recognition of the present invention introduces an interactive mechanism by designing an interactive classroom teaching effect enhancement system, so that teachers can independently set a classroom evaluation feedback mechanism according to the actual needs of the classroom and flexibly and in real time adjust the teaching method.

[0040] The interactive classroom teaching effect enhancement system includes a teacher control terminal, a high-definition camera, a dedicated teaching server, and a display. The teacher control terminal is used to set the time interval or the start and end point of a time period, and to display classroom teaching effect data in real time. The teacher control terminal can be a wearable terminal or a mobile terminal APP. The high-definition camera is installed in the front of the classroom to shoot student videos. The dedicated teaching server is used to calculate the classroom teaching effect. The display is installed in the back of the classroom to display classroom teaching effect data in real time.

[0041] The present invention is further described in detail below in conjunction with the accompanying drawings:

[0042] refer to Figure 1 As shown, the interactive classroom teaching effect enhancement method based on expression and behavior recognition of the present invention comprises the following steps:

[0043] S1. When classroom teaching begins, the camera starts to collect videos containing students’ facial expression information;

[0044] S2, the teacher sets the starting and ending points of a time period;

[0045] S3, extracting facial image data of students in the classroom from the collected video according to a set time period;

[0046] S4, using a student expression and behavior recognition algorithm, identifying the student expression and behavior in each face image of each student, and classifying the student expression and behavior of each face image;

[0047] S5. Calculate the classroom teaching effect based on the student expression classification results and behavior classification results;

[0048] S6. Transmit the classroom teaching effect data corresponding to the set time period to the display, and display the feedback classroom teaching effect data in real time; repeat steps S2 to S6 until the classroom teaching is finished.

[0049] When setting the time period, the teacher can use specific devices such as smart watches, smartphones, tablets and computers, and issue instructions through the APP control platform. High-definition cameras installed on both sides of the front of the classroom are used to collect video data in the classroom.

[0050] The step S3 of extracting facial image data of students in the classroom from the collected video according to a set time period is as follows: intercepting the student video taken by the camera according to the starting point and end point of a set time period, and then extracting facial image data at a given frequency.

[0051] The step S4 is specifically an expression recognition method through a convolutional neural network: using a labeled face image training data set to train a convolutional neural network; using the trained convolutional neural network as a student expression recognition model to extract features from the face image data, and then using a linear support vector machine to classify the student expressions to obtain the expression results of each student during the learning process and the behavioral results of each student during the learning process.

[0052] The student expression categories include understanding, confusion, frustration, pain, and discomfort expressed in the teaching content; the student behavior categories include raising the head, turning the head, and lowering the head.

[0053] The classroom teaching effect calculation formula in step S5 is:

[0054]

[0055] Considering that in the process of classroom teaching, there will be situations where students need to turn their heads to look at different blackboards or screens according to the teacher's instructions. At this time, turning the head or lowering the head in some special cases is not within the indicator range of not listening to the class; a given threshold of the non-indicator range in which most students turn their heads or lower their heads is set. When the percentage of students who are not listening to the class is greater than the given threshold, the number of head turning images and head lowering images at the corresponding time point are recorded as invalid data; the value of the given threshold is not less than 50%.

[0056] In an improved example, the method of the present invention further evaluates individual learning effects based on existing collected data. Figure 2 As shown, step S3 also includes extracting facial image data of students in the classroom, and according to the pre-stored student image sequence table of the corresponding class, by comparing the facial features (such as facial features proportions, facial contours, texture features, etc.) in the image sequence with the student image sequence table pre-entered into the database, the corresponding learner identity information can be accurately matched.

[0057] The step S5 further includes: calculating the individual learning effect according to the expression classification results and behavior classification results of each student; the individual learning effect calculation formula is:

[0058]

[0059]

[0060] Based on the personal learning effect data, the personal learning effect is calculated according to the formula: personal learning effect = personal listening concentration percentage * A + personal listening understanding percentage * B; when the personal learning effect is less than the set threshold, the student will be included in the list of students for teaching attention; A is the weight of the personal listening concentration, and B is the weight of the personal listening understanding.

[0061] The step S6 also includes: transmitting the list of students of teaching concern corresponding to the set time period to the display, and displaying the feedback list of students of teaching concern in real time.

[0062] During the course, teachers pay targeted attention to the learning status of students on the teaching attention list based on the individual learning results of each student. After observing the change in teaching methods, they pay targeted attention to the changes in the learning status of students on the teaching attention list. After the course is over, the system summarizes the individual learning results of each student in the teaching process, providing a detailed data basis for subsequent teachers to conduct classroom analysis. According to the individual learning results of each student during the course, teachers accurately identify the students' cognitive difficulties and periods of distraction, provide them with personalized teaching plans, and improve the understanding of students on the teaching attention list by combining the dual teaching plans in class and after class.

[0063] The second aspect of the present invention also provides a system for enhancing the interactive classroom teaching effect based on expression and behavior recognition, the system comprising

[0064] A time setting unit, configured to collect a start signal and an end signal of a time period set by the control terminal;

[0065] A video acquisition unit is configured to acquire student videos through a camera device, wherein the student videos include student facial expression information;

[0066] An image processing unit is configured to extract face image data from the student video taken according to the received data within a set time period, and send the face image data to the expression behavior recognition unit;

[0067] an expression and behavior recognition unit, configured to recognize the student expression and behavior in each face image of each student through a student expression and behavior recognition algorithm, and classify the student expression and behavior in each face image;

[0068] A classroom teaching effect calculation unit is configured to calculate the classroom teaching effect according to the student expression classification data and behavior classification data, wherein the classroom teaching effect data includes the percentage of students who have not attended the class and the percentage of students who have attended the class to understand the class;

[0069] The classroom teaching effect display unit is configured to display the feedback classroom teaching effect data and the feedback teaching focus student list in real time at the end of a set time period.

[0070] The student expression classification data includes understanding, confusion, frustration, pain, and discomfort expressed towards the teaching content (how to specifically judge the expression category through images, and how to define the above-mentioned various expression characteristics through images); the behavior classification data includes raising the head, turning the head, and lowering the head.

[0071] In another example, the system further includes

[0072] The student identification unit is configured to extract facial image data of students in the classroom, and perform portrait recognition on the extracted facial image data of students in the classroom according to a pre-stored student image sequence table of the corresponding class to obtain identity information of the students corresponding to the facial images;

[0073] A personal learning effect calculation unit, configured to calculate the personal learning effect according to the expression classification results and behavior classification results of each student;

[0074] The teaching attention student list generating unit is configured to determine whether the individual needs to be included in the teaching attention student list according to the individual learning effect score, and if so, the individual needs to be included in the teaching attention student list.

[0075] In addition, the third aspect of the present invention also discloses an interactive classroom teaching effect enhancement device based on expression and behavior recognition, and the device includes a computer platform and the above-mentioned interactive classroom teaching effect enhancement system.

[0076] The interactive classroom teaching effect enhancement system and device based on expression and behavior recognition provided by the embodiment of the present invention can implement the method embodiment provided above. For the specific function implementation, please refer to the description in the method embodiment, which will not be repeated here. A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program. The program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0077] The above technical solution is only one implementation mode of the present invention. For those skilled in the art, it is easy to make various types of improvements or modifications based on the principles disclosed in the present invention, and it is not limited to the technical solution described in the above specific embodiments of the present invention. Therefore, the above description is only preferred and does not have a restrictive meaning.

Claims

1. A method for enhancing interactive classroom teaching effect based on expression and behavior recognition, comprising the following steps: S1. When classroom teaching begins, the camera starts to collect videos containing students’ facial expression information; S2, the teacher sets the starting and ending points of a time period; S3, extracting facial image data of students in the classroom from the collected video according to a set time period; S4, using a student expression and behavior recognition algorithm, identifying the student expression and behavior in each face image of each student, and classifying the student expression and behavior of each face image; S5. Calculate the classroom teaching effect based on the student expression classification results and behavior classification results; S6. Transmit the classroom teaching effect data corresponding to the set time period to the display, and display the feedback classroom teaching effect data in real time; Repeat steps S2 to S6 until the classroom teaching is finished.

2. The interactive classroom teaching effect enhancement method based on expression and behavior recognition as claimed in claim 1, characterized in that: The step S3 of extracting facial image data of students in the classroom from the collected video according to a set time period is as follows: intercepting the student video taken by the camera according to the starting point and end point of a set time period, and then extracting facial image data at a given frequency.

3. The interactive classroom teaching effect enhancement method based on expression and behavior recognition as claimed in claim 1, characterized in that: The step S4 is specifically an expression recognition method through a convolutional neural network: using a labeled face image training data set to train a convolutional neural network; using the trained convolutional neural network as a student expression recognition model to extract features from the face image data, and then using a linear support vector machine to classify the student expressions.

4. The interactive classroom teaching effect enhancement method based on expression and behavior recognition as claimed in claim 1, characterized in that: The student expression categories include understanding, confusion, frustration, pain, and discomfort expressed towards the teaching content; the student behavior categories include raising the head, turning the head, and lowering the head.

5. The interactive classroom teaching effect enhancement method based on expression and behavior recognition as claimed in claim 1, characterized in that: The step S3 also includes: for the extracted facial image data of students in the classroom, according to the pre-stored student image sequence table of the corresponding class, performing portrait recognition on the extracted facial image data of students in the classroom to obtain identity information of the students corresponding to the facial images.

6. The interactive classroom teaching effect enhancement method based on expression and behavior recognition as claimed in claim 5, characterized in that: The classroom teaching effect calculation formula in step S5 is: When the percentage of students not attending the class is greater than a given threshold, the number of head-turning images and head-down images at the corresponding time point are recorded as invalid data; the value of the given threshold is not less than 50%; The step S5 further includes: calculating the individual learning effect according to the expression classification results and behavior classification results of each student; the individual learning effect calculation formula is: Based on the personal learning effect data, the personal learning effect is calculated according to the formula: personal learning effect = personal listening concentration percentage * A + personal listening understanding percentage * B; when the personal learning effect is less than the set threshold, the student will be included in the list of students for teaching attention; A is the weight of the personal listening concentration, and B is the weight of the personal listening understanding.

7. The interactive classroom teaching effect enhancement method based on expression and behavior recognition as claimed in claim 6, characterized in that: The step S6 also includes: transmitting the list of students of teaching concern corresponding to the set time period to the display, and displaying the feedback list of students of teaching concern in real time.

8. A system using the interactive classroom teaching effect enhancement method based on expression and behavior recognition as described in claim 1, characterized in that: The system comprises A time setting unit, configured to collect a start signal and an end signal of a time period set by the control terminal; A video acquisition unit is configured to acquire student videos through a camera device, wherein the student videos include student facial expression information; An image processing unit is configured to extract face image data from the student video taken according to the received data within a set time period, and send the face image data to the expression behavior recognition unit; an expression and behavior recognition unit, configured to recognize the student expression and behavior in each face image of each student through a student expression and behavior recognition algorithm, and classify the student expression and behavior in each face image; A classroom teaching effect calculation unit is configured to calculate the classroom teaching effect according to the student expression classification data and behavior classification data, wherein the classroom teaching effect data includes the percentage of students who have not attended the class and the percentage of understanding degree of students who have attended the class; The classroom teaching effect display unit is configured to display the feedback classroom teaching effect data and the feedback teaching focus student list in real time at the end of a set time period.

9. The system according to claim 8, characterized in that The system also includes The student identification unit is configured to extract facial image data of students in the classroom, and perform portrait recognition on the extracted facial image data of students in the classroom according to a pre-stored student image sequence table of the corresponding class to obtain identity information of the students corresponding to the facial images; A personal learning effect calculation unit, configured to calculate the personal learning effect according to the expression classification results and behavior classification results of each student; The teaching attention student list generating unit is configured to determine whether the individual needs to be included in the teaching attention student list according to the individual learning effect score, and if so, the individual needs to be included in the teaching attention student list.

10. An interactive classroom teaching effect enhancement device based on expression and behavior recognition, characterized in that: The device includes a device applying the interactive classroom teaching effect enhancement system described in claims 8-9.

Images