A teaching feedback method, device and server for facial expression information analysis

Abstract

The present application discloses a teaching feedback method, device and server for analyzing human face expression information, belonging to the field of classroom teaching effect feedback. The method includes: obtaining a teaching video containing the face expression information of each student during the class time period of the classroom whose teaching effect is to be judged; recognizing, according to the teaching video, a set of head portraits of each student arranged in chronological order during the teaching; performing expression recognition on the set of head portraits of each student, and correspondingly obtaining a high-frequency time series of micro-expressions of each student arranged in chronological order along with the time of the classroom whose teaching effect is to be judged, and processing the high-frequency time series to obtain face expression numerical information; processing the face expression numerical information of each student to obtain an expression value grayscale image; and obtaining the teaching effect feedback of the classroom whose teaching effect is to be judged through the expression value grayscale image. The present application can simultaneously pay attention to the facial expressions of each student in the classroom and directly observe the changes in the emotional expressions of the students to judge the teaching effect.

Description

Technical Field

[0001] This application relates to the technical field of classroom teaching effect feedback, and particularly to a teaching feedback method, device, and server for analyzing human face expression information. Background Art

[0002] People have well-developed facial muscles, which can express various expressions or emotions. During the teaching process, students not only have many behavioral language actions, but also show various expressions on their faces. Different teaching effects will make students have different emotional feelings, which are manifested in expressions and their changes in real time. Teachers can judge the teaching effect by observing students' words, deeds, facial expressions, etc. However, in practice, it is impossible for teachers to pay attention to every student at the same time.

[0003] On the other hand, online classes are becoming increasingly popular and are constantly integrated into students' learning lives. Different from traditional classroom teaching, online classes cannot directly observe the changes in students' emotional expressions to judge the teaching effect, making it impossible for teachers to obtain students' feedback. Summary of the Invention

[0004] Embodiments of this application provide a teaching feedback method, device, and server for analyzing human face expression information, which can solve the problem that in existing classrooms, it is impossible for teachers to pay attention to the facial expressions of every student at the same time, or in online classes, teachers cannot directly observe the changes in students' emotional expressions to judge the teaching effect.

[0005] In the first aspect, embodiments of the present invention provide a teaching feedback method for analyzing human face expression information, including:

[0006] Obtain a teaching video containing the face expression information of each student during the class time period of the class whose teaching effect is to be judged;

[0007] Identify a collection of head portraits of each student arranged in chronological order during the teaching period according to the teaching video;

[0008] Perform expression recognition on the collection of head portraits of each student, and correspondingly obtain a high-frequency time series of micro-expressions of each student arranged in chronological order with respect to the time of the class whose teaching effect is to be judged, and process the high-frequency time series to obtain human face expression numerical information;

[0009] Process the human face expression numerical information of each student to obtain an expression value grayscale image;

[0010] Obtain the teaching effect feedback of the class whose teaching effect is to be judged through the expression value grayscale image.

[0011] In combination with the first aspect, in a possible implementation, the head portrait collection of each student recognized according to the teaching video in chronological order during the teaching process includes:

[0012] Extract images from the teaching video at a set frequency and save them in picture format in chronological order;

[0013] Identify the faces of students from the pictures and establish a head portrait coordinate frame for each student;

[0014] Obtain the face portraits of each student according to all the recognized head portrait coordinate frames in each picture in chronological order, and establish a folder for each student that stores the face portrait information in chronological order.

[0015] In combination with the first aspect, in a possible implementation, for the head portrait collection of each student, perform micro-expression recognition, and correspondingly obtain a high-frequency time series of the micro-expressions of each student changing with time in chronological order, and process the high-frequency time series to obtain face expression numerical information, including:

[0016] Use an expression recognition tool to recognize the images in the head portrait collection of each student and correspondingly obtain a high-frequency time series of each micro-expression of each student changing with time;

[0017] Remove white noise from the high-frequency time series of each student to correspondingly obtain the face expression numerical information of each student.

[0018] In combination with the first aspect, in a possible implementation, after removing white noise from the high-frequency time series of each student, perform normalization processing to transform the absolute value of the high-frequency time series into its rate of change with time, that is, correspondingly obtain the face expression numerical information of each student.

[0019] In combination with the first aspect, in a possible implementation, obtain the teaching effect feedback of the classroom whose teaching effect is to be judged through the expression value grayscale map, including:

[0020] Set the thresholds for each expression value to obtain the proportion of each expression value;

[0021] Formulate an expression value grayscale map threshold judgment rule, and judge the recognition degree of each student for the teaching effect according to the expression value grayscale map threshold judgment rule.

[0022] In combination with the first aspect, in a possible implementation, obtaining the teaching effect feedback of the classroom whose teaching effect is to be judged through the expression value grayscale map further includes:

[0023] Obtain various expression combinations of each student and the time when each expression combination appears based on the grayscale expression map of each student, calculate the correlation and characteristic time between any two expressions to obtain the proportion of the duration of each expression combination, and

[0024] Mark the knowledge point time periods in the content of the teaching.

[0025] Formulate a determination rule for associating grayscale expression maps, and determine the mastery of each knowledge point by each student according to the time and duration proportion of each expression combination appearing within the marked knowledge point time periods.

[0026] Combined with the first aspect, in a possible implementation manner, the obtaining of the teaching effect feedback of the classroom for which the teaching effect is to be judged through the grayscale expression map of the expression value further includes:

[0027] Statistically analyze the mastery of all knowledge points by all students, and determine the teaching effect according to the degree of mastery of the knowledge points.

[0028] In a second aspect, an embodiment of the present invention provides a teaching feedback device for analyzing human face expression information, including:

[0029] An obtaining module, configured to obtain a teaching video including the human face expression information of each student during the class time period of the classroom for which the teaching effect is to be judged;

[0030] An identification module, configured to identify, according to the teaching video, a collection of head portraits of each student arranged in chronological order during the teaching;

[0031] A first obtaining module, configured to perform expression recognition on the collection of head portraits of each student, correspondingly obtain a high-frequency time series of micro-expressions of each student arranged in chronological order along with the time of the classroom for which the teaching effect is to be judged, and process the high-frequency time series to obtain human face expression numerical information;

[0032] A processing module, configured to process the human face expression numerical information of each student to obtain a grayscale expression map;

[0033] A second obtaining module, configured to obtain the teaching effect feedback of the classroom for which the teaching effect is to be judged through the grayscale expression map of the expression value.

[0034] In a third aspect, an embodiment of the present invention provides a server, including: a memory and a processor;

[0035] The memory is used to store program instructions;

[0036] The processor is configured to execute the program instructions in the receiver, so that the receiver executes the teaching feedback method for analyzing human face expression information described above.

[0037] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium storing executable instructions, and when a computer executes the executable instructions, it can implement the teaching feedback method for analyzing facial expression information described above.

[0038] One or more technical solutions provided in the embodiments of the present invention have at least the following technical effects or advantages:

[0039] An embodiment of the present invention provides a teaching feedback method for analyzing facial expression information, including: obtaining a teaching video containing the facial expression information of each student during the class time period of the class whose teaching effect is to be judged; identifying a collection of head portraits of each student arranged in chronological order during the teaching according to the teaching video; performing facial expression recognition on the collection of head portraits of each student, and correspondingly obtaining a high-frequency time series of micro-expressions of each student arranged in chronological order with respect to the time of the class whose teaching effect is to be judged, and processing the high-frequency time series to obtain facial expression numerical information; processing the facial expression numerical information of each student to obtain an expression value grayscale image; obtaining a teaching effect feedback of the class whose teaching effect is to be judged through the expression value grayscale image. This method analyzes by identifying the facial expression information of each student during the class, obtaining quantitative expression information, making the classroom teaching feedback more intelligent. This method is especially suitable for establishing a set of teaching evaluation mechanisms that are convenient for students and teachers to use in college classrooms, urging students to study independently, providing feedback on the class effect for teachers and students, and promoting the improvement of teaching effects. Whether it is a face-to-face class or an online class, it can identify and analyze the expression information of each student in the class, providing more real-time and accurate feedback on students' emotions for teaching. The teaching effect feedback obtained according to the facial expression value grayscale image comes from the feedback analysis results of each student in each class, which is more targeted at the teaching content and more helpful for teachers to improve their teaching methods. This method can simultaneously pay attention to the facial expressions of each student in both face-to-face classes and online classes, and can directly observe the changes in students' emotional expressions to judge the teaching effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following will briefly introduce the drawings required for description in the embodiments of the present invention. Obviously, the following drawings are some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained according to these drawings.

[0041] Figure 1 It is a flowchart of the teaching feedback method for analyzing facial expression information provided in an embodiment of the present application;

[0042] Figure 2Schematic diagram of the teaching feedback device for analyzing human face expression information provided by the embodiment of the present application;

[0043] Figure 3 Avatar coordinate frames corresponding to each student obtained by face recognition provided by the embodiment of the present application;

[0044] Figure 4 Avatar set of the human face wall of students with a 7*7 specification provided by the embodiment of the present application;

[0045] Figure 5 High-frequency time series of seven micro-expressions of students changing with time in a certain time period provided by the embodiment of the present application;

[0046] Figure 6 Time series after denoising and smoothing of the happy expression of students in a certain time period provided by the embodiment of the present application;

[0047] Figure 7 Time series of the relative strength index of the happy expression of students in a certain time period provided by the embodiment of the present application;

[0048] Figure 8 Gray scale map of the time series of the relative strength index of seven micro-expressions of students in a certain time period provided by the embodiment of the present application;

[0049] Figure 9 Relevance and characteristic time between calm and happy expressions of students in a certain time period provided by the embodiment of the present application;

[0050] Figure 10 Relevance and characteristic time between happy and fearful expressions of students in a certain time period provided by the embodiment of the present application.

[0051] Icons: 201 - Acquisition module; 202 - Recognition module; 203 - First acquisition module; 204 - Processing module; 205 - Second acquisition module. Specific implementation manners

[0052] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0053] Please refer to Figure 1 As shown, the embodiment of the present invention provides a teaching feedback method for analyzing human face expression information, including:

[0054] Step 101: Acquire a teaching video containing the human face expression information of each student during the class time period of the class for which the teaching effect is to be judged.

[0055] In practice, during the class time period of the class for which the teaching effect is to be judged, a teaching video containing the facial expression information of each student is recorded by a camera. The class for which the teaching effect is to be judged can be a face-to-face class or an online class. For an online class, all students can have classes in a classroom, or each student can have classes at their respective homes. In the embodiment of the present invention, all students have classes in a classroom. A certain classroom in a certain university is selected, and the experimental subjects are authorized to use their portraits. During the recording process, the audio of the teacher's teaching content is recorded to ensure the clarity of the teaching content. The video recorded during the teaching process of a certain knowledge point is selected as an example for processing.

[0056] Step 102: Identify the head portrait collection of each student arranged in chronological order during the teaching period according to the teaching video.

[0057] Further, step 102 includes step 1021: Extract images from the teaching video at a set frequency and save them in picture format in chronological order.

[0058] For example, use the readframe function of MATLAB to extract the first frame in each second of the video recording and save it in picture format in chronological order.

[0059] Step 1022: Identify the faces of the students from the pictures and establish a head portrait coordinate frame for each student.

[0060] For example, use a face detection program to identify the faces of the students from the pictures. Figure 3 Under the environment of the class for which the teaching effect is to be judged provided by the embodiment of the present application, the head portrait coordinate frame corresponding to each student is obtained by face recognition in the classroom environment based on Haar-Like features. The face of the student that can be recognized is within the white frame. Basically, all students whose front clear faces can be photographed are recognized. The algorithm will also number each tracked face, and the number on each white frame is the number of the student in this class.

[0061] Step 1023: According to all the recognized head portrait coordinate frames in each picture in chronological order, obtain the face portraits of each student, and establish a folder containing face portrait information for each student and save it in chronological order.

[0062] Exemplarily, use the imcrop function in MATLAB to intercept the head coordinate frames of all recognized students in each picture in chronological order. According to the head coordinate frames of the faces, intercept the face portraits of each student. Then, save the intercepted face portraits within the entire class period separately for different students in their respective corresponding folders. Next, make a face wall portrait set according to the requirements of different expression analysis websites. The expression analysis website used in the embodiments of this application requires a face wall portrait set with a 7*7 specification. Therefore, for the same student, 49 portraits corresponding to every 49 seconds of his / hers are combined into a face wall portrait set with a 7*7 specification, that is, each portrait set corresponds to the face portraits of this student with a time length of 49 seconds. The portraits per second are named after their corresponding moments. Then, name the above face wall portrait set after the time corresponding to its first portrait. The portraits of this student within the entire class period will be saved into a collection of face wall portrait sets named in chronological order according to the above method, and stored in a folder named after the student number. Figure 4 This is the 7*7 specification face wall portrait set of a certain student detected and recognized during a certain period provided by the embodiments of this application.

[0063] Step 103: Perform expression recognition on the portrait collection of each student, and correspondingly obtain the high-frequency time series of the micro-expressions of each student arranged in chronological order along with the classroom time to be judged for teaching effect, and process the high-frequency time series to obtain face expression numerical information.

[0064] Among them, Step 103 includes:

[0065] Step 1031: Use an expression recognition tool to recognize the images in the portrait collection of each student and correspondingly obtain the high-frequency time series of each micro-expression of each student changing with time.

[0066] Specifically, upload the portrait collection of each student to an expression recognition tool, such as a network server expression analysis container, and correspondingly obtain the high-frequency time series of each micro-expression (seven basic micro-expressions: calm, angry, disgusted, frightened, happy, sad, surprised) of each student changing with classroom time. Figure 5 This is for facilitating the observation of the high-frequency time series of the seven micro-expressions of a certain student changing with time during a certain period.

[0067] In the embodiments of the present invention, the steps of uploading the face wall portrait set and sorting out data include:

[0068] 1. Comprehensively analyze factors such as the openness, stability, and price of the business results of each family's face expression information, and select Microsoft Face Recognition as the website for analyzing students' face expression information. We upload the face wall portrait sets with a 7*7 specification in sequence to obtain the returned face expression information data. The data corresponding to one face wall portrait set is saved as an Excel sheet.

[0069] II. Since the facial expression information data returned by the website is saved in a disordered manner in the Excel table, it needs to be re-sorted in the original time order. Analyzing the information returned by the website shows the pixel positions of each avatar. Since the pixel size of one avatar is 100*100, the pixel positions of a set of avatars on the facial wall are x*y = (0 - 700)*(0 - 700), and each avatar corresponds to a different pixel position range according to the 7*7 specification. For example, if the returned pixel position range of the avatar is x*y = (0 - 100)*(600 - 700), it corresponds to the first column in the seventh row. When sorting, according to the different pixel position ranges of the avatars in the Excel table, the loop algorithm is used to sort the 49 sets of seven micro-expression values in ascending order of time into seven time series.

[0070] III. After sorting all the facial wall avatar sets with the 7*7 specification into continuous time series according to the above method, then sort them in ascending order of time according to the file names of each facial wall avatar set to form a complete continuous time series. That is to say, a student corresponds to a set of facial wall avatar collections during the entire class period. After uploading to the website, a set of Excel table facial expression information data is obtained. After sorting, a set of high-frequency time series of seven micro-expressions is obtained.

[0071] Step 1032: Remove white noise from the high-frequency time series of each student, and correspondingly obtain the facial expression numerical information of each student.

[0072] The micro-expression numerical values of students observed in real classes inevitably contain noise. Therefore, band-pass filtering is used to remove white noise from the high-frequency time series of micro-expressions. Figure 6 It is the time series of the happy expression of a student after denoising and smoothing in a certain period.

[0073] Furthermore, after removing white noise from the high-frequency time series of each student, normalization processing is also performed to transform the absolute values of the high-frequency time series into their rates of change over time, that is, the facial expression numerical information of each student is correspondingly obtained.

[0074] The personalities of different students are different, and the absolute values reflected by their micro-expressions should also be different. That is, extroverted students may show their emotions more obviously, while introverted students may be relatively calm. Therefore, it is necessary to perform normalization processing on the high-frequency time series of different students after removing white noise, and transform the absolute values of the high-frequency time series into their rates of change over time, that is, the time series of the relative strength index of a certain expression.

[0075] In an embodiment of the present invention, using the Relative Strength Index (RSI) theory, the absolute values of the high-frequency time series of different target students are transformed into their rates of change over time. The relative strength index of the original theory varies from 0 to 100, and 50 is the point where the rate of change is 0. It is improved and applied to the processing of the micro-expression values of different students. Specifically, the micro-expression value is set to the 0 value of the relative strength index at the moving average value over a period of time. Here, the moving average is recalculated every 200 ms to 300 ms to be more accurate. If the expression value is greater than the moving average, the rate of change is positive; if the expression value is less than the moving average, the rate of change is negative. Thus, the expression values of different students are made comparable, which can more accurately reflect the feedback of each student on classroom learning and reduce the judgment gap caused by the individual personalities of the students.

[0076] Figure 7 This is the time series of the relative strength index of a certain student's happy expression provided by the embodiment of the present application. The abscissa is time, and the ordinate is the rate of change of the micro-expression value over time.

[0077] Step 104: Process the facial expression numerical information of each student to obtain an expression value grayscale image.

[0078] The characteristic changes of the seven micro-expressions over time are interrelated. In order to clearly show the characteristic changes and interrelationships of each expression for later judgment, in the embodiment of the present invention, the time series of the relative strength of the seven micro-expressions of the student constitutes a grayscale image.

[0079] Figure 8 This is the grayscale image of the time series of the relative strength index of the seven micro-expressions of a certain student in a certain period. The grayscale color bar in the grayscale image represents the change intensity of the expression value.

[0080] Step 105: Obtain the teaching effect feedback of the classroom where the teaching effect to be judged is given through the expression value grayscale image.

[0081] Among them, step 105 includes:

[0082] Set the threshold values of each expression value to obtain the proportion of each expression value.

[0083] To a certain extent, the rate of change of the same expression over time can also reflect the emotional changes of the student. For example, for the "happy" expression value, when the rate of change over time is positive, it indicates that the student's happiness level is increasing at this time, which belongs to a positive emotion; when the rate of change over time is negative, it indicates that the student's happiness level is decreasing.

[0084] Formulate a threshold judgment rule for the expression value grayscale image, and judge the recognition degree of each student for the teaching effect of the given teaching according to the threshold judgment rule of the expression value grayscale image.

[0085] In an embodiment of the present invention, the expression value grayscale map threshold determination rule includes:

[0086] Referring to and improving the relative strength index (RSI) theory, the relative strength index values in the normal distribution are set between -20 and 20, and it is considered that the change in the expression value is not obvious within this range; if the relative strength index value reaches 30, it is considered that the expression value dominates at this time; otherwise, it is considered that the expression value does not dominate. Thus, the dominant position of various expression values of students throughout the time period is determined. For example, if the dominant position of the "happy" expression value among the seven micro-expressions is not less than 30%, and the dominant position of the "disgust" expression value is not higher than 5%, it can be considered that the students recognize the teaching method of the teacher in this class, that is, the students' recognition of the teacher's teaching method is positive; otherwise, the students' recognition of the teacher's teaching method is negative.

[0087] Through the expression value grayscale map threshold determination rule, the recognition degree of the teacher's teaching among students can be judged to a certain extent. If the positive recognition degree of the class students is less than 60%, the teacher should reflect on his / her teaching method and make appropriate adjustments.

[0088] Furthermore, step 105 further includes:

[0089] Obtain various expression combinations of each student and the time when each expression combination appears according to the expression grayscale map of each student, and calculate the correlation degree and characteristic time between any two expressions to obtain the proportion of the duration of each expression combination appearance.

[0090] Process the expression value grayscale map of each student, and use the non-stationary time correlation function to calculate the correlation degree and characteristic time between any two expressions. Among them, various expression combinations include all combinations of any two selected expressions, and generally record the expression combinations with high correlation degrees. These can reflect the frequency of the simultaneous or sequential appearance of two expressions in time, thereby reflecting several expression combinations that frequently appear in the learning process of the target student.

[0091] Figure 9 For the correlation degree between the calm and happy expressions of a student at a certain time in the embodiment of the present application, Figure 10 For the correlation degree between the happy and fearful expressions of a student at a certain time in the embodiment of the present application. Figure 9 And Figure 10 In, the time corresponding to the highest point on the vertical axis of the figure, Figure 9 In which Lag: -10 is marked, Figure 10 In which Lag: 7 marked is the characteristic time, that is, Figure 9 In, the correlation degree between the happy and calm characteristic times of about -10s is relatively high, Figure 10The correlation between the time of neutral, happy, and fearful expressions is relatively high at around 7 seconds. This indicates that the target student frequently shows a happy expression about 10 seconds after a neutral expression, and a fearful expression 7 seconds after a happy expression. This reflects that the student frequently exhibits "neutral - happy" and "happy - fearful" expression combinations during the learning process.

[0092] Mark the time periods of knowledge points in the content of the teaching.

[0093] Formulate the association judgment rule for the expression grayscale map, and determine the mastery of each knowledge point by each student according to the appearance time and duration ratio of each expression combination within the marked time period of the knowledge point.

[0094] This time is the time period corresponding to the explanation of the knowledge point. The duration ratio is the ratio of the appearance time of the expression combination within this time period to the total time of this time period. Accurately calculate the appearance time and duration ratio of the expression combination in each marked knowledge point time period, formulate the association judgment rule for the expression value grayscale map, qualitatively give the mastery degree of each knowledge point by the student, and qualitatively give the evaluation of the teaching effect of the whole class for the student.

[0095] Associate and mark the time of the knowledge point elaboration in the recorded teaching video of the classroom whose teaching effect is to be judged, and extract the expression combinations in each knowledge point elaboration time period. By calculating the correlation and characteristic time between any two expressions in each knowledge point elaboration time period, count the expression combinations that appear during the student's learning process of each knowledge point, so as to calculate the duration ratio of various expression combinations that appear. For example, if the current knowledge point elaboration time period is 0 - 400 seconds, the expression correlation and characteristic time are calculated every 80 seconds. For a certain two expressions, if the expression correlations calculated at 0 - 80 seconds and 160 - 240 seconds are both higher than 0.4, it is considered that the combination of these two expressions frequently appears within 0 - 80 seconds and 160 - 240 seconds. Also, according to the calculated characteristic time, if the appearance order of these two expressions in these two time periods is the same, it is considered to be the same group of expression combinations. At this time, record that this expression combination appears in the knowledge point time period of 0 - 400 seconds, and the duration ratio is 40%.

[0096] During the student's listening process, the seven micro - expressions are interrelated, and various expression combinations often appear frequently. Students tend to be relatively "neutral" towards newly learned knowledge, relatively "fearful" when encountering difficulties, and relatively "happy" when problems are solved smoothly... For example, getting a "happy - fearful" expression combination may mean that the student is happy when solving a problem but then encounters a new difficult - to - understand problem and feels fearful.

[0097] Therefore, the specific determination method is as follows: when the proportion of the duration of "calm - happy" in a certain student's facial expression combination is higher than 60% during the period when a certain knowledge point is being elaborated, the mastery level of this knowledge point for this student is given as good; when the proportion of the duration of "calm - happy" in the facial expression combination is lower than 40%, the mastery level of this knowledge point for this student is given as poor. Similarly, the proportion of the duration of each group of facial expression combinations such as "fear - calm", "happy - fear", etc. of this student is also calculated, and the mastery level of the knowledge point is given. Finally, the mastery levels of the knowledge points determined for all facial expression combinations are statistically analyzed, and the mastery level (good, poor) of this student for this knowledge point is qualitatively given.

[0098] Furthermore, the teaching feedback method for analyzing facial expression information provided in the embodiment of the present application, step 105 further includes:

[0099] Statistically analyze the mastery situations of all students regarding all knowledge points, and determine the teaching effect of the lecture according to the mastery levels of the knowledge points.

[0100] Statistically analyze the mastery levels of all knowledge points of all students in the entire class, calculate the proportion of the two results of the mastery levels of the knowledge points. If the proportion of those with good mastery level is higher than 60%, then qualitatively determine that the classroom effect of this class is good. On the contrary, if it is lower than 40%, then qualitatively determine that the classroom effect of this class is poor. Then the teacher needs to reflect on the elaboration of the knowledge points in their teaching and make appropriate adjustments.

[0101] An embodiment of the present invention provides a teaching feedback method for analyzing human face expression information, including: obtaining a teaching video containing the human face expression information of each student during the class time period of the class whose teaching effect is to be judged; identifying, according to the teaching video, a collection of head portraits of each student arranged in chronological order during the teaching; performing expression recognition on the collection of head portraits of each student, correspondingly obtaining a high-frequency time series of micro-expressions of each student arranged in chronological order with respect to the class time of the class whose teaching effect is to be judged, and processing the high-frequency time series to obtain human face expression numerical information; processing the human face expression numerical information of each student to obtain an expression value grayscale image; obtaining a teaching effect feedback of the class whose teaching effect is to be judged through the expression value grayscale image. This method makes the classroom teaching feedback more intelligent by identifying the human face expression information of each student during the class and obtaining and analyzing the quantified expression information. This method is particularly suitable for establishing a teaching evaluation mechanism that is convenient for students and teachers to use in a university classroom, urging students to study independently, providing feedback on the class effect for teachers and students, and promoting the improvement of teaching effects. Whether it is a face-to-face class or an online class, the expression information of each student in the class can be identified and analyzed, providing more real-time and accurate feedback on the emotions of students for teaching. The teaching effect feedback obtained according to the human face expression value grayscale image comes from the feedback analysis results of each student in each class, is more targeted at the teaching content, and is more helpful for teachers to improve their teaching methods. This method can simultaneously pay attention to the facial expressions of each student in both face-to-face classes and online classes, and can directly observe the changes in the emotional expressions of students to judge the teaching effect.

[0102] As Figure 2 shown, an embodiment of the present application provides a teaching feedback device for analyzing human face expression information, including:

[0103] An acquisition module 201, configured to obtain a teaching video containing the human face expression information of each student during the class time period of the class whose teaching effect is to be judged.

[0104] An identification module 202, configured to identify, according to the teaching video, a collection of head portraits of each student arranged in chronological order during the teaching.

[0105] Among them, the identification module 202 includes:

[0106] An extraction and saving sub-module, configured to extract images in the teaching video at a set frequency and save them in the form of pictures in chronological order.

[0107] An identification and establishment sub-module, configured to identify the human face of a student from the pictures and establish a head portrait coordinate frame for each student.

[0108] A obtaining sub-module, configured to obtain the face avatars of each student according to all the recognized face coordinate frames in each of the pictures in chronological order, and create a folder for each student that stores face avatar information in chronological order.

[0109] A first obtaining module 203, configured to perform expression recognition on the avatar collection of each student, correspondingly obtain the high-frequency time series of micro-expressions of each student arranged in chronological order along with the classroom time for judging the teaching effect to be judged, and process the high-frequency time series to obtain face expression numerical information.

[0110] Among them, the first obtaining module 203 includes:

[0111] An obtaining sub-module, configured to use an expression recognition tool to recognize the images in the avatar collection of each student and correspondingly obtain the high-frequency time series of each micro-expression of each student changing over time;

[0112] A noise removal sub-module, configured to remove white noise from the high-frequency time series of each student, and correspondingly obtain the face expression numerical information of each student.

[0113] Furthermore, after the noise removal sub-module removes white noise from the high-frequency time series of each student, it also performs normalization processing, transforms the absolute value of the high-frequency time series into its rate of change over time, that is, correspondingly obtains the face expression numerical information of each student.

[0114] A processing module 204, configured to process the face expression numerical information of each student to obtain an expression value grayscale image.

[0115] A second obtaining module 205, configured to obtain the teaching effect feedback of the classroom for judging the teaching effect to be judged through the expression value grayscale image.

[0116] Among them, the second obtaining module 205 includes:

[0117] A setting sub-module, configured to set the thresholds of each expression value to obtain the proportion of each expression value.

[0118] A first formulating and judging sub-module, configured to formulate a threshold judgment rule for the expression value grayscale image, and judge the recognition degree of each student for the teaching effect according to the threshold judgment rule of the expression value grayscale image.

[0119] Furthermore, the second obtaining module 205 further includes:

[0120] A calculation sub-module, configured to obtain various expression combinations of each student and the time when each expression combination appears according to the expression grayscale image of each student, calculate the correlation degree and characteristic time between any two expressions to obtain the proportion of the appearance duration of each expression combination.

[0121] A marking sub-module for marking the time periods of knowledge points in the content of the teaching.

[0122] A second formulating and judging sub-module for formulating an association judgment rule for the expression grayscale map, and judging the mastery of each knowledge point by each student according to the appearance time and duration ratio of each expression combination within the marked time period of the knowledge point.

[0123] Optionally, the second obtaining module 205 further includes:

[0124] A statistics sub-module for statistics the mastery of all knowledge points by all students, and judging the teaching effect according to the mastery degree of the knowledge points.

[0125] An embodiment of the present invention provides a server, including: a memory and a processor.

[0126] The memory is used for storing program instructions.

[0127] The processor is used for executing the program instructions in the receiver, so that the receiver executes the teaching feedback method for analyzing the facial expression information.

[0128] An embodiment of the present invention provides a computer-readable storage medium, and the computer-readable storage medium stores executable instructions, and when the computer executes the executable instructions, it can implement the teaching feedback method for analyzing the facial expression information.

[0129] The above storage medium includes but is not limited to Random Access Memory (RAM), Read-Only Memory (ROM), Cache, Hard Disk Drive (HDD) or Memory Card. The memory can be used for storing computer program instructions.

[0130] Although the present application provides method operation steps as described in the embodiments or flowcharts, more or fewer operation steps may be included based on routine or non-creative labor. The step order listed in this embodiment is only one way among the execution orders of many steps, and does not represent the only execution order. When the actual device or client product executes, it can be executed in the order of the method described in this embodiment or shown in the drawings, or executed in parallel (such as in an environment of parallel processors or multi-threaded processing).

[0131] The devices or modules illustrated in the above embodiments can be specifically implemented by computer chips or entities, or by products with certain functions. For the convenience of description, when describing the above devices, they are divided into various modules according to functions and described separately. When implementing the present application, the functions of each module can be implemented in the same or multiple software and / or hardware. Of course, the module that implements a certain function can also be implemented by combining multiple sub-modules or sub-units.

[0132] The methods, devices or modules described in the present application can be implemented in the form of computer-readable program codes. The controller can be implemented in any appropriate manner. For example, the controller can take the form of, for example, a microprocessor or a processor, and a computer-readable medium that stores computer-readable program codes (such as software or firmware) executable by the (micro)processor, logic gates, switches, application specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art also know that in addition to implementing the controller in the form of pure computer-readable program codes, the method steps can be logically programmed to enable the controller to be implemented in the form of logic gates, switches, application specific integrated circuits, programmable logic controllers, and embedded microcontrollers to achieve the same function. Therefore, such a controller can be regarded as a hardware component, and the devices included therein for implementing various functions can also be regarded as the structures within the hardware component. Or even, the devices for implementing various functions can be regarded as both software modules for implementing the method and the structures within the hardware component.

[0133] Some modules in the devices described in the present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, classes, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in a distributed computing environment, where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0134] As can be seen from the description of the above embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary hardware. Based on such an understanding, the technical solution of this application, in essence or the part that contributes to the prior art, can be embodied in the form of a software product, or can also be reflected in the implementation process of data migration. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions for causing a computer device (which can be a personal computer, mobile terminal, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of this application.

[0135] The various embodiments in this specification are described in a progressive manner. For the same or similar parts between the various embodiments, reference can be made to each other. The key point of each embodiment is to illustrate the differences from other embodiments. All or part of this application can be used in many general-purpose or special-purpose computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, mobile communication terminals, multi-processor systems, microprocessor-based systems, programmable electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and so on.

[0136] The above embodiments are only used to illustrate the technical solutions of this application, rather than limiting this application; although this application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements on some or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of this application.

Claims

1. A teaching feedback method for analyzing facial expression information, characterized in that Including: Obtain a teaching video of the class whose teaching effect is to be judged, which includes the facial expression information of each student during the class time period; Identify the head portrait collection of each student arranged in chronological order during the teaching according to the teaching video; Perform facial expression recognition on the head portrait collection of each student, and correspondingly obtain the high-frequency time series of the micro-expressions of each student arranged in chronological order along with the time of the class whose teaching effect is to be judged, and process the high-frequency time series to obtain facial expression numerical information; Process the facial expression numerical information of each student to obtain an expression value grayscale image; Obtain the teaching effect feedback of the class whose teaching effect is to be judged through the expression value grayscale image; Among them, the obtaining of the teaching effect feedback of the class whose teaching effect is to be judged through the expression value grayscale image includes: Set the threshold of each expression value to obtain the proportion of each expression value; Formulate an expression value grayscale image threshold judgment rule, and judge the recognition degree of each student for the teaching effect according to the expression value grayscale image threshold judgment rule; Obtain various expression combinations of each student and the time when each expression combination appears according to the expression grayscale image of each student, calculate the correlation and characteristic time between any two expressions to obtain the duration proportion of each expression combination; Mark the knowledge point time periods in the content of the teaching; Formulate an expression grayscale image association judgment rule, and judge the mastery of each knowledge point by each student according to the time and duration proportion of each expression combination appearing within the marked knowledge point time period.

2. The teaching feedback method for analyzing human face expression information according to claim 1, characterized in that The identifying the head portrait collection of each student arranged in chronological order during the teaching according to the teaching video includes: Extract the images in the teaching video at a set frequency and save them in picture format in chronological order; Identify the faces of the students from the pictures and establish a head portrait coordinate frame for each student; Obtain the face head portraits of each student according to all the identified head portrait coordinate frames in each picture in chronological order, and establish a folder including face head portrait information for each student saved in chronological order.

3. The teaching feedback method for analyzing human face expression information according to claim 1, characterized in that, The performing facial expression recognition on the head portrait collection of each student, and correspondingly obtaining the high-frequency time series of the micro-expressions of each student changing with time, and processing the high-frequency time series to obtain facial expression numerical information includes: Use a facial expression recognition tool to recognize the images in the head portrait collection of each student and correspondingly obtain the high-frequency time series of each micro-expression of each student changing with time; Remove white noise from the high-frequency time series of each student to correspondingly obtain the facial expression numerical information of each student.

4. The teaching feedback method for facial expression information analysis according to claim 3, characterized in that After removing white noise from the high-frequency time series of each student, perform normalization processing, and transform the absolute value of the high-frequency time series into its change rate with time, that is, correspondingly obtain the facial expression numerical information of each student.

5. The teaching feedback method for analyzing human face expression information according to claim 1, wherein The obtaining of the teaching effect feedback of the class whose teaching effect is to be judged through the expression value grayscale image further includes: Statistically analyze the mastery of all students regarding all knowledge points, and determine the effectiveness of the teaching based on the degree of knowledge point mastery.

6. A teaching feedback device for analyzing human face expression information for implementing the method according to any one of claims 1-5, characterized in that, Including: An acquisition module for acquiring a teaching video containing the facial expression information of each student during the class time period of the teaching whose effectiveness is to be judged. An identification module for identifying, based on the teaching video, a collection of head portraits of each student arranged in chronological order during the teaching. A first acquisition module for performing facial expression recognition on the collection of head portraits of each student, correspondingly obtaining a high-frequency time series of micro-expressions of each student arranged in chronological order with respect to the time of the teaching whose effectiveness is to be judged, and processing the high-frequency time series to obtain facial expression numerical information. A processing module for processing the facial expression numerical information of each student to obtain an expression value grayscale image. A second acquisition module for obtaining the teaching effectiveness feedback of the teaching whose effectiveness is to be judged through the expression value grayscale image. Among them, obtaining the teaching effectiveness feedback of the teaching whose effectiveness is to be judged through the expression value grayscale image includes: Setting thresholds for each expression value to obtain the proportion of each expression value. Formulating an expression value grayscale image threshold judgment rule, and judging the recognition degree of each student for the teaching effectiveness according to the expression value grayscale image threshold judgment rule. Obtaining various expression combinations of each student and the time when each expression combination appears according to the expression grayscale image of each student, calculating the correlation and characteristic time between any two expressions to obtain the duration proportion of each expression combination appearance. Marking the knowledge point time periods in the content of the teaching. Formulating an expression grayscale image association judgment rule, and judging the mastery of each knowledge point by each student according to the time and duration proportion of each expression combination appearance within the marked knowledge point time period.

7. A server, characterized in that, Including: A memory and a processor; The memory is used for storing program instructions; The processor is used for executing the program instructions in the receiver, so that the receiver executes the teaching feedback method for facial expression information analysis as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores executable instructions, and when the computer executes the executable instructions, it can implement the teaching feedback method for facial expression information analysis as described in any one of claims 1 to 5.

Images