Student classroom attention analysis system based on facial multi-modal emotion fusion

Abstract

The invention provides a student classroom attention analysis system based on facial multi-modal emotion fusion, aiming at solving the problem that an effective micro-expression recognition method for judging the attention analysis of students on teaching content is lacked in the current intelligent education classroom environment. The invention mainly solves the technical problem of providing a facial micro-expression recognition method based on multi-modal fusion aiming at the deficiency of facial micro-expression recognition analysis in an intelligent teaching classroom at present, and the facial micro-expression recognition method comprises the following steps of: dynamically enhancing local detail features of a facial image by utilizing coding features of a facial action unit; according to the method, the image difference subtle influence caused by low intensity of micro-expression motion is reduced, whether the students hide the real emotional states by using micro-expressions or not is analyzed, the attention condition of the students in class is analyzed according to the recognition condition of the micro-expressions and information such as the usual performance of the students and the teacher-student communication frequency, and then the teaching content of teachers is dynamically adjusted.

Description

Technical Field

[0001] The present invention belongs to the field of intelligent education technology, and relates to a method and system for identifying multimodal emotion fusion in classroom teaching, and in particular to a method and system for analyzing students' classroom attention through facial multimodal emotion fusion. Background Art

[0002] With the rapid development of artificial intelligence and machine learning technologies, intelligent human-computer interaction has gradually entered people's daily lives. The future society will become more highly intelligent, and intelligent human-computer interaction will permeate all aspects of human life. In the field of smart education, the combination of artificial intelligence methods and traditional education can improve the quality of classroom teaching by paying attention to students' classroom performance and dynamically adjusting teaching content.

[0003] In the classroom teaching process of smart education, the current classroom management system can analyze students' attention and interest in teaching content through their daily behavior data, but students may use micro-expressions to conceal their true emotional changes during the class. To address this problem, the present invention will conduct an in-depth analysis of the multimodal data of students' faces during the learning process based on the student facial image data obtained by the classroom management system, explore the correlation of multimodal data, thereby achieving robust emotion recognition and attention analysis, and providing new means and new ways for the development of classroom teaching in smart education. Summary of the invention

[0004] In view of the lack of effective micro-expression recognition methods in the current smart education classroom environment to judge students' attention analysis on the teaching content, the present invention proposes a student classroom attention analysis system based on facial multimodal emotion fusion. The main technical problem solved by the present invention is that in view of the lack of facial micro-expression recognition and analysis in the current smart teaching classroom, a facial micro-expression recognition method based on multimodal fusion is proposed, which dynamically enhances the local detail features of the facial image using facial action unit coding features, reduces the subtle impact of image differences caused by low-intensity micro-expression movements, analyzes whether students use micro-expressions to hide their true emotional state, and analyzes students' attention to the class based on the recognition of micro-expressions and students' usual performance, the frequency of communication between teachers and students, and other information, thereby improving the dynamic adjustment of teachers' teaching content.

[0005] In order to achieve the above objectives, the present invention is implemented by adopting the following technical methods.

[0006] A student classroom attention analysis system based on facial multimodal emotion fusion, characterized by comprising a smart classroom management system module, a student classroom attention analysis module, and a teaching content adjustment module; The smart classroom management system module is used by teachers, and various information about students can be obtained through this module: The student classroom attention analysis module can determine the student's learning status by analyzing the student's micro-expressions and the frequency of communication with the teacher; The teaching content adjustment module makes corresponding adjustments to the teaching content based on the students' learning status, the frequency of communication between teachers and students, their usual performance and previous grades in class.

[0007] On the other hand, the present invention provides a method for analyzing students' classroom attention based on facial multimodal emotion fusion, comprising the following steps: S1: Facial image data collection: Obtain facial image sequence data of students learning in the smart classroom; S2: Extracting facial emotion features: inputting the preprocessed facial image sequence data collected in step S1 into the image coding model to extract the facial emotion change features of the user, wherein the preprocessing includes noise reduction, segmentation, and normalization; S3: Facial action unit recognition: by accurately locating the region of interest of the facial action unit when the micro-expression occurs in the facial emotion change features obtained in step S2, the discriminative local features are learned to obtain the micro-expression recognition sequence; S4: learnable masked autoencoder model processing: inputting the peak frame and surrounding images in the micro-expression recognition sequence into the learnable masked autoencoder model and processing them to obtain action unit encoding features; S5: Dynamic attention fusion model processing: The features extracted by the multi-head self-attention model and the action unit encoding features are fused with the local micro-expression image features with high attention weights through the dynamic attention fusion model to improve the distinctiveness of facial emotion features.

[0008] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention and implement it according to the contents of the specification, the following is a detailed description of the preferred embodiments of the present invention in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The present invention will be further described below with reference to the accompanying drawings and embodiments, in which: Figure 1 It is a schematic diagram of the working process of an embodiment of the present invention.

[0010] Figure 2 Schematic diagram of facial emotion feature extraction according to an embodiment of the present invention.

[0011] Figure 3 Schematic diagram of facial action unit recognition according to an embodiment of the present invention.

[0012] Figure 4Schematic diagram of a learnable masked self-encoding model according to an embodiment of the present invention.

[0013] Figure 5 It is a schematic diagram of a dynamic attention fusion model according to an embodiment of the present invention.

[0014] Figure 6 It is a schematic diagram of the teaching content adjustment process based on micro-expression recognition according to an embodiment of the present invention. DETAILED DESCRIPTION

[0015] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described below in conjunction with the accompanying drawings and preferred embodiments. However, the protection scope of the present invention is not limited to this embodiment, and any modifications and changes made within the scope of the claims should be covered within the protection scope of the present invention. Example 1

[0016] A student classroom attention analysis system based on facial multimodal emotion fusion: the classroom management system module collects students' personal information, course information, teaching content release, course assignments and video data in the teaching process; then the student classroom attention analysis module extracts facial emotion features of video images during the teacher's teaching process, and performs facial action unit recognition and micro-expression recognition; when the system finds that the student's emotional state continues to change to a negative emotional category, the teaching content is dynamically adjusted through the teaching content adjustment module to attract the student's attention. When the student's emotional state is positive, the student's attention level in the learning process is analyzed based on the high frequency of teacher-student communication, good performance in normal times and previous grades, and the teaching content is continuously promoted until the end of the course. Otherwise, the teaching content is dynamically adjusted to focus the student's attention on the teaching content.

[0017] like Figure 2-6 As shown, a method for analyzing students' classroom attention based on facial multimodal emotion fusion includes the following steps: S1: Facial image data acquisition: The facial image data of students is acquired through an 80-megapixel high-definition camera, which is mainly used to obtain facial image sequence data of students learning in the smart classroom.

[0018] S2: Facial emotion feature extraction step: The peak frame is mapped to the starting frame through the cyclic generative adversarial network, and the identity attribute features and emotion change features in the peak frame image are decoupled through image generation. The identity attribute features are restored to the starting frame, and the emotion features are used for the location of the region of interest of the facial action unit and micro-expression recognition. The cyclic generative adversarial network uses paired image data as a training set to automatically learn the mapping relationship between the two.

[0019] S3: Facial action unit recognition: Aiming at the problem of locating the regions of interest of facial action units in micro-expression recognition, a local region of interest localization method of recursively generating attention networks is proposed for facial action unit recognition. This method can promote the learning of discriminative local features by accurately locating the regions of interest of facial action units when micro-expressions occur (e.g., corners of the mouth, eyes, nose tip, etc.), thereby further determining the subtle feature changes of the regions of interest of these facial action units when micro-expressions occur. The present invention uses a mutually reinforcing method to recursively learn the local regions of interest of discriminative facial action units and the feature representations of these regions from global to local.

[0020] S4: Learnable Masked Autoencoder Model: In view of the problem that the small dataset of micro-expressions seriously limits the model fitting problem, a model based on a learnable masked autoencoder is proposed. By inputting the peak frame and surrounding images in the micro-expression recognition sequence into the learnable masked autoencoder model, the peak frame image is first masked by the learnable masked autoencoder model, and then the high-weight image sub-block is input into the encoder. In the model, the encoder uses a visual self-attention encoder structure to extract the features of each sub-block, and then restores it to its original position and splices it with the feature representation of the masked sub-block in the original image, and then inputs the decoder structure to restore it to the starting frame.

[0021] S5: Dynamic attention fusion model: In order to solve the problem of inaccurate positioning and redundancy of the region of interest of facial action units in micro-expression recognition, the encoding features of facial action units are used to enhance the feature process of micro-expression images. This paper proposes a local detail feature enhancement method based on a dynamic visual self-attention model for micro-expression recognition. A learnable mask labeling model is used to decouple the local areas with implicit emotional expressions in micro-expression images. The features extracted by the multi-head self-attention model and the action unit encoding features are fused with the local micro-expression image features with high attention weights through a dynamic attention fusion model. The action unit encoding is applied to the micro-expression image feature extraction process in a dynamic weighted manner, which helps to improve the distinguishability of facial emotional features.

[0022] like Figure 6As shown in the diagram of the teaching content adjustment process based on micro-expression recognition, in the process of student learning, the user's micro-expression and the frequency of communication with the teacher are a kind of feedback on the degree of attention to the current teaching content, which has a certain universality. When students are not interested in the teaching content, they will use positive facial micro-expressions to hide their negative emotions. The method of micro-expression recognition is used to judge the student's true emotional state, and the frequency of interaction between students and teachers is low. When students pay more attention to the teaching content, their micro-expressions are usually more positive, and the frequency of interaction between students and teachers is high. According to the emotional state of students in the classroom, the frequency of communication between teachers and students, their usual performance and previous grades, the corresponding teaching content is adjusted. When the student's emotions in the course learning are positive, the frequency of communication between teachers and students is high, the usual performance is good and the previous grades are stable, the teacher maintains the teaching progress. When the student is in a negative emotional state, the frequency of communication between teachers and students is low, the usual performance is poor and the previous grades fluctuate greatly, the teacher will dynamically adjust the current teaching plan.

[0023] The technical solution of the present invention will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0024] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

Claims

1. A student classroom attention analysis system based on facial multimodal emotion fusion, characterized in that: It includes smart classroom management system module, student classroom attention analysis module, and teaching content adjustment module; The smart classroom management system module is used by teachers, and various information of students can be obtained through this module: The student classroom attention analysis module can determine the student's learning status by analyzing the student's micro-expressions and the frequency of communication with the teacher; The teaching content adjustment module makes corresponding adjustments to the teaching content based on the students' learning status, the frequency of communication between teachers and students, their usual performance and previous grades in class.

2. A method for analyzing students' classroom attention based on facial multimodal emotion fusion, characterized in that: The following steps are involved: S1: Facial image data collection: Obtain facial image sequence data of students learning in the smart classroom; S2: Extracting facial emotion features: inputting the preprocessed facial image sequence data collected in step S1 into the image coding model to extract the facial emotion change features of the user, wherein the preprocessing includes noise reduction, segmentation, and normalization; S3: Facial action unit recognition: by accurately locating the region of interest of the facial action unit when the micro-expression occurs in the facial emotion change features obtained in step S2, the discriminative local features are learned to obtain the micro-expression recognition sequence; S4: learnable masked autoencoder model processing: inputting the peak frame and surrounding images in the micro-expression recognition sequence into the learnable masked autoencoder model and processing them to obtain action unit encoding features; S5: Dynamic attention fusion model processing: The action unit encoding features are injected into the multi-head self-attention model, and the features extracted by the multi-head self-attention model and the action unit encoding features are fused with the local micro-expression image features with high attention weights through the dynamic attention fusion model to improve the distinctiveness of facial emotion features.

Images