A method for detecting emotional engagement for positive emotion perception

Abstract

The present invention is a method for detecting emotional engagement for positive emotion perception, which relates to the field of data analysis technology. It extracts features from data of different dimensions in students' classroom learning, realizes information recognition of each dimension, and then extracts features from data of each dimension in students' classroom learning in decision fusion and result application analysis. Then, a fusion strategy based on a deep learning network is adopted, which is applied to subsequent supervised classification to construct a model of students' positive emotional engagement. Different multimodal information such as students' cognition, emotion and thinking obtained from different channels is integrated into a framework to comprehensively reflect students' positive emotions.

Description

Technical Field

[0001] The present invention relates to the technical field of data analysis, and in particular to an emotion engagement detection method for positive emotion perception. Background Art

[0002] Emotional engagement is an important quantitative indicator for the process evaluation of students' learning. By tracking and evaluating the emotional state of students throughout the learning process, we can discover students' emotional intentions and tap their learning potential. Positive emotions, as a special kind of emotion, play an important role in human learning. Studies have found that learners can improve their learning efficiency and cognitive ability, stimulate students' learning interest, and improve teacher-student relationships under positive emotional experiences. Positive emotions can promote individual development potential and virtues, enhance individuals' positive emotional experiences, broaden cognitive scope, and improve cognitive flexibility. Therefore, it is of great significance to pay attention to students' positive emotional changes in the learning process, enhance students' positive emotional experiences in the learning process, and promote the learning process.

[0003] The research on the detection and analysis of students' positive emotions in the classroom teaching environment is currently in its infancy, and the combination of theory and practice is a necessary means to explore this field. First of all, the current intelligent research on learners' positive emotions focuses on the recognition and analysis of learners' facial emotions or body postures, and does not comprehensively consider the cognitive activities and thinking expansion activities in the learners' positive emotional process, which cannot fully reflect the students' positive emotional state; on the other hand, in the large-scale classroom teaching scene, there are many students and complex behaviors. There are still many technical difficulties in effectively analyzing the emotional state of students under positive emotions through existing computer vision and pattern recognition technologies. Therefore, how to construct a theoretical model that can describe students' positive emotions in the teaching environment from multiple dimensions.

[0004] Improving the positive emotions of classroom learners and stimulating their vitality are issues that educational researchers are widely concerned about. With the accelerated integration, innovation and fusion of technologies such as artificial intelligence, big data, mobile communications, virtual reality and the Internet of Things with other technologies, students' learning behaviors, body language, learning emotional states and other classroom process situational information can be captured by intelligent monitoring devices, learning devices, mobile devices, environmental sensing devices, etc. Therefore, it is possible to assist teachers in timely understanding the positive emotional changes of students in the classroom through computers and other intelligent devices.

[0005] At present, the recognition of students' positive emotional state is mainly based on physiological information recognition, image and video recognition, and voice feature recognition. Although the above different methods can realize the detection of learners' positive emotional state to a certain extent, these methods still have the following shortcomings:

[0006] (1) Lack of non-invasive and natural detection methods in large-scale classroom teaching scenarios

[0007] The method based on human physiological signal analysis is to extract learners' physiological signals through wearable devices such as smart bracelets and smart helmets, and extract information related to learners' emotions through signal processing, feature analysis and other methods to monitor learners' emotional states. Although these methods can predict learners' emotional states to a certain extent, they require learners to wear or carry invasive signal acquisition devices, which is not conducive to learners' natural interaction and is not suitable for emotional detection of a large number of learners in large classroom teaching scenarios.

[0008] The method based on speech recognition is to extract acoustic features that can express emotions in speech signals and find the mapping relationship between these acoustic features and human emotions. Ultimately, the speaker's emotional state can be estimated through speech recognition. However, due to technical limitations, the current speech recognition rate is generally low. Secondly, due to the shortcomings of the speech emotion database itself, the annotation results are not accurate enough, and the generalization performance of the speech model is insufficient. Therefore, the speech recognition method does not have good application scalability.

[0009] (2) Psychologists’ six basic emotions are not applicable to classroom teaching

[0010] Research based on image and video analysis tends to identify six basic emotions, namely happiness, sadness, surprise, fear, anger and disgust, but lacks recognition of positive emotions. Secondly, the traditional six basic emotions do not often appear in the classroom, or even almost never appear, so they are not suitable for classroom teaching. Secondly, this expression recognition method does not take into account the intensity of students' emotional experience, that is, different students have different emotional reactions under the same positive emotional experience.

[0011] (3) Positive emotions cannot be estimated only through general emotion recognition

[0012] Different from general emotions, positive emotions refer to positive emotions or emotions with positive valence. In an information-based classroom where students actively participate and are guided by positive emotions, in addition to listening, frustration, confusion and other emotions, students are more likely to have cognitive involvement and thinking expansion under the guidance of positive emotions, which is positive. Therefore, positive emotions are essentially different from general emotion recognition, and positive emotions cannot be estimated only through emotion recognition.

[0013] (4) The one-sidedness of the single-dimensional emotional engagement level test

[0014] Although positive emotions are a kind of positive emotions, they are pleasant emotions generated when events meet individual needs. They include basic emotions such as happiness, interest, satisfaction, and love. However, Fredrickson clearly stated in the expand-build theory of positive emotions that positive emotions not only include pleasant emotional experiences, but also expand the attention and cognitive scope of individuals. Existing studies have not only found that positive emotions expand the spatial and temporal attention scope and increase attention flexibility, but also found that individuals show attention bias to positive stimuli under positive emotions. At the same time, the expand-build theory of positive emotions believes that positive emotions can expand the instantaneous thinking-action scope of individuals. Therefore, emotion recognition through a single dimension such as facial expressions, body postures, or physiological recognition cannot fully reflect students' positive emotions and emotional experiences in the learning process. Summary of the invention

[0015] In order to overcome the problem that existing classroom teachers often focus on classroom teaching and guidance, but are unable to accurately grasp the positive emotional experience of learners in the entire classroom in real time and are unable to understand the students' positive emotional participation, the main purpose of the present invention is to provide an emotional participation detection method for positive emotion perception.

[0016] To achieve the above object, the present invention adopts the following technical scheme, a method for identifying emotional engagement of positive emotion perception in a classroom, comprising the following steps:

[0017] Obtain the head posture direction, smile intensity and access records of students in the classroom;

[0018] The attention mechanism in fine-grained image recognition is introduced into the deep learning network, a smile intensity estimation model is constructed, and the smile intensity of the students in the classroom is used as the input of the smile intensity estimation model to obtain the emotion intensity;

[0019] A multi-task head posture estimation model is constructed using the FSANet network, and the head posture directions of the students in the classroom are used as inputs of the multi-task head posture estimation model to obtain cognitive attention;

[0020] Screen the access records of students in the classroom learning system to obtain thinking activities;

[0021] The cognitive attention, the emotion intensity and the thinking activity are based on the positive emotion engagement model to construct a positive emotion engagement recognition model for classroom students on decision fusion;

[0022] as well as,

[0023] Obtain classroom video images and teaching resource dual-end system data, regard the head posture direction and smile intensity of classroom students in classroom video images and the access records of the learning system in the teaching resource dual-end system data as mutually compensating information of different modalities as input, and use deep network modality high-level semantic learning and multi-modal feature fusion to obtain homogeneous fusion expression;

[0024] Different objective functions are formulated for learning based on the acquired head posture direction, smile intensity and access records of the learning system of the classroom students. Through homogeneous fusion expression, a positive emotional engagement recognition model of classroom students based on multimodal fusion is constructed.

[0025] Using the decision-fusion classroom students' positive emotion engagement recognition model and the multimodal learning fusion classroom students' positive emotion engagement recognition model to jointly construct a classroom students' positive emotion engagement recognition model;

[0026] The smile intensity, the head posture, and the cognitive attention are identified by the positive emotion engagement recognition model of the classroom students to confirm the emotional engagement of the classroom positive emotion perception.

[0027] The smile intensity estimation model suppresses invalid information learned by the convolution layer through the attention mechanism in fine-grained image recognition, and enhances the network's learning of key area features. The key area is the area where the facial muscles that produce smiley expressions are located.

[0028] , the characteristics of the area where the facial muscles that produce the smiling expression are located include the coordinate changes of the feature points of the corners of the mouth and the coordinate changes of the feature points of the corners of the eyes during the smiling face movement;

[0029] The deep networks in the smile intensity estimation model include two types. One deep network VGGNet has pre-trained weights and uses feature weights as the starting point for learning to make up for the shortcoming of the small number of smile data sets and capture detailed information of smiles. The other deep network ResNet does not load pre-trained weights, learns image features from scratch, and focuses on training smile intensity information.

[0030] The smile intensity estimation model introduces focal loss to improve the cross entropy loss function.

[0031] In the homogeneous fusion expression, feature learning is performed through an asymmetric dual-stream branch, and the asymmetric dual-stream branch includes branch one and branch two. Branch one uses a deep cascaded autoencoder to learn feature mapping of the original data through a network structure to achieve automatic high-level feature learning. Branch two extracts low-level detail information of each information modality;

[0032] The high-level features of each generated information modality are connected to its low-level features through skip-layer connections, and the attention-guided feature cross-fusion module is used to generate shared sparse fusion features. Then, the features of multiple modalities are associated together in a complex nonlinear way, and the enhanced feature representation after fusion is used to obtain a homogeneous fusion expression.

[0033] Compared with the prior art, the beneficial effects of the present invention are:

[0034] 1. The present invention can obtain real-time and accurate grasp of the positive emotional experience of learners in the entire classroom based on the students' smile intensity, head posture estimation, and recognition of thinking activities, and understand the students' positive emotional participation, and enrich the research on students' positive emotional participation.

[0035] 2. The present invention constructs a positive emotional participation recognition model for classroom students in two ways. One is to extract features from data of different dimensions in students' classroom learning, realize information recognition of each dimension, and then perform decision fusion and result application analysis. The other is to extract features from data of various dimensions in students' classroom learning, and then adopt a fusion strategy based on a deep learning network, which is applied to subsequent supervised classification to construct a model of students' positive emotional participation. Both methods are to study and discover the positive emotional mechanism hidden behind classroom data to make correct predictions or judgments on future lesson plan design, emotional tracking, and classroom personalized assistance, improve classroom teaching effectiveness, and improve the accuracy of classroom participation prediction. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 is a schematic diagram of the three-dimensional positive emotional engagement model in the present invention;

[0037] Figure 2 It is a schematic diagram of the key technology research in the present invention;

[0038] Figure 3 It is a smile intensity estimation model based on dual attention mechanism and dual network in the present invention;

[0039] Figure 4 It is a multi-task head posture estimation model based on heterogeneous dual streams in the present invention;

[0040] Figure 5 It is a schematic diagram of multimodal information decision fusion based on the positive emotion engagement model in the present invention;

[0041] Figure 6 It is a schematic diagram of multimodal information fusion based on deep network in the present invention. DETAILED DESCRIPTION

[0042] The present invention will be further described below in conjunction with the accompanying drawings and implementation modes.

[0043] Example:

[0044] See also Figure 1-5 The existing intelligent research on students' positive emotional participation in class does not comprehensively consider students' cognitive activities and thinking-action activities under positive emotional states, and cannot fully reflect students' positive emotional participation. This application proposes a three-dimensional positive emotional participation model that includes cognition, emotion and thinking, such as Figure 1 As shown, cognition reflects learners' important motivation and cognitive attention orientation, and emotion reflects the intensity of positive emotional experience during the learning process. Thinking reflects the expansion of thinking-action activities during the learning process under the guidance of positive emotions.

[0045] Regarding cognitive attention, existing studies have found that positive emotions expand the spatial and temporal attention range, increase attention flexibility, and show that individuals show attention bias towards positive stimuli under positive emotions. The direction of students' head posture in the classroom can reflect the direction of students' attention. Studies have shown that the contribution of human horizontal head posture to the direction of attention is 40.3%, while the contribution of vertical head posture to the direction of attention is 28.4%, and the direction of the eyeballs accounts for 31.1%. Therefore, students' attention bias can be estimated by detecting the direction of students' head posture.

[0046] Regarding the intensity of emotions, positive emotions can be defined as the feelings generated when making progress in the process of goal achievement or receiving positive comments from others. From the perspective of discrete emotion theory, positive emotions include joy, happy, contentment, interest, pride, gratitude and love, etc. The specific facial expression produced with this feeling is smile. Among various facial expressions, the smiley face expression is the most widely understood positive expression. Nakamura et al. from Kyoto University in Japan found in an interactive recognition experiment of peer mutual assistance agent dialogue that the frequency of smiles occurred most frequently when learners were experiencing positive emotions. Therefore, this application uses the intensity of student smiles to estimate the emotional experience of students in a positive emotional state in the classroom.

[0047] Regarding the thinking activities: In this embodiment, the teaching resource dual-end system - Super Star Cloud Platform is introduced into classroom teaching, and the content of students' thinking-action activities is recorded through the system platform. Since the expansion-construction theory of positive emotions believes that positive emotions can expand the individual's instantaneous thinking-action range and build personal resources, including intellectual resources (such as problem-solving skills, new knowledge acquired), psychological resources (such as resilience, optimism, identity, goal orientation, etc.), etc., thus bringing more benefits to individuals. A large number of experimental studies have shown that positive emotions can make individuals' thinking patterns unusual, flexible, inclusive, creative, integrated, open, efficient, forward-looking and high-level. Positive emotions expand individuals' action tendencies by increasing their tendency to seek diversity and being open to more behavioral options. Therefore, thinking activities and learning activities guided by positive emotions, as well as the exploration, access, clicks, comments, and other related information on students' thinking-action, etc., can be recorded on the system platform.

[0048] This application has extensively referred to the latest research progress at home and abroad, combined with the research results of the application in the field of computer vision, based on existing technical means, among which, according to the information fusion process, it needs to include three steps: information extraction; information fusion; fusion application and analysis. The overall research plan of this application is divided into two parts based on the information fusion method. One part is to extract features from data of different dimensions in students' classroom learning, realize information identification of each dimension, and then analyze the decision fusion and result application. The other part is to extract features from data of various dimensions in students' classroom learning, and then adopt a fusion strategy based on deep learning network, which is applied to subsequent supervised classification to construct a student positive emotional participation model. Both methods are to study and discover the positive emotional mechanism hidden behind the classroom data, so as to make correct predictions or judgments on future lesson plan design, emotional tracking and classroom personalized assistance, and improve classroom teaching effectiveness. Specific key technology research includes Figure 2 shown.

[0049] A smile intensity estimation model based on dual network dual attention mechanism is constructed. Figure 3 As shown in the figure, if only smile recognition is performed, the emotional information obtained is very limited, and a more detailed classification of smiles is required. Different from smile recognition, the difference between the classes of smile intensity samples is small, the recognition is more difficult, and there is no dedicated public dataset for smile intensity.

[0050] For the task of estimating the intensity of smiles, combined with the characteristics of small differences between classes and fuzzy boundaries between subclasses, the attention mechanism in fine-grained image recognition is introduced to suppress the invalid information learned by the convolution layer and enhance the network's learning of key regional features, especially the feature learning of the facial muscles that produce smiley expressions, so as to approach the research goal. The specific steps include the following steps

[0051] (1) In previous research, it was found that the coordinate changes of the feature points at the corners of the mouth and eyes are the most obvious in the movement of smiling faces. Therefore, adding several geometric features guides the network to learn favorable feature areas, that is, suppressing useless information and enhancing the learning of useful information.

[0052] (2) Smile intensity estimation model Different deep networks are used in the model, and features are learned separately. The difference is that the VGGNet of deep network 1 has pre-trained weights, and uses rich feature weights as the starting point for learning, which makes up for the shortcoming of the small number of smile data sets, helps to suppress overfitting, and can capture the detailed information of smiles. The ResNet of deep network 2 does not load pre-trained weights, learns image features from scratch, and focuses on training smile intensity information. Then, by fusing the features of the two, the stability of the network model can be enhanced, avoiding the learned attention trend from being too extreme. At the same time, the transfer learning weights called can alleviate the impact of the small number of training sets on the model.

[0053] (3) Different from the smile recognition task, the difference between different smile samples in the smile intensity estimation model is very small, and there are many samples that are difficult to distinguish. Secondly, due to the imbalance of the categories of training samples, the number of intermediate smile intensity images is small. During network training, due to the imbalance of the number of samples, it is difficult to learn the effective feature information of small samples. Therefore, the study introduces focal loss to improve the cross entropy loss function.

[0054] (4) In order to solve the problem of insufficient smile intensity annotation samples and incomplete annotation information, and to further study the fine-grained image recognition problem of smile recognition, the smile intensity estimation problem is solved. For the expression video image sequence of the public dataset, referring to the intensity annotation information of AU, it is proposed to annotate based on the relative position of the key frames of the video sequence.

[0055] Based on FSANet, a multi-task head pose estimation model based on heterogeneous dual streams is proposed. Figure 4 As shown. Specifically including the following steps:

[0056] (1) Split the yaw angle, pitch angle, and roll angle that represent the head posture into three subtasks and learn the characteristics of each angle separately;

[0057] (2) In the process of head posture feature extraction, for the input head posture dataset, the FSANet network is used to learn the original data through the network structure to generate different feature maps, thereby realizing spatial feature and fine-grained feature extraction.

[0058] (3) Different from the general fine-grained feature extraction, this study introduces the attention mechanism in fine-grained image recognition to locate and weight the features of key areas of facial posture, guiding the subsequent extraction of fine-grained features.

[0059] (4) Fusion and further aggregation. Then formulate different objective functions for learning respectively, so that the fused expression can be further used in subsequent learning tasks to build a prediction model.

[0060] Based on the fusion of multimodal estimation models guided by the positive emotional engagement model, the Super Nebula platform extracts information related to students' exploration, clicks, comments, etc. of teaching resources and students' thinking-action expansion to estimate students' thinking activities; the smile intensity is estimated to extract students' classroom learning emotions; and the head posture is estimated to obtain students' cognitive attention. Based on the results identified by the above methods, the system model diagram for decision fusion is based on the positive emotional engagement model, as shown in the figure below: Figure 5 shown.

[0061] The multimodal learning fusion research based on deep networks regards unstructured classroom video images and teaching resource dual-end system data as mutually compensating information of different modalities, and uses deep network modality high-level semantic learning and multimodal feature fusion to provide a more comprehensive description of students' positive emotional characteristics by mining the correlation between multimodalities. The specific steps include:

[0062] Propose an asymmetric hybrid autoencoder deep network model:

[0063] (1) Using asymmetric dual-stream branches, for each modality, branch 1 uses a deep cascade autoencoder to learn feature mapping of the original data through the network structure to achieve automatic high-level feature learning. Branch 2 extracts low-level detailed information of each modality.

[0064] (2) The high-level features of each modality are skip-linked with the low-level features and the attention-guided feature cross-fusion module is used to generate shared sparse fusion features. The features of multiple modalities are then associated together in a complex nonlinear way to better utilize the enhanced feature representation after fusion and improve the generalization ability of the model.

[0065] After obtaining the homogeneous fusion expression, different objective functions are formulated for learning according to different data analysis problems, so that the homogeneous fusion expression can be further used for subsequent learning tasks and construct prediction models such as Figure 6 shown.

[0066] Finally, the entire network can be learned by formulating different objective functions. After effective optimization and solution, positive emotion classification can be achieved in the output layer, providing a scientific basis for the exploration of the potential mechanism of positive emotions.

[0067] It should be noted that, in the present invention, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprises" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0068] The above embodiments are merely examples of the present invention and do not limit the protection scope of the present invention. All designs that are the same or similar to the present invention fall within the protection scope of the present invention.

Claims

1. A method for identifying emotional engagement in classroom positive emotion perception, characterized in that: The following steps are involved: Obtain the head posture direction, smile intensity and access records of students in the classroom; The attention mechanism in fine-grained image recognition is introduced into the deep learning network, a smile intensity estimation model is constructed, and the smile intensity of the students in the classroom is used as the input of the smile intensity estimation model to obtain the emotion intensity; A multi-task head posture estimation model is constructed using the FSANet network, and the head posture directions of the students in the classroom are used as inputs of the multi-task head posture estimation model to obtain cognitive attention; Screen the access records of students in the classroom learning system to obtain thinking activities; The cognitive attention, the emotional intensity and the thinking activity are based on the positive emotional engagement model to construct a positive emotional engagement recognition model for classroom students on decision fusion; as well as, Obtain classroom video images and teaching resource dual-end system data, regard the head posture direction and smile intensity of classroom students in classroom video images and the access records of the learning system in the teaching resource dual-end system data as mutually compensating information of different modalities as input, and use deep network modality high-level semantic learning and multi-modal feature fusion to obtain homogeneous fusion expression; Different objective functions are formulated for learning based on the acquired head posture direction, smile intensity and access records of the learning system of the classroom students. Through homogeneous fusion expression, a positive emotional engagement recognition model of classroom students based on multimodal fusion is constructed. Using the decision-fusion classroom students' positive emotion engagement recognition model and the multimodal learning fusion classroom students' positive emotion engagement recognition model to jointly construct a classroom students' positive emotion engagement recognition model; The smile intensity, the head posture, and the cognitive attention are identified by the positive emotion engagement recognition model of the classroom students to confirm the emotional engagement of the classroom positive emotion perception; The smile intensity estimation model suppresses invalid information learned by the convolution layer through the attention mechanism in fine-grained image recognition, and enhances the network's learning of key area features, where the facial muscles that produce smiley expressions are located; The characteristics of the area where the facial muscles that produce the smiling expression are located include the coordinate changes of the feature points of the corners of the mouth and the coordinate changes of the feature points of the corners of the eyes during the smiling face movement; The deep networks in the smile intensity estimation model include two types, one of which is a VGGNet with pre-trained weights, which uses feature weights as the starting point for learning, to make up for the shortcoming of a small number of smile data sets and capture detailed information about smiles, and the other is a ResNet with no pre-trained weights, which learns image features from scratch and focuses on training smile intensity information; Focal loss is introduced into the deep network of the smile intensity estimation model to improve the cross entropy loss function.

2. The method for detecting emotional engagement for positive emotion perception according to claim 1, characterized in that: In the homogeneous fusion expression, feature learning is performed through an asymmetric dual-stream branch, and the asymmetric dual-stream branch includes a branch one and a branch two. The branch one uses a deep cascade autoencoder to learn feature mapping of the original data through a network structure to achieve automatic high-level feature learning, and the branch two extracts low-level detail information of each information modality; The high-level features of each generated information modality are connected to its low-level features through skip-layer connections, and the attention-guided feature cross-fusion module is used to generate shared sparse fusion features. Then, the features of multiple modalities are associated together in a complex nonlinear way, and the enhanced feature representation after fusion is used to obtain a homogeneous fusion expression.

Images