Facial expression and context-based academic emotion recognition method and device

Abstract

The invention discloses an academic emotion recognition method and device based on facial expressions and contexts. According to the method, accurate emotion modeling is realized by utilizing a visual language pre-training model CLIP and combining student facial expressions in a video sequence and contextual information of a learning situation. Specifically, according to the method, firstly, collaborative modeling is carried out on facial expressions and context information in a learning scene, so that the ability of recognizing the learning state (such as focusing on learning and distracting) of students is remarkably improved; and secondly, the strong language and visual feature alignment capability of the CLIP pre-training model is utilized, so that the dependence on large-scale video training data is effectively reduced, and the generalization capability and practicability of the model are improved. An accurate and feasible solution is provided for academic emotion recognition, and the method is suitable for monitoring and analysis of academic emotions of students in an education scene.

Description

Technical Field

[0001] The present invention belongs to the field of student academic emotion recognition, and in particular relates to an academic emotion recognition method and device based on facial expressions and contextual situations. Background Art

[0002] Academic emotions are students' emotional states during learning or academic-related activities, which play a vital role in the learning process as they are directly related to factors such as motivation, all of which greatly affect students' learning efficiency and learning outcomes. Therefore, accurately identifying learners' emotional states helps to analyze their level of engagement and understand their learning process, so as to take appropriate interventions and promote positive learning outcomes. This not only leads to better academic outcomes, but also creates a more supportive and adaptive learning environment that meets the personalized needs of students. Although academic emotions are learners' intrinsic psychological reactions, experienced teachers are good at identifying these emotional states by observing facial expressions and other situational clues, and adjusting teaching strategies accordingly. This makes it natural to develop methods that can automatically identify students' academic emotions or engagement levels from their facial expressions. Current academic emotion recognition mainly uses video-based dynamic expression recognition to identify students' academic emotion categories, lacking the modeling of contextual information that is crucial to academic emotions.

[0003] Facial expression recognition is divided into static facial expression recognition and dynamic facial expression recognition. Static facial expression recognition uses a single frame of a face image to identify the emotion in the image. Dynamic facial expression recognition (DFER) is different from static facial expression recognition (SFER) because DFER not only needs to extract spatial features from each frame of the image, but also needs to consider the changes in temporal information. To this end, many methods use convolutional neural networks (CNNs) to extract spatial features from single frames of images, and then use recurrent neural networks (RNNs) to capture dynamic relationships in time. In order to better jointly learn spatial and temporal features, 3D convolutional neural networks (3DCNNs) have been proposed to model the spatiotemporal characteristics in 3D data. In recent years, Transformer-based networks have received increasing attention in the extraction of spatial and temporal information. Transformer-based networks can capture more complex temporal and spatial relationships through a global attention mechanism.

[0004] Former-DFER proposed a method for dynamic facial expression recognition based on Transformer. The network processes spatial and temporal information respectively through convolutional spatial Transformer (CS-Former) and temporal Transformer (T-Former). CS-Former consists of five convolutional blocks and N spatial encoders. The input part divides the video sequence into U segments and extracts V frames from the middle of each segment, forming a total of U × V frames as the input of the model. The input frame first passes through four convolutional blocks to extract local feature maps, and then pulls the features on the channel dimension into a sequence form, and then adds position embedding to the SpatialEncoder encoder module. Spatial Encoder extracts global features through the self-attention mechanism. The role of the four convolutional blocks is to extract local features. The additional convolutional blocks connected subsequently further refine the features to generate a more robust spatial representation. After each frame is processed by CS-Former, a feature representation is generated, and the entire group of frames forms a sequence. In order to perform the final classification, the model adds a Class Token to the sequence, as shown in the figure. Then, by adding TemporalPositionalEmbedding to represent the time sequence, and using the time encoder to capture the dynamic relationship between frames, the expression classification is finally completed. However, Former-DFER is designed for expression recognition, and only models the face information without considering the context information. In addition, the model is based on Transformer. The learning of deep networks often requires a large amount of data. The cost of collecting data collected from the real world, such as academic emotions, is relatively expensive, so the amount of data is often small.

[0005] DFER-CLIP is a method that uses the visual language pre-training (V-LP) model CLIP to complete the dynamic expression recognition task. CLIP can obtain rich visual representations by learning the semantic relationship between images and text, which provides potential advantages for the recognition of dynamic facial expressions. Specifically, DFER-CLIP combines dynamic facial features and text descriptions related to expressions in order to achieve higher accuracy in facial expression recognition. As a well-known visual language pre-training model, CLIP (Contrastive Language-Image Pre-training) is trained by contrastive learning, enabling the model to learn the correspondence between images and text. DFER-CLIP makes full use of this feature of CLIP and shows good potential in the understanding and recognition of dynamic facial expressions by integrating its powerful visual and language modeling capabilities.

[0006] Using pre-trained models requires less data. However, similar to Former-DFER, DFER-CLIP is mainly designed for basic expression recognition tasks and does not model contextual information, which may limit its applicability in some scenarios. Summary of the invention

[0007] The present invention aims to solve some problems existing in the existing facial expression recognition models in academic expression recognition. For example, some deep learning models require a large amount of labeled data, which are not suitable for smaller academic expression data sets, and existing facial expression recognition models often only model facial expressions, but not the contextual information that is crucial for academic emotion recognition. To this end, an academic emotion recognition algorithm based on facial expressions and contextual situations is proposed. The algorithm can significantly reduce the demand for a large amount of labeled data by existing algorithms, while modeling important contextual information and facial expressions in learning scenarios, and using contextual information to help identify students' overall learning status (such as the ability to identify whether students are studying hard or absent-minded), thereby improving the accuracy of academic emotion recognition.

[0008] The objective of the present invention is achieved through the following technical solution: a method for identifying academic emotions based on facial expressions and contextual situations, the method comprising the following steps:

[0009] S1. Segment the face region from the video frame to generate the facial expression frame. The original video frame is used as the context information frame to construct the emotion category text description containing the facial expression and context, and obtain the academic emotion video dataset.

[0010] S2. Construct an academic emotion recognition network based on facial expressions and contextual situations. The network first extracts spatial features from the facial expression frames and contextual information frames in the input video, and generates a unified visual feature token by fusing the spatial features with the temporal information. The network uses the text description to input the text encoder and extracts the text feature tokens of the corresponding emotion categories. Finally, by calculating the cosine similarity between the visual feature token and the text feature token of each emotion category and normalizing the similarity, the probability distribution of all emotion categories is generated, and finally the classification of academic emotions is completed;

[0011] S3. Constrain and optimize the academic emotion recognition network based on facial expression and context through the classification loss function and the academic emotion video dataset, complete the network training, and generate the final academic emotion recognition model based on facial expression and context;

[0012] S4. Based on the trained academic emotion recognition model based on facial expressions and contextual situations, the input video clips are analyzed to identify the academic emotion categories of students.

[0013] Furthermore, the academic emotion video dataset includes different learning environments and learning links, wherein the academic emotion categories include: neutral, pleasant, confused, tired and distracted; wherein, neutral, pleasant, confused and tired are specific emotion categories in the learning state, and distraction covers all emotions in non-learning states.

[0014] Furthermore, the academic emotion recognition network based on facial expression and contextual situation includes: a CLIP visual encoder, a temporal relationship learning module, a visual feature fusion module, a CLIP text encoder and a feature alignment module;

[0015] The CLIP visual encoder comprises a 2D convolutional block and 12 Transformer layers, and inputs a sequence of facial expression images and a context information image; obtains facial features and context features;

[0016] The temporal relationship learning module includes two temporal learning units with the same structure, which are used to independently perform temporal modeling on facial features and context features, respectively, and are recorded as a facial temporal modeling module and a context temporal modeling module. Each temporal learning unit is composed of a Transformer layer, and facial features and context features that are integrated with temporal information are obtained respectively.

[0017] The visual feature fusion module includes a fully connected layer; the facial features fused with time information are spliced ​​with context features, the feature dimensions are adjusted and aligned to obtain visual feature tokens;

[0018] The CLIP text encoder contains 12 Transformer layers, which input the text description of each category to obtain the text feature token of each category;

[0019] The feature alignment module achieves cross-modal spatial alignment by calculating the cosine similarity between the visual feature token and the text feature token of each category. The video academic emotion category corresponding to the text feature with the highest cosine similarity is the final recognition result.

[0020] Furthermore, the CLIP visual encoder includes the following specific processing steps:

[0021] For the input facial expression image sequence and context information image sequence, single-frame feature extraction is performed through the CLIP visual encoder with shared weights to obtain the corresponding visual feature representation;

[0022] The single-frame feature extraction process includes:

[0023] First, the image is divided into blocks through 2D convolution and embedded to obtain the feature token of the block. Then a learnable category token is generated with the same dimension as the block image and initialized as a random vector to represent the global information of the entire image. The category token and all the block feature tokens are input into the Transformer layer together. After interactive calculations of multi-layer attention mechanisms, global and local information are fused, and finally the CLIP visual encoder outputs the updated category token.

[0024] Furthermore, the specific processing flow of the time relationship learning module includes:

[0025] For the facial expression feature token sequence extracted by the CLIP visual encoder, a learnable category token is first introduced. The category token and the facial expression feature token sequence are input into the face temporal relationship learning module. The module captures the dynamic relationship between frames and the global temporal information through temporal modeling. In this process, the weight of the category token is updated through training, and finally the category token representation containing temporal information is obtained, which is used to characterize the facial expression information with time-dependent characteristics.

[0026] For the context information feature token sequence, a randomly initialized learnable category token is introduced and input into the context temporal relationship learning module together with the context feature token sequence. The module learns the inter-frame dynamics and temporal dependencies of context features through temporal modeling and updates the weights of the category tokens through training. Finally, the updated category tokens are used to represent the context features with temporal information.

[0027] The learnable category token is a learnable vector of the same length as the CLIP visual encoder output.

[0028] Furthermore, the CLIP text encoder includes the following specific processing steps:

[0029] First, generate M randomly initialized learnable text prompts to learn additional details not covered in the text description. Then, concatenate these learnable prompt words with the text description after word segmentation to form a new input sequence, and add a start tag and an end tag before and after the concatenated sequence respectively.

[0030] Next, the concatenated sequences are length aligned to ensure that the number of tokens in all sentences is consistent, and the concatenated sequences are input into the Transformer model to capture global temporal dependencies. In the output of the Transformer, the output feature token corresponding to the end tag is used as the text representation of the entire sentence, which will be aligned with the visual feature token to achieve cross-modal alignment training;

[0031] During the training process, the pre-trained weights of the CLIP model are loaded first, and the weights of the CLIP text encoder are frozen so that they are not updated. The weights of the rest are then updated.

[0032] Furthermore, the feature alignment module has a specific processing flow including:

[0033] First compute the visual feature tokens Text feature token with category k The cosine similarity between ,in, is a visual feature token, is the text feature token of the kth type of academic sentiment, Represents the cosine similarity calculation operation. Then, the cosine similarities of all categories are normalized by the softmax function to obtain the probability distribution of each category:

[0034]

[0035] Among them, K represents the total number of academic emotion categories, k represents a specific emotion category, It indicates the probability that the predicted emotion category y is equal to category k when given an input video S. is the text feature token of category j.

[0036] Furthermore, the classification loss function L loss for:

[0037]

[0038] Where N represents the total number of samples, K represents the total number of academic emotion categories, is a category indicator variable, which takes the value of 1 when the true category of sample i is equal to k, otherwise it takes the value of 0. Indicates that given an input video , the probability that the predicted emotion category y is equal to k.

[0039] On the other hand, the present invention provides an academic emotion recognition device based on facial expressions and contextual situations, including a memory and one or more processors, wherein the memory stores executable code, and when the processor executes the executable code, it implements the academic emotion recognition method based on facial expressions and contextual situations.

[0040] On the other hand, the specification of the present invention provides a computer-readable storage medium having a program stored thereon, and when the program is executed by a processor, the method for academic emotion recognition based on facial expressions and contextual situations is implemented.

[0041] Beneficial effects of the present invention: The present invention first introduces a dual-branch architecture to collaboratively model facial expressions and contextual information of learning scenarios, fully explore and integrate students' facial expression features with key information in the learning environment, and strengthen the understanding of context-related factors. The attention mechanism is used to perform temporal modeling of facial expressions and contextual information, and a feature fusion module is designed to deeply integrate facial expression features with contextual information, thereby improving the model's understanding of the learning state. This method can not only accurately capture students' subtle emotional changes during the learning process, but also combine behavioral clues and environmental factors to improve the ability to accurately identify academic emotions. Compared with traditional methods that rely solely on facial expressions, the present invention effectively utilizes key contextual clues in academic emotion recognition (such as looking at mobile phones or learning materials, etc.), significantly enhances the model's ability to distinguish different learning states (such as focusing on learning or being absent-minded), and improves the robustness and generalization of academic emotion recognition.

[0042] Secondly, in terms of text feature modeling, the present invention designs a learnable text prompt mechanism. By introducing task-adaptive text embedding, the model can autonomously learn context-related emotional descriptions and supplement additional detailed information that fixed text prompts cannot cover. This method makes full use of CLIP's powerful feature extraction capabilities, reduces dependence on large-scale training data, and improves the stability and practicality of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 A flow chart of a method for identifying academic emotions based on facial expressions and contextual situations provided by an embodiment of the present invention;

[0044] Figure 2 A network structure diagram of a CLIP visual encoder provided by an embodiment of the present invention;

[0045] Figure 3 A network structure diagram of a time relationship learning module provided by an embodiment of the present invention;

[0046] Figure 4 A network structure diagram of a CLIP text encoder provided by an embodiment of the present invention;

[0047] Figure 5 A schematic diagram of an academic emotion recognition device based on facial expressions and contextual scenarios provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0048] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0049] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0050] like Figure 1 As shown, the present invention proposes a method for academic emotion recognition based on facial expressions and contextual situations, and the method specifically comprises the following steps:

[0051] Step 1: Construct the dataset required for training the academic emotion recognition model based on facial expressions and contextual situations;

[0052] The step 1 of constructing the data required for the academic emotion recognition model training includes the following steps:

[0053] This dataset is built based on the self-collected academic emotion video dataset RAER. The RAER dataset was collected in real learning scenarios, covering a variety of actual learning environments such as libraries, classrooms, laboratories, dormitories, and includes self-study scenarios and teaching sessions. The dataset contains rich contextual information, such as students' hand movements (such as using mobile phones, flipping through textbooks, or operating tablets), and all data is annotated with high quality to generate accurate data labels.

[0054] The label set of the RAER dataset contains five common academic emotion categories: neutral, pleasant, confused, tired, and distracted. Among them, neutral, pleasant, confused, and tired reflect specific emotion categories in the learning state, while distraction covers all emotions in the non-learning state. In order to simplify the annotation, distracting emotions are not further subdivided. The dataset is divided into a training set and a test set: the training set is used to train the model, and the test set is used to evaluate the model's recognition ability for unseen data. The construction of the RAER dataset provides important support for research and application in the field of academic emotion recognition. For the five academic emotion categories, ChatGpt is used to generate corresponding text descriptions, including corresponding facial expression features (such as "corners of the mouth raised") and contextual scene features (such as "operating the mobile phone").

[0055] Step 2: Construct an academic emotion recognition network based on facial expressions and context;

[0056] The academic emotion recognition network based on facial expression and context in step 2 includes:

[0057] The network is built based on the CLIP model and includes the following five modules: CLIP visual encoder (CLIP-IE), temporal relationship learning module (Temporal Encoder), visual feature fusion module (Feature Fusion Module, FFM), CLIP text encoder (CLIP-TE) and feature alignment module (Feature Alignment Module, FAM).

[0058] The CLIP visual encoder is used to extract the spatial features of a single frame from the input video frame sample and generate feature tokens for each frame for subsequent temporal feature modeling. The input video sample S consists of a sequence of N frames of facial expression images. , and N frames of context information image sequence Composition, of which The encoder adopts a VisionTransformer (ViT)-based structure, which includes a 2D convolutional block and 12 layers of Transformer modules. Each Transformer layer consists of a multi-head self-attention network and a feedforward neural network.

[0059] In the workflow of CLIP vision encoder, Figure 2 As shown, the input image is first divided into blocks of size 32×32 through a 2D convolution block. Each block is embedded as a feature token of length 512. For an image of size 224×224, a total of 49 patches are segmented, and 49 feature tokens are generated accordingly. In addition, the encoder generates a learnable category token (class token) with the same dimension as the block feature, which is used to represent the global information of the entire frame image. The category token is input into the Transformer module together with all the block feature tokens. After 12 layers of multi-head self-attention calculations to capture the global and local information interactions of the image, the final output of the category token combines the global and local features as the feature representation of the frame image. The CLIP visual encoder extracts features from the facial expression image sequence and the context image sequence respectively.

[0060]

[0061]

[0062] in represents the CLIP visual encoder, which finally outputs two sets of feature token sequences: one set is the facial expression feature token sequence, represented as ={ , ,.... }, and the other group is the context feature token sequence, expressed as ={ , ,.... }, used for subsequent timing information modeling.

[0063] The temporal relationship learning module is used to perform temporal feature modeling on the input video frame sequence to characterize the dynamic changes and global temporal information of the face time series and context time series. The module adopts the classic Transformer architecture and consists of two temporal learning units with the same structure, which are used to perform independent temporal modeling on face features and context features, respectively, and are recorded as the face time modeling module and the context time modeling module. Each temporal learning unit contains a layer of Transformer, including a multi-head self-attention network and a feedforward neural network.

[0064] In the processing flow of the temporal relationship learning module, Figure 3 As shown, for the facial expression feature token sequence extracted by the CLIP visual encoder First, we introduce a randomly initialized learnable class token, whose dimension is the same as the feature token output by the CLIP visual encoder, to represent the overall face video features with temporal information. The class token and the face feature tokens of all frames together constitute the input sequence ={ , , ,.... },in Represents the category token. The sequence is input into the face temporal modeling module, which captures the dynamic relationship between frames and global temporal dependencies through Transformer temporal modeling. In this process, the category token interacts with other feature tokens in the sequence through the self-attention mechanism, and its weight is continuously updated during the training process.

[0065]

[0066] Finally, the updated category token The temporal information of all frames is integrated as the temporal feature representation of the complete face video, where Represents the self-attention mechanism.

[0067] For context feature token sequence , taking a similar approach. The process is similar. A randomly initialized learnable category token is introduced and concatenated with the context feature token sequence as ={ , ,.... The sequence is input to the contextual temporal modeling module, which uses Transformer to learn the dynamic relationship and temporal dependency between context frames.

[0068]

[0069] Finally, the updated category token As a representation of contextual temporal features, it is used to characterize contextual features with temporal information, where Represents the self-attention mechanism.

[0070] The visual feature fusion module is mainly used to fuse facial features and context features, and includes a fully connected layer (Fully Connected Layer) to generate a unified overall visual feature representation to provide support for cross-modal alignment. The module includes a fully connected layer (Fully Connected Layer) to adjust the feature dimension and achieve alignment.

[0071] In the processing flow of the visual feature fusion module, the face feature tokens output by the temporal relationship learning module are first (length 512) and contextual feature tokens (length is 512) to form a new feature vector. The length of the concatenated feature token is 1024. Then, the feature token is input into the fully connected layer, and the feature is reduced in dimension through linear transformation to output an overall visual feature token with a length of 512. The overall visual feature vector after dimensionality reduction not only contains the temporal information of facial expression features, but also integrates the temporal dynamic relationship of contextual features, so as to more comprehensively represent the visual information of the input video. Finally, the overall visual feature vector is aligned with the text feature vector in the shared feature space for the cross-modal classification task of academic emotions.

[0072] The CLIP text encoder is used to extract independent text features for each category and align them with visual features to complete classification. The text encoder consists of 12 Transformer layers, each of which contains a self-attention network and a feedforward neural network. In addition, the encoder introduces learnable text prompts to capture detailed information not covered by the fixed text description; in the specific processing flow of the CLIP text encoder, Figure 4 As shown, for each category k, the text description First, generate M learnable text prompts, and then use them after tokenization by Tokenizer Concatenate to form category-independent input sequences:

[0073]

[0074] Where k represents the kth category, represents the input sequence of the kth category, M represents the number of learnable text prompts, Tokenizer represents the word segmentation operation, Represents the text description of the kth category. Add a start tag ([CLS]) and an end tag ([SEP]) before and after the concatenated sequence to make the start and end positions of the sequence clear. And input it into the CLIP text encoder model to capture global timing dependencies.

[0075]

[0076] In the output of the CLIP text encoder, the output feature token corresponding to the end tag ([SEP]) is used as the text representation of the entire sentence, which will be aligned with the visual feature token to achieve cross-modal alignment training. Finally, the feature token of each category text description is represented as , k represents the kth category.

[0077] During the training process, the pre-trained weights of the CLIP model are loaded first, and the weights of the CLIP text encoder are frozen so that they do not participate in the update. The weights of the rest are updated. This mechanism ensures that the model can focus on learning the potential detailed features of the text through the cue words based on the fixed pre-trained language model, thereby optimizing the alignment of text and visual features.

[0078] The feature alignment module is mainly used for cross-modal spatial alignment and finally completes the classification of academic emotions. The module calculates the cosine similarity between the visual features and the text features of all categories, and selects the category with the highest similarity as the final prediction result. In the specific processing flow of the feature alignment module, the visual features are first calculated. With each category k text features The cosine similarity between ,in Represents the cosine similarity calculation operation. Then, the cosine similarities of all categories are normalized through the softmax function to obtain the probability distribution of each category:

[0079]

[0080] Among them, K represents the total number of academic emotion categories, k represents a specific emotion category, It indicates the probability that the predicted emotion category y is equal to category k when given an input video S. is the text feature token of category j.

[0081] Step 3: Constrain and optimize the academic emotion recognition network based on facial expressions and contextual situations through the classification loss function, and train the final context-aware academic emotion recognition model;

[0082] In step 3, the academic emotion recognition network based on facial expressions and context is constrained and optimized by the classification loss function, and the training to obtain the final context-aware academic emotion recognition model includes the following steps:

[0083] After obtaining the probability that the video belongs to each sample after feature alignment, the model can be supervised by the real label. The specific formula of the loss used is as follows:

[0084]

[0085] is the most commonly used cross entropy loss function in classification tasks, where N represents the total number of samples, K represents the total number of academic emotion categories, is a category indicator variable, which takes the value of 1 when the true category of sample i is equal to k, otherwise it takes the value of 0. Indicates that given an input video , the probability that the predicted emotion category y is equal to k.

[0086] Step 4: Estimating the academic emotion of the input video clip based on the trained context-aware academic emotion recognition model;

[0087] Corresponding to the aforementioned embodiment of an academic emotion recognition method based on facial expressions and contextual situations, the present invention also provides an embodiment of an academic emotion recognition device based on facial expressions and contextual situations.

[0088] See also Figure 5 An embodiment of the present invention provides an academic emotion recognition device based on facial expressions and contextual situations, including a memory and one or more processors, wherein the memory stores executable codes, and when the processor executes the executable codes, it is used to implement an academic emotion recognition method based on facial expressions and contextual situations in the above-mentioned embodiment.

[0089] An embodiment of an academic emotion recognition device based on facial expressions and contextual situations provided by the present invention can be applied to any device with data processing capabilities, and the device with data processing capabilities can be a device or apparatus such as a computer. The device embodiment can be implemented through software, or through hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, it is formed by the processor of any device with data processing capabilities in which it is located reading the corresponding computer program instructions in the non-volatile memory into the internal memory for execution. From a hardware perspective, if Figure 5 As shown, it is a hardware structure diagram of a device for identifying academic emotions based on facial expressions and context provided by the present invention, in which any device with data processing capability is located, except Figure 5 In addition to the processor, memory, network interface, and non-volatile memory shown, any device with data processing capabilities in which the apparatus in the embodiments is located may also include other hardware, generally based on the actual functions of the device with data processing capabilities, which will not be described in detail.

[0090] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0091] For the device embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment. The device embodiment described above is only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of the present invention. Ordinary technicians in this field can understand and implement it without paying creative work.

[0092] An embodiment of the present invention also provides a computer-readable storage medium having a program stored thereon. When the program is executed by a processor, an academic emotion recognition method based on facial expressions and contextual situations in the above embodiment is implemented.

[0093] The computer-readable storage medium may be an internal storage unit of any device with data processing capability described in any of the aforementioned embodiments, such as a hard disk or a memory. The computer-readable storage medium may also be an external storage device of any device with data processing capability, such as a plug-in hard disk, a smart media card (SMC), an SD card, a flash card, etc. equipped on the device. Furthermore, the computer-readable storage medium may also include both an internal storage unit and an external storage device of any device with data processing capability. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capability, and may also be used to temporarily store data that has been output or is to be output.

[0094] The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements the academic emotion recognition method based on facial expressions and contextual situations.

[0095] Those skilled in the art will readily appreciate other embodiments of the present application after considering the description and practicing the contents disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The description and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the claims.

[0096] It should be understood that the above general description and the detailed description below are only exemplary and explanatory and cannot limit the present application. The present application is not limited to the precise structure described above and shown in the drawings, and various modifications and changes can be made without departing from the scope thereof. The scope of the present application is limited only by the attached claims.

Claims

1. A method for academic emotion recognition based on facial expressions and contextual situations, characterized in that: The method comprises the following steps: S1. Segment the face region from the video frame to generate the facial expression frame. The original video frame is used as the context information frame to construct the emotion category text description containing the facial expression and context, and obtain the academic emotion video dataset. S2. Construct an academic emotion recognition model based on facial expressions and contextual situations. The model first extracts spatial features from the facial expression frames and contextual information frames in the input video, and generates a unified visual feature token by fusing the spatial features with the temporal information. The network uses the text description to input the text encoder to extract the text feature tokens of the corresponding emotion category. Finally, by calculating the cosine similarity between the visual feature token and the text feature token of each emotion category and normalizing the similarity, the probability distribution of all emotion categories is generated, and finally the classification of academic emotions is completed. S3. Constrain and optimize the academic emotion recognition network based on facial expression and context through the classification loss function and the academic emotion video dataset, complete the network training, and generate the final academic emotion recognition model based on facial expression and context; S4. Based on the trained academic emotion recognition model based on facial expressions and contextual situations, the input video clips are analyzed to identify the academic emotion categories of students.

2. The method for identifying academic emotions based on facial expressions and contextual situations according to claim 1, characterized in that: The academic emotion video dataset includes different learning environments and learning links, wherein the academic emotion categories include: neutral, pleasant, confused, tired and distracted; wherein, neutral, pleasant, confused and tired are specific emotion categories in the learning state, and distraction covers all emotions in non-learning states.

3. The method for identifying academic emotions based on facial expressions and contextual situations according to claim 1, characterized in that: The academic emotion recognition network based on facial expression and contextual situation includes: CLIP visual encoder, temporal relationship learning module, visual feature fusion module, CLIP text encoder and feature alignment module; The CLIP visual encoder comprises a 2D convolutional block and 12 Transformer layers, and inputs a sequence of facial expression images and a context information image; obtains facial features and context features; The temporal relationship learning module includes two temporal learning units with the same structure, which are used to independently model the facial features and context features in time series. Each temporal learning unit consists of a Transformer layer, and obtains facial features and context features that integrate time series information. The visual feature fusion module includes a fully connected layer; the facial features fused with time information are spliced ​​with context features, the feature dimensions are adjusted and aligned to obtain visual feature tokens; The CLIP text encoder contains 12 Transformer layers, which input the text description of each category to obtain the text feature token of each category; The feature alignment module achieves cross-modal spatial alignment by calculating the cosine similarity between the visual feature token and the text feature token of each category. The video academic emotion category corresponding to the text feature with the highest cosine similarity is the final recognition result.

4. The method for identifying academic emotions based on facial expressions and contextual situations according to claim 3, characterized in that: The CLIP visual encoder, the specific processing flow includes: For the input facial expression image sequence and context information image sequence, single-frame feature extraction is performed through the CLIP visual encoder with shared weights to obtain the corresponding visual feature representation; The single-frame feature extraction process includes: First, the image is divided into blocks through 2D convolution and embedded to obtain the feature token of the block. Then a learnable category token is generated with the same dimension as the block image and initialized as a random vector to represent the global information of the entire image. The category token and all the block feature tokens are input into the Transformer layer together. After interactive calculations of multi-layer attention mechanisms, global and local information are fused, and finally the CLIP visual encoder outputs the updated category token.

5. The method for identifying academic emotions based on facial expressions and contextual situations according to claim 3, characterized in that: The specific processing flow of the temporal relationship learning module includes: For the facial expression feature token sequence extracted by the CLIP visual encoder, a learnable category token is first introduced. The category token and the facial expression feature token sequence are input into the face temporal relationship learning module. The module captures the dynamic relationship between frames and the global temporal information through temporal modeling. In this process, the weight of the category token is updated through training, and finally the category token representation containing temporal information is obtained, which is used to characterize the facial expression information with time-dependent characteristics. For the context information feature token sequence, a randomly initialized learnable category token is introduced and input into the context temporal relationship learning module together with the context feature token sequence. The module learns the inter-frame dynamics and temporal dependencies of context features through temporal modeling and updates the weights of the category tokens through training. Finally, the updated category tokens are used to represent the context features with temporal information. The learnable category token is a learnable vector of the same length as the CLIP visual encoder output.

6. The method for identifying academic emotions based on facial expressions and contextual situations according to claim 3, characterized in that: The CLIP text encoder includes the following specific processing steps: First, generate M randomly initialized learnable text prompts to learn additional details not covered in the text description. Then, concatenate these learnable prompt words with the text description after word segmentation to form a new input sequence, and add a start tag and an end tag before and after the concatenated sequence respectively. Next, the concatenated sequences are length aligned to ensure that the number of tokens in all sentences is consistent, and the concatenated sequences are input into the Transformer model to capture global temporal dependencies. In the output of the Transformer, the output feature token corresponding to the end tag is used as the text representation of the entire sentence, which will be aligned with the visual feature token to achieve cross-modal alignment training; During the training process, the pre-trained weights of the CLIP model are loaded first, and the weights of the CLIP text encoder are frozen so that it does not participate in the update, and the weights of the remaining parts are updated.

7. The method for identifying academic emotions based on facial expressions and contextual situations according to claim 3, characterized in that: The feature alignment module specifically includes the following processing steps: First, the cosine similarity between the visual feature token and the text feature token of the category is calculated. Then, the cosine similarity of all categories is normalized by the softmax function to obtain the probability that the predicted emotion category is equal to each text feature category when given an input video.

8. The method for identifying academic emotions based on facial expressions and contextual situations according to claim 1, characterized in that: The classification loss function L loss for: ； Where N represents the total number of samples, K represents the total number of academic emotion categories, is a category indicator variable, which takes the value of 1 when the true category of sample i is equal to k, otherwise it takes the value of 0. Indicates that given an input video , the probability that the predicted emotion category y is equal to k.

9. An academic emotion recognition device based on facial expression and context, comprising a memory and one or more processors, wherein the memory stores executable code, characterized in that: When the processor executes the executable code, it implements an academic emotion recognition method based on facial expressions and contextual situations as described in any one of claims 1-8.

10. A computer-readable storage medium having a program stored thereon, characterized in that: When the program is executed by a processor, an academic emotion recognition method based on facial expressions and contextual situations as described in any one of claims 1-8 is implemented.

Images