Method and apparatus for recognizing academic emotion based on facial expression and context situation

Abstract

The application discloses a learning emotion recognition method and device based on facial expressions and context situations. The method uses a visual language pre-training model CLIP to combine the facial expressions of students in video sequences with the context information of learning situations to achieve accurate emotion modeling. Specifically, the application first co-models the facial expressions and the context information in learning situations to significantly improve the ability to recognize the learning states of students (such as focused learning and distraction). Secondly, by using the powerful language and visual feature alignment capability of the CLIP pre-training model, the dependence on large-scale video training data is effectively reduced, and the generalization ability and practicability of the model are improved. The application provides a precise and feasible solution for learning emotion recognition, which is suitable for monitoring and analyzing the learning emotions of students in educational scenarios.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of student academic emotion recognition, and particularly relates to a method and device for recognizing academic emotion based on facial expression and context. BACKGROUND

[0002] Academic emotion is the emotional state of students during learning or academic-related activities, which plays a crucial role in the learning process as they are directly related to factors such as motivation, all of which greatly affect the learning efficiency and effectiveness of students. Therefore, accurately recognizing the emotional state of learners helps to analyze their level of engagement, understand their learning process, and thus take appropriate intervention measures to promote positive learning outcomes. This not only brings better academic results, but also creates a more supportive and adaptive learning environment that meets the individual needs of students. Although academic emotion is the internal psychological response of learners, experienced teachers are good at recognizing these emotional states by observing facial expressions and other contextual cues, and adjusting teaching strategies accordingly. This makes it natural to develop methods that can automatically recognize the academic emotions or engagement levels of students from their facial expressions. Current academic emotion recognition mainly uses video-based dynamic expression recognition methods to recognize the academic emotion categories of students, lacking modeling of contextual information that is crucial to academic emotion.

[0003] Expression recognition is divided into static expression recognition and dynamic expression recognition. Static expression recognition uses single-frame face images to recognize emotions in images. Dynamic facial expression recognition (DFER) is different from static facial expression recognition (SFER) because DFER not only needs to extract spatial features from each frame of image, but also needs to consider the changes in temporal information. For this reason, many methods have adopted convolutional neural networks (CNN) to extract spatial features from single-frame images, and then use recurrent neural networks (RNN) to capture dynamic relationships over time. In order to better jointly learn spatial and temporal features, 3D convolutional neural networks (3DCNN) have been proposed to model the spatio-temporal characteristics in 3D data. In recent years, Transformer-based networks have received increasing attention in the extraction of spatial and temporal information. Based on the Transformer-based network, through the global attention mechanism, it can capture more complex temporal and spatial relationships.

[0004] Former-DFER proposed a method for dynamic facial expression recognition based on Transformer. This network processes spatial and temporal information through Convolutional Spatial Transformer (CS-Former) and Temporal Transformer (T-Former), respectively. CS-Former consists of five convolutional blocks and N spatial encoder groups. The input part divides the video sequence into U segments and extracts V frames from the middle of each segment, forming a total of U × V frames as the input of the model. The input frames are first passed through four convolutional blocks to extract local feature maps, then the features in the channel dimension are pulled into a sequence form, and after adding position embedding, they are sent to the SpatialEncoder encoder module. Spatial Encoder extracts global features through self-attention mechanism. The role of the four convolutional blocks is to extract local features, and the additional convolutional blocks connected subsequently further refine the features to generate more robust spatial representations. Each frame processed by CS-Former generates a feature representation, and the entire set of frames forms a sequence. To perform the final classification, the model adds a Class Token in the sequence, as shown in the figure. Then, by adding TemporalPositional Embedding to represent the time sequence, the temporal encoder captures the dynamic correlation between frames, and finally completes the expression classification. However, Former-DFER is designed for expression recognition and only models facial information, without considering contextual information. Moreover, the model is based on Transformer, and deep network learning often requires a large amount of data. The cost of collecting real-world data such as academic emotions is relatively high, so the amount of data is usually small.

[0005] DFER-CLIP is a method that uses the visual language pre-training (V-LP) model CLIP to complete the dynamic expression recognition task. CLIP can obtain rich visual representations by learning the semantic relationship between images and text, which provides potential advantages for dynamic facial expression recognition. Specifically, DFER-CLIP combines dynamic facial features and expression-related text descriptions to achieve higher accuracy in facial expression recognition. As a famous visual language pre-training model, CLIP (Contrastive Language-Image Pre-training) is trained through contrastive learning, enabling the model to learn the correspondence between images and text. DFER-CLIP takes full advantage of this feature of CLIP by integrating its powerful visual and language modeling capabilities, showing good potential in understanding and recognizing dynamic facial expressions.

[0006] Pre-trained models require less data, however, similar to Former-DFER, DFER-CLIP is mainly designed for basic expression recognition task and does not model context information, which may limit its applicability in some scenarios. SUMMARY

[0007] The present application aims to solve some problems existing in the current facial expression recognition model in academic expression recognition, for example, some deep learning models require a large amount of labeled data, which is not suitable for small-scale academic expression data sets, and existing facial expression recognition models often only model facial expressions, without modeling the context information which is crucial for academic emotion recognition. To this end, an academic emotion recognition algorithm based on facial expression and context is proposed. This algorithm can significantly reduce the demand for a large amount of labeled data of existing algorithms, while modeling important context information in learning situations and facial expressions, using context information to help identify the overall learning state of students (such as the ability to identify whether students are studying seriously or slacking off), and improve the accuracy of academic emotion recognition.

[0008] The purpose of the present application is achieved by the following technical scheme: an academic emotion recognition method based on facial expression and context, comprising the following steps:

[0009] S1, segmenting the face region from the video frame to generate the facial expression frame, the original video frame as the context information frame, constructing the emotion category text description containing facial expression and context, and obtaining the academic emotion video data set;

[0010] S2, constructing an academic emotion recognition network based on facial expression and context, which first extracts spatial features from the facial expression frame and the context information frame in the input video, respectively, and generates unified visual feature tokens by fusing spatial features and temporal information, the network uses a text encoder to input text to extract text feature tokens corresponding to the emotion category, and finally, by calculating the cosine similarity between the visual feature tokens and each emotion category text feature token, and normalizing the similarity, the probability distribution of all emotion categories is generated, and the classification of academic emotion is finally completed;

[0011] S3, constrain and optimize the academic emotion recognition network based on facial expression and context through the classification loss function and the academic emotion video data set, complete network training, and generate the final academic emotion recognition model based on facial expression and context;

[0012] S4, based on the academic emotion recognition model based on facial expression and context trained, analyze the input video segment, and identify the academic emotion category of students.

[0013] Further, the academic emotion video dataset includes different learning environments and learning links, wherein the academic emotion categories include: neutral, happy, confused, tired and distracted; wherein the neutral, happy, confused and tired are specific emotion categories in the learning state, and the distracted covers all emotions in the non-learning state.

[0014] Further, the academic emotion recognition network based on facial expression and context situation includes: a CLIP visual encoder, a time relationship learning module, a visual feature fusion module, a CLIP text encoder and a feature alignment module.

[0015] The CLIP visual encoder includes a 2D convolution block and 12 Transformer layers, and inputs a face expression image sequence and context information images; face features and context features are obtained;

[0016] The time relationship learning module includes two time learning units with the same structure, which are respectively used for independent time sequence modeling of face features and context features, and are recorded as face time modeling module and context time modeling module. Each time learning unit is composed of a Transformer layer, and face features and context features fused with time information are respectively obtained;

[0017] The visual feature fusion module includes a full connection layer; the face features and context features fused with time information are spliced, the feature dimension is adjusted, and the visual feature token is obtained by alignment;

[0018] The CLIP text encoder includes 12 Transformer layers, and inputs text descriptions of each category to obtain text feature tokens of each category;

[0019] The feature alignment module realizes cross-modal space alignment by calculating the cosine similarity of the visual feature token and the text feature token of each category, and the video academic emotion category corresponding to the text feature with the highest cosine similarity is the final recognition result.

[0020] Further, the CLIP visual encoder includes:

[0021] For the input face expression image sequence and context information image sequence, single-frame feature extraction is performed through the CLIP visual encoder with shared weights respectively, and the corresponding visual feature representation is obtained;

[0022] The single-frame feature extraction process includes:

[0023] First, the picture is regionally blocked by 2D convolution, and the feature token of the block is obtained by embedding operation; then a learnable class token is generated, which has the same dimension as the blocked picture and is initialized as a random vector to represent the global information of the whole image; the class token and all the blocked feature tokens are input into the Transformer layer, and after the interaction calculation of multiple attention mechanisms, the global and local information are fused, and finally the updated class token is output by the CLIP visual encoder.

[0024] Further, the time relationship learning module specifically includes the following processing procedures:

[0025] For the facial expression feature token sequence extracted by the CLIP visual encoder, a learnable class token is first introduced, which is input into the facial time relationship learning module together with the facial expression feature token sequence. The module captures the dynamic relationship and global timing information between frames through timing modeling. In this process, the weight of the class token is updated through training, and finally the class token representation containing timing information is obtained, which is used to represent the facial expression information with time-dependent features;

[0026] For the context information feature token sequence, a randomly initialized learnable class token is introduced, and is input into the context time relationship learning module together with the context feature token sequence. The module learns the inter-frame dynamics and time dependence of the context features through timing modeling. Similarly, the weight of the class token is updated through training. Finally, the updated class token is used to represent the context features with timing information.

[0027] The learnable class token is a learnable vector with the same length as the output of the CLIP visual encoder.

[0028] Further, the CLIP text encoder specifically includes the following processing procedures:

[0029] First, M randomly initialized learnable text prompts are generated to learn additional detailed information not covered in the text description. Then, these learnable prompt words are spliced with the text description after tokenization to form a new input sequence. Start and end markers are added before and after the spliced sequence, respectively.

[0030] Next, the length of the spliced sequence is aligned to ensure that the number of tokens in all sentences is consistent. The spliced sequence is then input into the Transformer model to capture global timing dependencies. In the output of the Transformer, the output feature token corresponding to the end marker is used as the text representation of the entire sentence. This representation will be aligned with the visual feature token to achieve cross-modal alignment training.

[0031] During training, the pre-trained weights of the CLIP model are first loaded, and the weights of the CLIP text encoder are frozen so as not to participate in updating. The weights of the remaining parts are updated.

[0032] Further, the feature alignment module specifically includes the following processing procedures:

[0033] First, the cosine similarity between the visual feature token and the text feature token of class k is calculated wherein, is the visual feature token, is the text feature token of the kth academic emotion class, represents the cosine similarity calculation operation, and then the cosine similarities of all classes are normalized by a softmax function to obtain the probability distribution of each class:

[0034]

[0035] wherein K represents the total number of academic emotion classes, and k represents a specific emotion class, represents the probability that the emotion class y is equal to the class k when the input video S is given, is the text feature token of class j.

[0036] Further, the classification loss function L loss is:

[0037]

[0038] wherein N represents the total number of samples, and K represents the total number of academic emotion classes, is a class indicator variable, which takes a value of 1 when the real class of sample i is equal to k, and 0 otherwise, represents the probability that the emotion class y is equal to k when the input video is given.

[0039] On the other hand, the specification of the present application provides an academic emotion recognition device based on facial expressions and contextual situations, which comprises a memory and one or more processors, the memory stores executable code, and the processor executes the executable code to realize the academic emotion recognition method based on facial expressions and contextual situations.

[0040] On the other hand, the specification of the present application provides a computer readable storage medium, which stores a program, and the program is executed by a processor to realize the academic emotion recognition method based on facial expressions and contextual situations.

[0041] ​​The beneficial effects of the present application are: firstly, the present application introduces a double-branch architecture, which cooperatively models the facial expressions and the context information of the learning scene, fully excavates and fuses the facial expression features of students and the key information in the learning environment, and strengthens the understanding of the situation-related factors. The attention mechanism is used to model the facial expressions and the context information in time sequence, and a feature fusion module is designed to deeply fuse the facial expression features and the context information, so as to improve the understanding of the learning state by the model. This method not only can accurately capture the subtle emotional changes of students in the learning process, but also can combine the behavior clues and environmental factors to improve the accurate recognition ability of academic emotions. Compared with the traditional method which only relies on facial expressions, the present application effectively utilizes the key context clues (such as looking at the mobile phone or learning materials, etc.) in academic emotion recognition, significantly enhances the ability of the model to distinguish different learning states (such as focused learning or distraction), and improves the robustness and generalization ability of academic emotion recognition.

[0042] Secondly, in the aspect of text feature modeling, the present application designs a learnable text prompt mechanism, which introduces task-adaptive text embedding to enable the model to autonomously learn the situation-related emotional description, and supplements the additional detailed information that cannot be covered by the fixed text prompt. This method fully utilizes the powerful feature extraction capability of CLIP, reduces the dependence on large-scale training data, and improves the stability and practicality of the model. BRIEF DESCRIPTION OF DRAWINGS

[0043] Figure 1 A flow chart of an academic emotion recognition method based on facial expressions and context situations is provided for the embodiments of the present application.

[0044] Figure 2 A network structure diagram of a CLIP visual encoder is provided for the embodiments of the present application.

[0045] Figure 3 A network structure diagram of a time relationship learning module is provided for the embodiments of the present application.

[0046] Figure 4 A network structure diagram of a CLIP text encoder is provided for the embodiments of the present application.

[0047] Figure 5 A schematic diagram of an academic emotion recognition device based on facial expressions and context situations is provided for the embodiments of the present application. DETAILED DESCRIPTION

[0048] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the specific embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0049] In the following description, a lot of specific details are set forth in order to provide a thorough understanding of the present application, however, the present application can be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the present application, therefore, the present application is not limited to the specific embodiments disclosed below.

[0050] As shown in Figure 1 The present application proposes a learning emotion recognition method based on facial expression and context situation, which specifically includes the following steps:

[0051] Step 1: Constructing a learning emotion recognition model training data set based on facial expression and context situation;

[0052] The learning emotion recognition model training data constructed in the step 1 includes the following steps:

[0053] The data set is constructed based on the self-collected learning emotion video data set RAER. The RAER data set is collected in real learning scenarios, covering various actual learning environments such as libraries, classrooms, laboratories, dormitories, etc., including self-study scenes and teaching links. The data set contains rich context information, such as students' hand movements (such as using mobile phones, flipping through textbooks or operating tablets), and all data are labeled with high quality to generate accurate data labels.

[0054] The label set of the RAER data set contains five common learning emotion categories: neutral, happy, confused, tired, and distracted. Among them, neutral, happy, confused, and tired reflect specific emotion categories in learning state, while distraction covers all emotions in non-learning state. In order to simplify the annotation, the distraction emotion is not further subdivided. The data set is divided into training set and test set: the training set is used to train the model, and the test set is used to evaluate the recognition ability of the model on unseen data. The construction of the RAER data set provides important support for the research and application of learning emotion recognition field. For the five learning emotion categories, ChatGpt is used to generate corresponding text descriptions, including corresponding facial expression features (such as "smiling") and context scene features (such as "operating a mobile phone").

[0055] Step 2: Constructing a learning emotion recognition network based on facial expression and context situation;

[0056] The learning emotion recognition network based on facial expression and context situation in the step 2 includes:

[0057] The network is constructed based on a CLIP model and includes the following five modules: a CLIP visual encoder (CLIP-IE), a temporal relationship learning module (Temporal Encoder), a visual feature fusion module (Feature Fusion Module, FFM), a CLIP text encoder (CLIP-TE), and a feature alignment module (Feature Alignment Module, FAM).

[0058] The CLIP visual encoder is used to extract the spatial features of single frames from the input video frame samples, and generate feature tokens for each frame for subsequent time feature modeling. The input video sample S is composed of N frame facial expression image sequences and N frame context scene information image sequences , wherein The encoder adopts a structure based on VisionTransformer (ViT), including a 2D convolution block and 12 layers of Transformer modules, each of which is composed of a multi-head self-attention network and a feedforward neural network.

[0059] In the workflow of the CLIP visual encoder, as shown in Figure 2 , the input image is first divided into blocks with a size of 32x32 by the 2D convolution block, and each block is embedded into a feature token with a length of 512. For an image with a size of 224x224, a total of 49 patches are segmented, corresponding to the generation of 49 feature tokens. In addition, the encoder generates a learnable class token with the same dimension as the block features, which is used to represent the global information of the entire frame image. The class token and all block feature tokens are input into the Transformer module, and after 12 layers of multi-head self-attention calculation, the output of the class token is fused with the global and local features, serving as the feature representation of the frame image. The CLIP visual encoder extracts features from the facial expression image sequences and the context image sequences,

[0060]

[0061]

[0062] wherein represents the CLIP visual encoder, and finally outputs two sets of feature token sequences: one set of facial expression feature token sequences, represented as = , ,.... }, and the other group is the context feature token sequence, denoted as ={ , ,.... } for subsequent temporal information modeling.

[0063] The time relationship learning module is configured to model the temporal features of the input video frame sequence to represent the dynamic changes and global temporal information of the face time sequence and the context time sequence. The module adopts a classic Transformer architecture and is composed of two structurally identical time learning units, which are respectively configured to independently model the temporal features of the face and the context, and are denoted as a face time modeling module and a context time modeling module. Each time learning unit includes a layer of Transformer, including a multi-head self-attention network and a feedforward neural network.

[0064] In the processing flow of the time relationship learning module, as shown in Figure 3 , for the face expression feature token sequence extracted by the CLIP visual encoder, a randomly initialized learnable class token is first introduced, which has the same dimension as the feature token output by the CLIP visual encoder, and is used to represent the overall face video features with temporal information. The class token and the face feature tokens of all frames form an input sequence ={ , , ,.... }, wherein denotes the class token. The sequence is input into the face time modeling module to capture the dynamic relationship and global temporal dependency between frames through the time modeling of the Transformer. In this process, the class token interacts with other feature tokens in the sequence through the self-attention mechanism, and its weight is constantly updated during the training process.

[0065]

[0066] Finally, the updated class token fuses the temporal information of all frames as the temporal feature representation of the complete face video, wherein denotes the self-attention mechanism.

[0067] For the context feature token sequence , a similar method is adopted. The processing process is similar. A randomly initialized learnable class token is introduced, which is concatenated with the context feature token sequence to form ={ , ,....} The sequence is input into the context temporal modeling module to learn the dynamic relationship and temporal dependency between context frames using the Transformer.

[0068]

[0069] Finally, the updated class token is used as a representation of the context temporal feature to characterize the context feature with time information, where represents the self-attention mechanism.

[0070] The visual feature fusion module is mainly used to fuse the facial feature and the context feature, and contains a fully connected layer (Fully Connected Layer) to generate a unified overall visual feature representation and provide support for cross-modal alignment. The module contains a fully connected layer (Fully Connected Layer) to adjust the feature dimension and realize alignment.

[0071] In the processing flow of the visual feature fusion module, the facial feature token with a length of 512 output by the temporal relationship learning module and the context feature token with a length of 512 are first spliced to form a new feature vector, and the spliced feature token has a length of 1024. Then, the feature token is input into the fully connected layer, and the feature is reduced in dimension through linear transformation, and an overall visual feature token with a length of 512 is output. The reduced overall visual feature vector contains not only the time sequence information of the facial expression feature, but also the temporal dynamic relationship of the context feature, so it can more comprehensively represent the visual information of the input video. Finally, the overall visual feature vector and the text feature vector are aligned in the shared feature space for cross-modal classification of learning emotions.

[0072] The CLIP text encoder is used to extract independent text features for each class, and the visual features are spatially aligned to complete classification. The text encoder consists of 12 Transformer layers, each containing a self-attention network and a feedforward neural network. In addition, the encoder introduces a learnable text prompt to capture details not covered in the fixed text description; in the specific processing flow of the CLIP text encoder, as shown in Figure 4 , for the text description of each class k, first generate M learnable text prompts, and then concatenate them with the after tokenization by the Tokenizer to form a class-independent input sequence:

[0073]

[0074] where k denotes the k-th category, denotes the input sequence of the k-th category, M denotes the number of learnable text prompts, Tokenizer denotes the tokenization operation, denotes the text description of the k-th category. Start token ([CLS]) and end token ([SEP]) are added before and after the concatenated sequence to clearly indicate the start and end positions of the sequence. It is input into the CLIP text encoder model to capture global temporal dependencies.

[0075]

[0076] In the output of the CLIP text encoder, the output feature token corresponding to the end token ([SEP]) is used as the text representation of the entire sentence, which will be aligned with the visual feature token to achieve cross-modal alignment training. Finally, the feature token representation of each category text description is , where k denotes the k-th category.

[0077] During training, the pre-trained weights of the CLIP model are first loaded, and the weights of the CLIP text encoder are frozen so that they do not participate in updating. The weights of the remaining parts are updated. This mechanism ensures that the model can focus on learning the detailed features of the text through the prompt words based on the fixed pre-trained language model, thereby optimizing the alignment effect of text and visual features.

[0078] The feature alignment module is mainly used for cross-modal spatial alignment and finally completes the classification of academic emotions. The module selects the category with the highest similarity as the final prediction result by calculating the cosine similarity between the visual feature and the text features of all categories. In the specific processing process of the feature alignment module, first, the cosine similarity between the visual feature and the text feature of each category k is calculated, where denotes the cosine similarity calculation operation. Then, the cosine similarities of all categories are normalized by the softmax function to obtain the probability distribution of each category:

[0079]

[0080] where K denotes the total number of academic emotion categories, k denotes a specific emotion category, denotes the probability that the emotion category y equals category k when given input video S, is the text feature token of category j.

[0081] Step 3: constraint and optimization of the facial expression and context-based academic emotion recognition network through a classification loss function, and training of a final context-aware academic emotion recognition model;

[0082] The step 3 of constraint and optimization of the facial expression and context-based academic emotion recognition network through a classification loss function, and training of a final context-aware academic emotion recognition model includes the following steps:

[0083] After obtaining the probability of the video belonging to each sample after feature alignment, the model can be supervised through the real label, and the specific formula of the loss used is as follows:

[0084]

[0085] The cross-entropy loss function is most commonly used in classification tasks, where N represents the total number of samples, K represents the total number of academic emotion categories, is a category indicator variable, which takes a value of 1 when the real category of sample i is equal to k, and 0 otherwise, represents the probability that the predicted emotion category y is equal to k when the input video is given.

[0086] Step 4: academic emotion estimation of an input video segment based on the context-aware academic emotion recognition model trained.

[0087] Corresponding to the aforementioned embodiment of the facial expression and context-based academic emotion recognition method, the present application also provides an embodiment of a facial expression and context-based academic emotion recognition device.

[0088] Referring to Figure 5 , the embodiment of the present application provides a facial expression and context-based academic emotion recognition device, which comprises a memory and one or more processors, the memory stores executable code, and the processor executes the executable code to implement the facial expression and context-based academic emotion recognition method in the above embodiment.

[0089] The embodiment of the facial expression and context-based academic emotion recognition device provided by the present application can be applied to any device with data processing capability, which can be a device or apparatus such as a computer. The device embodiment can be realized by software, or by hardware or a combination of software and hardware. Taking software implementation as an example, as a logical device, it is formed by reading the corresponding computer program instructions in the non-volatile memory into the memory and running by the processor of the device with data processing capability. From the hardware point of view, for example,Figure 5 Fig. 1 is a hardware structure diagram of a device where the learning emotion recognition device based on facial expression and context situation provided by the present application is located in any data processing capable device, in addition to the processor, the memory, the network interface, and the non-volatile memory shown in the figure. Figure 5 In addition to the processor, the memory, the network interface, and the non-volatile memory shown in the figure, any data processing capable device where the device in the embodiment is located can also include other hardware according to the actual functions of the data processing capable device, and no further description is given here.

[0090] The implementation process of the functions and roles of each unit in the above device is specifically described in the implementation process of the corresponding steps in the above method, and no further description is given here.

[0091] For the device embodiment, since it basically corresponds to the method embodiment, the relevant part is described in the part of the method embodiment. The above described device embodiment is only schematic, and the units described as separate components can or can not be physically separated, and the components displayed as units can or can not be physical units, i.e. they can be located in one place or distributed on multiple network units. Part or all of the modules can be selected to achieve the purpose of the present application scheme according to actual needs. Those skilled in the art can understand and implement without creative labor.

[0092] The embodiment of the present application also provides a computer readable storage medium, which stores a program, and the program is executed by a processor to realize the learning emotion recognition method based on facial expression and context situation in the above embodiment.

[0093] The computer readable storage medium can be an internal storage unit of any data processing capable device, such as a hard disk or a memory. The computer readable storage medium can also be an external storage device of any data processing capable device, such as a plug-in hard disk, a smart media card (SMC), an SD card, a flash card, etc. Further, the computer readable storage medium can include both the internal storage unit and the external storage device of any data processing capable device. The computer readable storage medium is used to store the computer program and other programs and data required by the data processing capable device, and can also be used to temporarily store data that has been output or will be output.

[0094] The present application also provides a computer program product, which includes a computer program, and the computer program is executed by a processor to realize the learning emotion recognition method based on facial expression and context situation.

[0095] Other embodiments of the application will be apparent to those skilled in the art from consideration of the specification and practice of the application disclosed herein. It is intended that the specification and examples be considered as exemplary only, with the true scope and spirit of the application being indicated by the following claims.

[0096] It is to be understood that the above general description and the subsequent detailed description are merely exemplary and explanatory and are not restrictive of the application. The application is not restricted to the precise construction and combinations of parts and steps described above and shown in the accompanying drawings, as such variations and modifications are intended to be included within the scope of the application. The scope of the application is limited only by the claims that follow.

Claims

1. A method for academic emotion recognition based on facial expressions and contextual information, characterized in that, The method includes the following steps: S1. Segment the face region from the video frame to generate facial expression frames. Use the original video frame as context information frame to construct an emotion category text description containing facial expression and context, and obtain the academic emotion video dataset. S2. Construct an academic emotion recognition model based on facial expressions and contextual information. The model first extracts spatial features from facial expression frames and contextual information frames in the input video, and generates a unified visual feature token by fusing spatial features and temporal information. The network uses a text description input text encoder to extract text feature tokens corresponding to the emotion category. Finally, by calculating the cosine similarity between the visual feature token and the text feature token of each emotion category, and normalizing the similarity, the probability distribution of all emotion categories is generated, and the classification of academic emotions is finally completed. The academic emotion recognition model based on facial expressions and context includes: CLIP visual encoder, temporal relationship learning module, visual feature fusion module, CLIP text encoder and feature alignment module; The CLIP visual encoder contains a 2D convolutional block and 12 Transformer layers. It takes a sequence of facial expression images and a contextual information image as input and obtains facial features and contextual features. The temporal relationship learning module contains two identical temporal learning units, which are used to perform independent temporal modeling of facial features and contextual features, respectively. These are referred to as the facial temporal modeling module and the contextual temporal modeling module. Each temporal learning unit consists of a Transformer layer, which respectively obtains facial features and contextual features that have been fused with temporal information. The visual feature fusion module includes a fully connected layer; it concatenates facial features with fused time information and contextual features, adjusts feature dimensions, and aligns them to obtain a visual feature token; The CLIP text encoder contains 12 Transformer layers, and inputs text descriptions of each category to obtain text feature tokens for each category; The feature alignment module achieves cross-modal alignment by calculating the cosine similarity between the visual feature token and the text feature token of each category. The video academic emotion category corresponding to the text feature with the highest cosine similarity is the final recognition result. The processing flow of the time relationship learning module includes: For the facial expression feature token sequence extracted by the CLIP visual encoder First, a randomly initialized, learnable category token is introduced, with the same dimension as the feature token output by the CLIP visual encoder. This token represents the overall face video features with temporal information. The category token, together with the face feature tokens from all frames, constitutes the input sequence. Where f f The category token is input into the face temporal modeling module, which uses Transformer temporal modeling to capture the dynamic relationships and global temporal dependencies between frames. During this process, the category token interacts with other feature tokens in the sequence through a self-attention mechanism, and its weights are continuously updated during training. Finally, the updated category tokens The temporal information of all frames is integrated as a temporal feature representation of the complete face video, where S-ATT represents the self-attention mechanism; For context feature token sequence Introduce a randomly initialized learnable category token f. c , concatenated with the context feature token sequence They are input together into the contextual time modeling module, which uses Transformer to learn the dynamic relationships and temporal dependencies between context frames; Finally, the updated category tokens As a representation of contextual temporal features, it is used to characterize contextual features with temporal information; S3. Constrain and optimize the academic emotion recognition network based on facial expressions and context by using a classification loss function and an academic emotion video dataset, complete the network training, and generate the final academic emotion recognition model based on facial expressions and context. S4. Based on the trained academic emotion recognition model based on facial expressions and context, analyze the input video clips to identify the student's academic emotion category.

2. The academic emotion recognition method based on facial expressions and contextual information as described in claim 1, characterized in that, The academic emotion video dataset includes different learning environments and learning stages, and the academic emotion categories include: neutral, pleasant, confused, tired, and distracted; wherein, neutral, pleasant, confused, and tired are specific emotion categories in the learning state, while distracted covers all emotions in the non-learning state.

3. The academic emotion recognition method based on facial expressions and contextual information according to claim 1, characterized in that, The CLIP visual encoder's specific processing flow includes: For the input sequence of facial expression images and the sequence of context information images, single-frame feature extraction is performed using a CLIP visual encoder with shared weights to obtain the corresponding visual feature representations. The single-frame feature extraction process includes: First, the image is divided into regions using 2D convolution, and an embedding operation is performed to obtain the feature token for each region. Then, a learnable category token is generated, with the same dimension as the segmented image, and initialized as a random vector to represent the global information of the entire image. The category token and all segmented feature tokens are input into the Transformer layer. After interactive computation through multiple attention mechanisms, global and local information are fused, and finally, the CLIP visual encoder outputs the updated category token.

4. The academic emotion recognition method based on facial expressions and contextual information as described in claim 1, characterized in that, The CLIP text encoder's specific processing flow includes: First, generate M randomly initialized learnable text prompts to learn additional details not covered in the text description. Then, concatenate these learnable prompts with the segmented text description to form a new input sequence. Add start and end markers before and after the concatenated sequence, respectively. Next, the concatenated sequence is length aligned to ensure that the number of tokens in all sentences is consistent. The concatenated sequence is then input into the Transformer model to capture global temporal dependencies. In the output of the Transformer, the output feature token corresponding to the end marker is used as the text representation of the entire sentence. This representation is aligned with the visual feature token to achieve cross-modal alignment training. During training, the pre-trained weights of the CLIP model are loaded first, and the weights of the CLIP text encoder are frozen so that they do not participate in the update. The weights of the remaining parts are then updated.

5. The academic emotion recognition method based on facial expressions and contextual information according to claim 1, characterized in that, The feature alignment module, in its specific processing flow, includes: First, the cosine similarity between the visual feature token and the text feature token of the category is calculated. Then, the cosine similarity of all categories is normalized using the softmax function to obtain the probability that the predicted sentiment category is equal to each text feature category when given an input video.

6. The academic emotion recognition method based on facial expressions and contextual information according to claim 1, characterized in that, The classification loss function L loss for: Where N represents the total number of samples, K represents the total number of academic emotion categories, and y i,k P(y=k|S) is a category indicator variable, taking a value of 1 when the true category of sample i is equal to k, and 0 otherwise. i ) indicates that given the input video S i When predicting the probability that the emotion category y equals k.

7. A device for academic emotion recognition based on facial expressions and contextual information, comprising a memory and one or more processors, wherein the memory stores executable code, characterized in that... When the processor executes the executable code, it implements an academic emotion recognition method based on facial expressions and contextual information as described in any one of claims 1-6.

8. A computer-readable storage medium having a program stored thereon, characterized in that, When the program is executed by the processor, it implements an academic emotion recognition method based on facial expressions and contextual information as described in any one of claims 1-6.

Images