Method and apparatus for recognizing expression of online learner

Abstract

This invention belongs to the field of artificial intelligence and discloses an online learner facial expression recognition method and related apparatus. The method includes: acquiring a static face image extracted from the online learner's learning video to obtain a picture to be recognized; inputting the picture to be recognized into a pre-set multi-view facial expression recognition model based on contrastive learning to obtain the online learner's facial expression recognition result; the multi-view facial expression recognition model based on contrastive learning is obtained by performing comparative learning on a pre-trained multi-view facial expression recognition model through intra-class and inter-class contrastive learning; the multi-view facial expression recognition model includes a key point feature and facial feature fusion module, a multi-scale feature fusion module, a classification vector embedding module, and a classifier connected sequentially. The introduction of key point features and multi-scale features solves the problem of low facial expression recognition accuracy caused by diverse learner poses and perspectives, varying face sizes, and poor model generalization ability. The comparative learning through intra-class and inter-class contrastive learning solves the problems of imbalanced facial expression recognition datasets and low facial expression activation, effectively improving facial expression recognition accuracy.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence and relates to a method and related device for recognizing the facial expressions of online learners. Background Technology

[0002] In recent years, with the rapid development of information technology, online education has experienced explosive growth globally due to its advantages of transcending time and space boundaries and offering abundant and diverse resources. This has greatly broadened learners' access to knowledge and promoted the popularization and personalized development of education. Against this backdrop, effectively assessing students' learning status has become a crucial aspect of improving the quality of online teaching. Facial expressions, as one of the most intuitive and natural ways of human emotional communication, play a vital role in online learning scenarios. They not only reflect students' focus, comprehension level, and even confusion and interest, but also provide teachers with immediate feedback.

[0003] However, directly applying traditional facial expression recognition technology to complex online learning environments faces numerous challenges. First and foremost is the diversity of student postures. Students may participate in learning in various positions, such as sitting, lying down, or even standing, which significantly impacts the accuracy of facial feature capture and recognition. Secondly, the variability of camera position is another major challenge. Different camera angles, lighting conditions, and background environments can all interfere with facial expression recognition algorithms, increasing the risk of misjudgments. Furthermore, teachers often pay more attention to negative expressions that directly reflect students' learning difficulties or dissatisfaction, such as confusion, fatigue, or dissatisfaction. However, traditional models often struggle to handle these subtle and complex facial expressions, resulting in lower accuracy in facial expression recognition. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings of the prior art and provide an online learner facial expression recognition method and related device.

[0005] To achieve the above objectives, the present invention employs the following technical solution:

[0006] In a first aspect, the present invention provides an online learner facial expression recognition method, comprising: acquiring a static face image extracted from a learning video of an online learner to obtain an image to be recognized; inputting the image to be recognized into a preset multi-view facial expression recognition model based on contrastive learning to obtain the facial expression recognition result of the online learner; wherein, the multi-view facial expression recognition model based on contrastive learning is obtained by: performing contrastive learning on a pre-trained multi-view facial expression recognition model through intra-class contrastive learning and inter-class contrastive learning to obtain the multi-view facial expression recognition model based on contrastive learning; wherein, the multi-view facial expression recognition model includes a key point feature and facial expression feature fusion module, a multi-scale feature fusion module, a classification vector embedding module, and a classifier connected in sequence; the key point feature and facial expression feature fusion module is used to extract key point features and facial expression features of the image to be recognized and perform feature fusion to obtain a fused feature map; the multi-scale feature fusion module is used to perform multi-scale feature fusion on the fused feature map to obtain a multi-scale fused feature map; the classification vector embedding module is used to embed classification vectors into the multi-scale fused feature map to obtain a learned classification vector; and the classifier is used to obtain the facial expression recognition result based on the learned classification vector.

[0007] Optionally, the key point feature and expression feature fusion module is specifically used for: extracting expression features of the image to be recognized using an IR50 model pre-trained on a face recognition dataset; extracting key point features of the image to be recognized using a pre-trained Mobile Face Net model; performing a linear transformation on the key point features and expression features, mapping the key point features to a query vector, and mapping the expression features to a key vector and a value vector respectively; calculating an attention score by the inner product of the query vector and the key vector, scaling and normalizing the attention score to obtain an attention weight map; and weighted summing the attention weight map and the value vector to obtain a fused feature map.

[0008] Optionally, the multi-scale feature fusion module includes several consecutive downsampling modules and upsampling modules, and uses skip connections to connect the feature maps in the downsampling modules and upsampling modules at the corresponding levels.

[0009] Optionally, the classification vector embedding module is specifically used to: embed the classification vector into a multi-scale fusion feature map, and use two multi-head self-attention layers to enable the classification vector to learn the multi-scale fusion expression features of the multi-scale fusion feature map to obtain the learned classification vector.

[0010] Optionally, the step of performing comparative learning on the pre-trained multi-view expression recognition model through intra-class and inter-class comparative learning to obtain a multi-view expression recognition model based on comparative learning includes: traversing each training image in the training dataset and performing comparative learning steps respectively: obtaining the attention weight map of the current training image through the multi-view expression recognition model; selecting regions with a high number of weights before the current training image as retained regions from the attention weight map, and cropping and saving the retained regions from the training image; randomly selecting random training images from the training dataset, and pasting the retained regions cropped from the current training image to the corresponding positions in the random training images to obtain resampled training images; obtaining the feature vectors of the current training image and the resampled training image based on the multi-view expression recognition model. The process involves several steps: First, an intra-class contrastive learning loss is obtained for the current training image based on the similarity between the feature vectors of the current training image and the resampled training image. Second, an inter-class contrastive learning loss is obtained for the current training image based on the optimization objective of making the feature vectors of training images of the same category closer and the feature vectors of training images of different categories further apart. Third, a weighted sum of the intra-class and inter-class contrastive learning losses for each training image is used to obtain a reweighted loss for the training dataset. The multi-view expression recognition model is then optimized based on this reweighted loss. These steps are iterated until a preset number of iterations is reached or the optimized multi-view expression recognition model meets preset requirements, resulting in a contrastive learning-based multi-view expression recognition model.

[0011] Optionally, obtaining the intra-class contrastive learning loss of the current training image includes: obtaining the intra-class contrastive learning loss of the current training image through the following...

[0012]

[0013] Where λ is the preset weight parameter, CosSim() is the cosine similarity function, p1 and p2 are two randomly augmented views of the current training image, and t1 and t2 are two randomly augmented views of the resampled training image.

[0014] The inter-class contrast learning loss for obtaining the current training image includes: obtaining the inter-class contrast learning loss for the current training image through the following...

[0015]

[0016] Where I ≡ {1, ..., N} is the training dataset, N is the total number of training images, i represents the current training image, and p iA(i) represents the feature vector obtained by the multi-view expression recognition model for the current training image; A(i) is the set of the remaining training images in the training dataset excluding the current training image. Let |T(i)| be the set of training images in the training dataset that belong to the same category as the current training image, and |T(i)| be the number of training images in T(i). a Let p be the feature vector obtained from the a-th training image through the multi-view expression recognition model. t Let be the feature vector obtained from the t-th training image using the multi-view facial expression recognition model.

[0017] Optionally, the reweighted loss for obtaining the training dataset includes:

[0018] The reweighted loss of the training dataset is calculated using the following formula.

[0019]

[0020] Among them, E n λ is the number of valid samples out of n samples; β is a preset hyperparameter; λ1 and λ2 are both preset weight parameters; For intra-class contrastive learning loss, The loss is for inter-class comparison learning.

[0021] In a second aspect, the present invention provides an online learner facial expression recognition system, comprising: an image acquisition module for acquiring static facial images extracted from online learners' learning videos to obtain images to be recognized; and an expression recognition module for inputting the images to be recognized into a preset multi-view expression recognition model based on contrastive learning to obtain the online learner's expression recognition result; wherein the multi-view expression recognition model based on contrastive learning is obtained by performing contrastive learning on a pre-trained multi-view expression recognition model through intra-class contrastive learning and inter-class contrastive learning to obtain the multi-view expression recognition model based on contrastive learning; wherein the multi-view expression recognition model includes a key point feature and expression feature fusion module, a multi-scale feature fusion module, a classification vector embedding module, and a classifier connected in sequence; the key point feature and expression feature fusion module is used to extract key point features and expression features from the images to be recognized and perform feature fusion to obtain a fused feature map; the multi-scale feature fusion module is used to perform multi-scale feature fusion on the fused feature map to obtain multi-scale fused expression features; the classification vector embedding module is used to embed classification vectors into the multi-scale fused feature map to obtain a learned classification vector; and the classifier is used to obtain the expression recognition result based on the learned classification vector.

[0022] In a third aspect, the present invention provides a computer device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described online learner facial expression recognition method.

[0023] In a fourth aspect, the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the above-described online learner facial expression recognition method.

[0024] Compared with the prior art, the present invention has the following beneficial effects:

[0025] This invention presents an online learner facial expression recognition method. It utilizes a multi-view facial expression recognition model based on contrastive learning to recognize facial expressions from static facial images captured in online learners' learning videos, thereby obtaining the learners' facial expressions. The multi-view facial expression recognition model comprises a keypoint feature and facial expression feature fusion module, a multi-scale feature fusion module, a classification vector embedding module, and a classifier, all connected sequentially. By incorporating keypoint and multi-scale features through the keypoint feature and facial expression feature fusion module and the multi-scale feature fusion module, it addresses the low accuracy issues caused by diverse learner poses and perspectives, varying face sizes, and poor model generalization ability. Furthermore, the pre-trained multi-view facial expression recognition model undergoes comparative learning through intra-class and inter-class contrastive learning methods, resolving issues such as imbalanced facial expression datasets and low facial expression activation, thereby further improving facial expression recognition accuracy. Attached Figure Description

[0026] Figure 1 This is a flowchart of the online learner facial expression recognition method according to an embodiment of the present invention.

[0027] Figure 2 This is a schematic diagram of the model training process in an embodiment of the present invention.

[0028] Figure 3 This is a network structure diagram of the multi-view facial expression recognition model according to an embodiment of the present invention.

[0029] Figure 4 This is a diagram of the contrastive learning network structure for training the unbalanced facial expression recognition model according to an embodiment of the present invention.

[0030] Figure 5 This is a block diagram of the online learner facial expression recognition system according to an embodiment of the present invention. Detailed Implementation

[0031] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0032] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0033] The present invention will now be described in further detail with reference to the accompanying drawings:

[0034] See Figure 1 In one embodiment of the present invention, an online learner facial expression recognition method is provided to improve the accuracy of facial expression recognition for online learners, thereby providing a basis for teaching analysis for learners and educators.

[0035] Specifically, the online learner facial expression recognition method of the present invention includes the following steps:

[0036] S1: Obtain static face images extracted from online learners' learning videos to obtain the image to be recognized.

[0037] S2: Input the image to be recognized into a preset multi-view facial expression recognition model based on contrastive learning to obtain the facial expression recognition results of the online learner.

[0038] The multi-view facial expression recognition model based on contrastive learning is obtained in the following way:

[0039] A pre-trained multi-view facial expression recognition model is subjected to comparative learning through intra-class and inter-class contrastive learning to obtain a contrastive learning-based multi-view facial expression recognition model. This model comprises a keypoint feature and facial expression feature fusion module, a multi-scale feature fusion module, a classification vector embedding module, and a classifier, all connected sequentially. The keypoint feature and facial expression feature fusion module extracts keypoint and facial expression features from the image to be recognized and fuses them to obtain a fused feature map. The multi-scale feature fusion module performs multi-scale feature fusion on the fused feature map to obtain a multi-scale fused feature map. The classification vector embedding module embeds classification vectors into the multi-scale fused feature map to obtain a learned classification vector. The classifier then uses the learned classification vector to obtain the facial expression recognition result.

[0040] This invention presents an online learner facial expression recognition method. It utilizes a multi-view facial expression recognition model based on contrastive learning to recognize facial expressions from static facial images captured in online learners' learning videos, thereby obtaining the learners' facial expressions. The multi-view facial expression recognition model comprises a keypoint feature and facial expression feature fusion module, a multi-scale feature fusion module, a classification vector embedding module, and a classifier, all connected sequentially. By incorporating keypoint and multi-scale features through the keypoint feature and facial expression feature fusion module and the multi-scale feature fusion module, it addresses the low accuracy issues caused by diverse learner poses and perspectives, varying face sizes, and poor model generalization ability. Furthermore, the pre-trained multi-view facial expression recognition model undergoes comparative learning through intra-class and inter-class contrastive learning methods, resolving issues such as imbalanced facial expression datasets and low facial expression activation, thereby further improving facial expression recognition accuracy.

[0041] See Figure 2 The multi-view facial expression recognition model based on contrastive learning in this invention relies on two processes: training a multi-view facial expression recognition model and training an imbalanced facial expression recognition model. Specifically, the imbalanced facial expression recognition model training is performed through intra-class and inter-class contrastive learning.

[0042] The training of the multi-view facial expression recognition model involves the following steps: First, facial expression data is acquired by using static images extracted from online learners' learning videos as input images for facial expression recognition. Then, the facial expression recognition images undergo data preprocessing, primarily through face alignment and data augmentation methods. Subsequently, multiple CNN layers are used to extract expression features and key point features, and a self-attention mechanism is used to fuse the two types of features. Finally, seven discrete facial expression categories are output, resulting in the trained multi-view facial expression recognition model.

[0043] Imbalanced facial expression recognition model training mainly includes two stages: feature learning and classifier. During the training period, feature learning is mainly carried out to learn the feature information of the sample data. During the testing period, a classifier is added to the model to verify the training results and facial expression recognition is performed on the test set data.

[0044] In one possible implementation, see Figure 3 The network structure of the multi-view expression recognition model is shown, which includes a key point feature and expression feature fusion module, a multi-scale feature fusion module, a classification vector embedding module, and a classifier connected in sequence.

[0045] In one possible implementation, the keypoint feature and expression feature fusion module is specifically used for: extracting expression features of the image to be recognized using an IR50 model pre-trained on a face recognition dataset; extracting keypoint features of the image to be recognized using a pre-trained Mobile Face Net model; performing a linear transformation on the keypoint features and expression features, mapping the keypoint features to a query vector, and mapping the expression features to a key vector and a value vector respectively; calculating an attention score by the inner product of the query vector and the key vector, scaling and normalizing the attention score to obtain an attention weight map; and weighted summing the attention weight map and the value vector to obtain a fused feature map.

[0046] Optionally, the multi-scale feature fusion module includes several consecutive downsampling modules and upsampling modules, and a skip connection is used to connect the feature maps in the downsampling modules and upsampling modules at the corresponding levels.

[0047] Optionally, the classification vector embedding module is specifically used to: embed the classification vector into a multi-scale fusion feature map, and use two multi-head self-attention layers to enable the classification vector to learn the multi-scale fusion expression features of the multi-scale fusion feature map to obtain the learned classification vector.

[0048] Specifically, the MTCNN network is used for face alignment, followed by data augmentation operations, primarily employing random image cropping, random horizontal flipping, random vertical flipping, and histogram equalization. The resulting image size is 224*224*3. Then, a model pre-trained on a face recognition dataset is used for transfer learning, with IR50 and Mobile FaceNet selected as the models for extracting expression and keypoint features, respectively, resulting in an output feature size of 49*512. The keypoint and expression feature fusion module mainly consists of a self-attention module. First, a linear transformation is performed on the keypoint and expression features, mapping the keypoint features to a query vector and the expression features to a key vector and a value vector, respectively. Second, an attention score is calculated using the inner product of the query and key vectors, and the score is scaled and normalized to obtain an attention weight map. Finally, the obtained attention weights and value vectors are weighted and summed to obtain the final fused feature map.

[0049] The multi-scale feature fusion module mainly consists of multiple consecutive downsampling and upsampling operations. Skip connections are used to connect feature maps at different levels during downsampling and upsampling, assisting the upsampling process. The feature map resolution is gradually reduced through two consecutive downsampling operations, with each downsampling stage reducing the feature map size to half its original size. Then, two consecutive upsampling operations restore the feature map resolution. Each upsampling stage is skip-connected to its corresponding downsampling stage, ensuring that the final output multi-scale fused feature map has the same size as the input map.

[0050] The classification vector embedding module implements delayed embedding of classification vectors and sets up two layers of multi-head self-attention modules for learning facial expression features from the classification vectors. The classification vectors are then fed into the classifier for facial expression category recognition. The classification vector is a vector fixed at the beginning of the input features, representing the classification or summary information of the entire feature sequence. It is typically added to the feature map in the early stages of model training and gradually adjusted to a suitable classification vector through backpropagation and optimization algorithms during training. Delayed embedding of classification vectors ensures that they are not interfered with by keypoint features, avoiding instability caused in the early stages of modeling, making them closer to the parts relevant to the facial expression recognition task, and better capturing task-related features and contextual information. Separating feature fusion from feature classification allows the model to focus more on learning the correlation between different features in the early stages, while focusing on the global classification vector in the later stages. Finally, after feature fusion and classification, the output after passing through a softmax layer yields seven discrete categories of facial expressions.

[0051] In one possible implementation, the step of performing comparative learning on the pre-trained multi-view expression recognition model through intra-class and inter-class comparative learning to obtain a multi-view expression recognition model based on comparative learning includes: traversing each training image in the training dataset and performing comparative learning steps for each image: obtaining the attention weight map of the current training image through the multi-view expression recognition model; selecting regions with a high number of weights before a preset number from the attention weight map of the current training image as retained regions, and cropping and saving the retained regions from the training image; randomly selecting random training images from the training dataset, and pasting the retained regions cropped from the current training image into the corresponding positions in the random training images to obtain resampled training images; and obtaining the features of the current training image and the resampled training image based on the multi-view expression recognition model. The model employs the following steps: first, it calculates the feature vectors of the current training image and the resampled training image, and then performs inter-class contrastive learning on the current training image based on the similarity between the feature vectors of the current training image and the resampled training image. Next, it performs inter-class contrastive learning on the current training image based on the optimization objective of making the feature vectors of training images of the same class closer and the feature vectors of training images of different classes further apart, resulting in an inter-class contrastive learning loss for the current training image. Finally, it weights and sums the intra-class and inter-class contrastive learning losses of each training image to obtain a reweighted loss for the training dataset, and optimizes the multi-view expression recognition model based on this reweighted loss. These steps are iterated until a preset number of iterations is reached or the optimized multi-view expression recognition model meets preset requirements, resulting in a multi-view expression recognition model based on contrastive learning.

[0052] For details, see Figure 4 This paper demonstrates the contrastive learning network structure for training an imbalanced facial expression recognition model. The multi-view facial expression recognition model is used as the encoder in this contrastive learning network for feature learning, and four modules are added to it: resampling, intra-class contrastive learning, inter-class contrastive learning, and reweighted loss function.

[0053] The resampling module introduces the Attentive Cutmix method. First, the selected input image, i.e., the current training image, is processed by a pre-classification network model to obtain and output the attention feature map of the selected image. The pre-classification network model can be a pre-trained Mobile Face Net model, which is used to guide the stitching position of the input image and random images. Second, the regions with the highest weights are selected from the attention feature map and cropped from the input image as the retention regions. Finally, images are randomly selected from the dataset, i.e., random training images. The retention regions cropped from the input image are pasted into the corresponding positions in the random training images to obtain resampled training images that retain facial expression features to the greatest extent.

[0054] The intra-class contrastive learning module is used to compare resampled training images with the original input images to evaluate the quality of the resampled training images. After the encoder generates feature vectors for the two images, the cosine similarity between them is calculated to learn the similarity between the two images. At the same time, a threshold is set to distinguish between features with low and high similarity. Resampled training images with low feature similarity are discarded as poor-quality resampled data, and only high-quality resampled training images are used as valid samples for learning.

[0055] In one possible implementation, the intra-class contrastive learning loss

[0056]

[0057] Where λ is the preset weight parameter, CosSim() is the cosine similarity function, p1 and p2 are two randomly augmented views of the current training image, and t1 and t2 are two randomly augmented views of the resampled training image.

[0058] The main purpose of the inter-class contrastive learning module is to make the features of different images from the same category similar, while the features of images from different categories are dissimilar. Based on the category labels, the sample data in the same batch is divided: samples from the same category are treated as positive samples, and samples from other categories are treated as negative samples for contrastive learning.

[0059] In one possible implementation, the inter-class contrastive learning loss

[0060]

[0061] Where I ≡ {1, ..., N} is the training dataset, N is the total number of training images, i represents the current training image, and p i A(i) represents the feature vector obtained by the multi-view expression recognition model for the current training image; A(i) is the set of the remaining training images in the training dataset excluding the current training image. Let |T(i)| be the set of training images in the training dataset that belong to the same category as the current training image, and |T(i)| be the number of training images in T(i). a Let p be the feature vector obtained from the a-th training image through the multi-view expression recognition model. t Let be the feature vector obtained from the t-th training image using the multi-view facial expression recognition model.

[0062] The main purpose of the reweighted loss function is to make the model pay more attention to the training results of the minority class during the training process, so as to ensure that the negative emotions that teachers need to focus on are accurately identified without sacrificing the model's accuracy in the majority class. The reciprocal of the effective sample size is used as the weight to weight each class in the dataset, making the class division more neutral and maximizing the accuracy of minority class identification.

[0063] In one possible implementation, reweighted loss

[0064]

[0065] Among them, E n λ1 represents the number of valid samples out of n samples; β is a preset hyperparameter used to control the sampling degree of valid samples; λ1 and λ2 are preset weight parameters used to control the importance of intra-class contrastive learning and inter-class contrastive learning. For intra-class contrastive learning loss, The loss is for inter-class comparison learning.

[0066] In one possible implementation, to demonstrate the effectiveness of the proposed contrastive learning-based multi-view facial expression recognition model, facial expression recognition experiments were conducted using the proposed model and a series of classic algorithms. The experimental datasets were derived from internationally used public datasets RAF-DB and KDEF in facial expression recognition research. Furthermore, considering the specific characteristics of this research scenario, a private facial expression dataset (Online Learning Multiview Facial Expression Recognition, abbreviated as OL-MFER) for an online classroom scenario was constructed. Since facial expression recognition falls under the field of image classification, to evaluate the performance of different algorithms, the average accuracy of seven facial expression categories was used as the metric. The average accuracy in the experimental results represents the average value obtained after training the model multiple times.

[0067] Referring to Table 1, the model proposed in this invention outperforms existing facial expression recognition networks on the RAF-DB, KDEF, and OL-MFER datasets. The highest accuracy of the model is 0.9185 on the RAF-DB dataset, 0.9755 on the KDEF dataset, and 0.9348 on the OL-MFER dataset.

[0068] Table 1

[0069] Experimental model Model release time RAF-DB KDEF OL-MFER DACL 2021 0.8778 0.9663 0.8922 KTN 2021 0.8807 0.9684 0.8874 TransFER 2021 0.9091 0.9724 0.9099 CMCNN 2022 0.9036 0.9749 0.9157 DAN 2023 0.8970 0.9711 0.9044 This invention 2024 0.9185 0.9755 0.9348

[0070] Comparative experiments on public and private datasets fully demonstrate that the model of this invention, by combining Vision Transformer and CNN structures, improves the training accuracy of the model on the frontal facial expression dataset; at the same time, the introduction of key point features to assist in facial expression feature extraction improves the robustness of the model to multi-view problems.

[0071] In one possible implementation, to further illustrate the effectiveness of the proposed multi-view facial expression recognition model based on contrastive learning, the model was trained using a series of classic algorithms. The datasets used in the experiment were RAF-DB and OL-MFER. To measure the model's recognition performance across different categories, this implementation used the F1 score and G-mean as the main metrics to verify model performance, and the average value obtained after multiple training iterations was used as the experimental result. Simultaneously, to comprehensively consider the model compression effect, this implementation also selected metrics such as the number of model parameters and model training time as evaluation indicators.

[0072] Referring to Table 2, the model proposed in this invention achieved good performance on both datasets. On the RAF-DB dataset, it achieved 87.28% and 89.10% of the two evaluation metrics, respectively, and on the OL-MFER dataset, it achieved 85.80% and 88.03% of the two evaluation metrics, respectively. The comparative experiments fully demonstrate that the model's good performance is due to: the model resampling at the image level and filtering newly generated data through contrastive learning; simultaneously, it expands the boundary of the minority class samples in the feature space through contrastive learning, achieving clustering of similar classes and separation of different classes, thus improving the model's robustness on imbalanced datasets.

[0073] Table 2

[0074]

[0075]

[0076] Referring to Table 3, using the number of model parameters as a metric for computational cost and the training time as the time taken for one training cycle on the OL-MFER dataset, the changes in model complexity after adding each module are calculated. The results show that adding the intra-class contrastive learning module increases the number of model parameters by 2.01 times. However, since the intra-class contrastive learning module is only used by a minority of classes and is not generalized to the entire dataset, the training time only increases by 1.35 times. Furthermore, the imbalanced facial expression recognition model based on contrastive learning only has a large number of model parameters during the training phase; in practical applications, only the original model needs to be deployed.

[0077] Table 3

[0078] Resampling Intra-class contrastive learning Inter-class comparative learning Function reweighting Model parameter count Training time 75.127M 97s √ 75.127M 98s √ √ 151.370M 131s √ 75.288M 104s √ 75.127M 97s √ √ √ √ 151.531M 142s

[0079] The following are embodiments of the apparatus of the present invention, which can be used to execute embodiments of the method of the present invention. For details not disclosed in the apparatus embodiments, please refer to the embodiments of the method of the present invention.

[0080] See Figure 5 In another embodiment of the present invention, an online learner facial expression recognition system is provided, which can be used to implement the above-mentioned online learner facial expression recognition method. Specifically, the online learner facial expression recognition system includes an image acquisition module and an facial expression recognition module.

[0081] The image acquisition module is used to acquire static face images extracted from the learning videos of online learners to obtain the images to be recognized; the expression recognition module is used to input the images to be recognized into a preset multi-view expression recognition model based on contrastive learning to obtain the expression recognition results of online learners.

[0082] The multi-view expression recognition model based on contrastive learning is obtained as follows: the pre-trained multi-view expression recognition model is subjected to contrastive learning through intra-class and inter-class contrastive learning to obtain the multi-view expression recognition model based on contrastive learning. The multi-view expression recognition model includes a key point feature and expression feature fusion module, a multi-scale feature fusion module, a classification vector embedding module, and a classifier, which are connected sequentially. The key point feature and expression feature fusion module extracts key point features and expression features from the image to be recognized and performs feature fusion to obtain a fused feature map. The multi-scale feature fusion module performs multi-scale feature fusion on the fused feature map to obtain multi-scale fused expression features. The classification vector embedding module embeds classification vectors into the multi-scale fused feature map to obtain learned classification vectors. The classifier obtains the expression recognition result based on the learned classification vectors.

[0083] All relevant content of each step involved in the aforementioned embodiments of the online learner facial expression recognition method can be referenced to the functional description of the corresponding functional module of the online learner facial expression recognition system in the embodiments of the present invention, and will not be repeated here.

[0084] The module division in this embodiment of the invention is illustrative and represents only one logical functional division. In actual implementation, other division methods may be used. Furthermore, the functional modules in the various embodiments of the invention can be integrated into a single processor, exist as separate physical entities, or be integrated into a single module. The integrated modules described above can be implemented in hardware or as software functional modules.

[0085] In another embodiment of the present invention, a computer device is provided, comprising a processor and a memory. The memory stores a computer program, which includes program instructions. The processor executes the program instructions stored in the computer storage medium. The processor may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing and control core of the terminal, suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions from the computer storage medium to achieve a corresponding method flow or function. The processor described in this embodiment of the present invention can be used for the operation of an online learner facial expression recognition method.

[0086] In another embodiment of the present invention, a storage medium is provided, specifically a computer-readable storage medium (Memory), which is a memory device in a computer device used to store programs and data. It is understood that the computer-readable storage medium here can include both the built-in storage medium in the computer device and extended storage media supported by the computer device. The computer-readable storage medium provides storage space that stores the terminal's operating system. Furthermore, the storage space also stores one or more instructions suitable for loading and execution by a processor. These instructions can be one or more computer programs (including program code). It should be noted that the computer-readable storage medium here can be high-speed RAM or non-volatile memory, such as at least one disk storage device. The processor can load and execute one or more instructions stored in the computer-readable storage medium to implement the corresponding steps of the online learner facial expression recognition method in the above embodiments.

[0087] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0088] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0089] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0090] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0091] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A method for recognizing facial expressions of online learners, characterized in that, include: Obtain static face images extracted from online learners' learning videos to obtain the image to be recognized; The image to be recognized is input into a preset multi-view facial expression recognition model based on contrastive learning to obtain the facial expression recognition results of the online learner; The multi-view facial expression recognition model based on contrastive learning is obtained in the following way: A pre-trained multi-view facial expression recognition model is subjected to comparative learning through intra-class and inter-class contrastive learning to obtain a contrastive learning-based multi-view facial expression recognition model. This model comprises a keypoint feature and facial expression feature fusion module, a multi-scale feature fusion module, a classification vector embedding module, and a classifier, all connected sequentially. The keypoint feature and facial expression feature fusion module extracts keypoint and facial expression features from the image to be recognized and fuses them to obtain a fused feature map. The multi-scale feature fusion module performs multi-scale feature fusion on the fused feature map to obtain a multi-scale fused feature map. The classification vector embedding module embeds classification vectors into the multi-scale fused feature map to obtain a learned classification vector. The classifier then uses the learned classification vector to obtain the facial expression recognition result.

2. The online learner facial expression recognition method according to claim 1, characterized in that, The key point feature and facial expression feature fusion module is specifically used for: The IR50 model, pre-trained on a face recognition dataset, is used to extract facial expression features from the images to be recognized; and the Mobile Face Net model, pre-trained, is used to extract key point features from the images to be recognized. A linear transformation is performed on the key point features and expression features, mapping the key point features to the query vector and the expression features to the key vector and value vector respectively; The attention score is calculated by the inner product of the query vector and the key vector, and the attention score is scaled and normalized to obtain the attention weight map; the attention weight map and the value vector are weighted and summed to obtain the fused feature map.

3. The online learner facial expression recognition method according to claim 1, characterized in that, The multi-scale feature fusion module includes several consecutive downsampling and upsampling modules, and uses skip connections to connect the feature maps in the downsampling and upsampling modules at the corresponding levels.

4. The online learner facial expression recognition method according to claim 1, characterized in that, The classification vector embedding module is specifically used for: The classification vector is embedded into a multi-scale fusion feature map, and the classification vector learns the multi-scale fusion facial expression features of the multi-scale fusion feature map through two multi-head self-attention layers to obtain the learned classification vector.

5. The online learner facial expression recognition method according to claim 1, characterized in that, The step of performing comparative learning on the pre-trained multi-view facial expression recognition model through intra-class and inter-class comparative learning to obtain a multi-view facial expression recognition model based on comparative learning includes: The process iterates through each training image in the training dataset and performs comparative learning steps: First, the attention weight map of the current training image is obtained using a multi-view expression recognition model. Then, regions with a high number of pre-defined weights are selected from the attention weight map of the current training image as retained regions, and these regions are cropped and saved from the training image. Next, random training images are randomly selected from the training dataset, and the cropped regions from the current training image are pasted onto their corresponding positions in the random training image to obtain resampled training images. Feature vectors of the current training image and the resampled training image are obtained based on the multi-view expression recognition model. The intra-class contrastive learning loss of the current training image is obtained based on the similarity between the feature vectors of the current training image and the resampled training image. Finally, based on the optimization objective of making the feature vectors of training images of the same category closer and the feature vectors of training images of different categories further apart, inter-class contrastive learning of the multi-view expression recognition model is performed on the current training image and the training dataset to obtain the inter-class contrastive learning loss of the current training image. The intra-class contrast learning loss and inter-class contrast learning loss of each training image are weighted and superimposed to obtain the reweighted loss of the training dataset, and the multi-view expression recognition model is optimized based on the reweighted loss. The above steps are repeated iteratively until the preset number of iterations is reached or the optimized multi-view expression recognition model meets the preset requirements, thus obtaining a multi-view expression recognition model based on contrastive learning.

6. The online learner facial expression recognition method according to claim 5, characterized in that, The intra-class contrastive learning loss for obtaining the current training image includes: The intra-class contrast learning loss for the current training image is obtained as follows. Where λ is the preset weight parameter, CosSim() is the cosine similarity function, p1 and p2 are two randomly augmented views of the current training image, and t1 and t2 are two randomly augmented views of the resampled training image; The inter-class contrast learning loss for obtaining the current training image includes: obtaining the inter-class contrast learning loss for the current training image through the following... Where I ≡ {1, ..., N} is the training dataset, N is the total number of training images, i represents the current training image, and p i A(i) represents the feature vector obtained by the multi-view expression recognition model for the current training image; A(i) is the set of the remaining training images in the training dataset excluding the current training image. Let |T(i)| be the set of training images in the training dataset that belong to the same category as the current training image, and |T(i)| be the number of training images in T(i). a Let p be the feature vector obtained from the a-th training image through the multi-view expression recognition model. t Let be the feature vector obtained from the t-th training image using the multi-view facial expression recognition model.

7. The online learner facial expression recognition method according to claim 5, characterized in that, The reweighted loss for obtaining the training dataset includes: The reweighted loss of the training dataset is calculated using the following formula. Among them, E n λ is the number of valid samples out of n samples; β is a preset hyperparameter; λ1 and λ2 are both preset weight parameters; For intra-class contrastive learning loss, The loss is for inter-class comparison learning.

8. An online learner facial expression recognition system, characterized in that, include: The image acquisition module is used to acquire static face images extracted from online learners' learning videos to obtain the image to be recognized; The facial expression recognition module is used to input the image to be recognized into a preset multi-view facial expression recognition model based on contrastive learning to obtain the facial expression recognition results of the online learner; The multi-view facial expression recognition model based on contrastive learning is obtained in the following way: A pre-trained multi-view expression recognition model is subjected to comparative learning through intra-class and inter-class contrastive learning to obtain a contrastive learning-based multi-view expression recognition model. This model comprises a keypoint feature and expression feature fusion module, a multi-scale feature fusion module, a classification vector embedding module, and a classifier, all connected sequentially. The keypoint feature and expression feature fusion module extracts keypoint and expression features from the image to be recognized and fuses them to obtain a fused feature map. The multi-scale feature fusion module performs multi-scale feature fusion on the fused feature map to obtain multi-scale fused expression features. The classification vector embedding module embeds classification vectors into the multi-scale fused feature map to obtain learned classification vectors. The classifier uses the learned classification vectors to obtain the expression recognition result.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the online learner facial expression recognition method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the online learner facial expression recognition method as described in any one of claims 1 to 7.

Images