Online learner lightweight expression recognition method and related apparatus

Abstract

This invention belongs to the field of facial expression recognition and discloses a lightweight facial expression recognition method and related apparatus for online learners. The method includes: acquiring an image of a facial expression to be recognized from an online learner; inputting the image of the facial expression to be recognized into a pre-trained lightweight facial expression recognition model to obtain the facial expression recognition result of the online learner. The pre-trained lightweight facial expression recognition model achieves accurate facial expression recognition for online learners. It compresses the pre-trained occluded facial expression recognition model using a multi-layer knowledge distillation method, then further reduces the model's computational load by combining depthwise separable convolution, and finally uses model quantization to improve the model's running speed, thus obtaining a more lightweight occluded facial expression recognition model. The occluded facial expression recognition model is constructed using a feature extraction module, an attention branch fusion module, and an output layer module. By fusing several spatial attention branch modules, channel random occlusion modules, feature pyramid network modules, and fusion modules, the model's recognition accuracy under occlusion conditions can be further improved.

Description

Technical Field

[0001] This invention belongs to the field of facial expression recognition, and relates to a lightweight facial expression recognition method and related device for online learners. Background Technology

[0002] Facial expression recognition has wide applications, especially in education, where it holds immense promise. Facial expressions are one of the most effective, widespread, and natural ways for humans to convey emotional states and intentions. Facial expression recognition analyzes the features of facial expressions, determines their emotional category, and based on the results, can predict human behavior, infer psychological activities, and thus make appropriate behavioral responses.

[0003] Most students primarily learn online via mobile phones, and the learning environment is often inconsistent, making occlusion a common issue. Furthermore, the hardware performance of mobile devices is relatively limited, so the deployment of facial expression recognition models needs to consider the practical requirement of lightweight algorithm networks. However, current online learner facial expression recognition methods suffer from diverse types and varying regions of facial image occlusion, leading to low accuracy. Additionally, existing facial expression recognition models are quite complex, making effective deployment difficult. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings of the prior art and provide a lightweight facial expression recognition method and related device for online learners.

[0005] To achieve the above objectives, the present invention employs the following technical solution:

[0006] In a first aspect, the present invention provides a lightweight facial expression recognition method for online learners, comprising: acquiring a facial expression image to be recognized from an online learner; inputting the facial expression image to be recognized into a pre-trained lightweight facial expression recognition model to obtain the facial expression recognition result of the online learner; wherein the pre-trained lightweight facial expression recognition model is obtained by: acquiring a pre-trained occluded facial expression recognition model, compressing the pre-trained occluded facial expression recognition model using a multi-layer knowledge distillation method, and combining depthwise separable convolution to reduce the computational load of the model, and quantizing the compressed occluded facial expression recognition model to obtain the pre-trained lightweight facial expression recognition model; wherein the occluded facial expression recognition model includes a feature extraction module, an attention branch fusion module, and an output layer module, the attention branch fusion module includes several spatial attention branch modules, a channel random occlusion module, a feature pyramid network module, and a fusion module, the input ends of several spatial attention branch modules are all connected to the output ends of the feature extraction module, the output ends of several spatial attention branch modules are all connected to the input ends of the fusion module through the channel random occlusion module, the input ends of the feature pyramid network module are connected to the output ends of the feature extraction module, and the output ends of the feature pyramid network module are connected to the input ends of the fusion module.

[0007] Optionally, obtaining the pre-trained occluded expression recognition model includes: acquiring several facial expression images and constructing an occluded expression recognition model; aligning the several facial expression images using an MTCNN network, and then performing data augmentation operations to obtain several training images; wherein, the data augmentation operations include the following operations: random image cropping, random horizontal flipping, random vertical flipping, and histogram equalization; and training the occluded expression recognition model using the several training images to obtain the pre-trained occluded expression recognition model.

[0008] Optionally, the feature extraction module is constructed using the first three main feature extraction networks in a face recognition network pre-trained on a face recognition dataset; the spatial attention branch module consists of two convolutional layers. The feature map output by the feature extraction module is reduced in dimensionality by the first convolutional layer to obtain an intermediate feature map, then subjected to a ReLU activation function to add non-linear changes, and then subjected to a second convolutional layer for dimensionality reduction and a Sigmoid activation function to obtain a spatial attention map; the channel random masking module consists of a channel masking operation layer, a maximum value layer, and an average value layer. The spatial attention map is randomly reduced by a preset factor by the channel masking operation layer, then the maximum value is obtained by the maximum value layer, and then the average value layer is obtained by the average value layer to obtain the attention map; the feature pyramid network module consists of a downsampling layer and an upsampling layer. The feature map output by the feature extraction module is subjected to a downsampling layer to obtain four feature maps of different scales, then subjected to two 1*1 convolution operations to increase the dimensionality, and then subjected to an upsampling layer before being concatenated and fused into a multi-scale feature map.

[0009] Optionally, the facial expression recognition result of the online learner can be: angry, disgusted, afraid, happy, sad, surprised, or neutral.

[0010] Optionally, the step of compressing the pre-trained occluded expression recognition model using a multi-layer knowledge distillation method, combining it with depthwise separable convolution to reduce model computation, and quantizing the compressed occluded expression recognition model to obtain a pre-trained lightweight expression recognition model includes: using the pre-trained occluded expression recognition model as the teacher model in the knowledge distillation network, reducing the stacking number of the backbone CNN network to serve as the student network; performing knowledge distillation training based on a predefined knowledge guidance path; wherein, knowledge distillation training includes intermediate layer knowledge distillation and output layer knowledge distillation; during knowledge distillation training, extracting knowledge from the teacher network and student network through a spatial attention network, and fusing the feature maps of multiple layers of the teacher network using a multi-layer attention fusion network, ensuring that the fused feature map is the same size as the feature map of the student network at the same level; replacing the backbone network in the student network with a lightweight network MobileNetV2; and quantizing the final student network using a model quantization method to obtain the pre-trained lightweight expression recognition model.

[0011] Optionally, the loss function for knowledge distillation training is the sum of the intermediate layer knowledge distillation loss and the output layer distillation loss; the intermediate layer knowledge distillation loss is:

[0012]

[0013] L mid =L AT1 +λ1L AT2 +λ2L AT3

[0014] L kd =L distill +βL mid

[0015] Among them, L AT L represents the attention loss after the fusion of teacher network features and the network of students at the same level. AT1 L AT2 and L AT3 These represent different feature extraction layers; This represents the feature matrix output by the teacher model at layer j. This represents the feature matrix output by the student model at layer j; λ1, λ2, and β represent the balance coefficients; L distill L is the output layer distillation loss function; mid L represents the intermediate layer distillation loss; I represents the number of layers in the model; L represents the intermediate layer distillation loss. kd This is the total distillation loss function.

[0016] The output layer distillation loss is:

[0017]

[0018] L distill =αL soft +(1-α)L hard

[0019] in, Represents the probability distribution of the teacher model. c represents the probability distribution of the student model. j Represents the true label, This represents the case where temperature T is 1, α is the equilibrium factor, and L soft L represents the loss after distillation of the student model and the teacher model. hard L represents the loss between the student model's predicted values ​​and the true label values. distill Z represents the final total loss from knowledge distillation; N is the total number of samples; z i z is the logit output of the teacher model for the i-th class; j z is the logit output of the student model for the j-th class;k This is the logit output for the student model for the k-th class.

[0020] Optionally, the loss function of the occluded expression recognition model is:

[0021]

[0022] L cls =L s +λ 11 L intra +β 11 L inter

[0023] Among them, L s This represents the classic softmax loss function, where m is the total number of samples and n is the number of classes. The predicted probability of the sample not being normalized; Here is the weight matrix; x i b is the feature vector of sample i; n For bias terms; Let b be the weight matrix corresponding to category j; j For the bias term corresponding to category j; x i Let L be the feature vector of the i-th sample; intra L represents the distance between a sample and the cluster center of a feature cluster; inter The distance between the cluster centers of different samples; It is category y i Feature center; c k and c j These are the centers of categories k and j, respectively; L cls λ represents the final loss function. 11 and β 11 This represents the balance coefficient.

[0024] In a second aspect, the present invention provides a lightweight facial expression recognition system for online learners, comprising: an image acquisition module for acquiring an image of a facial expression to be recognized from an online learner; and an expression recognition module for inputting the image of the facial expression to be recognized into a pre-trained lightweight expression recognition model to obtain the expression recognition result of the online learner; wherein the pre-trained lightweight expression recognition model is obtained by: acquiring a pre-trained occluded expression recognition model, compressing the pre-trained occluded expression recognition model using a multi-layer knowledge distillation method, and combining depthwise separable convolution to reduce the computational load of the model, and quantizing the compressed occluded expression recognition model to obtain the pre-trained lightweight expression recognition model; wherein the occluded expression recognition model includes a feature extraction module, an attention branch fusion module, and an output layer module, the attention branch fusion module including several spatial attention branch modules, a channel random occlusion module, a feature pyramid network module, and a fusion module, the input ends of several spatial attention branch modules are all connected to the output ends of the feature extraction module, the output ends of several spatial attention branch modules are all connected to the input ends of the fusion module through the channel random occlusion module, the input ends of the feature pyramid network module are connected to the output ends of the feature extraction module, and the output ends of the feature pyramid network module are connected to the input ends of the fusion module.

[0025] In a third aspect, the present invention provides a computer device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the steps of the above-described lightweight facial expression recognition method for online learners.

[0026] In a fourth aspect, the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the above-described lightweight facial expression recognition method for online learners.

[0027] Compared with the prior art, the present invention has the following beneficial effects:

[0028] This invention presents a lightweight facial expression recognition method for online learners, achieving accurate facial expression recognition for online learners based on a pre-trained lightweight facial expression recognition model. The pre-trained lightweight facial expression recognition model is derived from a pre-trained occluded facial expression recognition model. First, the pre-trained occluded facial expression recognition model is compressed using a multi-layer knowledge distillation method. Then, depthwise separable convolution is combined to further reduce the model's computational load. Finally, model quantization is used to improve the model's running speed, resulting in a more lightweight occluded facial expression recognition model. The occluded facial expression recognition model is constructed using a feature extraction module, an attention branch fusion module, and an output layer module, while incorporating several spatial attention branch modules, channel random occlusion modules, feature pyramid network modules, and fusion modules. Several spatial attention branch modules extract local facial features, and the random channel occlusion module improves robustness to occlusion. Simultaneously, the feature pyramid network module extracts global multi-scale features. Facial expression classification is achieved through the fusion of local and global features, thereby further improving the model's recognition accuracy under occlusion conditions. Attached Figure Description

[0029] Figure 1 This is a flowchart of a lightweight facial expression recognition method for online learners according to an embodiment of the present invention.

[0030] Figure 2 This is a schematic diagram of an occluded facial expression recognition model according to an embodiment of the present invention.

[0031] Figure 3 This is a schematic diagram of the feature pyramid network module in an embodiment of the present invention.

[0032] Figure 4 This is a schematic diagram of the spatial attention branch module in an embodiment of the present invention.

[0033] Figure 5 This is a schematic diagram of the channel random masking module according to an embodiment of the present invention.

[0034] Figure 6 This is a schematic diagram illustrating the principle of the multilayer knowledge distillation method according to an embodiment of the present invention.

[0035] Figure 7 This is a schematic diagram of a multi-layer attention fusion network according to an embodiment of the present invention.

[0036] Figure 8 This is a schematic diagram of a spatial attention network according to an embodiment of the present invention.

[0037] Figure 9 This is a schematic diagram of the Knowledge Guidance Path (KGP) in an embodiment of the present invention.

[0038] Figure 10 This is a block diagram of a lightweight facial expression recognition system for online learners, as described in an embodiment of the present invention. Detailed Implementation

[0039] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0040] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0041] The present invention will now be described in further detail with reference to the accompanying drawings:

[0042] See Figure 1 In one embodiment of the present invention, a lightweight facial expression recognition method for online learners is provided, which can effectively improve the accuracy of facial expression recognition for online learners and reduce the size of the facial expression recognition model.

[0043] Specifically, the lightweight facial expression recognition method for online learners of this invention includes the following steps:

[0044] S1: Obtain the facial expression image of the online learner to be recognized.

[0045] S2: Input the image of the expression to be recognized into a pre-trained lightweight expression recognition model to obtain the expression recognition result of the online learner.

[0046] The pre-trained lightweight facial expression recognition model is obtained in the following way:

[0047] A pre-trained occluded facial expression recognition model is obtained. This model is then compressed using a multi-layer knowledge distillation method, and depthwise separable convolution is combined to reduce computational complexity. Finally, the compressed model is quantized to obtain a lightweight pre-trained facial expression recognition model. The occluded facial expression recognition model includes a feature extraction module, an attention branch fusion module, and an output layer module. The attention branch fusion module comprises several spatial attention branch modules, a channel random occlusion module, a feature pyramid network module, and a fusion module. The inputs of the spatial attention branch modules are all connected to the output of the feature extraction module, and the outputs are all connected to the input of the fusion module through the channel random occlusion module. The input of the feature pyramid network module is connected to the output of the feature extraction module, and its output is connected to the input of the fusion module.

[0048] This invention presents a lightweight facial expression recognition method for online learners, achieving accurate facial expression recognition for online learners based on a pre-trained lightweight facial expression recognition model. The pre-trained lightweight facial expression recognition model is derived from a pre-trained occluded facial expression recognition model. First, the pre-trained occluded facial expression recognition model is compressed using a multi-layer knowledge distillation method. Then, depthwise separable convolution is combined to further reduce the model's computational load. Finally, model quantization is used to improve the model's running speed, resulting in a more lightweight occluded facial expression recognition model. The occluded facial expression recognition model is constructed using a feature extraction module, an attention branch fusion module, and an output layer module. It integrates several spatial attention branch modules, channel random occlusion modules, feature pyramid network modules, and fusion modules. The spatial attention branch modules extract local facial features, and the random channel occlusion module improves robustness to occlusion. Simultaneously, the feature pyramid network module extracts global multi-scale features. Facial expression classification is achieved through the fusion of local and global features, thereby further improving the model's recognition accuracy under occlusion conditions.

[0049] In one possible implementation, obtaining the pre-trained occluded facial expression recognition model includes:

[0050] Acquire several facial expression images and construct an occluded expression recognition model; perform face alignment on the several facial expression images using the MTCNN network, and then perform data augmentation operations to obtain several training images; wherein, the data augmentation operations include one or more of the following: random image cropping, random horizontal flipping, random vertical flipping, and histogram equalization; use the several training images to train the occluded expression recognition model to obtain a pre-trained occluded expression recognition model.

[0051] Specifically, the facial expression images include seven discrete categories of facial expressions: anger, disgust, fear, happiness, sadness, surprise, and neutral. Still images extracted from online learners' learning videos can be used as facial expression images.

[0052] In one possible implementation, see Figures 2 to 5 The feature extraction module is constructed using the first three main feature extraction networks in a face recognition network pre-trained on a face recognition dataset. The spatial attention branch module consists of two convolutional layers. The feature map output by the feature extraction module is reduced in dimensionality by the first convolutional layer to obtain an intermediate feature map, then subjected to a ReLU activation function to add non-linear changes, and then subjected to a second convolutional layer for dimensionality reduction and a Sigmoid activation function to obtain a spatial attention map. The channel random masking module consists of a channel masking operation layer, a maximum value layer, and an average value layer. The spatial attention map is randomly reduced by a preset factor by the channel masking operation layer, then the maximum value is obtained by the maximum value layer, and then the average value layer is obtained by the average value layer to obtain the attention map. The feature pyramid network module consists of a downsampling layer and an upsampling layer. The feature map output by the feature extraction module is processed by the downsampling layer to obtain four feature maps of different scales, then each is subjected to two 1*1 convolution operations for dimensionality increase, and then the feature maps are spliced ​​and fused into a multi-scale feature map after being processed by the upsampling layer.

[0053] Specifically, after acquiring the facial expression image, the MTCNN network is used for face alignment, followed by data augmentation operations, mainly using random image cropping, random horizontal flipping, random vertical flipping, and histogram equalization. At this point, the facial expression image is represented as 224*224*3.

[0054] Transfer learning was performed using a model pre-trained on a face recognition dataset, primarily utilizing the first three main feature extraction networks in the face recognition network, with an output feature size of 28*28*1024.

[0055] The spatial attention branch module SSNet mainly consists of two convolutions. The input feature map is of size h*w*c. After 1*1 convolution in the first convolutional layer, the intermediate feature map with size h*w*c / r is obtained by dimensionality reduction. Then, it is activated by ReLU to add non-linear changes. Finally, it is activated by 1*1 convolution in the second convolutional layer and activated by Sigmoid to obtain the spatial attention map h*w*1.

[0056] The channel random masking module RCMask consists of channel masking operations and maximum and average value operations. The input feature map is a feature map extracted by a multi-branch SSNet with a size of h*w*n. After the channel masking operation, the value of a certain channel is randomly reduced by a factor of p to reduce the importance of that channel. The feature map size is still h*w*n. Then, the maximum and average value operations are used to extract the final attention map with a size of h*w*1.

[0057] The Feature Pyramid Network module PNet consists of downsampling layers and upsampling layers. The input feature map is of size h*w*c. After downsampling, four feature maps of different scales are obtained, with the number of channels reduced to one-quarter of the original. These are then upsampled by two 1*1 convolution operations, and finally upsampled and concatenated to form the final multi-scale feature map of size h*w*c, keeping the input and output feature map sizes unchanged.

[0058] After the attention branch fusion module fuses local and global features, it outputs the results through the softmax output layer module to obtain seven discrete categories of facial expressions.

[0059] In one possible implementation, see Figures 6 to 8 This paper describes a method to compress a pre-trained occluded facial expression recognition model using multi-layer knowledge distillation, and combines this with depthwise separable convolution to reduce computational cost. The compressed occluded facial expression recognition model is then quantized to obtain a pre-trained lightweight facial expression recognition model. This process involves: using the pre-trained occluded facial expression recognition model as the teacher model in the knowledge distillation network, and reducing the stacking number of the backbone CNN network to serve as the student network; performing knowledge distillation training based on a predefined knowledge guidance path; including intermediate layer knowledge distillation and output layer knowledge distillation; extracting knowledge from the teacher and student networks using a spatial attention network during training, and fusing the feature maps of multiple layers of the teacher network using a multi-layer attention fusion network, ensuring that the fused feature maps are the same size as the feature maps of the student networks at the same level; replacing the backbone network in the student network with a lightweight MobileNetV2 network; and quantizing the final student network using a model quantization method to obtain the pre-trained lightweight facial expression recognition model.

[0060] Specifically, knowledge distillation primarily employs the Knowledge Guidance Path (KGP), the Spatial Attention Network (SANet), and the Multilayer Attention Fusion Network (MFF). In the intermediate layer knowledge distillation, the KGP is defined, allowing the knowledge from the multilayer teacher network to guide the training of the student network, enabling the teacher network's previous knowledge to also help improve the student network's performance. The Spatial Attention Network (SANet) aims to extract knowledge from both the teacher and student networks. It extracts richer attention features through a dual-branch approach. The input feature map is h*w*c, and after max pooling and average pooling, two feature maps of size h*w*1 are obtained. These are then concatenated to obtain a feature map of size h*w*2, which is then convolved and activated to obtain the attention map of size h*w*1. The Multilayer Attention Fusion Network (MFF) aims to fuse features from multiple layers of the teacher network while reducing the model's computational cost. The input feature maps are multiple feature maps of varying sizes from the intermediate layers of the teacher network. After convolution, each feature map is converted to a feature map of size h*w*c, consistent with the size of the student network's feature maps at the same level. These feature maps are then concatenated and convolved to obtain the fused feature map of size h*w*c. The backbone network in the student network was replaced with the lightweight MobileNetV2 network, reducing the computational cost of the model. Finally, model quantization was used to accelerate model inference, improving the model's compression and speedup rates.

[0061] Previous knowledge distillation methods have mostly focused on distilling knowledge within the same intermediate layer or only using the knowledge of the last intermediate layer of the teacher network. These studies have neglected the contributions of knowledge from other layers of the teacher network. Intermediate layer knowledge distillation defines a Knowledge Guidance Path (KGP), allowing knowledge from multiple layers of the teacher network to guide student network training, enabling previous knowledge from the teacher network to also help improve the performance of the student network. For details, see [link to documentation]. Figure 9 Teacher networks at the same and previous levels are used to guide students' network training.

[0062] In one possible implementation, the loss function for the knowledge distillation training is the sum of the intermediate layer knowledge distillation loss and the output layer distillation loss.

[0063] The knowledge distillation loss in the intermediate layer is:

[0064]

[0065] L mid =L AT1 +λ1L AT2 +λ2L AT3

[0066] L kd =L distill +βL mid

[0067] Among them, L AT L represents the attention loss after the fusion of teacher network features and the network of students at the same level. AT1 L AT2 and L AT3 These represent different feature extraction layers. In this embodiment, three main feature extraction layers are set during training, namely L... AT1 L AT2 and L AT3 . This represents the feature matrix output by the teacher model at layer j. This represents the feature matrix output by the student model at layer j; λ1, λ2, and β represent the balance coefficients; L distill L is the output layer distillation loss function; mid L represents the intermediate layer distillation loss; I represents the number of layers in the model; L represents the intermediate layer distillation loss. kd This is the total distillation loss function.

[0068] The output layer distillation loss is:

[0069]

[0070] L distill =αL soft +(1-α)L hard

[0071] in, Represents the probability distribution of the teacher model. c represents the probability distribution of the student model. j Represents the true label, This represents the case where temperature T is 1, α is the equilibrium factor, and L soft L represents the loss after distillation of the student model and the teacher model. hard L represents the loss between the student model's predicted values ​​and the true label values. distill Z represents the final total loss from knowledge distillation; N is the total number of samples; z i z is the logit output of the teacher model for the i-th class; j z is the logit output of the student model for the j-th class; k This is the logit output for the student model for the k-th class.

[0072] In one possible implementation, the loss function of the occluded facial expression recognition model is:

[0073]

[0074] L cls =L s +λ 11 L intra +β11 L inter

[0075] Among them, L s This represents the classic softmax loss function, where m is the total number of samples and n is the number of classes. The predicted probability of the sample not being normalized; Here is the weight matrix; x i b is the feature vector of sample i; n For bias terms; Let b be the weight matrix corresponding to category j; j For the bias term corresponding to category j; x i Let L be the feature vector of the i-th sample. intra L represents the distance between a sample and the cluster center of a feature group, used to constrain intra-class distances. inter The distance between the cluster centers of different samples is used to constrain the inter-class distance. It is mainly achieved by calculating the cosine distance between the cluster centers. The +1 is to change the value range to [0, 2]. The closer it is to 0, the greater the difference between the classes. It is category y i The feature center. It is category y i Feature center; c k and c j These are the centers of categories k and j, respectively. N is the total number of samples. L cls λ represents the final loss function. 11 and β 11 This represents the balance coefficient, which takes values ​​of 0.01 and 0.1 in actual code training.

[0076] In one possible implementation, to compare the effectiveness of the lightweight facial expression recognition method for online learners proposed in this invention, facial expression recognition experiments were conducted using this method against a series of classic algorithms. The experimental datasets were derived from the internationally used public datasets JAFFE and RAF-DB for facial expression recognition research. Additionally, considering the specific characteristics of the research scenario, a private facial expression dataset for an online classroom scenario (Online Learning FER, abbreviated as OLFER) was constructed. Since facial expression recognition falls under the field of image classification, to measure the performance of different algorithms, the average accuracy of seven facial expression categories was used as the evaluation metric. The average accuracy in the experimental results represents the average value obtained after training the model multiple times.

[0077] The experimental results are shown in Table 1:

[0078] Table 1

[0079] Model JAFFE (mask) RAF-DB(mask) OLFER R50 0.756 0.762 0.805 SENet 0.805 0.788 0.817 CBAM 0.829 0.790 0.839 GAM 0.854 0.794 0.845 gACNN 0.829 0.791 0.842 SCAN 0.902 0.801 0.856 PVS 0.878 0.798 0.849 AFNet 0.926 0.803 0.859

[0080] Among them, JAFFE(mask) is obtained by occlusion processing of the JAFFE dataset, RAF-DB(mask) is obtained by occlusion processing of the RAF-DB dataset, and OLFER is a separately constructed expression recognition dataset for online learning scenarios, which already has real occlusion.

[0081] As shown in Table 1, the pre-trained lightweight facial expression recognition model proposed in this invention outperforms the compared attention network and occluded facial expression recognition network on the JAFFE(mask), RAF-DB(mask), and OLFER datasets. Compared to the baseline model (R50), the pre-trained lightweight facial expression recognition model of this invention has significant advantages on the JAFFE(mask), RAF-DB(mask), and OLFER datasets. The highest accuracy of the pre-trained lightweight facial expression recognition model of this invention is 0.926 on the JAFFE(mask) dataset, an improvement of 22.5%; the highest accuracy is 0.803 on the RAF-DB(mask) dataset, an improvement of 5.4%; and the highest accuracy is 0.859 on the OLFER dataset, an improvement of 6.7%.

[0082] Comparative experiments on public and private datasets fully demonstrate the effectiveness of the lightweight facial expression recognition model pre-trained in this invention, while also improving the model's robustness to simulated and real occlusion.

[0083] In one possible implementation, to demonstrate the effectiveness of the MKD compression network method for the pre-trained lightweight facial expression recognition model proposed in this invention, facial expression recognition model compression experiments were conducted using the proposed pre-trained lightweight facial expression recognition model and a series of classic algorithms. The datasets used in the experiments were RAF-DB(mask) and OLFER. To measure the performance of different knowledge distillation algorithms, the average accuracy of seven facial expression recognition classes was used as the main evaluation metric, and the average value obtained after multiple training iterations was taken as the experimental result. Furthermore, to comprehensively consider the model compression effect, this implementation also selected metrics such as the number of model parameters, model computational cost, model memory usage, parameter compression ratio, and model acceleration rate as evaluation indicators.

[0084] The experimental results are shown in Tables 2 to 4.

[0085] Table 2

[0086]

[0087] In Table 2, the numbers in the MKD+KD column represent the best performance achieved by the algorithm on this dataset. Both the teacher and student models in the table are baseline+AFNet models, omitting redundant AFNet implementations for data simplicity. R50 and R18 are abbreviations for ResNet50 and ResNet18, respectively. MKD is the compressed network method for the pre-trained lightweight facial expression recognition model proposed in this invention. KD, FitNet, AT, PKT, CC, and SimKD are classic knowledge distillation algorithms used for comparison.

[0088] Referring to Table 2, when the baseline networks have similar structures (i.e., isomorphic knowledge distillation), the MKD method outperforms other comparative methods. When the teacher network is R50+AFNet and the student network is R18+AFNet, the accuracy of the ordinary AT knowledge distillation method is 0.787, the MKD method is 0.795, and the MKD+KD method is 0.799, representing a 3.1% improvement in accuracy compared to the original directly trained student network. When the baseline networks have dissimilar structures (i.e., heterogeneous knowledge distillation), the MKD method also outperforms other state-of-the-art methods. When the teacher network is R50+AFNet and the student network is MobileNetV2+AFNet, the accuracy of the ordinary AT knowledge distillation method is 0.788, the MKD method is 0.804, and the MKD+KD method is 0.806, representing a 5.2% improvement in accuracy compared to the original directly trained student network.

[0089] Table 3

[0090]

[0091] Referring to Table 3, when the baseline networks have similar structures (i.e., isomorphic knowledge distillation), the MKD method outperforms other comparative methods. When the teacher network is R50+AFNet and the student network is R18+AFNet, the accuracy of the ordinary AT knowledge distillation method is 0.838, the MKD method is 0.852, and the MKD+KD method is 0.854, representing a 3.8% improvement in accuracy compared to the original directly trained student network. When the baseline networks have dissimilar structures (i.e., heterogeneous knowledge distillation), the MKD method also outperforms other state-of-the-art methods. When the teacher network is R50+AFNet and the student network is MobileNetV2+AFNet, the accuracy of the ordinary AT knowledge distillation method is 0.836, the MKD method is 0.849, and the MKD+KD method is 0.852, representing a 4.9% improvement in accuracy compared to the original directly trained student network.

[0092] Table 4

[0093] Model accuracy computational load Parameters Reasoning time Memory Compression ratio Acceleration rate Initial model 0.859 3.86G 25.86M 231ms 104.12M 1.00× 1.00× Lightweight model + distillation 0.871 0.32G 3.62M 145ms 3.55M 7.14× 1.59× Lightweight model + distillation + quantization 0.862 0.32G 3.62M 86ms 0.73M 7.14× 2.69×

[0094] Referring to Table 4, the initial model had a computational cost of 3.86G, 25.86M parameters, and an image inference time of 231ms. After lightweighting and knowledge distillation, the computational cost was significantly reduced to only 0.32G, the number of parameters to only 3.62G, the image inference time to 145ms, the parameter compression rate to 7.14, and the model acceleration rate to 1.59. After further model quantization, the image inference time was reduced to only 86ms, and the model acceleration rate to 2.69. Experiments using the hybrid model compression algorithm fully verified the superior performance of model compression in this invention.

[0095] The following are embodiments of the apparatus of the present invention, which can be used to execute embodiments of the method of the present invention. For details not disclosed in the apparatus embodiments, please refer to the embodiments of the method of the present invention.

[0096] See Figure 10 In another embodiment of the present invention, a lightweight facial expression recognition system for online learners is provided, which can be used to implement the above-mentioned lightweight facial expression recognition method for online learners. Specifically, the lightweight facial expression recognition system for online learners includes an image acquisition module and an facial expression recognition module.

[0097] The system comprises several modules: an image acquisition module for acquiring images of the online learner's facial expressions to be recognized; and an expression recognition module for inputting these images into a pre-trained lightweight expression recognition model to obtain the online learner's expression recognition results. The pre-trained lightweight expression recognition model is obtained by: acquiring a pre-trained occluded expression recognition model; compressing the pre-trained occluded expression recognition model using a multi-layer knowledge distillation method; combining depthwise separable convolution to reduce model computation; and quantizing the compressed occluded expression recognition model. The occluded expression recognition model includes a feature extraction module, an attention branch fusion module, and an output layer module. The attention branch fusion module includes several spatial attention branch modules, a channel random occlusion module, a feature pyramid network module, and a fusion module. The inputs of the spatial attention branch modules are all connected to the output of the feature extraction module, and the outputs of the spatial attention branch modules are all connected to the input of the fusion module through the channel random occlusion module. The input of the feature pyramid network module is connected to the output of the feature extraction module, and its output is connected to the input of the fusion module.

[0098] All relevant content of each step involved in the aforementioned embodiments of the lightweight facial expression recognition method for online learners can be referenced to the functional description of the corresponding functional module of the lightweight facial expression recognition system for online learners in the embodiments of the present invention, and will not be repeated here.

[0099] The module division in this embodiment of the invention is illustrative and represents only one logical functional division. In actual implementation, other division methods may be used. Furthermore, the functional modules in the various embodiments of the invention can be integrated into a single processor, exist as separate physical entities, or be integrated into a single module. The integrated modules described above can be implemented in hardware or as software functional modules.

[0100] In another embodiment of the present invention, a computer device is provided, comprising a processor and a memory. The memory stores a computer program, which includes program instructions. The processor executes the program instructions stored in the computer storage medium. The processor may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing and control core of the terminal, suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions from the computer storage medium to achieve a corresponding method flow or function. The processor described in this embodiment of the present invention can be used for the operation of a lightweight facial expression recognition method for online learners.

[0101] In another embodiment of the present invention, a storage medium is provided, specifically a computer-readable storage medium (Memory), which is a memory device in a computer device used to store programs and data. It is understood that the computer-readable storage medium here can include both the built-in storage medium in the computer device and extended storage media supported by the computer device. The computer-readable storage medium provides storage space that stores the terminal's operating system. Furthermore, the storage space also stores one or more instructions suitable for loading and execution by a processor. These instructions can be one or more computer programs (including program code). It should be noted that the computer-readable storage medium here can be high-speed RAM or non-volatile memory, such as at least one disk storage device. The processor can load and execute one or more instructions stored in the computer-readable storage medium to implement the corresponding steps of the lightweight facial expression recognition method for online learners in the above embodiments.

[0102] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0103] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0104] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0105] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0106] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A lightweight facial expression recognition method for online learners, characterized in that, include: Acquire images of facial expressions to be recognized from online learners; The image of the expression to be recognized is input into a pre-trained lightweight expression recognition model to obtain the expression recognition results of the online learner; The pre-trained lightweight facial expression recognition model is obtained in the following way: A pre-trained occluded facial expression recognition model is obtained. This model is then compressed using a multi-layer knowledge distillation method, and depthwise separable convolution is combined to reduce computational complexity. Finally, the compressed model is quantized to obtain a lightweight pre-trained facial expression recognition model. The occluded facial expression recognition model includes a feature extraction module, an attention branch fusion module, and an output layer module. The attention branch fusion module comprises several spatial attention branch modules, a channel random occlusion module, a feature pyramid network module, and a fusion module. The inputs of the spatial attention branch modules are all connected to the output of the feature extraction module, and the outputs are all connected to the input of the fusion module through the channel random occlusion module. The input of the feature pyramid network module is connected to the output of the feature extraction module, and its output is connected to the input of the fusion module.

2. The lightweight facial expression recognition method for online learners according to claim 1, characterized in that, The process of acquiring the pre-trained occluded facial expression recognition model includes: Acquire several facial expression images and construct an occluded expression recognition model; Several facial expression images were aligned using the MTCNN network, and then data augmentation was performed to obtain several training images. The data augmentation operations included the following: random image cropping, random horizontal flipping, random vertical flipping, and histogram equalization. An occluded expression recognition model is trained using several training images to obtain a pre-trained occluded expression recognition model.

3. The lightweight facial expression recognition method for online learners according to claim 1, characterized in that, The feature extraction module is constructed using the first three main feature extraction networks in a face recognition network pre-trained on a face recognition dataset; The spatial attention branch module consists of two convolutional layers. The feature map output by the feature extraction module is reduced in dimensionality by the first convolutional layer to obtain an intermediate feature map, which is then subjected to the ReLU activation function to add non-linear changes, and then subjected to the second convolutional layer for dimensionality reduction and the Sigmoid activation function to obtain the spatial attention map. The channel random occlusion module consists of a channel occlusion operation layer, a maximum value layer, and an average value layer. The spatial attention map is obtained by randomly reducing the value of a certain channel by a preset number through the channel occlusion operation layer, then taking the maximum value through the maximum value layer, and finally taking the average value through the average value layer. The feature pyramid network module consists of a downsampling layer and an upsampling layer. The feature map output by the feature extraction module is passed through the downsampling layer to obtain four feature maps of different scales. Then, each of them is subjected to two 1*1 convolution operations to increase the dimensionality. Finally, after passing through the upsampling layer, they are concatenated and fused into a multi-scale feature map.

4. The lightweight facial expression recognition method for online learners according to claim 1, characterized in that, The facial expression recognition results of the online learners are as follows: Anger, disgust, fear, happiness, sadness, surprise, or neutral.

5. The lightweight facial expression recognition method for online learners according to claim 1, characterized in that, The method of compressing the pre-trained occluded expression recognition model using multi-layer knowledge distillation, combining it with depthwise separable convolution to reduce the computational cost of the model, and quantizing the compressed occluded expression recognition model to obtain a pre-trained lightweight expression recognition model includes: The pre-trained occluded facial expression recognition model is used as the teacher model in the knowledge distillation network, and the backbone CNN network with reduced stacking is used as the student network. Knowledge distillation training is performed based on a predefined knowledge guidance path. The knowledge distillation training includes intermediate layer knowledge distillation and output layer knowledge distillation. During knowledge distillation training, knowledge is extracted from the teacher network and student network through a spatial attention network, and a multi-layer attention fusion network is used to fuse the feature maps of multiple layers of the teacher network, so that the size of the fused feature map is consistent with that of the feature map of the student network at the same level. Replace the backbone network in the student network with the lightweight MobileNetV2 network; The final student network is quantized using model quantization methods to obtain a pre-trained lightweight facial expression recognition model.

6. The lightweight facial expression recognition method for online learners according to claim 5, characterized in that, The loss function for the knowledge distillation training is the sum of the intermediate layer knowledge distillation loss and the output layer distillation loss; The knowledge distillation loss in the intermediate layer is: L mid =L AT1 +λ1L AT2 +λ2L AT3 L kd =L distill +βL mid Among them, L AT L represents the attention loss after the fusion of teacher network features and the network of students at the same level. AT1 L AT2 and L AT3 These represent different feature extraction layers; This represents the feature matrix output by the teacher model at layer j. This represents the feature matrix output by the student model at layer j; λ1, λ2, and β represent the balance coefficients; L distill L is the output layer distillation loss function; mid L represents the intermediate layer distillation loss; I represents the number of layers in the model; L represents the intermediate layer distillation loss. kd This is the total distillation loss function; The output layer distillation loss is: L distill =αL soft +(1-α)L hard in, Represents the probability distribution of the teacher model. c represents the probability distribution of the student model. j Represents the true label, This represents the case where temperature T is 1, α is the equilibrium factor, and L soft L represents the loss after distillation of the student model and the teacher model. hard L represents the loss between the student model's predicted values ​​and the true label values. distill Z represents the final total loss from knowledge distillation; N is the total number of samples; z i z is the logit output of the teacher model for the i-th class; j z is the logit output of the student model for the j-th class; k This is the logit output for the student model for the k-th class.

7. The lightweight facial expression recognition method for online learners according to claim 1, characterized in that, The loss function of the occluded facial expression recognition model is: L cls =L s +λ 11 L intra +b 11 L inter Among them, L s This represents the classic softmax loss function, where m is the total number of samples and n is the number of classes. The predicted probability of the sample not being normalized; Here is the weight matrix; x i b is the feature vector of sample i; n For bias terms; Let b be the weight matrix corresponding to category j; j For the bias term corresponding to category j; x i Let L be the feature vector of the i-th sample; intra L represents the distance between a sample and the cluster center of a feature cluster; inter The distance between the cluster centers of different samples; It is category y i Feature center; c k and c j These are the centers of categories k and j, respectively; L cls λ represents the final loss function. 11 and β 11 This represents the balance coefficient.

8. A lightweight facial expression recognition system for online learners, characterized in that, include: The image acquisition module is used to acquire images of the facial expressions of online learners to be recognized. The facial expression recognition module is used to input the facial expression image to be recognized into a pre-trained lightweight facial expression recognition model to obtain the facial expression recognition results of the online learner; The pre-trained lightweight facial expression recognition model is obtained in the following way: A pre-trained occluded facial expression recognition model is obtained. This model is then compressed using a multi-layer knowledge distillation method, and depthwise separable convolution is combined to reduce computational complexity. Finally, the compressed model is quantized to obtain a lightweight pre-trained facial expression recognition model. The occluded facial expression recognition model includes a feature extraction module, an attention branch fusion module, and an output layer module. The attention branch fusion module comprises several spatial attention branch modules, a channel random occlusion module, a feature pyramid network module, and a fusion module. The inputs of the spatial attention branch modules are all connected to the output of the feature extraction module, and the outputs are all connected to the input of the fusion module through the channel random occlusion module. The input of the feature pyramid network module is connected to the output of the feature extraction module, and its output is connected to the input of the fusion module.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the lightweight facial expression recognition method for online learners as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the lightweight facial expression recognition method for online learners as described in any one of claims 1 to 7.

Images