Lightweight expression recognition method for online learner and related device
Through technologies such as multi-layer knowledge distillation and depth separation convolution, a lightweight expression recognition model is built, which solves the accuracy of online learners' expression recognition in the case of occlusion, and realizes efficient expression recognition and model deployment.
Patent Information
- Application Number
- CN202411101388.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-12
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2044-08-12
AI Technical Summary
The existing online learner expression recognition methods are not very accurate when facing occlusion, and the model is complex and difficult to deploy, so it cannot effectively deal with inconsistent learning environments and limited mobile hardware performance.
The pre-trained occlusion expression recognition model is compressed by multi-layer knowledge distillation method, and combined with depth separation convolution and model quantization, a lightweight expression recognition model is constructed, and the feature extraction module, attention branch fusion module and output layer module are used to extract Local and global features of the face to improve recognition accuracy.
It realizes high-accurate expression recognition in occlusion situations, reduces the calculation amount and running speed of the model, improves the recognition ability of occlusion robustness, and is suitable for online learners' lightweight expression recognition needs.
Smart Images

Figure CN118968595A_ABST
Abstract
Description
Technical Field
[0001] The invention belongs to the field of expression recognition, and relates to a lightweight expression recognition method for online learners and a related device. Background Art
[0002] Facial expression recognition has a wide range of applications, especially in the field of education, and has great prospects. Facial expression is one of the most effective, common and natural ways for humans to convey emotional states and intentions. Facial expression recognition is to analyze the characteristics of facial expressions and determine the emotional category of expressions. Based on the results obtained, people's behavior can be predicted and their psychological activities can be inferred, so as to make corresponding correct behavioral responses.
[0003] Most students mainly use their mobile phones for online classroom learning. The learning environment is often not uniform and prone to occlusion. At the same time, the hardware performance of mobile devices is relatively limited, so the deployment of expression recognition models also needs to consider the actual needs of lightweight algorithm networks. However, the current online learner expression recognition methods have problems such as various types of facial image occlusion and variable regions, resulting in low accuracy of recognition results. At the same time, existing expression recognition models are relatively complex and difficult to deploy effectively. Summary of the invention
[0004] The purpose of the present invention is to overcome the above-mentioned shortcomings of the prior art and provide a lightweight expression recognition method for online learners and related devices.
[0005] In order to achieve the above object, the present invention adopts the following technical solutions:
[0006] The first aspect of the present invention provides a lightweight expression recognition method for online learners, comprising: obtaining an expression image to be recognized of an online learner; inputting the expression image to be recognized into a pre-trained lightweight expression recognition model to obtain an expression recognition result of the online learner; wherein the pre-trained lightweight expression recognition model is obtained by the following method: obtaining a pre-trained occluded expression recognition model, compressing the pre-trained occluded expression recognition model using a multi-layer knowledge distillation method, and combining a deep separable convolution to reduce the model calculation amount, and quantizing the compressed occluded expression recognition model to obtain the pre-trained lightweight expression recognition model; wherein the occluded expression recognition model comprises a feature extraction module, an attention branch fusion module and an output layer module, the attention branch fusion module comprises a plurality of spatial attention branch modules, a channel random masking module, a feature pyramid network module and a fusion module, the input ends of the plurality of spatial attention branch modules are all connected to the output end of the feature extraction module, the output ends are all connected to the input end of the fusion module through the channel random masking module, the input end of the feature pyramid network module is connected to the output end of the feature extraction module, and the output end is connected to the input end of the fusion module.
[0007] Optionally, obtaining a pre-trained occluded expression recognition model includes: obtaining a number of facial expression images and constructing an occluded expression recognition model; aligning the facial expression images using an MTCNN network, and then performing a data enhancement operation to obtain a number of training images; wherein the data enhancement operation includes the following operations: random image cropping, random horizontal flipping, random vertical flipping, and histogram equalization method; using a number of training images to train the occluded expression recognition model to obtain a pre-trained occluded expression recognition model.
[0008] Optionally, the feature extraction module is constructed using the first three main feature extraction networks in the face recognition network pre-trained on the face recognition data set; the spatial attention branch module is composed of two convolutional layers, and the feature map output by the feature extraction module is reduced in dimension by the first convolutional layer to obtain an intermediate feature map, and then increased by the ReLU activation function, and then reduced in dimension by the second convolutional layer and the Sigmoid activation function to obtain the spatial attention map; the channel random masking module is composed of a channel masking operation layer, a maximum value layer and an average value layer, and the spatial attention map is randomly reduced by a preset multiple of the value of a certain channel through the channel masking operation layer, and then the maximum value is taken through the maximum value layer, and then the average value is taken through the average value layer to obtain the attention map; the feature pyramid network module is composed of a downsampling layer and an upsampling layer, and the feature map output by the feature extraction module is obtained through the downsampling layer. Feature maps of four different scales are obtained, and then they are respectively increased in dimension through two 1*1 convolution operations, and then spliced and fused into a multi-scale feature map after passing through the upsampling layer.
[0009] Optionally, the expression recognition result of the online learner is: anger, disgust, fear, happiness, sadness, surprise or neutral.
[0010] Optionally, the method uses a multi-layer knowledge distillation method to compress the pre-trained occluded expression recognition model, combines it with deep separable convolution to reduce the model calculation amount, and quantizes the compressed occluded expression recognition model to obtain a pre-trained lightweight expression recognition model, including: using the pre-trained occluded expression recognition model as the teacher model in the knowledge distillation network, reducing the number of stacking of the backbone CNN network as the student network; performing knowledge distillation training based on a predefined knowledge guidance path; wherein the knowledge distillation training includes intermediate layer knowledge distillation and output layer knowledge distillation; during knowledge distillation training, extracting knowledge from the teacher network and the student network through a spatial attention network, and using a multi-layer attention fusion network to fuse the feature maps of the multiple layers of the teacher network, and making the size of the fused feature map consistent with the feature map of the student network at the same level; using a lightweight network MobileNetV2 to replace the backbone network in the student network; and using a model quantization method to quantize the final student network to obtain a pre-trained lightweight expression recognition model.
[0011] Optionally, the loss function of the knowledge distillation training is the sum of the intermediate layer knowledge distillation loss and the output layer distillation loss; the intermediate layer knowledge distillation loss is:
[0012]
[0013] L mid =L AT1 +λ1L AT2 +λ2L AT3
[0014] L kd =L distill +βL mid
[0015] Among them, L AT represents the attention loss of the teacher network after feature fusion and the student network at the same level, L AT1 , L AT2 and L AT3 Represent different feature extraction layers respectively; represents the feature matrix output by the teacher model at layer j, represents the feature matrix output by the student model at the jth layer; λ1, λ2, and β represent the balance coefficients; L distill is the output layer distillation loss function; L mid is the intermediate layer distillation loss; I is the number of layers in the model; L kd is the total distillation loss function.
[0016] The output layer distillation loss is:
[0017]
[0018] L distill =αL soft +(1-α)L hard
[0019] in, represents the probability distribution of the teacher model, represents the probability distribution of the student model, c j represents the true label, represents the case where the temperature T is 1, α is the equilibrium factor, L soft represents the loss after distillation of the student model and the teacher model, L hard Represents the loss between the student model prediction value and the true label value, L distill represents the final total loss of knowledge distillation; N is the total number of samples; z i is the logit output of the teacher model for the i-th category; z j is the logit output of the student model for the jth category; zk is the logit output of the student model for the kth category.
[0020] Optionally, the loss function of the occluded expression recognition model is:
[0021]
[0022] L cls =L s +λ 11 L intra +β 11 L inter
[0023] Among them, L s represents the classic softmax loss function, m is the total number of samples; n is the number of categories; is the unnormalized predicted probability of the sample; is the weight matrix; x i is the feature vector of sample i; b n is the bias term; is the weight matrix corresponding to category j; b j is the bias term corresponding to category j; x i is the feature vector of the i-th sample; L intra Represents the distance between the sample and the feature cluster center; L inter Represents the distance between the cluster centers of different samples; is category y i The characteristic center of k and c j are the centers of category k and category j respectively; L cls represents the final loss function, λ 11 and β 11 Represents the balance coefficient.
[0024] In a second aspect, the present invention provides a lightweight expression recognition system for online learners, comprising: an image acquisition module, used to acquire an expression image to be recognized of an online learner; an expression recognition module, used to input the expression image to be recognized into a pre-trained lightweight expression recognition model to obtain an expression recognition result of the online learner; wherein the pre-trained lightweight expression recognition model is obtained by the following method: acquiring a pre-trained occluded expression recognition model, compressing the pre-trained occluded expression recognition model using a multi-layer knowledge distillation method, and combining a deep separable convolution to reduce the model calculation amount, and quantizing the compressed occluded expression recognition model to obtain the pre-trained lightweight expression recognition model; wherein the occluded expression recognition model comprises a feature extraction module, an attention branch fusion module and an output layer module, the attention branch fusion module comprises a plurality of spatial attention branch modules, a channel random masking module, a feature pyramid network module and a fusion module, the input ends of the plurality of spatial attention branch modules are all connected to the output end of the feature extraction module, the output ends are all connected to the input end of the fusion module through the channel random masking module, the input end of the feature pyramid network module is connected to the output end of the feature extraction module, and the output end is connected to the input end of the fusion module.
[0025] In a third aspect of the present invention, a computer device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned method for lightweight expression recognition of online learners when executing the computer program.
[0026] According to a fourth aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned method for lightweight expression recognition of online learners are implemented.
[0027] Compared with the prior art, the present invention has the following beneficial effects:
[0028] The lightweight expression recognition method for online learners of the present invention realizes accurate expression recognition of online learners based on a pre-trained lightweight expression recognition model. The pre-trained lightweight expression recognition model is obtained based on a pre-trained occluded expression recognition model. First, the pre-trained occluded expression recognition model is compressed by using a multi-layer knowledge distillation method, and then the model calculation amount is further reduced by combining a deep separable convolution. Finally, the model quantization method is used to improve the model running speed, thereby obtaining a more lightweight occluded expression recognition model. The occluded expression recognition model is constructed by using a feature extraction module, an attention branch fusion module and an output layer module, and a plurality of spatial attention branch modules, a channel random masking module, a feature pyramid network module and a fusion module are fused. Local features of the face are extracted by a plurality of spatial attention branch modules, and the robustness to occlusion is improved by a random channel masking module. At the same time, global multi-scale features are extracted by a feature pyramid network module, and expression classification is realized by fusing local and global features, thereby further improving the recognition accuracy of the model under occlusion. BRIEF DESCRIPTION OF THE DRAWINGS
[0029] Figure 1 This is a flow chart of a lightweight expression recognition method for online learners according to an embodiment of the present invention.
[0030] Figure 2 Schematic diagram of an occluded expression recognition model according to an embodiment of the present invention.
[0031] Figure 3 Schematic diagram of a feature pyramid network module according to an embodiment of the present invention.
[0032] Figure 4 Schematic diagram of the spatial attention branch module of an embodiment of the present invention.
[0033] Figure 5 Schematic diagram of a channel random masking module according to an embodiment of the present invention.
[0034] Figure 6 Schematic diagram of the principle of the multi-layer knowledge distillation method according to an embodiment of the present invention.
[0035] Figure 7 Schematic diagram of a multi-layer attention fusion network according to an embodiment of the present invention.
[0036] Figure 8 Schematic diagram of a spatial attention network according to an embodiment of the present invention.
[0037] Fig. 9 Schematic diagram of the knowledge guidance path KGP according to an embodiment of the present invention.
[0038] Fig.10 This is a structural block diagram of a lightweight expression recognition system for online learners according to an embodiment of the present invention. DETAILED DESCRIPTION
[0039] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.
[0040] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0041] The present invention is further described in detail below in conjunction with the accompanying drawings:
[0042] See also Figure 1 In one embodiment of the present invention, a lightweight expression recognition method for online learners is provided, which can effectively improve the expression recognition accuracy of online learners and reduce the magnitude of the expression recognition model.
[0043] Specifically, the lightweight expression recognition method for online learners of the present invention comprises the following steps:
[0044] S1: Obtain the facial expression image to be recognized of the online learner.
[0045] S2: Inputting the expression image to be recognized into a pre-trained lightweight expression recognition model to obtain an expression recognition result of the online learner.
[0046] Among them, the pre-trained lightweight expression recognition model is obtained by the following method:
[0047] A pre-trained occluded expression recognition model is obtained, the pre-trained occluded expression recognition model is compressed using a multi-layer knowledge distillation method, and the model calculation amount is reduced by combining a deep separable convolution, and the compressed occluded expression recognition model is quantized to obtain a pre-trained lightweight expression recognition model; wherein, the occluded expression recognition model includes a feature extraction module, an attention branch fusion module and an output layer module, the attention branch fusion module includes several spatial attention branch modules, a channel random masking module, a feature pyramid network module and a fusion module, the input ends of the several spatial attention branch modules are all connected to the output end of the feature extraction module, the output ends are all connected to the input end of the fusion module through the channel random masking module, the input end of the feature pyramid network module is connected to the output end of the feature extraction module, and the output end is connected to the input end of the fusion module.
[0048] The lightweight expression recognition method for online learners of the present invention realizes accurate expression recognition of online learners based on a pre-trained lightweight expression recognition model. The pre-trained lightweight expression recognition model is obtained based on a pre-trained occluded expression recognition model. First, the pre-trained occluded expression recognition model is compressed by using a multi-layer knowledge distillation method, and then the model calculation amount is further reduced by combining a deep separable convolution. Finally, the model quantization method is used to improve the model running speed, thereby obtaining a more lightweight occluded expression recognition model. The occluded expression recognition model is constructed by using a feature extraction module, an attention branch fusion module and an output layer module, and a plurality of spatial attention branch modules, a channel random masking module, a feature pyramid network module and a fusion module are integrated. Local features of the face are extracted by a plurality of spatial attention branch modules, and the robustness to occlusion is improved by a random channel masking module. At the same time, global multi-scale features are extracted by a feature pyramid network module, and expression classification is realized by fusing local and global features, thereby further improving the recognition accuracy of the model under occlusion.
[0049] In a possible implementation, obtaining a pre-trained occluded expression recognition model includes:
[0050] Acquire a number of facial expression images and construct an occluded expression recognition model; perform face alignment on the facial expression images using the MTCNN network, and then perform data enhancement operations to obtain a number of training images; wherein the data enhancement operations include one or more of the following: random image cropping, random horizontal flipping, random vertical flipping, and histogram equalization methods; use a number of training images to train the occluded expression recognition model to obtain a pre-trained occluded expression recognition model.
[0051] Specifically, the plurality of facial expression images include seven discrete categories of facial expressions, namely anger, disgust, fear, happiness, sadness, surprise and neutral. Static images captured from learning videos of online learners can be used as facial expression images.
[0052] In one possible implementation, see Figures 2 to 5 The feature extraction module is constructed by using the first three main feature extraction networks in the face recognition network pre-trained on the face recognition data set; the spatial attention branch module consists of two convolutional layers, and the feature map output by the feature extraction module is reduced in dimension by the first convolutional layer to obtain an intermediate feature map, and then increased by the ReLU activation function, and then reduced in dimension by the second convolutional layer and the Sigmoid activation function to obtain the spatial attention map; the channel random masking module is composed of a channel masking operation layer, a maximum value layer and an average value layer. The spatial attention map is randomly reduced by a preset multiple of the value of a certain channel through the channel masking operation layer, and then the maximum value is taken through the maximum value layer, and then the average value is taken through the average value layer to obtain the attention map; the feature pyramid network module is composed of a downsampling layer and an upsampling layer. The feature map output by the feature extraction module is obtained through the downsampling layer. Feature maps of four different scales are obtained, and then they are respectively increased in dimension by two 1*1 convolution operations, and then spliced and fused into a multi-scale feature map after passing through the upsampling layer.
[0053] Specifically, after obtaining the facial expression image, the MTCNN network is used for face alignment, followed by data enhancement operations, mainly using image random cropping, random horizontal flipping, random vertical flipping and histogram equalization methods. At this time, the facial expression image is represented as 224*224*3.
[0054] Use the model pre-trained on the face recognition dataset for transfer learning, mainly using the first three main feature extraction networks in the face recognition network, and the output feature size is 28*28*1024.
[0055] The spatial attention branch module SSNet is mainly composed of two convolutions. The input feature map size is h*w*c. After the 1*1 convolution of the first convolutional layer, the intermediate feature map is obtained with a size of h*w*c / r. Then it passes through the ReLU activation function to increase nonlinear changes. Finally, it passes through the 1*1 convolution of the second convolutional layer to reduce the dimension and the Sigmoid activation function to obtain the spatial attention map h*w*1.
[0056] The channel random masking module RCMask consists of channel masking operation and maximum and average operations. The input feature map is the feature map extracted by multi-branch SSNet, with a size of h*w*n. After the channel masking operation, the value of a certain channel is randomly reduced by p times to reduce the importance of the channel. The feature map size is still h*w*n, and then the final attention map is extracted through maximum and average operations, with a size of h*w*1.
[0057] The feature pyramid network module PNet consists of a downsampling layer and an upsampling layer. The input feature map size is h*w*c. After the downsampling operation, four feature maps of different scales are obtained, and the number of channels is one-fourth of the original. Then, they are upgraded by two 1*1 convolution operations respectively, and then spliced and fused into the final multi-scale feature map after the upsampling operation. The size is h*w*c, keeping the size of the input and output feature maps unchanged.
[0058] After the attention branch fusion module fuses local features and global features, it outputs through the output layer module using softmax to obtain 7 discrete types of facial expressions.
[0059] In one possible implementation, see Figures 6 to 8 , a pre-trained occluded expression recognition model is compressed using a multi-layer knowledge distillation method, and the model calculation amount is reduced by combining the deep separable convolution, and the compressed occluded expression recognition model is quantized to obtain a pre-trained lightweight expression recognition model, including: using the pre-trained occluded expression recognition model as the teacher model in the knowledge distillation network, reducing the number of stacking of the backbone CNN network as the student network; performing knowledge distillation training based on a predefined knowledge guidance path; wherein the knowledge distillation training includes intermediate layer knowledge distillation and output layer knowledge distillation; during knowledge distillation training, knowledge is extracted from the teacher network and the student network through a spatial attention network, and the feature maps of the multiple layers of the teacher network are fused using a multi-layer attention fusion network, and the size of the fused feature maps is kept consistent with the feature maps of the student network at the same level; the backbone network in the student network is replaced by a lightweight network MobileNetV2; the final student network is quantized using a model quantization method to obtain a pre-trained lightweight expression recognition model.
[0060] Specifically, knowledge distillation mainly uses the knowledge guidance path KGP, the spatial attention network SANet and the multi-layer attention fusion network MFF. The knowledge guidance path KGP is defined in the middle layer knowledge distillation, so that the knowledge of the multi-layer teacher network guides the training of the student network, and the previous knowledge of the teacher network can also help the student network improve its performance. The main purpose of the spatial attention network SANet is to extract knowledge from the teacher network and the student network, and extract richer attention features through dual branches. The input feature map size is h*w*c. After maximum pooling and average pooling, two feature maps of size h*w*1 are obtained, and then after splicing operations, feature maps of size h*w*2 are obtained. After convolution and activation functions, the attention map of size h*w*1 is obtained. The main purpose of the multi-layer attention fusion network MFF is to fuse the features of the teacher network layers and reduce the calculation amount of the model. The input feature map is a plurality of feature maps of different sizes in the middle layer of the teacher network. After convolution operations, feature maps of size h*w*c are obtained, which are consistent with the size of the student network feature map at the same level. After the feature maps are spliced, the fused feature map of size h*w*c is obtained through convolution operations. The backbone network in the student network is replaced with a lightweight network MobileNetV2 to reduce the amount of model calculation. Finally, the model quantization method is used to speed up model reasoning and improve the compression rate and acceleration rate of the model.
[0061] Previous knowledge distillation methods mostly focused on the distillation of knowledge at the same level in the middle layer or only used the knowledge of the last middle layer of the teacher network. These studies ignored the contribution of knowledge from other layers of the teacher network. The knowledge guidance path KGP is defined in the middle layer knowledge distillation to allow the multi-layer teacher network knowledge to guide the student network training, so that the previous knowledge of the teacher network can also help the student network improve its performance. For details, see Fig. 9 , use the teacher network of the same layer and the previous layer to guide the student network training.
[0062] In a possible implementation, the loss function of the knowledge distillation training is the sum of the intermediate layer knowledge distillation loss and the output layer distillation loss.
[0063] Among them, the intermediate layer knowledge distillation loss is:
[0064]
[0065] L mid =L AT1 +λ1L AT2 +λ2L AT3
[0066] L kd =L distill +βL mid
[0067] Among them, L AT represents the attention loss of the teacher network after feature fusion and the student network at the same level, L AT1 , L AT2 and L AT3 They represent different feature extraction layers. In this embodiment, three main feature extraction layers are set during training, namely L AT1 , L AT2 and L AT3 . represents the feature matrix output by the teacher model at layer j, represents the feature matrix output by the student model at the jth layer; λ1, λ2, and β represent the balance coefficients; L distill is the output layer distillation loss function; L mid is the intermediate layer distillation loss; I is the number of layers in the model; L kd is the total distillation loss function.
[0068] Among them, the output layer distillation loss is:
[0069]
[0070] L distill =αL soft +(1-α)L hard
[0071] in, represents the probability distribution of the teacher model, represents the probability distribution of the student model, c j represents the true label, represents the case where the temperature T is 1, α is the equilibrium factor, L soft represents the loss after distillation of the student model and the teacher model, L hard Represents the loss between the student model prediction value and the true label value, L distill represents the final total loss of knowledge distillation; N is the total number of samples; z i is the logit output of the teacher model for the i-th category; z j is the logit output of the student model for the jth category; z k is the logit output of the student model for the kth category.
[0072] In a possible implementation, the loss function of the occluded expression recognition model is:
[0073]
[0074] L cls =L s +λ 11 L intra +β11 L inter
[0075] Among them, L s represents the classic softmax loss function, m is the total number of samples; n is the number of categories; is the unnormalized predicted probability of the sample; is the weight matrix; x i is the feature vector of sample i; b n is the bias term; is the weight matrix corresponding to category j; b j is the bias term corresponding to category j; x i is the feature vector of the i-th sample. intra Represents the distance between the sample and the feature cluster center, which is used to constrain the distance within the class. inter Represents the distance between the cluster centers of different samples, which is used to constrain the distance between classes. It is mainly achieved by calculating the cosine distance between classes. +1 is to change the value range to [0, 2]. The closer to 0, the greater the difference between classes. is category y i feature center. is category y i The characteristic center of k and c j are the centers of category k and category j respectively. N is the total number of samples. L cls represents the final loss function, λ 11 and β 11 Represents the balance coefficient, and its value is 0.01 and 0.1 in actual code training.
[0076] In a possible implementation, in order to compare the effectiveness of the lightweight expression recognition method for online learners proposed in the present invention, the lightweight expression recognition method for online learners of the present invention is used to conduct expression recognition experiments with a series of classic algorithms. The experimental data set is derived from the public data sets JAFFE and RAF-DB commonly used in international expression recognition research. At the same time, combined with the particularity of the research scenario, a private expression data set (Online Learning FER, abbreviated as OLFER) in the online classroom scenario is constructed. The expression recognition task belongs to the field of image classification. In order to measure the performance of different algorithms, the experiment uses the average accuracy of 7 types of expression recognition as a measurement indicator. The average accuracy in the experimental results is the average value obtained after training the model multiple times.
[0077] The experimental results are shown in Table 1:
[0078] Table 1
[0079] Model JAFFE(mask) RAF-DB(mask) OLFER R50 0.756 0.762 0.805 SENet 0.805 0.788 0.817 CBAM 0.829 0.790 0.839 GAM 0.854 0.794 0.845 gACNN 0.829 0.791 0.842 SCAN 0.902 0.801 0.856 PVS 0.878 0.798 0.849 AFNet 0.926 0.803 0.859
[0080] Among them, JAFFE (mask) is obtained after occlusion processing of the JAFFE dataset, and RAF-DB (mask) dataset is obtained after occlusion processing of the RAF-DB dataset. OLFER is an expression recognition dataset constructed separately in an online learning scenario, and the dataset itself already has real occlusion.
[0081] As can be seen from Table 1, the pre-trained lightweight expression recognition model proposed by the online learner lightweight expression recognition method of the present invention is superior to the comparative attention network and occluded expression recognition network on the JAFFE (mask), RAF-DB (mask) and OLFER datasets. Compared with the baseline model (R50), the pre-trained lightweight expression recognition model of the present invention has great advantages on the JAFFE (mask), RAF-DB (mask) and OLFER datasets. The pre-trained lightweight expression recognition model of the present invention has a maximum accuracy of 0.926 on the JAFFE (mask) dataset, an improvement of 22.5%, a maximum accuracy of 0.803 on the RAF-DB (mask) dataset, an improvement of 5.4%, and a maximum accuracy of 0.859 on the OLFER dataset, an improvement of 6.7%.
[0082] Comparative experiments on public and private datasets fully demonstrate the effectiveness of the lightweight expression recognition model pre-trained by the present invention, while also improving the robustness of the model to simulated occlusion and real occlusion.
[0083] In a possible implementation, in order to prove the effectiveness of the compression network MKD method of the pre-trained lightweight expression recognition model proposed in the present invention, the pre-trained lightweight expression recognition model proposed in the present invention is used with a series of classic algorithms to carry out expression recognition model compression experiments. The data sets used in the experiment are RAF-DB (mask) and OLFER. In order to measure the performance of different knowledge distillation algorithms, the experiment uses the average accuracy of 7 types of expression recognition as the main measurement indicator, and the average value obtained after multiple model training is used as the experimental result. At the same time, in order to comprehensively consider the model compression effect, this implementation also selects indicators such as model parameter quantity, model calculation amount, model memory usage, parameter compression rate and model acceleration rate as evaluation indicators.
[0084] The experimental results are shown in Tables 2 to 4.
[0085] Table 2
[0086]
[0087] The numbers in the MKD+KD column in Table 2 represent the best performance of the algorithm on the dataset. The teacher and student models in the table are both baseline+AFNet models, and the repeated AFNet is omitted for data simplicity. R50 and R18 are abbreviations of ResNet50 and ResNet18, respectively. MKD is a compression network method for the pre-trained lightweight expression recognition model proposed in this invention. KD, FitNet, AT, PKT, CC and SimKD are classic knowledge distillation algorithms used for comparison.
[0088] As shown in Table 2, when the baseline network is a similar structure, that is, isomorphic knowledge distillation, the MKD method outperforms other comparison methods. When the teacher network is R50+AFNet and the student network is R18+AFNet, the accuracy of the ordinary AT knowledge distillation method is 0.787, the MKD method is 0.795, and the MKD+KD method is 0.799, which is 3.1% higher than the original directly trained student network. When the baseline network is a dissimilar structure, that is, heterogeneous knowledge distillation, the MKD method is also better than other advanced methods. When the teacher network is R50+AFNet and the student network is MobileNetV2+AFNet, the accuracy of the ordinary AT knowledge distillation method is 0.788, the MKD method is 0.804, and the MKD+KD method is 0.806, which is 5.2% higher than the original directly trained student network.
[0089] Table 3
[0090]
[0091] As shown in Table 3, when the baseline network is a similar structure, that is, isomorphic knowledge distillation, the MKD method outperforms other comparison methods. When the teacher network is R50+AFNet and the student network is R18+AFNet, the accuracy of the ordinary AT knowledge distillation method is 0.838, the MKD method is 0.852, and the MKD+KD method is 0.854, which is 3.8% higher than the original directly trained student network. When the baseline network is a dissimilar structure, that is, heterogeneous knowledge distillation, the MKD method is also better than other advanced methods. When the teacher network is R50+AFNet and the student network is MobileNetV2+AFNet, the accuracy of the ordinary AT knowledge distillation method is 0.836, the MKD method is 0.849, and the MKD+KD method is 0.852, which is 4.9% higher than the original directly trained student network.
[0092] Table 4
[0093] Model Accuracy Calculation Amount Parameter quantity Reasoning time Memory Compression Acceleration rate Initial Model 0.859 3.86G 25.86M 231ms 104.12M 1.00× 1.00× Lightweight model + distillation 0.871 0.32G 3.62M 145ms 3.55M 7.14× 1.59× Lightweight model + distillation + quantization 0.862 0.32G 3.62M 86ms 0.73M 7.14× 2.69×
[0094] As shown in Table 4, the initial model has a computational complexity of 3.86G, a parameter complexity of 25.86M, and an image reasoning time of 231ms. After the lightweight model and knowledge distillation, the computational complexity is greatly reduced to only 0.32G, the parameter complexity is only 3.62G, the image reasoning time is 145ms, the parameter compression rate is 7.14, and the model acceleration rate is 1.59. After model quantization, the image reasoning time is only 86ms, and the model acceleration rate is 2.69. The present invention has fully verified the superior performance of model compression through experiments on the hybrid model compression algorithm.
[0095] The following are device embodiments of the present invention, which can be used to implement the method embodiments of the present invention. For details not disclosed in the device embodiments, please refer to the method embodiments of the present invention.
[0096] See also Fig.10 In another embodiment of the present invention, a lightweight expression recognition system for online learners is provided, which can be used to implement the above-mentioned lightweight expression recognition method for online learners. Specifically, the lightweight expression recognition system for online learners includes an image acquisition module and an expression recognition module.
[0097] The image acquisition module is used to acquire the expression image to be recognized of the online learner; the expression recognition module is used to input the expression image to be recognized into the pre-trained lightweight expression recognition model to obtain the expression recognition result of the online learner. The pre-trained lightweight expression recognition model is obtained by the following method: obtaining the pre-trained occluded expression recognition model, compressing the pre-trained occluded expression recognition model using the multi-layer knowledge distillation method, and combining the deep separable convolution to reduce the model calculation amount, and quantizing the compressed occluded expression recognition model to obtain the pre-trained lightweight expression recognition model; wherein the occluded expression recognition model includes a feature extraction module, an attention branch fusion module and an output layer module, the attention branch fusion module includes several spatial attention branch modules, a channel random masking module, a feature pyramid network module and a fusion module, the input ends of the several spatial attention branch modules are all connected to the output end of the feature extraction module, the output ends are all connected to the input end of the fusion module through the channel random masking module, the input end of the feature pyramid network module is connected to the output end of the feature extraction module, and the output end is connected to the input end of the fusion module.
[0098] All relevant contents of each step involved in the embodiment of the aforementioned online learner lightweight expression recognition method can be referred to the functional description of the functional modules corresponding to the online learner lightweight expression recognition system in the embodiment of the present invention, and will not be repeated here.
[0099] The division of modules in the embodiments of the present invention is schematic and is only a logical function division. There may be other division methods in actual implementation. In addition, each functional module in each embodiment of the present invention may be integrated into one processor, or may exist physically separately, or two or more modules may be integrated into one module. The above-mentioned integrated modules may be implemented in the form of hardware or in the form of software functional modules.
[0100] In another embodiment of the present invention, a computer device is provided, the computer device including a processor and a memory, the memory is used to store a computer program, the computer program includes program instructions, and the processor is used to execute the program instructions stored in the computer storage medium. The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc., which are the computing core and control core of the terminal, which are suitable for implementing one or more instructions, and are specifically suitable for loading and executing one or more instructions in the computer storage medium to implement the corresponding method flow or corresponding function; the processor described in the embodiment of the present invention can be used for the operation of the lightweight expression recognition method for online learners.
[0101] In another embodiment of the present invention, the present invention also provides a storage medium, specifically a computer-readable storage medium (Memory), which is a memory device in a computer device for storing programs and data. It can be understood that the computer-readable storage medium here can include both built-in storage media in the computer device and, of course, extended storage media supported by the computer device. The computer-readable storage medium provides a storage space, which stores the operating system of the terminal. In addition, one or more instructions suitable for being loaded and executed by the processor are also stored in the storage space, and these instructions can be one or more computer programs (including program codes). It should be noted that the computer-readable storage medium here can be a high-speed RAM memory or a non-volatile memory, such as at least one disk memory. The processor can load and execute one or more instructions stored in the computer-readable storage medium to implement the corresponding steps of the lightweight expression recognition method for online learners in the above embodiment.
[0102] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0103] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0104] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0105] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0106] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.
Claims
1. A lightweight expression recognition method for online learners, It is characterized in that include: Obtaining facial expression images to be recognized from online learners; Inputting the expression image to be recognized into a pre-trained lightweight expression recognition model to obtain an expression recognition result of the online learner; Among them, the pre-trained lightweight expression recognition model is obtained by the following method: A pre-trained occluded expression recognition model is obtained, the pre-trained occluded expression recognition model is compressed using a multi-layer knowledge distillation method, and the model calculation amount is reduced by combining a deep separable convolution, and the compressed occluded expression recognition model is quantized to obtain a pre-trained lightweight expression recognition model; wherein, the occluded expression recognition model includes a feature extraction module, an attention branch fusion module and an output layer module, the attention branch fusion module includes several spatial attention branch modules, a channel random masking module, a feature pyramid network module and a fusion module, the input ends of the several spatial attention branch modules are all connected to the output end of the feature extraction module, the output ends are all connected to the input end of the fusion module through the channel random masking module, the input end of the feature pyramid network module is connected to the output end of the feature extraction module, and the output end is connected to the input end of the fusion module.
2. The lightweight expression recognition method for online learners according to claim 1, It is characterized in that The step of obtaining a pre-trained occluded expression recognition model includes: Obtain several facial expression images and build an occluded expression recognition model; Using the MTCNN network to perform face alignment on a number of facial expression images, and then performing data augmentation operations to obtain a number of training images; wherein the data augmentation operations include the following operations: random image cropping, random horizontal flipping, random vertical flipping, and histogram equalization method; A plurality of training images are used to train the occluded expression recognition model to obtain a pre-trained occluded expression recognition model.
3. The lightweight expression recognition method for online learners according to claim 1, It is characterized in that The feature extraction module is constructed using the first three main feature extraction networks in the face recognition network pre-trained on the face recognition dataset; The spatial attention branch module consists of two convolutional layers. The feature map output by the feature extraction module is reduced in dimension by the first convolutional layer to obtain an intermediate feature map, and then the nonlinear change is increased by the ReLU activation function, and then the spatial attention map is obtained by reducing the dimension by the second convolutional layer and the Sigmoid activation function. The channel random masking module is composed of a channel masking operation layer, a maximum value layer and an average value layer. The spatial attention map is randomly reduced by a preset multiple of the value of a certain channel through the channel masking operation layer, and then the maximum value is taken through the maximum value layer, and then the average value is taken through the average value layer to obtain the attention map; The feature pyramid network module consists of a downsampling layer and an upsampling layer. The feature map output by the feature extraction module is passed through the downsampling layer to obtain feature maps of four different scales, which are then upgraded by two 1*1 convolution operations respectively, and then spliced and fused into a multi-scale feature map after passing through the upsampling layer.
4. The lightweight expression recognition method for online learners according to claim 1, It is characterized in that The expression recognition result of the online learner is: Angry, disgusted, scared, happy, sad, surprised, or neutral.
5. The lightweight expression recognition method for online learners according to claim 1, It is characterized in that The method uses a multi-layer knowledge distillation method to compress the pre-trained occluded expression recognition model, combines the depth-separable convolution to reduce the model calculation amount, and quantizes the compressed occluded expression recognition model to obtain a pre-trained lightweight expression recognition model, including: The pre-trained occluded expression recognition model is used as the teacher model in the knowledge distillation network, and the backbone CNN network is reduced in number of stacks and used as the student network; Knowledge distillation training is performed based on a predefined knowledge guidance path; wherein, knowledge distillation training includes intermediate layer knowledge distillation and output layer knowledge distillation; during knowledge distillation training, knowledge is extracted from the teacher network and the student network through a spatial attention network, and the feature maps of multiple layers of the teacher network are fused using a multi-layer attention fusion network, and the size of the fused feature map is kept consistent with that of the feature map of the student network at the same level; Use the lightweight network MobileNetV2 to replace the backbone network in the student network; The final student network is quantized using the model quantization method to obtain a pre-trained lightweight expression recognition model.
6. The lightweight expression recognition method for online learners according to claim 5, It is characterized in that The loss function of the knowledge distillation training is the sum of the intermediate layer knowledge distillation loss and the output layer distillation loss; The intermediate layer knowledge distillation loss is: L mid =L AT1 +λ 1 L AT2 +λ 2 L AT3 L kd =L distill +βL mid Among them, L AT represents the attention loss of the teacher network after feature fusion and the student network at the same level, L AT1 , L AT2 and L AT3 They represent different feature extraction layers respectively; represents the feature matrix output by the teacher model at layer j, represents the feature matrix output by the student model at layer j; 1 , 2 and β represent the equilibrium coefficient; L distill is the output layer distillation loss function; L mid is the intermediate layer distillation loss; I is the number of layers in the model; L kd is the total distillation loss function; The output layer distillation loss is: L distill =αL soft +(1-α)L hard in, represents the probability distribution of the teacher model, represents the probability distribution of the student model, c j represents the true label, represents the case where the temperature T is 1, α is the equilibrium factor, L soft represents the loss after distillation of the student model and the teacher model, L hard Represents the loss between the student model prediction value and the true label value, L distill represents the final total loss of knowledge distillation; N is the total number of samples; z i is the logit output of the teacher model for the i-th category; z j is the logit output of the student model for the jth category; z k is the logit output of the student model for the kth category.
7. The lightweight expression recognition method for online learners according to claim 1, It is characterized in that The loss function of the occluded expression recognition model is: L cls =L s +λ 11 L intra +b 11 L inter Among them, L s represents the classic softmax loss function, m is the total number of samples; n is the number of categories; is the unnormalized predicted probability of the sample; is the weight matrix; x i is the feature vector of sample i; b n is the bias term; is the weight matrix corresponding to category j; b j is the bias term corresponding to category j; x i is the feature vector of the i-th sample; L intra Represents the distance between the sample and the feature cluster center; L inter Represents the distance between the cluster centers of different samples; is category y i The characteristic center of k and c j are the centers of category k and category j respectively; L cls represents the final loss function, λ 11 and β 11 Represents the balance coefficient.
8. A lightweight expression recognition system for online learners, It is characterized in that include: An image acquisition module is used to acquire the facial expression image to be recognized of the online learner; An expression recognition module, used for inputting the expression image to be recognized into a pre-trained lightweight expression recognition model to obtain an expression recognition result of the online learner; Among them, the pre-trained lightweight expression recognition model is obtained by the following method: A pre-trained occluded expression recognition model is obtained, the pre-trained occluded expression recognition model is compressed using a multi-layer knowledge distillation method, and the model calculation amount is reduced by combining a deep separable convolution, and the compressed occluded expression recognition model is quantized to obtain a pre-trained lightweight expression recognition model; wherein, the occluded expression recognition model includes a feature extraction module, an attention branch fusion module and an output layer module, the attention branch fusion module includes several spatial attention branch modules, a channel random masking module, a feature pyramid network module and a fusion module, the input ends of the several spatial attention branch modules are all connected to the output end of the feature extraction module, the output ends are all connected to the input end of the fusion module through the channel random masking module, the input end of the feature pyramid network module is connected to the output end of the feature extraction module, and the output end is connected to the input end of the fusion module.
9. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, It is characterized in that When the processor executes the computer program, the steps of the method for lightweight expression recognition of online learners according to any one of claims 1 to 7 are implemented.
10. A computer-readable storage medium storing a computer program. It is characterized in that When the computer program is executed by a processor, the steps of the method for lightweight expression recognition of online learners according to any one of claims 1 to 7 are implemented.
Citation Information
Patent Citations
Real-time facial expression recognition method based on multi-scale kernel convolutional neural network
CN110414371A
Natural occlusion expression recognition method and system with accurate attention
CN114360026A
Human-computer interaction control method and device based on lightweight facial expression intensity recognition
CN115620363A
Self-attention mechanism-based shielded face expression deep convolution recognition method
CN116758621A
Facial expression recognition method based on attention mechanism and self-distillation
GB202404148D0
