Facial expression recognition method based on lightweight label assisted learning network
By adopting a lightweight tag assisted learning network and ShuffleNet v2 model in the facial expression recognition model, combining the tag distribution generator and auxiliary task tag space, the problem of difficult to balance the real-time, accuracy and generalization of the model in the prior art is solved, and more efficient and accurate facial expression recognition is achieved.
Patent Information
- Application Number
- CN202411856192.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-17
- Publication Date
- 2025-04-11
AI Technical Summary
In the prior art, facial expression recognition model based on lightweight convolutional neural networks is difficult to balance in real-time, accuracy and generalization, especially in ambiguous expressions and obscured facial expression detection.
The facial expression recognition method based on lightweight tag assisted learning network is adopted, and global-local feature extraction is performed through the ShuffleNet v2 model, and the tag distribution generator is used to generate auxiliary task label space, combining auxiliary task label distribution loss and cross entropy loss for model optimization.
It improves the accuracy and generalization of facial expression recognition, reduces computational overhead and redundant information, enhances the real-time nature of the model, and can more reliably identify ambiguous and obstructed expressions.
Smart Images
Figure CN119810886A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of computer vision, and more specifically to a facial expression recognition method based on a lightweight label-assisted learning network. Background Art
[0002] In recent years, with the continuous development of computer vision technology, facial expression recognition systems based on deep learning have been widely used. At present, convolutional neural networks (CNNs) are used as feature extractors in the field of expression recognition. Traditional CNN-based facial expression recognition algorithms often have too many parameters, resulting in complex calculations, making it difficult to use in the real world; the feature extraction algorithm of lightweight convolutional neural networks (LCNNs) has a simple process, fewer parameters, and is easier to achieve the balance between reasoning speed and accuracy in actual use environments, so it has become a commonly used feature extraction method. However, the commonly used LCNN feature extraction method is not a network designed for expression recognition, so the recognition ability of ambiguous expressions is relatively poor, and the ability to detect facial expressions with occlusions that are common in real environments is poor. Therefore, in the field of facial expression recognition, the performance of LCNN still needs to be further improved.
[0003] Facial expression recognition method in the wild based on LCNN: Prior art literature Jiang CS, Liu ZT, Wu M, et al. Efficient facial expression recognition with representation reinforcement network and transfer self-training for human–machine interaction [J]. IEEE Transactions on Industrial Informatics, 2023, 19 (9): 9943-9952. In order to overcome the high time cost and strong demand for high-quality datasets in display facial expression recognition, a representation reinforcement network is proposed to extract facial expression features. The specific implementation method is to design a neuron energy aggregation module to aggregate low-dimensional texture features and high-dimensional abstract features to extract rich facial expression features and simultaneously capture the dependencies between long-distance facial regions in the spatial dimension, effectively solving the common problem of low computational efficiency in industry. However, this model still has limitations and is limited to the learning of high-quality facial expression datasets. The generalization needs to be further improved. Prior art literature LeN, Nguyen K, TranQ, et al. Uncertainty-aware label distribution learning for facial expression recognition [C] / / Proceedings of the IEEE / CVF winter conference on applications of computer vision. 2023: 6088-6097. Aiming at the problem that the labels of the training dataset are noisy and inconsistent, and the facial expressions are ambiguous, resulting in poor detection results, an efficient label distribution learning with valence-arousal (LDLVA) is proposed to improve the robustness to uncertain labels and ambiguous expressions. This method first aggregates a set of neighboring data for each sample data in the training set; then aggregates the neighborhood information with the provided single label to generate the target label distribution; finally, uses the constructed distribution as a supervisory signal to optimize the model through label distribution learning, but this method has a large number of parameters and a complex network structure, resulting in poor real-time performance. In summary, the real-time performance, accuracy and generalization of the model in the field of facial expression recognition are the key and difficult points of current research.
[0004] Therefore, it is an urgent problem for those skilled in the art to propose a facial expression recognition method based on a lightweight label-assisted learning network to solve the problems existing in the prior art. Summary of the invention
[0005] In view of this, the present invention provides a facial expression recognition method based on a lightweight label-assisted learning network to solve the problem that it is difficult to achieve a balance among the real-time performance, accuracy and generalization of the model.
[0006] In order to achieve the above object, the present invention provides the following technical solutions:
[0007] A facial expression recognition method based on a lightweight label-assisted learning network comprises the following steps:
[0008] S1. Capturing facial expression images through an image acquisition device;
[0009] S2. Input the facial expression image into the ShuffleNet v2 model for global-local F overall Feature extraction;
[0010] S3. Input the facial expression image captured in S1 into the label distribution generator to generate the auxiliary task label space, and search for multiple nearest neighbor samples and labels as references for the main task prediction;
[0011] S4. Jointly optimize the ShuffleNetv2 model using auxiliary task label distribution loss and cross entropy loss;
[0012] S5. Obtain facial expression image recognition results based on the ShuffleNetv2 model optimized by S4.
[0013] In the above method, optionally, the specific content of S2 is:
[0014] The ShuffleNetv2 model includes: Conv1, Stage2, Stage3, Stage4, Conv5;
[0015] S21. After Conv1, the underlying global mapping F is obtained conv1 ∈ H×W×C , where H, W, and C represent the length, width, and number of channels of the mapping feature, respectively;
[0016] S22. Bottom-level global mapping F conv1 ∈ H×W×C After passing through the local feature extractor, it is divided into 4 equal blocks. Where i∈{1,2,3,4}, the 4 global maps are processed by two corresponding 3×3 convolutions to obtain local facial features Local facial features Splicing along the spatial axis to finally obtain local features
[0017] In the above method, optionally, H=W=56, C=29, H1=W1=28.
[0018] In the above method, optionally, S2 further includes:
[0019] S23. After the facial expression image passes through the Stage 2 layer, high-level global features are obtained
[0020] S24. Advanced global features After channel-space modulation, the global-local total feature F is obtained overall .
[0021] In the above method, optionally, the specific content of S24 is:
[0022] S241. Advanced global features Calculate channel heatmaps separately in two parallel branches and spatial heatmap The channel heat map H(Fc2)∈ C 1 and spatial heat map The whole heat map is obtained by adding and normalizing. The specific calculation formula is as follows:
[0023] H(F s 2)=σ(H c (F s 2) +H s (F s twenty one)
[0024] Where σ(·) is the Sigmoid function, H c (F c 2) is the channel heat map, H s (F s 2) is the spatial heat map, H(F s 2) The whole heat map after merging and normalizing the two heat maps;
[0025] S242. Modulated global feature map F mod The calculation is as follows:
[0026]
[0027] In the formula, represents element-wise multiplication, F l It is a local feature;
[0028] S243. Extracting local features F l and modulate the global feature Fmod After that, the final global-local feature calculation process is as follows:
[0029] F overall =F s 2+F mod (3)
[0030] The final output is the global-local total feature F overall .
[0031] The above method, optionally, in S241, channel heat map and spatial heatmap Before the merger, they were adjusted to Achieve dimensional consistency in operations.
[0032] The above method is optional, and the specific content of S3 is:
[0033] The auxiliary tasks are action unit recognition (AUR) and facial landmark detection (FLD), and the auxiliary tasks describe facial structure and movement from different perspectives.
[0034] In the above method, optionally, the label distribution loss in S4 is defined as follows:
[0035]
[0036] In the formula, x represents the extracted instance features, x i represents a specific instance, y represents the label variable, and y j represents the jth label value, Represents x i The logical label vector of , c is the number of labels, and the specific j-th label value of auxiliary task t is recorded as θ is a parameter in the neural network, and the probability value P(x i |θ) and the probability value of the jth adjacent image P(x j |θ) to guide the update of ShuffleNetv2 model parameters.
[0037] It can be seen from the above technical solution that, compared with the prior art, the present invention discloses a facial expression recognition method based on a lightweight label-assisted learning network, which has the following beneficial effects:
[0038] (1) The ShuffleNet v2 model used is a lightweight model, and the processed features are divided into four small parts, which further highlights the focus on the facial area, thereby improving the accuracy and generalization of expression recognition;
[0039] (2) The features are convolved before being concatenated, which reduces computational overhead;
[0040] (3) The global features obtained after the Stage 2 layer processing are used to calculate the global spatial heat map, so that the model only focuses on the key areas and reduces the attention to unnecessary areas, thereby reducing the redundant information during calculation and improving the real-time performance of the model;
[0041] (4) Generate auxiliary label task space as a reference for main task prediction, reduce the occurrence of main task prediction errors, and provide a more reliable prediction method for difficult-to-recognize expressions, thereby improving the accuracy and generalization of the model;
[0042] (5) The auxiliary task label distribution loss and cross entropy loss are combined to jointly optimize the entire model, which does not increase excessive calculations and effectively improves the performance of the recognition model, meeting the requirements of real-time, accuracy, and generalization of the actual application of the project. BRIEF DESCRIPTION OF THE DRAWINGS
[0043] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.
[0044] Figure 1 A flow chart of a facial expression recognition method based on a lightweight label-assisted learning network provided by the present invention;
[0045] Figure 2 The overall framework of a facial expression recognition method based on a lightweight label-assisted learning network disclosed in the present invention;
[0046] Figure 3 A schematic diagram of an auxiliary task learning module provided in an embodiment disclosed in the present invention;
[0047] Figure 4 A schematic diagram of constructing an auxiliary task label space provided in an embodiment disclosed in the present invention;
[0048] Figure 5 A schematic diagram of the expression recognition process provided for the disclosed example of the present invention. DETAILED DESCRIPTION
[0049] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0050] Combination Figure 5 The present invention discloses a facial expression recognition method based on a lightweight label-assisted learning network, comprising the following steps. Figure 1 As shown:
[0051] S1. Capturing facial expression images through an image acquisition device;
[0052] S2. Input the facial expression image into the ShuffleNet v2 model for global-local F overall Feature extraction;
[0053] S3. Input the facial expression images captured in S1 into the label distribution generator to generate the auxiliary task label space (see Figure 4 ), search for multiple nearest neighbor samples and labels as references for the main task prediction;
[0054] S4. Jointly optimize the ShuffleNetv2 model using auxiliary task label distribution loss and cross entropy loss;
[0055] S5. Obtain facial expression image recognition results based on the ShuffleNetv2 model optimized by S4.
[0056] For further information, see Figure 2 As shown, the specific content of S2 is:
[0057] The ShuffleNetv2 model includes: Conv1, Stage2, Stage3, Stage4, Conv5;
[0058] S21. After Conv1, the underlying global mapping F is obtained conv1 ∈ H×W×C , where H, W, and C represent the length, width, and number of channels of the mapping feature, respectively;
[0059] S22. Bottom-level global mapping F conv1 ∈ H×W×C After passing through the local feature extractor, it is divided into 4 equal blocks. Where i∈{1,2,3,4}, the 4 global maps are processed by two corresponding 3×3 convolutions to obtain local facial features Local facial features Splicing along the spatial axis to finally obtain local features
[0060] Further, H=W=56, C=29, H1=W1=28.
[0061] For further information, see Figure 2As shown, S2 also includes:
[0062] S23. After the facial expression image passes through the Stage 2 layer, high-level global features are obtained
[0063] S24. Advanced global features After channel-space modulation, the global-local total feature F is obtained overall .
[0064] Furthermore, the specific content of S24 is:
[0065] S241. Advanced global features Calculate channel heatmaps separately in two parallel branches and spatial heatmap The channel heat map H(Fc2)∈ C 1 and spatial heat map The whole heat map is obtained by adding and normalizing. The specific calculation formula is as follows:
[0066] H(F s 2)=σ(H c (F s 2) +H s (F s twenty one)
[0067] Where σ(·) is the Sigmoid function, H c (F c 2) is the channel heat map, H s (F s 2) is the spatial heat map, H(F s 2) The whole heat map after merging and normalizing the two heat maps;
[0068] S242. Modulated global feature map F mod The calculation is as follows:
[0069]
[0070] In the formula, represents element-wise multiplication, F l It is a local feature;
[0071] S243. Extracting local features F l and modulate the global feature F mod After that, the final global-local feature calculation process is as follows:
[0072] F overall =F s 2+F mod (3)
[0073] The final output is the global-local total feature F overall .
[0074] Furthermore, in S241, channel heatmap and spatial heatmap Before the merger, they were adjusted to Achieve dimensional consistency in operations.
[0075] For further information, see Figure 3 and Figure 4 As shown, the specific content of S3 is:
[0076] The auxiliary tasks are action unit recognition (AUR) and facial landmark detection (FLD), and the auxiliary tasks describe facial structure and movement from different perspectives.
[0077] Specifically, first, an approximate k-nearest neighbor graph of the training data is pre-established. Each graph contains all the images in the training set. The indexes of each image and its adjacent images are stored in a list, which is an index similarity list. Figure 4 As shown, node 0 is the center image. Nodes 1 and 2 are the two images closest to node 0 in the action unit space. Similarly, nodes 3 and 4 are the two images closest to node 0 in the facial point space.
[0078] Furthermore, the label distribution loss in S4 is defined as follows:
[0079]
[0080] In the formula, x represents the extracted instance features, x i represents a specific instance, y represents the label variable, and y j represents the jth label value, Represents x i The logical label vector of , c is the number of labels, and the specific j-th label value of auxiliary task t is recorded as θ is a parameter in the neural network, and the probability value P(x i |θ) and the probability value of the jth adjacent image P(x j |θ) to guide the update of ShuffleNetv2 model parameters.
[0081] Furthermore, the topological information of the auxiliary task label space is used to guide the update of the ShuffleNetv2 model parameters. Specifically, the label distribution loss is first used to calculate and optimize the entire classification model, and the deviation between the predicted probability value of the central image and the probability value of the adjacent images in the auxiliary task is used to guide the update of the ShuffleNetv2 model parameters. Then the similarity index between the central image and the adjacent images is calculated. The similarity index describes the relative importance of the ShuffleNetv2 model prediction to the adjacent images.
[0082] Updating the ShuffleNet v2 model parameters is achieved by estimating the distance or similarity, which describes the relative importance of the ShuffleNet v2 model predictions to adjacent images. The similarity metric is shown in the following formula:
[0083]
[0084] In the formula, the similarity measurement index is where is the set of neighbors in the auxiliary task label space, and x in auxiliary task t i The label vector is recorded as Images that are close to each other in the auxiliary label space have similar expression distributions. Larger means P(x i |θ) and P(x j |θ) is closer. The auxiliary task label strategy is the basis of the guidance loss formula, which is shown as follows:
[0085]
[0086] The auxiliary task label strategy is transformed into an optimization framework through equations (4) and (6), and the total loss function T(θ) is obtained as shown below:
[0087]
[0088] Where argminL(θ) represents the value of the independent variable required when finding the minimum value of the function, and λ is the parameter that balances the lightweight network loss function and the auxiliary task label loss function.
[0089] In one specific implementation, see Figure 5 As shown, it is a schematic diagram of the expression recognition process disclosed in the present invention.
[0090] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.
[0091] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A facial expression recognition method based on a lightweight label-assisted learning network, characterized in that: The following steps are involved: S1. Capturing facial expression images through an image acquisition device; S2. Input the facial expression image into the ShuffleNetv2 model for global-local F overall Feature extraction; S3. Input the facial expression image captured in S1 into the label distribution generator to generate the auxiliary task label space, and search for multiple nearest neighbor samples and labels as references for the main task prediction; S4. Jointly optimize the ShuffleNetv2 model using auxiliary task label distribution loss and cross entropy loss; S5. Obtain facial expression image recognition results based on the ShuffleNetv2 model optimized by S4.
2. A facial expression recognition method based on a lightweight label-assisted learning network according to claim 1, characterized in that: The specific contents of S2 are: The ShuffleNetv2 model includes: Conv1, Stage2, Stage3, Stage4, Conv5; S21. After Conv1, the underlying global mapping F is obtained conv1 ∈ H×W×C , where H, W, and C represent the length, width, and number of channels of the mapping feature, respectively; S22. Bottom-level global mapping F conv1 ∈ H×W×C After passing through the local feature extractor, it is divided into 4 equal blocks. Where i∈{1,2,3,4}, the 4 global maps are processed by two corresponding 3×3 convolutions to obtain local facial features Local facial features Splicing along the spatial axis to finally obtain local features 3. A facial expression recognition method based on a lightweight label-assisted learning network according to claim 2, characterized in that: H=W=56, C=29, H1=W1=28.
4. A facial expression recognition method based on a lightweight label-assisted learning network according to claim 3, characterized in that: S2 also includes: S23. After the facial expression image passes through the Stage 2 layer, high-level global features are obtained S24. Advanced global features After channel-space modulation, the global-local total feature F is obtained overall .
5. A facial expression recognition method based on a lightweight label-assisted learning network according to claim 4, characterized in that: The specific contents of S24 are: S241. Advanced global features Calculate channel heatmaps separately in two parallel branches and spatial heatmap Channel heatmap and spatial heatmap The whole heat map is obtained by adding and normalizing. The specific calculation formula is as follows: H(F s2 )=σ(H c (F s2 )+H s (F s2 )) (1) Where σ(·) is the Sigmoid function, H c (F c2 ) is the channel heat map, H s (F s2 ) is the spatial heat map, H(F s2 ) is the entire heat map after the two heat maps are merged and normalized; S242. Modulated global feature map F mod The calculation is as follows: In the formula, represents element-wise multiplication, F l It is a local feature; S243. Extracting local features F l and modulate the global feature F mod After that, the final global-local feature calculation process is as follows: F overall =F s2 +F mod (3) The final output is the global-local total feature F overall .
6. A facial expression recognition method based on a lightweight label-assisted learning network according to claim 5, characterized in that: In S241, channel heat map and spatial heatmap Before merging, they are adjusted to H1×W1×C1 to achieve consistent dimensions of operations.
7. A facial expression recognition method based on a lightweight label-assisted learning network according to claim 6, characterized in that: The specific contents of S3 are: The auxiliary tasks are action unit recognition (AUR) and facial landmark detection (FLD), and the auxiliary tasks describe facial structure and movement from different perspectives.
8. A facial expression recognition method based on a lightweight label-assisted learning network according to claim 7, characterized in that: The label distribution loss in S4 is defined as follows: In the formula, x represents the extracted instance features, x i represents a specific i-th instance, y represents the label variable, yj represents the j-th label value, Represents x i The logical label vector of , c is the number of labels, and the specific j-th label value of auxiliary task t is recorded as θ is a parameter in the neural network, and the probability value P(x i |θ) and the probability value of the jth adjacent image P(x j |θ) to guide the update of ShuffleNet v2 model parameters.