Facial expression recognition method and system based on deep joint model
By constructing a deep joint model for facial expression recognition, combining the MobileNetV2 backbone network and dynamic bottleneck blocks, and utilizing labeled and unlabeled datasets, the problem of small training data volume is solved, and high-precision facial expression recognition is achieved.
Patent Information
- Application Number
- CN202210891705.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-07-27
- Publication Date
- 2025-11-11
- Estimated Expiration
- 2042-07-27
AI Technical Summary
Existing facial expression recognition methods suffer from limited training data and the rich facial expression information in existing datasets is not effectively utilized, which limits the improvement of deep models in facial expression recognition accuracy.
A facial expression recognition method based on a deep joint model is constructed, which combines deep learning technology and expression-guided face clustering. By constructing labeled and unlabeled datasets, the network is trained and features are extracted using the MobileNetV2 backbone network and dynamic bottleneck blocks, and the dataset is optimized by expression-guided face clustering algorithm.
It achieves high-precision facial expression recognition under small data conditions, reduces the need for labeled data, improves the separability and clustering effect of the feature space, and reduces the amount of computation.
Smart Images
Figure CN115240258B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of facial emotion recognition methods, specifically to a facial expression recognition method and system based on a deep joint model. Background Technology
[0002] Facial features are the most commonly used features in expression recognition and personal identification. Facial expression recognition is one of the most fundamental tasks in face analysis, playing a crucial role in understanding emotional states and intentions. Facial expression recognition technology mainly consists of two parts: feature extraction and recognition. Extracting useful features from facial expression images has a significant impact on the recognition accuracy: highly separable features can greatly improve recognition accuracy; conversely, poorly separable features will significantly reduce recognition performance.
[0003] In recent years, deep learning has achieved remarkable results in feature extraction. To improve the accuracy of facial expression recognition, various deep learning models have been introduced into the recognition of facial expression images. By using deep learning models to autonomously extract deep features from facial expression images, the accuracy of facial expression recognition can be effectively improved.
[0004] However, existing methods for extracting deep features from facial expression images using deep models require a large amount of training data for network training. Labeling each facial expression image is extremely time-consuming and challenging for psychologists. Therefore, existing datasets for facial expression recognition suffer from limited data volume, a significant obstacle to the development of deep models in this field. Meanwhile, existing face recognition datasets contain a wealth of unused facial expression information. Therefore, researching how to simultaneously utilize the abundant unlabeled data in rich face datasets and the limited labeled data in facial expression datasets to build deep models for high-accuracy facial expression image recognition is a challenging task. Summary of the Invention
[0005] The purpose of this invention is to provide a facial expression recognition method based on a deep joint model. This method aims to address the problem of small training data volume in facial expression recognition methods by combining deep learning-related technologies to design a deep joint learning method based on a novel efficient network and expression-guided face clustering.
[0006] This invention provides a facial expression recognition method based on a deep joint model, comprising the following steps:
[0007] Construct a training dataset, which includes a labeled dataset and an unlabeled dataset;
[0008] A facial expression recognition network model is constructed based on the MobileNetV2 backbone network and using dynamic bottleneck blocks. The facial expression recognition network model is trained using a labeled dataset.
[0009] The trained facial expression recognition network model is used to obtain the facial expression depth feature representation of the training dataset, and the initial labels of the facial expression depth features of the unlabeled dataset are obtained.
[0010] The initial labels of facial expression depth features in the unlabeled dataset and the clustering results of facial expression depth features in the unlabeled dataset are compared. Facial expression data with high confidence and their facial expression depth feature labels in the unlabeled dataset are added to the labeled dataset.
[0011] The facial expression recognition network model was tuned using the updated labeled training dataset;
[0012] The adjusted facial expression recognition network model is used to perform expression recognition on the facial expression images to be recognized, and the recognition results are obtained.
[0013] Furthermore, the construction of the training dataset specifically includes:
[0014] Obtain the dataset of facial expression images and the dataset of face recognition images to be processed;
[0015] The Retinaface algorithm is used to detect and align the facial expression image dataset and the face recognition image dataset to be processed, respectively.
[0016] The sizes of the facial expression image dataset and the face recognition image dataset to be processed are normalized respectively;
[0017] Data augmentation was performed on the facial expression image dataset and the face recognition image dataset to be processed, using random cropping and random horizontal flipping with a 50% probability.
[0018] A portion of the facial expression image dataset was divided into a labeled training dataset XL, and the MS-Celeb-1M-v1c dataset from the face recognition image dataset was divided into an unlabeled training dataset Xu.
[0019] Furthermore, the facial expression recognition network model constructed based on the MobileNetV2 backbone network and utilizing dynamic bottleneck blocks includes the following steps:
[0020] Construct dynamic convolution kernels;
[0021] A dynamic bottleneck block is constructed by directly inserting an attention mechanism block into each dynamic convolutional kernel;
[0022] Dynamic bottleneck blocks are grouped into different layers in the MobileNetV2 backbone network based on the size of the input feature map of the dynamic bottleneck blocks; the first layer of the MobileNetV2 backbone network is a standard convolutional layer, and the remaining layers are dynamic bottleneck blocks.
[0023] The feature maps of the MobileNetV2 backbone network are transformed into 1024-dimensional feature vectors using average pooling and convolutional layers.
[0024] Furthermore, the construction of the dynamic convolution kernel specifically includes:
[0025] Dynamic convolution is used to generate intermediate feature maps of the input data.
[0026] Y′=X*f′(x)+b′ (1)
[0027] in,
[0028] Y′ is an intermediate feature map. h′, w′, and m represent the length, width, and number of channels of the intermediate feature mapping data, respectively;
[0029] X represents the length, width, and number of channels of the input data. h, w, and c represent the length, width, and number of channels of the input data, respectively;
[0030] f′ represents the dynamic convolution kernel operation. Its convolution kernel size is k×k;
[0031] Based on the intermediate feature maps, a new set of feature maps is generated through linear operations of grouped convolution;
[0032] The intermediate feature maps and the new feature maps are stacked together to obtain the final output data, which includes the length, width, and number of channels of the output data. h′, w′, and n represent the length, width, and number of channels of the output data, respectively.
[0033] Furthermore, when utilizing the labeled dataset X... L Before training the facial expression recognition network model, the model is pre-trained using a face recognition image dataset to obtain the initial parameters of the facial expression recognition network model.
[0034] Furthermore, a face clustering method guided by facial expressions is used to perform deep clustering on the unlabeled dataset, resulting in the clustering result {X} of the unlabeled dataset. u c (m)}
[0035] Furthermore, the use of labeled dataset X LTraining a facial expression recognition network model includes:
[0036] The labeled training dataset X L The data is input in batches into the facial expression recognition network model;
[0037] Facial expression features are extracted from a facial expression recognition network model, and the prediction results of the input data are calculated. The loss function between the predicted results and the actual results of the input data is expressed as L. combined ,but,
[0038] L combined =L C +λL A (2)
[0039] Where L C For center loss function; L A The additional angle margin loss function is λ, which is a scaling parameter and is set to 0.05.
[0040] Calculate the partial derivatives of the network weights of the facial expression recognition network model, and update the network parameters using the stochastic gradient descent algorithm.
[0041] Furthermore, it also includes using another portion of the facial expression image dataset to be processed as a test dataset to test the trained facial expression recognition network model.
[0042] This invention provides a facial expression recognition system based on a deep joint model, comprising:
[0043] The dataset building module is used to build the training dataset, which includes the labeled dataset X. L and unlabeled dataset X u ;
[0044] The model building module is used to build a facial expression recognition network model based on the MobileNetV2 backbone network and using dynamic bottleneck blocks, and to train the facial expression recognition network model using a labeled dataset.
[0045] The feature extraction module is used to obtain the facial expression depth feature representation of the training dataset using the trained facial expression recognition network model, and to obtain the initial labels {X} of the facial expression depth features of the unlabeled dataset. u p (m)};
[0046] The dataset update module is used for the initial labeling of facial expression deep features in the unlabeled dataset {X}. u p (m)} and clustering results of facial expression deep features from unlabeled datasets {Xu c (m) The dataset X is compared, and the facial expression data with high confidence and their facial expression depth feature labels from the unlabeled dataset are added to the labeled dataset X. L ;
[0047] The model tuning module is used to tune the facial expression recognition network model using the updated labeled training dataset;
[0048] The facial expression recognition module is used to perform facial expression recognition on the facial expression image to be recognized using the adjusted facial expression recognition network model, and obtain the recognition result.
[0049] Compared with the prior art, the beneficial effects of the present invention are as follows:
[0050] This invention constructs a facial expression recognition network model using a novel dynamic convolutional kernel. This dynamic kernel significantly reduces the number of parameters and, compared to other convolutional kernels used in lightweight networks, exhibits stronger feature representation capabilities. The facial expression recognition network model reduces the need for labeled data during network training and provides a more separable deep feature space, facilitating recognition and clustering within that space. Furthermore, the expression-guided face clustering algorithm effectively reduces the dependence of face clustering on facial identity information features, making it more suitable for large-scale face clustering. It delivers excellent clustering results with lower computational cost, providing more favorable conditions for improving network training performance. This invention achieves autonomous extraction of deep features and labeled data from facial expression images under limited data conditions, resulting in high-precision recognition. Compared to existing deep learning-based facial expression recognition methods, this invention requires less data and achieves higher accuracy. Attached Figure Description
[0051] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:
[0052] Figure 1 This is a flowchart of the facial expression recognition method based on a deep joint model proposed in this invention;
[0053] Figure 2 This is a schematic diagram of the dynamic and efficient convolutional structure of the facial expression recognition method based on a deep joint model proposed in this invention;
[0054] Figure 3 This is a schematic diagram of the dynamic bottleneck block of the facial expression recognition method based on a deep joint model proposed in this invention. Detailed Implementation
[0055] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. However, it should be understood that the scope of protection of the present invention is not limited to the specific implementation.
[0056] Example 1
[0057] like Figure 1-3 As shown, the facial expression recognition method based on a deep joint model includes the following steps:
[0058] Step 1: Construct the training dataset, which includes: a labeled dataset X L and unlabeled dataset X u Constructing the training dataset involves the following steps:
[0059] Step 1.1: Obtain the dataset of facial expression images to be processed and the dataset of face recognition images;
[0060] Step 1.2: Use the Retinaface algorithm to detect and align the facial expression image dataset and the face recognition image dataset to be processed, respectively;
[0061] Step 1.3: Normalize the size of the facial expression image dataset and the face recognition image dataset to be processed;
[0062] Step 1.4: Perform data augmentation on the facial expression image dataset and the face recognition image dataset to be processed using random cropping and random horizontal flipping with a 50% probability;
[0063] Step 1.5: Divide a portion of the facial expression image dataset into a labeled training dataset X. L The MS-Celeb-1M-v1c dataset in the face recognition image dataset is divided into an unlabeled training dataset X. u .
[0064] In this embodiment, all facial image data in the facial expression image dataset and the face recognition image dataset are adjusted to a size of 224×224 pixels as input.
[0065] Step 2: Construct a facial expression recognition network model based on the MobileNetV2 backbone network and utilizing dynamic bottleneck blocks, using the labeled dataset X. L Training a facial expression recognition network model;
[0066] Step 2.1 Constructing dynamic convolutional kernels, specifically including:
[0067] Dynamic convolution is used to generate intermediate feature maps of the input data.
[0068] Y′=X*f′(x)+b′ (1)
[0069] Where Y′ represents the intermediate feature map, h′, w′, and m represent the length, width, and number of channels of the intermediate feature mapping data, respectively;
[0070] X represents the length, width, and number of channels of the input data. h, w, and c represent the length, width, and number of channels of the input data, respectively;
[0071] f′ represents the dynamic convolution kernel operation. Its convolution kernel size is k×k;
[0072] Based on the intermediate feature maps, a new set of feature maps is generated through linear operations of grouped convolution;
[0073] The intermediate feature maps and the new feature maps are stacked together to obtain the final output data, which includes the length, width, and number of channels of the output data. h′, w′, and n represent the length, width, and number of channels of the output data, respectively.
[0074] Step 2.2: Construct a dynamic bottleneck block by directly inserting an attention mechanism block into each dynamic convolution kernel;
[0075] Step 2.3: Group the dynamic bottleneck blocks into different layers of the MobileNetV2 backbone network according to the size of the input feature map of the dynamic bottleneck block; the first layer of the MobileNetV2 backbone network is a standard convolutional layer, and the remaining layers are dynamic bottleneck blocks, including:
[0076] Step 2.4: Use average pooling and convolutional layers to convert the feature maps of the MobileNetV2 backbone network into 1024-dimensional feature vectors.
[0077] Step 3: Using the labeled dataset X L Before training the facial expression recognition network model, the model is pre-trained using a face recognition image dataset to obtain the initial parameters of the facial expression recognition network model.
[0078] Step 4: Utilize the labeled dataset X L Training a facial expression recognition network model includes:
[0079] The labeled training dataset X L The data is input in batches into the facial expression recognition network model;
[0080] Facial expression features are extracted from a facial expression recognition network model, and the prediction results of the input data are calculated. The loss function between the predicted results and the actual results of the input data is expressed as L. combined ,but,
[0081] L combined =L C +λL A (2)
[0082] Where L C For center loss function; L A The additional angle margin loss function is λ, which is a scaling parameter and is set to 0.05.
[0083] Calculate the partial derivatives of the network weights of the facial expression recognition network model, and update the network parameters using the stochastic gradient descent algorithm.
[0084] In this embodiment, 10-20 data points are randomly and non-repeatingly selected from the labeled training dataset XL to form a batch of training data. This training data is input into the facial expression recognition network model to extract features and calculate prediction results. The partial derivatives of the network weights are calculated, and the network parameters are updated using the stochastic gradient descent algorithm. One training epoch is defined as traversing the entire training set once. The entire training process consists of 200 epochs, with an initial learning rate of 0.001, which decays to 0.0001 in the last 10 epochs. Throughout the training process, the momentum term is set to 0.9.
[0085] Step 5: Use the trained facial expression recognition network model as a feature extractor for the training dataset to obtain the deep feature representation of expressions in the training dataset, and obtain the unlabeled dataset X. u The initial selection tag {X u p (m)};
[0086] Step 6: Perform deep clustering on the unlabeled dataset using an expression-guided face clustering method to obtain the clustering results {X} of the unlabeled dataset. u c (m)};
[0087] Step 7: For the unlabeled dataset X u The initial selection tag {X u p (m)} and unlabeled dataset X u Clustering results {X u c (m) The data is compared, and facial expression labels of the facial expression data with high confidence are selected and labeled. The facial expression data with high confidence and their labels are then added to the labeled dataset X. L ;
[0088] Step 6: Use the updated labeled training dataset X L Adjust the facial expression recognition network model.
[0089] Step 7: Iterate through steps 3 to 6 until the maximum number of iterations t is reached, where t = 3.
[0090] Step 8: Based on the final trained facial expression recognition network model, use another part of the facial expression image dataset to be processed as the test dataset to test the trained facial expression recognition network model.
[0091] Step 9: Use a facial expression recognition network model to perform expression recognition on the facial expression image to be recognized, and obtain the recognition result.
[0092] Example 2
[0093] This invention provides a facial expression recognition system based on a deep joint model, comprising:
[0094] The dataset building module is used to build the training dataset, which includes the labeled dataset X. L and unlabeled dataset X u ;
[0095] The model building module is used to build a facial expression recognition network model based on the MobileNetV2 backbone network and using dynamic bottleneck blocks, and to train the facial expression recognition network model using a labeled dataset.
[0096] The feature extraction module is used to obtain the facial expression depth feature representation of the training dataset using the trained facial expression recognition network model, and to obtain the initial labels {X} of the facial expression depth features of the unlabeled dataset. u p (m)};
[0097] The dataset update module is used for the initial labeling of facial expression deep features in the unlabeled dataset {X}. u p (m)} and clustering results of facial expression deep features from unlabeled datasets {X u c (m) The dataset X is compared, and the facial expression data with high confidence and their facial expression depth feature labels from the unlabeled dataset are added to the labeled dataset X. L ;
[0098] The model tuning module is used to tune the facial expression recognition network model using the updated labeled training dataset;
[0099] The facial expression recognition module is used to perform facial expression recognition on the facial expression image to be recognized using the adjusted facial expression recognition network model, and obtain the recognition result.
[0100] Finally, it should be noted that the above-disclosed embodiment is only one specific embodiment of the present invention. However, the embodiments of the present invention are not limited thereto, and any variations that can be conceived by those skilled in the art should fall within the protection scope of the present invention.
Claims
1. A facial expression recognition method based on a deep joint model, characterized in that, Includes the following steps: Construct a training dataset, which is an image dataset, including: a labeled dataset and an unlabeled dataset; A facial expression recognition network model is constructed based on the MobileNetV2 backbone network and using dynamic bottleneck blocks. The facial expression recognition network model is trained using a labeled dataset. The trained facial expression recognition network model is used to obtain the facial expression depth feature representation of the training dataset, and the initial labels of the facial expression depth features of the unlabeled dataset are obtained. The initial labels of facial expression depth features in the unlabeled dataset and the clustering results of facial expression depth features in the unlabeled dataset are compared. Facial expression data with high confidence and their facial expression depth feature labels in the unlabeled dataset are added to the labeled dataset. The facial expression recognition network model was tuned using the updated labeled training dataset; The adjusted facial expression recognition network model is used to perform expression recognition on the facial expression image to be recognized, and the recognition result is obtained. The facial expression recognition network model based on the MobileNetV2 backbone network and utilizing dynamic bottleneck blocks includes the following steps: Construct dynamic convolution kernels; A dynamic bottleneck block is constructed by directly inserting an attention mechanism block into each dynamic convolutional kernel; Dynamic bottleneck blocks are grouped into different layers in the MobileNetV2 backbone network based on the size of the input feature map of the dynamic bottleneck blocks; the first layer of the MobileNetV2 backbone network is a standard convolutional layer, and the remaining layers are dynamic bottleneck blocks. The feature maps of the MobileNetV2 backbone network are transformed into 1024-dimensional feature vectors using average pooling and convolutional layers. The construction of the dynamic convolution kernel specifically includes: Dynamic convolution is used to generate intermediate feature maps of the input data. Y'=X*f'(x)+b' (1) in, Y' is an intermediate feature map. h', w', and m represent the length, width, and number of channels of the intermediate feature mapping data, respectively; X is the input data. h, w, and c represent the length, width, and number of channels of the input data, respectively; f' represents the dynamic convolution kernel operation. Its convolution kernel size is k×k; Based on the intermediate feature maps, a new set of feature maps is generated through linear operations of grouped convolution; The intermediate feature maps and the new feature maps are stacked together to obtain the final output data, where the output data... h', w', and n represent the length, width, and number of channels of the output data, respectively.
2. The facial expression recognition method based on a deep joint model according to claim 1, characterized in that: The construction of the training dataset specifically includes: Obtain the dataset of facial expression images and the dataset of face recognition images to be processed; The Retinaface algorithm is used to detect and align the facial expression image dataset and the face recognition image dataset to be processed, respectively. The sizes of the facial expression image dataset and the face recognition image dataset to be processed are normalized respectively; Data augmentation was performed on the facial expression image dataset and the face recognition image dataset to be processed, using random cropping and random horizontal flipping with a 50% probability. A portion of the facial expression image dataset is divided into a labeled training dataset X. L The MS-Celeb-1M-v1c dataset in the face recognition image dataset is divided into an unlabeled training dataset X. u .
3. The facial expression recognition method based on a deep joint model according to claim 1, characterized in that: Using the labeled dataset X L Before training the facial expression recognition network model, the model is pre-trained using a face recognition image dataset to obtain the initial parameters of the facial expression recognition network model.
4. The facial expression recognition method based on a deep joint model according to claim 1, characterized in that: The unlabeled dataset was subjected to deep clustering using an expression-guided face clustering method, yielding the clustering results {X}. u c1 (m1) } 5. The facial expression recognition method based on a deep joint model according to claim 3, characterized in that: The use of labeled dataset X L Training a facial expression recognition network model includes: The labeled training dataset X L The data is input in batches into the facial expression recognition network model; Facial expression features are extracted from a facial expression recognition network model, and the prediction results of the input data are calculated. The loss function between the predicted results and the actual results of the input data is expressed as L. combined ,but, THE combined =L C +λL A (2) Where L C For center loss function; L A The additional angle margin loss function is λ, which is a scaling parameter and is set to 0.
05. Calculate the partial derivatives of the network weights of the facial expression recognition network model, and update the network parameters using the stochastic gradient descent algorithm.
6. The facial expression recognition method based on a deep joint model according to claim 5, characterized in that: It also includes using another portion of the facial expression image dataset to be processed as a test dataset to test the trained facial expression recognition network model.
7. A facial expression recognition system based on a deep joint model, used to implement the facial expression recognition method based on a deep joint model according to claim 1, characterized in that: include: The dataset building module is used to build the training dataset, which includes the labeled dataset X. L and unlabeled dataset X u ; The model building module is used to build a facial expression recognition network model based on the MobileNetV2 backbone network and using dynamic bottleneck blocks, and to train the facial expression recognition network model using a labeled dataset. The feature extraction module is used to obtain the facial expression depth feature representation of the training dataset using the trained facial expression recognition network model, and to obtain the initial labels {X} of the facial expression depth features of the unlabeled dataset. u p (m1) }; The dataset update module is used to initially select labels {X} for the facial expression deep features of the unlabeled dataset. u p (m1) } and clustering results of facial expression deep features from unlabeled datasets {X u c1 (m1) The dataset X is compared, and the facial expression data with high confidence and their facial expression depth feature labels from the unlabeled dataset are added to the labeled dataset X. L ; The model tuning module is used to tune the facial expression recognition network model using the updated labeled training dataset; The facial expression recognition module is used to perform facial expression recognition on the facial expression image to be recognized using the adjusted facial expression recognition network model, and obtain the recognition result; The facial expression recognition network model based on the MobileNetV2 backbone network and utilizing dynamic bottleneck blocks includes the following steps: Construct dynamic convolution kernels; A dynamic bottleneck block is constructed by directly inserting an attention mechanism block into each dynamic convolutional kernel; Dynamic bottleneck blocks are grouped into different layers in the MobileNetV2 backbone network based on the size of the input feature map of the dynamic bottleneck blocks; the first layer of the MobileNetV2 backbone network is a standard convolutional layer, and the remaining layers are dynamic bottleneck blocks. The feature maps of the MobileNetV2 backbone network are transformed into 1024-dimensional feature vectors using average pooling and convolutional layers. The construction of the dynamic convolution kernel specifically includes: Dynamic convolution is used to generate intermediate feature maps of the input data. Y'=X*f'(x)+b' (1) in, Y' is an intermediate feature map. h', w', and m represent the length, width, and number of channels of the intermediate feature mapping data, respectively; X is the input data. h, w, and c represent the length, width, and number of channels of the input data, respectively; f' represents the dynamic convolution kernel operation. Its convolution kernel size is k×k; Based on the intermediate feature maps, a new set of feature maps is generated through linear operations of grouped convolution; The intermediate feature maps and the new feature maps are stacked together to obtain the final output data, where the output data... h', w', and n represent the length, width, and number of channels of the output data, respectively.