An expression recognition method and system fusing lightweight convolution and feature weighting mechanism

Abstract

This invention discloses a facial expression recognition method and system that integrates lightweight convolution and feature weighting mechanisms. It includes the following steps: acquiring a dataset of facial expressions and performing feature enhancement preprocessing on the dataset; inputting the dataset into a feature extraction network, using deformable convolution to perform shallow feature extraction on the image, obtaining shallow features; adaptively adjusting the convolution sampling position to extract shallow adaptive features of the facial image, obtaining a feature map; based on the shallow features, constructing a depth-separable residual feature extraction module including an attention mechanism, performing weighted modeling on the features to obtain deep semantic features of the facial expression; performing multi-scale feature fusion on the weighted features, fusing the shallow features with the deep semantic features to obtain fused features; inputting the fused features into a classifier to output the expression recognition result. The advantages are: improved accuracy and stability of facial expression recognition.

Description

Technical Field

[0001] This invention belongs to the field of computer vision, and in particular relates to an expression recognition method and system that integrates lightweight convolution and feature weighting mechanisms. Background Technology

[0002] Facial expressions are a typical form of nonverbal communication and play a crucial role in sentiment analysis. Studies show that facial expressions account for 55% of emotional expression, making them a key characteristic of emotional communication. With the rapid development and application of artificial intelligence, facial expression recognition research has been widely applied in fields such as intelligent teaching, human-computer interaction, intelligent monitoring, safe driving, and medical diagnosis. Facial expression recognition methods mainly employ traditional methods and deep learning-based methods. Traditional methods involve manually designing features or using shallow learning to extract image features, such as Gabor wavelets, histogram of oriented gradients, principal component analysis, Haar features, and local binary patterns. Traditional feature extraction methods rely heavily on the researcher's subjective experience, are only effective for simple tasks, have limitations, and poor generalization ability.

[0003] With the development of computer vision and deep learning technologies, deep learning-based facial expression recognition methods have gradually become the mainstream research direction in this field. Existing methods typically rely on neural network models, using multi-layer nonlinear transformations to extract and represent features from facial images, thereby obtaining more discriminative expression features and reducing the risk of feature information loss, thus improving the accuracy of expression recognition. Among existing technologies, one type of method constructs a Siamese network or multi-branch network structure to map facial images to an embedded feature space and achieves expression recognition through feature similarity metrics; another type combines traditional handcrafted features with deep features, introducing feature constraints or auxiliary loss functions during network training to guide the learning process of deep features. Furthermore, some methods optimize the network structure or hyperparameters to improve the overall performance of the model. To enhance the model's ability to focus on expression regions, some existing technologies introduce Region of Interest (ROI) guidance mechanisms or attention mechanisms, enabling the network to focus on key facial regions to improve its ability to model expression changes. Simultaneously, for computationally limited or practical application scenarios, some technologies combine edge computing frameworks, utilizing convolutional neural networks and recurrent neural networks for end-to-end training of facial images or image sequences to achieve emotion recognition functionality. Although the above methods have improved the performance of facial expression recognition to some extent, they still have the following shortcomings: First, the feature extraction process does not make full use of key local expression features, making it difficult to take into account both global information and local details; Second, some network structures, while pursuing deep feature expression, are prone to introducing redundant features or causing the decay of effective feature information; Third, it is difficult to achieve a balance between the need for lightweight design and high recognition accuracy, which affects the application effect of the model in real-world scenarios. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to provide an expression recognition method and system that can effectively enhance the expression ability of key features while ensuring the model is lightweight. This method integrates lightweight convolution and feature weighting mechanisms.

[0005] To address the aforementioned technical problems, the facial expression recognition method of the present invention, which integrates lightweight convolution and feature weighting mechanisms, includes the following steps:

[0006] Step 1: Collect facial expressions to obtain a dataset, and perform feature enhancement preprocessing on the dataset;

[0007] Step 2: Input the preprocessed dataset with feature enhancement into the feature extraction network. Use deformable convolution to perform shallow feature extraction on the image to obtain shallow features. By adaptively adjusting the convolution sampling position, extract shallow adaptive features of the face image to obtain the feature map. ;

[0008] Step 3: Based on the shallow features, construct a deep separable residual feature extraction module with an attention mechanism, perform weighted modeling on the features, and obtain the deep semantic features of facial expressions;

[0009] Step 4: Perform a multi-scale feature fusion operation on the weighted features to fuse the shallow features with the deep semantic features to obtain fused features;

[0010] Step 5: Input the fused features into the classifier and output the expression recognition result.

[0011] The feature enhancement preprocessing in step 1 includes random horizontal flipping, random rotation, color jittering, and Gaussian blurring.

[0012] Step 2, the process of shallow feature extraction of the image using deformable convolution, specifically includes:

[0013] 2.1 Generate position offset for each sampling point of the deformable convolution kernel in deformable convolution. ;

[0014] 2.2 Based on the aforementioned position offset Adjust the sampling position of the deformable convolution kernel to obtain the new sampling position;

[0015] 2.3 The bilinear interpolation method is used to calculate the feature values ​​at the new sampling location. ;

[0016] 2.4 Assign the feature values ​​of each sampling point to their corresponding weights Weighted summation outputs shallow features ,in:

[0017]

[0018] Where R is the set of sampling point offsets of the deformable convolution kernel. For deformable convolution kernels in Weighting on.

[0019] Step 3 includes a depthwise separable residual feature extraction module with an attention mechanism. This module comprises feature extraction modules at different depths, and the process specifically includes:

[0020] 3.1 Feature Map The input is fed into the first group of 3×3 separable convolutional layers for feature extraction, and the first intermediate feature map is output. ;

[0021] 3.2 Transfer the first intermediate feature map The input is fed into a second group of 3×3 separable convolutional layers, which output a second intermediate feature map. :

[0022] 3.3 The second intermediate feature map Input dual-branch channel attention mechanism branches:

[0023] 3.3.1 to Perform global max pooling and global average pooling respectively to obtain two one-dimensional channel feature vectors;

[0024] 3.3.2 The two one-dimensional channel feature vectors are respectively input into a one-dimensional convolutional layer with adaptively adjusted kernel size and a ReLU activation function for processing, generating channel feature weight matrices. and ,

[0025]

[0026] in, Indicates global max pooling. Indicates global average pooling. This represents the ReLU activation function. This represents a one-dimensional convolution operation that adaptively adjusts the kernel size.

[0027] 3.3.3 will and , The enhanced channel features are obtained by multiplying them separately and then adding them together. The formula is as follows:

[0028]

[0029] 3.4 Transfer the first intermediate feature map Input space attention mechanism branch:

[0030] 3.4.1 Pair Max pooling and average pooling are performed sequentially to obtain the spatial feature map. and , where (H,W) is the feature map size;

[0031] 3.4.2 will and After concatenation, the data is processed through a 7×7 convolutional layer and the ReLU activation function to generate a spatial feature weight matrix. ,

[0032]

[0033] In the formula, Concat represents channel concatenation, Maxpool(.) represents max pooling, and Avgpool(.) represents average pooling;

[0034] 3.4.3 The spatial feature weight matrix With the first intermediate feature map Multiplication yields the spatial domain output features. The formula is as follows:

[0035]

[0036] 3.5 Enhance the channel features Spatial domain output features After addition, the sums are processed by batch normalization and the Mish activation function to output the final weighted feature map. ,

[0037] Represented as:

[0038]

[0039] Where BN represents the batch normalization layer function.

[0040] The multi-scale fusion process in step 4 specifically includes:

[0041] 4.1 The feature maps output by multiple feature extraction modules at different depths are downsampled by pooling or strided convolution to keep their spatial resolution consistent;

[0042] 4.2 After downsampling, each feature map is transformed by convolution to ensure that its channel dimensions remain consistent;

[0043] 4.3 Multi-layer feature maps with consistent channel dimensions and spatial resolution are fused by channel splicing or weighted fusion to obtain multi-scale fused feature maps.

[0044] In step 4.1, the spatial size of the feature map after downsampling is 3×3.

[0045] In step 4.2, the channel dimension of each feature map is uniformly set to 1024.

[0046] A system for facial expression recognition that integrates lightweight convolution and feature weighting mechanisms includes the following steps:

[0047] A. Obtain the image data to be classified;

[0048] B. Perform standardization and feature enhancement preprocessing on the image data;

[0049] C. Extracting spatial adaptive features using deformable convolutional networks;

[0050] D. Use a deep separable residual attention network to weighted fuse features;

[0051] E. Feature-based image category discrimination;

[0052] F. Output the classification results.

[0053] Advantages of this invention:

[0054] By introducing a deformable convolutional structure in the shallow feature extraction stage, the convolutional sampling position is adaptively adjusted to adapt to the local geometric changes of facial expressions. This effectively preserves detailed information of key areas such as eyes, eyebrows, and corners of the mouth in the early stages of feature extraction, reducing the loss of original expression features. Building upon this, a deep separable residual feature extraction module incorporating channel attention and spatial attention mechanisms is constructed. This module weights and models features while maintaining network lightweightness, making the network focus more on key information related to expression discrimination and suppressing redundant features, further enhancing the discriminative power of feature representation. Simultaneously, this invention employs a multi-scale feature fusion strategy to effectively fuse shallow local subtle features with deep high-level semantic features, fully utilizing feature information from different receptive fields to improve the model's robustness and generalization ability to complex expression changes. Therefore, this invention improves the accuracy and stability of facial expression recognition while reducing model parameters and computational complexity. It is suitable for deployment in application scenarios with limited computing resources or high real-time requirements, and has high practical application value. Attached Figure Description

[0055] Figure 1 This is a technical roadmap of the present invention;

[0056] Figure 2 This is a schematic diagram of the deformable convolution processing in this invention;

[0057] Figure 3This is a schematic diagram of the feature extraction process of the depth-separable residual attention module in this invention;

[0058] Figure 4 This is a schematic diagram of the multi-scale fusion strategy processing in this invention;

[0059] Figure 5 This is a visualization of the attention mechanism in this invention. Detailed Implementation

[0060] The following detailed description of the facial expression recognition method and system integrating lightweight convolution and feature weighting mechanisms of the present invention, with reference to the accompanying drawings and specific embodiments, is further provided.

[0061] Example 1:

[0062] An expression recognition method that integrates lightweight convolution and feature weighting mechanisms includes the following steps:

[0063] Step 1: Obtain a dataset of facial expression images, standardize the image data, and perform feature enhancement preprocessing to generate an enhanced training sample set; wherein, the feature enhancement preprocessing includes random horizontal flipping, random rotation, color jittering, and Gaussian blurring.

[0064] Step 2: Input the obtained enhanced training sample set into the deformable convolutional network. By adaptively adjusting the convolution sampling position, shallow adaptive features of the face image are extracted to obtain the feature map. ;

[0065] Step 3: Based on the shallow features obtained in Step 2, a deep separable residual attention mechanism network is constructed as a feature extraction module. The deep separable residual attention mechanism network maps features through residual connections and introduces channel attention mechanism and spatial attention mechanism to perform weighted processing on features in the channel domain and spatial domain, and outputs weighted features.

[0066] Step 4: Perform multi-scale feature fusion on the weighted features to fuse shallow fine-grained features with deep high semantic features, so as to make full use of the complementary information between features of different scales and improve the integrity and discriminability of feature representation.

[0067] Step 5: Input the fused features into the classification network for classification and discrimination, and output the corresponding facial expression category results.

[0068] Specifically, such as Figure 2 As shown, step 2 includes:

[0069] Deformable convolution adds an offset to each position of the original convolution sampling. The shape of the convolution kernel is dynamically and adaptively adjusted to better adapt to image features of different shapes and orientations. Therefore, the shallow feature extraction module of this invention uses deformable convolution to remove the influence of non-critical features such as background on subsequent feature extraction. Deformable convolution can be represented as...

[0070]

[0071] in, This is the offset. For the convolution kernel on The weights, the implementation process of the 3×3 deformable convolution in this invention is as follows: Figure 2 As shown, the offset layer generates the offset of each pixel. , used to adjust the position of the convolution kernel; The pixels applied to each convolution kernel are moved to a new position, which is obtained by adding the corresponding pixel's offset to the original convolution kernel position. Since the offset is usually not an integer, bilinear interpolation is used to calculate it. To obtain the feature values ​​of the sampled points after the offset.

[0072] Specifically, such as Figure 3 As shown, step 3 includes:

[0073] Feature map Where (H,W) is the feature map size, and C is the number of channels. The data consists of real numbers (all values ​​in a neural network are real numbers). Two concatenated 3×3 depthwise separable convolutions are used to extract features. The output feature map of the first convolution is denoted as... The output feature map of the second convolution is denoted as... Each convolutional operation is followed by a batch normalization (BN) layer and the Mish activation function.

[0074] Will A dual-branch channel attention mechanism is used. The first branch acquires local channel features through global max pooling, while the second branch acquires global channel features through global average pooling. The pooling results are then input into a one-dimensional convolutional function with adaptively adjusted kernel size and a ReLU activation function to obtain the channel feature weight matrix. and :

[0075]

[0076] in, Indicates global max pooling. Indicates global average pooling. This represents the ReLU activation function. This represents a one-dimensional convolution operation that adaptively adjusts the kernel size.

[0077] Will respectively with and Multiply and then add to obtain the enhanced channel features. :

[0078]

[0079] Will The input spatial attention mechanism module obtains spatial feature maps through max pooling and average pooling. (H,W) represents the feature map dimensions) and (H,W) represents the feature map size. The two feature maps are concatenated and then input into a 7×7 convolution followed by ReLU activation to obtain the spatial feature weight matrix. :

[0080]

[0081] In the formula, Concat represents channel concatenation, Maxpool(.) represents max pooling, and Avgpool(.) represents average pooling.

[0082] Enhanced channel characteristics Spatial feature weight matrix Multiplication yields the spatial domain output features. :

[0083]

[0084] The final output feature can be represented as:

[0085]

[0086] Where BN represents the batch normalization layer function.

[0087] Specifically, such as Figure 4 As shown, step 4 includes:

[0088] A multi-scale feature fusion module is employed to fuse image features from different receptive fields. This module includes four Depthally Separable Residual Attention (DSRAM) modules, used to extract shallow and deep feature information respectively. Shallow feature maps are larger in size and have fewer channels, containing subtle image information, while deep feature maps are smaller in size and have more channels, containing abstract semantic information. The multi-scale feature fusion module downsamples the feature maps output from the first three DSRAM modules to a 3×3 spatial size, then upsamples them to 1024 channels using 1×1 convolution. These are then fused with the feature map output from the last DSRAM module using a channel concatenation (Concat) method, resulting in an output feature map of size 3×3×1024. This output feature map contains multi-scale feature information, thereby improving the utilization rate of features at each level.

[0089] Specific experiment:

[0090] 1. Experimental Datasets: This invention uses three publicly available datasets: FER2013, CK+, and RAF-DB, among which:

[0091] The FER2013 dataset contains 35,887 facial expression images with a size of 48×48, divided into 7 categories. The training set has 28,708 images, while the public and private validation sets each have 3,589 images.

[0092] The CK+ dataset contains 593 video sequences from 123 different subjects (aged 18 to 50) showing the transition of facial expressions from neutral to target peak expressions. This invention uses 327 labeled image sequences (a total of 981 images).

[0093] The RAF-DB dataset is a large-scale facial expression dataset consisting of 29,672 images obtained from thousands of people in non-laboratory environments. The experiments in this invention used images representing seven basic facial expression categories. This dataset features more natural environmental and lighting conditions, making it closer to real-world scenarios.

[0094] 2. Parameter settings, step 1, are as follows:

[0095] Randomly flipping the image horizontally improves the model's adaptability to different image angles; randomly rotating the image improves the model's adaptability to image rotation; color dithering increases the image's diversity and robustness; and Gaussian blurring improves the model's generalization ability. Image enhancement parameters are shown in Table 1.

[0096] Table 1 Data Augmentation Parameter Table

[0097]

[0098] The remaining parameters are as follows: two deformable convolutional layers (3×3), four depthwise separable residual attention modules, with a depthwise separable convolutional kernel size of 3×3, a stride of 2, and padding of 1. The feature map size and parameter variations are shown in Table 2.

[0099] Table 2. Variation of Feature Map Size and Parameters

[0100]

[0101] 3. Experimental Content and Results

[0102] The experimental comparison methods included: SNNs, AWHFLNet, RAPNET, CNNHO, VGGNet, DeepEmotion, DAAL, IDSCNet, MCARNet, VGG19-CBAM, MBACNet, ROI-Guided, SE-Net50, NSVT, DRL, CUDL, VGG16Net, HCIA, AMT, DoubleNet, E2-CapsNet, FixMatch, and Transformer-VIT. The experimental results are shown in Table 3.

[0103] Table 3 Comparison of Experimental Results for Each Network

[0104]

[0105] To verify the effectiveness of the method of this invention, three publicly available datasets—FER2013, CK+, and RAF-DB—were standardized, including image scaling, normalization, and necessary data augmentation. The processed data was then input into DSRANT for training, and comparisons were made with methods such as VGG16Net, IDSCNet, and SE-Net50. Experimental results show that the recognition accuracy of the method of this invention is higher than that of the comparative methods on all three datasets, thus verifying the effectiveness of the invention.

[0106] To verify the accuracy and advancement of the recognition method of the present invention, the results of the real-time performance test of the network model using a computer device are shown in the figure. It can be clearly seen that the recognition network of the present invention can distinguish each expression well and has a good recognition effect.

[0107] Example 2:

[0108] A system for facial expression recognition that integrates lightweight convolution and feature weighting mechanisms includes the following steps:

[0109] A. Obtain the image data to be classified;

[0110] B. Perform standardization and feature enhancement preprocessing on the image data;

[0111] C. Extracting spatial adaptive features using deformable convolutional networks;

[0112] D. Use a deep separable residual attention network to weighted fuse features;

[0113] E. Feature-based image category discrimination;

[0114] F. Output the classification results.

Claims

1. A facial expression recognition method integrating lightweight convolution and feature weighting mechanisms, characterized in that, Includes the following steps: Step 1: Collect facial expressions to obtain a dataset, and perform feature enhancement preprocessing on the dataset; Step 2: Input the preprocessed dataset with feature enhancement into the feature extraction network. Use deformable convolution to perform shallow feature extraction on the image to obtain shallow features. By adaptively adjusting the convolution sampling position, extract shallow adaptive features of the face image to obtain the feature map. ; Step 3: Based on the shallow features, construct a deep separable residual feature extraction module with an attention mechanism, perform weighted modeling on the features, and obtain the deep semantic features of facial expressions; Step 4: Perform a multi-scale feature fusion operation on the weighted features to fuse the shallow features with the deep semantic features to obtain fused features; Step 5: Input the fused features into the classifier and output the expression recognition result.

2. The facial expression recognition method according to claim 1, which integrates lightweight convolution and feature weighting mechanisms, is characterized in that: The feature enhancement preprocessing in step 1 includes random horizontal flipping, random rotation, color jittering, and Gaussian blurring.

3. The facial expression recognition method according to claim 1, which integrates lightweight convolution and feature weighting mechanisms, is characterized in that: Step 2, the process of shallow feature extraction of the image using deformable convolution, specifically includes: 2.1 Generate position offset for each sampling point of the deformable convolution kernel in deformable convolution. ; 2.2 Based on the aforementioned position offset Adjust the sampling position of the deformable convolution kernel to obtain the new sampling position; 2.3 The bilinear interpolation method is used to calculate the feature values ​​at the new sampling location. ; 2.4 Assign the feature values ​​of each sampling point to their corresponding weights Weighted summation outputs shallow features ,in: ； Where R is the set of sampling point offsets of the deformable convolution kernel. For deformable convolution kernels in Weighting on.

4. The facial expression recognition method according to claim 1, which integrates lightweight convolution and feature weighting mechanisms, is characterized in that: Step 3 includes a depthwise separable residual feature extraction module with an attention mechanism. This module comprises feature extraction modules at different depths, and the process specifically includes: 3.1 Feature Map The input is fed into the first group of 3×3 separable convolutional layers for feature extraction, and the first intermediate feature map is output. ; 3.2 Transfer the first intermediate feature map The input is fed into a second group of 3×3 separable convolutional layers, which output a second intermediate feature map. : 3.3 The second intermediate feature map Input dual-branch channel attention mechanism branches: 3.3.1 to Perform global max pooling and global average pooling respectively to obtain two one-dimensional channel feature vectors; 3.3.2 The two one-dimensional channel feature vectors are respectively input into a one-dimensional convolutional layer with adaptively adjusted kernel size and a ReLU activation function for processing, generating channel feature weight matrices. and , ； in, Indicates global max pooling. Indicates global average pooling. This represents the ReLU activation function. This represents a one-dimensional convolution operation that adaptively adjusts the kernel size. 3.3.3 will and , The enhanced channel features are obtained by multiplying them separately and then adding them together. The formula is as follows: ； 3.4 Transfer the first intermediate feature map Input space attention mechanism branch: 3.4.1 Pair Max pooling and average pooling are performed sequentially to obtain the spatial feature map. and , where (H,W) is the feature map size; 3.4.2 will and After concatenation, the data is processed through a 7×7 convolutional layer and the ReLU activation function to generate a spatial feature weight matrix. , ； In the formula, Concat represents channel concatenation, Maxpool(.) represents max pooling, and Avgpool(.) represents average pooling; 3.4.3 The spatial feature weight matrix With the first intermediate feature map Multiplication yields the spatial domain output features. The formula is as follows: ； 3.5 Enhance the channel features Spatial domain output features After addition, the sums are processed by batch normalization and the Mish activation function to output the final weighted feature map. , Represented as: ； Where BN represents the batch normalization layer function.

5. The facial expression recognition method according to claim 4, which integrates lightweight convolution and feature weighting mechanisms, is characterized in that: The multi-scale fusion process in step 4 specifically includes: 4.1 The feature maps output by multiple feature extraction modules at different depths are downsampled by pooling or strided convolution to keep their spatial resolution consistent; 4.2 After downsampling, each feature map is transformed by convolution to ensure that its channel dimensions remain consistent; 4.3 Multi-layer feature maps with consistent channel dimensions and spatial resolution are fused by channel splicing or weighted fusion to obtain multi-scale fused feature maps.

6. The facial expression recognition method according to claim 5, which integrates lightweight convolution and feature weighting mechanisms, is characterized in that: In step 4.1, the spatial size of the feature map after downsampling is 3×3.

7. The facial expression recognition method according to claim 5, which integrates lightweight convolution and feature weighting mechanisms, is characterized in that: In step 4.2, the channel dimension of each feature map is uniformly set to 1024.

8. A system for facial expression recognition based on the lightweight convolution and feature weighting mechanism described in any one of claims 1-7, characterized in that, Includes the following steps: A. Obtain the image data to be classified; B. Perform standardization and feature enhancement preprocessing on the image data; C. Extracting spatial adaptive features using deformable convolutional networks; D. Use a deep separable residual attention network to weighted fuse features; E. Feature-based image category discrimination; F. Output the classification results.

Images