Low-resolution facial expression super-resolution and recognition method based on deep learning

Abstract

The invention discloses a low-resolution facial expression super-resolution and recognition method based on deep learning, and particularly relates to the technical field of image processing. According to the method, the network is improved based on cavity convolution, soft pooling and an attention mechanism, the receptive field of the network is expanded through the cavity convolution, and the capture capability of the model on fine changes of facial expressions is enhanced. In the soft pooling, weights are distributed for feature values in each pooling area through a softmax function, the response of key channels and important spatial positions in an input feature map is enhanced, and irrelevant or redundant information is inhibited at the same time. And the attention mechanism module enhances the modeling capability of the dependency relationship between the channels by sharing the weight of a convolution kernel. The method can achieve the super-resolution reconstruction and precise recognition of the facial expression image, and is suitable for various scenes such as public safety, human-computer interaction, driver state monitoring and the like.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to a low-resolution facial expression super-resolution and recognition method based on deep learning. BACKGROUND

[0002] Facial expression, as an important non-verbal communication method, can intuitively reflect a person's emotional state and psychological activity. Through the subtle activities of facial muscles, the face will show various expressions, which are rich and complex. In recent years, with the rapid development of deep learning technology, the field of facial expression recognition has also made important progress. However, most of the existing expression recognition models mainly rely on high-resolution images for training, which faces several challenges in practical applications. Low-resolution facial expression images will reduce the recognition accuracy of these models, thereby limiting the wide application of expression recognition technology in practical scenarios. The research of facial expression recognition involves multiple disciplines such as psychology, cognitive science and artificial intelligence. This multi-disciplinary cross-fusion not only promotes the in-depth development of related fields, but also provides new ideas and methods for solving complex problems. Some super-resolution algorithms can recover detailed images from low-resolution images, which provides a new idea for solving the problem of low-resolution facial expression recognition.

[0003] In the FER (Facial Expression Recognition) task, existing research has attempted to use super-resolution technology to solve the problem of decreased expression recognition accuracy caused by insufficient image resolution. In view of the challenges faced by low-resolution facial expression image recognition, exploring the combination of super-resolution technology and FER method has become an urgent and important research topic. This not only requires technical innovation, but also requires in-depth analysis and optimization of existing super-resolution algorithms and facial expression recognition algorithms. SUMMARY

[0004] The present application discloses a low-resolution facial expression super-resolution and recognition method based on deep learning, which realizes the super-resolution reconstruction and accurate recognition of facial expression images.

[0005] The technical scheme adopted by the present application is:

[0006] A low-resolution facial expression super-resolution and recognition method based on deep learning, the method comprising the following steps:

[0007] Step 1, based on the super-resolution network built, the input low-resolution facial expression image is super-resolved and reconstructed;

[0008] The super-resolution network sequentially comprises a shallow feature extraction module, a deep feature extraction module, and an image reconstruction module; the shallow feature extraction module performs shallow feature extraction on the input low-resolution facial expression image based on a progressive feature extraction manner; the deep feature extraction module sequentially comprises a plurality of RSTB (Residual SwinTransformer block) modules and a convolution layer, and adopts a residual connection mechanism to fuse the input shallow features and the deep features output by the convolution layer to obtain fused features output by the deep feature extraction module; and the image reconstruction module is used to reconstruct the fused features into a high-resolution facial expression image.

[0009] Step 2: input the high-resolution facial expression image obtained by super-resolution reconstruction into the built facial expression recognition network to obtain a facial expression recognition result.

[0010] The facial expression recognition network comprises three parts, the first part performs feature extraction based on a convolution layer and performs down-sampling on a spatial dimension; the second part performs feature mapping expression on the feature map extracted by the first part based on a plurality of bottleneck layer structures; and the third part outputs an expression category based on a convolution layer and a soft pooling layer.

[0011] Further, the shallow feature extraction module is specifically configured as a plurality of stacked convolution blocks and a plurality of stacked first bottleneck modules, wherein the convolution block sequentially comprises a 3×3 convolution layer, a batch normalization layer, and an activation function layer.

[0012] Preferably, the activation function of the activation function layer is set to ReLU.

[0013] Preferably, the number of convolution blocks of the shallow feature extraction module is set to 3.

[0014] Further, the structure of the first bottleneck module is that the input feature map is divided into four sub-feature maps X1-X4 of the same dimension; from the second sub-feature map, it is respectively sent into three convolution layers with a 3×3 convolution kernel; and from the third sub-feature map, the output of the convolution layer of the previous sub-feature map is added to the current sub-feature Figure One and sent into a convolution layer; finally, the sub-feature map and the outputs of the three convolution layers with a 3×3 convolution kernel are output together through a convolution layer with a 1×1 convolution kernel to obtain the output of the first bottleneck module.

[0015] Further, the deep feature extraction module comprises six RSTB modules.

[0016] Further, the convolution kernel of the convolution layer of the deep feature extraction module is set to 3×3.

[0017] Further, the image reconstruction module sequentially comprises a convolution layer, a lightweight up-sampling module and a convolution layer.

[0018] Further, the first part of the facial expression recognition network sequentially comprises a convolution layer, a batch normalization layer and an activation function.

[0019] Preferably, the convolution kernel of the convolution layer is set to 3*3, and the activation function adopts h-swish.

[0020] Further, the second part of the facial expression recognition network comprises 15 bottleneck layer structures, the first to the third adopt the first bottleneck layer structure, and the fourth to the fifteenth adopt the second bottleneck layer structure.

[0021] Further, the first bottleneck layer structure sequentially comprises a convolution layer with a convolution kernel of 1*1, a dilated convolution layer with a kernel of 3*3, a CBAM (Convolutional Block Attention Module) module, a convolution layer with a convolution kernel of 1*1, a TiedSE (Tied Squeeze and Excitation) module and a convolution layer with a convolution kernel of 1*1, wherein the input of the last convolution layer is the fusion of the input and output of the TiedSE module; and there is a skip connection between the input of the first bottleneck layer structure and the output of the last convolution layer.

[0022] Further, the second bottleneck layer structure sequentially comprises a convolution layer with a convolution kernel of 1*1, a depth convolution layer with a kernel of 3*3, a CBAM module, a convolution layer with a convolution kernel of 1*1, a TiedSE module and a convolution layer with a convolution kernel of 1*1, wherein the input of the last convolution layer is the fusion of the input and output of the TiedSE module; and there is a skip connection between the input of the first bottleneck layer structure and the output of the last convolution layer.

[0023] Further, the third part of the facial expression recognition network sequentially comprises a convolution layer with a convolution kernel of 1*1, a soft pooling layer and two convolution layers with a convolution kernel of 1*1.

[0024] The technical scheme provided by the present application at least brings the following beneficial effects:

[0025] The application is based on improvement of the network by using cavity convolution, soft pooling, and attention mechanism, expands the network receptive field by using cavity convolution, and enhances the model's ability to capture subtle changes in facial expressions. Soft pooling assigns weights to feature values in each pooling region by using a softmax function, enhances the response of key channels and important spatial positions in the input feature map, and suppresses irrelevant or redundant information. The attention mechanism module enhances the modeling ability of the inter-channel dependency by sharing the convolution kernel weights. The application can realize super-resolution reconstruction and accurate recognition of facial expression images, and is suitable for public safety, human-computer interaction, driver state monitoring and other scenes. BRIEF DESCRIPTION OF DRAWINGS

[0026] In order to more clearly illustrate the technical solutions in the embodiments of the application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative labor.

[0027] Figure 1 Res2Net-Lite network structure schematic diagram;

[0028] Figure 2 Super-resolution model structure diagram;

[0029] Figure 3 Expression recognition model structure diagram. DETAILED DESCRIPTION

[0030] In order to make the purpose, technical scheme and advantages of the embodiments of the application clearer, the technical solutions in the embodiments of the application will be described in detail and completely below in combination with the drawings in the embodiments of the application. Obviously, the described embodiments are only some embodiments of the application, not all embodiments. Generally, the components of the embodiments of the application described and shown in the drawings can be arranged and designed using different configurations. Therefore, the following detailed description of the embodiments of the application provided in the drawings is not only to limit the scope of the claimed application, but only to represent selected embodiments of the application.

[0031] In order to realize super-resolution reconstruction and accurate recognition of facial expression images, the embodiment of the application proposes a low-resolution facial expression super-resolution and recognition method based on deep learning, which is suitable for public safety, human-computer interaction, driver state monitoring and other scenes. The specific technical scheme is as follows:

[0032] Step 1: super-resolution reconstruction of facial expression images.

[0033] On the basis of SwinIR network, Res2Net-Lite and CARAFE technologies are introduced to obtain the improved RC-SwinIR network. Among them, the Res2Net-Lite network is based on the Res2Net network and is obtained by optimizing the network structure, which can more effectively extract the multi-scale detail features of the image. The CARAFE up-sampling technology realizes the dynamic adjustment of the features of different regions of the image through the learned reorganization kernel, so as to better preserve the detail information of the image.

[0034] Step 2: facial expression image recognition.

[0035] First, the super-resolution reconstruction method of the previous step is applied to the low-resolution facial expression dataset for super-resolution reconstruction. The training and testing of the facial expression recognition algorithm are based on the facial expression dataset after RC-SwinIR super-resolution reconstruction.

[0036] Secondly, on the basis of the MobileNetV3_Large network, the improved MobileNetV3-DSCB network is obtained by introducing the technologies of hole convolution, soft pooling, CBAM attention mechanism and TiedSE algorithm.

[0037] Finally, the improved network is used to perform facial expression recognition on the images after super-resolution reconstruction processing.

[0038] In one embodiment, step 1 specifically comprises the following steps:

[0039] Step 1.1: The input image is first subjected to shallow feature extraction by the Res2Net-Lite network.

[0040] Res2Net-Lite is a lightweight feature extraction network, as described in Figure 1 The core structure thereof includes multiple convolution blocks and several Bottleneck blocks. Each convolution block includes a convolution layer, a batch normalization (BN) and an activation function layer in sequence; the structure of the Bottleneck block is as shown in Figure 1Res2Net-Lite, as shown on the left side, is composed of several convolutional layer combinations, that is, the Res2Net network residual unit structure is used. In this structure, first, the input feature map is divided into four sub-feature maps X1-X4 through the Split operation: the dimension of each feature map is BxHxWxC / 4, where B is the number of images input into the network, HxW represents the width and height of the feature map respectively, and C is the number of channels of the feature map. Y1-Y4 are the outputs of each sub-feature map after multi-scale feature extraction, where Y1=X1, Y2=K(X2), Y3=K(X3+Y2), and Y4=K(X4+Y3). K() represents a convolution operation using a 3x3 convolution kernel, that is, the output of a 3x3 convolution layer. Finally, Y1-Y4 are input into a 1x1 convolution layer to obtain the output of the Bottleneck block. Res2Net-Lite can reduce the computational complexity while extracting preliminary feature information of the image, laying a foundation for subsequent deep feature extraction. In addition, the multi-scale feature extraction capability of Res2Net-Lite enables it to capture both local details and global structure information in the image, which is particularly important for the super-resolution task of facial expression images.

[0041] Step 1.2: The shallow layer features extracted in step 1.1 are input into a deep feature extraction module composed of several RSTB (Residual SwinTransformer block) modules, as shown in Figure 2 In this embodiment, there are 6 RSTBs. The RSTB module is composed of several stacked STL (Swin Transformer Layer) and a convolution layer, and there is a jump connection between the input and output of the RSTB module. One of the core modules of RC-SwinIR is RSTB, which contains multiple STLs. Each STL is composed of a multi-head self-attention mechanism (SW-MSA), a layer normalization (LayerNorm), and a multi-layer perceptron (MLP). Among them, SW-MSA can capture long-range dependencies in the image, thereby better understanding the global structure of the image; MLP further enhances the expression ability of the features through nonlinear transformation. By stacking multiple RSTBs, the network can gradually extract higher-level semantic information, thereby providing more rich context information support for the image super-resolution task.

[0042] Step 1.3: The features of the shallow and deep layers extracted in steps 1.1 and 1.2 are further fused through a convolutional layer and added to the initial shallow features to obtain the fused features. This residual connection mechanism not only effectively alleviates the gradient vanishing problem, but also fully utilizes the information of shallow and deep features, thereby further enhancing the stability and expression ability of the network. Specifically, shallow features mainly contain local detail information of the image, while deep features capture higher-level semantic information. By combining the two through residual connection, the network can better understand the global structure of the image while preserving the details, thereby achieving higher quality reconstruction in the image super-resolution task.

[0043] Step 1.4: The fused features from step 1.3 are passed through a reconstruction module to generate a high-resolution image. The reconstruction module includes two 3x3 convolutional layers and a CARAFE (Content-Aware ReAssembly of FEatures) upsampling operation, as shown in Figure 2 CARAFE is a content-aware upsampling method that can dynamically generate a recombination kernel based on the input features, thereby achieving efficient upsampling while preserving image details.

[0044] In one embodiment, step 2 includes the following steps:

[0045] Step 2.1: First, a 3x3 convolutional layer is used to extract features from the preprocessed facial expression image, followed by batch normalization and a Hardswish activation function to accelerate training convergence and enhance the model's non-linear expression ability.

[0046] Step 2.2: The initial features extracted in step 2.1 are passed through a series of inverted residual blocks for feature learning. Each inverted residual block uses a depthwise separable convolution and a linear bottleneck structure, and integrates a TiedSE module and a CBAM (Convolutional Block Attention Module) module to further enhance feature extraction capabilities. In particular, in the first 1 to 3 inverted residual blocks, a dilated convolution is enabled with a dilation rate of 2, while in the 4th to 5th inverted residual blocks, the dilation rate is set to 1, equivalent to a standard depthwise separable convolution. This means that only the first 3 of the 15 residual blocks have a dilated convolution enabled, and the remaining 9 do not.

[0047] Step 2.3: The feature map obtained in step 2.2 is compressed into a single vector through a soft pooling layer. Subsequently, the feature vector is passed through two 1x1 convolutional layers to complete the final classification output, not only integrating global feature information but also further enhancing the model's ability to capture subtle changes in facial expressions.

[0048] In one embodiment, the super-resolution network for implementing super-resolution reconstruction of facial expression images adopted by the embodiment is as shown in Figure 1 Specifically, as shown in

[0049] The network architecture of RC-SwinIR mainly consists of three core modules: a shallow feature extraction module, a deep feature extraction module, and an image reconstruction module. RC-SwinIR network first extracts shallow features from the input low-resolution image through the Res2Net-Lite network. The improvements to the Res2Net network mainly include the following two aspects: first, at the feature extraction layer, replace the first 7x7 convolutional layer with three 3x3 convolutional layers, each with a normalization layer and a ReLU activation function after convolution, and expand the output channel number from 64 to 180. This progressive feature extraction method not only enhances the model's expression ability, but also improves the image detail feature capture ability through multi-level convolution structure. Second, in terms of model structure optimization, by simplifying the network level and only retaining a single residual block layer, the original maximum pooling layer and redundant multiple residual block layers of the Res2Net network are removed, reducing the model's parameter quantity, not only realizing the lightweight of the model, but also improving the training and inference efficiency, making it more suitable for resource-constrained environments. The Res2Net-Lite network structure diagram is as shown in Figure 1 Then, 6 RSTB modules and a convolutional layer are used for deep feature extraction. In the feature fusion stage, residual connection mechanism is used to fuse shallow features and deep features, and finally high-resolution images are output through the image reconstruction module. The RC-SwinIR network structure diagram is as shown in Figure 2 Based on the above network construction, in the model training process, a two-stage training strategy is adopted. The first stage is the freezing training stage, based on the pre-trained model, the parameter freezing strategy is adopted, and the initial training is carried out by selectively freezing part of the parameters. The second stage is the global fine-tuning stage, all parameters are unfrozen to further optimize the model performance, and the loss function uses L1 loss function. After optimization, the PSNR index of the model performance is improved by 2.84 dB at most, and the SSIM index is improved by 0.0504 at most.

[0050] In one embodiment, the facial expression recognition network for implementing facial expression image recognition adopted by the embodiment is as shown in Figure 3 Specifically, as shown in

[0051] First, the face expression image after super-resolution processing is input to a 3x3 convolution layer for processing, which sets the output channel number to 16 and realizes spatial dimension down-sampling through a step of 2, effectively reducing the computational complexity while retaining key features. Subsequently, the network adopts a 16-bottleneck layer structure. Each bottleneck layer is flexibly configured according to different convolution kernel sizes (3x3 or 5x5), expansion ratios, output channel numbers, and whether to enable the TiedSE module. In the first three bottleneck layers of the network, the hole convolution technique is introduced, which expands the receptive field by adjusting the dilation rate, enabling the capture of more rich contextual information without increasing computational complexity, which is particularly important for extracting expression features in low-resolution images. In the bottleneck layers of the network, ReLU and Hard Swish activation functions are used alternately. ReLU accelerates model convergence through its simple nonlinear characteristics, while Hard Swish enhances the model's expression ability for complex features through its smooth gradient characteristics. In addition, some bottleneck layers are further down-sampled in spatial dimension by setting different steps. This strategy reduces the amount of calculation while retaining key spatial feature information. Finally, the expression category is output through 3 1x1 Conv and soft pooling layers, and the MobileNetV3-DSCB network structure is as shown in Figure 3 Based on the above network construction, the loss function uses the cross-entropy loss function, and the accuracy of the optimized model is improved by 1.48% compared with the original model.

[0052] In the description of the present specification, the description of the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" and the like means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In the present specification, the illustrative description of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any appropriate manner in any one or more embodiments or examples. In addition, the skilled in the art can combine and combine the different embodiments or examples described in the present specification and the features of the different embodiments or examples without contradiction.

[0053] In addition, the terms "first", "second", and the like are used for descriptive purposes only and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Therefore, the features defined with "first", "second", etc. can explicitly or implicitly include at least one of the features.

[0054] Any process or method described in the specification can be understood as representing a module, segment, or portion of code that includes one or more executable instructions for implementing the specified logical function(s) or process(es) and that the alternative implementations of the preferred embodiments of the application include other implementations that can not be exactly as described in the specification but that can be equivalent in terms of the functionality implemented, and that the scope of the preferred embodiments of the application encompasses these and other alternative implementations.

[0055] Those skilled in the art can understand that all or part of the steps of the method carried out by the above-mentioned embodiments can be instructed by programs to relevant hardware, and the programs can be stored in a computer readable storage medium. When the program is executed, it includes one or a combination of steps of the method embodiments.

[0056] Note that all features disclosed in the specification can be replaced by alternative features serving the same, equivalent or similar purpose unless otherwise explicitly stated. Therefore, each disclosed feature is only one example of a set of equivalent or similar features unless explicitly stated otherwise. In the case of use, further, preferably, further and more preferably are simply the starting point for another embodiment described on the basis of the preceding embodiment, and the content following the further, preferably, further or more preferably is combined with the preceding embodiment as a complete configuration of another embodiment. The combination of several further, preferably, further or more preferably settings following the same embodiment can form another embodiment.

[0057] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

[0058] The above only describes some embodiments of the present application. For those skilled in the art, without departing from the concept of the present application, several modifications and improvements can be made, which are within the scope of protection of the present application.

Claims

1. A deep learning-based low-resolution facial expression super-resolution and recognition method, characterized in that, The method comprises the following steps: Step 1, performing super-resolution reconstruction on the input low-resolution facial expression image based on the built super-resolution network; The super-resolution network comprises a shallow feature extraction module, a deep feature extraction module and an image reconstruction module in sequence; the shallow feature extraction module performs shallow feature extraction on the input low-resolution facial expression image based on a progressive feature extraction manner; the deep feature extraction module comprises a plurality of RSTB (Residual Swin Transformer block) modules and a convolution layer in sequence, and adopts a residual connection mechanism to fuse the input shallow features and the deep features output by the convolution layer to obtain fused features output by the deep feature extraction module; and the image reconstruction module is used for reconstructing the fused features into a high-resolution facial expression image; Step 2, inputting the high-resolution facial expression image obtained by the super-resolution reconstruction into the built facial expression recognition network to obtain a facial expression recognition result; The facial expression recognition network comprises three parts, the first part performs feature extraction based on a convolution layer and performs spatial dimension down-sampling; the second part performs feature mapping expression on the feature map extracted by the first part based on a plurality of bottleneck layer structures; and the third part outputs an expression category based on a convolution layer and a soft pooling layer.

2. The method of claim 1, wherein, The shallow feature extraction module is specifically provided as a plurality of stacked convolution blocks and a plurality of stacked first bottleneck modules, wherein the convolution block comprises a 3×3 convolution layer, a batch normalization layer and an activation function layer in sequence.

3. The method of claim 2, wherein, The structure of the first bottleneck module is that the input feature map is divided into four sub-feature maps X1-X4 of the same dimension, from the second sub-feature map, the three convolution layers with a 3×3 convolution kernel are respectively fed into the three convolution layers with a 3×3 convolution kernel; and from the third sub-feature map, the output of the convolution layer of the previous sub-feature map is fed into the convolution layer together with the current sub-feature map; finally, the sub-feature map and the output of the three convolution layers with a 1×1 convolution kernel are output together through the convolution layer with a 1×1 convolution kernel to obtain the output of the first bottleneck module.

4. The method of claim 1, wherein, The deep feature extraction module comprises six RSTB modules.

5. The method of claim 1, wherein, The convolution kernel of the convolution layer of the deep feature extraction module is set to 3×3.

6. The method of claim 1, wherein, The image reconstruction module comprises a convolution layer, a lightweight up-sampling module and a convolution layer in sequence.

7. The method of claim 1, wherein, The first part of the facial expression recognition network comprises a convolution layer, a batch normalization layer and an activation function in sequence.

8. The method of claim 1, wherein, The second part of the facial expression recognition network comprises 15 bottleneck layer structures, the first to third bottleneck layer structures are the first bottleneck layer structures; and the fourth to fifteenth bottleneck layer structures are the second bottleneck layer structures.

9. The method of claim 8, wherein, The first bottleneck layer structure comprises, in sequence, a convolution layer with a convolution kernel of 1*1, a dilated convolution layer with a kernel of 3*3, a CBAM (Convolutional Block Attention Module) module, a convolution layer with a convolution kernel of 1*1, a TiedSE (Tied Squeeze and Excitation) module and a convolution layer with a convolution kernel of 1*1, wherein the input of the last convolution layer is the fusion of the input and output of the TiedSE module; and there is a skip connection between the input of the first bottleneck layer structure and the output of the last convolution layer. The second bottleneck layer structure comprises, in sequence, a convolution layer with a convolution kernel of 1*1, a depth convolution layer with a kernel of 3*3, a CBAM module, a convolution layer with a convolution kernel of 1*1, a TiedSE module and a convolution layer with a convolution kernel of 1*1, wherein the input of the last convolution layer is the fusion of the input and output of the TiedSE module; and there is a skip connection between the input of the first bottleneck layer structure and the output of the last convolution layer.

10. The method of claim 1, wherein, The third part of the face expression recognition network comprises, in sequence, a convolution layer with a convolution kernel of 1*1, a soft pooling layer and two convolution layers with a convolution kernel of 1*1.

Images