A multi-view expression recognition method based on dynamic neural network

Abstract

The application relates to a multi-view expression recognition method based on a dynamic neural network, which comprises the following steps: acquiring an image to be detected; performing face detection on the image, segmenting a face image and performing face region alignment; performing image preprocessing on the aligned face image; constructing an expression recognition model based on a deep convolutional neural network and performing training; the expression recognition model comprises a shallow feature extraction layer, a plurality of blocks composed of a convolution sampling module and a residual unit and a full connection layer which are connected in sequence, the preprocessed image is taken as input, and an expression recognition result is output, wherein the convolution sampling module dynamically generates a mask based on an input feature map, the generated mask is used for controlling the convolution position of a convolution layer in a corresponding residual unit, and dynamic range convolution is realized; and the trained expression recognition model is used for multi-view expression recognition. Compared with the prior art, the application has the advantages of high recognition accuracy, good robustness and the like.

Description

Technical Field

[0001] This invention relates to the field of image recognition technology, and in particular to a multi-view facial expression recognition method based on dynamic neural networks. Background Technology

[0002] Facial expressions are a crucial means of conveying human emotions, and accurate recognition of facial expressions can promptly reflect a person's psychological state and emotional information. Facial expression recognition is an important application area of ​​advanced intelligent systems. Intelligent systems utilize sensors such as cameras to perceive human emotions, thereby improving the emotion recognition and natural interaction capabilities of artificial intelligence systems and achieving better human-computer interaction. Therefore, facial expression recognition is becoming an increasingly popular problem in advanced intelligent systems and has crucial applications in fields such as healthcare, education, human-computer interaction, and driver fatigue monitoring.

[0003] Current facial expression recognition research is well-based on the classification of six basic emotions: anger, disgust, fear, happiness, sadness, and surprise. Traditional expression recognition methods primarily rely on manual feature extraction, including Gabor textures and local binary patterns. With the development of deep learning, end-to-end learning methods such as convolutional neural networks are becoming increasingly popular.

[0004] While many facial expression recognition systems achieve high accuracy in controlled laboratory environments, deployment in the real world is complex, as variations in race, gender, age, and posture can degrade recognition performance. Particularly regarding posture variations, most research relies on a frontal view as a foundation. However, this premise may not hold true in most real-world applications. For example, in natural human-computer interaction, cameras are often fixed to tables or integrated into screens, while people can move freely in space, causing frontal expression recognition systems to fail. Therefore, multi-view facial expression recognition is crucial in the real world. Multi-view facial expression recognition faces two significant challenges: firstly, high-quality multi-view expression datasets are scarce; secondly, it is difficult to design effective feature representations for multi-view expressions. Consequently, existing multi-view expression recognition technologies suffer from low accuracy and poor robustness. Summary of the Invention

[0005] The purpose of this invention is to provide a multi-view expression recognition method based on dynamic neural networks. The expression recognition model network is designed using a dynamic neural network architecture so that the network can be dynamically adapted to facial expressions from different perspectives. At the same time, the network can also dynamically adjust the feature map range of convolution calculation in the network according to different input images to achieve sparse convolution, thereby reducing the amount of computation and solving the problem of poor robustness of expression recognition methods to facial pose changes in the real world.

[0006] The objective of this invention can be achieved through the following technical solutions:

[0007] A multi-view facial expression recognition method based on dynamic neural networks includes the following steps:

[0008] S1. Acquire the image to be detected;

[0009] S2. Perform face detection on the image, segment the face image, and align the face regions.

[0010] S3. Perform image preprocessing on the aligned facial images;

[0011] S4. Construct and train an expression recognition model based on a deep convolutional neural network. The expression recognition model includes a shallow feature extraction layer, multiple blocks composed of convolutional sampling modules and residual units, and a fully connected layer connected in sequence. The preprocessed image is used as input, and the recognized expression result is output. The convolutional sampling module dynamically generates a mask based on the input feature map. The generated mask is used to control the convolution position of the convolutional layer in the corresponding residual unit to achieve dynamic range convolution.

[0012] S5. Use the trained facial expression recognition model to perform multi-view facial expression recognition.

[0013] Step S2 specifically involves: using a multi-task convolutional neural network to detect faces in the image, performing facial key point detection on the detected faces, segmenting the face region, and aligning the face region using the relative positions of the facial key points.

[0014] Step S3 specifically involves: performing image preprocessing on the aligned facial image, converting the image into an RGB image of a preset size, and performing normalization processing.

[0015] The backbone network of the facial expression recognition model is a ResNet18 network, which includes four blocks consisting of convolutional sampling modules and residual units. The input of the convolutional sampling module of the first block is the feature map output by the shallow feature sampling layer, and the input of the remaining convolutional sampling modules is the mask output by the previous convolutional sampling module and the feature map output by the previous residual unit.

[0016] The convolution sampling module processes the input image through a convolutional layer with 1 channel and then inputs it into the softmax layer. By setting a sampling threshold, the pixels in the softmax layer output image with values ​​greater than the sampling threshold are set to 1, and the pixels with values ​​less than the sampling threshold are set to 0, thus outputting a two-dimensional matrix mask composed of 0 and 1.

[0017] The convolutional layer within the residual unit performs convolution operations only on pixels with a value of 1 at the corresponding mask position in the image. For the remaining pixels, values ​​are assigned through interpolation.

[0018]

[0019] Where p is the pixel to be interpolated, s i For the pixels within the 8-neighborhood of pixel p, f(s) i ) represents s in the original graph i The value of a pixel after convolution, if s i Not sampled, f(s) i ) is 0, |ps i | is the p pixel and s i The absolute value between pixels.

[0020] Each residual unit of the facial expression recognition model consists of two residual blocks. The residual blocks are of the following types: ordinary residual blocks or residual blocks with downsampling. The ordinary residual block consists of two convolutional layers and a skip connection. The residual block with downsampling consists of a 1 / 2 downsampling convolutional layer, an ordinary convolutional layer, and a skip connection with 1 / 2 downsampling.

[0021] The number of channels in the four residual units are 64, 128, 256, and 512, respectively. The residual unit with 64 channels is composed of two ordinary residual blocks connected together, while the other residual units are composed of a residual block with downsampling and an ordinary residual block connected together in sequence.

[0022] The overall loss function of the facial expression recognition model is:

[0023] L = L FER +αL mask

[0024] Among them, L FER It is the loss function for the recognition results of the facial expression recognition model, representing the cross-entropy loss between the predicted facial expression result and the actual result; L mask α is the loss function of the convolutional sampling module, used to ensure the sparsity of the mask output by the convolutional sampling module; α is the scale factor, used to control the loss function L. mask The importance of.

[0025] The loss function L FER The expression is:

[0026]

[0027] Where y is the actual label value. These are the label values ​​predicted by the model, where x represents the sample and n is the total number of labels;

[0028] The loss function L mask Expressed using L1 norm regularization:

[0029]

[0030] Where l represents different network layers in the convolutional sampling module, w l The parameters represent the network, and ||·||1 represents the 1-norm.

[0031] Compared with the prior art, the present invention has the following beneficial effects:

[0032] (1) This invention proposes a multi-view facial expression recognition method based on dynamic neural networks. Dynamic neural networks can dynamically change the model parameters in the network for different input images, so that the network can dynamically adjust the convolution calculation range of the convolution layer according to the different poses of the input face, and extract more effective non-frontal expression features. This greatly improves the problems of low accuracy and poor robustness of existing expression recognition technology for non-frontal facial expression recognition.

[0033] (2) The multi-view facial expression recognition method based on dynamic neural network proposed in this invention dynamically adjusts the calculation range of convolution according to the different input images. It can reduce the convolution calculation of unimportant positions on the entire feature map, achieve the purpose of sparse convolution, reduce the number of model parameters and computation, realize the lightweighting of the model, and facilitate its deployment on embedded devices. Attached Figure Description

[0034] Figure 1 This is a flowchart of the method of the present invention;

[0035] Figure 2 This is a schematic diagram of the network structure of the facial expression recognition model in an embodiment of the present invention;

[0036] Figure 3 This is a schematic diagram of the structure of the residual block in the residual unit in the embodiment of the present invention, wherein (3a) is a schematic diagram of the structure of a normal residual block, and (3b) is a schematic diagram of the structure of a residual block with downsampling;

[0037] Figure 4 This is a schematic diagram of the structure of the convolution sampling module in an embodiment of the present invention. Detailed Implementation

[0038] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. These embodiments are based on the technical solution of the present invention and provide detailed implementation methods and specific operating procedures. However, the scope of protection of the present invention is not limited to the following embodiments.

[0039] To address the issue of poor robustness of facial expression recognition systems to facial pose changes in the real world, this invention proposes a multi-view facial expression recognition method based on dynamic neural networks. This method mainly comprises three modules: a face detection module, which uses a multi-task convolutional neural network to detect faces in images and segment face regions; an image convolution sampling module, which dynamically generates a mask based on the input image. This mask is a two-dimensional matrix of 0s and 1s, downsampled to the size of the input image, used to determine which pixel locations in the image require convolution operations; and an expression recognition module, which uses a ResNet18 network as its backbone. Specifically, this module consists of a 7×7 convolutional layer and four blocks of the same channel size, each containing two residual blocks. Each block undergoes dynamic range convolution under the guidance of the image sampling module to achieve expression recognition. Specifically, as shown... Figure 1 As shown, it includes the following steps:

[0040] S1. Obtain the image to be detected.

[0041] The images obtained in this embodiment are real-world images. These images were obtained under natural conditions, so the facial poses in the images may be from multiple angles, rather than all being frontal faces, and not all parts of the face appear in the image.

[0042] S2. Perform face detection on the image, segment the face image, and align the face regions.

[0043] A multi-task convolutional neural network is used to detect faces in images, and facial key points are detected to segment face regions. The face regions are then aligned using the relative positions of the facial key points.

[0044] S3. Perform image preprocessing on the aligned facial images.

[0045] The aligned facial image is preprocessed to convert it into a 224-pixel * 224-pixel RGB image and then normalized.

[0046] S4. Construct and train an expression recognition model based on a deep convolutional neural network.

[0047] In this embodiment, the facial expression recognition model includes a shallow feature extraction layer, four blocks consisting of convolutional sampling modules and residual units, and a fully connected layer connected in sequence. It takes a preprocessed image as input and outputs the recognized facial expression result, as shown in the structure below. Figure 2 As shown.

[0048] The convolution sampling module dynamically generates masks for different poses of faces based on the input feature map. The generated masks are used to control the convolution position of the convolutional layer in the corresponding residual unit. In other words, the expression recognition module only performs convolution calculations on the positions in the input image activated by the mask, thereby achieving dynamic range convolution.

[0049] In this embodiment, as Figure 2 As shown, the backbone network of the facial expression recognition model is a ResNet18 network, consisting of four blocks composed of convolutional sampling modules and residual units. The input to the convolutional sampling module in the first block is the feature map output from the shallow feature sampling layer. The inputs to the remaining convolutional sampling modules are the mask output from the previous convolutional sampling module and the feature map output from the previous residual unit. The number of channels in the four residual units are 64, 128, 256, and 512, respectively, and the kernel size is 3*3 for each.

[0050] The convolution sampling module processes the input image through a convolutional layer with 1 channel and then inputs it into the softmax layer. By setting a sampling threshold, the pixels in the softmax layer output image with values ​​greater than the sampling threshold are set to 1, and the pixels with values ​​less than the sampling threshold are set to 0. The output is a two-dimensional matrix mask composed of 0 and 1.

[0051] like Figure 3 As shown, each residual unit of the facial expression recognition model consists of two residual blocks. The residual blocks are either ordinary residual blocks or residual blocks with downsampling. The ordinary residual block consists of two 3*3 convolutional layers and a skip connection, as shown in Figure (3a). The residual block with downsampling consists of a 3*3 1 / 2 downsampling convolutional layer, a 3*3 ordinary convolutional layer and a 1*1 skip connection with 1 / 2 downsampling, as shown in Figure (3b).

[0052] Among them, the residual unit with 64 channels is composed of two ordinary residual blocks connected together, and the remaining residual units are composed of a residual block with downsampling and an ordinary residual block connected in sequence.

[0053] like Figure 4The diagram shown is a structural diagram of the convolutional sampling model and a schematic diagram of how it guides the convolutional layers to perform dynamic convolution. The input to the convolutional sampling module is the combination of the output of the previous convolutional sampling module and the feature map output of the previous residual unit (where the first input is the feature map output by the shallow feature extraction layer). Taking the input of the second convolutional sampling module as an example, the size of the output of the previous convolutional sampling module is (1, 56, 56), and the size of the feature map output by the previous residual unit is (64, 56, 56). The two are superimposed in the channel dimension to obtain a tensor of size (64, 56, 56) which is input into the convolutional sampling module. Then, it goes through a convolutional layer with a kernel size of 3*3 and an output channel of 1 to obtain an output feature map of size (1, 56, 56). This feature map is a two-dimensional matrix. This matrix is ​​subjected to a softmax operation, and a sampling threshold is set. Values ​​greater than the threshold are set to 1, and values ​​less than the threshold are set to 0 to obtain the image mask to be convolved. Then, the mask is input into the corresponding residual unit. The convolutional layer in the residual unit only performs convolution operations on the pixels in the image with a value of 1 at the corresponding position of the mask. For the remaining pixels, values ​​are assigned using interpolation, i.e.:

[0054]

[0055] Where p is the pixel to be interpolated, s i For the pixels within the 8-neighborhood of pixel p, f(s) i ) represents s in the original graph i The value of a pixel after convolution, if s i Not sampled, f(s) i ) is 0, |ps i | is the p pixel and s i The absolute value between pixels.

[0056] In one embodiment, the specific recognition steps of the multi-view expression recognition model for the input image x are as follows:

[0057] The input image x has dimensions (3, 224, 224). x is input into a shallow feature extraction layer, passing sequentially through a convolutional layer with a kernel size of 7*7, 64 output channels, and a stride of 2, followed by a max-pooling layer, resulting in a feature map of dimensions (64, 56, 56). This feature map is then input into the first convolutional sampling module, yielding a sampling convolutional mask of dimensions (1, 56, 56). This mask controls the range of convolutional calculations in the corresponding residual unit with 64 output channels. Convolutional calculations are then performed on the activated range of the input feature map. This process continues, with the feature map output from the current module input into residual units with output channels of 128, 256, and 512, respectively. The mask output from the current convolutional sampling module is also input into the corresponding residual unit and the next convolutional sampling module. After downsampling, the residual units with output channels of 128, 256, and 512 undergo downsampling operations, and the corresponding convolutional sampling modules also perform downsampling operations with the same stride on their output masks. The feature map output by the last residual unit has a size of (512,7,7). Projecting it onto a one-dimensional line yields a one-dimensional feature vector of length 25088. This feature vector passes through a fully connected layer to obtain an expression label of length 7. Finally, it passes through a softmax layer to output the final recognition result.

[0058] In this embodiment, the overall loss function of the facial expression recognition model is:

[0059] L = L FER +αL mask

[0060] Among them, L FER It is the loss function for the recognition results of the facial expression recognition model, representing the cross-entropy loss between the predicted facial expression result and the actual result; L mask α is the loss function of the convolutional sampling module, used to ensure the sparsity of the mask output by the convolutional sampling module; α is the scale factor, used to control the loss function L. mask The importance of.

[0061] Loss function L FER The expression is:

[0062]

[0063] Where y is the actual label value. These are the label values ​​predicted by the model, where x represents the sample and n is the total number of labels;

[0064] Loss function L mask The L1 norm regularization is used to ensure the sparsity of the sampling mask output by the image sampling module. A sparse mask reduces the computational cost of convolution within the image sampling module, thereby reducing the number of parameters and accelerating inference and training.

[0065]

[0066] Where l represents different network layers in the convolutional sampling module, w l The parameters represent the network, and ||·||1 represents the 1-norm.

[0067] By training the model by minimizing the above loss function, we can obtain an expression recognition model with high accuracy in expression recognition and sparse sampling masks.

[0068] S5. Use the trained facial expression recognition model to perform multi-view facial expression recognition.

[0069] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.

Claims

1. A multi-view facial expression recognition method based on dynamic neural networks, characterized in that, Includes the following steps: S1. Acquire the image to be detected; S2. Perform face detection on the image, segment the face image, and align the face regions. S3. Perform image preprocessing on the aligned facial images; S4. Construct and train an expression recognition model based on a deep convolutional neural network. The expression recognition model includes a shallow feature extraction layer, multiple blocks composed of convolutional sampling modules and residual units, and a fully connected layer connected in sequence. The preprocessed image is used as input, and the recognized expression result is output. The convolutional sampling module dynamically generates a mask based on the input feature map. The generated mask is used to control the convolution position of the convolutional layer in the corresponding residual unit to achieve dynamic range convolution. S5. Use the trained facial expression recognition model to perform multi-view facial expression recognition; The convolution sampling module processes the input image through a convolutional layer with 1 channel and then inputs it into the softmax layer. By setting a sampling threshold, the pixels in the output image of the softmax layer are set to 1 if the value is greater than the sampling threshold and 0 if the value is less than the sampling threshold, and outputs a two-dimensional matrix mask composed of 0 and 1. The convolutional layer within the residual unit only performs convolution operations on pixels with a value of 1 at the corresponding mask position in the image, and assigns values ​​to the remaining pixels through interpolation.

2. The multi-view facial expression recognition method based on a dynamic neural network according to claim 1, characterized in that, Step S2 specifically involves: using a multi-task convolutional neural network to detect faces in the image, performing facial key point detection on the detected faces, segmenting the face region, and aligning the face region using the relative positions of the facial key points.

3. The multi-view facial expression recognition method based on a dynamic neural network according to claim 1, characterized in that, Step S3 specifically involves: performing image preprocessing on the aligned facial image, converting the image into an RGB image of a preset size, and performing normalization processing.

4. The multi-view facial expression recognition method based on a dynamic neural network according to claim 1, characterized in that, The backbone network of the facial expression recognition model is a ResNet18 network, which includes four blocks consisting of convolutional sampling modules and residual units. The input of the convolutional sampling module of the first block is the feature map output by the shallow feature sampling layer, and the input of the remaining convolutional sampling modules is the mask output by the previous convolutional sampling module and the feature map output by the previous residual unit.

5. The multi-view facial expression recognition method based on a dynamic neural network according to claim 1, characterized in that, The remaining pixels are assigned values ​​through interpolation: in, p For the pixels to be interpolated, for p Pixels within the 8-neighborhood of pixel , For the original image The value of a pixel after convolution, if Not sampled. =0, for p Pixels and The absolute value between pixels.

6. The multi-view facial expression recognition method based on a dynamic neural network according to claim 4, characterized in that, Each residual unit of the facial expression recognition model consists of two residual blocks. The residual blocks are of the following types: ordinary residual blocks or residual blocks with downsampling. The ordinary residual block consists of two convolutional layers and a skip connection. The residual block with downsampling consists of a 1 / 2 downsampling convolutional layer, an ordinary convolutional layer, and a skip connection with 1 / 2 downsampling.

7. The multi-view facial expression recognition method based on a dynamic neural network according to claim 6, characterized in that, The number of channels in the four residual units are 64, 128, 256, and 512, respectively. The residual unit with 64 channels is composed of two ordinary residual blocks connected together, while the other residual units are composed of a residual block with downsampling and an ordinary residual block connected together in sequence.

8. The multi-view facial expression recognition method based on a dynamic neural network according to claim 1, characterized in that, The overall loss function of the facial expression recognition model is: in, It is a loss function for the recognition results of the facial expression recognition model, which represents the cross-entropy loss between the predicted facial expression result and the actual result; It is the loss function of the convolutional sampling module, used to ensure the sparsity of the mask output by the convolutional sampling module; It is a scaling factor used to control the loss function. The importance of.

9. The multi-view facial expression recognition method based on a dynamic neural network according to claim 8, characterized in that, The loss function The expression is: in, y This is the actual tag value. These are the label values ​​predicted by the model. x Indicates a sample, n It represents the total number of tags; The loss function Expressed using L1 norm regularization: in, l Representing different network layers in the convolutional sampling module, Parameters representing the network, It is represented by the 1 norm.

Images