Facial expression recognition method based on convolutional neural networks

By combining preprocessing with convolutional neural networks and improved residual networks, the problem of low efficiency in facial expression recognition in existing technologies is solved, achieving efficient and accurate facial expression recognition results.

CN115937926BActive Publication Date: 2026-11-17ZHEJIANG GONGSHANG UNIVERSITY
View PDF 1 Cites 0 Cited by

Patent Information

Application Number
CN202211265533.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-10-17
Publication Date
2026-11-17
Estimated Expiration
2042-10-17

AI Technical Summary

Technical Problem

Existing facial expression recognition systems are inefficient in feature extraction and classification, making it difficult to quickly and accurately identify expressions.

Method used

An expression recognition method based on convolutional neural networks is adopted. The expression images are preprocessed by adjustable filtering, skin color adaptive grayscale conversion and low-attenuation interpolation. The improved residual neural network and spatial attention mechanism are combined for feature extraction and classification.

Benefits of technology

It improves the accuracy and efficiency of facial expression recognition, reduces computation and training time, avoids the manual feature extraction steps in traditional methods, and enhances sensitivity to yellow-brown skin tones and low attenuation during image magnification.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN115937926B_ABST
    Figure CN115937926B_ABST
Patent Text Reader

Abstract

This invention discloses an expression recognition method based on convolutional neural networks, comprising the following steps: S1: Adjustable filtering is applied to the expression image. S2: Grayscale processing is performed on the filtered expression image using skin color adaptive grayscale conversion. S3: Low-attenuation interpolation is performed on the grayscale processed expression image. S4: The processed expression image is divided into a training set and a dataset. S5: The constructed convolutional neural network model for expression recognition is trained using the training set and the dataset. This invention's expression recognition method based on convolutional neural networks utilizes the superior feature extraction capabilities of deep learning networks, thus avoiding the tedious manual feature extraction process. Furthermore, deep learning, in addition to prediction, also plays a role in feature engineering, further eliminating the need for manual feature extraction.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention belongs to the field of image processing, specifically relating to an expression recognition method based on convolutional neural networks. Background Technology

[0002] With the development of machine vision theory and technology, image content recognition and understanding has become a research hotspot with a broad application market. Especially in the field of facial recognition, "expression" is a frequently used term in our daily lives. In interpersonal communication, people can enhance communication effectiveness by controlling their facial expressions. Facial expressions are an important way to convey human emotional information and coordinate interpersonal relationships, and the demand for facial expression recognition is increasingly strong. Currently, facial expression recognition systems mainly consist of four parts: facial image acquisition, face detection, feature extraction, and feature classification. Progress in feature extraction and feature classification is relatively slow. Summary of the Invention

[0003] This invention provides an expression recognition method based on convolutional neural networks to solve the aforementioned technical problems, specifically employing the following technical solution:

[0004] A facial expression recognition method based on convolutional neural networks includes the following steps:

[0005] S1: Perform adjustable filtering on emoticon images;

[0006] S2: Perform grayscale processing on the filtered facial expression image by skin color adaptive grayscale conversion;

[0007] S3: Perform low-attenuation interpolation on the grayscale emoji image;

[0008] S4: Divide the processed facial expression images into a training set and a dataset;

[0009] S5: Train the constructed convolutional neural network model for facial expression recognition using the training set and dataset.

[0010] Furthermore, the specific method for adjusting the filtering process on emoticon images is as follows:

[0011] The filtering result is adjusted using the mean of pixel values ​​in the filtering window according to the following formula.

[0012] M=Sort[window(n-num),K window(n),K window(n+num)]

[0013]

[0014]

[0015] Where window(n-num) is the pixel value at position n-num in the image within the filter window, Sort() sorts the sequence from smallest to largest, M is the sequence of pixel values ​​in the filter window after sorting from smallest to largest, M(n) is the median of the sorted sequence in the filter window, M(n-1) is the pixel value to the left of the median, and M(n+1) is the pixel value to the right of the median. is the mean of pixel values ​​in the filtering window, 2num+1 is the size of the filtering window, the range (0.75, 1.25) is selected as the adjustable range, and filter(n) is the output value at the nth position of the filtering window after the filter is adjusted.

[0016] Furthermore, the specific method for grayscale processing of the filtered facial expression image using skin color adaptive grayscale conversion is as follows:

[0017] The following formula is used to process emoji images.

[0018]

[0019] Where R, G, and B are the RGB color values ​​corresponding to the image.

[0020] Furthermore, the specific method for performing low-attenuation interpolation on the grayscale-processed emoji image is as follows:

[0021] Low-attenuation interpolation processing is performed on grayscale emoji images by combining bilinear interpolation and weighted nearest neighbor interpolation.

[0022] Furthermore, the specific method for low-attenuation interpolation processing of grayscale expression images using a weighted combination of bilinear interpolation and nearest neighbor interpolation is as follows:

[0023] S31: Process the facial expression image using the following formulas to calculate bilinear interpolation and nearest neighbor interpolation.

[0024]

[0025] Where Q1 = (x1, y1), Q2 = (x1, y2), Q3 = (x2, y1), and Q4 = (x2, y2) are four points on the emoticon image, and (x′, y′) is the position of the unassigned pixel value within the neighborhood formed by the four points Q1, Q2, Q3, and Q4. i From (x′, y′) to point Q i The distance, m is d i Let i be the minimum value in the matrix, f be the pixel value at the corresponding position, f1(x′,y′) be the pixel value at (x′,y′) obtained by bilinear interpolation, and f2(x′,y′) be the pixel value at (x′,y′) obtained by nearest neighbor interpolation.

[0026] S32: Calculate Q using the following companies i The standard deviation σ is used to reflect the edge detail information of the neighborhood constructed by the four points.

[0027]

[0028] in, This represents the average pixel value corresponding to the four points Q1, Q2, Q3, and Q4.

[0029] S33: Calculate the weight w using the following formula, in the range [0,1).

[0030]

[0031] Where k is the adjustment coefficient, which is a small fluctuation factor added to the weight, set to 0.01.

[0032] S34: Calculate the final pixel value using the following formula.

[0033] f3(x′,y′)=(1-w)f1(x′,y′)+wf2(x′,y′)

[0034] Where f3(x′,y′) is the pixel value at (x′,y′) interpolated with low attenuation.

[0035] Furthermore, the specific method for training the constructed convolutional neural network model for facial expression recognition using the training set and dataset is as follows:

[0036] S51: The model first convolves the processed 48×48 emoji image with 64 7×7 convolution kernels, outputting 48×48×64 data, and then inputs the 48×48×64 data into the BN layer.

[0037] S52: Generate the corresponding spatial attention matrix for the facial expression images input to the model;

[0038] S53: Input the output of step S51 into a three-layer residual network with spatial attention, and combine it with the spatial attention matrix obtained in step S52 to output 48×48×1024 data.

[0039] S54: Input 48×48×1024 data into the adaptive average pooling layer and output 1×1×1024 data;

[0040] S55: Input 1×1×1024 data into a linear layer, then perform Prelude and dropout on the data, and perform softmax regression on the output to finally obtain the probability of different expressions.

[0041] Furthermore, in step S52, the specific method for generating the corresponding spatial attention matrix is ​​as follows:

[0042] Apply a two-dimensional Gaussian function to key points in specific locations on the face in the emoji image.

[0043]

[0044] Where A is the magnitude of the two-dimensional Gaussian function, (x0, y0) are the peak coordinates of the two-dimensional Gaussian function, and σ x ,σ y It is the variance of the two-dimensional Gaussian function, σ x ,σ y For the same value, as the receptive field of the designed network increases with the number of layers, σ is set accordingly. x ,σ y The numbers are 5, 7, and 9.

[0045] The Gaussian function diffusion values ​​of all key points are superimposed at various points in the facial expression image to generate a spatial attention matrix specifically for facial expression recognition.

[0046]

[0047]

[0048] Where N is the number of key points, f j (0,0) represents the diffusion value of the pixel at position 0 (x-coordinate, y-coordinate) in the input image, corresponding to the Gaussian function with the j-th keypoint as the peak point. The diffusion values ​​of the Gaussian function of all keypoints are superimposed at each point in the image. The sigmoid function is used to control the weight value obtained after superposition within the interval (0,1). Z is the generated spatial attention matrix. Given the three Gaussian functions with different variances, the corresponding spatial attention matrices Z1, Z2, and Z3 are generated.

[0049] Furthermore, in step S53, the specific operations in the input three-layer residual network with spatial attention are as follows:

[0050] The result of step S51 is input into the three-layer improved residual network.

[0051]

[0052] Where, x l The input to the corresponding layer l of the network is F, where F represents the residual network and W is the input to the network. l Z represents the parameters of layer l in the network. l The spatial attention matrix generated in step S52;

[0053] The first layer of the improved residual network is processed in two branches:

[0054]

[0055] 1) Branch 1: The input 48×48×64 data is convolved with 64 1×1 convolution kernels. The result of the convolution is then fed into a BN layer and LeakyReLU to accelerate the convergence of the network. Then, 64 3×3 convolution kernels are used for convolution. The result of the convolution is then fed into a BN layer and LeakyReLU to accelerate the convergence of the network. Finally, 256 1×1 convolution kernels are used for convolution. The output result is 48×48×256 data. The spatial attention matrix Z is then multiplied with the output result of 48×48×256 data. The final result is 48×48×256.

[0056] 2) Branch 2: Convolve the input 48×48×64 data using 256 1×1 convolution kernels, and then input the convolution result into the BN layer. The output result is 48×48×256 data.

[0057] The data output from the two branches are summed and input into LeakyReLU to obtain a 48×48×256 data set. This 48×48×256 data set is then input into the second-layer improved residual network, which operates on two branches:

[0058]

[0059]

[0060] 1) Branch 1: The input 48×48×256 data is convolved with 128 1×1 convolution kernels. The result of the convolution is then fed into a BN layer and LeakyReLU to accelerate the convergence of the network. Then, 128 3×3 convolution kernels are used for convolution. The result of the convolution is then fed into a BN layer and LeakyReLU to accelerate the convergence of the network. Finally, 512 1×1 convolution kernels are used for convolution. The output result is 48×48×512 data. The spatial attention matrix Z is then multiplied with the output result of 48×48×512 data. The final result is 48×48×512.

[0061] 2) Branch 2: Convolve the input 48×48×64 data using 512 1×1 convolution kernels, and then input the convolution result into the BN layer. The output result is 48×48×512 data.

[0062] The output data from the two branches are summed and input into LeakyReLU to obtain 48×48×512 data. This 48×48×512 data is then input into the third-layer improved residual network, which is divided into two branches:

[0063]

[0064]

[0065] 1) Branch 1: The input 48×48×512 data is convolved with 256 1×1 convolution kernels. The result of the convolution is then fed into a BN layer and LeakyReLU to accelerate the convergence of the network. Then, 256 3×3 convolution kernels are used for convolution. The result of the convolution is then fed into a BN layer and LeakyReLU to accelerate the convergence of the network. Finally, 1024 1×1 convolution kernels are used for convolution. The output result is 48×48×1024 data. The spatial attention matrix Z is then multiplied with the output result of 48×48×1024 data. The final result is 48×48×1024.

[0066] 2) Branch 2: Convolve the input 48×48×64 data using 1024 1×1 convolution kernels, and then input the convolution result into the BN layer. The output data is 48×48×1024.

[0067] The data output from the two branches are added together and input into LeakyReLU, resulting in a data set of 48×48×1024.

[0068] Further, in step S55,

[0069] res=maxindex([softmax(1),L,softmax(14)])

[0070]

[0071] Where q is the output probability, c is the number of expressions predicted by the network, softmax is used to normalize the probabilities of all predicted expressions, maxindex is used to obtain the index value of the maximum probability after normalization, and res is the index value of the final predicted expression.

[0072] Furthermore, before performing adjustable filtering on the facial expression images, the facial expression recognition method based on convolutional neural networks also includes;

[0073] MTCNN is used to detect faces in face images. The detected face regions and key facial points are then cropped and aligned to obtain expression images.

[0074] The advantages of this invention are:

[0075] 1. Image preprocessing: First, adjustable filtering is used to reduce image noise. Then, skin color adaptive grayscale processing is used to increase the sensitivity of the grayscale image to yellow-brown skin tones. Finally, low-attenuation interpolation, which combines bilinear interpolation and nearest neighbor interpolation with weights, is used to effectively avoid the problems of the two interpolation methods and achieve the goal of low pixel attenuation when the image is enlarged, which is conducive to better model learning.

[0076] 2. The model design initially used ordinary convolutions for image feature extraction, followed by a modified residual neural network. Compared to traditional residual neural networks, this network incorporates a spatial attention mechanism specifically for facial expression recognition to better learn facial features. Compared to traditional neural networks, this network is leaner, controls the number of parameters, reduces computation, uses fewer pooling layers, extensively uses downsampling to improve propagation efficiency, and utilizes Batch Normalization (BN) and global average pooling for regularization, accelerating training speed. Deep learning networks have a strong ability to extract features from images, thus avoiding the tedious manual feature extraction process. Furthermore, in addition to prediction, deep learning often plays a role in feature engineering, further eliminating the need for manual feature extraction. Attached Figure Description

[0077] Figure 1 This is a schematic diagram of an expression recognition method based on a convolutional neural network according to the present invention;

[0078] Figure 2 This is a schematic diagram of the specific locations of low-attenuation interpolation;

[0079] Figure 3 It is a convolutional neural network model used for facial expression recognition;

[0080] Figure 4 This is a heatmap of the spatial attention matrix under Gaussian functions of different variances;

[0081] Figure 4 a:σ x ,σ y Heatmap of the spatial attention matrix Z1 constructed using a 2D Gaussian function of 5;

[0082] Figure 4 b: σ x ,σ y Heatmap of the spatial attention matrix Z2 constructed using a 2D Gaussian function of 7;

[0083] Figure 4 c: σ x ,σ y Heatmap of the spatial attention matrix Z3 constructed using a two-dimensional Gaussian function of 9;

[0084] Figure 5 The facial landmark coordinates were obtained through MTCNN. Detailed Implementation

[0085] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0086] like Figure 1 The diagram illustrates a facial expression recognition method based on a convolutional neural network, comprising the following steps: S1: Adjustable filtering is applied to the facial expression image. S2: Grayscale processing is performed on the filtered facial expression image using skin color adaptive grayscale conversion. S3: Low-attenuation interpolation is applied to the grayscale processed facial expression image. S4: The processed facial expression image is divided into a training set and a dataset. S5: The constructed convolutional neural network model for facial expression recognition is trained using the training set and the dataset. The specific steps described below are detailed below.

[0087] For step S1: Perform adjustable filtering on the emoticon image.

[0088] Adjustable filtering is applied to emoji images primarily to reduce image noise.

[0089] Specifically, adjustable filtering is used to reduce image noise, mainly by adjusting the filtering result based on the mean pixel value in the filtering window. The operation is as follows:

[0090] M=Sort[window(n-num),K window(n),K window(n+num)]

[0091]

[0092]

[0093] Where window(n-num) is the pixel value at position n-num in the image within the filter window, Sort() sorts the sequence from smallest to largest, M is the sequence of pixel values ​​in the filter window after sorting from smallest to largest, M(n) is the median of the sorted sequence in the filter window, M(n-1) is the pixel value to the left of the median, and M(n+1) is the pixel value to the right of the median. is the mean of pixel values ​​in the filtering window, 2num+1 is the size of the filtering window, the range (0.75, 1.25) is selected as the adjustable range, and filter(n) is the output value at the nth position of the filtering window after the filter is adjusted.

[0094] For step S2: perform grayscale processing on the filtered facial expression image by skin color adaptive grayscale conversion.

[0095] As a preferred embodiment, the specific method for grayscale processing of the filtered facial expression image using skin color adaptive grayscale conversion is as follows:

[0096] The following formula is used to process emoji images.

[0097]

[0098] Here, R, G, and B represent the RGB color values ​​of the image. Since most facial colors are concentrated in yellow-brown, and yellow-brown is primarily composed of R and G from R, G, and B, to increase the sensitivity of the grayscale image to yellow-brown, the weights of color values ​​R and G are increased during the grayscale process, while the weight of color value B is decreased. This increases the sensitivity of the grayscale image to yellow-brown, reduces image dimensionality, and alleviates the computational burden of model training.

[0099] For step S3: perform low-attenuation interpolation on the grayscale image of the expression.

[0100] In this application, a weighted combination of bilinear interpolation and nearest neighbor interpolation is used to perform low-attenuation interpolation processing on grayscale emoji images. Specifically:

[0101] S31: Process the facial expression image using the following formulas to calculate bilinear interpolation and nearest neighbor interpolation.

[0102]

[0103] Where Q1 = (x1, y1), Q2 = (x1, y2), Q3 = (x2, y1), and Q4 = (x2, y2) are four points on the emoticon image, such as... Figure 2 As shown. (x′, y′) represents the position of the unassigned pixel value within the neighborhood formed by the four points Q1, Q2, Q3, and Q4, d i From (x′, y′) to point Q i The distance, m is d i The minimum value corresponds to the value of i, f is the pixel value at the corresponding position, f1(x′,y′) is the pixel value of bilinear interpolation at (x′,y′), and f2(x′,y′) is the pixel value of nearest neighbor interpolation at (x′,y′).

[0104] S32: Calculate Q using the following companies i The standard deviation σ is used to reflect the edge detail information of the neighborhood constructed by the four points.

[0105]

[0106] in, This represents the average pixel value corresponding to the four points Q1, Q2, Q3, and Q4.

[0107] S33: Calculate the weight w using the following formula, in the range [0,1).

[0108]

[0109] Where k is the adjustment coefficient, which is a small fluctuation factor added to the weight, set to 0.01.

[0110] S34: Calculate the final pixel value using the following formula.

[0111] f3(x′,y′)=(1-w)f1(x′,y′)+wf2(x′,y′)

[0112] Here, f3(x′,y′) represents the pixel value at (x′,y′) obtained through low-attenuation interpolation. Bilinear interpolation can cause mosaic and jagged edges in high-frequency edge details and nearest-neighbor interpolation. By adjusting the weights using the standard deviation σ, the problems of both interpolation methods can be effectively avoided. When σ is small, edge details are low-frequency, so the weight of bilinear interpolation is increased; when σ is large, edge details are high-frequency, so the weight of nearest-neighbor interpolation is increased.

[0113] For step S4: Divide the processed facial expression images into a training set and a dataset.

[0114] All images were preprocessed using the first three steps, resulting in preprocessed images of size 48×48. The preprocessed data was then divided into two sets: a training set and a test set.

[0115] For step S5: Train the constructed convolutional neural network model for facial expression recognition using the training set and dataset.

[0116] like Figure 3 The diagram shows the convolutional neural network model for facial expression recognition constructed in this application. Its specific structure is shown in the table below:

[0117] Table 1. Architecture diagram of convolutional neural networks with spatial attention

[0118]

[0119]

[0120] The processed 48×48 image is input into the model, and the model parameters are trained using backpropagation via SGD (Gradient Descent). Specifically, the method for training the constructed convolutional neural network model for facial expression recognition using the training set and dataset is as follows:

[0121] S51: The model first convolves the processed 48×48 emoji image with 64 7×7 convolution kernels, outputting 48×48×64 data, and then inputs the 48×48×64 data into the BN layer for batch normalization.

[0122] Specifically, the data is input into the Batch Normalization (BN) layer. Normalization and linear transformation ensure that the mean and variance of the input data for each layer are within a certain range, so that the feature map after convolution satisfies a distribution pattern with a mean of 0 and a variance of 1. Then, LeakyReLU (an activation function that introduces non-linear features) is used to accelerate the model convergence speed.

[0123] S52: Generate the corresponding spatial attention matrix for the facial expression images input to the model.

[0124] In step S52, the specific method for generating the corresponding spatial attention matrix is ​​as follows:

[0125] By utilizing the spatial diffusion characteristic of a two-dimensional Gaussian function (diffusion in a downward direction), a two-dimensional Gaussian function is applied simultaneously to key points in specific locations on the face in an expression image.

[0126]

[0127] Where A is the magnitude of the two-dimensional Gaussian function, set to 10. (x0, y0) are the peak coordinates of the two-dimensional Gaussian function. In this application, the coordinates of facial key points (obtained through MTCNN) are selected as the peak points of the Gaussian function. σ x ,σ y This is the variance of the two-dimensional Gaussian function. To achieve uniform diffusion of the two-dimensional Gaussian function, σ is set... x ,σ y For the same value, as the receptive field of the designed network increases with the number of layers, σ is set accordingly. x ,σ y The receptive fields of the residual networks are adjusted to 5, 7, and 9 to improve the network's attention to the regions formed by facial key points.

[0128] The coordinates of key points are used as the peak points of a Gaussian function. Spatial diffusion is then performed, and the diffusion values ​​of the Gaussian function of all key points at various points in the facial expression image are superimposed to finally generate a spatial attention matrix specifically for facial expression recognition.

[0129]

[0130]

[0131] Where N is the number of key points, f j(0,0) represents the diffusion value of the pixel at position 0 (x-coordinate, y-coordinate) in the input image, corresponding to the Gaussian function with the j-th keypoint as the peak point. The diffusion values ​​of the Gaussian functions at all keypoints are superimposed. The sigmoid function is used to control the weight values ​​obtained after superposition within the interval (0,1). Z represents the generated spatial attention matrix. Given three Gaussian functions with different variances, corresponding spatial attention matrices Z1, Z2, and Z3 are generated. For example... Figure 4 The image shows a heatmap of the spatial attention matrix under Gaussian functions with different variances.

[0132] S53: Input the output of step S51 into a three-layer residual network with spatial attention, and combine it with the spatial attention matrix obtained in step S52 to output 48×48×1024 data.

[0133] In step S53, the specific operations in the input three-layer residual network with spatial attention are as follows:

[0134] The result of step S51 is input into the three-layer improved residual network.

[0135]

[0136] Where, x l The input to the corresponding layer l of the network is F, where F represents the residual network and W is the input to the network. l Z represents the parameters of layer l in the network. l The spatial attention matrix generated in step S52.

[0137] The first layer of the improved residual network is processed in two branches:

[0138]

[0139] 1) Branch 1: The input 48×48×64 data is convolved with 64 1×1 convolution kernels. The result of the convolution is then fed into a BN layer and LeakyReLU to accelerate the convergence of the network. Then, 64 3×3 convolution kernels are used for convolution. The result of the convolution is then fed into a BN layer and LeakyReLU to accelerate the convergence of the network. Finally, 256 1×1 convolution kernels are used for convolution. The output result is 48×48×256 data. The spatial attention matrix Z is then multiplied with the output 48×48×256 data. The final result is 48×48×256.

[0140] 2) Branch 2: Convolve the input 48×48×64 data using 256 1×1 convolution kernels, and then input the convolution result into the BN layer. The output result is 48×48×256 data.

[0141] The data output from the two branches are summed and input into LeakyReLU to obtain a 48×48×256 data set. This 48×48×256 data set is then input into the second-layer improved residual network, which operates on two branches:

[0142]

[0143]

[0144] 1) Branch 1: Convolve the input 48×48×256 data using 128 1×1 convolution kernels, then input the convolution result into a BN layer and LeakyReLU to accelerate network convergence. Then convolve it using 128 3×3 convolution kernels, and input the convolution result into a BN layer and LeakyReLU to accelerate network convergence. Then convolve it using 512 1×1 convolution kernels, and the output result is 48×48×512 data. Then multiply the spatial attention matrix Z with the output 48×48×512 data, and the final result is 48×48×512.

[0145] 2) Branch 2: Convolve the input 48×48×64 data using 512 1×1 convolution kernels, and then input the convolution result into the BN layer. The output result is 48×48×512 data.

[0146] The output data from the two branches are summed and input into LeakyReLU to obtain 48×48×512 data. This 48×48×512 data is then input into the third-layer improved residual network, which is divided into two branches:

[0147]

[0148]

[0149] 1) Branch 1: The input 48×48×512 data is convolved using 256 1×1 convolution kernels. The result of the convolution is then fed into a BN layer and LeakyReLU to accelerate network convergence. Next, 256 3×3 convolution kernels are used for convolution, and the result of the convolution is fed into a BN layer and LeakyReLU to accelerate network convergence. Finally, 1024 1×1 convolution kernels are used for convolution, and the output result is 48×48×1024 data. The spatial attention matrix Z is then multiplied with the output 48×48×1024 data, and the final result is 48×48×1024.

[0150] 2) Branch 2: Convolve the input 48×48×64 data using 1024 1×1 convolution kernels, and then input the convolution result into the BN layer. The output data is 48×48×1024.

[0151] The data output from the two branches are added together and input into LeakyReLU, resulting in a data set of 48×48×1024.

[0152] S54: Input 48×48×1024 data into the adaptive average pooling layer and output 1×1×1024 data.

[0153] Specifically, the 48×48×1024 data obtained above is input into AdaptiveAvgPool2d. First, the pooling kernel size is dynamically generated, and the kernel size is obtained by rounding up using the "template maximization" principle. Then, equidistant position points are calculated on the feature map. When calculating the starting coordinates, the "minimum principle" is used to round down, and finally, 1×1×1024 data is output.

[0154] S55: Input 1×1×1024 data points into a linear layer, then perform PreLU and dropout on the data, and perform softmax regression on the output to finally obtain the probabilities of different expressions:

[0155] res=maxindex([softmax(1),L,softmax(14)])

[0156]

[0157] Where q is the output probability, c is the number of expressions predicted by the network, softmax is used to normalize the probabilities of all predicted expressions, maxindex is used to obtain the index value of the maximum probability after normalization, and res is the index value of the final predicted expression.

[0158] As a preferred embodiment, the facial expression recognition method based on convolutional neural networks further includes the following steps before performing adjustable filtering on the facial expression image:

[0159] MTCNN was used to detect faces in face images, detecting facial regions and key facial features, such as... Figure 5 As shown, the detected face regions are cropped and aligned to obtain expression images.

[0160] The following specific embodiments verify the facial expression recognition method based on convolutional neural networks of this application.

[0161] Images of infant facial expressions were collected from childcare facilities in a certain city. Different types of infant facial expressions were categorized, and an facial expression recognition model was established using the method of this invention. A total of 9012 infant images were collected, including fourteen common facial expression categories. Each category contained at least 600 samples, with a resolution of approximately 50*50 pixels, covering both well-lit and dimly lit conditions.

[0162] The implementation steps are as follows:

[0163] 1. Select 500 images for each of the 14 categories (excitement, anger, annoyance, hunger, discomfort, etc.) from the database for model training, and select 100 images for each category for model testing. The images must have a resolution of 1024*768.

[0164] 2. Image preprocessing:

[0165] (1) Use MTCNN to detect faces in each image. If the face region and key points are detected, the detected face region is cropped and aligned.

[0166] (2) The aligned images are processed by adjustable filtering, skin color adaptive grayscale conversion, and low-attenuation interpolation to form 48×48 images required by the model input. Then, all the preprocessed image data are divided into two sets: training set and test set.

[0167] 3. Model Training: The model parameters are preset. Training images are shuffled and input into the model in batches of 128. The model then undergoes forward propagation through convolutional layers, a modified residual layer, and linear layers. Finally, a softmax function outputs the predicted probabilities for various facial expressions. The loss between the predicted and actual results is calculated, and backpropagation using SGD updates the model parameters. Through continuous batch training in the preceding steps, the model can ultimately predict infant facial expressions with high accuracy. Finally, the model is tested using a test set of prediction data, achieving a prediction accuracy of 90.2%. This demonstrates that the facial expression recognition method based on convolutional neural networks proposed in this application achieves high accuracy.

[0168] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the above embodiments do not limit the present invention in any way, and all technical solutions obtained by equivalent substitution or equivalent transformation fall within the protection scope of the present invention.

Claims

1. A facial expression recognition method based on convolutional neural networks, characterized in that, Includes the following steps: S1: Perform adjustable filtering on emoticon images; S2: Perform grayscale processing on the filtered facial expression image by skin color adaptive grayscale conversion; S3: Perform low-attenuation interpolation on the grayscale emoji image; S4: Divide the processed facial expression images into a training set and a dataset; S5: Train the constructed convolutional neural network model for facial expression recognition using the training set and dataset; The specific method for training the constructed convolutional neural network model for facial expression recognition using the training set and dataset is as follows: S51: The model first convolves the processed 48×48 emoji image with 64 7×7 convolution kernels, outputting 48×48×64 data, and then inputs the 48×48×64 data into the BN layer. S52: Generate the corresponding spatial attention matrix for the facial expression images input to the model; S53: Input the output of step S51 into a three-layer residual network with spatial attention, and combine it with the spatial attention matrix obtained in step S52 to output 48×48×1024 data. S54: Input 48×48×1024 data into the adaptive average pooling layer and output 1×1×1024 data; S55: Input 1×1×1024 data into a linear layer, then perform Prelude and dropout on the data, and perform softmax regression on the output to finally obtain the probability of different expressions; In step S52, the specific method for generating the corresponding spatial attention matrix is ​​as follows: Apply a two-dimensional Gaussian function to key points in specific locations on the face in the emoji image. in, It is the magnitude of the two-dimensional Gaussian function. These are the peak coordinates of a two-dimensional Gaussian function. It is the variance of a two-dimensional Gaussian function. For the same value, the receptive field of the designed network increases with the number of layers, and is set separately. The numbers are 5, 7, and 9. The Gaussian function diffusion values ​​of all key points are superimposed at various points in the facial expression image to generate a spatial attention matrix specifically for facial expression recognition. Where N is the number of key points, The diffusion value of the pixel value at the x-coordinate and y-coordinate of the input image is the Gaussian function with the j-th keypoint as the peak point. The diffusion values ​​of the Gaussian function of all keypoints are superimposed at each point in the image. The sigmoid function is used to control the weight value obtained after superposition to be in the interval (0, 1). Z is the generated spatial attention matrix. Given the three Gaussian functions with different variances, the corresponding spatial attention matrices Z1, Z2, and Z3 are generated.

2. The facial expression recognition method based on convolutional neural networks according to claim 1, characterized in that, The specific method for performing adjustable filtering on emoticon images is as follows: The filtering result is adjusted using the mean of pixel values ​​in the filtering window according to the following formula. Where window(n-num) is the pixel value at position n-num in the image within the filter window, Sort() sorts the sequence from smallest to largest, M is the sequence of pixel values ​​in the filter window after sorting from smallest to largest, M(n) is the median of the sorted sequence in the filter window, M(n-1) is the pixel value to the left of the median, and M(n+1) is the pixel value to the right of the median. The mean of the pixel values ​​in the filter window. The size of the filter window is set to the range (0.75, 1.25), which is an adjustable range. filter(n) is the adjustable output value at filter window n after filtering.

3. The facial expression recognition method based on convolutional neural networks according to claim 1, characterized in that, The specific method for grayscale processing of the filtered facial expression image using skin color adaptive grayscale conversion is as follows: The following formula is used to process emoji images. Where R, G, and B are the RGB color values ​​corresponding to the image.

4. The facial expression recognition method based on a convolutional neural network according to claim 1, characterized in that, The specific method for performing low-attenuation interpolation on the grayscale-processed facial expression image is as follows: Low-attenuation interpolation processing is performed on grayscale emoji images by combining bilinear interpolation and weighted nearest neighbor interpolation.

5. The facial expression recognition method based on a convolutional neural network according to claim 4, characterized in that, The specific method for performing low-attenuation interpolation processing on grayscale expression images using a weighted combination of bilinear interpolation and nearest neighbor interpolation is as follows: S31: Process the facial expression image using the following formulas to calculate bilinear interpolation and nearest neighbor interpolation. in, The four dots on the emoji image for The location within the area formed by the four points where no pixel value has been assigned. for Time The distance, m is The minimum value of i in the middle. This represents the pixel value at the corresponding position. For bilinear interpolation in pixel values, Interpolation for nearest neighbors Pixel values; S32: Calculated through the following companies Standard deviation This is used to reflect the edge detail information of the neighborhood constructed by the four points. in, for The average pixel value corresponding to the four points. S33: Calculate the weights using the following formula The range is [0,1). Where k is the adjustment coefficient, which is a small fluctuation factor added to the weight, set to 0.01; S34: Calculate the final pixel value using the following formula. in, For low attenuation interpolation The pixel value.

6. The facial expression recognition method based on a convolutional neural network according to claim 1, characterized in that, In step S53, the specific operation of inputting the three-layer residual network with spatial attention is as follows: The result of step S51 is input into the three-layer improved residual network. in, For network correspondence Layer input, Represents the residual network. For network correspondence Layer parameters, Z l The spatial attention matrix generated in step S52; The first layer of the improved residual network is processed in two branches: 1) Branch 1: The input 48×48×64 data is convolved with 64 1×1 convolution kernels. The result of the convolution is then fed into a BN layer and LeakyReLU to accelerate the convergence of the network. Then, 64 3×3 convolution kernels are used for convolution. The result of the convolution is then fed into a BN layer and LeakyReLU to accelerate the convergence of the network. Finally, 256 1×1 convolution kernels are used for convolution. The output result is 48×48×256 data. The spatial attention matrix Z is then multiplied with the output result of 48×48×256 data. The final result is 48×48×256. 2) Branch 2: Convolve the input 48×48×64 data using 256 1×1 convolution kernels, and then input the convolution result into the BN layer. The output result is 48×48×256 data. The data output from the two branches are summed and input into LeakyReLU to obtain a 48×48×256 data set. This 48×48×256 data set is then input into the second-layer improved residual network, which operates on two branches: 1) Branch 1: Convolve the input 48×48×256 data using 128 1×1 convolution kernels, then input the convolution result into a BN layer and LeakyReLU to accelerate network convergence, then convolve it using 128 3×3 convolution kernels, then input the convolution result into a BN layer and LeakyReLU to accelerate network convergence, then convolve it using 512 1×1 convolution kernels, the output result is 48×48×512 data, then multiply the spatial attention matrix Z with the output result of 48×48×512 data, the final result is 48×48×512; 2) Branch 2: Convolve the input 48×48×64 data using 512 1×1 convolution kernels, and then input the convolution result into the BN layer. The output result is 48×48×512 data. The output data from the two branches are summed and input into LeakyReLU to obtain 48×48×512 data. This 48×48×512 data is then input into the third-layer improved residual network, which is divided into two branches: 1) Branch 1: The input 48×48×512 data is convolved with 256 1×1 convolution kernels. The result of the convolution is then fed into a BN layer and LeakyReLU to accelerate the convergence of the network. Then, 256 3×3 convolution kernels are used for convolution. The result of the convolution is then fed into a BN layer and LeakyReLU to accelerate the convergence of the network. Finally, 1024 1×1 convolution kernels are used for convolution. The output result is 48×48×1024 data. The spatial attention matrix Z is then multiplied with the output result of 48×48×1024 data. The final result is 48×48×1024. 2) Branch 2: Convolve the input 48×48×64 data using 1024 1×1 convolution kernels, and then input the convolution result into the BN layer. The output data is 48×48×1024. The data output from the two branches are added together and input into LeakyReLU, resulting in a data set of 48×48×1024.

7. The facial expression recognition method based on a convolutional neural network according to claim 1, characterized in that, In step S55, Where q is the output probability, c is the number of expressions predicted by the network, softmax is used to normalize the probabilities of all predicted expressions, maxindex is used to obtain the index value of the maximum probability after normalization, and res is the index value of the final predicted expression.

8. The facial expression recognition method based on a convolutional neural network according to claim 1, characterized in that, Before performing adjustable filtering on the facial expression image, the facial expression recognition method based on a convolutional neural network further includes: MTCNN is used to detect faces in face images. The detected face regions and key facial points are then cropped and aligned to obtain expression images.

Citation Information

Patent Citations

  • Expression recognition method based on attention modulation context space information

    CN114758383A