Facial expression recognition methods, devices, equipment and readable storage media

Abstract

This application relates to the field of computer vision technology and provides a facial expression recognition method, apparatus, terminal device, and readable storage medium. The method includes: determining the region images of each facial action unit on a target face image; determining the action intensity of each facial action unit based on the target face image; determining a facial expression enhancement image based on the intensity information of each facial action unit and the target face image; and inputting the region images corresponding to each facial action unit and the facial expression enhancement image into a preset neural network model for processing to obtain the expression recognition result of the target face image. This method can effectively improve the accuracy of facial expression recognition.

Description

Technical Field

[0001] This application belongs to the field of computer vision technology, and in particular relates to facial expression recognition methods, devices, equipment and readable storage media. Background Technology

[0002] Facial expressions are one of the most common ways humans express emotions and intentions. Basic facial expressions generally include seven types: anger, disgust, fear, happiness, sadness, surprise, and neutral expressions. Facial expression recognition has wide applications in human-computer interaction, such as driver fatigue detection, healthcare, and online teaching. In the field of computer vision, facial expression recognition using facial images has become a research hotspot.

[0003] Therefore, how to better perform facial expression recognition from face images has become a pressing technical problem that needs to be solved. Summary of the Invention

[0004] This application provides a facial expression recognition method, apparatus, device, and readable storage medium, which can solve the problem of how to effectively improve the accuracy of facial expression recognition based on face images.

[0005] In a first aspect, embodiments of this application provide a facial expression recognition method, including:

[0006] Identify the regions of each facial action unit on the target face image;

[0007] The motion intensity of each facial motion unit is determined based on the target face image;

[0008] A facial expression enhancement image is determined based on the action intensity of each facial action unit and the target face image;

[0009] The region images corresponding to each facial action unit and the facial expression enhancement images are input into a preset neural network model for processing to obtain the expression recognition results of the target face image.

[0010] In one possible implementation of the first aspect, the preset neural network model includes a local feature extraction model, a global feature extraction model, and a classification model.

[0011] The step of inputting the region image corresponding to each of the facial action units and the enhanced facial expression image into a preset neural network model for processing to obtain the expression recognition result of the target face image includes:

[0012] Each of the aforementioned region images is input into the local feature extraction model to obtain the local region features corresponding to each of the aforementioned region images;

[0013] The enhanced facial expression image is input into the global feature extraction model to obtain the global expression features corresponding to the target face image;

[0014] The features of each local region are fused with the global facial expression features to obtain fused facial expression features;

[0015] The fused facial expression features are input into the classification model to obtain the facial expression recognition result of the target face image.

[0016] In one possible implementation of the first aspect, the local feature extraction model includes a feature extraction sub-model and a feature dimensionality reduction sub-model;

[0017] The step of inputting the images of each region into the local feature extraction model to obtain the local region features corresponding to each facial action unit includes:

[0018] Each of the aforementioned regions is input into the feature extraction sub-model to obtain the initial local region features corresponding to each of the aforementioned facial action units;

[0019] Each of the initial local region features is input into the feature dimensionality reduction sub-model to perform feature dimensionality reduction, thereby obtaining the features of each of the local regions.

[0020] In one possible implementation of the first aspect, determining the region image containing each facial action unit on the target face image includes:

[0021] Identify the activity center point of each facial action unit in the target face image;

[0022] Each facial motion unit is assigned an image of a preset pixel size centered on the activity center point of its respective facial motion unit.

[0023] In one possible implementation of the first aspect, identifying the center of activity of each of the facial action units in the target face image includes:

[0024] Identify the facial key points corresponding to each facial action unit in the target face image;

[0025] The activity center point of each facial motion unit is determined based on the facial key points corresponding to each facial motion unit.

[0026] In one possible implementation of the first aspect, determining the activity center point of each facial motion unit based on the facial key points corresponding to each facial motion unit includes:

[0027] Obtain the distance between the two inner corners of the eyes of the target face in the target face image;

[0028] Based on the distance value and the facial key points corresponding to each facial motion unit, the activity center point of each facial motion unit is determined.

[0029] In one possible implementation of the first aspect, determining the facial expression enhancement image based on the action intensity of each of the facial action units and the target face image includes:

[0030] A Gaussian heatmap is generated based on the motion intensity and center point of each facial motion unit.

[0031] The Gaussian heatmap is fused with the target face image to obtain the facial expression enhancement image.

[0032] Secondly, embodiments of this application provide a facial expression recognition device, including:

[0033] The region image determination unit is used to determine the region image where each facial action unit is located on the face in the target face image;

[0034] An action intensity determination unit is used to determine the action intensity of each facial action unit based on the target face image.

[0035] An expression enhancement unit is used to determine a facial expression enhancement image based on the action intensity of each of the facial action units and the target face image;

[0036] The expression recognition unit is used to input the region image corresponding to each of the facial action units and the enhanced facial expression image into a preset neural network model for processing, so as to obtain the expression recognition result of the target face image.

[0037] Thirdly, embodiments of this application provide a facial expression recognition device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method described in any one of the first aspects above.

[0038] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in any one of the first aspects above.

[0039] Fifthly, embodiments of this application provide a computer program product that, when run on a data processing device, causes the data processing device to perform the method described in any one of the first aspects.

[0040] Based on the method described in the first aspect above, by inputting the facial expression enhancement image of the target face image, which includes the motion intensity of facial action units and the regional images of each facial action unit in the target face image, into a preset neural network model, the expression recognition result of the target face image is obtained. This method integrates the motion intensity of facial action units into expression recognition, making full use of the rich information contained in the motion intensity of facial action units, thereby improving the accuracy of facial expression recognition of face images.

[0041] It is understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant description of the beneficial effects of the method described in the first aspect, and will not be repeated here. Attached Figure Description

[0042] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0043] Figure 1 A schematic flowchart illustrating a facial expression recognition method provided in an embodiment of this application;

[0044] Figure 2 This is a schematic diagram showing the distribution of 68 facial key points in one embodiment of this application;

[0045] Figure 3 This is a schematic diagram showing the approximate location of the activity centers of 15 facial motion units calculated in one embodiment of this application in a face image;

[0046] Figure 4 This is a schematic diagram of the structure of a neural network model provided in an embodiment of this application;

[0047] Figure 5 The experimental results of the neural network model provided in one embodiment of this application on the RaFD dataset are shown in the figure.

[0048] Figure 6 The experimental results of the neural network model provided in one embodiment of this application on the Oulu-CASIA dataset are shown in the figure.

[0049] Figure 7 This is a structural block diagram of a facial expression recognition device provided in an embodiment of this application;

[0050] Figure 8 This is a schematic diagram of the structure of a facial expression recognition device provided in an embodiment of this application. Detailed Implementation

[0051] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0052] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0053] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0054] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0055] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0056] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0057] First, let me explain the terms used in this application:

[0058] Facial action units (AUs) are the outward representations of muscle movements at specific locations on the face. The concept of AUs was first described in 1978 by American psychologist Ekman in the Facial Action Coding System (FACS), proposed from a facial anatomy perspective. FACS defines 44 AUs, specifically defining the area of ​​action, motor characteristics, and composition of various facial expressions for each AU. Each AU represents a facial muscle movement with specific semantics; for example, AU1 represents raising the inner corner of the eyebrow, AU2 represents raising the outer corner of the eyebrow, AU4 represents gathering and lowering the eyebrows, AU9 represents wrinkling the nose, and AU25 represents opening the lips. Different facial expressions can be represented by different combinations of AUs. For instance, anger can be represented by a combination of AU4 (gathering and lowering the eyebrows), AU9 (wrinkling the nose), and AU25 (opening the lips), while a smile can be represented by a combination of AU12 (raising the corners of the mouth) and AU6 (raising the cheeks), and so on.

[0059] Face detection: Specifically, given an image, it outputs the approximate location of faces within the image. For example, a bounding box can be used to mark areas in the image where faces are present, and the location information of the bounding box can be provided; based on this location information, the face regions in the image can be determined. The location information of the bounding box can include the coordinates of the top-left and bottom-right corners of the bounding box, or the coordinates of the top-left corner and the length and width of the bounding box.

[0060] Face alignment refers to mapping faces across all images to a unified coordinate system. This may include the following steps: obtaining the coordinates of facial landmarks; and performing face alignment through affine transformation. Specifically, facial landmarks may be selected from the centers of the left and right eyes, the tip of the nose, and the corners of the mouth.

[0061] Affine transformation refers to the process in geometric space where one vector space is transformed into another vector space through a linear transformation and a translation operation. In this example, different face images can be transformed into a unified vector space by performing an affine transformation.

[0062] This application provides a facial expression recognition method. By inputting the facial expression enhancement image, which includes the motion intensity of facial action units and the regional images of each facial action unit in the target face image, into a preset neural network model, the expression recognition result of the target face image is obtained. This method integrates the motion intensity of facial action units into expression recognition, making full use of the rich information contained in the motion intensity of facial action units, thereby improving the accuracy of facial expression recognition of face images.

[0063] The facial expression recognition method provided in this application will be described exemplarily below with reference to specific embodiments.

[0064] Figure 1 A flowchart illustrating the facial expression recognition method provided in this application is shown below. Figure 1 The method includes:

[0065] Step S110: Determine the region image where each facial action unit is located on the face in the target face image.

[0066] In this embodiment, the target face image refers to the image containing a face to be recognized for expression recognition. The target face image can be a face image captured in real time, a face image stored in advance, or a face image obtained from the Internet.

[0067] Optionally, the target face image is an image obtained after preprocessing the original image. The specific preprocessing process includes: performing face detection on the original image to obtain the face region of the original image, wherein the face region includes a complete face; performing face alignment on the face region and cropping it to a preset size to obtain the target face image. Performing face detection and face alignment on the original image can avoid the adverse effects of image differences (such as different image sizes, different face region sizes on different images, and different rotation angles of faces on different images) on the expression recognition results, and ensure the stability of the expression recognition results.

[0068] In this embodiment, the facial action units in the target face image are the facial action units defined by FACS. To improve computational efficiency, some of the 44 facial action units defined by FACS can be selected for facial expression recognition. For example, in one embodiment, 15 facial action units are selected for expression recognition. These 15 facial action units are AU1 (inner corner of eyebrow raised), AU2 (outer corner of eyebrow raised), AU4 (eyebrows gathered and lowered), AU5 (upper eyelid raised), AU6 (cheek raised), AU7 (eyelid tightened), AU9 (nose wrinkled), AU10 (upper lip raised), AU12 (corner of mouth pulled up), AU15 (corner of mouth pulled down), AU17 (chin raised), AU20 (corner of mouth stretched), AU23 (lip tightened), AU25 (lips parted), and AU26 (chin lowered).

[0069] In this embodiment, the region image where each facial action unit is located refers to the region of interest image of the corresponding facial action unit in the target face image.

[0070] Optionally, the method for determining the images of each region can be as follows: identifying the center point of activity of each facial action unit in the target face image; and determining an image of a preset pixel size centered on the center point of activity of each facial action unit as the region image where each facial action unit is located. The preset pixel size can be set as needed; for example, in one embodiment, the preset pixel size is 20*20 pixels, meaning the size of each region image is 20*20 pixels.

[0071] The above method uses the activity center point to crop the region image, so that the obtained region image can well represent the corresponding facial action unit, thereby ensuring the accuracy of expression recognition.

[0072] Understandably, the center of activity is the point of maximum motion intensity within the corresponding facial motion unit. Since the human face is generally symmetrical about the midline, the motion intensity and position of each facial motion unit are also symmetrical about the midline. Therefore, two center of activity are defined for each facial motion unit, and these two center of activity points are symmetrical about the facial midline. Accordingly, each facial motion unit corresponds to two region images, and these two region images are symmetrical about the facial midline.

[0073] For example, a method for obtaining the activity center of each facial motion unit may include the following steps:

[0074] (1) Identify the facial key points corresponding to each facial action unit in the target face image.

[0075] (2) Determine the activity center point of each facial motion unit based on the facial key points corresponding to each facial motion unit.

[0076] In this embodiment, facial landmark detection specifically refers to locating key feature points of the face in a target face image. These key feature points are the locations of the eyebrows, eyes, nose, mouth, and facial contours within the face image. The obtained facial landmarks are, in particular, the coordinate values ​​of each key feature point in the target face image.

[0077] Before obtaining the facial key points corresponding to each facial action unit, it is necessary to first determine all facial key points on the target face image, for example, there can be N facial key points, where the size of N can be determined as needed, for example, N can be 21, 29, 68, 96, 98, 106, 108, etc. Those skilled in the art can use various existing facial key point detection techniques to obtain different numbers of facial key points, which will not be elaborated here.

[0078] For example, in one embodiment, the preset number of facial landmarks is 68, and the facial landmark detection technology used to obtain 68 facial landmarks can be Openface 2.0 toolkit, seetaFace 6.0 or Dlib library, etc.

[0079] Figure 2 This is a schematic diagram showing the distribution of 68 facial key points obtained using the Openface 2.0 toolbox in one embodiment of this application on a face image, where the numbers are the serial numbers of the facial key points, and * indicates the position of the facial key points.

[0080] In this embodiment, facial key points are distributed across the entire target face image, and the activity center point of each facial action unit can be determined by using the facial key points corresponding to each facial action unit.

[0081] To ensure accurate calculation of the coordinates of the activity center points of facial motion units, stable facial keypoints can be identified as the corresponding facial keypoints for each facial motion unit, thereby determining the activity center points of each facial motion unit. Stable facial keypoints are those whose coordinates do not change with changes in facial expression.

[0082] Below, with Figure 2 Taking 68 facial landmarks as an example, this section explains facial landmarks with stable positions. Figure 2 As shown, facial landmarks numbered 40 and 43, used to mark the inner corners of the eyes, and 37 and 46, used to mark the outer corners of the eyes, and facial landmarks numbered 28, 31, and 9, located on the center line of the face image, are stable facial landmarks.

[0083] To eliminate the size difference of faces in different face images, in an optional embodiment, the distance d between the two inner corners of the eyes in the target face image is used as the scaling distance, and the activity center point of the facial action unit corresponding to the face key point is obtained by calculating the coordinates of the stable facial key points and the scaling distance.

[0084] For example, the activity center point corresponding to each facial motion unit is obtained according to the following method: obtaining the distance value between the two inner corners of the eyes on the face region in the target face image; determining the activity center point corresponding to each facial motion unit based on the facial key points and distance values ​​corresponding to each facial motion unit. Specifically, the activity center point corresponding to each facial motion unit can be obtained by substituting the facial key points and distance values ​​corresponding to each facial motion unit into a preset calculation formula.

[0085] In one embodiment, 15 facial motion units from the target face image are used for facial expression recognition, and the number of facial key points is 68; the distance between the inner corners of the eyes is d, and the motion centers of the 15 facial motion units are calculated according to the formula in Table 1.

[0086] Table 1

[0087]

[0088]

[0089] Table 1 shows the formula for calculating the position value of the motion center point of each facial motion unit. Each facial motion unit is divided into left and right parts. The serial number indicates the number of each defined facial motion unit AU, AU represents the code of the facial motion unit in the corresponding FACS, x and y represent the coordinates of the facial key points at the corresponding serial number, and d represents the Euclidean distance between the two inner corners of the eyes. Figure 3 This is a schematic diagram showing the approximate location of the activity centers of 15 facial motion units calculated in one embodiment of this application in a face image.

[0090] Step S120: Determine the motion intensity of each facial motion unit based on the target face image.

[0091] In this embodiment, any available facial action unit intensity estimation algorithm in the prior art can be used to obtain the action intensity of each facial action unit on the target face image. For example, Openface 2.0 can be used to estimate the action intensity of each facial action unit on the target face image.

[0092] For example, the intensity of a facial motion unit can be represented by an intensity value, which is divided into 6 levels (the intensity value ranges from 0 to 5, and the higher the intensity value, the higher the intensity of the corresponding facial motion unit).

[0093] Step S130: Determine the facial expression enhancement image based on the motion intensity of each facial motion unit and the target face image.

[0094] In this embodiment, the facial expression enhancement image is obtained by fusing the motion intensity of each facial motion unit onto the target face image.

[0095] In one embodiment, a method for obtaining an enhanced facial expression image includes: generating a Gaussian heatmap based on the motion intensity and center point of each facial action unit; and fusing the Gaussian heatmap with a target face image to obtain the enhanced facial expression image.

[0096] For example, a method for obtaining an enhanced facial expression image may specifically include the following steps:

[0097] (1) Obtain the Gaussian function values ​​of multiple second pixel points corresponding to each facial action unit; the second pixel point is the pixel point in the square on the target face image with each activity center point as the center and the first preset number of pixels as the side length.

[0098] (2) Obtain the product of each Gaussian function value and the intensity value of the corresponding facial motion unit.

[0099] For example, the product of the Gaussian function value and the intensity value of the corresponding facial motion unit is obtained according to the following formula:

[0100]

[0101] Where g(m) represents the product of the Gaussian function value and the intensity value of the second pixel m, λ represents a constant coefficient, I represents the intensity value of the facial motion unit, σ is a standard deviation coefficient, and p i This represents the i-th active center point corresponding to the second pixel. This represents the second pixel m and the active center point p. i Euclidean distance.

[0102] (3) Assign values ​​to the pixels corresponding to each second pixel point on the initial template image to obtain a Gaussian heatmap; wherein the initial template image and the target face image are the same size and the value of each pixel is 0.

[0103] (4) The Gaussian heatmap is fused with the target face image to obtain an enhanced facial expression image.

[0104] For example, the target face image is an RGB three-channel color image, with each channel being a two-dimensional array. To achieve the fusion of the Gaussian heatmap and the target face image, the Gaussian heatmap can be plotted on the three-channel template image. Specifically, this involves: first, creating an initial two-dimensional array of the same size as the target face image, where each value is defaulted to 0; obtaining the product of the Gaussian function value of each second pixel in the target face image and the intensity value of the corresponding facial action unit; assigning each product to the point corresponding to the second pixel in the initial two-dimensional array to obtain the target two-dimensional array; copying the target two-dimensional data twice to obtain three target two-dimensional arrays; and multiplying the three target two-dimensional arrays with the arrays of the three channels of the target face image to obtain the facial expression enhancement image.

[0105] For example, in the facial expression enhancement image, the second pixel corresponding to each activity center point has a larger Gaussian value for a smaller distance from the activity center point. When the intensity value of the corresponding facial action unit is not 0, the value corresponding to this second pixel on the Gaussian heatmap is larger. This means that the trend of the distance between the second pixel and the corresponding activity center point is transformed into the trend of the corresponding value on the Gaussian heatmap. Thus, although each facial action unit corresponds to only one intensity value, different values ​​are assigned to different second pixels through the Gaussian function, accurately reflecting the differences in the intensity of facial action units among different second pixels. This fully utilizes the rich information contained in the intensity values ​​of facial action units, improving the accuracy of expression recognition.

[0106] Step S140: Input the region image and facial expression enhancement image corresponding to each facial action unit into a preset neural network model for processing to obtain the expression recognition result of the target face image.

[0107] In this embodiment, the preset neural network model can be a deep neural network model, which performs feature extraction and facial expression recognition classification on images of various regions and facial expression enhancement images.

[0108] Optionally, the deep neural network model can be a convolutional neural network (CNN) model, a graph neural network model, or a generative adversarial network (GAN) model.

[0109] Optionally, the convolutional neural network model can be a pyramid pattern network model, a Unet network model, a VGG network model (e.g., a VGG16 network model), a ResNet residual network model, or a convolutional autoencoder.

[0110] In the above embodiments, by inputting the region images of each facial action unit and the enhanced facial expression image in the target face image into a preset neural network model, the expression recognition result of the target face image is obtained. This method integrates the intensity information of the facial action unit into the expression recognition, making full use of the rich information contained in the intensity information of the facial action unit, thereby improving the accuracy of facial expression recognition of face images.

[0111] In one embodiment, the preset neural network model is a network model composed of multiple models. For example, the preset neural network model may include a local feature extraction model, a global feature extraction model, and a classification model. Step S104 includes:

[0112] (1) Input each region image into the local feature extraction model to obtain the local region features corresponding to each region image.

[0113] In this embodiment, the local region features corresponding to each region image represent the image features of the corresponding facial action unit. Optionally, the local feature extraction model is a convolutional neural network model.

[0114] In one embodiment, the local feature extraction model includes a feature extraction sub-model and a feature dimensionality reduction sub-model. The process of obtaining the local region features corresponding to each facial action unit includes: inputting the images of each region into the feature extraction sub-model to obtain the initial local region features corresponding to each facial action unit; and inputting the initial local region features into the feature dimensionality reduction sub-model to perform feature dimensionality reduction and obtain the features of each local region. Since there are multiple facial action units in the target face image, directly considering the local region features of each facial action unit would result in a large dimensionality of the input features, which could easily lead to overfitting of the neural network. The feature dimensionality reduction sub-model reduces the dimensionality of the initial local region features to avoid overfitting.

[0115] For example, the feature extraction sub-model includes 5 convolutional layers, each including a 2D convolution operation with a kernel of 3, a stride of 2, and padding of 1, batch normalization, and an activation function; the feature dimensionality reduction sub-model includes 4 convolutional layers, each including a 2D convolution operation with a kernel of 1, a stride of 1, and padding of 0, batch normalization, and an activation function. The activation function can be LeakReLU, sigmoid, tanh, etc. Preferably, LeakReLU is used.

[0116] In one embodiment, the first convolutional layer of the feature extraction sub-model has 3 channels as input and 64 channels as output; the second convolutional layer has 64 channels as input and 128 channels as output; the third convolutional layer has 128 channels as input and 256 channels as output; the fourth convolutional layer has 256 channels as input and 512 channels as output; and the fifth convolutional layer has 512 channels as input and 1024 channels as output. Similarly, the first convolutional layer of the feature dimensionality reduction sub-model has 1024 channels as input and 512 channels as output; the second convolutional layer has 512 channels as input and 256 channels as output; the third convolutional layer has 256 channels as input and 128 channels as output; and the fourth convolutional layer has 128 channels as input and 64 channels as output.

[0117] (2) Input the enhanced facial expression image into the global feature extraction model to obtain the global expression features corresponding to the target face image.

[0118] In this embodiment, since the facial expression enhancement image incorporates the intensity values ​​of each facial action unit, the global expression feature obtained by feature extraction from the facial expression enhancement image is a feature that includes the intensity values ​​of the facial action units.

[0119] In one embodiment, the global feature extraction model is a convolutional neural network model.

[0120] For example, the global feature extraction model may include seven convolutional layers. Each convolutional layer includes a two-dimensional convolution operation with a kernel size of 4, a stride of 2, and padding of 1, batch normalization, and an activation function. The activation function can be LeakReLU, sigmoid, tanh, etc. Preferably, the activation function is LeakReLU.

[0121] In one embodiment, the global feature extraction model includes a batch normalization operation before the first convolutional layer. Specifically, the first convolutional layer has a 3-channel input and a 64-channel output; the second convolutional layer has a 64-channel input and a 128-channel output; the third convolutional layer has a 128-channel input and a 256-channel output; the fourth convolutional layer has a 256-channel input and a 512-channel output; the fifth convolutional layer has a 512-channel input and a 1024-channel output; the sixth convolutional layer has a 1024-channel input and a 2048-channel output; and the seventh convolutional layer has a 2048-channel input and a 4096-channel output.

[0122] (3) The features of each local region are fused with the global facial expression features to obtain the fused facial expression features.

[0123] In the embodiments, the feature fusion methods can include feature concatenation, feature summation (mean, pooling, weighted summation), and element-wise multiplication between corresponding features.

[0124] Suppose that the dimensions of two input features x and y are p and q, respectively; if the two features can be directly concatenated, the dimension of the fused feature z is p+q; if the two features are combined into a composite vector, the dimension of the fused feature is p+qi, where i represents an imaginary number.

[0125] For example, the feature fusion method may include the following steps: stretching the global expression features into a single row of data; stretching each local region feature into a single row of data; and then placing the data corresponding to each local region feature after the global expression features to form a new row of data, which is the fused expression features.

[0126] In this embodiment, a preset classification model is used to process the input fused facial expression features and output the facial expression recognition result of the target face image. The facial expression recognition result can be any type of any preset facial expression. For example, it can be any expression among anger, disgust, fear, happiness, sadness, surprise, and neutral face.

[0127] (4) Input the fused facial expression features into the classification model to obtain the facial expression recognition results of the target face image.

[0128] In one embodiment, the classification model is a fully connected neural network classification model, such as a multilayer perceptron model. Specifically, the classification model can be a three-layer perceptron model, with each layer being a linear fully connected layer.

[0129] In the above method, local region features of each facial action unit are extracted from the target face image; global expression features are extracted from the enhanced facial expression image; and the local region features are fused with the global expression features to obtain fused expression features. These fused expression features are then input into a pre-defined classification model to obtain the expression recognition result of the target face image. This method fuses global expression features, including the intensity information of facial action units, with the features of each local region to achieve facial expression recognition of face images. It fully utilizes the intensity information of facial action units, thereby improving the accuracy of facial expression recognition in face images.

[0130] In one embodiment, the preset neural network models are all built based on the PyTorch framework. Figure 4 This is a schematic diagram of the structure of a neural network model provided in an embodiment of this application. The neural network model is a convolutional autoencoder. Figure 4 As shown, AU-RoI represents the feature extraction sub-model in the local feature extraction model, which is used to extract the initial local region features of each facial action unit; FD represents the feature dimensionality reduction sub-model in the local feature extraction model, which reduces the dimensionality of the initial local region features to obtain the features of each local region; AUI-E represents the global feature extraction model, which is used to extract global expression features from the facial expression enhancement image; and FC represents the classification model, which is used to classify the expression of the target face image based on the fusion features of each local region feature and the global expression feature.

[0131] The AU-RoI part takes as input the region of interest image of each facial action unit (FAI), which is 20*20 pixels in size, and outputs the local region features of each FAI. The AUI-E part takes as input the enhanced facial expression image after augmentation with FAI, and outputs the enhanced global expression features. The FC part takes as input the fused features of each local region feature and the global expression feature, and outputs a one-dimensional matrix with a length equal to the number of expression categories (e.g., 6 or 7). Finally, the softmax function selects the label with the largest value in the matrix as the result, which corresponds to the expression label.

[0132] In this embodiment, the preset training parameters of the neural network model include: 10 epochs, a batch size of 16, and an initial learning rate of 0.001. The optimizer is a stochastic gradient descent method with a decay weight of 0.0001 and a momentum of 0.9.

[0133] To demonstrate the effectiveness of the facial expression recognition method in the embodiments of this application, we utilize... Figure 6 The neural network model shown was validated on the RAFD and Oulu-Casia datasets.

[0134] The RAFD dataset is a facial expression dataset consisting of images taken of 67 subjects in a controlled laboratory environment. This dataset records facial poses from five angles, eye gaze directions from three directions, and eight facial expressions for each subject. It contains a total of 8056 high-resolution images. In our experiments, we selected only frontal facial images of seven expressions (excluding contempt). We used a total of 1407 experimental samples from the RAFD dataset. Specifically, we used seven expressions from 67 subjects across three gaze directions.

[0135] The Oulu-Casia dataset contains 480 video sequences collected from 80 subjects, each sequence consisting of one of six basic facial expressions, ranging from neutral to neutral. The dataset uses both near-infrared (NIR) and visible light (VIS) imaging systems to capture facial expressions under three different lighting conditions. Our experiment used only the last three frames and the first frame of each sequence, captured using the VIS system under normal indoor lighting, to capture neutral expressions. Therefore, the Oulu-Casia database contains 1440 facial images used in our experiment.

[0136] Ten-fold cross-validation was performed on two datasets. Specifically, all available images in one dataset were divided into 10 equal parts. Nine parts were used to train the neural network model, and one part was used to test it. This process was repeated 10 times on each dataset, and the average accuracy of the 10 tests was used as the evaluation metric. For example, the RaFD dataset has 1400 images, divided into 10 parts, each containing approximately 140 images. Nine parts (9*140) were used to train the model, and the remaining part was used for testing. This was repeated 10 times, with each test image inputting a different image to the model, thus making the resulting model more robust.

[0137] Figure 5 and Figure 6The figures show the experimental results of the neural network model in this application embodiment on the RaFD and Oulu-CASIA datasets. The figures show the confusion matrix obtained from the experiments, indicating which expression each different expression was identified as. If all expressions of a certain expression are correctly identified, 100% is displayed in the corresponding position. On the RaFD dataset, experiments were conducted on seven expressions (anger, disgust, fear, happiness, sadness, surprise, and neutral face). The expression recognition accuracy of the model in this application embodiment reached 99.79%. On the Oulu-CASIA dataset, experiments were conducted on six expressions (anger, disgust, fear, happiness, sadness, and surprise), and the expression recognition accuracy reached 100%.

[0138] In addition, the experimental results in the embodiments of this application are compared with the experimental results in existing public documents, and the comparison results are shown in Tables 2 and 3.

[0139] Table 2

[0140] Model Accuracy (%) k-Nearest Neighbor (KNN) model 94.60 Extreme Learning Machine (ELM) Model 98.65 Discriminative Deep Association Learning (DDAL) Model 94.63 Two-branch untangled generative adversarial network (TDGAN) model 99.32 Extended MiniExpNet model 96.12 This application model 99.79

[0141] Table 3

[0142]

[0143] Table 2 shows the comparison results of the model in this embodiment with other models for recognizing seven facial expressions on the RaFD dataset. Table 3 shows the comparison results of the model in this embodiment with other models for recognizing seven facial expressions on the RaFD dataset. As can be seen from Tables 2 and 3, the neural network model based on the facial expression recognition method in this embodiment outperforms other network models on both datasets, thus demonstrating the superiority of the facial expression recognition method in this embodiment.

[0144] In summary, compared with the prior art, the facial expression recognition method in this application embodiment has the following technical effects:

[0145] (1) The motion intensity of the facial motion unit is integrated into the expression recognition, which makes full use of the rich information contained in the motion intensity of the facial motion unit, thereby improving the accuracy of facial expression recognition of face images.

[0146] (2) The neural network model includes a local feature extraction model and a global feature extraction model, which makes the steps of extracting local region features and extracting global facial expression features independent to a certain extent, so that each feature retains its own characteristics; finally, the local region features and global facial expression features are fused and input into the classification model, so that the facial expression recognition result fully considers the two different features and ensures the accuracy of facial expression recognition.

[0147] (3) The local feature extraction model includes a feature extraction sub-model and a feature dimensionality reduction sub-model, which allows for the dimensionality reduction of the obtained local region features while extracting local region features, thereby avoiding overfitting.

[0148] (4) The images of each region are determined by the activity center point of each facial action unit, so that the obtained regional images correspond more strongly to each facial action unit, thereby ensuring that each regional image can accurately represent the corresponding facial action unit.

[0149] (5) The center point of each facial action unit is determined by facial key points, which is a simple and effective method.

[0150] (6) The face in the corresponding target face image is scaled by using the distance of the inner corner of the eye as a reference. The position of the activity center point is obtained by calculating the distance of the inner corner of the eye and the key points of the face. This avoids the influence of different face shapes on the position of the center point of the facial action unit and improves the universality of the calculation results.

[0151] (7) The intensity of facial action units is fused into the target face image by means of Gaussian heatmap, which is a simple and effective method.

[0152] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0153] Corresponding to the facial expression recognition method described in the above embodiments, Figure 7 A structural block diagram of the facial expression recognition device provided in the embodiments of this application is shown. For ease of explanation, only the parts related to the embodiments of this application are shown.

[0154] Reference Figure 7 The facial expression recognition device 700 includes: a region image determination unit 701, a motion intensity determination unit 702, an expression enhancement unit 703, and an expression recognition unit 704.

[0155] The region image determination unit 701 is used to determine the region image where each facial action unit is located on the face in the target face image;

[0156] The motion intensity determination unit 702 is used to determine the motion intensity of each of the facial motion units based on the target face image.

[0157] The expression enhancement unit 703 is used to determine a facial expression enhancement image based on the action intensity of each of the facial action units and the target face image;

[0158] The expression recognition unit 704 is used to input the region images corresponding to each facial action unit and the enhanced facial expression image into a preset neural network model for processing, so as to obtain the expression recognition result of the target face image.

[0159] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0160] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0161] Based on the same inventive concept, one embodiment of this application also provides an expression recognition device, such as... Figure 8 As shown, the facial expression recognition device 800 of this embodiment includes: a processor 801, a memory 802, and a computer program 804 stored in the memory 802 and executable on the processor 801. The computer program 804 can be executed by the processor 801 to generate instructions 803, which the processor 801 can use to implement the steps in the various facial expression recognition method embodiments described above. Alternatively, when the processor 801 executes the computer program 804, it implements the functions of each module / unit in the various device embodiments described above, for example... Figure 7 The functions of the area image determination unit 701 to the expression recognition unit 704 are shown.

[0162] The facial expression recognition device 800 can be a terminal device. For example, it can be a mobile terminal, desktop computer, laptop, PDA, etc. The facial expression recognition device may include, but is not limited to, a processor 801 and a memory 802. Those skilled in the art will understand that... Figure 8This is merely an example of a facial expression recognition device 800 and does not constitute a limitation on the facial expression recognition device 800. It may include more or fewer components than shown in the figure, or combine certain components, or different components, such as input / output devices, network access devices, etc.

[0163] The processor 801 may be a Central Processing Unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0164] In some embodiments, the memory 802 may be an internal storage unit of the facial expression recognition device 800, such as a hard disk or memory of the facial expression recognition device 800. In other embodiments, the memory 802 may be an external storage device of the facial expression recognition device 800, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the facial expression recognition device 800. Further, the memory 802 may include both internal and external storage units of the facial expression recognition device 800. The memory 802 is used to store the operating system, applications, bootloader, data, and other programs, such as the program code of the computer program. The memory 802 can also be used to temporarily store data that has been output or will be output.

[0165] This application also provides a network device, which includes: at least one processor, a memory, and a computer program stored in the memory and executable on the at least one processor, wherein the processor executes the computer program to implement the steps in any of the above method embodiments.

[0166] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps described in the various method embodiments above.

[0167] This application provides a computer program product that, when run on a data processing device, enables the data processing device to implement the steps described in the various method embodiments above.

[0168] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying computer program code to a device / data processing equipment, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.

[0169] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0170] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0171] In the embodiments provided in this application, it should be understood that the disclosed apparatus / devices and methods can be implemented in other ways. For example, the apparatus / device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0172] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0173] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A facial expression recognition method, characterized by, The method comprises the following steps: determining the region image where each facial action unit on the face in the target face image is located; determining the action intensity of each facial action unit according to the target face image; determining a facial expression enhancement image according to the action intensity of each facial action unit and the target face image; inputting the region image corresponding to each facial action unit and the facial expression enhancement image into a preset neural network model for processing to obtain an expression recognition result of the target face image; determining a facial expression enhancement image according to the action intensity of each facial action unit and the target face image, comprising: obtaining a plurality of second pixel point Gaussian function values corresponding to each facial action unit; the second pixel point is a pixel point in a square with a first preset number of pixel points as the side length and with the activity center point of each facial action unit as the center on the target face image; obtaining the product of each Gaussian function value and the intensity value of the action intensity of the corresponding facial action unit; assigning the pixel points corresponding to each second pixel point on the initial template image to obtain a Gaussian heat map; the initial template image is the same size as the target face image and the value of each pixel is 0; fusing the Gaussian heat map and the target face image to obtain the facial expression enhancement image; fusing the Gaussian heat map and the target face image to obtain the facial expression enhancement image, comprising: establishing an initial two-dimensional array with the same size as the target face image, and the value of each point in the initial two-dimensional array is 0 by default; obtaining the product of each second pixel point Gaussian function value in the target face image and the intensity value of the action intensity of the corresponding facial action unit; assigning each product to the point corresponding to the second pixel point in the initial two-dimensional array to obtain a target two-dimensional array; copying the target two-dimensional data twice to obtain three target two-dimensional arrays; multiplying the three target two-dimensional arrays with the three channel arrays of the target face image to obtain the facial expression enhancement image.

2. The method of claim 1, wherein, The preset neural network model comprises a local feature extraction model, a global feature extraction model and a classification model, inputting each region image into the local feature extraction model to obtain the local region feature corresponding to each region image; inputting the facial expression enhancement image into the global feature extraction model to obtain the global expression feature corresponding to the target face image; fusing each local region feature and the global expression feature to obtain a fused expression feature; inputting the fused expression feature into the classification model to obtain the expression recognition result of the target face image. The local feature extraction model comprises a feature extraction sub-model and a feature dimension reduction sub-model; 3. The method of claim 2, wherein, inputting each region image into the local feature extraction model to obtain the local region feature corresponding to each facial action unit, comprising: ​ input each of the region images to the feature extraction sub-model to obtain initial local region features corresponding to each of the facial action units; input each of the initial local region features to the feature dimension reduction sub-model for feature dimension reduction to obtain each of the local region features.

4. The method of claim 1, wherein, The region images in which each facial action unit on the face in the target face image are determined, comprising: identifying the activity center points of each facial action unit in the target face image; determining, respectively, images of a preset pixel size centered on the activity center points of each facial action unit as the region images in which each facial action unit is located.

5. The method of claim 4, wherein, The activity center points of each facial action unit in the target face image are identified, comprising: identifying the face key points corresponding to each facial action unit in the target face image; determining the activity center points of each facial action unit according to the face key points corresponding to each facial action unit, respectively.

6. The method of claim 5, wherein, The activity center points of each facial action unit are determined according to the face key points corresponding to each facial action unit, respectively, comprising: obtaining a distance value between two inner corners of the eyes of the face in the target face image; determining the activity center points of each facial action unit according to the distance value and the face key points corresponding to each facial action unit, respectively.

7. A facial expression recognition apparatus, characterized by comprising: Comprising: a region image determination unit configured to determine region images in which each facial action unit on the face in a target face image is located; an action intensity determination unit configured to determine action intensities of each facial action unit according to the target face image; an expression enhancement unit configured to determine a facial expression enhancement image according to the action intensities of each facial action unit and the target face image; an expression recognition unit configured to input the region images corresponding to each facial action unit and the facial expression enhancement image into a preset neural network model for processing to obtain an expression recognition result of the target face image; The facial expression enhancement image is determined according to the action intensities of each facial action unit and the target face image, comprising: obtaining a plurality of second pixel point Gaussian function values corresponding to each facial action unit; the second pixel point is a pixel point in a square centered on the activity center point of each facial action unit and having a first preset number of pixel points as the side length on the target face image; obtaining a product of each Gaussian function value and an intensity value of the action intensity of the corresponding facial action unit; assigning values to the pixel points on the initial template image corresponding to each second pixel point to obtain a Gaussian heat map; the initial template image is the same size as the target face image and each pixel has a value of 0; fusing the Gaussian heat map and the target face image to obtain the facial expression enhancement image; Fusing the Gaussian heat map and the target face image to obtain the facial expression enhancement image, comprising: establishing an initial two-dimensional array with the same size as the target face image, and each value in the initial two-dimensional array is defaulted to 0; obtaining a product of each second pixel point Gaussian function value in the target face image and a corresponding facial action unit action intensity value; assigning each product to a point corresponding to the second pixel point in an initial two-dimensional array to obtain a target two-dimensional array; copying the target two-dimensional data twice to obtain three target two-dimensional arrays; and multiplying the three target two-dimensional arrays with arrays of three channels of the target face image to obtain a facial expression enhanced image.

8. A facial expression recognition device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The processor executes the computer program to implement the method of any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, the computer-readable storage medium comprising: The computer program is executed by the processor to implement the method of any one of claims 1 to 6.

Images