Facial expression recognition method, system and storage medium with expression intensity variation

Abstract

The present invention discloses a facial expression recognition method, system and storage medium for expression intensity changes. The method comprises: obtaining a data sample set, the data sample set comprising peak expression samples and non-peak expression samples; extracting the feature vector of each sample in the data sample set, the feature vectors of all samples forming a feature matrix; constructing a first adjacency matrix and a second adjacency matrix; inputting the feature matrix, the first adjacency matrix and the second adjacency matrix into a graph convolutional neural network for training. The present invention can make full use of the local information between expressions with similar intensities, the peak-guided information between peak expressions and non-peak expressions, and can effectively improve the classification accuracy of expressions with varying intensities.

Description

Technical Field

[0001] The present invention belongs to the technical field of expression recognition, and more specifically, relates to a method for recognizing facial expressions with intensity variations and an application thereof. Background Art

[0002] Facial expression is one of the most important non-verbal communication methods, which can convey one's inner feelings. In order to make computers understand various facial expressions, facial expression recognition has become one of the research hotspots in the field of computer vision.

[0003] Graph convolutional neural network is a neural network that performs convolution operations on graph data structures. It is derived from convolutional neural networks in deep learning, but it can be used in non-Euclidean domains where convolutional neural networks are not good at. Graph convolutional neural networks inherit most of the advantages of deep learning and show strong manifold representation capabilities for nodes, edges or subgraphs of graphs.

[0004] Most of the existing deep learning methods focus on facial expression recognition with peak expressions. When applied to facial expression recognition with non-peak expressions, the performance of existing methods will be greatly challenged. When the expression intensity varies greatly compared to the peak expression, the intra-class distance of facial expressions will increase accordingly. In addition, when the expression intensity is weak, the changes in people's facial movements are also subtle and difficult to distinguish. At the same time, due to the lack of labeled data, the model cannot be trained with sufficient data, and its generalization ability is naturally insufficient. These problems lead to limitations in the application of existing methods and low recognition accuracy. Utilizing the advantages of graph neural networks is conducive to solving the above problems, but there is no public method in the prior art on how to further optimize graph neural networks for expression recognition with intensity changes. Summary of the invention

[0005] In response to at least one defect or improvement need in the prior art, the present invention provides a method, system and storage medium for recognizing facial expressions with varying expression intensity, which can fully utilize local information between expressions with similar intensities and peak-lead information between peak expressions and non-peak expressions, and can effectively improve the classification accuracy of expressions with varying intensity.

[0006] To achieve the above object, according to a first aspect of the present invention, a method for recognizing facial expressions with expression intensity changes is provided, comprising:

[0007] Acquire a data sample set, wherein the data sample set includes a peak expression sample and a non-peak expression sample;

[0008] Extracting feature vectors of samples in the data sample set, wherein the feature vectors of all samples form a feature matrix;

[0009] For each sample, calculate the distance between it and other samples, take the multiple samples closest to it as its first neighbor samples, and construct the first adjacency matrix according to whether there is a first neighbor relationship between every two samples; for each sample, calculate the distance between it and other peak expression samples, take the multiple samples closest to it as its second neighbor samples, and construct the second adjacency matrix according to whether there is a second neighbor relationship between every two samples;

[0010] The feature matrix, the first adjacency matrix and the second adjacency matrix are input into a graph convolutional neural network for training.

[0011] Furthermore, constructing the first adjacency matrix includes:

[0012] Let the i-th sample be x i , record the value of the i-th row and j-th column in the first adjacency matrix as If the sample x j is the sample x i The first neighbor sample of otherwise

[0013] Further, constructing the second adjacency matrix includes:

[0014] Let the i-th sample be x i , record the value of the i-th row and j-th column in the second adjacency matrix as If the sample x j is the sample x i The second neighbor sample of otherwise

[0015] Furthermore, the graph convolutional neural network includes a fusion module, a graph convolution module and a classification module, and the input to the graph convolutional neural network for training includes the steps of:

[0016] Inputting the first adjacency matrix and the second adjacency matrix into the fusion module for weighted fusion to obtain a weighted adjacency matrix;

[0017] Inputting the feature matrix and the weighted adjacency matrix into the graph convolution module;

[0018] The output data of the graph convolution module is input to the classification module.

[0019] Furthermore, the first adjacency matrix contains local information between sample-similar samples, and the second adjacency matrix contains peak guide information between non-peak samples and peak samples, and weighted weight parameters of the first adjacency matrix and the second adjacency matrix are adjusted.

[0020] Furthermore, the graph convolutional neural network includes a first graph convolutional module, a second graph convolutional module, an attention module and a classification module, and the input to the graph convolutional neural network for training includes the steps of:

[0021] Inputting the feature matrix and the first adjacency matrix into the first graph convolution module;

[0022] Inputting the feature matrix and the second adjacency matrix into the second graph convolution module;

[0023] Inputting the output data of the first graph convolution module and the second graph convolution module into the attention module;

[0024] The output data of the attention module is input into the classification module.

[0025] Furthermore, the calculation of the distance between it and other samples and the calculation of the distance between it and other peak expression samples are both calculations of the Euclidean distance.

[0026] Furthermore, the sample is preprocessed before extracting the feature vector, and the preprocessing includes:

[0027] Determine the positions of the eye centers and mouth centers visible in the input image through facial feature points;

[0028] Calculate the transformation matrix T between the input image and the aligned image points. The calculation formula of the transformation matrix T is:

[0029] set up is the horizontal center of the aligned image, w is the horizontal width of the aligned image, is the head posture angle,

[0030] S1=S mouth ,

[0031]

[0032] Among them, S1, S mouth is the coordinate of the center of the mouth in the original image, t1 is the coordinate of the center of the mouth in the aligned image, S2 is the coordinate of the center of the eye in the input image, t2 is the coordinate of the center of the eye in the aligned image, S l.eye is the coordinate of the left eye center of the input image, S r.eye is the coordinate of the right eye center of the input image, x center To align the horizontal center of the image, y eyes To align the horizontal center of the eye of the image, d xmouth To align the horizontal width of the mouth in the image, y mouthTo align the vertical coordinate of the center of the mouth in the image, when one eye is invisible, the coordinate of the visible eye is S v.eye To indicate that S mouth represents the coordinates of the center of the mouth. When only one eye is visible in the input image, S2 = S v.eye .

[0033] According to a second aspect of the present invention, there is provided a facial expression recognition system for expression intensity variation, comprising:

[0034] An acquisition module, used for acquiring a data sample set, wherein the data sample set includes a peak expression sample and a non-peak expression sample;

[0035] A feature extraction module is used to extract a feature vector of each sample in the data sample set, and the feature vectors of all samples form a feature matrix;

[0036] The adjacency matrix construction module is used to calculate the distance between each sample and other samples, take multiple samples with the closest distance to it as its first neighbor samples, and construct a first adjacency matrix according to whether there is a first neighbor relationship between every two samples; for each sample, calculate the distance between it and other peak expression samples, take multiple samples with the closest distance to it as its second neighbor samples, and construct a second adjacency matrix according to whether there is a second neighbor relationship between every two samples;

[0037] A training module is used to input the feature matrix, the first adjacency matrix and the second adjacency matrix into a graph convolutional neural network for training.

[0038] According to a third aspect of the present invention, there is provided a storage medium on which a computer program is stored, and when the computer program is executed by a processor, any of the above methods is implemented.

[0039] In general, compared with the prior art, the present invention has the following beneficial effects:

[0040] (1) The present invention makes full use of the peak guide information between non-peak samples and peak samples to construct the second adjacency matrix, thereby improving the recognition accuracy of expression by reducing the spatial distance between peak expression and non-peak expression;

[0041] (2) The present invention utilizes a subspace fusion model of a weighted adjacency matrix to fuse the first adjacency matrix with the second adjacency matrix. While utilizing the peak guide information, the present invention also considers the local information between expressions with similar intensities, thereby making full use of the effective information of expressions with different intensities in the data sample.

[0042] (3) The present invention also adopts a subspace fusion model of the attention mechanism. Compared with the subspace fusion method of the weighted adjacency matrix, the attention mechanism provides trainable weights, which promotes the effective fusion of peak expression information and peak-leading information between expressions with similar intensity in another way, greatly improving the accuracy and robustness of the facial expression recognition task.

[0043] (4) The present invention utilizes a semi-supervised neural network such as a graph convolutional neural network, which enables better recognition accuracy to be achieved while using less labeled data, thereby reducing the burden of sample labeling. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 is a flow chart of a method for recognizing facial expressions with intensity changes according to an embodiment of the present invention;

[0045] Figure 2 is a schematic diagram of a first adjacency matrix according to an embodiment of the present invention;

[0046] Figure 3 is a schematic diagram of a second adjacency matrix according to an embodiment of the present invention;

[0047] Figure 4 is a schematic diagram of a graph convolutional neural network module according to an embodiment of the present invention;

[0048] Figure 5 is a network structure diagram of a subspace fusion model of a weighted adjacency matrix according to an embodiment of the present invention;

[0049] Figure 6 It is a model network structure diagram of the subspace fusion model of the attention mechanism of an embodiment of the present invention. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0051] like Figure 1 As shown, a facial expression recognition method of a facial expression intensity change according to an embodiment of the present invention comprises the steps of:

[0052] S101, obtaining a data sample set, where the data sample set includes peak expression samples and non-peak expression samples.

[0053] In addition to the basic expression categories (anger, disgust, fear, happiness, sadness, and surprise), facial expressions also contain expression intensity, such as the degree of happiness. Peak expression samples refer to the image samples with the strongest degree of expression in a category, and samples other than peak expression samples are non-peak expression samples.

[0054] Furthermore, before inputting into the next step for feature extraction, the sample is preprocessed. The steps of data preprocessing method are as follows:

[0055] Crop the face area according to the facial feature points and normalize it;

[0056] Furthermore, the specific steps of face cropping and normalization are: determine the positions of the visible eye center and mouth center in the input image through the facial feature points; calculate the transformation matrix T between the input image and the aligned image points, and the calculation formula of the transformation matrix T is:

[0057] set up is the horizontal center of the aligned image, w is the horizontal width of the aligned image, is the head pose angle. The position in the input and aligned images is calculated as follows:

[0058] S1=S mouth ,

[0059]

[0060] Among them, S1, S mouth is the coordinate of the center of the mouth in the original image, t1 is the coordinate of the center of the mouth in the aligned image, S2 is the coordinate of the center of the eye in the input image, t2 is the coordinate of the center of the eye in the aligned image, S l.eye is the coordinate of the left eye center of the input image, S r.eye is the coordinate of the right eye center of the input image, x center To align the horizontal center of the image, y eyes To align the horizontal center of the eye of the image, d xmouth To align the horizontal width of the mouth in the image, y mouth is the ordinate of the center of the mouth in the aligned image. When one eye is invisible, the coordinate of the visible eye is S v.eye To indicate that S mouth represents the coordinates of the mouth center. In the case where only one eye of the subject is visible, the visible eye coordinates S are used v.eye Alternative That is, when the subject has only one eye visible, S2 = S v.eye The transformation matrix T is obtained by solving the linear equations given by the two point correspondences. This alignment method is suitable for face alignment under a wide range of head posture changes.

[0061] S102, extracting the feature vector of each sample in the data sample set, and the feature vectors of all samples form a feature matrix.

[0062] Furthermore, the feature extraction of the sample preferably adopts one of the following three methods.

[0063] (1) Feature extraction method using LBP operator.

[0064] LBP is an operator used to describe the local texture features of an image. It has significant advantages such as rotation invariance and grayscale invariance. The calculation of the LBP operator is as follows: in a 3*3 window, the grayscale value of the center pixel of the window is used as the threshold, and then the grayscale values ​​of the eight surrounding pixels are compared with it. If it is greater than the grayscale value of the center pixel, it is marked as 1, otherwise it is 0. In this way, after comparison, the 8 pixels in the 3*3 neighborhood can generate an 8-bit binary number. It is usually converted into a decimal number, namely the LBP code, which has a total of 256 types. Finally, the LBP value of the center pixel of the window can be obtained, and this value is used to reflect the texture information of the area. It can be expressed in formula:

[0065]

[0066] Where I(p) represents the gray value of the pth pixel in the window except the central pixel, I(c) represents the gray value of the central pixel, and s(·) is the threshold function, which is expressed as follows:

[0067]

[0068] Since LBP records the difference between the neighborhood pixels and the central pixel, when the illumination changes cause the grayscale values ​​of the pixels in the window to increase or decrease synchronously, the LBP value does not change significantly, so LBP is not sensitive to illumination changes.

[0069] (2) A deep feature extraction method that uses a public and trained face model to directly extract features of all data samples.

[0070] In this feature extraction method of the present embodiment, the ResNet18 network model is used to read the network parameters of the pre-trained face model Ms-Celeb-1M, all data samples are directly input into the entire network, and the 512-dimensional vector before the last fully connected layer in the entire network is output as the deep feature of the data sample in this embodiment.

[0071] (3) Use labeled samples to fine-tune the pre-trained face model Ms-Celeb-1M to obtain a deep model. Finally, use the obtained deep model to extract deep embedding features of all data samples.

[0072] In this feature extraction method of this embodiment, the ResNet18 network model is also used to read the network parameters of the pre-trained face model Ms-Celeb-1M, and all labeled samples are used as the training set data of the ResNet18 model. The parameters of the entire network are fine-tuned to obtain a fine-tuned deep model relative to the data samples used in this embodiment. Then all data samples are directly input into the fine-tuned deep model of this feature extraction method, and the 512-dimensional vector before the last fully connected layer in the entire network is output as the deep embedding feature of the current data sample.

[0073] The features obtained by the above three feature extraction methods can all be used as the feature vectors of the samples in this embodiment.

[0074] S103, for each sample, calculate the distance between it and other samples, take the multiple samples closest to it as its first neighbor samples, and construct a first adjacency matrix based on whether there is a first neighbor relationship between every two samples; for each sample, calculate the distance between it and other peak expression samples, take the multiple samples closest to it as its second neighbor samples, and construct a second adjacency matrix based on whether there is a second neighbor relationship between every two samples.

[0075] The adjacency matrix uses the value of the number in the i-th row and j-th column in the matrix to represent the connection relationship between the i-th node and the j-th node in the graph. If it is 1, it means there is a connection, and if it is 0, there is no connection.

[0076] By constructing the first adjacency matrix, we can make full use of the local information between the sample and its similar samples, that is, the information of other samples adjacent to the target sample in the feature space. By constructing the second adjacency matrix, we can make full use of the peak information between non-peak samples and peak samples, that is, the peak expression information adjacent to the target sample in the feature space.

[0077] Furthermore, the distance between the calculated sample and other samples, and the distance between the calculated sample and other peak expression samples are preferably calculated using Euclidean distance, but other distance calculation formulas that represent differences may also be used to achieve this.

[0078] The preferred implementation method for constructing the first adjacency matrix and the second adjacency matrix is ​​described in detail below.

[0079] (1) Construct the first adjacency matrix.

[0080] The principle of constructing the first adjacency matrix is ​​as follows Figure 2 As shown, for the image samples that have undergone data preprocessing, a local preservation projection adjacency matrix between the image samples is constructed based on the Euclidean distance between the feature vectors.

[0081] Take any image sample xi , calculate the Euclidean distance between it and all other image samples, and take the nearest k image samples as the first neighbor samples. In one embodiment, k = 20. j is the sample x i The first neighbor of x j ∈N(x i , k). Then the sample x j With sample x i The connection relationship (first neighbor relationship) between the two samples is represented by 1, and if there is no first neighbor relationship between the two samples, it is represented by 0. Finally, the first adjacency matrix is ​​obtained That is to say, the local projection adjacency matrix is ​​maintained, and the formula is expressed as follows:

[0082]

[0083] Let the first adjacency matrix be A L , A L The value of the jth column in the i-th row.

[0084] (2) Construct the second adjacency matrix.

[0085] The principle of constructing the second adjacency matrix is ​​as follows Figure 3 As shown, for the image samples that have undergone data preprocessing, a second adjacency matrix between the image samples is constructed according to the Euclidean distance between the feature vectors.

[0086] Take any image sample x i , calculate the Euclidean distance between it and all other peak expression samples, and take the nearest k peak expression samples as the second neighbor samples. In one embodiment, k = 20. j is the sample x i The second neighbor of x j ∈N(x i , k), and sample x j is the peak expression, that is, x j ∈X p . Then the sample x j With sample x i The connection relationship (second neighbor relationship) between the two samples is represented by 1, and if there is no second neighbor relationship between the two samples, it is represented by 0. Finally, the second adjacency matrix is ​​obtained That is, the second adjacency matrix, the formula is expressed as follows:

[0087]

[0088] Let the first adjacency matrix be A P , A PThe value of the jth column in the i-th row.

[0089] S104, inputting the feature matrix, the first adjacency matrix, and the second adjacency matrix into a graph convolutional neural network for training.

[0090] Construct a graph neural network recognition model. Input the image samples and adjacency matrix into the pre-built recognition model for training, and minimize the cross entropy loss between the true labels of the training samples and the labels predicted by the training sample model during training. Since the input data of the graph convolutional neural network includes the first adjacency matrix and the second adjacency matrix in addition to the feature matrix, it can make full use of the local information between expressions with similar intensities and the peak-guided information between peak expressions and non-peak expressions, and can effectively improve the classification accuracy of expressions with varying intensity.

[0091] The structure of the graph neural network recognition model can be implemented in many different ways, such as the subspace fusion model of the weighted adjacency matrix or the subspace fusion model of the attention mechanism.

[0092] In one embodiment, the graph convolutional neural network includes a fusion module, a graph convolution module and a classification module, and the input to the graph convolutional neural network for training includes the steps of: inputting the first adjacency matrix and the second adjacency matrix into the fusion module for weighted fusion to obtain a weighted adjacency matrix; inputting the feature matrix and the weighted adjacency matrix into the graph convolution module; and inputting the output data of the graph convolution module into the classification module.

[0093] The weighted adjacency matrix construction module is used to fuse the local information between samples with the peak-guide information between peak expressions and non-peak expressions in the form of an adjacency matrix. The module is sent to the graph convolutional neural network module to aggregate the local information with the peak-guide information, including the following steps:

[0094] (1) Constructing a weighted adjacency matrix

[0095] Figure 4 This is the network structure diagram of the subspace fusion module of the weighted adjacency matrix. The entire model first fuses the first adjacency matrix with the second adjacency matrix. The fusion is performed in the form of matrix addition. At the same time, each of the two adjacency matrices has an empirically set weight parameter to adjust and find the best fusion ratio. The specific formula is as follows:

[0096] A T =γ1A L +γ2A P

[0097] The parameters γ1 and γ2 are the weights of the first adjacency matrix and the second adjacency matrix respectively, and their value sets are {0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0}. In one embodiment, γ1=1, γ2=0.1.

[0098] The obtained weighted adjacency matrix can simultaneously contain the local information between samples with similar intensities and the peak-guided information between non-peak expressions and peak expressions. And thanks to the flexible values ​​of the two parameters γ1 and γ2, the weighted adjacency matrix can balance the proportion between local information and peak-guided information.

[0099] (2) The feature matrix and weighted adjacency matrix of the sample data are sent to the graph convolution module and the classification module.

[0100] The graph convolution module includes two layers of convolution operations.

[0101]

[0102] The input of the graph convolution module includes a feature matrix consisting of the feature vector of each data sample and an adjacency matrix that stores the adjacency relationship. (0) =X T It is the feature matrix input into the graph convolution module. If the feature representation of each node is x i , the number of nodes is n, the feature dimension is 512, then X T It is an n*512 matrix. (h) is the output of the hth layer in the neural network. (h) is the trainable weight matrix of the hth layer in the neural network. It is the normalized form of the adjacency matrix A input into the graph convolution module, and its specific calculation formula is as follows:

[0103]

[0104] Among them I n is the n-dimensional identity matrix. yes The degree matrix is ​​a diagonal matrix, and the elements on the diagonal are the degrees of each vertex, where for The value of the i-th row and i-th column in for The value of the i-th row and j-th column in . And σ is a nonlinear activation function, such as the ReLU function commonly used in neural networks, σ(x) = ReLU(x).

[0105] Since the graph convolutional neural network is trained in a semi-supervised manner, the training dataset is redefined as in are labeled samples in the dataset, and are the unlabeled samples in the dataset.

[0106] It is particularly noted that in the subspace fusion model of the weighted adjacency matrix, after the feature matrix composed of all sample data passes through the first layer of graph convolution, the dimension of the embedded feature matrix obtained is n*16, and after passing through the second layer of graph convolution, the dimension of the embedded feature matrix obtained is n*6. After passing through a soft-max layer, the category probability distribution of all samples is obtained.

[0107] It is particularly noted that in the present invention, the graph convolution modules used all have two graph convolution layers, and the activation functions used are all ReLU functions.

[0108] In another embodiment, a graph convolutional neural network includes a first graph convolution module, a second graph convolution module, an attention module and a classification module, and inputting into the graph convolutional neural network for training includes the steps of: inputting a feature matrix and a first adjacency matrix into the first graph convolution module; inputting a feature matrix and a second adjacency matrix into the second graph convolution module; inputting output data of the first graph convolution module and the second graph convolution module into the attention module; and inputting output data of the attention module into the classification module.

[0109] The attention module fuses the output features of the graph convolutional neural network module using local information and the graph convolutional neural network module using peak guide information, and fuses the local information between samples with the peak guide information between peak expressions and non-peak expressions at the feature level, including the following steps:

[0110] (1) The feature matrix of the data sample is sent to two graph convolution modules together with the first adjacency matrix and the second adjacency matrix respectively.

[0111] After these two graph convolution modules aggregate the local information between samples with similar intensities and the peak guide information between peak expressions and non-peak expressions, for a single sample, the dimension of the input into the graph convolution module is 512. After passing through the first graph convolution layer, the feature dimension is still 512. When it is sent to the second graph convolution layer, the output embedded feature dimension is 128.

[0112] (2) Using the attention mechanism

[0113] like Figure 6 As shown in the figure, the n*128 embedding feature matrix output by the two graph convolution modules is input into the attention module for fusion. The fused feature obtained is still 128-dimensional and can be expressed as:

[0114] f c =α1f1+α2f2

[0115] Among them, α1 and α2 are the self-attention weights, f c is the fused vector, f1 and f2 are the data vectors of the two graph convolution modules respectively, which can be learned by a soft-max activated feedforward neural network defined by the following formula.

[0116]

[0117] Where w is the trainable weight value in the feedforward neural network, i refers to the i-th dimension in the feature vector, and σ is a ReLU function. Finally, the fused features are input into a fully connected layer that can reduce 128 dimensions to 6 dimensions for expression classification.

[0118] The graph convolutional neural network is further constrained with cross entropy loss to optimize the trainable network parameters in the model. The cross entropy loss can be expressed as follows:

[0119]

[0120] Where n is the number of unlabeled data samples, <·,·> represents the inner product operation, and Y is the true label distribution of the samples in One-hot form. is the probability distribution of the model's prediction of the sample category, and log() is a logarithmic function with e as the base. In the process of minimizing the cross entropy loss L, the trainable parameters in the model can be optimized, thereby improving the performance of the model.

[0121] In one embodiment, the Oulu-CASIA expression library is used. The expression library collects 6 basic expressions (anger, disgust, fear, happiness, sadness and surprise) of 80 subjects under 3 different lighting conditions. This experiment recorded a total of 480 expression sequences under normal lighting conditions. Each sequence starts with a neutral expression and ends with a peak expression.

[0122] The present invention selects all subjects, and each subject has 6 expression sequences. However, only the 7th picture to the last picture in each sequence are selected as data input for the network model. The 7th, 8th, and 9th picture samples of each sequence are used as non-peak expressions, and the last 3 picture samples are used as peak expressions. Finally, a total of 7499 facial expression images are used as input. Using ten-fold cross validation, the samples are divided into 10 parts on an average basis based on the subjects. Each time, all facial expression picture samples of 72 subjects are selected as labeled data, and all facial expression picture samples of the remaining 8 subjects are selected as unlabeled data, and finally the recognition accuracy of all unlabeled data is calculated.

[0123] On the Oulu-CASIA dataset, the recognition accuracies of non-peak expressions, peak expressions and expressions of all intensities using ten-fold cross validation were 76.66%, 91.01% and 85.27% respectively under the subspace fusion model of the weighted adjacency matrix, and were 76.69%, 91.68% and 85.53% respectively under the subspace fusion model of the attention mechanism, which are better than the recognition results of other existing deep learning methods, indicating that the method provided by the present invention can effectively suppress the interference of intensity changes on facial expression recognition.

[0124] At the same time, in order to verify the recognition performance of the present invention with less labeled data, the present invention also adopts a five-fold cross validation, dividing the samples into five equal parts based on the subjects, each time selecting all facial expression picture samples of 16 subjects as labeled data, and all facial expression picture samples of the remaining 64 subjects as unlabeled data, and finally calculating the recognition accuracy of all unlabeled data.

[0125] In the Oulu-CASIA dataset, the recognition accuracies of non-peak expressions, peak expressions and expressions of all intensities using five-fold cross validation were 65.93%, 80.70% and 74.41% respectively under the subspace fusion model of the weighted adjacency matrix, and were 66.06%, 81.14% and 74.96% respectively under the subspace fusion model of the attention mechanism. The recognition results of the present invention with less labeled data are also higher than those of some existing deep learning methods that use more labeled data, indicating that the method provided by the present invention can achieve good results with only less labeled data.

[0126] A facial expression recognition system for expression intensity changes according to an embodiment of the present invention comprises an acquisition module, a feature extraction module, an adjacency matrix construction module and a training module.

[0127] An acquisition module is used to acquire a data sample set, wherein the data sample set includes a peak expression sample and a non-peak expression sample;

[0128] Feature extraction module, used to extract the feature vector of each sample in the data sample set, and the feature vectors of all samples form a feature matrix;

[0129] The adjacency matrix construction module is used to calculate the distance between each sample and other samples, take multiple samples with the closest distance to it as its first neighbor samples, and construct a first adjacency matrix according to whether there is a first neighbor relationship between every two samples; for each sample, calculate the distance between it and other peak expression samples, take multiple samples with the closest distance to it as its second neighbor samples, and construct a second adjacency matrix according to whether there is a second neighbor relationship between every two samples;

[0130] The training module is used to input the feature matrix, the first adjacency matrix and the second adjacency matrix into the graph convolutional neural network for training.

[0131] Furthermore, constructing the first adjacency matrix includes:

[0132] Let the i-th sample be x i , record the value of the i-th row and j-th column in the first adjacency matrix as If the sample x j is the sample x i The first neighbor sample of otherwise

[0133] Furthermore, forming a second adjacency matrix according to whether there is a second neighbor relationship between two samples includes:

[0134] Let the i-th sample be x i , record the value of the i-th row and j-th column in the second adjacency matrix as If the sample x j is the sample x i The second neighbor sample of otherwise

[0135] Furthermore, the graph convolutional neural network includes a fusion module, a graph convolution module and a classification module, and inputting into the graph convolutional neural network for training includes the following steps:

[0136] Inputting the first adjacency matrix and the second adjacency matrix into a fusion module for weighted fusion to obtain a weighted adjacency matrix;

[0137] Input the feature matrix and weighted adjacency matrix into the graph convolution module;

[0138] The output data of the graph convolution module is input into the classification module.

[0139] Furthermore, the graph convolutional neural network includes a first graph convolutional module, a second graph convolutional module, an attention module and a classification module, and inputting into the graph convolutional neural network for training includes the steps of:

[0140] Input the feature matrix and the first adjacency matrix into the first graph convolution module;

[0141] Input the feature matrix and the second adjacency matrix into the second graph convolution module;

[0142] Input the output data of the first graph convolution module and the second graph convolution module into the attention module;

[0143] The output data of the attention module is input into the classification module.

[0144] The implementation principle and technical effects of the system are similar to the above method and will not be repeated here.

[0145] The embodiment of the present invention also provides a storage medium on which a computer program is stored, and the computer program is executed by a processor to implement the technical solution of any of the above-mentioned facial expression recognition method embodiments with varying expression intensity. Its implementation principle and technical effect are similar to those of the above-mentioned method, and will not be repeated here.

[0146] It must be noted that in any of the above embodiments, the method is not necessarily executed in sequence according to the sequence number. As long as it cannot be inferred from the execution logic that it must be executed in a certain order, it means that it can be executed in any other possible order.

[0147] It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for recognizing facial expressions with varying expression intensity, characterized in that: include: Acquire a data sample set, wherein the data sample set includes a peak expression sample and a non-peak expression sample; Extracting a feature vector of each sample in the data sample set, and the feature vectors of all samples form a feature matrix; For each sample, calculate the distance between it and other samples, take multiple samples closest to it as its first neighbor samples, and construct the first adjacency matrix based on whether there is a first neighbor relationship between every two samples; For each sample, calculate the distance between it and other peak expression samples, take multiple samples with the closest distance to it as its second neighbor samples, and construct a second adjacency matrix based on whether there is a second neighbor relationship between every two samples; Inputting the feature matrix, the first adjacency matrix and the second adjacency matrix into a graph convolutional neural network for training; The graph convolutional neural network includes a fusion module, a graph convolution module and a classification module. The input to the graph convolutional neural network for training includes the steps of inputting the first adjacency matrix and the second adjacency matrix into the fusion module for weighted fusion to obtain a weighted adjacency matrix; inputting the feature matrix and the weighted adjacency matrix into the graph convolution module; and inputting the output data of the graph convolution module into the classification module; The graph convolutional neural network includes a first graph convolution module, a second graph convolution module, an attention module and a classification module, and the input to the graph convolutional neural network for training includes the steps of inputting the feature matrix and the first adjacency matrix into the first graph convolution module; inputting the feature matrix and the second adjacency matrix into the second graph convolution module; inputting the output data of the first graph convolution module and the second graph convolution module into the attention module; Inputting the output data of the attention module into the classification module; The sample is preprocessed before extracting the feature vector. The preprocessing includes determining the positions of the eye center and the mouth center visible in the input image through the facial feature points; calculating the transformation matrix T between the input image and the aligned image points. The calculation formula of the transformation matrix T is: set up To align the horizontal center of the image, w To align the horizontal width of the image, is the head posture angle, ， ， in, S 1. S mouth are the coordinates of the center of the mouth in the original image, t 1 is the coordinate of the center of the mouth in the alignment image, S 2 is the coordinate of the eye center in the input image, t 2 is the coordinate of the eye center in the alignment image, is the coordinate of the left eye center of the input image, is the coordinate of the right eye center of the input image, x center To align the horizontal center of the image, y eyes To align the horizontal center of the eye of the image, d x mouth To align the horizontal width of the mouth in the image, y mouth To align the vertical coordinate of the mouth center in the image, when one eye is invisible, the coordinates of the visible eye are used To express, represents the coordinates of the center of the mouth, when only one eye is visible in the input image. .

2. A facial expression recognition method based on expression intensity changes as claimed in claim 1, characterized in that: The constructing of the first adjacency matrix comprises: Let the i-th sample be , record the value of the i-th row and j-th column in the first adjacency matrix as , if the sample It is a sample The first neighbor sample of =1, otherwise =0.

3. A facial expression recognition method based on expression intensity changes as claimed in claim 1, characterized in that: The constructing the second adjacency matrix comprises: Let the i-th sample be , record the value of the i-th row and j-th column in the second adjacency matrix as , if the sample It is a sample The second neighbor sample of =1, otherwise =0.

4. A facial expression recognition method based on expression intensity changes as claimed in claim 1, characterized in that: The first adjacency matrix contains local information between sample-similar samples, and the second adjacency matrix contains peak guide information between non-peak samples and peak samples, and weighted weight parameters of the first adjacency matrix and the second adjacency matrix are adjusted.

5. A facial expression recognition method based on expression intensity changes as claimed in claim 1, characterized in that: Calculating the distance between it and other samples and calculating the distance between it and other peak expression samples are both calculating the Euclidean distance.

6. A facial expression recognition system with expression intensity variation, characterized in that: include: An acquisition module, used for acquiring a data sample set, wherein the data sample set includes a peak expression sample and a non-peak expression sample; A feature extraction module is used to extract a feature vector of each sample in the data sample set, and the feature vectors of all samples form a feature matrix; The adjacency matrix construction module is used to calculate the distance between each sample and other samples, take multiple samples with the closest distance to it as its first neighbor samples, and construct the first adjacency matrix according to whether there is a first neighbor relationship between each two samples; For each sample, calculate the distance between it and other peak expression samples, take multiple samples with the closest distance to it as its second neighbor samples, and construct a second adjacency matrix based on whether there is a second neighbor relationship between every two samples; A training module, used for inputting the feature matrix, the first adjacency matrix and the second adjacency matrix into a graph convolutional neural network for training; The graph convolution neural network includes a fusion module, a graph convolution module and a classification module. The training module is further used to input the first adjacency matrix and the second adjacency matrix into the fusion module for weighted fusion to obtain a weighted adjacency matrix; input the feature matrix and the weighted adjacency matrix into the graph convolution module; and input the output data of the graph convolution module into the classification module; The graph convolution neural network includes a first graph convolution module, a second graph convolution module, an attention module and a classification module, and the training module is further used to input the feature matrix and the first adjacency matrix into the first graph convolution module; input the feature matrix and the second adjacency matrix into the second graph convolution module; input the output data of the first graph convolution module and the second graph convolution module into the attention module; Inputting the output data of the attention module into the classification module; The preprocessing module is used to preprocess the samples before extracting the feature vectors. It determines the positions of the eye centers and mouth centers visible in the input image through the facial feature points; and calculates the transformation matrix T between the input image and the aligned image points. The calculation formula of the transformation matrix T is: set up To align the horizontal center of the image, w To align the horizontal width of the image, is the head posture angle, ， ， in, S 1. S mouth are the coordinates of the center of the mouth in the original image, t 1 is the coordinate of the center of the mouth in the alignment image, S 2 is the coordinate of the eye center in the input image, t 2 is the coordinate of the eye center in the alignment image, is the coordinate of the left eye center of the input image, is the coordinate of the right eye center of the input image, x center To align the horizontal center of the image, y eyes To align the horizontal center of the eye of the image, d x mouth To align the horizontal width of the mouth in the image, y mouth To align the vertical coordinate of the mouth center in the image, when one eye is invisible, the coordinates of the visible eye are used To express, represents the coordinates of the center of the mouth, when only one eye is visible in the input image. .

7. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images