A method and device for facial expression recognition

Abstract

A method and device for expression recognition, wherein the method comprises: obtaining a three-dimensional image of a target face, the three-dimensional image comprising third depth information of the target face and fifth color information of the target face; inputting the third depth information into a second neural network and inputting the fifth color information into a third neural network; the second neural network classifies the expression of the target face according to the third depth information and a second parameter and outputs first classification data, the third neural network classifies the expression of the target face according to the fifth color information and a third parameter and outputs second classification data, the second parameter comprising at least one facial expression category and second parameter data for identifying the expression category of the target face, the third parameter comprising at least one facial expression category and third parameter data for identifying the expression category of the target face; and outputting a classification result of the expression of the target face according to the first classification data and the second classification data.

Description

[0001] This application is a divisional application of application "Application No. 201710614130.8, application date 2017.07.25, invention name is A method and device for expression recognition". Technical Field

[0002] The present invention relates to an image processing method, and in particular to a method and a device for expression recognition. Background Art

[0003] As artificial intelligence technology is changing with each passing day, deep learning has brought new hope to technology, but it has also opened up technical bottlenecks. Expressions can be said to be a universal language, regardless of race or nationality. In human-computer interaction technology, expression recognition plays a very important role. For example, when taking care of the elderly and children, robots can judge whether what they have just done is in line with their wishes through their facial expressions, so that they can understand the living habits and personalities of children and the elderly more quickly.

[0004] In the prior art, facial expression recognition algorithms generally use two-dimensional image feature extraction and classification algorithms to classify expressions to obtain expression results. When the face appears at a certain angle or when there are poor lighting conditions, such as very dim or very strong light, the feature information extracted through the two-dimensional image features is very different and may be wrong, which will cause the algorithm to misjudge the expression. Summary of the invention

[0005] The present invention provides a method and device for facial expression recognition, which can effectively solve the problem of decreased accuracy of facial expression recognition caused by different facial postures and different lighting conditions.

[0006] According to a first aspect of the present invention, there is provided a method for facial expression recognition, the method comprising:

[0007] Acquire a three-dimensional image of a target face and a two-dimensional image of the target face, wherein the three-dimensional image includes first depth information of the target face and first color information of the target face, and the two-dimensional image includes second color information of the target face;

[0008] Inputting first depth information of the target face, first color information of the target face, and second color information of the target face into a first neural network;

[0009] The first neural network classifies the expression of the target face based on first depth information of the target face, first color information of the target face, second color information of the target face, and a first parameter, wherein the first parameter includes at least one facial expression category and first parameter data for identifying the expression category of the target face.

[0010] According to the first aspect of the present invention, in a first executable manner of the first aspect of the present invention, before inputting the first depth information of the target face, the first color information of the target face, and the second color information of the target face into the first neural network, the method further includes:

[0011] Performing the same first processing on the three-dimensional image of the target face and the two-dimensional image of the target face, wherein the first processing includes at least one of the following:

[0012] Determine feature points of the three-dimensional image of the target face and the two-dimensional image of the target face, and rotate the three-dimensional image of the target face and the two-dimensional image of the target face based on the feature points;

[0013] Performing mirror processing, linear transformation, and affine transformation on the three-dimensional image of the target face and the two-dimensional image of the target face;

[0014] Aligning feature points of the three-dimensional image of the target face and the two-dimensional image of the target face at a set position;

[0015] Performing grayscale stretching on the three-dimensional image of the target face and the two-dimensional image of the target face; and,

[0016] Image pixel value normalization processing is performed on the three-dimensional image of the target face and the two-dimensional image of the target face.

[0017] According to the first executable mode of the first aspect of the present invention, in the second executable mode of the first aspect of the present invention, the performing image pixel value normalization processing on the three-dimensional image of the target face and the two-dimensional image of the target face includes:

[0018] The pixel values ​​of each channel of the three-dimensional image of the target face and the two-dimensional image of the target face are normalized from [0, 255] to [0, 1].

[0019] According to the first aspect of the present invention, the first executable mode of the first aspect of the present invention, or the second executable mode, in a third executable mode of the first aspect of the present invention, the first parameter data for identifying the expression category of the target face is obtained by training the first neural network from three-dimensional images of a plurality of facial expression samples and two-dimensional images of the facial expression samples;

[0020] The three-dimensional image of the facial expression sample includes: second depth information of the facial expression sample and third color information of the facial expression sample;

[0021] The two-dimensional image of the facial expression sample includes: fourth color information of the facial expression sample.

[0022] According to the third executable mode of the first aspect of the present invention, in the fourth executable mode of the first aspect of the present invention, before the three-dimensional images of the plurality of facial expression samples and the two-dimensional images of the facial expression samples are trained by the first neural network, the method further comprises:

[0023] Performing the same second processing on the three-dimensional image of the facial expression sample and the two-dimensional image of the facial expression sample, wherein the second processing includes at least one of the following:

[0024] Determine feature points of the three-dimensional image of the facial expression sample and the two-dimensional image of the facial expression sample, and rotate the three-dimensional image of the facial expression sample and the two-dimensional image of the facial expression sample based on the feature points;

[0025] Performing mirror processing, linear transformation and affine transformation on the three-dimensional image of the facial expression sample and the two-dimensional image of the facial expression sample;

[0026] Aligning feature points of the three-dimensional image of the facial expression sample and the two-dimensional image of the facial expression sample to the set position;

[0027] Performing grayscale stretching on the three-dimensional image of the facial expression sample and the two-dimensional image of the facial expression sample; and,

[0028] Image pixel value normalization processing is performed on the three-dimensional image of the facial expression sample and the two-dimensional image of the facial expression sample.

[0029] According to the fourth executable mode of the first aspect of the present invention, in the fifth executable mode of the first aspect of the present invention, the performing image pixel value normalization processing on the three-dimensional image of the facial expression sample and the two-dimensional image of the facial expression sample comprises:

[0030] The pixel values ​​of each channel of the three-dimensional image of the facial expression sample and the two-dimensional image of the facial expression sample are normalized from [0, 255] to [0, 1].

[0031] According to the fourth or fifth executable mode of the first aspect of the present invention, in the sixth executable mode of the first aspect of the present invention, the facial expression sample has at least one of the following facial expression categories: fear, sadness, happiness, anger, disgust, surprise, naturalness, and contempt;

[0032] The facial expression sample, the second depth information of the facial expression sample, the third color information of the facial expression sample, and the fourth color information of the facial expression sample have the same facial expression category.

[0033] According to the first aspect of the present invention and any one of the first to sixth executable modes of the first aspect of the present invention, in the seventh executable mode of the first aspect of the present invention, the facial expression categories included in the first neural network include at least one of the following: fear, sadness, happiness, anger, disgust, surprise, naturalness, and contempt.

[0034] According to any one of the first to seventh executable modes of the first aspect of the present invention, in an eighth executable mode of the first aspect of the present invention, the feature point is an eye point.

[0035] According to the first aspect of the present invention and any one of the first to eighth executable modes of the first aspect of the present invention, in the ninth executable mode of the first aspect of the present invention, the first neural network includes a first convolutional neural network.

[0036] According to the ninth executable mode of the first aspect of the present invention, in the tenth executable mode of the first aspect of the present invention, the first convolutional neural network includes 4 convolutional layers, 4 downsampling layers, 1 downsampling layer and 2 fully connected layers.

[0037] According to the first aspect of the present invention and any one of the first to tenth executable modes of the first aspect of the present invention, in the eleventh executable mode of the first aspect of the present invention, the first color information and the second color information are images in RGB format or YUV format.

[0038] According to any one of the third executable mode to the eleventh executable mode of the first aspect of the present invention, in the twelfth executable mode of the first aspect of the present invention, the third color information and the fourth color information are images in RGB format or YUV format.

[0039] According to a second aspect of the present invention, there is provided a device for facial expression recognition, the device comprising:

[0040] a first acquisition module, configured to acquire a three-dimensional image of a target face and a two-dimensional image of the target face, wherein the three-dimensional image includes first depth information of the target face and first color information of the target face, and the two-dimensional image includes second color information of the target face;

[0041] A first input module, configured to input first depth information of the target face, first color information of the target face, and second color information of the target face into a first neural network;

[0042] A first neural network is used to classify the expression of the target face based on first depth information of the target face, first color information of the target face, second color information of the target face, and a first parameter, wherein the first parameter includes at least one facial expression category and first parameter data for identifying the expression category of the target face.

[0043] According to the second aspect of the present invention, in a first executable manner of the second aspect of the present invention, the device further includes a first processing module,

[0044] The first processing module is used to perform the same first processing on the three-dimensional image of the target face and the two-dimensional image of the target face, and input the three-dimensional image of the target face and the two-dimensional image of the target face after the first processing into the first input module;

[0045] The first processing module includes at least one of the following submodules: a first rotation submodule, a first transformation submodule, a first alignment submodule, a first grayscale stretching submodule, and a first normalization processing submodule;

[0046] The first rotation submodule is used to determine feature points of the three-dimensional image of the target face and the two-dimensional image of the target face, and rotate the three-dimensional image of the target face and the two-dimensional image of the target face based on the feature points;

[0047] The first transformation submodule is used to perform mirror processing, linear transformation and affine transformation on the three-dimensional image of the target face and the two-dimensional image of the target face;

[0048] The first alignment submodule is used to align feature points of the three-dimensional image of the target face and the two-dimensional image of the target face at a set position;

[0049] The first grayscale stretching submodule is used to perform grayscale stretching on the three-dimensional image of the target face and the two-dimensional image of the target face; and

[0050] The first normalization processing submodule is used to perform image pixel value normalization processing on the three-dimensional image of the target face and the two-dimensional image of the target face.

[0051] According to a first executable mode of the second aspect of the present invention, in a second executable mode of the second aspect of the present invention,

[0052] The first normalization processing submodule is specifically used to normalize the pixel values ​​of each channel of the three-dimensional image of the target face and the two-dimensional image of the target face from [0, 255] to [0, 1].

[0053] According to the second aspect of the present invention, and the first executable mode or the second executable mode of the second aspect of the present invention, in a third executable mode of the second aspect of the present invention,

[0054] The first parameter data for identifying the expression category of the target face is obtained by training the first neural network from three-dimensional images of a plurality of facial expression samples and two-dimensional images of the facial expression samples;

[0055] The three-dimensional image of the facial expression sample includes: second depth information of the facial expression sample and third color information of the facial expression sample;

[0056] The two-dimensional image of the facial expression sample includes: fourth color information of the facial expression sample.

[0057] According to the third executable mode of the second aspect of the present invention, in the fourth executable mode of the second aspect of the present invention, the device further includes a second processing module,

[0058] The second processing module is used to perform the same second processing on the three-dimensional image of the facial expression sample and the two-dimensional image of the facial expression sample, and input the three-dimensional image of the facial expression sample and the two-dimensional image of the facial expression sample after the second processing into the first input module;

[0059] The second processing module includes a second rotation submodule, a second transformation submodule, a second alignment submodule, a second grayscale stretching submodule, and a second normalization processing submodule;

[0060] The second rotation submodule is used to determine the feature points of the three-dimensional image of the facial expression sample and the two-dimensional image of the facial expression sample, and rotate the three-dimensional image of the facial expression sample and the two-dimensional image of the facial expression sample based on the feature points;

[0061] The second transformation submodule is used to perform mirror processing, linear transformation and affine transformation on the three-dimensional image of the facial expression sample and the two-dimensional image of the facial expression sample;

[0062] The second alignment submodule is used to align feature points of the three-dimensional image of the facial expression sample and the two-dimensional image of the facial expression sample to a set position;

[0063] The second grayscale stretching submodule is used to perform grayscale stretching on the three-dimensional image of the facial expression sample and the two-dimensional image of the facial expression sample; and

[0064] The second normalization processing submodule is used to perform image pixel value normalization processing on the three-dimensional image of the facial expression sample and the two-dimensional image of the facial expression sample.

[0065] According to a fourth executable mode of the second aspect of the present invention, in a fifth executable mode of the second aspect of the present invention,

[0066] The second normalization processing submodule is specifically used to normalize the pixel values ​​of each channel of the three-dimensional image of the facial expression sample and the two-dimensional image of the facial expression sample from [0, 255] to [0, 1].

[0067] According to any one of the third to fifth executable modes of the second aspect of the present invention, in a sixth executable mode of the second aspect of the present invention,

[0068] The facial expression sample has at least one of the following facial expression categories: fear, sadness, happiness, anger, disgust, surprise, neutral, and contempt;

[0069] The facial expression sample, the second depth information of the facial expression sample, the second color information of the facial expression sample, and the third color information of the facial expression sample have the same facial expression category.

[0070] According to the second aspect of the present invention and any one of the first to sixth executable modes of the second aspect of the present invention, in a seventh executable mode of the second aspect of the present invention,

[0071] The facial expression categories included in the first neural network include at least one of the following: fear, sadness, happiness, anger, disgust, surprise, neutral, and contempt.

[0072] According to the second aspect of the present invention and any one of the first to seventh executable modes of the second aspect of the present invention, in an eighth executable mode of the second aspect of the present invention, the feature point is an eye point.

[0073] According to the second aspect of the present invention and any one of the first to eighth executable modes of the second aspect of the present invention, in the ninth executable mode of the second aspect of the present invention, the first neural network includes a first convolutional neural network.

[0074] According to the ninth executable mode of the second aspect of the present invention, in the tenth executable mode of the second aspect of the present invention, the first convolutional neural network includes 4 convolutional layers, 4 downsampling layers, 1 downsampling layer and 2 fully connected layers.

[0075] According to the second aspect of the present invention and any one of the first to tenth executable modes of the second aspect of the present invention, in the eleventh executable mode of the second aspect of the present invention,

[0076] The first color information and the second color information are images in RGB format or YUV format.

[0077] According to any one of the third to eleventh executable modes of the second aspect of the present invention, in the twelfth executable mode of the second aspect of the present invention,

[0078] The third color information and the fourth color information are images in RGB format or YUV format.

[0079] According to a third aspect of the present invention, there is provided a method for facial expression recognition, the method comprising:

[0080] Acquire a three-dimensional image of a target face, wherein the three-dimensional image includes third depth information of the target face and fifth color information of the target face;

[0081] Inputting third depth information of the target face into the second neural network and inputting fifth color information of the target face into the third neural network;

[0082] The second neural network classifies the expression of the target face according to the third depth information of the target face and the second parameter and outputs first classification data, and the third neural network classifies the expression of the target face according to the fifth color information of the target face and the third parameter and outputs second classification data, the second parameter includes at least one facial expression category and second parameter data for identifying the expression category of the target face, and the third parameter includes at least one facial expression category and third parameter data for identifying the expression category of the target face;

[0083] A classification result of the expression of the target face is output according to the first classification data and the second classification data.

[0084] According to the third aspect of the present invention, in a first executable mode of the third aspect of the present invention,

[0085] The step of outputting a classification result of the target facial expression according to the first classification data and the second classification data comprises:

[0086] The support vector machine inputs the first classification data and the second classification data, and outputs a classification result of the expression of the target face according to the first classification data, the second classification data and the support vector machine parameter data, wherein the support vector machine includes the at least one facial expression category and the support vector machine parameter data for identifying the expression category of the target face.

[0087] According to the third aspect of the present invention or the first executable mode of the third aspect of the present invention, in the second executable mode of the third aspect of the present invention,

[0088] Before inputting the third depth information of the target face into the second neural network and inputting the fifth color information of the target face into the third neural network, the method further includes:

[0089] Performing a third process on the third depth information of the target face, wherein the third process includes at least one of the following:

[0090] Determine a feature point of the third depth information of the target face, and rotate the third depth information of the target face based on the feature point;

[0091] Performing mirror processing, linear transformation, and affine transformation on the third depth information of the target face;

[0092] Aligning feature points of the third depth information of the target face to a set position;

[0093] Performing grayscale stretching on the third depth information of the target face; and

[0094] Normalizing the third depth information of the target face to an image pixel value; or,

[0095] Before inputting the third depth information of the target face into the second neural network and inputting the fifth color information of the target face into the third neural network, the method further includes:

[0096] Performing the same third processing on the third depth information of the target face and the fifth color information of the target face, wherein the third processing includes at least one of the following:

[0097] Determine feature points of the third depth information of the target face and feature points of the fifth color information of the target face, and rotate the third depth information of the target face and the fifth color information of the target face based on the feature points;

[0098] Performing mirror processing, linear transformation, and affine transformation on the third depth information of the target face and the fifth color information of the target face;

[0099] Aligning feature points of the third depth information of the target face and the fifth color information of the target face at a set position;

[0100] Performing grayscale stretching on the third depth information of the target face or the fifth color information of the target face; and

[0101] The third depth information of the target face and the fifth color information of the target face are subjected to image pixel value normalization processing.

[0102] According to a second executable mode of the third aspect of the present invention, in a third executable mode of the third aspect of the present invention,

[0103] The step of normalizing the image pixel values ​​of the third depth information of the target face includes:

[0104] Normalize the pixel value of the third depth information of the target face from [0, 255] to [0, 1]; or,

[0105] The normalizing the pixel values ​​of each channel of the third depth information of the target face and the fifth color information of the target face includes:

[0106] The pixel values ​​of each channel of the third depth information of the target face and the fifth color information of the target face are normalized from [0, 255] to [0, 1].

[0107] According to the third aspect of the present invention or any one of the first to third executable modes of the third aspect of the present invention, in a fourth executable mode of the third aspect of the present invention,

[0108] The second parameter data is obtained by training the second neural network based on fourth depth information of a plurality of facial expression samples;

[0109] The third parameter data is obtained by training the third neural network based on the sixth color information of the plurality of facial expression samples.

[0110] According to a fourth executable mode of the third aspect of the present invention, in a fifth executable mode of the third aspect of the present invention,

[0111] Before the fourth depth information of the plurality of facial expression samples is trained by the second neural network, the method further includes:

[0112] Performing a fourth processing on the fourth depth information of the facial expression sample, wherein the fourth processing includes at least one of the following:

[0113] Determine a feature point of the fourth depth information of the facial expression sample, and rotate the fourth depth information of the facial expression sample based on the feature point;

[0114] Performing mirror processing, linear transformation and affine transformation on the fourth depth information of the facial expression sample;

[0115] Aligning a feature point of the fourth depth information of the facial expression sample to a set position;

[0116] Performing grayscale stretching on the fourth depth information of the facial expression sample; and

[0117] Performing image pixel value normalization processing on the fourth depth information of the facial expression sample;

[0118] Or, before the fourth depth information of the facial expression sample is trained by the second neural network and the sixth color information of the facial expression sample is trained by the third neural network, the method further includes:

[0119] Performing the same fourth processing on the fourth depth information of the facial expression sample and the sixth color information of the facial expression sample, wherein the fourth processing includes at least one of the following:

[0120] Determine a feature point of the fourth depth information of the facial expression sample and a feature point of the sixth color information of the facial expression sample, and rotate the fourth depth information of the facial expression sample and the sixth color information of the facial expression sample based on the feature point;

[0121] Performing mirror processing, linear transformation, and affine transformation on the fourth depth information of the facial expression sample and the sixth color information of the facial expression sample;

[0122] Aligning feature points of the fourth depth information of the facial expression sample and the sixth color information of the facial expression sample at a set position;

[0123] Performing grayscale stretching on the fourth depth information of the facial expression sample or the sixth color information of the facial expression sample; and,

[0124] Performing image pixel value normalization processing on the fourth depth information of the facial expression sample and the sixth color information of the facial expression sample.

[0125] According to a fifth executable mode of the third aspect of the present invention, in a sixth executable mode of the third aspect of the present invention,

[0126] The step of normalizing the image pixel values ​​of the fourth depth information of the facial expression sample comprises:

[0127] Normalizing the pixel value of the fourth depth information of the facial expression sample from [0, 255] to [0, 1]; or,

[0128] The step of performing image pixel normalization processing on the fourth depth information of the facial expression sample and the sixth color information of the facial expression sample comprises:

[0129] Normalize the pixel values ​​of each channel of the fourth depth information of the facial expression sample and the sixth color information of the facial expression sample from [0, 255] to [0, 1].

[0130] According to any one of the fourth to sixth executable modes of the third aspect of the present invention, in a seventh executable mode of the third aspect of the present invention,

[0131] The support vector machine parameter data for identifying the expression category of the target face is obtained by training the support vector machine using the output data of the second neural network when training the fourth depth information of multiple facial expression samples and the output data of the third neural network when training the sixth color information of the multiple facial expression samples.

[0132] According to any one of the fourth to seventh executable modes of the third aspect of the present invention, in an eighth executable mode of the third aspect of the present invention,

[0133] The facial expression sample has at least one of the following facial expression categories: fear, sadness, happiness, anger, disgust, surprise, neutral, and contempt;

[0134] The facial expression sample, the fourth depth information of the facial expression sample, and the sixth color information of the facial expression sample have the same facial expression category.

[0135] According to the third aspect of the present invention and any one of the first to eighth executable modes of the third aspect of the present invention, in a ninth executable mode of the third aspect of the present invention,

[0136] The facial expression categories included in the second neural network and the facial expression categories included in the third neural network include at least one of the following: fear, sadness, happiness, anger, disgust, surprise, neutral, and contempt.

[0137] According to any one of the second to ninth executable modes of the third aspect of the present invention, in a tenth executable mode of the third aspect of the present invention, the feature point is an eye point.

[0138] According to the third aspect of the present invention and any one of the first to tenth executable modes of the third aspect of the present invention, in the eleventh executable mode of the third aspect of the present invention,

[0139] The second neural network includes a second convolutional neural network, and the third neural network includes a third convolutional neural network.

[0140] According to the eleventh executable mode of the third aspect of the present invention, in the twelfth executable mode of the third aspect of the present invention,

[0141] The second convolutional neural network includes: 3 convolutional layers, 3 downsampling layers, 1 drop layer and 2 fully connected layers;

[0142] The third convolutional neural network includes: 4 convolutional layers, 4 downsampling layers, 1 drop layer and 2 fully connected layers.

[0143] According to the third aspect of the present invention and any one of the first to twelfth executable modes of the third aspect of the present invention, in the thirteenth executable mode of the third aspect of the present invention, the fifth color information is an image in RGB format or YUV format.

[0144] According to the third aspect of the present invention and any one of the fourth to thirteenth executable modes of the third aspect of the present invention, in the fourteenth executable mode of the third aspect of the present invention,

[0145] The sixth color information is an image in RGB format or YUV format.

[0146] According to a fourth aspect of the present invention, there is provided a device for expression recognition, the device comprising a second acquisition module, a second input module, a second neural network, a third neural network and a second classification module,

[0147] The second acquisition module is used to acquire a three-dimensional image of the target face, wherein the three-dimensional image includes third depth information of the target face and fifth color information of the target face;

[0148] The second input module is used to input the third depth information of the target face into the second neural network and input the fifth color information of the target face into the third neural network;

[0149] The second neural network is used to classify the expression of the target face according to the third depth information of the target face and the second parameter and output first classification data, and the third neural network is used to classify the expression of the target face according to the fifth color information of the target face and the third parameter and output second classification data, the second parameter includes at least one facial expression category and second parameter data for identifying the expression category of the target face, and the third parameter includes at least one facial expression category and third parameter data for identifying the expression category of the target face;

[0150] The second classification module is used to output a classification result of the expression of the target face according to the first classification data and the second classification data.

[0151] According to a fourth aspect of the present invention, in a first executable manner of the fourth aspect of the present invention, the second classification module includes a support vector machine,

[0152] The support vector machine is used to input the first classification data and the second classification data, and output a classification result of the expression of the target face according to the first classification data, the second classification data and the support vector machine parameter data. The support vector machine includes the at least one facial expression category and the support vector machine parameter data for identifying the expression category of the target face.

[0153] According to the fourth aspect of the present invention and the first executable mode of the fourth aspect of the present invention, in the second executable mode of the fourth aspect of the present invention, the device further includes a third processing module,

[0154] The third processing module is used to perform a third processing on the third depth information of the target face, and input the third depth information of the target face after the third processing into the second input module;

[0155] The third processing module includes at least one of a third rotation submodule, a third transformation submodule, a third alignment submodule, a third grayscale stretching submodule, and a third normalization processing submodule;

[0156] The third rotation submodule is used to determine feature points of the third depth information of the target face, and rotate the third depth information of the target face based on the feature points;

[0157] The third transformation submodule is used to perform mirror processing, linear transformation and affine transformation on the third depth information of the target face;

[0158] The third alignment submodule is used to align the feature points of the third depth information of the target face with the set positions;

[0159] The third grayscale stretching submodule is used to perform grayscale stretching on the third depth information of the target face;

[0160] The third normalization processing submodule is used to perform image pixel value normalization processing on the third depth information of the target face;

[0161] or,

[0162] The third processing module is further used to perform the same third processing on the third depth information of the target face and the fifth color information of the target face, and input the third depth information of the target face and the fifth color information of the target face after the third processing into the second input module;

[0163] The third rotation submodule is further used to determine feature points of the third depth information of the target face and feature points of the fifth color information of the target face, and rotate the third depth information of the target face and the fifth color information of the target face based on the feature points;

[0164] The third transformation submodule is further used to perform mirror processing, linear transformation and affine transformation on the third depth information of the target face and the fifth color information of the target face;

[0165] The third alignment submodule is further used to align feature points of the third depth information of the target face and the fifth color information of the target face at a set position;

[0166] The third grayscale stretching submodule is further used to perform grayscale stretching on the third depth information of the target face or the fifth color information of the target face; and

[0167] The third normalization processing submodule is further used to perform image pixel value normalization processing on the third depth information of the target face and the fifth color information of the target face.

[0168] According to the second executable mode of the fourth aspect of the present invention, in the third executable mode of the fourth aspect of the present invention,

[0169] The third normalization processing submodule is specifically used to normalize the pixel value of the third depth information of the target face from [0, 255] to [0, 1];

[0170] or,

[0171] The third normalization processing submodule is specifically used to normalize the pixel values ​​of each channel of the third depth information of the target face and the fifth color information of the target face from [0, 255] to [0, 1].

[0172] According to the fourth aspect of the present invention and the first to third executable modes of the fourth aspect of the present invention, in the fourth executable mode of the fourth aspect of the present invention,

[0173] The second parameter data is obtained by training the second neural network based on fourth depth information of a plurality of facial expression samples;

[0174] The third parameter data is obtained by training the third neural network based on the sixth color information of the plurality of facial expression samples.

[0175] According to a fourth executable mode of the fourth aspect of the present invention, in a fifth executable mode of the fourth aspect of the present invention, the device includes a fourth processing module,

[0176] The fourth processing module is used to perform a fourth processing on the fourth depth information of the facial expression sample, and input the fourth depth information of the facial expression sample after the fourth processing into the second input module;

[0177] The fourth processing module includes at least one of a fourth rotation submodule, a fourth transformation submodule, a fourth alignment submodule, a fourth grayscale stretching submodule, and a fourth normalization processing submodule;

[0178] The fourth rotation submodule is used to determine a feature point of the fourth depth information of the facial expression sample, and rotate the fourth depth information of the facial expression sample based on the feature point;

[0179] The fourth transformation submodule is used to perform mirror processing, linear transformation and affine transformation on the fourth depth information of the facial expression sample;

[0180] The fourth alignment submodule is used to align the feature points of the fourth depth information of the facial expression sample to a set position;

[0181] The fourth grayscale stretching submodule is used to perform grayscale stretching on the fourth depth information of the facial expression sample; and

[0182] The fourth normalization processing submodule is used to perform image pixel value normalization processing on the fourth depth information of the facial expression sample;

[0183] or,

[0184] The fourth processing module is further used to perform a fourth processing on the fourth depth information of the facial expression sample and the sixth color information of the facial expression sample, and input the fourth depth information of the facial expression sample and the sixth color information of the facial expression sample after the fourth processing into the second input module;

[0185] The fourth rotation submodule is further used to determine feature points of the fourth depth information of the facial expression sample and feature points of the sixth color information of the facial expression sample, and rotate the fourth depth information of the facial expression sample and the sixth color information of the facial expression sample based on the feature points;

[0186] The fourth transformation submodule is further used to perform mirror processing, linear transformation and affine transformation on the fourth depth information of the facial expression sample and the sixth color information of the facial expression sample;

[0187] The fourth alignment submodule is further used to align feature points of the fourth depth information of the facial expression sample and the sixth color information of the facial expression sample at a set position;

[0188] The fourth grayscale stretching submodule is further used to perform grayscale stretching on the fourth depth information of the facial expression sample or the sixth color information of the facial expression sample; and

[0189] The fourth normalization processing submodule is further used to perform image pixel value normalization processing on the fourth depth information of the facial expression sample and the sixth color information of the facial expression sample.

[0190] According to a fifth executable mode of the fourth aspect of the present invention, in a sixth executable mode of the fourth aspect of the present invention,

[0191] The fourth normalization processing submodule is specifically used to normalize the pixel value of the fourth depth information of the facial expression sample from [0, 255] to [0, 1];

[0192] or,

[0193] The fourth normalization processing submodule is specifically used to normalize the pixel values ​​of each channel of the fourth depth information of the facial expression sample and the sixth color information of the facial expression sample from [0, 255] to [0, 1].

[0194] According to any one of the fourth to sixth executable modes of the fourth aspect of the present invention, in a seventh executable mode of the fourth aspect of the present invention,

[0195] The support vector machine parameter data for identifying the expression category of the target face is obtained by training the support vector machine using the output data of the second neural network when training the fourth depth information of multiple facial expression samples and the output data of the third neural network when training the sixth color information of the multiple facial expression samples.

[0196] According to any one of the fourth to seventh executable modes of the fourth aspect of the present invention, in an eighth executable mode of the fourth aspect of the present invention,

[0197] The facial expression sample has at least one of the following facial expression categories: fear, sadness, happiness, anger, disgust, surprise, neutral, and contempt;

[0198] The facial expression sample, the fourth depth information of the facial expression sample, and the sixth color information of the facial expression sample have the same facial expression category.

[0199] According to the fourth aspect of the present invention and any one of the first to eighth executable modes of the fourth aspect of the present invention, in a ninth executable mode of the fourth aspect of the present invention,

[0200] The facial expression categories included in the second neural network and the facial expression categories included in the third neural network include at least one of the following: fear, sadness, happiness, anger, disgust, surprise, neutral, and contempt.

[0201] According to any one of the second to ninth executable modes of the fourth aspect of the present invention, in a tenth executable mode of the fourth aspect of the present invention, the feature point is an eye point.

[0202] According to the fourth aspect of the present invention and any one of the first to tenth executable modes of the fourth aspect of the present invention, in the eleventh executable mode of the fourth aspect of the present invention,

[0203] The second neural network includes a second convolutional neural network, and the third neural network includes a third convolutional neural network.

[0204] According to the eleventh executable mode of the fourth aspect of the present invention, in the twelfth executable mode of the fourth aspect of the present invention,

[0205] The second convolutional neural network includes: 3 convolutional layers, 3 downsampling layers, 1 drop layer and 2 fully connected layers;

[0206] The third convolutional neural network includes: 4 convolutional layers, 4 downsampling layers, 1 drop layer and 2 fully connected layers.

[0207] According to the fourth aspect of the present invention and the first to twelfth embodiments of the fourth aspect of the present invention, in the thirteenth embodiment of the fourth aspect of the present invention,

[0208] The fifth color information is an image in RGB format or YUV format.

[0209] According to the fourth to thirteenth embodiments of the fourth aspect of the present invention, in the fourteenth embodiment of the fourth aspect of the present invention,

[0210] The sixth color information is an image in RGB format or YUV format.

[0211] According to a fifth aspect of the present invention, there is provided a method for facial expression recognition, the method comprising:

[0212] Acquire a three-dimensional image of a target face, wherein the three-dimensional image includes fifth depth information of the target face and seventh color information of the target face;

[0213] Inputting the fifth depth information of the target face and the seventh color information of the target face into a fourth neural network;

[0214] The fourth neural network classifies the expression of the target face based on the fifth depth information of the target face, the seventh color information of the target face and the fourth parameter, wherein the fourth parameter includes at least one facial expression category and fourth parameter data for identifying the expression category of the target face.

[0215] According to a fifth aspect of the present invention, in a first executable mode of the fifth aspect of the present invention,

[0216] Before inputting the fifth depth information of the target face and the seventh color information of the target face into the fourth neural network, the method further includes:

[0217] Performing a fifth process on the three-dimensional image of the target face, wherein the fifth process includes at least one of the following:

[0218] Determine feature points of the three-dimensional image of the target face, and rotate the three-dimensional image of the target face based on the feature points;

[0219] Performing mirror processing, linear transformation and affine transformation on the three-dimensional image of the target face;

[0220] Aligning feature points of the three-dimensional image of the target face to a set position;

[0221] Performing grayscale stretching on the three-dimensional image of the target face; and

[0222] Performing image pixel value normalization processing on the three-dimensional image of the target face.

[0223] According to the first executable mode of the fifth aspect of the present invention, in the second executable mode of the fifth aspect of the present invention,

[0224] The step of normalizing the pixel values ​​of each channel of the three-dimensional image of the target face includes:

[0225] The pixel values ​​of each channel of the three-dimensional image of the target face are normalized from [0, 255] to [0, 1].

[0226] According to the fifth aspect of the present invention and the first executable mode or the second executable mode of the fifth aspect of the present invention, in the third executable mode of the fifth aspect of the present invention,

[0227] The fourth parameter data is obtained by training the three-dimensional images of a plurality of facial expression samples through the fourth neural network;

[0228] The three-dimensional image of the facial expression sample includes: sixth depth information of the facial expression sample and eighth color information of the facial expression sample.

[0229] According to a third executable mode of the fifth aspect of the present invention, in a fourth executable mode of the fifth aspect of the present invention,

[0230] Before the three-dimensional images of the plurality of facial expression samples are trained by the fourth neural network, the method further comprises:

[0231] Performing a sixth process on the three-dimensional image of the facial expression sample, wherein the sixth process includes at least one of the following:

[0232] Determine feature points of the three-dimensional image of the facial expression sample, and rotate the three-dimensional image of the facial expression sample based on the feature points;

[0233] Performing mirror processing, linear transformation and affine transformation on the three-dimensional image of the facial expression sample;

[0234] Aligning feature points of the three-dimensional image of the facial expression sample to the set position;

[0235] Performing grayscale stretching on the three-dimensional image of the facial expression sample; and,

[0236] Performing image pixel value normalization processing on the three-dimensional image of the facial expression sample.

[0237] According to a fourth executable mode of the fifth aspect of the present invention, in the fifth executable mode of the fifth aspect of the present invention,

[0238] The step of normalizing the pixel values ​​of the three-dimensional image of the facial expression sample comprises:

[0239] The pixel values ​​of each channel of the three-dimensional image of the facial expression sample are normalized from [0, 255] to [0, 1].

[0240] According to any one of the third to fifth executable modes of the fifth aspect of the present invention, in a sixth executable mode of the fifth aspect of the present invention,

[0241] The facial expression sample has at least one of the following facial expression categories: fear, sadness, happiness, anger, disgust, surprise, neutral, and contempt;

[0242] The facial expression sample, the sixth depth information of the facial expression sample, and the eighth color information of the facial expression sample have the same facial expression category.

[0243] According to the fifth aspect of the present invention and any one of the first to sixth executable modes of the fifth aspect of the present invention, in a seventh executable mode of the fifth aspect of the present invention,

[0244] The facial expression categories included in the fourth neural network include at least one of the following: fear, sadness, happiness, anger, disgust, surprise, neutral, and contempt.

[0245] According to any one of the first to seventh executable modes of the fifth aspect of the present invention, in an eighth executable mode of the fifth aspect of the present invention, the feature point is an eye point.

[0246] According to the fifth aspect of the present invention and any one of the first to eighth executable modes of the fifth aspect of the present invention, in a ninth executable mode of the fifth aspect of the present invention,

[0247] The fourth neural network includes a fourth convolutional neural network.

[0248] According to a ninth executable mode of the fifth aspect of the present invention, in a tenth executable mode of the fifth aspect of the present invention,

[0249] The fourth convolutional neural network includes 1 segmentation layer, 8 convolution layers, 8 downsampling layers, 2 reduction layers and 5 fully connected layers.

[0250] According to the fifth aspect of the present invention and the first to tenth executable modes of the fifth aspect of the present invention, in the eleventh executable mode of the fifth aspect of the present invention,

[0251] The seventh color information is an image in RGB format or YUV format.

[0252] According to the third to eleventh executable modes of the fifth aspect of the present invention, in the twelfth executable mode of the fifth aspect of the present invention, the eighth color information is an image in RGB format or YUV format.

[0253] According to a sixth aspect of the present invention, there is provided a device for facial expression recognition, characterized in that the device comprises:

[0254] a third acquisition module, configured to acquire a three-dimensional image of a target face, wherein the three-dimensional image includes fifth depth information of the target face and seventh color information of the target face;

[0255] A third input module, configured to input the fifth depth information of the target face and the seventh color information of the target face into a fourth neural network;

[0256] a fourth neural network, for classifying an expression of the target face based on the fifth depth information of the target face, the seventh color information of the target face, and a fourth parameter, wherein the fourth parameter includes at least one facial expression category and fourth parameter data for identifying the expression category of the target face.

[0257] According to a sixth aspect of the present invention, in a first executable manner of the sixth aspect of the present invention, the device further includes a fifth processing module,

[0258] The fifth processing module is used to perform a fifth processing on the three-dimensional image of the target face, and input the three-dimensional image of the target face after the fifth processing into the third input module;

[0259] The fifth processing module includes at least one of the following submodules: a fifth rotation submodule, a fifth transformation submodule, a fifth alignment submodule, a fifth grayscale stretching submodule, and a fifth normalization processing submodule;

[0260] The fifth rotation submodule is used to determine feature points of the three-dimensional image of the target face, and rotate the three-dimensional image of the target face based on the feature points;

[0261] The fifth transformation submodule is used to perform mirror processing, linear transformation and affine transformation on the three-dimensional image of the target face;

[0262] The fifth alignment submodule is used to align the feature points of the three-dimensional image of the target face to a set position;

[0263] The fifth grayscale stretching submodule is used to perform grayscale stretching on the three-dimensional image of the target face; and

[0264] The fifth normalization processing submodule is used to perform image pixel value normalization processing on the three-dimensional image of the target face.

[0265] According to the first executable mode of the sixth aspect of the present invention, in the second executable mode of the sixth aspect of the present invention,

[0266] The fifth normalization processing submodule is specifically used to normalize the pixel values ​​of each channel of the three-dimensional image of the target face from [0, 255] to [0, 1].

[0267] According to the sixth aspect of the present invention and the first executable mode or the second executable mode of the sixth aspect of the present invention, in the third executable mode of the sixth aspect of the present invention,

[0268] The fourth parameter data for identifying the expression category of the target face is obtained by training the fourth neural network with three-dimensional images of a plurality of facial expression samples;

[0269] The three-dimensional image of the facial expression sample includes: sixth depth information of the facial expression sample and eighth color information of the facial expression sample.

[0270] According to the third executable mode of the sixth aspect of the present invention, in the fourth executable mode of the sixth aspect of the present invention, the device further includes a sixth processing module,

[0271] The sixth processing module is used to perform a fifth processing on the three-dimensional image of the facial expression sample, and input the three-dimensional image of the facial expression sample after the fifth processing into the third input module;

[0272] The sixth processing module includes a sixth rotation submodule, a sixth transformation submodule, a sixth alignment submodule, a sixth grayscale stretching submodule, and a sixth normalization processing submodule;

[0273] The sixth rotation submodule is used to determine the feature points of the three-dimensional image of the facial expression sample, and rotate the three-dimensional image of the facial expression sample based on the feature points;

[0274] The sixth transformation submodule is used to perform mirror processing, linear transformation and affine transformation on the three-dimensional image of the facial expression sample;

[0275] The sixth alignment submodule is used to align the feature points of the three-dimensional image of the facial expression sample to the set position;

[0276] The sixth grayscale stretching submodule is used to perform grayscale stretching on the three-dimensional image of the facial expression sample; and

[0277] The sixth normalization processing submodule is used to perform image pixel value normalization processing on the three-dimensional image of the facial expression sample.

[0278] According to a fourth executable mode of the sixth aspect of the present invention, in a fifth executable mode of the sixth aspect of the present invention,

[0279] The sixth normalization processing submodule is specifically used to normalize the pixel values ​​of each channel of the three-dimensional image of the facial expression sample from [0, 255] to [0, 1].

[0280] According to any one of the third to fifth executable modes of the sixth aspect of the present invention, in the sixth executable mode of the sixth aspect of the present invention,

[0281] The facial expression sample has at least one of the following facial expression categories: fear, sadness, happiness, anger, disgust, surprise, neutral, and contempt;

[0282] The facial expression sample, the sixth depth information of the facial expression sample, and the eighth color information of the facial expression sample have the same facial expression category.

[0283] According to the sixth aspect of the present invention and any one of the first to sixth executable modes of the sixth aspect of the present invention, in a seventh executable mode of the sixth aspect of the present invention,

[0284] The facial expression categories included in the fourth neural network include at least one of the following: fear, sadness, happiness, anger, disgust, surprise, neutral, and contempt.

[0285] According to any one of the first executable mode of the sixth aspect of the present invention to the seventh executable mode of the sixth aspect of the present invention, in the eighth executable mode of the sixth aspect of the present invention, the feature point is an eye point.

[0286] According to the sixth aspect of the present invention and any one of the first executable mode of the sixth aspect of the present invention to the eighth executable mode of the sixth aspect of the present invention, in the ninth executable mode of the sixth aspect of the present invention,

[0287] The fourth neural network includes a fourth convolutional neural network.

[0288] According to a ninth executable mode of the sixth aspect of the present invention, in a tenth executable mode of the sixth aspect of the present invention,

[0289] The fourth convolutional neural network includes 1 segmentation layer, 8 convolution layers, 8 downsampling layers, 2 reduction layers and 5 fully connected layers.

[0290] According to the sixth aspect of the present invention and any one of the first to tenth executable modes of the sixth aspect of the present invention, in the eleventh executable mode of the sixth aspect of the present invention, the seventh color information is an image in RGB format or YUV format.

[0291] According to any one of the third executable mode of the sixth aspect of the present invention to the eleventh executable mode of the sixth aspect of the present invention, in the twelfth executable mode of the sixth aspect of the present invention,

[0292] The eighth color information is an image in RGB format or YUV format.

[0293] According to a seventh aspect of the present invention, there is provided a computer-readable storage medium, the computer-readable storage medium storing a computer program, characterized in that when the computer program is executed by a first processor, it implements the steps in any executable mode of the first aspect of the present invention and the first executable mode to the twelfth executable mode of the first aspect of the present invention, the third aspect of the present invention and the first executable mode to the fourteenth executable mode of the third aspect of the present invention, the fifth aspect of the present invention and the first executable mode to the twelfth executable mode of the fifth aspect of the present invention.

[0294] According to an eighth aspect of the present invention, there is provided an apparatus for facial expression recognition, comprising a memory, a second processor, and a computer program stored in the memory and executable on the second processor, wherein the computer program, when executed by the second processor, implements steps in any one of the first aspect of the present invention and the first to twelfth executable modes of the first aspect of the present invention, the third aspect of the present invention and the first to fourteenth executable modes of the third aspect of the present invention, the fifth aspect of the present invention and the first to twelfth executable modes of the fifth aspect of the present invention.

[0295] The present invention provides a method and device for facial expression recognition, which can effectively solve the problem of decreased accuracy of facial expression recognition caused by different facial postures and different lighting conditions, and improve the accuracy of facial expression recognition of a target face under different facial postures and different lighting conditions. BRIEF DESCRIPTION OF THE DRAWINGS

[0296] Figure 1 A flowchart of a method for facial expression recognition provided in Embodiment 1 of the present invention;

[0297] Figure 2 A flowchart of another method for facial expression recognition provided in Embodiment 2 of the present invention;

[0298] Figure 3 A flowchart of another method for facial expression recognition provided in Embodiment 3 of the present invention;

[0299] Figure 4 A schematic diagram of a device structure for expression recognition provided in a fourth embodiment of the present invention;

[0300] Figure 5A schematic diagram of the structure of another device for expression recognition provided in Embodiment 5 of the present invention;

[0301] Figure 6 A schematic diagram of a structure of another device for expression recognition provided in Embodiment 6 of the present invention;

[0302] Figure 7 A schematic diagram of the structure of another device for expression recognition provided in Embodiment 6 of the present invention;

[0303] Figure 8 A schematic diagram of the structure of another device for expression recognition provided in Embodiment 6 of the present invention. DETAILED DESCRIPTION

[0304] The technical solutions in the embodiments of the present invention will be described in detail below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0305] The description and claims of the present invention refer to the above-mentioned drawings and the terms "first", "second", etc., which are used to distinguish different objects rather than to limit a specific order.

[0306] Among them, the term "and / or" in the embodiment of the present invention is only a description of the association relationship of associated objects, indicating that there may be three relationships. For example, A and / or B can represent three situations: A exists alone, A and B exist at the same time, and B exists alone.

[0307] In the embodiments of the present invention, words such as "exemplary" or "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present invention should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0308] It should be noted that for the sake of simplicity and clarity of illustration, the elements shown in the drawings are not necessarily drawn to scale. For example, the size of some elements may be enlarged relative to other elements for clarity. In addition, where deemed appropriate, reference numerals may be repeated between the drawings to indicate corresponding or similar elements for that matter.

[0309] It should be noted that since videos and the like are composed of a number of pictures, the processing methods for pictures, imaging, images, etc. described in the embodiments of the present invention can be applied to videos and the like. Those skilled in the art can modify the methods disclosed by the present invention into processing methods applied to videos and the like without any creative effort, and such modified methods are within the scope of protection of the present invention.

[0310] The various embodiments of the present invention are described in detail using human faces as an example. The technical solution of the present invention is also applicable to the recognition of facial expressions of different objects, such as different animals, or target objects with similar features to human faces.

[0311] The following is combined with Figure 1 A method for facial expression recognition provided by the first embodiment of the present invention is specifically described. Figure 1 As shown, the method includes:

[0312] Step 101, obtaining a three-dimensional image of a target face and a two-dimensional image of the target face, wherein the three-dimensional image includes first depth information of the target face and first color information of the target face, and the two-dimensional image includes second color information of the target face.

[0313] Optionally, the acquiring step may be acquiring from a memory a three-dimensional image of the target face and a two-dimensional image of the target face taken by a camera device.

[0314] Optionally, the three-dimensional image of the target face and the two-dimensional image of the target face may be color images.

[0315] Optionally, the first color information and the second color information are images in RGB format or YUV format, and may also be images in other formats that can be converted into the RGB format or YUV format.

[0316] Step 102, input the first depth information of the target face, the first color information of the target face, and the second color information of the target face into a first neural network. Optionally, the input to the first neural network may be a depth map of the target face, an RGB map of a three-dimensional image of the target face, and an RGB map of a two-dimensional image of the target face; the input to the first neural network may also be a depth map of the target face, three channels of the RGB map of the three-dimensional image of the target face, and three channels of the RGB map of the two-dimensional image of the target face.

[0317] Optionally, the first neural network includes a first convolutional neural network, which includes 4 convolutional layers, 4 downsampling layers, 1 dropout layer and 2 fully connected layers.

[0318] Step 103, the first neural network classifies the expression of the target face according to the first depth information of the target face, the first color information of the target face, the second color information of the target face and the first parameter, wherein the first parameter includes at least one facial expression category and first parameter data for identifying the expression category of the target face. Since most expressions are compound expressions and can belong to at least one expression category, the first neural network includes the first parameter, and the facial expression category included in the first parameter includes at least one of the following: fear, sadness, happiness, anger, disgust, surprise, naturalness, and contempt. Optionally, in one embodiment, the first parameter may include eight facial expression categories of fear, sadness, happiness, anger, disgust, surprise, naturalness, and contempt, and first parameter data for identifying the eight facial expression categories. Specifically, the classification result output by the first neural network may be the probability that the target face belongs to the different expression categories, and the sum of the probabilities of belonging to the different expression categories is 1. The first neural network may sort the output classification results according to the level of the above-mentioned probabilities. The first parameter data may include the weight of at least one node of the neural network.

[0319] Optionally, in the case where the first parameter includes a facial expression category, the first neural network can be used to determine whether the expression of the target face is the facial expression category included in the first parameter.

[0320] Optionally, in order to cope with the situation that the obtained target face posture is not ideal or the lighting conditions are not ideal, the same first processing can be performed on the three-dimensional image of the target face and the two-dimensional image of the target face to approximately meet the requirements of the standard face or the requirements of use. Specifically, before the first depth information of the target face, the first color information of the target face, and the second color information of the target face are input into the first neural network, the method also includes: performing the same first processing on the three-dimensional image of the target face and the two-dimensional image of the target face, and the first processing includes at least one of the following: determining feature points of the three-dimensional image of the target face and the two-dimensional image of the target face, and rotating the three-dimensional image of the target face and the two-dimensional image of the target face based on the feature points; performing mirror processing, linear transformation and affine transformation on the three-dimensional image of the target face and the two-dimensional image of the target face; aligning the feature points of the three-dimensional image of the target face and the two-dimensional image of the target face to a set position; performing grayscale stretching on the three-dimensional image of the target face and the two-dimensional image of the target face; and performing image pixel value normalization processing on the three-dimensional image of the target face and the two-dimensional image of the target face.

[0321] The above-mentioned performing the same first processing on the three-dimensional image of the target face and the two-dimensional image of the target face may include: performing the same first processing on the two-dimensional image of the target face as the first processing on the three-dimensional image of the target face. Exemplarily, the above-mentioned performing the same first processing on the three-dimensional image of the target face and the two-dimensional image of the target face may include: performing linear transformation, affine transformation and grayscale stretching on the three-dimensional image of the target face, and also performing the same linear transformation, affine transformation and grayscale stretching on the two-dimensional image of the target face; for another example, performing mirror processing, linear transformation and image pixel value normalization processing on the three-dimensional image of the target face, and also performing mirror processing, linear transformation and image pixel value normalization processing on the two-dimensional image of the target face. Optionally, the above-mentioned same first processing of the three-dimensional image of the target face and the two-dimensional image of the target face can be: performing the same first processing on the depth information of the target face (such as a depth map), and the three channels of the RGB image of the three-dimensional image of the target face and the three channels of the RGB image of the two-dimensional image of the target face, respectively; or it can be performing the same first processing on the entire image of the three-dimensional image of the target face and the entire image of the two-dimensional image of the target face, and then decomposing them into the first depth information of the target face, the first color information of the target face, and the second color information of the target face, and inputting them into the first neural network.

[0322] Optionally, the feature points may be eye points, or other facial features such as nose tip points. The positions set for aligning the feature points of the three-dimensional image of the target face and the two-dimensional image of the target face may be feature points of a standard facial image, such as eye points, or may be pre-set positions, or may be feature points uniformly aligned with the facial expression samples input when training the first neural network, such as eye points.

[0323] Optionally, the grayscale stretching of the three-dimensional image of the target face and the two-dimensional image of the target face may include segmented grayscale stretching of the three-dimensional image of the target face and the two-dimensional image of the target face according to the characteristics of the three-dimensional image of the target face and / or the two-dimensional image of the target face. It may also include segmented grayscale stretching of the pixel values ​​according to the size of the pixel values ​​of the three-dimensional image of the target face and the two-dimensional image of the target face.

[0324] Optionally, the performing image pixel value normalization processing on the three-dimensional image of the target face and the two-dimensional image of the target face includes: normalizing the pixel values ​​of each channel of the three-dimensional image of the target face and the two-dimensional image of the target face from [0, 255] to [0, 1]. The above-mentioned channels may include: depth information of the three-dimensional image of the target face, three channels of the RGB image of the three-dimensional image of the target face, and three channels of the RGB image of the two-dimensional image of the target face.

[0325] Normally, taking the face as an example, the three-dimensional image and the two-dimensional image of the target face acquired by the camera device include redundant parts such as the neck and shoulders in addition to the face. It is necessary to locate the face frame through face detection, then deduct it, and then locate the above-mentioned facial features, such as the eye points, and then perform the above-mentioned first processing.

[0326] Optionally, the first parameter data for identifying the expression category of the target face is obtained by training the first neural network with a three-dimensional image of a plurality of facial expression samples and a two-dimensional image of the facial expression samples. The three-dimensional image of the facial expression sample includes: the second depth information of the facial expression sample and the third color information of the facial expression sample, and the two-dimensional image of the facial expression sample includes: the fourth color information of the facial expression sample. Specifically, the second depth information, the third color information and the fourth color information of the plurality of facial expression samples can be input into the first neural network and iterated. The plurality of facial expression samples have facial expression categories representing facial expression categories, and a parameter combination with a high expression accuracy of the facial expression samples is determined as the first parameter for identifying the expression category of the target face. The specific content of the first parameter can be found above. Optionally, the first parameter can be obtained by training the facial expression samples offline, and the facial expression samples can be excluded from the products provided for actual use for expression recognition.

[0327] Since most expressions are complex expressions and can belong to at least one expression category, the facial expression sample has at least one of the following facial expression categories: fear, sadness, happiness, anger, disgust, surprise, naturalness, and contempt. The facial expression sample, the second depth information of the facial expression sample, the third color information of the facial expression sample, and the fourth color information of the facial expression sample have the same facial expression category. The third color information and the fourth color information are images in RGB format or YUV format. Through the facial expression category of the facial expression sample, the facial expression category of the component of the facial expression sample input into the first neural network can be determined (the second depth information of the facial expression sample and the third color information of the facial expression sample are components of a three-dimensional image, and the fourth color information of the facial expression sample is a component of a two-dimensional image), then the first neural network can be trained to obtain the first parameter data corresponding to the above different facial expression categories.

[0328] Optionally, in order to cope with the situation that the posture of the acquired facial expression samples is not ideal or the lighting conditions are not ideal, the same second processing can be performed on the three-dimensional image of the facial expression samples and the two-dimensional image of the facial expression samples to approximately meet the requirements of the standard face or the requirements of use. Specifically, before the three-dimensional images of the plurality of facial expression samples and the two-dimensional images of the facial expression samples are trained by the first neural network, the method further includes: performing the same second processing on the three-dimensional image of the facial expression samples and the two-dimensional image of the facial expression samples, the second processing including at least one of the following: determining the three-dimensional image of the facial expression samples and the two-dimensional image of the facial expression samples The method comprises the following steps: taking the feature points of the two-dimensional image of the facial expression sample as a reference, rotating the three-dimensional image of the facial expression sample and the two-dimensional image of the facial expression sample based on the feature points; performing mirroring, linear transformation and affine transformation on the three-dimensional image of the facial expression sample and the two-dimensional image of the facial expression sample; aligning the feature points of the three-dimensional image of the facial expression sample and the two-dimensional image of the facial expression sample with the set position; performing grayscale stretching on the three-dimensional image of the facial expression sample and the two-dimensional image of the facial expression sample; and performing image pixel value normalization on the three-dimensional image of the facial expression sample and the two-dimensional image of the facial expression sample. The second processing described above may be the same as or different from the first processing.

[0329] Performing the same second processing on the three-dimensional image of the facial expression sample and the two-dimensional image of the facial expression sample may include: performing the same second processing on the two-dimensional image of the facial expression sample as the second processing performed on the three-dimensional image of the facial expression sample. Exemplarily, performing the same second processing on the three-dimensional image of the facial expression sample and the two-dimensional image of the facial expression sample may include: performing linear transformation, affine transformation and grayscale stretching on the three-dimensional image of the facial expression sample, and also performing the above-mentioned linear transformation, affine transformation and grayscale stretching on the two-dimensional image of the facial expression sample; for another example, performing mirror processing, linear transformation and image pixel value normalization processing on the three-dimensional image of the facial expression sample, and also performing mirror processing, linear transformation and image pixel value normalization processing on the two-dimensional image of the facial expression sample. Exemplarily, the above-mentioned same second processing of the three-dimensional image of the facial expression sample and the two-dimensional image of the facial expression sample can be: performing the same second processing on the second depth information (such as a depth map) of the facial expression sample, and the three channels of the RGB image of the three-dimensional image of the facial expression sample and the three channels of the RGB image of the two-dimensional image of the facial expression sample, respectively; it can also be performing the same second processing on the overall image of the three-dimensional image of the facial expression sample and the overall image of the two-dimensional image of the facial expression sample, and then decomposing them into the second depth information, the third color information and the fourth color information and inputting them into the first neural network.

[0330] Optionally, the above-mentioned feature points may be eye points, or other facial features such as nose tip points. The set positions for aligning the feature points of the three-dimensional images of the plurality of facial expression samples and the two-dimensional images of the plurality of facial expression samples may be feature points of a certain standard facial image, such as eye points, or may be pre-set positions, or may be feature points uniformly aligned with the facial expression samples input when training the above-mentioned first neural network, such as eye points.

[0331] Optionally, the grayscale stretching of the three-dimensional image of the facial expression sample and the two-dimensional image of the facial expression sample may include segmented grayscale stretching of the three-dimensional image of the facial expression sample and the two-dimensional image of the facial expression sample according to the characteristics of the three-dimensional image of the facial expression sample and / or the two-dimensional image of the facial expression sample. It may also include segmented grayscale stretching of the pixel values ​​according to the size of the pixel values ​​of the three-dimensional image of the facial expression sample and the two-dimensional image of the facial expression sample.

[0332] Optionally, the image pixel value normalization processing of the three-dimensional image of the facial expression sample and the two-dimensional image of the facial expression sample includes: normalizing the pixel value of each channel of the three-dimensional image of the facial expression sample and the two-dimensional image of the facial expression sample from [0, 255] to [0, 1]. The above-mentioned channels may include: the first depth information of the three-dimensional image of the facial expression sample, the three channels of the RGB image of the three-dimensional image of the facial expression sample, and the three channels of the RGB image of the two-dimensional image of the facial expression sample.

[0333] Normally, taking the human face as an example, the three-dimensional image of the facial expression sample and the two-dimensional image of the facial expression sample acquired by the camera device include redundant parts such as the neck and shoulders in addition to the face. It is necessary to locate the face frame position through face detection, then deduct it, and then locate the above-mentioned facial features, such as the eye points, and then perform the above-mentioned second processing.

[0334] The present invention provides a method and device for facial expression recognition, which can effectively solve the problem of decreased accuracy of facial expression recognition caused by different facial postures and different lighting conditions, and improve the accuracy of facial expression recognition of a target face under different facial postures and different lighting conditions.

[0335] Combine the following Figure 2 A method for facial expression recognition provided by the second embodiment of the present invention is specifically described. Figure 2 As shown, the method includes:

[0336] Step 201: Acquire a three-dimensional image of a target face, wherein the three-dimensional image includes third depth information of the target face and fifth color information of the target face.

[0337] Optionally, the acquiring step may be acquiring, from a memory, a three-dimensional image of the target face captured by a camera device.

[0338] Optionally, the three-dimensional image of the target face may be a color image.

[0339] Optionally, the fifth color information is an image in RGB format or YUV format, and may also be an image in other formats that can be converted into the RGB format or YUV format.

[0340] Step 202: Input the third depth information of the target face into the second neural network and input the fifth color information of the target face into the third neural network. Optionally, the input to the third neural network may be an RGB image of the target face or three channels of the RGB image of the target face.

[0341] Optionally, the second neural network includes: 3 convolutional layers, 3 downsampling layers, 1 downsampling layer and 2 fully connected layers. The third neural network includes: 4 convolutional layers, 4 downsampling layers, 1 downsampling layer and 2 fully connected layers.

[0342] Step 203: the second neural network classifies the expression of the target face according to the third depth information and the second parameter of the target face and outputs first classification data, and the third neural network classifies the expression of the target face according to the fifth color information and the third parameter of the target face and outputs second classification data, the second parameter includes at least one facial expression category and second parameter data for identifying the expression category of the target face, and the third parameter includes at least one facial expression category and third parameter data for identifying the expression category of the target face.

[0343] Since most expressions are compound expressions and can belong to at least one expression category, the second neural network includes the first classification data, and the facial expression categories included in the first classification data include at least one of the following: fear, sadness, happiness, anger, disgust, surprise, naturalness, and contempt. The third neural network includes the second classification data, and the facial expression categories included in the second classification data include at least one of the following: fear, sadness, happiness, anger, disgust, surprise, naturalness, and contempt. Optionally, the facial expression categories included in the first classification data and the second classification data are the same. Exemplarily, the first classification data and the second classification data both include eight facial expression categories of fear, sadness, happiness, anger, disgust, surprise, naturalness, and contempt, and eight groups of parameter data corresponding to the eight facial expression categories, and the eight groups of parameter data may include the probability of distribution belonging to the eight facial expression categories. The second parameter data and the third parameter data include second parameter data for identifying whether it belongs to the eight facial expression categories, such as the weight of at least one node of the neural network.

[0344] The second neural network includes a second convolutional neural network, and the third neural network includes a third convolutional neural network.

[0345] Step 204: outputting a classification result of the target facial expression according to the first classification data and the second classification data.

[0346] Optionally, the classification result of the expression of the target face is output based on the first classification data and the second classification data, including: a support vector machine inputs the first classification data and the second classification data, and outputs the classification result of the expression of the target face based on the first classification data, the second classification data and support vector machine parameter data, the support vector machine including the at least one facial expression category and the support vector machine parameter data for identifying the expression category of the target face.

[0347] Exemplarily, the first classification data may be a set of eight-dimensional data, i.e., data representing eight expression categories, and the eight expression categories may be fear, sadness, happiness, anger, disgust, surprise, naturalness, and contempt. Optionally, the data representing the eight expression categories may be eight probability values ​​that the expression of the target face belongs to the eight expression categories, and the sum of the eight probability values ​​is 1. Similarly, if the second classification data is the same eight expression categories, the input of the support vector machine is two sets of eight-dimensional data, and the support vector machine determines which type of expression the expression of the target face belongs to based on the two sets of eight-dimensional data and the support vector machine parameter data for identifying the expression category of the target face. The support vector machine may be a linear support vector machine. The classification result output by the support vector machine may be the probability that the target face belongs to the different expression categories, and the sum of the probabilities of belonging to the different expression categories is 1. The support vector machine may sort the output classification results according to the probabilities.

[0348] Optionally, in order to cope with the situation that the obtained target face posture is not ideal or the lighting condition is not ideal, the third processing can be performed only on the third depth information of the target face, or the third processing can be performed on both the third depth information of the target face and the fifth color information of the target face. Therefore, before inputting the third depth information of the target face into the second neural network and inputting the fifth color information of the target face into the third neural network, the method further includes:

[0349] Performing a third processing on the third depth information of the target face, the third processing comprising at least one of the following: determining feature points of the third depth information of the target face, and rotating the third depth information of the target face based on the feature points; performing mirror processing, linear transformation, and affine transformation on the third depth information of the target face; aligning the feature points of the third depth information of the target face to a set position; performing grayscale stretching on the third depth information of the target face; and performing image pixel value normalization processing on the third depth information of the target face;

[0350] or,

[0351] Before the third depth information of the target face is input into the second neural network and the fifth color information of the target face is input into the third neural network, the method further includes: performing the same third processing on the third depth information of the target face and the fifth color information of the target face, and the third processing includes at least one of the following: determining feature points of the third depth information of the target face and feature points of the fifth color information of the target face, and rotating the third depth information of the target face and the fifth color information of the target face based on the feature points; performing mirror processing, linear transformation and affine transformation on the third depth information of the target face and the fifth color information of the target face; aligning the feature points of the third depth information of the target face and the fifth color information of the target face with a set position; performing grayscale stretching on the third depth information of the target face or the fifth color information of the target face; and performing image pixel value normalization processing on the third depth information of the target face and the fifth color information of the target face.

[0352] The above-mentioned third depth information of the target face and the fifth color information of the target face are subjected to the same third processing, which may include: the fifth color information of the target face is subjected to the same third processing as the third depth information of the target face. Exemplarily, the third depth information of the target face may be subjected to linear transformation, affine transformation and grayscale stretching, and the fifth color information of the target face is also subjected to the same linear transformation, affine transformation and grayscale stretching. For another example, the third depth information of the target face is subjected to mirror processing, linear transformation and image pixel value normalization processing, and the fifth color information of the target face is also subjected to the same mirror processing, linear transformation and image pixel value normalization processing. Optionally, the above-mentioned third depth information of the target face and the fifth color information of the target face are subjected to the same third processing, which may be respectively subjected to the same third processing on the third depth information of the target face (such as a depth map) and the RGB map of the three-dimensional image of the target face; or respectively subjected to the same third processing on the third depth information of the target face and the three channels of the RGB map of the three-dimensional image of the target face.

[0353] Optionally, the above-mentioned feature points may be eye points, or other facial features such as nose tip points. The set positions for aligning the feature points of the third depth information of the target face and the fifth color information of the target face may be feature points of a standard facial image, such as eye points, or may be pre-set positions, or may be feature points uniformly aligned with the facial expression samples input when training the above-mentioned second neural network, and feature points uniformly aligned with the facial expression samples input when training the above-mentioned third neural network, such as eye points. Optionally, the set positions for aligning the feature points of the third depth information of the target face may be feature points of a standard facial image, such as eye points, or may be pre-set positions, or may be feature points uniformly aligned with the facial expression samples input when training the above-mentioned second neural network.

[0354] Optionally, the grayscale stretching of the third depth information of the target face and the fifth color information of the target face may include segmented grayscale stretching of the third depth information of the target face and the fifth color information of the target face according to the characteristics of the three-dimensional image of the target face. It may also include segmented grayscale stretching of the pixel values ​​according to the size of the pixel values ​​of the third depth information of the target face and the fifth color information of the target face.

[0355] Optionally, the image pixel value normalization processing of the third depth information of the target face and the fifth color information of the target face includes: normalizing the pixel values ​​of each channel of the third depth information of the target face and the fifth color information of the target face from [0, 255] to [0, 1]. The above-mentioned channels may include: the third depth information of the target face, and three channels of the RGB image of the three-dimensional image of the target face. The image pixel value normalization processing of the third depth information of the target face includes: normalizing the pixel values ​​of the third depth information of the target face from [0, 255] to [0, 1].

[0356] Normally, taking the face as an example, the three-dimensional image of the target face obtained by the camera device includes not only the face but also redundant parts such as the neck and shoulders. It is necessary to locate the face frame through face detection, then extract it, and then locate the above-mentioned facial features, such as the eye points, and then perform the above-mentioned third processing.

[0357] Optionally, the second parameter data is obtained by training the fourth depth information of a plurality of facial expression samples through the second neural network, and the third parameter data is obtained by training the sixth color information of the plurality of facial expression samples through the third neural network. The three-dimensional image of the facial expression sample includes: the fourth depth information of the facial expression sample and the sixth color information of the facial expression sample. The process of the second neural network training the fourth depth information to obtain the second parameter data and the process of the third neural network training the sixth color information to obtain the third parameter data can be parallel. Specifically, the fourth depth information and the sixth color information of the plurality of facial expression samples can be respectively input into the second neural network and the third neural network and iterated, and the plurality of facial expression samples have facial expression categories representing facial expression categories, and a parameter combination with a high expression accuracy of the facial expression samples is determined, such as the weight of at least one node of the neural network, as the second parameter data and the third parameter data for identifying the expression category of the target face, and the specific contents of the second parameter data and the third parameter data can be referred to above. Optionally, the second parameter data and the third parameter data can be obtained by training the facial expression samples offline, and the facial expression samples provided for actual use may not be included in the product for expression recognition.

[0358] Since most expressions are complex expressions and can belong to at least one expression category, the facial expression category included in the second neural network and the facial expression category included in the third neural network include at least one of the following: fear, sadness, happiness, anger, disgust, surprise, naturalness, and contempt. The facial expression sample, the fourth depth information of the facial expression sample, and the sixth color information of the facial expression sample have the same facial expression category. The above-mentioned sixth color information is an image in RGB format or YUV format. Through the facial expression category of the above-mentioned facial expression sample, the facial expression category of the component of the three-dimensional image of the above-mentioned facial expression sample input into the second neural network and the third neural network (the fourth depth information of the three-dimensional image of the facial expression sample, and the sixth color information of the three-dimensional image of the facial expression sample) can be determined, then the second neural network can be trained to obtain the second parameter data corresponding to the above-mentioned different facial expression categories, and the third neural network can be trained to obtain the third parameter data corresponding to the above-mentioned different facial expression categories.

[0359] Optionally, in order to cope with the situation that the posture of the acquired facial expression samples is not ideal or the lighting conditions are not ideal, the fourth depth information of the facial expression samples may be subjected to a fourth processing, and the fourth depth information of the facial expression samples and the sixth color information of the facial expression samples may be subjected to the same fourth processing to approximately meet the requirements of a standard face or the requirements of use. Specifically, before the fourth depth information of the plurality of facial expression samples is trained by the second neural network, the method further includes:

[0360] Performing a fourth processing on the fourth depth information of the facial expression sample, the fourth processing comprising at least one of the following: determining a feature point of the fourth depth information of the facial expression sample, and rotating the fourth depth information of the facial expression sample based on the feature point; performing mirror processing, linear transformation, and affine transformation on the fourth depth information of the facial expression sample; aligning the feature point of the fourth depth information of the facial expression sample to a set position; performing grayscale stretching on the fourth depth information of the facial expression sample; and performing image pixel value normalization processing on the fourth depth information of the facial expression sample;

[0361] Or, before the fourth depth information of the facial expression sample is trained by the second neural network and the sixth color information of the facial expression sample is trained by the third neural network, the method also includes: performing the same fourth processing on the fourth depth information of the facial expression sample and the sixth color information of the facial expression sample, and the fourth processing includes at least one of the following: determining the feature points of the fourth depth information of the facial expression sample and the feature points of the sixth color information of the facial expression sample, and rotating the fourth depth information of the facial expression sample and the sixth color information of the facial expression sample based on the feature points; performing mirror processing, linear transformation and affine transformation on the fourth depth information of the facial expression sample and the sixth color information of the facial expression sample; aligning the feature points of the fourth depth information of the facial expression sample and the sixth color information of the facial expression sample to a set position; performing grayscale stretching on the fourth depth information of the facial expression sample or the sixth color information of the facial expression sample; and performing image pixel value normalization processing on the fourth depth information of the facial expression sample and the sixth color information of the facial expression sample. The fourth process described above may be the same as or different from the third process.

[0362] Performing the same fourth processing on the fourth depth information of the facial expression sample and the sixth color information of the facial expression sample may include: performing the same fourth processing on the sixth color information of the facial expression sample as the fourth processing performed on the fourth depth of the facial expression sample. Exemplarily, performing the same fourth processing on the fourth depth information of the facial expression sample and the sixth color information of the facial expression sample may include: performing linear transformation, affine transformation and grayscale stretching on the fourth depth information of the facial expression sample, and also performing linear transformation, affine transformation and grayscale stretching on the sixth color information of the facial expression sample; for another example, performing mirror processing, linear transformation and image pixel value normalization processing on the fourth depth information of the facial expression sample, and also performing mirror processing, linear transformation and image pixel value normalization processing on the sixth color information of the facial expression sample. Exemplarily, the above-mentioned performing the same fourth processing on the fourth depth information of the facial expression sample and the sixth color information of the facial expression sample may include: performing the same fourth processing on the fourth depth information of the facial expression sample (such as a depth map) and the three channels of the RGB image of the three-dimensional image of the facial expression sample, respectively; it may also be performing the fourth processing on the overall image of the three-dimensional image of the facial expression sample, and then decomposing it into the fourth depth information of the facial expression sample and the sixth color information of the facial expression sample, and inputting them into the second neural network and the third neural network, respectively.

[0363] Optionally, the above-mentioned feature points may be eye points, or other facial features such as nose tip points. The set positions for aligning the feature points of the fourth depth information of the facial expression sample and the sixth color information of the facial expression sample, or the set positions for aligning the feature points of the fourth depth information of the facial expression sample may be feature points of a certain standard facial image, such as eye points, or may be pre-set positions, or may be feature points uniformly aligned with the facial expression samples input when training the above-mentioned second neural network and third neural network, such as eye points.

[0364] Optionally, the grayscale stretching of the fourth depth information of the facial expression sample, or the grayscale stretching of the fourth depth information of the facial expression sample and the sixth color information of the facial expression sample, may include: segmented grayscale stretching of the fourth depth information of the facial expression sample and the sixth color information of the facial expression sample according to the characteristics of the fourth depth information of the facial expression sample and / or the sixth color information of the facial expression sample. It may also include segmented grayscale stretching of the pixel value according to the size of the pixel value of the fourth depth information of the facial expression sample and the sixth color information of the facial expression sample.

[0365] Optionally, the image pixel value normalization processing of the fourth depth information of the facial expression sample includes: normalizing the pixel value of the fourth depth information of the facial expression sample from [0, 255] to [0, 1]; or, the image pixel normalization processing of the fourth depth information of the facial expression sample and the sixth color information of the facial expression sample includes: normalizing the pixel value of each channel of the fourth depth information of the facial expression sample and the sixth color information of the facial expression sample from [0, 255] to [0, 1]. The above-mentioned channels may include: the fourth depth information of the three-dimensional image of the facial expression sample, and three channels of the RGB image of the sixth color information of the facial expression sample.

[0366] Normally, taking a human face as an example, the three-dimensional image of a facial expression sample acquired by a camera device includes not only the face but also redundant parts such as the neck and shoulders. It is necessary to locate the face frame through face detection, extract it, and then locate the above-mentioned facial features, such as the eye points, and then perform the above-mentioned fourth processing.

[0367] The fifth color information is an image in RGB format or YUV format. The sixth color information is an image in RGB format or YUV format.

[0368] The support vector machine parameter data for identifying the expression category of the target face is obtained by training the support vector machine with the output data of the second neural network when training the fourth depth information of multiple facial expression samples and the output data of the third neural network when training the sixth color information of the multiple facial expression samples. Exemplarily, the output data of the second neural network when training the fourth depth information of multiple facial expression samples can be a set of eight-dimensional data, that is, data representing eight expression categories, and the eight expression categories can be fear, sadness, happiness, anger, disgust, surprise, nature, and contempt. Similarly, the output data of the third neural network when training the sixth color information of the multiple facial expression samples is the same eight expression categories, and the input of the support vector machine is the above two sets of eight-dimensional data. Since the above two sets of eight-dimensional data carry the facial expression category indicating the expression category, the support vector machine data of the facial expression category with the expression category can be trained through the above two sets of eight-dimensional data. The above two sets of eight-dimensional data can be the probability that the facial expression samples belong to different facial expression categories.

[0369] The present invention provides a method and device for facial expression recognition, which can effectively solve the problem of decreased accuracy of facial expression recognition caused by different facial postures and different lighting conditions, and improve the accuracy of facial expression recognition of a target face under different facial postures and different lighting conditions.

[0370] Combine the following Figure 3 A method for facial expression recognition provided by Embodiment 3 of the present invention is specifically described. Figure 3 As shown, the method includes:

[0371] Step 301: Acquire a three-dimensional image of a target face, wherein the three-dimensional image includes fifth depth information of the target face and seventh color information of the target face.

[0372] Optionally, the acquiring step may be acquiring, from a memory, a three-dimensional image of the target face captured by a camera device.

[0373] Optionally, the three-dimensional image of the target face may be a color image.

[0374] Optionally, the seventh color information is an image in RGB format or YUV format, and may also be an image in other formats that can be converted into the above RGB format or YUV format.

[0375] Step 302: Input the fifth depth information of the target face and the seventh color information of the target face into a fourth neural network. Optionally, the input to the fourth neural network may be a depth map of the target face and an RGB map of a three-dimensional image of the target face; the input to the first neural network may also be a depth map of the target face and three channels of an RGB map of a three-dimensional image of the target face.

[0376] Optionally, the fourth neural network includes a fourth convolutional neural network. The fourth convolutional neural network includes 1 segmentation layer, 8 convolutional layers, 8 downsampling layers, 2 downsampling layers and 5 fully connected layers.

[0377] Step 303: The fourth neural network classifies the expression of the target face according to the fifth depth information of the target face, the seventh color information of the target face, and a fourth parameter, wherein the fourth parameter includes at least one facial expression category and fourth parameter data for identifying the expression category of the target face.

[0378] Optionally, since most facial expressions are compound expressions and can belong to at least one expression category, the fourth neural network may include a fourth parameter, and the facial expression category included in the fourth parameter includes at least one of the following: fear, sadness, happiness, anger, disgust, surprise, naturalness, and contempt. Exemplarily, the fourth parameter may include eight facial expression categories of fear, sadness, happiness, anger, disgust, surprise, naturalness, and contempt, and fourth parameter data for identifying the eight facial expression categories, such as the weight of at least one node of the fourth neural network. Specifically, the classification result output by the fourth neural network may be the probability that the target face belongs to the different expression categories, and the sum of the probabilities of belonging to the different expression categories is 1. The fourth neural network may sort the output classification results according to the probabilities.

[0379] Optionally, in the case where the fourth parameter includes a facial expression category, the fourth neural network can be used to determine whether the expression of the target face is the facial expression category included in the fourth parameter.

[0380] Optionally, in order to cope with the situation where the obtained target face posture is not ideal or the lighting conditions are not ideal, the three-dimensional image of the target face can be subjected to fifth processing to approximately meet the requirements of a standard face or the requirements of use. Specifically, before the fifth depth information of the target face and the seventh color information of the target face are input into the fourth neural network, the method also includes: performing fifth processing on the three-dimensional image of the target face, and the fifth processing includes at least one of the following: determining the feature points of the three-dimensional image of the target face, and rotating the three-dimensional image of the target face based on the feature points; performing mirror processing, linear transformation and affine transformation on the three-dimensional image of the target face; aligning the feature points of the three-dimensional image of the target face to a set position; performing grayscale stretching on the three-dimensional image of the target face; and performing image pixel value normalization processing on the three-dimensional image of the target face.

[0381] The above-mentioned fifth processing on the three-dimensional image of the target face may be to perform the same fifth processing on the fifth depth information of the target face and the seventh color information of the target face, that is, the fifth processing performed on the fifth depth information of the target face is the same as the fifth processing performed on the seventh color information of the target face. Exemplarily, the same fifth processing on the fifth depth information of the target face and the seventh color information of the target face may be: linear transformation, affine transformation and grayscale stretching are performed on the fifth depth information of the target face, and linear transformation, affine transformation and grayscale stretching are also performed on the seventh color information of the target face. For another example, the fifth depth information of the target face is mirrored, linearly transformed and image pixel value normalized, and the seventh color information of the target face is mirrored, linearly transformed and image pixel value normalized. Optionally, the above-mentioned fifth processing of the three-dimensional image of the target face can be: performing the same fifth processing on the three channels of the RGB image of the fifth depth information (such as a depth map) of the target face and the seventh color information of the target face respectively; or performing the fifth processing on the entire image of the three-dimensional image of the target face, and then decomposing it into the fifth depth information and the seventh color information to input into the fourth neural network.

[0382] Optionally, the feature points may be eye points, or other facial features such as nose tip points. The positions set for aligning the feature points of the three-dimensional image of the target face may be feature points of a standard facial image, such as eye points, or may be pre-set positions, or may be feature points uniformly aligned with the facial expression samples input when training the fourth neural network, such as eye points.

[0383] Optionally, the above-mentioned grayscale stretching of the three-dimensional image of the target face may include performing segmented grayscale stretching on the three-dimensional image of the target face according to characteristics of the three-dimensional image of the target face, and may also include performing segmented grayscale stretching on pixel values ​​according to sizes of pixel values ​​of the three-dimensional image of the target face.

[0384] Optionally, the normalizing the pixel values ​​of the three-dimensional image of the target face includes normalizing the pixel values ​​of each channel of the three-dimensional image of the target face from [0, 255] to [0, 1]. The above-mentioned channels may include: depth information of the three-dimensional image of the target face, and three channels of the RGB image of the three-dimensional image of the target face.

[0385] Normally, taking the face as an example, the three-dimensional image of the target face obtained by the camera device includes redundant parts such as the neck and shoulders in addition to the face. It is necessary to locate the face frame through face detection, then deduct it, and then locate the above-mentioned facial features, such as the eye points, and then perform the above-mentioned fifth processing.

[0386] Optionally, the fourth parameter data is obtained by training the fourth neural network with a three-dimensional image of a plurality of facial expression samples. The three-dimensional image of the facial expression sample includes: the sixth depth information of the facial expression sample and the eighth color information of the facial expression sample. Specifically, the sixth depth information and the eighth color information of the above-mentioned plurality of facial expression samples can be input into the fourth neural network and iterated. The plurality of facial expression samples carry facial expression categories representing facial expression categories, and a parameter combination with a high expression accuracy rate for identifying facial expression samples is determined, such as the weight of at least one node of the neural network, as the fourth parameter for identifying the expression category of the target face. The specific content of the fourth parameter can be found above. Optionally, the fourth parameter can be obtained by training the above-mentioned facial expression samples offline, and the above-mentioned facial expression samples may not be included in the product provided for actual use for expression recognition.

[0387] Since most facial expressions are complex expressions and can belong to at least one expression category, the facial expression sample has at least one of the following facial expression categories: fear, sadness, happiness, anger, disgust, surprise, naturalness, and contempt. The facial expression sample, the sixth depth information of the facial expression sample, and the eighth color information of the facial expression sample have the same facial expression category. The eighth color information is an image in RGB format or YUV format. Through the facial expression category of the facial expression sample, the facial expression category of the component of the facial expression sample input into the fourth neural network (the sixth depth information of the facial expression sample and the eighth color information of the facial expression sample are components of a three-dimensional image) can be determined, and the fourth neural network can be trained to obtain the fourth parameters corresponding to the above different facial expression categories.

[0388] Optionally, in order to cope with the situation that the posture of the acquired facial expression sample is not ideal or the lighting condition is not ideal, the three-dimensional image of the facial expression sample can be subjected to a sixth processing to approximately meet the requirements of the standard face or the requirements of use. Specifically, before the three-dimensional images of the plurality of facial expression samples are trained by the fourth neural network, the three-dimensional image of the facial expression sample is subjected to a sixth processing, and the sixth processing includes at least one of the following: determining the feature points of the three-dimensional image of the facial expression sample, and rotating the three-dimensional image of the facial expression sample based on the feature points; performing mirror processing, linear transformation and affine transformation on the three-dimensional image of the facial expression sample; aligning the feature points of the three-dimensional image of the facial expression sample to the set position; performing grayscale stretching on the three-dimensional image of the facial expression sample; and performing image pixel value normalization processing on the three-dimensional image of the facial expression sample. The above-mentioned sixth processing may be the same as the fifth processing or different.

[0389] Optionally, performing the sixth processing on the three-dimensional image of the facial expression sample may include: performing the same sixth processing on the sixth depth information and the eighth color information of the facial expression sample, that is, the sixth processing performed on the sixth depth information of the facial expression sample is the same as the sixth processing performed on the eighth color information of the facial expression sample. Exemplarily, the sixth depth information of the facial expression sample may be subjected to linear transformation, affine transformation, and grayscale stretching, and the eighth color information of the facial expression sample may also be subjected to the above-mentioned linear transformation, affine transformation, and grayscale stretching; for another example, the sixth depth information of the facial expression sample may be subjected to mirror processing, linear transformation, and image pixel value normalization processing, and the eighth color information of the facial expression sample may also be subjected to mirror processing, linear transformation, and image pixel value normalization processing. Exemplarily, the above-mentioned same sixth processing of the sixth depth information of the facial expression sample and the eighth color information of the facial expression sample can be: performing the same sixth processing on the third channels of the sixth depth information of the facial expression sample (such as a depth map) and the eighth color information of the three-dimensional image of the facial expression sample, such as an RGB image, respectively; or it can be performing the same sixth processing on the entire image of the three-dimensional image of the facial expression sample, and then decomposing it into the sixth depth information and the eighth color information and inputting them into the fourth neural network.

[0390] Optionally, the feature points may be eye points, or other facial features such as nose points. The positions set for aligning the feature points of the three-dimensional images of the plurality of facial expression samples may be feature points of a certain standard facial image, such as eye points, or may be pre-set positions, or may be feature points uniformly aligned with the facial expression samples input when training the fourth neural network, such as eye points.

[0391] Optionally, the grayscale stretching of the three-dimensional image of the facial expression sample may include segmented grayscale stretching of the three-dimensional image of the facial expression sample according to the characteristics of the three-dimensional image of the facial expression sample, or segmented grayscale stretching of the pixel value according to the size of the pixel value of the three-dimensional image of the facial expression sample.

[0392] Optionally, the normalizing the pixel values ​​of the three-dimensional image of the facial expression sample includes: normalizing the pixel values ​​of each channel of the three-dimensional image of the facial expression sample from [0, 255] to [0, 1]. The above-mentioned channels may include: the sixth depth information of the three-dimensional image of the facial expression sample, the eighth color information of the three-dimensional image of the facial expression sample, such as three channels of an RGB image.

[0393] Normally, taking a human face as an example, the three-dimensional image of a facial expression sample acquired by a camera device includes not only the face but also redundant parts such as the neck and shoulders. It is necessary to locate the face frame through face detection, extract it, and then locate the above-mentioned facial features, such as the eye points, and then perform the above-mentioned sixth processing.

[0394] The present invention provides a method and device for facial expression recognition, which can effectively solve the problem of decreased accuracy of facial expression recognition caused by different facial postures and different lighting conditions, and improve the accuracy of facial expression recognition of a target face under different facial postures and different lighting conditions.

[0395] Combine the following Figure 4 A device for expression recognition provided by a fourth embodiment of the present invention is specifically described. The device 400 may include:

[0396] The first acquisition module 401 is used to acquire a three-dimensional image of a target face and a two-dimensional image of the target face, wherein the three-dimensional image includes first depth information of the target face and first color information of the target face, and the two-dimensional image includes second color information of the target face.

[0397] Optionally, the acquisition module 401 may acquire from a memory a three-dimensional image of the target face and a two-dimensional image of the target face taken by a camera device.

[0398] Optionally, the first color information and the second color information are images in RGB format or YUV format, and may also be images in other formats that can be converted into the RGB format or YUV format.

[0399] The first input module 402 is used to input the first depth information of the target face, the first color information of the target face, and the second color information of the target face into the first neural network. Optionally, the input to the first neural network may be a depth map of the target face, an RGB map of a three-dimensional image of the target face, and an RGB map of a two-dimensional image of the target face; the input to the first neural network may also be a depth map of the target face, three channels of the RGB map of the three-dimensional image of the target face, and three channels of the RGB map of the two-dimensional image of the target face.

[0400] Optionally, the first neural network includes a first convolutional neural network, which includes 4 convolutional layers, 4 downsampling layers, 1 dropout layer and 2 fully connected layers.

[0401] The first neural network 403 is used to classify the expression of the target face according to the first depth information of the target face, the first color information of the target face, the second color information of the target face, and the first parameter, wherein the first parameter includes at least one facial expression category and first parameter data for identifying the expression category of the target face. Since most expressions are compound expressions and can belong to at least one expression category, the first neural network includes the first parameter, and the facial expression category included in the first parameter includes at least one of the following: fear, sadness, happiness, anger, disgust, surprise, naturalness, and contempt. Optionally, in one embodiment, the first parameter includes eight facial expression categories of fear, sadness, happiness, anger, disgust, surprise, naturalness, and contempt, and the first parameter data for identifying the eight facial expression categories, such as the weight of at least one node of the first neural network. Specifically, the classification result output by the first neural network 403 can be the probability that the target face belongs to the different expression categories, and the sum of the probabilities of belonging to the different expression categories is 1. The first neural network 403 can sort the output classification results according to the level of the above-mentioned probability. Optionally, in the case where the first parameter includes a facial expression category, the first neural network can be used to determine whether the expression of the target face is the facial expression category included in the first parameter.

[0402] Optionally, in order to cope with the situation that the obtained target face posture is not ideal or the lighting conditions are not ideal, the same first processing can be performed on the three-dimensional image of the target face and the two-dimensional image of the target face to approximately meet the requirements of the standard face or the requirements of use. Specifically, the device also includes a first processing module, the first processing module is used to perform the same first processing on the three-dimensional image of the target face and the two-dimensional image of the target face, and input the three-dimensional image of the target face and the two-dimensional image of the target face after the first processing into the first input module. The first processing module includes at least one of the following submodules: a first rotation submodule, a first transformation submodule, a first alignment submodule, a first grayscale stretching submodule, and a first normalization processing submodule. The first rotation submodule is used to determine the feature points of the three-dimensional image of the target face and the two-dimensional image of the target face, and rotate the three-dimensional image of the target face and the two-dimensional image of the target face based on the feature points. The first transformation submodule is used to perform mirror processing, linear transformation and affine transformation on the three-dimensional image of the target face and the two-dimensional image of the target face. The first alignment submodule is used to align the feature points of the three-dimensional image of the target face and the two-dimensional image of the target face to a set position. The first grayscale stretching submodule is used to perform grayscale stretching on the three-dimensional image of the target face and the two-dimensional image of the target face. The first normalization processing submodule is used to perform image pixel value normalization processing on the three-dimensional image of the target face and the two-dimensional image of the target face.

[0403] The above-mentioned performing the same first processing on the three-dimensional image of the target face and the two-dimensional image of the target face may include: performing the same first processing on the two-dimensional image of the target face as the first processing performed on the three-dimensional image of the target face. Exemplarily, the above-mentioned first processing module performing the same first processing on the three-dimensional image of the target face and the two-dimensional image of the target face may include: performing linear transformation and affine transformation on the three-dimensional image of the target face through the first transformation submodule and performing grayscale stretching on the three-dimensional image of the target face through the first grayscale stretching submodule, and also performing the same linear transformation and affine transformation on the two-dimensional image of the target face through the first transformation submodule and performing grayscale stretching on the two-dimensional image of the target face through the first grayscale stretching submodule; for another example, performing mirror processing and linear transformation on the three-dimensional image of the target face through the first transformation submodule and performing image pixel value normalization processing through the first normalization processing submodule, and also performing mirror processing and linear transformation on the two-dimensional image of the target face through the first transformation submodule and performing image pixel value normalization processing through the first normalization processing submodule. Optionally, the first processing module can be specifically used to: perform the same first processing on the depth information of the target face (such as a depth map), the three channels of the RGB image of the three-dimensional image of the target face, and the three channels of the RGB image of the two-dimensional image of the target face; or perform the same first processing on the entire image of the three-dimensional image of the target face and the entire image of the two-dimensional image of the target face, and then decompose them into the first depth information of the target face, the first color information of the target face, and the second color information of the target face, and input them into the first neural network.

[0404] Optionally, the feature points may be eye points, or other facial features such as nose tip points. The positions set for aligning the feature points of the three-dimensional image of the target face and the two-dimensional image of the target face may be feature points of a standard facial image, such as eye points, or may be pre-set positions, or may be feature points uniformly aligned with the facial expression samples input when training the first neural network, such as eye points.

[0405] Optionally, the first grayscale stretching submodule can be specifically used to perform segmented grayscale stretching on the three-dimensional image of the target face and the two-dimensional image of the target face according to the characteristics of the three-dimensional image of the target face and / or the two-dimensional image of the target face, and can also be used to perform segmented grayscale stretching on the pixel values ​​according to the size of the pixel values ​​of the three-dimensional image of the target face and the two-dimensional image of the target face.

[0406] Optionally, the first normalization processing submodule can be specifically used to normalize the pixel values ​​of each channel of the three-dimensional image of the target face and the two-dimensional image of the target face from [0, 255] to [0, 1]. The above-mentioned channels may include: depth information of the three-dimensional image of the target face, three channels of the RGB image of the three-dimensional image of the target face, and three channels of the RGB image of the two-dimensional image of the target face.

[0407] Normally, taking the face as an example, the three-dimensional image and the two-dimensional image of the target face acquired by the camera device include redundant parts such as the neck and shoulders in addition to the face. It is necessary to locate the face frame through face detection, then deduct it, and then locate the above-mentioned facial features, such as the eye points, and then perform the above-mentioned first processing.

[0408] Optionally, the first parameter data for identifying the expression category of the target face is obtained by training the first neural network with a three-dimensional image of a plurality of facial expression samples and a two-dimensional image of the facial expression samples. The three-dimensional image of the facial expression sample includes: the second depth information of the facial expression sample and the third color information of the facial expression sample, and the two-dimensional image of the facial expression sample includes: the fourth color information of the facial expression sample. Specifically, the first input module 402 can input the second depth information, the third color information and the fourth color information of the plurality of facial expression samples into the first neural network 403 and iterate, and the plurality of facial expression samples have facial expression categories representing facial expression categories, and the first neural network 403 determines a parameter combination with a high expression accuracy of the facial expression samples, such as the weight of at least one node thereof, as the first parameter for identifying the expression category of the target face, and the specific content of the first parameter can be found above. Optionally, the first parameter can be obtained by training the facial expression samples offline, and the facial expression samples provided for actual use may not be included in the product for expression recognition.

[0409] Since most expressions are complex expressions and can belong to at least one expression category, the facial expression sample has at least one of the following facial expression categories: fear, sadness, happiness, anger, disgust, surprise, naturalness, and contempt. The facial expression sample, the second depth information of the facial expression sample, the third color information of the facial expression sample, and the fourth color information of the facial expression sample have the same facial expression category. The third color information and the fourth color information are images in RGB format or YUV format. The first neural network 403 can determine the facial expression category of the component of the facial expression sample input into the first neural network (the second depth information of the facial expression sample and the third color information of the facial expression sample are components of a three-dimensional image, and the fourth color information of the facial expression sample is a component of a two-dimensional image) through the facial expression category of the facial expression sample, and the first neural network 403 can be trained to obtain the first parameter data corresponding to the above different facial expression categories.

[0410] Optionally, in order to cope with the situation that the posture of the acquired facial expression sample is not ideal or the lighting condition is not ideal, the same second processing can be performed on the three-dimensional image of the facial expression sample and the two-dimensional image of the facial expression sample to approximately meet the requirements of the standard face or the requirements of use. Specifically, the device also includes a second processing module, the second processing module is used to perform the same second processing on the three-dimensional image of the facial expression sample and the two-dimensional image of the facial expression sample, and input the three-dimensional image of the facial expression sample and the two-dimensional image of the facial expression sample after the second processing into the first input module. The second processing module includes a second rotation submodule, a second transformation submodule, a second alignment submodule, a second grayscale stretching submodule, and a second normalization processing submodule. The second rotation submodule is used to determine the feature points of the three-dimensional image of the facial expression sample and the two-dimensional image of the facial expression sample, and rotate the three-dimensional image of the facial expression sample and the two-dimensional image of the facial expression sample based on the feature points. The second transformation submodule is used to perform mirror processing, linear transformation and affine transformation on the three-dimensional image of the facial expression sample and the two-dimensional image of the facial expression sample. The second alignment submodule is used to align the feature points of the three-dimensional image of the facial expression sample and the two-dimensional image of the facial expression sample to a set position. The second grayscale stretching submodule is used to perform grayscale stretching on the three-dimensional image of the facial expression sample and the two-dimensional image of the facial expression sample. The second normalization processing submodule is used to perform image pixel value normalization processing on the three-dimensional image of the facial expression sample and the two-dimensional image of the facial expression sample. The above-mentioned second processing module may be the same as the first processing module or may be different.

[0411] The second processing module can be specifically used to perform the same second processing on the two-dimensional image of the facial expression sample as the second processing on the three-dimensional image of the facial expression sample. Exemplarily, the second processing module can be specifically used to: perform linear transformation and affine transformation on the three-dimensional image of the facial expression sample through the second transformation submodule, and grayscale stretch the three-dimensional image of the facial expression sample through the second grayscale stretching submodule, and also perform the above-mentioned linear transformation and affine transformation on the two-dimensional image of the facial expression sample through the second transformation submodule, and grayscale stretch the two-dimensional image of the facial expression sample through the second grayscale stretching submodule; for another example, the three-dimensional image of the facial expression sample can be mirrored and linearly transformed through the second transformation submodule, and the image pixel values ​​of the three-dimensional image of the facial expression sample can be normalized through the second normalization processing submodule, and the two-dimensional image of the facial expression sample can be mirrored and linearly transformed through the second transformation submodule, and the image pixel values ​​of the two-dimensional image of the facial expression sample can be normalized through the second normalization processing submodule. Exemplarily, the above-mentioned second processing module can be specifically used to: perform the same second processing on the second depth information (such as a depth map) of the facial expression sample, the three channels of the RGB image of the three-dimensional image of the facial expression sample, and the three channels of the RGB image of the two-dimensional image of the facial expression sample; it can also be used to perform the same second processing on the overall image of the three-dimensional image of the facial expression sample and the overall image of the two-dimensional image of the facial expression sample, and then decompose them into the second depth information, the third color information and the fourth color information and input them into the first neural network.

[0412] Optionally, the above-mentioned feature points may be eye points, or other facial features such as nose tip points. The set positions for aligning the feature points of the three-dimensional images of the plurality of facial expression samples and the two-dimensional images of the plurality of facial expression samples may be feature points of a certain standard facial image, such as eye points, or may be pre-set positions, or may be feature points uniformly aligned with the facial expression samples input when training the above-mentioned first neural network, such as eye points.

[0413] Optionally, the second grayscale stretching submodule can be used to perform segmented grayscale stretching on the three-dimensional image of the facial expression sample and the two-dimensional image of the facial expression sample according to the characteristics of the three-dimensional image of the facial expression sample and / or the two-dimensional image of the facial expression sample. It can also include segmented grayscale stretching on the pixel values ​​according to the size of the pixel values ​​of the three-dimensional image of the facial expression sample and the two-dimensional image of the facial expression sample.

[0414] Optionally, the second normalization processing submodule can be specifically used to: normalize the pixel values ​​of each channel of the three-dimensional image of the facial expression sample and the two-dimensional image of the facial expression sample from [0, 255] to [0, 1]. The above-mentioned channels can include: the first depth information of the three-dimensional image of the facial expression sample, the three channels of the RGB image of the three-dimensional image of the facial expression sample, and the three channels of the RGB image of the two-dimensional image of the facial expression sample.

[0415] Normally, taking the human face as an example, the three-dimensional image of the facial expression sample and the two-dimensional image of the facial expression sample acquired by the camera device include redundant parts such as the neck and shoulders in addition to the face. It is necessary to locate the face frame position through face detection, then deduct it, and then locate the above-mentioned facial features, such as the eye points, and then perform the above-mentioned second processing.

[0416] The present invention provides a method and device for facial expression recognition, which can effectively solve the problem of decreased accuracy of facial expression recognition caused by different facial postures and different lighting conditions, and improve the accuracy of facial expression recognition of a target face under different facial postures and different lighting conditions.

[0417] Combine the following Figure 5 A device for facial expression recognition provided by Embodiment 5 of the present invention is specifically described. Figure 5 As shown, the device 500 includes a second acquisition module 501, a second input module 502, a second neural network 503, a third neural network 504 and a second classification module 505.

[0418] The second acquisition module 501 is used to acquire a three-dimensional image of the target face, wherein the three-dimensional image includes the third depth information of the target face and the fifth color information of the target face. Optionally, the three-dimensional image of the target face may be a color image. Optionally, the fifth color information may be an image in RGB format or YUV format, or may be an image in other formats that can be converted to and from the RGB format or YUV format. Optionally, the second acquisition module 501 may be used to acquire the three-dimensional image of the target face captured by the camera device from a memory.

[0419] The second input module 502 is used to input the third depth information of the target face into the second neural network 503 and input the fifth color information of the target face into the third neural network 504.

[0420] Optionally, the second neural network 503 includes: 3 convolutional layers, 3 downsampling layers, 1 downsampling layer and 2 fully connected layers. The third neural network 504 includes: 4 convolutional layers, 4 downsampling layers, 1 downsampling layer and 2 fully connected layers.

[0421] The second neural network 503 is used to classify the expression of the target face according to the third depth information of the target face and the second parameter and output first classification data, and the third neural network 504 is used to classify the expression of the target face according to the fifth color information of the target face and the third parameter and output second classification data, the second parameter includes at least one facial expression category and second parameter data for identifying the expression category of the target face, and the third parameter includes at least one facial expression category and third parameter data for identifying the expression category of the target face.

[0422] Since most expressions are complex expressions and can belong to at least one expression category, the second neural network includes the first classification data, and the facial expression categories included in the first classification data include at least one of the following: fear, sadness, happiness, anger, disgust, surprise, naturalness, and contempt. The third neural network includes the second classification data, and the facial expression categories included in the second classification data include at least one of the following: fear, sadness, happiness, anger, disgust, surprise, naturalness, and contempt. Optionally, the first classification data and the second classification data include the same facial expression categories. Exemplarily, the first classification data and the second classification data both include eight facial expression categories of fear, sadness, happiness, anger, disgust, surprise, naturalness, and contempt, and eight groups of parameter data corresponding to the eight facial expression categories, such as the probability that the expression of the target face belongs to the eight facial expression categories. The second parameter data and the third parameter data are used to identify which of the eight facial expression categories the expression of the target face belongs to, such as the weight of at least one node of the second neural network and the weight of at least one node of the third neural network.

[0423] The second neural network includes a second convolutional neural network, and the third neural network includes a third convolutional neural network.

[0424] The second classification module 505 is used to output a classification result of the target facial expression according to the first classification data and the second classification data.

[0425] Optionally, the second classification module 505 includes a support vector machine, which can be used to: input the first classification data and the second classification data, and output the classification result of the expression of the target face according to the first classification data, the second classification data and the support vector machine parameter data, the support vector machine including the at least one facial expression category and the support vector machine parameter data for identifying the expression category of the target face.

[0426] Exemplarily, the first classification data may be a set of eight-dimensional data, i.e., data representing eight expression categories, and the eight expression categories may be fear, sadness, happiness, anger, disgust, surprise, naturalness, and contempt. Optionally, the data representing the eight expression categories may be eight probability values ​​that the expression of the target face belongs to the eight expression categories, and the sum of the eight probability values ​​is 1. Similarly, if the second classification data is the same eight expression categories, the input of the support vector machine is two sets of eight-dimensional data, and the support vector machine determines which type of expression the expression of the target face belongs to based on the two sets of eight-dimensional data and the support vector machine parameter data for identifying the expression category of the target face. The support vector machine may be a linear support vector machine. The classification result output by the support vector machine may be the probability that the target face belongs to the different expression categories, and the sum of the probabilities of belonging to the different expression categories is 1. The support vector machine may sort the output classification results according to the probabilities.

[0427] Optionally, when the first classification data and the second classification data include a facial expression category, the support vector machine also includes a facial expression category, and the support vector machine can be used to determine whether the expression of the target face is the facial expression category included in the support vector machine.

[0428] Optionally, in order to cope with the situation that the obtained target face posture is not ideal or the lighting conditions are not ideal, the device further includes a third processing module, the third processing module is used to perform a third processing on the third depth information of the target face, and input the third depth information of the target face after the third processing into the second input module. The third processing module includes at least one of a third rotation submodule, a third transformation submodule, a third alignment submodule, a third grayscale stretching submodule, and a third normalization processing submodule. The third rotation submodule is used to determine the feature points of the third depth information of the target face, and rotate the third depth information of the target face based on the feature points. The third transformation submodule is used to perform mirror processing, linear transformation and affine transformation on the third depth information of the target face. The third alignment submodule is used to align the feature points of the third depth information of the target face to a set position. The third grayscale stretching submodule is used to perform grayscale stretching of the third depth information of the target face. The third normalization processing submodule is used to perform image pixel value normalization processing on the third depth information of the target face.

[0429] The third processing module is further used to perform the same third processing on the third depth information of the target face and the fifth color information of the target face, and input the third depth information of the target face and the fifth color information of the target face after the third processing into the second input module. The third rotation submodule is further used to determine the feature points of the third depth information of the target face and the feature points of the fifth color information of the target face, and rotate the third depth information of the target face and the fifth color information of the target face based on the feature points. The third transformation submodule is further used to perform mirror processing, linear transformation and affine transformation on the third depth information of the target face and the fifth color information of the target face. The third alignment submodule is further used to align the feature points of the third depth information of the target face and the fifth color information of the target face to a set position. The third grayscale stretching submodule is further used to perform grayscale stretching on the third depth information of the target face or the fifth color information of the target face. The third normalization processing submodule is further used to perform image pixel value normalization processing on the third depth information of the target face and the fifth color information of the target face.

[0430] The third processing module mentioned above can be specifically used to: perform the same processing on the fifth color information of the target face as the third processing performed on the third depth information of the target face. Exemplarily, the third processing module can perform linear transformation and affine transformation on the third depth information of the target face through the third transformation submodule, and grayscale stretch the third depth information of the target face through the third grayscale stretching submodule, and then the third transformation submodule can also perform the same linear transformation and affine transformation on the fifth color information of the target face, and grayscale stretch the fifth color information of the target face through the third grayscale stretching submodule. For another example, the third processing module can perform mirror processing and linear transformation on the third depth information of the target face through the third transformation submodule, and perform image pixel value normalization processing on the third depth information of the target face through the third normalization processing submodule, and then the third transformation submodule can also perform the same mirror processing and linear transformation on the fifth color information of the target face, and grayscale stretch the fifth color information of the target face through the third normalization processing submodule. Optionally, the above-mentioned third processing module may be to perform the same third processing on the third depth information (such as a depth map) of the target face and the RGB map of the three-dimensional image of the target face respectively; or may be to perform the same third processing on the third depth information of the target face and the three channels of the RGB map of the three-dimensional image of the target face respectively.

[0431] Optionally, the above-mentioned feature points may be eye points, or other facial features such as nose tip points. The set positions for aligning the feature points of the third depth information of the target face and the fifth color information of the target face may be feature points of a standard facial image, such as eye points, or may be pre-set positions, or may be feature points uniformly aligned with the facial expression samples input when training the above-mentioned second neural network, and feature points uniformly aligned with the facial expression samples input when training the above-mentioned third neural network, such as eye points. Optionally, the set positions for aligning the feature points of the third depth information of the target face may be feature points of a standard facial image, such as eye points, or may be pre-set positions, or may be feature points uniformly aligned with the facial expression samples input when training the above-mentioned second neural network.

[0432] Optionally, the above-mentioned third grayscale stretching submodule can be specifically used to perform segmented grayscale stretching on the third depth information of the target face and the fifth color information of the target face according to the characteristics of the three-dimensional image of the target face, and can also be used to perform segmented grayscale stretching on the pixel values ​​according to the size of the pixel values ​​of the third depth information of the target face and the fifth color information of the target face.

[0433] Optionally, the third normalization processing submodule can be specifically used to: normalize the pixel values ​​of each channel of the third depth information of the target face and the fifth color information of the target face from [0, 255] to [0, 1]. The above-mentioned channels may include: the third depth information of the target face, and three channels of the RGB image of the three-dimensional image of the target face. The third normalization processing submodule is specifically used to: normalize the pixel values ​​of the third depth information of the target face from [0, 255] to [0, 1].

[0434] Normally, taking the face as an example, the three-dimensional image of the target face acquired by the camera device includes not only the face but also redundant parts such as the neck and shoulders. It is necessary to locate the face frame through face detection, then extract it, and then locate the above-mentioned facial features, such as the eye points, and then perform the third processing through the above-mentioned third processing module.

[0435] Optionally, the second parameter data is obtained by training the fourth depth information of multiple facial expression samples through the second neural network, and the third parameter data is obtained by training the sixth color information of the multiple facial expression samples through the third neural network. The three-dimensional image of the facial expression sample includes: the fourth depth information of the facial expression sample and the sixth color information of the facial expression sample. The process of the second neural network training the fourth depth information to obtain the second parameter data and the process of the third neural network training the sixth color information to obtain the third parameter data can be parallel. Specifically, the second input module 502 can input the fourth depth information and the sixth color information of the above-mentioned multiple facial expression samples into the above-mentioned second neural network and the third neural network respectively and iterate, the multiple facial expression samples have facial expression categories representing facial expression categories, determine a parameter combination with a high expression accuracy of the facial expression sample, such as the weight of at least one node of the neural network, as the second parameter data and the third parameter data for identifying the expression category of the target face, and the specific contents of the second parameter data and the third parameter data can be referred to above. Optionally, the second parameter data and the third parameter data may be obtained by offline training with the facial expression samples, and the facial expression samples may not be included in the product provided for actual use for expression recognition.

[0436] Since most expressions are complex expressions and can belong to at least one expression category, the facial expression category included in the second neural network and the facial expression category included in the third neural network include at least one of the following: fear, sadness, happiness, anger, disgust, surprise, naturalness, and contempt. The facial expression sample, the fourth depth information of the facial expression sample, and the sixth color information of the facial expression sample have the same facial expression category. The above-mentioned sixth color information is an image in RGB format or YUV format. Through the facial expression category of the above-mentioned facial expression sample, the second neural network and the third neural network can determine the facial expression category of the component of the three-dimensional image of the above-mentioned facial expression sample input into the second neural network and the third neural network (the fourth depth information of the three-dimensional image of the facial expression sample, and the sixth color information of the three-dimensional image of the facial expression sample), then the second neural network can be trained to obtain the second parameter data corresponding to the above-mentioned different facial expression categories, and the third neural network can be trained to obtain the third parameter data corresponding to the above-mentioned different facial expression categories.

[0437] Optionally, in order to cope with the situation that the posture of the acquired facial expression sample is not ideal or the lighting condition is not ideal, the device includes a fourth processing module, the fourth processing module is used to perform a fourth processing on the fourth depth information of the facial expression sample, and input the fourth depth information of the facial expression sample after the fourth processing into the second input module. The fourth processing module includes at least one of a fourth rotation submodule, a fourth transformation submodule, a fourth alignment submodule, a fourth grayscale stretching submodule, and a fourth normalization processing submodule. The fourth rotation submodule is used to determine the feature points of the fourth depth information of the facial expression sample, and rotate the fourth depth information of the facial expression sample based on the feature points. The fourth transformation submodule is used to perform mirror processing, linear transformation and affine transformation on the fourth depth information of the facial expression sample. The fourth alignment submodule is used to align the feature points of the fourth depth information of the facial expression sample to a set position. The fourth grayscale stretching submodule is used to perform grayscale stretching of the fourth depth information of the facial expression sample. The fourth normalization processing submodule is used to perform image pixel value normalization processing on the fourth depth information of the facial expression sample.

[0438] The fourth processing module is further used to perform a fourth processing on the fourth depth information of the facial expression sample and the sixth color information of the facial expression sample, and input the fourth depth information of the facial expression sample and the sixth color information of the facial expression sample after the fourth processing into the second input module. The fourth rotation submodule is further used to determine the feature points of the fourth depth information of the facial expression sample and the feature points of the sixth color information of the facial expression sample, and rotate the fourth depth information of the facial expression sample and the sixth color information of the facial expression sample based on the feature points. The fourth transformation submodule is further used to perform mirror processing, linear transformation and affine transformation on the fourth depth information of the facial expression sample and the sixth color information of the facial expression sample. The fourth alignment submodule is further used to align the feature points of the fourth depth information of the facial expression sample and the sixth color information of the facial expression sample to a set position. The fourth grayscale stretching submodule is further used to perform grayscale stretching of the fourth depth information of the facial expression sample or the sixth color information of the facial expression sample. The fourth normalization processing submodule is further used to perform image pixel value normalization processing on the fourth depth information of the facial expression sample and the sixth color information of the facial expression sample. The fourth processing module mentioned above may be the same as or different from the third processing module.

[0439] The fourth processing module can be specifically used to: perform the fourth processing on the sixth color information of the facial expression sample in the same manner as the fourth processing performed on the fourth depth of the facial expression sample. Exemplarily, the fourth processing module can specifically perform linear transformation and affine transformation on the fourth depth information of the facial expression sample through the fourth transformation submodule, and grayscale stretch the fourth depth information of the facial expression sample through the fourth grayscale stretching submodule, and also perform linear transformation and affine transformation on the sixth color information of the facial expression sample through the fourth transformation submodule, and grayscale stretch the sixth color information of the facial expression sample through the fourth grayscale stretching submodule. For another example, the fourth depth information of the facial expression sample is mirrored and linearly transformed through the fourth transformation submodule, and the image pixel value of the fourth depth information of the facial expression sample is normalized through the fourth normalization processing submodule, and the sixth color information of the facial expression sample is mirrored and linearly transformed through the fourth transformation submodule, and the image pixel value of the sixth color information of the facial expression sample is normalized through the fourth normalization processing submodule. Exemplarily, the fourth processing module may be specifically used to: perform the same fourth processing on the fourth depth information (such as a depth map) of the facial expression sample and the three channels of the RGB image of the three-dimensional image of the facial expression sample; or perform the fourth processing on the overall image of the three-dimensional image of the facial expression sample, and then decompose it into the fourth depth information of the facial expression sample and the sixth color information of the facial expression sample, and input them into the second neural network and the third neural network through the second input module 502 respectively.

[0440] Optionally, the above-mentioned feature points may be eye points, or other facial features such as nose tip points. The set positions for aligning the feature points of the fourth depth information of the facial expression sample and the sixth color information of the facial expression sample, or the set positions for aligning the feature points of the fourth depth information of the facial expression sample may be feature points of a certain standard facial image, such as eye points, or may be pre-set positions, or may be feature points uniformly aligned with the facial expression samples input when training the above-mentioned second neural network and third neural network, such as eye points.

[0441] Optionally, the fourth grayscale stretching submodule can be specifically used to: perform segmented grayscale stretching on the fourth depth information of the facial expression sample and the sixth color information of the facial expression sample according to characteristics of the fourth depth information of the facial expression sample and / or the sixth color information of the facial expression sample; and can also be used to perform segmented grayscale stretching on the pixel value according to the size of the pixel value of the fourth depth information of the facial expression sample and the sixth color information of the facial expression sample.

[0442] Optionally, the fourth normalization processing submodule can be specifically used to: normalize the pixel value of the fourth depth information of the facial expression sample from [0, 255] to [0, 1]; or, the fourth normalization processing submodule can be specifically used to: normalize the pixel values ​​of each channel of the fourth depth information of the facial expression sample and the sixth color information of the facial expression sample from [0, 255] to [0, 1]. The above-mentioned channels may include: the fourth depth information of the three-dimensional image of the facial expression sample, and three channels of the RGB image of the sixth color information of the facial expression sample.

[0443] Normally, taking a human face as an example, the three-dimensional image of the facial expression sample acquired by the camera device includes redundant parts such as the neck and shoulders in addition to the face. It is necessary to locate the face frame through face detection, then extract it, and then locate the above-mentioned facial features, such as the eye points, and then perform the above-mentioned fourth processing through the above-mentioned fourth processing module.

[0444] The fifth color information is an image in RGB format or YUV format. The sixth color information is an image in RGB format or YUV format.

[0445] The support vector machine parameter data for identifying the expression category of the target face is obtained by training the support vector machine with the output data of the second neural network when training the fourth depth information of multiple facial expression samples and the output data of the third neural network when training the sixth color information of the multiple facial expression samples. Exemplarily, the output data of the second neural network when training the fourth depth information of multiple facial expression samples can be a set of eight-dimensional data, that is, data representing eight expression categories, and the eight expression categories can be fear, sadness, happiness, anger, disgust, surprise, naturalness, and contempt. Similarly, the output data of the third neural network when training the sixth color information of the multiple facial expression samples is the same eight expression categories, and the input of the support vector machine is the above two sets of eight-dimensional data. Since the above two sets of eight-dimensional data carry facial expression categories that indicate expression categories, support vector machine data with facial expression categories of the expression categories can be trained through the above two sets of eight-dimensional data.

[0446] The present invention provides a method and device for facial expression recognition, which can effectively solve the problem of decreased accuracy of facial expression recognition caused by different facial postures and different lighting conditions, and improve the accuracy of facial expression recognition of a target face under different facial postures and different lighting conditions.

[0447] Combine the following Figure 6The sixth embodiment of the present invention provides a device for facial expression recognition. The device 600 includes: a third acquisition module 601 , a third input module 602 and a fourth neural network 603 .

[0448] The third acquisition module 601 is used to acquire a three-dimensional image of a target face, where the three-dimensional image includes fifth depth information of the target face and seventh color information of the target face.

[0449] Optionally, the third acquisition module 601 may be configured to acquire a three-dimensional image of the target face captured by the camera device from a memory. Optionally, the three-dimensional image of the target face may be a color image. Optionally, the seventh color information may be an image in RGB format or YUV format, or may be an image in other formats that can be converted to or from the RGB format or YUV format.

[0450] The third input module 602 is used to input the fifth depth information of the target face and the seventh color information of the target face into the fourth neural network. Optionally, the input to the fourth neural network may be a depth map of the target face and an RGB map of a three-dimensional image of the target face; the input to the first neural network may also be a depth map of the target face and three channels of the RGB map of the three-dimensional image of the target face. Optionally, the fourth neural network includes a fourth convolutional neural network. The fourth convolutional neural network includes 1 segmentation layer, 8 convolutional layers, 8 downsampling layers, 2 downsampling layers and 5 fully connected layers.

[0451] The fourth neural network 603 is used to classify the expression of the target face according to the fifth depth information of the target face, the seventh color information of the target face and the fourth parameter, wherein the fourth parameter includes at least one facial expression category and fourth parameter data for identifying the expression category of the target face.

[0452] Optionally, since most facial expressions are compound expressions and can belong to at least one expression category, the fourth neural network may include a fourth parameter, and the facial expression category included in the fourth parameter includes at least one of the following: fear, sadness, happiness, anger, disgust, surprise, naturalness, and contempt. Exemplarily, the fourth parameter may include eight facial expression categories of fear, sadness, happiness, anger, disgust, surprise, naturalness, and contempt, and fourth parameter data for identifying the eight facial expression categories, such as the weight of at least one node of the neural network. Specifically, the classification result output by the fourth neural network 603 may be the probability that the target face belongs to the different expression categories, and the sum of the probabilities of belonging to the different expression categories is 1. The fourth neural network 603 may sort the output classification results according to the probabilities.

[0453] Optionally, in the case where the fourth parameter includes a facial expression category, the fourth neural network can be used to determine whether the expression of the target face is the facial expression category included in the fourth parameter.

[0454] Optionally, in order to cope with the situation that the obtained target face posture is not ideal or the lighting conditions are not ideal, the three-dimensional image of the target face can be processed to approximately meet the requirements of the standard face or the requirements of use. Specifically, the device also includes a fifth processing module, the fifth processing module is used to perform a fifth processing on the three-dimensional image of the target face, and input the three-dimensional image of the target face after the fifth processing into the third input module. The fifth processing module includes at least one of the following submodules: a fifth rotation submodule, a fifth transformation submodule, a fifth alignment submodule, a fifth grayscale stretching submodule, and a fifth normalization processing submodule. The fifth rotation submodule is used to determine the feature points of the three-dimensional image of the target face, and rotate the three-dimensional image of the target face based on the feature points. The fifth transformation submodule is used to perform mirror processing, linear transformation and affine transformation on the three-dimensional image of the target face. The fifth alignment submodule is used to align the feature points of the three-dimensional image of the target face to a set position. The fifth grayscale stretching submodule is used to perform grayscale stretching on the three-dimensional image of the target face. The fifth normalization processing submodule is used to perform image pixel value normalization processing on the three-dimensional image of the target face.

[0455] The fifth processing module mentioned above can be used to perform the same fifth processing on the fifth depth information of the target face and the seventh color information of the target face, that is, the fifth processing performed on the fifth depth information of the target face is the same as the fifth processing performed on the seventh color information of the target face. Exemplarily, the fifth processing module mentioned above can be used to perform linear transformation and affine transformation on the fifth depth information of the target face through the fifth transformation submodule, and grayscale stretch the fifth depth information of the target face through the fifth grayscale stretching submodule, and also perform linear transformation and affine transformation on the seventh color information of the target face through the fifth transformation submodule, and grayscale stretch the seventh color information of the target face through the fifth grayscale stretching submodule. For another example, the fifth depth information of the target face is mirrored and linearly transformed through the fifth transformation submodule, and the image pixel value of the fifth depth information of the target face is normalized through the fifth normalization processing submodule, and the seventh color information of the target face is mirrored and linearly transformed through the fifth transformation submodule, and the image pixel value of the seventh color information of the target face is normalized through the fifth normalization processing submodule. Optionally, the fifth processing module can be specifically used to perform the same fifth processing on the third channels of the RGB image of the fifth depth information (such as a depth map) of the target face and the seventh color information of the target face, and can also be used to perform the fifth processing on the overall image of the three-dimensional image of the target face, and then decompose it into the fifth depth information and the seventh color information and input it into the fourth neural network through the second input module 502.

[0456] Optionally, the feature points may be eye points, or other facial features such as nose tip points. The positions set for aligning the feature points of the three-dimensional image of the target face may be feature points of a standard facial image, such as eye points, or may be pre-set positions, or may be feature points uniformly aligned with the facial expression samples input when training the fourth neural network, such as eye points.

[0457] Optionally, the fifth grayscale stretching submodule can be specifically used to perform segmented grayscale stretching on the three-dimensional image of the target face according to the characteristics of the three-dimensional image of the target face, and can also be used to perform segmented grayscale stretching on the pixel values ​​according to the size of the pixel values ​​of the three-dimensional image of the target face.

[0458] Optionally, the fifth normalization processing submodule may be specifically used to normalize the pixel values ​​of each channel of the three-dimensional image of the target face from [0, 255] to [0, 1]. The above-mentioned channels may include: depth information of the three-dimensional image of the target face, and three channels of the RGB image of the three-dimensional image of the target face.

[0459] Normally, taking a human face as an example, the three-dimensional image of the target face acquired by the camera device includes redundant parts such as the neck and shoulders in addition to the face. It is necessary to locate the face frame through face detection, then extract it, and then locate the above-mentioned facial features, such as the eye points, and then the above-mentioned fifth processing can be performed through the fifth processing module.

[0460] Optionally, the fourth parameter data is obtained by training the fourth neural network with a three-dimensional image of a plurality of facial expression samples. The three-dimensional image of the facial expression sample includes: the sixth depth information of the facial expression sample and the eighth color information of the facial expression sample. Specifically, the sixth depth information and the eighth color information of the plurality of facial expression samples can be input into the fourth neural network and iterated. The plurality of facial expression samples carry facial expression categories representing facial expression categories. The fourth neural network can determine a parameter combination with a high expression accuracy rate for identifying facial expression samples, such as the weight of at least one node of the neural network, as a fourth parameter for identifying the expression category of the target face. The specific content of the fourth parameter can be found above. Optionally, the fourth parameter can be obtained by training the facial expression samples offline, and the facial expression samples provided for actual use may not be included in the product for expression recognition.

[0461] Since most facial expressions are complex expressions and can belong to at least one expression category, the facial expression sample has at least one of the following facial expression categories: fear, sadness, happiness, anger, disgust, surprise, naturalness, and contempt. The facial expression sample, the sixth depth information of the facial expression sample, and the eighth color information of the facial expression sample have the same facial expression category. The eighth color information is an image in RGB format or YUV format. Through the facial expression category of the facial expression sample, the fourth neural network can determine the facial expression category of the component of the input facial expression sample (the sixth depth information of the facial expression sample and the eighth color information of the facial expression sample are components of a three-dimensional image), and the fourth neural network can be trained to obtain the fourth parameters corresponding to the above different facial expression categories.

[0462] Optionally, in order to cope with the situation that the posture of the acquired facial expression sample is not ideal or the lighting condition is not ideal, the three-dimensional image of the facial expression sample can be processed to approximately meet the requirements of the standard face or the requirements of use. Specifically, the device further includes a sixth processing module, the sixth processing module is used to perform the fifth processing on the three-dimensional image of the facial expression sample, and input the three-dimensional image of the facial expression sample after the fifth processing into the third input module. The sixth processing module includes a sixth rotation submodule, a sixth transformation submodule, a sixth alignment submodule, a sixth grayscale stretching submodule, and a sixth normalization processing submodule. The sixth rotation submodule is used to determine the feature points of the three-dimensional image of the facial expression sample, and rotate the three-dimensional image of the facial expression sample based on the feature points. The sixth transformation submodule is used to perform mirror processing, linear transformation and affine transformation on the three-dimensional image of the facial expression sample. The sixth alignment submodule is used to align the feature points of the three-dimensional image of the facial expression sample to the set position. The sixth grayscale stretching submodule is used to perform grayscale stretching on the three-dimensional image of the facial expression sample. The sixth normalization processing submodule is used to perform image pixel value normalization processing on the three-dimensional image of the facial expression sample. The sixth processing may be the same as the fifth processing or may be different.

[0463] Optionally, the sixth processing module can be specifically used to: perform the same sixth processing on the sixth depth information and the eighth color information of the facial expression sample, that is, the same sixth processing is performed on the eighth color information of the facial expression sample as the sixth processing is performed on the sixth depth information of the facial expression sample. Exemplarily, the sixth processing module can perform linear transformation and affine transformation on the sixth depth information of the facial expression sample through the sixth transformation submodule, and grayscale stretch the sixth depth information of the facial expression sample through the sixth grayscale stretching submodule, and also perform the above-mentioned linear transformation and affine transformation on the eighth color information of the facial expression sample through the sixth transformation submodule, and grayscale stretch the eighth color information of the facial expression sample through the sixth grayscale stretching submodule; for another example, the sixth depth information of the facial expression sample is mirrored and linearly transformed through the sixth transformation submodule, and the sixth depth information of the facial expression sample is normalized by the sixth normalization processing submodule, and the eighth color information of the facial expression sample is mirrored and linearly transformed through the sixth transformation submodule, and the image pixel value is normalized by the sixth normalization processing submodule. Exemplarily, the above-mentioned sixth processing module can be specifically used to: perform the same sixth processing on the three channels of the sixth depth information (such as a depth map) of the facial expression sample and the eighth color information of the three-dimensional image of the facial expression sample, such as an RGB image; it can also be used to perform the same sixth processing on the overall image of the three-dimensional image of the facial expression sample, and then decompose it into the sixth depth information and the eighth color information to input into the fourth neural network.

[0464] Optionally, the feature points may be eye points, or other facial features such as nose points. The positions set for aligning the feature points of the three-dimensional images of the plurality of facial expression samples may be feature points of a certain standard facial image, such as eye points, or may be pre-set positions, or may be feature points uniformly aligned with the facial expression samples input when training the fourth neural network, such as eye points.

[0465] Optionally, the sixth grayscale stretching submodule can be specifically used to perform segmented grayscale stretching on the three-dimensional image of the facial expression sample according to the characteristics of the three-dimensional image of the facial expression sample, and can also be used to perform segmented grayscale stretching on the pixel values ​​according to the size of the pixel values ​​of the three-dimensional image of the facial expression sample.

[0466] Optionally, the sixth normalization processing submodule is specifically used to normalize the pixel values ​​of each channel of the three-dimensional image of the facial expression sample from [0, 255] to [0, 1]. The above-mentioned channels may include: the sixth depth information of the three-dimensional image of the facial expression sample, the eighth color information of the three-dimensional image of the facial expression sample, such as three channels of the RGB image.

[0467] Normally, taking a human face as an example, the three-dimensional image of the facial expression sample acquired by the camera device includes not only the face but also redundant parts such as the neck and shoulders. It is necessary to locate the face frame through face detection, then extract it, and then locate the above-mentioned facial features, such as the eye points, and then perform the above-mentioned sixth processing through the above-mentioned sixth processing module.

[0468] The present invention provides a method and device for facial expression recognition, which can effectively solve the problem of decreased accuracy of facial expression recognition caused by different facial postures and different lighting conditions, and improve the accuracy of facial expression recognition of a target face under different facial postures and different lighting conditions.

[0469] Combine the following Figure 7 A computer-readable storage medium 700 provided in an embodiment of the present invention is specifically described, wherein the computer-readable storage medium 700 stores a computer program, and is characterized in that when the computer program is executed by the first processor 701, the steps of the method described in any of the above embodiments 1-3 are implemented.

[0470] The computer-readable storage medium 700 provided by the present invention can effectively solve the problem of decreased accuracy in facial expression recognition caused by different facial postures and different lighting conditions, and improve the accuracy of facial expression recognition of the target face under different facial postures and different lighting conditions.

[0471] Combine the following Figure 8 An apparatus 800 for facial expression recognition provided by an embodiment of the present invention is specifically described. The apparatus 800 includes a memory 801, a second processor 802, and a computer program stored in the memory 801 and executable on the second processor 802. The computer program, when executed by the second processor 802, implements the steps of the method described in any one of embodiments 1-3.

[0472] The device 800 for facial expression recognition provided by the present invention can effectively solve the problem of decreased accuracy of facial expression recognition caused by different facial postures and different lighting conditions, and improve the accuracy of facial expression recognition of the target face under different facial postures and different lighting conditions.

[0473] Exemplarily, the computer program may be divided into one or more modules / units, which are stored in the memory and executed by the processor to implement the present invention. The one or more modules / units may be a series of computer program instruction segments capable of implementing specific functions, which are used to describe the execution process of the computer program in the device / terminal equipment.

[0474] The device / terminal device may be a computing device such as a mobile phone, a tablet computer, a desktop computer, a notebook, a PDA, and a cloud server. The device / terminal device may include, but is not limited to, a processor and a memory. Those skilled in the art will appreciate that the schematic diagram of the present invention is only an example of a device / terminal device and does not constitute a limitation on the device / terminal device. The device / terminal device may include more or fewer components than shown in the figure, or may combine certain components, or different components. For example, the device / terminal device may also include input / output devices, network access devices, buses, etc.

[0475] The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc. The processor is the control center of the device / terminal equipment, and various interfaces and lines are used to connect various parts of the entire device / terminal equipment.

[0476] The memory can be used to store the computer program and / or module, and the processor realizes various functions of the device / terminal equipment by running or executing the computer program and / or module stored in the memory, and calling the data stored in the memory. The memory may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as an image playback function, etc.), etc.; the data storage area may store data created according to the use of the mobile phone (such as video data, images, etc.), etc. In addition, the memory may include a high-speed random access memory, and may also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0477] If the module / unit integrated in the device / terminal equipment is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device that can carry the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electrical carrier signal, telecommunication signal and software distribution medium, etc.

[0478] The imaging of the target object in the above-mentioned embodiments may be partial imaging of the target object or overall imaging. Whether partial imaging or overall imaging is applicable or the method or device provided by the present invention is applicable after corresponding adjustments are made to the partial imaging or overall imaging. The above-mentioned adjustments do not require creative work by ordinary technicians in the field and should fall within the scope of protection of the present invention.

Claims

1. A method for facial expression recognition, characterized in that: include: Acquire a three-dimensional image of a target face, wherein the three-dimensional image includes third depth information of the target face and fifth color information of the target face; Inputting third depth information of the target face into the second neural network and inputting fifth color information of the target face into the third neural network; The second neural network classifies the expression of the target face according to the third depth information of the target face and the second parameter and outputs first classification data, and the third neural network classifies the expression of the target face according to the fifth color information of the target face and the third parameter and outputs second classification data, the second parameter includes a plurality of facial expression categories and second parameter data for identifying the expression category of the target face, and the third parameter includes the plurality of facial expression categories and third parameter data for identifying the expression category of the target face; outputting a classification result of the expression of the target face according to the first classification data and the second classification data; The step of outputting a classification result of the target facial expression according to the first classification data and the second classification data comprises: A support vector machine inputs the first classification data and the second classification data, and outputs a classification result of the expression of the target face according to the first classification data, the second classification data and support vector machine parameter data, wherein the support vector machine includes the plurality of facial expression categories and support vector machine parameter data for identifying the expression category of the target face; The facial expression categories included in the second neural network and the facial expression categories included in the third neural network include: fear, sadness, happiness, anger, disgust, surprise, neutral, and contempt; Before inputting the third depth information of the target face into the second neural network and inputting the fifth color information of the target face into the third neural network, the method further includes: Performing a third process on the third depth information of the target face, the third process comprising: Determine a feature point of the third depth information of the target face, and rotate the third depth information of the target face based on the feature point; Performing mirror processing, linear transformation, and affine transformation on the third depth information of the target face; Aligning feature points of the third depth information of the target face to a set position; Performing grayscale stretching on the third depth information of the target face; and Normalizing the pixel values ​​of the third depth information of the target face; or, Before inputting the third depth information of the target face into the second neural network and inputting the fifth color information of the target face into the third neural network, the method further includes: Performing the same third processing on the third depth information of the target face and the fifth color information of the target face, the third processing comprising: Determine feature points of the third depth information of the target face and feature points of the fifth color information of the target face, and rotate the third depth information of the target face and the fifth color information of the target face based on the feature points; Performing mirror processing, linear transformation, and affine transformation on the third depth information of the target face and the fifth color information of the target face; Aligning feature points of the third depth information of the target face and the fifth color information of the target face at a set position; Performing grayscale stretching on the third depth information of the target face or the fifth color information of the target face; and The third depth information of the target face and the fifth color information of the target face are subjected to image pixel value normalization processing.

2. The method according to claim 1, characterized in that The step of normalizing the image pixel values ​​of the third depth information of the target face includes: Normalize the pixel value of the third depth information of the target face from [0, 255] to [0, 1]; or, The normalizing the pixel values ​​of each channel of the third depth information of the target face and the fifth color information of the target face includes: The pixel values ​​of each channel of the third depth information of the target face and the fifth color information of the target face are normalized from [0, 255] to [0, 1].

3. The method according to claim 1, characterized in that: The second parameter data is obtained by training the second neural network based on fourth depth information of a plurality of facial expression samples; The third parameter data is obtained by training the third neural network based on the sixth color information of the plurality of facial expression samples.

4. The method according to claim 3, characterized in that: Before the fourth depth information of the plurality of facial expression samples is trained by the second neural network, the method further includes: Performing a fourth process on the fourth depth information of the facial expression sample, the fourth process comprising: Determine a feature point of the fourth depth information of the facial expression sample, and rotate the fourth depth information of the facial expression sample based on the feature point; Performing mirror processing, linear transformation and affine transformation on the fourth depth information of the facial expression sample; Aligning a feature point of the fourth depth information of the facial expression sample to a set position; Performing grayscale stretching on the fourth depth information of the facial expression sample; and Performing image pixel value normalization processing on the fourth depth information of the facial expression sample; Or, before the fourth depth information of the facial expression sample is trained by the second neural network and the sixth color information of the facial expression sample is trained by the third neural network, the method further includes: Performing the same fourth processing on the fourth depth information of the facial expression sample and the sixth color information of the facial expression sample, the fourth processing comprising: Determine a feature point of the fourth depth information of the facial expression sample and a feature point of the sixth color information of the facial expression sample, and rotate the fourth depth information of the facial expression sample and the sixth color information of the facial expression sample based on the feature point; performing mirror processing, linear transformation, and affine transformation on the fourth depth information of the facial expression sample and the sixth color information of the facial expression sample; Aligning feature points of the fourth depth information of the facial expression sample and the sixth color information of the facial expression sample at a set position; Performing grayscale stretching on the fourth depth information of the facial expression sample or the sixth color information of the facial expression sample; and, Performing image pixel value normalization processing on the fourth depth information of the facial expression sample and the sixth color information of the facial expression sample.

5. The method according to claim 4, characterized in that The step of normalizing the image pixel values ​​of the fourth depth information of the facial expression sample comprises: Normalizing the pixel value of the fourth depth information of the facial expression sample from [0, 255] to [0, 1]; or, The step of performing image pixel normalization processing on the fourth depth information of the facial expression sample and the sixth color information of the facial expression sample comprises: Normalize the pixel values ​​of each channel of the fourth depth information of the facial expression sample and the sixth color information of the facial expression sample from [0, 255] to [0, 1].

6. The method according to any one of claims 3 to 5, characterized in that: The support vector machine parameter data for identifying the expression category of the target face is obtained by training the support vector machine using the output data of the second neural network when training the fourth depth information of multiple facial expression samples and the output data of the third neural network when training the sixth color information of the multiple facial expression samples.

7. The method according to any one of claims 3 to 5, characterized in that: The facial expression sample has at least one of the following facial expression categories: fear, sadness, happiness, anger, disgust, surprise, neutral, and contempt; The facial expression sample, the fourth depth information of the facial expression sample, and the sixth color information of the facial expression sample have the same facial expression category.

8. The method according to any one of claims 1 to 5, characterized in that: The feature point is an eye point.

9. The method according to any one of claims 1 to 5, characterized in that: The second neural network includes a second convolutional neural network, and the third neural network includes a third convolutional neural network.

10. The method according to claim 9, characterized in that The second convolutional neural network includes: 3 convolutional layers, 3 downsampling layers, 1 drop layer and 2 fully connected layers; The third convolutional neural network includes: 4 convolutional layers, 4 downsampling layers, 1 drop layer and 2 fully connected layers.

11. The method according to any one of claims 1 to 5, characterized in that: The fifth color information is an image in RGB format or YUV format.

12. The method according to any one of claims 3 to 5, characterized in that: The sixth color information is an image in RGB format or YUV format.

13. A device for facial expression recognition, characterized in that: The device includes a second acquisition module, a second input module, a second neural network, a third neural network and a second classification module. The second acquisition module is used to acquire a three-dimensional image of the target face, wherein the three-dimensional image includes third depth information of the target face and fifth color information of the target face; The second input module is used to input the third depth information of the target face into the second neural network and input the fifth color information of the target face into the third neural network; The second neural network is used to classify the expression of the target face according to the third depth information of the target face and the second parameter and output first classification data, and the third neural network is used to classify the expression of the target face according to the fifth color information of the target face and the third parameter and output second classification data, the second parameter includes a plurality of facial expression categories and second parameter data for identifying the expression category of the target face, and the third parameter includes the plurality of facial expression categories and third parameter data for identifying the expression category of the target face; The second classification module is used to output a classification result of the expression of the target face according to the first classification data and the second classification data; The second classification module includes a support vector machine, The support vector machine is used to input the first classification data and the second classification data, and output a classification result of the expression of the target face according to the first classification data, the second classification data and the support vector machine parameter data, wherein the support vector machine includes the plurality of facial expression categories and the support vector machine parameter data for identifying the expression category of the target face; The facial expression categories included in the second neural network and the facial expression categories included in the third neural network include: fear, sadness, happiness, anger, disgust, surprise, neutral, and contempt; The device also includes a third processing module, The third processing module is used to perform a third processing on the third depth information of the target face, and input the third depth information of the target face after the third processing into the second input module; The third processing module includes a third rotation submodule, a third transformation submodule, a third alignment submodule, a third grayscale stretching submodule, and a third normalization processing submodule; The third rotation submodule is used to determine feature points of the third depth information of the target face, and rotate the third depth information of the target face based on the feature points; The third transformation submodule is used to perform mirror processing, linear transformation and affine transformation on the third depth information of the target face; The third alignment submodule is used to align the feature points of the third depth information of the target face with the set positions; The third grayscale stretching submodule is used to perform grayscale stretching on the third depth information of the target face; The third normalization processing submodule is used to perform image pixel value normalization processing on the third depth information of the target face; or, The third processing module is further used to perform the same third processing on the third depth information of the target face and the fifth color information of the target face, and input the third depth information of the target face and the fifth color information of the target face after the third processing into the second input module; The third rotation submodule is further used to determine feature points of the third depth information of the target face and feature points of the fifth color information of the target face, and rotate the third depth information of the target face and the fifth color information of the target face based on the feature points; The third transformation submodule is further used to perform mirror processing, linear transformation and affine transformation on the third depth information of the target face and the fifth color information of the target face; The third alignment submodule is further used to align feature points of the third depth information of the target face and the fifth color information of the target face at a set position; The third grayscale stretching submodule is further used to perform grayscale stretching on the third depth information of the target face or the fifth color information of the target face; and The third normalization processing submodule is further used to perform image pixel value normalization processing on the third depth information of the target face and the fifth color information of the target face.

14. The device according to claim 13, characterized in that The third normalization processing submodule is specifically used to normalize the pixel value of the third depth information of the target face from [0, 255] to [0, 1]; or, The third normalization processing submodule is specifically used to normalize the pixel values ​​of each channel of the third depth information of the target face and the fifth color information of the target face from [0, 255] to [0, 1].

15. The device according to claim 13, characterized in that The second parameter data is obtained by training the second neural network based on fourth depth information of a plurality of facial expression samples; The third parameter data is obtained by training the third neural network based on the sixth color information of the plurality of facial expression samples.

16. The device according to claim 15, characterized in that The device comprises a fourth processing module, The fourth processing module is used to perform a fourth processing on the fourth depth information of the facial expression sample, and input the fourth depth information of the facial expression sample after the fourth processing into the second input module; The fourth processing module includes a fourth rotation submodule, a fourth transformation submodule, a fourth alignment submodule, a fourth grayscale stretching submodule, and a fourth normalization processing submodule; The fourth rotation submodule is used to determine a feature point of the fourth depth information of the facial expression sample, and rotate the fourth depth information of the facial expression sample based on the feature point; The fourth transformation submodule is used to perform mirror processing, linear transformation and affine transformation on the fourth depth information of the facial expression sample; The fourth alignment submodule is used to align the feature points of the fourth depth information of the facial expression sample to a set position; The fourth grayscale stretching submodule is used to perform grayscale stretching on the fourth depth information of the facial expression sample; as well as, The fourth normalization processing submodule is used to perform image pixel value normalization processing on the fourth depth information of the facial expression sample; or, The fourth processing module is further used to perform a fourth processing on the fourth depth information of the facial expression sample and the sixth color information of the facial expression sample, and input the fourth depth information of the facial expression sample and the sixth color information of the facial expression sample after the fourth processing into the second input module; The fourth rotation submodule is further used to determine feature points of the fourth depth information of the facial expression sample and feature points of the sixth color information of the facial expression sample, and rotate the fourth depth information of the facial expression sample and the sixth color information of the facial expression sample based on the feature points; The fourth transformation submodule is further used to perform mirror processing, linear transformation and affine transformation on the fourth depth information of the facial expression sample and the sixth color information of the facial expression sample; The fourth alignment submodule is further used to align feature points of the fourth depth information of the facial expression sample and the sixth color information of the facial expression sample at a set position; The fourth grayscale stretching submodule is further used to perform grayscale stretching on the fourth depth information of the facial expression sample or the sixth color information of the facial expression sample; as well as, The fourth normalization processing submodule is further used to perform image pixel value normalization processing on the fourth depth information of the facial expression sample and the sixth color information of the facial expression sample.

17. The device according to claim 16, characterized in that The fourth normalization processing submodule is specifically used to normalize the pixel value of the fourth depth information of the facial expression sample from [0, 255] to [0, 1]; or, The fourth normalization processing submodule is specifically used to normalize the pixel values ​​of each channel of the fourth depth information of the facial expression sample and the sixth color information of the facial expression sample from [0, 255] to [0, 1].

18. The device according to any one of claims 15 to 17, characterized in that: The support vector machine parameter data for identifying the expression category of the target face is obtained by training the support vector machine using the output data of the second neural network when training the fourth depth information of multiple facial expression samples and the output data of the third neural network when training the sixth color information of the multiple facial expression samples.

19. The device according to any one of claims 15 to 17, characterized in that: The facial expression sample has at least one of the following facial expression categories: fear, sadness, happiness, anger, disgust, surprise, neutral, and contempt; The facial expression sample, the fourth depth information of the facial expression sample, and the sixth color information of the facial expression sample have the same facial expression category.

20. The device according to any one of claims 13 to 17, characterized in that: The feature point is an eye point.

21. The device according to any one of claims 12 to 17, characterized in that: The second neural network includes a second convolutional neural network, and the third neural network includes a third convolutional neural network.

22. The device according to claim 21, characterized in that The second convolutional neural network includes: 3 convolutional layers, 3 downsampling layers, 1 drop layer and 2 fully connected layers; The third convolutional neural network includes: 4 convolutional layers, 4 downsampling layers, 1 drop layer and 2 fully connected layers.

23. The device according to any one of claims 12 to 17, characterized in that: The fifth color information is an image in RGB format or YUV format.

24. The device according to any one of claims 15 to 17, characterized in that: The sixth color information is an image in RGB format or YUV format.

Images