A method, device, electronic device and storage medium for recognizing facial expression

Abstract

The present application relates to the field of facial recognition technology, and in particular to a method, device, electronic device and storage medium for facial expression recognition. The present application divides the image of the face to be recognized into the region of interest images corresponding to each facial action unit based on the three-dimensional face model corresponding to the face image to be recognized and the definition information of each facial action unit; determines the face fusion feature vector based on the face local feature vectors extracted from each region of interest image, the face expression feature vector extracted from the face image to be recognized, and the face detail feature vector; determines the target expression of the person in the face image to be recognized based on the face fusion feature vector, so that the accuracy of facial expression recognition can be improved by fusing the extracted face local feature vectors closely related to the facial action unit and the face global feature vector for facial expression recognition.

Description

Technical Field

[0001] The present application relates to the field of facial recognition technology, and in particular to a facial expression recognition method, device, electronic device and storage medium. Background Art

[0002] Facial expression recognition refers to identifying the expression corresponding to a given facial image, such as happiness, sadness, excitement, etc. It has very broad application prospects in human-computer interaction, autonomous driving, film and television or animation production, medical services and other fields.

[0003] Facial expressions can be divided into dozens of action units (AU) according to the movement of facial muscle groups, forming a complete facial action unit coding system (FACS). Due to the advantages of objective, fine-grained and quantifiable facial action units, they can more finely characterize human expressions and emotions. Therefore, any human expression can be represented as a combination of a set of action units and their different intensities.

[0004] At present, in the process of facial expression recognition, facial feature extraction mostly obtains information directly from the entire face of the face image, such as using manual design or deep convolutional networks to directly extract features from the entire face image. This feature extraction method often lacks the ability to perceive the features of local areas of the face, which leads to low accuracy in facial expression recognition. Summary of the invention

[0005] In view of this, the embodiments of the present application at least provide a facial expression recognition method, device, electronic device and storage medium, which can improve the accuracy of facial expression recognition.

[0006] This application mainly includes the following aspects:

[0007] In a first aspect, an embodiment of the present application provides a method for recognizing facial expressions, the method comprising:

[0008] Obtaining a face image to be recognized;

[0009] Based on the three-dimensional face model corresponding to the face image to be identified and the definition information of each facial action unit, dividing the region of interest image corresponding to each facial action unit from the face image to be identified;

[0010] Determine a face fusion feature vector based on the face local feature vectors extracted from each of the region of interest images, the face expression feature vector and the face detail feature vector extracted from the face image to be identified;

[0011] Based on the face fusion feature vector, a target expression of the person in the face image to be recognized is determined.

[0012] In a possible implementation, dividing the region of interest images corresponding to the facial action units from the facial image to be identified based on the three-dimensional facial model corresponding to the facial image to be identified and the definition information of each facial action unit, includes:

[0013] Determine the three-dimensional face model corresponding to the face image to be identified according to the standard face model and the face image to be identified;

[0014] Dividing the three-dimensional face model into various facial local areas according to definition information of each facial action unit;

[0015] Based on the region mask information of each of the local facial regions, the region of interest images corresponding to each of the facial action units are divided from the face image to be recognized.

[0016] In a possible implementation, the target facial expression recognition model includes a local feature extraction module, a global feature extraction module, a feature association module, and a facial action intensity detection module; the facial fusion feature vector is determined based on the facial local feature vectors extracted from each of the region of interest images, the facial expression feature vector extracted from the face image to be recognized, and the facial detail feature vector, including:

[0017] For each of the region of interest images, inputting the region of interest image into the local feature extraction module to determine a local feature vector of the face corresponding to the region of interest image;

[0018] Inputting the face image to be recognized into the global feature extraction module to determine the facial expression feature vector and the facial detail feature vector;

[0019] Inputting each of the local face feature vectors, the facial expression feature vector and the facial detail feature vector into the feature association module to determine the face fusion feature vector;

[0020] The step of determining a target expression of a person in the face image to be identified based on the face fusion feature vector comprises:

[0021] The face fusion feature vector is input into the facial action intensity detection module to determine the target expression of the person in the face image to be identified.

[0022] In a possible implementation, before acquiring the face image to be recognized, the target face expression recognition model is trained according to the following steps:

[0023] Using multiple sample face images to train the initial face expression recognition model to obtain an intermediate face expression recognition model; wherein the global feature extraction module in the initial face expression recognition model includes a face expression feature extraction network, a face detail feature extraction network and a face identity feature extraction network;

[0024] The facial identity feature extraction network in the intermediate facial expression recognition model is removed, and the feature association module and the facial action intensity detection module in the intermediate facial expression recognition model without the facial identity feature extraction network are trained for the next round to obtain the trained target facial expression recognition model.

[0025] In a possible implementation, before acquiring the face image to be recognized, a sample face image for training a facial expression recognition model is determined according to the following steps:

[0026] From the training data set, m training face images corresponding to each sample person are selected;

[0027] For each of the sample persons, n candidate face images corresponding to each facial action unit in an activated state are screened out from the m training face images corresponding to the sample person;

[0028] For each of the sample persons, from the n candidate face images corresponding to each facial action unit in an activated state, select target face images of each intensity level corresponding to the facial action unit;

[0029] Determine the target face image as the sample face image;

[0030] Wherein, m and n are both positive integers, and m>n.

[0031] In a possible implementation, the local feature extraction module includes at least two convolutional networks and a fully connected network; for each of the region of interest images, inputting the region of interest image into the local feature extraction module, and determining the local feature vector of the face corresponding to the region of interest image, including:

[0032] Inputting the region of interest image into a first convolutional network to determine a first local feature vector;

[0033] Inputting the first local feature vector into a second convolutional network to determine a second local feature vector;

[0034] Inputting the second local feature vector into the fully connected network to determine the local feature vector of the face;

[0035] The first convolutional network and the second convolutional network use different convolutional kernels.

[0036] In one possible implementation, the feature association module includes a target graph attention network; the target graph attention network is generated according to the following steps:

[0037] Each sample face local feature vector, sample face expression feature vector and sample face detail feature vector are used as nodes of the initial graph attention network respectively;

[0038] Counting the number of times each of the facial action units co-occurs in a plurality of sample face images used for training;

[0039] According to the counted number of times each of the facial action units co-occurs, a connection relationship between each node in the initial graph attention network is constructed to generate the target graph attention network.

[0040] In a possible implementation manner, inputting each of the local face feature vectors, the facial expression feature vector, and the facial detail feature vector into the feature association module to determine the face fusion feature vector includes:

[0041] For a first facial feature vector, determining attention coefficients between the first facial feature vector and the second facial feature vector, the facial expression feature vector, and the facial detail feature vector, respectively; the first facial feature vector and the second facial feature vector are different local facial feature vectors;

[0042] Using multiple attention coefficients corresponding to the first facial feature vector as weights, performing feature weighted calculation on the second facial feature vector, the facial expression feature vector, and the facial detail feature vector, to determine an intermediate fusion feature vector corresponding to the first facial feature vector;

[0043] Nonlinear activation processing is performed on the intermediate fusion feature vector corresponding to the first facial feature vector to determine the facial fusion feature vector corresponding to the first facial feature vector.

[0044] In a possible implementation, the step of identifying a target expression from the face image to be identified based on the face fusion feature vector includes:

[0045] Based on the face fusion feature vector, determining each facial action unit in the to-be-recognized face image that is in an activated state, and an intensity level corresponding to each facial action unit in an activated state;

[0046] The target expression of the person in the face image to be recognized is determined according to each facial action unit in an activated state and the intensity level corresponding to each facial action unit in an activated state.

[0047] In a second aspect, an embodiment of the present application further provides a facial expression recognition device, the facial expression recognition device comprising:

[0048] An acquisition unit, used for acquiring a face image to be recognized;

[0049] A dividing unit, configured to divide the image of the human face to be identified into images of regions of interest corresponding to the facial action units based on the three-dimensional face model corresponding to the facial image to be identified and the definition information of the facial action units;

[0050] A first determining unit is used to determine a face fusion feature vector based on the face local feature vectors extracted from each of the region of interest images, the face expression feature vector extracted from the face image to be identified, and the face detail feature vector;

[0051] The second determining unit is used to determine the target expression of the person in the face image to be identified based on the face fusion feature vector.

[0052] In a third aspect, an embodiment of the present application also provides an electronic device, comprising: a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate through the bus, and the machine-readable instructions are executed by the processor to execute the steps of the facial expression recognition method described in the first aspect or any possible implementation manner of the first aspect.

[0053] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of facial expression recognition described in the above-mentioned first aspect or any possible implementation manner of the first aspect are executed.

[0054] The embodiments of the present application provide a method, device, electronic device and storage medium for facial expression recognition. By using a three-dimensional face model corresponding to a face image to be recognized and definition information of each facial action unit, images of regions of interest corresponding to each facial action unit are divided from the face image to be recognized. Then, based on the local face feature vectors extracted from each region of interest image, the facial expression feature vectors extracted from the face image to be recognized and the facial detail feature vectors, a face fusion feature vector is jointly determined. Furthermore, based on the face fusion feature vector, a target expression of a person in the face image to be recognized is determined. Compared with the prior art solution of using manual design or deep convolutional network to directly extract features from the entire facial image and then complete expression recognition, the technical solution of the present application performs facial expression recognition by fusing the extracted local face feature vectors and the global face feature vectors closely related to the facial action units, thereby improving the overall perception of local face region features and global face features in the process of facial expression recognition and improving the accuracy of facial expression recognition.

[0055] Furthermore, the facial expression recognition method provided in the embodiment of the present application can improve the balance of sample facial images for model training by adopting a three-level balanced data sampling method based on facial identity, facial action unit category, and intensity level of facial action units. Therefore, it can be ensured that the model is not affected by samples of a certain facial identity feature, a certain facial action unit category, or a certain intensity level of facial action units during the training process, which can further improve the accuracy of facial expression recognition.

[0056] In addition, the facial expression recognition method provided in the embodiment of the present application also adopts two stages to train the facial expression recognition model. In the first stage, facial identity feature information, facial expression feature information, and facial detail feature information are extracted from the sample face image for training. In the second stage, facial identity feature information is not extracted for training. This can avoid the situation where the trained facial expression recognition model is overfitted to a specific character identity. In this way, the facial identity can be decoupled, and the accuracy of facial expression recognition can be further improved.

[0057] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0059] Figure 1 A flowchart of a facial expression recognition method provided by an embodiment of the present application is shown;

[0060] Figure 2 A flowchart of another facial expression recognition method provided by an embodiment of the present application is shown;

[0061] Figure 3 One of the functional module diagrams of a facial expression recognition device provided in an embodiment of the present application is shown;

[0062] Figure 4 A second functional module diagram of a facial expression recognition device provided in an embodiment of the present application is shown;

[0063] Figure 5 A schematic structural diagram of an electronic device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0064] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It should be understood that the drawings in the present application only serve the purpose of explanation and description and are not used to limit the scope of protection of the present application. In addition, it should be understood that the schematic drawings are not drawn in real proportion. The flowchart used in this application shows the operations implemented according to some embodiments of the present application. It should be understood that the operations of the flowchart can be implemented out of sequence, and the steps without logical context can be reversed in order or implemented simultaneously. In addition, those skilled in the art, under the guidance of the content of the present application, can add one or more other operations to the flowchart, or remove one or more operations from the flowchart.

[0065] In addition, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.

[0066] In order to enable those skilled in the art to use the contents of this application, the following implementation methods are given in combination with the specific application scenario of "facial expression recognition for facial images". For those skilled in the art, the general principles defined here can be applied to other embodiments and application scenarios without departing from the spirit and scope of this application.

[0067] The following methods, devices, electronic devices or computer-readable storage media of the embodiments of the present application can be applied to any scenario where facial expression recognition for facial images is required. The embodiments of the present application are not limited to specific application scenarios. Any scheme using the facial expression recognition method and device provided by the embodiments of the present application is within the scope of protection of the present application.

[0068] It is worth noting that before the present application was proposed, most of the related schemes directly obtained information from the entire face of the face picture, such as using manual design or deep convolutional networks to directly extract features from the entire face picture; there are also some schemes that coarsely divide the face image into several larger areas, and perform feature extraction on this basis, and then perform expression recognition based on facial features, but these schemes often result in the extracted facial features lacking the ability to perceive the features of local facial areas, and are easily affected by factors such as facial posture and image deformation, which affects the features of local facial areas, thereby resulting in low accuracy in recognizing facial expressions.

[0069] In response to the above problems, the embodiments of the present application provide a facial expression recognition method, device, electronic device and storage medium. By utilizing the three-dimensional face model corresponding to the face image to be recognized and the definition information of each facial action unit, an image of a region of interest closely related to each facial action unit is divided from the face image to be recognized. Then, the local face feature vectors extracted from each region of interest image, the facial expression feature vector extracted from the face image to be recognized, and the facial detail feature vector are fused to obtain a face fusion feature vector to determine the target expression of the person in the face image to be recognized. This can greatly improve the overall perception ability of the local area features and the global features of the face during the facial expression recognition process, and can improve the accuracy of facial expression recognition.

[0070] To facilitate the understanding of the present application, the technical solution provided by the present application is described in detail below in conjunction with specific embodiments.

[0071] Figure 1 A flowchart of a facial expression recognition method provided in an embodiment of the present application. Figure 1 As shown, the facial expression recognition method provided in the embodiment of the present application includes the following steps:

[0072] S101: Obtain a face image to be recognized.

[0073] In this step, a face image to be recognized that needs to be subjected to facial expression recognition is obtained.

[0074] Here, the face image to be identified may be a two-dimensional face image obtained by any shooting angle. Preferably, the accuracy of facial expression recognition using a frontal face image is higher than that using other face images.

[0075] S102: based on the three-dimensional face model corresponding to the face image to be identified and the definition information of each facial action unit, divide the region of interest image corresponding to each facial action unit from the face image to be identified.

[0076] In this step, a three-dimensional face model corresponding to the face image to be identified is first constructed, and then based on the three-dimensional face model and the definition information of each facial action unit, the region of interest images corresponding to each facial action unit are divided in the face image to be identified, that is, a plurality of region of interest images are divided in the identified face image, wherein each region of interest image corresponds to a facial action unit.

[0077] Here, the definition information of the facial action unit can be determined according to the definition of the facial action unit in the facial action coding system. Specifically, the facial action unit is a unit composed of the movement of one or more muscles of the human face, which can be used to represent small changes in movement on the human face. Different changes in movement can constitute different facial expressions. Therefore, a specific facial action unit has a certain correspondence with a specific local area of ​​the face, especially the forehead, eyebrows, nose, cheeks, mouth and chin, which are highly associated with expressions. In addition, some AUs with a high degree of correlation with expressions are given as examples, for example, AU0 represents the muscles of the upper lip and philtrum area, AU1 represents the lowering of the chin, AU2 represents the stretching of the corners of the mouth, AU3 represents the lowering and gathering of eyebrows, AU4 represents the pulling and tilting of the corners of the mouth, and AU5 represents the lifting of the outer corners of the eyebrows.

[0078] It should be noted that, when dividing the local area of ​​the face image to be identified, the present application strictly follows the definition of facial action units in the facial action coding system, and divides the two-dimensional region of interest image from the face image to be identified according to the muscle movement area corresponding to the definition of facial action units in the three-dimensional face model from an anatomical point of view. In this way, this division method can more flexibly and meticulously divide the local area that is more closely related to the facial action unit from the face image to be identified, and can avoid introducing noise as much as possible on the basis of retaining important feature areas. The local features of the face obtained from the region of interest image have clear semantic information and good interpretability. Therefore, on this basis, it is easy to improve the accuracy of facial expression recognition.

[0079] S103, determining a face fusion feature vector based on the local face feature vectors extracted from each of the region of interest images, the facial expression feature vector and the facial detail feature vector extracted from the face image to be identified.

[0080] In this step, for each region of interest image segmented from the face image to be identified, the corresponding local face feature vector can be extracted from each region of interest image. At the same time, a global face feature vector is extracted from the face image to be identified, where the global face feature vector includes a face expression feature vector and a face detail feature vector. Then, the local face feature vector, the face expression feature vector and the face detail feature vector are feature associated and fused to obtain a face fusion feature vector.

[0081] Here, for each region of interest image, an independent feature extraction network can be used to extract the local facial feature vector corresponding to each region of interest image, and another separate feature extraction branch can be used to extract the facial expression feature vector and facial detail feature vector corresponding to the face image to be identified.

[0082] Among them, the facial expression feature vector is used to reflect the facial expression features of the face in the face image to be identified, such as facial expression features related to happiness, sadness, and anger. Specifically, it can be represented from the extracted muscle movement information of the facial parts related to the expression; the facial detail feature vector is used to reflect the detail features of the face itself in the face image to be identified, such as the texture features and shape features of the face; the facial local feature vector is used to reflect the facial feature information corresponding to each area of ​​interest image segmented from the face image to be identified, such as: cheek lifting features, mouth corner upturned features, lip separation features, etc.

[0083] In this way, by comprehensively extracting the feature information of the face in the face image to be identified from the perspective of local perception and global perception, the extraction of facial features can be made more comprehensive and accurate. In this way, the face fusion feature vector combines the local facial features and the global facial features, and improves the accuracy of facial expression recognition by improving the overall perception ability of the local area features and the global facial features in the facial expression recognition process.

[0084] S104: Determine a target expression of the person in the face image to be identified based on the face fusion feature vector.

[0085] In this step, based on the face fusion feature vector that fuses the local face feature vector, the face expression feature vector and the face detail feature vector, the facial expression of the person in the face image to be identified is identified to determine the target expression of the person in the face image to be identified.

[0086] Here, the target expression of the person in the face image to be recognized may be one of a neutral expression, a happy expression, a sad expression, an angry expression, a disgusted expression, a surprised expression, a fearful expression, and the like.

[0087] The above scheme will be described below in conjunction with specific implementation methods.

[0088] In the embodiment of the present application, further, a three-dimensional deformable face reconstruction algorithm (3D Morphable Model, 3DMM) can be used in combination with the definition information of the facial action unit to divide the image of the face to be identified into the region of interest image corresponding to each of the facial action units. Specifically, step S102 includes the following steps 1021-1023:

[0089] Step 1021: Determine the three-dimensional face model corresponding to the face image to be identified based on the standard face model and the face image to be identified.

[0090] In step 1021, the average face calculated using the facial action unit data set, that is, the standard face model, can be used to perform three-dimensional reconstruction of the standard face model for the face image to be identified based on the 3DMM algorithm to obtain a three-dimensional face model corresponding to the face image to be identified.

[0091] Here, the 3DMM algorithm uses some fixed points to represent a three-dimensional face. Two face images are matched one by one in three-dimensional space, and can be obtained by weighted addition of many other face orthogonal bases. Therefore, each face in three-dimensional space can be represented by the weighted sum of the basis vectors and corresponding coefficients of all faces in the database. Among them, any face can be represented by an explicit superposition of a set of shape vectors and a set of texture vectors. The shape vector S and texture vector T can be shown as follows:

[0092] S=(X1,Y1,Z1,X2,Y2,Z2,…,X n ,Y n ,Z n );

[0093] T=(R1,G1,B1,R2,G2,B2,…,R n ,G n ,B n );

[0094] Among them, (X, Y, Z) represents the three-dimensional coordinates; (R, G, B) represents the color at the corresponding coordinates, R represents red, G represents green, and B represents blue.

[0095] Furthermore, the shape vector and texture vector of any 3D face can be obtained from the shape vectors S of m other face models in the dataset. i and texture T i Perform linear weighted summation to obtain the following formula:

[0096]

[0097]

[0098] Among them, a i and b i Represent the weighting coefficients, S mod Represents the shape vector of a 3D face, T mod Texture vector representing a 3D face.

[0099] In practical applications, it is necessary to first reduce the dimension and decompose the shape vectors and texture vectors of the m faces to achieve orthogonality between the vectors. The processed face can be expressed as follows:

[0100]

[0101]

[0102] in, and Represent the mean of the shape vector and texture vector of m faces respectively; and s i and t i Represent the original shape S i and texture T i Eigenvectors of the covariance matrix after subtracting the mean.

[0103] In this way, any three-dimensional face model in the three-dimensional space can be reconstructed through the estimated sets of coefficients and the corresponding basis vectors.

[0104] Step 1022: Divide the three-dimensional face model into various facial local areas according to the definition information of each facial action unit.

[0105] In step 1022, based on the definition information of each facial action unit, the local area of ​​interest of the face is obtained, and the three-dimensional face model corresponding to the face image to be identified is divided into various facial local areas. Here, the facial action unit, for example: AU1-eyebrow action unit, AU5-eye action unit, AU6-cheek action unit, AU9-nose action unit and AU26-lip action unit, etc., and accordingly, the facial local area may include: the eyebrow action area, eye action area, cheek action area, nose action area and lip action area of ​​the face in the face image to be identified.

[0106] Step 1023: based on the region mask information of each of the local facial regions, divide the region of interest images corresponding to each of the facial action units from the face image to be identified.

[0107] In step 1023, the region mask information corresponding to each local facial region is calculated, and combined with the region mask information corresponding to each local facial region, the region of interest images corresponding to each facial action unit are divided from the face image to be identified.

[0108] It should be noted that, since the present application strictly refers to the definition of each facial action unit in the FACS system when dividing the local facial areas related to the facial action units, compared with many solutions based on average area division in the related technology, the present application strictly divides according to the definition of the facial action unit, ensuring that the divided areas have a high degree of semantic consistency with the facial action units, and the irregular areas obtained by segmentation can avoid the redundant information interference caused by the regular square area division, have good discrimination, and can improve the accuracy of facial expression recognition.

[0109] As a possible implementation, after step 1023, the method further includes: performing noise reduction processing on each of the region of interest images, wherein the noise reduction processing includes: edge smoothing processing and denoising processing.

[0110] In this way, by performing denoising on the irregular image of the region of interest generated after segmentation, the redundant information interference caused by the regular square region segmentation method can be avoided, and the influence of noise such as edge jagged pixels on subsequent network feature extraction can be reduced. It has better discrimination and avoids introducing too much noise as much as possible while retaining the main feature areas.

[0111] In another embodiment of the present application, a target facial expression recognition model can be used to identify the target expression of a person in a facial image to be identified, wherein the target facial expression recognition model includes: a local feature extraction module, a global feature extraction module, a feature association module, and a facial action intensity detection module.

[0112] Here, the local feature extraction module is used to extract the corresponding local face feature vector from the input area of ​​interest image; the global feature extraction module is used to extract the facial expression feature vector and the facial detail feature vector from the input face image to be identified; the feature association module is used to perform feature association on the input local face feature vectors, facial expression feature vectors and facial detail feature vectors to determine the face fusion feature vector; the facial action intensity detection module is used to determine the target expression of the person in the face image to be identified based on the input face fusion feature vector.

[0113] Specifically, the step of determining the face fusion feature vector in step S103 based on the local face feature vectors extracted from each of the region of interest images, the facial expression feature vector extracted from the face image to be identified, and the facial detail feature vector may include the following steps 1031-1033.

[0114] Step 1031: For each of the region of interest images, input the region of interest image into the local feature extraction module to determine a local facial feature vector corresponding to the region of interest image.

[0115] Here, since the input of the local feature extraction module is the region of interest images corresponding to each facial action unit, the local face feature vector output by the local feature extraction module can reflect the features of the facial action unit corresponding to the input region of interest image.

[0116] In this way, redundant information interference caused by other facial action units that are irrelevant to the facial action units corresponding to the input region of interest image can be avoided to the greatest extent during the extraction of local facial feature vectors. For example, AU1 focuses on the eyebrow area corresponding to the eyebrow raising action, and only needs to pay attention to the pixel changes in this area. There is no need to care about the mouth, neck or even the background pixels of the image, that is, irrelevant areas.

[0117] As a possible implementation, step 1031 may specifically include: inputting the image of the region of interest into a first convolutional network to determine a first local feature vector; inputting the first local feature vector into a second convolutional network to determine a second local feature vector; inputting the second local feature vector into the fully connected network to determine the local feature vector of the face; wherein the first convolutional network and the second convolutional network use different convolution kernels.

[0118] Here, the local feature extraction module is composed of a first convolutional network, a second convolutional network and a fully connected network. For the image of the region of interest corresponding to any input facial action unit, the first convolutional network first performs the first convolution processing using the first convolution kernel to obtain a first local feature vector; then, for the image of the region of interest after the first convolution processing, the second convolutional network performs multiple convolution processing using the second convolution kernel to obtain a feature map corresponding to the image of the region of interest, and uses the global average pooling layer (Global Average Pool Layer, GAP) to obtain the feature mean of each convolution channel in the first convolutional network and the second convolutional network to form a second local feature vector; finally, the fully connected network is used to perform a linear transformation on the second local feature vector to generate a local feature vector of the face.

[0119] Exemplarily, the first convolution kernel size of the first convolution network can be set to 7, the step size to 3, the image padding to 1, and the number of channels to 32; the second convolution network includes 3 convolution layers, each of which uses a convolution kernel of 3×3 size, and the number of channels corresponding to each convolution layer is 64, 128, and 128, respectively. After the input region of interest image is processed by 4 convolutions in the first convolution network and the 3 convolution layers in the second convolution network, the size of the feature map obtained is 10×10×128. After the feature map is scaled by the global mean pooling layer, a 1×1×128 second local feature vector is obtained. A fully connected network without a nonlinear activation function is used for linear transformation of the second local feature vector, and the feature scale of the second local feature vector is scaled to 51 dimensions to obtain a local feature vector of the face.

[0120] In this way, in the process of extracting local feature vectors of the face, a separate feature extraction network is adopted between each feature extraction branch. Each feature extraction network is independent of each other and does not interfere with each other, which can reduce the interference of redundant information brought by other facial unit areas and ensure the distinguishability of features extracted by each feature extraction network. At the same time, the use of the global mean pooling layer will not introduce additional parameters, and the number of parameters of the fully connected network will be reduced after pooling to prevent model overfitting.

[0121] Step 1032: input the face image to be recognized into the global feature extraction module to determine the facial expression feature vector and the facial detail feature vector.

[0122] Here, the facial expression feature vector and the facial detail feature vector can describe the global features of the face in the face image to be recognized. Since the facial expressions often involve the actions of different muscle groups on the face, in the same expression, the facial units usually appear in combination. For example, when the face shows a smile, the two facial action units AU6 (cheek lift) and AU12 (corner of mouth lift) often appear at the same time. Therefore, in the process of expression recognition, it is necessary not only to observe the region of interest image corresponding to a single facial action unit, but also to consider the macroscopic global features in the entire face image.

[0123] In the specific implementation process, the global feature extraction module may include a facial expression feature extraction network and a facial detail feature extraction network. The facial expression feature extraction network may be a VGG19 network, and the facial detail feature extraction network may be a ResNet34 network.

[0124] The facial expression feature extraction network may be a convolutional neural network trained according to a preset expression training data set and corresponding expression classification labels. Optionally, the preset expression training data set may include facial image data of multiple expressions such as neutral, happy, sad, angry, disgusted, surprised, and fearful. The facial detail feature extraction network may be a convolutional neural network for identifying texture features, shape features, and the like of a face.

[0125] Step 1033: input each of the local face feature vectors, the facial expression feature vector, and the facial detail feature vector into the feature association module to determine the face fusion feature vector.

[0126] It should be noted that the embodiment of the present application constructs a multi-channel based local feature extraction module to obtain the local feature vector of the face, and combines the global feature vector of the face extracted by the global feature extraction module channel, so that the target facial expression recognition model has both local perception and global perception capabilities. The extracted features are subjected to feature association modeling and fusion by the feature association module for facial action unit detection or intensity estimation, so as to recognize the facial expression of the face image to be recognized, which can improve the accuracy of facial expression recognition.

[0127] Step S104: Based on the face fusion feature vector, the step of determining the target expression of the person in the face image to be identified may include the following step 1041: Step 1041, inputting the face fusion feature vector into the facial action intensity detection module to determine the target expression of the person in the face image to be identified.

[0128] Here, the feature association module can be implemented using a target graph attention. Each node of the graph attention network represents a category of facial action unit, and the association characteristics between different categories of facial action units are reflected through the connection relationship between nodes.

[0129] As a possible implementation, the target graph attention network is generated based on the following method: each sample face local feature vector, sample face expression feature vector and sample face detail feature vector are respectively used as nodes of the initial graph attention network; the number of times each facial action unit co-occurs in multiple sample face images used for training is counted; based on the counted number of times each facial action unit co-occurs, the connection relationship between each node in the initial graph attention network is constructed to generate the target graph attention network.

[0130] Here, the target graph attention network not only takes the local facial features of different categories of facial action units as nodes, but also incorporates global facial features such as facial expression features and facial detail features into the nodes of the target graph attention network, and associates them with other local facial feature nodes.

[0131] It should be noted that, for each sample face image, if at least two facial action units appear together in the sample face image, they are counted once, that is, it is counted that the at least two facial action units appear together once, that is, there is a correlation between the at least two facial action units.

[0132] Among them, in the target graph attention network, two nodes that appear together can be connected, if there is no two nodes that appear together, they will not be connected, and each node is associated with a different number of other nodes.

[0133] It should be noted that when generating the graph attention network, the present application makes good use of the prior knowledge of the mutual dependence between various facial action units to construct the graph network, and uses the training data set to drive the adjustment of the node connection relationship and weight of the graph network. Compared with the fusion method of directly splicing features in the related technology, it can effectively weigh the importance of various features to retain more useful information, fully model the relationship between various facial action units, reduce the difficulty of network convergence and help improve the network prediction accuracy.

[0134] On this basis, as a possible implementation method, step 1041 may specifically include: for the first facial feature vector, determining the attention coefficients between the first facial feature vector and the second facial feature vector, the facial expression feature vector and the facial detail feature vector respectively; the first facial feature vector and the second facial feature vector are different local facial feature vectors; using the multiple attention coefficients corresponding to the first facial feature vector as weights, performing feature weighted calculation on the second facial feature vector, the facial expression feature vector and the facial detail feature vector, and determining the intermediate fusion feature vector corresponding to the first facial feature vector; performing nonlinear activation processing on the intermediate fusion feature vector corresponding to the first facial feature vector, and determining the facial fusion feature vector corresponding to the first facial feature vector.

[0135] Among them, the attention coefficients between the first face feature vector, the second face feature vector, the face expression feature vector and the face detail feature vector can be obtained by configuring the target graph attention network through the attention mechanism during the training process.

[0136] Optionally, the output of the feature association module may be K F-dimensional face fusion feature vectors, where K represents the number of categories of facial action units, and F may be 51, that is, each category of facial action unit corresponds to a face fusion feature vector.

[0137] In step 1041, the facial action intensity detection module includes multiple facial action intensity estimation networks, each of which is responsible for estimating the intensity level of a facial action unit and mapping the intensity estimation result of each facial action unit into a numerical value between 0 and 1, which is used as the intensity level of the facial action unit. According to each facial action unit and the intensity level of each facial action unit, the target expression of the person in the face image to be identified is determined.

[0138] Here, the facial action unit intensity estimation task is to evaluate the types of facial action units appearing in the input face image and their corresponding intensity levels. When estimating the intensity level of a facial action unit, the facial action intensity detection module needs to activate the last layer of the fully connected network with a Sigmoid function and map the intensity estimation result of each facial action unit into a value between 0 and 1 to represent the intensity level of the facial action unit.

[0139] In another embodiment of the present application, during the process of facial expression recognition, since the identity of the person in the data set for training the target facial expression recognition model is limited, in order to avoid overfitting the target facial expression recognition model to the identity features of the person, before obtaining the face image to be recognized, the target facial expression recognition model can be trained according to the following steps:

[0140] The initial facial expression recognition model is trained using multiple sample facial images to obtain an intermediate facial expression recognition model; wherein the global feature extraction module in the initial facial expression recognition model includes a facial expression feature extraction network, a facial detail feature extraction network, and a facial identity feature extraction network.

[0141] Here, the facial identity feature is used to reflect the identity information of the person in the face image to be identified.

[0142] During the training process of this stage, the global feature extraction module in the initial facial expression recognition model includes three extraction networks, namely, the facial expression feature extraction network, the facial detail feature extraction network and the facial identity feature extraction network. The initial facial expression recognition model is trained using multiple sample facial images to obtain an intermediate facial expression recognition model. Here, the facial identity feature extraction network is used to extract feature information representing the identity of the person from the sample facial images, namely, the facial identity feature vector.

[0143] Among them, the face identity feature extraction network is pre-trained based on the data set of sample face images and the corresponding face identity labels, and the output is the classification result of the person identity information in the sample face images.

[0144] The facial identity feature extraction network in the intermediate facial expression recognition model is removed, and the feature association module and the facial action intensity detection module in the intermediate facial expression recognition model without the facial identity feature extraction network are trained for the next round to obtain the trained target facial expression recognition model.

[0145] During the training process of this stage, for the intermediate facial expression recognition model obtained by training the initial facial expression recognition model in the first stage, the face identity feature extraction network of the global feature extraction module in the intermediate facial expression recognition model is removed, and only the facial expression feature extraction network and the facial detail feature extraction network in the global feature extraction module of the intermediate facial expression recognition model are retained, and the feature association module and the facial action intensity detection module in the intermediate facial expression recognition model are continued to be trained for the next round to obtain the trained target facial expression recognition model.

[0146] In related technologies, the data sets related to facial action units are usually relatively small, and there are generally only dozens of character identities in the data sets. Using a deep network for training on such a data set with fewer character identities can easily cause the model to overfit to a specific character identity. For example, the model is likely to remember the unique features and facial action units that appear in a person's facial picture.

[0147] In response to the above-mentioned problem of overfitting of face identity, the present application adopts a two-stage training process when training the facial expression recognition model: in the first stage, the face identity feature network is added to the initial facial expression recognition model for training, and the face identity feature information is provided to the model, so that the feature association module and the facial action intensity detection module that use the face identity features as input in the subsequent process are overfitted to the character; in the second stage, the present application loads the network parameters obtained in the first stage and removes the face identity feature extraction network, and then fixes the network parameters of the global feature extraction module. At this time, the parameters of all local feature extraction modules and the global feature extraction module are fixed, and only the subsequent feature association module and the facial action intensity detection module are trained. Since there is no input of the face identity feature vector at this time, the feature association module and the facial action intensity detection module will only rely on the input face local features, face expression features and face detail features to identify whether the action unit is activated, and will no longer use the face identity features. Therefore, the risk of overfitting the entire face expression recognition model to the face identity can be alleviated. In this way, when the finally trained target facial expression recognition model is used to perform expression recognition on facial images, since the features related to the facial identity are not extracted in the facial action unit intensity estimation task, but the information related to the details and expressions is extracted, the facial identity can be decoupled. Therefore, it will not be affected by the facial identity and the accuracy of facial expression recognition can be improved.

[0148] In another embodiment of the present application, before obtaining the face image to be recognized in step S101, a sample face image for training the facial expression recognition model may be determined according to the following steps 1011 to 1014:

[0149] Step 1011: Filter out m training face images corresponding to each sample person from the training data set.

[0150] Step 1012: for each of the sample persons, n candidate face images corresponding to each facial action unit in an activated state are screened out from the m training face images corresponding to the sample person, wherein m and n are both positive integers, and m>n.

[0151] Step 1013: For each of the sample persons, select target face images of each intensity level corresponding to each facial action unit from the n candidate face images corresponding to each facial action unit in an activated state.

[0152] Step 1014: determine the target facial image as the sample facial image.

[0153] In the specific implementation, we first uniformly sample from multiple characters in the training data set to obtain K character IDs, then sample the categories of facial action units for each character ID, and finally continue to sample sample images of various intensities under the sampled character IDs and facial action unit categories.

[0154] In the related art, facial action unit datasets often have very serious data imbalance problems, especially in the facial action unit intensity estimation task, the data imbalance problem is particularly serious, because there are many categories of facial action units in the dataset, and each facial action unit has multiple intensity values. Therefore, whether it is face identity, facial action unit categories or intensity, the distribution is not balanced. For current facial action unit-related tasks, the commonly used methods to solve data imbalance include data enhancement, data resampling, loss function weighting or transfer learning, but these methods cannot fully solve the impact of data imbalance on facial action unit-related tasks. Among them, the facial action unit dataset contains the video data, picture data of all sample characters, and the intensity value annotations of the facial action units of the corresponding frames.

[0155] To address the problem of unbalanced sample data, the present application adopts a three-level balanced data resampling method based on face identity, facial action unit category, and intensity level of facial action units, which can improve the balance of sample face images for model training. Therefore, it can ensure that the model is not dominated by samples of a certain face identity feature, a certain facial action unit category, or a certain intensity value and intensity level of the facial action unit during training, and can further improve the accuracy of facial expression recognition of the tail-type facial action recognition unit.

[0156] Figure 2 Flow chart of another facial expression recognition method provided by the embodiment of the present application. Figure 2 As shown, another facial expression recognition method provided by an embodiment of the present application includes the following steps:

[0157] S201: Obtain a face image to be recognized.

[0158] S202: based on the three-dimensional face model corresponding to the face image to be identified and definition information of each facial action unit, divide the region of interest image corresponding to each facial action unit from the face image to be identified.

[0159] S203, determining a face fusion feature vector based on the local face feature vectors extracted from each of the region of interest images, the facial expression feature vector and the facial detail feature vector extracted from the face image to be identified.

[0160] Here, steps S201 to S203 are substantially the same as steps S101 to S103, have the same implementation methods, and can achieve the same technical effects, and will not be described in detail herein.

[0161] S204: Determine each facial action unit in an activated state in the face image to be identified and an intensity level corresponding to each facial action unit in an activated state based on the face fusion feature vector.

[0162] S205: Determine a target expression of the person in the to-be-recognized face image according to each facial action unit in an activated state and an intensity level corresponding to each facial action unit in an activated state.

[0163] It should be noted that the local face feature vector corresponding to each facial action unit will be fused with the local face feature vectors corresponding to other facial action units, the facial expression feature vectors, and the facial detail feature vectors after passing through the feature association modeling module. Therefore, the last layer of the feature map network outputs the face fusion feature vector of each node, which contains all the information that is helpful for the single intensity estimation of the facial action, and uses each face fusion feature vector as input to estimate the intensity of each facial action unit. Here, the input of the facial action intensity detection module is the K F-dimensional face fusion feature vectors output by the feature association module (K is the number of AU categories, and F can be 51). For example, 12 AUs correspond to 12 local face feature vectors. After passing through the feature association module, 12 AU features are also output. At this time, the AU feature is a feature after aggregating other local face feature vectors and global features associated with the AU. This feature is used as input to predict the intensity value of the AU. Therefore, there are 12 AU intensity estimation networks, each responsible for estimating one AU).

[0164] Based on the same application concept, the embodiments of the present application also provide a facial expression recognition device corresponding to the facial expression recognition method provided in the above embodiments. Since the principle of solving the problem by the device in the embodiments of the present application is similar to the facial expression recognition method in the above embodiments of the present application, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0165] like Figure 3 , 4 As shown, Figure 3 One of the functional module diagrams of a facial expression recognition device 300 provided in an embodiment of the present application; Figure 4 FIG2 is a functional module diagram of a facial expression recognition device 300 provided in an embodiment of the present application. Figure 3As shown, the facial expression recognition device 300 includes: an acquisition unit 310; a division unit 320; a first determination unit 330 and a second determination unit 340, wherein:

[0166] An acquisition unit 310 is used to acquire a face image to be recognized;

[0167] A dividing unit 320 is used to divide the image of the human face to be identified into the region of interest images corresponding to the facial action units based on the three-dimensional face model corresponding to the facial image to be identified and the definition information of each facial action unit;

[0168] A first determining unit 330 is used to determine a face fusion feature vector based on the face local feature vectors extracted from each of the region of interest images, the face expression feature vector extracted from the face image to be identified, and the face detail feature vector;

[0169] The second determining unit 340 is used to determine the target expression of the person in the face image to be recognized based on the face fusion feature vector.

[0170] In a possible implementation, Figure 3 As shown, the division unit 320 is specifically used for:

[0171] Determine the three-dimensional face model corresponding to the face image to be identified according to the standard face model and the face image to be identified;

[0172] Dividing the three-dimensional face model into various facial local areas according to definition information of each facial action unit;

[0173] Based on the region mask information of each of the local facial regions, the region of interest images corresponding to each of the facial action units are divided from the face image to be recognized.

[0174] In a possible implementation, the target facial expression recognition model includes a local feature extraction module, a global feature extraction module, a feature association module, and a facial action intensity detection module; Figure 3 As shown, the first determining unit 330 is specifically used for:

[0175] For each of the region of interest images, inputting the region of interest image into the local feature extraction module to determine a local feature vector of the face corresponding to the region of interest image;

[0176] Inputting the face image to be recognized into the global feature extraction module to determine the facial expression feature vector and the facial detail feature vector;

[0177] Inputting each of the local face feature vectors, the facial expression feature vector and the facial detail feature vector into the feature association module to determine the face fusion feature vector;

[0178] like Figure 3 As shown, the second determining unit 340 is specifically configured to:

[0179] The face fusion feature vector is input into the facial action intensity detection module to determine the target expression of the person in the face image to be identified.

[0180] In a possible implementation, Figure 4 As shown, the facial expression recognition device 300 further includes a model training unit 350; the model training unit 350 is used to train the target facial expression recognition model according to the following steps:

[0181] Using multiple sample face images to train the initial face expression recognition model to obtain an intermediate face expression recognition model; wherein the global feature extraction module in the initial face expression recognition model includes a face expression feature extraction network, a face detail feature extraction network and a face identity feature extraction network;

[0182] The facial identity feature extraction network in the intermediate facial expression recognition model is removed, and the feature association module and the facial action intensity detection module in the intermediate facial expression recognition model without the facial identity feature extraction network are trained for the next round to obtain the trained target facial expression recognition model.

[0183] In a possible implementation, Figure 4 As shown, the facial expression recognition device 300 further includes a sample screening unit 360; the sample screening unit 360 is used to determine the sample face images for facial expression recognition model training according to the following steps:

[0184] From the training data set, m training face images corresponding to each sample person are selected;

[0185] For each of the sample persons, n candidate face images corresponding to each facial action unit in an activated state are screened out from the m training face images corresponding to the sample person;

[0186] For each of the sample persons, from the n candidate face images corresponding to each facial action unit in an activated state, select target face images of each intensity level corresponding to the facial action unit;

[0187] Determine the target face image as the sample face image;

[0188] Wherein, m and n are both positive integers, and m>n.

[0189] In a possible implementation, the local feature extraction module includes at least two convolutional networks and a fully connected network; Figure 3 As shown, the first determining unit 330 is specifically used for:

[0190] Inputting the region of interest image into a first convolutional network to determine a first local feature vector;

[0191] Inputting the first local feature vector into a second convolutional network to determine a second local feature vector;

[0192] Inputting the second local feature vector into the fully connected network to determine the local feature vector of the face;

[0193] The first convolutional network and the second convolutional network use different convolutional kernels.

[0194] In one possible implementation, the feature association module includes a target graph attention network; Figure 4 As shown, the model training unit 350 is further used to generate the target graph attention network according to the following steps:

[0195] Each sample face local feature vector, sample face expression feature vector and sample face detail feature vector are used as nodes of the initial graph attention network respectively;

[0196] Counting the number of times each of the facial action units co-occurs in a plurality of sample face images used for training;

[0197] According to the counted number of times each of the facial action units co-occurs, a connection relationship between each node in the initial graph attention network is constructed to generate the target graph attention network.

[0198] In a possible implementation, Figure 3 As shown, the first determining unit 330 is further used for:

[0199] For a first facial feature vector, determining attention coefficients between the first facial feature vector and the second facial feature vector, the facial expression feature vector, and the facial detail feature vector, respectively; the first facial feature vector and the second facial feature vector are different local facial feature vectors;

[0200] Using multiple attention coefficients corresponding to the first facial feature vector as weights, performing feature weighted calculation on the second facial feature vector, the facial expression feature vector, and the facial detail feature vector, to determine an intermediate fusion feature vector corresponding to the first facial feature vector;

[0201] Nonlinear activation processing is performed on the intermediate fusion feature vector corresponding to the first facial feature vector to determine the facial fusion feature vector corresponding to the first facial feature vector.

[0202] In a possible implementation, Figure 3 As shown, the second determining unit 340 is specifically configured to:

[0203] Based on the face fusion feature vector, determining each facial action unit in the to-be-recognized face image that is in an activated state, and an intensity level corresponding to each facial action unit in an activated state;

[0204] The target expression of the person in the face image to be recognized is determined according to each facial action unit in an activated state and the intensity level corresponding to each facial action unit in an activated state.

[0205] Based on the same application concept, see Figure 5 As shown, it is a structural schematic diagram of an electronic device 500 provided in an embodiment of the present application, including: a processor 510, a memory 520 and a bus 530, the memory 520 stores machine-readable instructions executable by the processor 510, when the electronic device 500 is running, the processor 510 and the memory 520 communicate through the bus 530, and the machine-readable instructions are executed by the processor 510 when running to perform the steps of the facial expression recognition method described in any of the above embodiments.

[0206] Specifically, when the machine-readable instruction is executed by the processor 510, the following processing may be performed:

[0207] Obtain a face image to be recognized;

[0208] Based on the three-dimensional face model corresponding to the face image to be identified and the definition information of each facial action unit, dividing the region of interest image corresponding to each facial action unit from the face image to be identified;

[0209] Determine a face fusion feature vector based on the face local feature vectors extracted from each of the region of interest images, the face expression feature vector and the face detail feature vector extracted from the face image to be identified;

[0210] Based on the face fusion feature vector, a target expression of the person in the face image to be recognized is determined.

[0211] Based on the same application concept, an embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the facial expression recognition method provided in the above embodiment are executed.

[0212] Specifically, the storage medium can be a general storage medium, such as a mobile disk, a hard disk, etc. When the computer program on the storage medium is run, the above-mentioned facial expression recognition method can be executed. By fusing the extracted local facial feature vectors closely related to the facial action units and the global facial feature vectors, facial expression recognition is performed, thereby improving the accuracy of facial expression recognition.

[0213] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, the specific working process of the system and device described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here. In the several embodiments provided in the present application, it should be understood that the disclosed system, device and method can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0214] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0215] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0216] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc., various media that can store program codes.

[0217] The above are only specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method for recognizing facial expressions, characterized in that: The facial expression recognition method comprises: Obtaining a face image to be recognized; Based on the three-dimensional face model corresponding to the face image to be identified and the definition information of each facial action unit, dividing the region of interest images corresponding to each facial action unit from the face image to be identified; wherein each region of interest image corresponds to one facial action unit; Determine a face fusion feature vector based on the face local feature vectors extracted from each of the region of interest images, the face expression feature vector and the face detail feature vector extracted from the face image to be identified; Based on the face fusion feature vector, a target expression of the person in the face image to be recognized is determined.

2. The facial expression recognition method according to claim 1, characterized in that: The step of dividing the image of the human face to be identified into the region of interest image corresponding to each of the facial action units based on the three-dimensional face model corresponding to the face image to be identified and the definition information of each facial action unit comprises: Determine the three-dimensional face model corresponding to the face image to be identified according to the standard face model and the face image to be identified; Dividing the three-dimensional face model into various facial local areas according to definition information of each facial action unit; Based on the region mask information of each of the local facial regions, the region of interest images corresponding to each of the facial action units are divided from the face image to be recognized.

3. The facial expression recognition method according to claim 1, characterized in that: The target facial expression recognition model includes a local feature extraction module, a global feature extraction module, a feature association module and a facial action intensity detection module; the facial fusion feature vector is determined based on the facial local feature vectors extracted from each of the region of interest images, the facial expression feature vector extracted from the face image to be recognized and the facial detail feature vector, including: For each of the region of interest images, inputting the region of interest image into the local feature extraction module to determine a local feature vector of the face corresponding to the region of interest image; Inputting the face image to be recognized into the global feature extraction module to determine the facial expression feature vector and the facial detail feature vector; Inputting each of the local face feature vectors, the facial expression feature vector and the facial detail feature vector into the feature association module to determine the face fusion feature vector; The step of determining a target expression of a person in the face image to be identified based on the face fusion feature vector comprises: The face fusion feature vector is input into the facial action intensity detection module to determine the target expression of the person in the face image to be identified.

4. The facial expression recognition method according to claim 3, characterized in that: Before obtaining the face image to be recognized, the target face expression recognition model is trained according to the following steps: Using multiple sample face images to train the initial face expression recognition model to obtain an intermediate face expression recognition model; wherein the global feature extraction module in the initial face expression recognition model includes a face expression feature extraction network, a face detail feature extraction network and a face identity feature extraction network; The facial identity feature extraction network in the intermediate facial expression recognition model is removed, and the feature association module and the facial action intensity detection module in the intermediate facial expression recognition model without the facial identity feature extraction network are trained for the next round to obtain the trained target facial expression recognition model.

5. The facial expression recognition method according to claim 3, characterized in that: Before obtaining the face image to be recognized, a sample face image for training the facial expression recognition model is determined according to the following steps: From the training data set, m training face images corresponding to each sample person are selected; For each of the sample persons, n candidate face images corresponding to each facial action unit in an activated state are screened out from the m training face images corresponding to the sample person; For each of the sample persons, from the n candidate face images corresponding to each facial action unit in an activated state, select target face images of each intensity level corresponding to the facial action unit; Determine the target face image as the sample face image; Wherein, m and n are both positive integers, and m>n.

6. The facial expression recognition method according to claim 3, characterized in that: The local feature extraction module includes at least two convolutional networks and a fully connected network; for each of the region of interest images, the region of interest image is input into the local feature extraction module to determine the local feature vector of the face corresponding to the region of interest image, including: Inputting the region of interest image into a first convolutional network to determine a first local feature vector; Inputting the first local feature vector into a second convolutional network to determine a second local feature vector; Inputting the second local feature vector into the fully connected network to determine the local feature vector of the face; The first convolutional network and the second convolutional network use different convolutional kernels.

7. The facial expression recognition method according to claim 3, characterized in that: The feature association module includes a target graph attention network; The target graph attention network is generated according to the following steps: Each sample face local feature vector, sample face expression feature vector and sample face detail feature vector are used as nodes of the initial graph attention network respectively; Counting the number of times each of the facial action units co-occurs in a plurality of sample face images used for training; According to the counted number of times each of the facial action units co-occurs, a connection relationship between each node in the initial graph attention network is constructed to generate the target graph attention network.

8. The facial expression recognition method according to claim 3, characterized in that: The step of inputting each of the local face feature vectors, the facial expression feature vector, and the facial detail feature vector into the feature association module to determine the face fusion feature vector comprises: For a first facial feature vector, determining attention coefficients between the first facial feature vector and the second facial feature vector, the facial expression feature vector, and the facial detail feature vector, respectively; the first facial feature vector and the second facial feature vector are different local facial feature vectors; Using multiple attention coefficients corresponding to the first facial feature vector as weights, performing feature weighted calculation on the second facial feature vector, the facial expression feature vector, and the facial detail feature vector, to determine an intermediate fusion feature vector corresponding to the first facial feature vector; Nonlinear activation processing is performed on the intermediate fusion feature vector corresponding to the first facial feature vector to determine the facial fusion feature vector corresponding to the first facial feature vector.

9. The facial expression recognition method according to claim 1, characterized in that: The step of determining a target expression of a person in the face image to be identified based on the face fusion feature vector comprises: Based on the face fusion feature vector, determining each facial action unit in the to-be-recognized face image that is in an activated state, and an intensity level corresponding to each facial action unit in an activated state; The target expression of the person in the face image to be recognized is determined according to each facial action unit in an activated state and the intensity level corresponding to each facial action unit in an activated state.

10. A facial expression recognition device, characterized in that: The facial expression recognition device comprises: An acquisition unit, used for acquiring a face image to be recognized; A dividing unit, configured to divide the face image to be identified into region of interest images corresponding to each of the facial action units, based on the three-dimensional face model corresponding to the face image to be identified and the definition information of each facial action unit; wherein each region of interest image corresponds to one facial action unit; A first determining unit is used to determine a face fusion feature vector based on the face local feature vectors extracted from each of the region of interest images, the face expression feature vector extracted from the face image to be identified, and the face detail feature vector; The second determining unit is used to determine the target expression of the person in the face image to be identified based on the face fusion feature vector.

11. An electronic device, characterized in that: include: A processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate through the bus, and the machine-readable instructions are executed by the processor to perform the steps of the facial expression recognition method as described in any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the facial expression recognition method according to any one of claims 1 to 9 are executed.

Images