Expression recognition model, method and system based on facial key point feature fusion

Abstract

The present invention relates to the technical field of facial expression recognition of facial images, and in particular to an expression recognition model, method and system combined with facial key point feature fusion. The present invention proposes a training method for an expression recognition model combined with facial key point feature fusion, wherein the facial feature extraction module is a pre-trained model with fixed parameters; the expression feature extraction module, the feature fusion module and the expression classification module constitute the part to be trained of the basic model. During the model training process, only the part to be trained is trained. In the present invention, in a method based on deep learning, the pre-trained facial feature extraction module is used to extract facial features from facial images to improve attention to facial details, thereby reducing the sensitivity of the model to image scale and helping to promote the distinctiveness between samples between classes and the invariance of differences between samples within a class.

Description

Technical Field

[0001] The present invention relates to the technical field of facial expression recognition of human face images, and in particular to an expression recognition model, method and system combining the fusion of facial key point features. Background Art

[0002] Most traditional methods use features such as HOG, LBP and SIFT for facial expression recognition. However, these features often do not have good discriminative information, making the model not robust and accurate enough. At present, deep learning-based methods dominate facial expression recognition. Although these methods have made great progress, there are still some challenges: 1) Inter-class similarity, that is, there are subtle changes between similar images, often belonging to different expression categories, but are recognized as the same expression category; 2) Intra-class differences, that is, within the same expression category, images may have significant differences, resulting in different labels; 3) Scale sensitivity, that is, in practical applications, deep learning-based models are usually very sensitive to image quality and resolution, and often lead to recognition errors due to image quality and resolution. Summary of the invention

[0003] In order to overcome the defect that the accuracy of the expression recognition method based on deep learning in the above-mentioned prior art needs to be improved, the present invention proposes a training method for an expression recognition model combined with the fusion of facial key point features. The expression recognition model obtained by the training method can reduce the model's sensitivity to image scale and help promote the distinctiveness between samples between classes and the invariance of differences between samples within a class.

[0004] To achieve the above object, the present invention adopts the following technical solutions, including:

[0005] The present invention proposes a method for training an expression recognition model combining facial key point feature fusion, comprising the following steps:

[0006] S1. Construct a basic model and an image sample data set; the image samples in the image sample data set are face images marked with expression categories; the basic model includes a face feature extraction module, an expression feature extraction module, a feature fusion module and an expression classification module; the face feature extraction module is a pre-trained model with fixed parameters, which extracts face features X based on face images and outputs them; the expression feature extraction module, the feature fusion module and the expression classification module constitute the to-be-trained part of the basic model;

[0007] The expression feature extraction module extracts expression feature Y based on the face image and outputs it; the input of the feature fusion module is connected to the output of the face feature extraction module and the output of the expression feature extraction module respectively, the feature fusion module performs feature fusion based on the face feature X and the expression feature Y, and the feature fusion module outputs the fusion feature;

[0008] The input of the expression classification module is connected to the output of the feature fusion module, and the expression classification module recognizes the expression category based on the fusion features and outputs it;

[0009] S2. Let the basic model perform machine learning on the image samples in the image sample data set, and reversely update the model parameters of the training part in combination with the set loss function until the basic model converges, and use the converged basic model as the expression recognition model.

[0010] Preferably, the feature fusion module includes a local attention module, a global attention module and an activation function; the feature fusion module obtains the fusion feature f(N, X, Y) through the following steps S11-S14; N is the set fusion number threshold; X∈R C×H×W , Y∈R C×H×W , R C×H×W Represents image feature data with a C-dimensional length of H and a width of W;

[0011] S11, when the feature fusion module performs the nth fusion, f(n-1,X,Y) is input into the feature fusion module; the initial value of n is 1, and f(0,X,Y) is the image feature data after the elements of X and Y along the channel dimension are added;

[0012] S12, the local attention module extracts the local attention feature fl(n-1,X,Y) of f(n-1,X,Y), and the global attention module extracts the global attention feature gl(n-1,X,Y) of f(n-1,X,Y); fl(n-1,X,Y)∈R C×H×W ,gl(n-1,X,Y)∈R C×H×W ;

[0013] S13, adding the local attention feature fl(n-1,X,Y) and the global attention feature gl(n-1,X,Y) along the channel dimension elements and activating them through the activation function, substituting the activated feature data into the set calculation formula to obtain the face feature X and the expression feature Y after the nth fusion feature f(n,X,Y);

[0014] Set the calculation formula to:

[0015] f(n,X,Y)=X×M(f(n-1,X,Y))+Y×[{1} C×H×W -M(f(n-1,X,Y))]

[0016] Among them, {1} C×H×WRepresents the unit matrix with C-dimensional length H and width W; n is the number of fusions, 1≤n≤N; f(0,X,Y) is the image feature data after adding the elements of X and Y along the channel dimension; M(f(n-1,X,Y)) represents the activated feature data, M(f(n-1,X,Y))∈R C×H×W ;

[0017] S14. Determine whether n is equal to the set value N; if not, update n to n+1, and then return to step S11; if yes, the feature fusion module outputs f(n,X,Y) as the fused feature f(N,X,Y).

[0018] Preferably, the activation function is a sigmoid activation function, with N=2.

[0019] Preferably, the feature fusion module adds the facial features X and expression features Y of the same dimension along the channel dimension to obtain the fused features; or, the feature fusion module concatenates the facial features X and expression features Y along the channel dimension to obtain the fused features; or, the feature fusion module includes one or more attention feature extraction modules connected in sequence.

[0020] Preferably, the expression classification module includes a pooling layer and a fully connected classification network, and the pooling layer performs pooling processing on the fusion features to obtain pooling data h1∈R C×1×1 ,The fully connected classification network obtains the expression probability distribution based on the pooled data. The expression probability distribution includes the probability of each expression category corresponding to the face image, and the sum of all probabilities is 1.

[0021] Preferably, the acquisition of the image sample data set in S1 includes the following steps:

[0022] SA1. Obtain a face image of a known expression category from a standard facial expression dataset as a reference image, resize the obtained reference image, and obtain scaled images of different sizes;

[0023] SA2, normalize the scaled image to obtain a standard size image;

[0024] SA3. Use one or more image processing methods to process the standard size image; the image processing methods include flipping, rotating, adding noise and grayscale processing;

[0025] SA4. Associating the processed standard-size image with the expression category as an image sample, and collecting all the image samples to form an image sample data set; the expression category associated with the processed standard-size image is the expression category of its corresponding reference image.

[0026] Preferably, the loss function set in S2 is a cross entropy loss function; S2 includes the following sub-steps:

[0027] S21, dividing the image sample data set into a training set and a test set;

[0028] S22, selecting n1 image samples from the training set as training samples, and fitting the basic model to the training samples to train the model parameters; n1 is a set value;

[0029] S23, select n2 image samples from the test set as test samples, input the face images in the test samples into the basic model, and the basic model outputs the expression recognition results; n2 is a set value;

[0030] S24, determine whether the basic model has converged; if not, calculate the cross entropy loss based on the recognition results of the test samples, reversely update the parameters of the basic model according to the cross entropy loss, and then return to step S22; if yes, use the converged basic model as the expression recognition model.

[0031] Preferably, the facial feature extraction module is used to extract facial key point features as facial features.

[0032] The present invention proposes an expression recognition method combining facial key point feature fusion, which overcomes the defects of existing expression recognition such as inter-class similarity, intra-class difference and size sensitivity. The expression recognition method first obtains an expression recognition model by combining the training method of the expression recognition model combining facial key point feature fusion; then the face image to be recognized is input into the expression recognition module, and the expression category corresponding to the maximum probability in the expression probability distribution output by the expression recognition model is obtained as the expression recognition result.

[0033] The present invention proposes an expression recognition system combining the fusion of facial key point features, which provides a carrier for the promotion of the above-mentioned expression recognition method combining the fusion of facial key point features. The system stores an expression recognition module and a computer program, and the computer program is used to implement the above-mentioned expression recognition method combining the fusion of facial key point features when executed.

[0034] The advantages of the present invention are:

[0035] (1) The present invention proposes a training method for an expression recognition model that combines facial key point feature fusion. In a deep learning-based method, a pre-trained facial feature extraction module is used to extract facial features based on facial images to improve attention to facial details, reduce the model's sensitivity to image scale, and help promote the distinctiveness between inter-class samples and the invariance of differences between intra-class samples.

[0036] (2) The facial feature extraction module extracts the key point features of the facial image and can provide a sparse representation of the key facial area. It then fuses the features with the image visual features, i.e., the expression features, to obtain more discriminative facial expression image visual features, thereby effectively improving the accuracy of facial expression recognition.

[0037] (3) The feature fusion module of the present invention performs multiple attention feature fusions on the facial feature X and the expression feature Y, and performs fusion based on local attention and global attention, thereby achieving deep fusion of the facial feature X and the expression feature Y, thereby making the fused feature more discriminative. When performing expression recognition based on the fused feature, the influence of different facial features on expression recognition is taken into account, overcoming the inter-class similarity and intra-class difference, and greatly improving the expression recognition accuracy.

[0038] (4) The expression classification module first performs pooling processing on the fused features, compresses the features, removes redundant information, reduces the computational complexity of the model, and helps expand the model's field of view, thereby improving the model's generalization ability.

[0039] (5) The present invention achieves an increase in the number of face images of known categories by scaling and normalizing the size of the reference image, which facilitates the expansion of the number of image samples. Through image processing, the scenes of face images in the image samples are enriched, which is conducive to improving the robustness of the model.

[0040] (6) A facial expression recognition method that combines the fusion of facial key point features, using the facial expression recognition model provided by the present invention to perform facial expression recognition, has higher facial expression recognition accuracy, and overcomes the defects of existing facial expression recognition such as inter-class similarity, intra-class difference and size sensitivity. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 This is the structure diagram of the expression recognition model;

[0042] Figure 2 This is the working process of the fifth feature fusion module;

[0043] Figure 3 Flowchart of the training method for the expression recognition model;

[0044] Figure 4 is a flow chart of a method for obtaining an image sample data set;

[0045] Figure 5 It is a comparison chart of the accuracy of 7 models in the embodiment. DETAILED DESCRIPTION

[0046] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0047] like Figure 1 As shown, an expression recognition model proposed in this embodiment includes a face feature extraction module, an expression feature extraction module, a feature fusion module and an expression classification module; the face feature extraction module extracts and outputs a face feature X based on a face image; the expression feature extraction module extracts and outputs an expression feature Y based on a face image; the input of the feature fusion module is respectively connected to the output of the face feature extraction module and the output of the expression feature extraction module, the feature fusion module performs feature fusion based on the face feature X and the expression feature Y, and the feature fusion module outputs a fusion feature after the face feature X and the expression feature Y are fused; the input of the expression classification module is connected to the output of the feature fusion module, and the expression classification module recognizes and outputs the face expression based on the fusion feature;

[0048] Specifically, the expression classification module includes a pooling layer and a fully connected classification network. The pooling layer performs pooling processing on the fusion features to obtain the pooling data h1∈R C×1×1 , the fully connected classification network obtains the expression probability distribution based on the pooled data h1. The expression probability distribution includes the probability of each expression category corresponding to the face image, and the sum of all probabilities is 1. h1∈R C×1×1 Indicates that h1 is C-dimensional image feature data with both length and width of 1.

[0049] In this implementation, the feature fusion module has multiple implementation methods.

[0050] The first implementation method: The feature fusion module concatenates the facial feature X and the expression feature Y along the channel dimension to obtain the fusion feature. At this time, assuming that the facial feature X extracted by the facial feature extraction module is the image feature data of C1 dimension with a length of H and a width of W, and the expression feature Y extracted by the expression feature extraction module is the image feature data of C2 dimension with a length of H and a width of W, then the fusion feature is the image feature data of C1+C2 dimension with a length of H and a width of W; that is, X∈R C1×H×W , Y∈R C2×H×W , fusion features ∈ R (C1+C2)×H×W .

[0051] The second implementation method: Set X∈R C×H×W , Y∈R C×H×W , R C×H×WRepresents the image feature data with a length of H and a width of W in C dimension; the feature fusion module adds the face feature X and the expression feature Y along the channel dimension elements to obtain the fusion feature, the fusion feature ∈R C×H×W .

[0052] The third implementation method: The feature fusion module uses the attention extraction module to add or concatenate the facial feature X and the expression feature Y along the channel dimension as dual feature data, and extracts the attention value of the dual feature data through the attention extraction module as the fusion feature and outputs it.

[0053] The fourth implementation method: the feature fusion module includes multiple attention extraction modules, and the multiple attention extraction modules are connected in sequence. Each attention extraction module extracts an attention value based on the input data. The input of the latter attention extraction module is the output of the previous attention extraction module. The input of the first attention extraction module is the dual feature data formed by adding or concatenating the facial feature X and the expression feature Y along the channel dimension. The output of the last attention extraction module is the fusion feature output by the feature fusion module.

[0054] The fifth implementation method: the feature fusion module outputs the fused feature f(N,X,Y) after N fusions; the calculation formula of the fused feature f(N,X,Y) is as follows:

[0055] f(n,X,Y)=X×M(f(n-1,X,Y))+Y×[{1} C×H×W -M(f(n-1,X,Y))] (1)

[0056] {1} C×H×W represents the unit matrix with C-dimensional length H and width W; n is the number of fusions, 1≤n≤N; f(0,X,Y) is the image feature data after adding the elements of X and Y along the channel dimension; M(f(n-1,X,Y)) represents the attention feature value corresponding to f(n-1,X,Y), M(f(n-1,X,Y))∈R C×H×W .

[0057] When the feature fusion module is implemented using the fifth implementation method, the feature fusion module can be set to include a local attention module, a global attention module and an activation function; the input of the local attention module and the input of the global attention module are both connected to the input of the feature fusion module; the input of the activation function is respectively connected to the output of the local attention module and the output of the global attention module; the output of the feature fusion module is connected to the output of the activation function.

[0058] The local attention module is used to extract the local attention after the fusion of facial feature X and expression feature Y. The local attention after X and Y are fused n times is recorded as fl(n-1,X,Y);

[0059] The global attention module is used to extract the global attention after the fusion of facial feature X and expression feature Y. The global attention after X and Y are fused n times is recorded as gl(n-1,X,Y);

[0060] The activation function is used to activate the feature values ​​of the local attention fl(n-1,X,Y) and the global attention gl(n-1,X,Y) added along the channel dimension elements to output the fused feature f(n,X,Y).

[0061] 1≤n≤N, the output of the feature fusion module is f(N,X,Y).

[0062] like Figure 2 As shown, the working process of the fifth feature fusion module is shown in steps S11-S14.

[0063] S11, when the feature fusion module performs the nth fusion, f(n-1,X,Y) is input into the feature fusion module; the initial value of n is 1, and f(0,X,Y) is the image feature data after the elements of X and Y along the channel dimension are added.

[0064] S12, the local attention module extracts the local attention feature fl(n-1,X,Y) of f(n-1,X,Y), and the global attention module extracts the global attention feature gl(n-1,X,Y) of f(n-1,X,Y); fl(n-1,X,Y)∈R C×H×W ,gl(n-1,X,Y)∈R C×H×W .

[0065] S13, add the local attention feature fl(n-1,X,Y) and the global attention feature gl(n-1,X,Y) along the channel dimension elements and activate them through the activation function to obtain the activated feature data M(f(n-1,X,Y)), and then substitute the activated feature data into the set calculation formula (1) to obtain the feature f(n,X,Y) after the nth fusion of the facial feature X and the expression feature Y;

[0066] In this implementation, the activation function selects the sigmoid activation function;

[0067] M(f(n-1,X,Y))=sigmoid(fl(n-1,X,Y)+gl(n-1,X,Y))

[0068] fl(n-1,X,Y)+gl(n-1,X,Y) means adding fl(n-1,X,Y) and gl(n-1,X,Y) elements along the channel dimension.

[0069] S14. Determine whether n is equal to the set value N; if not, update n to n+1, and then return to step S11; if yes, the feature fusion module outputs f(n,X,Y) as the fused feature f(N,X,Y).

[0070] like Figure 3 As shown, the training method of the above expression recognition model includes the following steps S1-S2.

[0071] S1. Build the basic model and image sample dataset.

[0072] The structure of the basic model is the same as the structure of the above-mentioned expression recognition model.

[0073] The image sample dataset is used to store image samples, which are face images labeled with expression categories. Figure 4 As shown, image samples can be obtained based on an existing standard facial expression dataset, and specific reference may be made to the following steps SA1-SA4.

[0074] SA1. Obtain a face image of a known expression category from a standard facial expression dataset as a reference image, resize the obtained reference image, and obtain scaled images of different sizes. In this way, the number of face images can be increased based on the reference image by scaling the size.

[0075] SA2. Normalize the scaled image to obtain a standard size image. The size of the standard size image here is the size of the face image input to the basic model.

[0076] SA3. Use one or more image processing methods to process the standard size image; the image processing methods include flipping, rotating, adding noise and grayscale processing;

[0077] SA4. Associating the processed standard-size image with the expression category as an image sample, and collecting all the image samples to form an image sample data set; the expression category associated with the processed standard-size image is the expression category of its corresponding reference image.

[0078] In this way, by scaling and normalizing the size of the reference image, the number of face images of known categories is increased, which is convenient for expanding the number of image samples. Through image processing, the scenes of face images in the image samples are enriched, which is conducive to improving the generalization performance of the model.

[0079] S2. Combine the following steps S21-S24 to enable the basic model to perform machine learning on the image samples in the image sample data set so that the basic model converges, and use the converged basic model as the expression recognition model.

[0080] S21. Divide the image sample dataset into a training set and a test set.

[0081] S22, selecting n1 image samples from the training set as training samples, and fitting the basic model to the training samples to train the model parameters; n1 is a set value.

[0082] S23, select n2 image samples from the test set as test samples, input the face images in the test samples into the basic model, and the basic model outputs the expression recognition results; n2 is a set value.

[0083] S24, determine whether the basic model has converged; if not, calculate the cross entropy loss based on the recognition results of the test samples, reversely update the parameters of the basic model according to the cross entropy loss, and then return to step S22; if yes, use the converged basic model as the expression recognition model.

[0084] The convergence condition of the basic model can be set to the cross entropy loss being lower than the set loss, or it can be set to the number of basic model iterations reaching a set value.

[0085] The five expression recognition models and two comparative models proposed in the present invention are verified in combination with the RAFDB standard dataset and the FERPlus standard dataset.

[0086] RAFDB standard dataset: has 7 expression categories, namely neutral (NE), happy (HA), sad (SA), surprised (SU), afraid (FE), angry (AN), and disgust (DI). The number of samples in the training set is 12271, and the number of samples in the test set is 3068.

[0087] FERPlus standard dataset: It has 8 expression categories. Compared with the 7 expression categories of the RAFDB standard dataset, the FERPlus standard dataset adds contempt (CO, Contempt); the number of training set samples is 28709, and the number of test set samples is 3589.

[0088] The five expression recognition models proposed in this invention are:

[0089] Model MFFconcat: the expression recognition model proposed in the present invention, and the feature fusion module adopts the first implementation mode;

[0090] Model MFFdaf: the expression recognition model proposed in the present invention, and the feature fusion module adopts the second implementation mode;

[0091] Model MFFaff: the expression recognition model proposed in the present invention, and the feature fusion module adopts the third implementation mode;

[0092] Model MFFiaff: The expression recognition model proposed in the present invention, the feature fusion module adopts the fourth implementation mode, and the feature fusion module includes two attention extraction modules;

[0093] Model MFFiaff#: The expression recognition model proposed in the present invention, the feature fusion module adopts the fifth implementation method.

[0094] The facial feature extraction modules of the five models proposed in this embodiment adopt a facial key point feature extraction module to extract features of 68 key points of the labeled face.

[0095] The two comparison models are:

[0096] Model baseline: ResNet18 is used to perform machine learning on the constructed image sample dataset to obtain a recognition model with face images as input and expression categories as output;

[0097] Model MFFiaff-2: The structure of the model MFFiaff-2 is the same as that of the model MFFiaff provided in the present invention, but during the training process, the facial feature extraction module of the model MFFiaff-2 also performs parameter update.

[0098] In this embodiment, an image sample dataset constructed by using the above steps SA1-SA4 based on the RAFDB standard dataset is recorded as the first image sample dataset; an image sample dataset constructed by using the above steps SA1-SA4 based on the FERPlus standard dataset is recorded as the second image sample dataset.

[0099] In this embodiment, the image sample data set is divided into a training set and a test set, and the model is trained based on the training set and the test set using the above steps S21-S24 until convergence, and the accuracy of the model when it converges is obtained as an evaluation indicator.

[0100] In this embodiment, two groups of experiments are conducted. In the first group of experiments, the above seven models are used for machine learning based on the first image sample data set;

[0101] In the second set of experiments, the above seven models were used for machine learning based on the second image sample dataset.

[0102] In this embodiment, the test results of the accuracy of the seven models on two standard data sets are shown in Tables 1 and Figure 5 shown.

[0103] Table 1: Model test results statistics

[0104]

[0105] Combined with statistics, it can be seen that the accuracy of the five models proposed in the present invention is higher than that of the comparison model, which proves that the model training method of the present invention with a fixed face feature extraction module is conducive to improving the accuracy of expression recognition. The model MFFiaff# using the fifth feature fusion module performs best, proving the superiority of the feature fusion method proposed in the present invention.

[0106] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A training method for an expression recognition model combining facial key point feature fusion, characterized in that: The following steps are involved: S1. Construct a basic model and an image sample data set; the image samples in the image sample data set are face images annotated with expression categories; the basic model includes a face feature extraction module, an expression feature extraction module, a feature fusion module and an expression classification module; The face feature extraction module is a pre-trained model with fixed parameters, which extracts and outputs face features X based on face images; The expression feature extraction module, feature fusion module and expression classification module constitute the training part of the basic model; The expression feature extraction module extracts expression feature Y based on the face image and outputs it; the input of the feature fusion module is connected to the output of the face feature extraction module and the output of the expression feature extraction module respectively, the feature fusion module performs feature fusion based on the face feature X and the expression feature Y, and the feature fusion module outputs the fusion feature; The input of the expression classification module is connected to the output of the feature fusion module, and the expression classification module recognizes the expression category based on the fusion features and outputs it; S2, let the basic model perform machine learning on the image samples in the image sample data set, and reversely update the model parameters of the to-be-trained part in combination with the set loss function until the basic model converges, and use the converged basic model as the expression recognition model; The feature fusion module includes a local attention module, a global attention module and an activation function; the feature fusion module obtains the fusion feature f(N,X,Y) through the following steps S11-S14; N is the set fusion number threshold; X∈R C×H×W , Y∈R C×H×W , R C×H×W Represents image feature data with a C-dimensional length of H and a width of W; S11, when the feature fusion module performs the nth fusion, f(n-1,X,Y) is input into the feature fusion module; the initial value of n is 1, and f(0,X,Y) is the image feature data after the elements of X and Y along the channel dimension are added; S12, the local attention module extracts the local attention feature fl(n-1,X,Y) of f(n-1,X,Y), and the global attention module extracts the global attention feature gl(n-1,X,Y) of f(n-1,X,Y); fl(n-1,X,Y)∈R C×H×W ,gl(n-1,X,Y)∈R C×H×W ; S13, adding the local attention feature fl(n-1,X,Y) and the global attention feature gl(n-1,X,Y) along the channel dimension elements and activating them through the activation function, substituting the activated feature data into the set calculation formula to obtain the face feature X and the expression feature Y after the nth fusion feature f(n,X,Y); Set the calculation formula to: f(n,X,Y)=X×M(f(n-1,X,Y))+Y×[{1} C×H×W -M(f(n-1,X,Y))] Among them, {1} C×H×W Represents the unit matrix with C-dimensional length H and width W; n is the number of fusions, 1≤n≤N; f(0,X,Y) is the image feature data after adding the elements of X and Y along the channel dimension; M(f(n-1,X,Y)) represents the activated feature data, M(f(n-1,X,Y))∈R C×H×W ; S14. Determine whether n is equal to the set value N; if not, update n to n+1, and then return to step S11; if yes, the feature fusion module outputs f(n,X,Y) as the fused feature f(N,X,Y).

2. The training method of the expression recognition model combined with the fusion of facial key point features as claimed in claim 1, characterized in that: The activation function selects the sigmoid activation function, N=2.

3. The training method of the expression recognition model combined with the fusion of facial key point features as claimed in claim 1, characterized in that: The expression classification module includes a pooling layer and a fully connected classification network. The pooling layer performs pooling processing on the fusion features to obtain the pooling data h1∈R C×1×1 ,The fully connected classification network obtains the expression probability distribution based on the pooled data. The expression probability distribution includes the probability of each expression category corresponding to the face image, and the sum of all probabilities is 1.

4. The training method of the expression recognition model combined with the fusion of facial key point features as claimed in claim 1, characterized in that: The acquisition of the image sample dataset in S1 includes the following steps: SA1. Obtain a face image of a known expression category from a standard facial expression dataset as a reference image, resize the obtained reference image, and obtain scaled images of different sizes; SA2, normalize the scaled image to obtain a standard size image; SA3. Use one or more image processing methods to process the standard size image; the image processing methods include flipping, rotating, adding noise and grayscale processing; SA4. Associating the processed standard-size image with the expression category as an image sample, and collecting all the image samples to form an image sample data set; the expression category associated with the processed standard-size image is the expression category of its corresponding reference image.

5. The training method of the expression recognition model combined with the fusion of facial key point features as claimed in claim 1, characterized in that: The loss function set in S2 is the cross entropy loss function; S2 includes the following steps: S21, dividing the image sample data set into a training set and a test set; S22, selecting n1 image samples from the training set as training samples, and fitting the basic model to the training samples to train the model parameters; n1 is a set value; S23, select n2 image samples from the test set as test samples, input the face images in the test samples into the basic model, and the basic model outputs the expression recognition results; n2 is a set value; S24, determine whether the basic model converges; if not, calculate the cross entropy loss based on the recognition result of the test sample, reversely update the parameters of the basic model according to the cross entropy loss, and then return to step S22; If yes, the converged basic model is used as the expression recognition model.

6. The training method of the expression recognition model combined with the fusion of facial key point features as claimed in claim 1, characterized in that: The face feature extraction module is used to extract the key point features of the face as face features.

7. A facial expression recognition method combining facial key point feature fusion, characterized in that: First, an expression recognition model is obtained by combining the training method of the expression recognition model combined with the fusion of facial key point features as described in any one of claims 1 to 6; then the face image to be recognized is input into the expression recognition module, and the expression category corresponding to the maximum probability in the expression probability distribution output by the expression recognition model is obtained as the expression recognition result.

8. An expression recognition system combining facial key point feature fusion, characterized in that: An expression recognition module and a computer program are stored, and when the computer program is executed, it is used to implement the expression recognition method combined with the fusion of facial key point features as claimed in claim 7.

Images