Brain-inspired facial expression perception neural network model acquisition method and system

Abstract

The present disclosure relates to the field of computer vision and the field of brain neuroscience technology, and particularly relates to a brain-inspired expression perception neural network model acquisition method and system. The brain-inspired expression perception neural network model acquisition method comprises: acquiring at least one face region training data set; controlling at least one classifier channel in an initial expression perception neural network model to learn the at least one face region training data set respectively, to obtain at least one active parameter corresponding to the at least one classifier channel, wherein the face region training data set and the classifier channel are one-to-one corresponding; and determining a target expression perception neural network model according to the at least one active parameter. The present disclosure can establish a brain-inspired expression perception neural network model, which is used for expression recognition according to the contribution degree of a face region to a specific expression, so as to improve the accuracy of expression recognition.

Description

Technical Field

[0001] This disclosure relates to the fields of computer vision and neuroscience, and in particular to a method and system for obtaining a brain-inspired neural network model for facial expression perception. Background Technology

[0002] In related technologies, research on facial expressions treats the face as a whole, because there are indeed some interrelationships between different facial regions in expression and recognition. However, several studies involving neuroimaging, psychology, and neuroscience have found that different facial regions are responsible for expressing different facial expressions. This indicates that different facial regions contribute differently to specific expressions, and in single-category expression perception, the human brain shows a preference for different facial regions in the expression perception process.

[0003] However, all existing models treat the face as a whole for expression recognition, resulting in low accuracy. Therefore, how to establish a brain-inspired neural network model for expression perception, which can recognize expressions based on the contribution of facial regions to specific expressions, in order to improve the accuracy of expression recognition, has become a focus of attention. Summary of the Invention

[0004] This disclosure provides a method and system for obtaining a brain-inspired neural network model for facial expression perception. The main purpose is to establish a brain-inspired neural network model for facial expression perception, which is used to perform facial expression recognition based on the contribution of facial regions to specific facial expressions, thereby improving the accuracy of facial expression recognition.

[0005] According to one aspect of this disclosure, a method for obtaining a brain-inspired neural network model for facial expression perception is provided, comprising:

[0006] Obtain at least one face region training dataset;

[0007] At least one classifier channel in the initial facial expression perception neural network model is controlled to learn from the at least one face region training dataset to obtain at least one active parameter corresponding to the at least one classifier channel, wherein the face region training dataset and the classifier channel correspond one-to-one;

[0008] The target facial expression perception neural network model is determined based on the at least one active parameter.

[0009] Optionally, the at least one face region training dataset includes an eye region dataset, a mouth region dataset, and a whole face dataset, and obtaining the at least one face region training dataset includes:

[0010] Obtain an initial training dataset, wherein the initial training dataset includes at least one face image;

[0011] The initial training dataset is preprocessed to transform it into the eye region dataset, the mouth region dataset, and the overall face dataset.

[0012] Optionally, controlling at least one classifier channel in the initial expression-aware neural network model to learn from the at least one face region training dataset includes:

[0013] Using stochastic gradient descent, at least one classifier channel in the initial facial expression perception neural network model is controlled to iteratively learn the training dataset of at least one face region.

[0014] Optionally, the step of using stochastic gradient descent to iteratively learn the at least one classifier channel in the initial expression-perceived neural network model on the at least one face region training dataset includes:

[0015] In any iteration, the training dataset of at least one face region is input into the at least one classifier channel to obtain the loss value corresponding to the initial expression perception neural network model;

[0016] If the loss value satisfies the parameter update condition, the model parameters corresponding to the initial facial expression perception neural network model are updated, and the next iteration is performed.

[0017] If the loss value does not meet the parameter update conditions, then proceed directly to the next iteration until the number of iterations reaches the iteration threshold.

[0018] Optionally, the at least one face region training dataset includes an eye region dataset, a mouth region dataset, and a whole face dataset; the at least one classifier channel includes a first classifier channel, a second classifier channel, and a third classifier channel; and the step of inputting the at least one face region training dataset into the at least one classifier channel to obtain the loss value corresponding to the initial expression perception neural network model includes:

[0019] The eye region dataset is input into the first classifier channel to obtain the first activity parameter;

[0020] The mouth region dataset is input into the second classifier channel to obtain the second activity parameter;

[0021] The entire facial dataset is input into the third classifier channel to obtain the third activity parameter;

[0022] The loss value corresponding to the initial facial expression perception neural network model is determined based on the first activity parameter, the second activity parameter, and the third activity parameter.

[0023] Optionally, after obtaining the target expression perception neural network model, the method further includes:

[0024] Acquire the image to be tested;

[0025] The image to be tested is input into the target facial expression perception neural network model to obtain the facial expression perception result corresponding to the image to be tested.

[0026] Optionally, the step of inputting the image to be tested into the target expression perception neural network model to obtain the expression perception result corresponding to the image to be tested includes:

[0027] The image to be tested is input into the target expression perception neural network model, and the target expression perception neural network model is controlled to crop the image to be tested to obtain at least one feature sub-image;

[0028] The facial expression feature extractor in the target facial expression perception neural network model is controlled to extract features from the at least one feature sub-image to obtain at least one feature parameter corresponding to the at least one feature sub-image.

[0029] The classifier in the target facial expression perception neural network model is controlled to classify the at least one feature sub-image according to the at least one feature parameter, thereby obtaining at least one probability score set corresponding to the at least one feature sub-image;

[0030] Based on the at least one set of probability scores, the expression perception result corresponding to the image to be tested is determined.

[0031] According to another aspect of this disclosure, a brain-inspired neural network model acquisition system for facial expression perception is provided, comprising:

[0032] A dataset acquisition unit is used to acquire at least one face region training dataset.

[0033] The channel learning unit is used to control at least one classifier channel in the initial expression perception neural network model to learn the at least one face region training dataset, and obtain at least one active parameter corresponding to the at least one classifier channel, wherein the face region training dataset and the classifier channel correspond one-to-one.

[0034] The model determination unit is used to determine the target expression perception neural network model based on the at least one active parameter.

[0035] Optionally, the at least one face region training dataset includes an eye region dataset, a mouth region dataset, and a whole face dataset. When the dataset acquisition unit acquires the at least one face region training dataset, it is specifically used for:

[0036] Obtain an initial training dataset, wherein the initial training dataset includes at least one face image;

[0037] The initial training dataset is preprocessed to transform it into the eye region dataset, the mouth region dataset, and the overall face dataset.

[0038] Optionally, when the channel learning unit is used to control at least one classifier channel in the initial expression perception neural network model to learn the at least one face region training dataset, it is specifically used for:

[0039] Using stochastic gradient descent, at least one classifier channel in the initial facial expression perception neural network model is controlled to iteratively learn the training dataset of at least one face region.

[0040] Optionally, the channel learning unit is used to control at least one classifier channel in the initial expression-aware neural network model to iteratively learn the at least one face region training dataset using stochastic gradient descent, specifically for:

[0041] In any iteration, the training dataset of at least one face region is input into the at least one classifier channel to obtain the loss value corresponding to the initial expression perception neural network model;

[0042] If the loss value satisfies the parameter update condition, the model parameters corresponding to the initial facial expression perception neural network model are updated, and the next iteration is performed.

[0043] If the loss value does not meet the parameter update conditions, then proceed directly to the next iteration until the number of iterations reaches the iteration threshold.

[0044] Optionally, the at least one face region training dataset includes an eye region dataset, a mouth region dataset, and a whole face dataset; the at least one classifier channel includes a first classifier channel, a second classifier channel, and a third classifier channel; the channel learning unit is used to input the at least one face region training dataset into the at least one classifier channel respectively, and to obtain the loss value corresponding to the initial expression perception neural network model, specifically for:

[0045] The eye region dataset is input into the first classifier channel to obtain the first activity parameter;

[0046] The mouth region dataset is input into the second classifier channel to obtain the second activity parameter;

[0047] The entire facial dataset is input into the third classifier channel to obtain the third activity parameter;

[0048] The loss value corresponding to the initial facial expression perception neural network model is determined based on the first activity parameter, the second activity parameter, and the third activity parameter.

[0049] Optionally, the system further includes an image acquisition unit and a result acquisition unit, used after obtaining the target expression perception neural network model:

[0050] The image acquisition unit is used to acquire the image to be tested;

[0051] The result acquisition unit is used to input the image to be tested into the target expression perception neural network model to obtain the expression perception result corresponding to the image to be tested.

[0052] Optionally, when the result acquisition unit inputs the image to be tested into the target expression perception neural network model to obtain the expression perception result corresponding to the image to be tested, it is specifically used for:

[0053] The image to be tested is input into the target expression perception neural network model, and the target expression perception neural network model is controlled to crop the image to be tested to obtain at least one feature sub-image;

[0054] The facial expression feature extractor in the target facial expression perception neural network model is controlled to extract features from the at least one feature sub-image to obtain at least one feature parameter corresponding to the at least one feature sub-image.

[0055] The classifier in the target facial expression perception neural network model is controlled to classify the at least one feature sub-image according to the at least one feature parameter, thereby obtaining at least one probability score set corresponding to the at least one feature sub-image;

[0056] Based on the at least one set of probability scores, the expression perception result corresponding to the image to be tested is determined.

[0057] According to another aspect of this disclosure, a terminal is provided, comprising:

[0058] At least one processor; and

[0059] A memory communicatively connected to the at least one processor; wherein,

[0060] The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method described in any one of the preceding aspects.

[0061] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are used to cause the computer to perform the method described in any one of the preceding aspects.

[0062] According to another aspect of this disclosure, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the method described in any one of the preceding aspects.

[0063] In one or more embodiments of this disclosure, at least one face region training dataset is acquired; at least one classifier channel in an initial expression-aware neural network model is controlled to learn from the at least one face region training dataset to obtain at least one active parameter corresponding to the at least one classifier channel, wherein the face region training dataset and the classifier channel correspond one-to-one; and a target expression-aware neural network model is determined based on the at least one active parameter. Therefore, expression recognition can be performed based on the contribution of face regions to specific expressions, thereby improving the accuracy of expression recognition.

[0064] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0065] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0066] Figure 1 A flowchart illustrating the first brain-inspired neural network model acquisition method provided in this disclosure embodiment is shown.

[0067] Figure 2 A flowchart illustrating a second brain-inspired neural network model acquisition method provided in this disclosure embodiment is shown.

[0068] Figure 3 This diagram illustrates a training flowchart for an initial facial expression perception neural network model provided in an embodiment of this disclosure.

[0069] Figure 4(a) shows a schematic diagram of the accuracy curves for centralized training and validation of the FER2013 dataset provided in an embodiment of this disclosure;

[0070] Figure 4(b) shows a schematic diagram of the accuracy curves for centralized training and validation of a CK+ dataset provided in an embodiment of this disclosure;

[0071] Figure 4(c) shows a schematic diagram of the accuracy curves for centralized training and validation of a JAFFE dataset provided in an embodiment of this disclosure;

[0072] Figure 5 This diagram illustrates the workflow of a target facial expression perception neural network model provided in an embodiment of the present disclosure.

[0073] Figure 6 This diagram illustrates a visualization example of the active area distribution of a surprised expression provided in an embodiment of this disclosure.

[0074] Figure 7 This diagram illustrates the structure of a first brain-inspired neural network model acquisition system for facial expression perception provided in an embodiment of the present disclosure.

[0075] Figure 8 This diagram illustrates the structure of a second brain-inspired neural network model acquisition system for facial expression perception provided in an embodiment of this disclosure.

[0076] Figure 9 This is a block diagram of a terminal used to implement the brain-inspired facial expression perception neural network model acquisition method according to embodiments of the present disclosure. Detailed Implementation

[0077] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0078] The present disclosure will now be described in detail with reference to specific embodiments.

[0079] In the first embodiment, such as Figure 1 As shown, Figure 1 The diagram illustrates a flowchart of a first method for acquiring a brain-inspired neural network model for facial expression perception, according to an embodiment of this disclosure. This method can be implemented using a computer program and can run on a system performing the brain-inspired facial expression perception neural network model acquisition method. The computer program can be integrated into an application or run as a standalone utility application.

[0080] The brain-inspired facial expression perception neural network model acquisition system can be a terminal with brain-inspired facial expression perception neural network model acquisition capabilities. This terminal includes, but is not limited to, wearable devices, handheld devices, personal computers, tablets, in-vehicle devices, smartphones, computing devices, or other processing devices connected to a wireless modem. In different networks, the terminal may be called by different names, such as: user equipment, access terminal, user unit, user station, mobile station, mobile station, remote station, remote terminal, mobile device, user terminal, terminal, wireless communication equipment, user agent or user device, cellular phone, cordless phone, personal digital assistant (PDA), 5G network, 4G network, 3G network, or terminals in future evolved networks.

[0081] Specifically, the method for obtaining this brain-inspired facial expression perception neural network model includes:

[0082] S101, Obtain at least one face region training dataset;

[0083] According to some embodiments, the face region training dataset refers to a training dataset containing the entire face region or a portion thereof, used when training the initial expression perception neural network model. This face region training dataset does not specifically refer to a single, fixed dataset. It includes, but is not limited to, eye region datasets, mouth region datasets, and overall face datasets.

[0084] In some embodiments, the initial facial expression perception neural network model refers to an untrained facial expression perception neural network model.

[0085] It is easy to understand that when the terminal acquires a brain-inspired facial expression perception neural network model, the terminal can acquire at least one face region training dataset.

[0086] S102, control at least one classifier channel in the initial facial expression perception neural network model to learn at least one face region training dataset, and obtain at least one active parameter corresponding to at least one classifier channel;

[0087] In some implementations, the face region training dataset and classifier channels are in a one-to-one correspondence. The classifier channels are also in a one-to-one correspondence with the activity parameters.

[0088] In some embodiments, the process of controlling at least one classifier channel in the initial expression perception neural network model to learn from at least one face region training dataset is the training process of the initial expression perception neural network model.

[0089] According to some embodiments, the facial expression recognition neural network model may include at least one classifier channel. When a test image is input into the facial expression recognition neural network model, the at least one classifier channel can be used to identify facial expressions in different regions of the test image.

[0090] In some embodiments, when annotating data in the training dataset of facial regions, users have certain facial region attention preferences when annotating specific expressions. These preferences can be quantified by a large number of active parameters obtained through learning.

[0091] In some embodiments, the learning of activity parameters aims to explore the preferences arising from the human cognitive process of facial expressions, that is, to quantify the contribution of face regions to the classification prediction of the tested image, thereby examining and evaluating the degree of influence of preferences on the expression perception neural network model. Activity parameters have strong brain-like significance, and the size of the face region training dataset determines the reliability of the activity parameters.

[0092] According to some embodiments, the number of active parameters is not fixed as a preference simplification in modeling. The number of active parameters can depend on the type of face region, i.e., the size of the face region training dataset.

[0093] It is easy to understand that when the terminal obtains at least one face region training dataset, the terminal can control at least one classifier channel in the initial expression perception neural network model to learn from at least one face region training dataset, and obtain at least one active parameter corresponding to at least one classifier channel.

[0094] S103, Determine the target facial expression perception neural network model based on at least one active parameter.

[0095] According to some embodiments, the target expression-aware neural network model refers to a trained expression-aware neural network model. This target expression-aware neural network model does not specifically refer to a particular fixed expression-aware neural network model. For example, the target expression-aware neural network model can change when the activity parameters change.

[0096] It is easy to understand that when the terminal obtains at least one active parameter corresponding to at least one classifier channel, the terminal can determine the target expression perception neural network model based on the at least one active parameter.

[0097] In summary, the method provided in this disclosure involves acquiring at least one face region training dataset; controlling at least one classifier channel in an initial expression-aware neural network model to learn from the at least one face region training dataset, thereby obtaining at least one active parameter corresponding to the at least one classifier channel; and determining the target expression-aware neural network model based on the at least one active parameter. Therefore, expression recognition can be performed based on the contribution of face regions to specific expressions, thus improving the accuracy of expression recognition.

[0098] Please see Figure 2 , Figure 2 This diagram illustrates a flowchart of a second method for obtaining a brain-inspired neural network model for facial expression perception, as provided in an embodiment of this disclosure. Specifically, the method includes:

[0099] S201, Obtain the initial training dataset;

[0100] According to some embodiments, the initial training dataset includes at least one face image.

[0101] In some embodiments, a face image refers to an image containing a complete human face. This face image does not specifically refer to any particular image.

[0102] It is easy to understand that when a terminal acquires a brain-inspired neural network model for facial expression perception, it can acquire an initial training dataset.

[0103] S202, preprocess the initial training dataset to transform it into an eye region dataset, a mouth region dataset, and a whole face dataset;

[0104] It is easy to understand that when the terminal obtains the initial training dataset, the terminal can preprocess the initial training dataset and transform it into an eye region dataset, a mouth region dataset, and a whole face dataset.

[0105] S203, using stochastic gradient descent, each classifier channel in the initial facial expression perception neural network model is controlled to iteratively learn at least one face region training dataset to obtain at least one active parameter corresponding to at least one classifier channel;

[0106] According to some implementations, stochastic gradient descent (SGD) involves randomly selecting a set of samples, training the model, updating it once according to the gradient, then selecting another set and updating it again. With a very large sample size, it may be possible to obtain a model with an acceptable loss value without training all the samples. Here, "random" means that the samples are randomly shuffled during each iteration.

[0107] According to some embodiments, in any iteration process, at least one face region training dataset is input into at least one classifier channel to obtain the loss value corresponding to the initial expression perception neural network model; if the loss value meets the parameter update condition, the model parameters corresponding to the initial expression perception neural network model are updated and the next iteration is performed; if the loss value does not meet the parameter update condition, the next iteration is performed directly until the number of iterations reaches the iteration threshold.

[0108] In some embodiments, the parameter update condition can be that the loss value generated in the current iteration period (Epoch) is smaller than that in the previous period. That is, if the loss value generated in the current iteration period (Epoch) is smaller than that in the previous period, then all model parameters corresponding to the initial expression perception neural network model are updated; if the loss value generated in the current period is larger than that in the previous period, then the parameter update step is skipped.

[0109] In some embodiments, a classifier channel does not specifically refer to a particular fixed classifier channel. For example, at least one classifier channel may include a first classifier channel, a second classifier channel, and a third classifier channel.

[0110] According to some embodiments, Figure 3 This diagram illustrates a training flowchart for an initial facial expression perception neural network model provided in an embodiment of this disclosure. Figure 3 As shown, when the terminal acquires the raw data, it can perform data augmentation on the raw data to obtain preprocessed data 1 (overall face dataset), preprocessed data 2 (eye region dataset), and preprocessed data 3 (mouth region dataset). Then, the terminal can input preprocessed data 1 into the first classifier channel of classifier 1 to obtain the first activity parameter. The preprocessed data 2 is input into the second classifier channel of classifier 2 to obtain the second activity parameter. The preprocessed data 3 is input into the third classifier channel of classifier 3 to obtain the third activity parameter. Next, based on the first, second, and third activity parameters, the loss value and accuracy of the initial facial expression perception neural network model are determined using a loss function. If the loss value and accuracy do not improve compared to the previous iteration, the parameters are not updated in the next iteration; conversely, if the loss value and accuracy improve compared to the previous iteration, the parameters are updated in the next iteration.

[0111] In some embodiments, active parameters can be defined during the recognition of a specific facial expression category; that is, each facial expression category in the training set can correspond to a set of active parameters. The active parameters can take the form of... Where k is the activity parameter, i is the expression category index, and r is the facial local region index. In a single classifier channel, the activity parameter is related to the classification accuracy of the trained local part-based expression classifier, and its initial value can be obtained by the following formula:

[0112]

[0113] Where P is the prediction accuracy of the prediction network (initial facial expression perception neural network model) without the activity parameter. j Let be the prediction accuracy of the classifier for the j-th type of expression in the validation set.

[0114] In some embodiments, the validation set refers to the dataset used to validate the facial expression perception neural network model, such as the target facial expression perception neural network model. The facial expression labels in both the training set (face region training dataset) and the validation set remain consistent with the original data (initial training dataset).

[0115] It is easy to understand that when the terminal transforms the initial training dataset into an eye region dataset, a mouth region dataset, and a whole face dataset, the terminal can use stochastic gradient descent to control at least one classifier channel in the initial expression perception neural network model to iteratively learn at least one face region training dataset, thereby obtaining at least one active parameter corresponding to at least one classifier channel.

[0116] S204, Determine the target facial expression perception neural network model based on at least one active parameter;

[0117] In some implementations, the activity parameter represents the proportion of importance of each facial component during the classification process. The acquisition of the activity parameter needs to occur before final recognition; that is, a complete set of activity parameters needs to be obtained through the learning of the expression classifier, specifically the learning of the classifier's channels, and then fed back to the classifier to obtain the target expression perception neural network model.

[0118] For example, the backbone network used in the initial facial expression recognition neural network model could be a VGG-16 network pre-trained on ImageNet. The learning rate would be set to 10. -3 Updated to 10 after the 300th cycle. -4 The weight decay is set to 3×10. -4 The central processing unit (CPU) in the terminal is an Intel(R) Core(TM) i7-10700K, and the graphics processing unit (GPU) is an Nvidia GeForce RTX 3080.

[0119] To investigate the impact of different population data on activity parameters, this disclosure uses datasets with indistinct population characteristics and datasets with distinct population characteristics for experiments. Specifically, the datasets used are the FER2013 dataset, the CK+ dataset, and the JAFFE dataset. The data in the FER2013 dataset was obtained from an image search engine using emotion keywords. The dataset contains a total of 35,887 facial expression images, including 4,953 images of anger, 547 of disgust, 5,121 of fear, 8,989 of happiness, 6,077 of sadness, 4,002 of surprise, and 6,198 of neutral expressions. Due to the randomness of the data in the dataset, the commonalities among the subjects in the FER2013 dataset are very small. The CK+ dataset contains 593 facial expression sequences from 123 different subjects. Each sequence begins with a neutral expression and ends with the highest expression intensity. 327 of these expression sequences are labeled into seven categories: anger, contempt, disgust, fear, happiness, sadness, and surprise. Since the CK+ dataset does not directly provide validation and test sets, five-fold cross-validation is used as the model performance evaluation method. The CK+ dataset has a relatively small data volume, and the commonalities among the subjects are generally limited. The JAFFE dataset consists of 213 grayscale images of ten women. Each image is 256×256 pixels, and each person's expression is described by 2-4 samples. Similar to CK+, Jaffe also uses five-fold cross-validation to evaluate model performance. The subjects in the JAFFE dataset come from the same region, and the data shows a high degree of commonality in terms of culture, facial structure, and gender. The initial training and validation datasets can be determined from this dataset, and the face recognition accuracy is 65±5%.

[0120] In some embodiments, taking the expression "surprise" as an example, Figure 4(a) shows a schematic diagram of the accuracy curves for centralized training and validation on a FER2013 dataset provided by an embodiment of this disclosure; Figure 4(b) shows a schematic diagram of the accuracy curves for centralized training and validation on a CK+ dataset provided by an embodiment of this disclosure; and Figure 4(c) shows a schematic diagram of the accuracy curves for centralized training and validation on a JAFFE dataset provided by an embodiment of this disclosure. Figures 4(a) to 4(c) As shown, to avoid excessive overlap in the discounting, mean smoothing is performed. Here, Epochs refers to the number of iterations, Accuracy refers to the accuracy, "ori" is the overall facial data, "mask_1" is the mouth region data, "mask_2" is the eye region data, val_acc refers to the model's accuracy on the validation set, and train_acc refers to the model's accuracy on the training set.

[0121] To verify the accuracy of the target expression perception neural network model trained on the FER2013 dataset, the test set recommended by the Kaggle Facial Expression Recognition Challenge can be used as a public test set, which contains 3589 samples. Five-fold cross-validation was used to test the accuracy of the target expression perception neural network model trained by Jaffe and CK+. The accuracy comparison between the target expression perception neural network model and the related model (Fine-tuned VGG) is shown in Table (1);

[0122] Dataset FER2013 CK+ JAFFE Fine tuned VGG 67.55 93.02 98.99 Target expression perception neural network model 73.17 95.91 99.30

[0123] Table (1)

[0124] Therefore, the accuracy of the target facial expression perception neural network model trained according to the embodiments of this disclosure exceeds that of related models.

[0125] It is easy to understand that when the terminal obtains at least one active parameter, the terminal can determine the target expression perception neural network model based on the at least one active parameter.

[0126] S205, acquire the image to be tested;

[0127] According to some embodiments, the image to be tested refers to an image for which facial expression perception is required.

[0128] It is easy to understand that when the terminal obtains the target expression perception neural network model, the terminal can obtain the image to be tested.

[0129] S206, Input the image to be tested into the target expression perception neural network model to obtain the expression perception result corresponding to the image to be tested.

[0130] According to some embodiments, when the target expression perception neural network model recognizes the test image, the final integration method of the recognition result includes, but is not limited to, feature-level fusion, decision-level fusion, etc. Feature-level fusion can be, for example, stitching together features of different scales and modalities. Decision-level fusion can be, for example, a combined decision strategy of the classification outputs of multiple classifiers.

[0131] In some embodiments, when decision-level fusion is employed, the final output can be obtained by a single-channel classification probability weighted average. Figure 5 This diagram illustrates a workflow of a target facial expression perception neural network model provided in an embodiment of this disclosure. Figure 5 As shown, the image to be tested can be input into a target facial expression perception neural network model. The model then crops the image to obtain at least one feature sub-image. Next, the facial expression feature extractor within the model extracts features from the sub-image, obtaining at least one feature parameter corresponding to each sub-image. Then, the classifier in the model classifies the sub-image based on the feature parameter, obtaining at least one set of probability scores for each sub-image. Finally, based on the set of probability scores, the facial expression perception result corresponding to the image is determined, such as... Figure 5 As shown.

[0132] According to some embodiments, to ensure the capture of key facial details, the pooling degree of the expression feature extractor should not be too large when extracting expression features. Therefore, the expression feature extractor can use VGG-16 as the backbone network of the expression feature extractor, which facilitates the adjustment of parameters of each layer at any time.

[0133] In some embodiments, each classifier channel corresponds to a classifier, and each classifier consists of a fully connected (FC) layer and a softmax layer, performing a binary classification task (e.g., happy expression and unhappy expression), responsible for identifying one type of expression.

[0134] In some embodiments, Softmax is an activation function that normalizes a numerical vector into a probability distribution vector, where the sum of the probabilities is 1. Softmax can be used as the final layer of a neural network for outputting multi-class classification problems.

[0135] According to some embodiments, each classifier channel outputs a different classification probability, and each classifier channel outputs only two probabilities: the probability of whether or not it is a specific category of expression (e.g., happiness). When the terminal determines the expression perception result corresponding to the image under test based on at least one set of probability scores, the active parameter corresponding to each classifier channel can be activated. Next, weighted calculations yield the output prediction value (multi-path classification probability) corresponding to the image under test, i.e., the expression perception result, such as... Figure 5 As shown.

[0136] In some embodiments, the classifier may also choose whether to update the activity parameters during the expression recognition process. Specific update strategies can be found in step S203, and will not be elaborated here.

[0137] According to some embodiments, by extracting active parameters from the target facial expression perception neural network model and combining pooling layers with key pixel locations (localization information) in the feature map of a specific expression (e.g., surprise), a regional activity heatmap can be obtained in the facial expression image, identifying the active regions in the facial expression image during the perception process of that category of expression. Therefore, visualization of the feature intensity of facial expression regions for a specific expression can be achieved.

[0138] In some embodiments, key pixels can be obtained by propagating the loss back to the pixel values. Key pixels can be selected only from pixels that have a large impact on the loss value; these pixels represent the visual features that the convolutional neural network can capture from the input. For ease of observation, a 576×576 pixel expression image can be used, thereby capturing more active pixels.

[0139] In some embodiments, Figure 6 This diagram illustrates a visualization example of the active region distribution of a surprised expression, as provided in an embodiment of this disclosure. Figure 6 As shown, the target facial expression perception neural network model provided in this embodiment can achieve accurate perception of a single type of facial expression and can obtain the distribution of facial expression active areas that affect the model's prediction with relatively accurate results.

[0140] It is easy to understand that when the terminal acquires the image to be tested, the terminal can input the image to be tested into the target expression perception neural network model to obtain the expression perception result corresponding to the image to be tested.

[0141] In summary, the method provided in this disclosure involves: acquiring an initial training dataset; preprocessing the initial training dataset to transform it into an eye region dataset, a mouth region dataset, and a whole face dataset; using stochastic gradient descent to iteratively learn at least one classifier channel in the initial expression perception neural network model on at least one face region training dataset, obtaining at least one active parameter corresponding to at least one classifier channel; determining the target expression perception neural network model based on the at least one active parameter; acquiring a test image; and inputting the test image into the target expression perception neural network model to obtain the expression perception result corresponding to the test image. Therefore, by obtaining reliable active parameters through a multi-path perception process from different sources, expression recognition can be performed based on the contribution of face regions to specific expressions, thereby improving the accuracy of expression recognition. Furthermore, the method provided in this disclosure also has significant reference value for research on facial expression recognition with masking.

[0142] The collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0143] The following are system embodiments of this disclosure, which can be used to execute the method embodiments of this disclosure. For details not disclosed in the system embodiments of this disclosure, please refer to the method embodiments of this disclosure.

[0144] Please see Figure 7 This diagram illustrates the structure of a first brain-inspired neural network model acquisition system for facial expression perception provided in this embodiment. This brain-inspired neural network model acquisition system can be implemented as all or part of a system through software, hardware, or a combination of both. The brain-inspired neural network model acquisition system 700 includes a dataset acquisition unit 701, a channel learning unit 702, and a model determination unit 703, wherein:

[0145] Data set acquisition unit 701 is used to acquire at least one face region training dataset;

[0146] The channel learning unit 702 is used to control at least one classifier channel in the initial expression perception neural network model to learn at least one face region training dataset, and obtain at least one active parameter corresponding to at least one classifier channel, wherein the face region training dataset and the classifier channel correspond one-to-one.

[0147] The model determination unit 703 is used to determine the target expression perception neural network model based on at least one active parameter.

[0148] Optionally, at least one face region training dataset includes an eye region dataset, a mouth region dataset, and a whole face dataset. When the dataset acquisition unit 701 acquires at least one face region training dataset, it is specifically used for:

[0149] Obtain an initial training dataset, which includes at least one face image;

[0150] The initial training dataset is preprocessed to transform it into an eye region dataset, a mouth region dataset, and a whole face dataset.

[0151] Optionally, the channel learning unit 702 is used to control at least one classifier channel in the initial expression perception neural network model to learn from at least one face region training dataset, specifically for:

[0152] Using stochastic gradient descent, at least one classifier channel in the initial facial expression perception neural network model is controlled to iteratively learn at least one training dataset for a facial region.

[0153] Optionally, the channel learning unit 702 is used to control at least one classifier channel in the initial expression-aware neural network model to iteratively learn on at least one face region training dataset using stochastic gradient descent, specifically for:

[0154] In any iteration, at least one face region training dataset is input into at least one classifier channel to obtain the loss value corresponding to the initial expression perception neural network model;

[0155] If the loss value meets the parameter update condition, the model parameters corresponding to the initial expression perception neural network model are updated, and the next iteration is performed.

[0156] If the loss value does not meet the parameter update conditions, proceed directly to the next iteration until the number of iterations reaches the iteration threshold.

[0157] Optionally, at least one face region training dataset includes an eye region dataset, a mouth region dataset, and a whole face dataset; at least one classifier channel includes a first classifier channel, a second classifier channel, and a third classifier channel; the channel learning unit 702 is used to input at least one face region training dataset into at least one classifier channel respectively, and to obtain the loss value corresponding to the initial expression perception neural network model, specifically for:

[0158] Input the eye region dataset into the first classifier channel to obtain the first activity parameter;

[0159] Input the mouth region dataset into the second classifier channel to obtain the second activity parameter;

[0160] The entire facial dataset is input into the third classifier channel to obtain the third activity parameter;

[0161] Based on the first activity parameter, the second activity parameter, and the third activity parameter, determine the loss value corresponding to the initial expression perception neural network model.

[0162] Optional, Figure 8 This diagram illustrates the structure of a second brain-inspired neural network model acquisition system for facial expression perception provided in an embodiment of this disclosure. Figure 8 As shown, the brain-inspired facial expression perception neural network model acquisition system 700 also includes an image acquisition unit 704 and a result acquisition unit 705, used to: after obtaining the target facial expression perception neural network model:

[0163] Image acquisition unit 704 is used to acquire the image to be tested;

[0164] The result acquisition unit 705 is used to input the image to be tested into the target expression perception neural network model to obtain the expression perception result corresponding to the image to be tested.

[0165] Optionally, the result acquisition unit 705 is used to input the image to be tested into the target expression perception neural network model to obtain the expression perception result corresponding to the image to be tested, specifically for:

[0166] The image to be tested is input into the target expression perception neural network model, and the target expression perception neural network model is controlled to crop the image to be tested to obtain at least one feature sub-image;

[0167] The facial expression feature extractor in the target facial expression perception neural network model is controlled to extract features from at least one feature sub-map to obtain at least one feature parameter corresponding to at least one feature sub-map.

[0168] The classifier in the target facial expression perception neural network model classifies at least one feature sub-image based on at least one feature parameter, thereby obtaining at least one set of probability scores corresponding to at least one feature sub-image;

[0169] The expression perception result corresponding to the image under test is determined based on at least one set of probability scores.

[0170] It should be noted that the brain-inspired facial expression perception neural network model acquisition system provided in the above embodiments is only illustrated by the division of the above functional modules when executing the brain-inspired facial expression perception neural network model acquisition method. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the brain-inspired facial expression perception neural network model acquisition system and the brain-inspired facial expression perception neural network model acquisition method embodiments provided in the above embodiments belong to the same concept, and the implementation process is detailed in the method embodiments, which will not be repeated here.

[0171] In summary, the system provided in this embodiment acquires at least one face region training dataset through a dataset acquisition unit; a channel learning unit controls at least one classifier channel in the initial expression perception neural network model to learn from the at least one face region training dataset, obtaining at least one active parameter corresponding to the at least one classifier channel, wherein the face region training dataset and the classifier channel correspond one-to-one; and a model determination unit determines the target expression perception neural network model based on the at least one active parameter. Therefore, expression recognition can be performed based on the contribution of face regions to specific expressions, thereby improving the accuracy of expression recognition.

[0172] The collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0173] According to embodiments of this disclosure, this disclosure also provides a terminal, a readable storage medium, and a computer program product.

[0174] Figure 9 A schematic block diagram of an example terminal 900 that can be used to implement embodiments of the present disclosure is shown. The terminal is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The terminal may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0175] like Figure 9As shown, the terminal 900 includes a computing unit 901, which can perform various appropriate actions and processes based on a computer program stored in a read-only memory (ROM) 902 or a computer program loaded from a storage unit 908 into a random access memory (RAM) 903. The RAM 903 may also store various programs and data required for the operation of the terminal 900. The computing unit 901, ROM 902, and RAM 903 are interconnected via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0176] Multiple components in terminal 900 are connected to I / O interface 905, including: input unit 906, such as keyboard, mouse, etc.; output unit 907, such as various types of displays, speakers, etc.; storage unit 908, such as disk, optical disk, etc.; and communication unit 909, such as network card, modem, wireless transceiver, etc. Communication unit 909 allows terminal 900 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0177] The computing unit 901 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 901 performs the various methods and processes described above, such as the brain-inspired facial expression perception neural network model acquisition method. For example, in some embodiments, the brain-inspired facial expression perception neural network model acquisition method can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 908. In some embodiments, part or all of the computer program can be loaded and / or installed on terminal 900 via ROM 902 and / or communication unit 909. When the computer program is loaded into RAM 903 and executed by the computing unit 901, one or more steps of the brain-inspired facial expression perception neural network model acquisition method described above can be performed. Alternatively, in other embodiments, the computing unit 901 may be configured by any other suitable means (e.g., by means of firmware) to perform a brain-inspired facial expression perception neural network model acquisition method.

[0178] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0179] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0180] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0181] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0182] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), the Internet, and blockchain networks.

[0183] Computer systems can include clients and servers. Clients and servers are generally geographically separated and typically interact via communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. A server can be a cloud server, also known as a cloud computing server or cloud host, a hosting product within the cloud computing service ecosystem, addressing the shortcomings of traditional physical hosts and VPS (Virtual Private Server, or simply "VPS") services, such as high management difficulty and weak business scalability. Servers can also be servers for distributed systems or servers incorporating blockchain technology.

[0184] It should be understood that the various forms of processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0185] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A brain-inspired expression perception neural network model acquisition method, characterized in that, include: Obtain at least one face region training dataset; In any iteration, the training dataset of at least one face region is input into the at least one classifier channel to obtain the loss value corresponding to the initial expression perception neural network model. Specifically, the eye region dataset is input into the first classifier channel to obtain the first activity parameter, the mouth region dataset is input into the second classifier channel to obtain the second activity parameter, and the entire face dataset is input into the third classifier channel to obtain the third activity parameter. The loss value corresponding to the initial expression perception neural network model is determined based on the first activity parameter, the second activity parameter, and the third activity parameter. If the loss value satisfies the parameter update condition, the model parameters corresponding to the initial facial expression perception neural network model are updated, and the next iteration is performed. If the loss value does not meet the parameter update condition, the next iteration is performed directly until the number of iterations reaches the iteration threshold to obtain at least one active parameter corresponding to the at least one classifier channel. The active parameter is used to quantify the contribution of the face region to a specific expression to simulate the human brain's preference for different facial regions. The target facial expression perception neural network model is determined based on the at least one active parameter.

2. The method according to claim 1, characterized in that, The at least one face region training dataset includes an eye region dataset, a mouth region dataset, and a whole face dataset. Obtaining the at least one face region training dataset includes: Obtain an initial training dataset, wherein the initial training dataset includes at least one face image; The initial training dataset is preprocessed to transform it into the eye region dataset, the mouth region dataset, and the overall face dataset.

3. The method according to claim 1, characterized in that, After obtaining the target facial expression perception neural network model, the following is also included: Acquire the image to be tested; The image to be tested is input into the target facial expression perception neural network model to obtain the facial expression perception result corresponding to the image to be tested.

4. The method according to claim 3, characterized in that, The step of inputting the image to be tested into the target expression perception neural network model to obtain the expression perception result corresponding to the image to be tested includes: The image to be tested is input into the target expression perception neural network model, and the target expression perception neural network model is controlled to crop the image to be tested to obtain at least one feature sub-image; The facial expression feature extractor in the target facial expression perception neural network model is controlled to extract features from the at least one feature sub-image to obtain at least one feature parameter corresponding to the at least one feature sub-image. The classifier in the target facial expression perception neural network model is controlled to classify the at least one feature sub-image according to the at least one feature parameter, thereby obtaining at least one probability score set corresponding to the at least one feature sub-image; The expression perception result corresponding to the image to be tested is determined based on the at least one set of probability scores.

5. A brain-inspired facial expression perception neural network model acquisition system, characterized in that, include: A dataset acquisition unit is used to acquire at least one face region training dataset. The channel learning unit is used to input the training dataset of at least one face region into the at least one classifier channel during any iteration to obtain the loss value corresponding to the initial expression perception neural network model. Specifically, the eye region dataset is input into the first classifier channel to obtain a first activity parameter, the mouth region dataset is input into the second classifier channel to obtain a second activity parameter, and the entire face dataset is input into the third classifier channel to obtain a third activity parameter. Based on the first activity parameter, the second activity parameter, and the third activity parameter, the loss value corresponding to the initial expression perception neural network model is determined. If the loss value meets the parameter update condition, the model parameters corresponding to the initial expression perception neural network model are updated, and the next iteration is performed. If the loss value does not meet the parameter update condition, the next iteration is performed directly until the number of iterations reaches the iteration threshold to obtain at least one activity parameter corresponding to the at least one classifier channel. The activity parameter is used to quantify the contribution of the face region to a specific expression to simulate the human brain's preference characteristics for different facial regions. The model determination unit is used to determine the target expression perception neural network model based on the at least one active parameter.

6. A terminal, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-4.

7. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-4.

Images