A natural occluded expression recognition method and system with accurate attention

Abstract

The present invention provides a natural occluded expression recognition method and system with accurate attention, which obtains an image to be recognized and preprocesses the image; uses a pretrained natural occluded expression recognition network to process the preprocessed image to be recognized to obtain an expression recognition result; the construction and training process of the natural occluded expression recognition network includes detecting key points of the face for the preprocessed occluded expression image of a known expression, screening multiple points of interest in the key points, generating a Gaussian map based on each point of interest, and obtaining an occlusion indication map of the corresponding image; calculating an attention descriptor based on the neuron activation value of the deep feature of the occluded expression image, forcing the attention descriptor of the deep feature to be close to the occlusion indication map by constructing an attention loss, and forcing the attention of the deep feature to adapt to different expressions by constructing an expression classification loss. The recognition accuracy of the present invention is high.

Description

Technical Field

[0001] The present invention belongs to the technical field of expression recognition, and in particular relates to a method and system for naturally occluded expression recognition with accurate attention. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] In recent years, facial expression recognition has received extensive attention and achieved great success due to its important role in many applications such as affective computing, behavior prediction, human-computer interaction, and mental health services. When recognizing expressions in the real world, it is very common for faces to be occluded, such as by facial accessories such as sunglasses and scarves, or by other random objects such as hands, hair, and cups. Existing expression recognition methods work well for ideal unobstructed facial expressions, but when extended to natural occlusion conditions, the accuracy of expression recognition is greatly reduced. Therefore, it is necessary to study the problem of facial expression recognition under real occlusion conditions in a targeted manner.

[0004] Recognition of facial expressions under occlusion is a challenging problem, because facial occlusions can cause the original expression image to lose part of the expression information and even mislead the expression recognition. Recently, some methods have been proposed to deal with this challenge, which can be mainly divided into four categories: occluded image complementation method, unoccluded image providing privileged information method, unoccluded area attention method and feature reconstruction method.

[0005] The occluded image completion method uses a deep generative model to reconstruct facial images. Some studies use a pixel-level generative model as the bottom layer of a deep belief network (DBN). DBN can reconstruct a complete face from an occluded face, and then predict the expression category based on the complete face. Some studies construct a generator and two discriminators based on the WGAN model, and use reconstruction loss, triple loss, and adversarial loss to achieve the completion of occluded face images. However, due to the large number of occlusion locations and types, these methods cannot reconstruct facial images well.

[0006] Unoccluded images provide more information for expression recognition than occluded images, so unoccluded images can be used to assist the learning of occluded facial expression classifiers. Some studies combine triple loss with knowledge distillation to extract privileged information from the teacher network trained on unoccluded faces to the student network trained on occluded faces. Some studies introduce unoccluded face images paired with occluded face images, and use unoccluded face images as privileged information to guide the learning process of occluded classifiers in label space and feature space, so as to help occluded classifiers learn more robust feature representations and make better predictions. However, such methods require paired occluded and unoccluded images, and are only applicable to synthetic occluded facial expression datasets, but not to real occluded facial expression recognition.

[0007] Occlusions can interfere with expression classification. By making the expression classification network focus on unoccluded areas, the interference of occlusion can be eliminated. Some studies divide the face into different blocks and use regional gating units to predict the occlusion possibility of the corresponding blocks, and then adjust the features through the learned weights; some studies propose regional attention networks, which use self-attention modules and relational attention modules to adjust the importance of facial regions to alleviate problems such as occlusion. However, the accessibility scores and weighted weights of the above methods are learned without any occlusion information, which may be biased, and the network has difficulty in accurately locating unoccluded facial regions. Some studies propose an occlusion adaptive deep network. The global features are adjusted by 24 attention maps based on facial landmarks to guide the model to focus on important unoccluded facial regions and filter out occluded areas. However, this method is non-end-to-end, the attention map is uniquely determined by the position of the occlusion, is not supervised by the expression label, and is not adaptive to the expression.

[0008] The feature reconstruction method uses a detection algorithm to find the occluded area so that the occluded area can be reconstructed. Some studies have proposed a robust method to extract a set of Gabor-based partial face templates from the gallery image using the Monte Carlo method and convert these templates into template matching distance features. Some studies have used RPCA to reconstruct the occluded face area and extract census transform histogram features. K-nearest neighbor and support vector machine are then applied for classification. However, these methods have poor generalization ability.

[0009] For the problem of facial expression recognition under occlusion in natural scenes, among the many explorations mentioned above, the "attention to unoccluded areas method" has a more significant effect. In this type of method, how to obtain accurate attention is the focus and difficulty of the research. At present, such research either allows the network to learn attention weights without any occlusion information, and the learned attention weights are not accurate enough, or uses occlusion information to directly weight deep features, and the attention composed only of occlusion information is not suitable for facial expression recognition tasks, and is therefore not accurate enough. For example, the attention map learned without any occlusion information may still contain some occluded areas; for another example, the key expression information of an occluded disgust expression image is in the nose and mouth areas, and the key information of an occluded surprise expression image is in the eyes and mouth areas, while the occlusion information can only indicate which area is occluded, and treats the unoccluded key areas such as eyes, nose, and mouth equally, which is not suitable for specific expressions. Summary of the invention

[0010] In order to solve the above problems, the present invention proposes a natural occluded expression recognition method and system with accurate attention, and the recognition accuracy of the present invention is high.

[0011] According to some embodiments, the present invention adopts the following technical solutions:

[0012] A method for natural occluded expression recognition with accurate attention, comprising the following steps:

[0013] Obtain the image to be recognized and preprocess the image;

[0014] The pre-processed image to be recognized is processed using a pre-trained natural occlusion expression recognition network to obtain the expression recognition result;

[0015] The construction and training process of the natural occluded expression recognition network includes detecting facial key points for the preprocessed occluded expression image of the known expression, selecting a plurality of interest points from the key points, generating a Gaussian graph based on each interest point, and obtaining an occlusion indication graph of the corresponding image;

[0016] The attention descriptor is calculated based on the neuron activation value of the deep feature of the occluded expression image. The attention descriptor of the deep feature is forced to be close to the occlusion indicator map by constructing the attention loss. The attention of the deep feature is forced to adapt to different expressions by constructing the expression classification loss.

[0017] As an optional implementation, the preprocessing process includes using a face detection model to detect facial landmarks of the original image, obtaining an aligned face image after similarity transformation, and adjusting it to a specified pixel size.

[0018] As an optional implementation, the specific process of performing facial key point detection includes detecting various preliminary key points on the face, obtaining their coordinates and their confidence scores, and selecting points covering the areas around the eyes, eyebrows, nose and mouth from the preliminary key points, as well as points covering the cheeks and the area between the eyes and eyebrows as the final key points.

[0019] As an optional implementation, the specific process of screening multiple interest points among the key points and obtaining the occlusion indication map of the corresponding image includes setting a confidence score threshold, removing key points with confidence scores less than the threshold, obtaining interest points, and generating a Gaussian distribution map centered on the coordinates of each interest point. For those occluded points, the value of the Gaussian distribution map is zero everywhere.

[0020] As an optional implementation, the generation process of the occlusion indication map includes: calculating the statistical values ​​of each Gaussian distribution map at the same spatial position to obtain an occlusion indication map for an input expression image:

[0021]

[0022] in It refers to the Gaussian distribution map generated for the mth interest point of input image i, M represents the number of interest points, and |·| refers to the absolute value at the pixel level.

[0023] As an optional implementation, the specific process of calculating the attention descriptor is:

[0024] Attention Descriptor

[0025] in For the input image x i The p-th map of the k-th layer activation, C k represents the number of feature maps in the kth layer of the deep network, and |·| refers to the absolute value at the pixel level.

[0026] As an alternative implementation, the attention loss function is:

[0027]

[0028] Where Q S (x i )=vec(A S (x i )) is the vectorized form of the attention descriptor based on the neuron activation value, Q T (x i )=vec(resize(A T (x i))) is a vectorized form of the occlusion indication map based on the interest point. Before vectorization, the size of the occlusion indication map is first adjusted to H×W by linear interpolation to match the size of the attention descriptor extracted by the network;

[0029] The expression classification loss is:

[0030]

[0031] Among them, L E is the expression classification loss, is the expression prediction value, y i is the expression label value, and N is the number of expression categories.

[0032] A natural occluded expression recognition system with accurate attention, comprising:

[0033] An image preprocessing module is configured to obtain an image to be recognized and preprocess the image;

[0034] The expression recognition module is configured to process the pre-processed image to be recognized using a pre-trained natural occlusion expression recognition network to obtain an expression recognition result;

[0035] The expression recognition module further comprises:

[0036] The occlusion indication map generation module is configured to perform facial key point detection on the occluded expression image of the known expression, select a plurality of interest points from the key points, generate a Gaussian map based on each interest point, and obtain an occlusion indication map of the corresponding image;

[0037] an attention descriptor extraction module configured to calculate an attention descriptor based on neuron activation values ​​of deep features of the occluded expression image;

[0038] The attention guidance module is configured to force the attention descriptor of the deep feature to be close to the occlusion indication map by constructing an attention loss, and to force the attention of the deep feature to adapt to different expressions by constructing an expression classification loss.

[0039] An electronic device comprises a memory and a processor, and computer instructions stored in the memory and executed on the processor. When the computer instructions are executed by the processor, the steps in the above method are completed.

[0040] A computer-readable storage medium is used to store computer instructions, and when the computer instructions are executed by a processor, the steps in the above method are completed.

[0041] Compared with the prior art, the present invention has the following beneficial effects:

[0042] The present invention introduces occlusion information into the network, guides the network to learn expression features with accurate attention, and represents the attention of the convolutional neural network to a specific spatial position by calculating the attention descriptor based on neuron activation, thereby providing a channel for the subsequent introduction of occlusion information; by generating an occlusion indication map based on interest points, the occluded facial area and the unoccluded key area are indicated.

[0043] Based on two spatial attention maps, the present invention further proposes attention guidance, that is, by constructing an attention loss to force the attention descriptor of the deep feature to be close to the occlusion indication map. At the same time, by constructing an expression classification loss, the attention of the deep feature is forced to adapt to different expressions. In this way, the deep features learned by the network have accurate attention suitable for the occluded expression classification task, that is, the neuron activation value of the unoccluded area that is beneficial to a specific expression is small, while the neuron activation value that is not helpful to a specific expression is large, with good accuracy and efficiency.

[0044] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0046] Figure 1 The propagation path of the natural occluded expression recognition network with accurate attention in this embodiment;

[0047] Figure 2 Schematic diagram of the extracted attention descriptor of this embodiment;

[0048] Figure 3 Schematic diagram of 68 facial key points directly detected by SAN in this embodiment and 24 facial key points further obtained from the 68 key points;

[0049] Figure 4 is a schematic diagram of points of interest in this embodiment;

[0050] Figure 5 (a)-(c) are the confusion matrices of the RAF-DB, AffectNet, and FED-RO datasets of this embodiment;

[0051] Figure 6 Comparison of activation maps of Conv5_x of each model. The first row is the baseline and the second row is the method proposed in this embodiment.

[0052] Figure 7is the influence of confidence threshold T on the accuracy of expression recognition;

[0053] Figure 8 Comparison of activation maps of Conv4_x and Conv5_x of the natural occluded expression recognition network with accurate attention under different confidence score thresholds T. DETAILED DESCRIPTION

[0054] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0055] It should be noted that the following detailed descriptions are all illustrative and intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0056] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0057] The present invention utilizes a natural occluded expression recognition network with accurate attention to realize occluded expression recognition in natural scenes.

[0058] Specifically include:

[0059] Obtain the image to be recognized and preprocess the image;

[0060] The pre-processed image to be recognized is processed using the pre-trained natural occlusion expression recognition network to obtain the expression recognition result.

[0061] like Figure 1 As shown in the figure, the solid line is the forward propagation path in the training phase or the test phase, and the dotted line is the back propagation path in the training phase.

[0062] The following is a detailed introduction to each part.

[0063] The first step is to generate an occlusion indication map based on the point of interest:

[0064] For an input expression image, this embodiment proposes to generate an occlusion indication map based on facial interest points, which indicates the occluded facial area and the unoccluded key area. The occlusion indication map will introduce occlusion information into the expression recognition network by guiding the attention descriptor of the deep features in the training stage, so that the deep features learned by the network have more accurate attention.

[0065] This embodiment mainly generates an occlusion indication map through an occlusion indication map generation module. The generation of the occlusion indication map is divided into the following six steps:

[0066] (1) Image preprocessing: The five facial landmarks of the original image are detected using a standard multi-task convolutional neural network (MTCNN). After performing a similarity transformation, an aligned face image is obtained and resized to 224x224 pixels.

[0067] (2) Detect 68 key points. The preprocessed expression image is input into the network, and the SAN trained on the 300W dataset is used to obtain the coordinates of 68 facial key points of the expression image and their confidence scores.

[0068] (3) Calculate 24 key points. 24 facial key points and their confidence scores are calculated based on the 68 directly detected facial key points and their confidence scores. Specifically, 16 points are selected from the 68 key points to cover the areas around the eyes, eyebrows, nose, and mouth. In addition, 8 new points are further calculated based on the 68 key points to cover the cheeks and the area between the eyes and eyebrows. The confidence scores of these recalculated points are the minimum confidence scores of the points used to calculate them.

[0069] (4) Obtaining points of interest. The confidence score of a point reflects the degree of occlusion at this location to a certain extent. The lower the confidence score, the more serious the occlusion. By manually setting an appropriate threshold and removing key points with confidence scores less than the threshold, the points of interest are obtained. Specifically, the points of interest are obtained by the following formula:

[0070]

[0071] where p m represents the mth point of interest, x m ,y m represents the coordinates of the mth interest point, s m is the confidence score of the mth interest point, ranging from 0 to 1, and T is the threshold.

[0072] In this embodiment, the threshold T is set to 0.2, so as to remove key points with confidence scores less than 0.2. Figure 4 As shown, it can be seen that the key points in the occluded area are removed.

[0073] (5) Generate 24 Gaussian maps. Generate 24 Gaussian distribution maps centered on the coordinates of the interest point. For those obscured points, the values ​​of the Gaussian distribution maps are zero everywhere.

[0074] (6) Synthesize occlusion indication map. Calculate the statistical values ​​of 24 Gaussian distribution maps at the same spatial position to obtain an occlusion indication map for an input expression image. The calculation method is shown in the following formula:

[0075]

[0076] in is the Gaussian distribution map generated for the mth interest point of input image i, M represents the number of interest points, and |·| refers to the absolute value at the pixel level. As a result, A T (x i )∈R 224×224 .

[0077] It should be noted that the occlusion indication map is fixed after it is generated and is used to guide the attention descriptor later.

[0078] The above steps generate an occlusion indication map for the occluded face image. The occlusion indication map contains occlusion information and indicates the occluded facial area and the unoccluded key area.

[0079] This embodiment also proposes an attention descriptor extraction module, and the extracted attention descriptor can provide a path for the introduction of occlusion information in the next step.

[0080] Recent studies have shown that the activation tensor F(x i ), the absolute value of a hidden neuron’s activation can indicate the importance of the neuron for a particular input, and the attention descriptor of the activation tensor can be constructed by counting these values ​​in the channel dimension, such as Figure 2 shown.

[0081] The attention descriptor based on the neuron activation value is calculated according to the following formula:

[0082]

[0083] in For the input image x i The p-th map of the k-th layer activation, C k represents the number of feature maps in the kth layer of the deep network. |·| refers to the absolute value at the pixel level. As a result, the attention descriptor A S (x i )∈R H ×W .

[0084] During the training phase, the extracted deep features of the occluded expression images are input into the attention descriptor extraction module, and the attention descriptor is calculated based on the neuron activation value of the deep features. The attention descriptor indicates the area that the deep features focus on, providing a channel for the subsequent introduction of occlusion information. Our goal is to make the attention descriptor of the deep features more accurate through training, that is, to make the deep features have more accurate attention.

[0085] In the above process, based on the occlusion indication map of the interest points, which indicates the occluded facial areas and the unoccluded key areas, the attention descriptor based on neuron activation is calculated, which represents the attention of the convolutional neural network to a specific spatial location.

[0086] Based on these two spatial attention maps, this embodiment also proposes attention guidance, that is, by constructing an attention loss to force the attention descriptor to be close to the occlusion indication map, so that the deep features that take into account the occlusion information will have more accurate attention. Specifically, the attention loss function is as follows:

[0087]

[0088] Where Q S (x i )=vec(A S (x i )) is the vectorized form of the attention descriptor based on the neuron activation value, Q T (x i )=vec(resize(A T (x i ))) is a vectorized form of the occlusion indication map based on interest points.

[0089] It can be seen that before vectorization, the occlusion indication map is first resized to H×W by linear interpolation to match the size of the attention descriptor extracted by the network. It is worth emphasizing that the normalization of the occlusion indication map is crucial to the success of training.

[0090] The deep features used to extract the attention descriptor will also be used for expression classification, and the cross entropy loss function is used to calculate the expression classification loss:

[0091]

[0092] Among them, L E is the expression classification loss, is the expression prediction value, y i is the expression label value, and N is the number of expression categories.

[0093] Joint expression classification loss L E and attention loss L Q, and the overall loss function is constructed. The expression classification loss and the classification loss are back-propagated so that the occlusion indication map guides the attention descriptor of the deep feature and the expression label supervises the deep feature, so that the deep feature learned by the network has accurate attention suitable for the occluded expression classification task, that is, the neuron activation value of the unoccluded area that is beneficial to the recognition of a specific expression is large, while the neuron activation value of the occluded area or the area that is not beneficial to the recognition of a specific expression is small. The complete objective function of the natural occluded expression recognition network with accurate attention is:

[0094] L=λ1L E +λ2L Q

[0095] Where λ1 and λ2 are hyperparameters used to adjust the loss L E and L Q The weight, L Q and L E have been defined respectively.

[0096] The above parameters and specific setting values ​​can be adjusted or replaced.

[0097] The following comparative test is carried out.

[0098] The expression dataset used for the experiment is introduced, and the implementation of the experiment is described in detail. Then, the method provided in this embodiment is compared with the latest method, and experiments are conducted on the recognition of different expressions to verify the effect of the method of this embodiment. Finally, the attention map visualization is performed to show the superiority of the attention map obtained by the method of this embodiment, and an ablation experiment is constructed to study the impact of different thresholds on the experimental results.

[0099] First, the experimental data sets include:

[0100] The real facial expression dataset contains facial expressions in the real world with various occlusions, postures, lighting intensities and other uncontrollable conditions. The effectiveness of the method proposed in this embodiment is verified on the two largest real facial expression datasets RAF-DB and AffectNet. The method provided in this embodiment is also evaluated on three real-world occlusion datasets: Occlusion-AffectNet, Occlusion-FERPlus and FED-RO. The occlusion colors, shapes, positions and occlusion ratios of these real-world occlusion datasets are different.

[0101] RAF-DB contains 30,000 real facial expression images annotated with basic or compound expressions by 40 independent labelers. In this experiment, only images with seven basic expressions were used, including 12,271 samples for training and 3,068 samples for testing.

[0102] AffectNet is the largest dataset of real facial expressions. There are about 400,000 images, which are manually annotated with 7 expressions as well as the intensity of valence and arousal. Following the experimental settings, only images of neutral and six basic emotions are used, including 280,000 images for training and 3,500 images from the validation set for testing, because the test set is not publicly available. Wang et al. released the Occlusion-AffectNet and Pose-AffectNet datasets, and they only selected images with challenging conditions as test sets: for Occlusion-AffectNet, each image is occluded by at least one occlusion, such as wearing a mask, glasses, etc., with a total of 682 images; for Pose-AffectNet, images with posture degrees greater than 30° and 45° are collected, with the number of images being 1,949 and 985 respectively.

[0103] FED-RO is a facial expression dataset with real-world occlusions. It contains 400 expression images, each of which is labeled as one of the seven basic expressions. Each photo has natural occlusions, such as sunglasses, medical masks, hands, and hair. The network model provided in this embodiment is trained on the joint training data of RAF-DB and AffectNet, and then tested on FED-RO.

[0104] FERPlus is a dataset of real facial expressions introduced in the ICML 2013 challenge. It consists of 28,709 training images, 3,589 validation images, and 3,589 test images. Each image is labeled with one of eight expressions using 10 independent labelers. Recently, Wang et al. collected images under occlusion and large pose (>30° and >45°) conditions from the FERPlus test set and released the Occlusion-FERPlus and Pose-FERPlus datasets. Occlusion-FERPlus has a total of 605 images, and Pose-FERPlus has 1,171 images with poses greater than 30° and 634 images with poses greater than 45°, respectively. The network model provided in this embodiment was trained on the training data of FERPlus and tested on these challenging datasets.

[0105] In the experiment, this embodiment uses ResNet50 as the backbone of the network, and uses the weights pre-trained on ImageNet to initialize the model. The mini-batch size is set to 32, the momentum is 0.9, and the weight decay is 0.0005. The initial learning rate is 0.01, and starting from the 20th iteration, the learning rate decreases to 0.1 times that of the previous stage every 10 iterations. A total of 80 iterations were performed on the model. The optimization algorithm uses stochastic gradient descent (SGD). During training, only random flipping is used for data enhancement. The settings of all experiments are the same. The deep learning software used in the experiment is Pytorch.

[0106] The real facial expression datasets RAF-DB and AffectNet contain facial expressions in the real world with various occlusions, postures, light intensities and other uncontrollable conditions. Since occlusions, large postures, lighting, etc. can cause a certain degree of occlusion, the real facial expression dataset is used to verify the effectiveness of the method provided in this embodiment. On the real facial expression dataset, the method provided in this embodiment is compared with the latest PG-CNN, gACNN, RAN, and OADN methods, and is also compared with the ResNet-50 baseline pre-trained on the ImageNet dataset. Table 1 reports the comparison information. It can be seen that the method provided in this embodiment has obvious advantages over the baseline, with an increase of about 12% and 9% on RAF-DB and AffectNet, respectively. This shows that when using convolutional neural networks for facial expression recognition, using the method provided in this embodiment to fine-tune attention can significantly improve performance. It can also be clearly seen from Table 3 that the network model provided in this embodiment outperforms all other models on both the RAF-DB and AffectNet datasets, and is 2% and 3% higher than the next highest OADN on RAF-DB and AffectNet, respectively, which once again emphasizes that the method of the model of this embodiment with accurate attention is effective.

[0107] In this section, the proposed model is also compared with the most advanced methods on the real-world occlusion dataset to verify the robustness of the method of this embodiment to occlusion. Specifically, the proposed model is compared with the PG-CNN, gACNN, RAN, and OADN methods on the real occlusion dataset FED-RO, and compared with the RAN and OADN methods on Occlusion-AffectNet, Pose-AffectNet (>30° and >45°), Occlusion-FERPlus, and Pose-FERPlus (>30° and >45°).

[0108] Table 2 shows the comparison results. It can be seen that the method of this embodiment has good performance on the real-world occlusion dataset. On FED-RO, the accuracy of the method of this embodiment reaches 74.28%, which is 6% higher than OADN, which has the second best performance. On the challenging subtest sets of AffectNet and FERPlus: in the case of occlusion, the method of this embodiment is about 1% to 6% higher than RAN and OADN; in the case of posture changes, the method of this embodiment is about 1% to 9% higher than RAN and OADN.

[0109] Table 1 Performance comparison with the most advanced methods on real facial expression dataset

[0110]

[0111] Table 2 Performance comparison with state-of-the-art methods on real-world occlusion datasets

[0112]

[0113] In terms of expression discrimination, considering the imbalance of expression categories, the confusion matrix is ​​also used to explain the detailed classification results of all expressions, and the diagonal items represent the average recognition rate of each expression. Figure 5 The confusion matrices for the RAF-DB, AffectNet, and FED-RO datasets are plotted. In all three datasets, the recognition rate of the "happy" expression by the method provided in this embodiment is very high. The "disgusted" expression is one of the most easily confused expressions by the method provided in this embodiment, but this is also common in other models, mainly due to the subtlety of the "disgusted" expression and the uneven distribution of data samples.

[0114] Comparison of the attention maps of the baseline network and the natural occlusion expression recognition network with accurate attention. Conv5_x is the last convolutional layer of the baseline network and the natural occlusion expression recognition network with accurate attention, which is closely related to the final expression classification result. In order to further observe the effect of the method provided by this embodiment, the salient areas that each expression recognition model finally focuses on in the presence of occlusion are explored. Figure 6 Visualizes the attention graph of Conv5_x of the baseline network and the natural occluded expression recognition network with accurate attention. Figure 6It can be seen that the attention map of the baseline conv5_x may not fully cover the key areas that are beneficial to expression recognition, such as the mouth area in the 1st row, 1st column and the 4th column, the left cheek area and the eyebrow area in the 1st row, 3rd column, which may lead to the omission of expression information. The attention map of the baseline conv5_x may also focus on the occluded area, such as the book area in the 1st row, 2nd column and the beard area in the 1st row, 5th column. Paying attention to the occluded area is not only unhelpful for expression recognition, but may even mislead the expression recognition model. The attention map of conv5_x of the network model provided in this embodiment excludes the occluded area, and covers the key areas outside the occluded area that are beneficial to expression recognition in a relatively comprehensive manner. Obviously, compared with the baseline, the attention map of conv5_x of the network model provided in this embodiment is more accurate and more in line with the area that the human eye pays attention to when distinguishing expressions under occlusion conditions.

[0115] The influence of confidence threshold T. The confidence score of key points is used to filter points of interest in the unobstructed area. In order to study the influence of different thresholds on the experimental results, an ablation experiment was constructed. The experimental results are shown in Figure 7 As shown. Figure 7 It can be seen that when T = 0.2, the network model provided by this embodiment achieves the best performance; when T increases further, the performance drops rapidly because some important facial areas that may not be occluded are also discarded; on the other hand, when T is less than 0.2, the performance of the network model provided by this embodiment also begins to deteriorate, because the noise information from the occluded area is also included, which reduces the purity of the feature.

[0116] In order to further observe the influence of the confidence threshold T, we explore the area that the natural occluded expression recognition network with accurate attention finally focuses on under different confidence thresholds T. Figure 8 The activation graphs of Conv4_x and Conv5_x of the natural occlusion expression recognition network with accurate attention under different thresholds T are visualized in Figure 1. Among them, Conv5_x is the last convolutional layer before the model performs expression classification, which has a direct relationship with the expression classification result; while Conv4_x is the convolutional layer directly guided by the occlusion indicator map. Through Conv4_x, we can intuitively see the impact of the threshold T on the model. Figure 8 As can be seen in Figure 8 The first row is the activation map of Conv4_x, and the second row is the activation map of Conv5_x. The first to third columns correspond to thresholds 0, 0.2, and 0.5, respectively.

[0117] It can be seen that when the threshold is 0, the occluded area cannot be excluded; when the threshold is 0.5, key areas such as the left cheek and nose are not included; when the threshold is 0.2, the occluded area can be excluded and the unoccluded area can be completely covered. It can be seen that when the threshold T is set to 0.2, the model's attention is the most accurate.

[0118] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0119] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0120] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0121] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0122] Although the above describes the specific implementation mode of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without creative work are still within the scope of protection of the present invention.

Claims

1. A natural occluded expression recognition method with accurate attention, characterized in that, The following steps are involved: Obtain the image to be recognized and preprocess the image; The pre-processed image to be recognized is processed using a pre-trained natural occlusion expression recognition network to obtain the expression recognition result; The construction and training process of the natural occluded expression recognition network includes detecting facial key points for the preprocessed occluded expression image of the known expression, selecting a plurality of interest points from the key points, generating a Gaussian graph based on each interest point, and obtaining an occlusion indication graph of the corresponding image; The attention descriptor is calculated based on the neuron activation value of the deep feature of the occluded expression image. The attention descriptor of the deep feature is forced to be close to the occlusion indicator map by constructing the attention loss. The attention of the deep feature is forced to adapt to different expressions by constructing the expression classification loss. The specific process of calculating the attention descriptor is: Attention Descriptor in For the input image x i The p-th map of the k-th layer activation, C k represents the number of feature maps in the kth layer of the deep network, and |·| refers to the absolute value at the pixel level.

2. A method for recognizing natural occluded expressions with accurate attention as claimed in claim 1, characterized in that: The preprocessing process includes using a face detection model to detect facial landmarks of the original image, obtaining an aligned face image after similarity transformation, and adjusting it to a specified pixel size.

3. A method for recognizing natural occluded expressions with accurate attention as claimed in claim 1, characterized in that: The specific process of facial key point detection includes detecting various preliminary key points on the face, obtaining their coordinates and their confidence scores, and selecting points covering the areas around the eyes, eyebrows, nose and mouth, as well as points covering the cheeks and the area between the eyes and eyebrows from the preliminary key points as the final key points.

4. A natural occluded expression recognition method with accurate attention as claimed in claim 1, characterized in that The specific process of screening multiple interest points from the key points and obtaining the occlusion indication map of the corresponding image includes setting a confidence score threshold, removing key points with confidence scores less than the threshold, obtaining interest points, and generating a Gaussian distribution map centered on the coordinates of each interest point. For those occluded points, the value of the Gaussian distribution map is zero everywhere.

5. A method for recognizing natural occluded expressions with accurate attention as claimed in claim 1, characterized in that the occlusion The process of generating the indicator map includes: calculating the statistical values ​​of each Gaussian distribution map at the same spatial position to obtain an occlusion indicator map for an input expression image: in It refers to the Gaussian distribution map generated for the mth interest point of input image i, M represents the number of interest points, and |·| refers to the absolute value at the pixel level.

6. A method for recognizing natural occluded expressions with accurate attention as claimed in claim 1, characterized in that: The attention loss function is: Where Q S (x i )=vec(A S (x i )) is the vectorized form of the attention descriptor based on the neuron activation value, Q T (x i )=vec(resize(A T (x i ))) is a vectorized form of the occlusion indication map based on the interest point. Before vectorization, the size of the occlusion indication map is first adjusted to H×W by linear interpolation to match the size of the attention descriptor extracted by the network; Or, the expression classification loss is: Among them, L E is the expression classification loss, is the expression prediction value, y i is the expression label value, and N is the number of expression categories.

7. A natural occluded expression recognition system with accurate attention, characterized in that, include: An image preprocessing module is configured to obtain an image to be recognized and preprocess the image; The expression recognition module is configured to process the pre-processed image to be recognized using a pre-trained natural occlusion expression recognition network to obtain an expression recognition result; The expression recognition module further comprises: The occlusion indication map generation module is configured to perform facial key point detection on the occluded expression image of the known expression, select a plurality of interest points from the key points, generate a Gaussian map based on each interest point, and obtain an occlusion indication map of the corresponding image; an attention descriptor extraction module configured to calculate an attention descriptor based on neuron activation values ​​of deep features of the occluded expression image; The specific process of calculating the attention descriptor is: Attention Descriptor in For the input image x i The p-th map of the k-th layer activation, C k represents the number of feature maps in the kth layer of the deep network, with |·| referring to the absolute value at the pixel level; The attention guidance module is configured to force the attention descriptor of the deep feature to be close to the occlusion indication map by constructing an attention loss, and to force the attention of the deep feature to adapt to different expressions by constructing an expression classification loss.

8. An electronic device, characterized in that: The method comprises a memory and a processor, and computer instructions stored in the memory and executed on the processor, wherein when the computer instructions are executed by the processor, the steps in the method according to any one of claims 1 to 6 are completed.

9. A computer-readable storage medium, characterized in that: Used to store computer instructions, which, when executed by a processor, complete the steps of the method according to any one of claims 1 to 6.

Images