An expression recognition method in a natural scene based on brain-like technology

Abstract

The application provides an expression feature extraction device, an expression recognition method in a natural scene based on an expression feature extraction model, and an expression recognition method in a natural scene based on brain-like technology, wherein the expression feature extraction model improves the SCAN model; backbone ResNet50 is replaced by pre-trained ResNet18 to extract global and local texture information of expression samples; a mask mechanism in a BERT model is replaced by a sorting mechanism; LSTM is used to capture time sequence changes of the samples based on DenseSift key point expression; decision probability values based on global, local and time sequence information are fused to improve information extraction of decision; and cross-entropy loss function and chi-square estimation are combined to realize end-to-end model learning. The application enhances the robustness to face occlusion, head posture change and uneven illumination, and further improves the generalization of the model.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a method for facial expression recognition. Background Technology

[0002] With the rapid development of computer technology, artificial intelligence technology, and related disciplines, the level of automation in society is constantly increasing, and people's demand for human-computer interaction, similar to human-to-human communication, is growing stronger. If computers and robots can understand and express emotions like humans, this will fundamentally change the relationship between humans and computers, allowing computers to better serve humanity. Facial expression recognition is the foundation of emotion understanding, a prerequisite for computers to understand human emotions, and an effective way for people to explore and understand intelligence. Due to the diversity and complexity of facial expressions, facial expression recognition is difficult. Although some traditional feature engineering and deep learning models can perform expression recognition well under controlled laboratory conditions, recognizing expression samples in uncontrolled natural scenes still faces significant challenges in the following three aspects: 1. Uneven lighting; 2. Facial occlusion; 3. Multi-angle head posture. Summary of the Invention

[0003] In view of this, in order to achieve high performance and low power consumption in facial expression recognition in natural scenes, this invention proposes an facial expression feature extraction model, a facial expression recognition method in natural scenes based on the facial expression feature extraction model, and a facial expression recognition method in natural scenes based on brain-like technology.

[0004] The facial expression feature extraction device comprises two branches: a texture branch and a geometric branch. The texture branch extracts global spatial attention, local block spatial attention, and a self-attention representation for each local block of the face image based on an attention mechanism. Global spatial attention highlights the influence of different pixels in the input face image on global expression recognition, local block spatial attention highlights the importance of each pixel to its local block, and the self-attention representation for each local block highlights the contribution of different local blocks to global expression recognition when the face is divided into multiple local regions. The geometric branch describes the temporal changes of different regions of the face as the human eye views an image. The features extracted from the global spatial attention, local block representation, and geometric branch are fused at the feature level or at the decision level to obtain the expression recognition result. The local block representation includes local block spatial attention and the self-attention representation for each local block.

[0005] In general, the feature layers closer to the input layer in a neural network are called "low layers," and the feature layers closer to the output layer are called "high layers." The features extracted from the low layers to the high layers become more and more abstract and closer to the target task.

[0006] Feature-level fusion and decision-level fusion are existing technologies.

[0007] Furthermore, the texture branch works as follows: the face image to be recognized is input into the pre-trained ResNet18, and the feature map output from any of the later layers of ResNet18 is selected as the global feature map. Based on the global feature map, global spatial attention, local block spatial attention, and self-attention representations for each local block are generated respectively. The pre-training includes two steps: first, ResNet18 is pre-trained based on the face recognition dataset to acquire the ability to capture basic image features; then, transfer learning is performed to learn from the expression sample set.

[0008] Furthermore, the spatial attention template is used to calculate the global feature map to obtain the global spatial attention.

[0009] Furthermore, the spatial attention template is implemented using a 2D convolution kernel.

[0010] Furthermore, the global feature map is divided into multiple local blocks, and the spatial attention of each pixel in each local block is calculated using the same spatial attention template as the global one. Then, the weights of different local blocks are calculated to obtain the self-attention representation of each local block.

[0011] Furthermore, the weights of different local blocks are calculated using an improved BERT model. The improvement is that the mask design of the self-attention mechanism is changed to a sorting design, that is, the weights of the spatial attention of each local block are sorted in descending order using a multi-head self-attention mechanism with attention_heads of 8.

[0012] Furthermore, the geometric branch works by using the Dense SIFT operator to extract key points and their corresponding histogram representations from the face image to be recognized, fusing the two-dimensional coordinates of the key points with the histogram representations, and inputting the fused information into the temporal model LSTM to obtain the temporal information of the key points changing over time.

[0013] Furthermore, end-to-end model training is achieved by combining the cross-entropy loss function and the chi-square estimate used to describe whether the two distributions are consistent. The decision probability value based on global information and the decision probability value based on local block representation are used as the actual value and theoretical value of the chi-square estimate, respectively, and a coefficient λ is set to balance the cross-entropy loss function and the chi-square estimate.

[0014] A facial expression recognition method based on natural scenes, using an facial expression feature extraction device, specifically includes:

[0015] The face images to be recognized are preprocessed. Specifically, the expression samples are converted into grayscale images, and the illumination of the grayscale images is normalized based on the histogram equalization operator.

[0016] The preprocessed facial image is input into the facial expression feature extraction device to obtain the recognition result.

[0017] A method for facial expression recognition in natural scenes based on neuromorphic technology is proposed, which transfers a learned facial expression feature extraction device to an embedded device based on a neuromorphic chip for fast and accurate facial expression recognition.

[0018] Beneficial effects

[0019] 1) The backbone ResNet18 speeds up texture feature extraction, reduces computation, and lowers power consumption compared to ResNet50;

[0020] 3) The BERT model based on the ranking mechanism mitigates the impact of noisy representations on effective information extraction;

[0021] 3) LSTM tracks the temporal changes of facial key points as the view moves;

[0022] 4) The decision-level fusion strategy greatly enriches the diversity of sample information and improves the generalization ability of the model;

[0023] 5) Chi-square estimation drives different modules in the model to generate similar classification probability distributions, reducing the inconsistency between the model's description of global and local information, and helping to improve classification accuracy;

[0024] 6) This invention enhances robustness to facial occlusion, changes in head posture, and uneven lighting, further improving the generalization ability of the model. Attached Figure Description

[0025] Figure 1 SCAN structure diagram;

[0026] Figure 2 : Overall flowchart of emotion recognition methods based on natural scenes.

[0027] The dashed box indicates the differences between this invention and SCAN. Detailed Implementation

[0028] This invention proposes an expression recognition method based on natural scenes, which is based on an expression feature extraction model. The specific process is as follows:

[0029] First, texture feature extraction. Compared to the complex model structure of ResNet50, ResNet18, with its relatively simple module design and pre-trained on the face recognition dataset MS-Celeb-1M, was chosen as the backbone of the proposed framework. This backbone already has the ability to capture common basic features of images, such as details of edges and corners in sample images. To achieve accurate recognition of expression samples, a transfer learning mechanism was further used to transfer the backbone pre-trained on the face dataset to the expression sample set FER2013. To reduce the impact of lighting variations on facial expression recognition, facial expression samples are converted into grayscale images, and lighting normalization is performed on the grayscale images using a histogram equalization operator. The normalized samples are then used as input to the backbone for training, resulting in a backbone pre-trained based on the facial expression recognition sample set. After the first two stages of pre-training are completed, the backbone pre-trained based on facial expression samples is formally used for learning the RAFFasic facial expression sample set from natural scenes. Feature maps from any later layer of the backbone are extracted as global feature maps, with a size of 64*128*28*28, where 64 represents the batch size, 128 represents the number of feature maps extracted (i.e., the number of channels), and 28*... 28 represents the size of each feature map in the extraction layer. To highlight the impact of different pixels on global expression recognition, a 3*3 2D convolutional kernel with a stride of 1 is initialized. A spatial attention template is defined based on the set convolutional kernel, and global spatial attention is calculated based on the global feature map of size 28*28. Simultaneously, to capture the importance of each pixel to its local block, each 28*28 feature map in the extraction layer is divided into multiple local blocks of varying sizes, such as according to the "three courts and five eyes" principle. The specific division is not limited; in this embodiment, based on facial features, 25 local blocks of varying sizes are preferred: 16 x 6 pixels, 4 x 6 pixels, and 4 x 4 pixels. Each local block is calculated using a 4×4 pixel array and a 2D convolutional kernel with the same spatial attention template as the global array. To highlight the contribution of different local blocks to global expression recognition, the weights of spatial attention for different local blocks are calculated based on an improved BERT model. The original mask mechanism randomly generates a mask consisting of 0s and 1s and randomly selects weights for weighted ensemble. The improved BERT model sorts the weights calculated by the Multi-head self-attention mechanism with 8 attention heads in descending order and selects the top 5 weight values ​​to perform weighted ensemble of the spatial attention of the local blocks, generating the self-attention representation for each local block.Geometric feature extraction is performed to describe the temporal changes of various local facial regions as the human eye views an image. The input grayscale image is scaled to 64×64 pixels, and a Dense SIFT algorithm with a sliding window size of 6×6 pixels, a stride of 8 pixels, and 8 bins is applied. The operator extracts key points and their corresponding 128-dimensional histogram representations from the input grayscale samples, and fuses the two-dimensional coordinate values ​​of the key points with the histogram representations. The fused information is then input into a temporal LSTM model to learn the temporal information of key point changes over time. In this embodiment, three cascaded LSTMs are used, with the following parameter settings: First LSTM module: input layer feature dimension 128, hidden layer feature dimension 64, first dimension size of input data: mini-batch, dropout: 0.1, number of hidden layers (num_layers): 2, bidirectional; Second LSTM module: input layer feature dimension 128, hidden layer feature dimension: 128, first dimension size of input data: mini-batch, dropout: 0.1, number of hidden layers (num_layers): 2, bidirectional; Third LSTM module: input layer feature dimension 256, hidden layer feature dimension 256, first dimension size of input data: mini-batch, dropout: 0.1, number of hidden layers (num_layers): 2, bidirectional. Alternatively, an LSTM can be used; the choice depends on the specific circumstances.

[0030] To further improve the comprehensiveness of information representation, features from three modules—global spatial attention, local block representation, and LSTM—are fused. For the decision layer, classification is first performed based on the global, local block, and LSTM modules respectively, yielding three decision probability values. A fusion strategy is then employed, connecting these three decision probability values ​​and calculating the maximum, mean, and product of the corresponding points. Finally, to obtain more accurate recognition results, end-to-end model training is implemented by combining the cross-entropy loss function and the chi-square estimation, which describes the consistency between the two distributions. The decision probability values ​​based on global information and those based on local block information are used as the actual and theoretical values ​​of the chi-square estimation, respectively, with a coefficient λ=0.0001 set to balance the cross-entropy loss function and the chi-square estimation. After training on the RAFFasic training set, the model is validated on the validation set and tested on the test set. To further verify the model's generalization ability, validation will also be performed on the AFEW video expression dataset.

[0031] Setting hyperparameters for model training:

[0032] batch_size=32 or 64 (The model training process uses the gradient descent method of mini-batch, so the mini-batch size is 32 or 64. The optimal number of mini-batch samples is selected through experiments.)

[0033] epochs=40 (number of iterations)

[0034] Choose the optimal learning rate from the range [0.1, 0.01, 0.0001, 1e-4, 1e-5].

[0035] ReduceLROnPlateau: Learning Rate Strategy

[0036] Optimizer: Adam (Optimizer settings for model training)

[0037] weight_decay = 1e-4 (Model weight decay coefficient)

[0038] Using the loss function proposed in this invention for model training can improve the facial expression recognition rate. Based on the principle of the controlled variable method, the facial expression recognition rates of the seven types on the RAFFasic test set based on different loss functions are shown in Table 1. Among them, the cosine distance reflects the relative difference in the direction of the two probability distributions, the KL divergence is an indicator that measures the degree of matching between the two probability distributions, and the chi-square estimate represents the degree of deviation between the theoretical value and the actual value.

[0039] Table 1

[0040]

[0041] A neuromorphic technology-based facial expression recognition method for natural scenes is proposed. To meet the needs of fast and accurate recognition in real life, the learned model is transferred to an embedded device based on a neuromorphic chip and further verified. Experiments show that the proposed method can not only achieve high performance on GPU-based servers, but also be applied to embedded devices based on domestic chips, achieving a relative balance between recognition rate and speed.

Claims

1. A facial expression feature extraction device, characterized in that: The facial expression feature extraction model comprises two branches: a texture branch and a geometric branch. The texture branch extracts global spatial attention, local block spatial attention, and a self-attention representation for each local block from the face image based on an attention mechanism. Global spatial attention highlights the influence of different pixels in the input face image on global expression recognition, local block spatial attention highlights the importance of each pixel to its local block, and the self-attention representation for each local block highlights the contribution of different local blocks to global expression recognition when the face is divided into multiple local regions. The geometric branch describes the temporal changes of different regions of the face as the human eye views an image. The features extracted from the global spatial attention, local block representation, and geometric branch are fused at the feature level or at the decision level to obtain the expression recognition result. The local block representation includes local block spatial attention and a self-attention representation for each local block. The texture branch works as follows: the face image to be recognized is input into the pre-trained ResNet18, and the feature map output from any later layer of ResNet18 is selected as the global feature map. Based on the global feature map, global spatial attention, local block spatial attention, and self-attention representations for each local block are generated. The pre-training includes two steps: first, ResNet18 is pre-trained on a face recognition dataset to acquire the ability to capture basic image features; then, transfer learning is applied to the learning of expression sample sets. The geometric branch works by using the Dense SIFT operator to extract key points and their corresponding histogram representations from the face image to be recognized, fusing the two-dimensional coordinates of the key points with the histogram representations, and inputting the fused information into the temporal model LSTM to obtain the temporal information of the key points changing over time.

2. The facial expression feature extraction device according to claim 1, characterized in that: Global spatial attention is obtained by calculating the global feature map using a spatial attention template.

3. The facial expression feature extraction device according to claim 2, characterized in that: The spatial attention template is implemented using 2D convolution kernels.

4. The facial expression feature extraction device according to claim 2 or 3, characterized in that: The global feature map is divided into multiple local blocks. The spatial attention of each pixel in each local block is calculated using the same spatial attention template as the global block. Then, the weights of different local blocks are calculated to obtain the self-attention representation of each local block.

5. The facial expression feature extraction device according to claim 4, characterized in that: The improved BERT model is used to calculate the weights of different local blocks. The improvement is that the mask design of the self-attention mechanism is changed to a sorting design. That is, the weights of the spatial attention of each local block are calculated in descending order using a multi-head self-attention mechanism with attention_heads of 8.

6. The facial expression feature extraction device according to claim 1, characterized in that: End-to-end model training is achieved by combining the cross-entropy loss function and the chi-square estimate used to describe whether two distributions are consistent. The decision probability value based on global information and the decision probability value based on local block representation are used as the actual value and theoretical value of the chi-square estimate, respectively, and a coefficient λ is set to balance the cross-entropy loss function and the chi-square estimate.

7. A facial expression recognition method based on natural scenes, characterized in that: Based on the facial expression feature extraction device of claim 1, the facial image to be identified is preprocessed. Specifically, the facial expression sample is converted into a grayscale image, and the grayscale image is normalized for illumination based on the histogram equalization operator. The preprocessed facial image is input into the facial expression feature extraction device to obtain the recognition result.

8. A facial expression recognition method in natural scenes based on brain-like technology, characterized in that: The learned facial expression feature extraction device of claim 1 is transferred to a neuromorphic chip-based embedded device for fast and accurate facial expression recognition.

Images