A sentiment recognition method based on multi-scale atrous attention capsule network

Through the multi-scale hollow attention capsule network, key features and regional information of facial expressions are extracted, and the problem of degradation of recognition performance caused by facial occlusion and posture changes is solved, achieving efficient emotional recognition effect.

CN118334725BActive Publication Date: 2024-11-05NORTHEAST DIANLI UNIVERSITY
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410503115.6
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-04-25
Publication Date
2024-11-05
Estimated Expiration
2044-04-25

AI Technical Summary

Technical Problem

The existing emotion recognition technology is difficult to effectively capture the correlation between discontinuous facial areas under facial occlusion and posture changes, resulting in a degradation of recognition performance and requiring a large amount of facial image data.

Method used

Multi-scale hollow attention capsule network (MDA-CapsNet), including feature pre-extraction module, multi-scale hollow attention module and capsule classification module, extract key features and regional information of facial expressions through hollow convolution and position attention mechanisms, and capture facial differences Relative position and dynamic changes between parts.

Benefits of technology

It improves the performance and robustness of facial emotion recognition, reduces sensitivity to occlusion and facial posture changes, and allows you to understand and recognize complex expression changes more carefully without the need for a large amount of data training.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118334725B_ABST
    Figure CN118334725B_ABST
Patent Text Reader

Abstract

The invention discloses an emotion recognition method based on a multi-scale hole-attention capsule network, which relates to the field of computer science and technology, and includes establishing an MDA-CapsNet emotion recognition model. The MDA-CapsNet emotion recognition model includes the following modules: a feature pre-extraction module, which captures shallow features of a human face from an input facial image; a multi-scale hole-attention module, which extracts detailed information from tiny facial muscle changes to overall expression changes, so that the network can automatically identify the most critical features and regions in facial expressions when processing facial expression images. The emotion recognition method based on the multi-scale hole-attention capsule network proposes a multi-scale hole-attention capsule network from the perspective of extracting significant emotion features from global, local and discontinuous facial regions. The network can capture significant correlation features between discontinuous facial regions on the basis of solving the problems of occlusion and facial posture changes, so as to improve the performance and robustness of facial emotion recognition.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of computer science, and in particular to an emotion recognition method based on a multi-scale void attention capsule network. Background Art

[0002] Emotion, as a bridge of human-computer interaction, is considered to be a high-level cognitive activity of human beings and plays an important role in people's daily interpersonal communication and decision-making. In recent years, with the rapid development of brain-computer technology, emotion recognition has been widely used in medical diagnosis and evaluation, distance education and other fields to evaluate human emotional state. According to the requirements of different emotion recognition tasks for the evaluation method of emotional state, two types of emotion models are mainly used in existing research to represent emotions, namely dimensional and discrete emotion models. In dimensional models, 2D valence-arousal models are usually used to quantitatively describe emotional states. The valence dimension describes the negativity and positivity of emotions, and the arousal dimension describes the excitement and activity level of emotions. Discrete emotion models are more subdivided and are usually used to describe more concrete emotional states in daily life. Different emotional states present different physiological and non-physiological reactions. In past research work, emotion recognition research mainly uses methods based on physiological signals such as EEG, EMG, and cerebral blood oxygen, as well as non-physiological signals such as facial expressions and body language. Compared with the high-precision equipment and expensive costs required to collect physiological signals, emotion recognition based on facial expressions is more convenient and has real-time interpretation and is widely used.

[0003] Different sensory areas have different degrees of importance in terms of emotional information. Solving the problem of sensory area information loss caused by occlusion and facial posture changes to improve the expressiveness of features and narrow the recognition performance gap between controllable and uncontrollable scenes in emotion recognition research is still a challenge faced by emotion recognition research based on facial expression images.

[0004] In order to solve the problem of information loss caused by facial occlusion, in past studies, researchers generally adopted a facial reconstruction method that reshapes facial geometry and texture features by referring to the complete facial image. The localization of the facial region of interest based on facial landmarks may lead to inconsistent feature representation, resulting in inconsistent prediction results for the same facial image, reducing the robustness of recognition performance.

[0005] To address the information loss problem caused by facial pose changes, previous studies have proposed effective methods to explore potential key emotional features from both local and global perspectives. Although the above methods improve the performance of facial emotion recognition, they require a large amount of facial image data;

[0006] The facial perception mechanism is a process of perceiving local changes from global changes. Different facial regions convey different levels of emotional information. It is generally believed that the emotional information of the eye and mouth regions is more important, but this does not mean that the emotional information of other regions can be completely discarded, because the visual presentation of different facial expressions involves the coordinated cooperation of multiple facial sensory regions. Therefore, how to capture the correlation between discontinuous facial regions is an important issue. To this end, we propose an emotion recognition method based on a multi-scale atrous attention capsule network. Summary of the invention

[0007] The purpose of the present invention is to provide an emotion recognition method based on a multi-scale void attention capsule network to solve the problems raised in the above background technology.

[0008] To achieve the above object, the present invention provides the following technical solution: an emotion recognition method based on a multi-scale void attention capsule network, comprising establishing an MDA-CapsNet emotion recognition model, wherein the MDA-CapsNet emotion recognition model comprises the following modules:

[0009] A1: Feature pre-extraction module, which captures shallow features of the face from the input facial image;

[0010] A2: Multi-scale hole attention module, which extracts detailed information from small facial muscle changes to overall expression changes, so that the network can automatically identify the most critical features and regions of facial expressions when processing facial expression images;

[0011] A3: Capsule classification module, which understands the object's attributes such as posture, scale, and deformation, and helps capture the relative positions and dynamic changes between different parts of the face.

[0012] Preferably, the feature extraction module consists of a 5×5 two-dimensional convolution early layer and two basic blocks. The input image undergoes a two-dimensional convolution with a kernel size of 5×5 and is normalized and After linear activation, we get a low-level feature map ,use The basic building blocks in the model, the input feature map of the basic block After processing, a deeper feature map is obtained .

[0013] Preferably, the multi-scale hole attention module consists of 4 multi-scale hole blocks and 1 position attention block.

[0014] Preferably, in the multi-scale hole block, a parallel residual pyramid structure is established, and after a 3×3 convolution operation of Conv2D, the feature map obtained is , we use the split strategy to evenly divide it into feature map subsets, using Indicates that , each feature subset Compared with the input feature map, the spatial size is the same, and the number of channels is only the original , each feature subset There is a corresponding residual hole convolution layer, denoted as , the size of the convolution kernel is 3×3, and the void coefficient Use an incremental approach to increase the receptive field of the convolution kernel. Each feature subset The output of each feature subset Output Feature subset and The outputs are added and then input Get, the original convolution kernel size , through the segmentation of the segmentation strategy, the output of each feature subset With different numbers of subset features and different receptive field sizes, the number of segmentation subsets is set to After being processed by four multi-scale hole blocks, the feature map is processed to obtain a high-level feature map that integrates the local and global information of the input image. , the high-level feature map Send it to the position attention block to identify the most critical position area in the expression image;

[0015] The position attention block combines the maximum pooling operation and the global average pooling operation to modify the ECA-Net, and uses the modified ECA-Net to replace the channel attention block in the CBAM model.

[0016] Preferably, the output It can be calculated as:

[0017] ;

[0018] Since the different dilation coefficients will affect the effective size of the convolution kernel, the formula for the effective convolution kernel size obtained by dilation convolution is:

[0019] .

[0020] Preferably, the output feature of the multi-scale hole module is , the position attention module is As input, a one-dimensional convolution with an adaptive convolution kernel is used to learn the channel attention map :

[0021]

[0022] in, represents the activation function, represents the one-dimensional convolution operation of the adaptive convolution kernel, represents the global average pooling operation, Represents the maximum pooling operation, the convolution kernel k changes dynamically with the number of channels of the input feature, and can be expressed as:

[0023]

[0024] in, Indicates departure The nearest odd number;

[0025] Spatial attention uses two-dimensional convolution to generate a spatial attention map :

[0026]

[0027]

[0028] in, represents element-wise multiplication, represents the channel attention map, represents the activation function, express The final output of the position attention module is expressed as:

[0029]

[0030] in, Represents element-wise multiplication.

[0031] Preferably, the capsule classification module consists of a convolutional layer, a primary capsule layer and a sentiment capsule layer.

[0032] Preferably, the convolutional layer step size is set to 1, consisting of 256 The convolution kernel is The function is composed of activation functions.

[0033] Preferably, the primary capsule layer consists of 32 8D main capsules.

[0034] Preferably, the emotion capsule layer has A 16D emotional capsule, representing Emotion categories.

[0035] Compared with the prior art, the present invention has the following beneficial effects:

[0036] 1. From the perspective of extracting significant emotion features from global, local and discontinuous facial regions, this paper proposes a multi-scale void attention capsule network, which can capture significant correlation features between discontinuous facial regions on the basis of solving the problems of occlusion and facial posture changes, so as to improve the performance and robustness of facial emotion recognition.

[0037] 2. The present invention utilizes a multi-scale hole attention module. The scale hole attention module is robust against occlusion and facial posture changes. Different scales have different receptive fields. A larger receptive field can capture rich semantic features, but is easily affected by occlusion and facial posture changes. A smaller receptive field can capture rich geometric texture features and is less sensitive to occlusion and facial posture changes. The combination of large and small scales can effectively reduce the network's sensitivity to occlusion and posture changes, better capture local details of the face and contextual information of larger areas, thereby understanding information from tiny facial muscle changes to overall expression changes.

[0038] 3. The present invention introduces intervals in the convolution kernel by using dilated convolution. The use of dilated convolution makes it possible for information interaction between discontinuous facial regions.

[0039] 4. The present invention introduces a position attention module to help the model focus on key areas such as eyes and mouth, which are crucial for emotion recognition. The capsule network can capture the hierarchical and spatial relationships in the image, better understand facial attributes such as posture, scale and deformation, and thus better capture the relative positions and dynamic changes between different parts of the face, providing a more sophisticated understanding and recognition of complex and subtle changes in facial expressions. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 It is the overall framework diagram of the method of the present invention;

[0041] Figure 2 This is a basic architecture diagram of the present invention and the extraction module;

[0042] Figure 3 This is a basic architecture diagram of the multi-scale hole attention module of the present invention;

[0043] Figure 4 This is a basic architecture diagram of the location attention module of the present invention. DETAILED DESCRIPTION

[0044] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0045] Embodiment 1

[0046] See also Figure 1-4 The present invention provides a technical solution: an emotion recognition method based on a multi-scale void attention capsule network. The present invention makes corresponding improvements to the technical problems mentioned in the background technology, including establishing an MDA-CapsNet emotion recognition model, and the MDA-CapsNet emotion recognition model includes the following modules:

[0047] A1: Feature pre-extraction module, which captures shallow features of the face from the input facial image;

[0048] A2: Multi-scale hole attention module, which extracts detailed information from small facial muscle changes to overall expression changes, so that the network can automatically identify the most critical features and regions of facial expressions when processing facial expression images;

[0049] A3: Capsule classification module, which understands the attributes of the object such as posture, scale and deformation, and helps capture the relative positions and dynamic changes between different parts of the face;

[0050] The feature extraction module consists of a 5×5 two-dimensional convolution early layer and two basic blocks. The input image undergoes a 5×5 two-dimensional convolution and is normalized and After linear activation, we get a low-level feature map ,use The basic building blocks in the model, the input feature map of the basic block After processing, a deeper feature map is obtained ;

[0051] The multi-scale hole attention module consists of 4 multi-scale hole blocks and 1 position attention block;

[0052] In the multi-scale hole block, a parallel residual pyramid structure is established. After a 3×3 convolution operation of Conv2D, the feature map obtained is , we use the split strategy to evenly divide it into feature map subsets, using Indicates that , each feature subset Compared with the input feature map, the spatial size is the same, and the number of channels is only the original , each feature subset There is a corresponding residual hole convolution layer, denoted as , the size of the convolution kernel is 3×3, and the void coefficient Use an incremental approach to increase the receptive field of the convolution kernel. Each feature subset The output of each feature subset Output Feature subset and The outputs are added and then input Get, the original convolution kernel size , through the segmentation of the segmentation strategy, the output of each feature subset With different numbers of subset features and different receptive field sizes, the number of segmentation subsets is set to After being processed by four multi-scale hole blocks, the feature map is processed to obtain a high-level feature map that integrates the local and global information of the input image. , the high-level feature map Send it to the position attention block to identify the most critical position area in the expression image;

[0053] The position attention block combines the maximum pooling operation and the global average pooling operation to modify the ECA-Net, and uses the modified ECA-Net to replace the channel attention block in the CBAM model;

[0054] The output It can be calculated as:

[0055] ;

[0056] Since the different dilation coefficients will affect the effective size of the convolution kernel, the formula for the effective convolution kernel size obtained by dilation convolution is:

[0057] ;

[0058] The output characteristics of the multi-scale hole module are: , the position attention module is As input, a one-dimensional convolution with an adaptive convolution kernel is used to learn the channel attention map :

[0059]

[0060] in, represents the activation function, represents the one-dimensional convolution operation of the adaptive convolution kernel, represents the global average pooling operation, Represents the maximum pooling operation, the convolution kernel k changes dynamically with the number of channels of the input feature, and can be expressed as:

[0061]

[0062] in, Indicates departure The nearest odd number;

[0063] Spatial attention uses two-dimensional convolution to generate a spatial attention map :

[0064]

[0065]

[0066] in, represents element-wise multiplication, represents the channel attention map, represents the activation function, express The final output of the position attention module is expressed as:

[0067]

[0068] in, represents element-wise multiplication;

[0069] The capsule classification module consists of a convolutional layer, a primary capsule layer and a sentiment capsule layer;

[0070] The convolutional layer stride is set to 1, consisting of 256 The convolution kernel is The function is composed of activation functions;

[0071] The primary capsule layer consists of 32 8D main capsules;

[0072] The emotion capsule layer has A 16D emotional capsule, representing Emotion categories.

[0073] The specific implementation of this embodiment is as follows:

[0074] The MDA-CapsNet emotion recognition model consists of three modules, namely, feature pre-extraction module, multi-scale hole attention module and capsule classification module. The feature pre-extraction module is used to capture the shallow features of the face from the input facial image. Subsequently, we designed a multi-scale hole attention module to extract detailed information from small facial muscle changes to overall expression changes. The introduction of the attention module enables the network to automatically identify the most critical features and regions in facial expressions when processing facial expression images, effectively solving the problem of reduced recognition performance caused by occlusion and posture changes. Finally, the capsule classification module can effectively understand the attributes of the object such as posture, scale and deformation when processing data, which helps to capture the relative position and dynamic changes between different parts of the face, thereby providing a more refined understanding and recognition of complex and subtle expression changes;

[0075] (1) Feature pre-extraction module:

[0076] The feature pre-extraction module in the MDA-CapsNet model architecture consists of a 5×5 2D convolutional early layer and two basic blocks. Figure 2 The basic architecture of the pre-extraction module is shown in Figure 2. The input image undergoes a 2D convolution with a kernel size of 5×5 and is normalized and After linear activation, we get a low-level feature map In order to make the features extracted from the early layers continue to be used by the basic blocks and increase the reusability of the features, we used The basic building blocks in the model, the input feature map of the basic block After processing, a deeper feature map is obtained ,The design of the pre-extraction module enables the network to more ,effectively extract low-level features from the original image and convert them into ,higher-level feature representations, providing more representative features for ,subsequent operations, thereby improving the feature representation ,ability and recognition performance of the entire network model.

[0077] (2) Multi-scale hole attention module:

[0078] Atrous convolution, CNN's multi-scale features and attention mechanism are often used in natural language processing, object detection, semantic segmentation and other computer vision fields. Inspired by MA-Net, we designed a multi-scale atrous attention module, such as Figure 1 As shown in the figure, the multi-scale hole attention module proposed in this paper consists of 4 multi-scale hole blocks and 1 position attention block. The introduction of multi-scale holes can more effectively understand the information from tiny facial muscle changes to overall expression changes at different scales. At the same time, the attention mechanism further enhances the network's attention to key position information.

[0079] Figure 3The basic architecture of the multi-scale hole attention module is shown in Figure 1, which consists of two parts: a multi-scale hole block and a position attention block. In the multi-scale hole block, we design a parallel residual pyramid structure that can quickly learn multi-scale information in the feature map and increase the reusability of features. Specifically, after a 3×3 convolution operation of Conv2D, the feature map obtained in the channel dimension is , we use the split strategy to evenly divide it into feature map subsets, using Indicates that , each feature subset Compared with the input feature map, the spatial size is the same, but the number of channels is only the original , for each feature subset For each of them, there is a corresponding residual hole convolution layer, denoted as , the size of the convolution kernel is 3×3, and the void coefficient We use an incremental approach to increase the receptive field of the convolution kernel. Each feature subset The output needs to be clear that except for , each feature subset Output Feature subset and The outputs are added and then input So, the output can be calculated as:

[0080]

[0081] From the above equation, we can find that each residual hole convolution layer Captures all subsets In addition, for each residual hole convolution layer As for , the different dilation coefficients will affect the effective size of the convolution kernel. Therefore, the effective convolution kernel size obtained by dilation convolution is:

[0082]

[0083] In this method, the original convolution kernel size , the use of this segmentation strategy makes the output of each feature subset Subset features with different numbers and different receptive field sizes can more effectively capture local facial details and global context information at different scales for recognizing facial expressions of various sizes and shapes. In addition, the number of different segmentation subsets It determines the amount of scale information contained in the feature, but it also brings the problem of computational overhead. In order to achieve the best balance between the amount of information and computational overhead, in this study, we set the number of segmentation subsets to After being processed by four multi-scale hole blocks, the feature map is processed to obtain a high-level feature map that integrates the local and global information of the input image. , which is then fed into the position attention block to identify the most critical position area in the expression image;

[0084] In the process of facial expression recognition, since it is easily affected by some external environments, there will be problems of occlusion and posture changes in the process of collecting facial data, which will affect the recognition performance. Therefore, it is very important to focus on the key position areas of facial expression images. To this end, after the multi-scale void module, we designed a position attention module to focus on identifying the key position areas of facial expressions. Figure 4 The basic architecture of the position attention module is shown. Inspired by CBAM and ECA-Net, the combination of maximum pooling operation and global average pooling operation can highlight more refined feature information from the overall distribution. The original ECA-Net uses a one-dimensional lightweight sliding window convolution operation to learn cross-channel interaction information. Compared with the channel attention block in CBAM, which uses a multi-layer perceptron to learn attention weights, it can improve performance while reducing computational costs and the number of parameters. Based on this, we modify ECA-Net by combining the maximum pooling operation and the global average pooling operation, and use the modified ECA-Net to replace the channel attention block in the CBAM model to form the position attention module in this article.

[0085] The output features of the multi-scale hole module are , the position attention module is As input, a one-dimensional convolution with an adaptive convolution kernel is used to learn the channel attention map :

[0086]

[0087] in, represents the activation function, represents the one-dimensional convolution operation of the adaptive convolution kernel, represents the global average pooling operation, Represents the maximum pooling operation, the convolution kernel k changes dynamically with the number of channels of the input feature, and can be expressed as:

[0088]

[0089] in, Indicates departure The nearest odd number, this nonlinear mapping can automatically select the appropriate convolution kernel size according to the number of input channels, and the spatial attention generates a spatial attention map with two-dimensional convolution. :

[0090]

[0091]

[0092] in, represents element-wise multiplication, represents the channel attention map, represents the activation function, express The convolution kernel size is a two-dimensional convolution operation, so the final output of the position attention module is expressed as:

[0093]

[0094] in, Represents element-wise multiplication.

[0095] (3) Capsule classification module:

[0096] The capsule classification module can be used to capture the position information and dynamic changes between different facial regions obtained from the multi-scale hole attention module, so as to provide a more refined understanding and recognition of complex and subtle expression changes. The capsule classification module consists of a convolutional layer, a primary capsule layer, and an emotion capsule layer. The convolutional layer step size is set to 1, and it consists of 256 The convolution kernel is The function is composed of activation functions, which are used to detect the basic features of different position areas to be used as the input of the primary capsule layer;

[0097] The primary capsule layer consists of 32 8D primary capsules. In other words, each primary capsule contains 8 convolution operations, which are used to convert the scalar output of the convolution layer into a tensor form. That is, each output of the primary capsule layer is an 8D tensor. Depending on the size of the input image, the number of output capsules of the primary capsule layer is different. Assuming the input is , then the number of output capsules of the primary capsule layer is ,The modulus and direction of each primary capsule respectively represent the existence probability and posture attributes of the low-level features related to the emotional state.,The primary capsule layer as the cornerstone of the capsule classification module,is an essential operation to obtain high-level features related to the emotional,state;

[0098] The detailed steps are as follows:

[0099] First, each primary capsule output of the primary capsule layer is mapped to the sentiment capsule to obtain a high-level feature representation. This process is completed by multiplying the primary capsule with the weight matrix, which is formulated as:

[0100]

[0101] in, Indicates The output of the main capsule, Represents high-level features, Represents the weight matrix that describes the relative position relationship between low-level features and features.

[0102] Secondly, the input of each emotion capsule in the emotion capsule layer is achieved by applying all high-level features The weighted sum is obtained, and the calculation process is as follows:

[0103]

[0104] in is called the coupling coefficient between the main capsule and the emotion capsule, and The sum of the coupling coefficients between the main capsule and all the emotion capsules is 1. The calculations show that:

[0105]

[0106] in, Indicates The main capsule and The logarithmic prior probability of the emotion capsule feature transfer is 0, and the initial value is Activation function pair A nonlinear transformation is performed to control the length range of the output vector of the emotion capsule to , thus obtaining the output vector used to represent the probability of different emotion categories :

[0107]

[0108] Finally, by using To measure the output vector With the current output The coupling coefficient is updated iteratively based on the consistency of , the iteration formula is as follows:

[0109]

[0110] After the above dynamic routing mechanism iterates to the maximum number of iterations, all output emotion capsules can be obtained to achieve the emotion classification task. In addition, the present invention uses marginal loss to calculate the classification loss value of the entire model:

[0111]

[0112] in, represents the number of sentiment categories, Either 0 or 1, used to indicate whether the predicted category exists. and Respectively represent the boundary values ​​of whether the prediction category exists, we set and Indicates the upper limit of the predicted category as false positive and the lower limit of the false negative, the proportional coefficient , the total loss value of the MDA-CapsNet model is the sum of the emotion capsule losses of all categories.

[0113] The last part of the capsule classification module is the emotion capsule layer. The emotion capsule layer maps each 8D output capsule of the primary capsule layer to the 16D capsule output space to obtain high-level features related to the emotional state for emotion classification. The emotion capsule layer has a total of A 16D emotional capsule, representing The primary capsule layer and the emotion capsule layer are connected by a bridge called a dynamic routing mechanism. The dynamic routing mechanism continuously learns and improves weights to capture the relative spatial relationship between different location areas, thereby aggregating high-level features into different categories to form emotion capsules for emotion classification.

[0114] The following table shows the dynamic routing algorithm process:

[0115]

[0116] Embodiment 2

[0117] Emotion recognition research experiments were conducted on the DEAP and CK+ datasets. The following are the details of these two datasets:

[0118] DEAP dataset: The DEAP dataset is a public dataset for sentiment analysis research. It records the facial positive expression videos of 22 subjects at a rate of 50 frames per second while they watch 40 60-second video clips. The subjects are asked to quickly conduct self-assessment after watching each clip. Among them, the video data of 18 subjects are complete.

[0119] CK+ dataset: The CK+48 dataset is a public dataset used in the fields of computer vision and affective computing. 981 facial expression pictures from 118 subjects are labeled into seven basic emotions, namely angry (135), contempt (54), disgust (177), fear (75), happy (207), sad (84), and surprise (249).

[0120] The experimental results are compared with several state-of-the-art methods on DEAP and CK+ datasets:

[0121] (1) Comparison of DEAP dataset:

[0122] The emotion recognition accuracy achieved by MDA-CapsNet on the DEAP dataset and its comparison with some state-of-the-art methods are shown in the table below. Huang et al. first pre-trained a multi-task CNN classifier on the Fer2013 large image dataset and then fine-tuned the pre-trained model on the DEAP dataset. Zhu et al. designed a 3D neural network model to complete the emotion recognition task by pre-processing the input image. Zhang et al. designed a DeepVANet model and pre-trained it on the AFEW-VA dataset to adapt to the emotion recognition task based on the DEAP dataset. Wu et al. designed a multi-layer convolutional neural network model for facial expression emotion recognition. Wang et al. used a pre-trained convolutional model to extract facial features and then introduced an attention mechanism to extract key facial features to complete emotion recognition. The above studies have achieved satisfactory emotion recognition results, but our model achieved the highest recognition results in the arousal and valence dimensions by starting from the perspective of occlusion and facial posture changes and focusing on the correlation between discontinuous facial regions. Compared with the optimal results, our method improved the arousal and valence dimensions by 0.99% and 0.23%, respectively.

[0123] Comparison with the state-of-the-art results on the DEAP dataset

[0124]

[0125] (2) Comparison of CK+ datasets:

[0126] The emotion recognition accuracy achieved by MDA-CapsNet on the CK+ dataset and its comparison with some of the most advanced methods are shown in the table below. Consistent with other methods, we evaluate the effectiveness of our proposed model by identifying 7 discrete emotion categories. The experimental results show that our proposed MDA-CapsNet model achieves the best emotion recognition results. Compared with the recognition results of the best method, our method improves the recognition accuracy by 1.73%. This also proves that our model can still achieve good emotion recognition performance even on frontal facial expression datasets without any occlusion and facial posture changes.

[0127] Comparison with the state-of-the-art results on the CK+ dataset

[0128]

[0129] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0130] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for emotion recognition based on a multi-scale dilated attention capsule network, characterized in that: Including establishing an MDA-CapsNet emotion recognition model, the MDA-CapsNet emotion recognition model includes the following modules: A1: Feature pre-extraction module, which captures shallow features of the face from the input facial image; A2: Multi-scale hole attention module, which extracts detailed information from small facial muscle changes to overall expression changes, so that the network can automatically identify the most critical features and regional location information of facial expressions when processing facial expression images; A3: Capsule classification module, which understands the pose, scale and deformation properties of the object and helps capture the relative positions and dynamic changes between different parts of the face; The feature pre-extraction module consists of a 5×5 two-dimensional convolution early layer and two basic blocks. The input image undergoes a two-dimensional convolution with a kernel size of 5×5 and is normalized and After linear activation, we get a low-level feature map ,use The basic building blocks in the model, the input feature map of the basic block After processing, a deeper feature map is obtained ; The multi-scale hole attention module consists of four multi-scale hole blocks and one position attention block. The introduction of multi-scale holes can more effectively understand information from tiny facial muscle changes to overall expression changes at different scales. At the same time, the attention mechanism further enhances the network's attention to key position information. The capsule classification module is used to capture the position information and dynamic changes between different facial regions obtained from the multi-scale hole attention module, and provide a more sophisticated understanding and recognition of complex and subtle expression changes. The capsule classification module consists of a convolutional layer, a primary capsule layer, and an emotion capsule layer. The convolutional layer step size is set to 1, and it consists of 256 The convolution kernel is The function is composed of activation functions, which are used to detect the basic features of different position areas to be used as the input of the primary capsule layer; The primary capsule layer consists of 32 8D primary capsules, which are used to convert the scalar output of the convolutional layer into a tensor form. That is, each output of the primary capsule layer is an 8D tensor. Depending on the size of the input image, the number of output capsules of the primary capsule layer is different. The input is , then the number of output capsules of the primary capsule layer is ,The modulus and orientation of each primary capsule respectively represent the ,existence probability and posture attributes of low-level features related to ,emotional state; The emotion capsule layer maps each 8D output capsule of the primary capsule layer to the 16D capsule output space to obtain high-level features related to the emotional state for emotion classification. The emotion capsule layer has A 16D emotional capsule, representing The primary capsule layer and the emotion capsule layer are connected by a bridge called the dynamic routing mechanism. The dynamic routing mechanism continuously learns and improves weights to capture the relative spatial relationship between different location areas, thereby aggregating high-level features into different categories to form emotion capsules for emotion classification.

2. The emotion recognition method based on multi-scale void attention capsule network according to claim 1 is characterized in that: The multi-scale hole attention module builds a parallel residual pyramid structure. After a 3×3 convolution operation of Conv2D, the feature map obtained in the channel dimension is , we use the split strategy to evenly divide it into feature map subsets, using Indicates that , each feature subset Compared with the input feature map, the spatial size is the same, and the number of channels is only the original , each feature subset There is a corresponding residual hole convolution layer, denoted as , the size of the convolution kernel is 3×3, and the void coefficient Use an incremental approach to increase the receptive field of the convolution kernel. Each feature subset The output of each feature subset Output Feature subset and The outputs are added and then input Get, the original convolution kernel size , through the segmentation of the segmentation strategy, the output of each feature subset With different numbers of subset features and different receptive field sizes, the number of segmentation subsets is set to After being processed by four multi-scale hole blocks, the feature map is processed to obtain a high-level feature map that integrates the local and global information of the input image. , the high-level feature map Send it to the position attention block to identify the most critical position area in the expression image; The position attention block modifies ECA-Net by combining the maximum pooling operation with the global average pooling operation, and uses the modified ECA-Net to replace the channel attention block in the CBAM model.

3. The emotion recognition method based on multi-scale void attention capsule network according to claim 2 is characterized in that: The output It can be calculated as: ; Since the different dilation coefficients will affect the effective size of the convolution kernel, the formula for the effective convolution kernel size obtained by dilation convolution is: 。 4. The emotion recognition method based on multi-scale void attention capsule network according to claim 2, characterized in that: The output features of the multi-scale hole module are , the position attention module is As input, a one-dimensional convolution with an adaptive convolution kernel is used to learn the channel attention map : ; in, represents the activation function, represents the one-dimensional convolution operation of the adaptive convolution kernel, represents the global average pooling operation, Represents the maximum pooling operation, the convolution kernel k changes dynamically with the number of channels of the input feature, and can be expressed as: ; in, Indicates departure The nearest odd number; Spatial attention uses two-dimensional convolution to generate a spatial attention map : ; ; in, represents element-wise multiplication, represents the channel attention map, represents the activation function, express The final output of the position attention module is expressed as: ; in, Represents element-wise multiplication.

5. The emotion recognition method based on multi-scale void attention capsule network according to claim 1, characterized in that: Each primary capsule output of the primary capsule layer is mapped to the sentiment capsule to obtain high-level feature representation. This process is completed by multiplying the primary capsule with the weight matrix, which is formulated as: ; in, Indicates The output of the main capsule, Represents high-level features, Represents the weight matrix that describes the relative position relationship between low-level features and features.

6. The emotion recognition method based on multi-scale void attention capsule network according to claim 1, characterized in that:The input of each emotion capsule in the emotion capsule layer is achieved by applying all high-level features The weighted sum is obtained, and the calculation process is as follows: ; in is called the coupling coefficient between the main capsule and the emotion capsule, and The sum of the coupling coefficients between the main capsule and all the emotion capsules is 1. The calculations show that: ; in, Indicates The main capsule and The logarithmic prior probability of the emotion capsule feature transfer is 0, and the initial value is Activation function pair A nonlinear transformation is performed to control the length range of the output vector of the emotion capsule to , thus obtaining the output vector used to represent the probability of different emotion categories : ; Finally, by using To measure the output vector With the current output The coupling coefficient is updated iteratively based on the consistency of , the iteration formula is as follows: ; After the above dynamic routing mechanism iterates to the maximum number of iterations, all output emotion capsules can be obtained to achieve the emotion classification task.

7. The emotion recognition method based on multi-scale void attention capsule network according to claim 1, characterized in that: Use marginal loss to calculate the classification loss value of the entire model: ; in, represents the number of sentiment categories, Either 0 or 1, used to indicate whether the predicted category exists. and Respectively indicate whether the predicted category exists or not, set and Indicates the upper limit of the predicted category as false positive and the lower limit of the false negative, the proportional coefficient , the total loss value of the MDA-CapsNet model is the sum of the emotion capsule losses of all categories.