A facial and situational emotion recognition method based on attention mechanism
By introducing attention mechanisms and multi-scale features fusion in facial and situational emotion recognition, the problem of insufficient situational information in facial and situational emotion analysis is solved, and the accuracy and model performance of facial and situational emotion recognition are improved.
Patent Information
- Application Number
- CN202311573364.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2023-11-20
- Filing Date
- 2023-11-23
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2043-11-23
AI Technical Summary
In the prior art, in the analysis of facial emotions under natural environments, the situational information contained in a single picture is limited, and the video-based situational emotion recognition method ignores channel attention, resulting in limited improvement in model performance.
Using facial and situational emotion recognition methods based on attention mechanism, combined with deep learning and optimization methods, facial and situational emotion characteristics are extracted through ResNet18 and ResNeXt50 networks, channel attention and multi-scale feature fusion are introduced, and ECA modules and CA modules are weighted. Process the feature map and extract the timing information through MogrifierLSTM.
The accuracy and contextual emotion recognition effect driven by facial expressions are improved, the model's learning ability of key information and advanced semantic expression ability of image are improved, and it is proved that better recognition effect is achieved on the dataset than other methods.
Smart Images

Figure CN117636426B_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of facial capture technology, and in particular to a facial and situational emotion recognition method based on an attention mechanism. Background Art
[0002] In scenes under natural environments, if we analyze the emotions of the target subject from a visual perspective, it is not possible to accurately judge the emotional state by only observing facial expressions. Since human emotions are related to the environment in which they are located to a certain extent, the emotions of other people around them will also have more or less influence on the target person. Therefore, the context is also very important in multimodal sentiment analysis as auxiliary information for sentiment analysis. At present, most of the research on context information focuses on static images, but the context information contained in a single image is limited, and video data can contain more context information. At present, Lee et al. have studied the video-based context emotion recognition method, but the feature extraction model of context information is relatively simple, and only spatial attention is used to capture context information, ignoring the improvement of channel attention on model performance.
[0003] In recent years, facial expression recognition methods combined with scene context have become a relatively new research direction, making visual emotion recognition methods no longer rely solely on the face part, and can obtain clues related to the target person's emotional state from non-subject situational information in the image. Inspired by psychologists, some researchers believe that the environment in which the character is located is also important information related to the target person's emotional state, and emotions cannot be interpreted in isolation from the surrounding contextual information. Scene context includes the surrounding environment of a person, such as the category of place, the attributes of the place, objects, or behaviors around a person. For example, if a person is at the beach, then his emotion is likely to be positive, and if a person is in a hospital, the emotion may be more nervous and worried. In addition, when the facial expression of the target person is in a relatively ambiguous state, more features can be learned from the scene information. Kosti et al. constructed an image emotion dataset containing scene context information and proposed a scene-based end-to-end two-stream emotion recognition model, in which one branch extracts features from the human body part in the image, and the other branch takes the entire image as input to extract features related to the scene context information. Both branches use a truncated version of the low-rank filter convolutional network, and delete the last fully connected layer of the original network. The features extracted by the two branches are sent to the feature fusion module. First, a global average pooling layer is used on each feature map to reduce the number of features, and then the final classification result is obtained through two fully connected layers. Lee et al. proposed a two-stream encoding network, which includes two encoding branches of facial expression and scene, and uses scene context information in a joint and enhanced manner. It uses 3D convolution to extract features from the input image sequence, and introduces a spatial attention mechanism in the scene extraction branch. Finally, the features extracted by the two branches are assigned weights by using the fusion network, and feature fusion and classification are performed. In the context-aware multimodal emotion recognition method based on Mittal et al., ResNet18 is used in the context feature extraction branch to extract features from images with occluded target characters, and then the Attention Branch Network (ABN) is used to identify key areas in the features related to the context. The other two branches use visual information (face and posture) and depth map as inputs, respectively, and extract emotional features from them. Finally, the feature representations of the three branches are concatenated and fed into the classifier to obtain the emotional classification for the image. The literature proposes to use context to construct an emotional map to infer emotional states. First, a region proposal network (RPN) is used to extract the person nodes in the image as input to the graph convolutional network (GCN) to detect the context, and the convolution operation is converted from the Euclidean data structure to the non-Euclidean data structure. GCN learns emotional relationships during the back-propagation process. In addition, a convolutional neural network (CNN) is used to extract human features.Finally, the results of the two branches of GCN and CNN are spliced to infer the continuous dimensions and discrete emotion categories measured by VAD (Valence, Aluise, Contrance). Summary of the invention
[0004] In order to solve the above problems, the present application provides a facial and situational emotion recognition method based on attention mechanism, which combines deep learning and optimization methods to improve the accuracy of expression driving while meeting the real-time requirements.
[0005] In order to achieve the above purpose, the technical solution adopted in the embodiment of the present application is as follows:
[0006] In a first aspect, a facial and situational emotion recognition method based on an attention mechanism is provided, including a facial emotion recognition method and a situational emotion recognition method; the characteristics are:
[0007] The facial emotion recognition method comprises the steps of: extracting facial emotion features frame by frame from an input image sequence using a ResNet18 network to obtain a facial emotion feature map; inputting the facial emotion feature map extracted by the ResNet18 network into an ECA module to obtain a weighted facial emotion feature map; inputting the facial emotion feature map weighted by the ECA module into a feature selection fusion module for feature fusion processing to obtain a facial emotion recognition result P. f ;
[0008] The situation emotion recognition method comprises the steps of: taking frame images from each video sample in sequence as input; using ResNeXt50 network to extract situation emotion features from the image sequence frame by frame to obtain situation emotion feature map; inputting the situation emotion feature map extracted by ResNet18 network into CA module to obtain weighted situation emotion feature map; inputting the situation emotion feature map weighted by CA module into multi-scale feature fusion module to obtain feature vector x outputted by situation emotion t ; The above feature vector x t Input to the MogrifierLSTM module and process to get the facial emotion recognition result P f .
[0009] Furthermore, the facial emotion recognition method includes: using a face detection algorithm to capture facial images of the annotated person from a video sequence of a data set, scaling them to a uniform size, and saving their facial image sequences in folders named after their respective numbers according to the detection frame number of each annotated person.
[0010] Furthermore, the facial emotion recognition randomly extracts N frames of images from each face image sequence as input during the model training phase, and takes the image sequence of the entire video as input during the testing phase;
[0011] Furthermore, in the model training stage, the situational emotion recognition will take N frames of images from each video sample in sequence as input. Before the image is input, the target subject needs to be occluded so that the model can learn situational information other than the target subject. If a frame of the image does not contain the target subject, the entire image will be used as the input of the model; in the testing stage, the image sequence of the entire video will be used as the input of the testing stage.
[0012] Furthermore, ResNet18 extracts the expression features of each frame of the image, and the output feature map size obtained through the average pooling layer and the fully connected layer is W = 7 in width, H = 7 in height, and C = 512 in number of channels;
[0013] The feature map processed by ResNet18 is input into the ECA module. After global average pooling, a set of 1×1×C feature vectors is obtained. The feature vector is convolved using one-dimensional convolution. The size calculation formula of the convolution kernel k is:
[0014]
[0015] Among them, γ and b are parameters defined for nonlinear mapping, which are set to 2 and 1 respectively. In the formula, |t| odd The meaning of is to take the odd number closest to the absolute value of t.
[0016] Furthermore, after determining the k value, a feature vector of the same size is obtained by performing a one-dimensional convolution on the feature vector to achieve local cross-channel information interaction; after a sigmoid function, a weight representing the importance of each channel is obtained, which is multiplied with the corresponding channel feature map to obtain a weighted feature map. The feature vector corresponding to the weighted N image sequence is F = {f1,...,f N}.
[0017] Furthermore, the ECA module is integrated into the position before the residual connection in each Basicblock of ResNet by adding an Identity connection; the Identity connection is connected to the output of the ECA module by adding the elements at the corresponding positions.
[0018] Furthermore, the feature vector F is fed into a fully connected layer and a sigmoid function to generate the importance weight corresponding to each frame of the image, and the obtained weight is multiplied by the corresponding feature. The result is the weighted facial expression feature. The calculation process is as follows:
[0019]
[0020]
[0021] where α n represents the weight corresponding to the nth frame image, σ is the sigmoid activation function, W fc is the parameter matrix of the fully connected layer, x n is the expression feature of the nth frame image, is the expression feature obtained by weighted calculation of the n-th frame image;
[0022] Calculate the fusion features of weighted features in the time dimension. At the initial moment, the initial state vector h1 corresponds to the features of the first frame. The input at the nth moment is the feature vector of the nth frame image, and the fusion vector generated at the nth moment is h n , the state vector obtained at the last moment is the video-level fusion vector of the sample. The specific calculation process of feature fusion at each moment is as follows:
[0023]
[0024] When calculating the state vector at the current moment, first multiply the state vector of the previous moment by the sum of the weights of all past moments, then add it to the feature input at the current moment, and divide the result by the sum of the weights of all moments including the current moment. The result is the state vector of each moment. After all moments have passed, the final fusion feature is obtained, and the 28-dimensional classification result P is obtained through linear mapping of the fully connected layer. f , as the prediction result of facial emotion.
[0025] Furthermore, the ResNeXt50 network is composed of stacked residual blocks. Each time the input image passes through a group of residual blocks, the feature map size is reduced by half through downsampling. There are four groups of residual block combinations, and the number of residual blocks in each group is 3, 4, 6, and 3 respectively. Finally, the 2048-dimensional feature vector extracted from each frame is obtained through average pooling.
[0026] Furthermore, for each residual block, the output can be expressed by the following formula:
[0027]
[0028] Where x is the input feature of the residual block, y is the output of the residual block, C represents the cardinality, that is, the number of branches, T i A computation function that represents a topological structure.
[0029] Furthermore, the weighted processing process of the CA module is to use two one-dimensional convolutions of size (H, 1) and (1, W) on each channel of the input feature map, encode it along the horizontal and vertical directions of the feature map respectively, and obtain two independent attention maps; splice the two attention maps to obtain a size of C×(W+H), and then transform them using convolution; after a layer of 1×1 convolution and a nonlinear activation function, an encoded intermediate feature map is obtained; the intermediate feature map is re-formed into two feature maps along the spatial dimension at the previously spliced position, and sent to 1×1 convolution and Sigmoid activation function respectively, and the number of channels of the feature map is transformed to the same number as the input; obtain attention maps with sizes of C×H×1 and C×1×W in the horizontal and vertical directions respectively, and multiply them with the input feature map in the corresponding dimensions to obtain a recalibrated weighted feature map.
[0030] Furthermore, the multi-scale feature fusion fuses the features output by different layers of the network to obtain an output that contains both low-level features and high-level features; after the feature map passes through the convolution and maximum pooling layers of the ResNeXt network, four groups of blocks are obtained, and the outputs of the first three blocks are spliced with the feature map output by the last block to obtain a multi-scale fused feature map.
[0031] Furthermore, Mogrifier LSTM is used as the feature extraction model of time series information; -1 Represents the input feature x at the current moment t , h0 represents the hidden layer state h at the previous moment t-1 , the update round number r is a configurable integer hyperparameter. When its value is 4 or 5, the interaction effect is the best. When r = 0, the network structure is completely equivalent to the traditional LSTM; the calculation formula for the alternating interaction between the two is as follows:
[0032]
[0033]
[0034] Where x represents the input feature at time t, h prev Indicates h t-1 , update round number i∈[1...r], parameter matrix Q i and R i Generated by random initialization, the two matrices are respectively x of the previous round. t and h t-1 When the number of rounds i is an odd number, the hidden state updated in the previous round interacts with the input features updated two rounds ago to generate the input features of this round When the number of rounds is even, do the same transformation in reverse to generate After the interaction is completed, it is sent to LSTM for subsequent calculations. The features of the video sequence are sent to the two-layer Mogrifier LSTM, and the output is converted into a 28-dimensional feature vector P through the fully connected layer. g , as the prediction result of situational emotion.
[0035] In the technical solution provided in the embodiment of the present application, the attention mechanism is introduced to enable the model to pay more attention to the learning of key information during the feature extraction process, which effectively improves the recognition rate of the model in the emotion recognition of the two branches. In the facial expression recognition branch, the introduction of the channel attention mechanism improves the model's ability to express key information, and the feature selection fusion method alleviates the problem of poor model performance caused by the uncertainty of facial expressions. In the scenario path, the multi-scale fusion method is used to enhance the model's ability to express high-level semantics and spatial information of the image. Through the complementarity of high-level features and low-level features, the model's scenario emotion recognition effect is improved, and in the final comparative experiment, it is proved that the emotion recognition method of the present application can obtain better recognition effects on the data set than other methods. BRIEF DESCRIPTION OF THE DRAWINGS
[0036] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0037] The methods, systems and / or programs in the accompanying drawings will be further described according to exemplary embodiments. These exemplary embodiments will be described in detail with reference to the drawings. These exemplary embodiments are non-limiting exemplary embodiments, wherein example numbers represent similar mechanisms in the various views of the accompanying drawings.
[0038] Figure 1 It is a schematic diagram of the overall architecture of facial expression recognition provided by an embodiment of the present application;
[0039] Figure 2 It is a schematic diagram of the ResNet18 structure provided in an embodiment of the present application;
[0040] Figure 3 It is a schematic diagram of the ECA module structure provided in an embodiment of the present application;
[0041] Figure 4 It is a schematic diagram of a Basic block after an ECA module is embedded provided in an embodiment of the present application;
[0042] Figure 5 It is a schematic diagram of the structure of the feature fusion module provided in the embodiment of the present application;
[0043] Figure 6 is a schematic diagram of the situational emotion recognition network structure provided in an embodiment of the present application;
[0044] Figure 7 Schematic diagram of the ResNeXt residual block structure provided in an embodiment of the present application;
[0045] Figure 8 It is a schematic diagram of the CA module structure provided in an embodiment of the present application;
[0046] Fig. 9 is a schematic diagram of a residual structure of an embedded CA module provided in an embodiment of the present application;
[0047] Fig.10 It is a schematic diagram of the CA module structure provided in an embodiment of the present application;
[0048] Fig.11 It is a schematic diagram of the Mogrifier LSTM structure provided in an embodiment of the present application. DETAILED DESCRIPTION
[0049] In order to better understand the above technical scheme, the technical scheme of the present application is described in detail below through the accompanying drawings and specific embodiments. It should be understood that the embodiments of the present application and the specific features in the embodiments are detailed descriptions of the technical scheme of the present application, rather than limitations on the technical scheme of the present application. In the absence of conflict, the embodiments of the present application and the technical features in the embodiments can be combined with each other.
[0050] In the following detailed description, numerous specific details are set forth by way of example in order to provide a comprehensive understanding of the relevant guidance. However, it will be apparent to those skilled in the art that the present application may be practiced without these details. In other cases, well-known methods, procedures, systems, compositions and / or circuits have been described at a relatively high level, without detail, in order to avoid unnecessarily obscuring aspects of the present application.
[0051] Flowcharts are used in the present application to illustrate the execution process performed by the system according to the embodiment of the present application. It should be clearly understood that the execution process of the flowchart may not be performed in order. On the contrary, these execution processes may be performed in reverse order or simultaneously. In addition, at least one other execution process may be added to the flowchart. One or more execution processes may be deleted from the flowchart.
[0052] Before further describing the embodiments of the present invention in detail, the nouns and terms involved in the embodiments of the present invention are described. The nouns and terms involved in the embodiments of the present invention are subject to the following interpretations.
[0053] (1) In response, it is used to indicate the conditions or states on which the executed operations depend. When the dependent conditions or states are met, one or more operations executed may be in real time or with a set delay. Unless otherwise specified, there is no restriction on the order in which the multiple operations executed are executed.
[0054] (2) Based on is used to indicate the conditions or states on which the executed operations depend. When the dependent conditions or states are met, one or more operations executed may be in real time or have a set delay. Unless otherwise specified, there is no restriction on the order in which the multiple operations executed are executed.
[0055] Facial emotion recognition
[0056] Facial expressions account for 55% of emotional expression, which shows that facial expression analysis is necessary. Facial expression recognition requires the use of deep learning models to learn and analyze facial expression features. Therefore, the input data should not contain information irrelevant to the face as much as possible to prevent the network from learning irrelevant features. In data processing, the face detection algorithm is used to capture the facial images of the annotated persons from the video sequence of the data set and scale them to a uniform size. Each video may contain 1 to 4 annotated persons. According to the detection frame number of each annotated person, their facial image sequence is saved in a folder named with their respective numbers to facilitate subsequent data reading.
[0057] The SVCEmotion dataset is a video data. Each sample consists of a large number of image sequences, and the average length of the sample is about 10 seconds. The changes in human facial expressions are affected by the surrounding environment and events. Therefore, longer video samples can retain a more complete expression change process and richer emotional expressions, providing more information for multi-label sentiment analysis. Because facial expressions change slowly, the difference between two adjacent frames may be small. Considering the large amount of data in the entire video, N frames are randomly selected from each face image sequence as input in the model training phase, where N = 16. In the test phase, the image sequence of the entire video is used as input.
[0058] The facial emotion recognition part uses ResNet18 as the basic network for image feature extraction. Considering that as the network deepens, the number of channels gradually increases, but the feature maps learned by different channels do not contribute exactly the same to the total classification, redundant information will appear. To alleviate this situation, the channel attention (ECA) module is introduced. After extracting the feature representation of each frame, it is necessary to learn the temporal information of the features of each frame. Since the facial angles of the characters in the dataset of this article vary greatly, and the sparse sampling method will lead to large feature differences between two consecutive frames, in addition, the facial expressions of the characters are not at their peak throughout the process, misjudgment may occur. Therefore, the importance weight generation method is introduced to solve this problem. Finally, the sliding fusion method is used to perform temporal fusion according to the importance weights corresponding to each frame of the image, and the fusion results are sent to the classifier to obtain the final classification result. The overall architecture of facial expression recognition is shown below. Figure 1 shown.
[0059] Expression feature extraction network based on channel attention
[0060] In the feature extraction part of facial expressions, a convolutional neural network is first used to extract emotional features from the image sequence frame by frame, and then a network that can process time series information is used to learn the interdependence between features at different times to obtain the final recognition result. ResNet18 is used as the backbone network to extract the expression features of each frame of the image. The network has a total of four residual block combinations, each of which contains two Basic Blocks. Finally, the output is obtained through the average pooling layer and the fully connected layer. Due to the changes in the distance of the camera in the video, the size of the captured facial images is not uniform, generally between 200 and 300 pixels, so the image size is uniformly scaled to the standard input size of ResNet, that is, 224×224. The feature map output by the last residual block has a size of 7×7 and a number of channels of 512. The output of this part of the network takes the 512-dimensional feature vector of the average pooling layer. The specific structure of the network and the size of the feature map are as follows: Figure 2 shown.
[0061] In this network, the number of channels of the feature map doubles every time the input passes through a group of residual blocks, and the maximum number of channels reaches 512. Different channels learn features of different types and regions in the image, but some channels may learn redundant information. The contribution of the feature map on each channel to the learning of expression features is not exactly the same, and the network cannot distinguish and focus on information related to emotional expression like humans. In order to highlight the feature maps with greater contributions, the channel attention mechanism is introduced into the network. By explicitly constructing the correlation between channels and assigning different importance weights to each channel, the model can focus more on feature extraction in key areas.
[0062] The literature has improved the problems of SENet and proposed a lightweight Efficient Channel Attention (ECA) module. Wang et al. found through experiments that the way of modeling the dependencies between all channels at once through the dimension reduction of the fully connected layer in the excitation process of SENet is inefficient, and this dimension reduction operation is unnecessary. In the ECA module, after using global average pooling for each input feature map, the local interaction information between each channel and its adjacent channels is captured by using a one-dimensional convolution kernel, avoiding the dimension reduction operation of the fully connected layer, and achieving better results than SENet with fewer parameters. The specific structure of the ECA module is as follows: Figure 3 shown.
[0063] The input of the module is a set of feature maps with C channels, W width and H height respectively. After global average pooling, a set of 1×1×C feature vectors is obtained, in which each element represents the global information of the feature map of its corresponding channel. By using one-dimensional convolution to perform convolution operation on the obtained vector, the method of fully connected layer is replaced to capture the correlation between adjacent channels within a certain range and realize cross-channel information interaction. The size of the convolution kernel determines the number of channels covered by each calculation of local interaction, and its specific size is determined in an adaptive way. Based on the assumption that the coverage of local interactions is proportional to the number of channels, there may be a nonlinear mapping relationship between the convolution kernel size k and the number of channels C, which is specifically expressed as follows:
[0064] C=φ(k)=2 (γ*k-b)
[0065] (4-7)
[0066] From this, we can get the size calculation formula of the convolution kernel k:
[0067]
[0068] Among them, γ and b are parameters defined for nonlinear mapping, which are set to 2 and 1 respectively. In the formula, |t| odd The meaning of is to take the odd number closest to the absolute value of t. After determining the value of k, a feature vector of the same size is obtained by performing a one-dimensional convolution on the feature vector, thereby realizing local cross-channel information interaction. After a sigmoid function, the weight representing the importance of each channel is obtained, and it is multiplied with the corresponding channel feature map to obtain a weighted feature map.
[0069] The ECA module is integrated into the position before the residual connection in each Basicblock of ResNet by adding an Identity connection. The Identity connection is connected to the output of the ECA module by adding the elements at the corresponding positions. The attention weight processed by the sigmoid function is converted into a value between 0 and 1. This connection method can enhance the weight of the key channel on the original basis, which is equivalent to uniformly increasing the channel attention weight by 1, further highlighting the information expression of the feature map. The specific structure is as follows Figure 4 shown.
[0070] Image sequence feature fusion module
[0071] By using ResNet18 with the channel attention mechanism, we get the feature vector F={f1,...,f N}. Since the image sequence is randomly extracted from each video sample in order, there may be large differences in facial posture, expression intensity, and emotion category between two adjacent frames, and the multimodal emotion dataset contains multiple ways of expressing emotions. The expression intensity of different modalities at the same time may be different, so the facial expression data is not always at the peak of the emotional state. The above situations have a certain degree of impact on the analysis of facial expressions. Therefore, the use of conventional recursive neural networks to mine the interdependencies between image sequences cannot effectively integrate the emotional features at different times. In order to suppress the noise introduced by the mislabeling of the dataset labels, the literature uses a feature extraction network to obtain the facial feature representation of each sample, and then sends the feature vector to the fully connected layer and sigmoid function to assign importance weights to each sample. Samples with lower weights indicate that their emotional categories do not match the labels and will be re-identified and labeled. Based on this method, this section considers introducing the idea of importance weight generation to reallocate the importance weights of the feature vectors obtained through the backbone network. Frames with complete facial expressions and strong emotional expressions will be assigned higher weights, and frames with side faces or unclear emotional expressions will be assigned lower weights. The feature vectors are combined with the corresponding weights to achieve feature vector weighting for expression uncertainty. Then, a feature selection fusion method is used to fuse the recalibrated feature vectors in time series to obtain the final emotion recognition result. The specific structure of the Feature Selection Fusion Module (FSFM) is as follows: Figure 5 shown.
[0072] The input of the network is a set of N-frame image sequences. After feature extraction by the backbone model, N feature vectors with a dimension of 512 are obtained. The feature vectors are sent to a fully connected layer and a sigmoid function to generate the importance weights corresponding to each frame of the image, and the obtained weights are multiplied by the corresponding features. The result is the weighted facial expression feature. The calculation process is as follows:
[0073]
[0074]
[0075] where α n represents the weight corresponding to the nth frame image, σ is the sigmoid activation function, W fc is the parameter matrix of the fully connected layer, x n is the expression feature of the nth frame image, is the expression feature obtained by weighted calculation of the nth frame image. Next, the fusion feature of the weighted feature in the time dimension is calculated. Each frame image corresponds to a moment. At the initial moment, the initial state vector h1 corresponds to the feature of the first frame. The input at the nth moment is the feature vector of the nth frame image, and the fusion vector generated at the nth moment is h n , the state vector obtained at the last moment is the video-level fusion vector of the sample. The specific calculation process of feature fusion at each moment is as follows:
[0076]
[0077] When calculating the state vector at the current moment, first multiply the state vector of the previous moment by the sum of the weights of all past moments, then add it to the feature input at the current moment, and divide the result by the sum of the weights of all moments including the current moment. The result is the state vector of each moment. After all moments have passed, the final fusion feature is obtained, and the 28-dimensional classification result P is obtained through linear mapping of the fully connected layer. f , as the prediction result of facial expression.
[0078] Situational emotion recognition
[0079] Humans will show different emotional states in different environments, and the scene or surrounding things will have a certain impact on emotions. When analyzing the emotions of the target subject, it is not possible to analyze them separately from the current situation. It is necessary to consider the factors that have a decisive influence on the emotional state. Therefore, the situational information contained in the video samples in the data set can be used to assist in sentiment analysis. In the training stage of the model, this branch also uses random sampling to take N frames of images from each video sample in sequence as input. Before the image is input, the target subject needs to be blocked so that the model can learn the situational information other than the target subject. If a frame of the image does not contain the target subject, the entire image is used as the input of the model. The video sample is composed of an image sequence, which contains a large amount of complex situational information, but not all of the content is helpful for sentiment analysis. Only some key areas have a greater impact on the emotional state of the target subject. At the same time, since the data set is captured from film and television works, there will inevitably be camera switching in the video. At this time, the situational content of the image sequence will change, and the key areas that affect the emotional state of the target subject will also change. Based on the above analysis, if a simple backbone network is used to extract features of scene information in an image, the model cannot effectively distinguish which parts of the image are important.
[0080] Based on this, this section uses ResNeXt as the image sequence feature extraction model and embeds a hybrid attention module to make the model pay more attention to the key areas that affect the target subject's emotions. In order to make the features extracted at different levels of the model complement each other, the features extracted from different network layers are averaged and pooled, and then spliced with the final output feature map to obtain a multi-scale fused feature representation. The fused features are mapped to a 1024-dimensional feature vector through a fully connected layer, and then sent to the Mogrifier LSTM that processes time series information. The connection between the situational information at different times is mined, and the final output is sent to the fully connected layer to obtain the final emotion recognition result. The overall network structure is shown below. Figure 6 shown.
[0081] Context feature extraction network embedded with hybrid attention
[0082] Since the input of scene sentiment analysis is the entire image, using a network with fewer layers cannot obtain good classification results. In recent years, researchers have been committed to deepening and widening the network model to improve the recognition effect of the network. ResNeXt is an improved network based on ResNet. Inspired by the Inception network, Inception uses convolution kernels of different sizes in different branches in each layer of the network, and then aggregates the branches, so that the network can obtain receptive fields of different sizes in each layer, which can improve the model's ability to mine information for images. However, since the hyperparameters of Inception are highly targeted and not easy to extend to other tasks, ResNeXt proposes an aggregation transformation structure based on this design idea, changing the original residual block structure to a grouped convolution form, divided into several branches, using parallel stacking of the same topological structure, and then merging the branches before the residual connection. This method reduces the number of hyperparameters and improves the model performance without significantly increasing the number of parameters. The number of branches is determined by the parameter cardinality, which is set to 32 in this article. There are three equivalent structures of residual blocks. The specific structure of the residual block used in this article is as follows: Figure 7 When the number of input channels is 256, the number of channels is reduced to 128 by using 1×1 convolution, and then it is divided into 32 branches, each with 4 channels. After passing through the convolution layer with a convolution kernel size of 3×3, the number of channels is increased to 256 by using 1×1 convolution. The feature maps of the 32 branches are added by position and then added to the direct connection path.
[0083] For each residual block, the output can be expressed by the following formula:
[0084]
[0085] Where x is the input feature of the residual block, y is the output of the residual block, C represents the cardinality, that is, the number of branches, T i The operation function representing the topological structure maps the input to a lower-dimensional integration, then implements the same transformation on each branch, and finally adds the outputs of the 32 topological structures bit by bit to obtain the output of each residual block. This paper selects ResNeXt50 as the feature extraction model. Figure 6 The residual blocks shown are stacked. After each group of residual blocks, the feature map size is reduced by half through downsampling. There are four groups of residual block combinations, and the number of residual blocks in each group is 3, 4, 6, and 3 respectively. Finally, the 2048-dimensional feature vector extracted from each frame is obtained through average pooling.
[0086] Since the information in the image contributes differently to the final sentiment classification, the number of channels increases exponentially as the network deepens, and the importance of different channels to the task is also different, the model needs to allocate more computing resources to key areas. The Coordinate Attention (CA) module belongs to the hybrid attention module. By embedding the position information into the channel attention, the model obtains accurate position information. Figure 8 The specific structure of the CA module is shown. For a given input feature map, two one-dimensional convolutions of size (H, 1) and (1, W) are used on each channel to encode the feature map in the horizontal and vertical directions, respectively, to obtain two independent attention maps. This step is equivalent to decomposing the global average pooling operation into feature aggregation operations in two directions. This transformation method enables the attention module to capture long-distance dependencies while retaining accurate location information, which helps the network focus on key areas more accurately. Next, the two attention maps are concatenated to a size of C×(W+H), and then transformed using convolution, where r is the downsampling ratio. After a layer of 1×1 convolution and nonlinear activation function transformation process, an encoded intermediate feature map is obtained. The intermediate feature map is re-formed into two feature maps along the spatial dimension at the previously concatenated position, and is respectively fed into 1×1 convolution and Sigmoid activation function to transform the number of channels of the feature map to the same number as the input. After the above steps, we get the attention maps in the horizontal and vertical directions, and also get the interdependence between channels. The sizes of the attention maps are C×H×1 and C×1×W respectively. We multiply them with the input feature map in the corresponding dimensions to get the recalibrated weighted feature map. Compared with the direct use of the global average pooling operation, the attention map generation method of this module avoids the loss of position information, explicitly constructs the relationship between channels, and obtains an attention map that is sensitive to both direction and position, realizing the combination of spatial attention and channel attention.
[0087] The coordinated attention module is embedded in each residual structure of ResNeXt50, and the connection method is the same as Section 3.2. After each residual block completes the group convolution and branch aggregation, before the residual addition, it first passes through the CA module to strengthen the feature representation of key information in space and channels, and then the residual connection is performed. The specific structure is as follows Fig. 9 shown.
[0088] Multi-scale feature fusion
[0089] The features extracted by each layer of the convolutional neural network have different properties. The receptive field of the shallow network is small, and it generally extracts common features such as color, texture and lines in the image, which are also called low-level features. The receptive field of the deep features gradually increases, and the final output extracts high-level features with the global receptive field of the image, which contains the semantic information of the entire image. In theory, both the spatial information contained in the low-level features and the semantic information in the high-level features contain key information that contributes to sentiment analysis. The semantic information identifies the content of the entire image, and the low-level features capture the spatial information in the image. The two convey sentiment features from different angles. However, as the number of layers of the network increases, the key information contained in some low-level features is ignored. Therefore, this paper considers using the multi-scale feature fusion method to fuse the features output by different layers of the network to obtain an output that contains both low-level features and high-level features, so that local information and global information complement each other and obtain a richer and more comprehensive feature expression. Since ResNeXt is composed of stacked residual blocks, after 7×7 convolution and maximum pooling layers, four groups of blocks are stacked, and the outputs of the first three blocks are spliced with the feature map of the last output to achieve multi-scale fusion of low-level features and high-level features. Due to the different sizes of feature maps at different levels, adaptive average pooling (Adaptive AvgPool) is used to process each group of feature maps during connection. The particularity of this method is that the size of its output feature map is given, and the output feature map is generated by automatically selecting the appropriate step size and sampling size, and the number of channels before and after does not change. The size of each layer of feature map is transformed to the size of the feature map of the last Block, and they are spliced in the channel dimension, and finally the output is obtained through the average pooling layer. The specific structure of this part is as follows: Fig.10 shown.
[0090] Time series feature processing module
[0091] After using the above feature extraction network to obtain the frame-level features of the video sequence, it is necessary to learn the feature dependencies between different moments through a model that can process temporal information. In the traditional LSTM structure, the input x at each time step is t and the hidden layer state h at the previous moment t-1 are independent of each other. Mogrifier LSTM
[80] makes improvements based on the LSTM network. Before entering the LSTM operation at each time step, the current input x t With h t-1The two models are modulated in an alternating manner, and the results of the interaction are then fed into LSTM. The purpose is to enhance the model's ability to mine the correlation between contextual information through the interaction between the two models, enhance significant features, and weaken irrelevant redundant features. This section uses MogrifierLSTM as a feature extraction model for time series information. Fig.11 This is a calculation process with 5 alternating update rounds.
[0092] x in the figure -1 Represents the input feature x at the current moment t , h0 represents the hidden layer state h at the previous moment t-1 , the update round number r is a configurable integer hyperparameter. When its value is 4 or 5, the interaction effect is the best. When r = 0, the network structure is completely equivalent to the traditional LSTM. The calculation formula for the alternating interaction between the two is as follows:
[0093]
[0094]
[0095] Where x represents the input feature at time t, h prev Indicates h t-1 , update round number i∈[1...r], parameter matrix Q i and R i Generated by random initialization, the two matrices are respectively x of the previous round. t and h t-1 When the number of rounds i is an odd number, the hidden state updated in the previous round interacts with the input features updated two rounds ago to generate the input features of this round When the number of rounds is even, do the same transformation in reverse to generate After the interaction is completed, it is sent to LSTM for subsequent calculations. The features of the video sequence are sent to the two-layer Mogrifier LSTM, and the output is converted into a 28-dimensional feature vector P through the fully connected layer. g , and obtain the prediction results for each emotion category.
[0096] This example will introduce the experimental results of facial expression recognition and situational emotion recognition. Due to the complex scenes in the data set, some samples may not be able to detect faces. Therefore, for samples with missing data, the corresponding annotations will be deleted, and only the data that can detect faces will be retained. This section verifies the effectiveness of the attention mechanism for facial expression and situational emotion recognition, the improvement of the feature selection fusion module for the problem of facial expression uncertainty, and the improvement effect of multi-scale feature fusion on situational emotion recognition through experimental results, and analyzes the experimental results. During the training process of facial expression and situational emotion recognition, the initial learning rate is set to 0.001.
[0097] In the dataset built based on real environments, the composition of audio signals is very complex. Audio data collected under laboratory conditions usually only contains the dialogue sounds of actors, but in real environments, in addition to verbal communication between people, it may also contain sounds in the environment. The audio of the SVCEmotion dataset collected from film and television dramas often contains sounds from some scenes, and judging the scene category can also help with sentiment analysis.
[0098] The facial expression image sequence in the SVCEmotion dataset is used as the experimental data, and the ResNet18+feature selection fusion module is used as the basic model for feature extraction. Since the dataset mainly considers whether the video as a whole has significant emotional expression, such as voice intonation and character behavior during the collection process, it is judged whether to use it as a dataset by combining the performance of multiple modalities. Therefore, in some samples, the facial posture of the character will change during the camera switching process, and it is easy to have unclear expressions, which will lead to misjudgment of expressions. The feature selection fusion module used in this chapter can assign higher weights to frames with more significant emotional features, and can make full use of discriminative video frames. In order to improve the model's ability to learn key areas of different channel features, the ECA channel attention module is introduced. Table 4.1 compares the experimental results of the basic model and the introduction of the ECA module.
[0099] Table 4.1 Facial expression recognition experimental results
[0100]
[0101] The dataset constructed in this paper is currently small in scale. Due to the lack of some facial data, the sample size is smaller. The training of the model requires sufficient data as support. Due to the lack of sufficient training data, the model designed in this section still performs poorly when directly applied to the SVCEmotion dataset. Therefore, the convolutional network part of the model is used. After pre-training on the HEUEmotion dataset, the parameters are migrated to the training task of the dataset. HEU Emotion is a large-scale video dataset. A total of 19,004 videos are collected from the Internet and film and television works. 10 categories of emotions are annotated. The labels are in the form of single labels. The facial angle range of the characters is large, which is similar to the situation of the SVCEmotion dataset. At the same time, there are changes in lighting conditions and clarity, which are similar to the problems encountered in the real environment. The large data scale is suitable for pre-training tasks, enabling the model to be trained on the new target task with a better initial point, quickly converge and stabilize, and obtain results close to audio emotion recognition. Table 4.2 shows the results of AP and RA of each expression category after the introduction of the attention mechanism.
[0102] It can be seen from the table that the recognition effect of categories with fewer samples is poor, which may be related to the distribution of the dataset itself and the selection of fragments.
[0103] Table 4.2 Facial emotion recognition results embedded in the ECA module
[0104]
[0105]
[0106] The global image of the video in the dataset is used as the input of the scene branch, and ResNeXt50+MogrifierLSTM is used as the basic network for feature extraction. In order to make the feature extraction network pay more attention to the key targets in the image, the coordinated attention module CA is introduced, and the multi-scale feature fusion method is used to enable the model to learn feature representations of different receptive fields. Table 4.3 compares the experimental results of the basic model, basic model + CA, and basic model + CA + multi-scale feature fusion.
[0107] Table 4.3 Experimental results of situational sentiment analysis
[0108]
[0109] From the data in the table, we can observe that situational information can provide a certain degree of auxiliary effect for emotion recognition. There is a connection between the scene and the emotion category in the video, and some samples show the interaction between the target subject and the non-target subject. Therefore, there may be emotional sharing or mutual influence. The facial expressions of the non-target subject also assist emotion recognition to a certain extent. The addition of the attention mechanism and the multi-scale feature fusion method have improved the performance of the model to a certain extent, indicating that the attention mechanism can effectively help the model enhance the learning of key targets. The multi-scale feature fusion model enables the model to enhance the comprehensive expression ability of features at all levels, so that it can learn more situational targets related to the target person's emotions. Since situational information only plays an auxiliary role in emotion analysis, and in the data set constructed in this paper, there are more close-ups, the target person accounts for a large proportion of the shots, and the proportion of distant shots is not high, so the model learns less situational information in the feature extraction process, and the overall recognition rate is low. In the subsequent expansion of the data set, we can focus on the key information of the video as a whole.
[0110] In order to verify that the situational emotion recognition method proposed in this chapter has an auxiliary effect on facial expression recognition, the recognition results of the two branches are directly cascaded for emotion recognition and compared with the video-based situational emotion recognition method CAERNet. The experimental results are compared as follows:
[0111] Table 4.4 Comparison of experimental results between visual modality emotion recognition network and CAERNet
[0112]
[0113] The CAERNet in the table is a two-stream encoding network. It introduces spatial attention in the scene feature extraction branch, and fuses the features of the face and scene branches through an adaptive fusion network and sends them to the classifier. It can be seen from the table that the feature extraction network designed for SVCEmotion in this paper achieves better results. Since the dataset in this paper contains more complex scene information and the facial expressions of the target characters are uncertain, it is easy to cause misjudgment. This paper alleviates the impact of this problem by using channel attention and feature selection fusion, and improves the model's ability to discriminate and extract complex scene information through methods such as mixed attention and multi-scale feature fusion.
[0114] The above embodiments can be implemented in whole or in part by software, hardware (such as circuits), firmware or any other combination. When implemented using software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a tape), an optical medium (for example, a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state hard disk.
[0115] In this application, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can be represented by: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple.
[0116] It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.
[0117] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.
[0118] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.
[0119] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.
[0120] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0121] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.
[0122] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage media include: various media that can store program codes, such as USB flash drives, mobile hard disks, read-only memories (ROM), random access memories (RAM), magnetic disks or optical disks.
[0123] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.
Claims
1. A facial and situational emotion recognition method based on an attention mechanism, comprising a facial emotion recognition method and a situational emotion recognition method; characterized in that: The facial emotion recognition method comprises the steps of: extracting facial emotion features frame by frame from an input image sequence using a ResNet18 network to obtain a facial emotion feature map; inputting the facial emotion feature map extracted by the ResNet18 network into an ECA module to obtain a weighted facial emotion feature map; inputting the facial emotion feature map weighted by the ECA module into a feature selection fusion module for feature fusion processing to obtain a facial emotion recognition result P. f ; ResNet18 extracts the expression features of each frame of the image. After the average pooling layer and the fully connected layer, the output feature map size is W = 7, H = 7, and the number of channels is C = 512; The feature map processed by ResNet18 is input into the ECA module. After global average pooling, a set of 1×1×C feature vectors is obtained. The feature vector is convolved using one-dimensional convolution. The size calculation formula of the convolution kernel k is: Among them, γ and b are parameters defined for nonlinear mapping, which are set to 2 and 1 respectively. |t| odd The meaning of is to take the odd number closest to the absolute value of t. After determining the k value, a feature vector of the same size is obtained by performing a one-dimensional convolution on the feature vector to achieve local cross-channel information interaction; after a sigmoid function, a weight representing the importance of each channel is obtained, which is multiplied with the corresponding channel feature map to obtain a weighted feature map. The feature vector corresponding to the weighted N image sequence is F = {f1,...,f N }; The ECA module is integrated into the position before the residual connection in each Basic block of ResNet by adding an Identity connection; the Identity connection is connected to the output of the ECA module by adding the elements at the corresponding positions; The feature vector F is sent to a fully connected layer and a sigmoid function to generate the importance weight corresponding to each frame of the image, and the obtained weight is multiplied by the corresponding feature. The result is the weighted facial expression feature. The calculation process is as follows: where α n represents the weight corresponding to the nth frame image, σ is the sigmoid activation function, W fc is the parameter matrix of the fully connected layer, x n is the expression feature of the nth frame image, is the expression feature obtained by weighted calculation of the n-th frame image; Calculate the fusion features of weighted features in the time dimension. At the initial moment, the initial state vector h1 corresponds to the features of the first frame. The input at the nth moment is the feature vector of the nth frame image, and the fusion vector generated at the nth moment is h n , the state vector obtained at the last moment is the video-level fusion vector of the sample. The specific calculation process of feature fusion at each moment is as follows: When calculating the state vector at the current moment, first multiply the state vector of the previous moment by the sum of the weights of all past moments, then add it to the feature input at the current moment, and divide the result by the sum of the weights of all moments including the current moment. The result is the state vector of each moment. After all moments have passed, the final fusion feature is obtained, and the 28-dimensional classification result P is obtained through linear mapping of the fully connected layer. f , as the prediction result of facial emotion; The situation emotion recognition method comprises the steps of: taking frame images from each video sample in sequence as input; extracting situation emotion features from the image sequence frame by frame using a ResNeXt50 network to obtain a situation emotion feature map; inputting the situation emotion feature map extracted by the ResNeXt50 network into a CA module to obtain a weighted situation emotion feature map; inputting the situation emotion feature map weighted by the CA module into a multi-scale feature fusion module to obtain a feature vector x output by the situation emotion. t ; The above eigenvector x t Input to the MogrifierLSTM module and process to obtain the situation emotion recognition result P g ; The weighted processing process of the CA module is to use two one-dimensional convolutions of size (H, 1) and (1, W) on each channel of the input feature map, encode it along the horizontal and vertical directions of the feature map respectively, and obtain two independent attention maps; splice the two attention maps to obtain a size of C×(W+H), and then use convolution to transform them; after a layer of 1×1 convolution and nonlinear activation function transformation process, an encoded intermediate feature map is obtained; the intermediate feature map is re-formed into two feature maps along the spatial dimension at the previously spliced position, and is respectively sent to 1×1 convolution and Sigmoid activation function to transform the number of channels of the feature map to the same number as the input; obtain attention maps with sizes of C×H×1 and C×1×W in the horizontal and vertical directions respectively, and multiply them with the input feature map in the corresponding dimensions to obtain a recalibrated weighted feature map; The multi-scale feature fusion fuses the features output by different layers of the network to obtain an output that contains both low-level features and high-level features. After the feature map passes through the convolution and maximum pooling layers of the ResNeXt network, four groups of blocks are obtained. The outputs of the first three blocks are spliced with the feature map output by the last block to obtain a multi-scale fused feature map. Use MogrifierLSTM as the feature extraction model for time series information; -1 Represents the input feature x at the current moment t , h0 represents the hidden layer state h at the previous moment t-1 , the update round number r is a configurable integer hyperparameter. When its value is 4 or 5, the interaction effect is the best; when r = 0, the network structure is completely equivalent to the traditional LSTM; the calculation formula for the alternating interaction between the two is as follows: Where x represents the input feature at time t, h prev Indicates h t-1 , update round number i∈[1...r], parameter matrix Q i and R i Generated by random initialization, the two matrices are respectively x of the previous round. t and h t-1 Realize linear transformation; when the round number i is an odd number, the hidden state updated in the previous round interacts with the input features updated two rounds ago to generate the input features of this round When the number of rounds is even, do the same transformation in reverse to generate After the interaction is completed, it is sent to LSTM for subsequent calculations; the features of the video sequence are sent to the two-layer MogrifierLSTM, and the output is converted into a 28-dimensional feature vector P through the fully connected layer. g , as the prediction result of situational emotion.
2. A facial and situational emotion recognition method based on attention mechanism according to claim 1, characterized in that: The facial emotion recognition method comprises: using a face detection algorithm to capture facial images of the annotated person from a video sequence of a data set, scaling them to a uniform size, and saving their facial image sequences in folders named with their respective numbers according to the detection frame numbers of each annotated person.
3. A facial and situational emotion recognition method based on attention mechanism according to claim 1, characterized in that: The facial emotion recognition randomly extracts N frames of images from each face image sequence as input during the model training phase, and takes the image sequence of the entire video as input during the testing phase; The scenario emotion recognition will take N frames of images from each video sample in sequence as input during the model training phase. Before the image is input, the target subject needs to be blocked so that the model can learn scenario information other than the target subject. If a frame of the image does not contain the target subject, the entire image will be used as the input of the model. During the testing phase, the image sequence of the entire video will be used as the input of the testing phase.
4. The method for facial and situational emotion recognition based on attention mechanism according to claim 1, characterized in that: The ResNeXt50 network is composed of stacked residual blocks. Each time the input image passes through a group of residual blocks, the feature map size is reduced by half through downsampling. There are four groups of residual block combinations, and the number of residual blocks in each group is 3, 4, 6, and 3 respectively. Finally, the 2048-dimensional feature vector extracted from each frame is obtained through average pooling.
5. The method for facial and situational emotion recognition based on attention mechanism according to claim 3, characterized in that: For each residual block, the output can be expressed by the following formula: Where x is the input feature of the residual block, y is the output of the residual block, C represents the cardinality, that is, the number of branches, T i A computation function that represents a topological structure.
