A multi-feature expression recognition method based on deep learning

Abstract

A multi-feature expression recognition method based on deep learning belongs to the field of machine learning and deep learning technology, including the following: obtaining a clear image containing a human body and an environment and performing simple preprocessing, compressing the image to a primary color RGB image of 640*640 or smaller, retaining the color channel, and using the public dataset Emotic expression recognition dataset; the overall network is composed of a target detection network in parallel with a scene information recognition network, and finally outputting to a classification network. The target detection network consists of a backbone network CSPdarknet53, a feature pyramid structure for feature extraction in parallel with a feature enhancement network, and finally outputting the position information and 0 / 1 confidence information of the human body, which are then sent to the classification network for classification and output of the final result together with the scene information. The present invention uses a multi-scale context information extraction module, a context attention module, and a content attention module to achieve the purpose of improving the foreground weight of the image and increase the detection ratio of people in the image.

Description

Technical Field

[0001] The present invention belongs to the technical field of machine learning and deep learning, and specifically relates to a multi-feature expression recognition method based on deep learning. Background Art

[0002] Existing deep learning-based expression recognition systems can only identify a few basic expressions. These judgments are typically based on obvious facial changes compared to network-learned features. This results in the network's reliance on a single facial feature for expression recognition, and the system is often categorized as a single basic expression, making it less practical in medicine. Other expression analysis methods that incorporate voice, movement, or patient age information are also limited by the need for excessive manual calibration and analysis, which increases labor and time costs, and prevents automated, real-time, and accurate expression analysis.

[0003] In today's society, people face immense pressures in their daily lives and work, and psychological issues are often overlooked. Psychological and mental health issues such as bipolar disorder, anxiety, and neurasthenia are becoming increasingly prevalent among young people. Diagnosis is often based on subjective impressions, coupled with medical intervention. Conventional psychological counseling and treatment involves observing the patient's facial changes, recording their movements, and engaging in verbal communication to understand their psychological and mental state. This requires regular follow-up visits by professional physicians, a complex process. Currently, there is no automated diagnostic method for psychological issues, particularly facial expression recognition. Facial expression recognition is a common method for diagnosing psychological issues and represents the most intuitive expression of the human body's inner emotions. Both traditional and existing expression recognition methods utilize basic neural network models or deep learning models. Classification also relies on Ekman's basic expression classification method, focusing primarily on identifying distinct facial features, such as facial features and muscle movements. These are indeed the primary means of expressing human expressions. However, the deep learning training process, which uses images and corresponding labels, easily loses a lot of feature information. This results in large feature gaps between the images used. For example, happiness requires a noticeable upward turn of the corners of the mouth, and the overall face must exhibit distinct signs of happiness. Deep learning models can only recognize these expressions during forward propagation, resulting in a basic expression. These distinct expressions, such as anger and fear, are generally categorized as basic expressions, making them of limited medical practicality. Diagnosis of mental illness is rarely based on a patient's overt facial expressions. Patients with mental illness often struggle to express their emotions, and their facial expressions are often difficult to predict. Accurately assessing a person's mental state relies more on a combination of behavioral changes, environmental influences, and even physiological characteristics such as age and gender.

[0004] Existing technical solutions have high requirements for images, and the quality of images directly affects the recognition results. At the same time, existing solutions rarely achieve real-time face tracking and detection. This solution uses a variant of the object detection network YOLO to solve this problem, realizing automated, arbitrary and real-time emotion recognition.

[0005] Among existing technical solutions, convolutional neural networks or machine learning are commonly used as the main framework of the expression recognition model, using Ekman's expression classification into six basic categories, including happiness, sadness, surprise, anger, disgust, and fear. This is a commonly used classification method in the field of intelligent recognition, but it all uses pure facial features to identify basic expressions. This resulted in low accuracy rates for early expression recognition methods. Therefore, in later research, more features were used to enhance expression recognition results. For example, the Chinese invention patent application with publication number CN114783034A proposes a facial expression recognition method based on the fusion of local sensitive features and global features. The method divides the face into different parts and uses an attention mechanism model to strengthen the weights of the facial features to achieve the purpose of improving recognition rate. However, it is still a single-modal single-feature recognition method and is greatly affected by facial images. The Chinese invention patent application with publication number CN114724222A uses a multimodal approach to randomly combine facial expressions, voice, and text to enhance recognition randomness, and take the average to obtain the final result to improve the recognition rate. Multimodality has improved the defects of traditional recognition methods to a certain extent, but it still cannot achieve multiple classifications and real-time recognition of expressions. More existing technical solutions are described in detail in the papers "A Review of Facial Expression Recognition Technology" and "Deep Facial Expression Recognition: A Survey". The review of facial expression recognition technology mentions three shortcomings of existing technologies: in single-modal and single-feature facial expression recognition, the data set is insufficient and confusing, and it mostly remains in the laboratory stage, with low recognition rate and high error rate, and cannot be used in daily life. This has led to the development of multimodal methods, which combine more information to assist facial expression recognition to improve the accuracy of facial expression recognition, but still cannot solve various problems of the data set. At this time, the proposed data set enhancement method and network model can more effectively help the recognition effect of the multimodal network, such as collecting more field environment data so that the model can fit this data method for recognition, or using more data enhancement methods and using generative adversarial networks to generate more data composed of network learning features.

[0006] The present invention is inspired by the existing Emotic dataset (Reference: Context Based Emotion Recognition Using EMOTIC Dataset) and the Places dataset (Reference: Places: A 10 million Image Database for Scene The emoticon dataset proposes a method for facial expression recognition and classification in a multimodal network. It provides 26 sub-expressions based on six basic expressions. These 26 expressions represent all human expressions summarized in psychology books and literature. Three continuous quantities representing degree of expression are also added. The continuous dimension of emotion classification includes three aspects of emotion description: Pleasure (P) measures the degree of pleasure of a person's emotions. The size of the P value indicates the person's emotions from negative to positive. It is a relatively direct method to express emotional state. Arousal (A) measures a person's current calmness. The size of the A value indicates whether the person is anxious, excited, or idle. It can represent a variety of emotional states and is a reference value for judging diseases such as ADHD and depression. Dominance (D) indicates the degree of control over a person's expression and whether they can quickly control spontaneous changes in expression. It is used to judge the influence of psychology or mental state on emotions. The Places dataset proposes a method for scene recognition. Scene recognition is also very helpful for expression recognition because the probability of a person making a certain expression in a specific scene is relatively high. This will increase the model's weight in a certain category, while narrowing the recognition range of the model's results and improving the overall recognition accuracy.

[0007] Defects in the existing technology:

[0008] 1. Existing single-modal methods only focus on facial changes, but deep learning networks tend to lose a lot of details during the learning and training process. Therefore, facial features must be very clear and free of excessive interference.

[0009] 2. The expression classification used in existing multimodal and unimodal methods is too general and cannot express the subtle differences between expressions. It can only be roughly divided into six types of expressions, which is too simple.

[0010] 3. The quality and quantity of the dataset are too low, so the network cannot well obtain the inter-class features and gaps during training;

[0011] 4. Existing methods cannot achieve real-time, arbitrary expression recognition and can only use images in a complete laboratory environment; Summary of the Invention

[0012] In order to solve the above problems, the present invention combines a variety of information such as human motion features, environmental features and facial expressions, and divides this information into 26 different emotion categories, including 3 continuous classifications representing the current emotional state of the human body, such as the size of the aura, the intensity of the emotion and the degree of excitement. It uses the target detection model and the attention model to capture and detect human motion and facial expressions in real time, and uses the scene information, motion information and expression information to judge the current emotion of the person, thus realizing a method of auxiliary diagnosis in the medical field. The present invention proposes: a multi-feature expression recognition method based on deep learning,

[0013] The beneficial effects of the present invention are as follows: the present invention innovatively proposes a multimodal expression recognition method, which incorporates human body posture and scene information, makes multi-faceted judgments on emotions, and makes more detailed classifications of expressions, and adds a continuous dimension to judge the intensity of emotions.

[0014] This invention innovatively combines target detection methods with traditional convolutional networks to achieve real-time and continuous recognition of facial expressions.

[0015] This paper uses an improved object detection network. These improvements are based on YOLOv3 and integrate the multi-scale pooling method and the decoupled detection head of YOLOX to improve the performance of object detection tasks.

[0016] The present invention uses a multi-scale context information extraction module, a context attention module and a content attention module to achieve the purpose of improving the foreground weight of the image and increase the detection ratio of people in the image.

[0017] This paper utilizes a multi-feature facial expression recognition method based on deep learning. Compared to traditional methods that suffer from poor recognition rates, inadequate classification, simplistic recognition criteria, and insufficient datasets, this invention improves traditional facial expression recognition methods by incorporating auxiliary gesture and scene recognition, reducing its reliance on facial features while improving recognition accuracy. It also uses an expression classification method more consistent with medical classification, resulting in more detailed expression recognition results and avoiding the categorization of images with relatively small differences. The public dataset used, Emotic, is a large-scale dataset of images from various target detection tasks, encompassing complete human motion, facial, and environmental information. This allows network training to capture greater inter-class differences and learn features corresponding to subtle changes in the human body. These advantages make the overall framework applicable to medical diagnosis, providing highly accurate human expression changes. Furthermore, based on extensive data from psychology, neuroscience, and computer vision, a context extraction and detection method suitable for such sentiment analysis and psychological diagnosis has been designed, promoting further research on deep learning in psychological issues and human anatomy. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 A clear image of the human body of the present invention;

[0019] Figure 2 The three sizes of image feature maps are output after being processed by the feature coarse processing backbone network based on the deep convolutional network of the present invention, and their sizes are 80×80, 40×40, and 20×20 respectively;

[0020] Figure 3 This is a structural diagram of the feature coarse processing backbone network based on the deep convolutional network of the present invention;

[0021] Figure 4 This is a structural diagram of the feature extraction network based on the deep convolutional network of the present invention;

[0022] Figure 5 The heat map of the prominent weights of the image after the feature enhancement network based on deep void convolution and attention mechanism of the present invention includes the feature enhancement points of the human face and the body posture feature enhancement points;

[0023] Figure 6 This is a structural diagram of the feature enhancement network based on deep hole convolution and attention mechanism of the present invention;

[0024] Figure 7 This is a structural diagram of the multi-branch output module based on the deep convolutional network of the present invention;

[0025] Figure 8 This is a state diagram of the scene information recognition results after pre-training of the scene information recognition network based on the deep convolutional network of the present invention;

[0026] Figure 9 The feature fusion module of the present invention is Figure 1 The output diagram of the final recognition result. DETAILED DESCRIPTION

[0027] A multi-feature expression recognition method based on deep learning, such as Figures 1-9 As shown, the method includes the following steps: obtaining a clear image containing a human body and the environment and performing preprocessing, compressing the image to a primary color RGB image of 640*640 or smaller size, retaining the color channels, and using the public dataset Emotic expression recognition dataset;

[0028] The overall network consists of a target detection network in parallel with a scene information recognition network, and finally outputs it to a classification network. The target detection network consists of the backbone network CSPdarknet53, a feature pyramid structure for feature extraction in parallel with a feature enhancement network, and finally outputs the human body's position information and 0 / 1 confidence information, which is then sent to the classification network together with the scene information for classification and output of the final result.

[0029] Among them, the backbone network CSPdarknet53 is improved. CSPdarknet consists of multiple residual structures, keeping the output of the native network unchanged. Before the output of ResBlock_F3, ResBlock_F4, and ResBlock_F5, a spatial pyramid pooling that can collect multi-scale information is added to simplify network parameters and fuse multi-scale information; and three feature maps with sizes of 80*80, 40*40, and 20*20 are output in ResBlock_F3, ResBlock_F4, and ResBlock_F5, respectively, so that the feature extraction network can perform sampling and integration to realize the subsequent feature extraction process;

[0030] CSPdarknet53 is the backbone network of the object detection model YOLO. To address the characteristics of human feature extraction, the backbone network's loss function was modified to the mish function without increasing the overall number of parameters in the backbone network. A new multi-feature pooling branch was added to enhance the network's multi-scale perception capabilities. The input image is an RGB color image with three channels retained. Scaling, translation, and mosaic are used as data preprocessing methods. The pre-trained weights of CSPdarknet53 on the MS-COCO dataset are used as the initial weights, and hyperparameters are fine-tuned. The dataset is divided into three parts of different sizes: training set, test set, and validation set. The training weights with the smallest loss are finally selected as the training result.

[0031] Among them, the feature extraction network adopts a feature pyramid structure, and performs a bottom-up upsampling process on previously output feature maps of different sizes, which contain different semantic information and feature information; in the feature extraction network, upsampling is performed on the feature map with the largest size, and at the same time, a top-down fusion method is used for the output information, and the semantic information is propagated to the low-level features, so that the low-level features also contain rich semantic information; using the nearest neighbor interpolation method, the neighboring pixel value closest to the pixel to be calculated does not need to be calculated among the four neighboring pixels of the pixel to be calculated, and the pixel to be calculated is assigned to the pixel to be calculated.

[0032] The feature enhancement network uses dilated convolution and attention mechanisms to extract multi-scale, high-weight features of human targets, ensuring that more attention is paid to human features and the main body of the person is highlighted during network training.

[0033] Parallel multi-scale dilated convolution and positive sampling structure are used to further extract features from the feature map after channel regularization in the feature extraction network;

[0034] The dilated convolutions of multiple scales maximize the perception of regions of different sizes on the image. After parallel upsampling and fully connected layers, the original feature information is retained to the maximum extent.

[0035] The attention mechanism is a hybrid attention mechanism, and the parallel connection of multiple attention mechanisms will increase the partial weight of the face in the feature map of the deep void convolution part.

[0036] Among them, the feature enhancement network is a modular additional network, which is inserted at the output of the backbone network, the output of the feature pyramid structure and the sampling end of the feature pyramid structure. It includes a set of fully connected and up-sampled simple output networks and a set of multiple variable-scale void convolution layers. The size of the void convolution is 3×3, and the expansion rate is a variable value from 3 to 24. They are stacked to expand the features to learn variable geometric feature modeling capabilities.

[0037] Among them, the feature enhancement module is divided into a context information extraction module, a context attention module and a content attention module. After obtaining the feature map of step 4, the rich context information is used to input it into the context information extraction module. The context information extraction module consists of multi-path void convolutions with different expansion rates. The separated convolution layers obtain multiple feature maps in different receptive fields.

[0038] Among them, a deformable convolutional layer is introduced in each path, which ensures that the context information extraction module learns transformation-invariant features from the given data.

[0039] Among them, dense connections are used in the context information extraction module to merge multi-scale information, where the output of each expansion layer is connected to the input feature map and then sent to the next expansion layer; dense connections are used to solve the problem of gradient disappearance and enhance feature propagation when the deep learning model becomes deeper and deeper; the output of the void convolution group is connected to the upsampled input, the coarse-grained information of the initial input is maintained, and they are input into the 1×1 convolution layer to fuse coarse-grained and fine-grained features; a context attention module based on the self-attention mechanism is introduced to actively capture the semantic dependencies between sub-regions. Unlike the semantic segmentation of dual-path attention, the above-mentioned features containing multi-scale receptive field information generated by the context information extraction module are input into the context attention module; based on these information features, the context attention module adaptively pays more attention to the relationship between related sub-regions; the output features of the context attention module will have clear semantics and contain context dependencies in the surrounding objects. For a given feature image Transformed into the latent convolutional network space W q and W k , the converted feature map is:

[0040]

[0041] and Afterwards, Q and K are transformed into In this case, N = H × W. In order to capture the relationship between each sub-region, a correlation matrix is ​​calculated as:

[0042]

[0043] and At the same time, it is transformed into After normalizing R through sigmoid activation function and average pooling, an attention matrix R′ is established, and

[0044] At the same time, using the convolutional layer W v Convert the feature map F into another representation V:

[0045]

[0046] and Finally, we perform element-wise product of R′ and V matrix and get:

[0047] E i =R′⊙V i (4)

[0048] Among them E i For the i th feature maps, the channel dimension is C;

[0049] The content attention module is introduced to solve the problem of position offset caused by the complete destruction of the geometric characteristics of a given image, so as to maintain the precise position information of each object. Similar to the context attention module, it uses a convolutional layer to transform a given feature map. Instead of using the feature map F to generate the attention matrix, it uses the feature map Capture the exact location of each object,

[0050] Apply two convolutional layers W p and W z Get the attention matrix and transform F5 into latent space respectively:

[0051]

[0052] and Reshape the dimensions of P and Z into The correlation matrix similar to formula (2) is obtained:

[0053]

[0054] and After reshaping S into After normalizing S through the sigmoid activation function and average pooling, an attention matrix S′ is established, and Combine the extracted features V and perform dot product on the results:

[0055] D i =S′⊙V i (7)

[0056] and D i Indicates the i th The output feature map.

[0057] After the above network processing is completed, information about the human body position and confidence will be output. The multi-branch output module based on the deep convolutional network contains the final required confidence and coordinate information. Different loss functions are used in these two branches. The confidence output is a binary absolute confidence result. This branch uses the binary cross entropy loss function. The coordinates output by the network is a regression task that uses the difference between the predicted value and the true value. This branch uses the CIoU loss function to represent the offset of the prediction box.

[0058] The scene information network and object detection network are connected in parallel. The scene information network uses ResNet34 and ResNet50 as the main network. Pre-trained weights on the Places365CNN scene information dataset are used as initial weights, and hyperparameters are fine-tuned. The dataset is divided into three parts of different sizes: training set, test set, and validation set. The training weight with the lowest loss is selected as the final training result. This scene information helps analyze location attributes and categories. Different emotions have significant differences in different environments.

[0059] The final feature fusion module is composed of three fully connected networks of different lengths. The feature fusion module combines the features of the two feature extraction modules to estimate discrete emotion categories and continuous emotion dimensions. Both feature extraction modules are one-dimensional filtered CNNs. Each network consists of 16 convolutional layers, in which the one-dimensional kernel alternates between horizontal and vertical directions, and the two-dimensional kernel is used to effectively model 8 layers; a global average pooling layer is used to reduce the features of the last convolution layer; a batch normalization layer and a rectified linear unit are added after each convolution layer to speed up training and avoid internal covariance offset.

[0060] The above description is only a preferred specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any technician familiar with the technical field, within the technical scope disclosed in the present invention, who makes equivalent replacements or changes based on the technical solutions and concepts of the present invention, should be covered by the scope of protection of the present invention.

Claims

1. A multi-feature expression recognition method based on deep learning, characterized in that: The method includes the following steps: obtaining a clear image containing a human body and an environment and performing preprocessing, compressing the image to a primary color RGB image of 640*640 or smaller size, retaining the color channels, and using the public dataset Emotic expression recognition dataset; The overall network consists of a target detection network in parallel with a scene information recognition network, and finally outputs to a classification network. The target detection network consists of a backbone network CSPdarknet53, a feature pyramid structure for feature extraction in parallel with a feature enhancement network, and finally outputs the body's position information and 0 / 1 confidence information, which are then sent to the classification network together with the scene information for classification and output of the final result. The backbone network CSPdarknet53 is composed of multiple residual structures, keeping the output of the native network unchanged. Before the output of the third layer residual block, the fourth layer residual block and the fifth layer residual block, a spatial pyramid pooling for collecting multi-scale information is added. The spatial pyramid pooling simplifies the network parameters and fuses multi-scale information. The third layer residual block, the fourth layer residual block and the fifth layer residual block output three feature maps of sizes 80*80, 40*40 and 20*20 respectively, so that the feature extraction network performs sampling and integration to realize the subsequent feature extraction process; CSPdarknet53 is the backbone network of the object detection model YOLO. To address the characteristics of human feature extraction, the loss function of the backbone network is modified to the mish function while ensuring that the overall number of parameters of the backbone network does not increase. A new multi-feature pooling branch is added to enhance the network's multi-scale perception capabilities. The feature enhancement network uses dilated convolution and attention mechanism to extract multi-scale, high-weight features of human targets, ensuring that more attention is paid to human features and the main body of the person is highlighted during network training. Parallel multi-scale dilated convolution and positive sampling structure are used to further extract features from the feature map after channel regularization in the feature extraction network; The dilated convolutions of multiple scales maximize the perception of regions of different sizes on the image. After parallel upsampling and fully connected layers, the original feature information is retained to the maximum extent. The feature enhancement network is a modular additional network inserted at the output of the backbone network, the output of the feature pyramid structure, and the sampling end of the feature pyramid structure. It includes a set of fully connected and up-sampled simple output networks and a set of multiple variable-scale dilated convolution layers. The dilated convolution layers are all 3×3 in size and have a dilation rate of 3 to 24. They are stacked to expand features and learn variable geometric feature modeling capabilities. The feature enhancement is divided into a context information extraction module, a context attention module, and a content attention module. After obtaining the feature map extracted by the feature pyramid, the rich context information is used to input it into the context information extraction module. The context information extraction module consists of multi-path dilated convolutions with different dilation rates. The separated convolution layers obtain multiple feature maps in different receptive fields. Deformable convolutional layers are introduced in each path, which ensure that the context information extraction module learns translation-invariant features from the given data.

2. The multi-feature expression recognition method based on deep learning according to claim 1, wherein The input image is an RGB color image with three channels retained. Scaling, translation, and mosaic are used as data preprocessing methods. The pre-trained weights of CSPdarknet53 on the MS-COCO dataset are used as the initial weights, and the hyperparameters are fine-tuned. The dataset is divided into three parts of different sizes: training set, test set and validation set; finally, the training weight with the smallest loss is selected as the final training result.

3. The multi-feature expression recognition method based on deep learning according to claim 2, wherein The feature extraction network adopts a feature pyramid structure, and performs a bottom-up upsampling process on previously output feature maps of different sizes, which contain different semantic information and feature information. In the feature extraction network, upsampling is performed on the feature map with the largest size, and a top-down fusion method is used for the output information. The semantic information is propagated to the lower-level features, so that the lower-level features also contain rich semantic information. The nearest neighbor interpolation method is used, and the neighboring pixel value closest to the pixel to be calculated does not need to be calculated among the four neighboring pixels of the pixel to be calculated.

4. The multi-feature expression recognition method based on deep learning according to claim 1, wherein The context information extraction module uses dense connections to merge multi-scale information, where the output of each expansion layer is connected to the input feature map and then fed into the next expansion layer. Dense connections are used to address the problem of vanishing gradients and enhance feature propagation as the deep learning model gets deeper. The output of the dilated convolution group is connected to the upsampled input to maintain the coarse-grained information of the initial input and is fed into a 1×1 convolution layer to fuse coarse-grained and fine-grained features. A contextual attention module based on the self-attention mechanism is introduced to actively capture the semantic dependencies between sub-regions. Unlike the semantic segmentation of dual-path attention, the features containing multi-scale receptive field information generated by the contextual information extraction module are input into the contextual attention module. Based on these information features, the contextual attention module adaptively pays more attention to the relationship between related sub-regions; the output features of the contextual attention module will have clear semantics and contain contextual dependencies in the surrounding objects. Transformed into the latent convolutional network space W q and W k , the converted feature map is: and Afterwards, Q and K are transformed into At this time, N = H × W; in order to capture the relationship between each sub-region, a correlation matrix is ​​calculated as: and At the same time, it is transformed into After normalizing R through the sigmoid activation function and average pooling, an attention matrix R' is established, and At the same time, using the convolutional layer W v Convert the feature map F into another representation V: and Finally, perform element-wise product of R' and V matrices to obtain: E i =R'⊙V i (4) Among them E i For the i th feature maps, the channel dimension is C; The content attention module is introduced to solve the problem of position offset caused by the complete destruction of the geometric characteristics of a given image, so as to maintain the precise position information of each object. Similar to the context attention module, it uses a convolutional layer to transform a given feature map. Instead of using the feature map F to generate the attention matrix, it uses the feature map Capture the exact location of each object, Apply two convolutional layers W p and W z Get the attention matrix and transform F5 into latent space respectively: and Reshape the dimensions of P and Z into The correlation matrix similar to formula (2) is obtained: and After reshaping S into After normalizing S through the sigmoid activation function and average pooling, an attention matrix S′ is established, and Combine the extracted features V and perform dot product on the results: D i =S'⊙V i (7) and D i Indicates the i th The output feature map.

5. The multi-feature expression recognition method based on deep learning according to claim 4 is characterized in that, After the network processing is completed, it will output information about the human body position and confidence. The multi-branch output module based on the deep convolutional network contains the final required confidence and coordinate information. Different loss functions are used in these two branches. The confidence output is a binary absolute confidence result. The confidence branch uses the binary cross entropy loss function. The coordinates output by the network is a regression task that uses the difference between the predicted value and the true value. The coordinate information branch uses the CIoU loss function to represent the offset of the prediction box. The scene information network and the object detection network are connected in parallel. The scene information network uses ResNet34 and ResNet50 as the main network. The pre-trained weights on the Places 365CNN dataset are used as the initial weights, and the hyperparameters are fine-tuned. The dataset is divided into three parts of different sizes: training set, test set, and validation set; the training weight with the smallest loss is finally selected as the final training result; The final feature fusion module is composed of three fully connected networks of different lengths. The feature fusion module combines the features of the two feature extraction modules to estimate discrete emotion categories and continuous emotion dimensions. Both feature extraction modules are one-dimensional filtered CNNs. Each network consists of 16 convolutional layers, in which the one-dimensional kernel alternates between horizontal and vertical directions, and the two-dimensional kernel is used to effectively model 8 layers; a global average pooling layer is used to reduce the features of the last convolution layer; a batch normalization layer and a rectified linear unit are added after each convolution layer to speed up training and avoid internal covariance offset.

Images