A visual emotion recognition method integrating self-supervised learning and attention mechanism

Abstract

The present invention discloses a visual emotion recognition method integrating self-supervised learning and attention mechanism. The method comprises the following steps: constructing an emotion recognition model, including a backbone feature extraction network, an emotion recognition classifier and a rotation self-supervised classifier; constructing a self-supervised learning task; constructing an emotion classification task; respectively constructing a first loss function and a second loss function for optimizing the self-supervised learning task and the emotion classification task, and constructing a total loss function; optimizing the self-supervised learning task and the emotion classification task through the total loss function, and obtaining an optimized backbone feature extraction network and an emotion calculation classifier; obtaining an image to be recognized, inputting the optimized backbone feature extraction network and the emotion calculation classifier, and completing visual emotion recognition. The present invention fully considers the global and local features of the image, enhances the feature representation through the attention mechanism, and enriches the feature semantic information through the self-supervised auxiliary task, thereby improving the effect of visual emotion classification.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a visual emotion recognition method integrating self-supervised learning and attention mechanism. Background Art

[0002] With the rapid development of information technology and the revolutionary transformation of Internet media, the Internet has become an important way to exchange information in modern society. People tend to post images on social media to express their emotions and opinions. Sentiment analysis plays an indispensable role in many activities such as human perception, reasoning, decision-making, planning, creation and social interaction. Therefore, visual sentiment analysis has also attracted great attention from researchers. Understanding the emotions conveyed by such visual content will be beneficial to a wide range of applications, such as multimedia retrieval, advertising recommendation, etc. In recent years, great progress has been made in visual sentiment analysis tasks, which attempt to mine the emotions contained in images.

[0003] For visual sentiment analysis, how to extract effective discriminant features is a key factor. In early studies, manually extracted features were usually used, including element information such as color, composition, texture, low-level overall features, and high-level features such as adjective-noun pairs. In recent years, with the rapid development of deep networks, researchers have designed many deep learning models to explore emotional information in vision. The features extracted by deep models have stronger semantic representation capabilities and better performance than traditional manual features.

[0004] Initially used in machine translation, attention has become an important concept in the field of neural networks. In the field of artificial intelligence, attention has become an important component of neural network architecture and has a large number of applications in natural language processing, statistical learning, speech, and computer vision. The attention mechanism can be intuitively explained using the human visual mechanism. For example, the visual system tends to focus on part of the information in the image that assists judgment and ignores irrelevant information. Similarly, in problems involving language or vision, some parts of the input may be more helpful for decision making than others. For example, in translation and summarization tasks, only certain words in the input sequence may be relevant to predicting the next word.

[0005] Due to the subjectivity of human cognition, different regions and contents of an image may contribute differently to the arousal of emotions. Early studies often extracted global features from images, and therefore failed to delve into the possibility that different regions of an image may trigger different emotions. Some studies have adopted attention mechanisms to extract more effective local features, but most current studies still do not refine the response of each emotion to different regions or explore the potential relationship between each emotion and the emotion-perceiving region. Therefore, it is still challenging to effectively locate relevant image regions for each emotion and capture the dependencies between regions to aid visual sentiment analysis.

[0006] Self-supervised learning mainly uses auxiliary tasks to mine its own supervisory signals from large-scale unsupervised data, and trains the network through this constructed supervisory information, so that it can learn representations that are valuable for downstream tasks. For self-supervised learning, there are three main methods: based on contextual constraints, such as word order, relative positions of parts of the picture, etc.; based on temporal constraints, such as frame similarity, multiple perspectives of objects, etc.; based on contrast constraints, such as learning to encode the similarity or dissimilarity of two things.

[0007] In recent years, most studies on visual emotion have used attention mechanisms to locate emotion-stimulating areas to improve emotion classification results. However, the semantic information in the features obtained through deep learning networks is still not rich enough. Adding semantic labels to the dataset is time-consuming and labor-intensive, while it is much easier to only implement image emotion labels, which limits the research on visual emotion (WSCNet: Weakly Supervised Coupled Networks for Visual Sentiment Classification and Detection). Summary of the invention

[0008] The purpose of the present invention is to provide a visual emotion recognition method that integrates self-supervised learning and attention mechanism, so as to solve the problems mentioned in the above background technology that the semantic information of emotional features is limited in the existing deep learning-based image emotion classification method, and adding semantic labels is time-consuming and labor-intensive.

[0009] The study found that the self-supervised rotation classification task provides a powerful supervision signal for image semantic feature learning, enabling the feature extraction network to extract rich semantic information from the image. The present invention adds a self-supervised auxiliary task to the main task of sentiment classification, extracts image features through the same feature extraction network, and performs multi-task learning on it, so that the backbone network can extract richer semantic features such as the type, position, and posture of objects in the image under limited sentiment labels, and strengthens features based on channel and spatial attention mechanisms in the main task, solving the problem of limited semantic information of sentiment features under limited sentiment labels.

[0010] The purpose of the present invention is achieved by at least one of the following technical solutions.

[0011] A visual emotion recognition method integrating self-supervised learning and attention mechanism includes the following steps:

[0012] S1. Construct an emotion recognition model, including a backbone feature extraction network, an emotion recognition classifier, and a rotational self-supervised classifier;

[0013] S2, construct self-supervised learning tasks;

[0014] S3, construct sentiment classification task;

[0015] S4, constructing the first loss function and the second loss function for optimizing the self-supervised learning task and the sentiment classification task respectively, and constructing the total loss function;

[0016] S5. Optimize the self-supervised learning task and the sentiment classification task through the total loss function to obtain the optimized backbone feature extraction network and sentiment calculation classifier;

[0017] S6. Obtain the image to be recognized, input it into the optimized backbone feature extraction network and sentiment computing classifier, and complete visual emotion recognition.

[0018] Furthermore, in step S1, the construction of the backbone feature extraction network is as follows:

[0019] A trained convolutional neural network is obtained through pre-training, and the last two layers of the trained convolutional neural network, namely the global average pooling layer and the fully connected layer, are removed to obtain a backbone feature extraction network.

[0020] Furthermore, in step S2, a public original data set is obtained, and images in the original data set are expanded, including flipping and transposing operations, and a corresponding rotation angle label is added to each image obtained by the expansion operation to obtain an expanded data set;

[0021] The self-supervised learning task includes the following steps:

[0022] S2.1. Extract the first-level features f of the image samples in the expanded dataset through the backbone feature extraction network rot ;

[0023] S2.2. For the first level feature f rot Downsample and generate feature vector d rot , the feature vector d rot Input into the rotation self-supervised classifier to calculate the rotation angle of the image sample.

[0024] Furthermore, in step S2.2, the first-level feature f rot Through the global spatial pooling layer and the fully connected layer in turn, the feature vector d is generated rot , the feature vector d rot Input into the rotation self-supervised classifier, and calculate the rotation angle prediction probability of the input image sample through the softmax function The details are as follows:

[0025]

[0026] Where i = 1,…,N rot , N rot represents the total number of images in the expanded dataset, r represents the rth rotation label, represents the i-th image after rotation, It indicates the probability that the i-th image after rotation corresponds to the r-th rotation label.

[0027] Furthermore, in step S3, each image in the original data set is subjected to random rotation, cropping, and normalization preprocessing to obtain images of the same size, thereby obtaining a preprocessed data set;

[0028] The sentiment classification task includes the following steps:

[0029] S3.1. Extract the second-level features f of the image samples in the preprocessed dataset through the backbone feature extraction network 1 ;

[0030] S3.2. Attention to the second-level features f through space and channel 1 Perform reinforcement representation and obtain attention reinforcement characteristics;

[0031] S3.3. Combine the attention enhancement feature with the second-level feature f 1 Coupled fusion is performed to obtain discriminative features, which are then input into the emotion recognition classifier to calculate the emotion category.

[0032] Further, step S3.2 includes the following steps:

[0033] S3.2.1. Compress the input second-level features f through convolution operation 1 The spatial dimension of the third level feature f 2 ;

[0034] S3.2.2. For the third level feature f 2 Perform average pooling layer and maximum pooling layer operations on the spatial dimension to obtain the average pooling layer features on the spatial dimension. and the maximum pooling layer features in the spatial dimension The average pooling layer features in the spatial dimension and the maximum pooling layer features in the spatial dimension The channel attention map is generated by inputting it into the shared network, i.e., the multi-layer perceptron. The channel attention map after passing through the multi-layer perceptron is summed element by element to obtain the channel attention enhancement feature M1. The calculation formula is as follows:

[0035]

[0036] Among them, sigmoid is the activation function, MLP stands for multi-layer perceptron;

[0037] S3.2.3. For the third level feature f 2 Perform average pooling layer and maximum pooling layer operations on the channel dimension to obtain the average pooling layer features on the channel dimension. and the maximum pooling layer features in the channel dimension And the average pooling layer features obtained on the channel dimension and the maximum pooling layer features in the channel dimension Connect them to get the feature descriptor, and input the feature descriptor into a convolutional layer to get the spatial attention enhancement feature M2, which is calculated as follows:

[0038]

[0039] Among them, sigmoid is the activation function, and conv represents the convolution operation.

[0040] Further, step S3.3 includes the following steps:

[0041] S3.3.1. The third level feature f 2 , channel attention enhancement feature M1 and spatial attention enhancement feature M2 are multiplied to obtain the fourth level feature f 3 , and the third level feature f 2 In comparison, the fourth level feature f 3 Contains the location and validity information of the emotional activation areas;

[0042] S3.3.2. The second level feature f 1 Each feature map in is related to the fourth level feature f 3 Multiply element by element to get the coupling characteristic map u=[u1,u2,……,u n ], where n represents the number of channels of the feature, u n It represents the characteristics of the nth channel after coupling, and the formula is as follows:

[0043]

[0044] in, Represents element-wise multiplication;

[0045] S3.3.3, the second level feature f 1 It is connected to the coupled feature map u and input into the global average pooling layer to obtain the fifth level feature f 4 , has richer semantic information, the formula is as follows:

[0046] f 4 =GAP(f 1 ; u),

[0047] Among them, GAP represents the global average pooling layer operation;

[0048] S3.3.4, the fifth level feature f 4 The discriminative feature vector d is generated through the fully connected layer, and then the sentiment category prediction probability p(y k =c), where y k represents the kth picture in the preprocessed dataset, and c represents the cth sentiment label; the calculation formula is as follows:

[0049]

[0050] Among them, j = 1, 2, ..., m, m represents the total number of sentiment labels, d j represents the jth value of the discriminative feature vector d, d c represents the value in the discriminative feature vector d corresponding to the sentiment label c.

[0051] Furthermore, in step S4, the self-supervised learning task is optimized by minimizing the cross entropy loss function. The first loss function L rot The details are as follows:

[0052]

[0053] Among them, i represents the i-th picture in the expanded dataset, N rot represents the total number of images in the expanded dataset, r represents the rth rotation label, and R represents the total number of rotation labels. represents the i-th image after rotation, It indicates the probability that the i-th image after rotation corresponds to the r-th rotation label, It is 1 if the equation holds, and 0 otherwise.

[0054] Furthermore, in step S4, the sentiment classification task is optimized by minimizing the cross entropy loss function. The second loss function L cls The details are as follows:

[0055]

[0056] Among them, k represents the kth picture in the original data set, N represents the total number of pictures in the original data set, c represents the cth emotion label, C represents the total number of emotion labels, p(y k =c) represents the probability that the kth picture in the original data set corresponds to the cth emotion label; 1(y k =c) means it is 1 if the equation holds, otherwise it is 0.

[0057] Furthermore, in step S4, the first loss function L is combined rotand the second loss function L cls By optimizing the self-supervised learning task and the sentiment classification task at the same time, the optimized backbone feature extraction network can extract more semantically rich information through the self-supervised learning task, further improving the accuracy of sentiment classification; construct the total loss function L, and minimize the total loss function L through stochastic gradient descent, as follows:

[0058] L=L cls +γL rot ,

[0059] Among them, γ represents the first loss function L rot The constraint coefficient.

[0060] Compared with the prior art, the present invention has the following beneficial effects:

[0061] Image features are extracted through a deep residual network; enhanced features are processed through spatial and channel attention mechanisms and coupled with the original features so that the features contain overall and local information of the image; self-supervised auxiliary tasks are added so that the backbone network can accurately locate areas with rich semantic information in the image, reducing the label burden and making the extracted feature information more discriminative, thereby improving the effect of visual emotion classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 This is an overall structural diagram of the emotion recognition model in an embodiment of the present invention;

[0063] Figure 2 It is a structural diagram of the self-supervision module of the present invention;

[0064] Figure 3 This is a structural diagram of the attention module of the present invention;

[0065] Figure 4 This is a structural diagram of the sentiment classification module of the present invention. Specific implementation methods

[0066] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical methods in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0067] Embodiment 1:

[0068] A visual emotion recognition method that combines self-supervised learning and attention mechanism, such as Figure 1 As shown, the following steps are included:

[0069] S1. Construct an emotion recognition model, including a backbone feature extraction network, an emotion recognition classifier, and a rotational self-supervised classifier;

[0070] The construction of the backbone feature extraction network is as follows:

[0071] In this embodiment, a trained ResNet-101 network is obtained through pre-training, and the last two layers of the trained ResNet-101 network, namely the global average pooling layer and the fully connected layer, are removed to obtain a backbone feature extraction network.

[0072] S2, construct self-supervised learning tasks;

[0073] Obtain a publicly available original data set, expand the images in the original data set, including using flip and transpose operations, and add corresponding rotation angle labels to each image obtained through the expansion operation to obtain an expanded data set;

[0074] In this embodiment, for a 90-degree clockwise rotation, the image is first transposed and then flipped vertically (inverted); for a 180-degree clockwise rotation, the image is first flipped vertically and then flipped horizontally (left-right flipped); for a 270-degree clockwise rotation, the image is first flipped vertically and then transposed; each image in the data set is rotated clockwise by 0, 90, 180, and 270 degrees, respectively, and the image obtained by the i-th expansion operation is supplemented with the i-th label, which are 0, 1, 2, and 3, respectively.

[0075] The present invention uses the image rotation angle prediction, i.e., the self-supervised learning task, as an auxiliary task in the sentiment classification task. The study found that the self-supervised learning task enables the backbone feature extraction network to quickly and accurately locate the semantically rich areas of the image, and enables the backbone feature extraction network to learn the rich semantic features of the type, position, posture, etc. of the object in the image. The extracted features are more discriminative and have more transferable visual representations. Through self-supervision, the present invention can improve the ability of visual emotion recognition.

[0076] By adding a self-supervised learning task, the rotation angle of the image is identified, so that the features extracted by the backbone feature extraction network contain richer semantic information and the extracted features are more discriminative, so as to increase the sentiment classification accuracy of the sentiment classification task; the self-supervised learning task includes the following steps:

[0077] S2.1. Extract the first-level features f of the image samples in the expanded dataset through the backbone feature extraction network rot ;

[0078] S2.2. For the first level feature f rot Downsample and generate feature vector d rot , the feature vector d rotInput into the rotation self-supervised classifier to calculate the rotation angle of the image sample.

[0079] like Figure 2 As shown, the first level feature f rot Through the global spatial pooling layer and the fully connected layer in turn, the feature vector d is generated rot , the feature vector d rot Input into the rotation self-supervised classifier, and calculate the rotation angle prediction probability of the input image sample through the softmax function The details are as follows:

[0080]

[0081] Where i = 1,…,N rot , N rot represents the total number of images in the expanded dataset, r represents the rth rotation label, represents the i-th image after rotation, It indicates the probability that the i-th image after rotation corresponds to the r-th rotation label.

[0082] S3, construct sentiment classification task;

[0083] Perform random rotation, cropping, and normalization preprocessing on each image in the original data set to obtain images of the same size, which are adjusted to 448×448×3 to obtain a preprocessed data set;

[0084] In this embodiment, the size of the expanded data set is four times that of the preprocessed data set.

[0085] In order to obtain more discriminative image features, the image is first input into the backbone feature extraction network to extract the features of the image, and then the features are input into the sentiment classification task based on the attention mechanism. The enhanced features are obtained through the spatial attention mechanism and the channel attention mechanism, so that the enhanced features contain the information of the emotion-stimulating area, can focus on the area in the image that can highlight the emotion more, and select more important channel features; the sentiment classification task includes the following steps:

[0086] S3.1. Extract the second-level features f of the image samples in the preprocessed dataset through the backbone feature extraction network 1 ;

[0087] S3.2, such as Figure 3 As shown, the second-level feature f is 1 Performing enhanced representation to obtain attention enhancement features includes the following steps:

[0088] S3.2.1. Compress the input second-level features f through convolution operation 1The spatial dimension of the third level feature f 2 ;

[0089] S3.2.2. For the third level feature f 2 Perform average pooling layer and maximum pooling layer operations on the spatial dimension to obtain the average pooling layer features on the spatial dimension. and the maximum pooling layer features in the spatial dimension The average pooling layer features in the spatial dimension and the maximum pooling layer features in the spatial dimension The channel attention map is generated by inputting it into the shared network, i.e., the multi-layer perceptron. The channel attention map after passing through the multi-layer perceptron is summed element by element to obtain the channel attention enhancement feature M1. The calculation formula is as follows:

[0090]

[0091] Among them, sigmoid is the sigmoid activation function, MLP means multi-layer perceptron;

[0092] S3.2.3. For the third level feature f 2 Perform average pooling layer and maximum pooling layer operations on the channel dimension to obtain the average pooling layer features on the channel dimension. and the maximum pooling layer features in the channel dimension And the average pooling layer features obtained on the channel dimension and the maximum pooling layer features in the channel dimension Connect them to get the feature descriptor, and input the feature descriptor into a convolutional layer to get the spatial attention enhancement feature M2, which is calculated as follows:

[0093]

[0094] Among them, sigmoid is the sigmoid activation function, and conv represents the convolution operation.

[0095] S3.3, if Figure 3 and Figure 4 As shown, the attention enhancement feature is combined with the second-level feature f 1 Coupled fusion is performed to obtain discriminative features, which are then input into the emotion recognition classifier to calculate the emotion category, including the following steps:

[0096] S3.3.1. The third level feature f 2 , channel attention enhancement feature M1 and spatial attention enhancement feature M2 are multiplied to obtain the fourth level feature f 3 , and the third level feature f 2 In comparison, the fourth level feature f3 Contains the location and validity information of the emotional activation areas;

[0097] S3.3.2. The second level feature f 1 Each feature map in is related to the fourth level feature f 3 Multiply element by element to get the coupling characteristic map u=[u1,u2,……,u n ], where n represents the number of channels of the feature, u n It represents the characteristics of the nth channel after coupling, and the formula is as follows:

[0098]

[0099] in, Represents element-wise multiplication;

[0100] S3.3.3, the second level feature f 1 It is connected to the coupled feature map u and input into the global average pooling layer to obtain the fifth level feature f 4 , has richer semantic information, the formula is as follows:

[0101] f 4 =GAP(f 1 ; u),

[0102] Among them, GAP represents the global average pooling layer operation;

[0103] S3.3.4, the fifth level feature f 4 The discriminative feature vector d is generated through the fully connected layer, and then the sentiment category prediction probability p(y k =c), where y k represents the kth picture in the dataset, and c represents the cth sentiment label; the calculation formula is as follows:

[0104]

[0105] Among them, j = 1, 2, ..., m, m represents the total number of sentiment labels, d j represents the jth value of the discriminative feature vector d, d c represents the value in the discriminative feature vector d corresponding to the sentiment label c.

[0106] S4, constructing the first loss function and the second loss function for optimizing the self-supervised learning task and the sentiment classification task respectively, and constructing the total loss function;

[0107] The self-supervised learning task is optimized by minimizing the cross entropy loss function. The first loss function L rot The details are as follows:

[0108]

[0109] Among them, i represents the i-th picture in the expanded dataset, N rot represents the total number of images in the expanded dataset, r represents the rth rotation label, and R represents the total number of rotation labels. represents the i-th image after rotation, It indicates the probability that the i-th image after rotation corresponds to the r-th rotation label. It is 1 if the equation holds, and 0 otherwise.

[0110] The sentiment classification task is optimized by minimizing the cross entropy loss function. The second loss function L cls The details are as follows:

[0111]

[0112] Among them, k represents the kth picture in the original data set, N represents the total number of pictures in the original data set, c represents the cth emotion label, C represents the total number of emotion labels, p(y k =c) represents the probability that the kth picture in the original data set corresponds to the cth emotion label; 1(y k =c) means it is 1 if the equation holds, otherwise it is 0.

[0113] Combined first loss function L rot and the second loss function L cls By optimizing the self-supervised learning task and the sentiment classification task at the same time, the optimized backbone feature extraction network can extract more semantically rich information through the self-supervised learning task, further improving the accuracy of sentiment classification; construct the total loss function L, and minimize the total loss function L through stochastic gradient descent, as follows:

[0114] L=L cls +γL rot ,

[0115] Among them, γ represents the first loss function L rot The constraint coefficient.

[0116] S5. Optimize the self-supervised learning task and the sentiment classification task through the total loss function to obtain the optimized backbone feature extraction network and sentiment calculation classifier;

[0117] S6. Obtain the image to be recognized, input it into the optimized backbone feature extraction network and sentiment computing classifier, and complete visual emotion recognition.

[0118] In this embodiment, an evaluation is performed on a Twitter I dataset to demonstrate the effectiveness of the present invention; the dataset is randomly divided into a training set of 80% and a test set of 20%.

[0119] In this embodiment, the feature extraction network is a pre-trained ResNet-101, the input batch-size is set to 4, the stochastic gradient descent algorithm is used for optimization in the model, the weight decay is set to 0.0005, the learning rate of the feature extraction network is set to 0.0001, the learning rate of the rotation classification task and the sentiment classification task is set to 0.01, and γ is set to 1.

[0120] In this embodiment, in order to prove the effectiveness of the multiple methods proposed in the present invention, a variety of different image sentiment classification methods are compared on the same data set. The main comparison methods are described as follows:

[0121] SPN comes from reference 1 (see: Y. Zhu, Y. Zhou, Q. Ye, Q. Qiu, and J. Jiao, "Soft proposal networks for weakly supervised object localization," in Proc. Int. Conf. Comput. Vis., 2017, pp. 1859–1868.);

[0122] WILDCAT comes from literature 2 (see: T.Durand, T.Mordan, N.Thome, and M.Cord, "Wildcat: Weakly supervised learning of deep ConvNets for image classification, pointwise localization and segmentation," in Proc.IEEE Conf.Comput.Vis.PatternRecognit, 2017, pp.5957–5966.);

[0123] CAM_Res101 comes from reference 3 (see: B.Zhou, A.Khosla, Lapedriza, A. Oliva, and A. Torralba, “Learning deep features for discriminative localization,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit, 2016, pp. 2921–2929.);

[0124] WSCNet comes from literature 4 (for details, see: D.She, J.Yang, M.-M.Cheng, Y.-K.Lai, PLRosin andL.Wang, "WSCNet: Weakly Supervised Coupled Networks for Visual SentimentClassification and Detection," in IEEE Transactions on Multimedia, vol.22, no.5, pp.1358-1371, May 2020,doi:10.1109 / TMM.2019.2939744.).

[0125] Embodiment 2:

[0126] This embodiment is different from Embodiment 1 in that an evaluation is performed on the Twitter II dataset to demonstrate the effectiveness of the present invention.

[0127] Embodiment 3:

[0128] This embodiment is different from Embodiment 1 in that an evaluation is performed on the EmotionROI dataset to demonstrate the effectiveness of the present invention.

[0129] Table 1 shows the classification results of the present invention and other comparison methods on the emotional image datasets Twitter I, Twitter II and EmotionROI. The classification results are evaluated by accuracy, which is defined as the ratio of the number of correctly classified samples to the total number of samples.

[0130] As shown in Table 1, the classification accuracy of the present invention on Twitter I is 84.98%, the classification accuracy on Twitter II is 83.19%, and the classification accuracy on EmotionROI is 59.43%. Compared with other models, the classification accuracy of the present invention on these three data sets is improved, which shows that under limited labels, the present invention can discover more semantically rich information and extract more discriminative features, thereby improving the image sentiment classification effect.

[0131] Table 1 Classification accuracy of different methods on Twitter I, Twitter II and EmotionROI datasets

[0132]

[0133] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be equivalent replacement methods and are included in the protection scope of the present invention.

Claims

1. A visual emotion recognition method integrating self-supervised learning and attention mechanism, characterized in that: The following steps are involved: S1. Construct an emotion recognition model, including a backbone feature extraction network, an emotion recognition classifier, and a rotational self-supervised classifier; S2, construct self-supervised learning tasks; Obtain a publicly available original data set, expand the images in the original data set, including flipping and transposing operations, and add corresponding rotation angle labels to each image obtained through the expansion operation to obtain an expanded data set; The self-supervised learning task includes the following steps: S2.

1. Extract the first-level features f of the image samples in the expanded dataset through the backbone feature extraction network rot ; S2.

2. For the first level feature f rot Downsample and generate feature vector d rot , the feature vector d rot Input into the rotation self-supervised classifier to calculate the rotation angle of the image sample; The first level feature f rot Through the global spatial pooling layer and the fully connected layer in turn, the feature vector d is generated rot , the feature vector d rot Input into the rotation self-supervised classifier, and calculate the rotation angle prediction probability of the input image sample through the softmax function The details are as follows: Where i = 1, ..., N rot , N rot represents the total number of images in the expanded dataset, r represents the rth rotation label, represents the i-th image after rotation, Indicates the probability that the i-th image after rotation corresponds to the r-th rotation label; S3, construct sentiment classification task; Performing random rotation, cropping, and normalization preprocessing on each image in the original data set to obtain images of the same size, thereby obtaining a preprocessed data set; The sentiment classification task includes the following steps: S3.

1. Extract the second-level features f of the image samples in the preprocessed dataset through the backbone feature extraction network 1 ; S3.

2. Attention to the second-level features f through space and channel 1 Performing enhanced representation to obtain attention enhancement features includes the following steps: S3.2.

1. Compress the input second-level features f through convolution operation 1 The spatial dimension of the third level feature f 2 ; S3.2.

2. For the third level feature f 2 Perform average pooling layer and maximum pooling layer operations on the spatial dimension to obtain the average pooling layer features on the spatial dimension. and the maximum pooling layer features in the spatial dimension The average pooling layer features in the spatial dimension and the maximum pooling layer features in the spatial dimension The channel attention map is generated by inputting it into the shared network, i.e., the multi-layer perceptron. The channel attention map after passing through the multi-layer perceptron is summed element by element to obtain the channel attention enhancement feature M1. The calculation formula is as follows: Among them, sigmoid is the activation function, MLP stands for multi-layer perceptron; S3.2.

3. For the third level feature f 2 Perform average pooling layer and maximum pooling layer operations on the channel dimension to obtain the average pooling layer features on the channel dimension. and the maximum pooling layer features in the channel dimension And the average pooling layer features obtained on the channel dimension and the maximum pooling layer features in the channel dimension Connect them to get the feature descriptor, and input the feature descriptor into a convolutional layer to get the spatial attention enhancement feature M2, which is calculated as follows: Among them, sigmoid is the activation function, and conv represents the convolution operation; S3.

3. Combine the attention enhancement feature with the second-level feature f 1 Coupled fusion is performed to obtain discriminative features, which are then input into the emotion recognition classifier to calculate the emotion category, including the following steps: S3.3.

1. The third level feature f 2 , channel attention enhancement feature M1 and spatial attention enhancement feature M2 are multiplied to obtain the fourth level feature f 3 ; S3.3.

2. The second level feature f 1 Each feature map in is related to the fourth level feature f 3 Multiply element by element to get the coupling characteristic map u = [u1, u2, ...u i ...,u n ], where n represents the number of channels of the feature, u n It represents the characteristics of the nth channel after coupling, and the formula is as follows: in, Represents element-wise multiplication; S3.3.3, the second level feature f 1 It is connected to the coupled feature map u and input into the global average pooling layer to obtain the fifth level feature f 4 , the formula is as follows: f 4 =GAP(f 1 ;u), Among them, GAP represents the global average pooling layer operation; S3.3.4, the fifth level feature f 4 The discriminative feature vector d is generated through the fully connected layer, and then the sentiment category prediction probability p(y k =c), where y k represents the kth picture in the preprocessed dataset, and c represents the cth sentiment label; the calculation formula is as follows: Where j = 1, 2, ..., m, m represents the total number of sentiment labels, d j represents the jth value of the discriminative feature vector d, d c represents the value corresponding to the sentiment label c in the discriminative feature vector d; S4, constructing the first loss function and the second loss function for optimizing the self-supervised learning task and the sentiment classification task respectively, and constructing the total loss function; The self-supervised learning task is optimized by minimizing the cross entropy loss function. The first loss function L rot The details are as follows: Among them, i represents the i-th picture in the expanded dataset, N rot represents the total number of images in the expanded dataset, r represents the rth rotation label, and R represents the total number of rotation labels. represents the i-th image after rotation, It indicates the probability that the i-th image after rotation corresponds to the r-th rotation label, It is 1 when the equation is true, otherwise it is 0; S5. Optimize the self-supervised learning task and the sentiment classification task through the total loss function to obtain the optimized backbone feature extraction network and sentiment calculation classifier; S6. Obtain the image to be recognized, input it into the optimized backbone feature extraction network and sentiment computing classifier, and complete visual emotion recognition.

2. A visual emotion recognition method integrating self-supervised learning and attention mechanism according to claim 1, characterized in that: In step S1, the construction of the backbone feature extraction network is as follows: A trained convolutional neural network is obtained through pre-training, and the last two layers of the trained convolutional neural network, namely the global average pooling layer and the fully connected layer, are removed to obtain a backbone feature extraction network.

3. The visual emotion recognition method integrating self-supervised learning and attention mechanism according to claim 1, characterized in that: In step S4, the sentiment classification task is optimized by minimizing the cross entropy loss function. The second loss function L cls The details are as follows: Among them, k represents the kth picture in the original data set, N represents the total number of pictures in the original data set, c represents the cth emotion label, C represents the total number of emotion labels, p(y k =c) represents the probability that the kth picture in the original data set corresponds to the cth emotion label; I(y k =c) means it is 1 if the equation holds, otherwise it is 0.

4. The visual emotion recognition method integrating self-supervised learning and attention mechanism according to claim 3 is characterized in that: In step S4, the first loss function L is combined rot and the second loss function L cls By optimizing the self-supervised learning task and the sentiment classification task at the same time, the optimized backbone feature extraction network can extract more semantically rich information through the self-supervised learning task, further improving the accuracy of sentiment classification; construct the total loss function L, and minimize the total loss function L through stochastic gradient descent, as follows: L=L cls +γL rot , Among them, γ represents the first loss function L rot The constraint coefficient.

Images