Visual emotion recognition method based on prototype learning

Abstract

The invention discloses a visual emotion recognition method based on prototype learning, and the method comprises the following steps: 1, extracting a prototype of a learnable cue word from multiple aspects, and constructing a clear text feature for each emotion category, thereby reducing the semantic difference; 2, capturing the most representative image features of each category, providing more robust and discriminative feature representation, and meanwhile, enabling samples to be more aggregated to reduce overfitting; 3, a label fuzziness problem is relieved based on a label smoothing algorithm of a prototype distance; 4, in a model training stage, comprehensively utilizing a text prototype, a visual prototype and label smoothing, constructing a multi-task learning framework, and training a text encoder, an image encoder and a classifier; in the inference stage, the trained image encoder and classifier are used for visual emotion recognition. According to the method, through joint learning of prototypes of text emotion expressions and visual features, discriminative emotion expressions are explored, so that the accuracy of visual emotion recognition is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision and relates to a visual emotion recognition method, in particular to a visual emotion recognition method based on prototype learning. Background Art

[0002] Visual Emotion Recognition (VER) aims to identify human emotions evoked by images and has received widespread attention in various applications, such as depression detection and opinion mining.

[0003] Existing visual emotion recognition methods mainly include the following categories: methods based on traditional visual models, such as AlexNet and ResNet, but these methods lack the guidance of textual emotional expression; methods based on visual language models (such as CLIP), which show significant potential in aligning text-image features, but mainly focus on constructing diverse prompt words, resulting in a cumbersome process and insufficient flexibility; methods based on fine-grained feature extraction, which are prone to overfitting to noise.

[0004] Due to the subjectivity and complexity of emotions, existing visual emotion recognition methods have the following problems: semantic gap: there is a large emotional gap between visual cues and emotional expressions; overfitting: the model is not robust enough due to unclear features in emotion category samples; label ambiguity: there is overlap between categories due to diverse emotional responses.

[0005] To solve the above problems, a new visual emotion recognition method is needed that can effectively narrow the semantic gap, reduce model overfitting, and alleviate label ambiguity. Summary of the Invention

[0006] The present invention provides a prototype learning-based visual emotion recognition method named ProtoEmotion (PoE), which explores discriminative emotion representation by jointly learning prototypes of textual emotion expression and visual features, thereby improving the accuracy of visual emotion recognition.

[0007] The purpose of the present invention is achieved through the following technical solutions:

[0008] A visual emotion recognition method based on prototype learning includes the following steps:

[0009] Step 1: Text prototype construction: By extracting prototypes of learnable prompt words from multiple aspects, we construct clear text features for each sentiment category to reduce semantic differences. The specific steps are as follows:

[0010] Step 1: Design three types of prompts: emotion prompts, valence prompts, and arousal prompts. Connect these three prompts into a unified prompt and obtain the corresponding text feature representation through the text encoder of the CLIP model.

[0011] Step 1 and 2: In order to provide more representative and robust feature representation, the exponential moving average (EMA) method is used to continuously update the text prompt prototype. The formula for updating the text prototype is:

[0012]

[0013] in, represents the updated k-th text prototype, represents the currently learned prototype, and α is the smoothing factor;

[0014] Step 13: Guide the alignment of image features with text prototypes, prompting the image encoder to extract more semantically rich emotional features;

[0015] Step 2: Visual Prototype Capture: Capture the most representative image features of each category, provide more robust and discriminative feature representations, and make samples more clustered to reduce overfitting. The specific steps are as follows:

[0016] Step 21: Obtain image feature representation through the image encoder of the CLIP model;

[0017] Step 22: Use the memory bank mechanism to gradually update and improve the visual prototype of each emotion category during the training process;

[0018] Steps 2 and 3: To build more accurate and robust category centers, the confidence of the classifier is used as a weight to guide visual prototype learning. For each emotion category, the classifier will give a confidence score to the sample. The higher the confidence score, the greater the contribution of the sample in building the category prototype.

[0019] Step 3: Use a label smoothing algorithm based on prototype distance to alleviate label ambiguity. The specific steps are as follows:

[0020] Step 3. Calculate the similarity between visual prototypes of different emotion categories and construct a similarity matrix;

[0021] Step 3.2: Set the confidence threshold for screening to ensure that the main emotion categories still have significant weights;

[0022] Step 3. Multiply the similarity matrix by the threshold weight matrix to obtain a smoothed label matrix for model training;

[0023] Step 3 and 4: Use the Kullback-Leibler divergence loss function to guide the model to learn the correlation between emotion categories, thereby improving the recognition ability of samples with blurred emotion boundaries;

[0024] Step 4: Training and inference:

[0025] Step 41: During the model training phase, a unified multi-task learning framework is constructed by comprehensively utilizing text prototypes, visual prototypes, and label smoothing to train the text encoder, image encoder, and classifier.

[0026] Step 4.2: In the inference phase, use the trained image encoder and classifier for visual emotion recognition.

[0027] Compared with the prior art, the present invention has the following advantages:

[0028] 1. Multifaceted learnable text prototypes narrow the semantic gap;

[0029] 2. Visual prototypes distinguish category features to reduce model overfitting;

[0030] 3. Label smoothing strategy reduces label ambiguity;

[0031] 4. Extensive experiments on multiple datasets demonstrate the effectiveness of the present invention, which improves the existing technology by 1.37% on the FI dataset and 1.52% on the EmotionROI dataset. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 Figure 2 shows the overall architecture of ProtoEmotion. The left side shows the text prototype part of multi-faceted learning, including the three prompt words [emotion], [valence], and [arousal] and their text features generated by the text encoder; the middle part shows the image input and its image features generated by the image encoder; the right side shows the prototype distance-based label smoothing component; the solid line indicates the path used in both training and inference stages, and the dotted line indicates the path used only in training. DETAILED DESCRIPTION

[0033] The technical solution of the present invention is further described below with reference to the accompanying drawings, but is not limited thereto. Any modification or equivalent replacement of the technical solution of the present invention that does not depart from the spirit and scope of the technical solution of the present invention should be included in the scope of protection of the present invention.

[0034] This paper presents a method for visual emotion recognition based on prototype learning. The main goal of this method is to address abstractions in emotional data by constructing clear category prototypes, thereby facilitating classification. Two prototypes, textual and visual, are explored: a learnable textual prototype guides text-image semantic alignment, while a visual prototype constructs emotional centers for explicit image features. Furthermore, based on the obtained visual prototype distance, a label smoothing strategy is proposed to improve the classification of images with overlapping emotional regions.

[0035] The present invention is based on the CLIP (Contrastive Language-Image Pre-Training) model for visual emotion recognition. CLIP is a popular image-text model. Its main components include an image encoder E I and text encoder E T , receiving image and text inputs respectively. In the visual emotion recognition task, CLIP first incorporates emotion labels into pre-designed text prompts. Considering that manual prompt design is time-consuming, the present invention adopts a multi-faceted learnable prompt prototype, which is both informative and computationally efficient.

[0036] like Figure 1 As shown, the visual emotion recognition method based on prototype learning of the present invention specifically includes the following steps:

[0037] Step 1: Text prototype learning:

[0038] Considering that emotional expression is described not only by emotion labels but also by valence (i.e., positive-negative polarity) and arousal (i.e., calmness-excitement level), the present invention generates a set of multi-dimensional prompt words to provide richer semantic information.

[0039] Step 1: Text prototype construction: By extracting prototypes of learnable prompt words from multiple aspects, clear text features are constructed for each sentiment category, thereby reducing semantic differences.

[0040] Specifically, the present invention designs three types of prompts: emotion prompts, valence prompts, and arousal prompts. The template for the valence prompt is "A photo seems to express a polarity, such as [valence]," and the template for the arousal prompt is "A photo seems to express a degree of arousal, such as [arousal]." These prompts contain learnable parameters and corresponding label word embeddings. In practical applications, the present invention combines these three prompts into a unified prompt and obtains the corresponding feature representation through the CLIP text encoder.

[0041] In order to provide a more representative and robust feature representation, the present invention uses the exponential moving average (EMA) method to continuously update the text prompt prototype. This method can smoothly integrate historical information and currently learned information, making the prototype more stable. The formula for updating the text prototype is:

[0042]

[0043] in, represents the updated k-th text prototype, represents the currently learned prototype, and α is a smoothing factor (usually set to 0.9). Since text features usually have more obvious feature expressions, the present invention guides the alignment of image features with text prototypes, prompting the image encoder to extract more semantically rich emotional features.

[0044] Step 2: Visual prototype learning:

[0045] To address the problem of unclear emotional image features that can lead to model overfitting, this paper proposes a visual prototype learning method. Unlike text prototypes, visual prototypes are learned directly from image data and can capture the most representative visual features of each emotion category.

[0046] Specifically, this invention uses a memory bank mechanism to gradually update and refine the visual prototype for each emotion category during training. To construct more accurate and robust category centers, this invention innovatively uses the classifier's confidence as a weight to guide prototype learning. For each emotion category, the classifier assigns a confidence score to each sample; samples with higher confidence scores contribute more to the construction of the category prototype.

[0047] This weighting ensures that the visual prototype is primarily determined by samples that best represent the emotion of the category, and is less susceptible to interference from samples with blurred boundaries. By gradually aligning image features with their corresponding visual prototypes, this method effectively reduces the model's overfitting to noise in the data while enhancing its ability to extract true emotional features.

[0048] Step 3: Prototype-based label smoothing:

[0049] A key challenge in emotion recognition is label ambiguity. Due to the subjective nature of emotion, a single image can often evoke multiple emotions rather than just a single one. Traditional methods use one-hot encoded labels for supervised learning, which cannot properly handle the overlap between emotion categories.

[0050] To address this issue, this paper proposes a label smoothing strategy based on prototype distance. The core idea is that if the visual prototypes of two emotion categories are close in feature space, they are likely to have semantic overlap. Based on this assumption, this paper calculates the similarity between visual prototypes and generates a soft label for each sample, replacing the traditional hard label.

[0051] The specific approach is to first calculate the similarity between visual prototypes of different emotion categories and construct a similarity matrix. Then, a confidence threshold (set to 0.2 in this paper) is set for screening to ensure that the main emotion categories still have significant weights. Finally, the similarity matrix is multiplied by the threshold weight matrix to obtain a smoothed label matrix for model training.

[0052] This label smoothing method is particularly suitable for samples with low confidence (i.e., samples that the classifier has difficulty making a clear judgment). It guides the model to learn the correlation between sentiment categories through the Kullback-Leibler divergence loss function, thereby improving the recognition ability of samples with blurred sentiment boundaries.

[0053] Step 4: Training and inference:

[0054] During the model training phase, this paper comprehensively utilizes the three core components (text prototypes, visual prototypes, and label smoothing) to construct a unified multi-task learning framework. The overall training loss function is the weighted sum of the alignment loss between image features and text prototypes, the alignment loss between image features and visual prototypes, and the classification loss with label smoothing.

[0055] This multi-task learning approach enables the model to simultaneously focus on semantic alignment, feature clustering, and label smoothing, thereby addressing the three major issues of semantic gaps, overfitting, and label ambiguity. Notably, during the inference phase, the present invention only uses the trained image encoder and classifier for visual emotion recognition, without requiring text prototypes. This results in fast inference speed and low computational resource requirements.

[0056] We conduct extensive experiments on four visual emotion recognition datasets: FI (containing 22,683 images classified into Mikels' eight emotion categories), EmotionROI (containing 1,980 images classified according to Ekman's six emotion categories), Twitter I and Twitter II (containing 1,269 and 603 images respectively, both classified into binary categories).

[0057] Experimental results demonstrate that our method achieves state-of-the-art results on all datasets. For the largest dataset, FI, our method significantly outperforms the existing state-of-the-art, achieving gains of 1.37% and 1.56% on eight categories and two classes, respectively. For smaller datasets, our method also outperforms existing CLIP-based work, particularly on the EmotionROI dataset, achieving gains of 1.52% and 1.51% on six categories and two classes, respectively.

Claims

1. A visual emotion recognition method based on prototype learning, characterized by The method comprises the following steps: Step 1: Text prototype construction: By extracting prototypes of learnable prompt words from multiple aspects, clear text features are constructed for each sentiment category to reduce semantic differences; Step 2: Visual prototype capture: Capture the most representative image features of each category, provide more robust and discriminative feature representation, and make samples more clustered to reduce overfitting; Step 3: Label smoothing algorithm based on prototype distance to alleviate label ambiguity problem; Step 4: Training and inference: Step 41: During the model training phase, a unified multi-task learning framework is constructed by comprehensively utilizing text prototypes, visual prototypes, and label smoothing to train the text encoder, image encoder, and classifier. Step 4.2: In the inference phase, use the trained image encoder and classifier for visual emotion recognition.

2. The visual emotion recognition method based on prototype learning according to claim 1 is characterized in that The specific steps of step one are as follows: Step 1: Design three types of prompts: emotion prompts, valence prompts, and arousal prompts. Connect these three prompts into a unified prompt and obtain the corresponding text feature representation through the text encoder of the CLIP model. Step 1 and 2: To provide more representative and robust feature representation, the exponential moving average method is used to continuously update the text prompt prototype; Step 1 and 3: Guide the image features to align with the text prototype, prompting the image encoder to extract more semantically rich emotional features.

3. The visual emotion recognition method based on prototype learning according to claim 2 is characterized in that In steps 1 and 2, the formula for updating the text prompt prototype is: in, represents the updated k-th text prototype, represents the currently learned prototype, and α is the smoothing factor.

4. The visual emotion recognition method based on prototype learning according to claim 1 is characterized in that The specific steps of step 2 are as follows: Step 21: Obtain image feature representation through the image encoder of the CLIP model; Step 22: Use the memory bank mechanism to gradually update and improve the visual prototype of each emotion category during the training process; Steps 2 and 3: In order to build a more accurate and robust category center, the confidence of the classifier is used as a weight to guide the learning of the visual prototype. For each emotion category, the classifier will give a confidence score to the sample. The higher the confidence score, the greater the contribution of the sample in constructing the category prototype.

5. The visual emotion recognition method based on prototype learning according to claim 1 is characterized in that The specific steps of step three are as follows: Step 3. Calculate the similarity between visual prototypes of different emotion categories and construct a similarity matrix; Step 3.2: Set the confidence threshold for screening to ensure that the main emotion categories still have significant weights; Step 3. Multiply the similarity matrix by the threshold weight matrix to obtain a smoothed label matrix for model training; Step 3 and 4: Use the Kullback-Leibler divergence loss function to guide the model to learn the correlation between sentiment categories, thereby improving the recognition ability of samples with blurred sentiment boundaries.

Images