An image emotion recognition method based on abstract relational scene graphs

Abstract

This invention discloses an image emotion recognition method based on an abstract relationship scene graph, comprising the following steps: constructing a new object and attribute detector and extracting object features and their corresponding attribute features from the image; inferring the intimacy and abstract relationship features between objects through object features and constructing an abstract relationship scene graph; constructing a graph neural network to infer the abstract relationship scene graph so that the features of each object include emotional factors; constructing a scene feature extractor to extract the scene features of the image and designing a progressive attention mechanism to fuse multiple object features to obtain a holistic object feature; stitching together the scene features and the holistic object feature of the image and feeding them into an emotion classifier to obtain the image emotion category. Compared with existing technologies, this invention utilizes the relationships between objects and the interaction between objects and the scene to identify image emotions, effectively narrowing the gap between low-level vision and high-level emotions, and improving the classification accuracy of image emotion recognition.

Description

Technical Field

[0001] This invention relates to the field of image emotion recognition, and in particular to an image emotion recognition method based on an abstract relational scene graph. Background Technology

[0002] With the widespread use of social networks, more and more people are conveying emotions through sharing images. To understand how emotions are perceived through visual stimuli, a growing number of researchers are focusing on image emotion recognition. Because emotion perception is a subjective and complex process, image emotion recognition has always been a highly challenging task. In recent years, researchers have introduced convolutional neural networks (CNNs) and attention mechanisms into image emotion recognition, resulting in significant improvements in emotion classification accuracy. Furthermore, improving image emotion recognition can also promote the development of other related fields, including stylized image description, image retrieval, and opinion mining.

[0003] Traditional image emotion recognition methods attempt to predict emotions by manually extracting features such as color, shape, and texture. However, manually extracted features cannot cover all important emotional factors. After more than 20 years of development, image emotion recognition has shifted from traditional methods to deep learning methods. Deep learning methods utilize CNNs for feature extraction, achieving significant breakthroughs. Early deep learning methods focused on how to enable CNNs to extract overall features from emotional images, achieving great progress compared to manual feature extraction, but neglecting important local regions expressing emotions, i.e., emotion regions. In recent years, by combining object detection with attention mechanisms, focusing on emotion regions, the performance of image emotion recognition has been further improved. Since multiple emotion regions in an image can evoke opposing emotions, existing methods that directly combine local and global features as image feature representations are insufficient to bridge the emotional gap between low-level visual features and high-level emotions.

[0004] Beyond computer vision, psychology researchers have also conducted extensive studies on how emotions are perceived through visual stimuli. Existing research has confirmed that emotions can be independently evoked by scenes and objects. Furthermore, studies have shown that humans perceive the world within scenes, and visual objects typically appear in rich environments and are often embedded in the context of other related objects. Clearly, the relationships between objects and the interactions between objects and scenes can all serve as emotional stimuli. Due to the diversity of visual objects contained in images and the complexity of the relationships between them, image emotion recognition faces the following main challenges:

[0005] 1. Emotional perception is a subjective process, meaning that different people evoke different emotions from the same image. How can we bridge the gap between low-level visual features and high-level emotions?

[0006] 2. Previous methods directly combined local and global features as image feature representations. How can we utilize the relationships between objects and the interactions between scenes and objects to identify image emotions?

[0007] 3. The classification of relationships between objects is complex and diverse. How can we deduce the abstract relationships between objects without any annotations on the relationships between them?

[0008] 4. In an image, multiple objects have different weights in conveying the image's emotional expression. How can we find the important emotional regions without weight labeling information? Summary of the Invention

[0009] The purpose of this invention is to provide an image emotion recognition method based on abstract relationship scene graphs, which aims to narrow the gap between low-level vision and high-level emotion by utilizing the relationships between objects and the interaction between objects and scenes, thereby improving the classification accuracy of image emotion recognition.

[0010] To achieve the above objectives, the present invention provides an image emotion recognition method based on an abstract relational scene graph, the method comprising the following steps:

[0011] S101 is based on Faster R-CNN and Bottom-Up Attention to construct a new object and attribute detector to extract the features of each object in the image and its corresponding attribute features.

[0012] S102 uses the characteristics of objects to infer the closeness and abstract relationship characteristics between objects, and constructs an abstract relationship scene graph;

[0013] S103 constructs a graph neural network to infer abstract relationship scene graphs, so that the features of each object include emotional factors;

[0014] S104 constructs a scene feature extractor to extract scene features from the image; then it designs a progressive attention mechanism to fuse the features of multiple objects into one feature, thus obtaining the overall object features of the image.

[0015] After the scene features and overall object features of the stitched image are obtained by S105, they are fed into the sentiment classifier to obtain the sentiment category of the image.

[0016] In the image emotion recognition method of the present invention, the specific method described in S101 for constructing a new object and attribute detector based on Faster R-CNN and Bottom-UpAttention to extract the features of each object in the image and its corresponding attribute features is as follows:

[0017] First, Faster R-CNN is used to extract features of each object. Then, based on the Bottom-Up Attention concept, non-maximum suppression is performed on the category of each object using an IoU threshold. After the above operations, the feature set of each object in the image is obtained. and their corresponding confidence levels and category labels ;

[0018] Then, the characteristics of each object Reorder the features according to confidence level in descending order. For convenience, the set of features of the sorted objects will still be denoted as... Its category label is still recorded as ;

[0019] The object category embedding is then added to the object's features and fed into an attribute classifier to identify the object's corresponding attributes and their confidence levels. This attribute classifier includes a fully connected layer and employs... An additional multi-class cross-entropy loss function is added as the activation function to train the embedding layer and attribute classifier. After training, the object category embedding is added to the object's features to obtain the attribute features of each object. .

[0020] In the image emotion recognition method of the present invention, the specific method of S102, which involves using the features of objects to infer the intimacy and abstract relationship features between objects and constructing an abstract relationship scene graph, is as follows:

[0021] First, define the abstract relational scene graph for each image as... , , These represent the sets of points and edges in the abstract relational scene graph, respectively.

[0022] It includes three types of nodes: object nodes, attribute nodes, and abstract relation nodes. Object features and attribute features are used as the values ​​of object nodes and attribute nodes, respectively. Abstract relation features between objects are inferred using object features, and these are used as the values ​​of abstract relation nodes. The set of abstract relation features is denoted as […]. , ;

[0023] There are two types of edges: one is the edge between an object node and an attribute node, and the other is the edge between an object node and an abstract relation node. There is no edge between an attribute node and an abstract relation node. The weight of the edge between an object node and an attribute node is set to 1, and then the object is calculated. With object intimacy between As the weight of the edge between an object node and an abstract relation node

[0024] Furthermore, the specific method for inferring abstract relationship features between objects using object features is as follows:

[0025] First, add the features of the two objects and then... Regularization as an abstract relational feature between them :

[0026]

[0027] Then, the abstract relational features are projected from the visual space to the emotional space using the following formula:

[0028]

[0029] in These are learnable weights. It is a learnable bias.

[0030] Furthermore, the computing object With object intimacy between The specific method is:

[0031] First, through two different embedding functions and Calculate the affinity between object i and object j. ;

[0032] Then use the sigmoid activation function to adjust the intimacy level. Mapped to the range of 0-1;

[0033]

[0034] in, , These are two different learnable embeddings. and These are learnable weights that can be learned through backpropagation;

[0035] Finally, we obtain the set of intimacy levels between the objects. .

[0036] In the image emotion recognition method of the present invention, the specific method for constructing a graph neural network reasoning abstract relationship scene graph in step S103, so that the features of each object include emotional factors, is as follows:

[0037] Using the following formula for the GCN layer:

[0038]

[0039] Update object characteristics ,in: Indicates GCN number The first layer The value of each object node, Indicates the first Layer objects With object The values ​​of the abstract relationship nodes between them; Nonlinear activation function , and It is GCN's first Learnable weights of the layer;

[0040] Then, update the abstract relationship features between objects based on the aforementioned method of inferring the features of objects. Then, based on the aforementioned calculation of intimacy... Method update .

[0041] In the image emotion recognition method of this invention, the specific method for constructing a scene feature extractor in step S104 to extract scene features of the image, and then designing a progressive attention mechanism to fuse the features of multiple objects into one feature to obtain the overall object features of the image is as follows:

[0042] First, ResNet-101 is used as a scene feature extractor to obtain the scene features of the image. Reuse Regularization regularizes the object features obtained from GCN inference to prevent overfitting;

[0043] Then, using the following two linear functions and The characteristics of each object and scene features Mapping them to the same space makes them as close as possible, in order to narrow the gap between them:

[0044]

[0045]

[0046]

[0047]

[0048] in and These are learnable weights. and It is a learnable bias. This is called embedded scene feature. This is called the set of features of the embedded object;

[0049] The scene can be considered as being gradually formed by the sequential addition of multiple objects; therefore, the attention weights of each object are calculated sequentially.

[0050]

[0051] After obtaining the attention weights for each object, the embedded scene features are updated according to the following formula. :

[0052]

[0053] Once all objects have been calculated;

[0054] Finally, the attention weight set of each object is used. Characteristics of the object The overall object features of the image are calculated according to the following formula. :

[0055] .

[0056] In the image emotion recognition method of the present invention, the specific method for obtaining the emotion category of the image by inputting the scene features and overall object features of the stitched image into the emotion classifier in step S105 is as follows:

[0057] First, the scene features of the image extracted using ResNet-101 are... And the overall object feature obtained by fusing multiple object features through a progressive attention mechanism The emotional features of the image are obtained by stitching together. :

[0058]

[0059] Then, the emotional features of the image Feed into the sentiment classifier:

[0060]

[0061] Where cl is the emotion category, and CL is the number of emotion categories. These are the learnable weights in the sentiment classifier, and finally, a multi-class cross-entropy loss function is added to train the entire model.

[0062] This invention improves the classification accuracy of image emotion recognition by leveraging relationships between objects and interactions between objects and scenes. First, it constructs object and attribute detectors to incorporate more semantic information into features. Then, without object relationship annotations, it uses object features to infer the intimacy and abstract relationship features between objects, constructing an abstract relationship scene graph. Next, it constructs an unsupervised graph neural network to infer the abstract relationship scene graph, ensuring that the features of each object include emotional factors. Finally, it designs a progressive attention mechanism using the interactions between objects and scenes to find important emotional regions. Attached Figure Description

[0063] Figure 1 This is a flowchart of the image emotion recognition method according to an embodiment of the present invention;

[0064] Figure 2 This is a schematic diagram of the algorithm structure of the image emotion recognition method according to an embodiment of the present invention;

[0065] Figure 3 This is a schematic diagram of the object and attribute detector structure according to an embodiment of the present invention;

[0066] Figure 4 This is a diagram showing the detection effect of the object and attribute detector according to an embodiment of the present invention;

[0067] Figure 5 This is a schematic diagram illustrating the process of constructing an abstract relational scene graph according to an embodiment of the present invention;

[0068] Figure 6 This is a schematic diagram of the progressive attention mechanism structure according to an embodiment of the present invention. Detailed Implementation

[0069] The present invention will be further described in detail below with reference to the embodiments and accompanying drawings, but this is not intended to limit the present invention.

[0070] Example

[0071] Please see Figures 1-6 An image emotion recognition method based on abstract relational scene graphs includes the following steps, detailed in [reference needed]. Figure 1-2 :

[0072] S101 is based on Faster R-CNN and Bottom-Up Attention, constructing a new object and attribute detector and extracting the features of each object in the image and its corresponding attribute features;

[0073] Object and property detectors such as Figure 3 As shown, the specific method is as follows:

[0074] First, Faster R-CNN is used to extract... Figure 3The features of each object in the image are analyzed, and non-maximum suppression is performed on the category of each object based on the Bottom-Up Attention idea using an IoU threshold. After the above operations, a set of features of each object in the image is obtained. and their corresponding confidence levels and category labels ,in , , , This is the dimension of the vector, and the IoU threshold is set to 0.3;

[0075] Then, the characteristics of each object Reorder the features according to confidence level in descending order. For convenience, the set of features of the sorted objects will still be denoted as... Its category label is still recorded as ;

[0076] The object category embedding is then added to the object's features and fed into the attribute classifier, which includes a fully connected layer and employs... As the activation function, a multi-class cross-entropy loss function is used to train the embedding layer and attribute classifier to identify the corresponding attributes of objects and their confidence scores. After training, the embedding of the object category is added to the object's features to obtain the attribute features of each object. ,in The object and attribute detectors were pre-trained on the Visual Genome dataset. The detection results of the object and attribute detectors are shown in the image below. Figure 4 As shown.

[0077] S102 uses the characteristics of objects to infer the closeness and abstract relationship characteristics between objects, and constructs an abstract relationship scene graph;

[0078] The process of constructing an abstract relationship scene graph is as follows: Figure 5 As shown, the specific method is as follows:

[0079] First, define the abstract relational scene graph for each image as... , , These represent the sets of points and edges in the abstract relational scene graph, respectively.

[0080] It includes three types of nodes: object nodes, attribute nodes, and abstract relation nodes. Object features and attribute features are used as the values ​​of object nodes and attribute nodes, respectively. Abstract relation features between objects are inferred using object features, and these are used as the values ​​of abstract relation nodes. The set of abstract relation features is denoted as […]. ,in ;

[0081] There are two types of edges: one is between object nodes and attribute nodes, and the other is between object nodes and abstract relation nodes. There are no edges between attribute nodes and abstract relation nodes. The weight of the edge between an object node and an attribute node is set to 1, and then the object... With object intimacy between As the weight of the edge between an object node and an abstract relation node.

[0082] Furthermore, the specific method for inferring abstract relationship features between objects using object features is as follows:

[0083] First, add the features of the two objects and then... Regularization as an abstract relational feature between them :

[0084]

[0085] Then, the following formula is used to abstract the relational features. Projecting from visual space to emotional space:

[0086]

[0087] in These are learnable weights. It is a learnable bias.

[0088] Furthermore, the computing object With object intimacy between The specific method is:

[0089] First, two different embedding functions are implemented using 1x1 convolution. and and calculate the object With object intimacy between ;

[0090] Then use the sigmoid activation function to adjust the intimacy level. Mapped to the range of 0-1;

[0091]

[0092] in, , These are two different learnable embeddings. and These are learnable weights that can be learned through backpropagation;

[0093] Finally, we obtain the set of intimacy levels between the objects. .

[0094] S103 constructs a graph neural network to infer abstract relationship scene graphs, so that the features of each object include emotional factors;

[0095] Specifically, using the following formula for the GCN layer...

[0096]

[0097] Update object characteristics ,in: Indicates GCN number The first layer The value of each object node, Indicates the first Layer objects With object The values ​​of the abstract relationship nodes between them; Nonlinear activation function , and It is GCN's first Learnable weights of the layer;

[0098] Then, update according to the method in step S102. and .

[0099] S104 constructs a scene feature extractor to extract scene features from the image; then a progressive attention mechanism is designed to fuse the features of multiple objects to obtain the overall object features of the image;

[0100] A schematic diagram of the progressive attention mechanism is shown below. Figure 6 As shown, the specific method is:

[0101] First, a ResNet-101 pre-trained on the ImageNet dataset is used as a scene feature extractor to obtain the scene features of the image. ,in ;

[0102] Reuse Regularization regularizes the object features O obtained from GCN inference to prevent overfitting;

[0103] Then, using the following two linear functions and The characteristics of each object and scene features Mapping them to the same space makes them as close as possible, in order to narrow the gap between them:

[0104]

[0105]

[0106]

[0107]

[0108] in and These are learnable weights. and It is a learnable bias. This is called embedded scene feature. This is called the set of features of the embedded object;

[0109] The scene can be considered as being gradually formed by the sequential addition of multiple objects; therefore, the attention weights of each object are calculated sequentially.

[0110]

[0111] After obtaining the attention weights for each object, the embedded scene features are updated according to the following formula. :

[0112]

[0113] Once all objects have been calculated;

[0114] Finally, the attention weight set of each object is used. With object characteristics The overall object features of the image are calculated according to the following formula. :

[0115]

[0116] in .

[0117] The scene features and overall object features of the stitched image from S105 are then fed into an emotion classifier to obtain the emotion category of the image.

[0118] Specifically, firstly, the scene features of the image extracted using ResNet-101 are... and the overall object features obtained in step S104 The emotional features of the image are obtained by stitching together. :

[0119]

[0120] in ;

[0121] Then, the emotional features of the image Feed into the sentiment classifier:

[0122]

[0123] Where cl is the emotion category, and CL is the number of emotion categories. These are the learnable weights in the sentiment classifier.

[0124] The effects of the method of the present invention will be further explained below with reference to the embodiments.

[0125] The image emotion recognition method based on abstract relational scene graphs proposed in this invention has the following test environment and experimental results:

[0126] 1) Test environment:

[0127] System environment: Ubuntu 16.04; Hardware environment: 256GB RAM, GPU: Tesla V100×4, CPU: 1.70GHz Intel(R) Xeon(R) E5-2609, Hard drive: 8TB;

[0128] 2) Experimental data:

[0129] To verify the effectiveness of this invention, three of the most widely used publicly available emotion image datasets in the field of image emotion recognition were used: FI, EmotionRoI, and Twitter I. Based on the number of samples contained in the datasets, they can be divided into large-scale and small-scale datasets, with only the FI dataset being a large-scale dataset and the others being small-scale datasets. Except for EmotionRoI, which has a specific partitioning, the FI dataset was randomly divided into 80% training set, 5% validation set, and 15% test set, and the Twitter I dataset was randomly divided into 80% training set and 20% test set. Furthermore, the FI dataset was divided into binary categories, with amusement, awe, contentment, and excitement classified as positive emotions, and anger, disgust, fear, and sadness as negative emotions. Images in EmotionRoI were labeled with six emotions: anger, disgust, fear, joy, sadness, and surprise; joy and surprise were classified as positive emotions, and the others as negative emotions. For the small-scale datasets, we first initialized the model with the parameters trained on the FI dataset, and then fine-tuned the model parameters using the small dataset.

[0130] 3) Implementation details:

[0131] We implemented the entire model using PyTorch and trained it using the Adam optimizer with weight decay of 0.001 and the cross-entropy loss function. The total number of training epochs was set to 30. At the start of training, the learning rate was initialized to 0.0001 and decayed to 0.1 times the current learning rate every 7 epochs.

[0132] 4) Comparison with other algorithms:

[0133] Experimental results show that the classification accuracy of this invention is superior to existing methods on three publicly available datasets, which fully demonstrates that combining concrete visual information with abstract relationships can improve the performance of image emotion recognition. This is due to the fact that this invention considers the influence of inter-object relationships and interactions on emotion.

[0134] Other methods:

[0135] [1]Zhang Hao, Xu Dan, Luo Gaifang, et al. Learning multi-level representations for affective image recognition [J]. Neural Computing andApplications, 2022, 34(16): 14107-14120;

[0136] [2]Yang Jufeng, She Dongyu, Sun Ming, et al. Visual sentimentprediction based on automatic discovery of affective regions [J]. IEEE Transon Multimedia, 2018, 20(9): 2513-2525;

[0137] [3]Xiong Haitao, Liu Qing, Song Shaoyi, et al. Region-basedconvolutional neural network using group sparse regularization for imagesentiment classification [J]. EURASIP Journal on Image and Video Processing, 2019, 2019(1): 1-9.

[0138] Table 1. Sentiment classification accuracy of this invention and other methods on public datasets.

[0139] .

[0140] In summary, the image emotion recognition method of the present invention, on the one hand, effectively narrows the gap between low-level vision and high-level emotion by utilizing the relationships between objects and the interaction between objects and the scene; on the other hand, it improves the classification accuracy of image emotion recognition by combining specific visual features and abstract relational features.

Claims

1. An image emotion recognition method based on abstract relational scene graphs, characterized in that, The method includes the following steps: S101 is based on Faster R-CNN and Bottom-Up Attention to construct a new object and attribute detector to extract the features of each object in the image and its corresponding attribute features. S102 uses the characteristics of objects to infer the closeness and abstract relationship characteristics between objects, and constructs an abstract relationship scene graph; S103 constructs a graph neural network to infer abstract relationship scene graphs, so that the features of each object include emotional factors; S104 constructs a scene feature extractor to extract scene features from the image; then it designs a progressive attention mechanism to fuse the features of multiple objects into one feature, thus obtaining the overall object features of the image. After the scene features and overall object features of the stitched image are obtained by S105, they are fed into the sentiment classifier to obtain the sentiment category of the image.

2. The image emotion recognition method as described in claim 1, characterized in that, The specific method described in S101 for constructing a new object and attribute detector based on Faster R-CNN and Bottom-Up Attention to extract the features of each object in the image and its corresponding attribute features is as follows: First, Faster R-CNN is used to extract features of each object. Then, based on the Bottom-Up Attention concept, non-maximum suppression is performed on the category of each object using an IoU threshold. After the above operations, a set of features of each object in the image is obtained. and their corresponding confidence levels and category labels ; Then, the characteristics of each object Reorder the features according to confidence level in descending order. For convenience, we will still denote the sorted set of object features as follows: Its category label is still recorded as ; The object category embedding is then added to the object's features and fed into an attribute classifier to identify the object's corresponding attributes and their confidence levels. This attribute classifier includes a fully connected layer and employs... As an activation function, an additional multi-class cross-entropy loss function is added to train the embedding layer and attribute classifier. After training, the object category embedding is added to the object's features to obtain the attribute features of each object. The set of attribute features is denoted as... .

3. The image emotion recognition method as described in claim 1, characterized in that, The specific method described in S102 for inferring the intimacy and abstract relationship features between objects using their characteristics, and for constructing an abstract relationship scene graph, is as follows: First, define the abstract relational scene graph for each image as... , , These represent the sets of points and edges in the abstract relational scene graph, respectively. It includes three types of nodes: object nodes, attribute nodes, and abstract relation nodes. Object features and attribute features are used as the values ​​of object nodes and attribute nodes, respectively. Abstract relation features between objects are inferred using object features, and these are used as the values ​​of abstract relation nodes. The set of abstract relation features is denoted as […]. ; There are two types of edges: one is the edge between an object node and an attribute node, and the other is the edge between an object node and an abstract relation node. There is no edge between an attribute node and an abstract relation node. The weight of the edge between an object node and an attribute node is set to 1, and then the object is calculated. With object intimacy between As the weight of the edge between an object node and an abstract relation node.

4. The image emotion recognition method as described in claim 3, characterized in that, The specific method for inferring abstract relationship features between objects using object features is as follows: First, add the features of the two objects and then... Regularization as an abstract relational feature between them : Then, the following formula is used to abstract the relational features. Projecting from visual space to emotional space: in These are learnable weights. It is a learnable bias.

5. The image emotion recognition method as described in claim 3, characterized in that, The computing object With object intimacy between The specific method is: First, through two different embedding functions and , computational object With object intimacy between ; Then use the sigmoid activation function to Mapped to the range of 0-1 in, , These are two different learnable embeddings. and These are learnable weights that can be learned through backpropagation; Finally, we obtain the set of intimacy levels between the objects. .

6. The image emotion recognition method as described in claim 5, characterized in that, The specific method described in S103 for constructing a graph neural network reasoning abstract relationship scene graph, so that the features of each object include emotional factors, is as follows: Using the following formula for GCN layers: Update object characteristics ,in: Indicates GCN number The first layer The value of each object node, Indicates the first Layer objects With object The values ​​of the abstract relationship nodes between them; Nonlinear activation function , and It is GCN's first Learnable weights of the layer; Then, update according to the method in step S102. and .

7. The image emotion recognition method as described in claim 1, characterized in that, The specific method described in S104 for constructing a scene feature extractor to extract scene features from an image, and then designing a progressive attention mechanism to fuse the features of multiple objects into a single feature to obtain the overall object features of the image, is as follows: First, ResNet-101 is used as a scene feature extractor to obtain the scene features of the image. Reuse Regularization regularizes the object features obtained from GCN inference to prevent overfitting; Then, using the following two linear functions and The characteristics of each object and scene features Map them to the same space so that they are as close as possible, in order to narrow the gap between them: in, and These are learnable weights. and It is a learnable bias. This is called embedded scene feature. This is called the set of features of the embedded object; The scene can be considered as being gradually formed by the sequential addition of multiple objects; therefore, the attention weights of each object are calculated sequentially. After obtaining the attention weights for each object, the embedded scene features are updated according to the following formula. : Once all objects have been calculated; Finally, the attention weight set of each object is used. Characteristics of the object The overall object features of the image are calculated according to the following formula. : 。 8. The image emotion recognition method as described in claim 7, characterized in that, The specific method for obtaining the sentiment category of an image by inputting the scene features and overall object features of the stitched image as described in S105 into a sentiment classifier is as follows: First, the scene features of the image extracted using ResNet-101 are... And the overall object feature obtained by fusing multiple object features through a progressive attention mechanism The emotional features of the image are obtained by stitching together. : Then, the emotional features of the image Feed into the sentiment classifier: Where cl is the emotion category, and CL is the number of emotion categories. These are the learnable weights in the sentiment classifier, and finally, a multi-class cross-entropy loss function is added to train the entire model.

Images