An image sentiment classification method based on feature complementation under sentiment gain

Abstract

The invention relates to the field of computer vision technology, and discloses an image emotion classification method with feature complementation under emotion gain, image preprocessing, dividing an image data set FI into a training set A1 and a test set A2, randomly scaling and cropping the images of the training set A1, and finally obtaining an image set B with a size of 448×448×3; and the image emotion classification method, in order to enhance the network's ability to mine image emotion features, adopts a visual attention network as a backbone network, and introduces an emotional visual attention mechanism to improve the model's sensitivity to subtle emotional differences in images, in order to obtain significant features and latent features in the image, on the one hand, enhances the significant area of ​​the image to obtain the significant features, and on the other hand, suppresses the significant area to mine the latent features in the non-significant area of ​​the image, and finally combines the significant features and the latent features for image emotion classification.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision, and in particular to an image emotion classification method with feature complementation under emotion gain. Background Art

[0002] With the rapid development of the Internet, more and more images are used to convey information and express emotions. Research and analysis of these images to identify the emotions contained in them have been widely used in business, education, medical and other fields, such as public opinion monitoring, opinion mining, business intelligence, entertainment assistance, mental health and behavior assessment. Therefore, image emotion recognition has become one of the focuses of academic and business circles. Existing studies have shown that the mapping process from the underlying visual features of images to high-level human emotional expressions requires the construction of mapping relationships in more dimensions. Therefore, how to deeply explore the local and potential features of images to improve the accuracy of image emotion recognition is the current challenge.

[0003] Most existing technologies use deep learning methods to perform image sentiment analysis, but existing image sentiment classification still does not fully mine contextual information in images, and focuses on significant features while ignoring potential features, making it more difficult to capture tiny details and local features of images. ViT (Vision Transformer) came into being. ViT is a classification model based on the Transformer architecture. Compared with the traditional CNN network model, it can achieve global perception through the Transformer's self-attention mechanism, and can take into account all position information in the image. Since the subject of the image awakening emotion is people, in addition to the significant features that can be perceived in the image, those difficult-to-find details will also awaken emotions, such as artistic principles in works of art, composition in photographs, and colors in paintings. At present, the ViT network is mainly used in fine-grained image classification, and its application in the field of image sentiment classification is still relatively small. Since the image block information embedding of the ViT network lacks emotional information, it cannot exert the maximum potential of the network when performing image sentiment classification tasks. Therefore, an image sentiment classification method with feature complementation under emotional gain is proposed to solve the above problems. Summary of the invention

[0004] In view of the shortcomings of the prior art, the present invention provides an image sentiment classification method with feature complementation under sentiment gain, which has the advantages of high classification accuracy and high prediction precision, and solves the problems mentioned in the above background technology.

[0005] In order to achieve the above-mentioned purpose of high classification accuracy and high prediction precision, the present invention provides the following technical solution: an image emotion classification method with feature complementation under emotion gain, comprising the following steps:

[0006] S1: Image preprocessing, dividing the image dataset FI into a training set A 1 and test set A 2 , the training set A 1 The images are randomly scaled and cropped, and finally an image set B of size 448×448×3 is obtained;

[0007] S2: Divide the image set B of size 448×448×3 into 16×16 image blocks, and extract the context and global information features of the 16×16 image blocks through the vit network model , the emotional gain feature X is obtained by the emotional gain method temo , context and global information features With emotional gain feature X temo Fusion obtains image feature X 1 ;

[0008] S3: Image feature X 1 Obtain the salient feature X through the CBR convolution block E And the potential feature X is obtained by the potential feature mining method S , and then connect the salient features to L Transformer encoding blocks to obtain the final salient feature X E1 and L potential features Finally, these two features are used to mine the emotional features of the image in a complementary form to obtain the final emotional feature X FIN ;

[0009] S4: The obtained emotional feature X FIN Input into a multi-layer perceptron classification head, the labeled image set B trains the multi-layer perceptron classification head, and the test set A 2 After the test image is processed by fixed-size scaling and center cropping, it is input into the trained vit network model to obtain the corresponding emotion category;

[0010] S5: Take the image dataset Emotion6 and repeat steps S1-S4 to obtain the corresponding emotion category.

[0011] Preferably, the image data set FI includes eight emotions, which are pleasure, awe, satisfaction, excitement, anger, disgust, fear, and sadness. The image data set Emotion6 includes happiness, sadness, surprise, anger, disgust, and fear.

[0012] Preferably, the specific steps in step S2 are:

[0013] S2.1: Given an input image of size 448×448×3, split it into multiple non-overlapping image blocks Pi , the size of each image block is 16×16, let N be the total number of image blocks, and each image block P i Represented as a vector vec(P i ), for each image block P i , flatten it into a vector of dimension M, for each vectorized image block vec(P i ) Add location embed E pos,i , and then introduce a category embedding E cls , and finally obtain the context of the 16×16 image and the global information feature X P , the expression is:

[0014]

[0015] Among them, vec(P i ) is each vectorized image block, E pos,i is the position embedding, E cls for category embedding;

[0016] S2.2.1: Use ResNet50 network to capture the spatial features X of image set B res , the expression is:

[0017]

[0018] Among them, B i is one of the images in the image set B, with a size of W×H×C, where W is the width, H is the height, and C is the number of channels;

[0019] S2.2.2: Using emotional visual attention mechanism to perceive spatial features X res Emotional characteristics of X emo , the expression is:

[0020]

[0021] S2.2.3: Enhance the spatial features X using a CBR convolutional block consisting of a 1×1 convolution, a BN layer, and a ReLU activation function res , and obtain the enhanced spatial feature X cbn1 , the expression is:

[0022]

[0023] Among them, X cbn1 is the enhanced spatial feature, K is the convolution kernel size, Conv represents the convolution operation, BN is the batch normalization operation, and Relu is the activation function;

[0024] S2.2.4: Transform the emotional feature Xemo With the enhanced spatial feature X cbn1 Fusion gets the emotional gain feature X temo , the expression is:

[0025]

[0026] S2.3: Integrate context and global information features With emotional gain feature X temo Fusion obtains image feature X 1 , the expression is:

[0027]

[0028] Preferably, the specific steps in step S3 are:

[0029] S3.1: Input image feature X 1 Through a 1×1 convolution, a BN layer and a Relu activation function, the emotional gain feature X is obtained. temo The significant feature X in cbn2 , the expression is:

[0030]

[0031] S3.2: For the significant feature X cbn2 Indicates global average pooling, and then uses the softmax activation function for normalization to obtain the significant feature representation X cbn2的权重α1 , the expression is:

[0032]

[0033] Among them, GAP represents the global average pooling operation;

[0034] S3.3: Obtain significant feature X cbn2 The weight α 1 After that, the feature X is enhanced by weighting 1 The significant part of E , the expression is:

[0035]

[0036] in, is the significant feature in image X1;

[0037] S3.4: By suppressing the significant feature X cbn2 The weight α 1 To obtain the latent feature X S The weight α 2, and then obtain the potential feature X by weighting S , the expression is:

[0038]

[0039]

[0040] Among them, α 2 is the potential feature weight, is the latent feature of the image;

[0041] S3.5: Distinctive features Connect to L Transformer encoding blocks to obtain the final salient feature X E1 and L potential features ;

[0042] S3.6: Design a learning parameter {T, U i ∈B},U i =[U 1 ,U 2 ,…U L ],U 1 +U 2 +…+U L =1, used to represent L potential features Contribution to the vit network model, making the vit network model better and more potential The expression is:

[0043]

[0044] in, Represents the Lth potential feature The weight of represents the i-th latent feature, i=[1,2,…,L].

[0045] S3.7: The significant feature X E1 and L potential features Mining the emotional features of images in a complementary manner FIN , the expression is:

[0046]

[0047] Among them, X FIN is the final emotional feature of the image.

[0048] Compared with the prior art, the present invention provides an image emotion classification method with feature complementation under emotion gain, which has the following beneficial effects:

[0049] This image sentiment classification method with feature complementation under sentiment gain adopts a visual attention network as the backbone network to enhance the network's ability to mine image sentiment features, and introduces an emotional visual attention mechanism to improve the model's sensitivity to subtle emotional differences in images. In order to obtain the salient features and latent features in the image, on the one hand, the salient areas of the image are enhanced to obtain the salient features, and on the other hand, the salient areas are suppressed to mine the latent features in the non-salient areas of the image. Finally, the salient features and latent features are combined for image sentiment classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 This is a comparison diagram of the eight-class confusion matrix on the image dataset FI for an image sentiment classification method with complementary features under sentiment gain proposed by the present invention;

[0051] Figure 2 This is a comparison diagram of the confusion matrices of six categories on the image dataset Emtion6 for an image sentiment classification method with complementary features under sentiment gain proposed by the present invention;

[0052] Figure 3 This is a confusion matrix diagram of different network layer combinations in the image dataset FI for an image sentiment classification method with feature complementation under sentiment gain proposed by the present invention; among them, (a) Baseline, (b) Baseline+RCAM, (c) Baseline+PFMM, (d) Baseline+ASCFF+PFMM;

[0053] Figure 4 This is a structural framework diagram of an image emotion classification method with feature complementation under emotion gain proposed by the present invention;

[0054] Figure 5 It is an emotional gain network diagram of an image emotion classification method with complementary features under emotional gain proposed by the present invention;

[0055] Figure 6 This is a network diagram of a feature enhancement and suppression module of an image emotion classification method with feature complementation under emotion gain proposed by the present invention;

[0056] Figure 7 This is a feature complementary module network diagram of an image sentiment classification method with feature complementarity under sentiment gain proposed by the present invention. DETAILED DESCRIPTION

[0057] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0058] See also Figure 1-2 , an image sentiment classification method with feature complementation under sentiment gain, comprising the following steps:

[0059] S1: Image preprocessing, dividing the image dataset FI into a training set A 1 and test set A 2 , the training set A 1 The images are randomly scaled and cropped, and finally an image set B of size 448×448×3 is obtained;

[0060] The FI dataset was constructed by You et al. based on Mikels's emotional theory. It has about 23,185 images and is one of the largest well-labeled datasets available. They collected a total of more than 3 million weakly labeled images by querying image search engines (Flick and Instagram) with eight emotions as keyword queries, and then deleted images with any label and two different emotions. After that, 225 workers were selected from 1,000 workers for verification, where each image was assigned to five AMT participants. Each participant needed to select one of the eight emotions to describe the image. If and only if the image has three or more emotional votes of the same category, the category can be used as the emotional label of the image.

[0061] S2: Divide the image set B of size 448×448×3 into 16×16 image blocks, and extract the context and global information features of the 16×16 image blocks through the vit network model , the emotional gain feature X is obtained by the emotional gain method temo , context and global information features With emotional gain feature X temo Fusion obtains image feature X 1 ;

[0062] The emotion gain method can make up for the spatial information loss and edge information loss problems of the ViT network model when slicing images, and at the same time improve the model's sensitivity to subtle emotional differences in images.

[0063] S3: Image feature X 1 Obtain the salient feature X through the CBR convolution block EAnd the potential feature X is obtained by the potential feature mining method S , and then connect the salient features to L Transformer encoding blocks to obtain the final salient feature X E1 and L potential features Finally, these two features are used to mine the emotional features of the image in a complementary form to obtain the final emotional feature X FIN ;

[0064] S4: The obtained emotional feature X FIN Input into a multi-layer perceptron classification head, the labeled image set B trains the multi-layer perceptron classification head, and the test set A 2 After the test image is processed by fixed-size scaling and center cropping, it is input into the trained vit network model to obtain the corresponding emotion category;

[0065] S5: Take the image dataset Emotion6 and repeat steps S1-S4 to obtain the corresponding emotion category.

[0066] The Emotion6 dataset was constructed by Peng et al. and contains 1980 images obtained from Flickr using 6 classification keywords and corresponding synonyms. For each image, each image was annotated by 15 participants with a value arousal score and a discrete emotion distribution.

[0067] The image dataset FI includes eight emotions, namely pleasure, awe, satisfaction, excitement, anger, disgust, fear, and sadness. The image dataset Emotion6 includes happiness, sadness, surprise, anger, disgust, and fear.

[0068] The specific steps in step S2 are:

[0069] S2.1: Given an input image of size 448×448×3, split it into multiple non-overlapping image blocks P i , the size of each image block is 16×16, let N be the total number of image blocks, and each image block P i Represented as a vector vec(P i ), for each image block P i , flatten it into a vector of dimension M, for each vectorized image block vec(P i ) Add location embed E pos,i , and then introduce a category embedding E cls , and finally obtain the context of the 16×16 image and the global information feature X P , the expression is:

[0070]

[0071] Among them, vec(P i ) is each vectorized image block, E pos,i is the position embedding, E cls for category embedding;

[0072] S2.2.1: Use ResNet50 network to capture the spatial features X of image set B res , the expression is:

[0073]

[0074] Among them, B i is one of the images in the image set B, with a size of W×H×C, where W is the width, H is the height, and C is the number of channels;

[0075] S2.2.2: Using emotional visual attention mechanism to perceive spatial features X res Emotional characteristics of X emo , the expression is:

[0076]

[0077] S2.2.3: Enhance the spatial features X using a CBR convolutional block consisting of a 1×1 convolution, a BN layer, and a ReLU activation function res , and obtain the enhanced spatial feature X cbn1 , the expression is:

[0078]

[0079] Among them, X cbn1 is the enhanced spatial feature, K is the convolution kernel size, Conv represents the convolution operation, BN is the batch normalization operation, and Relu is the activation function;

[0080] S2.2.4: Transform the emotional feature X emo With the enhanced spatial feature X cbn1 Fusion gets the emotional gain feature X temo , the expression is:

[0081]

[0082] S2.3: Integrate context and global information features With emotional gain feature X temo Fusion obtains image feature X 1 , the expression is:

[0083]

[0084] The specific steps in step S3 are:

[0085] S3.1: Input image feature X 1 Through a 1×1 convolution, a BN layer and a Relu activation function, the emotional gain feature X is obtained. temo The significant feature X in cbn2 , the expression is:

[0086]

[0087] S3.2: For the significant feature X cbn2 Indicates global average pooling, and then uses the softmax activation function for normalization to obtain the significant feature representation X cbn2的权重α1 , the expression is:

[0088]

[0089] Among them, GAP represents the global average pooling operation;

[0090] S3.3: Obtain significant feature X cbn2 The weight α 1 After that, the feature X is enhanced by weighting 1 The significant part of E , the expression is:

[0091]

[0092] in, is the significant feature in image X1;

[0093] S3.4: By suppressing the significant feature X cbn2 The weight α 1 To obtain the latent feature X S The weight α 2 , and then obtain the potential feature X by weighting S , the expression is:

[0094]

[0095]

[0096] Among them, α 2 is the potential feature weight, is the potential feature of the image;

[0097] S3.5: Distinctive features Connect to L Transformer encoding blocks to obtain the final salient feature X E1 and L potential features ;

[0098] S3.6: Design a learning parameter {T, U i ∈B},U i =[U1,U2,…UL], U1+U2+…+UL=1, used to represent L potential features Contribution to the vit network model, making the vit network model better and more potential The expression is:

[0099]

[0100] in, Represents the Lth potential feature The weight of represents the i-th latent feature, i=[1,2,…,L].

[0101] S3.7: The significant feature X E1 and L potential features Mining the emotional features of images in a complementary manner FIN , the expression is:

[0102]

[0103] in, is the final emotional feature of the image.

[0104] An eight-category and two-category comparison experiment was conducted on the FI dataset. The two-category dataset classified anger, expectation, disgust, fear, joy, sadness, and surprise in the original FI eight-category dataset as positive categories, and disgust, excitement, fear, and sadness as negative categories. The method in this chapter was compared with the existing excellent image emotion classification methods on the two datasets, and the confusion matrices of the PES-ViT network and the ViT network on the FI eight-category dataset were compared. The "-" in the table indicates that no relevant experiments were conducted, and the bold indicates the highest classification accuracy, as shown in Table 1:

[0105] Table 1 shows the experimental results of different algorithms on the FI dataset for eight-classification and two-classification

[0106]

[0107] like Figure 1 The confusion matrix of the ViT network and the MSP-ViT network for the 8-category FI dataset is shown. From the results in the figure, we can see that the prediction accuracy of the MSP-ViT network is higher than that of the ViT network in most categories, especially in the sadness category, which is 11% higher. At the same time, the prediction accuracy in the happiness category reaches 91%.

[0108] pass Figure 1 As can be seen from Table 1, the performance of this technical solution on the dataset FI is always better than the existing most advanced methods.

[0109] A six-category and two-category comparison experiment was conducted on the Emotion6 dataset. The two-category dataset is to classify happiness, sadness, and surprise in the original Emotion6 six-category dataset as positive categories, and anger, disgust, and fear as negative categories. The technical solution is compared with the existing excellent image emotion classification methods on the two datasets, and the confusion matrices of the PES-ViT network and the ViT network on the Emotion6 six-category dataset are compared, as shown in Table 2, where "-" indicates that no relevant experiments were conducted, and bold indicates the highest classification accuracy.

[0110] Table 2 shows the experimental results of different algorithms on the eight-classification and two-classification of the Emotion6 dataset

[0111]

[0112] Figure 2 The following figure shows the comparison of the six-category confusion matrix of the ViT network and the MSP-ViT network on the Emotion6 dataset. It can be seen from the figure that the prediction accuracy of the MSP-ViT network in most categories is higher than that of the ViT network, indicating that the performance of the MSP-ViT network has obvious advantages. Figure 2 As can be seen from Table 2, the performance of this technical solution on the Emotion6 dataset is always better than the existing state-of-the-art methods.

[0113] In summary, the PES-ViT network achieved a classification accuracy of 76.57% and 94.03% on the FI eight-category and FI two-category datasets, respectively, which is 1.99% and 1.79% higher than the best DRSN-ViT network of the same type; it achieved a classification accuracy of 68.68% and 91.93% on the Emotion6 six-category and classification datasets, which is 6.18% and 5.2% higher than the best DRSN-ViT network of the same type, respectively, indicating that the classification performance of the PES-ViT network is better than the existing state-of-the-art methods. At the same time, in the comparison of the confusion matrix with the baseline network ViT, the PES-ViT network proposed in this chapter has higher prediction accuracy in most categories, proving the effectiveness of the emotion gain module and the potential feature mining module in the PES-ViT network.

[0114] In order to verify the effectiveness of the network sentiment gain method and the potential feature mining method, this paper sets an ablation experiment on the FI eight-category dataset with the ViT network as the Baseline. The experimental results are shown in Table 3. B, R, and F in the first column of Table 3 represent Baseline, RCAM, and PFMM, respectively. "+" means that the corresponding component is embedded in the network, and "-" means that the corresponding component is not embedded in the network.

[0115] The method of emotion gain is as follows: first, the spatial features of the image are captured by the ResNet network, and then the emotional visual attention mechanism CAM is used to obtain the emotional features in the image. Then, the enhanced spatial features are obtained by using the CBR convolution block, and finally, the enhanced spatial features are combined with the emotional features to obtain the emotional gain features.

[0116] Method for mining latent features: First, the image features are processed through a 1×1 convolution, a BN layer and a Relu activation function to obtain the significant feature representation of the image. Then, the significant feature weights are obtained through global average pooling and softmax activation function normalization. Then, the significant parts of the image features are enhanced in a weighted manner to mine the significant features in the image. Finally, the latent feature weights are obtained by suppressing the significant feature weights, and the latent features are obtained again through weighting.

[0117] Table 3 Comparison of sentiment classification accuracy in ablation experiments

[0118]

[0119] From the results in Table 3, we can see that embedding sentiment gain on the basis of Baseline can improve sentiment classification performance to a certain extent, because sentiment gain increases the model's sensitivity to subtle differences in sentiment in images and helps the model learn better. After connecting PFMM to the subsequent Transformer encoder inference process of Baseline, the sentiment classification accuracy is improved by 4.54%. In addition to the significant features of the image that can arouse the viewer's emotions, there are also some potential features that are easy to be ignored but still play a role. PFMM can extract the significant features of the image while paying attention to the potential features in the image. The complementarity of the two features can more fully mine the emotional information in the image. By adding RCAN and PFMM to Baselline, the model in this chapter can achieve significant classification performance improvement compared to Baseline, which proves the effectiveness of the model in this chapter in image sentiment classification.

[0120] With from Figure 3The results show that compared with the basic network, the accuracy of most categories of different module combinations has been improved. In the "pleasure" category, all methods have achieved high accuracy. The three methods composed of different modules have achieved a classification accuracy of 91%, but there is a certain degree of decrease in the negative emotion categories of "fear" and "anger". In other categories, the more samples there are, the more the accuracy improves. This is because the emotion gain module and the special enhancement inhibition module accumulate more emotional features of categories with a large number of samples in the process of continuously optimizing the model, and exchange the classification performance of a small number of samples for a higher overall classification performance of the model.

[0121] Figure 4 First, given an input image X, it is divided into multiple non-overlapping image blocks, and each image block is vectorized; each vectorized image block is embedded with position and category, and finally the context and global information features X of the image are obtained. P ; For the input image X, the emotional gain feature X is obtained by the RCAM emotional labeling module temo , and then fuse these two features to get the image feature X 1 Image feature X 1 Mining the significant emotional features in the image through the feature enhancement and suppression module E1 and potential emotional features, the two features are complemented to obtain the final emotional feature X FIN Finally, the obtained emotional feature X FIN Input into a multi-layer perceptron classification head to achieve the classification effect.

[0122] Figure 5 First, the input image X is passed through the ResNet50 network to obtain the spatial feature X res , and then use the emotional visual attention mechanism to perceive the spatial feature X res Emotional characteristics of X emo ; Spatial feature X res The enhanced emotional feature X is obtained by the CBR convolution block emo1 , and finally the sentiment feature X emo and the enhanced emotional feature X emo1 Fusion obtains image feature X 1 .

[0123] Figure 6 First, the image feature X 1 After the CBR convolution block, the salient feature representation X is obtained cbn2 , and then the significant feature weight α is obtained through GAP and Softmax operations 1 , and then enhance the feature X by weighting 1 The significant part of E; Then by suppressing the significant feature X cbn2 The weight α 1 To obtain the latent feature X S The weight α 2 , and then obtain the potential feature X by weighting S .

[0124] Figure 7 In the above example, the salient features are connected to L Transformer encoding blocks and PFMM potential feature mining modules to obtain the final salient feature X E1 And L potential features, then the two features will be used in a complementary form to obtain the emotional feature X of the image FIN .

[0125] The method of sentiment gain and the method of latent feature mining can help the model better accumulate advantages in the model iteration process by enhancing the model's sensitivity to sentiment and combining significant features and latent features, thereby improving the classification accuracy of the model in the overall category.

[0126] In summary, the image sentiment classification method with feature complementation under sentiment gain adopts the visual attention network as the backbone network to enhance the network's ability to mine image sentiment features, and introduces the emotional visual attention mechanism to improve the model's sensitivity to subtle emotional differences in images. In order to obtain the salient features and latent features in the image, on the one hand, the salient areas of the image are enhanced to obtain the salient features, and on the other hand, the salient areas are suppressed to mine the latent features in the non-salient areas of the image. Finally, the salient features and latent features are combined for image sentiment classification.

[0127] It should be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0128] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for image sentiment classification with feature complementation under sentiment gain, characterized in that: The following steps are involved: S1: Image preprocessing: divide the image dataset FI into a training set A1 and a test set A2, randomly scale and crop the images in the training set A1, and finally obtain an image set B with a size of 448×448×3; S2: Divide the image set B of size 448×448×3 into 16×16 image blocks, and extract the context and global information features of the 16×16 image blocks through the vit network model , the emotional gain feature X is obtained by the emotional gain method temo , context and global information features With emotional gain feature X temo Fusion obtains image feature X1; S3: Obtain the significant feature X through the CBR convolution block for the image feature X1 E And the potential feature X is obtained by the potential feature mining method S , and then connect the salient features to L Transformer encoding blocks to obtain the final salient feature X E1 and L potential features Finally, these two features are used to mine the emotional features of the image in a complementary form to obtain the final emotional feature X FIN ; S4: The obtained emotional feature X FIN Input into a multi-layer perceptron classification head, the labeled image set B is used to train the multi-layer perceptron classification head, and the test images of the test set A2 are processed by fixed-size scaling and center cropping, and then input into the trained vit network model to obtain the corresponding emotion category; S5: Take the image dataset Emotion6 and repeat steps S1-S4 to obtain the corresponding emotion category; The specific steps in step S2 are: S2.1: Given an input image of size 448×448×3, split it into multiple non-overlapping image blocks P i , the size of each image block is 16×16, let N be the total number of image blocks, and each image block P i Represented as a vector vec(P i ), for each image block P i , flatten it into a vector of dimension M, for each vectorized image block vec(P i )Add location embed E pos,i , and then introduce a category embedding E cls , and finally obtain the context of the 16×16 image and the global information feature X P , the expression is: Among them, vec(P i ) is each vectorized image block, E pos,i is the position embedding, E cls for category embedding; S2.2.1: Use ResNet50 network to capture the spatial features X of image set B res , the expression is: Among them, B i is one of the images in the image set B, with a size of W×H×C, where W is the width, H is the height, and C is the number of channels; S2.2.2: Using emotional visual attention mechanism to perceive spatial features X res Emotional characteristics of X emo , the expression is: S2.2.3: Enhance the spatial features X using a CBR convolutional block consisting of a 1×1 convolution, a BN layer, and a ReLU activation function res , and obtain the enhanced spatial feature X cbn1 , the expression is: Among them, X cbn1 is the enhanced spatial feature, K is the convolution kernel size, Conv represents the convolution operation, BN is the batch normalization operation, and Relu is the activation function; S2.2.4: Transform the emotional feature X emo With the enhanced spatial feature X cbn1 Fusion gets the emotional gain feature X temo , the expression is: S2.3: Integrate context and global information features With emotional gain feature X temo The image feature X1 is obtained by fusion, and the expression is: 。 2. The image emotion classification method with feature complementation under emotion gain according to claim 1, characterized in that: The image dataset FI includes eight emotions, namely pleasure, awe, satisfaction, excitement, anger, disgust, fear, and sadness. The image dataset Emotion6 includes happiness, sadness, surprise, anger, disgust, and fear.

3. The image emotion classification method with feature complementation under emotion gain according to claim 1, characterized in that: The specific steps in step S3 are: S3.1: The input image feature X1 is passed through a 1×1 convolution, a BN layer and a Relu activation function to obtain the emotional gain feature X temo The significant feature X in cbn2 , the expression is: S3.2: For the significant feature X cbn2 Indicates global average pooling, and then uses the softmax activation function for normalization to obtain the significant feature representation X cbn2的权重α1 , the expression is: Among them, GAP represents the global average pooling operation; S3.3: Obtain significant feature X cbn2 After adding the weight α1, the significant part of feature X1 is weighted to obtain the significant feature X E , the expression is: in, is the significant feature in image X1; S3.4: By suppressing the significant feature X cbn2 The weight α1 is used to obtain the latent feature X S The weight α2 is used to obtain the potential feature X again by weighting S , the expression is: Among them, α2 is the potential feature weight, is the potential feature of the image; S3.5: Distinctive features Connect to L Transformer encoding blocks to obtain the final salient feature X E1 and L potential features ; S3.6: Design a learning parameter {T, U i ∈B},U i =[U1,U2,…U L ],U1+U2+…+U L =1, used to represent L potential features Contribution to the vit network model, making the vit network model better and more potential The expression is: in, Represents the Lth potential feature The weight of represents the i-th potential feature, i=[1,2,…,L]; S3.7: The significant feature X E1 and L potential features Mining the emotional features of images in a complementary manner FIN , the expression is: in, is the final emotional feature of the image.

Images