An image emotion classification method based on self-attention interactive fusion

Abstract

The present invention belongs to the technical field of image emotion classification, and specifically relates to an image emotion classification method based on self-attention interaction fusion. The present invention extracts the feature map output by the object detector to represent the low-level semantic features of the object, and generates the object semantic feature representation of the emotional image through the spatial Fisher vector method; uses a fully convolutional neural network to extract the saliency map, performs binary segmentation on it, and sends the minimum circumscribed rectangle part of the white area of ​​the binary map to the pre-trained CNN, takes the 2048-dimensional vector of the second-to-last fully connected layer as the salient area feature, and uses the output of the CNN fully connected layer as the global appearance feature of the emotional image; these three deep features are sent to the self-attention conversion network to generate the final image emotion classification result. The present invention can provide more accurate image emotion classification results and help machines understand the emotions implied in the image.

Description

Technical Field

[0001] The present invention belongs to the technical field of image emotion classification, and in particular relates to an image emotion classification method based on self-attention interaction fusion. Background Art

[0002] In recent decades, people have been trying to find effective representations that can improve image classification performance. In early work, hand-crafted low-level features were the first choice for image classification. Machajdik and Hanbury developed a method for extracting low-level features, defining a rich set of hand-crafted features based on art and psychology theories, including composition, color, and texture, to represent the emotional content of images. Ko et al. designed two hand-crafted features based on psychology and experiments: color features and shape features of images. First, two visual dictionaries were established using color image scaling and hierarchical clustering methods. Then, two features were extracted from the image and combined. Finally, the number of occurrences of each feature in the dictionary was calculated, and the extracted features were converted into a histogram to explore the relationship between image color, shape, and image emotion labels. In order to complete the task of recognizing image emotion labels, Colombo et al. proposed to fuse the image features and other underlying image features according to a set of rules. First, the image was transformed using the Hough transform, and then the distribution of the slope of the line segment detected by the Hough transform was converted into a histogram. The slope of the line segment was mapped to human emotions. Finally, the correlation between the shape features of the image and human emotions was analyzed. Image emotion can be regarded as a kind of high-level semantics of the image. The "semantic gap" problem between high-level semantics and low-level visual features of the image will also have a significant impact on the effect of large-scale image sentiment analysis.

[0003] Objects are another important information that represents the emotions of images. Compared with fixed-size partitions of image segmentation, the results of object detection (category, size, and layout) can provide more effective spatial context features to distinguish images of different emotion categories. Farhadi uses intermediate attributes of objects to build image representations, which describe parts, shapes, and materials; high-level representations are defined by combining low-level semantic elements. Li uses the response maps of 200 object detectors of different scales, called Object Bank (OB), as attributes of images for image recognition.

[0004] Recently, deep learning theory has achieved great success in image and video analysis. Compared with shallow learning, deep learning models always stack many intermediate layers from input data to classification layer, and can learn high-level semantic features from a large amount of training data. In image recognition, researchers use the deep activations of deep neural networks as low-level features of images and combine these features to form high-level representations by using generative models such as Fisher vectors. There are some works dedicated to designing new deep neural network models to extract more identifiable features from emotional images. There are also some methods that provide mixed representations of emotional images by combining different types of deep features. At present, these deep learning-based methods have achieved the best image emotion classification results on benchmark datasets. Summary of the invention

[0005] The purpose of the present invention is to provide an image emotion classification method based on self-attention interactive fusion.

[0006] A method for image emotion classification based on self-attention interaction fusion, comprising the following steps:

[0007] Step 1: Use 177 object categories of images to train the YOLO v2 object detector.

[0008] The image is stretched to 608*608 dimensions and input into YOLO v2. The dimension of the output feature map is 328510. Maxpooling is applied to the 5 anchor boxes of each unit of the feature map to obtain a feature map with a dimension of 65702. 65702 dimensions can be regarded as a local descriptor with 182 dimensions multiplied by 361. The 65702-dimensional feature map is converted to 47616 dimensions through the spatial Fisher vector method, and principal component analysis is performed on it using 128 Gaussian kernels to reduce the dimension, obtaining a 2048-dimensional object semantic feature.

[0009] Step 2: The original input image is scaled to a size of 224*224*3 and sent to a pre-trained fully convolutional network FCN to obtain a saliency map of size 224*224*1, which is then binarized and segmented;

[0010] Step 3: Send the image content of the minimum bounding rectangle of the white area of ​​the binary image to the pre-trained convolutional neural network CNN, and take the 2048-dimensional vector of the second-to-last fully connected layer as the salient area feature;

[0011] Step 4: Input the original input image into the pre-trained convolutional neural network CNN, and use the 2048-dimensional vector output by the CNN fully connected layer as the global appearance feature of the image;

[0012] Step 5: Input the object semantic features, salient region features, and global appearance features into the self-attention conversion network for training. The self-attention conversion network interactively fuses and generates feature vectors that are mapped to eight emotion categories to obtain the probability corresponding to each emotion. The emotion with the highest corresponding probability is taken as the image emotion.

[0013] Step 6: After scaling, the image to be classified is input into the trained YOLO v2 object detector to obtain the object semantic features, and then input into the trained pre-trained convolutional neural network CNN to obtain the salient area features and global appearance features; the object semantic features, salient area features, and global appearance features of the image to be classified are input into the trained self-attention conversion network to obtain the probability corresponding to each emotion, and the emotion with the highest corresponding probability is taken as the image emotion.

[0014] The beneficial effects of the present invention are:

[0015] The present invention extracts the feature map output by the object detector to represent the low-level semantic features of the object, and generates the object semantic feature representation of the emotional image through the spatial Fisher vector method; uses a fully convolutional neural network to extract the salient map, performs binary segmentation on it, and sends the minimum circumscribed rectangle part of the white area of ​​the binary map to the pre-trained CNN, takes the 2048-dimensional vector of the second-to-last fully connected layer as the salient area feature, and uses the output of the CNN fully connected layer as the global appearance feature of the emotional image; these three deep features are sent to the self-attention conversion network to generate the final image emotion classification result. The present invention can provide more accurate image emotion classification results, help machines understand the emotions implied by images, and make predictions about human decisions. It can be widely used in opinion mining, business recommendations, mental health assessment and other aspects. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is a flow chart of the present invention. DETAILED DESCRIPTION

[0017] The present invention is further described below in conjunction with the accompanying drawings.

[0018] The present invention proposes an image emotion classification method based on self-attention interactive fusion, and constructs a comprehensive representation of emotion images composed of three deep features to improve the performance of image emotion classification, namely object semantic features, salient features, and global appearance features. The feature map output by the object detector is extracted to represent the low-level semantic features of the object, and the object semantic feature representation of the emotion image is generated by the spatial Fisher vector method. The present invention first uses a fully convolutional neural network to extract the salient map, performs binary segmentation on it, and sends the image content of the minimum circumscribed rectangle part of the white area of ​​the binary map to a pre-trained convolutional neural network (CNN), and takes the 2048-dimensional vector of the second-to-last fully connected layer as the salient area feature. The output of the CNN fully connected layer is used as the global appearance feature of the emotion image. These three deep features are sent to the self-attention transformation network (Transformer) to generate the final image emotion classification result. The present invention can provide more accurate image emotion classification results and help machines understand the emotions implied by the image.

[0019] The present invention provides an efficient and accurate image emotion classification method. The user inputs the image to be classified as the input of the model into a trained deep neural network model, and the model automatically gives the classification result of the image. The specific steps are as follows:

[0020] Step 1: Extract feature maps output by the object detector YOLO v2, which represent low-level semantic features of objects. These low-level features are trained using the spatial Fisher vector method to generate object semantic feature representations of emotional images that contain spatial layout information.

[0021] Specifically, 177 object images are used to train the YOLO v2 object detector to generate object-oriented low-level features. The image is stretched to 608*608 dimensions and input to YOLO v2. The dimension of the output feature map is 328510. Maxpooling is applied to the 5 anchor boxes of each unit of the feature map to obtain a feature map with a dimension of 65702. 65702 dimensions can be regarded as a local descriptor with 182 dimensions multiplied by 361. The 65702-dimensional feature map is converted to 47616 dimensions through the spatial Fisher vector method, and the principal component analysis is performed on it using 128 Gaussian kernels to reduce the dimension, and the object semantic features of 2048 dimensions are obtained.

[0022] Step 2: Use the fully convolutional neural network to extract the salient map, perform binary segmentation on it, and send the image content of the minimum bounding rectangle of the white area of ​​the binary map to the pre-trained convolutional neural network (CNN), and take the 2048-dimensional vector of the second-to-last fully connected layer as the salient area feature.

[0023] Specifically, the input image is scaled to 224*224*3 and fed into a pre-trained fully convolutional network (FCN) to obtain a salient map of size 224*224*1, which is then binarized and segmented. The image content of the minimum bounding rectangle of the white area of ​​the binary image is fed into a pre-trained convolutional neural network (CNN), and the 2048-dimensional vector of the penultimate fully connected layer is taken as the salient region feature.

[0024] Step 3: Send the original image to CNN, and use the output of the CNN fully connected layer to obtain the 2048-dimensional global appearance features of the image.

[0025] Step 4: The three features of object semantics, salient regions, and global features are sent to the attention conversion network (Transformer), interactively fused to generate feature vectors mapped to eight emotion categories, and the probability corresponding to each emotion is obtained. The emotion with the highest corresponding probability is taken as the image emotion;

[0026] Step 5: After scaling, the image to be classified is input into the trained YOLO v2 object detector to obtain the object semantic features, and then input into the trained pre-trained convolutional neural network CNN to obtain the salient area features and global appearance features; the object semantic features, salient area features, and global appearance features of the image to be classified are input into the trained self-attention conversion network to obtain the probability corresponding to each emotion, and the emotion with the highest corresponding probability is taken as the image emotion.

[0027] The present invention can provide more accurate image emotion classification results, help machines understand the emotions implied in images, and make predictions about human decisions. It can be widely used in opinion mining, business recommendations, mental health assessment, and other aspects.

[0028] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for image emotion classification based on self-attention interaction fusion, characterized in that: The following steps are involved: Step 1: Use 177 object categories of images to train the YOLO v2 object detector. The image is stretched to 608*608 dimensions and input into YOLO v2. The dimension of the output feature map is 328510. Maxpooling is applied to the 5 anchor boxes of each unit of the feature map to obtain a feature map with a dimension of 65702. 65702 dimensions can be regarded as a local descriptor with 182 dimensions multiplied by 361. The 65702-dimensional feature map is converted to 47616 dimensions through the spatial Fisher vector method, and principal component analysis is performed on it using 128 Gaussian kernels to reduce the dimension, obtaining a 2048-dimensional object semantic feature. Step 2: The original input image is scaled to a size of 224*224*3 and sent to a pre-trained fully convolutional network FCN to obtain a saliency map of size 224*224*1, which is then binarized and segmented; Step 3: Send the image content of the minimum bounding rectangle of the white area of ​​the binary image to the pre-trained convolutional neural network CNN, and take the 2048-dimensional vector of the second-to-last fully connected layer as the salient area feature; Step 4: Input the original input image into the pre-trained convolutional neural network CNN, and use the 2048-dimensional vector output by the CNN fully connected layer as the global appearance feature of the image; Step 5: Input the object semantic features, salient region features, and global appearance features into the self-attention conversion network for training. The self-attention conversion network interactively fuses and generates feature vectors that are mapped to eight emotion categories to obtain the probability corresponding to each emotion. The emotion with the highest corresponding probability is taken as the image emotion. Step 6: After scaling, the image to be classified is input into the trained YOLO v2 object detector to obtain the 2048-dimensional object semantic features, and then input into the trained pre-trained convolutional neural network CNN to obtain the salient area features and global appearance features; The object semantic features, salient area features, and global appearance features of the image to be classified are input into the trained self-attention conversion network to obtain the probability corresponding to each emotion, and the emotion with the highest corresponding probability is taken as the image emotion.

Images