A method for image sentiment polarity classification combined with graph convolutional neural network

Abstract

A method for classifying image emotion polarity combined with a graph convolutional neural network, relates to the fields of intelligent media computing and computer vision technology; firstly, object information is extracted from training samples, and a graph model is established using the object information and visual features in each image; secondly, the graph convolutional network is used to extract the object interaction information contained in the graph model, and is fused with the features of the convolutional neural network; then, the training samples are pre-processed and passed into the network, and the parameters of the model are iteratively updated using a loss function and an optimizer until convergence is reached, completing the training; finally, the test data is sent into the network to obtain the model's prediction results for the test data and the classification accuracy. The present invention extracts the interaction features of objects in the image in the emotion space to make the classification features more consistent with the emotional features of the objects and the human emotion triggering mechanism, and adds high-level semantic features on the basis of visual features, which helps to improve the performance of the emotion classification algorithm in practical application scenarios.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision and relates to an image emotion polarity classification method combined with a graph convolutional neural network. Background Art

[0002] With the rapid development of image social networks, more and more people like to use pictures to express their moods. Images containing emotional information play an important role in strengthening the views conveyed by the content and effectively influencing the audience. Faced with a large amount of image data containing user emotions, analyzing the emotional information contained in the images will greatly promote the development of social media and have a wide range of applications in education, advertising, entertainment and other fields. Therefore, image sentiment analysis has become one of the recent research hotspots.

[0003] Earlier image sentiment analysis methods mainly used the statistical characteristics of images, such as artificial features such as color, texture, and lines, to achieve image sentiment classification. However, due to the large semantic and emotional gaps between such low-level image features and human emotions, they did not achieve good sentiment polarity analysis results. In recent years, with the rapid development of social media and electronic computers, deep learning has been continuously developed in the field of computer vision, and has achieved good results in image classification, object detection, target tracking, and other fields. The role of high-level visual features extracted by convolutional neural networks in the field of computer vision has also attracted some researchers to apply them to image sentiment classification. In 2015, You et al. designed an image sentiment classification algorithm using convolutional neural networks, which achieved better classification results than traditional methods, but due to the inherent limitations of the self-learning theory used, its performance improvement was relatively limited. With the in-depth study of convolutional neural networks and human emotion principles, Yang et al. proposed an image sentiment classification algorithm combined with an image instance segmentation algorithm in 2018. By combining the current image instance segmentation algorithm with good results, the area containing rich semantic and emotional content in the image is extracted, and the visual features in the image sentiment classification algorithm are enhanced and the accuracy of the image sentiment classification algorithm is improved. In 2019, Wu et al. designed a convolutional neural network that combines image saliency detection algorithm with image sentiment classification. Based on the connection between the principles of human attention mechanism and human emotions, the saliency detection algorithm is used to imitate humans to obtain the most attention-grabbing areas in the image, and the error back-propagation mechanism polarity feedback adjustment is used to further improve the accuracy of image sentiment classification.

[0004] Some of the latest achievements mentioned above and the research results in the field of human emotion principles inspired us. The existing image sentiment classification algorithm improves the classification algorithm effect by acquiring and enhancing the visual features contained in some objects in the image. However, the existing research also has certain limitations. It enriches and enhances the visual features of objects based on object segmentation and saliency detection, but its use of object features is limited to visual features, and does not use the interactive relationship of objects in the emotional space. This also leads to the limitations of the designed model and limited performance improvement. On this basis, we designed a sentiment classification method that combines the emotional relationship of objects with visual features using a graph convolution model. While using the visual features of the image, it also considers the mutual influence of each object in the image in the emotional space, fully exploring the features of the image at the emotional semantic level, and improving the accuracy of image sentiment classification. Summary of the invention

[0005] The purpose of this invention is to design an image sentiment polarity classification method combined with a graph convolutional neural network. Figure 1 shown.

[0006] In response to the problems of existing research methods, we designed a model that uses the emotional relationship features of objects in the image, combining the graph convolutional neural network with the convolutional neural network, so that the model can simultaneously obtain the relationship features of objects in the image in the emotional space and the visual features of the image. By combining the open source panoramic segmentation algorithm Detectron2, we designed a method to establish a corresponding graph model based on the image segmentation results. Using the graph convolution model, we can express the interaction features of objects in the image in the emotional space, that is, the relationship features, while enriching the visual features of the image. Combined with the basic convolutional neural network, it can enrich the high-level semantic features of the image and improve the accuracy of image emotional polarity analysis.

[0007] The specific steps of the present invention are as follows:

[0008] Step 1, obtain image object information: use the panoramic segmentation model to perform panoramic segmentation on each image in the data set to obtain information such as the category, position, and area of ​​the object in the image, and use the word annotation results in SentiWordNet to annotate the sentiment polarity and intensity of the object category words obtained by panoramic segmentation in the data set;

[0009] Step 2, establish a graph model: use objects as nodes, the inverse of the emotional space distance between object words and other objects as edge weights, and the brightness and texture features of the corresponding areas of the objects as node features to establish the corresponding graph model.

[0010] Step 3: Build a deep network model: Use the basic convolutional neural network model VGG-16 and the graph convolution model GCN, merge the outputs of the two models and input them into the fully connected layer, and replace the last classification layer of the original model with the number of categories to be classified as the output dimension of the fully connected layer;

[0011] Step 4, training model: After preprocessing the image by scaling and random flipping, input it into the network model, optimize it using the stochastic gradient descent method, evaluate the model performance using the cross entropy function and learn the model parameters;

[0012] Step 5, obtaining the emotion category of the image to be tested: After the image in the database has undergone the same preprocessing steps as step 4, it is input into the model trained in step 4 to obtain the corresponding emotion category.

[0013] Compared with the prior art, the present invention has the following obvious outstanding substantive features and significant technical progress:

[0014] This paper proposes a new image sentiment classification algorithm, which combines graph convolutional networks with basic deep convolutional neural networks, and adds high-level semantic features such as object sentiment relationship features on the basis of high-level visual features. In view of the problem that existing public sentiment datasets do not annotate the object information contained in the pictures, the research results in the field of panoptic segmentation are used to design a method to achieve the conversion from pictures to graph models, and the graph convolutional network is used to update, enhance and fuse the graph model with visual features, and the emotional trigger mechanism is used to accurately extract the emotional features in the picture to obtain better sentiment classification results. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments:

[0016] Figure 1 The architecture diagram of the convolutional neural network for training image sentiment classification based on this method.

[0017] Figure 2 This is the overall flow chart for emotional image classification based on this method. DETAILED DESCRIPTION

[0018] The present invention proposes an image sentiment polarity classification method combined with a graph convolutional neural network. The overall structure of the present invention is as follows Figure 1 The present invention is simulated in Windows 10 and Pyhcharm environment. The specific implementation process of the present invention is as follows Figure 2 As shown, the specific implementation steps are as follows:

[0019] Step 1, obtain image object information: use the panoramic segmentation model to perform panoramic segmentation on each image in the data set to obtain information such as the category, position, and area of ​​the object in the image, and use the word annotation results in SentiWordNet to annotate the sentiment polarity and intensity of the object category words obtained by panoramic segmentation in the data set;

[0020] Step 2, establish a graph model: use objects as nodes, the inverse of the emotional space distance between object words and other objects as edge weights, and the brightness and texture features of the corresponding areas of the objects as node features to establish the corresponding graph model.

[0021] Step 3: Build a deep network model: Use the basic convolutional neural network model VGG-16 and the graph convolution model GCN, merge the outputs of the two models and input them into the fully connected layer, and replace the last classification layer of the original model with the number of categories to be classified as the output dimension of the fully connected layer;

[0022] Step 4, training model: After preprocessing the image by scaling and random flipping, input it into the network model, optimize it using the stochastic gradient descent method, evaluate the model performance using the cross entropy function and learn the model parameters;

[0023] Step 5, obtaining the emotion category of the image to be tested: After the image in the database has undergone the same preprocessing steps as step 4, it is input into the model trained in step 4 to obtain the corresponding emotion category.

[0024] In step 1, a method for obtaining image object information using a panoramic segmentation model is designed:

[0025] This method can be used for sentiment classification of images in large real social networks. Therefore, in this example, the general public sentiment dataset Flickr&Instagram (hereinafter referred to as FI dataset) extracted from Flickr and Instagram is selected. This dataset has the characteristics of large data scale and accurate sentiment annotation, which is more in line with the real network environment.

[0026] Using the panoramic segmentation model in Deterton2, we segment each image in the FI dataset and save information such as the object category, location, and pixel location. For the category words of objects, we designed a word sentiment tagging method using the sentiment dictionary SentiWordNet. The specific calculation method is as follows:

[0027]

[0028] S W =S p -S n

[0029] Where S p is the positive sentiment intensity of the object word W, where p represents positive, S n The negative sentiment intensity of the object is represented by the subscript n, S′ is the noun and adjective containing the current word W in SentiWordNet, and m is the number of S′. ip , S′ in is the positive and negative sentiment intensity of S′ marked in SentiWordNet, where the subscript i represents the order of n words, p and n represent positive and negative respectively, and their values ​​are between 0 and 1. The sentiment value of the current word W is calculated by the sum and average method. At the same time, the difference between the positive and negative sentiment intensity of each word represents the sentiment intensity S of the current word.

[0030] In step 2, a method for establishing a graph model using image object information is designed:

[0031] Using the object information obtained by panoptic segmentation, the objects contained in the image are used as nodes of the graph model, and the inverse of the distance between words in the emotional space is used as the edge weight between nodes in the graph model. The following formula is used to calculate the distance between words in the emotional space:

[0032]

[0033] Where S i , S j Represents two words W i , W j When the sentiment polarity of two words is opposite, we use the absolute value of the difference in sentiment intensity of the two words plus 1 to describe the distance between the words in the sentiment space. When the sentiment polarity of two words is the same, the absolute value of the difference in sentiment intensity is used as the sentiment distance. In particular, when the sentiment intensity of two words is 0 at the same time, we stipulate that the sentiment distance between the two words is 0.5. Because the sentiment value of a word is between 0 and 1, 0.5 is used to distinguish between two neutral words and two words of the same polarity.

[0034] The brightness feature and texture feature are used as node features in the graph model. The image area where each object is located is obtained using the object border position information obtained in step 1. The brightness histogram of the pixels in the image is used as the brightness feature, that is, the pixel RGB value of the image area where the object is located is converted into the hue, saturation and brightness of the HSI space, and the brightness value is quantized and the quantized brightness value distribution curve is used as the brightness feature. Here, the brightness is quantized to 0-255, and finally a 256-dimensional feature vector is obtained.

[0035] At the same time, for texture features, the gray level co-occurrence matrix method is used to calculate the area where each object is located. Here we recommend calculating at 45° or 135°. The feature quality calculated at 0° or 90° is lower, and the calculated result is quantized into a 256-dimensional feature vector.

[0036] Finally, the brightness feature and the texture feature are concatenated to obtain a 512-dimensional feature vector as the feature corresponding to each node in the graph model, and the edge weight matrix A is used as the edge of the graph model to establish the graph model.

[0037] In step 3, a deep network model is established:

[0038] The GCN model is used to extract relational features. The stacked GCN structure is used to implement this method. In this method, a two-layer GCN structure is used, where the input feature H of the current layer k is k is the output of the previous layer, and the output result H is calculated as follows k+1 :

[0039]

[0040] in is the adjacency matrix, which is the result of adding the edge weight A calculated according to the object sentiment value above to the identity matrix; W k is the weight matrix of the current convolutional layer k, which is obtained by random initialization and trained and adjusted according to the loss function during the training process until the end of training; σ is a nonlinear activation function, Calculated by the following formula:

[0041]

[0042] in for The elements in i represent The coordinates in for The elements of ij are The coordinates in .

[0043] After calculation by the GCN model, the relationship feature is obtained as a 2048-dimensional feature vector.

[0044] For visual features, VGG16 is used. The VGG16 model pre-trained on ImageNet is obtained. After loading, the last classification layer with an input dimension of 2048*1024 and an output dimension of 1*1024 is removed, and its output dimension is adjusted to 2048*1 as the visual feature. The GCN network contains two graph convolution layers. The input dimension of the first graph convolution layer is 512*1 and the output is 1024*1. The input dimension of the second convolution layer is 1024*1 and the output is 2048*1, which is the final relational feature vector. The visual features and the relational features are concatenated to obtain a 4096*1 feature vector, which is input into the final classification layer. The output dimension is 1*2, that is, the predicted probability of each category. The corresponding emotion category of the position of the largest item is taken as the emotion category of the image output;

[0045] In step 4, the model training is implemented through operations such as data preprocessing, data input, and calculation of loss function:

[0046] The training set data is preprocessed by scaling and randomly flipping the image. In this example, the parameters of random cropping are set to 224*224, and the probability of random flipping is set to 0.5. The fixed size is used as a batch and input into the network model. Finally, the samples that are less than the fixed size batch are used as a batch. Setting the fixed batch size as large as possible can slightly improve the training effect of the model to a certain extent. However, due to the limitations of the experimental platform, it is recommended to select 8, 16 or 32. In this example, the fixed batch size is set to 16. Through the final classification layer, the output prediction results are automatically compared with the input training set labels, and the proportion of the number of correct samples to the total training samples is recorded as the accuracy of the training set in this round. At the same time, when the output vector is obtained, the loss value of the current model can be calculated using the loss function shown below. The loss value will be fed back to the optimizer for processing and then back-propagation to update the various parameters in the model.

[0047] In the calculation of the loss function, the cross entropy loss function shown below is used to maintain the distance between classes and make the images between different emotional categories farther apart:

[0048]

[0049] Where m is the number of images per batch during training, N is the number of emotion categories in the dataset, and x i is the feature of the i-th image in this batch obtained from the basic backbone network in the previous 3 before the classification layer. w and b are the weight and bias parameter values ​​in the classification layer. The subscript y i represents the category predicted after the classification layer, and j represents the category number corresponding to the prediction result, for example, Indicates that the i-th image in this batch is judged as yi The value of the bias parameter in the classification layer when the category is

[0050] Considering the convergence speed and convergence effect, the optimizer in this method selects stochastic gradient descent as the optimization method. The optimizer parameter settings mainly include the initial learning rate and momentum. The initial learning rate is generally selected from 0.1, 0.01, 0.0001, 0.00001, etc. depending on the convergence of the model. In this example, 0.01 is recommended. The convergence effect is more stable at this initial value. In principle, the momentum takes a value between 0 and 1. In this example, it is recommended to select the default value of 0.9 in the stochastic gradient descent method. Since the setting of a fixed learning rate is not conducive to the deep network finding better parameters in the second half of training, this method adds a fixed round to reduce the learning rate strategy during the training process. The recommended number of rounds is reduced 1-2 times in 20-30 rounds, and the total number of training rounds is recommended to be 50-80 rounds. In this example, the optimizer is set to reduce the learning rate every 20 rounds and every 30 rounds, and the model parameters are trained for 80 rounds to ensure effective convergence of the training effect. If the number of rounds is too small, convergence may not occur. If the number of rounds is too large, the training time will be increased but the effect will not be improved.

[0051] After each round of training samples, the model parameters are fixed and the validation set data in the FI dataset is used to scale and crop the data to a fixed size and then passed into the network model. In this example, the cropping parameters are set to 224*224. The model output is compared with the sample label and the proportion of correct samples, i.e., the validation set accuracy, is counted. If the validation set accuracy of the current round is higher than the highest validation set accuracy of the previous round, the current accuracy is saved as the highest validation set accuracy, and the model trained in the current round is saved. When all rounds of training are completed, the model with the highest validation set accuracy is the optimal model trained.

[0052] In step 5, the emotion category of the image to be tested is obtained:

[0053] After the test set data or any image in the FI dataset is preprocessed with the fixed-size scaling and center cropping steps in the same way as the validation set images in step 4, it can be input into the model one by one or in batches of fixed quantities. In this example, the fixed-size scaling and center cropping parameters are set to 224*224. In order to improve the processing efficiency under the same experimental conditions, it is recommended to use a batch size of 16 for the test set data in this example, and output the test images in batches to the model for testing. After model processing, the output results after the classification layer are compared with the labels of the samples and the proportion of correct samples, i.e., the accuracy of the test set, are counted. The emotion category corresponding to the output result is the image emotion category judged by the model.

[0054] The test set in the FI dataset was tested by the model in this example, and the accuracy result was 0.8808, which is higher than the best result in the current research content of similar methods: the accuracy of "Visual Sentiment Prediction based on Automatic Discovery of Affective Regions" published in the high-level journal IEEE TRANSACTIONS ON MULTIMEDIA in 2018 was 0.8635.

Claims

1. The image sentiment polarity classification method combined with graph convolutional neural network is characterized by: The following steps are included: Step 1, obtain image object information: use the panoramic segmentation model to perform panoramic segmentation on each picture in the data set to obtain the category and location information of the object in the picture, and use the word annotation results in SentiWordNet to annotate the sentiment polarity and intensity of the object category words obtained by panoramic segmentation in the data set; Step 2, establish a graph model: take objects as nodes, the inverse of the distance between object words in the emotional space as edge weights, and the brightness and texture features of the corresponding areas of the objects as node features to establish the corresponding graph model; Step 3: Build a deep network model: Use the basic convolutional neural network model VGG-16 and the graph convolution model GCN, merge the outputs of the two models and input them into the fully connected layer, and replace the last classification layer of the original model with the number of categories to be classified as the output dimension of the fully connected layer; Step 4, training model: After preprocessing, the image is input into the network model, optimized using the stochastic gradient descent method, and the cross entropy function is used to evaluate the model performance and learn the model parameters; Step 5, obtaining the emotion category of the image to be tested: After preprocessing, the image in the database is input into the model trained in step 4 to obtain the corresponding emotion category.

2. The method according to claim 1, characterized in that: In step 1, the panoramic segmentation algorithm is used to identify the object information of the image, and the word sentiment tagging method of the sentiment dictionary SentiWordNet is used. The specific calculation method is as follows: S W =S p -S n Where S p is the positive sentiment intensity of the object word W, the subscript p represents positive, S n is the negative sentiment intensity of the object, the subscript n represents negative, S′ is the noun and adjective containing the current word W in SentiWordNet, m is the number of S′, S′ ip , S′ in is S′ annotated in SentiWordNet i The positive and negative sentiment intensity of the word W is between 0 and 1, and the sentiment value of the current word W is calculated by summing and averaging. The difference between the positive and negative sentiment intensity of each word represents the sentiment intensity S of the current word. W .

3. The method according to claim 1, characterized in that: In step 2, the object information obtained by the panoramic segmentation algorithm is used, the object features contained in the picture are used as the nodes of the graph model, and the inverse of the distance between the words in the emotional space is used as the edge weight between the nodes in the graph model; for multiple object words W1, W2... contained in the picture, the following formula is used to calculate the weight of two words W i , W j Distance in emotional space S i , S j Represents the word W calculated according to the method in step 1 i , W j Corresponding to the sentiment value, when the sentiment polarities of two words are opposite, the absolute value of the difference in sentiment intensity of the two words plus 1 is used to describe the distance between the words in the sentiment space. When the sentiment polarities of two words are the same, the absolute value of the difference in sentiment intensity is used as the sentiment distance. When the sentiment intensity of two words is 0 at the same time, the sentiment distance between the two words is set to 0.5 to distinguish between two neutral words and two words of the same polarity. Finally, with two words W i , W j The inverse of the sentiment distance is used as the edge weight A of the corresponding node in the graph model ij , and repeat this step to calculate the edge weights between all words W1, W2, etc., and obtain the edge weight matrix A of each picture; Brightness features and texture features are used as node features in the graph model; The object position information obtained in the panoramic segmentation algorithm is used to obtain the image area where each object is located; the brightness histogram of the pixels in the picture is used as the brightness feature, that is, the pixel RGB value of the image area where the object is located is converted into the hue, saturation and brightness of the HSI space, and the brightness value is quantized and the quantized brightness value distribution curve is used as the brightness feature, and the brightness is quantized to 0-255, and finally a 256-dimensional feature vector is obtained; At the same time, for texture features, the gray-level co-occurrence matrix method is used to calculate the area where each object is located; the calculation is performed at 45° or 135°, and the calculated results are quantized into a 256-dimensional feature vector; Finally, the brightness feature and the texture feature are concatenated to obtain a 512-dimensional feature vector as the feature corresponding to each node in the graph model, and the edge weight matrix A is used as the edge of the graph model to establish the graph model.

4. The method according to claim 1, characterized in that: In step 3, Use the GCN model to extract relational features in the graph model; The stacked GCN structure is used to implement the two-layer GCN structure, where the input feature H of the current layer k is k is the output of the previous layer, and the output result H is calculated as follows k +1 : in is the adjacency matrix, which is the result of adding the edge weight A calculated according to the object sentiment value above to the identity matrix; W k is the weight matrix of the current convolutional layer k, which is obtained by random initialization and trained and adjusted according to the loss function during the training process until the end of training; σ is a nonlinear activation function, Calculated by the following formula: in for The elements in i represent The coordinates in for The elements of ij are The coordinates in ; After calculation by the GCN model, the relationship feature is obtained as a feature vector of 2048*1 dimension; For visual features, VGG16 is used. The VGG16 model pre-trained on ImageNet is obtained. After loading, the last classification layer with an input dimension of 2048*1024 and an output dimension of 1024*1 is removed, and its output dimension is adjusted to 2048*1 as the visual feature; The relational features are concatenated with the visual features to obtain a 4096*1 feature vector and input it into the fully connected layer to classify the image sentiment.

5. The method according to claim 1, characterized in that: In step 4, the loss function uses cross entropy loss to perform basic loss measurement; the specific loss function is as follows: Where o is the number of images per batch during training, N is the number of emotion categories in the dataset, and x p is the feature of the pth image in the batch obtained from the basic backbone network in step 3 before the classification layer; w and b are the weight and bias parameter values ​​in the classification layer, which are obtained by random initialization and trained and adjusted according to the loss function during the training process until the training is completed, with the subscript y q represents the category predicted after the classification layer, and q represents the category number corresponding to the prediction result. Indicates that the i-th image in this batch is judged as y p The value of the bias parameter in the classification layer when the category is During the training process, the initial learning rate is 0.01, and the learning rate is reduced to one tenth of the current learning rate every 20 rounds. The training is terminated after the number of training times reaches more than 80.

Images