An image sentiment analysis method based on a bidirectional connection
By using a method based on bidirectional feature connections, deep convolutional neural networks are used to extract image features and fuse features at different scales, which solves the problem of low accuracy in emotion classification in visual emotion analysis and achieves higher classification accuracy.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- HANGZHOU DIANZI UNIV
- Filing Date
- 2022-12-27
- Publication Date
- 2026-05-15
AI Technical Summary
Existing visual sentiment analysis technologies struggle to effectively simulate human emotional processes, as they map from a concrete image space to an abstract emotional space, resulting in low accuracy in sentiment classification.
A method based on bidirectional feature connectivity is adopted, which extracts feature tensors of different scales through a deep convolutional neural network, and uses attention modules and convolutions to downsample layer by layer. The feature pyramid structure is combined to perform feature fusion, and finally the sentiment classification result is output through a classification network.
It improves the accuracy of sentiment classification, enhances the model's ability to perceive abstract features, breaks the linear structure of a single classification network, and is suitable for more complex sentiment classification problems.
Smart Images
Figure CN115995029B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of image sentiment analysis, specifically relating to a visual sentiment analysis method based on bidirectional feature connectivity. Background Technology
[0002] Rich emotions are a uniquely human psychological phenomenon. Emotion is part of biological intelligence and a derivative of it. Generally, we believe that the richness of emotions is directly related to the level of intelligence; organisms capable of generating and expressing rich emotions possess higher intelligence, while those unable to express emotions or with limited emotional expression have lower intelligence. In recent years, with the booming development of artificial intelligence and robotics, affective computing has become increasingly important in human-computer interaction research within AI. Natural interaction methods with emotions not only enable more user-friendly interfaces but are also an essential path to achieving strong AI. AI that combines emotion and intelligence has higher practical value and real-world significance. For example, in management, affective computing can be used to obtain the emotions of leaders and employees, thereby improving overall corporate efficiency; in trade, analyzing customer reviews to understand their emotions allows for more precise promotions, helping companies build their brands more effectively; in the health field, emotion prediction based on doctor-patient dialogues can help doctors analyze patient psychology, assist in psychological interviews, and ultimately diagnose and treat mental illnesses and alleviate negative emotions such as suicidal ideation. To achieve true artificial intelligence, it is essential to realize natural human-computer interaction that integrates intelligence and emotion. In recent years, thanks to breakthroughs in computer technology and deep neural network technology, artificial intelligence research has reached a new peak. With the booming development of artificial intelligence and robotics, affective computing is playing an increasingly important role in human-computer interaction research. Natural interaction methods with emotions not only enable more user-friendly interfaces but are also an essential path to achieving strong artificial intelligence. Artificial intelligence that combines emotion and intelligence has higher practical value and real-world significance. For example, in the management industry, affective computing can be used to obtain the emotions of leaders and employees, thereby improving the overall efficiency of enterprises; in trade, analyzing customer reviews to understand customer emotions and conduct precise promotions can help companies build their brands more accurately; in the health field, emotion prediction based on doctor-patient dialogues can help doctors analyze patient psychology, assist in psychological interviews, and thus diagnose and treat mental illnesses and alleviate negative emotions such as suicidal ideation. To achieve true artificial intelligence, it is essential to realize natural human-computer interaction that integrates intelligence and emotion. Against this backdrop, affective computing, an emerging scientific field, has begun to attract the attention of many information science and psychology researchers. More and more researchers are focusing on emotion as a derivative of intelligence, hoping to use affective computing research to recreate biological emotional intelligence and endow machines with rich emotions and higher intelligence.Visual sentiment analysis is an important part of the field of affective computing. It aims to use computers and specific algorithms to predict the psychological changes that people will experience when they see an image [1]. It can predict and control the emotional changes of readers when browsing and reading to a certain extent. It has important application prospects in the fields of social media analysis, multimedia analysis, and public opinion prediction. It has become an important research direction in the current field of affective computing.
[0003] Visual sentiment analysis, as a part of sentiment computing, shares some similarities with classic classification problems in machine learning, but also has its differences. The similarity lies in the fact that both visual sentiment analysis and classic classification problems require neural networks to have a certain ability to understand the semantic information in images, i.e., the model's ability to extract image features. Like most other classification problems, sentiment classification requires analyzing and understanding image content to arrive at a classification result. The difference lies in the significant gap between the emotional space and the visual feature space. Classic classification problems identify concrete objects existing in the image's visual space, while the key to visual sentiment analysis is how to map from the concrete image space to the abstract emotional space, simulating the process of humans generating emotions when viewing images, thereby deriving emotional classification labels. Therefore, the process of visual sentiment analysis must maintain objectivity while referencing and reproducing human thought processes and characteristics, completing the image sentiment computing task from multiple perspectives and multiple thinking modes. Summary of the Invention
[0004] To address the shortcomings of existing technologies, this invention proposes an image sentiment analysis method based on bidirectional connectivity.
[0005] This invention designs a sentiment analysis method based on bidirectional feature connections to improve the accuracy of abstract classification problems by expanding the breadth of the network. First, two deep convolutional neural networks (CNNs) are used as the feature extraction backbone networks to obtain feature tensors of different scales during the vertical stacking of modules. Then, the two models are connected in parallel, and multi-level horizontal connections are performed on feature tensors of the same scale to expand the extracted semantic information. Simultaneously, an attention module is used to add attention features to the feature information, and convolution is used for layer-by-layer downsampling and jointing to finally obtain a complex feature vector output from the two bidirectionally combined deep CNNs. Finally, a classification network outputs the sentiment classification result. This sentiment classification method using bidirectional feature connections strengthens the feature extraction capability of the backbone network, absorbs the advantages of feature pyramids in comprehensively considering feature maps of different scales, and breaks the linear structure of a single classification network that repeatedly stacks modules, making it more suitable for abstract sentiment classification problems.
[0006] A bidirectional connectivity-based image sentiment analysis method, which specifically includes the following steps:
[0007] Step 1: Deep neural network weight pre-training and image dataset preprocessing;
[0008] Step 2: Complete feature merging through the deep feature lateral connection module;
[0009] Multiple deep neural network models are laterally connected through a deep feature lateral connection module to complete feature merging; the multiple deep neural network models have convolutional layers with the same output size; the deep feature lateral connection module consists of convolutional layers, connection layers, and SE attention modules;
[0010] Step 3: Perform joint feature fusion using the vertical feature concatenation module;
[0011] The joint features obtained by horizontally connecting deep features are used as input to the cascaded structure, and joint features of different sizes are fused in a manner similar to a feature pyramid. An attention layer is also used to add channel attention to improve model accuracy. The vertical joining method is similar to the feature pyramid structure, where the joint feature vectors are vertically joined from shallow to deep. This process is represented as:
[0012]
[0013]
[0014]
[0015] In the formula Q I f is the output of the I-th layer of the joint feature pyramid; Q () represents the downsampling operation in the joint feature pyramid. The joint feature pyramid is the feature pooling layer; Avgpool() is the global average pooling layer; Out is the overall output feature vector of the joint feature pyramid; P I This is the input tensor of the I-th layer of the deep feature lateral connection module;
[0016] Step 4: After the vertically cascaded module, a classification network is formed by two fully connected layers with 4096 neurons each, one fully connected layer with 1000 neurons each, and one fully connected layer with 2 neurons and the sigmoid activation function.
[0017] Step 5: Obtain the sentiment binary classification result by passing the normalized image through steps 2-4.
[0018] Preferably, the deep neural network weight pre-training and dataset preprocessing specifically include:
[0019] Step 1-1: Train a deep neural network feature extraction model on the ImageNet image dataset to enhance the basic image perception capabilities of the image sentiment analysis model;
[0020] Steps 1-2: For the target dataset, normalize each sample in the dataset to the same size, and expand the sample size in the training set of each cross-validation by image processing; use the pre-trained model on the processed dataset to extract the depth features of each sample image.
[0021] Preferably, the number of deep neural network models is two: a deep residual network and a neural structure search network.
[0022] Preferably, the method of laterally connecting the deep residual network and the neural structure search network model through the deep feature lateral connection module specifically includes:
[0023] For the input image Y, feature extraction is performed using deep convolutional models Res-Net and NAS-Net to obtain its output at each complete module layer, and then connected using lateral connection modules. The lateral connection part concatenates the outputs of the 3rd, 4th, and 5th RES modules of the deep residual network with the output features of the 2nd, 3rd, and 4th NAS modules of the neural architecture search network. To reduce computational cost, a 1×1 convolutional layer is used to process the features to be merged before lateral connection at each unit, enabling cross-channel information exchange while limiting channel expansion. This process is represented as:
[0024]
[0025] In the formula, PI is the input tensor of the I-th layer; FA ,I FB represents the output tensor of the i-th specific layer of model A. ,I fA and fB represent the output tensor of the i-th specific layer of model B; fA and fB are the normalization operations of model A and model B, consisting of convolutional layers with kernel sizes of 1×1 and 3×3, used to add nonlinearity to the joint feature tensor. denoted as joint feature; g is a normalization operation used to limit the number of channels of the joint feature tensor, consisting of a convolutional layer with a kernel size of 1×1; through this step, several joint feature vectors are obtained from the input image Y for subsequent use.
[0026] Preferably, in the process of vertical joining, the first horizontal joint input tensor requires an additional convolutional layer with a kernel size of 3×3 for processing, while the joint input tensors of other layers are processed by a convolutional layer with a kernel size of 1×1. The purpose is to limit the expansion of the number of feature channels in the process of vertical joining. After the vertical cascade module, a large-scale joint feature vector is obtained.
[0027] Beneficial results of the present invention:
[0028] 1. For abstract target classification problems such as image sentiment analysis, the model's breadth is expanded by using a lateral connection feature extraction model to replace the deep module stacking in traditional target classification problems, thereby enhancing the model's perceptual surface and its ability to perceive abstract features.
[0029] 2. An improved vertical feature connection model based on FPN is used to vertically cascade multi-layer horizontal joint features. This absorbs the advantages of feature pyramids in comprehensively considering feature maps of different scales, breaks the linear structure of repeated stacking of modules in a single classification network, and makes it more suitable for abstract sentiment classification problems, thus improving the accuracy of the model in sentiment classification problems. Attached Figure Description
[0030] Figure 1 This is an overall flowchart of the present invention;
[0031] Figure 2 This diagram illustrates the formation process of horizontal joint features and the network structure of horizontal connection modules.
[0032] Figure 3 This is a network structure diagram of the vertical cascading process and the vertical connection modules; Detailed Implementation
[0033] This invention proposes a deep learning-based image sentiment analysis method. The overall structure of the method is as follows: Figure 1 As shown, the main content includes two parts: a feature lateral connection module based on a parallel network of bidirectional feature connections and a feature pyramid-based vertical connection module. First, the lateral connection module uses two pre-trained backbone networks on large datasets to extract features, and then laterally connects the feature tensors extracted by the feature extraction networks at different module levels, using feature tensors of the same feature scale from the two models. Second, the vertical connection module concatenates the joint features obtained from the lateral connection module vertically. The vertical connection adopts a joint feature pyramid structure similar to a feature pyramid, obtaining the output features of the joint feature pyramid through progressive convolutional downsampling, and using an attention module to add channel-level attention. Finally, a classification network composed of global average pooling layers and fully connected layers is used to obtain the classification result.
[0034] The present invention will now be described in detail with reference to the accompanying drawings and examples. The specific steps are as follows:
[0035] Step 1: Deep neural network weight pre-training and dataset preprocessing.
[0036] We used Res-Net and NAS-Net to pre-train on a large dataset on ImageNet to enhance the basic image perception capabilities of the image sentiment analysis model. For reproducibility, we used the pre-trained weights provided by the official model as the initial weights.
[0037] Furthermore, the dataset is split into a training set and a test set. The training set samples are used as the training target, and the test set samples are used to validate the model. Since the number of training set samples is relatively small, the sample capacity is expanded through simple data augmentation. Specifically, the sample capacity is expanded to 3 times by randomly scaling, randomly translating, and randomly rotating each training set sample to meet the training requirements.
[0038] Step 2: Lateral connection of deep features.
[0039] Deep feature lateral connection is performed using a deep feature lateral connection module. For the input image Y, features are extracted using deep convolutional models Res-Net and NAS-Net to obtain the output of each complete module layer, and then connected using the lateral connection module. Specifically, as follows... Figure 2 As shown, the lateral connection part concatenates the outputs of the 3rd, 4th, and 5th RES modules of the Deep Residual Network (Res-Net) with the output features of the 2nd, 3rd, and 4th NAS modules of the Neural Architecture Search Network (NAS-Net). To reduce computational cost, a 1×1 convolutional layer is used to process the features to be merged before each unit performs the lateral connection, enabling cross-channel information exchange while limiting the expansion of the number of channels. This process can be represented as:
[0040]
[0041] In the formula, PI is the input tensor of the I-th layer; FA ,I fA represents the output tensor of the i-th specific layer of model A; fA is the normalization operation of model A, consisting of convolutional layers with kernel sizes of 1×1 and 3×3, used to add nonlinearity to the joint feature tensor; This represents the joint feature vector; g is the normalization operation used to limit the number of channels in the joint feature tensor, consisting of convolutional layers with a kernel size of 1×1. Through this step, several joint feature vectors can be obtained from the input image Y for subsequent use.
[0042] Step 3: Vertical feature concatenation and classification result output
[0043] Use a vertical cascading module to perform vertical cascading and use a classification network to output the classification results. For example... Figure 3As shown, for the joint feature vectors obtained in step 2, a network structure similar to a feature pyramid is used to merge them level by level. The difference between the two is that the input to the feature pyramid comes from the output tensor of a specific layer of a convolutional neural network, while the input to the vertical cascade module comes from the output tensor obtained by horizontally connecting layers of the same size in two different neural networks, and then they are vertically combined. The vertical combination method is similar to the feature pyramid structure, where the joint feature vectors are vertically combined from shallow to deep. This process can be represented as:
[0044]
[0045]
[0046]
[0047] In the formula, QI is the output of the I-th layer of the joint feature pyramid; fQ() is the downsampling operation in the joint feature pyramid, which mainly consists of a convolutional layer with a kernel size of 1×1 and a pooling layer with a stride of 2, in order to realize cross-space information exchange and determine the training weights between joint feature vectors. The first layer is the concatenate layer; the second layer is the global average pooling layer; and the third layer is the overall output feature vector of the joint feature pyramid. A classification network maps the feature vector to the classification result. During the vertical concatenation process, the first horizontally concatenated input tensor requires an additional 3×3 convolutional layer for processing. The input tensors of other layers are processed by a single 1×1 convolutional layer to limit the expansion of the number of feature channels during vertical concatenation. After the vertical concatenation module, we obtain a large-scale joint feature vector.
[0048] Furthermore, a classification network is used to process the joint feature vector. The classification network consists of two fully connected layers with 4096 neurons each and one fully connected layer with 1000 neurons. Finally, a binary classification result of sentiment is output through a fully connected layer with 2 neurons and a sigmoid activation function.
[0049] We validated our image sentiment analysis results through both ablation and comparative experiments. The experimental datasets used were the Twitter I dataset and the Artphoto dataset, both well-known public datasets in the field of image sentiment analysis. The ablation results are as follows:
[0050] Table 1 Ablation Experiment Based on Bidirectional Feature Connectivity Visual Emotion Analysis
[0051]
[0052]
[0053] Ablation experiments demonstrate that the bidirectional connection-based image sentiment analysis method proposed in this invention achieves higher classification accuracy compared to methods that directly use deep neural network models or simply connect the last layer of a deep neural network, thus confirming the superiority of this invention. Comparative experiments were conducted using some of the more advanced methods in the field of image sentiment analysis in recent years, and the comparison results are as follows:
[0054] Table 2 Comparative Experiments
[0055]
[0056] Comparative experiments show that the present invention performs relatively well on the ArtPhoto and Twitter I datasets. Furthermore, the method presented in this paper is an improved scheme for feature extraction and feature connection, which is not entirely contradictory to existing image sentiment classification methods and has good development prospects.
Claims
1. An image sentiment analysis method based on bidirectional connectivity, characterized in that, The method specifically includes the following steps: Step 1: Deep neural network weight pre-training and image dataset preprocessing; Step 2: Complete feature merging through the deep feature lateral connection module; Two deep neural network models are laterally connected through a deep feature lateral connection module to merge features; multiple deep neural network models have convolutional layers with the same output size; the deep feature lateral connection module connects the output features of corresponding layers of the two deep neural network models together; the deep feature lateral connection module consists of convolutional layers, connection layers, and SE attention modules; Step 3: Perform joint feature fusion using the vertical feature concatenation module; The joint features obtained by horizontally connecting deep features are used as input to the cascaded structure, and joint features of different sizes are fused in a manner similar to a feature pyramid. An attention layer is also used to add channel attention to improve model accuracy. The vertical joining method is similar to the feature pyramid structure, where the joint feature vectors are vertically joined from shallow to deep. This process is represented as: (6) (7) (8) In the formula This is the output of the I-th layer of the joint feature pyramid; For downsampling operations in the joint feature pyramid, For feature joint layer; This is a global average pooling layer; This is the overall output feature vector of the joint feature pyramid; For the first deep feature lateral connection module The input tensor of the layer; Step 4: After the vertically cascaded module, a classification network is formed by two fully connected layers with 4096 neurons each, one fully connected layer with 1000 neurons each, and one fully connected layer with 2 neurons and the sigmoid activation function. Step 5: Obtain the sentiment binary classification result by passing the normalized image through steps 2-4.
2. The image sentiment analysis method based on bidirectional connectivity according to claim 1, characterized in that: The aforementioned deep neural network weight pre-training and dataset preprocessing specifically include: Step 1-1: Train a deep neural network feature extraction model on the ImageNet image dataset; To enhance the basic image perception capabilities of image sentiment analysis models; Steps 1-2: For the target dataset, normalize each sample in the dataset to the same size, and expand the sample size in the training set of each cross-validation by image processing; use the pre-trained model on the processed dataset to extract the depth features of each sample image.
3. The image sentiment analysis method based on bidirectional connectivity according to claim 1, characterized in that: The deep neural network model described is a deep residual network and a neural structure search network.
4. The image sentiment analysis method based on bidirectional connectivity according to claim 3, characterized in that: The method of laterally connecting the deep residual network and the neural structure search network model through the deep feature lateral connection module is as follows: For the input image Y, feature extraction is performed using deep convolutional models Res-Net and NAS-Net to obtain its output at each complete module layer, and then connected using lateral connection modules. The lateral connection part concatenates the outputs of the 3rd, 4th, and 5th RES modules of the deep residual network with the output features of the 2nd, 3rd, and 4th NAS modules of the neural architecture search network. To reduce computational cost, a 1×1 convolutional layer is used to process the features to be merged before lateral connection at each unit, enabling cross-channel information exchange while limiting channel expansion. This process is represented as: , I =1...n(5) In the formula For the first The input tensor of the layer; Representative model The A specific layer output tensor Representative model The A specific layer output tensor; For the normalization operation of model A and model B, the kernel size is... and The convolutional layers are used to add nonlinearity to the joint feature tensor; Indicates feature union; To perform normalization operations that limit the number of channels in the joint feature tensor, the kernel size is... The convolutional layers consist of several layers; through this step, several joint feature vectors are obtained from the input image Y for subsequent use.
5. The image sentiment analysis method based on bidirectional connectivity according to claim 1, characterized in that: In the process of vertical joining, the first horizontal joint input tensor requires an additional convolutional layer with a kernel size of 3×3 for processing, while the joint input tensors of other layers are processed by a convolutional layer with a kernel size of 1×1. The purpose is to limit the expansion of the number of feature channels during the vertical joining process. After the vertical cascade module, a large-scale joint feature vector is obtained.