An image sentiment analysis method based on pre-training optimization
By defining low-level, mid-level, and high-level feature levels and conducting emotion-related training tasks, combined with feature map regularization and probabilistic activation regularization, the problem of insufficient pre-trained features in backbone networks for image sentiment analysis is solved, thereby improving the performance of sentiment analysis.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- NANKAI UNIV
- Filing Date
- 2023-03-08
- Publication Date
- 2026-05-15
AI Technical Summary
Existing image sentiment analysis methods suffer from poor sentiment recognition performance because the features learned by the backbone network during pre-training cannot adequately provide guidance for sentiment identification.
By defining low-level, mid-level, and high-level feature levels, emotion-related training tasks are performed respectively. The emotion-related prior knowledge in the pre-trained backbone network models of these feature levels is then aggregated into the downstream target backbone network, and feature map regularization and probabilistic activation regularization are used for knowledge aggregation.
It significantly improves the performance of downstream sentiment analysis tasks, enhancing the accuracy and effectiveness of sentiment recognition.
Smart Images
Figure CN116342934B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of computer vision technology, specifically relating to an image sentiment analysis method based on pre-training optimization. Background Technology
[0002] Image sentiment analysis (ISA) aims to understand the emotions conveyed in an image and has become an increasingly important computer vision task with applications in various fields such as opinion mining, entertainment assistance, and business intelligence. With the development of deep learning, deep neural networks (DNNs) have gradually become important tools in the field of image analysis, greatly promoting the development of image sentiment analysis. Currently, most image sentiment analysis models are built on DNNs. Generally, these methods use a backbone network for feature extraction, obtaining a multi-dimensional feature vector, and then use one or more fully connected neural networks for sentiment classification or to output the sentiment label distribution.
[0003] In the methods described above, the initialization of the backbone network is particularly crucial. The backbone network is typically pre-trained on a large upstream dataset and then fine-tuned in the downstream sentiment analysis task to ensure good sentiment recognition performance. However, if a randomly initialized model is directly used for training on the sentiment task, the lack of a large number of samples and prior knowledge of image feature extraction often leads to poor sentiment recognition results. In existing sentiment analysis methods, pre-training is often performed on the large image classification dataset ImageNet. This dataset primarily classifies objects. After pre-training on this dataset, the backbone network often learns how to capture features with object discrimination. While these features have some use in the downstream sentiment analysis task, they still cannot provide sufficient sentiment-oriented features, thus failing to fully realize the potential of the pre-trained model. Summary of the Invention
[0004] The purpose of this invention is to improve upon the shortcomings of existing sentiment pre-training methods by proposing a sentiment-oriented image sentiment analysis method based on pre-training optimization, thereby enhancing the performance of multiple downstream sentiment analysis tasks.
[0005] This invention is achieved through the following technical solution:
[0006] A pre-trained optimization-based image sentiment analysis method includes the following steps:
[0007] Step 1: First, define the three feature levels that affect image sentiment analysis: low-level, mid-level, and high-level. Each feature level contains image attributes related to sentiment.
[0008] Step 2: Based on the three feature levels that affect image sentiment analysis as defined in Step 1 and the image attributes of each layer, determine the corresponding sentiment training task, and then obtain the pre-trained backbone network models of the low-level, mid-level, and high-level feature levels respectively.
[0009] Step 3: Aggregate the sentiment-related prior knowledge from the three feature levels of the pre-trained backbone network models obtained in Step 2 into the same downstream target backbone network model.
[0010] In the above technical solution, in step 1, the low-level feature level image attributes include color and texture, the mid-level feature level image attributes include scene and geometric arrangement, and the high-level feature level image attributes include object state and logical reasoning.
[0011] In the above technical solution, in step 2, the color image attribute in the low-level feature level corresponds to the image coloring training task, and the texture image attribute in the low-level feature level corresponds to the super-resolution training task; the first backbone network model is used to train the two training tasks of the low-level feature level simultaneously, thereby obtaining the low-level pre-trained backbone network model.
[0012] The scene image attributes in the mid-level feature level correspond to the training task of scene classification, and the geometric arrangement image attributes in the mid-level feature level correspond to the training task of jigsaw puzzle. The second backbone network model is used to train the two training tasks of the above mid-level feature level simultaneously, thereby obtaining the mid-level pre-trained backbone network model.
[0013] The object state image attribute in the high-level feature level corresponds to the training task of adjective noun pair classification, and the logical reasoning image attribute in the high-level feature level corresponds to the training task of image description generation. The three backbone network models are used to train the two training tasks of the above high-level feature level simultaneously, thereby obtaining the high-level pre-trained backbone network model.
[0014] In the above technical solution, for the image colorization task, given any RGB image I∈R H×W×3 Where H is the height of the image and W is the width of the image, the image is converted to the LAB color space through color space conversion, where the brightness matrix X in the L space is... cr ∈R H×W×1 It is the input to the model, and the color matrix Y in the AB space. cr ∈R H×W×2 This is the model's prediction target, and the training loss uses the pixel-level average mean squared error.
[0015] In the above technical solution, for the super-resolution task, given an RGB image X sr ∈R H×W×3 Where H is the height of the image and W is the width of the image, the image is first downsampled to the original image. The data is then fed into the network for feature extraction, followed by upsampling using transposed convolution, and the training loss is calculated using pixel-level average mean square error.
[0016] In the above technical solution, for the scene classification task, the label of each image is Y. sc ∈{0,1} C The model predicts the probability for each category as follows: Model training is performed using cross-entropy loss.
[0017] In the above technical solution, for the jigsaw puzzle task, given an image, it is uniformly divided into 9 sub-blocks of 3×3 size. Then, the order of the sub-blocks is randomly shuffled, and all the shuffled sub-blocks are input into the model to predict the shuffle index. Let Y be the index label of the shuffling method used for an image. jp ∈{0,1} C Where C represents the pre-defined disorder type, and the probability of each disordered index predicted by the model is... Then the training loss is Where L jp It is the training loss of the jigsaw puzzle task.
[0018] In the above technical solution, the prior knowledge aggregation described in step 3 is achieved by introducing feature map regularization and probabilistic activation regularization during the image sentiment analysis training process.
[0019] In the above technical solution, for feature map regularization, the feature extraction of the downstream target backbone network model and the pre-trained backbone network models at each level is first divided into three stages. The feature map output by each stage includes three dimensions: length, width, and channels. This is the feature map of the downstream target backbone network model in the i-th stage. These are the feature maps of the low-level, mid-level, and high-level pre-trained backbone network models at the i-th stage, respectively, where C... i H i W i These represent the number of channels, length, and width of the feature map in the i-th stage, respectively.
[0020] The feature map regularization consists of three groups: regularization for the downstream target backbone network model and the low-level pre-trained backbone network model in the first stage; regularization for the downstream target backbone network model and the mid-level pre-trained backbone network model in the second stage; and regularization for the downstream target backbone network model and the high-level pre-trained backbone network model in the third stage. The overall feature map regularization training loss is calculated as follows:
[0021]
[0022] In the above technical solution, the probabilistic activation regularization is applied between the downstream target backbone network model and the high-level pre-trained backbone network model. Let the output probabilistic activations of the downstream target backbone network model and the three pre-trained backbone network models be L. b ,L l ,L m ,L h ∈R N Here, N is the number of sentiment categories in the image sentiment analysis training. The calculation method for the probabilistic activation regularization training loss is as follows:
[0023] In the above technical solution, during the process of performing prior knowledge aggregation for image sentiment analysis training, all pre-trained backbone network models and downstream target backbone network models perform loss calculations for image sentiment analysis training and perform gradient backpropagation updates.
[0024] The advantages and beneficial effects of this invention are as follows:
[0025] This invention addresses the problem that previous applications of deep neural networks in image sentiment analysis lacked sentiment-related pre-training parameters. The proposed sentiment-oriented pre-training method can fully capture sentiment-related features at various semantic levels. Furthermore, this invention proposes a knowledge aggregation method. When training downstream sentiment tasks, for the pre-trained backbone networks obtained after the pre-training phase, this aggregation method can combine the sentiment prior knowledge from different semantic levels carried by these backbone networks into the same downstream backbone network, thereby improving the performance of the downstream backbone network in sentiment analysis. Experiments were conducted on numerous sentiment analysis tasks and corresponding datasets, and the results show that, compared to previous pre-training methods, this invention can significantly improve the performance of these tasks. Attached Figure Description
[0026] Figure 1 This is a general implementation block diagram of the present invention.
[0027] Figure 2 This is a schematic diagram illustrating the implementation methods of various pre-training tasks in the pre-training phase of this invention.
[0028] Figure 3 This is a schematic diagram illustrating the knowledge aggregation operation performed on the input training images according to the present invention.
[0029] Figure 4 This is a visualization of the prediction results of the present invention in three sentiment analysis tasks.
[0030] For those skilled in the art, other related figures can be obtained from the above figures without any creative effort. Detailed Implementation
[0031] To enable those skilled in the art to better understand the present invention, the technical solution of the present invention will be further described below with reference to specific embodiments.
[0032] The image sentiment analysis method based on pre-training optimization proposed in this invention includes the following steps:
[0033] Step 1: First, define the feature levels and image attributes that affect image sentiment analysis at three levels.
[0034] We define three feature levels affecting image sentiment analysis: low-level, mid-level, and high-level. Each feature level contains two key representative image attributes that are closely related to sentiment. The low-level image attributes include color and texture, the mid-level image attributes include scene and geometric arrangement, and the high-level image attributes include object state and logical reasoning.
[0035] Step 2: Determine the corresponding sentiment training task based on the three-layer feature levels affecting image sentiment analysis defined in Step 1 and the image attributes of each layer. See Table 1 for specific task information.
[0036] Table 1
[0037]
[0038] Table 1 contains information such as feature level, image attributes, training task, dataset, image type, and supervision type. The training tasks include image colorization, super-resolution, scene classification, jigsaw puzzle, adjective / noun pair classification, and image description generation. The datasets include Places365, VSO, and ArtEmis. The image types include scene photographs, web images, and artwork. Specifically, Places365 contains scene photographs, VSO contains web images, and ArtEmis contains artwork. The supervision types include fully supervised and self-supervised supervision.
[0039] The implementation method for each training task is as follows: Figure 2 As shown below, each training task will be explained in detail.
[0040] Step 2.1: The color image attribute in the low-level feature level corresponds to the image coloring training task, using scene photos from Places365 as the dataset, with self-supervised supervision. The texture image attribute in the low-level feature level corresponds to the super-resolution training task, also using scene photos from Places365 as the dataset, with self-supervised supervision. The first backbone network model (e.g., ResNet50 model) is used to simultaneously train the two training tasks at the low-level feature level, thus obtaining the low-level pre-trained backbone network model.
[0041] Specifically, the image colorization task is the process of converting a black and white image into a color image, and the super-resolution task is the process of sampling a given low-resolution image into a high-resolution image. Both training tasks are implemented using a fully convolutional neural network architecture.
[0042] For the image coloring task, given any RGB image I∈R H×W×3 Where H is the height of the image and W is the width of the image, the image is converted to the LAB color space through color space conversion, where the brightness matrix X in the L space is... cr ∈R H×W×1 The input to the model is a black and white image, while the color matrix Y in the AB space is... cr ∈R H×W×2 This is the model's prediction target; the training loss uses pixel-level average mean squared error.
[0043] in, It is the model's predicted output, L cr It is the training loss for the image coloring task.
[0044] For super-resolution tasks, given an RGB image X sr ∈R H×W×3 Where H is the height of the image and W is the width of the image, the image is first downsampled to the original image. The image is then fed into a network for feature extraction, followed by upsampling using transposed convolution to restore the original image. Assume the network output is... Calculate the training loss using pixel-level average mean squared error:
[0045] Where L sr It is the training loss for super-resolution tasks.
[0046] Step 2.2: The scene image attributes in the mid-level feature hierarchy correspond to the scene classification training task, using scene photos from Places365 as the dataset, with fully supervised supervision. The geometric arrangement image attributes in the mid-level feature hierarchy correspond to the jigsaw puzzle training task, using scene photos from Places365 as the dataset, with self-supervised supervision. The second backbone network model is used to simultaneously train the two training tasks at the mid-level feature hierarchy, thus obtaining the mid-level pre-trained backbone network model.
[0047] Specifically, for a scene classification task, let the label of each image be Y. sc ∈{0,1} C Where C = 1000 represents the pre-defined number of scene categories, and the prediction probability of the classification model for each scene category is... Model training using cross-entropy loss:
[0048] Where L sc It represents the training loss for the scene recognition task.
[0049] For the jigsaw puzzle task, given an image, it is uniformly divided into 9 sub-blocks (3x3), and then the order of the sub-blocks is randomly shuffled. All the shuffled sub-blocks are then input into the network to predict the shuffled index. The jigsaw puzzle task can be defined as a classification task. Assume the shuffling method used on an image is indexed by the label Y. jp ∈{0,1} C Where C = 1000 is a pre-set type of out-of-order index, and the probability of each out-of-order index predicted by the network is... Therefore, the training loss is:
[0050] Where L jp It is the training loss of the jigsaw puzzle task.
[0051] Step 2.3: The object state image attribute in the high-level feature hierarchy corresponds to the training task of adjective-noun pair classification. The dataset uses the network images of VSO, and the supervision type is fully supervised. The logical reasoning image attribute in the high-level feature hierarchy corresponds to the training task of image description generation. The dataset uses the artworks from ArtEmis, and the supervision type is fully supervised. The third backbone network model is used to train the two training tasks of the above high-level feature hierarchy simultaneously, thereby obtaining the high-level pre-trained backbone network model.
[0052] Specifically, the classification of adjective-noun pairs is similar to object recognition and classification, but the difference lies in the more refined classification of adjective-noun pairs. For objects of the same category, they are further subdivided into multiple categories based on the object's state, such as sad cat, happy cat, angry cat, etc. This invention uses the VSO dataset for training this task, where all adjectives are sentiment adjectives, which is more conducive to capturing sentiment features; the adjective-noun pair classification task is a type of classification task.
[0053] Image caption generation is a cross-modal task that aims to automatically generate a natural language description of the content of a given image using a model. This invention uses a convolutional network to encode features on the input image, and the decoded result is the natural language description. This task is also defined as a classification task.
[0054] Step 3: After completing the training tasks in Step 2, three levels of pre-trained backbone network models—low-level, mid-level, and high-level—are obtained. Then, a knowledge aggregation method is used to aggregate the sentiment-related prior knowledge from these three pre-trained backbone network models into the same downstream target backbone network model.
[0055] Specifically, the prior knowledge aggregation is achieved by introducing feature map regularization and probabilistic activation regularization during the image sentiment analysis training process.
[0056] For feature map regularization, the feature extraction of the downstream target backbone network model and the pre-trained backbone network models at each level is first divided into three stages. The feature map output by each stage includes three dimensions: length, width, and channels. Let... This is the feature map of the downstream target backbone network model in the i-th stage. These are the feature maps of the low-level, mid-level, and high-level pre-trained backbone network models at the i-th stage, respectively, where C... i H i W i These represent the number of channels, length, and width of the feature map in the i-th stage, respectively.
[0057] The feature map regularization consists of three groups: regularization for the downstream target backbone network model and the low-level pre-trained backbone network model in the first stage; regularization for the downstream target backbone network model and the mid-level pre-trained backbone network model in the second stage; and regularization for the downstream target backbone network model and the high-level pre-trained backbone network model in the third stage. The overall feature map regularization training loss is calculated as follows:
[0058]
[0059] in
[0060] The probabilistic activation regularization is applied between the downstream target backbone network model and the high-level pre-trained backbone network model. Let the output probabilistic activations of the downstream target backbone network model and the three pre-trained backbone network models be L. b ,L l ,L m ,L h ∈R N Here, N is the number of sentiment categories in the image sentiment analysis training. The calculation method for the probabilistic activation regularization training loss is as follows:
[0061]
[0062] Furthermore, during the prior knowledge aggregation process of image sentiment analysis training, in order to enable each level of pre-trained backbone network model to adaptively update its parameters based on the data characteristics of the image sentiment analysis training, all pre-trained backbone network models and downstream target backbone network models perform loss calculations and gradient backpropagation updates for the image sentiment analysis training. The target loss of each level of pre-trained backbone network model and the downstream target backbone network model is determined by the image sentiment analysis training, using L... tarl ,L tarm ,L tarh ,L tarb These represent the training losses of the low-level pre-trained backbone network model, the mid-level pre-trained backbone network model, the high-level pre-trained backbone network model, and the downstream target backbone network model, respectively.
[0063] In summary, the overall loss L in the process of aggregating prior knowledge during image sentiment analysis training is:
[0064] L = L fr +L lr +L tarl +L tarm +L tarh +L tarb .
[0065] Referring to Table 2, the technical effects of the present invention can be illustrated through experiments.
[0066] Table 2
[0067]
[0068] Table 2 shows the performance of this invention in three downstream sentiment analysis tasks: single-label classification, multi-label classification, and label distribution. Representative datasets were selected for each task, including five commonly used sentiment analysis datasets: FI, UnbiasedEmo (UB), Emotic (EM), Emotion6 (E6), and Abstract (AB). The single-label classification task used the Accuracy (Acc) metric, the multi-label classification task used Hamming, Ranking, MicroF1, and MacroF1, and the label classification task used Chebyshev, Clark, Canberra, Kullback-Leibler (KL), Cosine, and Intersection. For each metric, an "↑" symbol indicates a higher metric and a better model performance, while a "↓" symbol indicates a lower metric and a better model performance. The backbone network used in the experiments covered various architectures, including five different structures and sizes of convolutional networks: Vgg16, Vgg19, ResNet18, ResNet50, and ResNet101. The numbers on the left of Table 2 represent the performance using conventional pre-training methods, i.e., pre-training on the ImageNet dataset, then directly loading the trained model parameters for retraining on downstream sentiment analysis tasks, and finally testing the model performance on the dataset. The numbers on the right of Table 2 represent the performance using the pre-training and aggregation methods proposed in this invention. As can be seen from the tables, compared to traditional pre-training methods, the sentiment-oriented pre-training and knowledge aggregation methods proposed in this invention can effectively improve the performance of different models on various sentiment analysis tasks. Figure 4 The invention demonstrates the prediction format and visualization results for a subset of samples in three downstream sentiment analysis tasks.
[0069] The present invention has been described above by way of example. It should be noted that any simple modifications, alterations or other equivalent substitutions that can be made by those skilled in the art without creative effort without departing from the core of the present invention fall within the protection scope of the present invention.
Claims
1. An image sentiment analysis method based on pre-training optimization, characterized in that, Includes the following steps: Step 1: First, define the three feature levels that affect image sentiment analysis: low-level, mid-level, and high-level. Each feature level contains image attributes related to sentiment. Low-level feature-level image attributes include color and texture; mid-level feature-level image attributes include scene and geometric arrangement; and high-level feature-level image attributes include object state and logical reasoning. Step 2: Based on the three feature levels that affect image sentiment analysis as defined in Step 1 and the image attributes of each layer, determine the corresponding sentiment training task, and then obtain the pre-trained backbone network models of the low-level, mid-level, and high-level feature levels respectively. Step 3: Aggregate the sentiment-related prior knowledge from the three feature levels of the pre-trained backbone network model obtained in Step 2 into the same downstream target backbone network model. The aforementioned prior knowledge aggregation is achieved by introducing feature map regularization and probabilistic activation regularization during the image sentiment analysis training process; For feature map regularization, the feature extraction of the downstream target backbone network model and the pre-trained backbone network models at each level is first divided into three stages. The feature map output by each stage includes three dimensions: length, width, and channels. Let... For the downstream target backbone network model in the first Feature maps of each stage These represent the low-level pre-trained backbone network model, the mid-level pre-trained backbone network model, and the high-level pre-trained backbone network model, respectively, at the [missing information - likely a specific stage or timeframe]. Feature maps of each stage, where , , The first The number of channels, length, and width of the feature map at each stage; The feature map regularization consists of three groups: regularization for the downstream target backbone network model and the low-level pre-trained backbone network model in the first stage; regularization for the downstream target backbone network model and the mid-level pre-trained backbone network model in the second stage; and regularization for the downstream target backbone network model and the high-level pre-trained backbone network model in the third stage. The overall feature map regularization training loss is calculated as follows: ； The probabilistic activation regularization is applied between the downstream target backbone network model and the high-level pre-trained backbone network model. Let the output probabilistic activations of the downstream target backbone network model and the three pre-trained backbone network models be respectively... ,here If the number of sentiment categories is used in image sentiment analysis training, then the calculation method for the probabilistic activation regularization training loss is as follows: 。 2. The image sentiment analysis method based on pre-training optimization according to claim 1, characterized in that: In step 2, the color image attribute in the low-level feature level corresponds to the image coloring training task, and the texture image attribute in the low-level feature level corresponds to the super-resolution training task; the first backbone network model is used to train the two training tasks of the low-level feature level simultaneously, thereby obtaining the low-level pre-trained backbone network model. The scene image attributes in the mid-level feature level correspond to the training task of scene classification, and the geometric arrangement image attributes in the mid-level feature level correspond to the training task of jigsaw puzzle. The second backbone network model is used to train the two training tasks of the above mid-level feature level simultaneously, thereby obtaining the mid-level pre-trained backbone network model. The object state image attribute in the high-level feature level corresponds to the training task of adjective noun pair classification, and the logical reasoning image attribute in the high-level feature level corresponds to the training task of image description generation. The three backbone network models are used to train the two training tasks of the above high-level feature level simultaneously, thereby obtaining the high-level pre-trained backbone network model.
3. The image sentiment analysis method based on pre-training optimization according to claim 2, characterized in that: For image coloring tasks, given any RGB image... ,in It is the height of the image. This refers to the width of the image. The image is converted to the LAB color space using color space conversion, where the L space represents the brightness matrix. It is the input to the model, and the color matrix in the AB space. This is the model's prediction target, and the training loss uses the pixel-level average mean squared error.
4. The image sentiment analysis method based on pre-training optimization according to claim 2, characterized in that: For super-resolution tasks, given an RGB image ,in It is the height of the image. It is the width of the image. First, downsample the image to the width of the original image. The data is then input into the network for feature extraction, followed by upsampling using transposed convolution, and the training loss is calculated using pixel-level average mean square error.
5. The image sentiment analysis method based on pre-training optimization according to claim 2, characterized in that: For the jigsaw puzzle task, given an image, cut it evenly into... common The image is divided into several sub-blocks, and their order is randomly shuffled. All the shuffled sub-blocks are then input into the model to predict the shuffled index. Let the shuffling method used for an image be denoted as . , in For each pre-defined type of out-of-order index, the model predicts the probability of each type of out-of-order index as follows: Then the training loss is ,in It is the training loss of the jigsaw puzzle task.
6. The image sentiment analysis method based on pre-training optimization according to claim 1, characterized in that: During the prior knowledge aggregation process of image sentiment analysis training, all pre-trained backbone network models and downstream target backbone network models perform loss calculations and gradient backpropagation updates for image sentiment analysis training.