Image sentiment analysis method fusing multi-dimensional features

Abstract

The invention relates to an image sentiment analysis method in the technical field of electronics. According to the method, image features extracted by a deep learning model ResNet-18, features extracted by a Haar cascade classifier and features extracted by OpenCV and Numpy are fused, and a multi-dimensional information fusion sentiment analysis system is constructed. The system extracts image features through a pre-trained ResNet-18 and maps the image features to different emotion categories to generate an initial score; face and smile detection is carried out by using a Haar cascade classifier, and a smile detection result reflects the pleasure degree of figures in the image; if the human face is detected and the smile is not detected, using Canny edge detection and Sobel gradient analysis to judge the negative degree of the figure in the image; calculating the brightness, contrast, texture complexity, symmetry, color variance, blue tone, dark tone and other attributes of the image by means of OpenCV and Numpy libraries, and dynamically adjusting the emotion score; and finally, the relationship between the emotion score and the image feature is displayed in a graphical mode by using a knowledge graph, so that the accuracy and interpretability of emotion analysis are improved.

Description

Technical Field

[0001] Electronic technology field: Description: The present invention relates to the field of electronic technology, and in particular to a method for fusing image features extracted by a deep learning model ResNet-18, features extracted by a Haar cascade classifier, and features extracted by OpenCV and Numpy, and a method for visualizing the results using a knowledge graph. Background Art

[0002] The field of image sentiment analysis currently faces numerous challenges. Methods that leverage the physical properties of images primarily rely on analyzing physical attributes such as color distribution and texture. These methods suffer from significant accuracy limitations. Analyzing only certain physical properties of an image may not accurately distinguish between different emotions, as the same physical property may correspond to multiple emotions, leading to biased analysis results. While models such as convolutional neural networks have demonstrated powerful capabilities for image feature extraction with the advancement of deep learning technology, using only deep learning models for sentiment analysis also presents certain limitations. Deep learning models typically require extensive amounts of labeled data for training, but acquiring large-scale sentiment-labeled datasets is often challenging. Furthermore, deep learning models are often viewed as "black boxes," making their decision-making processes difficult to interpret. This makes it difficult for users to understand how the models derive emotion from image features. Consequently, existing sentiment analysis methods often lack a clear visual representation of the relationship between emotion and image features, making it difficult for users to gain a deeper understanding of sentiment analysis results. Summary of the Invention

[0003] This invention aims to address the shortcomings of existing image sentiment analysis methods by providing an image sentiment analysis method that integrates multidimensional features. By fusing image features extracted by the deep learning model ResNet-18, features extracted by a Haar cascade classifier, and features extracted by OpenCV and Numpy, the accuracy of sentiment analysis is improved. Furthermore, knowledge graph technology is used to enhance interpretability and the logic of conclusion analysis, enabling visual display of sentiment analysis results. The technical solution of this invention includes the following aspects.

[0004] The first step is to develop the system architecture. This involves building an image sentiment analysis system that includes an image preprocessing module, a feature extraction module, a sentiment analysis module, a knowledge graph construction module, and a front-end interactive interface. The image preprocessing module performs operations such as resizing, cropping, and normalizing the input image to adapt it to the needs of subsequent modules. The feature extraction module extracts image features using image processing techniques such as the ResNet-18 model and the Haar cascade classifier. The sentiment analysis module calculates an initial sentiment score based on the extracted features and dynamically adjusts the sentiment score based on expression detection results and image features. The knowledge graph construction module graphically displays the sentiment score, image features, and the relationships between them, forming a knowledge graph.

[0005] The second step is feature extraction and fusion. A pre-trained ResNet-18 model is used to extract image features. The model's last fully connected layer is removed, retaining the convolutional layer as the feature extractor. The extracted feature vectors are evenly distributed across different emotion categories, and an initial emotion score is calculated for each category. A Haar cascade classifier is used for face and smile detection. If a face is detected but a smile is not, the system performs upper and lower face difference analysis, eyebrow region edge density detection, mouth contour roundness detection, Canny edge detection, and Sobel gradient analysis to determine negative emotion. The emotion score is adjusted based on the detection results. If a smile is detected, the happy emotion score is increased based on the confidence level, while the negative emotion score is decreased. If a negative emotion is detected, the corresponding negative emotion score is increased based on the confidence level, while the happy emotion score is decreased. Then, the physical features of the image are extracted, such as brightness, contrast, color variance, texture complexity, symmetry, blue tone, dark tone, etc. These features are closely related to emotional expression. By analyzing these features, the emotional score is further adjusted. High brightness enhances the emotional weights of happiness and neutrality, high contrast enhances the emotional weights of anger and surprise, high color variance enhances the emotional weights of surprise, anger and happiness, high texture complexity enhances the emotional weights of anger, surprise and fear, high symmetry enhances the emotional weights of neutrality and happiness, high blue tone ratio enhances the emotional weights of sadness, disgust and fear, and high dark tone ratio enhances the emotional weights of sadness, fear and disgust.

[0006] The third step is to build and visualize the knowledge graph. Create a directed graph with images, emotions, features, etc. as nodes. The edge weight from the image node to the emotion node is the normalized emotion probability value. The edge weight from the image node to the feature node is the ratio of the feature value to the total feature value. The edge weight from the feature node to the emotion node is calculated according to predefined rules: Weight = (eigenvalue / total eigenvalue) × (sentiment score / total sentiment score) × rule factor The setting of rule factors is based on the correlation between image features and emotions, and is determined through experience, psychological research, and understanding of image emotional expression. Rule factors include: Texture complexity was correlated with anger, surprise, and fear, with correlation coefficients of 1.4, 1.3, and 1.1, respectively. Brightness was correlated with happiness and neutrality, with correlation coefficients of 1.2 and 1.1, respectively. Contrast was correlated with anger and surprise, with correlation coefficients of 1.8 and 1.6, respectively. The proportion of blue tones was correlated with sadness, disgust, and fear, with correlation coefficients of 1.6, 1.4, and 1.2, respectively. The proportion of dark tones was correlated with sadness, fear, and disgust, with correlation coefficients of 1.7, 1.5, and 1.3, respectively. Symmetry was correlated with happiness and neutrality, with correlation coefficients of 1.4 and 3, respectively. Color variance was correlated with surprise, anger, and happiness, with correlation coefficients of 1.9, 1.2, and 1.3, respectively.

[0007] Normalize the nodes and edges in the knowledge graph to ensure the rationality and comparability of the weights. Convert the knowledge graph data into the data format required by ECharts, including the name, type, size of the node, the source node, target node, and weight of the edge. Display the knowledge graph to the user through visualization technology, allowing users to intuitively understand the results and basis of image sentiment analysis.

[0008] Compared with existing technologies, this invention has the following advantages: Improved accuracy. By integrating image features extracted by the deep learning model ResNet-18, features extracted by the Haar cascade classifier, and features extracted by OpenCV and Numpy, it fully leverages the advantages of different features and improves the accuracy of sentiment analysis. Strong interpretability. Leveraging knowledge graph technology, the relationship between sentiment analysis results and image features is intuitively displayed, allowing users to clearly understand the basis of sentiment analysis and enhancing the interpretability of sentiment analysis. Enhanced logic. The connections between nodes and edges in the knowledge graph reflect the logical relationship between sentiment and features, making the conclusions of sentiment analysis more logical. BRIEF DESCRIPTION OF THE DRAWINGS Figure 1 is a flow chart of the method of the present invention; Figure 2 Detection flow chart for Haar cascade classifier; Figure 3 This is a diagram of the knowledge graph construction process; Figure 4 This is a visualization example diagram of the knowledge graph. DETAILED DESCRIPTION Step 1: Build the system, using Python as the development language and combining it with the PyTorch framework to load and extract the ResNet-18 model. Use the OpenCV library to implement face detection and expression detection, loading the pre-trained Haar cascade classifier file. Use the Numpy library to extract and calculate image physical features. Use the Flask framework to build a front-end interactive interface, allowing users to upload images and display sentiment analysis results and knowledge graphs to users. Use the NetworkX library to build a knowledge graph and convert it into a data format suitable for front-end visualization.

[0009] Step 2: Image preprocessing: resize the input image to a square with a side length of 256 pixels; capture a centered valid area of ​​224×224 pixels from the resized image; convert the image data into Tensor format and normalize it so that its value range is between "0" and "1".

[0010] Step 3: Feature extraction and sentiment analysis: Input the preprocessed image into the ResNet-18 model to extract image features and calculate the initial sentiment score; use OpenCV's Haar cascade classifier to detect faces and expressions in the image, and adjust the sentiment score based on the detection results; extract physical features of the image, such as brightness, contrast, color variance, etc., and further adjust the sentiment score based on these features; comprehensively analyze the features extracted by the deep learning model, the features extracted by the Haar cascade classifier, and the features extracted by OpenCV and Numpy to form a complete feature representation for sentiment analysis.

[0011] Step 4: Build a knowledge graph. Create a directed graph with images, emotions, features, etc. as nodes, and emotion scores and feature values ​​as node weights. Create edges between nodes based on the association rules between features and emotions, and calculate the weights of the edges. Normalize the nodes and edges in the knowledge graph to ensure the rationality and comparability of the weights. Convert the knowledge graph data into a knowledge graph format based on ECharts that is suitable for front-end visualization, and display it to users through the front-end interactive interface.

[0012] Step 5: Display the results of sentiment analysis on the front-end interactive interface, including information such as sentiment distribution, dominant sentiment, and key features; through the visualization of the knowledge graph, the relationship between sentiment and features is displayed, allowing users to intuitively understand the results and basis of sentiment analysis.

[0013] The image sentiment analysis system and analysis method of the present invention achieve high accuracy, strong interpretability and logic of sentiment analysis by integrating multidimensional features and knowledge graph technology, providing a new solution for the field of image sentiment analysis, with broad application prospects and important practical significance.

Claims

1. A method for image sentiment analysis integrating multidimensional features comprises the following steps: Step S1: Extract image features through the pre-trained ResNet-18 model; Step S2: Use Haar cascade classifier to detect face area and smile. If face is detected but smile is not detected, negative emotion detection is performed. Step S3: Integrate the features extracted by the deep learning model with the expression detection results, dynamically adjust the initial emotion score, and generate a probability distribution of seven categories of emotions, including anger, disgust, fear, happiness, neutrality, sadness, and surprise; Step S4: extracting physical features of the image, including brightness, contrast, color variance, texture complexity, symmetry, blue tone ratio, and dark tone ratio; Step S5: Construct a dynamic knowledge graph, where nodes are divided into four categories, including image nodes, sentiment nodes, feature nodes, and sentiment overall evaluation nodes. Edge weights are dynamically calculated based on sentiment scores and feature values, and normalized to ensure that the sum of node-input edge weights is "1." The connection weights between feature nodes and sentiment nodes are calculated according to the rules, as shown in the formula: Weight = (eigenvalue / total eigenvalue) × (sentiment score / total sentiment score) × rule factor Step S6: Use ECharts to visualize the knowledge graph.

2. The method according to claim 1, characterized in that The negative emotion detection process described in step S2 is as follows: Negative emotions are judged by analyzing the difference between the upper and lower halves of the face, detecting the edge density of the eyebrow area, and the roundness of the mouth contour. The accuracy of expression feature extraction is enhanced by combining Canny edge detection and Sobel gradient analysis.

3. The method for dynamically adjusting sentiment scores according to claim 1, wherein: The process of dynamically adjusting the sentiment score when different emotion-related characteristics are detected in step S3 is as follows: S3.1: If a smile is detected, increase the happy emotion score and decrease the negative emotion score according to the confidence level; S3.2: If negative emotions are detected, the corresponding negative emotion score is increased according to the confidence level, and the happy emotion score is decreased; S3.3: Further modify the emotional score based on physical features: high brightness enhances the emotional weights of happiness and neutrality, high contrast enhances the emotional weights of anger and surprise, high color variance enhances the emotional weights of surprise, anger and happiness, high texture complexity enhances the emotional weights of anger, surprise and fear, high symmetry enhances the emotional weights of neutrality and happiness, high blue tone ratio enhances the emotional weights of sadness, disgust and fear, and high dark tone ratio enhances the emotional weights of sadness, fear and disgust.

4. The method for dynamically adjusting sentiment scores according to claim 1, wherein: The process of dynamically adjusting the sentiment score when different emotion-related characteristics are detected in step S3 is as follows: S3.1: If a smile is detected, increase the happy emotion score and decrease the negative emotion score according to the confidence level; S3.2: If negative emotions are detected, the corresponding negative emotion score is increased according to the confidence level, and the happy emotion score is decreased; S3.3: Further modify the emotional score based on physical features: high brightness enhances the emotional weights of happiness and neutrality, high contrast enhances the emotional weights of anger and surprise, high color variance enhances the emotional weights of surprise, anger and happiness, high texture complexity enhances the emotional weights of anger, surprise and fear, high symmetry enhances the emotional weights of neutrality and happiness, high blue tone ratio enhances the emotional weights of sadness, disgust and fear, and high dark tone ratio enhances the emotional weights of sadness, fear and disgust.

Images