A Visual Sentiment Analysis Method and System Based on Frequency Domain Enhancement and Multi-Attribute Inference

Abstract

This invention discloses a visual sentiment analysis method and system based on frequency domain enhancement and multi-attribute inference. The method includes the following steps: extracting multi-attribute features from a sentiment image; constructing an attribute node matrix and an adjacency matrix; passing the attribute node matrix through a hierarchical attribute graph inference network to obtain a final attribute node matrix; using a spectrum modulation module to transform and modulate the attribute features in the frequency domain to obtain frequency domain enhanced features; an attribute collaborative perception module extracting collaborative perception features of adjacent attribute nodes, fusing them element-wise with the frequency domain enhanced features to obtain fused features; an attribute attention module adjusting the retention ratio of each attribute feature in the fused features before fusing them with the attribute node matrix to obtain an enhanced attribute node matrix; a graph convolutional layer performing weighted aggregation on the enhanced attribute node matrix based on the adjacency matrix; finally, the attribute node matrix is ​​converted into a global sentiment feature vector for sentiment category prediction, outputting the sentiment prediction result. This invention enhances the sentiment analysis performance in complex scenes.

Description

Technical Field

[0001] This invention relates to the field of image sentiment analysis technology, specifically to a visual sentiment analysis method and system based on frequency domain enhancement and multi-attribute reasoning. Background Technology

[0002] Visual sentiment analysis aims to automatically parse the abstract emotional semantics conveyed in images using computer algorithms. Existing visual sentiment analysis methods mainly extract local textures, object edges, or region features in the spatial domain, making it difficult to effectively separate the global emotional atmosphere (such as hue and lighting) from complex local details, resulting in limited model representation capabilities of global emotional cues. In contrast, frequency domain analysis has a natural advantage, as frequency domain transformation can effectively decouple the global atmosphere from local details from the perspective of signal frequency. However, directly applying frequency domain processing to general high-level features that have not been attribute-decoupled is difficult to meet the needs of visual sentiment analysis tasks. Since the effective frequency patterns corresponding to different emotional attributes are not consistent—for example, attributes such as brightness and color usually rely more on low-frequency smooth changes, while object semantic attributes often rely on high-frequency edge details—if a uniform frequency domain enhancement method is used, it may amplify some attributes while suppressing others, thus causing semantic interference between attributes.

[0003] Furthermore, existing multi-attribute inference models typically only stitch together or aggregate global graphs of attributes such as scene, object, and color. They lack modeling of local collaborative relationships between attribute nodes and mechanisms to adjust the contribution of different attributes based on the image. They are unable to flexibly adjust inference strategies according to different scenarios and establish robust and discriminative high-level sentiment semantic representations. Summary of the Invention

[0004] To overcome the shortcomings and deficiencies of existing technologies, this invention provides a visual sentiment analysis method and system based on frequency domain enhancement and multi-attribute inference. This invention constructs an attribute node matrix using multi-attribute features of sentiment images, inputs the attribute node matrix into a hierarchical attribute graph inference network for multi-layer graph inference, utilizes a spectrum modulation module to perform frequency domain transformation and modulation of attribute features, enhances sentiment-related frequency components at the attribute node level, extracts interaction features between adjacent attribute nodes using an attribute collaborative perception module, captures local collaborative relationships between different sentiment attributes, adjusts attribute contributions through an attribute attention module, and combines graph convolutional layers to achieve multi-attribute inference. In multi-layer graph inference, it achieves collaborative optimization of intra-attribute frequency selection, inter-attribute collaborative modeling, and attribute contribution adjustment, effectively improving the representational capability of multi-attribute features, establishing a robust and discriminative high-level sentiment semantic representation, and enhancing sentiment analysis performance in complex scenarios.

[0005] To achieve the above objectives, the present invention adopts the following technical solution:

[0006] This invention provides a visual sentiment analysis method based on frequency domain enhancement and multi-attribute reasoning, comprising the following steps:

[0007] Extract multi-attribute features from the input emotion image and construct an initial attribute node matrix;

[0008] Construct an adjacency matrix based on the relationships between various attribute features;

[0009] The initial attribute node matrix is ​​input into the hierarchical attribute graph inference network for multi-layer graph inference to obtain the final attribute node matrix.

[0010] The hierarchical attribute graph reasoning network includes multiple spectrum attribute collaborative enhancement units and graph convolutional layers. Each spectrum attribute collaborative enhancement unit is equipped with a spectrum modulation module, an attribute collaborative perception module, and an attribute attention module.

[0011] The spectrum modulation module performs frequency domain transformation and modulation on the attribute features to obtain frequency domain enhanced features. The attribute collaborative perception module extracts the interaction features between adjacent attribute nodes to obtain collaborative perception features. The frequency domain enhanced features and collaborative perception features are fused element by element to obtain fused features. The attribute attention module adjusts the retention ratio of each attribute feature in the fused features and then fuses it with the attribute node matrix to obtain an enhanced attribute node matrix.

[0012] Each graph convolutional layer performs weighted aggregation of the enhanced attribute node matrix based on the adjacency matrix;

[0013] After converting the final attribute node matrix into a global sentiment feature vector, sentiment category prediction is performed, and the sentiment prediction result is output.

[0014] As a preferred technical solution, an adjacency matrix is ​​constructed based on the correlation between various attribute features, specifically including:

[0015] The extracted attribute features are connected in pairs to transmit information, and an original adjacency matrix is ​​constructed. The elements of the original adjacency matrix represent the association between different attribute features.

[0016] The original adjacency matrix is ​​row normalized to obtain the adjacency matrix.

[0017] As a preferred technical solution, the hierarchical attribute graph inference network sequentially includes a first spectrum attribute collaborative enhancement unit, a first graph convolutional layer, a second spectrum attribute collaborative enhancement unit, a second graph convolutional layer, and a third spectrum attribute collaborative enhancement unit.

[0018] The initial attribute node matrix is ​​passed sequentially through the first spectral attribute co-enhancement unit, the first graph convolutional layer, the second spectral attribute co-enhancement unit, the second graph convolutional layer, and the third spectral attribute co-enhancement unit. The third spectral attribute co-enhancement unit outputs the final attribute node matrix.

[0019] As a preferred technical solution, the spectrum modulation module performs frequency domain transformation and modulation on the attribute features to obtain frequency domain enhanced features, specifically including:

[0020] The spectrum modulation module maps the attribute features in the attribute node matrix from the spatial domain to the frequency domain, decouples them into real and imaginary parts, performs nonlinear weighting adjustment on the real part, re-integrates the adjusted real and imaginary parts into a spectrum, and multiplies the fused spectrum element-wise with the weighting factor to obtain the complex spectrum after weight adjustment.

[0021] The complex spectrum is restored to the spatial domain through inverse fast Fourier transform, resulting in frequency domain enhancement features.

[0022] As a preferred technical solution, nonlinear weighted adjustment of the real part is performed, specifically including:

[0023] A nonlinear spectrum filter is constructed based on a multilayer perceptron, and the real part of the nonlinear spectrum filter is nonlinearly weighted and adjusted.

[0024] The nonlinear spectrum filter consists of a first fully connected layer, a ReLU activation function, a second fully connected layer, and a Tanh activation function. The first fully connected layer performs a weighted combination of the real responses at different frequency positions, thus... The spectrum of the real part is mapped as follows Wei stated that the second fully connected layer will Wei indicates restoration to The real part spectrum, Indicates the dimension of the current input attribute feature.

[0025] As a preferred technical solution, the attribute collaborative perception module extracts the interaction features between adjacent attribute nodes to obtain collaborative perception features, specifically including:

[0026] The attribute collaborative perception module extracts the interaction features of the current attribute node and its neighboring nodes based on convolution operations, and then processes the interaction features through linear mapping and activation functions to obtain collaborative perception features.

[0027] As a preferred technical solution, the attribute attention module adjusts the retention ratio of each attribute feature in the fused feature and then fuses it with the attribute node matrix to obtain an enhanced attribute node matrix, specifically including:

[0028] The attribute attention module calculates the sentiment recognition saliency score of each attribute in the fused feature based on the multilayer perceptron, and adjusts the retention ratio of each attribute feature in the fused feature based on the sentiment recognition saliency score.

[0029] A weight vector is constructed based on the sentiment recognition saliency score corresponding to each attribute;

[0030] The weight vector is row-weightedly multiplied with the fused features. The result of the row-weighted multiplication is then multiplied with the residual scaling factor and fused with the attribute node matrix to obtain the enhanced attribute node matrix.

[0031] As a preferred technical solution, each graph convolutional layer performs weighted aggregation of the enhanced attribute node matrix based on the adjacency matrix, specifically including:

[0032] Each graph convolutional layer performs a linear mapping on the enhanced attribute node matrix to obtain the mapping features of different attribute nodes;

[0033] The node feature matrix after inference is obtained by weighted aggregation of the mapping features of nodes with different attributes based on the adjacency matrix.

[0034] As a preferred technical solution, the final attribute node matrix is ​​converted into a global sentiment feature vector before sentiment category prediction is performed, and the sentiment prediction result is output, specifically including:

[0035] The final attribute node matrix is ​​transformed into a global sentiment feature vector of uniform dimension through a flattening operation.

[0036] The global sentiment feature vector is input into the classification head for sentiment category prediction. The classification head performs a non-linear mapping on the global sentiment feature vector and projects it onto a predefined sentiment category space. The Softmax activation function is used to calculate the predicted probability distribution of the image belonging to each sentiment category. Based on the predicted probability distribution, the category index with the largest response value is selected as the final sentiment prediction result.

[0037] The present invention also provides a visual sentiment analysis system based on frequency domain enhancement and multi-attribute reasoning, for implementing the above-mentioned visual sentiment analysis method based on frequency domain enhancement and multi-attribute reasoning, including: an attribute node matrix construction module, an adjacency matrix construction module, a multi-layer graph reasoning module, and a sentiment category prediction module;

[0038] The attribute node matrix construction module is used to extract multi-attribute features of the input emotion image and construct an initial attribute node matrix;

[0039] The adjacency matrix construction module is used to construct an adjacency matrix based on the association relationships between various attribute features;

[0040] The multi-layer graph reasoning module is used to input the initial attribute node matrix into the hierarchical attribute graph reasoning network for multi-layer graph reasoning to obtain the final attribute node matrix.

[0041] The hierarchical attribute graph reasoning network includes multiple spectrum attribute collaborative enhancement units and graph convolutional layers. Each spectrum attribute collaborative enhancement unit is equipped with a spectrum modulation module, an attribute collaborative perception module, and an attribute attention module.

[0042] The spectrum modulation module performs frequency domain transformation and modulation on the attribute features to obtain frequency domain enhanced features. The attribute collaborative perception module extracts the interaction features between adjacent attribute nodes to obtain collaborative perception features. The frequency domain enhanced features and collaborative perception features are fused element by element to obtain fused features. The attribute attention module adjusts the retention ratio of each attribute feature in the fused features and then fuses it with the attribute node matrix to obtain an enhanced attribute node matrix.

[0043] Each graph convolutional layer performs weighted aggregation of the enhanced attribute node matrix based on the adjacency matrix;

[0044] The sentiment category prediction module is used to convert the final attribute node matrix into a global sentiment feature vector and then perform sentiment category prediction, outputting the sentiment prediction result.

[0045] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0046] (1) This invention inputs the attribute node matrix constructed based on the multi-attribute features of emotional images into the hierarchical attribute graph reasoning network for multi-level graph reasoning. It uses the spectrum modulation module to perform frequency domain transformation and modulation on the attribute features, strengthens the emotional related frequency components at the attribute node level, uses the attribute collaborative perception module to extract the interaction features between adjacent attribute nodes, captures the local collaborative relationship between different emotional attributes, realizes attribute contribution adjustment through the attribute attention module, and realizes multi-attribute reasoning by combining graph convolutional layers. In multi-level graph reasoning, it realizes the collaborative optimization of frequency selection within attributes, collaborative modeling between attributes, and attribute contribution adjustment, effectively improving the representation ability of multi-attribute features, establishing a robust and discriminative high-level emotional semantic representation, and enhancing the emotional analysis performance in complex scenarios.

[0047] (2) Unlike existing technologies that perform unified frequency domain processing on general high-level features, this invention captures the global emotional atmosphere in parallel through a spectrum modulation module for each attribute feature in the attribute node matrix. This can effectively separate global emotional atmosphere cues from complex local details. Spectrum modulation is performed at the attribute node level, enabling different attributes to learn their corresponding effective frequency responses in their respective semantic spaces. This avoids attribute conflicts caused by unified frequency domain enhancement and solves the problem that traditional spatial domain methods have difficulty separating global atmosphere from local details and have limited ability to represent global emotional cues.

[0048] (3) This invention utilizes the attribute collaborative perception module to model the local collaborative relationship between adjacent attributes in the attribute node dimension, explicitly captures the emotional cues of different attribute combinations, avoids different attributes being modeled in isolation or simply spliced ​​together, thereby improving the discriminative power of multi-attribute emotional representation, and enhances the robustness of sentiment analysis in variable scenarios by adjusting the contribution of different attribute nodes through the attribute attention module. Attached Figure Description

[0049] Figure 1 This is a flowchart illustrating the visual sentiment analysis method based on frequency domain enhancement and multi-attribute reasoning of the present invention. Detailed Implementation

[0050] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0051] Example 1

[0052] like Figure 1 As shown, this embodiment provides a visual sentiment analysis method based on frequency domain enhancement and multi-attribute reasoning, including the following steps:

[0053] S1: Extract multi-attribute features from the input emotion image and construct an initial attribute node matrix;

[0054] In this embodiment, multiple heterogeneous pre-trained networks are used to extract multi-attribute features from the input emotion image, specifically:

[0055] A scene encoder built using a pre-trained ResNet-18 network extracts scene context features from images. In order to establish environmental constraints;

[0056] An object encoder constructed using the CLIP model visual encoder extracts semantic features from the object. ;

[0057] Luminance features extracted using an appearance encoder built on a MobileNet-V2 network With color richness characteristics The appearance encoder is pre-trained using a multi-task learning architecture and supervised learning through the cross-entropy loss function.

[0058] In this embodiment, an initial attribute node matrix is ​​constructed based on the extracted multi-attribute features. , represented as:

[0059]

[0060] S2: Construct the original adjacency matrix of the attribute graph based on the extracted multi-attribute features, and perform row normalization on the original adjacency matrix to obtain the adjacency matrix used for graph convolutional inference, specifically including:

[0061] The extracted multi-attribute features are connected pairwise for information transfer. In this embodiment, an original adjacency matrix is ​​constructed based on the four attribute features in the initial attribute node matrix. The original adjacency matrix is ​​represented as follows:

[0062] ;

[0063] To describe the relationships between different attribute features, elements in the original adjacency matrix... middle, Indicates the first The attribute feature and the first There are information transmission relationships between the attribute features. This indicates that there is no direct connection between the two.

[0064] To avoid the situation where the magnitude of the aggregated feature increases with the number of connected nodes due to the simple summation of each attribute feature when aggregating the features of neighboring nodes, the original adjacency matrix is ​​row-normalized to obtain the adjacency matrix used for graph convolutional inference. , represented as:

[0065] ;

[0066] in, for The degree matrix, whose diagonal elements Represented as:

[0067] ;

[0068] S3: Input the initial attribute node matrix into the hierarchical attribute graph inference network for multi-layer graph inference to obtain the final attribute node matrix. Then, transform the final attribute node matrix into a global sentiment feature vector of a unified dimension through a flattening operation.

[0069] In this embodiment, the hierarchical attribute graph inference network includes a first spectrum attribute collaborative enhancement unit, a first graph convolutional layer, a second spectrum attribute collaborative enhancement unit, a second graph convolutional layer, and a third spectrum attribute collaborative enhancement unit. The network structure of each spectrum attribute collaborative enhancement unit is the same, and each unit is equipped with a spectrum modulation module, an attribute collaborative perception module, and an attribute attention module. The processing procedures corresponding to each module are also the same.

[0070] In this embodiment, the initial attribute node matrix is ​​input into the hierarchical attribute graph inference network for multi-layer graph inference. The specific process is as follows:

[0071] S31: Initialize the attribute node matrix The input is processed by the first spectral attribute collaborative enhancement unit. The attribute features in the initial attribute node matrix are captured in parallel by the spectral modulation module to capture the global emotional atmosphere, resulting in frequency domain enhanced features. By utilizing the attribute collaboration perception module to capture the collaborative relationships between attributes, collaborative perception features with local collaboration perception capabilities are generated. Enhance frequency domain features and collaborative perception features Element-by-element fusion is performed to obtain fusion characteristics. fusion features The attribute saliency weights are calculated by the attribute attention module. This is used to adjust the contribution of each attribute, and is achieved through residual connections to the initial attribute node matrix. The fusion yields a preliminary enhanced attribute node matrix. :

[0072] In this embodiment, addressing the difficulty of effectively separating global atmosphere from local details using traditional spatial domain feature extraction methods, the spectrum modulation module enhances the emotion-related frequency components in each attribute node from a frequency domain perspective. This is achieved by performing frequency domain transformation and modulation on each attribute feature to mine emotion-related frequency components, specifically including:

[0073] The spectrum modulation module in the first spectrum attribute collaborative enhancement unit takes the initial attribute node matrix as input, performs a one-dimensional fast Fourier transform (1D-FFT) on each row of attribute features in the attribute node matrix, maps the attribute features from the spatial domain to the frequency domain, and decouples them to their real parts. and the virtual part Considering that the perception of the emotional atmosphere of an image (such as the overall tone and brightness distribution) is mainly affected by the amplitude spectrum, in order to balance computational efficiency and theoretical accuracy, this embodiment only enhances the real part and keeps the imaginary part unchanged as an anchor point to reduce the disturbance to the original phase structure information.

[0074] By using a multilayer perceptron (MLP) as a learnable nonlinear spectral filter to adaptively adjust the real part, the processed real part features are obtained. , represented as:

[0075] ;

[0076] in, , The weight matrix is ​​a learnable matrix. , The learnable bias parameters are uniformly and randomly initialized along the input dimension by the learnable weight matrix and bias parameters. This adaptive adjustment is driven by end-to-end supervised training without the need for manual specification of frequency weights. The learnable weight matrix and bias parameters are updated during training. Backpropagation is performed during training based on the cross-entropy loss of sentiment classification to fit the frequency response curve that best suits the sentiment attribute. The nonlinear spectrum filter can reshape the frequency response distribution of the signal. It only performs nonlinear weighting on the real part after frequency domain decoupling, strengthening the frequency components that are strongly correlated with the current sentiment expression, while suppressing redundant noise and irrelevant frequencies that are not related to sentiment expression.

[0077] In this embodiment, a multilayer perceptron (MLP) is used as a learnable nonlinear spectral filter, comprising a first fully connected layer, a ReLU activation function, a second fully connected layer, and a Tanh activation function. Let the dimension of the current input attribute feature be... The first fully connected layer is used to weight and combine the real responses at different frequency locations to establish the correlation between frequency components. The spectrum of the real part is mapped as follows The dimension represents the weight matrix that can therefore be learned. and learnable bias parameters Initial value from interval Internal random sampling, the second fully connected layer mapping will Wei indicates restoration to The spectrum of the real part is given, thus the weight matrix can be learned. and learnable bias parameters Initial value from interval Internal random sampling; the activation function is used to constrain the amplitude of the real part of the modulated response to avoid the spectral response being too large and affecting the stability of subsequent graph inference.

[0078] Processed real features With the reserved imaginary part Reintegrated into a complete spectrum To achieve precise fine-tuning across different frequency bands and further enhance emotion-related frequency bands, this embodiment introduces a learnable weighting factor. , For attribute feature dimensions, Each element corresponds to a frequency position, and each element is initialized to 0.5. During training, it is updated by backpropagation based on the cross-entropy loss of sentiment classification, and is used to scale the corresponding frequency band in the complex spectrum element by element.

[0079] Weighting factors With the fused spectrum Element-wise multiplication yields the weighted complex spectrum. , represented as:

[0080] ;

[0081] complex spectrum The frequency-enhanced features are obtained by restoring the spatial domain using the inverse fast Fourier transform (IFFT). ;

[0082] To further explicitly model the local collaborative relationships between different attribute nodes, an attribute collaboration perception module captures the local collaborative relationships between adjacent attributes along the attribute node dimension. This module applies a one-dimensional sliding window convolution with a kernel size of 3 along the node dimension of the attribute node matrix. As the convolution kernel slides along the node dimension, it can simultaneously cover and extract the interaction features of the current attribute node and its adjacent nodes (such as object and scene nodes). The interaction features extracted by the convolution operation are processed through linear mapping and activation functions to generate collaborative perception features with local collaboration perception capabilities. The formula is as follows:

[0083] ;

[0084] in, This represents a one-dimensional sliding window convolution performed along the attribute node dimension, used to extract interaction features between adjacent attribute nodes. This represents linear mapping convolution, used to restore the feature dimension to the same dimension as the input attribute node matrix. express Activation function;

[0085] Enhance frequency domain features With collaborative perception features By performing element-by-element fusion, a fusion feature containing both global atmosphere and local collaboration information is obtained. , represented as:

[0086] ;

[0087] in, This indicates an element-by-element merging operation;

[0088] To further adjust the contributions of different attribute nodes based on image content, the attribute attention module generates attention weights for different attribute nodes according to the current image content. The attribute attention module utilizes a multilayer perceptron (MLP) with shared parameters to fuse features. The processing involves calculating the sentiment recognition saliency score for the i-th attribute node feature. The calculation is as follows:

[0089] ;

[0090] in, Indicates fusion features The i-th row vector in the data, , The weight matrix is ​​a learnable matrix. and These are learnable bias parameters, which are updated during training by backpropagation based on the sentiment classification cross-entropy loss and are uniformly and randomly initialized according to the input dimension.

[0091] Let the feature dimension of the current input attribute be... The first fully connected layer mapping of the multilayer perceptron (MLP) will Dimensional node feature mapping is 3D hidden representation, therefore weight matrix and bias parameters Initial value from interval Internal random sampling, the second fully connected layer will dimensional hidden representation is mapped as Significance score Therefore, the weight matrix and bias parameters Initial value from interval Internal random sampling is used to obtain the weight vector. This weight vector can adaptively adjust the inference focus based on the semantic bias of the image content (e.g., images that convey emotions through specific objects). Specifically, since the saliency score is calculated by a multilayer perceptron (MLP) with shared parameters from the fused features, its value varies with the input. When a certain attribute node contributes more to the correct category prediction, the backpropagation of the cross-entropy loss will improve its corresponding saliency score, thereby improving the node's contribution to the fused features. The retention ratio in the middle, and vice versa, reduces its impact.

[0092] To ensure the stability of the original semantic structure while introducing enhanced features, a learnable residual injection structure is used to control the proportion of enhanced features written into the original attribute node representation. Specifically, a learnable residual scaling factor is introduced. The initial value is 0.1, which is mapped to the injection ratio using the sigmoid function. During training, it is updated via backpropagation based on the sentiment classification cross-entropy loss. Finally, the first spectral attribute co-enhancement unit outputs the initially enhanced attribute node matrix. , represented as:

[0093] ;

[0094] in, This represents the sigmoid activation function. Represents the weight vector With fusion features row weighted multiplication, This represents the residual scaling factor, which is designed to balance the ratio between extracted attribute features and spectral attribute-assisted enhancement features, thereby enhancing the robustness of the model in variable scenarios.

[0095] S32: Initially enhance the attribute node matrix The first graph is input into a convolutional layer for attribute propagation, resulting in the inferred node feature matrix. ;

[0096] Specifically, the first convolutional layer first utilizes a learnable weight matrix. Perform a linear mapping on the enhanced attribute node matrix, and then utilize the adjacency matrix. The mapping features of nodes with different attributes are weighted and aggregated to obtain the node feature matrix after inference from the first convolutional layer. , represented as:

[0097] ;

[0098] in, Represents the learnable weight matrix;

[0099] S33: Analyze the node feature matrix after inference from the convolutional layer of the first image. Input the second spectral attribute co-enhancement unit to obtain the enhanced attribute node matrix. ;

[0100] S34: Enhanced attribute node matrix Input the second convolutional layer to obtain the node feature matrix after inference from the second convolutional layer. , represented as:

[0101] ;

[0102] in, Represents the learnable weight matrix;

[0103] In this embodiment, the weight matrix and weight matrix During training, backpropagation updates are performed based on the sentiment classification cross-entropy loss. The parameters mentioned above are uniformly and randomly initialized according to the output dimension. The first convolutional layer maps the 512-dimensional attribute node features to the 1024-dimensional graph inference feature space. Therefore, the weight matrix... The initial value is from the interval Internal random sampling, the second graph convolutional layer maps the 1024-dimensional graph inference features to a 2048-dimensional high-level graph inference feature space, therefore the weight matrix The initial value is from the interval Internal random sampling.

[0104] S35: Transform the node feature matrix The third spectral attribute collaborative enhancement unit is input to obtain the final attribute node matrix. The final attribute node matrix is ​​then transformed into a global sentiment feature vector of uniform dimension through a flattening operation.

[0105] S4: Input the global sentiment feature vector into a classification head consisting of three fully connected layers for sentiment category prediction. The classification head projects the global sentiment feature vector onto a predefined sentiment category space through a non-linear mapping, and uses the Softmax activation function to calculate the predicted probability distribution of the image belonging to each sentiment category. , represented as:

[0106] ;

[0107] in, Represents the ReLU activation function. Represents the global sentiment feature vector. , , This represents the weight matrix of the fully connected layer corresponding to the classification head. , , These represent the bias vectors of the fully connected layer corresponding to the classification head, which are updated via backpropagation during training based on the sentiment classification cross-entropy loss.

[0108] Based on this predicted probability distribution, the category index with the largest response value is selected as the final sentiment prediction result, expressed as:

[0109] ;

[0110] In this embodiment, the final attribute node matrix, after flattening, yields an 8192-dimensional global sentiment feature vector. The first fully connected layer of the classification head maps the 8192-dimensional global sentiment features to a 256-dimensional hidden feature space, with the weight matrix... The initial value is from the interval Internal random sampling, the second fully connected layer of the classification head maps 256-dimensional hidden features to a 64-dimensional sentiment discrimination feature space, weight matrix The initial value is from the interval Internal random sampling, the third fully connected layer of the classification head maps the 64-dimensional sentiment discrimination features to a predefined sentiment category space, weight matrix The initial value is from the interval For example, in an eight-class emotion recognition task, the output dimension of the third fully connected layer of the classification head is 8, based on internal random sampling.

[0111] In this embodiment, the standard cross-entropy loss function is used to calculate the predicted probability distribution during the training phase. Deviation from actual emotional labels:

[0112] ;

[0113] Where B represents the number of image samples input into the model each time during the training phase and used to jointly calculate the loss. Preferably, B=16, that is, 16 images are input each time to calculate the cross-entropy loss. This represents the total number of emotion categories. Indicates the first The sample at the th Authentic labels in each emotional category This represents the corresponding predicted probability.

[0114] In this embodiment, the total number of emotion categories It can be flexibly configured according to different application scenarios. The application scenarios can be selected as eight-class classification tasks for the Mikels dataset, six-class classification tasks for the Ekman dataset, or binary classification tasks for sentiment polarity.

[0115] This invention constructs a visual emotion attribute node matrix that includes scene, object, brightness, and color richness. It utilizes a spectrum modulation module to enhance emotion-related frequency components at the attribute node level, captures local collaborative relationships between different emotion attributes based on an attribute collaborative perception module, and adjusts attribute contributions through an attribute attention module and a learnable residual injection structure. This invention unifies spectrum modulation, attribute collaborative perception, attribute attention allocation, and learnable residual injection into a spectrum attribute collaborative enhancement unit, embedding this unit into the hierarchical attribute graph reasoning process. This achieves synergistic optimization of intra-attribute frequency selection, inter-attribute collaborative modeling, and attribute contribution adjustment, effectively improving the representation capability of multi-attribute features. In complex scenarios, the emotion analysis performance of this invention significantly surpasses existing technologies, providing a solution for visual emotion analysis that is closer to the human emotion perception mechanism.

[0116] Example 2

[0117] This embodiment provides a visual sentiment analysis system based on frequency domain enhancement and multi-attribute reasoning, which is used to implement the visual sentiment analysis method based on frequency domain enhancement and multi-attribute reasoning in Embodiment 1 above. It includes: an attribute node matrix construction module, an adjacency matrix construction module, a multi-layer graph reasoning module, and a sentiment category prediction module.

[0118] In this embodiment, the attribute node matrix construction module is used to extract multi-attribute features of the input emotion image and construct an initial attribute node matrix;

[0119] In this embodiment, the adjacency matrix construction module is used to construct an adjacency matrix based on the association relationships between various attribute features;

[0120] In this embodiment, the multi-layer graph reasoning module is used to input the initial attribute node matrix into the hierarchical attribute graph reasoning network for multi-layer graph reasoning to obtain the final attribute node matrix.

[0121] The hierarchical attribute graph reasoning network includes multiple spectrum attribute collaborative enhancement units and graph convolutional layers. Each spectrum attribute collaborative enhancement unit is equipped with a spectrum modulation module, an attribute collaborative perception module, and an attribute attention module.

[0122] The spectrum modulation module performs frequency domain transformation and modulation on the attribute features to obtain frequency domain enhanced features. The attribute collaborative perception module extracts the interaction features between adjacent attribute nodes to obtain collaborative perception features. The frequency domain enhanced features and collaborative perception features are fused element by element to obtain fused features. The attribute attention module adjusts the retention ratio of each attribute feature in the fused features and then fuses it with the attribute node matrix to obtain an enhanced attribute node matrix.

[0123] Each graph convolutional layer performs weighted aggregation of the enhanced attribute node matrix based on the adjacency matrix;

[0124] In this embodiment, the sentiment category prediction module is used to convert the final attribute node matrix into a global sentiment feature vector and then perform sentiment category prediction, outputting the sentiment prediction result.

[0125] Example 3

[0126] This embodiment selects several publicly available visual sentiment analysis datasets for comparative experiments, including the Emoset-8 dataset, the EmotionROI-6 dataset, the Twitter I dataset, and the Twitter II dataset. Among them, the Emoset-8 dataset is a large-scale visual sentiment dataset containing rich attribute information such as scene, object, brightness, and color. The EmotionROI-6 dataset is constructed based on Ekman's six basic sentiments and provides annotation information related to the emotional stimulus region, which is suitable for verifying the model's ability to jointly model local emotional stimuli and global emotional atmosphere. The Twitter I and Twitter II datasets are derived from social media scenes, with complex image content and a lot of noise, which can be used to verify the robustness of the model in real complex scenes.

[0127] During the experiment, the training and test sets of the Emoset-8 dataset were used as officially defined. For the EmotionROI-6, Twitter I, and Twitter II datasets, the training and test sets were defined according to common visual sentiment analysis experimental settings. During the training phase, data augmentation methods such as random cropping and random horizontal flipping were used, and the input images were normalized to a uniform size.

[0128] The main comparison methods involved in the comparative experiments include: AlexNet network, VGG16 network (VisualGeometry Group 16-layer Network), ResNet101 residual network, WSCNet (Weakly Supervised Coupled Networks), Stimuli-aware visual emotion analysis framework, SOLVER (Scene-Object interrelated Visual Emotion Reasoning Network), OSANet (Object Semantic Attention Network), MDAN (Multi-level Dependent Attention Network), PT-DPC (Learning to Compose Diversified Prompts for Image Emotion Classification) pre-trained model, VSCNet (Visual Sentiment Analysis with Semantic Correlation Enhancement) visual emotion analysis model, AttDNet (Attribute-driven Deep Learning Network) deep learning network, and BOVIS (Bias-mitigated Object-enhanced Visual Emotion) visual emotion analysis. The table below shows a comparison of the classification accuracy of different visual sentiment analysis methods, including (Analysis) and (MAJOR) multi-attribute joint reasoning. The abbreviations of the algorithms used in the table are as follows:

[0129] Table 1. Comparison of classification accuracy results of different visual emotion analysis methods

[0130]

[0131] As shown in Table 1, this invention achieved the best classification accuracy on the Emoset-8 dataset, EmotionROI-6 dataset, Twitter I dataset, and Twitter II dataset. By constructing a visual sentiment attribute node matrix and introducing spectral modulation, attribute collaborative perception, and attribute attention mechanisms in the hierarchical attribute graph inference process, this invention can more fully capture the global emotional atmosphere, local structural details, and multi-attribute collaborative information in the image, thereby improving the accuracy of visual sentiment analysis.

[0132] This embodiment also selects the EmotionROI-6 dataset, FI dataset, and Twitter II dataset for ablation experiments. Among them, the FI dataset consists of images from the Flickr and Instagram platforms, which has a natural class imbalance distribution that closely resembles real-world scenes. It can be used to verify the model's adaptability to imbalanced sentiment data. As shown in Table 2 below, the ablation experiment results of the key structure of this invention are obtained:

[0133] Table 2 Ablation Experiment Results of Key Structures in this Invention

[0134]

[0135] As shown in Table 2, the ablation experiments of the key structures of this invention further verified the effectiveness of each component. First, when using only attribute graph inference, the accuracy on the EmotionROI-6, FI, and Twitter II datasets was relatively low. Adding the attribute collaborative perception module improved the accuracy, indicating that modeling local collaborative relationships at the attribute node level can enhance the expression of multi-attribute features. Further adding the spectrum modulation module increased the model accuracy to 0.6869, 0.7759, and 0.8430, respectively, demonstrating that modulating emotion-related frequency components at the attribute node level helps enhance global emotional atmosphere cues. In contrast, adding only the attribute attention module also improved the model, indicating that adjusting attribute node contributions based on current image content has a positive effect. When the attribute collaborative perception module, spectrum modulation module, and attribute attention module are used together, the highest accuracy is achieved on all three datasets. This shows that this invention does not simply stack multiple modules, but rather achieves complementary collaboration between spectrum modulation, attribute collaborative perception, and attribute attention allocation through the spectrum attribute collaborative enhancement unit.

[0136] This embodiment conducted experiments at different spectral modulation positions, as shown in Table 3 below, and obtained the experimental results corresponding to different spectral modulation positions:

[0137] Table 3 Experimental results at different spectral modulation positions

[0138]

[0139] Table 3 shows that different spectral modulation positions have a significant impact on model performance. Without spectral modulation, the accuracy on the EmotionROI-6 dataset is 0.6734. When spectral processing is performed on low-level inputs, intermediate-level features, or general high-level features before attribute decoupling, the accuracy is lower than the attribute node feature spectral modulation method used in this invention. In particular, the performance degradation is significant when spectral processing is performed on intermediate-level features without attribute decoupling, indicating that direct spectral processing of intermediate-level features without attribute decoupling may destroy the original semantic representation. In contrast, when spectral modulation is performed on decoupled attribute node features, the accuracy reaches 0.7071, achieving the optimal result.

[0140] Furthermore, to further illustrate that the design of key structural parameters is not arbitrary, this embodiment conducted parameter sensitivity experiments on the EmotionROI-6 dataset regarding the number of graph convolutional layers, the number of fully connected layers in the nonlinear spectral filter, and the residual injection method. The experimental results for each key structural parameter are shown in Table 4 below.

[0141] Table 4. Key structural parameters are set based on experimental results.

[0142]

[0143] Table 4 shows that the highest accuracy is achieved when the number of graph convolutional layers is 2, indicating that two-layer graph inference can achieve a better balance between attribute relationship modeling ability and representation stability. The highest accuracy is achieved when the nonlinear spectral filter adopts a two-layer fully connected structure, indicating that it has the necessary nonlinear frequency modulation capability while avoiding performance fluctuations caused by excessively deep structures. The accuracy of using a learnable residual scaling mechanism is higher than that of fixed scaling values ​​of 0.2, 0.6, and 1.0, indicating that using training-driven adaptive gating to enhance the feature writing ratio is more reasonable. Therefore, the relevant structure depth, residual control method, and parameter settings are all supported by clear experiments.

[0144] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. A visual sentiment analysis method based on frequency domain enhancement and multi-attribute reasoning, characterized in that, Includes the following steps: Extract multi-attribute features from the input emotion image and construct an initial attribute node matrix; Construct an adjacency matrix based on the relationships between various attribute features; The initial attribute node matrix is ​​input into the hierarchical attribute graph inference network for multi-layer graph inference to obtain the final attribute node matrix. The hierarchical attribute graph inference network includes multiple spectrum attribute collaborative enhancement units and graph convolutional layers. The hierarchical attribute graph inference network sequentially includes a first spectrum attribute collaborative enhancement unit, a first graph convolutional layer, a second spectrum attribute collaborative enhancement unit, a second graph convolutional layer, and a third spectrum attribute collaborative enhancement unit. The initial attribute node matrix is ​​passed sequentially through the first spectral attribute collaborative enhancement unit, the first graph convolutional layer, the second spectral attribute collaborative enhancement unit, the second graph convolutional layer, and the third spectral attribute collaborative enhancement unit. The third spectral attribute collaborative enhancement unit outputs the final attribute node matrix. Each spectrum attribute collaborative enhancement unit is equipped with a spectrum modulation module, an attribute collaborative perception module, and an attribute attention module; The spectrum modulation module performs frequency domain transformation and modulation on the attribute features to obtain frequency domain enhanced features. The attribute collaborative perception module extracts the interaction features between adjacent attribute nodes to obtain collaborative perception features. The frequency domain enhanced features and collaborative perception features are fused element by element to obtain fused features. The attribute attention module adjusts the retention ratio of each attribute feature in the fused features and then fuses it with the attribute node matrix of the input current spectrum attribute collaborative enhancement unit to obtain the enhanced attribute node matrix. Each graph convolutional layer performs weighted aggregation of the enhanced attribute node matrix based on the adjacency matrix; After converting the final attribute node matrix into a global sentiment feature vector, sentiment category prediction is performed, and the sentiment prediction results are output, including: The final attribute node matrix is ​​transformed into a global sentiment feature vector of uniform dimension through a flattening operation. The global sentiment feature vector is input into the classification head for sentiment category prediction. The classification head performs a non-linear mapping on the global sentiment feature vector and projects it onto a predefined sentiment category space. The Softmax activation function is used to calculate the predicted probability distribution of the image belonging to each sentiment category. Based on the predicted probability distribution, the category index with the largest response value is selected as the final sentiment prediction result.

2. The visual sentiment analysis method based on frequency domain enhancement and multi-attribute reasoning according to claim 1, characterized in that, An adjacency matrix is ​​constructed based on the relationships between various attribute features, specifically including: The extracted attribute features are connected in pairs to transmit information, and an original adjacency matrix is ​​constructed. The elements of the original adjacency matrix represent the association between different attribute features. The original adjacency matrix is ​​row normalized to obtain the adjacency matrix.

3. The visual sentiment analysis method based on frequency domain enhancement and multi-attribute reasoning according to claim 1, characterized in that, The spectrum modulation module performs frequency domain transformation and modulation on the attribute features to obtain frequency domain enhanced features, specifically including: The spectrum modulation module maps the attribute features in the attribute node matrix from the spatial domain to the frequency domain, decouples them into real and imaginary parts, performs nonlinear weighting adjustment on the real part, re-integrates the adjusted real and imaginary parts into a spectrum, and multiplies the fused spectrum element-wise with the weighting factor to obtain the complex spectrum after weight adjustment. The complex spectrum is restored to the spatial domain through inverse fast Fourier transform, resulting in frequency domain enhancement features.

4. The visual sentiment analysis method based on frequency domain enhancement and multi-attribute reasoning according to claim 3, characterized in that, The real part is subjected to nonlinear weighting adjustment, specifically including: A nonlinear spectrum filter is constructed based on a multilayer perceptron, and the real part of the nonlinear spectrum filter is nonlinearly weighted and adjusted. The nonlinear spectrum filter consists of a first fully connected layer, a ReLU activation function, a second fully connected layer, and a Tanh activation function. The first fully connected layer performs a weighted combination of the real responses at different frequency positions, thus... The spectrum of the real part is mapped as follows Wei stated that the second fully connected layer will Wei indicates restoration to The real part spectrum, Indicates the dimension of the current input attribute feature.

5. The visual sentiment analysis method based on frequency domain enhancement and multi-attribute reasoning according to claim 1, characterized in that, The attribute collaboration perception module extracts the interaction features between adjacent attribute nodes to obtain collaboration perception features, specifically including: The attribute collaborative perception module extracts the interaction features of the current attribute node and its neighboring nodes based on convolution operations, and then processes the interaction features through linear mapping and activation functions to obtain collaborative perception features.

6. The visual sentiment analysis method based on frequency domain enhancement and multi-attribute reasoning according to claim 1, characterized in that, The attribute attention module adjusts the retention ratio of each attribute feature in the fused feature set and then fuses it with the attribute node matrix of the input current spectral attribute co-enhancement unit to obtain the enhanced attribute node matrix, which specifically includes: The attribute attention module calculates the sentiment recognition saliency score of each attribute in the fused feature based on the multilayer perceptron, and adjusts the retention ratio of each attribute feature in the fused feature based on the sentiment recognition saliency score. A weight vector is constructed based on the sentiment recognition saliency score corresponding to each attribute; The weight vector is row-weightedly multiplied with the fused features. The result of the row-weighted multiplication is then multiplied with the residual scaling factor and fused with the attribute node matrix of the input current spectral attribute co-enhancement unit to obtain the enhanced attribute node matrix.

7. The visual sentiment analysis method based on frequency domain enhancement and multi-attribute reasoning according to claim 1, characterized in that, Each graph convolutional layer performs weighted aggregation of the enhanced attribute node matrix based on the adjacency matrix, specifically including: Each graph convolutional layer performs a linear mapping on the enhanced attribute node matrix to obtain the mapping features of different attribute nodes; The node feature matrix after inference is obtained by weighted aggregation of the mapping features of nodes with different attributes based on the adjacency matrix.

8. A visual sentiment analysis system based on frequency domain enhancement and multi-attribute reasoning, characterized in that, The method for implementing the visual sentiment analysis method based on frequency domain enhancement and multi-attribute reasoning as described in any one of claims 1-7 includes: an attribute node matrix construction module, an adjacency matrix construction module, a multi-layer graph reasoning module, and a sentiment category prediction module. The attribute node matrix construction module is used to extract multi-attribute features of the input emotion image and construct an initial attribute node matrix; The adjacency matrix construction module is used to construct an adjacency matrix based on the association relationships between various attribute features; The multi-layer graph reasoning module is used to input the initial attribute node matrix into the hierarchical attribute graph reasoning network for multi-layer graph reasoning to obtain the final attribute node matrix. The hierarchical attribute graph reasoning network includes multiple spectrum attribute collaborative enhancement units and graph convolutional layers. Each spectrum attribute collaborative enhancement unit is equipped with a spectrum modulation module, an attribute collaborative perception module, and an attribute attention module. The spectrum modulation module performs frequency domain transformation and modulation on the attribute features to obtain frequency domain enhanced features. The attribute collaborative perception module extracts the interaction features between adjacent attribute nodes to obtain collaborative perception features. The frequency domain enhanced features and collaborative perception features are fused element by element to obtain fused features. The attribute attention module adjusts the retention ratio of each attribute feature in the fused features and then fuses it with the attribute node matrix of the input current spectrum attribute collaborative enhancement unit to obtain the enhanced attribute node matrix. Each graph convolutional layer performs weighted aggregation of the enhanced attribute node matrix based on the adjacency matrix; The sentiment category prediction module is used to convert the final attribute node matrix into a global sentiment feature vector and then perform sentiment category prediction, outputting the sentiment prediction result.

Images