A visual sentiment analysis method, system, terminal and storage medium based on attribute guidance

Abstract

The present invention relates to the field of data analysis technology, and discloses a visual emotion analysis method, system, terminal and storage medium based on attribute guidance. The method comprises: obtaining an emotion image to be analyzed of a target user, performing characterization extraction to obtain multiple visual attribute representations, and performing characterization to obtain multiple text attribute representations; aligning all visual attribute representations and text attribute representations to obtain multiple target visual attribute representations, and performing attribute association analysis to obtain attribute association information; constructing an attribute emotion map, optimizing to obtain a target attribute emotion map, and performing emotion prediction based on the target attribute emotion map to obtain an emotion prediction result. The present invention improves the accuracy of the emotion analysis results by optimizing the extraction of visual representations under the guidance of text attribute representations, dynamically adjusting the weight of each attribute representation through attribute association analysis, and constructing an attribute emotion map to deeply analyze the relationship between emotions and attributes.

Description

Technical Field

[0001] The present invention relates to the field of data analysis technology, and in particular to a method, system, terminal and computer-readable storage medium for attribute-guided visual sentiment analysis. Background Art

[0002] Emotions are a unique and indispensable human trait, permeating every aspect of daily life. With the rise of social media, more and more people are expressing their emotional experiences by sharing images. Consequently, visual sentiment analysis has garnered widespread attention, aiming to deeply understand individuals' emotional responses to various visual stimuli and to provide explanations for these predictions. An inherent challenge in visual sentiment analysis research is known as the "emotion gap." Emotions are complex responses of the brain to external stimuli and internal states, while images contain visual elements of varying modalities. While pixel-level semantic information is relatively easy to discern, perceiving highly complex emotions is more challenging. This discrepancy, primarily due to the representational discrepancy between visual elements and human emotional experiences, is the so-called emotion gap.

[0003] While deep learning methods have made significant progress in extracting visual representations, they have overlooked subtle differences in emotional expression. For example, low-level visual representations such as color, lighting, and composition can provide certain emotional cues. To address this issue, existing technologies have used multimodal learning, attention mechanisms, and large-scale data-driven methods to narrow this emotional gap. For example, language descriptions are used to enhance emotional semantic understanding, or generative models are used to construct more emotionally perceptive visual representations. However, existing technologies use neural networks to extract global representation information and do not mention the use of dedicated attribute networks to obtain attribute representations. Instead, the corresponding attribute representations are obtained through the proposed network. Secondly, sentiment analysis does not analyze the association between attributes, but directly performs sentiment inference analysis. It also directly classifies and predicts the sentiment category of the representation, resulting in inaccurate sentiment analysis results.

[0004] Therefore, the existing technology still needs to be improved and developed. Summary of the Invention

[0005] The main purpose of the present invention is to provide a visual sentiment analysis method, system, terminal and storage medium based on attribute guidance, aiming to solve the problem that the existing technology does not use a dedicated attribute network to obtain attribute representation in sentiment analysis, does not analyze the relationship between attributes, and does not perform sentiment reasoning analysis, thereby resulting in inaccurate sentiment analysis results.

[0006] To achieve the above object, the present invention provides a visual emotion analysis method based on attribute guidance, which includes the following steps:

[0007] Acquire an emotion image to be analyzed of a target user, perform characterization extraction on the emotion image to be analyzed to obtain multiple visual attribute characterizations, and perform text characterization extraction on the emotion image to be analyzed to obtain multiple text attribute characterizations;

[0008] Aligning all the visual attribute representations and all the text attribute representations to obtain a plurality of target visual attribute representations, and performing attribute association analysis on all the target visual attribute representations to obtain attribute association information;

[0009] An attribute emotion map is constructed according to the attribute association information, the attribute emotion map is optimized to obtain a target attribute emotion map, and emotion prediction is performed according to the target attribute emotion map to obtain an emotion prediction result.

[0010] Optionally, the attribute-guided visual sentiment analysis method, wherein the steps of obtaining the target user's emotion image to be analyzed, extracting a representation of the emotion image to be analyzed to obtain multiple visual attribute representations, and calculating a text representation of the emotion image to be analyzed to obtain multiple text attribute representations, specifically include:

[0011] Obtain the target user's emotional image to be analyzed and build a multi-level attribute expert module;

[0012] Extracting a representation of the emotion image to be analyzed by the multi-level attribute expert module to obtain a multi-level attribute representation, wherein the multi-level attribute representation includes a low-level attribute representation, a mid-level attribute representation, and a high-level attribute representation;

[0013] Performing nonlinear transformation on the multi-level attribute representation to obtain multiple visual attribute representations;

[0014] Text representation extraction is performed on the emotional image to be analyzed to obtain multiple text attribute representations.

[0015] Optionally, in the attribute-guided visual sentiment analysis method, the nonlinear transformation of the multi-level attribute representation is performed as follows:

[0016] ;

[0017] The text representation extraction of the emotional image to be analyzed is specifically:

[0018] ;

[0019] in, is the visual attribute representation set, is the activation function, is a low-level attribute representation, is the intermediate attribute representation, For high-level attribute representation, and are all learnable parameters, is the number of attribute modules, is the text attribute representation set, For text adapters, For text encoder, Encode attribute embeddings.

[0020] Optionally, the attribute-guided visual sentiment analysis method, wherein the alignment processing of all the visual attribute representations and all the text attribute representations to obtain multiple target visual attribute representations, specifically includes:

[0021] performing scale assimilation on all the visual attribute representations and all the text attribute representations, and aligning all the scale-assimilated visual attribute representations and text attribute representations to obtain a plurality of aligned visual attribute representations;

[0022] Loss calculation is performed on the visual attribute representations and text attribute representations after assimilation at all scales to obtain attribute alignment loss, and all aligned visual attribute representations are optimized according to the attribute alignment loss to obtain multiple target visual attribute representations.

[0023] Optionally, the attribute-guided visual sentiment analysis method, wherein the attribute association analysis is performed on all the target visual attribute representations to obtain attribute association information, specifically includes:

[0024] Calculating the attribute relevance of all the target visual attribute representations to obtain a plurality of relevance scores;

[0025] Attribute association analysis is performed on all attributes according to all the correlation scores to obtain attribute association information.

[0026] Optionally, the attribute-guided visual sentiment analysis method, wherein the step of constructing an attribute sentiment map based on the attribute association information, optimizing the attribute sentiment map to obtain a target attribute sentiment map, and performing sentiment prediction based on the target attribute sentiment map to obtain a sentiment prediction result, specifically includes:

[0027] Performing spatial mapping on the attribute association information to obtain an attribute mapping result, and constructing an attribute emotion map based on the attribute mapping result;

[0028] Performing loss calculation on all the visual attribute representations to obtain sentiment classification loss, and performing total loss calculation on the sentiment classification loss and the attribute alignment loss to obtain a target total loss;

[0029] The attribute emotion map is optimized according to the target total loss to obtain a target attribute emotion map, and emotion prediction is performed according to the target attribute emotion map to obtain an emotion prediction result, wherein the emotion prediction result includes emotion information and emotion interpretation.

[0030] Optionally, in the attribute-guided visual sentiment analysis method, the loss calculation for all the visual attribute representations is performed as follows:

[0031] ;

[0032] The total loss calculation of the sentiment classification loss and the attribute alignment loss is specifically as follows:

[0033] ;

[0034] in, is the sentiment classification loss, is the total number of sentiment categories, is the number of attribute modules, For the True labels, is a nonlinear mapping, F is a set of visual attribute representations, For the association analysis between attributes, is the number of sentiment categories, is the target total loss, is a hyperparameter, is the attribute alignment loss.

[0035] Optionally, in the attribute-guided visual sentiment analysis method, the attribute-guided visual sentiment analysis system includes:

[0036] a representation processing module, configured to obtain an emotion image to be analyzed of a target user, perform representation extraction on the emotion image to be analyzed to obtain a plurality of visual attribute representations, and perform text representation extraction on the emotion image to be analyzed to obtain a plurality of text attribute representations;

[0037] an attribute analysis module, configured to align all the visual attribute representations and all the text attribute representations to obtain a plurality of target visual attribute representations, and perform attribute association analysis on all the target visual attribute representations to obtain attribute association information;

[0038] The emotion prediction module is used to construct an attribute emotion map based on the attribute association information, optimize the attribute emotion map to obtain a target attribute emotion map, and perform emotion prediction based on the target attribute emotion map to obtain an emotion prediction result.

[0039] In addition, to achieve the above-mentioned purpose, the present invention also provides a terminal, wherein the terminal includes: a memory, a processor, and an attribute-guided visual emotion analysis program stored on the memory and runnable on the processor, and when the attribute-guided visual emotion analysis program is executed by the processor, the steps of the attribute-guided visual emotion analysis method as described above are implemented.

[0040] In addition, to achieve the above-mentioned purpose, the present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores an attribute-guided visual sentiment analysis program, and when the attribute-guided visual sentiment analysis program is executed by a processor, it implements the steps of the attribute-guided visual sentiment analysis method as described above.

[0041] In the present invention, an emotional image to be analyzed of a target user is obtained, a representation extraction is performed on the emotional image to be analyzed to obtain multiple visual attribute representations, and a textual representation extraction is performed on the emotional image to be analyzed to obtain multiple textual attribute representations; all the visual attribute representations and all the textual attribute representations are aligned to obtain multiple target visual attribute representations, and attribute association analysis is performed on all the target visual attribute representations to obtain attribute association information; an attribute emotion map is constructed based on the attribute association information, the attribute emotion map is optimized to obtain a target attribute emotion map, and emotion prediction is performed based on the target attribute emotion map to obtain an emotion prediction result. The present invention uses a multi-level attribute expert feature extraction method and introduces a text-guided multi-level alignment module, and simultaneously extracts multiple visual attribute representations under the guidance of textual attribute representations to optimize the extraction of visual attribute representations; in the emotion reasoning task, attribute association analysis is proposed to analyze all attributes and dynamically adjust the weight of each attribute representation, and the attribute association map is mapped to the emotion space for emotion reasoning. By constructing an attribute sentiment map, we can deeply analyze the relationship between emotions and attributes. It not only considers a single representation, but also integrates the interactions between multiple attributes, enhances the interpretability of emotion recognition, and thus improves the accuracy of sentiment analysis results. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 is a schematic diagram of an existing visual emotion analysis method and a preferred embodiment of the present invention;

[0043] Figure 2 Flowchart of a preferred embodiment of the attribute-guided visual sentiment analysis method of the present invention;

[0044] Figure 3 Schematic diagram of the overall visual sentiment analysis method based on attribute guidance of the present invention;

[0045] Figure 4This is a schematic diagram of a single-dimensional attribute association module in a preferred embodiment of the present invention;

[0046] Figure 5 1 is a structural diagram of a preferred embodiment of the attribute-guided visual sentiment analysis system of the present invention;

[0047] Figure 6 Schematic diagram of the operating environment of a preferred embodiment of the terminal of the present invention. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solutions and advantages of the present invention more clear and distinct, the present invention is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0049] It should be noted that if the embodiments of the present invention involve directional indications (such as up, down, left, right, front, back, etc.), such directional indications are only used to explain the relative position relationship, movement status, etc. between the various components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication will also change accordingly.

[0050] In addition, if there are descriptions involving "first", "second", etc. in the embodiments of the present invention, the descriptions of "first", "second", etc. are only for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features specified as "first" and "second" may explicitly or implicitly include at least one of such features. In addition, the technical solutions between the various embodiments can be combined with each other, but this must be based on the fact that ordinary technicians in this field can implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such a combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0051] Existing technologies construct visual elements based on artistic and psychological principles, attempting to analyze the relationship between constructed visual elements and human emotions. However, due to the complexity of emotions and the diversity of visual elements that can drive emotions, artificially constructed visual elements are difficult to cover all and may cause errors in real-world recognition. With the continuous development of deep learning networks, convolutional neural networks have superior performance in tasks such as classification, and use neural networks to extract visual representations from images for emotion prediction. Convolutional neural networks can effectively extract global visual representations from images and perform emotion classification, such as Figure 1 As shown in (a), relying solely on global information for sentiment prediction may ignore important sentiment clues embedded in local representations. Therefore, local information is also considered to be crucial in sentiment prediction. In order to improve the accuracy of prediction, local representation extraction is combined with global information, as shown in Figure 1(b) of the figure. However, these methods still have limitations in visual emotion prediction because they oversimplify the complexity of emotional stimulation. Simply combining global and local representations is still not enough to establish meaningful connections with emotions. To solve this problem, this paper proposes attribute-guided visual emotion analysis, which analyzes the relationship between attribute representations at different levels and constructs an emotional attribute graph for reasoning, as shown in Figure 2. Figure 1 As shown in (c).

[0052] The attribute-guided visual sentiment analysis method described in the preferred embodiment of the present invention is as follows: Figure 2 As shown, the attribute-guided visual sentiment analysis method includes the following steps:

[0053] Step S10: Acquire the target user's emotion image to be analyzed, perform representation extraction on the emotion image to be analyzed to obtain multiple visual attribute representations, and perform text representation extraction on the emotion image to be analyzed to obtain multiple text attribute representations.

[0054] Specifically, in the embodiment of the present invention, Figure 3 As shown in the figure, a visual sentiment analysis method based on attribute guidance is proposed to analyze the relationship between attribute representations in an image and explain the impact of attributes on emotion. The multi-level attribute guidance module uses multiple expert networks to capture attribute-related visual representations based on text representation guidance, and maps these representations to a latent representation space through adaptive layers and nonlinear mapping, thereby mapping the image and text representations into the same space and extracting accurate visual attribute representations. The second is the association reasoning module, which analyzes the relationship between attributes. Finally, the emotion reasoning module uses these attribute representations to construct an attribute emotion map, which can explain the changing importance of different attributes to emotion during the emotion reasoning process. The specific analysis process is that in a deep neural network, as the network depth increases, its representation processing gradually tends to semantic information. To this end, the present invention proposes a multi-level extraction network, in which attributes at different levels correspond to different parts of the network. The higher the attribute level, the deeper the network depth. In addition, how to accurately extract visual attribute representations remains a key challenge. The present invention uses proprietary submodules, that is, each type of attribute has a corresponding network module, for example, two low-level attributes include color and brightness, two mid-level attributes include objects and scenes, and two high-level attributes include facial expressions and human movements.

[0055] Get the target user's emotional image to be analyzed, and use Representation; and construct a multi-level attribute expert module; the multi-level attribute expert module extracts the representation of the emotional image to be analyzed according to the representation extraction formula to obtain a multi-level attribute representation, wherein the multi-level attribute representation includes a low-level attribute representation, a mid-level attribute representation and a high-level attribute representation; the representation extraction formula is:

[0056] ;

[0057] in, is a low-level attribute representation, is the intermediate attribute representation, For high-level attribute representation, is an image encoder. In an embodiment of the present invention, a dedicated submodule for each attribute is responsible for extracting the corresponding attribute representation. Since multiple attribute representations are different, an attribute space is constructed for each attribute. Secondly, different attribute-specific representations are also different, and a more fine-grained representation of each attribute can be learned. The submodule uses a multi-layer perceptron to extract a unique fine-grained representation of each attribute and performs a nonlinear transformation, that is, a nonlinear transformation is performed on the multi-level attribute representation to obtain multiple visual attribute representations. The nonlinear transformation of the multi-level attribute representation is specifically as follows:

[0058] ;

[0059] in, is the visual attribute representation set, is the activation function, and are all learnable parameters, is the number of attribute modules, is a set of visual attribute representations, including six attribute representations, ,in, They are represented by 6 visual attributes, namely brightness, color, scene type, object category, facial expression and human motion.

[0060] Subsequently, CLIP (Contrastive Language-Image Pre-training) is a powerful language-image contrast learning model that can associate text and semantic representations in CLIP space. In this invention, a CLIP text encoder is used to encode labels for six attributes within the attribute's text representation, resulting in an accurate visual representation. However, to facilitate matching of text representations with visual representations in the attribute space, a text adapter is proposed for text representation extraction. Specifically, text representation extraction is performed on the emotion image to be analyzed, resulting in multiple text attribute representations. The text representation extraction for the emotion image to be analyzed is as follows:

[0061] ;

[0062] in, is the text attribute representation set, For text adapters, For text encoder, Encode attribute embeddings. ,in, They are represented by 6 text attributes, namely brightness, color, scene type, object category, facial expression and human motion.

[0063] Step S20 : ​​aligning all the visual attribute representations and all the text attribute representations to obtain a target representation alignment result, and performing attribute association analysis based on the target representation alignment result to obtain attribute association information.

[0064] Specifically, after obtaining the visual attribute representation and the text attribute representation, the visual attribute representation and the text attribute representation need to be kept at the same scale to ensure the reliability and accuracy of the alignment, that is, all the visual attribute representations and all the text attribute representations are scale-assimilated to obtain multiple scale-assimilated visual attribute representations (using representation) and multi-scale assimilated text attribute representation (using Representation), multi-level text-guided alignment can achieve alignment of visual and textual representations at different levels, thereby enhancing the accuracy of visual representation; and align the visual attribute representations and textual attribute representations after assimilation at all scales to obtain multiple aligned visual attribute representations. Under the guidance of text representation, attribute loss is formulated to ensure the accuracy and diversity of visual attribute representations extracted from images, that is, the loss is calculated for the visual attribute representations and textual attribute representations after assimilation at all scales to obtain the attribute alignment loss, which is used The corresponding expression is:

[0065] ;

[0066] in, is the total number of attribute categories, is the total number of attributes in a batch, For the visual attribute features, For the Text attribute features, is the similarity calculation, where the similarity calculation formula is:

[0067] ;

[0068] As the attribute loss converges, the visual semantic representation will become more and more accurate. Afterwards, all aligned visual attribute representations are optimized according to the attribute alignment loss to obtain multiple target visual attribute representations, that is, multiple accurate visual attribute representations.

[0069] Afterwards, attribute association analysis can analyze all attributes and dynamically adjust the weight of each attribute representation. Important attribute categories are amplified and irrelevant attribute categories are weakened. The correlation between attributes can reveal potential emotional predictions to help the model better interpret complex emotional expressions in images. For example, subtle changes in color, brightness, and facial expressions can change the expression of emotions; and the attribute association module analyzes the relationship between attributes by calculating the correlation scores between attributes. Specifically, the attribute correlation of all the target visual attribute representations is scored to obtain multiple correlation scores; attribute association analysis is performed on all attributes based on all the correlation scores to obtain attribute association information. Figure 4 As shown in the figure, "brightness" is associated with other attributes. Self-attention can dynamically adjust the association between embeddings and automatically update the importance of each embedding. The multi-head attention mechanism obtains the relationship between attributes from multiple dimensions and forms a relationship graph between attributes, where Figure 4 middle 、 、 、 、 、 and They are all visual attribute representations. , , , , , , , , and All are weight matrices, X is matrix multiplication, + is matrix addition, and softmax is the normalized exponential function.

[0070] Step S30: construct an attribute emotion map based on the attribute association information, optimize the attribute emotion map to obtain a target attribute emotion map, and perform emotion prediction based on the target attribute emotion map to obtain an emotion prediction result.

[0071] Specifically, in the emotion reasoning task, it not only involves emotion recognition, but also requires the model to understand the causes of emotions and important triggers of emotions. Emotion is a high-dimensional and complex representation, and the attribute association information is mapped to the emotion space, so that an attribute emotion map is constructed in the emotion space. The constructed emotion map is a highly refined attribute key factor of emotion, and the knowledge of the emotion map can be applied to any field and data analysis visual emotion, which is conducive to the understanding and analysis of emotions. Nonlinear mapping can map low-level attribute association information to a high-dimensional emotion space to realize the construction of an attribute emotion map, specifically, spatial mapping the attribute association information to obtain an attribute mapping result, and constructing an attribute emotion map based on the attribute mapping result. Afterwards, loss calculation is performed on all the visual attribute representations to obtain the emotion classification loss, wherein the loss calculation for all the visual attribute representations is specifically as follows:

[0072] ;

[0073] in, is the sentiment classification loss, is the total number of sentiment categories, is the number of attribute modules, For the True labels, is a nonlinear mapping, F is a set of visual attribute representations, For the association analysis between attributes, is the number of sentiment categories; as the sentiment classification loss function converges, the construction of the attribute sentiment graph is gradually completed.

[0074] In the visual sentiment analysis task, the best performance of both is required, so a trade-off needs to be made between the attribute alignment loss and the sentiment classification loss. In the embodiment of the present invention, a hyperparameter is introduced to dynamically adjust the impact of the optimization between the two losses on the construction of the sentiment map. Specifically, the total loss of the sentiment classification loss and the attribute alignment loss is calculated to obtain the target total loss. The total loss calculation of the sentiment classification loss and the attribute alignment loss is specifically:

[0075] ;

[0076] in, is the target total loss, is a hyperparameter, is the attribute alignment loss; then, emotion prediction is performed according to the target attribute emotion map to obtain an emotion prediction result, wherein the emotion prediction result includes emotion information and emotion explanation.

[0077] This paper uses a multi-level attribute expert-level feature extraction method and introduces a text-guided multi-level alignment module. It simultaneously extracts multiple visual attribute representations under the guidance of text attribute representations, optimizing the extraction of visual attribute representations. In the sentiment reasoning task, it proposes attribute association analysis to analyze all attributes and dynamically adjust the weight of each attribute representation. The attribute association graph is then mapped to the sentiment space for sentiment reasoning. By constructing an attribute sentiment graph, it is possible to deeply analyze the relationship between emotions and attributes, not only considering a single representation but also integrating the interactions between multiple attributes. This enhances the interpretability of emotion recognition and thus improves the accuracy of sentiment analysis results.

[0078] Furthermore, if Figure 5 As shown, based on the above-mentioned attribute-guided visual emotion analysis method, the present invention also provides a corresponding attribute-guided visual emotion analysis system, wherein the attribute-guided visual emotion analysis system includes:

[0079] The representation processing module 51 is used to obtain the target user's emotion image to be analyzed, perform representation extraction on the emotion image to be analyzed to obtain multiple visual attribute representations, and perform text representation extraction on the emotion image to be analyzed to obtain multiple text attribute representations;

[0080] an attribute analysis module 52 for aligning all the visual attribute representations and all the text attribute representations to obtain a plurality of target visual attribute representations, and performing attribute association analysis on all the target visual attribute representations to obtain attribute association information;

[0081] The emotion prediction module 53 is used to construct an attribute emotion map based on the attribute association information, optimize the attribute emotion map to obtain a target attribute emotion map, and perform emotion prediction based on the target attribute emotion map to obtain an emotion prediction result.

[0082] Furthermore, if Figure 6 As shown, based on the above-mentioned attribute-guided visual sentiment analysis method, the present invention also provides a terminal, which includes a processor 10, a memory 20 and a display 30. Figure 6 Only some of the components of the terminal are shown, but it should be understood that implementation of all of the shown components is not required, and more or fewer components may be implemented instead.

[0083] In some embodiments, the memory 20 may be an internal storage unit of the terminal, such as a hard disk or memory of the terminal. In other embodiments, the memory 20 may also be an external storage device of the terminal, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the terminal. Furthermore, the memory 20 may also include both an internal storage unit of the terminal and an external storage device. The memory 20 is used to store application software and various types of data installed on the terminal, such as the program code of the installation terminal. The memory 20 may also be used to temporarily store data that has been output or is to be output. In one embodiment, the memory 20 stores an attribute-guided visual emotion analysis program 40, and the attribute-guided visual emotion analysis program 40 can be executed by the processor 10, thereby realizing the attribute-guided visual emotion analysis method in the present application.

[0084] In some embodiments, the processor 10 may be a central processing unit (CPU), a microprocessor, or other data processing chip, configured to execute program codes or process data stored in the memory 20, such as executing the attribute-guided visual sentiment analysis method.

[0085] In some embodiments, the display 30 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen. The display 30 is used to display information on the terminal and to display a visual user interface. The components of the terminal communicate with each other via a system bus.

[0086] In one embodiment, when the processor 10 executes the attribute-guided visual sentiment analysis program 40 in the memory 20, the following steps are implemented:

[0087] Acquire an emotion image to be analyzed of a target user, perform characterization extraction on the emotion image to be analyzed to obtain multiple visual attribute characterizations, and perform text characterization extraction on the emotion image to be analyzed to obtain multiple text attribute characterizations;

[0088] Aligning all the visual attribute representations and all the text attribute representations to obtain a plurality of target visual attribute representations, and performing attribute association analysis on all the target visual attribute representations to obtain attribute association information;

[0089] An attribute emotion map is constructed according to the attribute association information, the attribute emotion map is optimized to obtain a target attribute emotion map, and emotion prediction is performed according to the target attribute emotion map to obtain an emotion prediction result.

[0090] The step of obtaining the target user's emotion image to be analyzed, extracting a representation of the emotion image to be analyzed to obtain a plurality of visual attribute representations, and extracting a text representation of the emotion image to be analyzed to obtain a plurality of text attribute representations specifically includes:

[0091] Obtain the target user's emotional image to be analyzed and build a multi-level attribute expert module;

[0092] Extracting a representation of the emotion image to be analyzed by the multi-level attribute expert module to obtain a multi-level attribute representation, wherein the multi-level attribute representation includes a low-level attribute representation, a mid-level attribute representation, and a high-level attribute representation;

[0093] Performing nonlinear transformation on the multi-level attribute representation to obtain multiple visual attribute representations;

[0094] Text representation extraction is performed on the emotional image to be analyzed to obtain multiple text attribute representations.

[0095] The nonlinear transformation of the multi-level attribute representation is specifically performed as follows:

[0096] ;

[0097] The text representation extraction of the emotional image to be analyzed is specifically:

[0098] ;

[0099] in, is the visual attribute representation set, is the activation function, is a low-level attribute representation, is the intermediate attribute representation, For high-level attribute representation, and are all learnable parameters, is the number of attribute modules, is the text attribute representation set, For text adapters, For text encoder, Encode attribute embeddings.

[0100] The step of aligning all the visual attribute representations and all the text attribute representations to obtain a plurality of target visual attribute representations specifically includes:

[0101] performing scale assimilation on all the visual attribute representations and all the text attribute representations, and aligning the scale-assimilated visual attribute representations and text attribute representations to obtain a plurality of aligned visual attribute representations;

[0102] Loss calculation is performed on the visual attribute representations and text attribute representations after assimilation at all scales to obtain attribute alignment loss, and all aligned visual attribute representations are optimized according to the attribute alignment loss to obtain multiple target visual attribute representations.

[0103] The performing of attribute association analysis on all the target visual attribute representations to obtain attribute association information specifically includes:

[0104] Calculating the attribute relevance of all the target visual attribute representations to obtain a plurality of relevance scores;

[0105] Attribute association analysis is performed on all attributes according to all the correlation scores to obtain attribute association information.

[0106] The step of constructing an attribute emotion map based on the attribute association information, optimizing the attribute emotion map to obtain a target attribute emotion map, and performing emotion prediction based on the target attribute emotion map to obtain an emotion prediction result specifically includes:

[0107] Performing spatial mapping on the attribute association information to obtain an attribute mapping result, and constructing an attribute emotion map based on the attribute mapping result;

[0108] Performing loss calculation on all the visual attribute representations to obtain sentiment classification loss, and performing total loss calculation on the sentiment classification loss and the attribute alignment loss to obtain a target total loss;

[0109] The attribute emotion map is optimized according to the target total loss to obtain a target attribute emotion map, and emotion prediction is performed according to the target attribute emotion map to obtain an emotion prediction result, wherein the emotion prediction result includes emotion information and emotion interpretation.

[0110] The loss calculation for all the visual attribute representations is specifically performed as follows:

[0111] ;

[0112] The total loss calculation of the sentiment classification loss and the attribute alignment loss is specifically as follows:

[0113] ;

[0114] in, is the sentiment classification loss, is the total number of sentiment categories, is the number of attribute modules, For the True labels, is a nonlinear mapping, F is a set of visual attribute representations, For the association analysis between attributes, is the number of sentiment categories, is the target total loss, is a hyperparameter, is the attribute alignment loss.

[0115] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores an attribute-guided visual sentiment analysis program, and when the attribute-guided visual sentiment analysis program is executed by a processor, it implements the steps of the attribute-guided visual sentiment analysis method as described above.

[0116] In summary, the present invention provides a method, system, terminal and storage medium for visual sentiment analysis based on attribute guidance, the method comprising: obtaining an emotional image to be analyzed of a target user, extracting a representation of the emotional image to be analyzed to obtain multiple visual attribute representations, and extracting a textual representation of the emotional image to be analyzed to obtain multiple textual attribute representations; aligning all the visual attribute representations and all the textual attribute representations to obtain multiple target visual attribute representations, and performing attribute association analysis on all the target visual attribute representations to obtain attribute association information; constructing an attribute sentiment map based on the attribute association information, optimizing the attribute sentiment map to obtain a target attribute sentiment map, and performing sentiment prediction based on the target attribute sentiment map to obtain a sentiment prediction result. The present invention adopts a multi-level attribute expert feature extraction method and introduces a text-guided multi-level alignment module, while extracting multiple visual attribute representations under the guidance of textual attribute representations to optimize the extraction of visual attribute representations; proposes attribute association analysis in the sentiment reasoning task to analyze all attributes and dynamically adjust the weight of each attribute representation, and maps the attribute association map to the sentiment space for sentiment reasoning. By constructing an attribute sentiment map, we can deeply analyze the relationship between emotions and attributes. It not only considers a single representation, but also integrates the interactions between multiple attributes, enhances the interpretability of emotion recognition, and thus improves the accuracy of sentiment analysis results.

[0117] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or terminal comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or terminal. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or terminal comprising the element.

[0118] Of course, those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing related hardware (such as a processor, controller, etc.) through a computer program. The program can be stored in a computer-readable storage medium that can be read by a computer. When executed, the program can include the processes in the above-described method embodiments. The computer-readable storage medium can be a memory, a magnetic disk, an optical disk, etc.

[0119] It should be understood that the application of the present invention is not limited to the above examples. For those skilled in the art, improvements or changes can be made based on the above description. All these improvements and changes should fall within the scope of protection of the claims attached to the present invention.

Claims

1. A visual sentiment analysis method based on attribute guidance, characterized in that: The attribute-guided visual sentiment analysis method includes: Acquire an emotion image to be analyzed of a target user, perform characterization extraction on the emotion image to be analyzed to obtain multiple visual attribute characterizations, and perform text characterization extraction on the emotion image to be analyzed to obtain multiple text attribute characterizations; Aligning all the visual attribute representations and all the text attribute representations to obtain a plurality of target visual attribute representations, and performing attribute association analysis on all the target visual attribute representations to obtain attribute association information; The aligning of all the visual attribute representations and all the text attribute representations to obtain multiple target visual attribute representations specifically includes: performing scale assimilation on all the visual attribute representations and all the text attribute representations, and aligning all the scale-assimilated visual attribute representations and text attribute representations to obtain a plurality of aligned visual attribute representations; Performing loss calculation on the assimilated visual attribute representations and text attribute representations at all scales to obtain attribute alignment loss, and optimizing all aligned visual attribute representations based on the attribute alignment loss to obtain multiple target visual attribute representations; Constructing an attribute emotion map based on the attribute association information, optimizing the attribute emotion map to obtain a target attribute emotion map, and performing emotion prediction based on the target attribute emotion map to obtain an emotion prediction result; The step of constructing an attribute emotion map based on the attribute association information, optimizing the attribute emotion map to obtain a target attribute emotion map, and performing emotion prediction based on the target attribute emotion map to obtain an emotion prediction result specifically includes: Performing spatial mapping on the attribute association information to obtain an attribute mapping result, and constructing an attribute emotion map based on the attribute mapping result; Performing loss calculation on all the visual attribute representations to obtain sentiment classification loss, and performing total loss calculation on the sentiment classification loss and the attribute alignment loss to obtain a target total loss; Optimizing the attribute emotion map according to the target total loss to obtain a target attribute emotion map, and performing emotion prediction based on the target attribute emotion map to obtain an emotion prediction result, wherein the emotion prediction result includes emotion information and emotion interpretation; The loss calculation for all the visual attribute representations is specifically as follows: ； The total loss calculation of the sentiment classification loss and the attribute alignment loss is specifically as follows: ； in, is the sentiment classification loss, is the total number of sentiment categories, is the number of attribute modules, For the True labels, is a nonlinear mapping, F is a set of visual attribute representations, For the association analysis between attributes, is the number of sentiment categories, is the target total loss, is a hyperparameter, is the attribute alignment loss.

2. The attribute-guided visual sentiment analysis method according to claim 1 is characterized in that: The step of obtaining an emotional image to be analyzed of a target user, extracting a representation of the emotional image to be analyzed to obtain a plurality of visual attribute representations, and extracting a textual representation of the emotional image to be analyzed to obtain a plurality of textual attribute representations specifically includes: Obtain the target user's emotional image to be analyzed and build a multi-level attribute expert module; Extracting a representation of the emotion image to be analyzed by the multi-level attribute expert module to obtain a multi-level attribute representation, wherein the multi-level attribute representation includes a low-level attribute representation, a mid-level attribute representation, and a high-level attribute representation; Performing nonlinear transformation on the multi-level attribute representation to obtain multiple visual attribute representations; Text representation extraction is performed on the emotional image to be analyzed to obtain multiple text attribute representations.

3. The attribute-guided visual sentiment analysis method according to claim 2 is characterized in that: The nonlinear transformation of the multi-level attribute representation is specifically performed as follows: ； The text representation extraction of the emotional image to be analyzed is specifically: ； in, is the visual attribute representation set, is the activation function, is a low-level attribute representation, is the intermediate attribute representation, For high-level attribute representation, and are all learnable parameters, is the number of attribute modules, is the text attribute representation set, For text adapters, For text encoder, Encode attribute embeddings.

4. The attribute-guided visual sentiment analysis method according to claim 1, characterized in that: The performing attribute association analysis on all the target visual attribute representations to obtain attribute association information specifically includes: Calculating the attribute relevance of all the target visual attribute representations to obtain a plurality of relevance scores; Attribute association analysis is performed on all attributes according to all the correlation scores to obtain attribute association information.

5. A visual sentiment analysis system based on attribute guidance, characterized in that: The attribute-guided visual emotion analysis system is applied to the attribute-guided visual emotion analysis method according to any one of claims 1 to 4, and the attribute-guided visual emotion analysis system includes: a representation processing module, configured to obtain an emotion image to be analyzed of a target user, perform representation extraction on the emotion image to be analyzed to obtain a plurality of visual attribute representations, and perform text representation extraction on the emotion image to be analyzed to obtain a plurality of text attribute representations; an attribute analysis module, configured to align all the visual attribute representations and all the text attribute representations to obtain a plurality of target visual attribute representations, and perform attribute association analysis on all the target visual attribute representations to obtain attribute association information; The emotion prediction module is used to construct an attribute emotion map based on the attribute association information, optimize the attribute emotion map to obtain a target attribute emotion map, and perform emotion prediction based on the target attribute emotion map to obtain an emotion prediction result.

6. A terminal, characterized in that: The terminal includes: a memory, a processor, and an attribute-guided visual emotion analysis program stored in the memory and executable on the processor. When the attribute-guided visual emotion analysis program is executed by the processor, the steps of the attribute-guided visual emotion analysis method according to any one of claims 1 to 4 are implemented.

7. A computer-readable storage medium, characterized in that The computer-readable storage medium stores an attribute-guided visual sentiment analysis program, and when the attribute-guided visual sentiment analysis program is executed by a processor, the steps of the attribute-guided visual sentiment analysis method according to any one of claims 1 to 4 are implemented.

Images