A method and system for generating subjective visual emotion interpretation

Abstract

This invention relates to the field of artificial intelligence technology and discloses a method and system for generating subjective visual sentiment interpretations. The construction and training process of the sentiment interpretation model includes: statistically analyzing the normalized frequencies of each sentiment label to obtain a consensus ratio, which is then mapped to multiple consensus levels; extracting global semantic features and local patch features of the image; encoding the consensus levels into bucket cues, concatenating them with the global semantic features, and inputting them into a visual sentiment encoder to generate visual sentiment features, which are then fused with the local patch features to form a dual-space visual representation; concatenating the bucket cues with the cue word text to form a consensus cue; and inputting the consensus cue and the dual-space visual representation together into a large language model to output the sentiment interpretation text. This invention models the degree of sentiment consensus among different observers, enabling the system to capture diverse subjective emotional responses triggered by visual content, thereby improving the model's ability to perceive the subjectivity and diversity of emotions.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, specifically to a method and system for generating subjective visual emotion interpretation. Background Technology

[0002] In recent years, the rapid development of artificial intelligence has increased the demand for emotional interaction. For example, companion robots capable of recognizing and expressing emotions can provide more harmonious companionship, especially for the elderly and children. To endow machines with human-like emotions, understanding human emotional experiences in response to different visual stimuli is crucial. The focus of visual emotion interpretation generation is understanding the emotions viewers perceive from images, especially the reasons behind these emotions. This task can infer and interpret human emotional states, laying the foundation for applications such as mental health monitoring, personalized emotional intervention, and empathic virtual agents.

[0003] Recently, thanks to the development of Large Language Models (LLMs), this field has made initial progress. By combining CLIP-based image encoders with large language models such as GPT, sentiment interpretation generation can be transformed into a text generation task. After further fine-tuning on sentiment datasets, large language models can predict sentiment categories based on images and generate explanatory sentences. However, existing methods still face challenges in providing fair and diverse image-evoked sentiment understanding. Psychological research shows that human emotional perception is influenced by multiple factors, including neurophysiology, cultural background, personal past experiences, and social and even environmental factors. These factors determine the inherent subjectivity challenges in visual sentiment interpretation tasks. Existing methods directly optimize the fitting of true labels, causing the model to favor common dominant emotions while neglecting less common but equally important sentiment perspectives. This leads to two main problems: 1) Human-provided interpretations have significant emotional uniqueness, and existing models perform poorly in learning low-frequency emotions; 2) Many images lack a consistent perception of the dominant emotion within a group, making it difficult for the model to determine the dominant emotion. Summary of the Invention

[0004] To address the aforementioned technical problems, this invention provides a method and system for generating subjective visual sentiment interpretation based on consensus prompts and a large language model, which solves this problem by introducing an additional control signal (i.e., sentiment consensus level).

[0005] This invention calculates the normalized sentiment tag frequency as the consensus rate for each "image-sentiment tag-explanatory text" data set, and then discretizes it into predefined consensus levels as control signals during training. This allows the model to learn to generate diverse sentiment interpretations at different consensus levels, balancing mainstream and minority viewpoints.

[0006] To bridge the emotional gap between visual content and abstract emotions, this invention introduces an emotional space, which organizes features by emotional categories, promoting consistency of features within the same emotion while enhancing the distinguishability between different emotional features. A dual-space visual representation method is proposed, fusing the emotional and semantic features of an image to provide effective emotional cues and visual evidence for generating emotional interpretations.

[0007] To solve the above-mentioned technical problems, the present invention adopts the following technical solution:

[0008] This invention provides a method for generating subjective visual emotion interpretations. The construction and training process of the emotion interpretation model includes:

[0009] Construct an image dataset, where each image sample includes the image itself, along with its corresponding sentiment label and explanatory text;

[0010] Based on multiple sentiment tags for the same image, the consensus ratio is obtained by calculating the normalized frequency of each sentiment tag. The consensus ratio is then mapped to multiple predefined consensus levels using a discretization function. ;

[0011] Extract image The global semantic features and local patch features are used; the consensus level is encoded as bucket prompts, which are concatenated with the global semantic features and input into the visual sentiment encoder to generate visual sentiment features, which are then fused with the local patch features to form a dual-space visual representation;

[0012] The bucket cue is connected with the cue word text to form a consensus cue. The consensus cue and the dual-space visual representation are input into the decoder of the large language model. The decoder of the large language model outputs an autoregressive sentiment explanation text that includes sentiment category and explanation text.

[0013] We optimize the sentiment interpretation model by combining language generation loss based on autoregressive prediction of target text, sentiment feature alignment loss based on cross-modal sentiment space feature matching, and sentiment classification loss based on supervision of real sentiment label classification.

[0014] In one embodiment, based on multiple sentiment tags of the same image, the normalized frequency of each sentiment tag is counted to obtain a consensus ratio, and the consensus ratio is mapped to multiple predefined consensus levels through a discretization function. Specifically, it includes:

[0015] For a given image And the corresponding source Emotional interpretation texts for different audiences Calculate emotions consensus ratio :

[0016] ;

[0017] in, This represents the sentiment label given to the image by the nth viewer. express Corresponding explanatory text; It is an indicator function, when When it is 1, when The time is 0;

[0018] Using bucketing technology Convert to Discrete consensus levels :

[0019] ;

[0020] in, It is a Bucketization function for equally spaced boundaries.

[0021] In one embodiment, the image extraction Global semantic features and local patch features, specifically including:

[0022] Image Input to image encoder Extracting global and patch-level semantics:

[0023] ;

[0024] For global semantic features of the image, This represents the Mth local patch feature of the image, where M represents the total number of local patches in the image.

[0025] In one embodiment, the step of encoding the consensus level as bucket prompts, concatenating them with global semantic features, inputting them into a visual sentiment encoder to generate visual sentiment features, and then fusing them with local patch features to form a dual-space visual representation specifically includes:

[0026] Use learnable bucket embedding layers Consensus Level Encode to obtain bucket hints ;

[0027] Bucket prompt With global semantic features Connect them and input them into a visual emotion encoder implemented based on linear layers. To obtain visual emotional features :

[0028] ;

[0029] Indicates column-level concatenation;

[0030] Visual emotional features Combined with local patch features, a dual-space visual representation is obtained. :

[0031] ;

[0032] Indicates concatenation by line; This represents the Mth local patch feature of the image, where M represents the total number of local patches in the image.

[0033] In one embodiment, the step of concatenating the bucket cue with the cue text to form a consensus cue specifically includes:

[0034] Bucket prompt By concatenating the word embeddings with the cue text, we obtain the consensus cue T:

[0035] ;

[0036] in This indicates concatenation by line. Represents word embedding layer, It is the Lth word segment of the prompt text, where L is the total number of words in the prompt text.

[0037] In one embodiment, the process of inputting consensus cues and dual-space visual representations into the decoder of a large language model, and then outputting sentiment explanation text containing sentiment categories and explanatory text through the autoregressive output of the decoder of the large language model, specifically includes:

[0038] The consensus cue T and the dual-space visual representation V are input into the decoder of the large language model. In order to predict sentiment categories and interpretations:

[0039] ;

[0040] in, and These are the hidden states of the sentiment category and the explanatory text; and The input is fed into a linear layer and a softmax activation function to derive the probability distribution over the vocabulary. After passing through the argmax function, the sentiment interpretation text of the image is predicted. The sentiment interpretation text includes the sentiment category and the explanatory text.

[0041] In one embodiment, the language generation loss based on target-interpretive text autoregressive prediction specifically includes:

[0042] Language generation loss for:

[0043] ;

[0044] For the first part of the text used to predict the sentiment interpretation of the text The probability distribution of each target word The first part of the predicted sentiment interpretation text represents the... One target word, This represents the first t-1 target words in the predicted sentiment interpretation text.

[0045] In one embodiment, the sentiment feature alignment loss based on cross-modal sentiment space feature matching specifically includes:

[0046] Sentiment feature alignment loss for:

[0047] ;

[0048] For the aforementioned visual emotional features, Indicates the emotional characteristics of the explanatory text;

[0049] ;

[0050] The hidden state of the explanatory text output by the decoder of the large language model is denoted as . This indicates the semantic features of the explanatory text. Indicates average pooling. For bucket hints, Indicates column-level concatenation. This represents a text sentiment encoder implemented based on linear layers.

[0051] In one embodiment, the sentiment classification loss based on real sentiment label classification supervision specifically includes:

[0052] Sentiment Classification Loss :

[0053] ;

[0054] in, and These represent the learnable visual sentiment classifiers. and text sentiment classifier Predicted sentiment distribution For the aforementioned visual emotional features, Indicates the emotional characteristics of the explanatory text. It is the sentiment label of the image sample.

[0055] In a second aspect, the present invention provides a computer system including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the method of any embodiment of the first aspect.

[0056] This invention, by introducing a consensus-guided mechanism and combining visual emotion recognition and language generation models, achieves subjective modeling and accurate interpretation of visually induced emotions, and has the following technical effects:

[0057] 1. Enhanced understanding of subjective emotions: By modeling the degree of emotional consensus among different observers, the system is able to capture diverse subjective emotional responses triggered by visual content, thereby improving the model's ability to perceive the subjectivity and diversity of emotions.

[0058] 2. Achieve controllable emotional interpretation generation: Utilize consensus level as a control signal to guide the system to generate explanatory text consistent with specific emotional cognitive levels, thereby achieving fine-grained control and diversified output of emotional expression.

[0059] 3. Enhance sentiment consistency and semantic credibility: By designing feature-level and label-level sentiment consistency constraint mechanisms, the generated text is made to better match the real emotions evoked by the input image, thereby enhancing the semantic accuracy and sentiment credibility of the interpreted text.

[0060] 4. Excellent application scalability: The method of this invention can be widely applied to fields such as emotional health assessment, educational companionship, virtual human interaction, and digital art understanding, providing effective technical support for building more humanized and emotionally interactive intelligent systems. Attached Figure Description

[0061] Figure 1 This is a flowchart of the method in an embodiment of the present invention.

[0062] Figure 2 This is a schematic diagram of the framework of the present invention. Detailed Implementation

[0063] A preferred embodiment of the present invention will now be described in detail with reference to the accompanying drawings.

[0064] like Figure 1 As shown, this invention provides a method for generating subjective visual emotion interpretations. The construction and training process of the emotion interpretation model includes the following steps:

[0065] S1, Construct an image dataset. Each image sample in the image dataset includes the image, its corresponding sentiment label, and explanatory text.

[0066] S2, based on multiple sentiment tags of the same image, calculates the normalized frequency of each sentiment tag to obtain the consensus ratio, and maps the consensus ratio to multiple predefined consensus levels through a discretization function. ;

[0067] S3, Extract Image The global semantic features and local patch features are used; the consensus level is encoded as bucket prompts, which are concatenated with the global semantic features and input into the visual sentiment encoder to generate visual sentiment features, which are then fused with the local patch features to form a dual-space visual representation;

[0068] S4. Connect the bucket cue with the cue word text to form a consensus cue. Input the consensus cue and the dual-space visual representation into the decoder of the large language model. The decoder of the large language model outputs an autoregressive sentiment explanation text containing sentiment category and explanation text.

[0069] S5 combines language generation loss based on autoregressive prediction of target explanatory text, sentiment feature alignment loss based on cross-modal sentiment space feature matching, and sentiment classification loss based on supervision of real sentiment label classification to optimize the sentiment explanation model.

[0070] Through the above technical solutions, this invention achieves integrated processing of visually induced emotions recognition and interpretation, breaking through the problem that traditional visual emotion analysis focuses on classification while neglecting interpretation, and providing a theoretical basis and technical support for building an emotional intelligence system with greater human-computer interaction value.

[0071] The purpose of this invention is to provide a method for recognizing and interpreting visually induced emotions, addressing the problems of existing methods' inability to effectively model emotional subjectivity and lack of high-quality emotional interpretation. By introducing a consensus-guided control mechanism, dual-space visual representation, and an emotion consistency learning strategy, accurate recognition and interpretation of visual emotions are achieved.

[0072] This invention introduces a consensus-level subjective modeling mechanism: the group's emotional response is discretized and graded according to the degree of consensus, and used as a control signal to guide the generation of emotional interpretations, thereby improving the model's ability to model subjectivity and diversity and supporting the controllable generation of emotional interpretations.

[0073] This invention enhances semantic and emotional perception capabilities by constructing a dual-space visual representation: it integrates visual semantic features and subjective emotional features to construct a unified dual-space visual representation structure, enabling the model to simultaneously perceive the objective attributes of image content and the subjective emotions it evokes, thereby improving the accuracy of emotional interpretation and consistency with human cognition.

[0074] This invention designs an emotion consistency learning mechanism to ensure the reliability of emotion expression: introducing consistency constraints at the feature level and label level to ensure that the generated text interpretation is consistent with the image content in terms of semantic expression and emotion attribution, effectively improving the emotion fidelity and semantic relevance of the emotion interpretation.

[0075] This invention possesses strong generative capabilities and scalability: by combining a large-scale pre-trained language model with a controllable generation strategy, the method of this invention not only achieves good results in sentiment classification accuracy, but also has the ability to generate natural, fluent, and emotionally relevant interpretations, and has broad application potential.

[0076] In one embodiment, the construction of the image dataset in step S1 specifically includes:

[0077] The input image database was obtained using two art image datasets, ArtEmisv1.0 and ArtEmisv2.0, and the Affection real-world image dataset, containing image works and their corresponding sentiment labels and explanatory texts. The Affection dataset was constructed using real-world images from the MS-COCO dataset, the Flickr30k Entities dataset, the Visual Genome dataset, and the FI dataset.

[0078] For the ArtEmisv1.0 and ArtEmisv2.0 datasets, the standard preprocessing workflow provided in the official repository was followed. To comprehensively evaluate the model of this invention, five test set segments were defined based on the sentiment dominance and diversity of the images. Sentiment dominance includes test images where a particular sentiment is overwhelmingly dominant among their annotators, while other test images are classified as non-dominant. For sentiment diversity, the test images in the dataset are categorized into those containing a single sentiment, consistent sentiment, and contradictory sentiment.

[0079] In one embodiment, the image extraction in step S3 Global semantic features and local patch features, specifically including:

[0080] This invention employs a CLIP model image encoder with frozen parameters, designed to align image and text modalities. In the CLIP space, semantically relevant features are close to each other. For example... Figure 2 As shown, given an image The present invention extracts image-level and patch-level semantics as follows:

[0081] ;

[0082] in, For global semantic features of the image, Represents the dimensions of visual features. This represents the Mth local patch feature of the image, where M represents the total number of local patches in the image.

[0083] This invention achieves fair and diverse visual emotion reasoning by explicitly modeling the subjectivity of emotional experiences among different groups. It proposes a consensus-based emotion modeling method to guide the model to adapt to different emotional perspectives.

[0084] In one embodiment, step S2 involves calculating the normalized frequencies of each sentiment tag to obtain a consensus ratio. This consensus ratio is then mapped to multiple predefined consensus levels using a discretization function. Specifically, this includes:

[0085] The consensus level is approximated by the normalized frequency of each sentiment label in the image; this frequency is called the consensus ratio. For a given image in the image dataset... and its origin Emotional interpretation texts for different audiences Calculate emotions consensus ratio The formula is as follows:

[0086] ;

[0087] in, It is an indicator function, when It equals 1 if it is true, otherwise it is 0.

[0088] To obtain the consensus ratio of all training samples The present invention constructs Each bucket is used to capture different samples, that is... Convert to Discrete consensus levels The specific method is to use the bucketing technique, and the formula is as follows:

[0089] .

[0090] in, It is a A bucketing function with equally spaced boundaries. This bucketing causes low-overlap training triples to be reassigned to buckets with lower consensus (e.g., ...). Training triples with high overlap are grouped into buckets with higher consensus (e.g., ...). ).

[0091] In one embodiment, step S3, which encodes the consensus level as bucket prompts, concatenates them with global semantic features, inputs them into a visual sentiment encoder to generate visual sentiment features, and then fuses them with local patch features to form a dual-space visual representation, specifically includes:

[0092] To generate sentiment interpretations, models require comprehensive visual understanding, capturing both semantic clarity and sentiment accuracy. While traditional visual encoders are effective at extracting scene semantics, they struggle to model subtle emotional relationships. To bridge this gap, this invention proposes jointly utilizing a semantic encoder to extract objective semantic features and a visual sentiment encoder to capture nuanced emotional differences.

[0093] The raw visual features provided by the CLIP model's image encoder do not contain emotional meaning. Therefore, this invention introduces a visual emotion encoder to construct an emotion representation space, clustering similar emotions together and separating dissimilar emotions. Based on the fact that emotions originate from the overall visual environment, this invention uses global semantic features. Initialize sentiment encoding. However, in sentiment interpretation, the same image can evoke multiple emotions depending on individual experience, meaning that visual features alone cannot fully capture the different perceptions of emotions among different people. To address this, this invention explicitly conditionalizes sentiment representation to a level of consistency among a given image-sentiment label-interpretive text triplet (image sample). This invention incorporates bucket hints. With global semantic features Connect them and feed them into the visual emotion encoder to obtain visual emotion features:

[0094] ;

[0095] in This indicates column-level concatenation. This represents a visual emotion encoder, implemented through a linear layer.

[0096] Visual emotional features By combining local patch features, a dual-space visual representation is obtained. :

[0097] .

[0098] This invention believes It captures the emotional tone of the image, while local patch features It provides the fine-grained visual evidence needed to generate reasonable explanations.

[0099] In one embodiment, S4 concatenates the bucket cue with the cue word text to form a consensus cue. The consensus cue and the dual-space visual representation are then input into the decoder of the large language model. The decoder of the large language model then outputs the sentiment category and explanatory text through autoregression. Specifically, this includes:

[0100] To enable the model to dynamically adjust sentiment inference based on different consensus levels, this invention introduces consensus-based bucket hints as control signals. A learnable bucket embedding layer is used. Consensus Level Encode and bucket hints By connecting it with the prompt text, a consensus prompt is obtained. :

[0101] ;

[0102] in, The dimension representing the word embedding can be set to 768 in a preferred embodiment; This indicates concatenation by line. Represents word embedding layer, This is the Lth word segment of the cue word text. Using the cue word text, the GPT decoder is guided to generate interpretations matching different consensus levels. This controllability allows the model of this invention to function as an implicit mixture of experts (but sharing parameters), where one expert (e.g., has...) The model focuses on a set of highly consensus-based image-sentiment label-explanatory text triples (image samples), while parameter sharing among experts allows all experts to share the learned knowledge to cover a rich range of sentiment perspectives.

[0103] Consensus prompts and dual-space visual representation The input is fed into a GPT-based decoder, specifically a GPT2 decoder. To predict sentiment categories and interpret text:

[0104] ;

[0105] in, and These are the hidden states for sentiment categories and explanatory text, respectively. After being processed through a linear layer and input into a softmax activation function, the probability distribution on the vocabulary can be derived from the hidden states. .

[0106] In one embodiment, step S5, which combines the language generation loss based on autoregressive prediction of the target explanatory text, the sentiment feature alignment loss based on cross-modal sentiment space feature matching, and the sentiment classification loss based on real sentiment label classification supervision, optimizes the sentiment explanation model, specifically including:

[0107] The text generated by the sentiment interpretation model should indicate the expected sentiment category and provide an image-based explanation consistent with the sentiment. To achieve this goal, this invention first applies a language generation loss. To encourage the generation of fluent and meaningful sentences, this invention proposes a sentiment interpretation model based on sentiment consensus, unlike typical loss functions, thereby generating a new objective as follows:

[0108] ;

[0109] in, It is the first of the predicted sentiment interpretation texts One target word. However, Since there is no explicit guarantee that the interpretation is consistent with the target emotion, this invention proposes an emotion consistency constraint involving feature-level and label-level regularization.

[0110] Since emotions are inherently subjective and not explicitly encoded, feature-level emotion consistency constraints in raw visual features are achieved by minimizing the distance between two spatial feature representations:

[0111] ;

[0112] and All features reside within the same emotional space, allowing for feature-level alignment. This invention achieves this emotional consistency through mean squared error loss, yielding an emotional feature alignment loss. :

[0113] .

[0114] Unlike direct feature matching, alignment in the sentiment space ensures that the generated interpretation retains the emotional essence of the image, rather than just reflecting semantic similarity. Binding the interpretation to the visual input also prevents it from drifting toward a general emotional expression.

[0115] To ensure that sentiment features can accurately represent different sentiment categories, this invention designs a label-level sentiment consistency constraint. An unconstrained sentiment space can become chaotic, with semantically similar sentiments overlapping, leading to ambiguous sentiment representations. To address this issue, this invention introduces a constraint applied to… and Sentiment classification loss This is to enhance the uniqueness of the emotional space learned. Specifically, Encourage from and The predicted sentiment category matches the actual label:

[0116] ;

[0117] in, and These represent the learnable visual sentiment classifiers. and text sentiment classifier The predicted sentiment distribution. Each sentiment classifier is a combination of ReLU layers, linear layers, and a softmax function. It is the sentiment label in the input triples (image samples). By minimizing The model learned to maintain clear emotional boundaries and generate interpretations that match its emotional labels.

[0118] To train the model in an end-to-end manner, the ultimate goal of this invention is to combine all losses to obtain a total loss function. :

[0119] ;

[0120] in, and It is a hyperparameter used to balance the loss.

[0121] This invention uses the AdamW optimizer for end-to-end training. If the BLEU-4 metric does not improve within 10 consecutive rounds on the validation set, the model is considered to have converged, and training is stopped.

[0122] During the inference phase, given the input image and consensus level, the sentiment interpretation model generates sentiment interpretation text through autoregression. The sentiment interpretation text includes sentiment category and interpretation text, and the output format is "sentiment category because interpretation text".

[0123] It should be understood that although the steps in the flowcharts of the accompanying drawings are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated in this application, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some of the steps in the flowcharts of the accompanying drawings may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least a portion of the steps or stages of other steps.

[0124] In one embodiment, the present invention provides a computer system. The computer system includes a processor, memory, and a network interface connected via a system bus. The processor of the computer system provides computing and control capabilities. The memory of the computer system includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The database of the computer system stores data used in the methods described above. The network interface of the computer system is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements the methods described above.

[0125] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0126] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention, and no reference numerals in the claims should be construed as limiting the scope of the claims.

[0127] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

Claims

1. A method for generating subjective visual emotion interpretation, characterized in that, The construction and training process of the sentiment interpretation model used includes: Construct an image dataset, where each image sample includes the image itself, along with its corresponding sentiment label and explanatory text; Based on multiple sentiment tags for the same image, the consensus ratio is obtained by calculating the normalized frequency of each sentiment tag. The consensus ratio is then mapped to multiple predefined consensus levels using a discretization function. ; Extract image The global semantic features and local patch features are used; the consensus level is encoded as bucket prompts, which are concatenated with the global semantic features and input into the visual sentiment encoder to generate visual sentiment features, which are then fused with the local patch features to form a dual-space visual representation; The bucket cue is connected with the cue word text to form a consensus cue. The consensus cue and the dual-space visual representation are input into the decoder of the large language model. The decoder of the large language model outputs an autoregressive sentiment explanation text that includes sentiment category and explanation text. We optimize the sentiment interpretation model by combining language generation loss based on autoregressive prediction of target text, sentiment feature alignment loss based on cross-modal sentiment space feature matching, and sentiment classification loss based on supervision of real sentiment label classification.

2. The subjective visual emotion interpretation generation method according to claim 1, characterized in that, The consensus ratio is obtained by statistically analyzing the normalized frequencies of multiple sentiment tags based on the same image, and then mapping the consensus ratio to multiple predefined consensus levels using a discretization function. Specifically, it includes: For a given image And the corresponding source Emotional interpretation texts for different audiences Calculate emotions consensus ratio : ； in, This represents the sentiment label given to the image by the nth viewer. express Corresponding explanatory text; It is an indicator function, when When it is 1, when The time is 0; Using bucketing technology Convert to Discrete consensus levels : ； in, It is a Bucketization function for equally spaced boundaries.

3. The subjective visual emotion interpretation generation method according to claim 1, characterized in that, The extracted image Global semantic features and local patch features, specifically including: Image Input to image encoder Extracting global and patch-level semantics: ； For global semantic features of the image, This represents the Mth local patch feature of the image, where M represents the total number of local patches in the image.

4. The subjective visual emotion interpretation generation method according to claim 1, characterized in that, The process of encoding consensus levels into bucket prompts, concatenating them with global semantic features, inputting them into a visual sentiment encoder to generate visual sentiment features, and then fusing them with local patch features to form a dual-space visual representation specifically includes: Use learnable bucket embedding layers Consensus Level Encode to obtain bucket hints ; Bucket prompt With global semantic features Connect them and input them into a visual emotion encoder implemented based on linear layers. To obtain visual emotional features : ； Indicates column-level concatenation; Visual emotional features Combined with local patch features, a dual-space visual representation is obtained. : ； Indicates concatenation by line; This represents the Mth local patch feature of the image, where M represents the total number of local patches in the image.

5. The subjective visual emotion interpretation generation method according to claim 1, characterized in that, The process of concatenating bucket prompts with prompt text to form consensus prompts specifically includes: Bucket prompt By concatenating the word embeddings with the cue text, we obtain the consensus cue T: ； in This indicates concatenation by line. Represents word embedding layer, It is the Lth word segment of the prompt text, where L is the total number of words in the prompt text.

6. The method for generating subjective visual emotion interpretation according to claim 1, characterized in that, The consensus cues and dual-space visual representations are jointly input into the decoder of the large language model. The decoder of the large language model outputs an autoregressive sentiment explanation text containing sentiment categories and explanatory text, specifically including: The consensus cue T and the dual-space visual representation V are input into the decoder of the large language model. In order to predict sentiment categories and interpretations: ； in, and These are the hidden states of the sentiment category and the explanatory text; and The input is fed into a linear layer and a softmax activation function to derive the probability distribution over the vocabulary. After passing through the argmax function, the sentiment interpretation text of the image is predicted. The sentiment interpretation text includes the sentiment category and the explanatory text.

7. The method for generating subjective visual emotion interpretation according to claim 1, characterized in that, The language generation loss based on target-interpretive text autoregressive prediction specifically includes: Language generation loss for: ； For the first part of the text used to predict the sentiment interpretation of the text The probability distribution of each target word The first part of the predicted sentiment interpretation text represents the... One target word, This represents the first t-1 target words in the predicted sentiment interpretation text.

8. The method for generating subjective visual emotion interpretation according to claim 1, characterized in that, The sentiment feature alignment loss based on cross-modal sentiment space feature matching is specifically... include: Sentiment feature alignment loss for: ； For the aforementioned visual emotional features, Indicates the emotional characteristics of the explanatory text; ； The hidden state of the explanatory text output by the decoder of the large language model is denoted as . This indicates the semantic features of the explanatory text. Indicates average pooling. For bucket hints, Indicates column-level concatenation. This represents a text sentiment encoder implemented based on linear layers.

9. The method for generating subjective visual emotion interpretation according to claim 1, characterized in that, The sentiment classification loss based on real sentiment label classification supervision specifically includes: Sentiment Classification Loss : ； in, and These represent the learnable visual sentiment classifiers. and text sentiment classifier Predicted sentiment distribution For the aforementioned visual emotional features, This indicates the emotional characteristics of the explanatory text. It is the sentiment label of the image sample.

10. A computer system comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 9.

Images