Art image emotion understanding modeling method based on background knowledge retrieval enhancement
By introducing a background knowledge retrieval mechanism and a multimodal sentiment prediction network, the problem of lack of background knowledge in the understanding of painting emotions in existing technologies is solved, and a deep understanding and interpretable generation of emotions in art images is achieved.
Patent Information
- Application Number
- CN202511917012.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-18
- Publication Date
- 2026-03-03
AI Technical Summary
Existing methods for understanding artistic emotions lack effective integration with the background of painting creation, making it difficult to capture the complex, ambiguous, and symbolic emotional information in paintings.
By employing a background knowledge-based retrieval enhancement method, high-dimensional visual features of art images are extracted using a frozen CLIP-ViT model. A feature index is then established using FAISS to perform image semantic nearest neighbor retrieval, obtain structured art metadata, and sentiment understanding is achieved through a multimodal art sentiment prediction and interpretation network.
It effectively compensates for the lack of external knowledge support in sentiment reasoning in existing models, improves the ability to discriminate sentiment understanding and the semantic depth and coherence of interpretation generation, and enhances the interpretability and practicality of the model.
Smart Images
Figure SMS_1 
Figure SMS_6 
Figure SMS_10
Abstract
Description
Technical Field
[0001] This invention belongs to the field of image sentiment understanding technology, specifically relating to a method for modeling sentiment understanding in artistic images based on background knowledge retrieval enhancement. Background Technology In the field of artificial intelligence, the core task of art emotion understanding is to combine computer vision and natural language processing technologies to identify the emotion categories (such as joy, sadness, anger, awe, etc.) of paintings, and to explain why these emotions are evoked, generating natural language explanations that conform to human cognition, clarifying which elements triggered the perception of specific emotions. This task goes beyond traditional image classification, requiring machines not only to "judge" emotions but also to have the ability to "reason," that is, to reveal the source and expression of emotions in images in an interpretable way, thereby bridging the gap between art appreciation and AI interpretability.
[0002] In recent years, image sentiment analysis has made progress in fields such as social media, advertising, and emotion recognition. However, most existing methods focus on low-level visual features of natural or figurative images, making it difficult to capture the complex, ambiguous, and symbolic emotional information in paintings. Emotional expression in art is often built upon the synergy of multi-level knowledge, such as the visual composition of the image, the textual description of the work, historical and cultural background, the artist's personal style, and even the underlying symbolic symbols. This knowledge intertwines to construct a complete emotional semantic space. Therefore, constructing a multimodal, multi-level sentiment understanding model for art images that integrates visual information (such as color and composition), textual descriptions (such as painting titles and reviews), and background knowledge (such as the artist's school, subject matter, and symbolic meaning) is of significant research value. Current methods generally rely on surface-level visual features of images in the sentiment modeling process, lacking effective integration of the painting's creative background. Summary of the Invention
[0003] The purpose of this invention is to propose an art image sentiment understanding method based on background knowledge retrieval enhancement, in order to solve the problem that existing art sentiment understanding methods lack background knowledge consideration.
[0004] To solve the above-mentioned technical problems, the present invention adopts the following technical solution: A method for modeling sentiment understanding of art images based on background knowledge retrieval enhancement includes the following steps: Step 1: Collect multiple original art images and preprocess each original art image; Step 2: Use a visual encoder to extract visual features from each preprocessed original image, generate corresponding visual feature vectors, and associate the metadata of each preprocessed image with its visual feature vectors to form an image feature library for similarity retrieval. Step 3: Extract the visual feature vector of the target art image to be processed. Select the original image with the highest similarity to the visual feature vector from the image feature library obtained in Step 2. Use the metadata corresponding to the original image as the background metadata of the target art image to be processed. Obtain the background context text related to the background metadata. Step 4: Construct a multimodal art emotion prediction and interpretation network, which includes an image encoder, a text encoder, a feature fusion unit, and an emotion decoding unit. The image encoder is used to extract image features from the target artistic image; The text encoder is used to extract the background text features of the background context text obtained in step 3; The feature fusion unit is used to fuse image features with background text features to generate a context feature representation; The emotion decoding unit outputs the emotion classification result and the corresponding emotion interpretation based on the contextual feature representation.
[0005] The present invention also has the following features: Furthermore, the preprocessing in step 1 specifically involves normalizing the size of all original artistic images and adjusting them to a fixed resolution of 224×224.
[0006] Further, step 2 specifically involves using the frozen CLIP-ViT / 32 visual encoder to extract visual features from each preprocessed original image and generate corresponding visual feature vectors; then, an index is built based on FAISS, and the metadata of each preprocessed image is associated with its visual feature vector to form an image feature library for similarity retrieval. The metadata includes the artist, title, style, and year of creation corresponding to the original image.
[0007] Furthermore, step 3 includes the following sub-steps: Step 31: Use the frozen CLIP-ViT / 32 visual encoder to extract the visual feature vector of the target art image to be processed, as the target visual feature vector; Step 32: Calculate the feature similarity between the target visual feature vector and any visual feature vector in the image feature library using the following formula:
[0008] in, Represents the target visual feature vector; This represents any visual feature vector in the image feature library; It represents the similarity between the target visual feature vector and visual features in the image feature library; This indicates the total number of original art images; Step 33: Traverse each visual feature vector in the image feature library, select the original image with the highest similarity to the target visual feature vector, and use the metadata corresponding to the original image as the background metadata of the target art image; Step 34: Input the background metadata of the target art image as query text into the DrQA-based retrieval system to obtain multiple candidate articles. Calculate the cosine similarity score between the target visual feature vector and the visual feature vector of each candidate article using the following formula:
[0009] in, The feature vector representing the candidate article; Represents the target visual feature vector; This represents the similarity between the target visual feature vector and the candidate article feature vector; M represents the number of candidate articles; Step 35: Select the top five candidate articles with the highest similarity as context candidate articles, which serve as background context text related to obtaining background metadata.
[0010] Furthermore, in step 4, the image encoder specifically uses the CLIP-ViT / 16 pre-trained model, and the text encoder specifically uses the BERT model.
[0011] Furthermore, the feature fusion unit described in step 4 specifically uses an adaptive weighted fusion method. The concatenated image features and background text features are fed into a fully connected network MLP after passing through an adaptive pooling layer. The output is a two-dimensional vector value to determine the weights of the visual and background features, as shown in the following formula:
[0012] Wherein, APool represents adaptive pooling; Image features that represent artistic images; Background text features representing artistic images; Indicates the weight value; This indicates vector concatenation; Represents a 2-dimensional real vector; The feature fusion unit fuses image features with background text features to generate a contextual feature representation, as shown below:
[0013] in, Represents the visual characteristics of the target artistic image; Features representing background information of the target artistic image; This indicates the weight values of the output; This indicates the final fusion characteristic.
[0014] Furthermore, the sentiment decoding unit in step 4 outputs the sentiment classification result and the corresponding sentiment interpretation based on the contextual feature representation, specifically including the following sub-steps: Step a: Receive contextual feature representation, represent the sentiment sentence to be generated as a structured input text composed of sentiment category and sentiment interpretation, and obtain the corresponding structured input text embedding representation by mapping each token in the text to the corresponding embedding vector; Step b involves using a multi-layer Transformer structure to perform self-attention modeling and cross-attention modeling (fusion of visual and contextual features) on the structured input text embedding representation obtained in step a, thereby obtaining the sentiment output representation. Step c: Use a language prediction head consisting of a linear layer and a softmax layer to generate sentiment classification results and corresponding sentiment interpretations based on the sentiment output representation.
[0015] Compared with the prior art, the present invention has the following technical effects: (I) The art image sentiment understanding method based on background knowledge retrieval enhancement of the present invention introduces background knowledge to enhance the sentiment understanding process. It introduces the background knowledge retrieval mechanism into the art image sentiment understanding process. Through a two-stage image and text retrieval strategy, it combines visual meta-information and structured text knowledge to obtain background text that is highly relevant to the context of the image content. It uses the frozen CLIP-ViT / 32 model to extract high-dimensional visual features of the art image and combines FAISS to establish an efficient feature index, thereby realizing the rapid retrieval of semantic neighbors of the image and obtaining structured art metadata. This effectively makes up for the lack of external knowledge support in the sentiment reasoning of existing models and solves the problem that existing art sentiment understanding methods lack background knowledge consideration.
[0016] (II) The art image sentiment understanding method based on background knowledge retrieval enhancement proposed in this invention proposes a visual-text fusion method based on adaptive weighting. It uses an adaptive pooling mechanism to jointly learn the importance weights of visual features and background text features, effectively improving the discriminative ability of context representation and providing more semantically deep joint features for sentiment decoding.
[0017] (III) The art image sentiment understanding method based on background knowledge retrieval enhancement of the present invention jointly models sentiment category and sentiment interpretation as structured text input, adopts a multi-layer Transformer decoding structure, and introduces image visual features, contextual text features and image description features layer by layer through self-attention and multi-round cross-attention mechanism, which significantly enhances the semantic integrity and sentiment rationality of the generated sentence.
[0018] (IV) The art image sentiment understanding method based on background knowledge retrieval enhancement of the present invention integrates image encoding, text retrieval, feature fusion and sentiment generation tasks into a unified training framework. It is optimized by the joint cross-entropy loss of sentiment classification and interpretation generation, so that the model can improve the coherence and readability of interpretation generation while maintaining discriminability. Detailed Implementation
[0019] The following are specific embodiments of the present invention. It should be noted that the present invention is not limited to the following specific embodiments. All equivalent modifications made based on the technical solutions of this application fall within the protection scope of the present invention.
[0020] The overall technical concept of this embodiment is as follows: First, the art image data is preprocessed to normalize the images to a uniform size to adapt to the input requirements of the visual encoder. Then, a frozen visual encoding model (such as CLIP-ViT) is used to extract deep visual features of the images to capture emotion-related visual cues. To incorporate contextual information, this method employs a two-stage retrieval mechanism: retrieving image metadata (such as author, title, style, etc.) from the art image database, and further retrieving background text information related to the images from a general encyclopedia corpus using the metadata as query conditions to supplement the cultural and semantic background of the images. In the feature modeling stage, the visual features of the image and its background text features are respectively input into the pre-trained image encoder and text encoder for representation learning. To enhance the fusion effect of multimodal features, an adaptive weighting mechanism is adopted to dynamically fuse image and text features, retaining key semantic information and suppressing redundant interference, thereby obtaining a context-aware joint representation. Subsequently, the fused feature representation is processed using an emotion decoder, and a joint training strategy is employed to simultaneously generate emotion classification and emotion explanation text. By constructing a structured output format, the model can generate explanatory text while outputting sentiment categories, thereby enhancing the model's interpretability and usability.
[0021] Specifically, a method for modeling sentiment understanding of artistic images based on background knowledge retrieval enhancement includes the following steps: Step 1: Collect multiple original art images, and preprocess each original art image; In this step, an art image dataset is acquired, and all image samples undergo uniform preprocessing. Specifically, the original image data is normalized to a fixed resolution of 224×224 to suit the input requirements of the subsequent visual encoder (ViT).
[0022] Step 2: Use a visual encoder to extract visual features from each preprocessed original image and generate a visual feature vector. Associate the metadata of each preprocessed image with its visual feature vector to form an image feature library for similarity retrieval. Visual features were extracted from all art images in the Wikipedia database using a frozen CLIP-ViT / 32 visual encoder to generate visual feature vectors, and a feature index library was built using FAISS (Vector Database). Metadata information for each image (including artist, title, style, and year of creation) was associated with its visual features to form a feature database for similarity retrieval.
[0023] Step 3: Extract the visual feature vector of the target art image to be processed, select the original image with the highest similarity to the visual feature vector in the image feature library, use the metadata corresponding to the original image as the background metadata of the target art image, and obtain the background context text related to the background metadata. Step 4: Construct a multimodal art sentiment prediction and interpretation network, which includes an image encoder, a text encoder, a feature fusion unit, and a sentiment decoding unit. The image encoder is used to extract image features from the target artistic image; The text encoder is used to extract background text features that are consistent with the background context text; The feature fusion unit fuses image features with background text features to generate a contextual feature representation. The feature fusion unit employs an adaptive weighted fusion strategy. After concatenating the image and text features, the data passes through an adaptive pooling module and is input into a multilayer perceptron (MLP) network, outputting two weight vectors to adjust the fusion ratio of the visual and textual modalities. Finally, the fused contextual feature representation is obtained. The sentiment decoding unit outputs the sentiment classification result and the corresponding sentiment interpretation based on this contextual feature representation.
[0024] As a preferred approach, in step 1, the original artistic images are normalized in size, adjusting all original artistic images to a fixed resolution of 224×224.
[0025] As a preferred approach, step 2 specifically involves using a frozen CLIP-ViT / 32 visual encoder to extract visual features from each preprocessed original image, generating corresponding visual feature vectors; establishing an index based on FAISS, and then associating the metadata of each preprocessed image with its visual feature vector to form an image feature library for similarity retrieval. The metadata includes the original image artist, title, style, and year of creation.
[0026] Specifically, step 3 includes the following sub-steps: Step 31: Use the frozen CLIP-ViT / 32 visual encoder to extract the visual feature vector of the target art image to be processed, as the target visual feature vector; Step 32: Calculate the feature similarity between the target visual feature vector and any visual feature vector in the image feature library using the following formula:
[0027] in, Represents the target visual feature vector; This represents any visual feature vector in the image feature library; It represents the similarity between the target visual feature vector and visual features in the image feature library; This indicates the total number of original art images; Step 33: Traverse each visual feature vector in the image feature library, select the original image with the highest similarity to the target visual feature vector, and use the metadata corresponding to the original image as the background metadata of the target art image; Step 34: Input the background metadata of the target art image as query text into the DrQA-based retrieval system to obtain multiple candidate articles. Calculate the cosine similarity score between the target visual feature vector and the visual feature vector of each candidate article using the following formula:
[0028] in, The feature vector representing the candidate article; Represents the target visual feature vector; This represents the similarity between the target visual feature vector and the candidate article feature vector; M Indicates the number of candidate articles; Among them, a text retrieval system based on DrQA was developed. This system is an existing system that performs TF-IDF vectorization on the Wikipedia corpus, calculates the cosine similarity between the query text and candidate articles, and selects the top 5 articles with the highest similarity as candidate background texts.
[0029] Step 35: Select the top five candidate articles with the highest similarity as context candidate articles to form background context text related to the target art image.
[0030] As a preferred approach, the image encoder in step 4 is specifically a CLIP-ViT / 16 pre-trained model, and the text encoder is specifically a BERT model. The image encoder uses the CLIP-ViT / 16 pre-trained model to divide the input image into fixed-size image patches and extracts its global visual features through a multi-layer Transformer network.
[0031] The feature fusion unit specifically uses an adaptive weighted fusion method. The concatenated visual and background text features are passed through an adaptive pooling layer and then fed into a fully connected MLP network. The output is a two-dimensional vector value to determine the weights of the visual and background features, as shown in the following equation:
[0032] in, APool Indicates adaptive pooling; Image features that represent artistic images; Features that represent background information in artistic images; Indicates the weight value; This indicates vector concatenation; Represents a 2-dimensional real vector; The feature fusion unit fuses image features with background text features to generate a contextual feature representation, as shown below:
[0033] in, Represents the visual characteristics of the target artistic image; Features representing background information of the target artistic image; This indicates the weight values of the output; This indicates the final fusion characteristic.
[0034] Specifically, the sentiment decoding unit outputs the sentiment classification result and the corresponding sentiment interpretation based on the contextual feature representation, which includes the following sub-steps: Step a: Receive contextual feature representation, represent the sentiment sentence to be generated as a structured input text composed of sentiment category and sentiment interpretation, and obtain the corresponding structured input text embedding representation by mapping each token in the text to the corresponding embedding vector; Furthermore, the sentiment decoding unit constructs a structured sentiment sentence template in the format: "The emotion is <sentiment category> because <sentiment explanation>". This template is input into the model and first undergoes three embedding layers to initialize the representation: the word embedding layer converts each token into a vector representation; the position embedding layer introduces sequence position information; and the paragraph embedding layer distinguishes the sentiment category part from the explanation part.
[0035] Step b: Using a multi-layer Transformer structure, self-attention modeling and cross-attention modeling fused with visual and contextual features are performed sequentially on the structured input text embedding representation to obtain the sentiment output representation; Step c: Using a language prediction head consisting of a linear layer and a softmax layer, generate sentiment classification results and corresponding sentiment interpretations based on the sentiment output representation.
[0036] The obtained contextual representations are fed into a feedforward neural network and a layer normalization module to further refine high-level semantics. The final generated hidden states are mapped to the vocabulary space through a language prediction head, completing the joint generation task of sentiment categories and explanatory text.
[0037] In this embodiment, model training and optimization are specifically performed using a manually labeled dataset of artistic images. Each image is equipped with a sentiment category label and explanatory text, and the training objective is to jointly optimize two sub-tasks: The sentiment classification subtask involves predicting the subjective sentiment category of an image by fusing multimodal features. A cross-entropy loss function is used to optimize classification accuracy.
[0038] Text generation subtask: Generate sentiment explanation text in natural language form based on sentiment category and contextual features. An autoregressive text generation mechanism is employed, with cross-entropy loss from language modeling as the optimization objective.
[0039] By jointly training the two tasks mentioned above, we can achieve unified modeling and performance improvement in artistic image emotion recognition and text interpretation generation.
[0040] The following is a more specific example: The model performance was evaluated on two benchmark datasets, Artemis v1.0 and v2.0. This dataset contains 80,031 artworks from 1,119 artists, covering 27 art styles. Each artwork includes meta-information such as artist, style, and title, and sentiment labels and textual expressions are provided by multiple annotators. Artemis v1.0 and v2.0 contain 454,684 and 455,000 sentiment expressions, respectively. v2.0 provides more granular text and a more balanced sentiment distribution compared to v1.0. There are eight sentiment labels: amusement, awe, contentment, excitement, fear, sadness, anger, and disgust.
[0041] The following metrics were used to evaluate the model: Accuracy (ACC): Measures the model's performance in emotion recognition tasks; Emotional Alignment (EA): Evaluates whether the emotion expressed by the generated interpretation is consistent with the dominant emotion of the image; BLEU, METEOR, ROUGE-L (B1-B4, M, R): These measures the similarity between the generated text and the manually annotated reference description at the linguistic level. Higher values for these metrics indicate better model performance.
[0042] Following the experimental setup of existing research, the Artemis dataset was randomly divided into training, validation, and test sets, with proportions of 85%, 5%, and 10%, respectively. The main model was implemented on an NVIDIA GeForce RTX 3080 and developed using the PyTorch framework. For image processing, this embodiment used a frozen ViT-B / 16 model to extract image features. The input image was resized to 224×224 pixels and divided into 196 image patches, each represented as a 768-dimensional feature vector. For text processing, both the context encoder and title encoder used the BERT-Base default tokenizer, with a maximum input length of 80 tokens and a feature dimension of 768. The maximum length of the generated explanatory text was 30 tokens.
[0043] This implementation plan mainly includes two types of experiments: the first is an ablation experiment with and without background information to evaluate the impact of background knowledge on the model's sentiment understanding ability; the second is a comparative experiment with existing mainstream methods to verify the comprehensive performance advantages of the proposed method in terms of sentiment classification accuracy and interpretability under the same data partitioning and evaluation indicators. Through these two types of experiments, the effectiveness of this method in the task of sentiment understanding of artistic images is comprehensively analyzed.
[0044] Ablation experiments with and without background information: No-context-information model: Using only image features as input, this model serves as a baseline model to evaluate performance in the absence of external semantic support.
[0045] The background information model: Based on the above, semantic information obtained by the background retrieval module is introduced, including the artist's life background, creative era, cultural context, etc., and integrated into the main model through a weighted fusion mechanism.
[0046] Training and evaluation were conducted under both settings, comparing the differences in sentiment classification accuracy, interpretation quality, and other metrics. Experimental results show that introducing background information significantly improves the model's ability to understand complex emotions, especially when processing images requiring cultural or historical context, validating the important role of background knowledge in understanding the emotions of artistic images. The results are shown in Tables 1 and 2.
[0047] Table 1. Comparison of sentiment classification accuracy and sentiment consistency between no background information and with background information.
[0048] Table 2. Comparison of sentiment interpretability metrics between no background information and with background information.
[0049] Comparative experiments with existing mainstream methods: In comparative experiments with existing mainstream methods, this embodiment selected several representative artistic image sentiment understanding models as controls, including unimodal methods based on visual features, multimodal methods incorporating joint image and text modeling, and recently advanced models that incorporate contextual information. The model in this embodiment demonstrates leading advantages across multiple evaluation dimensions, especially in images involving complex semantic emotions, exhibiting higher accuracy and more interpretable generation results compared to purely visual models, as shown in Tables 3 and 4.
[0050] Table 3. Evaluation metrics of this embodiment and other algorithms on the Artemisv1.0 dataset.
[0051] Table 4. Evaluation metrics of this embodiment and other algorithms on the Artemisv2.0 dataset.
[0052] As shown in the table above, after adopting the method of this embodiment, all evaluation indicators are higher than those of the existing mainstream methods.
Claims
1. A method for modeling sentiment understanding of artistic images based on background knowledge retrieval enhancement, characterized in that, Includes the following steps: Step 1: Collect multiple original art images and preprocess each original art image; Step 2: Use a visual encoder to extract visual features from each preprocessed original image, generate corresponding visual feature vectors, and associate the metadata of each preprocessed image with its visual feature vectors to form an image feature library for similarity retrieval. Step 3: Extract the visual feature vector of the target art image to be processed. Select the original image with the highest similarity to the visual feature vector from the image feature library obtained in Step 2. Use the metadata corresponding to the original image as the background metadata of the target art image to be processed. Obtain the background context text related to the background metadata. Step 4: Construct a multimodal art emotion prediction and interpretation network, which includes an image encoder, a text encoder, a feature fusion unit, and an emotion decoding unit. The image encoder is used to extract image features from the target artistic image; The text encoder is used to extract the background text features of the background context text obtained in step 3; The feature fusion unit is used to fuse image features with background text features to generate a context feature representation; The emotion decoding unit outputs the emotion classification result and the corresponding emotion interpretation based on the contextual feature representation.
2. The art image sentiment understanding modeling method based on background knowledge retrieval enhancement as described in claim 1, characterized in that, The preprocessing in step 1 specifically involves normalizing the size of all original artistic images and adjusting them to a fixed resolution of 224×224.
3. The art image sentiment understanding modeling method based on background knowledge retrieval enhancement as described in claim 1, characterized in that, Step 2 specifically involves using the frozen CLIP-ViT / 32 visual encoder to extract visual features from each preprocessed original image, generating corresponding visual feature vectors; then, an index is built based on FAISS, and the metadata of each preprocessed image is associated with its visual feature vector to form an image feature library for similarity retrieval. The metadata includes the artist, title, style, and year of creation corresponding to the original image.
4. The art image sentiment understanding modeling method based on background knowledge retrieval enhancement as described in claim 3, characterized in that, Step 3 includes the following sub-steps: Step 31: Use the frozen CLIP-ViT / 32 visual encoder to extract the visual feature vector of the target art image to be processed, as the target visual feature vector; Step 32: Calculate the feature similarity between the target visual feature vector and any visual feature vector in the image feature library using the following formula: in, Represents the target visual feature vector; This represents any visual feature vector in the image feature library; It represents the similarity between the target visual feature vector and visual features in the image feature library; This indicates the total number of original art images; Step 33: Traverse each visual feature vector in the image feature library, select the original image with the highest similarity to the target visual feature vector, and use the metadata corresponding to the original image as the background metadata of the target art image; Step 34: Input the background metadata of the target art image as query text into the DrQA-based retrieval system to obtain multiple candidate articles. Calculate the cosine similarity score between the target visual feature vector and the visual feature vector of each candidate article using the following formula: in, The feature vector representing the candidate article; Represents the target visual feature vector; This represents the similarity between the target visual feature vector and the candidate article feature vector; M represents the number of candidate articles; Step 35: Select the top five candidate articles with the highest similarity as context candidate articles, which serve as background context text related to obtaining background metadata.
5. The art image sentiment understanding modeling method based on background knowledge retrieval enhancement as described in claim 4, characterized in that, In step 4, the image encoder specifically uses the CLIP-ViT / 16 pre-trained model, and the text encoder specifically uses the BERT model.
6. The art image sentiment understanding modeling method based on background knowledge retrieval enhancement as described in claim 5, characterized in that, The feature fusion unit described in step 4 specifically uses an adaptive weighted fusion method. The concatenated image features and background text features are passed through an adaptive pooling layer and then fed into a fully connected network (MLP). The output is a two-dimensional vector value to determine the weights of the visual and background elements, as shown in the following formula: Wherein, APool represents adaptive pooling; Image features that represent artistic images; Background text features representing artistic images; Indicates the weight value; This indicates vector concatenation; Represents a 2-dimensional real vector; The feature fusion unit fuses image features with background text features to generate a contextual feature representation, as shown below: in, Represents the visual characteristics of the target artistic image; Features representing background information of the target artistic image; This indicates the weight values of the output; This indicates the final fusion characteristic.
7. The method for modeling artistic image sentiment understanding based on background knowledge retrieval enhancement as described in claim 6, characterized in that, The sentiment decoding unit in step 4 outputs the sentiment classification result and the corresponding sentiment interpretation based on the contextual feature representation, specifically including the following sub-steps: Step a: Receive contextual feature representation, represent the sentiment sentence to be generated as a structured input text composed of sentiment category and sentiment interpretation, and obtain the corresponding structured input text embedding representation by mapping each token in the text to the corresponding embedding vector; Step b involves using a multi-layer Transformer structure to perform self-attention modeling and cross-attention modeling (fusion of visual and contextual features) on the structured input text embedding representation obtained in step a, thereby obtaining the sentiment output representation. Step c: Use a language prediction head consisting of a linear layer and a softmax layer to generate sentiment classification results and corresponding sentiment interpretations based on the sentiment output representation.