An interpretable image aesthetic sentiment prediction method based on a language large model

Abstract

The application discloses a kind of based on language big model's explainable image aesthetic sentiment prediction method: step 1, obtain aesthetic dataset, data cleaning and pretreatment are carried out to aesthetic dataset, obtain the image ID of each picture, text description information and aesthetic quality score information;Step 2, construct image aesthetic explainability model, the model includes visual feature extraction module, prompt text feature extraction module, text-visual fusion module, aesthetic sentiment feature extraction module, aesthetic adaptive module and aesthetic description generation decoding module;Step 3 is trained to image aesthetic explainability model using training set, and the trained image aesthetic explainability model is obtained.Step 4, the image to be measured is input into the trained image aesthetic explainability model obtained in step 3, and output result is obtained.The application effectively improves the ability of aesthetic explainability.

Description

Technical Field

[0001] This invention belongs to the field of interpretable research technology in image aesthetics, and specifically relates to an interpretable image aesthetic sentiment prediction method based on a large language model. Background Technology

[0002] Aesthetics is a subjective feeling obtained by an individual's emotions acting on an aesthetic object. The content of aesthetic research is the individual's emotional response to an object or phenomenon. Aesthetic value is based on the sensory level of discrimination, while aesthetic judgment transcends the senses. Aesthetic judgments are often related to feelings of liking and disliking, and emotions are constrained by certain cultural and aesthetic backgrounds, making aesthetic factors overly complex. Generally speaking, aesthetics is related to emotional responses. The research process of using computational techniques to predict human emotional responses to visual stimuli, enabling computers to mimic the human aesthetic process, and thus quantitatively describe aesthetics, is called computational aesthetics. As early as the 1930s, mathematician George David Birkhoff created a method of aesthetic measurement using the ratio of order to complexity as an aesthetic metric. Subsequent work introduced new related attributes, such as connections to information processing and information theory, improving the scientific nature of aesthetics by verifying and identifying repetition, symmetry, etc. With the rise of technologies such as deep learning, more and more work focuses on automatic aesthetic evaluation. Deep learning utilizes large amounts of manually labeled data for training, enabling models to acquire the ability of humans to evaluate aesthetics.

[0003] Among numerous aesthetic fields, the aesthetics of images is arguably the most widely applied and extensively studied. Thanks to the widespread use of images in art for preservation and dissemination, the field of image aesthetics has developed rapidly. Professional photographers use certain rules to make their photographs more appealing, and these rules are often used as important factors in measuring aesthetic value. Examples include the rule of thirds, visual weight balance, focus, and ISO speed rating. Some studies indicate that works that conform to human visual stimulation provide higher aesthetic quality. The aesthetic characteristics of an image are also a significant influencing factor. These characteristics can be categorized into low-level and high-level features, based on salience, category, and object. Low-level features include color, brightness, and edges, intuitively describing an image with low spatiotemporal complexity. High-level features include regions and content.

[0004] There are many different methods in the process of aesthetic evaluation, including but not limited to evaluating the overall aesthetic quality of an image as high or low, or using scoring to express the beauty of an image. Developments in the field of natural language processing have provided a way to use text descriptions to explain the aesthetic content related to an image. Current relevant research in this field, such as the MiniGPT4 model proposed by Deyao Zhu*, Jun Chen*, and others, can explain the objective content of an image to some extent and describe its abstract subjective features to some extent. However, its ability to generate corresponding aesthetic evaluations and interpret these evaluations is relatively poor.

[0005] In summary, considering how to enhance the aesthetic accuracy of a model's evaluation of an image, and how to improve the interpretability of generated descriptions, are key aspects of research on the interpretability of image aesthetics in this field. Summary of the Invention

[0006] The purpose of this invention is to provide an interpretable image aesthetic sentiment prediction method based on a large language model, in order to solve the problem of poor aesthetic interpretability of existing models.

[0007] To achieve the above objectives, the present invention employs the following technical solution:

[0008] An interpretable image aesthetic sentiment prediction method based on a large language model, comprising the following steps:

[0009] Step 1: Obtain the aesthetic dataset. Perform data cleaning and preprocessing on the aesthetic dataset to obtain the image ID, text description information, and aesthetic quality score information for each image.

[0010] Step 2: Construct an image aesthetic interpretability model, which includes a visual feature extraction module, a prompt text feature extraction module, a text-visual fusion module, an aesthetic emotion feature extraction module, an aesthetic adaptation module, and an aesthetic description generation and decoding module.

[0011] The visual feature extraction module is used to extract visual features from the images in the aesthetic dataset processed in step 1.

[0012] The aforementioned prompt text feature extraction module is used to extract relevant text features of the Prompt.

[0013] The text-visual fusion module is used to fuse the relevant text features extracted by the prompt text feature extraction module and the visual features of the image extracted by the visual feature extraction module to obtain the image features of the Prompt prompt text.

[0014] The aforementioned aesthetic and emotional feature extraction module serves as an aesthetic and emotional feature extraction space, processing the visual features of the image extracted by the visual feature extraction module to obtain features related to aesthetics and emotions.

[0015] The aforementioned aesthetic adaptation module is used to perform forward and backward propagation of the image features of the Prompt text obtained by the text-visual fusion module. This enhances the accuracy of image feature extraction, improves the accuracy of the aesthetic quality assessment generated by the decoder, and simultaneously improves the aesthetic quality of the generated text.

[0016] The aesthetic description generation and decoding module is used to perform feature decoding on the output of the aesthetic adaptation module, generate an image-related aesthetic quality assessment, and a textual explanation of the assessment.

[0017] Step 3: Use each image, its text description, and corresponding aesthetic score label from the aesthetic dataset processed in Step 1 as the training set to train the image aesthetic interpretable model constructed in Step 2, thus obtaining the trained image aesthetic interpretable model.

[0018] Step 4: Input the image to be tested into the trained image aesthetic interpretable model obtained in Step 3 to obtain the output result.

[0019] The present invention also has the following technical features:

[0020] Specifically, step 1 includes the following sub-steps:

[0021] Step 1.1: Process the scores in the original dataset. Based on the scores, divide the images in the original dataset into high-quality images and low-quality images.

[0022] Step 1.2: Using the mind chain approach, process the text description information of each image in the original dataset of Step 1, which is needed for the second stage of fine-tuning in Step 3, into the following pattern: caption = (IS, OD, SD, OD);

[0023] In the formula:

[0024] IS: Indicates an evaluation of the aesthetic quality of the image;

[0025] OD: Represents an objective description of an image in the text;

[0026] SD: Indicates a subjective description of the image in the text;

[0027] OD: Indicates additional descriptions about the image.

[0028] Specifically, step 2 includes the following sub-steps:

[0029] Step 2.1, the visual feature extraction module uses the images from the original dataset in Step 1 as input, and uses VIT-G / 14 as the image feature extractor:

[0030]

[0031] I' image =LN(I image )+E position Formula 2;

[0032] f' image =Transformer(I' image Formula 3;

[0033] f image =MLP(f' image Formula 4;

[0034] In the formula:

[0035] I image The image is represented by a digitized image, divided into multiple patches;

[0036] This means that a digitized image is input into VIT-G / 14 and becomes 14*14, which is one of the 196 patches;

[0037] LN represents a linear layer;

[0038] E position Indicates positional encoding;

[0039] f' image This represents the image features after passing through the Transformer Encoder block;

[0040] MLP stands for Multilayer Perceptron;

[0041] f image This represents the features of an image after it has been processed by an image feature extractor.

[0042] Step 2.2: To improve the model's ability to learn aesthetic features, this invention adds an aesthetic sentiment feature extraction module. This module is first pre-trained on a sentiment classification dataset, and then extracts image features f. image This module significantly improves the model's accuracy in predicting aesthetic emotions. The calculation formula is as follows:

[0043] f Aesthetic =Down(LN(δ(Up(f)) image Formula 5;

[0044] fEmotion =Down(LN(δ(Up(f)) image Formula 6;

[0045] f output =W Aes f Aesthetic +W Emo f Emotion Formula 7;

[0046] In the formula:

[0047] Up represents an upsampling layer;

[0048] Down indicates a downsampling layer;

[0049] δ represents an activation function;

[0050] LN represents a linear layer;

[0051] f Aesthetic This represents the extracted aesthetically relevant visual features;

[0052] f Emotion This represents the extracted emotion-related visual features;

[0053] W Aes This represents a learnable parameter used as f. Aesthetic The participation weight;

[0054] W Emo This represents a learnable parameter used as f. Emotion The participation weight.

[0055] f output This represents the output of the current module.

[0056] Step 2.3, extract the aesthetic and emotional features f of the image obtained in Step 2.2. output and Prompt text features f prompt The concatenation is represented as eigenvalue f words and the eigenvalue f words Perform cross-attention calculation. Use the Prompt text features extracted by the prompt text feature extraction module. prompt and the text feature f prompt and image features f image The corresponding and fusion processes are then performed. The specific calculation method is as follows:

[0057] f prompt =BertTokenizer(prompt) Formula 8;

[0058] f words =fprompt +f output Equation 9;

[0059] f ImgAndWords =CrossAttention(f image ,f words Formula 10;

[0060] In the formula:

[0061] f words The text feature f that represents the prompt text prompt f obtained from the aesthetic emotion feature extraction module output The concatenated feature values ​​output;

[0062] CrossAttention refers to the cross-attention mechanism;

[0063] f ImgAndWords This represents the features corresponding to the image and text obtained through the cross-attention mechanism;

[0064] Step 2.4: To better memorize aesthetic and emotional information during training, and to align the input from the previous stage with the feature dimensions in the decoder, a new module, the aesthetic adaptation module, is added. Its specific calculation steps are as follows:

[0065] f' input =Up(f ImgAndWords Formula 11;

[0066] f input =LN(Gelu(LN(f') input )))+f' input Equation 12;

[0067] f output =Down(Gelu(LN(f) input Formula 13;

[0068] In the formula:

[0069] Up indicates the upsampling layer;

[0070] LN represents a linear layer;

[0071] Gelu represents the activation function;

[0072] f' input This represents the input features after upsampling;

[0073] Down indicates the downsampling layer;

[0074] f iutput This represents the result of upsampling and intermediate layer residual linking;

[0075] f output This indicates the output result.

[0076] Step 2.5, in the subsequent aesthetic description, the decoding module generates the output feature result f from step 2.4. output Decode it into the final result.

[0077] Here's a brief introduction to the computation layer RMSnorm:

[0078]

[0079] In the formula:

[0080] x represents the input feature;

[0081] Mean represents the function for calculating the average value;

[0082] eps represents a small value, typically 1e-6;

[0083] W represents the weight parameters that can be learned;

[0084] Next, we will introduce the calculation formula in LLaMA:

[0085] f' rmsnorm =RMSNorm(E(x)) Equation 15;

[0086] In the formula: E represents the Embedded layer.

[0087] The results after RMSnorm are processed using a self-attention mechanism. The specific calculation formula is as follows:

[0088] f rope =RoPE(Q)*CausalMask(RoPE(K)) Formula 16;

[0089] f' block =Softmax(f rope )*V Formula 17;

[0090] f' b_output =f' block +E(x) Equation 18;

[0091] f final_output =f' b_output +FFN(RMSNorm(f' b_output Formula 19;

[0092] In the formula:

[0093] Softmax represents the normalization function;

[0094] Q = W Q ·f' input , representing the Query in self-attention;

[0095] K = W K ·f' input , representing the Key in self-attention;

[0096] V = W V ·f' input , representing the Value in self-attention;

[0097] f' rmsnorm This represents the input processed through the RMSNorm layer;

[0098] Rope represents a rotation encoding operation;

[0099] CausalMask represents a random mask operation;

[0100] W Q W K and W V These are three randomly initialized weight matrices;

[0101] FFN stands for Forward Propagation Layer, which consists of two MLPs.

[0102] Step 3, as a preferred embodiment of the present invention, trains the image aesthetic interpretability model constructed in Step 2 using a training set. Then, the trained model is fine-tuned with a reduced learning rate to better align with the interpretability of image aesthetics. The output text of the fine-tuned image aesthetic interpretability model is used as the final output, enabling the evaluation of the aesthetic quality of an image and the interpretation of that evaluation. Furthermore, during training, the present invention uses cross-entropy loss as the loss function, treating each word in the caption used as the true label as a prediction result. The specific calculation formula is as follows:

[0103]

[0104] In the formula:

[0105] C represents the number of categories, which in this case is the number of words in the text dictionary;

[0106] p i Indicates the predicted value;

[0107] q i Represents the actual value.

[0108] Compared with the prior art, the present invention has the following technical effects:

[0109] (I) This invention enables the model to generate detailed textual descriptions of the aesthetic quality evaluation of an image, solving the problem that the aesthetic quality evaluation of an image generated by existing models is unrelated to the textual description. At the same time, the generated subjective descriptions are relevant to the image content, and the extended and expanded subjective aspects are consistent with the overall theme of the image, effectively improving the ability of aesthetic interpretability.

[0110] (II) The present invention can achieve the accuracy of the entire model in emotion prediction and aesthetic prediction by separately training the aesthetic emotion feature extraction module when the large model is frozen and the modules that require a lot of training. At the same time, the overall computational cost and computing power are relatively small compared with the existing technology.

[0111] (III) This invention can generate a detailed aesthetic description sentence to explain the aesthetic quality evaluation of an image, while this description sentence conforms to the image content. Furthermore, from a textual perspective, it avoids the problems of consistent sentence structure and repetitive words in previous descriptions, ensuring the quality of the generated sentence. Attached Figure Description

[0112] Figure 1 It is the overall architecture of the aesthetic interpretability model of images.

[0113] Figure 2 This is the aesthetic and emotional feature extraction module AE-Plugins involved in this invention.

[0114] Figure 3 This is the Aesthetic-Adapter, an aesthetic adaptive module involved in this invention.

[0115] Figure 4 These are the subjective experimental results of this invention, and the subjective description of the aesthetic interpretability of the images. Specific implementation methods

[0116] The method of this invention solves the problem in previous methods that the aesthetic evaluation of images and the generated descriptions are not closely related. It also proposes an aesthetic emotion feature extraction module and an aesthetic adaptation module to enhance the accuracy of the model in aesthetic emotion classification. In addition, it solves the problem of the generated descriptive sentences having the same sentence structure and monotonous content when generating explanatory descriptive sentences.

[0117] The meanings of the relevant technical terms in this invention are as follows:

[0118] Training dataset: refers to the DPC-Caption dataset and 80% of the AVA dataset randomly selected;

[0119] Test dataset: consisting of the remaining AVA dataset and the IAE dataset;

[0120] Pre-training: Training performed before the model is actually trained, where the parameters and weights are changed once.

[0121] Fine-tuning: Instead of training all parameters, select some parameters and reduce the learning rate for training;

[0122] VIT: Vision Transformer is a model of Transformer in the field of computer vision;

[0123] BERT model: refers to a pre-trained language representation model, whose full English name is Bidirectional Encoder Representation from Transformers. In this invention, BertTokenizer is mainly used to extract text features from the Prompt text.

[0124] BLIP-2 model: refers to a multimodal framework that effectively combines features from both visual and textual modalities. Its full name is Bootstrapping Language-Image Pretraining-2.

[0125] Prompt: This indicates the content of the prompt text, which is usually a question used to guide the direction of the large model in answering the question;

[0126] Softmax: Normalization function.

[0127] CrossAttention: Cross-attention mechanism is a type of attention mechanism used for weighted calculations across multiple modalities;

[0128] Q, K, V: Three weight matrices in the attention mechanism;

[0129] RMSNorm: A normalization method

[0130] GELU: Activation function used to introduce nonlinear changes and improve the expressive power of the model;

[0131] MLP: Multilayer Perceptron, which consists of linear layers and activation functions;

[0132] FFN: Feed-forward layer, consisting of two MLP layers;

[0133] LLaMA: A pure language decoding model used to describe the generated decoding module;

[0134] CrossEntropyLoss: A loss function used to calculate the difference between the true value and the predicted value.

[0135] Minigpt4: The model compared to this invention

[0136] AE-Miminigpt4: An Image Aesthetic Emotional Interpretability Model of the Invention

[0137] Following the technical solution of the present invention described above, specific embodiments are given below.

[0138] Example 1:

[0139] This embodiment presents an interpretable image aesthetic sentiment prediction method based on a large language model, such as... Figures 1-3 As shown, it includes the following steps:

[0140] Step 1: Obtain the aesthetic dataset. Perform data cleaning and preprocessing on the aesthetic dataset to obtain the image ID, text description information, and aesthetic quality score information for each image. Step 1 specifically includes the following sub-steps:

[0141] Step 1.1: Calculate the aesthetic score for each image in the AVA dataset. Images with an aesthetic score of 5 or higher are considered high-quality images, while those with a score lower are considered low-quality images. The high-quality and low-quality binary classification results are used as labels.

[0142] Step 1.2: Summarize the high and low quality classification results from Step 1.1 into a single text describing the high and low quality of the image, making it consistent with the score evaluation description, and place it at the beginning of the sentence. Then, process the original description information of the AVA aesthetic dataset from Step 1, removing abbreviations, numbers, and other words, and standardize its format.

[0143] Step 2 involves constructing an aesthetic interpretability model for the images using the AVA dataset processed in Step 1. This model includes a visual feature extraction module, a cue text feature extraction module, a text-visual fusion module, an aesthetic sentiment feature extraction module, an aesthetic adaptation module, and an aesthetic description generation and decoding module. Step 2 specifically includes the following sub-steps:

[0144] Step 2.1: Normalize the input image to a size of 224×224. Use the VIT-G / 14 feature extractor with a feature dimension of 196×768. Each input image is divided into 16×16 patches and passes through a 14-head attention mechanism. After that, each dimension has a size of 768. This output serves as the visual feature content of the image.

[0145] Step 2.2: The visual features of the image obtained in Step 2.1 are input into the Aesthetic-Emotion Feature Extraction module AE-Plugins. This module consists of an Aesthetic-Extractor and an Emotion-Extractor. Each feature extraction unit first undergoes upsampling, an activation function, a linear fusion layer, and a downsampling layer. Through pre-training, different parameter weights are assigned to these two parts, allowing them to favor aesthetics and emotion respectively throughout the computation process. Finally, an Aesthetic Feature of size 196×768 and an Emotion Feature of the same size are obtained. The weights W are then added together. Aes ×Aesthetic Feature and W Emo The value of ×Emotion Feature is used to obtain the final feature value of aesthetic emotional feature, represented as f. output The feature dimension size is [196,768].

[0146] Step 2.3: Use BertTokenizer to extract the features of the Prompt text and add them to the aesthetic sentiment feature value, denoted as f. words , will f words and visual features f image Perform CrossAttention to highlight image features relevant to the text content. Denote the entire output feature as f. ImgAndWords .

[0147] Step 2.4, take the output f from step 2.3. ImgAndWords The input is fed into the Aesthetic-Adapter module. Through linear layers, activation functions, and residual links from Equation 11 in step 2.4, the dimension is adapted to the input dimension of the LLaMA decoder, which is 4396. At the same time, the learning of relevant features is placed on the parameter weights of this part.

[0148] Step 2.5: Input the features obtained in Step 2.4 into the aesthetic description generation and decoding module. Process the input features through the Embedded layer, then normalize them using RMSnorm. Finally, compare the output features with W... Q W K and W V Multiply to obtain Q, K, and V, then multiply them using the rotation encoding RoPE and the mask, and finally perform a softmax operation to obtain the output feature f'. block Then, the feature f' is summed using the residual addition method. block The input features are added to the embedded layer and then passed through an RMSnorm layer and an FFN layer consisting of two MLPs to obtain the final output f.output .

[0149] Step 3: Use the image, text description, and high / low aesthetic quality of each image in the aesthetic dataset as the training set to train the image aesthetic emotion interpretability model. Then, test the generated descriptions on other datasets using the trained model.

[0150] This embodiment of the performance test for the image aesthetic emotion interpretability model mainly includes two parts: subjective experiments and objective experiments. In the objective experiment, the accuracy of the aesthetic emotion quality evaluation is used to judge the model's ability to grasp aesthetic emotion content. The specific formula is as follows:

[0151]

[0152] In the formula:

[0153] TP indicates that the actual case was positive and the prediction was positive.

[0154] TN indicates that the actual case was negative and the predicted case was negative.

[0155] FP indicates that the actual example is positive and the prediction is negative;

[0156] FN indicates that the actual example is negative and the prediction is positive.

[0157] In the subjective experiment section, the generated sentences are judged manually, and the judgment results are given.

[0158] The objective experimental results are as follows:

[0159] Table 1. Minigpt4, Minigpt4 (fine-tuned), AE-Minigpt4 Aesthetic Emotion Classification Results (%)

[0160]

[0161] Table 2 shows whether AE-Plugins were pre-trained in the sentiment classification of the IAE dataset.

[0162]

[0163] This embodiment uses Minigpt4 as the comparison model, and a Minigpt4 model fine-tuned on the AVA dataset in the second stage. This example calculates the accuracy of the models on both datasets (AVA, IAE). The results on the AVA dataset show that the AE-Minigpt4 model of this invention and the Minigpt4 model fine-tuned on the AVA dataset improve the results in the binary classification of aesthetic high and low quality.

[0164] This embodiment did not use the IAE dataset for fine-tuning AE-Minigpt4. The IAE dataset lacks corresponding text descriptions (captions), making it more suitable as a zero-shot dataset. It can be seen that the accuracy is lower than the AVA dataset. This is because, although the images in the IAE and AVA datasets come from the same website, the aesthetic quality of each image in the IAE dataset was re-evaluated, thus changing the patterns of aesthetic quality evaluation and affecting the model's accuracy after switching datasets. In this embodiment, the AE-Plugins module was pre-trained separately on the IAE dataset, and then the pre-trained AE-Plugins was reintroduced into the AE-Minigpt4 model, resulting in a significant improvement in the results. This demonstrates the effectiveness of the aesthetic sentiment feature extraction module.

[0165] In this embodiment, the subjective experimental results are as follows: Figure 4 As shown, Minigpt4 initially generates a relatively short aesthetic description sentence that captures the objective content of the image, but the aesthetic description is not detailed enough and lacks interpretability. The AE-Minigpt4 of this invention generates more detailed sentences, significantly improving interpretability. Furthermore, the study employed subjective evaluation of sentence fluency and aesthetic accuracy through manual testing. In the generated aesthetic description sentences, the output sentences of AE-Minigpt4 were consistently considered superior to those of Minigpt4. Regarding aesthetic accuracy, individual aesthetic differences led to variations in the aesthetic evaluations of some images between the two models. Overall, AE-Minigpt4 outperforms Minigpt4 in terms of aesthetic interpretability.

Claims

1. An interpretable image aesthetic sentiment prediction method based on a large language model, characterized in that, Includes the following steps: Step 1: Obtain the aesthetic dataset, perform data cleaning and preprocessing on the aesthetic dataset, and obtain the image ID, text description information and aesthetic quality score information for each image; Step 2: Construct an image aesthetic interpretability model, which includes a visual feature extraction module, a cue text feature extraction module, a text-visual fusion module, an aesthetic emotion feature extraction module, an aesthetic adaptation module, and an aesthetic description generation and decoding module. The visual feature extraction module is used to extract visual features from the images in the aesthetic dataset processed in step 1; The aforementioned prompt text feature extraction module is used to extract relevant text features of the Prompt; The text-visual fusion module is used to fuse the relevant text features extracted by the prompt text feature extraction module and the visual features of the image extracted by the visual feature extraction module to obtain the image features of the Prompt prompt text. The aforementioned aesthetic and emotional feature extraction module serves as an aesthetic and emotional feature extraction space, processing the visual features of the image extracted by the visual feature extraction module to obtain features related to aesthetics and emotions. The aesthetic adaptive module is used to perform forward and backward propagation of the image features of the Prompt prompt text obtained by the text-visual fusion module; The aesthetic description generation and decoding module is used to perform feature decoding on the output of the aesthetic adaptation module, generate an image-related aesthetic quality evaluation, and a textual explanation of the evaluation. Step 2 includes the following sub-steps: Step 2.1, the visual feature extraction module uses the images from the original dataset in Step 1 as input, and uses VIT-G / 14 as the image feature extractor: Formula 1; Formula 2; Formula 3; Equation 4; In the formula: The image is represented by a digitized image, divided into multiple patches; This means inputting a digitized image into VIT-G / 14 and converting it to 14. 14 means it is one of the 196 patches; Indicates a linear layer; Indicates positional encoding; This represents the image features after passing through the Transformer Encoder block; Represents a multilayer perceptron; This represents the features of an image after it has been processed by an image feature extractor. Step 2.2: The aesthetic emotion feature extraction module first undergoes pre-training on an emotion classification dataset, and then extracts image features. This module significantly improves the model's accuracy in predicting aesthetic emotions; the calculation formula is as follows: Formula 5; Formula 6; Formula 7; In the formula: Indicates an upsampling layer; This represents a downsampling layer; Represent an activation function; Indicates a linear layer; This represents the extracted aesthetically relevant visual features; This represents the extracted emotion-related visual features; This represents a learnable parameter used as... Participation weight; This represents a learnable parameter used as... Participation weight; This represents the output of the current module; Step 2.3: Analyze the aesthetic and emotional characteristics of the image obtained in Step 2.

2. and Prompt prompt text features The concatenation is represented as eigenvalues. and the eigenvalues Perform cross-attention calculation; use the Prompt text features extracted by the Prompt text feature extraction module. and the text features and image features The specific calculation method for corresponding and merging is as follows: Formula 8; Equation 9; Formula 10; In the formula: Text features that indicate the prompt text The aesthetic and emotional feature extraction module obtained The concatenated feature values ​​output; This represents the cross-attention mechanism; This represents the features corresponding to the image and text obtained through the cross-attention mechanism; Step 2.4, the specific calculation steps of the aesthetic adaptive module are as follows: Formula 11; Equation 12; Equation 13; In the formula: Indicates the upsampling layer; Indicates a linear layer; Indicates the activation function; This represents the input features after upsampling; Indicates the downsampling layer; This represents the result of upsampling and intermediate layer residual linking; Indicates the output result; Step 2.5, the aesthetic description generation decoding module generates the output feature results of step 2.

4. Decode the final result; the specific steps are as follows: Equation 14; In the formula: Represents input features; This represents the function for calculating the average value. This represents a small numerical value, typically 1e-6; This represents the weight parameters that can be learned; Formula 15; In the formula: Indicates the Embedded layer; The results after RMSnorm are processed using a self-attention mechanism. The specific calculation formula is as follows: Formula 16; Equation 17; Formula 18; Formula 19; In the formula: Softmax represents the normalization function; , representing the Query in self-attention; , representing the Key in self-attention; , representing the Value in self-attention; This represents the input processed through the RMSNorm layer; Indicates a rotation encoding operation; Indicates causal masking operation; , and These are three randomly initialized weight matrices; This indicates the forward propagation layer, which consists of two MLPs; Step 3: Use each image, the text description of each image, and the corresponding aesthetic score label in the aesthetic dataset processed in Step 1 as the training set to train the image aesthetic interpretability model constructed in Step 2, and obtain the trained image aesthetic interpretability model. Step 4: Input the image to be tested into the trained image aesthetic interpretability model obtained in Step 3 to obtain the output result.

2. The interpretable image aesthetic sentiment prediction method based on a large language model as described in claim 1, characterized in that, Step 1 includes the following sub-steps: Step 1.1: Process the scores in the original dataset. Based on the scores, divide the images in the original dataset into high-quality images and low-quality images. Step 1.2: Using the mind chain approach, process the text description information of each image in the original dataset of Step 1, which is needed for the second stage of fine-tuning in Step 3, into the following pattern: caption=(IS, OD, SD, OT); In the formula: IS: Indicates an evaluation of the aesthetic quality of the image; OD: Represents an objective description of an image in the text; SD: Indicates a subjective description of the image in the text; OT: Indicates additional descriptions about the image.

3. The interpretable image aesthetic sentiment prediction method based on a large language model as described in claim 1 or 2, characterized in that, Step 3 is as follows: The image aesthetic interpretability model constructed in step 2 is trained using the training set, and then the trained model is fine-tuned by reducing the learning rate. The output text of the fine-tuned image aesthetic interpretability model is used as the final output.

4. The interpretable image aesthetic sentiment prediction method based on a large language model as described in claim 1 or 2, characterized in that, In step 3 of the training process, cross-entropy loss is used as the loss function, and each word in the caption used as the true label is used as each prediction result. The specific calculation formula is as follows: Formula 20; In the formula: C represents the number of categories, which in this case is the number of words in the text dictionary; Indicates the predicted value; Represents the actual value.

Images