An image aesthetic evaluation method based on text extraction and multi-granularity alignment of CLIP

Abstract

The present application relates to a kind of image aesthetic evaluation methods based on CLIP of text extraction and multi-granularity alignment, belong to image processing and computer vision field.The method, the user comment in the existing large language model is extracted from the text in the aesthetic dataset, and the core description related to aesthetics is analyzed and summarized, and then aesthetic abstract dataset is constructed;Second, the CLIP framework of multi-granularity alignment is designed, based on the framework, the original CLIP model is fine-tuned using the paired image-detailed comment-aesthetic abstract, so that the model better understands the semantic relationship between image and text description in the field of aesthetics, so as to learn more rich aesthetic feature representation, obtain the optimized CLIP model for aesthetic task;Finally, based on the fine-tuned CLIP model, image aesthetic quality evaluation network is designed to realize the effective prediction of image aesthetic quality.The present application avoids the problem that cross-modal aesthetic evaluation method needs to input image text pair simultaneously in reasoning stage.

Description

Technical Field

[0001] This invention belongs to the fields of image processing and computer vision, specifically relating to an image aesthetic evaluation method based on text extraction and multi-granularity alignment CLIP. Background Technology

[0002] In the fields of modern image processing and computer vision, image aesthetic evaluation techniques are receiving increasing attention. Image aesthetic evaluation is typically used to assess the visual appeal and artistic value of an image, and is widely applied in areas such as photo album management, image retrieval, and digital photography. In practical applications, image aesthetic evaluation needs to comprehensively consider various aesthetic elements such as composition, color, texture, and lighting to provide objective and accurate aesthetic scores, thereby assisting users in selecting high-aesthetic-quality images. Due to the highly subjective and diverse nature of image aesthetics, effectively capturing and quantifying these aesthetic characteristics has become a core research challenge.

[0003] Early traditional methods of image aesthetic evaluation relied primarily on hand-designed features such as color, texture, contrast, and sharpness. While these features can reflect the basic aesthetic attributes of an image to some extent, their expressive power is limited, making it difficult to effectively capture the complex relationships among the various factors influencing aesthetic quality evaluation.

[0004] With the development of deep learning technology, image aesthetic evaluation methods based on convolutional neural networks have been widely researched and applied. These methods, through an end-to-end learning process, can automatically extract high-level features of images and have achieved significant performance improvements on multiple aesthetic evaluation datasets. However, these methods often ignore the rich aesthetic semantic information contained in user comment text, leading to certain limitations. To address this issue, many deep learning-based cross-modal aesthetic evaluation methods have been proposed. These methods process both image and text modalities simultaneously, fusing the visual features of images with the semantic features of text, thereby capturing the aesthetic attributes of images more comprehensively and further improving the performance of image aesthetic evaluation tasks. However, these methods require simultaneous input of image and text pairs during the inference phase, which presents a limitation in practical applications.

[0005] In recent years, visual-language pre-trained models such as CLIP have made breakthroughs in the field of multimodal semantic understanding, demonstrating strong capabilities in understanding semantic information related to image and text, thus bringing new research opportunities to image aesthetic evaluation tasks. CLIP, trained on large-scale image-text pairing data, can not only extract semantic features from images but also possesses excellent language processing and content understanding capabilities, providing potential cross-modal support for aesthetic evaluation tasks. However, because the training data for CLIP models mainly comes from general domains, it lacks targeted modeling of fine aesthetic features in image aesthetics, making it difficult to directly apply to complex aesthetic evaluation tasks. Therefore, how to improve the ability of CLIP models to understand and capture aesthetic features and effectively apply them to image aesthetic evaluation tasks is a key issue. Summary of the Invention

[0006] The purpose of this invention is to better apply the CLIP model to aesthetic evaluation tasks and further address the limitations of existing image aesthetic evaluation methods. It provides an image aesthetic evaluation method based on text extraction and multi-granularity alignment using CLIP. This method utilizes existing large language models to extract text from user comments in an aesthetic dataset, analyzing and summarizing core aesthetic-related descriptions to construct an aesthetic summary dataset. Secondly, a multi-granularity alignment CLIP framework is designed. Based on this framework, the original CLIP model is fine-tuned using paired image-detailed comment-aesthetic summary models, enabling the model to better understand the semantic relationships between images and text descriptions in the aesthetic domain, thereby learning richer aesthetic feature representations and obtaining an optimized CLIP model for aesthetic tasks. Finally, an image aesthetic quality evaluation network is designed based on the fine-tuned CLIP model to achieve effective prediction of image aesthetic quality.

[0007] To achieve the above objectives, the technical solution of this invention is: an image aesthetic evaluation method based on text extraction and multi-granularity alignment CLIP, comprising:

[0008] Step S1: Preprocess the images and text in the aesthetic image-comment dataset to obtain a paired image-comment dataset for training;

[0009] Step S2: Construct an aesthetic summary dataset using a large language model; design the CLIP framework with multi-granularity alignment;

[0010] Step S3: Using the aesthetic summary dataset described in step S2 and the paired image-comment dataset described in step S1, fine-tune the original CLIP model using the multi-granularity alignment CLIP framework described in step S2 to obtain an optimized CLIP model suitable for aesthetic tasks.

[0011] Step S4: Design an image aesthetic quality evaluation network, including an image branch and a text branch. The image encoder in the image branch and the text encoder in the text branch are derived from the CLIP model fine-tuned in step S3.

[0012] Step S5: Use the images in the paired image-comment dataset described in step S1 to train the image aesthetic quality evaluation network described in step S4, and obtain the trained image aesthetic quality evaluation network.

[0013] Step S6: Input the image to be tested into the trained image aesthetic quality evaluation network, output the corresponding aesthetic quality evaluation score distribution, and calculate the aesthetic evaluation score accordingly.

[0014] In one embodiment of the present invention, step S1 is specifically implemented as follows:

[0015] Step S11: Scale each image in the aesthetic image-comment training dataset to a uniform size of H×W×3, and normalize the RGB three channels of each scaled image, where H and W are the height and width of the image, respectively.

[0016] Step S12: Convert all words in the comment text of the aesthetic image-comment training dataset to lowercase and remove irrelevant symbols; then, perform word segmentation on the text using the byte pair encoding algorithm BPE, splitting the text into word or sub-word tags. Each word or sub-word is mapped to a unique numerical ID in the vocabulary, that is, each ID represents the index of a word or sub-word in the vocabulary; in addition, add start tag "[SOS]" and end tag "[EOS]" at the beginning and end of the text sequence, respectively; the final size of each text sequence is 1×len, where len represents the maximum length of the input sequence supported by the model, including the start and end tags. Parts exceeding len are deleted, and otherwise, they are padded with 0.

[0017] In one embodiment of the present invention, step S2 is specifically implemented as follows:

[0018] Step S21: Use the large language model LLaMA-7B to extract text from user comments in the aesthetic image-comment training dataset, analyze and summarize the descriptions related to aesthetics, and construct an aesthetic summary dataset. The aesthetic summaries in the aesthetic summary dataset are paired one-to-one with the images and comments in the aesthetic image-comment training dataset.

[0019] Step S22: Preprocess the text in the aesthetic summary dataset obtained in step S21 using the same method as in step S12. The size of each processed text sequence is 1×len.

[0020] Step S23: Design a multi-granularity alignment CLIP framework. This framework can achieve fine-grained alignment by comparing and learning the fine-grained features of an image with its corresponding detailed comment text features, and simultaneously achieve coarse-grained alignment by comparing and learning the coarse-grained features of an image with its corresponding aesthetic summary features. This allows the model to capture the detailed aesthetic attributes of an image while also identifying key aesthetic features. In addition, this framework requires multiple pairs of samples as input, and thus compares positive and negative sample pairs.

[0021] In one embodiment of the present invention, step S21 is specifically implemented as follows:

[0022] Step S211: Construct the prompt text as input to the large language model LLaMA-7B. The prompt text consists of three parts: The first part is a task instruction description, informing the model that the current specific task is to analyze and summarize the various aesthetic-related descriptions involved in the given user comment text; the second part provides a pair of examples separated by the ">" symbol, with the original comment text before the symbol and the aesthetic summary text expected after executing the task instruction after the symbol. Based on the specific examples, the large language model can learn and imitate the required reconstruction behavior; the last part includes the comment text that needs to be summarized and the separating symbol ">".

[0023] Step S212: Input the prompt text described in step S211 into the large language model LLaMA-7B. The large language model LLaMA-7B will automatically complete the text content after the separator “=>” in the prompt text, thus obtaining the aesthetic summary from the current comment text.

[0024] Step S213: Process all user comments in the aesthetic image-comment training dataset as described in step S212 to obtain an aesthetic summary dataset that is paired one-to-one with each user comment.

[0025] In one embodiment of the present invention, step S23 is specifically implemented as follows:

[0026] Step S231: Extract fine-grained image features using the CLIP model image encoder for a given batch of B images. Fine-grained image feature matrix is ​​obtained through an image encoder. Represented as F Ifine =Image_Encoder(I B ), where Image_Encoder represents the image encoder, which is based on the visual Transformer architecture; specifically, for each input image First, it undergoes convolution with a kernel size of P×P and a stride of P, then is flattened into a series of patch sequences, resulting in... Where C represents the number of channels, P 2 This indicates the size of each image patch. This indicates the number of generated patches. Each patch is then transformed into a d-dimensional embedding vector through a linear mapping. Simultaneously, a learnable [CLS] token is added to the beginning of the embedding sequence to capture global semantic information, and positional embeddings are incorporated to preserve the spatial location information of the patch, thus generating an image embedding representation. Subsequently, i' is processed by the L-layer Transformer to obtain... Finally, I' Lcls After layer normalization and projection onto the multimodal embedding space via linear mapping, the final image feature representation is obtained. For B input images, a fine-grained image feature matrix representation is obtained. Let represent the fine-grained image features of the i-th image; the specific formula is as follows:

[0027] I p =Flatten(Conv P×P,P (I)),

[0028]

[0029] I′1=Transformer(I′),

[0030] I' l =Transformer(I′) l-1 ), I = 2, ..., L,

[0031]

[0032] Among them, Conv P×P,P (·) denotes a convolution with kernel size P×P and stride P, and Flatten(·) denotes a flattening operation; E, E pos and E I Let E represent a learnable matrix, used to map each patch to a d-dimensional embedding vector. pos Used to learn the spatial location information of patches, E I Used to project image features into a multimodal embedding space; Transformer(·) represents a Transformer layer, l represents the specific number of Transformer layers passed through; LN(·) represents the layer normalization operation;

[0033] Step S232: Obtain coarse-grained image features, and use principal component analysis to refine the fine-grained image features obtained in step S231. Principal component feature extraction is performed to obtain the dimensionality-reduced feature representation. At this point, each image has lost some detail information; subsequently, an inverse transform is used to map it back to the original spatial dimension, resulting in a coarse-grained image feature representation. Specifically, firstly, for feature F Ifine The feature matrix is ​​centered by subtracting the mean from each feature. Subsequently, through F Icenter Singular value decomposition (SVD) is performed to obtain the principal component directions of the data. The column vectors of the right singular matrix after SVD are the eigenvectors, i.e., the principal component directions. The first dim principal components are selected from these directions to retain the most representative information in the data. Next, the data is projected onto the selected principal component directions to achieve feature dimensionality reduction, resulting in the dimensionality-reduced feature representation F. IPCA Finally, the dimensionality-reduced data is inversely projected using the selected principal component direction, mapped back to the original dimensions, and the mean of the original features is added back to obtain a coarse-grained image feature representation. Let represent the coarse-grained image features of the i-th image; the specific formula is as follows:

[0034]

[0035] F ICenter =F Ifine -u,

[0036]

[0037] V dim =V[:,:dim],

[0038] F IPCA =F Icenter V dim ,

[0039]

[0040] in, The expression represents the singular value decomposition operation, where U represents the left singular vector matrix, ∑ represents the singular value diagonal matrix, and V represents the right singular vector matrix; dim represents the number of principal components selected, i.e., the first dim ones.

[0041] Step S233: Extract fine-grained text features using the CLIP model's text encoder. For a given batch of comment text containing segment B in detail... A fine-grained text feature matrix is ​​obtained through a text encoder. Represented as F Tfine =Text_Encoder(T B ), where Text_Encoder represents the text encoder, which is based on the Transformer architecture; specifically, for a text sequence after word segmentation in step S12 First, a pre-initialized word embedding layer maps each ID to a d-dimensional word embedding vector, thus obtaining the text embedding representation. Subsequently, T' is processed by the L-layer Transformer to obtain... Finally, T' Llen[EOS] After layer normalization and projection onto the multimodal embedding space via linear mapping, the final text feature representation is obtained. For the input of comment text B, a fine-grained text feature matrix representation is obtained. This represents the fine-grained text features of the i-th text segment; the specific formula is as follows:

[0042] T′=Embedding(T),

[0043] T′1 = Transformer(T′),

[0044] T′ l =Transformer(T′) l-1 ),l=2,...,L,

[0045]

[0046] Where Embedding(·) represents the embedding layer, Transformer(·) represents the Transformer layer, l represents the specific number of Transformer layers passed through; LN(·) represents the layer normalization operation; E T This represents a learnable matrix used to project text features into a multimodal embedding space;

[0047] Step S234: Extract coarse-grained text features using a text encoder consistent with that described in step S233; for a given batch containing B-segment aesthetic summaries The coarse-grained text feature matrix is ​​obtained through a text encoder. Represented as F Tcoarse =Text_Encoder(A B );

[0048] Step S235: Design a fine-grained contrast loss function lfine For a given batch containing B pairs (image, detailed comment, aesthetic summary), F is obtained according to the method described in steps S231 to S234. Ifine F Icoarse F Tfine and F Tcoarse , represented as For fine-grained image and text features There are B × B possible pairings, of which B are true pairings; e fine The aim is to maximize the true pairing of B pairs in a batch. and Cosine similarity between them, while minimizing B 2 -B mismatches are used to achieve fine-grained alignment of image text features; the specific formula is as follows:

[0049]

[0050] Among them, l fi2t (·) represents the fine-grained image-to-text contrastive learning loss, l ft2i (·) represents the fine-grained text-to-image contrastive learning loss; Sim(·) represents the cosine similarity calculation, and τ represents the temperature coefficient used to scale the similarity score;

[0051] Step S236: Design coarse-grained contrast loss function l coarse For a given batch containing B pairs (image, detailed comment, aesthetic summary), F is obtained according to the method described in steps S231 to S234. Ifine F Icoarse F Tfine and F Tcoarse , represented as For coarse-grained image and text features There are B × B possible pairings, of which B are true pairings; coarse The aim is to maximize the true pairing of B pairs in a batch. and Cosine similarity between them, while minimizing B 2 -B mismatches are used to achieve coarse-grained alignment of image text features; the specific formula is as follows:

[0052]

[0053] Among them, l ci2t (·) represents the coarse-grained image-to-text contrastive learning loss, l ct2i (·) represents the coarse-grained text-to-image contrastive learning loss; Sim(·) represents the cosine similarity calculation, and τ represents the temperature coefficient used to scale the similarity score;

[0054] Step S237: Design a multi-granularity contrast loss function l, which is used as the overall objective function of the model. The multi-granularity contrast loss function l is a fine-granularity contrast loss l. fine Compared with coarse-grained loss l coarse The sum; the specific formula is as follows:

[0055] l = l fine +l coarse .

[0056] In one embodiment of the present invention, step S3 is specifically implemented as follows:

[0057] Step S31: Randomly divide the paired (image, detailed comment, aesthetic summary) dataset used for training described in Step S1 and Step S2 to obtain several batches containing B pairs (image, detailed comment, aesthetic summary);

[0058] Step S32: Using the pre-trained CLIP model as the initial model, input the images and text from the dataset described in Step S31 into the image encoder and text encoder respectively in batches, and obtain the results according to the method described in Step S23. And calculate the loss l according to the formula;

[0059] Step S33: Calculate the gradient of the parameters in the model using the backpropagation method based on the loss function, and update the model parameters using the Adam optimization method;

[0060] Step S34: Repeat steps S31 to S33 until the preset number of training rounds is reached to obtain an optimized CLIP model suitable for aesthetic tasks.

[0061] In one embodiment of the present invention, step S4 is specifically implemented as follows:

[0062] Step S41: Design the text branch, including a parameter-frozen text encoder and five learnable cue words as input. Specifically, first, introduce five learnable cue words representing different aesthetic quality categories. These cue words correspond to the five quality categories of "terrible", "bad", "average", "good", and "perfect". Each cue word consists of several learnable embedding vectors and words representing the corresponding quality category; specifically represented as "[X]1[X]2[X]3...[X]". M "[Quality Category]", where each [X] i(i∈1,...M) is a learnable text token with the same dimension as the word embedding; M represents the number of learnable text tokens. These prompts are then input into the text encoder to generate five corresponding feature vectors, represented as follows:

[0063] Step S42: Design the image branch, which includes a parameter-frozen image encoder and an adapter module. The adapter module includes four sub-adaptors. Each adapter consists of a fully connected layer that maps input features to a lower dimension, a GELU activation function, and a fully connected layer that maps the low-dimensional features back to the original dimension. The inputs of each adapter are I' from the four different stages of the image encoder. lcls Specifically, the image encoder consists of L Transformer layers, which are first divided into four stages, each containing L... s layer, The output of the first stage is the Lth stage. s The [CLS] token in the layer output represents... Using this as input to the first adapter, we obtain feature F. A1 The output of the second stage is the 2nd L. S The [CLS] token in the layer output represents... Using this as input to the second adapter, we obtain feature F. A2 The output of the third stage is the 3L. s The [CLS] token in the layer output represents... Using this as input to the third adapter, we obtain feature F. A3 The output of the fourth stage is the [CLS] token in the output of the Lth layer, denoted as I'. lcls Using this as input to the fourth adapter, we obtain feature F. A4 Next, the output features of the four adapters are added and fused, and the final image features are obtained through linear mapping. The specific formula is expressed as follows:

[0064]

[0065] F A4 =FC Up(GELU(FC Down(I′) Lcls ))),

[0066]

[0067] Where FC DOwn(·) represents a fully connected layer used to map input features to a lower dimension, GELU·) represents the GELU activation function, and FC Up(·) represents a fully connected layer used to map the low-dimensional features back to the original dimension; E F This represents a learnable matrix used to incorporate features F. I Projected onto text features F P Same dimensions;

[0068] Step S43: Transfer the text features F obtained in step S41 to... P As the key and value, the image features F obtained in step S42 are used as the key and value respectively. I As a query, cross-attention is used to calculate features. To capture the correlation between image content and different aesthetic quality categories; subsequently F C After passing through a feedforward network and image features F I The final output features are obtained by fusion. This data is then further refined using another feedforward network to ultimately predict the aesthetic rating distribution of the image; the specific formula is as follows:

[0069] F C =CrossAttention(F I ,F p ),

[0070] F O =FFN(F C )+F I ,

[0071]

[0072] Where CrossAttention(·) represents cross-attention calculation, and FFN(·) represents feedforward network; This represents the distribution of predicted aesthetic scores;

[0073] Step S44: Design the loss function for the image aesthetic quality assessment network. To more accurately measure the difference between the predicted distribution and the true distribution, and to capture the order and distance relationships between scores, EMD loss is used as the target loss function for the image aesthetic quality assessment network; the specific formula is as follows:

[0074]

[0075] Where T represents the number of aesthetic score categories; Let p represent the cumulative distribution function of the true distribution P. i This represents the probability of the i-th score in the true distribution; Represents the predicted distribution The cumulative distribution function, This represents the probability of the i-th score in the prediction distribution.

[0076] In one embodiment of the present invention, step S5 is specifically implemented as follows:

[0077] Step S51: Randomly divide the images in the paired image-comment dataset described in step S1 to obtain several batches containing B images;

[0078] Step S52: Input the image into the image aesthetic quality assessment network in step S4 to obtain the predicted aesthetic score distribution, and calculate the loss l using the formula in step S42. EMD ;

[0079] Step S53: Calculate the gradients of the unfrozen parameters in the network using the backpropagation method based on the loss function, and update these parameters using the Adam optimization method.

[0080] Step S54: Repeat steps S51 to S53 in batches to obtain the trained image aesthetic quality evaluation network.

[0081] In one embodiment of the present invention, step S6 is specifically implemented as follows:

[0082] Step S61: Input the image to be tested into the trained image aesthetic quality evaluation network described in step S5 to obtain the aesthetic score distribution predicted by the model. Let represent the probability of the i-th score in the predicted distribution. Based on the aesthetic rating distribution predicted by the model, the aesthetic evaluation score (Score) is obtained by weighted summing of the scores of all categories and their corresponding predicted probabilities. Let the score of each rating category be . Where s i If is the score for the i-th rating category, then the specific formula is as follows:

[0083]

[0084] The present invention also provides a computer-readable storage medium having stored thereon computer program instructions that can be executed by a processor, wherein when the processor executes the computer program instructions, it can implement the steps of the method described above.

[0085] Compared to existing technologies, this invention offers the following advantages: First, it constructs an aesthetic summary dataset and utilizes existing large language models to analyze and summarize user comments within the dataset, extracting core descriptions related to aesthetics and providing more data support for subsequent aesthetic evaluation tasks. Second, it designs a multi-granularity alignment CLIP framework, achieving fine-grained alignment by comparing and learning the fine-grained features of images with their corresponding detailed comment text features, and simultaneously achieving coarse-grained alignment by comparing and learning the coarse-grained features of images with their corresponding aesthetic summary features. This allows the model to capture the detailed aesthetic attributes of images while also identifying key aesthetic features. Finally, based on the CLIP model fine-tuned using the multi-granularity alignment framework, this invention designs an image aesthetic quality evaluation network. This network freezes the text encoder and image encoder for feature extraction. Furthermore, considering the subjectivity and diversity of image aesthetic evaluation, five learnable cue words representing different aesthetic quality categories are introduced as auxiliary features in the text branch to improve the accuracy of the aesthetic evaluation task. An adapter module is introduced in the image branch for fine-tuning to improve the model's generalization ability. Unlike other deep learning-based image aesthetic evaluation methods, this invention fully leverages the cross-modal advantages of the visual-language model and employs a multi-granularity alignment fine-tuning strategy to enable the CLIP model to better capture and understand the semantic connections between images and diverse textual descriptions, learning rich aesthetic feature representations. Furthermore, learnable cue words are introduced as auxiliary features to further improve the performance of the aesthetic evaluation task, avoiding the problem of existing cross-modal aesthetic evaluation methods requiring simultaneous input of image and text pairs during the inference stage, thus providing a more efficient and flexible solution for aesthetic evaluation. Attached Figure Description

[0086] Figure 1 This is a flowchart illustrating the implementation of the method of the present invention.

[0087] Figure 2 This is a framework diagram of text extraction in an embodiment of the present invention.

[0088] Figure 3 This is a structural diagram of the CLIP framework with multi-granularity alignment in an embodiment of the present invention.

[0089] Figure 4 This is a structural diagram of the image aesthetic quality evaluation network in an embodiment of the present invention. Detailed Implementation

[0090] The technical solution of the present invention will now be described in detail with reference to the accompanying drawings.

[0091] It should be noted that the following detailed descriptions are exemplary and intended to provide further explanation of this application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains.

[0092] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments according to this application. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0093] This invention provides an image aesthetic evaluation method based on text extraction and multi-granularity alignment (CLIP), comprising:

[0094] Step S1: Preprocess the images and text in the aesthetic image-comment dataset to obtain a paired image-comment dataset for training;

[0095] Step S2: Construct an aesthetic summary dataset using a large language model; design the CLIP framework with multi-granularity alignment;

[0096] Step S3: Using the aesthetic summary dataset described in step S2 and the paired image-comment dataset described in step S1, fine-tune the original CLIP model using the multi-granularity alignment CLIP framework described in step S2 to obtain an optimized CLIP model suitable for aesthetic tasks.

[0097] Step S4: Design an image aesthetic quality evaluation network, including an image branch and a text branch. The image encoder in the image branch and the text encoder in the text branch are derived from the CLIP model fine-tuned in step S3.

[0098] Step S5: Use the images in the paired image-comment dataset described in step S1 to train the image aesthetic quality evaluation network described in step S4, and obtain the trained image aesthetic quality evaluation network.

[0099] Step S6: Input the image to be tested into the trained image aesthetic quality evaluation network, output the corresponding aesthetic quality evaluation score distribution, and calculate the aesthetic evaluation score accordingly.

[0100] The following is a detailed implementation process of the present invention.

[0101] This invention provides an image aesthetic evaluation method based on text extraction and multi-granularity alignment (CLIP), such as... Figure 1 As shown, it includes the following steps:

[0102] Step S1: Preprocess the images and text in the aesthetic image-comment dataset to obtain a paired image-comment dataset for training;

[0103] Step S2: Construct an aesthetic summary dataset using a large language model; design the CLIP framework with multi-granularity alignment;

[0104] Step S3: Using the aesthetic summary dataset described in step S2 and the image-comment dataset described in step S1, fine-tune the original CLIP model using the multi-granularity alignment framework described in step S2 to obtain an optimized CLIP model suitable for aesthetic tasks.

[0105] Step S4: Design an image aesthetic quality evaluation network, which includes an image branch and a text branch. The image encoder in the image branch and the text encoder in the text branch are derived from the CLIP model fine-tuned in step S3.

[0106] Step S5: Use the images in the training dataset described in step S1 to train the image aesthetic quality evaluation network described in step S4, thereby obtaining the trained image aesthetic quality evaluation network.

[0107] Step S6: Input the image to be tested into the trained image aesthetic quality evaluation network, output the corresponding aesthetic quality evaluation score distribution, and calculate the aesthetic evaluation score accordingly.

[0108] Further, step S1 includes the following steps:

[0109] Step S11: Scale each image in the aesthetic image-comment training dataset to a uniform size of H×W×3, and perform normalization operations on the three RGB channels of each scaled image.

[0110] Step S12: Convert all words in the comment text of the aesthetic image-comment training dataset to lowercase and remove irrelevant symbols. Then, perform word segmentation using the Byte-Pair Encoding (BPE) algorithm, splitting the text into word or sub-word tokens. Each word or sub-word is mapped to a unique numerical ID in the vocabulary, meaning each ID represents the index of a word or sub-word in the vocabulary. Furthermore, add a start marker "[SOS]" and an end marker "[EOS]" to the beginning and end of the text sequence, respectively. The final size of each text sequence is 1×len, where len represents the maximum length of the input sequence supported by the model (including start and end markers). Parts exceeding len are deleted; otherwise, they are padded with zeros.

[0111] Further, step S2 includes the following steps:

[0112] Step S21: Use the large language model LLaMA-7B to extract text from user comments in the aesthetic image-comment training dataset, analyze and summarize the descriptions related to aesthetics, and construct an aesthetic summary dataset. The aesthetic summaries in this dataset are paired one-to-one with the images and comments in the aesthetic image-comment training dataset. Figure 2 This is a framework diagram of text extraction in an embodiment of the present invention;

[0113] Step S22: Preprocess the text in the aesthetic summary dataset obtained in step S21 using the same method as described in step S12. The size of each processed text sequence is 1×len.

[0114] Step S23: Design a multi-granularity alignment CLIP framework. This framework can achieve fine-grained alignment by comparing and learning the fine-grained features of an image with its corresponding detailed comment text features, and simultaneously achieve coarse-grained alignment by comparing and learning the coarse-grained features of an image with its corresponding aesthetic summary features. This allows the model to capture the detailed aesthetic attributes of an image while also identifying key aesthetic features. Furthermore, this framework requires multiple pairs of samples as input, thus allowing for comparison through positive and negative sample pairs. Figure 3 This is a structural diagram of the CLIP framework with multi-granularity alignment in an embodiment of the present invention;

[0115] Further, step S21 includes the following steps:

[0116] Step S211: Construct prompt text as input to the large language model. The text content mainly includes three parts: The first part is a task instruction description, informing the model that the current specific task is to analyze and summarize the various aesthetic-related descriptions involved in the given user comment text; the second part provides a pair of examples, separated by the ">" symbol. The part before the symbol represents the original comment text, and the part after the symbol represents the aesthetic summary text expected after executing the task instruction. Based on the specific examples, the large language model can learn and imitate the required reconstruction behavior; the last part includes the comment text that needs to be summarized and the separating symbol ">".

[0117] Step S212: Input the prompt text described in step S211 into the large language model LLaMA-7B. The model will automatically complete the text content after the separator “=>” in the prompt text, thus obtaining the aesthetic summary summarized from the current comment text.

[0118] Step S213: Process all user comments in the aesthetic image-comment training dataset as described in step S212 to obtain an aesthetic summary dataset that is paired one-to-one with each user comment.

[0119] Further, step S23 includes the following steps:

[0120] Step S231: Extract fine-grained image features using the CLIP model image encoder. For a given batch containing B images... Fine-grained image feature matrix is ​​obtained through an image encoder. Represented as F Ifine =Image_Encoder(I B ), where Image_Encoder represents the image encoder. This image encoder is based on the visual Transformer architecture. Specifically, for each input image First, it undergoes convolution with a kernel size of P×P and a stride of P, then is flattened into a series of patch sequences. Where C represents the number of channels, P 2 This indicates the size of each image patch. This indicates the number of generated patches. Each patch is then transformed into a d-dimensional embedding vector through a linear mapping. Simultaneously, a learnable [CLS] token is added to the beginning of the embedding sequence to capture global semantic information, and positional embeddings are incorporated to preserve the spatial location information of the patch, thus generating an image embedding representation. Subsequently, I' is processed by the L-layer Transformer to obtain... Finally, I' Lcls After layer normalization and projection onto the multimodal embedding space via linear mapping, the final image feature representation is obtained. For B input images, a fine-grained image feature matrix representation is obtained. This represents the fine-grained image features of the i-th image. The specific formula is as follows:

[0121] I p =Flatten(Conv P×P,P (I)),

[0122]

[0123] I′1=Transformer(I′),

[0124] I' l =Transformer(I′) l-1 ),l=2,...,L,

[0125]

[0126] Among them, Conv P×P,P(·) denotes a convolution with kernel size P×P and stride P, and Flatten(·) denotes the flattening operation; E, E pos and E I Let E represent a learnable matrix, used to map each patch to a d-dimensional embedding vector. pos Used to learn the spatial location information of the patch; EI is used to project image features into the multimodal embedding space; Transformer(·) represents a Transformer layer, l represents the specific number of Transformer layers passed through; LN(·) represents the layer normalization operation;

[0127] Step S232: Obtain coarse-grained image features. Principal component analysis is used to refine the fine-grained image features obtained in step S231. Principal component feature extraction is performed to obtain the dimensionality-reduced feature representation. At this point, each image has lost some detail. Subsequently, an inverse transform is used to map it back to the original spatial dimensions, yielding a coarse-grained image feature representation. Specifically, firstly, for feature F Ifine The feature matrix is ​​centered by subtracting the mean from each feature. Subsequently, through F Icenter Singular value decomposition (SVD) is performed to obtain the principal component directions of the data (the column vectors of the right singular matrix after SVD are the eigenvectors, i.e., the principal component directions). The first dim principal components are selected to retain the most representative information in the data. Next, the data is projected onto the selected principal component directions to achieve feature dimensionality reduction, resulting in the dimensionality-reduced feature representation F. IPCA Finally, the dimensionality-reduced data is inversely projected using the selected principal component directions, mapped back to the original dimensions, and the mean of the original features is added back to obtain a coarse-grained image feature representation. This represents the coarse-grained image features of the i-th image. The specific formula is as follows:

[0128]

[0129] F ICenter =F Ifine -u,

[0130]

[0131] V dim =V[:,:dim],

[0132] F IPCA =F Icenter V dim ,

[0133]

[0134] in, The expression represents the singular value decomposition operation, where U represents the left singular vector matrix, ∑ represents the singular value diagonal matrix, and V represents the right singular vector matrix; dim represents the number of principal components selected, i.e., the first dim ones.

[0135] Step S233: Extract fine-grained text features using the CLIP model's text encoder. For a given batch of comment text containing segment B of detailed text... A fine-grained text feature matrix is ​​obtained through a text encoder. Represented as F Tfine =Text_Encoder(T B ), where Text_Encoder represents the text encoder. This text encoder is based on the Transformer architecture. Specifically, for a text sequence after word segmentation in step S12... First, a pre-initialized word embedding layer maps each ID to a d-dimensional word embedding vector, thus obtaining the text embedding representation. Subsequently, T' is processed by the L-layer Transformer to obtain... Finally, T' Llen[EOS] After layer normalization and projection onto the multimodal embedding space via linear mapping, the final text feature representation is obtained. For the comment text input in segment B, a fine-grained text feature matrix representation is obtained. This represents the fine-grained text features of the i-th text segment. The specific formula is as follows:

[0136] T′=Embedding(T),

[0137] T′1 = Transformer(T′),

[0138] T′ l =Transformer(T′) l-1 ),l=2,...,L,

[0139]

[0140] Where Embedding(·) represents the embedding layer, Transformer(·) represents the Transformer layer, l represents the specific number of Transformer layers passed through; LN(·) represents the layer normalization operation; E TThis represents a learnable matrix used to project text features into a multimodal embedding space;

[0141] Step S234: Extract coarse-grained text features using a text encoder consistent with that described in step S233. For a given batch containing a B-segment aesthetic summary... The coarse-grained text feature matrix is ​​obtained through a text encoder. Represented as F Tcoarse =Text_Encoder(A B );

[0142] Step S235: Design a fine-grained contrast loss function l fine For a given batch containing B pairs (image, detailed comment, aesthetic summary), F is obtained according to the method described in steps S231 to S234. Ifine F Icoarse F Tfine and F Tcoarse , can be represented as For fine-grained image and text features There are B × B possible pairings, of which B are true pairings. fine The aim is to maximize the true pairing of B pairs in a batch. and Cosine similarity between them, while minimizing B 2 -B mismatches are used to achieve fine-grained alignment of image text features. The specific formula is as follows:

[0143]

[0144] Among them, l fi2t (·) represents the fine-grained image-to-text contrastive learning loss, l ft2i (·) represents the fine-grained text-to-image contrastive learning loss; Sim(·) represents the cosine similarity calculation, and τ represents the temperature coefficient used to scale the similarity score;

[0145] Step S236: Design coarse-grained contrast loss function l coarse For a given batch containing B pairs (image, detailed comment, aesthetic summary), F is obtained by following the methods described in steps S231 to S234. Ifine F Icoarse F Tfine and F Tcoarse , can be represented as For coarse-grained image and text features There are B × B possible pairings, of which B are true pairings. coarse The aim is to maximize the true pairing of B pairs in a batch. and Cosine similarity between them, while minimizing B 2 -B incorrect pairings are used to achieve coarse-grained alignment of image text features. The specific formula is as follows:

[0146]

[0147]

[0148] Among them, l ci2t (·) represents the coarse-grained image-to-text contrastive learning loss, l ct2i (·) represents the coarse-grained text-to-image contrastive learning loss; Sim(·) represents the cosine similarity calculation, and τ represents the temperature coefficient used to scale the similarity score.

[0149] Step S237: Design a multi-granularity contrastive loss function l, and use it as the overall objective function of the model. This loss is a fine-grained contrastive loss l. fine Compared with coarse-grained loss l coarse The sum. The specific formula is as follows:

[0150] l = l fine +l coarse .

[0151] Further, step S3 includes the following steps:

[0152] Step S31: Randomly divide the paired (image, detailed comment, aesthetic summary) dataset used for training described in Step S1 and Step S2 to obtain several batches containing B pairs (image, detailed comment, aesthetic summary);

[0153] Step S32: Using the pre-trained CLIP model as the initial model, input the images and text from the dataset described in Step S31 into the image encoder and text encoder respectively in batches, and obtain the results according to the method described in Step S23. And calculate the loss l according to the formula;

[0154] Step S33: Calculate the gradient of the parameters in the model using the backpropagation method based on the loss function, and update the model parameters using the Adam optimization method.

[0155] Step S34: Repeat steps S31 to S33 until the preset number of training rounds is reached to obtain an optimized CLIP model suitable for aesthetic tasks.

[0156] Further, step S4 includes the following steps:

[0157] Step S41: Design the text branch. This branch includes a parameter-frozen text encoder and five learnable cue words as input. The text encoder is derived from the CLIP model fine-tuned in step S3. Specifically, five learnable cue words representing different aesthetic quality categories are introduced, corresponding to the quality categories of "terrible", "bad", "average", "good", and "perfect". Each cue word consists of several learnable embedding vectors and a word representing the corresponding quality category. Specifically, it can be represented as "[X]1[X]2[X]3...[X]". M "[Quality Category]", where each [X] i (i∈1,...M) is a learnable text token with the same dimension as the word embedding; M represents the number of learnable text tokens. These prompts are then input into a text encoder to generate five corresponding feature vectors, represented as follows:

[0158] Step S42: Design the image branch. This branch includes a parameter-frozen image encoder and an adapter module. The image encoder is derived from the CLIP model fine-tuned in step S3. The adapter module includes four sub-adaptors, each consisting of a fully connected layer that maps input features to a lower dimension, a GELU activation function, and a fully connected layer that maps the low-dimensional features back to the original dimension. The inputs to each adapter are I' from the four different stages of the image encoder. lcls Specifically, the image encoder consists of L Transformer layers, which are first divided into four stages, each containing L... s layer, The output of the first stage is the Lth stage. s The [CLS] token in the layer output represents... Using this as input to the first adapter, we obtain feature F. A1 The output of the second stage is the 2nd L. s The [CLS] token in the layer output represents... Using this as input to the second adapter, we obtain feature F. A2 The output of the third stage is the 3L. s The [CLS] token in the layer output represents... Using this as input to the third adapter, we obtain feature F. A3 The output of the fourth stage is the [CLS] token in the output of the Lth layer, denoted as I'. Lcls Using this as input to the fourth adapter, we obtain feature F. A4Next, the output features of the four adapters are added and fused, and the final image features are obtained through linear mapping. The specific formula is expressed as follows:

[0159]

[0160] F A4 =FC Up(GELU(FC Down(I′) Lcls ))),

[0161]

[0162] Where FC DOwn(·) represents a fully connected layer used to map input features to a lower dimension, GELU·) represents the GELU activation function, and FC Up(·) represents a fully connected layer used to map the low-dimensional features back to the original dimension; E F This represents a learnable matrix used to incorporate features F. I Projected onto text features F P Same dimensions;

[0163] Step S43: Transfer the text features F obtained in step S41 to... P As the key and value, the image features F obtained in step S42 are used... I As a query, cross-attention is performed to obtain features. To capture the correlation between image content and different aesthetic quality categories. Then F C After passing through a feedforward network and image features F I The final output features are obtained by fusion. The result is then further refined using another feedforward network to ultimately predict the aesthetic rating distribution of the image. The specific formula is as follows:

[0164] F C =CrossAttention(F I ,F p ),

[0165] F O =FFN(F C )+F I ,

[0166]

[0167] Where CrossAttention(·) represents cross-attention calculation, and FFN(·) represents feedforward network; This represents the distribution of predicted aesthetic scores;

[0168] Step S44: Design the loss function for the image aesthetic quality assessment network. To more accurately measure the difference between the predicted distribution and the true distribution, and to capture the order and distance relationships between scores, EMD loss is used as the target loss function for the image aesthetic quality assessment network. The specific formula is as follows:

[0169]

[0170] Where T represents the number of aesthetic score categories; Let p represent the cumulative distribution function of the true distribution P. i This represents the probability of the i-th score in the true distribution; Represents the predicted distribution The cumulative distribution function, This represents the probability of the i-th score in the prediction distribution.

[0171] Figure 4 This is a structural diagram of the image aesthetic quality evaluation network in an embodiment of the present invention.

[0172] Further, step S5 includes the following steps:

[0173] Step S51: Randomly divide the images in the training dataset described in step S1 to obtain several batches containing B images.

[0174] Step S52: Input the image into the image aesthetic quality assessment network in step S4 to obtain the predicted aesthetic score distribution, and calculate the loss l using the formula in step S42. EMD ;

[0175] Step S53: Calculate the gradients of the unfrozen parameters in the network using the backpropagation method based on the loss function, and update these parameters using the Adam optimization method.

[0176] Step S54: Repeat steps S51 to S53 in batches to obtain the image aesthetic quality evaluation network.

[0177] Further, step S6 includes the following steps:

[0178] Step S61: Input the image to be tested into the trained image aesthetic quality evaluation network described in step S5 to obtain the aesthetic score distribution predicted by the model. Based on this distribution, the aesthetic evaluation score can be obtained by weighted summing the scores of all categories with their corresponding predicted probabilities. Let the score of each category be... Where s i If is the score for the i-th rating category, then the specific formula is as follows:

[0179]

[0180] The present invention also provides a computer-readable storage medium having stored thereon computer program instructions that can be executed by a processor, wherein when the processor executes the computer program instructions, it can implement the steps of the method described above.

[0181] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0182] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0183] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0184] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0185] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any other way. Any person skilled in the art may make changes or modifications to the above-disclosed technical content to create equivalent embodiments. However, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the protection scope of the present invention.

Claims

1. An image aesthetic evaluation method based on CLIP of text distillation and multi-granularity alignment, characterized in that, The method comprises the following steps: Step S1, preprocessing the images and texts in the aesthetic image-comment data set to obtain a paired image-comment data set for training; Step S2, constructing an aesthetic summary data set using a large language model; designing a multi-granularity alignment CLIP framework; the specific implementation steps are as follows: Step S21, using a large language model LLaMA-7B to perform text extraction on the user comments in the aesthetic image-comment training data set, analyzing the descriptions related to aesthetics and summarizing them to construct an aesthetic summary data set, wherein the aesthetic summaries in the aesthetic summary data set are paired with the images and comments in the aesthetic image-comment training data set; Step S22, preprocessing the texts in the aesthetic summary data set obtained in step S21 using the same method as step S12, and the size of each processed text sequence is 1xlen, wherein len represents the maximum length of the input sequence supported by the model; Step S23, designing a multi-granularity alignment CLIP framework, which can perform fine-grained alignment by comparing the fine-grained features of the image with the detailed comment text features corresponding to the image, and perform coarse-grained alignment by comparing the coarse-grained features of the image with the aesthetic summary features corresponding to the image, so that the model can capture the detailed aesthetic properties of the image while recognizing the key aesthetic features; in addition, the framework needs multiple pairs of samples as input, so as to compare through positive and negative sample pairs; Step S3, using the aesthetic summary data set described in step S2 and the paired image-comment data set described in step S1 to fine-tune the original CLIP model using the multi-granularity alignment CLIP framework described in step S2, to obtain an optimized CLIP model suitable for aesthetic tasks; Step S4, designing an image aesthetic quality evaluation network, including an image branch and a text branch, wherein the image encoder in the image branch and the text encoder in the text branch are derived from the fine-tuned CLIP model in step S3; Step S5, using the images in the paired image-comment data set described in step S1 to train the image aesthetic quality evaluation network described in step S4, to obtain a trained image aesthetic quality evaluation network; Step S6, inputting the to-be-tested image into the trained image aesthetic quality evaluation network, outputting a corresponding aesthetic quality evaluation score distribution, and calculating the aesthetic evaluation score based on the same.

2. The image aesthetic evaluation method based on CLIP with text summarization and multi-granularity alignment according to claim 1, characterized in that, The specific implementation steps of step S1 are as follows: Step S11, scaling each image in the aesthetic image-comment training data set to a uniform size of HxWx3 image, and performing standardization operation on the three channels of RGB of each scaled image, wherein H and W are the height and width of the image, respectively; Step S12, convert all the words in the aesthetic image-comment training dataset into lowercase and remove irrelevant symbols; then, perform word segmentation processing on the text by using the byte pair encoding algorithm BPE, split the text into words or subword tokens, and map each word or subword to a unique numerical ID in the vocabulary, i.e., each ID represents the index of a word or subword in the vocabulary; in addition, add a start token "[SOS]" and an end token "[EOS]" at the beginning and end of the text sequence respectively; finally, the size of each text sequence obtained is 1xlen, where len represents the maximum length of the input sequence supported by the model, and the part exceeding len is deleted, otherwise, it is padded with 0.

3. The image aesthetic evaluation method based on CLIP with text summarization and multi-granularity alignment according to claim 1, characterized in that, The step S21 is specifically implemented as follows: Step S211, construct a prompt text as input of the large language model LLaMA-7B, and the prompt text content includes three parts: the first part is a task instruction description, which informs the model that the current specific task is to analyze and summarize various aesthetic-related descriptions involved in the given user comment text; the second part gives an example pair, which is separated by the symbol "=>", and the symbol before represents the original comment text, and the symbol after represents the aesthetic summary text expected after executing the task instruction operation; the last part includes the comment text that needs to be summarized at present and the separator symbol "=>"; Step S212, input the prompt text in step S211 into the large language model LLaMA-7B, and the large language model LLaMA-7B will automatically complete the text content after the separator symbol "=>" in the prompt text, i.e., obtain the aesthetic summary from the current comment text; Step S213, process all user comments in the aesthetic image-comment training dataset by step S212 to obtain an aesthetic summary dataset paired with each user comment.

4. The image aesthetic evaluation method based on CLIP with text summarization and multi-granularity alignment according to claim 1, characterized in that, The step S23 is specifically implemented as follows: Step S231, fine-grained image features are extracted by using an image encoder of the CLIP model, for a given batch containing B images A fine-grained image feature matrix is obtained by the image encoder denoted as F Ifine = Image_Encoder(I B ), where Image_Encoder represents an image encoder based on a visual Transformer architecture; specifically, for each input image First, a convolutional processing with a kernel size of P x P and a step size of P is performed, and then the image is flattened into a series of patch sequences to obtain where C represents the number of channels, P 2 represents the size of each image patch, represents the number of generated patches; then each patch is converted into a d-dimensional embedding vector through a linear mapping, a learnable [CLS] token is added at the front end of the embedding sequence to capture global semantic information, and position embedding is added to preserve the spatial position information of the patch, thereby generating an image embedding representation Subsequently, I' is processed through L Transformer layers to obtain Finally, I' Lcls is normalized by layer and projected into a multi-modal embedding space through a linear mapping to obtain the final image feature representation And for B input images, a fine-grained image feature matrix representation is obtained denotes the fine-grained image features of the i-th image; Step S232, obtaining coarse-grained image features, using principal component analysis method to obtain fine-grained image features in step S231 Perform principal component feature extraction to obtain the reduced dimension feature representation At this time, each image has lost part of the detailed information; then, using inverse transformation to map it back to the original space dimension to obtain coarse-grained image feature representation Specifically, first, the feature F Ifine Centralized processing, that is, each feature is subtracted from its mean to obtain the centralized feature matrix Then, by singular value decomposition of F Icenter Obtain the principal component direction of the data, and the column vectors of the right singular matrix after singular value decomposition are the eigenvectors, that is, the principal component direction, and the first dim principal components are selected to retain the most representative information in the data; then, the data is projected onto the selected principal component direction to realize feature reduction, and the reduced dimension feature representation F IPCA ; Finally, the reduced dimension data is inversely projected using the selected principal component direction, mapped back to the original dimension and the mean of the original feature is added back, and the coarse-grained image feature representation is obtained representing the coarse-grained image features of the i-th image; Step S233, extract fine-grained text features using the text encoder of the CLIP model, for a given batch of B pieces of detailed review text Obtain the fine-grained text feature matrix through the text encoder denoted as F Tfine = Text_Encoder(T B ), where Text_Encoder represents the text encoder based on the Transformer architecture; specifically, for a text sequence after tokenization processing in step S12 First, map each ID to a d-dimensional word embedding vector through a pre-initialized word embedding layer, thereby obtaining a text embedding representation Subsequently, T' is processed through L layers of Transformer layers to obtain Finally, T' Llen[EOS] is normalized through the layer and projected to the multi-modal embedding space through linear mapping to obtain the final text feature representation For the input of B pieces of review text, the fine-grained text feature matrix representation is obtained denotes the fine-grained text features of the i-th piece of text; Step S234, extracting coarse-grained text features with the text encoder consistent with step S233; for a given batch containing B pieces of aesthetic abstracts Obtaining coarse-grained text feature matrix through text encoder is denoted as F Tcoarse = Text_Encoder(A B ); Step S235, design fine-grained contrast loss function l fine For a given batch containing B pairs [image, detailed comment, aesthetic summary], according to the method of steps S231 to S234, F Ifine , F Icoarse , F Tfine , and F Tcoarse , is expressed as For fine-grained image, text features There are Bx B possible pairs, of which B pairs are real pairs; l fine Aim to maximize the cosine similarity between And , while minimizing B 2 B error pairs to achieve alignment of fine-grained image and text features; Step S236, design coarse-grained contrast loss function l coarse For a given batch containing B pairs [image, detailed comment, aesthetic summary], according to the method of steps S231 to S234, F Ifine , Icoarse , Tfine , and F Tcoarse , are expressed as For coarse-grained image, text features There are Bx B possible pairs, of which B pairs are real pairs; l coarse Aim to maximize the cosine similarity between and , while minimizing B 2 B error pairs to achieve alignment of coarse-grained image and text features; Step S237, design a multi-granularity contrast loss function l, which is used as the total objective function of the model. The multi-granularity contrast loss function l is the sum of the fine-grained contrast loss l fine and the coarse-grained contrast loss l coarse .

5. The image aesthetic evaluation method based on CLIP with text summarization and multi-granularity alignment according to claim 4, characterized in that, The step S3 is specifically implemented as follows: Step S31, randomly divide the paired [image, detailed comment, aesthetic summary] dataset for training into several batches containing B [image, detailed comment, aesthetic summary]; Step S32, using the pre-trained CLIP model as the initial model, inputting the images and texts in the data set in step S31 into the image encoder and the text encoder respectively in batches, and obtaining the image feature vector and the text feature vector according to the method in step S23 and calculating the loss l according to the formula Step S33, calculate the gradient of the parameters in the model using the back propagation method according to the loss function, and update the model parameters using the Adam optimization method; Step S34, repeat steps S31 to S33 until the preset number of training rounds is reached, and obtain the optimized CLIP model suitable for aesthetic tasks.

6. The image aesthetic evaluation method based on CLIP with text summarization and multi-granularity alignment according to claim 1, characterized in that, The step S4 is specifically implemented as follows: Step S41, design a text branch, including a parameter frozen text encoder, and five learnable prompt words as input; specifically, first introduce five learnable prompt words representing different aesthetic quality categories, which correspond to the five quality categories of "terrible", "bad", "average", "good" and "perfect" respectively, and each prompt word is composed of several learnable embedding vectors and a word representing the corresponding quality category; specifically represented as "[X]1[X]2[X]3...[X] M [quality category]", wherein each [X] i (i∈1,...M) is a learnable text token, which has the same dimension as the word embedding; M represents the number of learnable text tokens, and then the prompt words are input into the text encoder to generate five feature vectors corresponding thereto, represented as Step S42, design the image branch, including a parameter frozen image encoder and an adapter module, the adapter module includes four sub-adapters, each adapter is composed of a fully connected layer that maps input features to a lower dimension, a GELU activation function and a fully connected layer that maps low-dimensional features back to the original dimension, the input of each adapter is I' lcls ; Specifically, the image encoder is composed of L layers of Transformer layers, which are first divided into four stages, each stage contains L s layers, The output of the first stage is the [CLS] token in the L s layer output, denoted as Take it as the input of the first adapter to get the feature F A1 ; The output of the second stage is the [CLS] token in the 2L s layer output, denoted as Take it as the input of the second adapter to get the feature F A2 ; The output of the third stage is the [CLS] token in the 3L s layer output, denoted as Take it as the input of the third adapter to get the feature F A3 ; The output of the fourth stage is the [CLS] token in the L layer output, denoted as I' Lcls , take it as the input of the fourth adapter to get the feature F A4 , then add and fuse the output features of the four adapters, and get the final image feature I Step S43, the text feature F obtained in step S41 is taken as the key Key and the image feature F obtained in step S42 is taken as the value Value, to perform cross-attention calculation to obtain a feature F P As the query Query, the image feature F obtained in step S42 is taken as the key Key and the text feature F obtained in step S41 is taken as the value Value, to perform cross-attention calculation to obtain a feature F I As the query Query, the image feature F obtained in step S42 is taken as the key Key and the text feature F obtained in step S41 is taken as the value Value, to perform cross-attention calculation to obtain a feature F to capture the correlation between the image content and different aesthetic quality categories; then F C The image feature F is taken as the input of a feedforward network to obtain a feature F I The image feature F is taken as the input of a feedforward network to obtain a feature F The image feature F is taken as the input of a feedforward network to obtain a feature F Step S44, design a loss function for the image aesthetic quality evaluation network, in order to more accurately measure the difference between the predicted distribution and the true distribution, and capture the order and distance relationship between the scores, use EMD loss as the target loss function of the image aesthetic quality evaluation network.

7. The image aesthetic evaluation method based on CLIP with text summarization and multi-granularity alignment according to claim 6, characterized in that, The step S5 is specifically implemented as follows: Step S51, randomly divide the images in the paired image-comment data set described in step S1 to obtain a plurality of batches containing B images; Step S52, the image input step S4 in the image aesthetic quality evaluation network to get the predicted aesthetic score distribution, using step S42 in the formula to calculate the loss l EMD ; Step S53, calculate the gradient of the parameters in the network that are not frozen using the back propagation method according to the loss function, and update these parameters using the Adam optimization method; Step S54, repeat steps S51 to S53 in batches to obtain a trained image aesthetic quality evaluation network.

8. The image aesthetic evaluation method based on CLIP with text summarization and multi-granularity alignment according to claim 1, characterized in that, The step S6 is specifically implemented as follows: Step S61, input the to-be-tested image into the trained image aesthetic quality evaluation network of step S5 to obtain an aesthetic score distribution predicted by the model The probability of the i-th score in the prediction distribution, based on the model-predicted aesthetic score distribution, the scores of all categories are weighted and summed with their corresponding prediction probabilities to obtain the aesthetic evaluation score Score; let the score of the i-th category be Where s i is the score of the i-th category. 9.A computer readable storage medium having stored thereon computer program instructions capable of being executed by a processor, when the processor executes the computer program instructions, steps of the method according to any one of claims 1-8 can be implemented.

Images