CLIP image aesthetic evaluation method based on text extraction and multi-granularity alignment

Abstract

The invention relates to a CLIP image aesthetic evaluation method based on text extraction and multi-granularity alignment, and belongs to the field of image processing and computer vision. The method comprises the following steps: performing text extraction on user comments in an aesthetics data set by utilizing an existing large language model, analyzing and summarizing aesthetics-related core description from the user comments, and further constructing an aesthetics abstract data set; secondly, designing a multi-granularity aligned CLIP framework, and performing fine adjustment on an original CLIP model by utilizing the paired image-detailed comment-aesthetic abstract based on the framework, so that the model better understands the semantic relationship between the image and text description in the field of aesthetics, thereby learning richer aesthetic feature representation, and improving the accuracy of aesthetics. Obtaining an optimized CLIP model for the aesthetic task; and finally, designing an image aesthetics quality evaluation network based on the fine-tuned CLIP model so as to realize effective prediction of the image aesthetics quality. The problem that image text pairs need to be input at the same time in the reasoning stage in a cross-modal aesthetic evaluation method is avoided.

Description

Technical Field

[0001] The invention belongs to the field of image processing and computer vision, and in particular relates to an image aesthetic evaluation method based on CLIP of text extraction and multi-granularity alignment. Background Art

[0002] In the field of modern image processing and computer vision, image aesthetic evaluation technology has received increasing attention. Image aesthetic evaluation is usually used to evaluate the visual appeal and artistic value of an image, and is widely used in many fields such as album management, image retrieval, and digital photography. In practical applications, image aesthetic evaluation needs to comprehensively consider multiple aesthetic elements such as image composition, color, texture, light and shadow, in order to provide an objective and accurate aesthetic score, thereby assisting users in selecting images of high aesthetic quality. Since image aesthetics is highly subjective and diverse, how to effectively capture and quantify these aesthetic features has become a core challenge in research.

[0003] Early traditional image aesthetic evaluation methods mainly rely on manually designed features, such as color, texture, contrast, clarity, etc. Although these features can reflect the basic aesthetic attributes of images to a certain extent, their expressive power is limited and it is difficult to effectively capture the complex relationship between multiple influencing factors of aesthetic quality evaluation.

[0004] With the development of deep learning technology, image aesthetic evaluation methods based on convolutional neural networks have been widely studied and applied. These methods can automatically extract high-level features of images through an end-to-end learning process and have achieved significant performance improvements on multiple aesthetic evaluation datasets. However, such methods usually ignore the rich aesthetic semantic information contained in user comment texts, resulting in certain limitations. To solve this problem, many cross-modal aesthetic evaluation methods based on deep learning have been proposed. Such methods process image and text modalities simultaneously, fuse the visual features of the image with the semantic features of the text, thereby capturing the aesthetic attributes of the image more comprehensively and further improving the performance of image aesthetic evaluation tasks. However, such methods require simultaneous input of image and text pairs in the inference stage, which is limited in practical applications.

[0005] In recent years, visual-language pre-training models such as CLIP have made breakthroughs in the field of multimodal semantic understanding, and have demonstrated strong capabilities in understanding semantic information related to image text, bringing new research opportunities for image aesthetic evaluation tasks. CLIP is trained through large-scale image-text pairing data, which can not only extract the semantic features of images, but also has excellent language processing and content understanding capabilities, providing potential cross-modal support for aesthetic evaluation tasks. However, since the training data of the CLIP model mainly comes from general fields, it lacks targeted modeling of fine aesthetic features in image aesthetics and is difficult to directly apply to complex aesthetic evaluation tasks. Therefore, how to improve the CLIP model's ability to understand and capture aesthetic features and effectively apply it to image aesthetic evaluation tasks is a key issue. Summary of the invention

[0006] The purpose of the present invention is to better apply the CLIP model to aesthetic evaluation tasks, and further solve the limitations of existing image aesthetic evaluation methods. An image aesthetic evaluation method based on text extraction and multi-granularity alignment CLIP is provided. The method uses an existing large language model to extract text from user comments in an aesthetic dataset, analyzes and summarizes the core descriptions related to aesthetics, and then constructs an aesthetic summary dataset; secondly, a multi-granularity aligned CLIP framework is designed, and based on the framework, the original CLIP model is fine-tuned using paired images-detailed comments-aesthetic summaries, so that the model can better understand the semantic relationship between images and text descriptions in the aesthetic field, thereby learning richer aesthetic feature representations and obtaining an optimized CLIP model for aesthetic tasks; finally, an image aesthetic quality evaluation network is designed based on the fine-tuned CLIP model to achieve effective prediction of image aesthetic quality.

[0007] To achieve the above object, the technical solution of the present invention is: an image aesthetic evaluation method based on text extraction and multi-granularity alignment CLIP, comprising:

[0008] Step S1, preprocessing the images and texts in the aesthetic image-comment dataset to obtain a paired image-comment dataset for training;

[0009] Step S2: construct an aesthetic summary dataset using a large language model; design a CLIP framework for multi-granular alignment;

[0010] Step S3, using the aesthetic summary dataset described in step S2 and the paired image-comment dataset described in step S1, the CLIP framework with multi-granular alignment described in step S2 is used to fine-tune the original CLIP model, so as to obtain an optimized CLIP model suitable for aesthetic tasks;

[0011] Step S4, designing an image aesthetic quality evaluation network, including an image branch and a text branch, wherein the image encoder in the image branch and the text encoder in the text branch are derived from the CLIP model fine-tuned in step S3;

[0012] Step S5, using the images in the paired image-comment dataset described in step S1 to train the image aesthetic quality evaluation network described in step S4, to obtain a trained image aesthetic quality evaluation network;

[0013] Step S6: input the image to be tested into the trained image aesthetic quality evaluation network, output the corresponding aesthetic quality evaluation score distribution, and calculate the aesthetic evaluation score based on it.

[0014] In one embodiment of the present invention, the specific implementation steps of step S1 are as follows:

[0015] Step S11, scaling each image in the aesthetic image-comment training dataset to an image of uniform size H×W×3, and performing standardization operations on the RGB channels of each scaled image, where H and W are the height and width of the image, respectively;

[0016] Step S12: convert the words of all the comment texts in the aesthetic image-comment training dataset into lowercase, and remove irrelevant symbols; then, segment the text using the byte pair encoding algorithm BPE to split the text into words or subword tags, each word or subword is mapped to a corresponding unique numerical ID in the vocabulary, that is, each ID represents the index of a word or subword in the vocabulary; in addition, add a start tag "[SOS]" and an end tag "[EOS]" at the beginning and end of the text sequence, respectively; the size of each text sequence finally obtained is 1×len, where len represents the maximum length of the input sequence supported by the model, including the start and end tags, and the part exceeding len is deleted, otherwise it is padded with 0.

[0017] In one embodiment of the present invention, the specific implementation steps of step S2 are as follows:

[0018] Step S21, using the large language model LLaMA-7B to perform text extraction on user comments in the aesthetic image-comment training dataset, analyzing and summarizing the descriptions related to aesthetics, and constructing an aesthetic summary dataset, wherein the aesthetic summaries in the aesthetic summary dataset are paired one by one with the images and comments in the aesthetic image-comment training dataset;

[0019] Step S22, preprocessing the text in the aesthetic summary data set obtained in step S21 using the method consistent with step S12, and the size of each text sequence after processing is 1×len;

[0020] Step S23, design a multi-granularity alignment CLIP framework, which can compare and learn the fine-grained features of the image with the corresponding detailed comment text features to achieve fine-grained alignment, and at the same time obtain the coarse-grained features of the image and the corresponding aesthetic summary features for comparative learning to achieve coarse-grained alignment, so that the model can capture the detailed aesthetic attributes of the image while identifying the key aesthetic features; in addition, the framework requires multiple pairs of samples as input, so as to compare through positive and negative sample pairs.

[0021] In one embodiment of the present invention, the specific implementation steps of step S21 are as follows:

[0022] Step S211, construct a prompt text as the input of the large language model LLaMA-7B, the prompt text content includes three parts: the first part is the task instruction description, telling the model that the current specific task is to analyze and summarize the various aesthetic-related descriptions involved in the given user comment text; the second part gives an example pair, the example pairs are separated by the "=>" symbol, the original comment text is represented before the symbol, and the aesthetic summary text expected after the task instruction operation is executed. According to the specific example, the large language model can learn and imitate the required reconstruction behavior; the last part includes the comment text that needs to be summarized and the separation symbol "=>";

[0023] Step S212: input the prompt text described in step S211 into the large language model LLaMA-7B, and the large language model LLaMA-7B automatically completes the text content after the separator "=>" in the prompt text, that is, obtains an aesthetic summary summarized from the current comment text;

[0024] Step S213: All user comments in the aesthetic image-comment training data set are processed through the operation described in step S212 to obtain an aesthetic summary data set that is paired one-to-one with each user comment.

[0025] In one embodiment of the present invention, the specific implementation steps of step S23 are as follows:

[0026] Step S231: Use the image encoder of the CLIP model to extract fine-grained image features. For a given batch of B images, Obtain fine-grained image feature matrix through image encoder Indicated as F Ifine =Image_Encoder(I B ), where Image_Encoder represents an image encoder based on the visual Transformer architecture; specifically, for each input image First, it is processed by convolution with a kernel size of P×P and a step size of P, and then flattened into a series of patch sequences. Where C represents the number of channels, P 2 Indicates the size of each image patch, Indicates the number of patches generated; then each patch is converted into a d-dimensional embedding vector through linear mapping, and a learnable [CLS] token is added to the front of the embedding sequence to capture global semantic information, and position embedding is added to retain the spatial position information of the patch, thereby generating an image embedding representation Then, i' is processed by L layers of Transformer layers to obtain Finally, I' Lcls After layer normalization and projection into the multimodal embedding space through linear mapping, the final image feature representation is obtained For B input images, the fine-grained image feature matrix representation is obtained Represents the fine-grained image features of the i-th image; the specific formula is as follows:

[0027] I p =Flatten(Conv P×P,P (I)),

[0028]

[0029] I′ 1 = Transformer(I′),

[0030] I′ l = Transformer(I′ l-1 ),I=2,...,L,

[0031]

[0032] Among them, Conv P×P,P (·) indicates a convolution with a kernel size of P×P and a stride of P, and Flatten(·) indicates a flattening operation; E, E pos and E I represents a learnable matrix, E is used to map each patch into a d-dimensional embedding vector, E pos Used to learn the spatial location information of the patch, E I It is used to project image features into the multimodal embedding space; Transformer(·) represents the Transformer layer, l represents the specific number of Transformer layers passed; LN(·) represents the layer normalization operation;

[0033] Step S232: Obtain coarse-grained image features, and use principal component analysis to transform the fine-grained image features obtained in step S231 into Perform principal component feature extraction to obtain feature representation after dimensionality reduction At this point, each image has lost some detail information; then, the inverse transform is used to map it back to the dimension of the original space to obtain a coarse-grained image feature representation. Specifically, first, we need to Ifine Perform centralization, that is, subtract the mean of each feature to obtain the centralized feature matrix Then, through the F Icenter Singular value decomposition is performed to obtain the principal component direction of the data. The column vector of the right singular matrix after singular value decomposition is the eigenvector, i.e., the principal component direction, and the first dim principal components are selected from them to retain the most representative information in the data; then, the data is projected onto the selected principal component direction to achieve feature dimensionality reduction and obtain the feature representation F after dimensionality reduction IPCA Finally, the reduced-dimensional data is back-projected using the selected principal component direction, mapped back to the original dimension and the mean of the original feature is added back to obtain the coarse-grained image feature representation. Represents the coarse-grained image features of the i-th image; the specific formula is as follows:

[0034]

[0035] F ICenter =F Ifine -u,

[0036]

[0037] V dim =V[:,:dim],

[0038] F IPCA =F Icenter V dim ,

[0039]

[0040] in, represents the singular value decomposition operation, U represents the left singular vector matrix, ∑ represents the singular value diagonal matrix, V represents the right singular vector matrix; dim represents the number of principal components selected, i.e. the first dim;

[0041] Step S233: Use the text encoder of the CLIP model to extract fine-grained text features. For a given batch of B detailed review texts, Get a fine-grained text feature matrix through the text encoder Indicated as F Tfine =Text_Encoder(T B ), where Text_Encoder represents a text encoder based on the Transformer architecture; specifically, for a text sequence after the word segmentation process in step S12 First, through the pre-initialized word embedding layer, each ID is mapped to a d-dimensional word embedding vector to obtain the text embedding representation Then, T' is processed by L layers of Transformer layers to obtain Finally, T' Llen[EOS] After layer normalization and projection to the multimodal embedding space through linear mapping, the final text feature representation is obtained For the input of segment B comment text, the fine-grained text feature matrix representation is obtained Represents the fine-grained text features of the i-th paragraph of text; the specific formula is as follows:

[0042] T′=Embedding(T),

[0043] T′ 1 = Transformer(T′),

[0044] T′ l = Transformer(T′ l-1 ),l=2,...,L,

[0045]

[0046] Where, Embedding(·) represents the embedding layer, Transformer(·) represents the Transformer layer, l represents the specific number of Transformer layers passed; LN(·) represents the layer normalization operation; E T represents a learnable matrix for projecting text features into a multimodal embedding space;

[0047] Step S234: extract coarse-grained text features using a text encoder consistent with step S233; for a given batch containing aesthetic summaries of segment B Get the coarse-grained text feature matrix through the text encoder Indicated as F Tcoarse =Text_Encoder(A B );

[0048] Step S235: Design fine-grained contrast loss function l fine For a given batch containing B pairs (image, detailed comment, aesthetic summary), follow the method described in steps S231 to S234 to obtain F Ifine 、F Icoarse 、F Tfine and F Tcoarse , expressed as For fine-grained image and text features There are B×B possible pairings, of which there are B pairs of real pairs; e fine Aims to maximize the number of true pairs of B in a batch and The cosine similarity between them, while minimizing B 2 -B wrong pairings to achieve the alignment of fine-grained image text features; the specific formula is as follows:

[0049]

[0050] Among them, l fi2t (·) represents the fine-grained image-to-text contrastive learning loss, l ft2i (·) represents the fine-grained text-to-image contrastive learning loss; Sim(·) represents the cosine similarity calculation, and τ represents the temperature coefficient used to scale the similarity score;

[0051] Step S236: Design a coarse-grained contrast loss function l coarse For a given batch containing B pairs (image, detailed comment, aesthetic summary), follow the method described in steps S231 to S234 to obtain F Ifine 、F Icoarse 、F Tfine and F Tcoarse , expressed as For coarse-grained image and text features There are B×B possible pairings, of which there are B pairs of real pairs; coarse Aims to maximize the number of true pairs of B in a batch and The cosine similarity between them, while minimizing B 2 -B wrong pairings are used to achieve the alignment of coarse-grained image text features; the specific formula is as follows:

[0052]

[0053] Among them, l ci2t (·) represents the coarse-grained image-to-text contrastive learning loss, l ct2i(·) represents the coarse-grained text-to-image contrastive learning loss; Sim(·) represents the cosine similarity calculation, and τ represents the temperature coefficient used to scale the similarity score;

[0054] Step S237: Design a multi-granularity contrast loss function l and use it as the overall objective function of the model. The multi-granularity contrast loss function l is the fine-grained contrast loss l fine Compared with the coarse-grained loss l coarse The specific formula is as follows:

[0055] l=l fine +l coarse .

[0056] In one embodiment of the present invention, the specific implementation steps of step S3 are as follows:

[0057] Step S31, randomly divide the paired (image, detailed comment, aesthetic summary) data set used for training described in step S1 and step S2 to obtain a plurality of batches containing B pairs (image, detailed comment, aesthetic summary);

[0058] Step S32: Use the pre-trained CLIP model as the initial model, input the images and texts in the data set in step S31 into the image encoder and the text encoder in batches, and obtain the image encoder according to the method described in step S23. And calculate the loss l according to the formula;

[0059] Step S33, using the back propagation method to calculate the gradient of the parameters in the model according to the loss function, and using the Adam optimization method to update the model parameters;

[0060] Step S34: repeat steps S31 to S33 until a preset number of training rounds is reached, thereby obtaining an optimized CLIP model suitable for aesthetic tasks.

[0061] In one embodiment of the present invention, the specific implementation steps of step S4 are as follows:

[0062] Step S41, design a text branch, including a text encoder with frozen parameters and five learnable prompt words as input; specifically, firstly introduce five learnable prompt words representing different aesthetic quality categories, which correspond to five quality categories of "terrible", "bad", "average", "good" and "perfect", respectively. Each prompt word consists of several learnable embedding vectors and words representing the corresponding quality category; specifically represented as "[X] 1 [X] 2 [X] 3 ...[X] M[quality category]", where each [X] i (i∈1,...M) is a learnable text token whose dimension is the same as the word embedding dimension; M represents the number of learnable text tokens. Subsequently, these prompt words are input into the text encoder to generate five corresponding feature vectors, which are expressed as

[0063] Step S42, design an image branch, including a parameter-frozen image encoder and an adapter module, the adapter module includes four sub-adapters, each adapter consists of a fully connected layer that maps input features to a lower dimension, a GELU activation function, and a fully connected layer that maps the low-dimensional features back to the original dimension, and the input of each adapter is I' of four different stages in the image encoder. lcls Specifically, the image encoder consists of L layers of Transformer layers, which are first divided into four stages, each of which contains L s layer, The output of the first stage is L s The [CLS] token in the layer output is represented by As the input of the first adapter, we get the feature F A1 ; The output of the second stage is 2L S The [CLS] token in the layer output is represented by It is used as the input of the second adapter to obtain the feature F A2 ; The output of the third stage is 3L s The [CLS] token in the layer output is represented by It is used as the input of the third adapter to obtain the feature F A3 ; The output of the fourth stage is the [CLS] token in the Lth layer output, represented by I' lcls , which is used as the input of the fourth adapter to obtain the feature F A4 Then, the output features of the four adapters are added and fused, and the final image features are obtained through linear mapping. The specific formula is as follows:

[0064]

[0065] F A4 =FC Up(GELU(FC Down(I′ Lcls ))),

[0066]

[0067] Among them, FC DOwn(·) represents the fully connected layer used to map the input features to a lower dimension, GELU·) represents the GELU activation function, and FC Up(·) represents the fully connected layer used to map the low-dimensional features back to the original dimension; E F Represents a learnable matrix used to transform the feature F I Projected to the text feature F P Same dimensions;

[0068] Step S43: The text feature F obtained in step S41 P As the key Key and value Value, the image feature F obtained in step S42 is I As the query Query, cross attention calculation is performed to obtain the feature to capture the correlation between image content and different aesthetic quality categories; then F C After a feed-forward network and image features F I Fusion is performed to obtain the final output features It is then further refined through another feed-forward network to finally predict the aesthetic score distribution of the image; the specific formula is as follows:

[0069] F C =CrossAttention(F I ,F p ),

[0070] F O =FFN(F C )+F I ,

[0071]

[0072] Among them, CrossAttention(·) represents cross attention calculation, FFN(·) represents feedforward network; represents the predicted aesthetic rating distribution;

[0073] Step S44, designing the loss function of the image aesthetic quality evaluation network. In order to more accurately measure the difference between the predicted distribution and the true distribution and capture the order and distance relationship between the scores, the EMD loss is used as the target loss function of the image aesthetic quality evaluation network. The specific expression formula is as follows:

[0074]

[0075] Where T represents the number of aesthetic score categories; represents the cumulative distribution function of the true distribution P, p i represents the probability of the i-th score in the true distribution; Represents the predicted distribution The cumulative distribution function of represents the probability of the i-th score in the prediction distribution.

[0076] In one embodiment of the present invention, the specific implementation steps of step S5 are as follows:

[0077] Step S51, randomly dividing the images in the paired image-comment dataset described in step S1 to obtain a plurality of batches containing B images;

[0078] Step S52: Input the image into the image aesthetic quality evaluation network in step S4 to obtain the predicted aesthetic score distribution, and use the formula in step S42 to calculate the loss l EMD ;

[0079] Step S53, using the back propagation method to calculate the gradients of the parameters that are not frozen in the network according to the loss function, and using the Adam optimization method to update these parameters;

[0080] Step S54: Repeat steps S51 to S53 in batches to obtain a trained image aesthetic quality evaluation network.

[0081] In one embodiment of the present invention, the specific implementation steps of step S6 are as follows:

[0082] Step S61: Input the image to be tested into the trained image aesthetic quality evaluation network described in step S5 to obtain the aesthetic score distribution predicted by the model. represents the probability of the i-th score in the predicted distribution. Based on the distribution of aesthetic scores predicted by the model, the aesthetic evaluation score Score is obtained by weighted summing the scores of all categories and their corresponding predicted probabilities. Suppose the score of the scoring category is where s i is the score of the i-th scoring category, the specific formula is as follows:

[0083]

[0084] The present invention also provides a computer-readable storage medium, on which computer program instructions that can be executed by a processor are stored. When the processor executes the computer program instructions, the method steps described above can be implemented.

[0085] Compared with the prior art, the present invention has the following beneficial effects: first, the present invention constructs an aesthetic summary dataset, uses the existing large language model to analyze and summarize the user comments in the aesthetic dataset, extracts the core descriptions related to aesthetics, and provides more data support for subsequent aesthetic evaluation tasks. Secondly, the present invention designs a multi-granularity aligned CLIP framework, compares and learns the fine-grained features of the image with the corresponding detailed comment text features to achieve fine-grained alignment, and obtains the coarse-grained features of the image and the corresponding aesthetic summary features for comparison and learning to achieve coarse-grained alignment, so that the model can capture the detailed aesthetic attributes of the image while also identifying the key aesthetic features. Finally, based on the CLIP model fine-tuned by the multi-granularity alignment framework, the present invention designs an image aesthetic quality evaluation network, which freezes the text encoder and the image encoder for feature extraction; at the same time, considering the subjectivity and diversity of image aesthetic evaluation, five learnable prompt words representing different aesthetic quality categories are introduced in the text branch as auxiliary features to improve the accuracy of the aesthetic evaluation task; and the image branch introduces an adapter module for fine-tuning to improve the generalization of the model. Different from other image aesthetic evaluation methods based on deep learning, the present invention makes full use of the cross-modal advantages of the vision-language model and adopts a multi-granularity alignment fine-tuning strategy to enable the CLIP model to better capture and understand the semantic connection between images and diverse text descriptions, and learn rich aesthetic feature representations; in addition, learnable prompt words are introduced as auxiliary features to further improve the performance of the aesthetic evaluation task, avoiding the problem that existing cross-modal aesthetic evaluation methods need to simultaneously input image-text pairs in the reasoning stage, providing a more efficient and flexible solution for aesthetic evaluation. BRIEF DESCRIPTION OF THE DRAWINGS

[0086] Figure 1 It is a flow chart for realizing the method of the present invention.

[0087] Figure 2 It is a framework diagram of text extraction in an embodiment of the present invention.

[0088] Figure 3 4 is a structural diagram of the CLIP framework for multi-granularity alignment in an embodiment of the present invention.

[0089] Figure 4 is a structural diagram of an image aesthetic quality evaluation network in an embodiment of the present invention. DETAILED DESCRIPTION

[0090] The technical solution of the present invention is described in detail below in conjunction with the accompanying drawings.

[0091] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present application belongs.

[0092] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0093] The present invention provides an image aesthetic evaluation method based on text extraction and multi-granularity aligned CLIP, comprising:

[0094] Step S1, preprocessing the images and texts in the aesthetic image-comment dataset to obtain a paired image-comment dataset for training;

[0095] Step S2: construct an aesthetic summary dataset using a large language model; design a CLIP framework for multi-granular alignment;

[0096] Step S3, using the aesthetic summary dataset described in step S2 and the paired image-comment dataset described in step S1, the CLIP framework with multi-granular alignment described in step S2 is used to fine-tune the original CLIP model, so as to obtain an optimized CLIP model suitable for aesthetic tasks;

[0097] Step S4, designing an image aesthetic quality evaluation network, including an image branch and a text branch, wherein the image encoder in the image branch and the text encoder in the text branch are derived from the CLIP model fine-tuned in step S3;

[0098] Step S5, using the images in the paired image-comment dataset described in step S1 to train the image aesthetic quality evaluation network described in step S4, to obtain a trained image aesthetic quality evaluation network;

[0099] Step S6: input the image to be tested into the trained image aesthetic quality evaluation network, output the corresponding aesthetic quality evaluation score distribution, and calculate the aesthetic evaluation score based on it.

[0100] The following is the specific implementation process of the present invention.

[0101] The present invention provides an image aesthetic evaluation method based on text extraction and multi-granularity alignment CLIP, such as Figure 1 As shown, the following steps are included:

[0102] Step S1, preprocessing the images and texts in the aesthetic image-comment dataset to obtain a paired image-comment dataset for training;

[0103] Step S2: construct an aesthetic summary dataset using a large language model; design a CLIP framework for multi-granular alignment;

[0104] Step S3, using the aesthetic summary dataset described in step S2 and the image-comment dataset described in step S1 to fine-tune the original CLIP model using the multi-granularity alignment framework described in step S2, to obtain an optimized CLIP model suitable for aesthetic tasks;

[0105] Step S4, designing an image aesthetic quality evaluation network, which includes an image branch and a text branch, wherein the image encoder in the image branch and the text encoder in the text branch are derived from the CLIP model fine-tuned in step S3;

[0106] Step S5, using the images in the training data set described in step S1 to train the image aesthetic quality evaluation network described in step S4, so as to obtain a trained image aesthetic quality evaluation network;

[0107] Step S6: input the image to be tested into the trained image aesthetic quality evaluation network, output the corresponding aesthetic quality evaluation score distribution, and calculate the aesthetic evaluation score based on it.

[0108] Furthermore, the step S1 includes the following steps:

[0109] Step S11, scaling each image in the aesthetic image-comment training dataset to an image of uniform size H×W×3, and performing standardization operations on the three RGB channels of each scaled image respectively;

[0110] Step S12: Convert all the words in the comment text in the aesthetic image-comment training data set to lowercase, and remove irrelevant symbols. Subsequently, the text is segmented by the byte pair encoding algorithm (BPE), and the text is split into word or subword tags. Each word or subword is mapped to a corresponding unique numerical ID in the vocabulary, that is, each ID represents the index of a word or subword in the vocabulary. In addition, the start tag "[SOS]" and the end tag "[EOS]" are added at the beginning and end of the text sequence, respectively. The size of each text sequence finally obtained is 1×len, where len represents the maximum length of the input sequence supported by the model (including the start and end tags). The part exceeding len will be deleted, otherwise it will be padded with 0.

[0111] Furthermore, the step S2 comprises the following steps:

[0112] Step S21, using the large language model LLaMA-7B to perform text extraction on user comments in the aesthetic image-comment training dataset, analyzing and summarizing the descriptions related to aesthetics, and constructing an aesthetic summary dataset, in which the aesthetic summaries in the dataset are paired one by one with the images and comments in the aesthetic image-comment training dataset; Figure 2 A framework diagram of text extraction in an embodiment of the present invention;

[0113] Step S22: preprocess the text in the aesthetic summary data set obtained in step S21 using the method consistent with that in step S12, and the size of each text sequence after processing is 1×len.

[0114] Step S23, designing a CLIP framework for multi-granularity alignment, which can compare and learn the fine-grained features of the image with the corresponding detailed review text features to achieve fine-grained alignment, and at the same time obtain the coarse-grained features of the image and the corresponding aesthetic summary features to achieve coarse-grained alignment, so that the model can capture the detailed aesthetic attributes of the image while also identifying the key aesthetic features; in addition, the framework requires multiple pairs of samples as input, so as to compare through positive and negative sample pairs; Figure 3 is a structural diagram of a CLIP framework for multi-granularity alignment in an embodiment of the present invention;

[0115] Furthermore, the step S21 includes the following steps:

[0116] Step S211, construct a prompt text as the input of the large language model, the text content mainly includes three parts: the first part is the task instruction description, telling the model that the current specific task is to analyze and summarize the various aesthetic-related descriptions involved in the given user comment text; the second part gives an example pair, and the example pairs are separated by the "=>" symbol, the original comment text is represented before the symbol, and the aesthetic summary text expected after the task instruction operation is executed. According to the specific example, the large language model can learn and imitate the required reconstruction behavior; the last part includes the comment text that needs to be summarized and the separation symbol "=>";

[0117] Step S212: input the prompt text described in step S211 into the large language model LLaMA-7B, and the model will automatically complete the text content after the separator "=>" in the prompt text, that is, obtain an aesthetic summary summarized from the current comment text;

[0118] Step S213: All user comments in the aesthetic image-comment training data set are processed through the operation described in step S212 to obtain an aesthetic summary data set that is paired one-to-one with each user comment.

[0119] Furthermore, the step S23 includes the following steps:

[0120] Step S231: Use the image encoder of the CLIP model to extract fine-grained image features. Obtain fine-grained image feature matrix through image encoder Indicated as F Ifine =Image_Encoder(I B ), where Image_Encoder represents the image encoder. The image encoder is based on the visual Transformer architecture. Specifically, for each input image First, it is processed by convolution with a kernel size of P×P and a step size of P, and then flattened into a series of patch sequences to obtain Where C represents the number of channels, P 2 Indicates the size of each image patch, Indicates the number of patches generated. Each patch is then converted into a d-dimensional embedding vector through linear mapping. At the same time, a learnable [CLS] token is added to the front of the embedding sequence to capture global semantic information, and position embedding is added to retain the spatial position information of the patch, thereby generating an image embedding representation. Then, I' is processed by L layers of Transformer layers to obtain Finally, I' Lcls After layer normalization and projection into the multimodal embedding space through linear mapping, the final image feature representation is obtained For B input images, the fine-grained image feature matrix representation is obtained Represents the fine-grained image features of the i-th image. The specific formula is as follows:

[0121] I p =Flatten(Conv P×P,P (I)),

[0122]

[0123] I′ 1 = Transformer(I′),

[0124] I′ l = Transformer(I′ l-1 ),l=2,...,L,

[0125]

[0126] Among them, Conv P×P,P(·) represents a convolution with a kernel size of P×P and a stride of P, and Flatten(·) represents a flattening operation; E, E pos and E I represents a learnable matrix, E is used to map each patch into a d-dimensional embedding vector, E pos It is used to learn the spatial position information of the patch, and EI is used to project the image features into the multimodal embedding space; Transformer (·) represents the Transformer layer, l represents the specific number of Transformer layers passed; LN (·) represents the layer normalization operation;

[0127] Step S232: Obtain coarse-grained image features. Use principal component analysis to convert the fine-grained image features obtained in step S231 into Perform principal component feature extraction to obtain feature representation after dimensionality reduction At this point, each image has lost some detail information. Then, the inverse transform is used to map it back to the dimension of the original space to obtain a coarse-grained image feature representation. Specifically, first, we need to Ifine Perform centralization, that is, subtract the mean of each feature to obtain the centralized feature matrix Then, through the F Icenter Perform singular value decomposition to obtain the principal component direction of the data (the column vector of the right singular matrix after singular value decomposition is the eigenvector, i.e., the principal component direction), and select the first dim principal components to retain the most representative information in the data. Then, project the data onto the selected principal component direction to achieve feature dimensionality reduction and obtain the feature representation F after dimensionality reduction. IPCA Finally, the reduced-dimensional data is back-projected using the selected principal component direction, mapped back to the original dimension and the mean of the original feature is added back to obtain the coarse-grained image feature representation. Represents the coarse-grained image features of the i-th image. The specific formula is as follows:

[0128]

[0129] F ICenter =F Ifine -u,

[0130]

[0131] V dim =V[:,:dim],

[0132] F IPCA =F Icenter V dim ,

[0133]

[0134] in, represents the singular value decomposition operation, U represents the left singular vector matrix, ∑ represents the singular value diagonal matrix, V represents the right singular vector matrix; dim represents the number of principal components selected, i.e. the first dim;

[0135] Step S233: Extract fine-grained text features using the text encoder of the CLIP model. Get a fine-grained text feature matrix through the text encoder Indicated as F Tfine =Text_Encoder(T B ), where Text_Encoder represents a text encoder. The text encoder is based on the Transformer architecture. Specifically, for a text sequence after the word segmentation process in step S12 First, through the pre-initialized word embedding layer, each ID is mapped to a d-dimensional word embedding vector to obtain the text embedding representation Then, T' is processed by L layers of Transformer layers to obtain Finally, T' Llen[EOS] After layer normalization and projection to the multimodal embedding space through linear mapping, the final text feature representation is obtained For the comment text input in segment B, the fine-grained text feature matrix representation is obtained Represents the fine-grained text features of the i-th paragraph of text. The specific formula is as follows:

[0136] T′=Embedding(T),

[0137] T′ 1 = Transformer(T′),

[0138] T′ l = Transformer(T′ l-1 ),l=2,...,L,

[0139]

[0140] Where, Embedding(·) represents the embedding layer, Transformer(·) represents the Transformer layer, l represents the specific number of Transformer layers passed; LN(·) represents the layer normalization operation; ET represents a learnable matrix for projecting text features into a multimodal embedding space;

[0141] Step S234: Extract coarse-grained text features using the text encoder consistent with step S233. Get the coarse-grained text feature matrix through the text encoder Indicated as F Tcoarse =Text_Encoder(A B );

[0142] Step S235: Design fine-grained contrast loss function l fine For a given batch containing B pairs (image, detailed comment, aesthetic summary), the method described in steps S231 to S234 is used to obtain F Ifine 、F Icoarse 、F Tfine and F Tcoarse , which can be expressed as For fine-grained image and text features There are B×B possible pairings, of which there are B pairs. fine Aims to maximize the number of true pairs of B in a batch and The cosine similarity between them, while minimizing B 2 -B wrong pairings to achieve the alignment of fine-grained image text features. The specific formula is as follows:

[0143]

[0144] Among them, l fi2t (·) represents the fine-grained image-to-text contrastive learning loss, l ft2i (·) represents the fine-grained text-to-image contrastive learning loss; Sim(·) represents the cosine similarity calculation, and τ represents the temperature coefficient used to scale the similarity score;

[0145] Step S236: Design a coarse-grained contrast loss function l coarse For a given batch containing B pairs (image, detailed comment, aesthetic summary), the method described in steps S231 to S234 is used to obtain F Ifine 、F Icoarse 、F Tfine and F Tcoarse , which can be expressed as For coarse-grained image and text features There are B×B possible pairings, of which there are B pairs. coarseAims to maximize the number of true pairs of B in a batch and The cosine similarity between them, while minimizing B 2 -B wrong pairings to achieve the alignment of coarse-grained image text features. The specific formula is as follows:

[0146]

[0147]

[0148] Among them, l ci2t (·) represents the coarse-grained image-to-text contrastive learning loss, l ct2i (·) denotes the coarse-grained text-to-image contrastive learning loss; Sim(·) denotes the cosine similarity calculation, and τ represents the temperature coefficient used to scale the similarity score.

[0149] Step S237: Design a multi-granularity contrast loss function l and use it as the overall objective function of the model. This loss is the fine-grained contrast loss l fine Compared with the coarse-grained loss l coarse The specific formula is as follows:

[0150] l=l fine +l coarse .

[0151] Furthermore, step S3 includes the following steps:

[0152] Step S31, randomly divide the paired (image, detailed comment, aesthetic summary) data set used for training described in step S1 and step S2 to obtain a plurality of batches containing B pairs (image, detailed comment, aesthetic summary);

[0153] Step S32: Use the pre-trained CLIP model as the initial model, input the images and texts in the data set in step S31 into the image encoder and the text encoder in batches, and obtain the image encoder according to the method described in step S23. And calculate the loss l according to the formula;

[0154] Step S33: Calculate the gradient of the parameters in the model using the back propagation method according to the loss function, and update the model parameters using the Adam optimization method.

[0155] Step S34: repeat steps S31 to S33 until a preset number of training rounds is reached, thereby obtaining an optimized CLIP model suitable for aesthetic tasks.

[0156] Furthermore, the step S4 comprises the following steps:

[0157] Step S41, design a text branch. This branch includes a text encoder with frozen parameters and five learnable prompt words as input. The text encoder is derived from the CLIP model fine-tuned in step S3. Specifically, five learnable prompt words representing different aesthetic quality categories are first introduced. These prompt words correspond to the five quality categories of "terrible", "bad", "average", "good" and "perfect". Each prompt word consists of several learnable embedding vectors and words representing the corresponding quality category. It can be specifically expressed as "[X] 1 [X] 2 [X] 3 ...[X] M [quality category]", where each [X] i (i∈1,...M) is a learnable text token, whose dimension is the same as the word embedding dimension; M represents the number of learnable text tokens. Subsequently, these prompt words are input into the text encoder to generate five corresponding feature vectors, represented as

[0158] Step S42, design an image branch. This branch includes a parameter-frozen image encoder and an adapter module. The image encoder is derived from the CLIP model fine-tuned in step S3. The adapter module includes four sub-adapters, each of which consists of a fully connected layer that maps input features to a lower dimension, a GELU activation function, and a fully connected layer that maps low-dimensional features back to the original dimension. The input of each adapter is I' at four different stages in the image encoder. lcls Specifically, the image encoder consists of L layers of Transformer layers, which are first divided into four stages, each of which contains L s layer, The output of the first stage is L s The [CLS] token in the layer output is represented by As the input of the first adapter, we get the feature F A1 ; The output of the second stage is 2L s The [CLS] token in the layer output is represented by It is used as the input of the second adapter to obtain the feature F A2 ; The output of the third stage is 3L s The [CLS] token in the layer output is represented by It is used as the input of the third adapter to obtain the feature F A3 ; The output of the fourth stage is the [CLS] token in the Lth layer output, represented by I'Lcls , which is used as the input of the fourth adapter to obtain the feature F A4 Next, the output features of the four adapters are added and fused, and the final image features are obtained through linear mapping. The specific formula is as follows:

[0159]

[0160] F A4 =FC Up(GELU(FC Down(I′ Lcls ))),

[0161]

[0162] Among them, FC DOwn(·) represents the fully connected layer used to map the input features to a lower dimension, GELU·) represents the GELU activation function, and FC Up(·) represents the fully connected layer used to map the low-dimensional features back to the original dimension; E F Represents a learnable matrix used to transform the feature F I Projected to the text feature F P Same dimensions;

[0163] Step S43: The text feature F obtained in step S41 P As the key (Key) and value (Value), the image feature F obtained in step S42 is I As a query, perform cross-attention calculation to obtain features to capture the correlation between image content and different aesthetic quality categories. C After a feed-forward network and image features F I Fusion is performed to obtain the final output features It is then further refined through another feed-forward network to finally predict the aesthetic score distribution of the image. The specific formula is as follows:

[0164] F C =CrossAttention(F I ,F p ),

[0165] F O =FFN(F C )+F I ,

[0166]

[0167] Among them, CrossAttention(·) represents cross attention calculation, FFN(·) represents feedforward network; represents the predicted aesthetic rating distribution;

[0168] Step S44, design the loss function of the image aesthetic quality evaluation network. In order to more accurately measure the difference between the predicted distribution and the true distribution, and capture the order and distance relationship between the scores, the EMD loss is used as the target loss function of the image aesthetic quality evaluation network. The specific expression formula is as follows:

[0169]

[0170] Where T represents the number of aesthetic score categories; represents the cumulative distribution function of the true distribution P, p i represents the probability of the i-th score in the true distribution; Represents the predicted distribution The cumulative distribution function of represents the probability of the i-th score in the prediction distribution.

[0171] Figure 4 is a structural diagram of an image aesthetic quality evaluation network in an embodiment of the present invention.

[0172] Furthermore, the step S5 comprises the following steps:

[0173] Step S51, randomly dividing the images in the training data set described in step S1 to obtain a plurality of batches containing B images;

[0174] Step S52: Input the image into the image aesthetic quality evaluation network in step S4 to obtain the predicted aesthetic score distribution, and use the formula in step S42 to calculate the loss l EMD ;

[0175] Step S53, using the back propagation method to calculate the gradients of the parameters that are not frozen in the network according to the loss function, and using the Adam optimization method to update these parameters;

[0176] Step S54: Repeat steps S51 to S53 in batches to obtain an image aesthetic quality evaluation network.

[0177] Furthermore, step S6 includes the following steps:

[0178] Step S61: Input the image to be tested into the trained image aesthetic quality evaluation network described in step S5 to obtain the aesthetic score distribution predicted by the model. Based on this distribution, the aesthetic evaluation score can be obtained by weighted summing up the scores of all categories and their corresponding prediction probabilities. where s i is the score of the i-th scoring category, the specific formula is as follows:

[0179]

[0180] The present invention also provides a computer-readable storage medium, on which computer program instructions that can be executed by a processor are stored. When the processor executes the computer program instructions, the method steps described above can be implemented.

[0181] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0182] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0183] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0184] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0185] The above is only a preferred embodiment of the present invention, and does not limit the present invention in other forms. Any technician familiar with the profession may use the above disclosed technical content to change or modify it into an equivalent embodiment with equivalent changes. However, any simple modification, equivalent change and modification made to the above embodiment according to the technical essence of the present invention without departing from the technical solution of the present invention still belongs to the protection scope of the technical solution of the present invention.

Claims

1. An image aesthetic evaluation method based on text extraction and multi-granularity alignment CLIP, characterized in that: include: Step S1, preprocessing the images and texts in the aesthetic image-comment dataset to obtain a paired image-comment dataset for training; Step S2: construct an aesthetic summary dataset using a large language model; design a CLIP framework for multi-granular alignment; Step S3, using the aesthetic summary dataset described in step S2 and the paired image-comment dataset described in step S1, the CLIP framework with multi-granular alignment described in step S2 is used to fine-tune the original CLIP model, so as to obtain an optimized CLIP model suitable for aesthetic tasks; Step S4, designing an image aesthetic quality evaluation network, including an image branch and a text branch, wherein the image encoder in the image branch and the text encoder in the text branch are derived from the CLIP model fine-tuned in step S3; Step S5, using the images in the paired image-comment dataset described in step S1 to train the image aesthetic quality evaluation network described in step S4, to obtain a trained image aesthetic quality evaluation network; Step S6: input the image to be tested into the trained image aesthetic quality evaluation network, output the corresponding aesthetic quality evaluation score distribution, and calculate the aesthetic evaluation score based on it.

2. The image aesthetic evaluation method based on text extraction and multi-granularity alignment CLIP according to claim 1, characterized in that: The specific implementation steps of step S1 are as follows: Step S11, scaling each image in the aesthetic image-comment training dataset to an image of uniform size H×W×3, and performing standardization operations on the RGB channels of each scaled image, where H and W are the height and width of the image, respectively; Step S12: convert the words of all the comment texts in the aesthetic image-comment training dataset into lowercase, and remove irrelevant symbols; then, perform word segmentation on the text through the byte pair encoding algorithm BPE, split the text into word or subword tags, and each word or subword is mapped to a corresponding unique numerical ID in the vocabulary, that is, each ID represents the index of a word or subword in the vocabulary; in addition, add a start tag "[SOS]" and an end tag "[EOS]" at the beginning and end of the text sequence respectively; the size of each text sequence finally obtained is 1×len, len represents the maximum length of the input sequence supported by the model, including the start and end tags, and the part exceeding len will be deleted, otherwise it will be padded with 0.

3. The image aesthetic evaluation method based on text extraction and multi-granularity alignment CLIP according to claim 2, characterized in that: The specific implementation steps of step S2 are as follows: Step S21, using the large language model LLaMA-7B to perform text extraction on user comments in the aesthetic image-comment training dataset, analyzing and summarizing the descriptions related to aesthetics, and constructing an aesthetic summary dataset, wherein the aesthetic summaries in the aesthetic summary dataset are paired one by one with the images and comments in the aesthetic image-comment training dataset; Step S22, preprocessing the text in the aesthetic summary data set obtained in step S21 using the method consistent with step S12, and the size of each text sequence after processing is 1×len; Step S23, design a multi-granularity alignment CLIP framework, which can compare and learn the fine-grained features of the image with the corresponding detailed comment text features to achieve fine-grained alignment, and at the same time obtain the coarse-grained features of the image and the corresponding aesthetic summary features for comparative learning to achieve coarse-grained alignment, so that the model can capture the detailed aesthetic attributes of the image while identifying the key aesthetic features; in addition, the framework requires multiple pairs of samples as input, so as to compare through positive and negative sample pairs.

4. The image aesthetic evaluation method based on text extraction and multi-granularity alignment CLIP according to claim 3 is characterized in that: The specific implementation steps of step S21 are as follows: Step S211, construct a prompt text as the input of the large language model LLaMA-7B, the prompt text content includes three parts: the first part is the task instruction description, telling the model that the current specific task is to analyze and summarize the various aesthetic-related descriptions involved in the given user comment text; the second part gives an example pair, and the example pairs are separated by the "=>" symbol, the original comment text is represented before the symbol, and the aesthetic summary text expected after the task instruction operation is executed after the symbol. According to the specific example, the large language model can learn and imitate the required reconstruction behavior; the last part includes the comment text that needs to be summarized and the separation symbol "=>"; Step S212: input the prompt text described in step S211 into the large language model LLaMA-7B, and the large language model LLaMA-7B automatically completes the text content after the separator "=>" in the prompt text, that is, obtains an aesthetic summary summarized from the current comment text; Step S213: All user comments in the aesthetic image-comment training data set are processed through the operation described in step S212 to obtain an aesthetic summary data set that is paired one-to-one with each user comment.

5. The image aesthetic evaluation method based on text extraction and multi-granularity alignment CLIP according to claim 3 is characterized in that: The specific implementation steps of step S23 are as follows: Step S231: Use the image encoder of the CLIP model to extract fine-grained image features. For a given batch of B images, Obtain fine-grained image feature matrix through image encoder Indicated as F Ifine =Image_Encoder(I B ), where Image_Encoder represents an image encoder based on the visual Transformer architecture; specifically, for each input image First, it is processed by convolution with a kernel size of P×P and a step size of P, and then flattened into a series of patch sequences to obtain I p =[I1; I2; ...; I N ], Where C represents the number of channels, P 2 Indicates the size of each image patch, Indicates the number of patches generated; then each patch is converted into a d-dimensional embedding vector through linear mapping, and a learnable [CLS] token is added to the front of the embedding sequence to capture global semantic information, and position embedding is added to retain the spatial position information of the patch, thereby generating an image embedding representation I′=[I′ cls ; I′1; I′2; …; I′ N ], Then, I′ is processed by L layers of Transformer layers to obtain I′ L =[I′ Lcls ; I′ L1 ; I′ L2 ;…;I′ LN ], Finally, I′ Lcls After layer normalization and projection into the multimodal embedding space through linear mapping, the final image feature representation is obtained For B input images, the fine-grained image feature matrix representation is obtained Represents the fine-grained image features of the i-th image; the specific formula is as follows: I p =Flatten(Cone P×P，P (I)), I′1=Transformer(I′), I′ l =Transformer(I′ l-1 ),l=2,...,L, Among them, Conv P×P，P (·) indicates a convolution with a kernel size of P×P and a stride of P, and Flatten(·) indicates a flattening operation; E, E pos and E I represents a learnable matrix, E is used to map each patch into a d-dimensional embedding vector, E pos Used to learn the spatial location information of the patch, E I It is used to project image features into the multimodal embedding space; Transformer(·) represents the Transformer layer, l represents the specific number of Transformer layers passed; LN(·) represents the layer normalization operation; Step S232: Obtain coarse-grained image features, and use principal component analysis to transform the fine-grained image features obtained in step S231 into Perform principal component feature extraction to obtain feature representation after dimensionality reduction At this point, each image has lost some detail information; then, the inverse transform is used to map it back to the dimension of the original space to obtain a coarse-grained image feature representation. Specifically, first, the feature F Ifine Perform centralization, that is, subtract the mean of each feature to obtain the centralized feature matrix Then, through the F Icenter Singular value decomposition is performed to obtain the principal component direction of the data. The column vector of the right singular matrix after singular value decomposition is the eigenvector, i.e., the principal component direction, and the first dim principal components are selected from them to retain the most representative information in the data; then, the data is projected onto the selected principal component direction to achieve feature dimensionality reduction and obtain the feature representation F after dimensionality reduction IPCA Finally, the reduced-dimensional data is back-projected using the selected principal component direction, mapped back to the original dimension and the mean of the original feature is added back to obtain the coarse-grained image feature representation. Represents the coarse-grained image features of the i-th image; the specific formula is as follows: F Icenter =F Ifine -u, V dim =V[:,:dim], F IPCA =F Icenter V dim , in, represents the singular value decomposition operation, U represents the left singular vector matrix, ∑ represents the singular value diagonal matrix, V represents the right singular vector matrix; dim represents the number of principal components selected, i.e. the first dim; Step S233: Use the text encoder of the CLIP model to extract fine-grained text features. For a given batch of B detailed review texts, Get a fine-grained text feature matrix through the text encoder Indicated as F Tfine =Text_Encoder(T B ), where Text_Encoder represents a text encoder based on the Transformer architecture; specifically, for a text sequence after the word segmentation process in step S12 First, each ID is mapped to a d-dimensional word embedding vector through a pre-initialized word embedding layer, thereby obtaining a text embedding representation T′=[T′ 1[SOS] ; T′2; T′3; …; T′ len[EOS] ], Then, T′ is processed by L layers of Transformer layers to obtain T′ L =[T′ L1[SOS] ; T′ L2 ; T′ L3 ;…;T′ Llen[EOS] ], Finally, T′ Llen[EOS] After layer normalization and projection to the multimodal embedding space through linear mapping, the final text feature representation is obtained For the input of segment B comment text, the fine-grained text feature matrix representation is obtained Represents the fine-grained text features of the i-th paragraph of text; the specific formula is as follows: T′=Embedding(T), T′1=Transformer(T′), T′ l =Transformer(T′ l-1 ),l=2,...,L, Where, Embedding(·) represents the embedding layer, Transformer(·) represents the Transformer layer, l represents the specific number of Transformer layers passed; LN(·) represents the layer normalization operation; E T represents a learnable matrix for projecting text features into a multimodal embedding space; Step S234: extract coarse-grained text features using a text encoder consistent with step S233; for a given batch containing aesthetic summaries of segment B Get the coarse-grained text feature matrix through the text encoder Indicated as F Tcoarse =Text_Encoder(A B ); Step S235: Design fine-grained contrast loss function For a given batch containing B pairs (image, detailed comment, aesthetic summary), the method described in steps S231 to S234 is used to obtain F Ifine 、F Icoarse 、F Tfine and F Tcoarse , expressed as For fine-grained image and text features There are B×B possible pairings, of which there are B pairs that are real; Aims to maximize the number of true pairs of B in a batch and The cosine similarity between them, while minimizing B 2 -B wrong pairings to achieve the alignment of fine-grained image text features; the specific formula is as follows: in, represents the fine-grained image-to-text contrastive learning loss, represents the contrastive learning loss of fine-grained text to image; Sim(·) represents the cosine similarity calculation, and τ represents the temperature coefficient used to scale the similarity score; Step S236: Design a coarse-grained contrast loss function For a given batch containing B pairs (image, detailed comment, aesthetic summary), the method described in steps S231 to S234 is used to obtain F Ifine 、F Icoarse 、F Tfine and F Tcoarse , expressed as For coarse-grained image and text features There are B×B possible pairings, of which there are B pairs that are real; Aims to maximize the number of true pairs of B in a batch and The cosine similarity between them, while minimizing B 2 -B wrong pairings are used to achieve the alignment of coarse-grained image text features; the specific formula is as follows: in, represents the coarse-grained image-to-text contrastive learning loss, represents the coarse-grained text-to-image contrastive learning loss; Sim(·) represents the cosine similarity calculation, and τ represents the temperature coefficient used to scale the similarity score; Step S237: Design a multi-granularity contrast loss function Take it as the overall objective function of the model, multi-granularity contrast loss function is the fine-grained contrast loss Compared with the coarse-grained loss The specific formula is as follows:

6. The image aesthetic evaluation method based on text extraction and multi-granularity alignment CLIP according to claim 5, characterized in that: The specific implementation steps of step S3 are as follows: Step S31, randomly divide the paired (image, detailed comment, aesthetic summary) data set used for training described in step S1 and step S2 to obtain a plurality of batches containing B pairs (image, detailed comment, aesthetic summary); Step S32: Use the pre-trained CLIP model as the initial model, input the images and texts in the data set in step S31 into the image encoder and the text encoder in batches, and obtain the image encoder according to the method described in step S23. And calculate the loss according to the formula Step S33, using the back propagation method to calculate the gradient of the parameters in the model according to the loss function, and using the Adam optimization method to update the model parameters; Step S34: repeat steps S31 to S33 until a preset number of training rounds is reached, thereby obtaining an optimized CLIP model suitable for aesthetic tasks.

7. The image aesthetic evaluation method based on text extraction and multi-granularity alignment CLIP according to claim 1, characterized in that: The specific implementation steps of step S4 are as follows: Step S41, design a text branch, including a text encoder with frozen parameters and five learnable prompt words as input; specifically, firstly introduce five learnable prompt words representing different aesthetic quality categories, which correspond to five quality categories of "terrible", "bad", "average", "good" and "perfect", respectively. Each prompt word consists of a number of learnable embedding vectors and words representing the corresponding quality category; specifically expressed as "[X]1[X]2[X]3…[X] M [quality category]", where each [X] i (i∈1,…M) is a learnable text token whose dimension is the same as the word embedding dimension; M represents the number of learnable text tokens. Subsequently, these prompt words are input into the text encoder to generate five corresponding feature vectors, denoted as F P =[F P1 ; F P2 ; F P3 ; F P4 ; F P5 ], Step S42, design an image branch, including a parameter-frozen image encoder and an adapter module, the adapter module includes four sub-adapters, each adapter consists of a fully connected layer that maps input features to a lower dimension, a GELU activation function, and a fully connected layer that maps low-dimensional features back to the original dimension, and the input of each adapter is I′ at four different stages in the image encoder. lcls Specifically, the image encoder consists of L layers of Transformer layers, which are first divided into four stages, each of which contains L s layer, The output of the first stage is L s The [CLS] token in the layer output is represented by As the input of the first adapter, we get the feature F A1 ; The output of the second stage is 2L s The [CLS] token in the layer output is represented by It is used as the input of the second adapter to obtain the feature F A2 ; The output of the third stage is 3L s The [CLS] token in the layer output is represented by It is used as the input of the third adapter to obtain the feature F A3 ; The output of the fourth stage is the [CLS] token in the Lth layer output, denoted as I′ Lcls , which is used as the input of the fourth adapter to obtain the feature F A4 Then, the output features of the four adapters are added and fused, and the final image features are obtained through linear mapping. The specific formula is as follows: F A4 =FC Up(GELU(FC Down(I′ Lcls )), Among them, FC Down (·) represents the fully connected layer used to map the input features to a lower dimension, GELU (·) represents the GELU activation function, and FC Up (·) represents the fully connected layer used to map the low-dimensional features back to the original dimension; E F Represents a learnable matrix used to transform the feature F I Projected to the text feature F P Same dimensions; Step S43: The text feature F obtained in step S41 P As the key Key and value Value, the image feature F obtained in step S42 is I As the query Query, cross attention calculation is performed to obtain the feature to capture the correlation between image content and different aesthetic quality categories; then F C After a feed-forward network and image features F I Fusion is performed to obtain the final output features It is then further refined through another feed-forward network to finally predict the aesthetic score distribution of the image; the specific formula is as follows: F C =CrossAttention(F I ,F p ), F O =FFN(F C ) + F I , Among them, CrossAttention(·) represents cross attention calculation, FFN(·) represents feedforward network; represents the predicted aesthetic rating distribution; Step S44, designing the loss function of the image aesthetic quality evaluation network. In order to more accurately measure the difference between the predicted distribution and the true distribution and capture the order and distance relationship between the scores, the EMD loss is used as the target loss function of the image aesthetic quality evaluation network. The specific expression formula is as follows: Where T represents the number of aesthetic score categories; represents the cumulative distribution function of the true distribution P, p i represents the probability of the i-th score in the true distribution; Represents the predicted distribution The cumulative distribution function of represents the probability of the i-th score in the prediction distribution.

8. The image aesthetic evaluation method based on text extraction and multi-granularity alignment CLIP according to claim 7, characterized in that: The specific implementation steps of step S5 are as follows: Step S51, randomly dividing the images in the paired image-comment dataset described in step S1 to obtain a plurality of batches containing B images; Step S52: Input the image into the image aesthetic quality evaluation network in step S4 to obtain the predicted aesthetic score distribution, and calculate the loss using the formula in step S42. Step S53, using the back propagation method to calculate the gradients of the parameters that are not frozen in the network according to the loss function, and using the Adam optimization method to update these parameters; Step S54: Repeat steps S51 to S53 in batches to obtain a trained image aesthetic quality evaluation network.

9. The image aesthetic evaluation method based on text extraction and multi-granularity alignment CLIP according to claim 1, characterized in that: The specific implementation steps of step S6 are as follows: Step S61: Input the image to be tested into the trained image aesthetic quality evaluation network described in step S5 to obtain the aesthetic score distribution predicted by the model. represents the probability of the i-th score in the predicted distribution. Based on the distribution of aesthetic scores predicted by the model, the aesthetic evaluation score Score is obtained by weighted summing the scores of all categories and their corresponding predicted probabilities. Suppose the score of the scoring category is where s i is the score of the i-th scoring category, the specific formula is as follows:

10. A computer-readable storage medium having stored thereon computer program instructions that can be executed by a processor, and when the processor executes the computer program instructions, the method steps according to any one of claims 1 to 9 can be implemented.

Images