An image aesthetic evaluation method fusing multi-modal aesthetics and attribute features

Abstract

The application provides an image aesthetic evaluation method fusing multi-modal aesthetic and attribute features. The method comprises the following steps: constructing image-aesthetic and image-attribute text pair data; constructing a multi-modal pre-training model combining attribute and aesthetic description; extracting original image features and conditional features by using a frozen image encoder and a conditional encoder; generating aesthetic or attribute text matching the conditional features by using a text decoder; minimizing the difference between the generated text and target text to obtain a pre-training model; obtaining and processing image and aesthetic score data, and constructing an aesthetic score model fusing multi-modal aesthetic and attribute features; extracting and fusing attribute and aesthetic features of the image to obtain attribute-aware aesthetic features; mapping the features through a linear layer to obtain an aesthetic score; optimizing model parameters by using a mean square error function to obtain a final aesthetic score model; and the application can not only obtain accurate aesthetic scores, but also only needs a single image in the reasoning stage.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of multimodal pre-training and image aesthetic evaluation, and particularly relates to an image aesthetic evaluation method fusing multimodal aesthetic and attribute features. BACKGROUND

[0002] As a direct social medium, images can more vividly and comprehensively convey information than text. However, different individuals have great subjective differences in the aesthetic perception of images. If automatic image aesthetic evaluation can be realized and beautiful images can be presented to users, the aesthetic experience of users can be greatly improved.

[0003] Deep learning has been gradually applied to the field of image aesthetic evaluation. However, existing methods only use a single aesthetic score to perceive image aesthetics, and thus it is difficult to understand complex aesthetic elements such as composition, color, and light. In addition, existing methods use a single evaluation standard, ignoring the fact that different attributes of images should use different aesthetic evaluation standards. For example, the aesthetic evaluation standard of a landscape photo is usually different from that of a portrait. Therefore, there is an urgent need to establish a model that has a relatively comprehensive perception of aesthetic elements; at the same time, attributes can also be introduced in the process of aesthetic evaluation, so that the model can select different aesthetic features according to different image attributes, thereby better understanding image aesthetics.

[0004] The present application provides an image aesthetic evaluation method fusing multimodal aesthetic and attribute features. First, aesthetic comments and attribute categories are introduced into the multimodal pre-training stage, so that the pre-training model can perceive the attributes of images while realizing aesthetic semantic understanding. In the downstream task of aesthetic evaluation, the influence of attribute categories on aesthetic evaluation is introduced by fusing aesthetic features and attribute features, greatly improving the rationality and accuracy of aesthetic evaluation. SUMMARY

[0005] The present application provides an image aesthetic evaluation method fusing multimodal aesthetic and attribute features, characterized in that the method comprises:

[0006] Step 1: constructing image-aesthetic text pair and image-attribute text pair data;

[0007] Step 2: constructing a multimodal pre-training model combining attribute and aesthetic description;

[0008] Step 3: extracting original image features using a frozen image encoder, and extracting conditional features using the interaction between the conditional encoder and the original image features under the guidance of conditional query;

[0009] Step 4, under the guidance of the condition characteristics, the text decoder generates aesthetic or attribute text matching the characteristics extracted by the condition encoder, optimizes the model parameters using the cross-entropy loss function, minimizes the difference between the generated text and the target text, and finally obtains a pre-trained model with both aesthetic perception and attribute perception;

[0010] Step 5, obtaining and processing images and corresponding aesthetic score data;

[0011] Step 6, constructing an aesthetic score model that fuses multi-modal aesthetic and attribute features;

[0012] Step 7, extracting attribute and aesthetic features of the image;

[0013] Step 8, fusing attribute features and aesthetic features to obtain attribute-aware aesthetic features;

[0014] Step 9, mapping the attribute-aware aesthetic features to one dimension using a linear layer to obtain an aesthetic score, and optimizing the model parameters using the mean square error loss function to finally obtain an aesthetic score model;

[0015] Specifically, in step 1, the image and its corresponding aesthetic description text information are obtained and organized in the form of image-aesthetic text pairs, and the classification dataset of the image and its corresponding attributes is obtained, the classification categories are converted into text information corresponding to the categories to construct the image-attribute text pair dataset, and the attribute data can be selected from scene, style, etc. related to aesthetic factors;

[0016] Specifically, in step 2, the model includes a frozen image encoder, a condition encoder, a text decoder, and a condition query module, wherein the condition query includes an attribute condition query and an aesthetic condition query, and the size of each condition query is 1×N dim , where N dim is the dimension information, the frozen image encoder selects EVA-CLIP, the condition encoder and the text decoder both adopt the Bert structure, which includes multiple layers of Transformer, the input in the condition encoder is set to a learnable query for extracting different aesthetic features, and the size is M×N dim , where M is the number of learnable queries;

[0017] Specifically, in step 3, first, after obtaining the image I in step 1, the frozen image encoder F is used to extract the features of the image I to obtain the original image features f, the calculation formula is f=F(I), then the corresponding condition query is selected according to the text information matched by the current image, wherein the aesthetic text selects the aesthetic condition query s c , the attribute text selects the attribute condition query t c , and finally f and the condition query are input into the condition encoder F cThe conditional features include aesthetic conditional features and attribute conditional features, and the attribute conditional features t f The calculation formula of the aesthetic conditional features s f c (f, t c ) and the calculation formula of the aesthetic conditional features s f f c (f, s c ) are as follows:

[0018] Specifically, in step 4, the text decoder interacts with the features extracted by the conditional encoder through a self-attention layer based on the conditional query to obtain text features that perceive image information, including attribute text features and aesthetic text features, wherein the attribute text features are obtained based on the attribute conditional query, and the aesthetic text features are obtained based on the aesthetic conditional query, and then the aesthetic text or attribute text corresponding to the text features is generated according to the text features, finally the difference between the generated text and the actual text is calculated using a cross-entropy loss function, and the difference is used as a loss to optimize the model parameters, and finally a pre-training model with aesthetic perception and attribute perception is obtained.

[0019] Specifically, in step 5, the obtained image is scaled to 224x224 pixels, and the aesthetic score data is processed by taking the mean value, and the formula is as follows: Where n represents the number of people participating in the image aesthetic scoring, and S i represents the aesthetic score of the image by the i-th person.

[0020] Specifically, in step 6, the aesthetic scoring model includes a frozen image encoder, an aesthetic feature extraction module, an attribute feature extraction module, and an attribute-perceived aesthetic feature extraction module, wherein the frozen image encoder is consistent with the frozen image encoder in step 2, the aesthetic feature extraction module is set as a trainable module, and the structure and parameters are initialized as the aesthetic conditional query and the aesthetic conditional encoder obtained in the pre-training stage, the attribute feature extraction module is set as a frozen module, and the parameters are not updated during the training process, and the structure and parameters are initialized as the attribute conditional query and the attribute conditional encoder obtained in the pre-training stage, and the attribute-perceived aesthetic feature extraction module includes a global attribute extraction module CA t , a global attribute query G, and an aesthetic attribute interaction module CA st , wherein CA t and CA st are both cross-attention layers, and G is initialized as an embedding of 1*N dim , wherein N dim is the dimension information.

[0021] ​​​Specifically, in step 7, first, the original image features are extracted by the frozen image encoder constructed in step 6, and second, the attribute condition query and the original image features are input into the frozen attribute condition encoder by the attribute feature extraction module constructed in step 6, and the attribute features t fn are extracted through interaction. fn Meanwhile, the aesthetic condition query and the original image features are input into the aesthetic condition encoder by the aesthetic feature extraction module constructed in step 6, and the aesthetic features s fn are extracted through interaction.

[0022] Specifically, in step 8, the attribute-aware aesthetic feature extraction module constructed in step 6 is used to fuse the attribute features t fn and the aesthetic features s fn extracted in step 7. fn In the fusion process, first, the global attribute query G and t t are input into the global attribute extraction module CA g constructed in step 6, and then the global attribute features t g are obtained, and the calculation formula is t t = CA fn (G, t fn , t g ), then t fn and s st are input into the aesthetic attribute interaction module CA g constructed in step 6, and then the attribute-aware aesthetic features s are obtained, and the calculation formula is s

[0023] Specifically, in step 9, first, the attribute-aware aesthetic features s obtained in step 8 are input into the linear layer Lin2ar to obtain the predicted aesthetic score o, and the calculation formula is o = Lin2ar(s The difference between the predicted aesthetic score o and the actual aesthetic score data o g obtained in step 5 is calculated by using the mean square error calculation model, and the difference is used as a loss to optimize the aesthetic model, and the calculation formula of the loss L is L = (o - o ), where K is the number of samples in each iteration, and finally the aesthetic score model is obtained.

[0024] The present application aims at the problem that the existing image aesthetic evaluation method only relies on a single aesthetic score to learn image aesthetics, leading to insufficient perception of different attribute images, difficulty in understanding complex aesthetic concepts, and difficulty in applying multi-modal aesthetic evaluation in practical scenarios due to the lack of aesthetic description. An image aesthetic evaluation method fusing multi-modal aesthetics and attribute features is proposed. By comprehensively understanding different aesthetic concepts and attribute features in the pre-training stage, and fusing aesthetic features and attribute features in the downstream task, the effect of automatically obtaining accurate aesthetic evaluation by inputting only a single image is realized, greatly improving the practicality. BRIEF DESCRIPTION OF DRAWINGS

[0025] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, below will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description only constitute a part of the embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of the provided drawings.

[0026] Figure 1 The step flow chart of the image aesthetic evaluation method fusing multi-modal aesthetics and attribute features; DETAILED DESCRIPTION

[0027] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments only constitute a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0028] Deep learning is gradually applied to the field of image aesthetic evaluation. However, the existing work only uses a single aesthetic score to perceive image aesthetics, leading to difficulty in understanding complex aesthetic elements such as composition, color, light, etc. In addition, the existing method uses a single evaluation standard, ignoring that different attribute images should use different aesthetic evaluation standards. Furthermore, in the practical application stage, the existing method often needs to input images and corresponding aesthetic descriptions simultaneously, leading to difficulty in application. The present application proposes an image aesthetic evaluation method fusing multi-modal aesthetics and attribute features. First, aesthetic comments and attribute categories are introduced into the multi-modal pre-training stage, so that the pre-training model can perceive the attributes of the image when understanding aesthetic semantics. In the downstream task of aesthetic evaluation, by fusing aesthetic features and attribute features, the influence of attribute categories on aesthetic evaluation is introduced, improving the rationality and comprehensiveness of aesthetic evaluation. Moreover, only a single image needs to be input to realize accurate aesthetic evaluation, improving the practicality of aesthetic evaluation. The specific steps are as follows:

[0029] S101: Constructing image-aesthetic text pair and image-attribute text pair data;

[0030] Specifically, the image and its corresponding aesthetic description text information are obtained and organized in the form of image-aesthetic text pair, and the image and its corresponding attribute classification data set are obtained, and the classification category is converted into text information corresponding to the category to construct the image-attribute text pair data set. The attribute data can select scene, style, etc. related to aesthetic factors;

[0031] S102: Building a multi-modal pre-training model combining attribute and aesthetic description;

[0032] Specifically, the model includes a frozen image encoder, a conditional encoder, a text decoder, and a conditional query module, wherein the conditional query includes an attribute conditional query and an aesthetic conditional query, and the feature size of each conditional query is 1xN dim , wherein N dim is the dimension information, the frozen image encoder selects EVA-CLIP, the conditional encoder and the text decoder both adopt the Bert structure, which includes multiple layers of Transformer, the input in the conditional encoder sets a learnable query to extract different aesthetic features, and the feature size is MxN dim , wherein M is the number of learnable queries;

[0033] S103: Extracting original image features using the frozen image encoder, and extracting conditional features using the interaction between the conditional encoder and the original image features under the guidance of the conditional query;

[0034] Specifically, after obtaining the image I in step 1, first scale it to 256x256 pixels and randomly crop it to 224x224 pixels, then convert it to RGB format I RGB , and then perform normalization processing, the formula is , wherein mean is the mean of the image, and std is the standard deviation of the image; secondly, use the frozen image encoder F to extract the features of I D to obtain the original image features f, the calculation formula is f=F(I D ); thirdly, select the corresponding conditional query according to the text information matched by the current image, wherein the aesthetic text selects the aesthetic conditional query s c , and the attribute text selects the attribute conditional query t c ; finally, f and the conditional query are input into the conditional encoder F c to obtain the conditional features, which include aesthetic conditional features and attribute conditional features, and the calculation formula of the attribute conditional features t f is t f =F c (f,t c), aesthetic condition feature s f The calculation formula of s f = F c (s c , f, s f and t f The feature size of s dim ;

[0035] S104: Under the guidance of the condition feature, the text decoder generates aesthetic or attribute text matching the features extracted by the condition encoder, optimizes the model parameters using the cross-entropy loss function, minimizes the difference between the generated text and the target text, and finally obtains a pre-training model with aesthetic perception and attribute perception;

[0036] Specifically, the text decoder interacts with the features extracted by the condition encoder through a self-attention layer based on the condition query, obtains text features that perceive image information, and the text features include attribute text features and aesthetic text features; wherein the attribute text features are obtained based on the attribute condition query, and the aesthetic text features are obtained based on the aesthetic condition query, and then the text features are decoded according to the text features, and the decoding process is word-by-word decoding, specifically: map the text features to the dimension C N of the vocabulary, the size of C N is consistent with the number of word units contained in the vocabulary, then obtain the final probability distribution S N , the size is 1xC N , then calculate the position of the word unit with the maximum probability in the vocabulary through the argmax function, take the word unit in the position as the next generated aesthetic text or attribute text, and finally calculate the difference between the generated text and the actual text using the cross-entropy loss function, and use it as the loss to optimize the model parameters, and finally obtain a pre-training model with aesthetic perception and attribute perception;

[0037] S105: Obtain and process images and corresponding aesthetic score data;

[0038] Specifically, the obtained image is scaled to 224x224 pixels, and the aesthetic score data is processed by averaging, the formula is Where n represents the number of people participating in the image aesthetic score, S i represents the aesthetic score of the image by the i-th person;

[0039] S106: Construct an aesthetic score model that integrates multi-modal aesthetic and attribute features;

[0040] Specifically, the aesthetic score model comprises a frozen image encoder, an aesthetic feature extraction module, an attribute feature extraction module, and an attribute-aware aesthetic feature extraction module, wherein the frozen image encoder is consistent with the frozen image encoder in S102, the aesthetic feature extraction module is set as a trainable module, the structure and parameters are initialized as the aesthetic condition query and the aesthetic condition encoder obtained in the pre-training stage, the attribute feature extraction module is set as a frozen module, the parameters are not updated in the training process, the structure and parameters are initialized as the attribute condition query and the attribute condition encoder obtained in the pre-training stage, and the attribute-aware aesthetic feature extraction module comprises a global attribute extraction module CA t , a global attribute query G, and an aesthetic attribute interaction module CA st , wherein CA t and CA st are cross-attention layers in structure, G is initialized as an embedding of 1*N dim , wherein N dim is dimension information.

[0041] S107: Extracting attribute and aesthetic features of the image;

[0042] Specifically, first, the frozen image encoder constructed by S106 is used to extract original image features, second, the attribute condition query and the original image features are input into the frozen attribute condition encoder by using the attribute feature extraction module constructed by S106, and attribute features t fn are extracted through interaction, and meanwhile, the aesthetic condition query and the original image features are input into the aesthetic condition encoder by using the aesthetic feature extraction module constructed by S106, and aesthetic features s fn are extracted through interaction.

[0043] S108: Fusing attribute features and aesthetic features to obtain attribute-aware aesthetic features;

[0044] Specifically, the attribute-aware aesthetic feature extraction module constructed by S106 fuses the attribute features t fn and the aesthetic features s fn extracted by S107, wherein in the fusion process, first, the global attribute query G constructed by S106 and t fn are input into the global attribute extraction module CA t constructed by S106, and then global attribute features t g are obtained, and the calculation formula is t g =CA t (G,t fn ,t fn ), then t g and s fn are input into the aesthetic attribute interaction module CA stIn this process, the aesthetic characteristics of attribute perception are obtained. The calculation formula is

[0045] S109: The linear layer is used to map the attribute-aware aesthetic features to one dimension to obtain an aesthetic score. The mean squared error loss function is then used to optimize the model parameters, and finally the aesthetic scoring model is obtained.

[0046] Specifically, firstly, based on the attribute-perceived aesthetic features obtained in S108... The predicted aesthetic score o is obtained through a linear layer, calculated using the following formula: Then, the aesthetic score predicted by the model is calculated using the mean squared error, and compared with the actual aesthetic score data obtained from S105. g The difference is used as the loss optimization aesthetic model, and the formula for calculating the loss L is: Where K is the number of samples in each iteration, and the final aesthetic scoring model is obtained;

[0047] The image aesthetic evaluation method proposed in this invention integrates multimodal aesthetics and attribute features. At the application level, it can combine image attributes with semantic understanding of aesthetics to obtain an image aesthetic score more comprehensively and accurately, outperforming existing methods. Furthermore, in the inference stage, only image information is needed to automatically evaluate the image aesthetic score, making it highly practical.

Claims

1. An image aesthetic evaluation method fusing multi-modal aesthetics and attribute features, characterized in that, The method comprises the following steps: Step 1, constructing image-aesthetic text pair and image-attribute text pair data; Step 2, constructing a multimodal pre-training model of joint attribute and aesthetic description; Step 3, extract the original image features by using the frozen image encoder, and extract the condition features by using the interaction between the condition encoder and the original image features under the guidance of the conditional query; Specifically: first, after obtaining the image I through step 1, the original image features f are extracted by using the frozen image encoder F to perform feature extraction on the image I, and the calculation formula is f=F(I), then the corresponding condition query is selected according to the text information matched by the current image, wherein the aesthetic text selects the aesthetic condition query s c , the attribute text selects the attribute condition query t c , finally, f and the condition query are input into the condition encoder F c to obtain the condition features, wherein the condition features include aesthetic condition features and attribute condition features, the calculation formula of the attribute condition feature t f is t f =F c (f,t c ), and the calculation formula of the aesthetic condition feature s f is s f =F c (f,s c ). Step 4, under the guidance of conditional characteristics, the text decoder generates aesthetic or attribute text matching the features extracted by the conditional encoder, and the model parameters are optimized by using a cross-entropy loss function to minimize the difference between the generated text and the target text, and finally a pre-training model with aesthetic perception and attribute perception is obtained; Specifically, the text decoder interacts with the features extracted by the conditional encoder through a self-attention layer based on a conditional query, obtains text features that perceive image information, and the text features include attribute text features and aesthetic text features, wherein the attribute text features are obtained based on attribute condition queries, and the aesthetic text features are obtained based on aesthetic condition queries, and then the corresponding aesthetic text or attribute text is generated according to the text features, and finally the difference between the generated text and the actual text is calculated by using a cross-entropy loss function, the model parameters are optimized, and finally a pre-training model with aesthetic perception and attribute perception is obtained; Step 5, obtaining and processing image and corresponding aesthetic score data; Step 6, constructing an aesthetic score model that fuses multimodal aesthetic and attribute features; Step 7, extracting the attribute and aesthetic features of the image; Step 8: Fuse attribute features and aesthetic features to obtain attribute-aware aesthetic features; specifically: use the constructed attribute-aware aesthetic feature extraction module to fuse the extracted attribute features t fn and aesthetic features fn During the fusion process, the constructed global attribute queries G and t are first performed. fn Input to the global attribute extraction module CA built t In this process, the global attribute feature t is obtained. g The calculation formula is t g =CA t (G,t fn ,t fn ), then t g and s fn Input into the aesthetic attribute interaction module CA st In this process, the aesthetic characteristics of attribute perception are obtained. The calculation formula is Step 9, mapping the attribute-perceived aesthetic features to one dimension by using a linear layer to obtain an aesthetic score, and then optimizing the model parameters by using a mean square error loss function to finally obtain an aesthetic score model. 2.The image aesthetic evaluation method of fusing multi-modal aesthetic and attribute features according to claim 1, wherein, In step 1, the image-aesthetic text pair and image-attribute text pair data are constructed, specifically: the image and its corresponding aesthetic description text information are obtained and organized in the form of image-aesthetic text pair, and at the same time, the image and its corresponding attribute classification data set are obtained, the classification categories are converted into text information corresponding to the categories to construct an image-attribute text pair data set. 3.The image aesthetic evaluation method of fusing multi-modal aesthetic and attribute features according to claim 1, wherein, In step 2, the multimodal pre-training model of joint attribute and aesthetic description is constructed, specifically: the model comprises a frozen image encoder, a conditional encoder, a text decoder, and a conditional query module, wherein the conditional query comprises an attribute condition query and an aesthetic condition query, and the frozen image encoder is EVA-CLIP. 4.The image aesthetic evaluation method of fusing multi-modal aesthetic and attribute features according to claim 1, wherein, In step 5, the image and the corresponding aesthetic score data are acquired and processed, specifically: the acquired image is scaled to 224x224 pixels, and the aesthetic score data is processed by averaging, with the formula being where n represents the number of people participating in the image aesthetic score, S i represents the aesthetic score of the image by the i-th person. 5.The image aesthetic evaluation method of fusing multi-modal aesthetic and attribute features according to claim 1, wherein, In step 6, the aesthetic score model of the fusion of multi-modal aesthetics and attribute features is constructed, specifically: the aesthetic score model includes a frozen image encoder, an aesthetic feature extraction module, an attribute feature extraction module, and an attribute-aware aesthetic feature extraction module, wherein the frozen image encoder is consistent with the frozen image encoder in step 2, the aesthetic feature extraction module is set as a trainable module, the structure and parameters are initialized as the aesthetic condition query and the aesthetic condition encoder obtained in the pre-training stage, the attribute feature extraction module is set as a frozen module, the parameters are not updated in the training process, the structure and parameters are initialized as the attribute condition query and the attribute condition encoder obtained in the pre-training stage, and the attribute-aware aesthetic feature extraction module includes a global attribute extraction module CA t , a global attribute query G, and an aesthetic attribute interaction module CA st , wherein CA t and CA st are cross-attention layers in structure, G is initialized as an embedding of 1xN dim , wherein N dim is the dimension information. 6.The image aesthetic evaluation method of fusing multi-modal aesthetic and attribute features according to claim 1, wherein, In step 7, the attributes and aesthetic features of the extracted image are extracted, specifically: first, the original image features are extracted using the frozen image encoder constructed in step 6, and second, the attribute condition query and the original image features are input into the frozen attribute condition encoder using the attribute feature extraction module constructed in step 6, and the attribute features t are extracted through interaction fn At the same time, the aesthetic condition query and the original image features are input into the aesthetic condition encoder using the aesthetic feature extraction module constructed in step 6, and the aesthetic features s are extracted through interaction fn .

7. The image aesthetic evaluation method fusing multi-modal aesthetic and attribute features as claimed in claim 1, wherein, In step 9, the aesthetic features perceived by the attributes are mapped to one dimension by the linear layer, and then the aesthetic score is obtained, and then the model parameters are optimized by the mean square error loss function, and finally the aesthetic score model is obtained, specifically: first, according to the aesthetic features perceived by the attributes obtained in step 8 The predicted aesthetic score o is obtained by the linear layer Linear, and the calculation formula is Then the difference between the predicted aesthetic score o calculated by the mean square error and the actual aesthetic score data o obtained in step 5 is used as the loss to optimize the aesthetic model, and the calculation formula of the loss L is g The calculation formula of the loss L is Where K is the number of samples per iteration, and the aesthetic score model is finally obtained.

Images