A general image aesthetic evaluation method and device based on attribute-aware relationship reasoning

Abstract

The present invention discloses a general image aesthetic evaluation method and device based on attribute-aware relationship reasoning, which is used to improve the accuracy of the general image aesthetic evaluation model. The present invention first uses a convolutional neural network to construct a feature extractor to obtain a global feature map of the image; and on the basis of the global feature map, the attribute-aware feature map is generated by learning the aesthetic attributes of the image; then the attribute relationship reasoning module is constructed using a self-attention mechanism, and the relationship feature map of the aesthetic attributes is further obtained; finally, the global feature map, the aesthetic attribute features and the attribute relationship feature map are combined, and the aesthetic distribution of the image is modeled and predicted; finally, the aesthetic distribution is converted into a general aesthetic score of the image by calculation; the present invention can effectively evaluate the general aesthetic score of the image to be tested, with high accuracy, and the present invention can effectively screen out images that meet the public aesthetics, and is easy to be applied in technologies such as image retrieval and image enhancement.

Description

Technical Field

[0001] The present invention relates to computer vision and image processing technology, and in particular to a general image aesthetics evaluation method and device based on attribute perception relationship reasoning. Background Art

[0002] With the popularity of social media, people are paying more and more attention to image aesthetics that can make images look more attractive. Image aesthetics assessment aims to simulate the public's visual aesthetics to evaluate the aesthetic quality of images. This research has attracted more and more attention from researchers in the fields of computer vision and multimedia. Image aesthetics assessment technology has many potential applications, such as advertising poster design, image retrieval, image enhancement, artistic photography, and image recommendation.

[0003] Early image aesthetic assessment methods mainly evaluated the aesthetic quality of images by extracting manual features of general rules in photography technology (such as depth of field and rule of thirds). With the popularity of deep learning technology, existing methods usually use convolutional neural networks (CNNs) to map the aesthetics of images into binary classifications (high or low) or aesthetic scores. However, the above two aesthetic assessment results cannot fully describe the diversity of the public's aesthetic appreciation of images. Therefore, recent image aesthetic assessment technologies have begun to focus on the prediction of image aesthetic distribution, that is, mapping the aesthetics of images into the score distribution of public image evaluation. Existing technologies mainly use some interpretable aesthetic attributes to evaluate the general aesthetic quality of images. These works believe that the aesthetic attributes of images are closely related to aesthetic quality, and aesthetic attributes can be directly used to assist image aesthetic assessment tasks. In 2016, Kong et al. proposed an image aesthetic ranking network that can regress and predict the aesthetic score of an image by jointly learning adaptive image aesthetic attributes and content. In 2019, Pan et al. proposed a multi-task regression network that can simultaneously learn image aesthetic attributes and aesthetic scores, and introduced a discriminator to enhance the learning ability of the proposed multi-task regression network. In 2020, Li et al. proposed a multi-task learning network to construct an image aesthetic distribution prediction model, which uses the personality attributes of images to further improve the performance of the image aesthetic distribution prediction model.

[0004] Although the above methods can effectively evaluate the beauty of images to a certain extent, due to the differences in the public's preferences for image aesthetic attributes, directly using image aesthetic attributes to predict the public's aesthetic distribution of images may have insufficient representation capabilities. In order to fully utilize the influence of aesthetic attributes on image aesthetic distribution, an effective solution is to explore the implicit relationship between various aesthetic attributes in the image to infer the aesthetic distribution of the public's evaluation of the image. Summary of the invention

[0005] Purpose of the invention: In order to overcome the deficiencies in the prior art, the present invention provides a general image aesthetic evaluation method and device based on attribute-perceptual relationship reasoning. Specifically, in order to capture the implicit relationship between different aesthetic attributes in an image, the present invention utilizes a self-attention mechanism to transform aesthetic attribute perception features into attribute relationship features, and then combines the attribute features, attribute relationship features and global features to jointly model the aesthetic distribution of image evaluation, and ultimately a general image aesthetic evaluation model with high accuracy can be obtained.

[0006] Technical solution: To achieve the above purpose, the technical solution adopted by the present invention is:

[0007] A general image aesthetic assessment method based on attribute-aware relational reasoning includes the following steps:

[0008] Step 1: Use the convolutional neural network (CNN) model and 1×1 convolution operation to build an image feature extractor f θ , using the image feature extractor f θ Extract the global feature map F of the image g ; Where: θ is the parameter of the convolutional neural network model;

[0009] Step 2: Construct an aesthetic attribute prediction module and transform the global feature map F g A 1×1 convolution operation is performed to obtain a feature map F containing more than two channels. a , the feature map F a Mapped to the aesthetic attributes of the image d a ; Feature map F obtained by training the aesthetic attribute prediction module a That is the attribute-aware feature map;

[0010] Step 3: Construct an attribute relationship reasoning module based on the self-attention mechanism, and obtain the relationship feature map F of the aesthetic attribute by learning the correlation between the attribute perception feature maps ar ;

[0011] Step 4: Use global average pooling (GAP) to pool the global feature map F g Transformed into global features d g , after size transformation, the relationship feature map F ar Transformed into relational features ar , the global feature d g , aesthetic attributes a and relationship feature d ar Merge into feature d; construct an aesthetic distribution prediction model to map feature d into the aesthetic distribution s of the image;

[0012] Step 5: Input the image to be tested into the aesthetic distribution prediction model and output the aesthetic distribution prediction value of the image The average of the predicted values ​​of the image's aesthetic distribution is used as the predicted value of the image's aesthetic score

[0013] Preferably, in Step 1, the convolutional neural network model can adopt the convolutional layer part of the ResNet-152 network pre-trained on ImageNet, wherein the global feature map F g The number of channels is 2048.

[0014] Preferably, in Step 2, m general rules in photography technology are used to characterize the aesthetic attributes of the image. a , 11 general rules of photography technology can be used, mainly including interesting content, emphasized objects, good lighting, harmonious colors, vivid colors, depth of field, motion blur, thirds composition, balanced elements, repetition degree and symmetry degree; constructing an aesthetic attribute prediction module, including the following steps:

[0015] Step 21: Input the image x into the image feature extractor f θ In the above example, we get the global feature map F of the image. g =f θ (x), and the global feature map F g A 1×1 convolution operation is performed to obtain a feature map F containing m channels. a ;

[0016] Step 22: Use global average pooling and tanh activation function to transform the feature map F a Mapped to the aesthetic attributes of the image d a =tanh(GAP(F a ));

[0017] Step 23: Use the aesthetic attribute dataset to supervise the training of the aesthetic attribute prediction module. Each training image in the aesthetic attribute dataset is labeled with the aesthetic attributes of the image. The following loss function L is used a Optimize the parameters of the aesthetic attribute prediction module:

[0018]

[0019] in: and d a are the predicted value and true value of the aesthetic attribute respectively. The parameters of the aesthetic attribute prediction module are updated by the stochastic gradient descent method. The feature map F obtained by training the aesthetic attribute prediction module is a That is the attribute-aware feature map.

[0020] Preferably, in the step 3, the attribute relationship reasoning module based on the self-attention mechanism obtains the relationship feature map F of the aesthetic attribute by learning the correlation between the attribute perception feature maps ar , including the following steps:

[0021] Step 31: Transform the feature map F by size a Convert to attribute-aware features L is the feature map F a The product of the width and height;

[0022] Step 32: Transpose the attribute-aware feature D to obtain D T , and then use the self-attention mechanism to learn the attribute-aware features D and its transposition D T The pairwise relationship between them is calculated as follows:

[0023] F ar =D T D

[0024] in: It is the relationship feature map between m aesthetic attributes of the image.

[0025] Preferably, in Step 4, the global feature map F is pooled using global average pooling. g Transformed into global features d g , after size transformation, the relationship feature map F ar Transformed into relational features ar , the global feature d g , aesthetic attributes a and relationship feature d ar Merge into feature d; construct an aesthetic distribution prediction model, including the following steps:

[0026] Step 41: First, the image feature extractor f θ The obtained global feature d g , the aesthetic attribute d obtained by the aesthetic attribute prediction module a and the relational features d obtained by the attribute relation reasoning module ar Perform feature combination and obtain feature d = [d g , d a , d ar ], and then use the fully connected layer and softmax activation function to build an aesthetic distribution prediction model to map the feature d to the aesthetic distribution of the image in represents the fully connected layer, softmax(·) represents the softmax activation function, θ s represents the parameters of the aesthetic distribution prediction model;

[0027] Step 42: Use the aesthetic evaluation dataset to supervise the aesthetic distribution prediction model. Each training image in the aesthetic evaluation dataset collects a large number of aesthetic scores annotated by users, and then obtains the aesthetic distribution s of the image. The loss function L is used to optimize the parameters of the aesthetic distribution prediction model:

[0028]

[0029] in: represents the Euclidean distance, N is the number of images in the aesthetic evaluation dataset, and s are the predicted value and true value of the aesthetic distribution, respectively. The parameter θ of the aesthetic distribution prediction model is quantized by the stochastic gradient descent method. s After updating, the aesthetic distribution prediction model is finally obtained.

[0030] A device for implementing any of the above-mentioned general image aesthetic evaluation methods based on attribute-aware relationship reasoning, comprising a global feature extraction unit, an attribute-aware feature acquisition unit, an attribute-relation feature acquisition unit, an aesthetic distribution prediction unit and a general aesthetic score prediction unit:

[0031] The global feature extraction unit uses a convolutional neural network model to obtain a global feature map F of the input image. g ;

[0032] The attribute perception feature acquisition unit constructs an aesthetic attribute prediction module and generates an aesthetic attribute prediction module based on the global feature map F g Based on the feature map F a Mapped to the aesthetic attributes of the image d a , the feature map F obtained by training the aesthetic attribute prediction module a That is the attribute-aware feature map;

[0033] The attribute relationship feature acquisition unit uses the self-attention mechanism to construct an attribute relationship reasoning module, and obtains the relationship feature map F of the aesthetic attribute by learning the correlation between the attribute perception feature maps. ar ;

[0034] The aesthetic distribution prediction unit is used to transform the global feature map F g , aesthetic attributes a And the relationship feature graph F ar Combine features and model the aesthetic distribution of images;

[0035] The universal aesthetic score prediction unit inputs the image to be tested into the aesthetic distribution prediction model and outputs the aesthetic distribution prediction value of the image The average of the predicted values ​​of the image's aesthetic distribution is used as the predicted value of the image's aesthetic score

[0036] Specifically, the attribute perception feature acquisition unit includes a feature conversion module and an aesthetic attribute training module; the feature conversion module converts the global feature map F of the image g Transformed into a feature map F with the same number of channels as the number of aesthetic attributes through convolution a ; The aesthetic attribute training module converts the feature map F a Mapped to the aesthetic attributes of the image d a , and use the aesthetic attribute dataset to supervise the training of the aesthetic attribute prediction module, optimize the parameters of the aesthetic attribute prediction module, and train the feature map F a That is the attribute-aware feature map.

[0037] Specifically, the attribute relationship feature acquisition unit includes a feature conversion module and a feature relationship learning module; the feature conversion module converts the feature map F a Converted into attribute-aware features D; the feature relationship learning module uses a self-attention mechanism to learn attribute-aware features D and their transposition D T The pairwise relationship between them, and the relationship feature graph F is obtained ar .

[0038] Specifically, the aesthetic distribution prediction unit includes a feature combination module and a model training module; the feature combination module combines the global feature d g , aesthetic attributes a and relationship feature d ar Perform feature combination; the model training module uses the aesthetic evaluation data set to perform supervised training on the aesthetic distribution prediction model, optimizes the parameters of the aesthetic distribution prediction model, and finally obtains the aesthetic distribution prediction model.

[0039] Beneficial effects: The general image aesthetic evaluation method and device based on attribute-perceived relationship reasoning provided by the present invention analyze the impact of the implicit relationship between image aesthetic attributes on the public's aesthetic evaluation, and mine the correlation between multiple aesthetic attributes of the image through the self-attention mechanism, so as to more efficiently model and predict the aesthetic distribution of the image, and then obtain the general aesthetic quality score of the image; the present invention more effectively predicts the diversity of the public's aesthetic evaluation of the image by reasoning on the perceived relationship of image attributes, and obtains a more accurate general aesthetic score of the image. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 Schematic diagram of the implementation process of the method of the present invention;

[0041] Figure 2 A schematic diagram of the network structure of the method of the present invention;

[0042] Figure 3 It is a schematic diagram of the structure of the device of the present invention. DETAILED DESCRIPTION

[0043] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0044] In the description of the present invention, it is necessary to understand that the terms "center", "longitudinal", "lateral", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like indicate positions or positional relationships based on the positions or positional relationships shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance.

[0045] like Figure 1 A general image aesthetic assessment method based on attribute-aware relational reasoning is shown, which includes the following steps:

[0046] Step S01: Construct an image feature extractor f using a convolutional neural network (CNN) model and a 1×1 convolution operation θ , using the image feature extractor f θ Extract the global feature map F of the image g ; Where: θ is the parameter of the convolutional neural network model;

[0047] Step S02: Construct an aesthetic attribute prediction module and transform the global feature map F g A 1×1 convolution operation is performed to obtain a feature map F containing more than two channels. a , the feature map F a Mapped to the aesthetic attributes of the image d a ; Feature map F obtained by training the aesthetic attribute prediction module a That is the attribute-aware feature map;

[0048] Step S03: Construct an attribute relationship reasoning module based on the self-attention mechanism, and obtain the relationship feature map F of the aesthetic attribute by learning the correlation between the attribute perception feature maps ar ;

[0049] Step S04: Use global average pooling (GAP) to pool the global feature map F g Transformed into global features d g , after size transformation, the relationship feature map F ar Transformed into relational features ar , the global feature d g , aesthetic attributes a and relationship feature dar Merge into feature d; construct an aesthetic distribution prediction model to map feature d into the aesthetic distribution s of the image;

[0050] Step S05: Input the image to be tested into the aesthetic distribution prediction model and output the aesthetic distribution prediction value of the image The average of the predicted values ​​of the image's aesthetic distribution is used as the predicted value of the image's aesthetic score

[0051] Based on the above steps, it can be seen that this method mainly includes the acquisition of aesthetic attribute perception features, the construction of attribute relationship reasoning module and the construction of aesthetic distribution prediction model. Figure 2 To implement the network structure diagram of the method, the technical solution of this embodiment is further described below in conjunction with the accompanying drawings.

[0052] Aesthetic attribute perception feature acquisition: The convolutional neural network used in this embodiment comes from the pre-trained ResNet-152 model with the fully connected layer removed, and then a 1×1 convolution operation is used to obtain a global feature map of 2048 channels, and then a 1×1 convolution operation is used to obtain a feature map of 11 channels, and then a global average pooling and tanh activation function are used to obtain the predicted 11 aesthetic attributes. In order to obtain aesthetic attribute perception features by training the above network, this embodiment obtains 11 aesthetic attributes of the image from the existing image aesthetic attribute dataset AADB, which comes from the document: Kong S, Shen X, Lin Z, et al. Photo aesthetics ranking network with attributes and content adaptation [C]. Proceedings of the European Conference on Computer Vision. 2016: 662-679. The dataset collects 11 aesthetic attributes of 10,000 images annotated with each other, and each attribute is measured by a numerical scalar in the range of [-1, 1]. The larger the value, the higher the aesthetic level of the image. In this example, 9000 images in the training set are selected to train the aesthetic attribute prediction module, and the remaining 1000 images are used for model testing. The following loss function is used to optimize the model parameters:

[0053]

[0054] in: and d a are the predicted values ​​and true values ​​of the aesthetic attributes respectively. The aesthetic attribute prediction module is trained 25 times with the data in the training set. Through the above training of the aesthetic attribute prediction module, the feature map containing 11 channels is the attribute perception feature map.

[0055] Constructing the attribute relationship reasoning module: First, the 11 aesthetic attribute perception feature maps are stretched by size transformation to obtain 11-dimensional attribute perception features. In order to explore the pairwise relationship between different aesthetic attributes, this embodiment is based on the idea of ​​self-attention mechanism, and generates an attribute relationship map by learning the mutual correlation between the attribute perception features of each dimension. The generated relationship feature map is a 11×11 dimensional feature, and the value at each position is the correlation degree of the corresponding two aesthetic attributes. The attribute relationship feature is an important factor affecting the public's evaluation of image aesthetics.

[0056] Constructing the aesthetic distribution prediction model: The aesthetic distribution prediction model consists of a fully connected layer and a softmax activation function. The fully connected layer includes two fully connected layers with 1024 and 10 nodes respectively. The input of the network model is the global features after global average pooling, the aesthetic attribute features and the attribute relationship features after size transformation. The output is the aesthetic distribution of the image. In order to train the aesthetic distribution prediction model, the aesthetic distribution of the image is obtained from the existing image aesthetic evaluation dataset AVA. The dataset comes from the literature: Murray N, Marchesotti L, Perronnin F. AVA: A large-scale database for aesthetic visual analysis [C]. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2012: 2408-2415. The dataset collects more than 250,000 images with annotated aesthetic scores. Each image receives professional aesthetic ratings from 78 to 549 different photography enthusiasts, with scores ranging from 1 to 10. Then, the scores obtained for each image are used to generate probability distributions for supervised training. By selecting 230,000 images and aesthetic distributions as training sets, the model parameters can be optimized according to the following loss function:

[0057]

[0058] in: represents the Euclidean distance, N is the number of images in the aesthetic evaluation dataset, and s are the predicted value and true value of the aesthetic distribution respectively. The regression model is trained 50 times with all the data in the training set, and the final aesthetic distribution prediction model is obtained. The model can input the aesthetic distribution of the image by inputting the test image, and obtain the universal aesthetic score of the image by calculating the average value of the aesthetic distribution.

[0059] The method of the present invention aims to solve the problem that the existing methods directly use aesthetic attributes to represent the aesthetic distribution of images insufficiently. The method uses the self-attention mechanism to mine the implicit relationship between different aesthetic attributes in the image, more effectively predicts the distribution of the public's aesthetic evaluation of the image, and finally obtains a universal image aesthetic score that meets the public's aesthetic taste. In addition, the method can effectively screen out images that meet the public's aesthetic taste and is easy to apply in technologies such as image retrieval and image enhancement.

[0060] This embodiment also relates to a device for implementing the above-mentioned general image aesthetic evaluation method based on attribute perception relationship reasoning. The structural schematic diagram of the device is shown in FIG. Figure 3 shown. Figure 3 The device includes a global feature extraction unit, an attribute perception feature acquisition unit, an attribute relationship feature acquisition unit, an aesthetic distribution prediction unit and a general aesthetic score prediction unit; wherein the global feature extraction unit uses a convolutional neural network model to obtain a global feature map F of the input image. g ; Attribute perception feature acquisition unit, build aesthetic attribute prediction module, in the global feature map F g Based on the feature map F a Mapped to the aesthetic attributes of the image d a , the feature map F obtained by training the aesthetic attribute prediction module a The attribute perception feature map is the attribute relationship feature acquisition unit. The attribute relationship reasoning module is constructed using the self-attention mechanism. The relationship feature map F of the aesthetic attribute is obtained by learning the correlation between the attribute perception feature maps. ar ; Aesthetic distribution prediction unit, used to transform the global feature map F g , aesthetic attributes a And the relationship feature graph F ar Combine features and model the aesthetic distribution of the image at the same time; the general aesthetic score prediction unit inputs the image to be tested into the aesthetic distribution prediction model and outputs the aesthetic distribution prediction value of the image The average of the predicted values ​​of the image's aesthetic distribution is used as the predicted value of the image's aesthetic score

[0061] In this embodiment, the attribute perception feature acquisition unit includes a feature conversion module and an aesthetic attribute training module; the feature conversion module converts the global feature map F of the image g Transformed into a feature map F with the same number of channels as the number of aesthetic attributes through convolution a ; The aesthetic attribute training module converts the feature map F a Mapped to the aesthetic attributes of the image d a, and use the aesthetic attribute dataset to supervise the training of the aesthetic attribute prediction module, optimize the parameters of the aesthetic attribute prediction module, and train the feature map F a That is the attribute-aware feature map.

[0062] In this embodiment, the attribute relationship feature acquisition unit includes a feature conversion module and a feature relationship learning module; the feature conversion module converts the feature map F a Converted into attribute-aware features D; the feature relationship learning module uses a self-attention mechanism to learn attribute-aware features D and their transposition D T The pairwise relationship between them, and the relationship feature graph F is obtained ar .

[0063] In this embodiment, the aesthetic distribution prediction unit includes a feature combination module and a model training module; the feature combination module combines the global features d g , aesthetic attributes a and relationship feature d ar Perform feature combination; the model training module uses the aesthetic evaluation data set to perform supervised training on the aesthetic distribution prediction model, optimizes the parameters of the aesthetic distribution prediction model, and finally obtains the aesthetic distribution prediction model.

[0064] The technical effects of the present invention are described in detail below in combination with performance tests and experimental analysis.

[0065] This paper uses the image aesthetic assessment dataset AVA for experimental verification, and uses 20,000 images in the dataset as a test set. In order to prove the effect of the present invention, the model prediction is performed on the images in the test set, and the correlation analysis is performed with the aesthetic distribution of the image annotations, and compared with the following four mainstream image aesthetic distribution prediction methods:

[0066] NIMA: Talebi H, Milanfar P. NIMA: Neural image assessment[J]. IEEE Transactions on Image Processing, 2018, 27(8): 3998-4011.

[0067] AFDC: Chen Q, Zhang W, Zhou N, et al. Adaptive fractional dilated convolution network for image aesthetics assessment[C]. Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2020: 14114-14123.

[0068] PA_IAA: Li L, Zhu H, Zhao S, et al. Personality-assisted multi-task learning for generic and personalized image aesthetics assessment[J]. IEEE Transactions on Image Processing, 2020. 29: 3898-3910.

[0069] HLA-GCN: She D, Lai Y K, Yi G, et al. Hierarchical Layout-Aware Graph Convolutional Network for Unified Aesthetics Assessment[C]. Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2021: 8475-8484.

[0070] Table 1 Performance comparison of image aesthetics assessment methods

[0071] method ACC(%) SROCC EMD NIMA 81.5 0.612 0.050 AFDC 83.0 0.649 0.045 PA_IAA 83.7 0.677 0.047 HLA-GCN 84.6 0.655 0.043 The present invention 85.1 0.683 0.041

[0072] The image aesthetics evaluation performance of the method of the present invention is compared with NIMA, AFDC, PA_IAA and HLA-GCN in the AVA test set. Since the aesthetic distribution of the image can also be converted into a universal aesthetic score, and the aesthetic score can be further converted into an aesthetic binary classification, the present invention uses the Earth Mover's Distance (EMD) to measure the performance of image aesthetic distribution prediction, the Spearman Rank Order Correlation Coefficient (SROCC) to measure the performance of image aesthetic score prediction, and the Accuracy (ACC) to measure the performance of image aesthetic binary classification, wherein the smaller the EMD value, the larger the SROCC and ACC values, the better the evaluation performance of the method. Table 1 shows the performance comparison results of the method of the present invention and the four image aesthetics evaluation methods. It can be seen from the table that the performance of the present invention in the three tasks of aesthetic distribution prediction, aesthetic score evaluation and aesthetic binary classification of images in the AVA test set is better than that of the other four methods, indicating that the method of the present invention has more accurate performance in the evaluation of universal aesthetics of images.

[0073] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0074] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0075] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the above embodiments do not limit the present invention in any form, and any technical solution obtained by equivalent replacement or equivalent transformation falls within the protection scope of the present invention.

Claims

1. A general image aesthetic assessment method based on attribute-aware relational reasoning, characterized by: The steps include: Step 1: Use the convolutional neural network model and 1×1 convolution operation to build an image feature extractor f θ , using the image feature extractor f θ Extract the global feature map F of the image g ; Where: θ is the parameter of the convolutional neural network model; Step 2: Construct an aesthetic attribute prediction module and transform the global feature map F g A 1×1 convolution operation is performed to obtain a feature map F containing more than two channels. a , the feature map F a Mapped to the aesthetic attributes of the image d a ; Feature map F obtained by training the aesthetic attribute prediction module a That is the attribute-aware feature map; Step 3: Construct an attribute relationship reasoning module based on the self-attention mechanism, and obtain the relationship feature map F of the aesthetic attribute by learning the correlation between the attribute perception feature maps ar ; Step 4: Use global average pooling to transform the global feature map F g Transformed into global features d g , after size transformation, the relationship feature map F ar Transformed into relational features ar , the global feature d g , aesthetic attributes a and relationship feature d ar Merge into feature d; construct an aesthetic distribution prediction model to map feature d into the aesthetic distribution s of the image; Step 5: Input the image to be tested into the aesthetic distribution prediction model and output the aesthetic distribution prediction value of the image The average of the predicted values ​​of the image's aesthetic distribution is used as the predicted value of the image's aesthetic score 2. The general image aesthetics assessment method based on attribute-aware relationship reasoning according to claim 1, characterized in that: In Step 2, m general rules in photography technology are used to characterize the aesthetic attributes of the image. a ,Constructing the aesthetic attribute prediction module includes the following steps: Step 21: Input the image x into the image feature extractor f θ In the above example, we get the global feature map F of the image. g =f θ (x), and the global feature map F g A 1×1 convolution operation is performed to obtain a feature map F containing m channels. a ; Step 22: Use global average pooling and tanh activation function to transform the feature map F a Mapped to the aesthetic attributes of the image d a =tanh(GAP(F a )); Step 23: Use the aesthetic attribute dataset to supervise the training of the aesthetic attribute prediction module. Each training image in the aesthetic attribute dataset is labeled with the aesthetic attribute of the image. The following loss function L is used a Optimize the parameters of the aesthetic attribute prediction module: in: and d a are the predicted value and true value of the aesthetic attribute respectively. The parameters of the aesthetic attribute prediction module are updated by the stochastic gradient descent method. The feature map F obtained by training the aesthetic attribute prediction module is a That is the attribute-aware feature map.

3. The general image aesthetics assessment method based on attribute-aware relationship reasoning according to claim 1, characterized in that: In the above Step 3, the attribute relationship reasoning module based on the self-attention mechanism obtains the relationship feature map F of the aesthetic attribute by learning the correlation between the attribute perception feature maps ar , including the following steps: Step 31: Transform the feature map F by size a Convert to attribute-aware features L is the feature map F a The product of the width and height; Step 32: Transpose the attribute-aware feature D to obtain Then the attribute-aware feature D and its transposition are learned using the self-attention mechanism The pairwise relationship between them is calculated as follows: in: It is the relationship feature map between m aesthetic attributes of the image.

4. The general image aesthetics assessment method based on attribute-aware relationship reasoning according to claim 1, characterized in that: In the Step 4, the global feature map F is pooled using global average pooling. g Transformed into global features d g , after size transformation, the relationship feature map F ar Transformed into relational features ar , the global feature d g , aesthetic attributes a and relationship feature d ar Merge into feature d; construct an aesthetic distribution prediction model, including the following steps: Step 41: First, the image feature extractor f θ The obtained global feature d g , the aesthetic attribute d obtained by the aesthetic attribute prediction module a and the relational features d obtained by the attribute relation reasoning module ar Perform feature combination and obtain feature d = [d g , d a , d ar ], and then use the fully connected layer and softmax activation function to build an aesthetic distribution prediction model to map the feature d to the aesthetic distribution of the image in represents the fully connected layer, softmax(·) represents the softmax activation function, θ s represents the parameters of the aesthetic distribution prediction model; Step 42: Use the aesthetic evaluation dataset to supervise the aesthetic distribution prediction model. Each training image in the aesthetic evaluation dataset collects the aesthetic scores annotated by the user, and then obtains the aesthetic distribution s of the image. The loss function L is used to optimize the parameters of the aesthetic distribution prediction model: in: represents the Euclidean distance, N is the number of images in the aesthetic evaluation dataset, and s are the predicted value and true value of the aesthetic distribution, respectively. The parameter θ of the aesthetic distribution prediction model is quantized by the stochastic gradient descent method. s After updating, the aesthetic distribution prediction model is finally obtained.

5. A device for implementing any of the general image aesthetic evaluation methods based on attribute perception relationship reasoning as described in claims 1 to 4, characterized in that: It includes global feature extraction unit, attribute perception feature acquisition unit, attribute relationship feature acquisition unit, aesthetic distribution prediction unit and general aesthetic score prediction unit: The global feature extraction unit uses a convolutional neural network model to obtain a global feature map F of the input image. g ; The attribute perception feature acquisition unit constructs an aesthetic attribute prediction module and generates an aesthetic attribute prediction module based on the global feature map F g Based on the feature map F a Mapped to the aesthetic attributes of the image d a , the feature map F obtained by training the aesthetic attribute prediction module a That is the attribute-aware feature map; The attribute relationship feature acquisition unit uses the self-attention mechanism to construct an attribute relationship reasoning module, and obtains the relationship feature map F of the aesthetic attribute by learning the correlation between the attribute perception feature maps. ar ; The aesthetic distribution prediction unit is used to transform the global feature map F g , aesthetic attributes a And the relationship feature graph F ar Combine features and model the aesthetic distribution of images; The universal aesthetic score prediction unit inputs the image to be tested into the aesthetic distribution prediction model and outputs the aesthetic distribution prediction value of the image The average of the predicted values ​​of the image's aesthetic distribution is used as the predicted value of the image's aesthetic score 6. The device according to claim 5, characterized in that: The attribute perception feature acquisition unit includes a feature conversion module and an aesthetic attribute training module; the feature conversion module converts the global feature map F of the image g Transformed into a feature map F with the same number of channels as the number of aesthetic attributes through convolution a ; The aesthetic attribute training module converts the feature map F a Mapped to the aesthetic attributes of the image d a , and use the aesthetic attribute dataset to supervise the training of the aesthetic attribute prediction module, optimize the parameters of the aesthetic attribute prediction module, and train the feature map F a That is the attribute-aware feature map.

7. The device according to claim 5, characterized in that: The attribute relationship feature acquisition unit includes a feature conversion module and a feature relationship learning module; the feature conversion module converts the feature map F a Converted into attribute-aware feature D; the feature relationship learning module uses the self-attention mechanism to learn the attribute-aware feature D and its transposition The pairwise relationship between them, and the relationship feature graph F is obtained ar .

8. The device according to claim 5, characterized in that: The aesthetic distribution prediction unit includes a feature combination module and a model training module; the feature combination module combines the global features d g , aesthetic attributes a and relationship feature d ar Perform feature combination; the model training module uses the aesthetic evaluation data set to perform supervised training on the aesthetic distribution prediction model, optimizes the parameters of the aesthetic distribution prediction model, and finally obtains the aesthetic distribution prediction model.

Images