Image aesthetics quality evaluation method and system based on region-level fine-grained understanding

Abstract

The invention provides an image aesthetic quality assessment method and system based on region-level fine-grained understanding, and the method comprises the following steps: S1, carrying out the understanding through combining with the aesthetic features of multiple regions based on an aesthetic quality scoring device of multi-region aesthetic quality perception, and generating region aesthetic quality scores for images of different scale regions; and S2, based on the regional aesthetics quality score, generating optimal and worst aesthetics regions in the picture by using an aesthetics region regression device, thereby realizing comprehensive evaluation of the aesthetics quality of the whole picture, and realizing positioning of the regions with the optimal and worst aesthetics effects in the picture. According to the image aesthetic quality evaluation training method based on the region-level fine grit, the problems that a previous aesthetic evaluation method lacks region-level fine grit understanding, the aesthetic defect positioning capability is lacked and the generalization is poor are solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision and image processing, and particularly relates to an image aesthetic quality evaluation method and system based on region-level fine-grained understanding. BACKGROUND

[0002] People are naturally attracted to images that are aesthetically pleasing and pleasing to the eye, so image aesthetics are crucial in the fields of photography, advertising, e-commerce, and social media. Image aesthetic quality evaluation [1][6]

[10]

[11] aims to achieve a deep learning model that can accurately predict human preferences for image aesthetics. However, current image aesthetic quality evaluation methods lack semantic-based region-level fine-grained understanding and aesthetic defect localization capabilities.

[0003] Existing image aesthetic quality evaluation methods can be roughly divided into two categories: (1) general image aesthetic quality evaluation [2][4][5][7][8]

[13] uses multi-person labeled aesthetic picture quality scores for model training to reflect human mass aesthetic preferences as much as possible; (2) personalized image aesthetic quality evaluation [3]

[12] uses personal labeled aesthetic picture quality scores for model training to address the problem of varying aesthetic preferences. However, personalized image aesthetic quality evaluation often requires further fine-tuning training on specific data based on general aesthetic quality evaluation models, so general image aesthetic quality evaluation is more fundamental and important. The present application focuses on semantic-based region-level fine-grained understanding for general image aesthetic quality evaluation.

[0004] Existing general image aesthetic quality evaluation methods all focus on network design to preserve the high resolution and original aspect ratio of the input image, reducing the loss of spatial information related to aesthetic features. CNN-based methods [2][3][4][7] aim to reduce the negative impact of cropping and resizing; while Transformer-based methods [9]

[10]

[11]

[12]

[13] treat the input image as a visual token and support variable-length sequence processing, thereby maintaining the original resolution and aspect ratio of the image. However, these methods lack fine-grained perception of different regions in the picture and aesthetic defect localization capabilities.

[0005] REFERENCES

[0006] [1] Naila Murray et al. AVA: A large-scale database for aesthetic visual analysis. IEEE Conference on Computer Vision and Pattern Recognition, 2012.

[0007] [2] Wang Wenguan et al. Deep Cropping via Attention Box Prediction and Aesthetics Assessment. IEEE International Conference on Computer Vision, 2018.

[0008] [3] Ren Jian et al. Personalized Image Aesthetics. IEEE International Conference on Computer Vision, 2017.

[0009] [4] Wei Zijun et al. Good View Hunting: Learning Photo Composition from Dense View Pairs, IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2018.

[0010] [5] Hosu Vlad et al. Effective Aesthetics Prediction With Multi-Level Spatially Pooled Features. IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2019.

[0011] [6] Yi Ran et al. Towards Artistic Image Aesthetics Assessment: a Large-scale Dataset and a New Method. arXiv preprint arXiv:2303.15166, 2023.

[0012] [7] Chen Qiuyu et al. Adaptive Fractional Dilated Convolution Network for Image Aesthetics Assessment. IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2020.

[0013] [8] She Dongyu et al. Hierarchical Layout-Aware Graph Convolutional Network for Unified Aesthetics Assessment. IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2021.

[0014] [9] Tu Zhengzhong et al. MaxViT: Multi-Axis Vision Transformer. arXiv preprint arXiv:2204.01697, 2022.

[0015]

[10] He Shuai et al. Thinking Image Color Aesthetics Assessment: Models, Datasets and Benchmarks. IEEE / CVF International Conference on Computer Vision, 2023

[0016]

[11] Wu Haoning et al. Exploring Video Quality Assessment on User-Generated Contents from Aesthetic and Technical Perspectives. IEEE / CVF International Conference on Computer Vision, 2023.

[0017]

[12] Yun Jooyeol et al. Scaling Up Personalized Image Aesthetic Assessment via Task Vector Customization. arXiv preprint arXiv:2407.07176, 2024.

[0018]

[13] Behrad Fatemeh et al. Charm: The Missing Piece in ViT fine-tuning for Image Aesthetic Assessment. arXiv preprint arXiv:2504.02522, 2025. SUMMARY

[0019] The present application is to solve the above problems, and aims to provide an image aesthetic quality evaluation method and system based on regional level fine-grained understanding, which solves the problem that the existing method does not fully utilize the aesthetic quality features of different regions of the image, and lacks aesthetic defect positioning capability.

[0020] The present application provides an image aesthetic quality evaluation method based on regional level fine-grained understanding, which has the following characteristics, including the following steps: S1, based on the aesthetic quality score of multi-region aesthetic quality perception, combining the aesthetic features of multi-region to understand, generating regional aesthetic quality score of image of different scale regions; S2, based on regional aesthetic quality score, using aesthetic region regressor to generate the best and worst aesthetic region in the picture, realizing comprehensive evaluation of whole picture aesthetic quality, realizing positioning of the best and worst aesthetic region in the picture.

[0021] In the image aesthetic quality evaluation method based on regional level fine-grained understanding provided by the present application, it can also have the following characteristics: wherein, step S1 includes the following substeps:

[0022] S1-1, using a pre-trained backbone network MaxViT to extract the whole image feature map F of the image global ;

[0023] S1-2, introducing a regional feature extraction module, which selects N key regions on F global Applies regional adaptive pooling to extract the aesthetic quality feature vector F i of each region R i , using a regional attention fusion module, which calculates the contribution weight a i of each regional feature F i to the final aesthetic score and weighted fusion:

[0024]

[0025] W a is a learnable weight vector;

[0026] S1-3, outputting the whole image score through the regression layer:

[0027] ​

[0028] W r ,b r is the regression layer parameter, and sigma (·) is a Sigmoid function.

[0029] In the image aesthetic quality evaluation method based on region-level fine-grained understanding provided by the application, the step S2 can further include the following sub-steps:

[0030] A defect heat map A = H(x, y) is generated, where H(x, y) is in 0, 1, and represents the defect probability at the coordinate (x, y), and a higher value represents a worse aesthetic quality,

[0031] Worst aesthetic region calculation:

[0032] A b = (x, y) | H(x, y) > p,

[0033] p is a threshold value, and a value exceeding the threshold value indicates a negative impact on the image aesthetic quality, and the best cropping region calculation:

[0034]

[0035] where C is a candidate cropping region set meeting a preset aspect ratio, S(C) is an aesthetic score of the region C, and lambda is a balance weight.

[0036] The application further discloses an image aesthetic quality evaluation system based on region-level fine-grained understanding, which has the following features: a region aesthetic quality scoring module, an aesthetic quality scorer based on multi-region aesthetic quality perception, understanding of aesthetic features of multiple regions, and generation of region aesthetic quality scores of images of different scales; and an aesthetic region positioning module, which generates the best and worst aesthetic regions in a picture based on the region aesthetic quality scores and using an aesthetic region regressor, realizes comprehensive evaluation of the overall picture aesthetic quality, and realizes positioning of the best and worst aesthetic regions in the picture.

[0037] Effects of the application

[0038] The image aesthetic quality evaluation method and system based on region-level fine-grained understanding according to the application solve the problems of lack of region-level fine-grained understanding, lack of aesthetic defect positioning capability, and poor generalization of previous aesthetic evaluation methods.

[0039] The application can quantify the contribution of different regions to the overall score through multi-region analysis, enhance the explainability of aesthetic scores, generate the best and worst aesthetic regions in a picture using an aesthetic region regressor, and further improve the robustness of overall picture aesthetic quality evaluation.

[0040] The image aesthetic quality evaluation method based on region-level fine-grained understanding of the embodiment of the present application can not only output the overall aesthetic score of the image, but also accurately locate the specific area that affects the aesthetic quality score based on the image trimmer for aesthetic defect positioning, can quickly give the user the defect area to obtain the optimization direction directly, makes the aesthetic quality evaluation have explainability, greatly reduces the improvement cost of image processing, photography post-processing and the like, and therefore has strong application value. BRIEF DESCRIPTION OF DRAWINGS

[0041] Figure 1 is a flowchart of an image aesthetic quality evaluation method based on region-level fine-grained understanding in an embodiment of the present application;

[0042] Figure 2 is a schematic diagram of an image aesthetic quality evaluation system based on region-level fine-grained understanding in an embodiment of the present application. DETAILED DESCRIPTION

[0043] In order to make the technical means, creative features, purposes and effects achieved by the present application easy to understand, the following embodiments will make a specific description of the image aesthetic quality evaluation method and system based on region-level fine-grained understanding of the present application in combination with the drawings.

[0044] The method of the present patent aims to understand the aesthetic score of the whole image through the aesthetic quality features of multiple regions, rather than a single feature such as spatial composition, color, etc. that affects the aesthetic quality.

[0045] Figure 1 is a flowchart of an image aesthetic quality evaluation method based on region-level fine-grained understanding in an embodiment of the present application.

[0046] As shown in Figure 1 , the image aesthetic quality evaluation method based on region-level fine-grained understanding in the embodiment includes the following steps:

[0047] S1, an aesthetic quality scorer based on multi-region aesthetic quality perception, combines the aesthetic features of multiple regions to understand and generates regional aesthetic quality scores for images of different scales.

[0048] The method aims to overcome the problem of insufficient local aesthetic quality perception of images in existing methods. The core lies in explicitly extracting and fusing the aesthetic quality features of multiple representative regions in the image, rather than relying only on global features or a single visual feature such as composition, color distribution, etc. that affects the aesthetic quality.

[0049] Step S1 includes the following sub-steps:

[0050] S1-1, using a pre-trained backbone network MaxViT to extract the full-image feature map F global of the image.

[0051] S1-2, introduce a region feature extraction module, which is F global N key regions are selected on the image For each region R i Apply region adaptive pooling to extract its aesthetic quality feature vector F i Use a region attention fusion module, which calculates the contribution weight a i of each region feature F i to the final aesthetic score and weighted fusion:

[0052]

[0053] W a is a learnable weight vector.

[0054] S1-3, output the whole image score through the regression layer:

[0055]

[0056] W r ,b r are the regression layer parameters, and σ(·) is the Sigmoid function.

[0057] S2, based on the region aesthetic quality score, use the aesthetic region regressor to generate the best and worst aesthetic regions in the picture, realize the comprehensive evaluation of the whole picture aesthetic quality, and realize the positioning of the best and worst aesthetic regions in the picture.

[0058] The image cropper designed by this method can locate the defect area of the image and output the best aesthetic region of the image.

[0059] Step S2 includes the following sub-steps:

[0060] Generate a defect heat map A = H(x, y), where H(x, y) ∈ 0, 1 represents the defect probability at coordinate (x, y), and the higher the value, the worse the aesthetic quality.

[0061] Worst aesthetic region calculation:

[0062] A b = (x, y) | H(x, y) > p,

[0063] p is a threshold value, which means that the image aesthetic quality is negatively affected.

[0064] Best cropping region calculation:

[0065]

[0066] Where C is a set of candidate cropping regions that meet the preset aspect ratio, S(C) is the aesthetic score of region C, and λ is the balance weight.

[0067] Experimental quantitative results

[0068] The present application is verified on the current largest image aesthetic quality evaluation dataset AVA [1] as shown in the following table 1:

[0069] Table 1

[0070] Method Backbone network PLCC SRCC ACC Hosu et al. [5] ]]> InceptionResNet 0.757 0.756 0.817 Charm

[13] ]]> Dinov2-small 0.779 0.777 0.826 The invention MaxViT-small 0.793 0.788 0.841

[0071] Wherein PLCC (Pearson Linear Correlation Coefficient) measures the linear consistency of the model score and the artificial score; SRCC (Spearman Rank Correlation Coefficient) measures the nonlinear consistency of the model score and the artificial score; ACC (Accuracy) maps the model score and the artificial score to two categories (1-5 points for low quality category, 6-10 points for high quality category), and calculates the accuracy of the model score.

[0072] At the same time, the present application also annotates the poor aesthetic quality region on the AVA test set, and obtains about 1000 region frames. The defect positioning accuracy verification of the present application is shown in the following table 2:

[0073] Table 2

[0074] Defect type IoU Overexposure 0.783 Subject blur 0.768 Poor composition 0.815 Color imbalance 0.734

[0075] Wherein IoU (Intersection over Union) is used to calculate the overlap degree of the model predicted region and the artificial annotated region.

[0076] Figure 2 is a schematic diagram of an image aesthetic quality evaluation system based on region-level fine-grained understanding in an embodiment of the present application.

[0077] As shown in Figure 2 , the present application also discloses an image aesthetic quality evaluation system based on region-level fine-grained understanding, which comprises a region aesthetic quality score module and an aesthetic region positioning module.

[0078] The region aesthetic quality score module is performed according to the above step S1, and the aesthetic quality scorer based on multi-region aesthetic quality perception is combined with the aesthetic features of multiple regions for understanding, so as to generate region aesthetic quality scores of images of different scales.

[0079] The aesthetic region positioning module is performed according to the above step S2, based on the region aesthetic quality score, uses the aesthetic region regressor to generate the best and worst aesthetic regions in the picture, realizes the comprehensive evaluation of the picture aesthetic quality, and realizes the positioning of the best and worst aesthetic regions in the picture.

[0080] Those skilled in the art should understand that the present application is not limited to the above-mentioned embodiments, and the above-mentioned embodiments and descriptions in the specification are only to illustrate the principles of the present application, and various changes and improvements can be made without departing from the spirit and scope of the present application, and these changes and improvements all fall within the scope of the claimed present application. The scope of protection of the present application is defined by the appended claims and their equivalents.

Claims

1. A method for image aesthetic quality assessment based on region-level fine-grained understanding, characterized by: The steps include: S1, an aesthetic quality scorer based on multi-region aesthetic quality perception, combines the aesthetic features of multiple regions to understand and generate regional aesthetic quality scores for images of different scale regions; S2, based on the regional aesthetic quality scores, uses an aesthetic region regressor to generate the best and worst aesthetic regions in the image, achieves a comprehensive evaluation of the aesthetic quality of the entire image, and locates the regions with the best and worst aesthetic effects in the image.

2. The image aesthetic quality assessment method based on region-level fine-grained understanding according to claim 1 is characterized by: in, Step S1 includes the following sub-steps: S1-1, use the pre-trained backbone network MaxViT to extract the full image feature map F of the image global ; S1-2, introduces a regional feature extraction module, which is based on F global Select N key areas For each region R i Apply region adaptive pooling to extract its aesthetic quality feature vector F i , using a regional attention fusion module, which calculates each region feature F i Contribution weight to the final aesthetic score a i And weighted fusion: W a is the learnable weight vector; S1-3, output the whole image score after the regression layer: W r ,b r is the regression layer parameter, and σ(·) is the Sigmoid function.

3. The image aesthetic quality assessment method based on region-level fine-grained understanding according to claim 1 is characterized by: in, Step S2 includes the following sub-steps: Generate a defect heat map A = H(x,y), where H(x,y)∈[0,1] represents the defect probability at the coordinate (x,y). Higher values ​​indicate worse aesthetic quality. Worst aesthetic area calculation: TO b =(x,y)∣H(x,y)>p, p is a threshold, exceeding this threshold indicates a negative impact on the aesthetic quality of the image. Optimal cropping area calculation: Where C is the set of candidate cropping regions that meet the preset aspect ratio, S(C) is the aesthetic score of region C, and λ is the balance weight.

4. An image aesthetic quality assessment system based on region-level fine-grained understanding, characterized by: include: The regional aesthetic quality scoring module is based on an aesthetic quality scorer based on multi-region aesthetic quality perception. It combines the aesthetic characteristics of multiple regions to understand and generate regional aesthetic quality scores for images of regions at different scales. The aesthetic region positioning module uses an aesthetic region regressor based on the regional aesthetic quality scores to generate the best and worst aesthetic regions in the image, achieve a comprehensive evaluation of the aesthetic quality of the entire image, and locate the best and worst aesthetic regions in the image.

Images