Art material aesthetic evaluation method and system fusing design rule and visual attention mechanism

Abstract

The invention discloses an artistic material aesthetic evaluation method and system fusing a design rule and a visual attention mechanism, and the method comprises the steps: simulating a human visual attention path through extracting the multi-level visual information of an image, and carrying out the collaborative analysis through combining four interpretable aesthetic dimensions, namely composition, color, content and image quality, extracting local and overall aesthetic representations; each expert module outputs intermediate aesthetic indexes according to visual features and classic design rules; under the guidance of a visual attention mechanism, the intermediate aesthetic indexes are fused into a unified comprehensive aesthetic representation; the model outputs aesthetic scores of the nine local areas, and the overall aesthetic evaluation score of the image is obtained through weighted average calculation. According to the method, an end-to-end multi-task joint training strategy is adopted, and the expression ability and evaluation precision of aesthetic features of the art image are effectively enhanced.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of artistic material aesthetic evaluation, and particularly relates to an artistic material aesthetic evaluation method and system fusing design rules and visual attention mechanisms. BACKGROUND

[0002] With the rapid development of computer vision and artificial intelligence technology, how to make computer models simulate human aesthetic ability, and automatically evaluate the aesthetics of images and videos, i.e. computational aesthetics, has become a research field that is full of challenges and attracts much attention. Current image aesthetic computing models generally rely on extracting multi-level visual features (such as color, composition, semantic information, etc.) from images, and using deep learning and other data-driven methods to make aesthetic scores. These methods perform well when dealing with natural images or photographic images, but when facing images with high artistic style (such as Van Gogh's Sunflowers, Piet Mondrian's Composition with Red, Yellow, and Blue, etc.), it is often difficult to accurately grasp their complex artistic language and aesthetic intentions, resulting in a large deviation between the evaluation results and human subjective aesthetics. From the perspective of psychology, image aesthetics is a highly subjective and dynamic perception process, involving many factors such as human emotions, experiences, interests, and attention. Among them, the visual attention mechanism is the key bridge connecting external image features and internal subjective feelings, which determines which areas are first focused on when viewing images, how information is processed in the brain, and thus deeply affects the final aesthetic judgment. Therefore, visual attention is the core embodiment of aesthetic psychological mechanism, which can show the perception path, interest focus and aesthetic preference when people watch images, and to a large extent determines the way in which image content is understood, perceived and evaluated. The current model fails to effectively model this mechanism, which is a common key problem that restricts image aesthetic computing to further approach human subjective feelings.

[0003] Existing image aesthetic computing models include: image feature-based aesthetic evaluation methods. Early research generally relies on manually designed image features, such as brightness, contrast, color composition, composition balance, texture distribution, etc., and image aesthetic scores are obtained through statistical analysis or traditional machine learning models (such as support vector machines, logistic regression, etc.). Convolutional neural network-based aesthetic evaluation methods. Deep learning methods such as convolutional neural networks (CNN) are widely used in image aesthetic evaluation. Representative models such as NIMA (Neural Image Assessment), A-Lamp, etc. usually predict image aesthetic scores or grades through regression or classification methods. Methods based on contrast learning and multi-task learning. Some studies attempt to model the subjective aesthetic judgment process by using contrast learning, ranking learning, or integrating subjective labels (such as preference). For example: using picture pairs (A is more beautiful than B) to train an aesthetic ranking model; a multi-task network simultaneously predicts aesthetic scores and image content labels. Aesthetic evaluation methods that introduce visual attention. Some studies simulate human visual attention mechanisms to extract saliency maps or attention heat maps from images, and combine the features of salient regions to obtain weighted aesthetic scores. These methods usually use saliency detection models or attention mechanisms (such as self-attention, visual Transformer) to capture the most important local regions for aesthetic judgment, enhance the understanding of local details and overall structure, and improve the accuracy and interpretability of aesthetic evaluation.

[0004] Limitations of existing technologies: Image feature-based aesthetic evaluation methods are highly dependent on feature selection and extraction quality, making it difficult to adapt to diverse artistic styles. When faced with stylized artistic images, especially paintings with strong artistic style expression (such as Van Gogh's "Sunflowers" and Mondrian's "Composition with Red, Yellow, and Blue"), it is often difficult to accurately reflect their aesthetic value and artistic characteristics. Convolutional neural network-based aesthetic scoring methods require data fitting and lack in-depth simulation of human subjective aesthetic cognitive processes, especially in accurately reflecting the visual attention and scanning behavior of viewers. In addition, these methods usually only give overall aesthetic scores, ignoring the aesthetic differences of local details, resulting in evaluation results lacking fine-grained interpretability. Methods based on contrast learning and multi-task learning do not start from the perspective of human perception mechanisms, only modeling aesthetic behavior from the label level such as user profile and scoring preference, ignoring psychological characteristics such as real perception path and attention distribution during image viewing. When faced with artistic materials with varying styles, the stability is poor and the generalization ability is limited. Aesthetic evaluation methods that introduce visual attention often only focus on salient regions, lack comprehensive analysis of multi-dimensional aesthetic features such as composition, color, and content, and the evaluation results are difficult to fully reflect the overall artistic beauty of the image. SUMMARY

[0005] To solve the above technical problems, the present application provides an aesthetic evaluation method and system for artistic materials by fusing design rules and visual attention mechanisms, which effectively enhances the expression ability and evaluation accuracy of aesthetic features.

[0006] In one aspect, the present application provides an aesthetic evaluation method for artistic materials by fusing design rules and visual attention mechanisms, which comprises:

[0007] The input image is input into a shared visual feature extraction module, which is based on the ConvNeXt architecture and consists of multiple ConvNeXt blocks stacked in an inverted bottleneck structure, and outputs multi-scale feature maps;

[0008] The feature maps are input into a visual attention module, which includes a visual system processing unit, a visual working memory processing unit, and a priority map generation unit, generates a time series of inhibition map and outputs the probability distribution of the next fixation point;

[0009] The visual feature maps output by the visual attention module are divided into 9 equal local region features;

[0010] The feature maps of different levels are input into multiple parallel expert analysis modules, including a composition layout expert, a color light and shadow expert, an image quality expert, and a content subject expert, which extract feature vectors corresponding to the aesthetic dimensions respectively;

[0011] The aesthetic feature vectors output by the expert modules are input as a whole into a fusion Transformer module together with the 9 equal local visual features segmented by the visual attention module, which performs multi-layer encoding processing on the input sequence and outputs comprehensive aesthetic feature vectors corresponding to the 9 local regions;

[0012] The 9 local comprehensive aesthetic feature vectors output by the fusion Transformer module are input into a scoring module, which outputs 9 local aesthetic scores respectively through a linear regression layer;

[0013] The 9 local aesthetic scores are weighted and averaged to obtain the final overall aesthetic score of the image.

[0014] In another aspect, the present application also provides an aesthetic evaluation system for artistic materials by fusing design rules and visual attention mechanisms, which comprises:

[0015] A shared visual feature extraction module is used to receive input images and extract multi-scale feature maps based on the ConvNeXt architecture;

[0016] a visual attention module, configured to generate a time series of inhibition maps and output a probability distribution of a next fixation point while dividing a visual feature map into nine equal local region features;

[0017] an expert analysis module, configured to extract feature vectors of corresponding aesthetic dimensions from the multi-scale feature map;

[0018] a fusion Transformer module, configured to perform multi-layer encoding processing on the input nine-equal local visual features and the sequence of expert feature vectors, and output comprehensive aesthetic feature vectors of the nine local regions;

[0019] a scoring module, configured to output nine local aesthetic scores respectively through a linear regression layer and perform weighted averaging to obtain a final overall aesthetic score.

[0020] The present application discloses an artistic material aesthetic evaluation method and system fusing design rules and visual attention mechanisms, simulates human visual attention mechanisms, fuses multi-dimensional design rules, and significantly improves the aesthetic evaluation accuracy for artistic materials; is robust for artistic images with complex composition and variable style; has an efficient model architecture, small feature extraction and fusion module calculation overhead, and is suitable for efficient image aesthetic evaluation; supports output based on multi-dimensional interpretable aesthetic indicators such as composition, color, content, and image quality, improves model transparency and result interpretability. Supports local aesthetic score output, which helps users understand the specific "good-looking" or "not good-looking" positions in the image, and provides clear guidance for subsequent aesthetic optimization and editing. BRIEF DESCRIPTION OF DRAWINGS

[0021] The accompanying drawings, which form a part of this application, are intended to provide further understanding of the application and are incorporated herein in their entirety. The schematic embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0022] Figure 1 a flowchart of an artistic material aesthetic evaluation method fusing design rules and visual attention mechanisms according to an embodiment of the present application;

[0023] Figure 2 a structural schematic diagram of an artistic material aesthetic evaluation system fusing design rules and visual attention mechanisms according to an embodiment of the present application. DETAILED DESCRIPTION

[0024] It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0025] It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described herein can be executed in an order different from that shown.

[0026] As shown in the embodiment, an art material aesthetic evaluation method combining design rules and visual attention mechanisms is provided, including: Figure 1

[0027] The input image is input to a shared visual feature extraction module, which is based on a ConvNeXt architecture and consists of multiple ConvNeXt blocks stacked in an inverted bottleneck structure, and outputs multi-scale feature maps.

[0028] The feature maps are input to a visual attention module, which includes a visual system processing unit, a visual working memory processing unit, and a priority map generation unit, generates a time series of inhibition map and outputs the probability distribution of the next fixation point.

[0029] The visual feature maps output by the visual attention module are divided into 9 equal local region features.

[0030] The feature maps of different levels are input to multiple parallel expert analysis modules, including a composition layout expert, a color light and shadow expert, an image quality expert, and a content subject expert, which extract feature vectors corresponding to aesthetic dimensions.

[0031] The aesthetic feature vectors output by the expert modules are input as a whole to a fusion Transformer module along with the 9 equal local visual features segmented by the visual attention module, which performs multi-layer encoding processing on the input sequence and outputs comprehensive aesthetic feature vectors corresponding to the 9 local regions.

[0032] The 9 local comprehensive aesthetic feature vectors output by the fusion Transformer module are input to a scoring module, which outputs 9 local aesthetic scores through a linear regression layer.

[0033] The 9 local aesthetic scores are weighted and averaged to obtain the final overall aesthetic score of the image.

[0034] Further, the shared visual feature extraction module includes four stages, each of which is implemented by stacking multiple ConvNeXt blocks.

[0035] The ConvNeXt block includes a 7x7 deep convolution layer, layer normalization, a 1x1 pointwise convolution layer, a GELU activation function, and a residual connection structure.

[0036] ​Further, in the visual attention module, the visual working memory processing unit locally suppresses the previous gaze position through a spatial mask;

[0037] The spatial mask acts on the feature map in the form of a Gaussian function, and a time-related attention feature map is generated through a convLSTM unit.

[0038] Further, the composition layout expert module divides the high-level semantic feature map into 3×3 regions; each region extracts a local structure representation through regional average pooling, and inputs the local structure representation into a multi-layer perceptron with shared parameters to output a composition feature vector.

[0039] Further, the color light shadow expert module divides the input image into 3×3 regions;

[0040] After color space conversion is performed on each region, a statistical histogram is calculated, a color distribution vector is formed by splicing, and the color distribution vector is input into a shallow MLP.

[0041] Further, the fusion Transformer module includes four layers of Transformer encoders;

[0042] Each layer of the encoder includes 8 heads of multi-head self-attention mechanism and a feedforward network, and the input sequence dimension is 512.

[0043] Further, the method adopts a multi-task joint training strategy; and loss functions including aesthetic score, visual attention path prediction, composition classification, color regression, image quality regression and content classification are jointly optimized.

[0044] Specifically, the shared visual feature extraction module core function is to convert the input original image pixels into a feature representation with rich hierarchical information that can be understood and processed by subsequent analysis modules. The module receives a pre-processed image tensor as input, which is composed of four consecutive, hierarchical stages, which are implemented by stacking multiple ConvNeXt blocks as the core basic unit. Each ConvNeXt block adopts an inverted bottleneck structure and contains a main branch and a short connection for residual learning. In the main branch, the feature map is first passed through a deep convolution layer with a large size of convolution kernel (such as 7x7), which aims to effectively expand the receptive field of the model to capture more rich contextual information. Followed by a layer normalization layer to stabilize the training process. Next, a 1x1 pointwise convolution layer expands the dimension of the feature channel (usually 4 times), and through the GELU activation function for nonlinear transformation, forms a wide intermediate layer to learn more rich features. Finally, another 1x1 pointwise convolution layer compresses the channel dimension back to the original size, completing the closed loop of the bottleneck structure. At the same time, a short connection passes the original input features of the block through element-wise addition to fuse the processing results of the main branch, and this residual learning mechanism makes it possible to train very deep networks.

[0045] By stacking such ConvNext blocks in four stages, the entire module systematically increases the channel dimension while gradually reducing the spatial resolution of the feature map. Finally, the module outputs a set of multi-scale feature maps {F1, F2, F3, F4}. Among them, the low-level feature map F1 with higher resolution retains rich details and textures, mainly for the image quality expert module; the feature map F2 with medium-low resolution combines part of the local structure and contour; F3 with medium resolution and strong semantics is mainly for the content subject expert module; and the low-resolution high-level feature map F4 with the most rich global semantic information is used for the composition layout expert module and the core fusion Transformer module.

[0046] The core function of the visual attention module is to simulate the human visual perception mechanism and generate a visual attention scanpath for the input image to be evaluated. The visual attention module includes three steps: visual system processing, visual working memory processing, and priority map generation. The visual system encodes scene information such as contrast, color, direction, and brightness. Visual working memory processing adds selected attention targets, as the brain's processing capacity is limited, it can only focus on one or a few targets at a time. At this point, selective attention will determine which regions of the image can pass through the processing bottleneck. Priority map generation is used to generate a fixation point distribution map, which is guided by "bottom-up" saliency based on scene information, the perceived value of previously attended regions, and the fixation history. All these guiding factors are combined in a weighted manner to generate a dynamic priority map. The visual system processing uses a modified version of the VGG-16 model trained on the ImageNet dataset to generate a feature representation F0 with an 8-fold downsampling ratio. To maintain appropriate spatial resolution, an upsampling layer is added to generate a feature map F1.

[0047] The visual working memory processing uses a convolutional LSTM (convLSTM) to simulate the human visual inhibition (IOR) mechanism. The IOR mechanism enables humans to suppress the processing of visited targets, thereby quickly analyzing the environment. ScanpathNet differs from other convLSTM models in that it uses convLSTM to learn the spatio-temporal dependencies of the fixation sequence, rather than using convLSTM to refine the saliency map. Instead of directly calculating an inhibition map, it implicitly learns the IOR mechanism by directly inhibiting previously fixated feature locations. The process of visual inhibition is achieved by a spatial mask g(x t-1 ,y t-1 ) before the ConvLSTM layer. This mask is a function centered on the previous fixation location (x t-1 ,y t-1 ), usually a Gaussian mask or other form of inhibition mask. This spatial mask will suppress the surrounding area of the previous fixation point (for example, by setting the values of these areas to near 0 or reducing their weights), thereby reducing the impact of previously fixated regions on the next decision. The specific formula is as follows,

[0048]

[0049] where ⊙ denotes element-wise multiplication, so that the previous fixation location in the feature map is inhibited.

[0050] The priority map generation uses a mixture density network (MDN) to explicitly model the randomness of human attention. The MDN generates the probability density of the next fixation point using K Gaussian distributions. The MDN takes the flattened hidden representation from the IOR graph as input and outputs a parametric Gaussian mixture model y t The model consists of the means μ i , standard deviations σ i , correlation coefficients ρ i and mixing weights π i of K Gaussian components. Mathematically, this can be represented as:

[0051]

[0052] To obtain valid probability distributions, the parameters of the MDN are constrained and normalized. The probability distribution of the next fixation location can be represented as:

[0053]

[0054] where N is a bivariate normal distribution.

[0055] To generate the initial priority map, the model extracts rich representations from the image and applies Gaussian blurring in the feature space. To generate the location of the next fixation point, the model randomly samples the output probability distribution. For the next fixation point, the model uses the feature map F1 and masks the (x t-1 , y t-1 ) location. Finally, the feature map is uniformly divided into 9 equal-sized regions (e.g., divided into left-top, top-center, right-top, left-center, …, right-bottom, for a total of 9 regions), each corresponding to a visual local of the input image.

[0056] Multiple parallel expert analysis modules are used to independently and meticulously deconstruct the input image from different aesthetic dimensions, providing multi-angle structured aesthetic information. This part is composed of at least four independently designed sub-networks, which model key dimensions such as composition layout, color and light, image quality, and content subject. These sub-modules all use lightweight and specialized neural network structures, do not share weights with each other, and can efficiently run in a parallel computing framework.

[0057] Among them, the composition layout expert receives the highest semantic layer output (i.e. high-level feature map, representing the global structure information of the whole image) from the shared visual feature extraction module, which retains the spatial distribution information. To realize local composition perception, this module no longer performs global average pooling, but divides the high-level feature map directly into 3x3 nine spatial regions. Each region extracts local structure representation through regional average pooling respectively, and sends it into a multi-layer perception (MLP, containing one to two layers of fully connected layers) with shared parameters, outputting the corresponding composition feature vector. Finally, this module outputs 9 local composition feature vectors, which can be used to measure the regional level composition matching degree of the image under the rules of three-point method, symmetry and visual balance, etc.

[0058] The color and light expert takes the original input image as input, first divides the image into 3x3 nine regions. Each region image is respectively converted in color space (such as RGB to HSV or Lab), and the hue, saturation and brightness channels are separately calculated and statistical histograms. The color histogram of each region is spliced to form a color distribution vector of fixed length, which is then input into a shallow MLP to extract color features at the perception level (such as dominant color, warm and cold tendency, light and dark contrast and harmony). Finally, this module outputs 9 local color feature vectors, corresponding to the nine equal division regions of the image, respectively, for modeling the color aesthetics of each region.

[0059] The image quality expert is used to evaluate the image in the technical level of clarity, noise and compression artifacts, etc. The input of this module is the high-resolution feature map (such as the output of the first few layers of ResNet) in the low-level of the backbone network, which retains rich local texture and edge information. The feature map is first divided into 3x3 nine sub-regions according to the spatial dimension, and each region is extracted by a convolutional coding network. Specifically, each sub-region is sent into three two-dimensional convolution layers (3x3 convolution kernel, output channel number is 32, 64, 128, stride is 1) with shared parameters, and Batch Normalization and ReLU are added in the middle. Finally, each region feature map is compressed by 2x2 max pooling, and flattened into a one-dimensional vector, which is sent into two MLPs to generate regional level image quality feature vectors (such as 64 dimensions). Finally, 9 local image quality vectors are output to describe the blur degree, compression trace and texture integrity in each region.

[0060] The content subject expert is responsible for analyzing the semantic category and subject structure of the image. The input is the hierarchical semantic feature map in the backbone network (such as the intermediate layer output of ResNet), which contains object-level mid-level semantic information. The feature map is first divided into 3x3 nine regions, each region is compressed into a channel-level feature vector through average pooling, and is sent to a multi-layer perception (MLP) with shared parameters to convert into a region-level semantic representation vector. The feature vector can be used to determine whether each region contains semantic subjects such as people, animals, buildings, or natural landscapes, and can also estimate the proportion, number, and distribution of subjects in the region. Finally, the module outputs 9 region-level content semantic vectors representing the subject perception information of each region.

[0061] The fusion Transformer module is a core component for implementing multi-dimensional aesthetic information fusion and final score decision-making, and its structure is based on the standard Transformer Encoder architecture. The input of this module is a set of expert feature vectors output by other sub-modules, forming a unified "expert opinion" sequence, including the global visual vector extracted from the visual attention module, and the feature representation output by the composition, color, image quality, and image content four-dimensional expert modules. In order to aggregate global information, a learnable classification token vector ([CLS] Token) is attached to the front end of the sequence. Therefore, the input sequence length of the Transformer is fixed at 6.

[0062] Before building the input, first use the visual attention scan path output by the visual attention module to weight the deep-level semantic feature map output by the backbone network pixel by pixel, and then extract a 512-dimensional attention-weighted visual feature vector through global average pooling. Each expert module also outputs a 512-dimensional feature vector to ensure consistency in dimensions.

[0063] The sequence is then input into the fusion Transformer encoder. The encoder is composed of 4 stacked Transformer encoding layers (L=4), each containing an 8-head multi-head attention mechanism (Multi-head Attention, h=8), with a scaled dot product attention calculation and a dimension division of 64 dimensions per head (512 / 8). Each layer also contains a feed-forward network (Feed-Forward Network, FFN) with an internal hidden layer dimension of 2048 and an output dimension of 512, using ReLU as the activation function. Each layer is equipped with Layer Normalization and residual connection mechanisms to enhance training stability and modeling depth. The position encoding uses a learnable absolute position embedding to ensure that the order of each expert vector in the sequence is explicitly expressed in space.

[0064] The final output vector corresponding to the first [CLS] token in the sequence output by the Transformer is regarded as the integrated aesthetic representation. The 512-dimensional feature vector is input into a linear regression layer (Fully Connected Layer, output dim = 1) to output the final image aesthetic score. The overall architecture can learn the context dependence, complementary relationship and relative weight between various aesthetic dimensions by means of the powerful inter-feature modeling capability of the Transformer, while keeping the sequence simple, thereby significantly improving the accuracy and discriminability of score prediction.

[0065] The scoring module is located at the output end of the entire system and is a direct execution unit for mapping the integrated aesthetic feature vector obtained by the fusion Transformer module into the final aesthetic score. The module adopts a simple and efficient linear regression head, and the input of the module is the high-dimensional feature vector corresponding to the [CLS] position in the output sequence of the Transformer encoder. The [CLS] vector has 512 dimensions and has included the deep fusion representation of the visual attention information and the information from each expert module (composition, color, quality, content), representing the overall aesthetic perception expression of the image. The internal structure of the scoring module is composed of a fully connected neural network with a single layer of parameters, the input layer dimension is 512, and the output layer dimension is 1, i.e. the aesthetic score of the artistic material.

[0066] In order to accurately perform aesthetic evaluation by using the artistic material aesthetic evaluation model with the fusion of design rules and visual attention mechanism, a multi-task joint training method needs to be used. The method aims to improve the understanding depth and generalization ability of the model for the main task (aesthetic score) by simultaneously optimizing multiple related subtasks. The detailed training steps are as follows:

[0067] The first step is the construction and preparation of the dataset. A multi-attribute training dataset needs to be constructed. Each image sample in the dataset must contain the following types of annotation information: a global aesthetic score (main task label), a continuous numerical value, such as 1 to 10, representing the overall aesthetic quality of the image; a visual attention scan path (attention task label), a grayscale image of the same size as the original image, generated from real human eye-tracking data; and expert task labels (auxiliary task labels): composition label (the composition category to which the image belongs, including "rule of thirds," "central symmetry," etc.), color label (a numerical value or category describing color harmony, color temperature, etc.), quality label (a numerical value describing image sharpness, noise level, etc.), and content label (the content category to which the artistic material belongs, including "clothing texture," "drawing," etc.). Since a single existing dataset usually cannot contain all of the above labels, it is necessary to construct this multi-attribute dataset by fusing multiple existing public datasets (such as using aesthetic scores from the AVA dataset, or visual attention scan path annotations from the SALICON dataset), or through manual annotation.

[0068] The second step is model initialization. To accelerate model convergence and improve performance, pre-trained weights are used for initialization: the shared visual feature extraction module (ConvNeXt) is loaded with its pre-trained weights on a large image classification dataset (such as ImageNet-22K); the visual attention module is loaded with model weights pre-trained on a large visual attention scan path prediction dataset; and other modules, including the newly added network layers in the expert analysis module, the fusion Transformer module, and the scoring module, use standard random initialization methods (such as Kaiming initialization or Xavier initialization).

[0069] The third step is to design the joint loss function. A joint loss function L_total is designed as the overall objective of model optimization. This function is a weighted sum of the loss functions of all subtasks:

[0070] L total =λ aes L aes +λ sal L sal +λ comp L comp +λ color L color +λ qual L qual +λ cont L cont ;

[0071] Among them, L aes For the aesthetic rating loss of the main task, L1 loss (MAE Loss) is used to calculate the difference between the model's predicted score and the true score.sal For visual attention scan path prediction loss, binary cross-entropy loss is used to calculate the pixel-level difference between the predicted visual attention scan path and the real image. comp and L cont are composition and content classification loss respectively, using cross-entropy loss. color and L qual are color and quality regression loss respectively, using mean square error loss or L1 loss. aes ,λ sal is a set of preset hyperparameters, used to balance the contribution weight of different tasks in the total loss, to ensure that the main task is fully optimized, while the auxiliary task can also play an effective regularization and guiding role.

[0072] Fourthly, iterative optimization training. Small batch stochastic gradient descent strategy is used to train the model end-to-end. First, a small batch of images and their corresponding multi-attribute labels are randomly selected from the prepared data set. Input the batch of images into the MAP-Net, perform a forward propagation, and get the model's prediction output for all tasks (including the final aesthetic score, visual attention scan path and expert dimension feature / classification result). Then, according to the definition of the joint loss function L total , the total loss between the model's prediction output and the real label is calculated. Perform a backward propagation, calculate the gradient of all trainable parameters in the model according to the total loss. Use the optimizer SGD with momentum to update the model's weights according to the calculated gradient. Finally, repeat the above steps until the model's performance on the validation set converges or reaches the pre-set training period. In the training process, learning rate warm-up and cosine annealing strategies are used to dynamically adjust the learning rate to obtain better training results.

[0073] As shown in Figure 2 , the embodiment also provides an artistic material aesthetic evaluation system integrating design rules and visual attention mechanism, which comprises:

[0074] A shared visual feature extraction module is configured to receive an input image and extract multi-scale feature maps based on a ConvNeXt architecture;

[0075] A visual attention module is configured to generate a time series of inhibition map and output a probability distribution of the next fixation point, while dividing the visual feature map into 9 equal local region features;

[0076] An expert analysis module is configured to extract feature vectors of corresponding aesthetic dimensions from the multi-scale feature maps;

[0077] The Transformer module is integrated to perform multi-layer encoding on the input sequence of 9 equal parts of local visual features and expert feature vectors, and output a comprehensive aesthetic feature vector of 9 local regions.

[0078] The scoring module is used to output nine local aesthetic scores through a linear regression layer and perform a weighted average to obtain the final overall aesthetic score.

[0079] The core of this invention is a multi-module cascaded neural network system, which mainly consists of the following functional modules: a shared visual feature extraction module, a visual attention module, multiple parallel expert analysis modules, a fusion Transformer module, and a scoring module.

[0080] The shared visual feature extraction module is responsible for converting the input preprocessed image into a multi-scale visual feature representation. Specifically, it is implemented as a ConvNeXt block consisting of four stacked stages. Each block employs an inverted bottleneck structure, including large-kernel depthwise convolutions, layer normalization, pointwise convolutions, and residual connections, progressively extracting multi-level feature maps from low-level texture details to high-level semantic information. The multi-scale features output by this module provide rich foundational representations for subsequent expert modules and the visual attention module.

[0081] The visual attention module simulates the human visual perception mechanism. It extracts salient features of the scene based on a pre-trained VGG-16 model and combines a convolutional LSTM to simulate the visual inhibition mechanism (IOR). Through spatial masking, it dynamically suppresses the gazed area, achieving dynamic generation of the attention focus. A hybrid density network (MDN) is used to model the gaze probability distribution, outputting a gaze path priority map. Then, a local gaze path priority map is obtained by dividing the region into nine equal parts (3×3), which guides subsequent feature weighting and attention enhancement.

[0082] Multiple parallel expert analysis modules include the following four sub-modules, each addressing different aesthetic dimensions for spatial perception modeling and semantic deconstruction of images: The composition layout expert, based on the high-level semantic feature map of the shared feature module, divides the image into nine equal parts (3×3), extracts regional composition features through local average pooling and a shared multilayer perceptron, and evaluates the matching degree of classic composition rules in each region; the color and lighting expert directly processes the nine spatial sub-blocks of the image, performs color space transformation, extracts regional color statistical vectors, and uses a shallow MLP to model the color harmony and brightness features of each region; the image quality expert utilizes the low-level high-resolution feature map from the shared module, divides the image into nine-grid sections, and extracts the image quality features of each region through a shared small convolutional network and fully connected layers, reflecting its sharpness, compression artifacts, and texture consistency; the content subject expert divides the image into nine regions from the mid-level semantic feature map, performs average pooling and multilayer perceptron modeling on each region, extracts the semantic category and subject distribution information of each region, thereby achieving spatial representation of the content dimension.

[0083] The fusion Transformer module is used as the core of multi-dimensional aesthetic feature fusion, receives the attention-weighted visual features output by the visual attention module and the 512-dimensional feature vector of each expert module, forms an expert feature sequence with a fixed length of 6, and fuses the information of each aesthetic dimension through a 4-layer stacked standard Transformer encoder (including multi-head self-attention and feedforward network), captures the context dependence and complementary relationship between features, and outputs local aesthetic features of 9 regions.

[0084] The scoring module is located at the end of the system, receives the comprehensive aesthetic feature vector ([CLS] Token corresponding to the 512-dimensional vector) output by the fusion Transformer module, maps the comprehensive features to the aesthetic evaluation scores of 9 local regions through a single-layer linear regression network, realizes the quantitative evaluation of artistic materials, and finally obtains the overall aesthetic score through weighted average.

[0085] In order to improve the generalization ability and scoring accuracy of the model, the system adopts a multi-task joint training strategy to optimize the joint loss function containing aesthetic score, visual attention path prediction, composition, color, image quality and content classification. In the training process, combined with multi-attribute annotation data, some modules are initialized using pre-training weights, and random gradient descent and learning rate scheduling strategies are used to realize end-to-end joint optimization.

[0086] The artistic material aesthetic evaluation model designed by fusing design rules and visual attention mechanism according to the present application has a data flow architecture of "main distribution-parallel analysis-center fusion" in structure. The structure is designed to efficiently deconstruct and process complex visual information, and the specific connection relationship between components and data flow direction are as follows.

[0087] In structure, after the input end of the system receives the image to be evaluated, the data flow first enters the shared visual feature extraction module (based on ConvNeXt). The module is designed as the core visual backbone of the entire system, and the output end of the module, i.e. a set of multi-scale feature maps F1, F2, F3, F4, is designed to form a "one-to-many" fan-out connection with multiple downstream modules.

[0088] Specifically, this set of multi-scale feature maps is precisely distributed to the corresponding analysis modules: the output end of the low-layer, high-resolution feature map F1 is unidirectionally connected to the input end of the image quality expert module, providing rich detailed texture information for it; the output end of the middle-layer, strong semantic feature map F3 is unidirectionally connected to the input end of the content subject expert module, providing the middle-level semantics required for resolving the object; and the output end of the high-layer, global semantic feature map F4 is simultaneously connected to the input end of the composition layout expert module and the input preprocessing unit of the subsequent fusion Transformer module. The color and light expert module is relatively independent in structure, and its input end is directly connected to the original image input end of the system, ensuring that its analysis is based on the most original color information

[0089] At the same time, the input end of the visual attention module is also connected to the middle-high layer feature maps F3, F4 of the shared backbone, and the visual attention scanning path generated by the output end of the visual attention module flows into the input preprocessing unit of the fusion Transformer module, which is used to generate a visual attention weighted visual feature vector.

[0090] The core hinge of the system structure is the fusion Transformer module. The input end of the module plays a "many-to-one" information collection role in structure. It collects information streams from five different sources: 1) attention weighted global visual vector; 2) output vector of the composition layout expert module; 3) output vector of the color and light expert module; 4) output vector of the image quality expert module; and 5) output vector of the content subject expert module. These vectors are arranged and combined into a sequence with a fixed length (6 tokens) in structure, serving as the unified input of the fusion Transformer module.

[0091] On the final output link, the aggregated output end (i.e., the feature vector corresponding to the [CLS] token) of the fusion Transformer module is unidirectionally connected to the input end of the scoring module. The scoring module serves as a simple linear mapping unit, and the output end thereof is the final aesthetic evaluation value of the entire system.

[0092] In summary, the system structure of the present application realizes efficient and independent analysis of multi-dimensional image information through initial parallelization design; and realizes deep and context-related association and integration of these discrete information through the centralized fusion Transformer hinge, and finally draws a conclusion through a simple output link. This "diversion-confluence" structure ensures the comprehensiveness of system analysis and the intelligence of decision-making.

[0093] Alternative one: This scheme realizes the global representation of image content by constructing a unified deep image feature extraction backbone (such as ResNet, DenseNet or EfficientNet, etc.). After the output of the backbone, multiple independent branch modules (i.e. multiple regression heads or classifiers) are connected to complete the prediction of the composition, color, content, clarity and other aesthetic dimensions respectively, and finally the total score is obtained through weighting or joint mechanism. The module structure includes an image feature extractor (Backbone) for extracting general image features, a multi-branch head structure, each head processing an aesthetic dimension, and a score fusion module: the outputs of each dimension are fused into the final score through simple weighting, logistic regression or small neural network.

[0094] Alternative two: Use a pre-trained large-scale image-text joint model (such as CLIP, BLIP, Florence, etc.) to convert the image into a semantic feature vector through an image encoder; at the same time, input the pre-defined aesthetic dimension text (such as "this is a colorful image") into the text encoder; calculate the semantic matching degree (usually cosine similarity) between the two as the basis for scoring the image in this aesthetic dimension. The module structure includes an image encoder from the CLIP or BLIP image branch (such as ViT-B / 32); a text encoder, a CLIP text branch corresponding to multiple aesthetic dimension prompts; a matching module that calculates the cosine similarity of each pair of image-text to obtain the score; a score aggregation that weights the multi-dimensional scores to output the final score.

[0095] The application discloses an artistic material aesthetic evaluation method and system fusing design rules and visual attention mechanisms, which solves the problem of insufficient accuracy of existing aesthetic evaluation methods for aesthetic evaluation of stylized artistic images. The proposed model is composed of a shared visual feature extraction backbone, a visual attention module, a multi-path parallel expert analysis module, and an attention-guided feature fusion and scoring module. This method extracts multi-level visual information from images, simulates human visual attention paths, and combines composition, color, content and image quality, four aesthetic dimensions with explainability for collaborative analysis, and extracts local and overall aesthetic representations. Each expert module outputs intermediate aesthetic indicators based on visual features and classic design rules; under the guidance of the visual attention mechanism, these indicators are fused into a unified comprehensive aesthetic representation. Finally, the model outputs nine aesthetic scores for local regions, and calculates the overall aesthetic evaluation score of the image through weighted average. The system adopts an end-to-end multi-task joint training strategy, effectively enhancing the expression ability and evaluation accuracy of aesthetic features.

[0096] The above merely provides the preferred embodiments of the present application, and the protection scope of the present application is not limited thereto, and any changes or substitutions within the technical scope disclosed by the present application should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. An art material aesthetic evaluation method that fuses design rules and visual attention mechanisms, characterized by, The method comprises the following steps: inputting an input image into a shared visual feature extraction module, which is composed of multiple ConvNeXt blocks stacked in an inverted bottleneck structure based on a ConvNeXt architecture, and outputting multi-scale feature maps; inputting the feature maps into a visual attention module, which comprises a visual system processing unit, a visual working memory processing unit and a priority map generation unit, generating a time series of inhibition map and outputting a probability distribution of the next fixation point; dividing the visual feature maps output by the visual attention module into 9 equal local region features; inputting the feature maps of different levels into multiple parallel expert analysis modules, including a composition layout expert, a color and light expert, an image quality expert and a content subject expert, to extract feature vectors of corresponding aesthetic dimensions; inputting the aesthetic feature vectors output by the expert modules as a whole into a fusion Transformer module together with the 9 equal local visual features segmented by the visual attention module, and performing multi-layer encoding on the input sequence to output comprehensive aesthetic feature vectors corresponding to the 9 local regions; inputting the 9 local comprehensive aesthetic feature vectors output by the fusion Transformer module into a scoring module, and outputting 9 local aesthetic scores through a linear regression layer respectively; performing weighted average on the 9 local aesthetic scores to obtain the final overall aesthetic score of the image.

2. The method of claim 1, wherein the method is characterized by: The shared visual feature extraction module comprises four stages, each of which is implemented by stacking multiple ConvNeXt blocks; The ConvNeXt block comprises a 7×7 deep convolution layer, a layer normalization, a 1×1 point convolution layer, a GELU activation function and a residual connection structure.

3. The method of claim 1, wherein the method is characterized by: In the visual attention module, the visual working memory processing unit performs local inhibition on the previous fixation position through a spatial mask; The spatial mask acts on the feature map in the form of a Gaussian function, and generates a time-dependent attention feature map through a convLSTM unit.

4. The method of claim 1, wherein the fusion of design rules and visual attention mechanism is an aesthetic evaluation of artistic material. The composition layout expert module divides the high-level semantic feature map into 3×3 regions; each region extracts a local structure representation through regional average pooling and inputs it into a multi-layer perceptron with shared parameters to output a composition feature vector.

5. The method of claim 1, wherein the fusion of design rules and visual attention mechanism is an aesthetic evaluation of artistic materials. The color and light expert module divides the input image into 3×3 regions; After color space conversion, a statistical histogram is calculated for each region, and a color distribution vector is formed and input into a shallow MLP.

6. The method of claim 1, wherein the method is characterized by: The fusion Transformer module comprises four layers of Transformer encoders; Each encoder contains 8 heads of multi-head self-attention mechanism and feedforward network, and the input sequence dimension is 512.

7. The method of claim 1, wherein the method is characterized by: The method adopts a multi-task joint training strategy; the loss functions including aesthetic score, visual attention path prediction, composition classification, color regression, image quality regression and content classification are jointly optimized.

8. A system for aesthetic evaluation of artistic material in fusion of design rules and visual attention mechanism according to any one of claims 1-7, characterized in that, The method comprises the following steps: a shared visual feature extraction module for receiving an input image and extracting multi-scale feature maps based on a ConvNeXt architecture; a visual attention module for generating a time series of inhibition maps and outputting a probability distribution of the next fixation point while dividing the visual feature map into 9 equal local region features; an expert analysis module for extracting feature vectors corresponding to aesthetic dimensions from the multi-scale feature map; a fusion Transformer module for multi-layer encoding processing of the input 9-equal local visual features and expert feature vector sequence, and outputting comprehensive aesthetic feature vectors of 9 local regions; a scoring module for outputting 9 local aesthetic scores respectively through a linear regression layer and performing weighted averaging to obtain a final overall aesthetic score.

Images