A general image aesthetic evaluation method based on state space model

Abstract

The application discloses a general image aesthetic evaluation method based on a state space model, and comprises the following steps: step 1, acquiring a training sample set S train and a test set S test according to a data set and a task type; step 2, constructing sub-modules of a general image aesthetic evaluation framework; step 3, constructing a specific general or personalized aesthetic evaluation model M in combination with a specific task according to the different sub-modules in step 2; step 4, iteratively training the network model M; and step 5, obtaining an aesthetic quality evaluation score prediction result through the model after the iterative training in step 4. The application effectively solves the problems that an existing image aesthetic evaluation model is difficult to combine local and global image features, the imbalance of multi-task training of the model, and the difficulty of efficient and flexible design of the model.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of image aesthetic evaluation, and particularly relates to a general image aesthetic evaluation method based on a state space model. BACKGROUND

[0002] The purpose of image aesthetic evaluation is to predict the quality of an image from an aesthetic perspective. It has wide applications in the fields of image editing, generation and shooting. However, a person's judgment of aesthetics is highly related to different visual attributes (such as color, composition and content) and multiple individual roles. It is still challenging to develop an effective and efficient method to comprehensively consider all these information in the reasoning process.

[0003] In this field, researchers have carried out a lot of work and proposed a large number of algorithms. Most of the existing works are focused on generic aesthetic assessment (GIAA), which aims to predict the average aesthetic evaluation scored by multiple individuals. On the generic aesthetic quality assessment, domestic and foreign researchers have carried out extensive and in-depth research. These researches can be mainly divided into three categories of algorithms: the aesthetic quality evaluation algorithm based on manual features refers to that researchers manually design features closely related to aesthetics. However, the selection process of manual features requires high precision and meticulous observation, which leads to the relative finiteness of feature dimension. The image aesthetic quality evaluation algorithm based on visual features is to more comprehensively capture the aesthetic features of images. Many studies have designed a double-flow network or a structure of multiple sub-networks to simultaneously focus on the global and local features of images. The multi-modal aesthetic quality evaluation algorithm based on user comments focuses on the massive user comments accompanying the images. These comments not only express the emotions of users, but also contain rich semantic content, which provides strong assistance for the aesthetic quality evaluation of images. The research results in this field open up a new direction for the field of aesthetic quality evaluation and provide valuable reference and inspiration for future research.

[0004] Recently, researchers have begun to explore personalized image aesthetic assessment (PIAA), which learns a specific model for each individual to predict personal aesthetic preferences. The training of the model is usually divided into two stages. In the first stage, a general image aesthetic assessment dataset is used for supervised training to obtain an aesthetic prior knowledge model. This model can learn general aesthetic rules and patterns, providing a basis for subsequent personalized fine-tuning. In the second stage, to address the subjective nature of user visual aesthetic experience, the aesthetic prior knowledge model is fine-tuned using a specific user's PIAA dataset. In addition, attempts have been made to develop fine-grained image aesthetic assessment (FIAA), which evaluates the quality or subjective preference of multiple visual attributes.

[0005] For the above three directions, the aesthetic quality evaluation field mainly exists three modalities of features: images, image description texts, and user portrait texts. The challenge for the algorithm is how to design a general method to fuse the above modal information and predict a more relevant aesthetic score. Existing methods usually focus on a single task and strive to improve feature representation, information fusion mechanism, reasoning architecture, learning strategy, and dataset, etc.

[0006] The patent application with publication number CN115272203A and the name of "a general image aesthetic evaluation method based on a state space model" discloses a general method that can be used in various image aesthetic evaluation tasks and can process multi-modal input for aesthetic evaluation;

[0007] However, the method has the following shortcomings: first, the introduction of balanced cross-entropy loss requires manual division of score categories, making the training preparation cumbersome. Second, for PIAA tasks, different user information processing requires dynamic and real-time construction of text templates, and training needs to be redeveloped. In summary, the existing technology has the following defects:

[0008] First: most existing image quality evaluation methods cannot efficiently model image feature representations that combine local and global information. Subjective judgments of image aesthetics are based on the integration of multiple visual information, from local details such as noise and color to overall perception such as composition and semantics. Local and global perception are essential for IAA. Current IAA methods mainly use convolutional neural networks (CNN) or attention-based vision transformers (ViT) to learn visual representations. However, these models cannot effectively combine local and global information.

[0009] Second: cannot effectively balance the imbalance of multi-task learning. The prediction of multi-attribute evaluation is a kind of multi-task learning. In the learning process, the attribute evaluation tasks may interfere with each other, which reduces the stability or effectiveness of representation learning. In addition, the distribution of aesthetic or attribute scores in the existing data set is seriously unbalanced. The model of IAA will be tilted to lower error, and the tail label occupies limited instances.

[0010] Third: cannot achieve efficient and flexible model design and deployment, inference. The existing PIAA method ignores the semantic information of the main character. In addition, it is difficult to modify the role flexibly to improve flexibility and accuracy of PIAA. SUMMARY

[0011] In order to overcome the deficiencies existing in the prior art, the purpose of the present application is to provide a general image aesthetic evaluation method based on state space model, which uses image, text information, user information multiple modalities to carry out feature extraction, fusion and then carries out multi-attribute aesthetic score evaluation. Through the combination of various basic modules, it is flexibly applied to different (Image Aesthetic Assessment, IAA) tasks, and is a general framework suitable for various image aesthetic evaluation tasks. Effectively solve the problems that the existing image aesthetic evaluation model is difficult to combine local and global image features, the imbalance of multi-task training of the model, and the difficulty of efficient and flexible design of the model. In order to achieve the above purpose, the technical scheme adopted by the present application is:

[0012] A general image aesthetic evaluation method based on state space model, comprising the following steps:

[0013] Step 1, according to the data set and the type of task, obtaining the training sample set S train And test set S test ;

[0014] Step 2, construct the sub-modules of the general image aesthetic evaluation framework:

[0015] Step 3: construct different sub-modules from step 2, and construct specific general or personalized aesthetic evaluation model M in combination with specific tasks;

[0016] Step 4, iteratively train the network model M:

[0017] Step 5, obtain the aesthetic quality evaluation score prediction result through the model after iterative training in step 4.

[0018] The data set in the step 1:

[0019] Experiments are conducted on the following benchmark datasets, namely, AVA, TAD66K, PARA, AADB and Photo.Net datasets;

[0020] AVA and Photo.Net include the aesthetic score distribution of each image;

[0021] While PARA and AADB include the average score;

[0022] The PARA dataset contains images, image text information, user portrait information, and user scores for images;

[0023] The dataset does not have a fixed task type, and in combination with the different data and content, the following experiments are conducted:

[0024] For the AADB dataset, only VIAA experiment is conducted;

[0025] For the TAD66K dataset, only VIAA experiment is conducted;

[0026] For the AVA dataset, VIAA and MIAA experiments are conducted;

[0027] For the PARA dataset, VIAA, MIAA, FIAA and PIAA experiments are conducted;

[0028] For the PhotoNet dataset, VIAA and MIAA experiments are conducted.

[0029] In the GIAA related tasks VIAA, FIAA and Multimodal Image Aesthetic Assessment (MIAA), the official training and test data division method is selected;

[0030] In the PIAA task, first, randomly select users from the annotators as experimental objects, conduct multiple tests, and calculate the average value of the evaluation index as the final experimental result, and then fine-tune the aesthetic score prediction model.

[0031] The aesthetic score prediction model conducts VIAA, MIAA, FIAA and PIAA experiments on the PARA dataset;

[0032] Among them:

[0033] The general aesthetic quality evaluation of GIAA includes VIAA, MIAA and FIAA;

[0034] VIAA is a pure visual aesthetic evaluation task, the model processes the input picture and predicts the aesthetic quality score of the corresponding picture;

[0035] MIAA, multi-modal aesthetic assessment task, model processes inputted picture and picture related text information (e.g. description of image, appreciation comment content of image), predicts corresponding picture aesthetic quality score;

[0036] FIAA, pure visual aesthetic assessment task, model processes inputted picture, predicts corresponding picture multi-attribute quality score, including aesthetic, composition, color, depth score;

[0037] PIAA, personalized aesthetic quality assessment, user information, and model processes inputted picture and picture related text information (may not have), predicts corresponding picture aesthetic quality score.

[0038] For different tasks, sample set elements are not the same, and the training sample set and the test sample set have the following three types of inputs: image, image description, user information;

[0039] For images: In the process of fine-grained aesthetic evaluation experiment, the images in the PARA dataset are uniformly preprocessed. Different preprocessing strategies are used in the training and testing stages. In the training stage, data augmentation techniques are used. First, the size of each image is adjusted using bicubic interpolation, and then the image is uniformly resampled. Subsequently, a pixel image block is randomly cropped to adapt to the input requirements of the model. The image is randomly flipped horizontally;

[0040] For image description: Select the image description information provided by the PARA dataset;

[0041] For user information: SMP (Subject Multi-character Prompts), Chinese definition: user multi-character prompt, text template based on user information.

[0042] Make full use of the three user information provided in the PARA dataset: artistic experience, photographic experience, and personality. Specifically, aesthetic experience and photographic experience both contain four levels: "beginner", "competent", "skilled", and "expert";

[0043] Design corresponding text prompts to incorporate this information into the model;

[0044] "My artistic experience is{art_exp}." and "My Photographic experience is{photo_exp}."

[0045] Where art exp and photo exp represent the user's corresponding experience level, and the personality traits include five aspects, namely "responsibility O", "affinity C", "extroversion E", "openness A", and "spirit N". For a given user, the corresponding text prompt information follows the following template:

[0046] "In the Big-Five personality traits test, my scores are as follows: openness score is {O}, conscientiousness score is {C}, extroversion score is {E}, agreeableness score is {A}, and Neuroticism score is {N}."

[0047] All the above text templates can be integrated together to represent the user's personal information. The PARA dataset is used for VIAA task, and the input information of the training set and test set is only pictures, and the output is the aesthetic score.

[0048] The step 2 is specifically:

[0049] Step (1). Visual encoder E based on SSM v Vmamba-Tiny is used as the backbone network, and the output image features are obtained after the classification head output layer of the model. The output features are averaged and pooled.

[0050] Step (2). Text encoder E of user information t First, the input review is fine-grained segmented using Word Piece Tokenizer (WPT) to generate a word sequence of length n. Then a special mark [cls] is added at the beginning of the sequence to represent the global feature, and the corresponding position encoding is added. The embedding vectors are then input into a Bidirectional Encoder Representations from Transformers (BERT) model containing 12 Transformer modules. Finally, a variable-length vector of dimension n is output, where n represents the length of the word. The 0th dimension of the output is truncated and taken as the output.

[0051] Step (3). Adaptive average pooling AAP, which compresses the input sequence dimension to 1 dimension through average pooling.

[0052] Step (4). Cross-modal fusion module CMF, using the visual features f v and text features f t extracted in the image and text encoders respectively

[0053] Step (5). Feature adapter Firstly, the feature f is received, and for a specific i-th task, the f is nonlinearly mapped to reduce the dimension, and after the GELU activation function, it is mapped back to the received dimension feature, and the residual connection is performed with the previous image feature to obtain the output feature The multi-scale adapter is called a global branch.

[0054]

[0055] Step (6). Task adapter φ: On the basis of the feature adapter , a plurality of local branches are added to form the task adapter.

[0056] In the step (4), in the process of information fusion, firstly, the visual features and the text features are subjected to layer normalization processing, and then the normalized visual features are input into the cross-attention layer CA, wherein f v is mapped to obtain a vector Q, f t is linearly mapped to obtain a vector K and V, and the calculation of the cross-attention is as follows:

[0057]

[0058] The output of the cross-attention CA is residual connected with the original visual features to obtain an intermediate feature f v , and then the intermediate feature is subjected to layer normalization again, and then a fully connected layer is passed to obtain a feature, which is residual connected with the intermediate feature to obtain a cross-modal fusion output feature f.

[0059]

[0060]

[0061] The feature adapter output feature needs different heads to adapt the learning process, and there are mainly three local branches:

[0062] The score regression branch (score prediction head): the input feature is mapped, and then the relu activation function is passed and nonlinearly mapped to obtain the predicted score.

[0063] The attribute prediction branch (classification head): the input feature is mapped, and then the relu activation function is passed and nonlinearly mapped to obtain a 4-dimensional score feature, and the predicted category is obtained through the softmax normalization.

[0064] Distribution prediction branch (distribution prediction head): map the input features, then pass through the relu activation function and linear mapping to get the score distribution features, and then pass through the softmax normalization to get the prediction distribution.

[0065] Different branches are selected according to different data sets, and the specific settings are as follows:

[0066] Using the distribution prediction branch, L EMD needs to be increased.

[0067] Using the attribute prediction branch, L Ba1-CE needs to be increased.

[0068] Using the score regression branch, L MSE needs to be increased.

[0069] L dst = L EMD + γL MSE + λL Ba1-CE ,

[0070] L reg = L MSE + λL Ba1-CE ,

[0071] There are two deployment methods in total, for PARA, AADB and TAD66K data sets, L reg method is adopted;

[0072] For AVA and PHOTONET, L dst method is adopted.

[0073] The step 3 is specifically:

[0074] The model receives three types of input, which are image, image description text information and user information; according to the difference of specific task, the model will adopt different construction method to build; according to the difference of predicted attribute, the model will adopt different adapter.

[0075] Further, the specific model selection:

[0076] AesMmba-V model: for pure visual aesthetic evaluation task VIAA, the input is picture and the output is aesthetic score, image encoder E v , a single global branch and a plurality of local branches are adopted;

[0077] AesMmba-M model: for visual and text fusion aesthetic evaluation task MIAA, the input is picture and image description related text information, and the output is aesthetic score, image encoder E v, text encoder E t , information fusion module CMF, single global branch and several local branches;

[0078] AesMmba-F model: for the task of fine-grained image aesthetic evaluation FIAA, the input is an image, and the scores of multiple attributes need to be predicted, then an image encoder E v , multiple global branches and several local branches;

[0079] AesMmba-P model: for the task of personalized visual aesthetic evaluation PIAA, the input is an image and user description information, an image encoder E v , text encoder E t , information fusion module CMF, multiple global branches and several local branches;

[0080] wherein the number of global branches i depends on the number of dataset attributes, the type of local branches and the type of single attribute score, distribution type or average value type. Different aesthetic evaluation tasks are carried out, and the model M is constructed in the above manner.

[0081] The step 4 is specifically:

[0082] Step 4.1, initialize the model in the training stage;

[0083] Initialize the number of iterations t, the maximum number of iterations T, and for the different models selected in step 3, let the current multi-granularity network-based no-reference image quality evaluation network model be M t , M is the model for the task of VIAA, initialize t = 1, M t = M;

[0084] Step 4.2, training stage, set to randomly select batch_size training samples from the training sample set S train without replacement as the input of the network model M t ;

[0085] Step 4.3, construct the loss function: the loss function has three types:

[0086] Mean square error loss function: wherein represents the predicted score, s i represents the true score, and n represents the number of samples;

[0087] Balanced cross-entropy loss function: wherein n l represents the number of instances, and C iwhere K represents the number of tasks, and represents the class label;

[0088] Global moving distance loss function:

[0089] where a is the predicted label, p is the correct label of the data set, CDF is the combined distribution function, and Len is the number of interval blocks. In the inference phase, the aesthetic evaluation score is calculated according to the predicted distribution;

[0090] For different data sets, different training strategies need to be carried out: L dst Carry out distribution loss, for AADB, PARA contains real score data set L reg Carry out regression loss.

[0091] L dst = L EMD + γL MSE + λL Ba1-CE ,

[0092] L reg = L MSE + λL Ba1-CE ,

[0093] For first calculating the predicted score corresponding to each training sample and the score label corresponding to the training sample, according to the different data sets, the above loss function is calculated. The weight parameters of each layer of the network model S t are updated;

[0094] Step 4.4, judge whether the training sample set S train has been traversed, if yes, execute step 4.5, otherwise, execute steps 4.2 and 4.3;

[0095] Step 4.5, judge whether t=T is true, if yes, obtain the trained multi-granularity network based on no-reference image quality evaluation network model M * =M t , otherwise, let t=t+1, and execute steps 4.3 and 4.4.

[0096] The step 5 is specifically:

[0097] The test sample set S test is input into the trained network model M * for forward inference to obtain the quality prediction score of each test sample. For the data set of regression prediction, the prediction output is the score regression branch output; for the data set of distribution prediction, the prediction output is the weighted combination of the distribution prediction output, and for the PIAA task, M iForward reasoning is performed, and the final evaluation index is averaged to verify the model effect.

[0098] The present application has the advantages of:

[0099] The present application utilizes image, text multi-modal information, and can flexibly construct a model for specific aesthetic evaluation tasks. Competitive or even optimal results can be achieved on each task.

[0100] The present application has the advantages of model predicting aesthetic scores highly consistent with human visual aesthetic perception, model adapting to various aesthetic evaluation tasks, and convenient and efficient model training and reasoning.

[0101] The present application proposes a general aesthetic evaluation architecture, which can flexibly process image input, image description, and user text input of multiple modalities. For specific aesthetic evaluation tasks, an adaptive model can be constructed by combining various adaptive modules. Multi-attribute aesthetic score prediction is carried out on multiple datasets and multiple related tasks. The score prediction performance on GIAA and PIAA is remarkable. BRIEF DESCRIPTION OF DRAWINGS

[0102] Figure 1 FIG. 1 is a structural schematic diagram of a cross-information fusion module CMF and an adapter φ of the present application.

[0103] Figure 2 FIG. 2 is a general IAA framework schematic diagram of the present application.

[0104] Figure 3 FIG. 3 is a model structure schematic diagram of different tasks of the present application.

[0105] Figure 4 FIG. 4 is a flowchart of the present application. DETAILED DESCRIPTION

[0106] The present application will be further described in detail below with reference to the accompanying drawings.

[0107] As shown in FIG. 1, a general image aesthetic evaluation method based on a state space model includes the following steps: Figure 4

[0108] Step 1: According to the data set and the task type, a training sample set S and a test set S are obtained. train test

[0109] The data set in step 1 is:

[0110] Experiments are carried out on the following benchmark data sets, i.e., AVA, TAD66K, PARA, AADB and Photo.Net data sets, which contain about 255K, 66K, 20K, 31K and 10K images, respectively. ​​​

[0111] AVA and Photo.Net include the aesthetic score distribution of each image; while PARA and AADB include the average score; PARA dataset contains images, image text information, user profile information, and user ratings of images.

[0112] In GIAA related tasks VIAA, FIAA and Multimodal Image Aesthetic Assessment (MIAA), the official training and test data partition method is selected, which ensures that there is no image scene and content in the training set and test set. The distribution of training and test data is similar.

[0113] In the PIAA task, first, 40 users are randomly selected from the annotators as experimental objects, and multiple tests are performed. In this embodiment, the number of tests for each user is 10, and the average value of the evaluation index is calculated as the final experimental result. Subsequently, the aesthetic score prediction model is fine-tuned in two ways. They are:

[0114] 1. Specify 10 pictures of users for training, and randomly select 50 pictures for testing,

[0115] 2. Specify 100 pictures of users for training, and randomly select 50 pictures for testing. The above two experimental configurations are used as the basis for model training.

[0116] The model performs VIAA, MIAA, FIAA and PIAA experiments on the PARA dataset;

[0117] Among them:

[0118] GIAA general aesthetic quality evaluation includes VIAA, MIAA, FIAA;

[0119] VIAA, a pure visual aesthetic evaluation task, the model processes the input picture and predicts the aesthetic quality score of the corresponding picture;

[0120] MIAA, a multimodal aesthetic evaluation task, the model processes the input picture and the text information related to the picture (such as the description of the image, the appreciation of the content of the image), and predicts the aesthetic quality score of the corresponding picture;

[0121] FIAA, a pure visual aesthetic evaluation task, the model processes the input picture and predicts the multi-attribute quality score of the corresponding picture, including aesthetic, composition, color, and depth score;

[0122] PIAA personalized aesthetic quality evaluation, user information, and model processing input picture and picture related text information (may not have), predict the user's aesthetic quality score of the picture.

[0123] The quality score is a distribution; for example, 1, 4, 7, 24, 15, 3; 1. The physical meaning is: 1 user scored 1, 2 users scored 4 for this picture, 7 users scored 3 for these pictures, and so on;

[0124] Or an average value, for example, 3.54, indicating that the average score of the image is 3.54.

[0125] The model conducts VIAA, MIAA, FIAA and PIAA experiments on the PARA dataset. For different tasks, the sample set elements are not the same, and the training sample set and the test sample set have the following three types of inputs: image, image description, and user information.

[0126] For images: During the fine-grained aesthetic evaluation experiment, the images in the PARA dataset are uniformly preprocessed. Different preprocessing strategies are used in the training and testing stages. In the training stage, data augmentation techniques are used. First, the size of each image is adjusted using bicubic interpolation, and the image is resampled to 256x256 pixels. Then, a 224x224 pixel image block is randomly cropped from the image to adapt to the input requirements of the model. The image is randomly horizontally flipped, and the flip probability is set to 0.5. For image description: Select the image description information provided by the PARA dataset.

[0127] For user information, the SMP of the present application makes full use of the three user information provided in the PARA dataset: artistic experience, photographic experience, and personality. Specifically, aesthetic experience and photographic experience both contain four levels: "beginner", "competent", "skilled", and "expert". In order to combine these information into the model, the present application designs the corresponding text prompts as follows:

[0128] "My artistic experience is{art_exp}." and "My Photographic experience is{photo_exp}."

[0129] Where art_exp and photo_exp represent the user's corresponding experience level. Personality includes five aspects: "responsibility O", "affinity C", "extroversion E", "openness A", and "spirit N". Given a user, the corresponding text prompt information should follow the following template:

[0130] "In the Big-Five personality traits test,my scores are as follows:openness score is{O},conscientiousness score is{C},extroversion score is{E},agreeableness score is{A},and Neuroticism score is{N}."

[0131] All the above text templates can be integrated together to represent the user's personal information. In addition, the present application has great flexibility in attribute selection and expansion.

[0132] In this embodiment, the PARA dataset is used for the VIAA task. The information input information of the training set and the test set is only the picture, and the output is the aesthetic score.

[0133] The training set and the test set in step 1;

[0134] When carrying out the PIAA task: the text template data needs to be manually constructed, for example, the text template information is constructed in combination with the attributes of the user on the PIAA task, and is included in the training set and the test set.

[0135] In addition, the aesthetic score is used to divide the belonging category, for example, if the aesthetic score is 1 to 2 points, it belongs to the 0th category, 2 to 3 points belongs to the 1st category, and so on. It depends on the score segment of the dataset.

[0136] The content of each dataset in step 1 is different;

[0137] For the VIAA task: there is an scv file responsible for storing the image path of the training data, scoring the aesthetic score, and the category of the aesthetic score;

[0138] For the MIAA task: there is an scv file responsible for storing the image path of the training data and the corresponding image text description information, scoring the aesthetic score, and the category of the aesthetic score;

[0139] For the FIAA task: there is an scv file responsible for storing the image path of the training data, scoring the aesthetic score and other attributes, and the category of the aesthetic score and other scores;

[0140] For PIAA task: there are 40 users, 80 csv files (train, test), take train as an example: including: the path of 100 training images, the corresponding text information constructed by SMP mode, the user's aesthetic score, and the category of aesthetic score.

[0141] The method of step 1 carries out IAA training by combining user information multi-modal learning and multi-character prompts, and the text template is constructed based on the SMP proposed in the application, wherein the text template specifically includes five personality characteristics.

[0142] Because the original csv file does not have text data on the PIAA task, we need to construct the text template SMP, and take the text template as a component part of the training data. This process is also part of constructing the training set and the test set.

[0143] Step 2, construct the sub-modules of the general image aesthetic evaluation framework:

[0144] 1. Visual encoder E based on SSM v , Vmamba-Tiny is used as the backbone network, and the output image feature is obtained after the classification head output layer of the model, which is 768*49 dimension, and the average value pooling is taken on the output feature to obtain the feature of 768 dimension.

[0145] 2. Text encoder E of user information t , first, the input comment is finely segmented by using (Word Piece Tokenizer, WPT) to generate a word sequence of length n. Then a special mark [cls] is added at the beginning of the sequence to represent the global feature. The corresponding position encoding is added, and it is converted into an embedding vector through linear mapping. These embedding vectors are then input into a (Bidirectional Encoder Representations from Transformers, BERT) model containing 12 Transformer modules, and finally a variable size 768*n dimension vector is output, n represents the length of the word. The dimension 768 feature cls of the output is intercepted as the output.

[0146] 3. Adaptive average pooling AAP, the input sequence dimension is compressed to 1 dimension through average pooling.

[0147] 4. Cross information fusion module CMF, the visual feature f v and the text feature f t extracted from the image and text encoders are used. The specific structure is shown in Figure 1 .

[0148] Fv

batch, 768, 49

batch, 768

batch, x, 768

batch, 768

[0149] In the process of information fusion (this is just a statement, which means f t and f v Feature fusion is performed by cross attention) First, the visual features and text features are normalized by layer normalization. Then, the normalized visual features are input into the cross attention layer CA, where f v After mapping, a 768-dimensional vector Q is obtained, f t After linear mapping, a 768x49-dimensional vector K, V is obtained. The calculation of cross attention is as follows:

[0150]

[0151] The output of the cross attention CA is connected with the original visual features by residual connection to obtain the intermediate feature Then the intermediate feature is normalized again by layer normalization, and then a 768-dimensional feature is obtained by full connection layer. The 768-dimensional cross information fusion output feature f is obtained by connecting the intermediate feature with the output feature.

[0152]

[0153] 5. Feature adapter First, the 768-dimensional feature f is received. For a specific i-th task, f is nonlinearly mapped to a 192-dimensional feature, which is mapped back to a 768-dimensional feature after GELU activation function. The output feature is obtained by connecting the previous image feature with the 768-dimensional feature by residual connection. Since the output feature of the multi-scale adapter of the model during the training process needs to be regressed by the local branch adapter, the multi-scale adapter is called the global branch.

[0154]

[0155] 6. Task adapter φ: based on the MBA of the present application, in order to fully balance the learning between attributes, a number of local branches are added to the feature adapter to form the task adapter. The feature adapter outputs a 768-dimensional feature f, which needs different heads to adapt the learning process, mainly including three local branches:

[0156] Score regression branch (score prediction head): map the input features to 256-dimensional features, pass through the relu activation function, and then nonlinearly map to get the predicted score.

[0157] Attribute prediction branch (classification head): map the input features to 256-dimensional features, pass through the relu activation function, and then nonlinearly map to get 4-dimensional score features, and then pass through the softmax normalization to get the predicted category.

[0158] Distribution prediction branch (distribution prediction head): map the input features to 256-dimensional features, pass through the relu activation function, and then linearly map to get 10-dimensional score features, and then pass through the softmax normalization to get the predicted distribution.

[0159] Different branches are selected according to different data sets, and the specific settings are as follows:

[0160] Using the distribution prediction branch, L EMD needs to be increased.

[0161] Using the attribute prediction branch, L Ba1-CE needs to be increased.

[0162] Using the score regression branch, L MSE needs to be increased.

[0163] L dst = L EMD + γL MSE + λL Ba1-CE ,

[0164] L reg = L MSE + λL Ba1-CE ,

[0165] There are two deployment methods in total. For PARA, AADB, and TAD66K data sets, L reg is used.

[0166] For AVA and PHOTONET, L dst is used.

[0167] The construction method of the module in step 2 includes efficient combination of global features and local features on the IAA task, and the first use of the Vmamba encoder, the cross-attention cross-information fusion module CMF, the multi-task balanced adapter φ, and the combination of multiple prediction heads corresponding to each adapter.

[0168] The Vmamba encoder based on the state space can effectively balance the global information and local information of the image, which is an efficient image encoder.

[0169] The balancing adapter φ can effectively balance the learning process of different tasks (different prediction heads), enabling the model to learn efficiently.

[0170] The cross-information fusion module CMF can effectively fuse user features and image features.

[0171] Step 3: Construct a specific, general or personalized aesthetic evaluation model based on the specific task.

[0172] The general framework for IAA tasks proposed in this invention is shown in Figure 2:

[0173] The image is processed by an image encoder to obtain vector F. v The text is processed by a BERT encoder to obtain F. t In VIAA and FIAA missions, F v [batch, 768, 49] f is obtained after adaptive pooling. v

batch, 768

[0174] Figure 2 The adapter section in the middle right half is already... Figure 1 There is a specific structure. Single-task learning is used by VIAA, MIAA, and PIAA, while multi-task learning is only used by FIAA.

[0175] The model receives three types of input: images, image description text, and user information. Depending on the specific task, the model will be constructed using different methods. Furthermore, depending on the predicted attribute, the model will employ different adapters.

[0176] Specific model selection, such as Figure 3 As shown.

[0177] The AesMmba-V model is used for the purely visual aesthetic evaluation task VIAA. The input is an image, and the output is an aesthetic score. An image encoder E is employed. v Single global branch And several local branches.

[0178] The AesMmba-M model is used for the visual-text fusion aesthetic evaluation task MIAA. The input consists of images and image descriptions, and the output is an aesthetic score. An image encoder E is employed. v Text Encoder Et a single global branch and several local branches.

[0179] AesMmba-F model: for the task of fine-grained image aesthetic evaluation FIAA, the input is an image, and the scores of multiple attributes need to be predicted, then an image encoder E v multiple global branches and several local branches.

[0180] AesMmba-P model: for the task of personalized image aesthetic evaluation PIAA, the input is an image and user description information, and an image encoder E v a text encoder E t a cross-modal fusion module CMF, multiple global branches and several local branches.

[0181] wherein the number of global branches i depends on the number of attribute data sets, the type of local branches, and the type of single attribute score, distribution type or average type. Different aesthetic evaluation tasks are carried out by constructing the model M in the above manner.

[0182] The specific construction method is different according to the type of task and the type of data set. Specific example: when the AVA data set carries out the MIAA task, the AesMmba-M model framework is adopted;

[0183] More detailed structure:

[0184] Cross-modal fusion module CMF is adopted because it is an MIAA task

[0185] Single task learning mode because the AVA data set only has an aesthetic score

[0186] Classification head, regression head and distribution prediction head are adopted because the aesthetic score of the AVA data set is a distribution, and there is also an average score and a category.

[0187] In this embodiment, the PARA data set is adopted to carry out the VIAA task, and the model adopts an image encoder E v a single global branch and several local branches are constructed. The number of global branches is 1, and the local branches select score prediction branches and attribute prediction branches.

[0188] The method of constructing a general or personalized aesthetic evaluation model for a specific task in step 3. It includes the following four kinds: module composition of pure visual aesthetic evaluation, module composition of visual and text multi-modal aesthetic evaluation, module composition of pure visual multi-attribute aesthetic evaluation, and module composition of user personalized aesthetic evaluation.

[0189] Currently there is no framework in the field of IAA that can handle all tasks of VIAA, MIAA, FIAA, PIAA at the same time. The present method can achieve this by flexibly calling various modules involved in step 2, and constructing the model in step 3 according to the different tasks and different data sets.

[0190] Step 4, iterative training of network model M:

[0191] Step 4.1, initialize the model in the training stage.

[0192] Initialize the number of iterations t, the maximum number of iterations T, and for different models selected in step 3, let the current multi-granularity network-based no-reference image quality assessment network model be M t In this embodiment, T = 50. M is a model for carrying out the VIAA task. Initialize t = 1, M t = M.

[0193] Step 4.2, in the training stage, randomly select batch_size training samples from the training sample set S train as the input of the network model M t without replacement; in this embodiment, batch_size = 64.

[0194] Step 4.3, construct the loss function: there are three kinds of loss functions in the present application:

[0195] Mean squared error loss function: where represents the predicted score, s i represents the true score, and n represents the number of samples.

[0196] Balanced cross-entropy loss function: where n l represents the number of instances, C i represents the class label, and K represents the number of tasks. It improves the importance of rare classes in the learning process. Therefore, this auxiliary quantity table classification task can improve the model's ability to correctly evaluate long-tail data. In this embodiment, C i = 0, 1, 2, 3, K = 1.

[0197] Global movement distance loss function: where a is the predicted label, p is the correct label of the data set, CDF is the combined distribution function, and Len is the number of interval blocks. In the inference stage, the aesthetic evaluation score is calculated according to the predicted distribution.

[0198] For different data sets, different training strategies need to be carried out: L dstThe distribution loss is carried out. For the AADB, PARA and other data sets containing real scores, the L reg The regression loss is carried out.

[0199] L dst = L EMD + γL MSE + λL Ba1-CE ,

[0200] L reg = L MSE + λL Ba1-CE ,

[0201] For each training sample, the predicted score corresponding to the training sample is first calculated, and the score label corresponding to the training sample is calculated according to the different data sets. The weight parameters of each layer of the network model S t are updated.

[0202] In this embodiment, the PARA data set is used, γ and λ are set to 10 and 0.001 respectively, and L reg is used as the loss function. The AdamW optimization algorithm is used, and β1=0.9, β2=0.99. Except for the learning rate, the remaining parameters use the default value. In order to ensure the convergence speed and learning accuracy of the model, the learning rate strategy of cosine annealing update is adopted in the training process. In this embodiment, the initial learning rate is set to 1×10-4, and the batch size is 64.

[0203] In the PIAA task, the initial learning rate is set to 4×10-5, and the batch size is 10 and 50 respectively under the training number setting of 10 or 100.

[0204] Step 4.4, judge whether the training sample set S train has been traversed, if yes, execute step 4.5, otherwise, execute steps 4.2 and 4.3;

[0205] Step 4.5, judge whether t=T is true, if yes, get the trained multi-granularity network based on the no-reference image quality evaluation network model M * =M t , otherwise, let t=t+1, and execute steps 4.3 and 4.4. For the training of PIAA task, each user trains a separate model. i=1, j=1, 2,..., 40 (randomly select 40 users) to obtain the training network M i,j , i=i+1. Continue to execute step 4.1 until the model M 10,j is obtained. A total of 400 models are trained. In this embodiment, the PIAA training is not carried out.

[0206] The step 4 is a training phase, and training of the GIAA (VIAA, FIAA, MIAA) follows a conventional procedure to carry out training, and first training and then verification in an epoch;

[0207] For the PIAA, the official training method proposed in the para data set paper is followed:

[0208] 40 users, and one model is trained for each user, that is, 40 times of training.

[0209] For a specific user id: frwet434adsa:

[0210] For each user, 10 times of training are also carried out respectively, and final models M1, M2 to M10 are obtained. For 40 users, 10 final models are obtained for each user, that is, 400 final models. The average value of the indicators PLCC and SRCC in the test phase is reported as the final result.

[0211] In the training process of step 4, different loss function combination methods are adopted for different data sets, L dist is adopted for the AVA data set, and L reg is adopted for the PARA data set. In addition, in order to solve the imbalance problem in the IAA, the MBA proposed in the present application is introduced, and an adapter and a balanced cross-entropy loss function are introduced in the IAA multi-attribute prediction.

[0212] Step 5, obtain the aesthetic quality evaluation score prediction result:

[0213] The test sample set S test is input into the trained network model M * for forward inference to obtain the quality prediction score of each test sample. For the data set of regression prediction, the prediction output is the score regression branch output; for the data set of distribution prediction, the prediction output is a weighted combination of the distribution prediction output. For the PIAA task, the M i is subjected to forward inference, and the average value of the final evaluation indicators is taken to verify the model effect.

[0214] In the present embodiment, the VIAA task is carried out, and the M * is subjected to forward inference. The technical effects of the present application are described below in combination with a simulation experiment:

[0215] 1. Simulation conditions and contents:

[0216] The hardware platform of the simulation experiment of the present application is: the graphics card is NVIDIA GeForce RTX4090;

[0217] The software platform of the simulation experiment of the application is: Ubuntu 20.04, Pytorch1.13.1+cu117, Python3.8.19.

[0218] 2. Simulation experiment: the simulation experiment of the application adopts three indexes of accuracy rate, Spearman Rank-order Correlation Coefficient (SRCC) and Pearson Linear Correlation Coefficient (PLCC) to respectively judge the image aesthetic evaluation effects of the application and the prior art. Specifically, the prior art and the application respectively select n samples from the test sample set S test to perform image aesthetic evaluation, output the predicted scores, and calculate the values of ACC, SROCC and PLCC from the predicted scores of the samples and the label scores corresponding to the samples. Wherein, the above indexes are calculated according to the following formula:

[0219]

[0220] ACC∈[0,1]. The higher the value is, the smaller the overall score gap between the true score and the predicted score is. In the calculation, the predicted and true scores are first normalized. TP represents that the true score and the predicted score of the corresponding picture are both greater than or equal to 0.5, FP represents that the true and predicted scores are both less than 0.5, and S represents the total number of test samples.

[0221]

[0222] SRCC∈[-1,1]. The higher the value is, the more correct the evaluation result of the aesthetic evaluation method being judged can reflect the degree of good or bad of the image. Wherein, n represents the total number of images, be e represents the difference between the sequence number of the aesthetic predicted score of the e-th image in the predicted score sequence of n images and the sequence number of the label score.

[0223]

[0224] PLCC∈[-1,1]. The higher the value is, the closer the result of the aesthetic evaluation method being judged is to the subjective aesthetic evaluation score of human. s r represents the quality predicted score of the r-th image, represents the average value of the quality predicted scores of all images, p r represents the subjective evaluation score of the r-th image, represents the average value of the subjective evaluation scores of all images.

[0225] To fully verify the performance of the present application, the simulation results are not limited to the embodiments, experiments are carried out on AVA, PARA, TAN66K, AADB, Photo.Net data sets in GIAA, PIAA tasks, and compared with advanced methods in various fields, and the final results are shown in Tables 1, 2, 3 and 4.

[0226]

[0227] Table 1. Comparison table of VIAA of PARA single task of the present application and other methods. * indicates that the model is trained using external IAA data sets. Except for the learning rate, the rest of the parameters use the default value.

[0228] Table 1 shows the performance of Aesmamba-V in VIAA and the comparison of existing algorithms. Note that some previous methods (indicated by *) such as TANet use external aesthetic-related data sets (such as FLICKR-AES and LAION-5B) for pre-training. In contrast, all other methods only use the standard training set on each data set. In all these data sets, the present model achieves the best or highly competitive performance.

[0229]

[0230] Table 2. MIAA of multi-modal input aesthetic quality evaluation of AVA and Photo.Net single task

[0231] Table 2 shows that AesMamba-M method achieves the best performance in all indicators. Compared with the previous optimal performance, AesMamba-M achieves a performance improvement of 2.9, 0.037 and 0.040 in ACC, PLCC and SRCC on AVA, and a performance improvement of 0.05 / 0.04 in PLCC / SRCC on Photo.Net. In addition, on these two data sets. This obvious advantage proves the effectiveness of the text template in IAA and the cross-modal fusion CMF used by the present application in the integrated visual language representation.

[0232]

[0233]

[0234] Table 3. Multi-attribute image aesthetic quality evaluation FIAA of multi-task

[0235] Table 3 shows the PLCC and SRCC for each attribute, as well as the average of the metrics across all attributes. It is clear that the model built with the present application consistently achieves the best PLCC and SRCC values, across all attributes. Moreover, the base model of the present application, namely VMamba-Tiny, consistently outperforms all other advanced visual backbones, whether CNNs or Transformers, demonstrating the effectiveness of using VMamba to learn effective aesthetic representations.

[0236]

[0237]

[0238] Table 4. Personalized Aesthetic Quality Assessment (PIAA) for single-task

[0239] The average and standard deviation of the PLCC / SRCC are shown in Table 4. It is clear that AesMamba-P achieves the best performance in both the 10-train and 100-train tasks, with the average PLCC and SRCC values reaching the highest performance. This advantage demonstrates the effectiveness of using multiple attributes of the user to construct a textual template for PIAA.

[0240] From the above tables, it can be seen that the method of the present application exhibits outstanding advantages, whether verified on various data sets or on multiple IAA tasks, both indicators have achieved very competitive results, which proves the effectiveness of the present application on all IAA tasks.

[0241] Cross-dataset verification experiment:

[0242] The results obtained by testing the model trained on one dataset on other datasets.

[0243] The final summary is shown in Figures 6 and 7.

[0244]

[0245]

[0246] Table 5. VIAA cross-dataset experiment, the vertical list indicates the test dataset

[0247]

[0248] Table 6. MIAA cross-dataset experiment, the vertical list indicates the test dataset.

[0249] The present application is used to solve the difficulty of combining local and global image features in the prior art, and the present application explores the use of Vmamba to effectively learn global visual representation. In addition, in order to alleviate the interference between tasks and balance the instability of multi-task training, the present application proposes a multi-task balanced adaptation (MBA), for multi-task learning, a variety of adapters are designed, an auxiliary classification task is used for each attribute score, and balanced cross-entropy loss is used for training to strengthen the learning focus on instances. Finally, in view of the problem that the model is difficult to design efficiently and flexibly, the present application relates to various modules such as related adapters and cross-information fusion, which can be flexibly combined to adapt to various IAA tasks. For PIAA, the present application proposes subject multi-character prompts (SMP), a text prompt template is designed based on multiple subject characters of the user; then the personalized aesthetic score is predicted according to the image and the text template. In this way, PIAA is expressed as a multi-modal learning task. In addition, the text prompt allows flexible combination.

Claims

1. A general image aesthetic evaluation method based on a state-space model, characterized in that, Includes the following steps; Step 1: Obtain the training sample set S according to the dataset and task type. train and test set S test ; Step 2, constructing sub-modules of a general image aesthetic evaluation framework: Step 3: Construct a specific, general or personalized aesthetic evaluation model M by building upon the different sub-modules from Step 2, and combining them with the specific task. Step 4: Iteratively train the network model M: Step 5: Obtain the predicted aesthetic quality evaluation score using the model trained iteratively in Step 4; Step 2 specifically involves: Step (1). SSM-based visual encoder E v Vmamba-Tiny is used as the backbone network. The output image features are obtained after the classification head output layer of the model, and the output features are subjected to average pooling. Step (2). Text encoder E for user information t First, the input comments are segmented into words using WordPieceTokenizer (WPT) to generate a word sequence of length n. Then, a special token [cls] is added to the beginning of the sequence to represent global features. The corresponding positional encoding is added to it and it is converted into an embedding vector through linear mapping. These embedding vectors are then input into a model containing 12 Transformer modules. Finally, the output is a vector with a variable sequence length dimension n, where n represents the word length. The 0th dimension feature cls is truncated as the output. Step (3). Adaptive Average Pooling (AAP) is used to compress the input sequence dimension to 1 dimension through average pooling. Step (4). Cross-information fusion module CMF, utilizing visual features f extracted from the image and text encoders respectively. v and text features f t ; Step (5). Feature adapter: First, receive feature f. For a specific i-th task, perform non-linear mapping on f to reduce its dimensionality. After passing through the GELU activation function, map it back to the feature of the received dimension. Perform residual connection with the previous image features to obtain the output feature. The multi-scale adapter is called the global branch; Step (6). Task adapter: Based on the feature adapter, add several local branches to form a task adapter; Step 3 specifically involves: The model receives three types of input: images, image description text, and user information. Depending on the specific task, the model will be constructed in different ways. Depending on the predicted attribute, the model will also use different adapters. Specific model selection: The AesMmba-V model is used to perform the purely visual aesthetic evaluation task VIAA. The input is an image, and the output is an aesthetic score. It employs an image encoder, E. v Single global branch and several local branches; The AesMmba-M model is used for the visual-text fusion aesthetic evaluation task MIAA. The input consists of images and image descriptions, and the output is an aesthetic score. It employs an image encoder, E. v Text Encoder E t Information Fusion Module (CMF), Single Global Branch and several local branches; The AesMmba-F model is used for the FIAA (Financial Assessment of Artificial Intelligence) task, a purely visual fine-grained image aesthetic evaluation task. The input is an image, and the model needs to predict scores for multiple attributes. Therefore, an image encoder E is employed. v Multiple global branches and several local branches; AesMmba-P model: Conducts the Personalized Visual Aesthetics Evaluation Task (PIAA) with inputs of images and user descriptions, employing an image encoder E. v Text Encoder E t Information Fusion Module (CMF), multiple global branches and several local branches; The number of global branches i depends on the number of attributes in the dataset, and the type of local branches depends on the type of single attribute score, whether it is distributed or average. Different aesthetic evaluation tasks are carried out, and the model M is constructed in the above manner. Step 4 specifically involves: Step 4.1, Initialize the model during the training phase; Let the initial number of iterations be t, and the maximum number of iterations be T. For the different models selected in step 3, let the current no-reference image quality assessment network model based on multi-granularity networks be M. t Let M be the model for carrying out the VIAA task. Initialize it by setting t=1, M t =M; Step 4.2, Training Phase: Set up the training sample set S train Batch_size training samples are randomly selected without replacement as the network model M. t Input; Step 4.3, Construct the loss function: There are three types of loss functions: Mean squared error loss function: in Represents the predicted score, s i Represents the true score, and n represents the sample size; Balanced cross-entropy loss function: Where n l Represents the number of instances, C i This represents the category label, and K represents the number of tasks; Global movement distance loss function: Where a is the predicted label, p is the correct label of the dataset, CDF is the combined distribution function, and Len is the number of interval blocks. During the inference phase, the aesthetic evaluation score is calculated based on the predicted distribution. Different training strategies are required for different datasets: AVA and Photo.Net datasets, which include the aesthetic score distribution for each image, employ L... dst We employ distributed loss; for AADB, PARA uses L... reg Perform regression loss. THE dst =L EMD +γL MSE +λL Ba1-CE , THE reg =L MSE +λL Ba1-CE , First, the predicted score and score label for each training sample are calculated, and then the loss function is calculated according to the different datasets. For network model S... t The weight parameters of each layer are updated; Step 4.4: Determine whether the training sample set S has been traversed. train If yes, proceed to step 4.5; otherwise, proceed to steps 4.2 and 4.

3. Step 4.5: Determine if t = T holds true. If so, obtain the trained no-reference image quality assessment network model M based on multi-granularity networks. * =M t Otherwise, let t = t + 1, and execute steps 4.3 and 4.4; Step 5 specifically involves: Test sample set S test The trained network model M * The input is used for forward inference to obtain the quality prediction score for each test sample. For regression prediction datasets, the prediction output is the score regression branch output; for distribution prediction datasets, the prediction output is a weighted combination of the distribution prediction outputs. For the PIAA task, the output is calculated based on M... i Forward inference is performed, and the final evaluation metric is the average value to verify the model's effectiveness.

2. The general image aesthetic evaluation method based on a state-space model according to claim 1, characterized in that, The dataset in step 1 includes: Experiments were conducted on the following benchmark datasets: AVA, TAD66K, PARA, AADB, and Photo.Net. AVA and Photo.Net include the aesthetic score distribution for each image; PARA and AADB include average scores; The PARA dataset contains images, image text information, user profile information, and user ratings of the images. The dataset does not have a fixed task type. Based on the different data and content, the following experiments were conducted: For the AADB dataset, only the VIAA experiment was conducted; For the TAD66K dataset, only the VIAA experiment was conducted; For the AVA dataset, conduct VIAA and MIAA experiments; For the PARA dataset, experiments were conducted using VIAA, MIAA, FIAA, and PIAA. For the PhotoNet dataset, we conducted VIAA and MIAA experiments.

3. The general image aesthetic evaluation method based on a state-space model according to claim 2, characterized in that, In GIAA-related tasks such as VIAA, FIAA, and multimodal image aesthetic evaluation, the official training and testing data splitting method was selected. In the PIAA task, users are first randomly selected from the annotators as experimental subjects, and multiple trials are conducted. The average value of the evaluation index is calculated as the final experimental result. Subsequently, the aesthetic score prediction model is fine-tuned. The aesthetic score prediction model was tested on the PARA dataset using VIAA, MIAA, FIAA, and PIAA. in: The GIAA's commonly used aesthetic quality assessment includes VIAA, MIAA, and FIAA; VIAA is a purely visual aesthetic evaluation task where the model processes the input image and predicts the corresponding aesthetic quality score. MIAA, a multimodal aesthetic evaluation task, involves a model that processes input images and related text information to predict the aesthetic quality score of the corresponding images. FIAA is a purely visual aesthetic evaluation task. The model processes the input image and predicts the corresponding multi-attribute quality score of the image, including aesthetics, composition, color, and depth of field score. PIAA Personalized Aesthetic Quality Assessment uses user information, along with the model's processing of the input image and related text information, to predict the user's aesthetic quality score for that image.

4. The general image aesthetic evaluation method based on a state-space model according to claim 3, characterized in that, For different tasks, the elements of the sample set are not the same. The training sample set and the test sample set have the following three types of input: images, image descriptions, and user information. For images: In the fine-grained aesthetic evaluation experiment, a uniform data preprocessing method was implemented for the images in the PARA dataset; different preprocessing strategies were adopted in the training and testing phases. In the training phase, data augmentation techniques were used. First, the size of each image was adjusted using bicubic interpolation, and uniform resampling was performed. Then, image blocks of pixels were randomly cropped from the images to adapt to the input requirements of the model, and the images were randomly horizontally flipped. For image descriptions: Select image description information from the PARA dataset; For user information: SMP: Multi-character prompts for users, which build text templates based on user information; Make full use of the three user information provided in the PARA dataset: artistic experience, photographic experience, and personality. Specifically, both aesthetic experience and photographic experience include four levels: "beginner", "competent", "skilled", and "expert". Design corresponding text prompts and incorporate this information into the model; "My artistic experience is{art_exp}." and "My Photographic experience is{photo_exp}." Where art_exp and photo_exp represent the user's corresponding experience level, and the personality traits include five aspects: "Conscientiousness O", "Affinity C", "Extroversion E", "Openness A", and "Mentality N". Given a user, the corresponding text prompts should follow the template below: "In the Big-Five personality traits test, my scores are as follows: openness score is {O}, conscientiousness score is {C}, extroversion score is {E}, agreeableness score is {A}, and Neuroticism score is {N}." All the above text templates can be integrated to represent a user's personal information. The PARA dataset is used for the VIAA task. The input information for the training and test sets consists only of images, and the output is an aesthetic score.

5. The general image aesthetic evaluation method based on a state-space model according to claim 1, characterized in that, In step (4), during the information fusion process, visual features and text features are first subjected to layer normalization. Then, the normalized visual features are input into the cross-attention layer (CA), where f... v After mapping, we obtain vectors Q and f. t After linear mapping, vectors K and V are obtained. The cross-attention is calculated as follows: The output of cross-attention (CA) is residually connected with the original visual features to obtain intermediate features. The intermediate features are then normalized again, and then passed through a fully connected layer to obtain the feature. The feature is then residually connected with the intermediate features to obtain the cross-information fusion output feature f. The feature adapter The output feature f requires different heads to adapt to the learning process, mainly involving three local branches: Score regression branch: The input features are mapped, then passed through the ReLU activation function and then nonlinearly mapped to obtain the predicted score; Attribute prediction branch: The input features are mapped, then passed through the ReLU activation function and non-linearly mapped to obtain 4-dimensional score features, which are then normalized by softmax to obtain the predicted category; Distribution prediction branch: The input features are mapped, then linearly mapped through the ReLU activation function to obtain the score distribution features, and then normalized by softmax to obtain the prediction distribution.

6. The general image aesthetic evaluation method based on a state-space model according to claim 5, characterized in that, Different branches are selected based on the specific dataset provided, as detailed below: Using distributed prediction branching requires adding L EMD ; Using attribute prediction branches requires adding L Ba1-CE ; Using the fractional regression branch requires adding L... MSE ; THE dst =L EMD +γL MSE +λL Ba1-CE , THE reg =L MSE +λL Ba1-CE , There are two deployment methods. For the PARA, AADB, and TAD66K datasets, L is used. reg Way; For AVA and PHOTONET, use L dst Way.

Images