A method for image aesthetic assessment based on self-supervised pre-training

Abstract

This invention discloses a method for image aesthetic assessment based on self-supervised pre-training. The method first inputs an unlabeled dataset to obtain an enhanced view. The enhanced view is then input into an aesthetic representation extraction network consisting of an encoder and an expander. The aesthetic representation extraction network is then trained using a self-supervised method to obtain a trained aesthetic representation extraction network encoder. The self-supervised trained encoder is then combined with a regression and classification head to form a teacher network model. The teacher network model is fine-tuned using a labeled dataset. Finally, self-training is used to address overfitting. This method addresses the drawback of supervised learning methods for image aesthetic assessment, which require a large amount of labeled data, effectively reducing overfitting and improving network performance.

Description

Technical Field

[0001] The present invention belongs to the field of deep learning technology, and specifically relates to an image aesthetics evaluation method. Background Art

[0002] Image aesthetic assessment is a long-standing problem in image processing and computer vision, which aims to model the aesthetic information of images and ultimately enable computers to automatically provide objective evaluations of the aesthetic quality of images. The great challenge of this task involves predicting aesthetic ratings or their distribution, which is a subjective opinion that is subject to many influences (photographic rhythm patterns, personal emotions and preferences) that interfere with human judgment. Deep learning is very suitable for this task, but existing deep learning methods are mostly supervised and require a large labeled dataset. However, in practice, collecting large-scale basic datasets is expensive, and using supervised learning can only partially utilize unlabeled data. The document "Adaptive Computational Aesthetic Assessment of Ink Paintings Based on Deep Learning, Journal of Computer-Aided Design and Graphics, 2021, Vol33(09), pp 1349-1360" discloses an adaptive computational aesthetic assessment method for ink paintings based on deep learning. This method first constructs a benchmark dataset for aesthetic evaluation of ink painting images. It then extracts global and local image patches as multiple inputs based on the aesthetic criteria of ink painting, and designs a multi-view parallel deep convolutional neural network to extract deep aesthetic features. Finally, based on the subject matter query mechanism of ink painting, an adaptive deep aesthetic evaluation model is constructed. The method described in the literature is an adaptive computational aesthetic evaluation framework based on deep learning, but its parameter adaptability still has shortcomings, making it less practical. Furthermore, this method requires the construction of a benchmark dataset for aesthetic evaluation of ink painting images for training. However, due to the difficulty in collecting data, it is difficult to obtain sufficient training data, which limits its practical application. Summary of the Invention

[0003] To overcome the shortcomings of existing technologies, this invention provides an image aesthetic assessment method based on self-supervised pre-training. The method first inputs an unlabeled dataset to generate enhanced views. The enhanced views are then fed into an aesthetic representation extraction network consisting of an encoder and an expander. The aesthetic representation extraction network is then trained using a self-supervised method to obtain a trained aesthetic representation extraction network encoder. The self-supervised trained encoder is then combined with a regression and classification head to form a teacher network model. The teacher network model is fine-tuned using a labeled dataset. Finally, self-training is used to address overfitting. This method addresses the drawback of supervised learning methods for image aesthetic assessment, which require a large amount of labeled data, effectively reducing overfitting and improving network performance.

[0004] The technical solution adopted by the present invention to solve the technical problem includes the following steps:

[0005] Step 1: Input the unlabeled dataset U and obtain the enhanced views t, t′;

[0006] Input unlabeled image (t i ,·)∈U and sample two transformations from distribution T to generate enhanced views t and t′; distribution T is a probability distribution generated by a data augmentation strategy, which contains two transformation functions, one of which is an affine transformation function T a , the other is a non-rigid transformation function T b , these two transformation functions are randomly sampled from T, and the process of generating enhanced views t and t′ using the transformation function is as follows:

[0007] t=T a (t i ),t′=T b (t i ) (1)

[0008] Step 2: Input the enhanced views t, t′ into the encoder and expander to obtain the output z, z′;

[0009] Views t and t′ are input into the encoder, and then mapped into embedding outputs z and z′ by the expander; the encoder and expander form the aesthetic representation extraction network;

[0010] Step 3: Use a self-supervised method to train the aesthetic representation extraction network to obtain a trained aesthetic representation extraction network encoder;

[0011] The self-supervised approach is to minimize the distance between two embedding outputs z and z′ from the same image, keeping the variance of each embedding variable above a set threshold, while the covariance between pairwise embedding variables is attracted to zero, making the variables de-correlated; the overall loss function is a weighted average of the invariance s(·), variance v(·), and covariance c(·) terms, as shown below:

[0012] l(Z,Z′)=λs(Z,Z′)+μ[v(Z)+v(Z′)]+γ[c(Z)+c(Z′)] (2)

[0013] Where λ, σ and γ are weights that control the importance of each term in the loss, Z = [z1,...,z n ] and Z′=[z1′,...,z′ n ] are two batches of size n;

[0014] The overall objective function obtained on all images in the unlabeled dataset U is given by:

[0015] L=∑ I∈U ∑ t,t′∈T l(Z I ,ZI ′) (3)

[0016] Where Z I With Z I ′ is the embedding output of the aesthetic representation extraction network for the two batches of image I corresponding to the transformations t and t′;

[0017] After completing self-supervised pre-training, remove the expander and only keep the encoder;

[0018] Step 4: Add the self-supervised trained encoder to the regression classification head to form the teacher network model f t ;

[0019] Teacher network model f t The linear regression head consists of two branches, the first branch consists of a linear layer, and the second branch consists of a linear layer with an activation function; the outputs of the two branches are added and input into the softmax function to obtain the final output f t ;

[0020] Step 5: Fine-tune the teacher network model f using the labeled dataset L t ;

[0021] Use the metric method - Earth Mover Distance EMD to fine-tune the model on the labeled dataset L and do not freeze the backbone network, that is, fine-tune the teacher network model f t ; Given the predicted value And the true score distribution y with N ordered categories, the EMD loss function is expressed as:

[0022]

[0023] Where, CDF y (k) and is the cumulative distribution function, r is the coefficient used to penalize the Euclidean distance between two cumulative distribution functions;

[0024] Step 6: Solve the overfitting problem through self-training;

[0025] The self-training is a process that combines supervised training with knowledge distillation; using the fine-tuning network f t As a teacher to generate the student network model f s The label of the fine-tuned network f t Student network model f s Perform knowledge distillation; the student model is initialized by the pre-trained weights, and the network structure is the same as the teacher network f t Same; given unlabeled data (u i ,·)∈U, the output of the teacher network and student network model is defined as and

[0026] Minimize distillation losses as follows:

[0027]

[0028] Where H(·) represents EMD loss, σ is the softmax function, and T represents the temperature parameter in knowledge distillation.

[0029] Student Network Modelf s Learning not only from the teacher model but also from labeled data, the total loss is defined as:

[0030]

[0031] Where λ is the weight of supervision loss and distillation loss, is the labeled data from the student model (x j ,y j )∈U’s predicted value.

[0032] Preferably, the T a Including rotation, scaling, translation affine transformation, T b Includes non-rigid transformations such as bending, twisting, and stretching.

[0033] Preferably, the encoder is a pre-trained model, a ResNet-50 network pre-trained by the ImageNet dataset; and the expander consists of three fully connected layers.

[0034] Preferably, the Earth Mover Distance (EMD) is a measurement method for measuring the difference between two probability distributions, and the EMD is used to measure the similarity between two images.

[0035] Preferably, r=2.

[0036] The beneficial effects of the present invention are as follows:

[0037] This method aims to efficiently predict accurate aesthetic scores using both labeled and large amounts of unlabeled data, addressing the drawback of supervised learning methods for image aesthetic assessment, which require large amounts of labeled data. It first uses unlabeled data for pre-training to obtain aesthetic representations that accurately describe internal and external features, serving as prior knowledge for supervised training. It then uses a small amount of labeled data for fine-tuning. To mitigate overfitting in supervised training with limited labeled data, this method proposes a two-branch structured regression head to learn the aesthetic representations. It also combines knowledge distillation with supervised learning, performing self-training to add additional pseudo-labels generated by a teacher network in knowledge distillation to the limited amount of labeled data in supervised learning. This guidance from the teacher network serves as additional features to enhance the student network's understanding of the limited labels, effectively reducing overfitting and improving network performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 Flowchart of the method of the present invention.

[0039] Figure 2 Schematic diagram of an unlabeled dataset U according to an embodiment of the present invention.

[0040] Figure 3 FIG. 1 is a schematic diagram of a labeled dataset L according to an embodiment of the present invention.

[0041] Figure 4 The image aesthetic evaluation score predicted by the embodiment of the present invention. DETAILED DESCRIPTION

[0042] The present invention will be further described below with reference to the accompanying drawings and examples.

[0043] A method for image aesthetic assessment based on self-supervised pre-training, comprising the following steps:

[0044] Step 1: Input unlabeled data U and obtain enhanced views t, t′;

[0045] Input unlabeled image (t i ,·)∈U and sample two transformations from distribution T to generate enhanced views t and t′. Distribution T is a probability distribution generated by a data augmentation strategy, which contains two transformation functions, one of which is an affine transformation function T a , the other is a non-rigid transformation function T b , where T a Contains affine transformations such as rotation, scaling, and translation, T b It includes non-rigid transformations such as bending, twisting, and stretching. Both transformation functions are randomly sampled from T. The process of generating enhanced views t and t′ using the transformation function is as follows:

[0046] t=T a (t i ),t′=T b (t i )(1)

[0047] Step 2: Input the enhanced views t and t′ into the encoder and expander to obtain outputs z and z′.

[0048] Views t and t′ are first input into the encoder, and then mapped into the embedding outputs z and z′ by the expander. The encoder is a pre-trained model, a ResNet-50 network pre-trained on the ImageNet dataset. The expander consists of three fully connected layers. The aesthetic representation extraction network consists of the encoder and the expander.

[0049] Step 3: Use a self-supervised method to train the aesthetic representation extraction network to obtain a trained aesthetic representation extraction network encoder.

[0050] Using a self-supervised training method, the distance between two embedding outputs z and z′ from the same image is minimized. The variance of each embedding variable in the batch is kept above a threshold, and the covariance between pairwise embedding variables in the batch is attracted to zero, thereby decorrelating the variables. The overall loss function is a weighted average of the invariance s(·), variance v(·), and covariance c(·) terms, as shown below:

[0051] l(Z,Z′)=λs(Z,Z′)+μ[v(Z)+v(Z′)]+γ[c(Z0+c(Z′)] (2)

[0052] Where λ, σ and γ are weights that control the importance of each term in the loss, Z = [z1,...,z n ] and Z′=[z1′,...,z′ n ] are two batches of size n. The overall objective function obtained on all images on the unlabeled data U is given by:

[0053] L=∑ I∈U ∑ t,t′∈T l(Z I ,Z I ′) (3)

[0054] Where Z I With Z I ′ is the embedding output of the aesthetic representation extraction network for the two batches of image I corresponding to the transformations t and t′. After self-supervised pre-training, the present invention removes the expander and only retains the encoder.

[0055] Step 4: Add the self-supervised trained encoder to the regression classification head to form the teacher network model f t .

[0056] Fine-tuning is to adjust a pre-processing network of an uncertain task to a specific downstream task, so a regression head is needed to adjust the abstract aesthetic representation output by the backbone network to predict the aesthetic score. Since a single linear layer can only perform simple feature processing on the backbone output of complex features, it cannot accurately capture the exact score of each threshold. Therefore, the linear regression head of the teacher network consists of two branches, the first branch consists of a linear layer, and the second branch consists of a linear layer with an activation function. The outputs of the two branches are added and input into the softmax function to obtain the final output f t .

[0057] Step 5: Fine-tune the teacher network model f using labeled data L t .

[0058] Use the metric method - Earth Mover Distance (EMD) to fine-tune the model on the labeled data L without freezing the backbone network, that is, fine-tune the teacher network model f t EMD is a measure of the difference between two probability distributions. In computer vision and image processing, EMD is often used to measure the similarity between two images. and the true score distribution y with N ordered categories, the EMD loss function can be expressed as:

[0059]

[0060] Where, CDF y (k) and is the cumulative distribution function, r is the coefficient used to penalize the Euclidean distance between two cumulative distribution functions, and is set to 2.

[0061] Step 6: Use the self-training method proposed in this invention to solve the overfitting problem.

[0062] When using a small amount of labeled aesthetic data for supervised fine-tuning, severe overfitting occurs. This is because the network has difficulty learning more useful aesthetic features from the limited labeled data and ignores useful information in the unlabeled data. To reduce overfitting, the present invention proposes a self-training method to utilize the complex aesthetic features in both labeled data and a large amount of unlabeled data. In the present invention, "self-training" refers to the process of combining supervised training with knowledge distillation. The present invention uses a fine-tuned network f t As a teacher to generate the network f for training the student s The label of the fine-tuned network f t Student Network s Perform knowledge distillation. The student model is initialized by the pre-trained weights, and the network structure is the same as the teacher network f tThe same. Given unlabeled data (u i ,·)∈U, the output of the teacher network and student online class model is defined as and Minimize distillation losses as follows:

[0063]

[0064] Where H(·) represents EMD loss, σ is the softmax function, and T represents the temperature parameter in knowledge distillation. In addition, the student model f s It is also important to learn not only from the teacher model but also from the labeled data. This paper combines the two so that the student model can learn some important missing representations in a complementary way. Therefore, the total loss can be defined as:

[0065]

[0066] Where λ is the weight of supervision loss and distillation loss, is the labeled data from the student model (x j ,y j )∈U’s predicted value. Specific embodiment:

[0068] Take the unlabeled dataset U and the labeled dataset L as examples. Figure 2 、 Figure 3 As shown, the efficient image aesthetics evaluation method based on self-supervised pre-training in this embodiment includes the following steps:

[0069] Step 1: Input unlabeled data U and obtain enhanced views t, t′.

[0070] Input unlabeled image (t i ,·)∈U and sample two transformations from distribution T to generate enhanced views t and t′. Distribution T is a probability distribution generated by a data augmentation strategy, which contains two transformation functions, one of which is an affine transformation function T a , the other is a non-rigid transformation function T b , where T a Contains affine transformations such as rotation, scaling, and translation, T b It includes non-rigid transformations such as bending, twisting, and stretching. Both transformation functions are randomly sampled from T. The process of generating enhanced views t and t′ using the transformation function is as follows:

[0071] t=T a (t i ),t′=T b (t i )(1)

[0072] Step 2: Input the enhanced views t and t′ into the encoder and expander to obtain outputs z and z′.

[0073] Views t and t′ are first input into the encoder, and then mapped into the embedding outputs z and z′ by the expander. The encoder is a pre-trained model, a ResNet-50 network pre-trained on the ImageNet dataset. The expander consists of three fully connected layers. The aesthetic representation extraction network consists of the encoder and the expander.

[0074] Step 3: Use a self-supervised method to train the aesthetic representation extraction network to obtain a trained aesthetic representation extraction network encoder.

[0075] Using a self-supervised training method, the distance between two embedding outputs z and z′ from the same image is minimized. The variance of each embedding variable in the batch is kept above a threshold, and the covariance between pairwise embedding variables in the batch is attracted to zero, thereby decorrelating the variables. The overall loss function is a weighted average of the invariance s(·), variance v(·), and covariance c(·) terms, as shown below:

[0076] l(Z,Z′)=λs(Z,Z′)+μ[v(Z)+v(Z′)]+γ[c(Z)+c(Z′)] (2)

[0077] Where λ, μ, and γ are weights that control the importance of each term in the loss, Z = [z1,...,z n ] and Z′=[z1′,...,z′ n ] are two batches of size n. The overall objective function obtained on all images on the unlabeled data U is given by:

[0078] L=∑ I∈U ∑ t,t′∈T l(Z I ,Z I ′) (3)

[0079] Where Z I With Z I ′ is the embedding output of the aesthetic representation extraction network for the two batches of image I corresponding to the transformations t and t′. After self-supervised pre-training, the present invention removes the expander and only retains the encoder.

[0080] Step 4: Add the self-supervised trained encoder to the regression classification head to form the teacher network model f t .

[0081] Fine-tuning is to adjust a pre-processing network of an uncertain task to a specific downstream task, so a regression head is needed to adjust the abstract aesthetic representation output by the backbone network to predict the aesthetic score. Since a single linear layer can only perform simple feature processing on the backbone output of complex features, it cannot accurately capture the exact score of each threshold. Therefore, the linear regression head of the teacher network consists of two branches, the first branch consists of a linear layer, and the second branch consists of a linear layer with an activation function. The outputs of the two branches are added and input into the softmax function to obtain the final output f t .

[0082] Step 5: Fine-tune the teacher network model f using labeled data L t .

[0083] Use the metric method - Earth Mover Distance (EMD) to fine-tune the model on the labeled data L without freezing the backbone network, that is, fine-tune the teacher network model f t EMD is a measure of the difference between two probability distributions. In computer vision and image processing, EMD is often used to measure the similarity between two images. and the true score distribution y with N ordered categories, the EMD loss function can be expressed as:

[0084]

[0085] Where, CDF y (k) and is the cumulative distribution function, r is the coefficient used to penalize the Euclidean distance between two cumulative distribution functions, and is set to 2.

[0086] Step 6: Use the self-training method proposed in this invention to solve the overfitting problem.

[0087] When using a small amount of labeled aesthetic data for supervised fine-tuning, severe overfitting occurs. This is because the network has difficulty learning more useful aesthetic features from the limited labeled data and ignores useful information in the unlabeled data. To reduce overfitting, the present invention proposes a self-training method to utilize the complex aesthetic features in both labeled data and a large amount of unlabeled data. In the present invention, "self-training" refers to the process of combining supervised training with knowledge distillation. The present invention uses a fine-tuned network f t As a teacher to generate the network f for training the student s The label of the fine-tuned network f t Student Network s Perform knowledge distillation. The student model is initialized by the pre-trained weights, and the network structure is the same as the teacher network f tThe same. Given unlabeled data (u i ,·)∈U, the output of the teacher network and student online class model is defined as and Minimize distillation losses as follows:

[0088]

[0089] Where H(·) represents EMD loss, σ is the softmax function, and T represents the temperature parameter in knowledge distillation. In addition, the student model f s It is also important to learn not only from the teacher model but also from the labeled data. This paper combines the two so that the student model can learn some important missing representations in a complementary way. Therefore, the total loss can be defined as:

[0090]

[0091] Where λ is the weight of supervision loss and distillation loss, is the labeled data from the student model (x j ,y j )∈U’s predicted value.

Claims

1. A method for image aesthetic assessment based on self-supervised pre-training, characterized in that: The steps include: Step 1: Input the unlabeled dataset U and obtain the enhanced views t, t′; Input unlabeled image (t i , ·)∈U and sample two transformations from distribution T to generate enhanced views t and t′; distribution T is a probability distribution generated by a data augmentation strategy, which contains two transformation functions, one of which is an affine transformation function T a , the other is a non-rigid transformation function T b , these two transformation functions are randomly sampled from T, and the process of generating enhanced views t and t′ using the transformation function is as follows: t=T a (t i ),t′=T b (t i ) (1) Step 2: Input the enhanced views t, t′ into the encoder and expander to obtain the output z, z′; Views t and t′ are input into the encoder, and then mapped into embedding outputs z and z′ by the expander; the encoder and expander form the aesthetic representation extraction network; Step 3: Use a self-supervised method to train the aesthetic representation extraction network to obtain a trained aesthetic representation extraction network encoder; The self-supervised approach is to minimize the distance between two embedding outputs z and z′ from the same image, keeping the variance of each embedding variable above a set threshold, while the covariance between pairwise embedding variables is attracted to zero, making the variables de-correlated; the overall loss function is a weighted average of the invariance s(·), variance v(·), and covariance c(·) terms, as shown below: l(Z, Z′)=λs(Z, Z′)+μ[v(Z)+v(Z′)]+γ[c(Z)+c(Z′)] (2) Where λ, μ, and γ are weights that control the importance of each term in the loss, Z = [z1, ..., z n ] and Z′=[z′1,...,z′ n ] are two batches of size n; The overall objective function obtained on all images in the unlabeled dataset U is given by: L=∑ I∈U ∑ t，t′∈T l(Z I ,Z′ I ) (3) Where Z I and Z′ I is the embedding output of the aesthetic representation extraction network for the two batches of image I corresponding to the transformations t and t′; After completing self-supervised pre-training, remove the expander and only keep the encoder; Step 4: Add the self-supervised trained encoder to the regression classification head to form the teacher network model f t ; Teacher network model f t The linear regression head consists of two branches, the first branch consists of a linear layer, and the second branch consists of a linear layer with an activation function; the outputs of the two branches are added and input into the softmax function to obtain the final output f t ; Step 5: Fine-tune the teacher network model f using the labeled dataset L t ; Use the metric method - Earth Mover Distance EMD to fine-tune the model on the labeled dataset L and do not freeze the backbone network, that is, fine-tune the teacher network model f t ; Given the predicted value And the true score distribution y with N ordered categories, the EMD loss function is expressed as: Where, CDF y (k) and is the cumulative distribution function, r is the coefficient used to penalize the Euclidean distance between two cumulative distribution functions; Step 6: Solve the overfitting problem through self-training; The self-training is a process that combines supervised training with knowledge distillation; using the fine-tuning network f t As a teacher to generate the student network model f s The label of the fine-tuned network f t Student network model f s Perform knowledge distillation; the student model is initialized by the pre-trained weights, and the network structure is the same as the teacher network f t Same; given unlabeled data (u i ,·)∈U, the output of the teacher network and student network model is defined as and Minimize distillation losses as follows: Where H(·) represents EMD loss, σ is the softmax function, and T represents the temperature parameter in knowledge distillation. Student Network Modelf s Learning not only from the teacher model but also from labeled data, the total loss is defined as: Where λ is the weight of supervision loss and distillation loss, is the labeled data from the student model (x j ,y j )∈U’s predicted value.

2. The image aesthetic assessment method based on self-supervised pre-training according to claim 1, characterized in that: The T a Including rotation, scaling, translation affine transformation, T b Includes non-rigid transformations such as bending, twisting, and stretching.

3. The image aesthetic assessment method based on self-supervised pre-training according to claim 1, characterized in that: The encoder is a pre-trained model, a ResNet-50 network pre-trained by the ImageNet dataset; the expander consists of three fully connected layers.

4. The image aesthetic assessment method based on self-supervised pre-training according to claim 1, characterized in that: The Earth Mover Distance (EMD) is a measurement method for measuring the difference between two probability distributions. The EMD is used to measure the similarity between two images.

5. The image aesthetic assessment method based on self-supervised pre-training according to claim 1, characterized in that: Said r=2.

Images