Photographic image aesthetic style classification method based on self-supervised learning and deep forest

Abstract

The application discloses a kind of based on self-supervised learning and deep forest's photographic image aesthetic style classification method, first constructs and trains self-supervised feature learning model, then constructs and trains deep forest model, finally completes photographic image aesthetic style classification by the feature extraction network and deep forest model of well-trained, the present application is pre-trained using self-supervised feature learning model, fully excavates aesthetic feature, it is favorable for deep forest model to find the optimal solution of multi-label photographic image aesthetic style classification better.The association information between multi-label is fully considered in the iterative training process of deep forest model, and the precision of multi-label photographic image aesthetic style classification is improved.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of image aesthetic style classification, and relates to a photographic image aesthetic style classification method, in particular to a multi-label photographic image aesthetic style classification method based on self-supervised feature learning and deep forest. BACKGROUND

[0002] Since multiple aesthetic style labels are always included in photographic images, the multi-label photographic image aesthetic style classification method is more practical than the single-label method, and the purpose is to predict all the aesthetic styles to which the photographic images belong. Since photographic images include multiple aesthetic style labels, the number of combinations of classification results increases exponentially compared to single-label, and the multi-label photographic image aesthetic style classification problem is more difficult and less accurate.

[0003] The existing method mainly has the following problems: the existing large-scale data set is not suitable for feature learning of aesthetic style classification tasks, and it is difficult to learn features valuable for aesthetic style classification tasks; the correlation information between labels cannot be fully utilized when learning multi-label tasks SUMMARY

[0004] In view of the deficiencies in the prior art, the present application provides a photographic image aesthetic style classification method based on self-supervised learning and deep forest, which comprises a self-supervised feature learning model and a deep forest model. A large-scale aesthetic image is used to train the self-supervised feature learning model. After training, the multi-label photographic image data set is inputted to obtain the corresponding feature vector matrix as the input of the deep forest model, and the deep forest model is trained to obtain the final multi-label prediction result.

[0005] The technical scheme of the present application is as follows:

[0006] Step 1: constructing and training a self-supervised feature learning model;

[0007] The self-supervised feature learning model is composed of a feature extraction network and a projection network.

[0008] Step 1-1: establishing a feature extraction network.

[0009] Step 1-2: establishing a projection network.

[0010] Step 1-3: establishing a self-supervised contrast learning loss function.

[0011] Step 2: constructing and training a deep forest model, specifically including:

[0012] Step 2-1: The dataset consists of 2000 multi-label photographic images, representing 14 photographic aesthetic styles. Each image has more than one photographic aesthetic style label. The feature vectors obtained from these 2000 images through the self-supervised feature learning model in Step 1 are concatenated column-wise to obtain the input matrix.

[0013] Step 2-2: Establish the hierarchical structure of the deep forest model.

[0014] Steps 2-3: Establish the connection methods between layers of the deep forest model.

[0015] Step 3: Complete the aesthetic style classification of photographic images using the trained feature extraction network and deep forest model.

[0016] The beneficial effects of this invention are:

[0017] 1. Pre-training with a self-supervised feature learning model fully explores aesthetic features, which helps the deep forest model find the optimal solution for classifying the aesthetic style of multi-label photographic images.

[0018] 2. Deep forest models fully consider the correlation information between multiple labels during iterative training, thereby improving the accuracy of multi-label photographic image aesthetic style classification. Attached Figure Description

[0019] Figure 1 This is a schematic diagram of the process of an embodiment of the method of the present invention. Detailed Implementation

[0020] The specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings.

[0021] like Figure 1 As shown, a photographic image aesthetic style classification method based on self-supervised learning and deep forest is presented, with the following steps:

[0022] Step 1: Construct and train a self-supervised feature learning model, as follows:

[0023] The self-supervised feature learning model consists of a feature extraction network and a projection network.

[0024] Step 1-1: Establish the feature extraction network.

[0025] The dataset uses the AVA aesthetic dataset, which contains 250,000 aesthetic images. For each input sample x, ResNet50 is used as the feature extraction network to obtain the representation vector. Normalize to Hypersphere.

[0026] Step 1-2: Establish the projection network.

[0027] After obtaining the representation vector r, the vector is obtained through the projection network. The projection network is a multilayer perceptron network, consisting of a 2048-dimensional hidden layer and a D layer. P =128 output layer. Normalize vector z again to the hypersphere.

[0028] Steps 1-3: Establish a self-supervised contrastive learning loss function.

[0029] Given a batch of randomly sampled data / label pairs {x} of size N. k ,y k} k＝1…N Then, the data used for training consists of a batch of 2N data / label pairs that have undergone data augmentation (rotation, translation, etc.). in and It is x k The labels were generated through two different data augmentations.

[0030] In the same batch of training data, if any data with index i∈I≡1…2N is selected as the baseline data, then j(i) is another data index obtained by data augmentation from the same source data sample as index i. The formula for the self-supervised contrastive learning loss function is as follows:

[0031]

[0032] in, The symbol · represents the inner product operation. Let A(i) represent the temperature coefficient. The data indexed by j(i) is considered as positive samples, and A(i) represents the set of the remaining data as negative samples.

[0033] Step 2: Build and train the deep forest model, as follows:

[0034] Step 2-1: The dataset consists of 2000 multi-label photographic images from the AVA aesthetic dataset, representing 14 photographic aesthetic styles. Each image has more than one photographic aesthetic style label. The 2000 corresponding 2048 feature vectors obtained from the feature extraction network in Step 1-1 are concatenated column-wise to obtain the input matrix.

[0035] Step 2-2: Establish the hierarchical structure of the deep forest model. Use the input matrix obtained in Step 2-1 as the input to the first layer. The deep forest model has a total of 5 layers, each consisting of 4 random forest models: 2 ordinary random forest models and 2 fully random forest models. The difference between the two lies in the feature selection during branching; ordinary random forests select... (n is the number of sample features) are used as candidates, and then the Gini coefficient is used to select the best feature for branching; while a completely random forest randomly selects one feature from n features for branching. Given dataset X mn Each row X i (i = 0, ..., m-1) = [x0, ..., x n-1 The corresponding tag set is Y. ml Each row is Y i (i = 0, ..., m-1) = [y0, ..., y 1-1 Where m = 2000 is the total number of samples, n = 2048 is the total number of features, and k = 14 is the total number of labels. Each layer of the random forest model generates a probability value for the input sample image belonging to a certain style label. If this probability value is greater than a set threshold, the image is determined to belong to that style label, and 1 is output; otherwise, it is determined not to belong, and 0 is output.

[0036] Steps 2-3: Establish the connections between layers of the deep forest model. For each layer, iterate through all four random forest models, using multi-fold cross-training for each. Each random forest model outputs an m x k ​​matrix, representing the probability of each image belonging to each style label. Therefore, each layer outputs a probability matrix for four labels, which are concatenated column-wise to obtain the output matrix. Where t represents the output matrix of the t-th layer, and m×4k represents the size of the matrix as m rows and 4*k columns.

[0037] The connections between layers employ a reuse mechanism. For the output of the first layer, the output matrix is ​​directly concatenated column-wise with the initial input matrix, serving as the input to the next layer in the deep forest model. The next layer is trained in the same manner as the first layer, generating... The resulting matrix. At this point, the reuse mechanism takes effect, performing operations on the training output of the current layer and the training output of the previous layer to generate a new matrix. The specific calculation is as follows:

[0038]

[0039] Depending on the evaluation metric for multi-label tasks, columns or rows with better performance in the previous layer are used to replace the corresponding columns or rows in the current layer's output. If Hamming loss is used as the evaluation metric, it is calculated iteratively column-by-column. The Hamming loss for each column is calculated, and if the loss of a column in the current layer's output matrix is ​​greater than the loss of the same column in the same random forest model from the previous layer (i.e., the performance is worse), the calculated confidence score for that column is added to a set. After iteration, the set of confidence scores for the poorly performing columns in the current prediction results is obtained. This set of confidence scores is then averaged to obtain the average confidence score. Therefore, for each column in the current output matrix, if the confidence score is less than the average confidence score, the result from the previous layer is used to replace that column.

[0040] When reaching the last layer, the maximum probability voting method is used to obtain the final predicted multi-label result, as shown below:

[0041]

[0042] Step 3: Complete the aesthetic style classification of photographic images using the trained feature extraction network and deep forest model.

[0043] Input a photographic image, obtain a feature vector through the feature extraction network in step 1-1, and then input it into the deep forest model in step 2 to obtain the predicted multi-label classification result.

Claims

1. A photographic image aesthetic style classification method based on self-supervised learning and deep forest, characterized in that, The steps are as follows: Step 1: constructing and training a self-supervised feature learning model; The self-supervised feature learning model is composed of a feature extraction network and a projection network; Step 1-1: establishing a feature extraction network; Step 1-2: establishing a projection network; Step 1-3: establishing a self-supervised contrastive learning loss function; Step 2: constructing and training a deep forest model, specifically including: Step 2-1: the data set adopts 2000 multi-label photographic images, a total of 14 photographic aesthetic styles, and each image has more than one photographic aesthetic style label; the feature vectors obtained by the self-supervised feature learning model in step 1 are concatenated according to the columns to obtain an input matrix; Step 2-2: establishing a deep forest model hierarchy; Step 2-3: establishing the connection mode between the layers of the deep forest model; Step 3: completing the photographic image aesthetic style classification through the trained feature extraction network and deep forest model; Step 1 specifically includes the following methods: The self-supervised feature learning model is composed of a feature extraction network and a projection network; Step 1-1: establishing a feature extraction network; The dataset adopts AVA aesthetic dataset with 250,000 aesthetic images. For each input sample x, ResNet50 is used as a feature extraction network to obtain a feature vector normalization to hypersphere; Step 1-2: establishing a projection network; After obtaining the representation vector r, the vector The projection network is a multilayer perceptron network with a 2048-dimensional hidden layer and an output layer of dimension D P = 128; the vector z is again normalized to the hypersphere; Step 1-3: establishing a self-supervised contrastive learning loss function; If a batch of randomly sampled data / label pairs {x k ,y k} k＝1…N is given with capacity N, then the data / label pairs with capacity 2N that is applied to training is generated by data augmentation where are generated by two different data augmentations on x k , respectively, and the label In the same batch of training data, any selected i∈I≡1…2N index data is used as reference data, then j(i) is another data index derived from the same source data sample as the index i data through data augmentation; the self-supervised contrastive learning loss function formula is as follows: wherein denotes an inner product operation, denotes the temperature coefficient; j(i) indexes the data as positive samples, A(i) denotes the set of the rest data as negative samples; Step 2 specifically includes the following methods: Step 2-1: the data set adopts 2000 multi-label photographic images in the AVA aesthetic data set, a total of 14 photographic aesthetic styles, and each image has more than one photographic aesthetic style label; the corresponding 2000 2048 feature vectors obtained by the feature extraction network in step 1-1 are concatenated according to the columns to obtain an input matrix; Step 2-2: Establish the deep forest model hierarchy; take the input matrix obtained in step 2-1 as the input of the first layer; the deep forest model has a total of 5 layers, each layer is composed of 4 random forest models, including 2 ordinary random forest models and 2 complete random forest models; the difference between the two lies in the difference in feature selection when branching; the ordinary random forest selects the best features for branching according to the Gini coefficient; while the complete random forest randomly selects one from n features for branching; given a data set X mn , where each row X i (i = 0, …, m-1) = [x0, …, x n-1 ] corresponds to a label set Y ml , where each row Y i (i = 0, …, m-1) = [y0, …, y 1-1 ]; where m = 2000 is the total number of samples, n = 2048 is the total number of features, and k = 14 is the total number of labels; the random forest model of each layer of the model will generate a probability value of the input sample image belonging to a certain style label, if the probability value is greater than a set threshold, it is judged that the image belongs to the style label, and 1 is output, otherwise it is judged not to belong, and 0 is output; Step 2-3: Establish the connection mode between the layers of the deep forest model; for each layer model, traverse all 4 random forest models, and each random forest model adopts the multi-fold cross-training mode; in this way, each random forest model outputs an m-row k-column matrix, which represents the probability value of each image belonging to each style label; therefore, each layer outputs a probability matrix of 4 labels in total, and the output matrix is obtained by concatenating by column where t represents the result matrix of the output of the tth layer, and m*4k represents that the scale of the matrix is m rows and 4*k columns; The connection mode between layers adopts a multiplexing mechanism; for the output of the first layer, the output matrix is directly spliced with the initial input matrix according to columns, as the input of the next layer in the deep forest model; the next layer is also trained according to the first layer, to generate a result matrix; at this time, the multiplexing mechanism takes effect, and the training output of the current layer is operated with the training output of the previous layer to generate a new matrix The specific operation is as follows: According to the different multi-label task evaluation indexes, the column or row with good performance in the last layer is replaced with the corresponding column or row of the output of the current layer; when reaching the last layer, the maximum probability voting method is used to obtain the final predicted multi-label result, which is specifically expressed as follows:

Images