A method for image complexity evaluation based on convolutional neural network

Abstract

This paper discloses a method for image complexity assessment based on convolutional neural networks. The method comprises the following steps: using a two-branch convolutional neural network to extract image detail features and semantic information respectively, and fusing these two features in the prediction phase; proposing a spatially distributed attention module specifically designed for image complexity, so that the extracted features can be adaptively optimized based on the spatial distribution of the features; in the prediction phase, the features are input into two prediction head branches, wherein the global complexity score prediction head predicts the global complexity of the input image using a fully connected neural network, and the local complexity heat map prediction head predicts the local complexity heat map of the input image using a convolutional neural network. The method demonstrates excellent complexity prediction performance on complexity assessment datasets, surpassing existing methods. For a given image, the method can accurately provide a global complexity score and a pixel-level complexity heat map.

Description

Technical Field

[0001] The present invention relates to the fields of deep learning and computer vision technology, and in particular to an image complexity evaluation method based on convolutional neural networks. Background Art

[0002] Image complexity (IC) refers to the complexity of an image. Objectively, complexity is the number of details and content types within an image. Subjectively, complexity can be defined as the difficulty for humans to perceive or describe the overall and detailed information of an image. Complexity is an important psychological perception that significantly influences the observer's visual aesthetics and emotional perception. Complexity is also a key image attribute, and automatic complexity assessment can be effectively used in tasks such as image segmentation, image steganography, web design, text detection, and image enhancement.

[0003] Some methods have been proposed in the early days for complexity assessment. Based on the assumption that the higher the complexity, the more information it contains, some methods use information entropy as a measure of complexity. Similarly, since simple images contain more redundant pixel information, and complex images are more difficult to predict the pixels at each position, the image compression ratio can be used to measure complexity. Specifically, the lower the image compression ratio, the less pixel redundancy there is, and therefore the more complex it is. In addition, some methods based on image features can also be used to evaluate complexity, such as image edge density, image color type, composition distribution histogram, etc. In addition, some researchers have tried to first extract a variety of manual features as feature vectors, and then input them into classifiers such as support vector machines and gradient boosting trees for complexity prediction.

[0004] With the development of Deep Convolutional Neural Networks (DCNNs), some methods have begun to use DCNNs to assess complexity. Some researchers believe that the more complex the image, the more activated the DCNN's activation layer will be, so the average activation value of the DCNN activation layer can be used to represent complexity. Other methods first use a DCNN trained on a large dataset for feature extraction, and then use a support vector machine or linear regressor for complexity prediction.

[0005] Most of these methods are based on traditional feature extraction and classification regression, which makes them difficult to achieve ideal results. Some methods use the latest DCNN technology, but they are unable to achieve end-to-end training. That is, DCNN is only used for feature extraction and does not participate in the training process, which in turn limits the DCNN's learning ability. Summary of the Invention

[0006] The purpose of the present invention is to provide an image complexity evaluation method based on convolutional neural network to address the technical defects in complexity evaluation in the prior art.

[0007] The technical solution adopted to achieve the purpose of the present invention is:

[0008] A method for evaluating image complexity based on a convolutional neural network comprises the following steps:

[0009] Step 1: Using a two-branch convolutional neural network including a detail branch network and a global branch network, for a given image, the image is input into the detail branch network and the global branch network respectively;

[0010] Step 2: Use the detail branch network to extract low-level detail features, and the global branch network to extract high-level semantic features. For the extracted detail features and semantic features, use the spatial distributed attention module to perform complexity-oriented detail and semantic feature optimization;

[0011] Step 3: Use bilinear upsampling to adjust the detail features and semantic features optimized in step 2 to the same spatial size, and then perform feature concatenation based on the channel dimension to obtain fused features.

[0012] Step 4: Use the Conv-BN-ReLU three-layer structure to balance and further map the fused features obtained in step 3 to obtain the input feature map F of the global complexity score prediction head. sp And the input feature map F of the local complexity heat map prediction head mp Then, F sp Input to the global complexity score prediction head to get the global complexity score, and F mp Input into the local complexity heat map prediction head to obtain the local complexity heat map;

[0013] In step 5, the loss functions of the global complexity score prediction head and the local complexity heat map prediction head are calculated respectively to obtain the total training loss, and the parameters of the two-branch convolutional neural network, the global complexity score prediction head, and the local complexity heat map prediction head are optimized by the gradient back propagation algorithm.

[0014] In the above technical solution, the dual-branch convolutional neural network in step 1 uses the ResNet network architecture, and the ResNet network architecture includes four stages: CS1, CS2, CS3, and CS4. The detail branch network includes CS1 and CS2, and the global branch network includes CS1, CS2, CS3, and CS4.

[0015] In the above technical solution, in step 1, a bilinear interpolation algorithm is used to scale the given image to two different sizes of 256×256 and 512×512 respectively, wherein the 512×512 size image is input into the detail branch network to facilitate the capture of local detail information, and the 256×256 size image is input into the global branch network to facilitate the capture of global semantic information.

[0016] In the above technical solution, in step 2, the input and output of the spatially distributed attention module are both feature maps. A feature map is first flattened according to its length and width spatial dimensions and then input into a fully connected layer with shared weights. Weights are adaptively predicted for each channel based on the spatial feature distribution. These weights are then used to weight the original feature map through an element-wise multiplication operation. This spatially distributed attention module can make the captured features more complex and discriminative.

[0017] In the above technical solution, in step 2, the steps of performing detail feature optimization and semantic feature optimization by the spatial distributed attention module are as follows:

[0018] 2.1 For a given three-dimensional feature map F∈R C×H×W , first flatten the feature map into a two-dimensional feature map F according to the spatial dimension fl ∈R C×(HW) ;

[0019] 2.2 Use MLP to transform the two-dimensional feature map F fl Mapped to a vector s∈R C , MLP has two layers. The weight and bias parameters of the first layer are W0, b0 respectively, and the weight and bias parameters of the second layer are W1, b1 respectively. Each channel dimension of is operated separately, and the weight is shared. For a certain channel dimension i, the calculation method is:

[0020]

[0021] Where σ0 and σ1 are ReLU and Sigmoid activation functions respectively, W0∈R (HW)×512 , W1∈R 512×1 , b0∈R 512 , b1∈R 1 ;

[0022] 2.3 Expand the obtained s to the same spatial size as F in each channel and obtain the attention weight S∈R C ×H×W , perform element-level tensor multiplication on S and F to obtain the output of the spatially distributed attention module.

[0023] In the above technical solution, in step 4, the global complexity score prediction head is composed of an average pooling layer, a fully connected neural network and an activation layer. The process of calculating the global complexity score is as follows: the global complexity score prediction head uses the global average pooling to transform the feature map F sp Average pooling is vector X sp ∈R 512 :

[0024]

[0025] in It's X sp The i-th element of It's F sp The xth row, yth column, and ith channel element of is then used to train a fully connected neural network with 512 input neurons and 1 output neuron. sp Map and get the result x sp , the weight of the neural network is W spfc ∈R 512×1 , the bias parameter is b spfc ∈R, the calculation process is:

[0026]

[0027] Finally, the SigMoid activation function is used to convert x sp Scaling to the normalized [0,1] interval, we get the complexity score S1, which represents the global complexity of the image. The activation function is calculated as:

[0028]

[0029] In the above technical solution, in step 4, the local complexity heat map prediction head consists of a convolutional layer with a length and width of 1 and an activation layer. The process of predicting the local complexity heat map is as follows:

[0030] The local complexity heat map prediction head first uses 1×1 convolution to transform the feature map F mp Mapped to 1-channel feature map F mps ∈R 64×64 The weight of the convolution here is W mpco ∈R 512×1 , the specific calculation process is:

[0031]

[0032] Then use a method applied to the entire feature map F mps The Sigmoid function on the complex local prediction heat map M is obtained ht :

[0033]

[0034] In the above technical solution, in step 5, the global complexity score prediction head is trained using a fully supervised method using a mean square error loss, and the loss function of the global complexity score prediction head is S gt is the true complexity label of the sample, is the batch size during training, and S1 is the complexity score.

[0035] In the above technical solution, in step 5, the local complexity heat map prediction head is trained based on the weak supervision method using the mean square error loss, and the loss function used by the local complexity heat map prediction head is in, S gt is the true complexity label of the sample and is the batch size during training.

[0036] In the above technical solution, in step 5, the total training loss L=λ×L1+(1-λ)×L2, and the value range of λ is 0 to 1, preferably 0.9.

[0037] Compared with the prior art, the present invention has the following beneficial effects:

[0038] 1. The present invention designs a dual-branch convolutional neural network consisting of a detail branch and a global branch for comprehensive feature extraction. It can extract global high-level semantic information and detailed low-level texture information, which are crucial for complexity assessment.

[0039] 2. The present invention also proposes a spatially distributed attention module designed for complexity. The module consists of flattening, multi-layer perceptron, expansion, and tensor element multiplication operations. It can further perform complexity-oriented feature optimization on the extracted features according to their spatial distribution.

[0040] 3. This paper proposes a dual prediction head architecture. Two prediction heads can simultaneously reuse extracted features and then predict a global score that measures the overall complexity of an image, as well as a complexity heat map that describes the complexity of each local pixel. Finally, this paper effectively utilizes both fully and weakly supervised training strategies during training, further leveraging the learning capabilities of DCNNs to achieve accurate complexity assessment results. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 The overall structure of the complexity evaluation model proposed by the present invention is formed;

[0042] Figure 2 The specific structure and implementation method of the spatial distributed attention module proposed in this invention;

[0043] Figure 3 This is the performance of the present invention on a self-built complexity evaluation dataset;

[0044] Figure 4 This is a demonstration of the evaluation effect of the present invention on actual images. DETAILED DESCRIPTION

[0045] The present invention will be further described in detail below with reference to specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0046] refer to Figure 1 The DCNN-based image complexity assessment model proposed in the present invention is mainly composed of a deep convolutional network. The image complexity assessment method based on convolutional neural network has the following specific implementation steps:

[0047] Step 1: Use a two-branch convolutional neural network including a detail branch network and a global branch network. For a given image, input the image into the detail branch network and the global branch network respectively.

[0048] 1.1 Using the ResNet-based network architecture as the feature extraction backbone network, that is, the dual-branch convolutional neural network, the ResNet18 backbone network is divided into four convolutional extraction stages (CS) from shallow to deep, including CS1, CS2, CS3, and VS4. The output feature map size of each stage is The detail branch network consists of CS1 and CS2, while the global branch consists of all 4 stages.

[0049] 1.2 Given an RGB three-channel image, use the bilinear interpolation algorithm to scale it to two different sizes of 256×256 and 512×512 respectively. The 512×512 size image is input into the detail branch network, and the 256×256 size image is input into the global branch network.

[0050] Step 2: Use the detail branch network to extract low-level detail features, and the global branch network to extract high-level semantic features. For the extracted detail features and semantic features, a spatial layout attention module (SLAM) is used to optimize the detail and semantic features based on complexity, so that these output feature maps have more complexity discrimination. The input and output of this module are both feature maps. Figure 2 , the specific operations of this module are as follows:

[0051] 2.1 For a given three-dimensional feature map F∈RC×H×W , first flatten the feature map into a two-dimensional feature map F according to the spatial dimension fl ∈R C×(HW) .

[0052] 2.2 Use multilayer perceptron (MLP) to transform the two-dimensional feature map F fl Mapped to a vector s∈R C MLP has two layers. The weight and bias parameters of the first layer are W0 and b0 respectively. The weight and bias parameters of the second layer are W1 and b1 respectively. Each channel dimension of is operated separately, and the weights are shared. Specifically, for a certain channel dimension i, the calculation method is:

[0053]

[0054] Where σ0 and σ1 are ReLU and Sigmoid activation functions respectively, W0∈R (HW)×512 , W1∈R 512×1 , b0∈R 512 , b1∈R 1 .

[0055] 2.3 Expand the obtained s to the same spatial size as F on each channel and obtain the attention weight S∈R C×H×W , perform element-level tensor multiplication on S and F to obtain the output of the spatially distributed attention module.

[0056] Denoting the flattening, multi-layer perceptron, and expansion operations as o1, o2, and o3 respectively, the overall processing of the spatial attention module can be expressed as:

[0057] F so =·o3(o2(o1(F))),

[0058] Among them F so ∈R C×H×W is the output feature map of the spatially distributed attention module.

[0059] In step 3, after feature extraction using a dual-branch convolutional network and optimization using the spatially distributed attention module, the resulting detail features are sized 128×64×64 (the three dimensions, in order, represent the number of channels, length, and width of the feature map, respectively, similarly below), and the semantic features are sized 512×8×8. Both features are mapped to 256 dimensions using a 1×1 convolution, resulting in a detail feature map of 256×64×64 and a semantic feature map of 256×8×8. Bilinear interpolation is then used to upsample the semantic features to 256×64×64, maintaining the same size for both feature maps. Finally, an aggregation operation is used to concatenate the two features along the channel dimension, resulting in a feature map of 512×64×64.

[0060] Step 4: The merged features are input into the global complexity score prediction head and the local complexity heat map prediction head respectively to predict the global complexity score and the local complexity heat map:

[0061] 4.1 Use the Conv-BN-ReLU (convolution-normalization-activation) three-layer structure to balance and further map the fused feature map to obtain the input feature map F of the global complexity score prediction head sp ∈R 512×64×64 And the input feature map F of the local complexity heat map prediction head mp ∈R 512×64×64 This feature map is then sent to the two subsequent prediction head branches for complexity prediction.

[0062] 4.2 Global complexity score prediction head uses global average pooling to transform the feature map F sp Average pooling is vector X sp ∈R 512 :

[0063]

[0064] in It's X sp The i-th element of It's F sp The xth row, yth column, and ith channel element of is then used to train a fully connected neural network with 512 input neurons and 1 output neuron. sp Map and get the result x sp , the weight of the neural network is W spfc ∈R 512×1 , the bias parameter is b spfc ∈R, the calculation process is:

[0065]

[0066] Finally, the SigMoid activation function is used to convert x sp Scaling to the normalized [0,1] interval, we get the complexity score S1, which represents the global complexity of the image. The activation function is calculated as:

[0067]

[0068] 4.3 Local Complexity Heat Map Prediction The first step is to use 1×1 convolution to transform the feature map F mp Mapped to 1-channel feature map D mps ∈R 64×64 The weight of the convolution here is W mpco ∈R 512×1 , the specific calculation process is:

[0069]

[0070] Then use a method applied to the entire feature map F mps The Sigmoid function on the complex local prediction heat map M is obtained ht :

[0071]

[0072] Step 5: The loss function L for model training consists of two parts, L1 and L2, corresponding to the global complexity score prediction head and the local complexity heat map prediction head. The loss is mainly calculated by the mean square error. Let the true complexity label of the sample be S gt , the batch size during training is N, then for the global complexity score prediction head:

[0073]

[0074] For the local complexity heat map prediction head, due to the lack of real pixel-level complexity labels, this paper proposes a training strategy based on weak supervision, that is, using M gt The average value and S gt The loss is calculated by the mean squared error:

[0075]

[0076]

[0077] Finally, the λ parameter is used to control the weight of L1 and L2 in the training process to obtain the total training loss L:

[0078] L = λ × L1 + (1-λ) × L2.

[0079] During the training process, λ is set to 0.9. The parameters of the neural network model are optimized by the gradient back propagation algorithm. The goal is to make the loss L as small as possible to improve the prediction accuracy of the model.

[0080] like Figure 3 The table shows the performance of the present invention on a self-built complexity evaluation dataset. The "↑" symbol in the table indicates that the higher the index, the better the complexity prediction effect, and the "↓" symbol indicates that the lower the index, the better the complexity prediction effect.

[0081] In this data set, the true complexity score is a normalized value in the interval [0,1]. The evaluation indicators of the model prediction results mainly include Pearson correlation coefficient (PCC), Spearman correlation coefficient (SRCC), root mean square error (RMSE) and root mean absolute error (RMAE). The higher the PCC and SRCC, the better the prediction effect, and the lower the RMSE and RMAE, the better the prediction effect. The table also shows a comparison between the present invention and 15 other proposed complexity evaluation methods, including 10 traditional methods and 5 deep learning-based methods. The results show that the complexity evaluation performance of the present invention surpasses all published results.

[0082] This invention is the first to generate pixel-level complexity heat maps. Figure 4 The image effect of the complexity heat map generated by the present invention is shown, where the numbers in the brackets at the bottom of each image represent the prediction results of the model, and the numbers outside the brackets represent the actual complexity of the image being annotated. It can be found that the present invention can accurately predict the global complexity score of the image and the complexity heat of local pixels. In addition, it can be seen that in flat and smooth parts, the method of the present invention can output low complexity, while in parts with more textures and elements, the method of the present invention outputs high complexity, which shows the effectiveness of the present invention.

[0083] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for image complexity evaluation based on convolutional neural network, characterized in that: The following steps are involved: Step 1: Using a two-branch convolutional neural network including a detail branch network and a global branch network, for a given image, the image is input into the detail branch network and the global branch network respectively; Step 2: Use the detail branch network to extract low-level detail features, and the global branch network to extract high-level semantic features. For the extracted detail features and semantic features, use the spatial distributed attention module to perform complexity-oriented detail and semantic feature optimization; Step 3: Use bilinear upsampling to adjust the detail features and semantic features optimized in step 2 to the same spatial size, and then perform feature concatenation based on the channel dimension to obtain fused features. Step 4: Use the Conv-BN-ReLU three-layer structure to balance and further map the fused features obtained in step 3 to obtain the input feature map F of the global complexity score prediction head. sp And the input feature map F of the local complexity heat map prediction head mp Then, F sp Input to the global complexity score prediction head to get the global complexity score, and F mp Input into the local complexity heat map prediction head to obtain the local complexity heat map; In step 5, the loss functions of the global complexity score prediction head and the local complexity heat map prediction head are calculated respectively to obtain the total training loss, and the parameters of the two-branch convolutional neural network, the global complexity score prediction head, and the local complexity heat map prediction head are optimized by the gradient back propagation algorithm.

2. The image complexity evaluation method based on convolutional neural network according to claim 1, characterized in that: The dual-branch convolutional neural network in step 1 uses the ResNet network architecture, which includes four stages: CS1, CS2, CS3, and CS4. The detail branch network includes CS1 and CS2, and the global branch network includes CS1, CS2, CS3, and CS4.

3. The image complexity evaluation method based on convolutional neural network according to claim 1, characterized in that: In step 1, a bilinear interpolation algorithm is used to scale the given image to two different sizes of 256×256 and 512×512, respectively. The 512×512 size image is input into the detail branch network to facilitate the capture of local detail information, and the 256×256 size image is input into the global branch network to facilitate the capture of global semantic information.

4. The image complexity evaluation method based on convolutional neural network according to claim 1, characterized in that: In step 2, the input and output of the spatially distributed attention module are both feature maps. A feature map is first flattened according to the spatial dimensions of length and width, and then input into a fully connected layer with shared weights. Weights are adaptively predicted for each channel based on the spatial feature distribution. The weights are then used to weight the original feature map through an element-level multiplication operation. The spatially distributed attention module can make the captured features more complex and discriminative.

5. The image complexity evaluation method based on convolutional neural network according to claim 4, characterized in that: In step 2, the steps of performing detail feature optimization and semantic feature optimization in the spatial distribution attention module are as follows: 2.1 For a given three-dimensional feature map F∈R C×H×W , first flatten the feature map into a two-dimensional feature map F according to the spatial dimension fl ∈R C×(HW) ; 2.2 Use MLP to transform the two-dimensional feature map F fl Mapped to a vector s∈R C , MLP has two layers. The weight and bias parameters of the first layer are W0, b0 respectively, and the weight and bias parameters of the second layer are W1, b1 respectively. Each channel dimension of is operated separately, and the weight is shared. For a certain channel dimension i, the calculation method is: Where σ0 and σ1 are ReLU and Sigmoid activation functions respectively, W0∈R (HW)×512 , W1∈R 512×1 , b0∈R 512 , b1∈R 1 ; 2.3 Expand the obtained s to the same spatial size as F in each channel and obtain the attention weight S∈R C×H×W , perform element-level tensor multiplication on S and F to obtain the output of the spatially distributed attention module.

6. The image complexity evaluation method based on convolutional neural network according to claim 1, characterized in that: In step 4, the global complexity score prediction head consists of an average pooling layer, a fully connected neural network and an activation layer. The process of calculating the global complexity score is as follows: the global complexity score prediction head uses the global average pooling to transform the feature map F sp Average pooling is vector X sp ∈R 512 : in It's X sp The i-th element of It's F sp The xth row, yth column, and ith channel element of is then quantized using a fully connected neural network with 512 input neurons and 1 output neuron. sp Map and get the result x sp , the weight of the neural network is W spfc ∈R 512×1 , the bias parameter is b spfc ∈R, the calculation process is: Finally, the SigMoid activation function is used to convert x sp Scaling to the normalized [0,1] interval, we get the complexity score S1, which represents the global complexity of the image. The activation function is calculated as:

7. The image complexity evaluation method based on convolutional neural network according to claim 1, characterized in that: In step 4, the local complexity heat map prediction head consists of a convolutional layer with a length and width of 1 and an activation layer. The process of predicting the local complexity heat map is as follows: The local complexity heat map prediction head first uses 1×1 convolution to transform the feature map F mp Mapped to 1-channel feature map F mps ∈R 64 ×64 , where the convolution weight is W mpco ∈R 512×1 , the specific calculation process is: Then use a method applied to the entire feature map F mps The Sigmoid function on the complex local prediction heat map M is obtained ht :

8. The image complexity evaluation method based on convolutional neural network according to claim 1, characterized in that: In step 5, the global complexity score prediction head is trained using a fully supervised method using a mean square error loss, and the loss function of the global complexity score prediction head is S gt is the true complexity label of the sample, N is the batch size during training, and S1 is the complexity score.

9. The image complexity evaluation method based on convolutional neural network according to claim 7, characterized in that: In step 5, the local complexity heat map prediction head is trained based on the weak supervision method using the mean square error loss. The loss function used by the local complexity heat map prediction head is in, S gt is the true complexity label of the sample, and N is the batch size during training.

10. The image complexity evaluation method based on convolutional neural network according to claim 1, characterized in that: In step 5, the total training loss L = λ×L1+(1-λ)×L2, where L1 is the loss function of the global complexity score prediction head, L2 is the loss function used by the local complexity heat map prediction head, and the value range of λ is 0 to 1, preferably 0.9.

Images