Blind reference image quality evaluation method based on content and distortion feature fusion

Abstract

The invention provides a blind reference image quality evaluation method based on content and distortion feature fusion. Content and distortion features are effectively extracted, and the global action relation of the two features in quality evaluation is accurately captured. The method comprises the following steps: firstly, combining a synthetic distortion data set and a real distortion data set to construct a pre-training data set, and learning distortion priori by adopting a pre-training model of a twin neural network architecture; in the quality prediction network, a distortion branch uses a backbone of a pre-training model as a distortion feature extractor, and a content branch uses a ResNet50 backbone trained on ImageNet as a content feature extractor; a deformable attention module (DCA) is used for sampling key information on a feature space and a channel and normalizing a multi-level feature map; and finally, introducing a Transform codec, and constructing a weight token to guide the two features to perform quality semantic weighted fusion so as to obtain better quality representation. Compared with a previous method, the method has higher precision on the quality evaluation data sets of the real distortion image and the synthetic distortion image.

Description

Technical Field

[0001] The invention discloses a blind reference image quality evaluation method based on the fusion of content and distortion features, and belongs to the field of computer low-level vision. Background Art

[0002] Image quality evaluation is one of the basic tasks in the field of computer vision. Its purpose is to evaluate image quality through algorithms and ensure that the evaluation results are consistent with human subjective feelings. With the advent of the information age, the Internet is flooded with a large number of pictures. Being able to accurately evaluate the quality of pictures is of great help to autonomous driving and media applications, and also has guiding significance for some advanced image algorithms.

[0003] According to whether there is reference information for image quality evaluation, existing algorithms can be divided into full reference, semi-reference and no reference. Because full reference images and semi-reference images have certain reference information, existing algorithms have been able to achieve good results. However, it is difficult to obtain reference information for images, so no reference image quality evaluation has important practical significance and is more challenging. According to the type of image evaluation database, existing public datasets can be divided into synthetic distortion datasets and real distortion datasets. Synthetic distortion datasets are generated in a laboratory environment by adding different types of distortion to an original high-quality reference image and setting different distortion levels for each distortion. Real distortion datasets are not generated in a laboratory environment. They are distorted images generated in real scenes due to equipment limitations or low light, overexposure, jitter, etc. during shooting. Some images also come from the Internet, and the quality of these images is reduced due to transmission and compression.

[0004] Image content and distortion are two important factors that determine image quality. At present, many algorithms have been explored for no-reference image quality evaluation, from early traditional methods to current deep learning-based methods. However, none of them captures the content features and distortion features that are closely related to image quality. Although some methods have made attempts at distortion feature perception, they often only consider real distortion or synthetic distortion. In addition, the global interaction between content features and distortion features in image evaluation has not been well explored. Since the above problems have not been well solved, most current solutions find it difficult to achieve good results on both real distortion datasets and synthetic distortion datasets. Summary of the invention

[0005] The purpose of the present invention is to propose a blind reference image quality assessment method based on the fusion of content and distortion features, which can achieve good performance on both real distortion data sets and synthetic distortion data sets. Content features and distortion features are accurately extracted through pre-training tasks and dual-branch structures; then the deformable attention module DCA is used to sample key information on multi-level feature map space and channels; finally, the Transformer codec architecture is introduced, combined with the constructed weight tokens to capture the global interaction relationship of different features in quality assessment and guide feature weighted fusion, so as to obtain better quality representation and improve the performance of the model in quality assessment tasks. The proposed method includes the following steps:

[0006] A mixed-distortion image quality assessment dataset is established. By pairing image pairs with the same distortion type, different image content, and different distortion levels, the influence of content factors is shielded, making it easier for the model to obtain the essential characteristics of distortion independent of the content during the learning process. The constructed dataset contains both synthetic distortion and real distortion, so the distortion knowledge prior is more generalizable.

[0007] A pre-trained model based on a twin neural network architecture. This model uses EfficientNet-B0 as a feature extractor and learns the distortion differences between feature maps of different distortion levels by sharing weights, thereby better perceiving the more generalized distortion knowledge priors in the pre-training dataset.

[0008] The quality prediction network based on the dual-branch structure uses the ResNet50 backbone network pre-trained on ImageNet as the feature extractor of the content branch, and the EfficientNet-B0 feature extractor backbone trained on the self-constructed distortion dataset as the feature extractor of the distortion branch of the quality prediction network.

[0009] The DCA module combines key features in space and channels to better aggregate multi-level feature maps from dual branches.

[0010] Based on the Transformer codec and the quality weighting module of the weight token, the global interaction relationship between content features and distortion features is captured to guide the quality-weighted fusion of two different features to obtain better quality representation and be used for quality regression.

[0011] Furthermore, the pre-training dataset construction strategy has different processing methods for synthetic distorted datasets and real distorted datasets.

[0012] Furthermore, for the synthetic distortion dataset, VCL@FER and CID-IQ datasets were used at the same time; the distortion level was directly used as the label, and pictures of the same type with different distortion levels from different reference images were combined to form synthetic distortion comparison pairs.

[0013] Furthermore, for the real distortion dataset, the LIVEFB dataset is used; because the distortion is not added in a laboratory environment, the distortion levels of different images cannot be directly derived; assuming that the distortion level of images with extremely good quality scores is less than that of images with extremely poor quality scores, the quality scores are used to approximate the distortion levels, and the batch of LIVEFB images with the best quality and the batch of images with the worst quality are paired two by two to form real distortion comparison pairs.

[0014] Furthermore, in the process of constructing the distortion pair, the position of the distorted image is randomly selected, and it is stipulated that if the distortion level of the left image in the pair is lower, the label is 1, otherwise it is 0; finally, the synthetic distortion pairs and the real distortion pairs are randomly mixed to obtain the distorted pre-training dataset.

[0015] Furthermore, the pre-trained model based on the twin neural network architecture uses the feature maps of the 3rd, 4th, 8th, and 9th stages in EfficientNet-B0 as the distortion feature maps, extracts these layers of feature maps from the two images in the distorted pair for element-by-element subtraction, and then the multi-scale feature fusion module MFF uses 1×3, 3×1, and 3×3 convolutions to extract fine-grained features and adds these fine-grained feature maps element-by-element. Different strides are used for feature maps of different sizes, so that after passing through MFF, the feature map sizes of each stage are unified, and then splicing operations are performed in the channel dimension. Finally, global average pooling, linear layer, and Sigmoid function are used to obtain the predicted value, which represents the probability that the distortion level of the left image in the image pair is lower.

[0016] Furthermore, the pre-trained EfficientNet-B0 backbone network and the pre-trained ResNet50 backbone network on ImageNet are loaded into the quality prediction network as distortion feature extractor and content feature extractor respectively, and their parameters are frozen; the feature maps of the 3rd, 4th, and 8th stages in EfficientNet-B0 and the feature maps of the first three stages of ResNet50 are used in the quality prediction stage.

[0017] Furthermore, deformable convolution and channel attention are introduced in the DCA module, which enables the module to well integrate the key information in the spatial and channel dimensions from any branch feature; use different strides to normalize the feature map and perform splicing operations in the channel dimension; use residual connections when sampling the channel dimension and feature dimension to avoid feature degradation.

[0018] Furthermore, a learnable weight token is constructed, concatenated with the content features and distortion features after dimensionality transformation, and sent to the Transformer encoder for self-attention operation, so that the weight token can learn the global interaction relationship between different features; in the decoding stage, cross attention is performed, and the weight token is used as the key and value, and the content features and distortion features from the two branches are used as queries respectively to obtain features with quality-weighted semantics; the weighted features of the two branches are concatenated to obtain a fused feature that can better represent the quality, and after global pooling, the linear layer obtains the quality prediction score. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 It is a schematic diagram of the process of constructing a pre-training dataset of distorted images.

[0020] Figure 2 This is a schematic diagram of a pre-trained model based on a twin neural network architecture.

[0021] Figure 3 It is a schematic diagram of the image quality prediction network structure. DETAILED DESCRIPTION

[0022] In order to more clearly describe the technical content of the present invention, the invention content is further described below together with the drawings and specific implementations.

[0023] For synthetic distortion construction, see Figure 1 Left. Because different degrees of distortion are added in a laboratory environment, different levels of distortion images generated under the same type of distortion are directly selected from different reference images to form a distortion comparison pair. Represents different reference image contents, j∈[1,C] represents the same distortion type, l,l′∈[1,L] represents different distortion levels; a synthetic distortion dataset with N original reference images, C distortion types, and L distortion levels for each distortion can generate (N-1)*N*L*(L+1) distortion pairs. For the VCL@FER and CID-IQ datasets used, a total of 45540 distortion pairs are generated.

[0024] Construct a distortion comparison dataset, see Figure 1 Right. For real distortion, there is no distortion level constructed in a laboratory environment. The quality score is used to approximate the distortion level. It is assumed that the distortion level of images with excellent quality scores is lower than that of images with extremely poor quality scores. The 300 images with the highest quality scores and the 300 images with the lowest quality scores are selected from the LIVEFB dataset and paired with each other to form real distortion comparison pairs. After removing grayscale images and image pairs with image sizes less than 224 pixels, a total of 87,607 image pairs are obtained.

[0025] In order to prevent data bias, the position of the distorted image in the pair is randomly selected; if the distortion level of the image on the left side of the pair is lower, the label is 1, otherwise it is 0; the constructed synthetic distortion pairs and real distortion pairs are randomly mixed to obtain the distortion comparison dataset.

[0026] Build a pre-trained model based on the twin network architecture, see Figure 2 The pre-trained model uses the EfficientNet-B0 backbone network as the feature extractor E, and the input distorted image pair I pair (I1, I2) share weight parameter w e , for the two images, extract the 4-layer feature maps f of the 3rd, 4th, 8th, and 9th stages respectively i j , i∈{1,2}, j∈{1,2,3,4}:

[0027]

[0028] Subtract the feature maps of the two images element by element to get the feature difference map:

[0029]

[0030] Four multi-scale feature fusion modules MFF are used to process F diff Feature maps of the 4 stages:

[0031]

[0032]

[0033] Where σ is the activation function RELU, bn is the batch normalization layer BatchNorm, K n×m is a convolution operation with a convolution kernel size of n×m, It is element-wise addition.

[0034] Concatenate the outputs in the channel dimension:

[0035] The fused feature map is passed through the global pooling and linear layer, and the output is constrained by the Sigmoid activation function to obtain the predicted probability value P:

[0036] P = Sigmoid(FC(Avg_pool(F out ))

[0037] The loss function in the pre-training stage uses the binary cross loss, and the formula is as follows:

[0038]

[0039] where y i, x i They represent the true label and predicted probability value of the i-th distortion pair in batch N respectively.

[0040] The loss function is used for gradient descent and back propagation to continuously update the pre-trained model parameters.

[0041] In the proposed quality prediction network, see Figure 3 For the distortion branch feature extractor, the EfficientNet-B0 backbone network in the pre-trained model is used and the weight parameters are frozen; for the content branch feature extractor, the ResNet50 backbone network pre-trained on ImageNet is directly used and the weight parameters are frozen; the feature maps of the 3rd, 4th, and 8th stages of EfficientNet-B0 and the feature maps of the 1st, 2nd, and 3rd stages of ResNet50 are used to represent the extracted distortion features D i and content feature C i , i∈{1,2,3}.

[0042] The multi-level content and distortion feature maps from the two branches are processed by DCA modules respectively, as Figure 3 As shown in the figure, the DCA module uses deformable convolution, which can drift the convolution position to select a more appropriate convolution point; combined with channel attention, the module can well sample key information in the spatial and channel dimensions; the deformable convolution operation is as follows:

[0043]

[0044] Among them, p0 represents the convolution center, p n represents the original convolution point, Δp n Represents the two-dimensional coordinate offset of the convolution point. The convolution kernel w used is 3×3, then R={-1, 0, 1}×{-1, 0, 1}, and x represents the feature map.

[0045] After the deformable convolution, channel attention is used to sample channel features. The process is as follows:

[0046]

[0047] Where MLP stands for multi-layer perceptron, It represents element-by-element multiplication, and σ is the activation function RELU.

[0048] In the DCA module, residual connections are used to avoid feature degradation during spatial and channel sampling, and different strides and channel adjustments are used for feature maps of different sizes, so that after passing through the DCA module, the multi-level feature maps of the two branches have the same size in width and height dimensions and the total number of channels of the multi-level feature maps of each branch is the same; the content feature and distortion feature are denoted by D′ i, C′ i , i∈{1, 2, 3}. These features are concatenated in the channel dimension to obtain:

[0049]

[0050] D f and C f Perform dimensional transformation (C, H, W) → (N, C) and positionally encode the two features, where C, H, and W represent channel dimension, height dimension, and width dimension, respectively, and N = C × H; in order to capture the global interaction between content features and distortion features in quality evaluation, construct a learnable weight token Splice WT, D f and C f get Use the Transformer encoder to obtain its self-attention relationship:

[0051] F cd =MSHA(LN(F cd ), LN(F cd ), LN(F cd ))+F cd

[0052] F′ cd =MLP(LN(F cd ))+F cd

[0053] Among them, MSHA represents the multi-head attention mechanism, and LN represents layer normalization LayerNorm.

[0054] Similar to Vision Transformer, F′ after self-attention mechanism cd The first dimension of is used as the weight token WT = F′ to learn the global relationship between content and distortion features. cd [0].

[0055] In the feature decoding stage, the weight token WT is used to guide different features to perform quality semantic weighting, which can better balance the different degrees of influence of content and distortion features on quality scores in quality evaluation.

[0056] The Transformer decoder performs cross attention, with WT as the key and value, and the position-encoded content feature C f and distortion characteristics D f As a query. f For example, first, f Do self-attention encoding:

[0057] Df =MSHA(LN(D f ), LN(D f ), LN(D f ))+D f

[0058] Perform a criss-cross attention query:

[0059] D′ f =MSHA(LN(D f ), WT, WT)+D f

[0060]

[0061] That is, it is the semantic distortion feature with quality weighting after WT guidance weighting. Similarly, for C f Perform the same operation to obtain the content features after quality semantic weighting

[0062] Will Perform dimensional transformation (N, C) → (C, H, W), and obtain the fused quality feature F after splicing in the channel dimension.

[0063] The final quality prediction score s is obtained through the global pooling layer and the linear layer:

[0064] s=FC(Avg_pool(F)

[0065] For the loss function of the quality prediction network, select SmothLlloss:

[0066]

[0067] where x n Indicates the predicted quality score of the nth image, y n Indicates the true quality score of the nth image.

[0068] The quality prediction network parameters are continuously updated through gradient descent and back propagation.

[0069] Following the existing standards, Pearson linear correlation coefficient PLCC and Spearman correlation coefficient SROCC are used as the performance evaluation criteria of the method. Experiments are conducted on synthetic distorted image quality assessment datasets CSIQ and KADID and real distorted image quality assessment dataset KONIQ.

[0070] The comparison results with the current state-of-the-art methods are as follows:

[0071] From the experimental results, we can see that the present invention has the most advanced effect on both real distorted data sets and synthetic distorted data sets. This proves that by mixing two types of data sets for pre-training tasks, combined with the effective information extraction of dual-branch features in the quality evaluation network and weighted fusion with quality semantics, the present invention can capture more generalized distortion features and have better quality perception capabilities.

Claims

1. A blind reference image quality evaluation method based on the fusion of content and distortion features can simultaneously extract image content features and distortion features and accurately evaluate image quality. Its main features are The following steps are involved: 1) Using two synthetic distortion datasets VCL@FER and CID-IQ and a real distortion dataset LIVEFB, and pairing the images according to their degree of distortion to construct a mixed distortion image distortion comparison pre-training dataset; 2) For the synthetic distortion dataset, the distortion comparison pairs are constructed using the same type of different levels of distortion images with different content reference images; for the real distortion dataset, the distortion level is approximated using the quality score, and the images with higher and lower quality scores in the LIVEFB dataset are paired to form real distortion comparison pairs; it is stipulated that if the distortion degree of the left image in the pair is lower, the pre-trained label is 1, otherwise it is 0; in the process of constructing the image pair, the position of the image in the pair is randomly selected; the real distortion pairs and synthetic distortion pairs are randomly mixed to form the pre-trained distortion comparison dataset; 3) Using EfficientNet-B0 as the backbone and building a distortion feature extractor based on the twin neural network architecture as a distortion branch pre-training model; 4) Extract the distorted feature maps of the 3rd, 4th, 8th, and 9th feature reduction stages of EfficientNet-B0, and subtract the feature maps of the four stages from the two images in the distorted pair to obtain the feature difference map; 5) Design a multi-scale feature fusion module MFF to use multi-scale convolution to extract feature information in a more fine-grained manner. MFF uses 1×3, 3×1, and 3×3 convolutions to extract fine-grained features and perform element-by-element addition operations on these feature maps; 6) For feature maps at different stages, use different convolution strides to normalize the feature maps at multiple stages so that the width and height of the feature maps at the four stages after the MFF module are the same, and concatenate them in the channel dimension; use pooling layers and linear layers to predict the probability that the left image in the image pair has a lower distortion level; 7) Sigmoid is used as the activation function to constrain the prediction range to be between 0 and 1, and binary cross entropy loss is used as the loss function in the pre-training stage; 8) For the content feature branch, ResNet50 pre-trained on the classification task on ImageNet is used as the feature extractor; 9) In the quality prediction network, the pre-trained ResNet50 backbone and EfficientNet-B0 backbone are used as feature extractors for the content branch and the distortion branch and the weight parameters are frozen; the feature maps of the first three stages of the ResNet50 backbone network and the feature maps of the 3rd, 4th, and 8th stages of the EfficientNet-B0 backbone network are used to represent the extracted content features and distortion features respectively; 10) Design a deformable attention module DCA to sample the spatial and channel information of multi-level feature maps and fuse these multi-level feature maps from the two branches respectively; the DCA module combines deformable convolution and channel attention, and uses 1×1 residual connections in the spatial feature and channel feature sampling stages to effectively avoid feature degradation; 11) The content features and distortion features obtained by the dual-branch feature extractor are transformed into dimensions (C, H, W) → (N, C), and position encoding is performed on these features respectively; 12) Use Transformer encoder-decoder and construct weight tokens to capture the global interaction relationship between the two features; 13) After the encoder performs self-attention encoding, the first dimension feature vector of the encoder output feature is extracted as the weight token of the learned weight relationship; when decoding the weight feature, the content features and distortion features of the original two branches are used as decoder queries respectively, and the weight token is used as the key and value to perform a cross-attention mechanism to obtain the quality-weighted content features and distortion features; 14) During regression prediction, the weighted features of the two branches are transformed in dimension (N, C) → (C, H, W) and concatenated in the channel dimension to obtain the final fusion features. The global average pooling and linear layer are used to predict the quality score. SmothL1loss is used as the loss function to make the loss change smoother during the training process.

Images