A no-reference image quality evaluation method based on self-supervised learning and transformer

Abstract

The application discloses a kind of based on self-supervised learning and referenceless image quality evaluation method of Transformer, using supervised contrast learning method is carried out self-supervised training on large-scale unmarked image quality evaluation dataset KADIS, to learn the degradation characteristics of different distortion types and level distortion images;Multi-stream backbone network structure is designed in combination with CNN and Transformer characteristics, to extract the distortion characteristics corresponding to the image;Block attention module is designed, the degradation characteristics and distortion characteristics are fused, to obtain the final distortion image quality characteristics, for the regression prediction of image objective score.The application combines self-supervised learning and Transformer and proposes a new type of image quality evaluation method, which can effectively improve the quality score prediction performance of the model on small sample dataset and real dataset, and has a wide application prospect in a series of image compression and image super-resolution and other fields requiring image quality evaluation.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of digital image processing, and particularly relates to a no-reference image quality evaluation method based on self-supervised learning and a Transformer. BACKGROUND

[0002] With the growing of social networks and digital multimedia technology, image data presents explosive growth and is more and more closely related to people's life. However, in the process of collection, transmission to application of digital images, image data may be disturbed by various external factors, resulting in different degrees of damage to the quality of images. Common distortions that cause image distortion include blurring, compression and noise. In addition, generating high-quality images consistent with human vision has always been the goal pursued by computer vision application technology, such as image super-resolution reconstruction, image repair and image denoising. Therefore, it is particularly important to evaluate the quality of images with different distortion types and distortion degrees.

[0003] Image quality evaluation refers to quantitative analysis of the content of an image, thereby quantifying the distortion degree of a distorted image in vision. Related evaluation methods are generally divided into subjective quality evaluation and objective quality evaluation. Subjective quality evaluation refers to obtaining the quality of an image through observation of a distorted image by a human eye vision system. Objective quality evaluation generally refers to obtaining the quality information of an image by calculating a distorted image through a corresponding objective evaluation algorithm. Among them, according to whether the evaluation algorithm needs to refer to image information in the calculation process, the objective quality evaluation is divided into three categories: full-reference image quality evaluation, semi-reference image quality evaluation and no-reference image quality evaluation. Among them, the full-reference image quality evaluation and semi-reference image quality evaluation methods mainly calculate the quality of an image by comparing the differences between a reference image and a distorted image. The no-reference image quality evaluation method evaluates the quality of an image only according to the distorted image itself without reference information, and therefore becomes the most difficult and challenging task in the field of image quality evaluation, and has more extensive application prospects in practical applications.

[0004] Since the performance of deep neural networks depends largely on the size of the training dataset, however, current image quality assessment datasets are small, such as the LIVE Challenge database only contains 1162 distorted images, which is obviously not comparable to the million datasets of other visual tasks, therefore the performance of the convolutional neural network-based no-reference image quality assessment method on these small sample datasets is generally poor. In addition, it is very difficult to obtain a large number of labeled human subjective image quality scores, which makes it more difficult to design a no-reference image quality assessment method. Generally, to solve the problem of small data sets, data augmentation (such as flipping, cropping, rotating, etc.) can be used, unfortunately, these operations will more or less affect the true quality of the image. It is worth noting that although the convolutional neural network-based no-reference image quality assessment method can indeed extract the distortion information of the image to some extent, due to its small receptive field, it cannot model the image globally, so it is inevitable to lose some global information when evaluating the quality. SUMMARY

[0005] The purpose of the application is to provide a no-reference image quality assessment method based on self-supervised learning and Transformer, which can effectively improve the quality score prediction performance of the model on small sample datasets and real datasets.

[0006] Technical scheme: The application provides a no-reference image quality assessment method based on self-supervised learning and Transformer, comprising the following steps:

[0007] (1) Construct a no-reference image quality assessment network based on self-supervised learning and Transformer; the evaluation network comprises a feature extraction module, a multi-stream backbone network, an encoder network, a block attention module and a score prediction module;

[0008] (2) According to the supervised contrast learning method, construct the encoder network and perform self-supervised training on a large-scale unlabeled image quality assessment dataset to learn the degradation features corresponding to images of different distortion types and levels;

[0009] (3) Design a multi-stream backbone network structure combining the characteristics of CNN and Transformer to extract the distortion features corresponding to the image;

[0010] (4) Through the block attention module, the degradation features in step (2) and the distortion features in step (3) are fused to obtain the final distortion image quality features; the regression prediction of the image objective score is obtained through the score prediction module.

[0011] Further, the multi-stream backbone network of step (1) comprises a combination of three-stage multi-stream modules and block down-sampling, and the third stage only comprises one multi-stream module.

[0012] Further, the step (2) is implemented as follows:

[0013] Definition I represents the distorted image, where u∈{1,...,U} represents different distortion types, and u∈{1,...,V} represents different distortion levels for each u; the network structure composed of the encoder f(·) and the projection layer g(·) is used in combination with the contrast loss function to learn the degradation characteristics of the image, and the loss function is defined as follows:

[0014]

[0015] Where N represents the number of images in each batch, II represents an indicator function, τ represents a temperature coefficient, P(i) represents a set belonging to the same class i, |P(i)| is its cardinality, and φ(m,n)=m T n / ||m||2||n||2.

[0016] Further, the step (3) is implemented as follows:

[0017] The feature map extracted by the feature extraction module is divided into three parts according to the channel According to the channel dimension, the three parts are combined and connected through the residual connection CBAM module. And The three parts; X1 is obtained by variable convolution and linear layer X1', X2 is obtained by depth separable convolution, maximum pooling and up sampling X2', and X3 is obtained by multi-head attention and linear layer X3':

[0018] χ1′=FC(Deform(Conv(χ1)

[0019] χ2′=UpSample(MaxPool(DwConv(χ2)))

[0020] χ3′=FC(MHSA(χ3))

[0021] According to the channel dimension, the three parts are combined and connected through the residual connection CBAM module.

[0022] Further, the step (4) is implemented as follows,

[0023] Before feature fusion, map the image distortion feature information into K and V, and map the image degradation feature into Q, then calculate them through the attention mechanism to obtain the image quality feature S; finally, the objective image score can be regressed through the feature S.

[0024] Further, for obtaining Q, K and V, the calculation process of S is as follows:

[0025]

[0026] wherein D represents the feature dimension; after S is obtained, a global average pooling operation and a linear layer can be used to regress to the final quality score; the evaluation network is regarded as a mapping function F, and the network parameters are learned by minimizing the loss L(θ) between the network prediction block F(X; θ) and the corresponding true label Y, and the absolute error MAE is used as the loss function:

[0027]

[0028] wherein N is the number of training samples in each batch of images.

[0029] Beneficial effects: compared with the prior art, the beneficial effects of the present application: the present application starts from improving the quality score prediction performance of the model on small sample data sets and real data sets, and proposes a no-reference image quality evaluation method based on self-supervised learning and Transformer; in order to utilize large-scale unlabeled data and improve the prediction performance of the model on small sample data sets, a self-supervised learning method is used to obtain the degradation features corresponding to the distorted images; in order to more effectively extract the global and local distortion information of the distorted images, the present application combines the characteristics of CNN and Transformer to propose a feature extraction backbone network; in order to effectively fuse the degradation features corresponding to the distorted images and the distortion information, the present application also proposes a block attention module to obtain quality features for score prediction. BRIEF DESCRIPTION OF DRAWINGS

[0030] Figure 1 The self-supervised learning and Transformer-based no-reference image quality evaluation network structure constructed by the present application intends to;

[0031] Figure 2 The self-supervised training flowchart proposed by the present application intends to;

[0032] Figure 3 The self-supervised training model framework diagram proposed by the present application intends to;

[0033] Figure 4 The multi-stream module diagram proposed by the present application intends to;

[0034] Figure 5 The method proposed by the present application and the DMOS prediction result comparison diagram of MANIQA intends to;

[0035] Figure 6 The Grad-CAM result diagram of the method proposed by the present application intends to;

[0036] Figure 7 The prediction value and true value scatter diagram of the method proposed by the present application and MANIQA intends to;

[0037] Figure 8 t-SNE results of the method proposed in the application on the CSIQ dataset. DETAILED DESCRIPTION

[0038] The application will be further described in detail below with reference to the accompanying drawings.

[0039] The application provides a no-reference image quality evaluation method based on self-supervised learning and a Transformer, specifically comprising the following steps:

[0040] Step 1: constructing a no-reference image quality evaluation network based on self-supervised learning and a Transformer as shown in Figure 1 The evaluation network comprises a feature extraction module, a multi-stream backbone network, an encoder network, a block attention module and a score prediction module.

[0041] The feature extraction module, i.e., a pre-trained ViT model, is used for image feature extraction; the multi-stream backbone network comprises a combination of multi-stream modules in three stages and block downsampling, and the third stage only comprises one multi-stream module; the encoder network used in the self-supervised pre-training process is trained in a self-supervised manner on the KADIS dataset to obtain degradation features corresponding to images of different distortion types and levels; the block attention module fuses the degradation features of the distorted images extracted by the backbone network and the degradation features extracted by the encoder; and the score prediction module predicts the final scores of the distorted images according to the fused features.

[0042] Step 2: using a supervised contrastive learning method to perform self-supervised training on a large-scale unlabeled image quality evaluation dataset KADIS, so as to learn the degradation features of distorted images of different distortion types and levels.

[0043] During the self-supervised training, first, different distorted images are classified according to distortion types and levels; then, a network model composed of an encoder and a projection layer is constructed; and finally, the training of the model is completed by using a normalized cross-entropy loss function with a temperature coefficient.

[0044] First, define I represents a distorted image, wherein u represents different distortion types, and u represents different distortion levels of each u. Thus, there are UxV+1 categories in total including the reference image. As shown in Figure 2 and Figure 3 Then, a network structure composed of an encoder f(·) and a projection layer g(·) is used to learn the degradation features of the image in combination with a contrastive loss function, and the loss function is defined as follows:

[0045]

[0046] where N denotes the number of images in each batch, denotes an indicator function, τ denotes a temperature coefficient, P(i) denotes a set belonging to the same class i, |P(i)| is its cardinality, φ(m, n) = m T n / ||m||2||n||2.

[0047] Step 3: A multi-stream backbone network structure combining the characteristics of CNN and Transformer is designed to extract the distortion features corresponding to the image.

[0048] First, the feature map after feature extraction is divided into three parts according to the channel dimension, the first part passes through variable convolution and linear layer, the second part passes through depth separable convolution, max pooling and up sampling layer, and the third part passes through multi-head attention and linear layer; then, it is merged according to the channel dimension and passes through the residual connection CBAM module, i.e., channel attention mechanism and spatial attention mechanism.

[0049] As shown in Figure 4 , first, the feature map extracted after the feature extraction module is divided into three parts and according to the channel. χ1 is obtained by variable convolution and linear layer, χ2 is obtained by depth separable convolution, max pooling and up sampling, and χ3 is obtained by multi-head attention and linear layer.

[0050] χ1′ = FC(Deform(Conv(χ1)

[0051] χ2′ = UpSample(MaxPool(DwConv(χ2)))

[0052] χ3′ = FC(MHSA(χ3))

[0053] Then, it is merged according to the channel dimension and passes through the residual connection CBAM module.

[0054] Step 4: A block attention module is designed to fuse the degradation features in step 2 and the distortion features in step 3 to obtain the final distortion image quality features for regression prediction of image objective score.

[0055] Before feature fusion, map the image distortion feature information into K and V, and map the image degradation feature into Q, then calculate them through attention mechanism to obtain the image quality feature S. Finally, the objective image score can be regressed through the feature S.

[0056] For obtaining Q, K and V, the calculation process of S is as follows:

[0057]

[0058] where D denotes the feature dimension. After obtaining S, the final quality score can be regressed by a global average pooling operation and a linear layer. The whole network is regarded as a mapping function F, and the network parameters are learned by minimizing the loss L(θ) between the network prediction block F(X; θ) and the corresponding true label Y, using the absolute error MAE as the loss function:

[0059]

[0060] where N is the number of training samples in each batch of images.

[0061] The method proposed in the application randomly crops 8 224x224 image blocks from each image during image preprocessing, and randomly flips and crops. The trained image blocks inherit the quality score of the original image. The encoder used in contrast learning is a modified ResNet-50, and the feature extraction part of the backbone network uses ViT-B / 8, where the size of the image block is 8. The model is trained on ImageNet-21k and fine-tuned on ImageNet-1k. The model contains 3 stages, the first two stages are composed of multi-stream modules and block downsampling, and the last stage only contains multi-stream modules, where D is set to 768, h4 and w4 are 7. In the three stages, the number of channels D1, D2 and D3 of each branch of the multi-stream module is: [256, 256, 256], [192, 192, 384], [48, 48, 672] respectively.

[0062] The training strategy used by the existing no-reference image quality evaluation algorithm is the same. First, randomly select 5 random seeds to divide the dataset into 80:20. 80% is used for training, and the rest is used for testing. For the synthetic dataset, divide the dataset according to the reference image. During training, use the ADAM optimizer with a learning rate of 2e-5, a weight decay of 1e-5, and a cosine annealing learning strategy. Tmax is set to 50, and etamin is 0. The loss function uses MAE, and the batch size is 8. When testing, the final objective quality score is generated by averaging the scores of 20 patches of the distorted image. By running five different seed experiments, the average value of the results is reported.

[0063] In order to verify the effectiveness of the application, taking the JPEG distortion type image on the CSIQ test set as an example, the DMOS value prediction results of MANIQA and the model proposed in the application are shown as follows: Figure 5As shown, the method proposed in this invention outperforms MANIQA in both image distortion level ranking and overall DMOS value prediction. MANIQA, in particular, produces errors in predicting the fourth and fifth distortion levels. Figure 6 The Grad-CAM results of the method proposed in this invention are plotted. It can be seen that the main focus of this invention is on the distorted parts of the image, which conforms to the human eye's judgment of distortion. Figure 7 As shown, the modified ResNet-50 model was trained using a self-supervised approach on the KADIS dataset and then tested on the CSIQ dataset. The model parameters were not adjusted during the testing process. The test results were plotted using t-SNE. Figure 7 In the image, (a) and (b) represent the feature clustering results of the model on the CSIQ dataset for six types of distortion. It can be seen that after training with self-supervised learning, the model has a strong ability to distinguish between different types of distortion. Figure 7 Images (c) and (d) show the visualization results of the model at five levels for a specific distortion type, demonstrating the model's strong discriminative ability across various distortion levels for AWGN and fnoise distortion types. This also indicates that the model proposed in this invention has strong generalization performance. Figure 8 In this study, the predicted scores and ground truth values ​​of MANIQA and the present invention on the CSIQ and KONIQ datasets were plotted as scatter plots and linearly fitted. Figure 8 As can be seen in (a) and (b), the prediction results of this invention have a stronger linear relationship with the true value on the CSIQ dataset. Figure 8 As can be seen in (c) and (d), MANIQA exhibits more outliers on the KONIQ dataset, while the method proposed in this invention yields more consistent results. In summary, this invention can effectively improve the score prediction performance of the model on different image quality assessment datasets.

[0064] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims.

[0065] In addition, it should be understood that, although the present specification is described in terms of embodiments, not every embodiment exhibits only one independent technical solution, and the present specification is described in this manner only for the sake of clarity, and those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that those skilled in the art can understand.

Claims

1. A no-reference image quality assessment method based on self-supervised learning and Transformer, characterized in that, Includes the following steps: (1) Construct a no-reference image quality evaluation network based on self-supervised learning and Transformer; the evaluation network includes a feature extraction module, a multi-stream backbone network, an encoder network, a block attention module and a score prediction module; (2) Based on the supervised contrastive learning method, an encoder network is constructed and self-supervised training is performed on a large-scale unlabeled image quality assessment dataset to learn the degradation features corresponding to images with different distortion types and levels. (3) Design a multi-stream backbone network structure by combining the characteristics of CNN and Transformer to extract the distortion features corresponding to the image; (4) The degradation features in step (2) and the distortion features in step (3) are fused through the block attention module to obtain the final distortion image quality features; the regression prediction of the objective image score is obtained through the score prediction module. The multi-stream backbone network in step (1) consists of a combination of three stages of multi-stream modules and block downsampling, with the third stage containing only one multi-stream module; The implementation process of step (3) is as follows: The feature map after feature extraction module Divided into three parts according to the passage. and The process consists of three parts: χ1 is obtained by variable convolution and linear layers to obtain χ1′; χ2 is obtained by depthwise separable convolution, max pooling and upsampling to obtain χ2′; and χ3 is obtained by multi-head attention and linear layers to obtain χ3′. χ1′=FC(Deform(Conv(χ1)) χ2'=UpSample(MaxPool(DwCon(χ2)) χ3′=FC(MHSA(χ3)) Then, the channels are merged and the residuals are connected to the CBAM module.

2. The method for evaluating the quality of a no-reference image based on self-supervised learning and Transformer as described in claim 1, characterized in that, The implementation process of step (2) is as follows: definition I represents a distorted image, where u∈{1,...,U} represents different distortion types, and v∈{1,...,V} represents different distortion levels for each u. A network structure consisting of an encoder f(·) and a projection layer g(·) is used in conjunction with a contrastive loss function to learn the degradation features of the image. The loss function is defined as follows: Where N represents the number of images in each batch, Let τ represent the indicator function, τ represent the temperature coefficient, P(i) represent the set of i belonging to the same class, |P(i)| is its cardinality, and φ(m,n) = m. T n / ||m||2||n||2.

3. The method for evaluating the quality of a no-reference image based on self-supervised learning and Transformer as described in claim 1, characterized in that, The implementation process of step (4) is as follows: Before feature fusion, the image distortion features are mapped to K and V, and the image degradation features are mapped to Q. Then, they are calculated through an attention mechanism to obtain the image quality features S. Finally, the objective image score can be regressed through the features S.

4. The method for evaluating the quality of a no-reference image based on self-supervised learning and Transformer according to claim 3, characterized in that, The calculation process for obtaining Q, K, and V is as follows: Where D represents the feature dimension; after obtaining S, the final quality score can be obtained by global average pooling and linear layers; the evaluation network is regarded as a mapping function F, and the network parameters θ are learned by minimizing the loss L(θ) between the network predicted block F(X; θ) and the corresponding true label Y, using the absolute error MAE as the loss function: Where N is the number of training samples in each batch of images.

Images