A multi-domain distortion learning based no-reference image quality assessment method

Abstract

The application discloses a kind of based on multi-domain distortion learning's no reference image quality evaluation method, first construct based on multi-domain distortion learning's no reference image quality evaluation model;Then the input distorted image is degraded and repaired;Complete multi-domain distortion learning, obtain corresponding frequency characteristics;Again carry out multi-scale semantic feature extraction and edge information extraction;Finally according to the multi-scale frequency characteristics, multi-scale semantic feature and image edge information obtained, realize feature fusion and quality regression.The application proposes using multi-domain distortion learning method, help model to identify various noises in image information, and effectively improve image quality.The application introduces frequency domain information to assist in predicting quality score on the basis of image semantic features.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image and video processing, and in particular to a feature learning method combining multi-domain distortion learning and a deep learning-based no-reference image quality assessment method. BACKGROUND

[0002] Image quality assessment (IQA) is a very basic and important task in the field of machine vision. IQA refers to evaluating the visual distortion degree of an image by analyzing the relevant characteristics of the image signal. IQA methods can be divided into subjective IQA methods and objective IQA methods. Subjective IQA methods refer to obtaining the visual quality of an image through subjective judgments of a large number of observers. Objective IQA refers to automatically calculating the visual quality of an image through an objective algorithm. In particular, objective IQA can be further divided into full-reference IQA, semi-reference IQA, and no-reference IQA. Compared with full-reference IQA and semi-reference IQA, no-reference IQA does not require reference picture information, and it has a broader application market and actual value. According to whether the subjective score of an image is needed for training when calculating the image degradation quality, no-reference IQA can be divided into supervised learning-based no-reference IQA and unsupervised learning-based no-reference IQA.

[0003] Supervised learning-based no-reference IQA mainly includes traditional machine learning-based methods and deep learning-based methods. Traditional machine learning-based methods aim to design an effective visual feature expression method, and learn a mapping model from visual features to image degradation quality through traditional machine learning methods such as support vector regression.

[0004] With the vigorous development of deep learning, computer vision has reached a new height, and there has been a great breakthrough in image and video processing. Traditional methods of image quality assessment relying on modeling of human visual effects (HVS) are complex and computationally intensive, and since user-generated outdoor shooting images contain a large number of distortion types, it is difficult for traditional modeling algorithms to achieve high prediction accuracy. Therefore, deep learning-based more efficient and more accurate video quality assessment methods have gained sustained research interest.

[0005] Existing deep learning-based NR-IQA methods mainly rely on image semantic information and local feature information for feature judgment, and pay little attention to the degradation of the image itself. We hope to pay more attention to the degradation information rather than just focusing on semantic information, and to be able to distinguish good and bad pictures in similar scenarios. For the NR-IQA task, it is an important and extremely challenging task for the algorithm to perceive the quality of the content change image, such as Figure 1 As shown in FIG. 1, the existing model prediction score often cannot correctly distinguish the quality of the pictures in a similar scenario. SUMMARY

[0006] In view of the deficiencies of the existing quality evaluation methods, the application provides a no-reference image quality evaluation method based on multi-domain distortion learning.

[0007] According to the research of HVS, it is shown that HVS will infer the environment according to the internal state. Inspired by this, we assume that HVS will predict the appearance of the original image when perceiving the degraded image, and then compare the degraded-imagined restored image pair to judge the degradation quality. Therefore, we think it is necessary to repair the features of the real shooting scene image based on a simple and effective model. The distorted degraded image is compared with the repaired image to help quality prediction. At the same time, the human eye is particularly sensitive to the medium and high frequency information in the image content and image edge information. Therefore, we can process the frequency domain information differently according to the different sensitivity of the human eye to different frequency domains. Therefore, it is necessary to extract the features of the medium and high frequency information to learn the image degradation quality, so as to help the model to correctly distinguish the image quality.

[0008] A no-reference image quality evaluation method based on multi-domain distortion learning, comprising the following steps:

[0009] Step (1), constructing a no-reference image quality evaluation model based on multi-domain distortion learning;

[0010] The no-reference image quality evaluation model based on multi-domain distortion learning comprises a distortion image restoration module (Restoration) and a network main module.

[0011] The network main module is composed of a multi-domain distortion extraction submodule and a feature fusion submodule. The multi-domain distortion extraction submodule is composed of a frequency domain distortion extraction module, a semantic feature extraction module and an edge information extraction module. The feature fusion submodule includes multi-scale feature connection and "Combiner" feature fusion module.

[0012] Step (2), distortion image restoration:

[0013] The distortion image restoration module (Restoration) is used to restore the input distorted image (i.e. quality degraded image).

[0014] Step (3), multi-domain distortion learning is completed through the multi-domain distortion extraction submodule to obtain the corresponding frequency features.

[0015] Step (4), multi-scale semantic feature extraction is completed through the semantic feature extraction module;

[0016] Step (5), edge information extraction is completed through the edge information extraction module;

[0017] Step (6): Based on the obtained multi-scale frequency features, multi-scale semantic features and image edge information, feature fusion and quality regression are achieved;

[0018] Step (7): Training method for constructing a no-reference image quality assessment model based on multi-domain distortion learning.

[0019] Furthermore, the overall structure of the distorted image restoration module is an "encoder-decoder" model. The encoder sub-module consists of three 3x3 convolutions, a Batch Norm layer, and a Leaky ReLU activation layer; the decoder sub-module consists of three 3x3 convolutions, a Batch Norm layer, and a ReLU activation layer. The restored image and the degraded image are paired and used as the input to the main network module.

[0020] Furthermore, the frequency distortion extraction module first obtains restored images and degraded images at different scales through three downsampling operations, and then uses a frequency feature extractor to extract frequency features from the restored images and degraded images at different scales to obtain the corresponding frequency features f1, f2, f3, f4.

[0021] Furthermore, the specific method for step (4) is as follows:

[0022] A pre-trained convolutional neural network is used as the semantic feature extraction module. The pre-trained convolutional neural network is a network built using an image classification model pre-trained on the ImageNet21k dataset using a ResNet50 network, which retains the feature extraction modules of stages 1-4.

[0023] Used to extract semantic features of content at multiple scales from image pairs. Specifically:

[0024]

[0025] in, Indicates that it comes from i th The ResNet50 of the block, where i∈{1,2,3,4}, b represents the batch size, c i ,m i ,n i This represents the channel size, width, and height. The output of the last layer of each stage is used as a multi-scale semantic feature extracted from the input image.

[0026] Furthermore, the edge information extraction module (ARM) consists of two asymmetric residual blocks, which are used to extract image edge information from the original image pairs. Specifically, each asymmetric residual block consists of three parallel 3*3, 3*1, and 1*3 convolutions. The outputs of the three convolutions are merged and then followed by a batch normalization (BN) layer and a pre-return (PReLU) layer.

[0027] Furthermore, the specific method for step (6) is as follows;

[0028] First, a concat operation is performed on frequency features and semantic features of the same scale. For the frequency features and semantic features extracted from the original scale image, a concat operation needs to be performed with the image edge information to obtain the corresponding summed features.

[0029] The summed features are input into the feature connection part of the feature fusion submodule. The multi-scale features are aggregated into the first-layer connection link through the "add" operation. Finally, the features of the four scales are fused by the "Combiner" feature fusion module, and the predicted quality score is output through the "Head" layer.

[0030] Furthermore, the specific method for step (7) is as follows:

[0031] The perceptual loss is calculated pairwise between the feature maps of the image to be evaluated and the feature maps of the restored image at four scales of the model, as expressed in the following formula:

[0032]

[0033] Where r and d represent the degraded image and the restored image, respectively; C j W j H j Let Ω represent the feature map size at the j-th layer, and Ω() represent the proposed model function.

[0034] The model is trained using the Adam optimizer with zero weight decay and an initial learning rate of 2*10. -5 After that, the model is boosted by 10% every 10 epochs. A pre-trained ResNet50 network is used to initialize the model's weights, and the L1 loss function is used during training, as follows:

[0035]

[0036] The overall loss function of the model is as follows:

[0037] L=λ1L1+λ2PLoss (5)

[0038] Where λ1 and λ2 are 0.5 respectively.

[0039] Specifically, during the training phase, for artificially synthesized distortion datasets, since reference images are already available, the reference images are directly input into the main model as the repaired images for joint training. For naturally distorted datasets, image inpainting operations are required for the distorted images.

[0040] The beneficial effects of this invention are as follows:

[0041] This invention proposes a multi-domain distortion learning method to help models identify various noises in image information and effectively improve image quality. Existing deep learning-based NR-IQA methods mainly rely on image semantic information and local feature information for feature judgment, rarely paying attention to the degradation of the image itself. We hope to focus not only on semantic information but also on degradation information, and be able to distinguish between good and bad images in similar scenes. This invention proposes a multi-domain distortion learning method, which introduces frequency domain information on top of image semantic features to assist in predicting quality scores. Attached Figure Description

[0042] Figure 1 This is an example illustrating how existing methods cannot correctly distinguish between images;

[0043] Figure 2 This is a framework diagram of the method of the present invention. Detailed Implementation

[0044] The invention will now be further described with reference to the accompanying drawings.

[0045] The method of this invention consists of an image inpainting module, a multi-scale frequency domain distortion learning module, a multi-scale semantic feature extraction module, a feature fusion module, and a quality regression model. Given a distorted RGB image I∈R3×w×h, it is input into the image inpainting module to obtain the inpainted image I. restore We input both the distorted and restored images as image pairs into the main network. A multi-scale frequency domain distortion learning module extracts the frequency domain distortion information of the input image pairs. A pre-trained convolutional neural network extracts multi-scale semantic features of the images, and we also extract edge information. Then, we fuse the multi-scale distortion prior information and semantic information and input it into the feature fusion module to generate feature maps. We then use the features of the restored image to perform perceptual loss learning on the features of the image to be evaluated. Finally, a quality regression module is used to aggregate the quality scores of the feature maps.

[0046] Step (1): Construct a no-reference image quality assessment model based on multi-domain distortion learning;

[0047] The aforementioned no-reference image quality assessment model based on multi-domain distortion learning includes a distortion image restoration module and a main network module.

[0048] The main network module consists of a multi-domain distortion extraction submodule and a feature fusion submodule. The multi-domain distortion extraction submodule comprises a frequency domain distortion extraction module, a semantic feature extraction module, and an edge information extraction module. The feature fusion submodule includes multi-scale feature concatenation and "Combiner" feature fusion.

[0049] Step (2), Distorted Image Restoration:

[0050] The distortion restoration module repairs the input distorted image (i.e., the image with degraded quality).

[0051] like Figure 2 As shown in the "Restoration" module, its overall structure is an "encoder-decoder" model. The encoder sub-module consists of three 3x3 convolutions, a Batch Normal layer, and a Leaky ReLU activation layer; the decoder sub-module also consists of three 3x3 convolutions, a Batch Normal layer, and a ReLU activation layer. We use the restored image and the degraded image as image pairs as input to the main network module for feature-supervised learning, helping the model better perceive quality degradation.

[0052] Step (3), multi-domain distortion learning, the specific method is as follows:

[0053] The frequency distortion extraction module first performs three downsampling operations, each time reducing the height H and width W of the image by half and increasing the number of channels C by a corresponding factor, to obtain restored images and degraded images of different scales ([C,H,W],[4C,H / 2,W / 2],[16C,H / 4,W / 4],[64C,H / 8,W / 8]). Then, a frequency feature extractor is used to extract frequency features from the restored images and degraded images of different scales to obtain the corresponding frequency features f1, f2, f3, f4.

[0054] A distorted image is composed of a clean image plus distorted feature information, which can be expressed by the formula (1):

[0055] I dis =I clear +N Dis (1)

[0056] Among them, I dis For distorted images, I clear For a clean image, N Dis The distortion characteristics were characterized. Block-DCT is an effective method for handling frequency domain correlation problems, therefore we will use N Dis The translation into the DCT domain is processed as follows:

[0057] (3-1) Convert the input to a grayscale image;

[0058] (3-2) Perform Discrete Cosine Transform (DCT) on the grayscale image;

[0059] (3-3) In the DCT domain, the low-frequency component is located in the upper left corner. If the coefficient of the low-frequency component is set to zero, then the result is an image that only contains mid-to-high frequency information.

[0060] Step (4) Semantic feature extraction, the specific method is as follows:

[0061] This invention utilizes a pre-trained convolutional neural network as a semantic feature extraction module. The pre-trained convolutional neural network involved is a network constructed using an image classification model pre-trained on the ImageNet21k dataset using a ResNet50 network, which retains the feature extraction modules of stages 1-4.

[0062] Used to extract semantic features of content at multiple scales from image pairs. Specifically:

[0063]

[0064] in, Indicates that it comes from i th The ResNet50 of the block, where i∈{1,2,3,4}, b represents the batch size, c i ,m i ,n i This represents the channel size, width, and height. We use the output of the last layer of each stage as multi-scale semantic features extracted from the input image. It's worth noting that features extracted from different stages have different scales, structures, and dimensions. Low-level semantic information about image structure represents basic image information such as the underlying layers, texture, shape, and color. This information is directly related to the IQA task; while deep semantic information represents the closest thing to human understanding in image representation, possessing a high degree of abstraction and a certain quality assessment capability, and can provide attention mechanisms corresponding to local distortions.

[0065] Step (5), edge information extraction, the specific method is as follows:

[0066] In human vision, image edge information is crucial, especially the positional information of edges. The human eye can easily perceive changes in edge position. Therefore, extracting image edge contour information at the original image scale helps models better perceive and locate target regions. However, common convolutional layers struggle to extract edge information effectively. Inspired by ACNet, our Edge Information Extraction Module (ARM) consists of two asymmetric residual blocks, which extract image edge information from original image pairs. Specifically, each asymmetric residual block consists of three parallel 3x3, 3x1, and 1x3 convolutions. The outputs of these three convolutions are merged and then followed by a Batch Normalization (BN) layer and PReLU.

[0067] Step (6), feature fusion and quality regression, the specific methods are as follows:

[0068] First, multi-scale frequency features, multi-scale semantic features, and image edge information are added together. Frequency features and semantic features of the same scale are then concatenated. For images at the original scale, the extracted frequency and semantic features need to be concatenated with the image edge information to obtain the corresponding added features.

[0069] The summed features are input into the feature fusion submodule's feature connection section. Multi-scale features are aggregated into the first-layer connection link using the "add" operation. Finally, the features from the four scales are aggregated through the "Combiner" layer. The "Combiner" layer uses the torch.cat method to concatenate features along the channel number.

[0070] This invention uses a feature connectivity model in the shape of a "feature pyramid" to enhance... and The model integrates multi-scale frequency domain information, semantic information, and image edge information to generate rich feature maps. Multiple feature maps are obtained during model propagation. These feature maps are input into the "Combiner" feature extraction module, which concatenates the features along the channel number direction to fuse them. Finally, the predicted quality score is output through the "Head" layer.

[0071] Step (7), the model training method is as follows:

[0072] Specifically, during the training phase, for artificially synthesized distortion datasets, since reference images are already available, the reference images are directly input into the main model as the repaired images for joint training. For naturally distorted datasets, we need to perform image inpainting on the distorted images.

[0073] Since the model input is a pair of images, and the restored image also yields corresponding feature maps through model inference, these feature maps extracted from the restored image do not participate in the quality regression task. We only use a perceptual loss function to constrain the training of these feature maps and their corresponding feature maps from the distorted image (i.e., the image to be evaluated). Specifically, the perceptual loss is calculated pairwise between the feature maps of the image to be evaluated and the feature maps of the restored image at each of the model's four scales, as expressed in the following formula:

[0074]

[0075] Where r and d represent the degraded image and the restored image, respectively; C j W j H j Let Ω represent the feature map size at the j-th layer, and Ω() represent the proposed model function.

[0076] The model in this invention is written using the PyTorch framework. The model is trained using the Adam optimizer, with weight decay of 0 and an initial learning rate of 2*10^6. -5 After that, the model is boosted by 10% every 10 epochs. A pre-trained ResNet50 network is used to initialize the model's weights, and the L1 loss function is used during training, as follows:

[0077]

[0078] The overall loss function of the model is as follows:

[0079] L=λ1L1+λ2PLoss (5)

[0080] Where λ1 and λ2 are 0.5 respectively.

[0081] Figure 1 Three image pairs are presented, each consisting of an upper and lower image pair, with the upper image of better quality and the lower image of poorer quality. We compare the quality predictions using HyperIQA, TReS models, and our proposed model, Ours, and the results are expressed as the mean subjective opinion score (MOS), ranging from [1, 10]. Figure 1 It can be seen that TReS and HyperIQA have the problem of incorrectly distinguishing image pairs with quality differences, while our method can correctly distinguish image pairs with quality differences.

[0082] The above description, in conjunction with specific / preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. Those skilled in the art can make various substitutions or modifications to these described embodiments without departing from the inventive concept, and all such substitutions or modifications should be considered within the scope of protection of the present invention.

[0083] The parts of this invention not described in detail are well-known to those skilled in the art.

Claims

1. A no-reference image quality assessment method based on multi-domain distortion learning, characterized in that, The steps include the following: Step (1): Construct a no-reference image quality assessment model based on multi-domain distortion learning; The aforementioned no-reference image quality assessment model based on multi-domain distortion learning includes a distortion image inpainting module and a network main module; The main network module consists of a multi-domain distortion extraction submodule and a feature fusion submodule; the multi-domain distortion extraction submodule is composed of a frequency domain distortion extraction module, a semantic feature extraction module, and an edge information extraction module; the feature fusion submodule includes multi-scale feature connection and a "Combiner" feature fusion module. Step (2), Distorted Image Restoration: The distorted image restoration module is used to restore the input distorted image. Step (3): Multi-domain distortion learning is completed through the multi-domain distortion extraction submodule to obtain the corresponding frequency features; Step (4): Multi-scale semantic feature extraction is completed through the semantic feature extraction module; Step (5): Edge information extraction is completed through the edge information extraction module; Step (6): Based on the obtained multi-scale frequency features, multi-scale semantic features and image edge information, feature fusion and quality regression are achieved; Step (7): Train the no-reference image quality assessment model based on multi-domain distortion learning; The overall structure of the distorted image restoration module is an "encoder-decoder" model. The encoder sub-module consists of three 3*3 convolutions, a Batch Norm layer, and a Leaky ReLU activation layer; the decoder sub-module consists of three 3*3 convolutions, a Batch Norm layer, and a ReLU activation layer; the restored image and the degraded image are combined to form an image pair, which serves as the input to the main network module. The frequency domain distortion extraction module first obtains restored and degraded images at different scales through three downsampling operations. Then, a frequency feature extractor is used to extract frequency features from the restored and degraded images at different scales to obtain the corresponding frequency features. .

2. The no-reference image quality assessment method based on multi-domain distortion learning according to claim 1, characterized in that, The specific method for step (4) is as follows: A pre-trained convolutional neural network is used as the semantic feature extraction module. The pre-trained convolutional neural network is a network built using an image classification model pre-trained on the ImageNet21k dataset using the ResNet50 network, which retains the feature extraction modules of stages 1-4. Used to extract multi-scale semantic features of content from image pairs; specifically as follows: （2） in, Indicates from The ResNet50 of the block, where 'b' represents the batch size. It represents the channel size, width, and height; the output of the last layer of each stage is used as a multi-scale semantic feature extracted from the input image.

3. The no-reference image quality assessment method based on multi-domain distortion learning according to claim 1, characterized in that, The edge information extraction module consists of two asymmetric residual blocks, which are used to extract image edge information from the original image pairs. Specifically, each asymmetric residual block consists of three parallel 3*3, 3*1 and 1*3 convolutions. The outputs of the three convolutions are merged and then followed by a BN layer and PReLU.

4. The no-reference image quality assessment method based on multi-domain distortion learning according to claim 1, characterized in that, The specific method for step (6) is as follows; First, a concat operation is performed on frequency features and semantic features of the same scale. For the frequency features and semantic features extracted from the original scale image, a concat operation needs to be performed with the image edge information to obtain the corresponding summed features. ; The summed features are input into the feature connection part of the feature fusion submodule. The multi-scale features are aggregated into the first-layer connection link through the "add" operation. Finally, the features of the four scales are fused by the "Combiner" feature fusion module, and the predicted quality score is output through the "Head" layer.

5. A no-reference image quality assessment method based on multi-domain distortion learning according to any one of claims 1-4, characterized in that, The specific method for step (7) is as follows: The perceptual loss is calculated pairwise between the feature maps of the image to be evaluated and the feature maps of the restored image at four scales of the model, as expressed in the following formula: （3） Where r and d represent the degraded image and the restored image, respectively; This represents the size of the feature map at layer j. This represents the proposed model function; The model is trained using the Adam optimizer with zero weight decay and an initial learning rate of 2*10. -5 After that, the model is boosted by 10% every 10 epochs; a pre-trained ResNet50 network is used to initialize the model weights, and the L1 loss function is used during training, as follows: （4） The overall loss function of the model is as follows: （5） in They are 0.5 respectively.

6. The no-reference image quality assessment method based on multi-domain distortion learning according to claim 5, characterized in that, During the training phase, since the artificially synthesized distortion dataset already has reference images, the reference images are directly input into the main model as repaired images for joint training; for the natural distortion dataset, image restoration operations are required for the distorted images.

Images