A no-reference image quality assessment method based on distortion prior learning

Abstract

The present invention discloses a no-reference image quality assessment method based on distortion prior learning. The present invention comprises the following steps: step (1). distortion prior learning; step (2). semantic feature extraction; step (3). edge information extraction; step (4). feature aggregation; step (5). distortion masking and quality regression; step (6). model training. The present invention proposes a distortion prior learning method, which introduces frequency domain prior information on the basis of image semantic features to assist in predicting quality scores. The method of the present invention has been extensively experimented on four public IQA datasets. The results show that the proposed model is significantly superior to the most advanced NR-IQA method in terms of generalization ability and evaluation accuracy. The results show that the proposed model is significantly superior to the most advanced NR-IQA method in terms of generalization ability and evaluation accuracy.

Description

Technical Field

[0001] The present invention relates to the field of image and video processing technology, and in particular to a method for performing reference-free image quality evaluation by combining a feature learning method with distortion prior learning and deep learning. Technical Background

[0002] Image quality assessment (IQA) is a fundamental and crucial task in the field of machine vision. Image quality is crucial to user experience and many image analysis applications. However, various factors, such as the shooting environment, data storage, compression, transmission, and processing, can introduce noise, affecting image quality. The goal of IQA is to enable computers to perceive image quality similarly to humans. IQA methods can be categorized into subjective and objective methods. Subjective IQA methods rely on subjective judgment to determine image visual quality, but this is time-consuming and labor-intensive. Objective IQA automatically calculates image quality using objective algorithms. Specifically, objective IQA can be further categorized into full-reference IQA, semi-reference IQA, and no-reference IQA. Compared to full-reference and semi-reference IQA, no-reference IQA does not require reference image information and has a broader application market and practical value. No-reference IQA can be categorized into non-reference IQA using specific distortion methods and non-reference IQA using general methods. Distortion-specific NR-IQA methods require some prior information, such as the image distortion type. Distorted images typically exhibit one or more distortion types, and image quality is assessed specifically based on the characteristics of the specific distortion. Examples include NR-IQA methods for JPEG images, contrast-distorted images, and blurred images. To effectively address the limitations of distortion-specific NR-IQA methods, general NR-IQA methods have emerged that assess image quality without knowing the distortion type.

[0003] Early general NR-IQA methods used traditional handcrafted feature-based NR-IQA methods to assess image quality and widely used statistical features of natural scenes. Natural scene statistics (NSS) refers to the fact that the visual features (such as brightness and gradient) of "good quality" images follow a certain distribution pattern, and different types or degrees of distortion will have a corresponding impact on this distribution pattern. Early methods based on traditional handcrafted features could only extract obvious features and were unable to effectively extract deep features that are invisible to the human eye. Therefore, when given a large dataset, the model performance was poor.

[0004] With the rapid development of deep learning, the field of computer vision has reached new heights, with significant breakthroughs in image and video processing. Traditional methods for image quality assessment, which rely on modeling the human visual effect (HVS), are complex and computationally intensive. Furthermore, due to the wide variety of distortion types inherent in user-generated, outdoor images, traditional modeling algorithms struggle to achieve high prediction accuracy. Consequently, more efficient and accurate video quality assessment methods based on deep learning continue to attract significant research interest.

[0005] Existing deep learning-based NR-IQA methods primarily rely on image semantics and local features for feature analysis. These methods are highly sensitive to scene variations and, therefore, are unable to effectively identify the underlying characteristics that cause quality degradation in distorted images. For NR-IQA, ensuring that algorithms perceive the quality of images with varying content is a crucial and challenging task. For example, a picture of blue sky and white clouds is superior to a blurred image of an animal, yet existing algorithms often misclassify them.

[0006] Currently, image prior learning has not been used in NR-IQA tasks. We believe that effective image prior information can help distinguish different distortion information from image content and alleviate the model's sensitivity to scene changes. Early research focused on learning specific distortion priors, which are often hand-crafted feature information and lack observations. With the development of deep learning, a large number of works have used deep learning to learn priors. The human eye is particularly sensitive to mid- and high-frequency information in image content and image edge information. Therefore, we can process frequency domain information differently based on the human eye's different sensitivities to different frequency domains. Previous research on HVS has shown that the human visual system iteratively performs a weighted sum operation on the signal, which is equivalent to passing the signal through a bandpass filter. Therefore, we believe it is necessary to introduce a learnable bandpass filter to learn distortion degradation priors in the NR-IQA task. Summary of the Invention

[0007] In view of the shortcomings of existing quality assessment methods, the present invention provides a method for no-reference image quality assessment using distortion prior learning.

[0008] Step (1). Distortion prior learning, the specific method is as follows:

[0009] 1-1. Using Learnable Bandpass Filters (LBF) as a distortion prior learning module. Assume that the distorted image I dis is a clean image I clear Combined with the distortion feature information, it can be expressed as follows:

[0010] I dis =Iclear +N Dis (1.1)

[0011] Among them, N Dis Characterizes the distortion characteristics. Block-DCT is an effective method for dealing with frequency domain related problems, so we Dis Translated into DCT domain for processing. Therefore, the pixel domain distortion N P and the distortion N of the feature domain f Can be expressed as:

[0012]

[0013] in Denotes IDCT operation, C denotes convolution operation. Inspired by implicit DCT, we can directly use a deep CNN block to estimate N Dis We noticed that and C are both linear operations, N f With N Dis Linearly related. Therefore, to obtain the distortion N of the feature domain f The distortion feature N can be estimated first Dis Then use a convolution to and C perform linear operations.

[0014] To solve the above problem, we use a learnable bandpass filter to learn some prior knowledge from the DCT domain. The formula is as follows:

[0015]

[0016] Here, θ represents the learnable parameters in the DCT domain.

[0017] Furthermore, the distortion prior learning module mentioned above includes three 3*3 residual connected dilated convolution layers, one 1*1 convolution layer and an implicit DCT domain unit (Implict DCT Unit, IDU). The dilated convolution layer structure of the residual connection is used to extract high-dimensional features and expand the receptive field of the following process; the 1*1 convolution layer is used as a feature fusion layer to fuse the convolution features generated by dense connections and reduce the number of channels; then the distortion prior learning module LBF attempts to extract the distorted image I from the distorted image I. dis The DCT domain learning prior is used, and the implicit DCT domain unit IDU performs the final transformation. Assuming that the size of IDU is p*p, then the corresponding DCT domain spectrum has p 2 frequencies, so the magnitude of θ is p 2 θ is initialized to 1 and is non-negative. The passband in the distortion prior learning module LBF is obtained through training.

[0018] Step (2). Semantic feature extraction, the specific method is as follows:

[0019] Use the pre-trained convolutional neural network as the semantic feature extraction module to extract the distorted image I dis The pre-trained neural network involved is a network built using the image classification model pre-trained on the ImageNet21k dataset using the ResNet50 network. It can extract multi-scale semantic features of the image content. The details are as follows:

[0020]

[0021] Among them, represents the ResNet50 network from block i, where i∈{1,2,3,4}, b represents the batch size, and c i ,m i ,n i Represent the channel size, width, and height, respectively. We use the last layer of each stage to extract multi-scale features from the input image. It is worth noting that the features extracted from different stages have different scales, structures, and dimensions. The low-level semantic information of the image structure represents the basic information of the image, such as the underlying layer, texture, shape, color, etc. This information is directly related to the IQA task; while the deep semantic information is the closest to human understanding of the image expression, with a high degree of abstraction and certain quality assessment capabilities, and can provide an attention mechanism corresponding to local distortions.

[0022] Step (3). Edge information extraction, the specific method is as follows:

[0023] In human vision, image edge information is very important, especially the position information of the edge. The human eye can easily perceive the position change of the edge. Therefore, extracting image edge information helps the model perceive image quality. However, it is difficult for the common convolutional layer to extract edge information well. Inspired by ACNet, the ARM module for extracting edge information in this paper consists of two asymmetric residual blocks. We use asymmetric residual blocks to better extract the edge information from the distorted image I dis Specifically, the 3*3 convolution in each asymmetric residual block is replaced by three parallel 3*3, 3*1, and 1*3 convolutions, followed by a BN layer and PReLU.

[0024] Step (4). Feature aggregation. The specific method is as follows:

[0025] The encoder-decoder generation module uses a UNet-like structure to fuse input distortion prior information, semantic features, and image edge information to generate a rich feature map. Specifically, the encoder uses a ConcatConv to fuse the input distortion prior information, semantic features, and image edge information, and then passes it through a LeakyReLU to form a downsampling module. The decoder uses a ConcatConv and ReLU to form an upsampling module.

[0026] Step (5). Distortion masking and quality regression. The specific method is as follows:

[0027] Image artifacts and spatial masking effects from content are often responsible for image quality degradation. Previous studies have shown that the human eye is insensitive to small pixel differences in complex backgrounds unless the differences between pixels are very large. Therefore, we multiply the generated feature map element-wise with a learnable, content-based adaptive distortion sensitivity map. Distortion is often more perceptible when the background masking effect is weak. We apply global average pooling to the masked distortion map to represent the degree of perceptual distortion. Finally, we input the feature map into the quality regression module to predict the final quality score.

[0028] Step (6). The model training method is as follows:

[0029] The model of the present invention is coded using the PyTorch framework. The model uses the Adam optimizer to train the entire model, with weight decay of 0 and an initial learning rate of 2*10 -5 , and then increased by 10% every 10 epochs. The model weights were initialized using the pre-trained ResNet50 network, and the L1 loss function was used in training, as follows:

[0030]

[0031] The beneficial effects of the present invention are as follows:

[0032] This paper proposes the use of a distortion prior learning method to help the model identify various noises in image information and effectively improve image quality. Existing methods rely heavily on image semantic information and local feature information for feature judgment, and are very sensitive to scene changes in content. Therefore, they cannot effectively explore the essential characteristic laws of the quality degradation of distorted images. This paper proposes a distortion prior learning method. We introduce frequency domain prior information based on image semantic features to assist in predicting quality scores. The method of this paper has been extensively experimented on four public IQA datasets. The results show that the proposed model is significantly superior to the most advanced NR-IQA method in terms of generalization ability and evaluation accuracy. At the same time, we proposed a dataset containing 1,198 real images. Test experiments based on a self-built real image dataset show that our model can effectively estimate massive distortion information in images. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 It is a framework diagram of the method of the present invention;

[0034] Figure 2 It is the distortion visualization graph learned by the multi-scale distortion prior learning module DETAILED DESCRIPTION

[0035] The present invention will be further described below with reference to the accompanying drawings.

[0036] The method of the present invention consists of a multi-scale distortion prior learning module, a multi-scale semantic feature extraction module, a multi-scale encoder-decoder generation module and a quality regression model. Given an RGB distorted image I∈R 3× w×h, input it into the model. Figure 1 As shown, f i represents the features of the downsampled multi-scale input image. First, the multi-scale distortion prior learning module extracts distortion prior information from the input image. A pre-trained convolutional neural network extracts multi-scale semantic features of the image, while also extracting edge information. We then fuse the multi-scale distortion prior information with the semantic information and input it into the multi-scale encoder-decoder generation module to generate feature maps. Finally, the quality regression module aggregates the quality scores of the feature maps to produce the output q.

[0037] like Figure 1 As shown, the method of the present invention is specifically as follows:

[0038] Step (1). Distortion prior learning, the specific method is as follows:

[0039] The present invention uses learnable bandpass filters (LBF) as a distortion prior learning module. Assuming that the distorted image is composed of a clean image plus distortion feature information, it can be expressed as:

[0040] I dis =I clear +N Dis (1.7)

[0041] where N Dis Characterizes the distortion characteristics. Block-DCT is an effective method for dealing with frequency domain related problems, so we Dis Translated into DCT domain for processing. Therefore, the pixel domain distortion N P and the distortion N of the feature domain f Can be expressed as:

[0042]

[0043] in Denotes IDCT operation, C denotes convolution operation. Inspired by implicit DCT, we can directly use a deep CNN block to estimate N Dis We noticed that and C are both linear operations, N f With N Dis Linearly related. Therefore, we need to obtain N f You can first estimate N Dis Then use a convolution to To solve the above problem, we use a learnable bandpass filter to learn some prior knowledge from the DCT domain. The formula is as follows:

[0044]

[0045] θ represents the learnable parameters in the DCT domain.

[0046] The distortion prior learning module mentioned above includes three 3*3 residual connected dilated convolution layers, a 1*1 convolution layer and an implicit DCT domain unit (Implict DCT Unit, IDU). The dilated convolution layer structure of the residual connection is used to extract high-dimensional features and expand the receptive field of the following process; the 1*1 convolution layer is used as a feature fusion layer to fuse the convolution features generated by dense connections and reduce the number of channels; then LBF attempts to learn the prior from the DCT domain of the distorted image, and the implicit DCT domain unit (IDU) performs the final transformation. We assume that the size of the IDU is p*p, and then the corresponding DCT domain spectrum has p^2 frequencies, so the size of θ is p^2. θ is initialized to 1 and is a non-negative value. The Passband in LBF is obtained through training. As Figure 2 As shown in the figure, the LBF we use has the ability to filter the passband layer by layer, and the deep LBF can effectively learn the deep distortion information.

[0047] Step (2). Semantic feature extraction, the specific method is as follows:

[0048] This paper uses a pre-trained convolutional neural network as a semantic feature extraction module. The pre-trained neural network involved is constructed using the image classification model pre-trained on the ImageNet21k dataset using the ResNet50 network. It can extract multi-scale semantic features of the image content. The details are as follows:

[0049]

[0050] Among them, represents ResNet50 from the i-th block, where i∈{1,2,3,4}, b represents the batch size, c i, m i ,n i Represents the channel size, width, and height. We use the last layer of each stage to extract multi-scale features from the input image. Notably, features extracted from different stages have different scales, structures, and dimensions. Low-level semantic information of the image structure represents basic information about the image, such as the underlying layer, texture, shape, and color. This information is directly relevant to IQA tasks; whereas deep semantic information is the closest to human understanding of image representation, possessing a high degree of abstraction and certain quality assessment capabilities, and can provide an attention mechanism corresponding to local distortions.

[0051] Step (3). Edge information extraction, the specific method is as follows:

[0052] In human vision, image edge information is very important, especially the position of edges. The human eye can easily perceive changes in the position of edges. Therefore, extracting image edge information helps the model perceive image quality. However, conventional convolutional layers have difficulty extracting edge information well. Inspired by ACNet, we use asymmetric residual blocks to better extract texture and edge information from images. Specifically, the 3*3 convolution in each asymmetric module is replaced by three parallel 3*#, 3*1, and 1*3 convolutions, followed by a batch normalization layer and pre-reduction unit (PReLU). The ARM module consists of two asymmetric modules.

[0053] Step (4). Feature aggregation. The specific method is as follows:

[0054] The present invention uses an encoder-decoder generation module similar to the UNet structure to fuse the input distortion prior information, semantic information and image edge information to generate rich feature maps.

[0055] Step (5). Distortion masking and quality regression. The specific method is as follows:

[0056] Image artifacts and spatial masking effects from content are often responsible for image quality degradation. Previous studies have shown that the human eye is insensitive to small pixel differences in complex backgrounds unless the differences between pixels are very large. Therefore, we multiply the generated feature map element-wise with a learnable, content-based adaptive distortion sensitivity map. Distortion is often more perceptible when the background masking effect is weak. We apply global average pooling to the masked distortion map to represent the degree of perceptual distortion. Finally, we input the feature map into the quality regression module to predict the final quality score.

[0057] Table 1 below shows the results of comparative experiments on the LIVE, CSIQ, TID2013, and KADID-10K public datasets. Our method surpasses most existing methods in the SROCC and PLCC metrics, two mainstream no-reference quality assessment criteria. On the TID2013 dataset, we see significant improvement. The average SROCC score surpasses the second-place TReS method by 0.031 points, while the PLCC score surpasses both the second-place TReS method and the RNSA method by 0.038 points. In summary, our method demonstrates significant improvement in no-reference image quality assessment.

[0058] Table 1:

[0059] Table 3.SROCC and PLCC results of IQA methods on four IQA datasets(the best results are shown in bold and the second results are shown inunderline).

[0060]

[0061] Step (6). The model training method is as follows:

[0062] The model was coded using the PyTorch framework. The model was trained using the Adam optimizer with a weight decay of 0 and an initial learning rate of 2 x 10, followed by a 10% increase every 10 epochs. The model weights were initialized using a pre-trained ResNet50 network, and the L1 loss function was used during training, as follows:

[0063]

Claims

1. A no-reference image quality assessment method based on distortion prior learning, characterized by The steps include: Step (1). Distortion prior learning; Step (2). Semantic feature extraction; Step (3). Edge information extraction; Step (4). Feature aggregation; Step (5). Distortion masking and quality regression; Step (6). Model training; The distortion prior learning described in step (1) is specifically performed as follows: Using a learnable bandpass filter as a distortion priori learning module, assuming that the distorted image I dis is a clean image I clear Combined with the distortion characteristics, it can be expressed as: I dis =I clear +N Dis (1.1) Among them, N Dis Characterizes the distortion characteristics; N Dis Translated into the DCT domain for processing, the pixel domain distortion N P and the distortion N of the feature domain f Can be expressed as: in Denotes IDCT operation, C denotes convolution operation; inspired by the implicit DCT domain, a deep CNN block is used to estimate N Dis ;in and C are both linear operations, so N f With N Dis Linearly related, so to obtain the distortion N of the feature domain f The distortion feature N can be estimated first Dis Then use a convolution to Perform linear operations with C; Use a learnable bandpass filter to learn some priors from the DCT domain, as follows: Where θ represents the learnable parameters in the DCT domain; The distortion prior learning module includes three 3*3 residual connected dilated convolution layers, a 1*1 convolution layer and an implicit DCT domain unit IDU; the dilated convolution layer with residual connection is used to extract high-dimensional features and expand the receptive field of the process; the 1*1 convolution layer is used as a feature fusion layer to fuse the convolution features generated by dense connections and reduce the number of channels; then the distortion prior learning module attempts to extract the distorted image I dis The DCT domain learning prior is used, and the implicit DCT domain unit IDU performs the final transformation; if the size of IDU is p*p, then the corresponding DCT domain spectrum has p 2 frequencies, so the magnitude of θ is p 2 ; θ is initialized to 1 and is non-negative; The semantic feature extraction described in step (2) is specifically performed as follows: Use the pre-trained convolutional neural network as the semantic feature extraction module to extract the distorted image I dis Semantic feature information; The pre-trained convolutional neural network is constructed using an image classification model pre-trained on the ImageNet21k dataset using the ResNet50 network. It can extract multi-scale semantic features of the content in the image; the details are as follows: Among them, represents the ResNet50 network from block i, where i∈{1,2,3,4}, b represents the batch size, and c i ,m i ,n i Represents channel size, width, and height respectively; the last layer of each stage is used to extract multi-scale semantic features from the input image; The specific method for extracting edge information in step (3) is as follows: Inspired by ACNet, the ARM module for extracting edge information consists of two asymmetric residual blocks; the 3*3 convolution in each asymmetric residual block is replaced by three parallel 3*3, 3*1, and 1*3 convolutions, followed by a batch normalization layer and pre-reduction linear unit (PReLU); The feature aggregation described in step (4) is specifically performed as follows: The Encoder-Decoder generation module with a UNet-like structure is used to fuse the input distortion prior information, semantic feature information, and image edge information to generate a rich feature map. The encoder consists of a ConcatConv to fuse the input distortion prior information, semantic feature information and image edge information, and then passes it through a LeakyReLU to form a downsampling module; the decoder consists of a ConcatConv and ReLU to form an upsampling module; The distortion masking and quality regression described in step (5) are specifically performed as follows: Global average pooling is used in the masked distorted image to represent the degree of perceptual distortion; and the feature map is input into the quality regression module to predict the final quality score; The model training method described in step (6) is as follows: The whole model is trained using the Adam optimizer, with weight decay set to 0 and an initial learning rate of 2*10 -5 , and then increased by 10% every 10 epochs; the pre-trained ResNet50 network was used to initialize the model weights, and the L1 loss function was used in training.

Images