A method, system, electronic device, and storage medium for evaluating image quality without reference

Abstract

This invention relates to a no-reference image quality assessment method, system, electronic device, and storage medium, comprising: S1, acquiring a plurality of sample image blocks, and for each sample image block, acquiring a corresponding pseudo-reference image and image restoration features based on a preset encoder and a preset decoder; S2, acquiring high-level features of the sample image blocks and the pseudo-reference image respectively through a preset feature extractor to obtain the difference features between the two; S3, acquiring multi-scale content features of the sample image blocks through the preset feature extractor to obtain fused multi-scale features; S4, performing global average pooling on the high-level features, fused multi-scale features, and difference features of the sample image blocks, and then performing fully connected neural network mapping to obtain a predicted quality score for the sample image blocks; S5, obtaining the quality assessment result of the image to be evaluated based on the preset quality scores of all sample image blocks. Implementing this invention can improve the accuracy of image quality assessment results.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and more specifically, to a method, system, electronic device, and storage medium for evaluating the quality of a referenceless image. Background Technology

[0002] Referenceless image quality assessment (NR-IQA) does not require reference image information during the evaluation process; instead, it evaluates image quality by analyzing the characteristics of the distorted image itself. NR-IQA is more practical because, in real-world applications, obtaining the original or reference image is often impossible. It also presents the greatest challenge, requiring consideration of the complex characteristics of distorted images and the visual characteristics of the human eye. Therefore, NR-IQA has become a research hotspot in objective image quality assessment methods, possessing significant theoretical and practical implications.

[0003] Due to the lack of reference image information, many traditional NR-IQA methods focus on specific distortion types in distorted images and propose corresponding evaluation algorithms based on these distortion types. For example, filtering-based methods are used to estimate noise in images, and sharpness and blur estimation algorithms are used to evaluate the quality of blurred images. If the image distortion process or the type of distortion in the image can be known, this method can achieve higher accuracy.

[0004] In addition, some NR-IQA methods do not target specific types of distortion, but instead extract general quality features that can describe multiple types of distortion to achieve quality assessment of images with various distortions. The key and challenge of this approach lies in selecting which features to use to measure the degree of distortion. In traditional methods, these features are typically extracted manually using Natural Scene Statistics (NSS), while in deep learning-based methods, they can be automatically learned through deep learning network models.

[0005] Among the methods based on manual feature extraction are BIQI, BRISQUE, NIQE, and IL-NIQE. The BIQI method uses a generalized Gaussian distribution (GGD) to fit the wavelet decomposition coefficients of the image, using the parameters of the GGD model as features, and employs a support vector machine (SVM) to classify the probabilities of five distortion types. Then, SVM is used to calculate the quality score for each of the five distortion types, and finally, the scores of each type are weighted and summed to obtain the overall image quality score. The BRISQUE method also adopts the two-stage framework of BIQI. First, it calculates the multi-scale mean-free contrast normalization (MSCN) coefficients of the distorted image, then uses an asymmetric generalized Gaussian distribution (AGGD) to fit the MSCN coefficients and their correlation coefficients in different directions, using the AGGD coefficients as features for quality score prediction. The NIQE method uses the same features as BRISQUE, then uses a multivariate Gaussian distribution (MVG) to fit the features, performing MVG fitting of NSS features on both the distorted and natural images, defining the distance between the MVG models of the two images as the quality of the distorted image. The IL-NIQE method uses MSCN coefficients, gradient statistics, Log-Gabor filter response, and color statistics as quality-perceived features, and then uses MVG for modeling, taking the distance between MVGs as image quality.

[0006] Deep learning-based methods include IQA-CNN, DIQaM-NR, HyperIQA, DB-CNN, RAN4IQA, Hall-IQA, and VCRNet. Among them, IQA-CNN was the first to use a convolutional neural network (CNN) for no-reference image quality assessment. This model consists of one convolutional layer, a max-min pooling layer, and two fully connected layers. It uses non-overlapping image patches as input to the CNN to augment the data volume, and takes the average quality scores of all image patches as the quality score of the distorted image. The DIQaM-NR method constructs 10 convolutional layers and 2 fully connected layers for feature extraction and score prediction. It also uses image patches as input, but the output is the quality score and weights of each image patch, which are then weighted and summed to obtain the overall image quality score. The HyperIQA method assesses the quality of truly distorted images, using ResNet-50 to extract content features from the distorted images. It uses a hypernetwork to generate adaptive parameters for different images to predict quality scores and employs multi-scale features to enhance the model's ability to perceive detailed features. The DB-CNN method is applicable to both synthetically distorted and real-world distorted images. It uses a VGG-16 pre-trained on ImageNet to extract real-world distortion features and a CNN pre-trained on the Waterloo Exploration dataset and PASCAL VOC 2012 to extract synthetic-world distortion features. Finally, it fuses the two features using bilinear pooling and maps them to a quality score through a fully connected layer. The RAN4IQA method is based on Generative Adversarial Networks (GANs). It uses a repairer to restore the reference image as closely as possible, a discriminator to distinguish between the reference image and the repaired image to ensure improved quality, and an evaluator predicts the final quality score based on the repair gains. The Hall-IQA method is also based on GANs, using a GAN to generate a pseudo-reference image. It then uses the distorted image and its difference map with the pseudo-reference image as input to a regression network for quality score prediction. The VCRNet method uses a visual reconstruction network to generate a pseudo-reference image, utilizing the multi-scale features of the distorted image and the recovery features during the reconstruction process to predict the quality score, but it fails to utilize the difference features between the pseudo-reference image and the distorted image.

[0007] The aforementioned NR-IQA extracts features only from distorted images, making it closest to real-world applications, which also makes NR-IQA more challenging. Some researchers have proposed GAN-based methods to generate pseudo-reference images of distorted images, thus addressing the problem of not being able to obtain reference image information. However, the training process of GANs is quite volatile, making it difficult to achieve good image restoration performance. Furthermore, when faced with severely distorted images, GANs struggle to effectively restore image quality. Summary of the Invention

[0008] The technical problem to be solved by the present invention is to provide a method, system, electronic device and storage medium for evaluating the quality of a referenceless image.

[0009] The technical solution adopted by this invention to solve its technical problem is: constructing a referenceless image quality assessment method, including the following steps:

[0010] S1. Based on the image to be evaluated, obtain several sample image blocks. For each sample image block, obtain the downsampling result of the sample image block based on a preset encoder, and obtain the upsampling result of the sample image block based on a preset decoder, so as to obtain the pseudo reference image corresponding to the sample image block and the image restoration feature corresponding to the pseudo reference image based on the upsampling result.

[0011] S2. Obtain the high-level features of the sample image block and the high-level features of the pseudo-reference image respectively through a preset feature extractor, so as to obtain the difference features between the sample image block and the pseudo-reference image based on the high-level features of the sample image block and the high-level features of the pseudo-reference image;

[0012] S3. Obtain the multi-scale content features of the sample image block through the preset feature extractor, and fuse the multi-scale content features based on the image restoration features to obtain the fused multi-scale features;

[0013] S4. Perform global average pooling on the high-level features, the fused multi-scale features, and the differential features of the sample image patch to obtain a stitched feature map, and perform fully connected neural network mapping on the stitched feature map to obtain the predicted quality score of the sample image patch.

[0014] S5. Obtain the quality evaluation result of the image to be evaluated based on the preset quality scores of all the sample image blocks.

[0015] Preferably, in the referenceless image quality evaluation method of the present invention, in step S1, the encoder includes a first single-layer convolutional layer, a first residual block, and four second residual blocks, as well as four residual connection modules corresponding one-to-one with the four second residual blocks; wherein,

[0016] The first end of the four second residual blocks after being cascaded is connected to the output end of the first single-layer convolutional layer via the first residual block, and the output end of the first single-layer convolutional layer is used to input the sample image block.

[0017] The second end of the four second residual blocks cascaded together is connected to the input end of the preset decoder;

[0018] The residual connection module is used to connect the input and output of the corresponding second residual block respectively.

[0019] Preferably, in the referenceless image quality assessment method of the present invention, in step S1, the decoder includes four first deconvolution layers and one second deconvolution layer; wherein,

[0020] The first end of the cascaded connection of the four first deconvolution layers is connected to the second deconvolution layer, and the pseudo reference image corresponding to the sample image block is output through the second deconvolution layer. The second end of the cascaded connection of the four first deconvolution layers is connected to the preset encoder.

[0021] Preferably, in the no-reference image quality assessment method of the present invention, in step S2, the preset feature extractor includes ResNet-50 pre-trained based on ImageNet;

[0022] In step S2, the high-level features of the sample image patch are the output of Conv5_9 in ResNet-50;

[0023] In step S4, the multi-scale content features of the sample image patch include the outputs of Conv2_10, Conv3_12 and Conv4_18 in ResNet-50.

[0024] Preferably, in the no-reference image quality assessment method of the present invention, in step S3, fusing the multi-scale content features based on the image restoration features to obtain the fused multi-scale features includes the following steps:

[0025] S31. Obtain the convolutional features of the image restoration features and the convolutional features of the multi-scale content features based on multi-scale convolution;

[0026] S32. Max pooling and average pooling are performed on the convolutional features of the image restoration features and the convolutional features of the multi-scale content features, respectively, to obtain the max pooling feature map and average pooling feature map corresponding to the image restoration features and the max pooling feature map and average pooling feature map corresponding to the multi-scale content features.

[0027] S33. The max pooling feature map and average pooling feature map corresponding to the image restoration feature are concatenated and the spatial attention weight of the image restoration feature is obtained through 5×5 convolution; the max pooling feature map and average pooling feature map corresponding to the multi-scale content feature are concatenated and the weights in the space are learned through 5×5 convolution to obtain the spatial attention weight of the multi-scale content feature.

[0028] S34. Based on the spatial attention weights of the image restoration features and the spatial attention weights of the multi-scale content features, the image restoration features and the multi-scale content features are concatenated to obtain a concatenated spatial attention feature map;

[0029] S35. Perform global average pooling on the spliced ​​spatial attention feature map to obtain a one-dimensional vector, and obtain the weight of each channel in the one-dimensional vector according to the fully connected layer to obtain the weight vector corresponding to the one-dimensional vector.

[0030] S36. Based on the spliced ​​spatial attention feature map and the weight vector, the fused multi-scale features are obtained.

[0031] Preferably, the referenceless image quality assessment method of the present invention further includes:

[0032] S11. Obtain a structural similarity map based on the Waterloo synthetic distortion dataset and the initial encoder and initial decoder, and train the initial encoder and initial decoder according to the structural similarity map to obtain the preset encoder and the preset decoder.

[0033] Preferably, in the no-reference image quality assessment method of the present invention, the decoder further includes a third deconvolution layer, the first end of the four first deconvolution layers cascaded together is connected to the third deconvolution layer, and the structural similarity map corresponding to the sample image block is output through the third deconvolution layer.

[0034] The present invention also constructs a referenceless image quality assessment system, comprising:

[0035] The first acquisition unit is used to acquire a number of sample image blocks based on the image to be evaluated, acquire downsampling results of the sample image block based on a preset encoder for each sample image block, and acquire upsampling results of the sample image block based on a preset decoder, so as to acquire the pseudo reference image corresponding to the sample image block and the image restoration features corresponding to the pseudo reference image based on the upsampling results.

[0036] The second acquisition unit is used to acquire the high-level features of the sample image block and the high-level features of the pseudo-reference image respectively through a preset feature extractor, so as to obtain the difference features between the sample image block and the pseudo-reference image based on the high-level features of the sample image block and the high-level features of the pseudo-reference image;

[0037] The third acquisition unit is used to acquire the multi-scale content features of the sample image block through the preset feature extractor, and fuse the multi-scale content features based on the image restoration features to obtain the fused multi-scale features;

[0038] The fourth acquisition unit is used to perform global average pooling on the high-level features, the fused multi-scale features and the difference features of the sample image patch to obtain a stitched feature map, and to perform fully connected neural network mapping on the stitched feature map to obtain the predicted quality score of the sample image patch.

[0039] The result output unit is used to obtain the quality evaluation result of the image to be evaluated based on the preset quality scores of all the sample image blocks.

[0040] The present invention also constructs a computer storage medium storing a computer program thereon, which, when executed by a processor, implements the referenceless image quality assessment method as described in any of the above.

[0041] The present invention also provides an electronic device comprising a memory and a processor;

[0042] The memory is used to store computer programs;

[0043] The processor is used to execute the computer program to implement the no-reference image quality assessment method as described in any of the above.

[0044] The referenceless image quality assessment method, system, electronic device, and storage medium of the present invention have the following beneficial effects: they can make fuller use of the information provided by the pseudo-reference image and improve the accuracy of the image quality assessment results. Attached Figure Description

[0045] The present invention will be further described below with reference to the accompanying drawings and embodiments. In the accompanying drawings:

[0046] Figure 1 This is a flowchart of an embodiment of a referenceless image quality assessment method according to the present invention;

[0047] Figure 2 This is a logic block diagram of an embodiment of a no-reference image quality assessment method of the present invention;

[0048] Figure 3 This is a schematic diagram of the structure of a decoder and encoder in an embodiment of a no-reference image quality assessment method of the present invention;

[0049] Figure 4 This is a schematic diagram of the structure of a residual block in a referenceless image quality assessment method of the present invention;

[0050] Figure 5 This is a flowchart of another embodiment of the referenceless image quality assessment method of the present invention;

[0051] Figure 6This is a flowchart of a local process in a no-reference image quality assessment method of the present invention;

[0052] Figure 7 This is a logic block diagram of an embodiment of a referenceless image quality evaluation device of the present invention;

[0053] Figure 8 This is a comparison diagram showing the effects of an embodiment of the referenceless image quality evaluation method of the present invention. Detailed Implementation

[0054] To provide a clearer understanding of the technical features, objectives, and effects of the present invention, specific embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0055] like Figure 1 and Figure 2 As shown, in a first embodiment of a no-reference image quality assessment method of the present invention, the method includes the following steps: S1, obtaining several sample image blocks based on the image to be evaluated, obtaining a downsampling result of the sample image block based on a preset encoder for each sample image block, and obtaining an upsampling result of the sample image block based on a preset decoder, so as to obtain a pseudo-reference image corresponding to the sample image block and an image restoration feature corresponding to the pseudo-reference image based on the upsampling result. Specifically, the image to be evaluated is sampled to obtain sample image blocks. The sample image blocks can be used as input for encoder downsampling, and the output result of encoder downsampling is output as a pseudo-reference image after upsampling by decoder, and the image restoration feature corresponding to the pseudo-reference image is output. Since no-reference image quality assessment cannot use a reference image, a pseudo-reference image is generated here through quality restoration, which provides prior information about the reference image. The image restoration feature is the feature extracted when generating the pseudo-reference image, used to restore the distorted image.

[0056] S2. High-level features of the sample image patch and the pseudo-reference image are obtained respectively using a preset feature extractor, so as to obtain the difference features between the sample image patch and the pseudo-reference image based on the high-level features of the sample image patch and the pseudo-reference image. Specifically, the true distortion features of the sample image patch can be obtained through the preset feature extractor, that is, based on the high-level features of the sample image patch and the pseudo-reference image obtained respectively by the preset feature extractor, the difference is obtained by subtracting the high-level features of the sample image patch from the high-level features of the pseudo-reference image, and the difference features between the sample image patch and the pseudo-reference image are obtained based on this difference.

[0057] S3. Obtain multi-scale content features of the sample image patch through the preset feature extractor, and fuse the multi-scale content features based on the image restoration features to obtain fused multi-scale features. Specifically, multi-scale content features of the sample image patch can also be obtained through the preset feature extractor, and fully fused based on the image restoration features obtained by the decoder to obtain fused multi-scale features. It can be understood that the number and correspondence between image restoration features and scale features can be set as needed, and fusion is performed based on the correspondence during the fusion process. For example, in one embodiment, three image restoration features and three scale features can be corresponded and fused.

[0058] S4. Global average pooling is performed on the high-level features, the fused multi-scale features, and the difference features of the sample image patch to obtain a stitched feature map. A fully connected neural network is then applied to the stitched feature map to obtain the prediction quality score of the sample image patch. Specifically, the high-level features, the corresponding difference features, and the fused multi-scale features of the obtained sample image patch are stitched together to obtain a corresponding stitched feature map. A fully connected neural network is then applied to this stitched feature map to finally obtain the prediction quality score of the sample image patch.

[0059] In one embodiment, the fully connected neural network can be a three-layer fully connected structure. The process of obtaining the predicted quality score of a sample image patch through this three-layer fully connected structure can be implemented based on the following formula.

[0060] q i =W3ρ(W2ρ(W1F+b1)+b2)+b3

[0061] Where W1, W2, and W3 are the parameters of the three fully connected layers, b1, b2, and b3 are the bias terms of the three fully connected layers, and ρ(·) is the ReLU activation function, as shown in the following equation.

[0062] ρ(x) = max(0,x)

[0063] F represents the concatenated features, i.e., the corresponding concatenated feature map, as shown in the following formula.

[0064]

[0065] Among them, F H For high-level features of distorted images, F M For the fused multi-scale features, F D The difference features between the sample image patch and the pseudo-reference image are represented.

[0066] S5. Obtain the quality evaluation result of the image to be evaluated based on the preset quality scores of all the sample image blocks. Specifically, the final quality score of the image to be evaluated, corresponding to its quality evaluation result, is obtained by performing data calculations based on the predicted quality scores of all the sample image blocks.

[0067] Optionally, the preset encoder includes a first single-layer convolutional layer, a first residual block, and four second residual blocks, as well as four residual connection modules corresponding one-to-one with the four second residual blocks. The first end of the cascaded connection of the four second residual blocks is connected to the output of the first single-layer convolutional layer via the first residual block, and the output of the first single-layer convolutional layer is used to input the sample image block. The second end of the cascaded connection of the four second residual blocks is connected to the input of the preset decoder. The residual connection modules are used to connect the input and output of the corresponding second residual blocks, respectively. Specifically, as shown... Figure 3 As shown, the preset encoder can be configured to consist of six cascaded convolutional modules, E1 to E6. Module E1 is a single-layer convolutional layer composed of 16 3×3 convolutional kernels with a stride of 1. Typically, a convolutional layer consists of multiple kernels, each performing a convolution operation on the input image or the feature map of the previous layer to generate an output feature map. To avoid gradient vanishing when deepening the preset encoder layers and to reuse low-level features, residual blocks can be used for downsampling in the convolutional modules E2 to E6 following module E1. Because residual blocks introduce cross-layer connections—that is, the output of one layer in the convolutional neural network is added to the output of multiple layers—residual connections can enhance the network's representational power, prevent gradient vanishing, increase the network's depth and performance, and make the neural network easier to train and optimize. Figure 4As shown, in one embodiment, the convolutional module E2 uses a first residual block corresponding to residual block 1. The first residual block may contain two 3×3 convolutional layers with a stride of 1. The processing of the first residual block does not change the size of the input feature map, so that the input and output feature maps of the convolutional module E2 can be directly added during the residual connection process. Convolutional modules E3-E6 use a second residual block corresponding to residual block 2. Each second residual block contains one 3×3 convolutional layer with a stride of 2 and one 3×3 convolutional layer with a stride of 1. During the processing of the second residual block, the size of its output feature map is halved compared to the size of its input feature map. That is, the input and output feature map sizes of convolutional modules E3 to E6 are inconsistent. Therefore, when performing residual connections between convolutional modules E3 to E6, it is necessary to perform a 1×1 convolution with a stride of 2 on the input feature map of each convolutional module in E3 to E6, i.e., connect them through residual modules, so that the size of the input and output feature maps of each convolutional module matches. For each residual block, reusing low-level features is equivalent to reusing input features. Residual block 1 is a direct addition, while residual block 2 is to perform a 1×1 convolution on the input features before adding them.

[0068] Optionally, in step S1, the decoder includes four first deconvolutional layers and one second deconvolutional layer; wherein, the first end of the cascaded connection of the four first deconvolutional layers is connected to the second deconvolutional layer, and the pseudo-reference image corresponding to the sample image block is output through the second deconvolutional layer; the second end of the cascaded connection of the four first deconvolutional layers is connected to the preset encoder. Specifically, as shown... Figure 3 As shown, the decoder can consist of five deconvolutional layers D1 to D5, upsampling the high-level features of the sample image patches to generate pseudo-reference images. Furthermore, through multi-level skip connections, the decoder can effectively preserve the detailed information in the sample image patches while avoiding the loss of feature details caused by pooling layers, thus improving the image restoration effect. Deconvolutional layers D1 to D5 are all composed of 3×3 convolutional layers, with a stride of 2 for deconvolutional layers D1 to D4 and a stride of 1 for deconvolutional layer D5. The multi-level skip connections can include connections from encoder E2 to decoder D4, encoder E3 to decoder D3, encoder E4 to decoder D2, and encoder E5 to decoder D1.

[0069] Optionally, the referenceless image quality assessment method of the present invention further includes: S11, obtaining a structural similarity map based on the Waterloo synthetic distortion dataset and the initial encoder and initial decoder, and training the initial encoder and initial decoder according to the structural similarity map to obtain the preset encoder and the preset decoder. Specifically, the encoder and decoder can be pre-trained on the Waterloo synthetic distortion dataset by generating structural similarity maps using an auxiliary task to obtain the preset encoder and preset decoder. The specific operation process can be as follows: randomly cropping 224×224 image patches from the image to be evaluated to expand the training data, setting the learning rate to 0.001, the batch size to 64, training with the Adam optimizer for 100 epochs, and using the loss function as the L1 loss between the generated structural similarity map and the real structural similarity map, as shown in the following formula:

[0070]

[0071] Where N1 is the number of training images in this training process. The structural similarity map between the i-th image patch and its corresponding reference image is calculated using the SSIM method. This is a structural similarity diagram of the i-th image block predicted based on the encoder and decoder described above.

[0072] After training the auxiliary task based on the above process, the obtained encoder and decoder parameters are able to learn the structural information in the image to be evaluated quite well. Based on this, the process of generating pseudo-reference images by the encoder and decoder can be further refined. This can also be trained on the Waterloo synthetic distortion dataset, using randomly cropped 224×224 image patches from the image to be evaluated, with a learning rate of 0.0001, a batch size of 64, and training with the Adam optimizer for 50 epochs. The loss function is the L1 loss between the pseudo-reference image and the reference image, as shown in the following equation:

[0073]

[0074] Where N2 is the number of training images. The reference image corresponding to the i-th image patch. A pseudo-reference image generated for the model.

[0075] Optional, such as Figure 3 As shown, the decoder also includes a third deconvolution layer. The first end of the cascaded connection of the four first deconvolution layers is connected to the third deconvolution layer, which outputs the structural similarity map corresponding to the sample image block. Specifically, the third deconvolution layer corresponds to deconvolution layer D6, which consists of a 3×3 convolution layer with a stride of 1.

[0076] In one specific embodiment, the structures of the encoder and decoder can be referred to Table 1.

[0077] Table 1. Encoder and Decoder Structural Parameters

[0078]

[0079] Optionally, in step S2, the preset feature extractor includes a ResNet-50 pre-trained on ImageNet. In step S2, the high-level features of the sample image patch are the outputs of Conv5_9 in the ResNet-50; in step S4, the multi-scale content features of the sample image patch include the outputs of Conv2_10, Conv3_12, and Conv4_18 in the ResNet-50. Specifically, a ResNet-50 can be pre-trained on ImageNet and used as the preset feature extractor to extract the true distortion features of the sample image patch. The outputs of Conv2_10, Conv3_12, and Conv4_18 in the ResNet-50 are used as the multi-scale content features of the sample image patch, and are fully fused with the image restoration features F1, F2, and F3 output by the preset decoder through a multi-scale feature fusion module to obtain the fused multi-scale features. Additionally, the high-level features of the sample image patch, i.e., the output of Conv5_9 in ResNet-50, can be subtracted from the high-level features of the pseudo-reference image. After dimensionality reduction through 1×1 convolution, the difference features between the sample image patch and the pseudo-reference image can be obtained.

[0080] Optional, such as Figure 5As shown, in step S3, fusing the multi-scale content features based on the image restoration features to obtain the fused multi-scale features includes the following steps: S31, obtaining the convolutional features of the image restoration features and the convolutional features of the multi-scale content features based on multi-scale convolution; S32, performing max pooling and average pooling on the convolutional features of the image restoration features and the convolutional features of the multi-scale content features, respectively, to obtain the max pooling feature map and average pooling feature map corresponding to the image restoration features and the max pooling feature map and average pooling feature map corresponding to the multi-scale content features; S33, concatenating the max pooling feature map and average pooling feature map corresponding to the image restoration features and obtaining the empty space of the image restoration features through a 5×5 convolution. Spatial attention weights; concatenating the max pooling feature map and average pooling feature map corresponding to the multi-scale content features and learning the weights in the space through a 5×5 convolution to obtain the spatial attention weights of the multi-scale content features; S34, concatenating the image restoration features and the multi-scale content features based on the spatial attention weights of the image restoration features and the spatial attention weights of the multi-scale content features to obtain the concatenated spatial attention feature map; S35, performing global average pooling on the concatenated spatial attention feature map to obtain a one-dimensional vector, and obtaining the weight vector corresponding to the one-dimensional vector based on the weight of each channel in the one-dimensional vector according to the fully connected layer; S36, obtaining the fused multi-scale features based on the concatenated spatial attention feature map and the weight vector.

[0081] Specifically, a multi-scale feature fusion module is used to fully integrate image restoration features and multi-scale content features of sample image patches. A schematic diagram of the multi-scale feature fusion module is shown below. Figure 6 As shown. Since the image restoration features and the multi-scale content features of the sample image patches come from different network structures, their feature scales may differ. Therefore, multi-scale convolutions are performed on these two types of features separately to extract features at each scale. Specifically, the multi-scale convolution can use two cascaded 3×3 convolutions to achieve the receptive field size of a 5×5 convolution, while reducing the number of parameters. The specific details can be found in the following formula:

[0082]

[0083] Where MC(·) represents a multi-scale convolution operation, F in The input feature map can be multi-scale content features and image reconstruction features of sample image patches, Conv 3×3 (·) represents a 3×3 convolution with a stride of 1 and a feature map padding size of 1. Figure 4Add one pixel to each feature map to increase its size. If the original feature map size was H×W, the padded size would be (H+2)×(W+2). This padding ensures that the output feature maps have the same size, allowing for stitching. 1×1 (·) represents a 1×1 convolution with a stride of 1. This is a feature splicing operation.

[0084] Next, a spatial attention module is used to extract the spatially salient features of the convolutional features of the image restoration features and the multi-scale content features, respectively. The spatial attention module can retain key information while ignoring unimportant regions. Specifically, the process involves first performing max-pooling and average-pooling operations on the feature maps obtained after each multi-scale convolution, generating two two-dimensional feature maps respectively. This can be understood as obtaining a set of two-dimensional feature maps corresponding to each convolutional feature, for example, obtaining the max-pooling and average-pooling feature maps corresponding to the image restoration features, and the max-pooling and average-pooling feature maps corresponding to the multi-scale content features.

[0085] For each convolutional feature, the two 2D feature maps are concatenated, and weights in the spatial dimension are learned through a 5×5 convolution. These weights are then multiplied by the corresponding input feature maps to obtain the corresponding spatial attention feature maps. This spatial dimension can be understood as the spatial location of the feature maps; typically, a feature map can be viewed as a cuboid of length × width × number of channels. Spatial attention assigns different weights to different locations in the feature maps to focus on key information. For example, the max-pooling and average-pooling feature maps of image restoration are concatenated, and their spatial weights are used to obtain the spatial attention weights for the image restoration features. Similarly, the max-pooling and average-pooling feature maps of multi-scale content features are concatenated, and their spatial weights are used to obtain the spatial attention weights for the multi-scale content features. Finally, the image restoration features and the multi-scale content features are concatenated based on their obtained spatial attention weights to obtain the concatenated spatial attention feature maps.

[0086] The specific process can be referred to in the following formula:

[0087]

[0088] Among them, F R For image restoration features, F C For multi-scale content features of images, Conv 5×5(·) represents a 5×5 convolution, and MaxPool(·) and AvgPool(·) represent the max pooling and average pooling operations, respectively. For feature concatenation, ⊙ represents feature multiplication, and F... MR F is the feature map after multi-scale convolution of the image recovery features. MC F is the feature map resulting from multi-scale convolution of image content features. WR F represents the spatial weights of the image reconstruction features. WC F represents the spatial weights of image content features. S This is the spliced ​​spatial attention feature map.

[0089] To further integrate the spatial attention feature maps, a channel attention module is used to learn the importance of the concatenated spatial attention feature map in different channels, thereby better capturing the relationships between different channels. First, global average pooling is applied to the concatenated spatial attention feature map to obtain a one-dimensional vector. Then, a fully connected layer is used to learn the weights of each channel, generating a weight vector. This weight vector is multiplied by the concatenated spatial attention feature map to obtain a fully integrated output feature. The process is shown in the following equation:

[0090]

[0091] Among them, F S This is the concatenated spatial attention feature map, where FC(·) is a fully connected layer, ⊙ represents the feature multiplication operation, and F... WS Let W be a one-dimensional vector after global average pooling, and F be the weight vector. M The output features are the fused multi-scale features.

[0092] Optionally, the average predicted quality scores of all the sample image patches can be used as the quality evaluation result of the image to be evaluated. Specifically, five 224×224 pixel image patches can be randomly sampled from the image to be evaluated. Then, predictions are made on these five image patches, and the average of the prediction results is taken to obtain the quality score of the test image. The specific calculation formula is shown below:

[0093]

[0094] Where N3 is the number of image patches sampled from the test image, q i Let be the predicted quality score of the i-th sample image patch. The number and size of the sampled image patches can be adjusted as needed.

[0095] Optionally, the above process can be trained based on the following function, which

[0096]

[0097] Where N4 is the number of training image patches in this training process. Let be the true quality score of the i-th image patch, and l1 represent L1 regularization. The Adam optimizer can be used in the above process, with a weight decay rate of 5 × 10⁻⁶. -4 The training rounds are 50, the batch size is 48, and the initial learning rate is 5×10. -5 The learning rate is multiplied by 0.9 for every 10 training iterations. For data augmentation, images can be randomly horizontally flipped during training, and five 224×224 image patches can be randomly sampled from each image to expand the number of training samples. It can be understood that the quality score of each image patch is the same as the quality score of the corresponding image to be evaluated, i.e., the distorted image. It can be understood that after each image flip, steps S1 to S4 are performed based on the flipped image.

[0098] like Figure 7 As shown, a referenceless image quality assessment system of the present invention includes:

[0099] The first acquisition unit 110 is used to acquire a number of sample image blocks based on the image to be evaluated, acquire the downsampling result of the sample image block based on a preset encoder for each sample image block, and acquire the upsampling result of the sample image block based on a preset decoder, so as to acquire the pseudo reference image corresponding to the sample image block and the image restoration feature corresponding to the pseudo reference image based on the upsampling result.

[0100] The second acquisition unit 120 is used to acquire the high-level features of the sample image block and the high-level features of the pseudo-reference image respectively through a preset feature extractor, so as to acquire the difference features between the sample image block and the pseudo-reference image based on the high-level features of the sample image block and the high-level features of the pseudo-reference image.

[0101] The third acquisition unit 130 is used to acquire the multi-scale content features of the sample image block through the preset feature extractor, and fuse the multi-scale content features based on the image restoration features to obtain the fused multi-scale features.

[0102] The fourth acquisition unit 140 is used to perform global average pooling on the high-level features, the fused multi-scale features and the difference features of the sample image patch to obtain a spliced ​​feature map, and to perform fully connected neural network mapping on the spliced ​​feature map to obtain the predicted quality score of the sample image patch.

[0103] The result output unit 150 is used to obtain the quality evaluation result of the image to be evaluated based on the preset quality scores of all the sample image blocks.

[0104] Specifically, the specific operational process of the various units in the no-reference image quality assessment system can be referred to the no-reference image quality assessment method described above, and will not be repeated here.

[0105] Furthermore, an electronic device according to the present invention includes a memory and a processor; the memory is used to store a computer program; the processor is used to execute the computer program to implement any of the referenceless image quality assessment methods described above. Specifically, according to embodiments of the present invention, the processes described in the flowcharts above can be implemented as computer software programs. For example, embodiments of the present invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, when the computer program is downloaded, installed, and executed by an electronic device, it performs the functions defined above in the methods of the embodiments of the present invention. The electronic device in the present invention can be a terminal such as a laptop, desktop computer, tablet computer, or smartphone, or it can be a server.

[0106] Furthermore, this invention provides a computer storage medium storing a computer program, which, when executed by a processor, implements any of the above-mentioned methods for evaluating image quality without reference. Specifically, it should be noted that the computer-readable medium described above can be a computer-readable signal medium, a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, system, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. In this invention, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. The transmitted data signal can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, system, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0107] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device. A specific embodiment

[0109] To avoid image content duplication between the training and test sets, the synthetic distortion dataset was divided into training and test sets at 80% and 20% based on the original images. For the real distortion dataset, the training and test sets were directly divided at the 80% and 20% ratio. To reduce the impact of random errors, each dataset was randomly divided 10 times according to the above rules, and the median of the 10 experimental results was taken as the final result. This process allows for a comprehensive evaluation of the performance of this embodiment on different datasets and avoids the problem of image content duplication between the training and test sets. The final result has higher reliability.

[0110] In this embodiment, the Spearman Rank Correlation Coefficient (SROCC) and the Pearson Correlation Coefficient (PLCC) are mainly used for performance comparison. SROCC evaluates the monotonicity of the prediction, while PLCC evaluates the linear correlation between the predicted score and the subjective quality score. Both range from -1 to 1, with larger absolute values ​​indicating better model performance. Their calculation methods are as follows:

[0111]

[0112]

[0113] Where M is the number of images, x i and y i Let $\mathbf{i}$ be the subjective evaluation score and the predicted score for the $i$-th image, respectively. and These are the mean subjective evaluation score and the mean predicted score, respectively, d i Let be the difference between the subjective evaluation score and the predicted score ranking of the i-th image.

[0114] Furthermore, on the large synthetic distortion dataset Waterloo, due to the lack of subjective quality scores, performance comparisons were conducted using the original / distorted image discrimination ability test (D-Test), ranking consistency test (L-Test), and image pair preference consistency test (P-Test). The calculation methods are shown in the following formulas:

[0115]

[0116] Where S p and S d S′ represents the reference image set and the distorted image set in the dataset, respectively. p and S′ d This is the set of reference and distorted images for model prediction.

[0117]

[0118] Where N is the number of reference images, C is the number of distortion types, and l ij and q ij These represent the actual distortion order and the predicted distortion order, respectively.

[0119]

[0120] Where Q is the total number of image pairs, Q c The number of image pairs that the model predicts in the correct order.

[0121] Performance comparison on a single dataset

[0122] Experiments on a single dataset used four synthetic distortion datasets: LIVE, CSIQ, TID2013, and KADID, as well as two real distortion datasets: LIVEC and KonIQ. The results are shown in Tables 2 and 3. Mainstream methods compared to the procedures in the examples include three traditional methods: PSNR, SSIM, and BRISQUE; seven deep learning-based methods: CNN, BIECON, MEON, DIQaM-NR, HyperIQA, DB-CNN, and TS-CNN; GAN-based methods RAN4IQA and Hall-IQA; and VCRNet, which is based on visual compensation and inpainting.

[0123] As can be seen from the experimental results in Tables 2 and 3, in this embodiment, the SROCC achieves a top-two ranking on all six datasets, and achieves the best performance on the KADID dataset. On other datasets, it maintains a small gap with the best method. Compared to traditional methods, this embodiment outperforms these methods on all six datasets. Compared to deep learning-based and GAN-based methods, this embodiment outperforms most methods on synthetic distortion datasets, and on real distortion datasets, it is only slightly inferior to Hyper-IQA for real distortion.

[0124] Table 2 SROCC results for a single dataset

[0125]

[0126] Table 3 PLCC results on a single dataset

[0127]

[0128] Based on the combined results of SROCC and PLCC, this embodiment demonstrates good performance on all six datasets and has certain advantages compared to methods based on deep learning and GAN.

[0129] Performance comparison on a single distortion type

[0130] To test the performance of the method presented in this chapter on specific distortion types, experiments were conducted on three datasets: LIVE, CSIQ, and TID2013. For each dataset, the model was trained on the entire dataset and then tested on a specific distortion type. The experimental results are shown in Tables 4, 5, and 6.

[0131] As shown in the table, this embodiment achieved top-two performance in four distortion types on both LIVE and CSIQ, outperforming other methods. On the TID2013 dataset, this embodiment achieved top-two performance in 13 out of 24 distortion types, second only to HyperIQA's 16. Furthermore, on more complex distortion types such as NPN, BW, MS, CC, and CCS, most methods struggle to achieve satisfactory results and rarely reach a SROCC of 0.500, while this embodiment still achieves excellent results, reaching SROCCs of 0.596, 0.728, 0.542, 0.786, and 0.719 for these distortion types, respectively.

[0132] Table 4. SROCC results for a single distortion type on the LIVE dataset.

[0133]

[0134]

[0135] Table 5 shows the SROCC results for a single distortion type on the CSIQ dataset.

[0136]

[0137] Table 6 shows the SROCC results for a single distortion type on the TID2013 dataset.

[0138]

[0139] Based on the experimental results of individual distortion types on the three datasets, this embodiment demonstrates good performance for specific distortion types. Even when faced with relatively complex distortion types, this embodiment can still maintain a high prediction accuracy.

[0140] Performance comparison based on cross-datasets

[0141] Cross-dataset experiments are a common method for testing a model's generalization ability. This paper conducts cross-dataset experiments on four datasets: LIVE, CSIQ, TID2013, and LIVEC. Specifically, the model is trained on one of the datasets and then tested on the others. For example, the model is trained on the LIVE dataset and then tested on the CSIQ, TID2013, and LIVEC datasets, and so on. This method can more comprehensively test the model's generalization performance. The SROCC results of the experiments are shown in Table 7.

[0142] Table 7. SROCC Results of the Cross-Dataset Experiment

[0143]

[0144] Overall, the method presented in this chapter achieved a top-two SROCC ranking 11 times out of 12 tests, outperforming the second-place DB-CNN and VCRNet in 7 times. When testing on synthetic distortion datasets, most methods achieved good performance due to the similar distortion types across datasets. However, TID2013 contains more complex distortion types, causing a significant performance drop for many methods on this dataset, yet this embodiment still achieved the highest SROCC. When cross-testing between synthetic and real distortion datasets, many methods experienced severe performance degradation, making it difficult to achieve good results. This embodiment, however, achieved SROCC rankings of 0.600, 0.467, 0.375, and 0.419, consistently ranking in the top two and surpassing other deep learning-based and GAN-based methods.

[0145] To further test the generalization performance of this embodiment on large-scale datasets, D-Test, P-Test, and L-Test were performed on the Waterloo dataset. To prevent cross-training between the training and test sets, the multi-task image restoration subnetwork was pre-trained on TID2013, and then the entire model was trained on LIVE. The experimental results are shown in Table 8. As can be seen from the table, this embodiment achieved the top two positions in all three metrics, demonstrating its good generalization performance.

[0146] Table 8 Results of D-Test, L-Test and P-Test

[0147]

[0148] To verify the impact of each module in the proposed method on the model performance, ablation experiments were conducted on LIVE, CSIQ, and LIVEC. The experimental results are shown in Table 9.

[0149] First, a score prediction subnetwork using only distorted images as input was used as the baseline model, resulting in the worst performance. Then, a single-task image quality restoration subnetwork was added, directly concatenating image restoration features with multi-scale content features of the distorted image. This allowed the model to utilize some information from the pseudo-reference image, improving performance. Next, a multi-task image quality restoration subnetwork was used, still using only image restoration features. This time, due to the improved quality of the pseudo-reference image, the model's performance further improved. Then, image difference features were introduced, allowing the model to utilize pseudo-reference image information more fully, further improving the model's SROCC. Finally, a multi-scale feature fusion module was introduced, enabling the full fusion of multi-scale image content features and restoration features, achieving the best performance.

[0150] Based on the experimental results above, the multi-task image restoration sub-network, image restoration features, image difference features, and multi-scale feature fusion module proposed in this embodiment can all effectively improve the performance of the model.

[0151] Table 9 Ablation Experiment Results

[0152]

[0153] To test the performance of the image restoration sub-network, image restoration performance experiments were conducted on the LIVE, CSIQ, and TID2013 datasets. The average PSNR and average SSIM between the distorted image and the pseudo-reference image and the reference image were used to evaluate the image restoration effect. This chapter tests the performance of the single-task and multi-task image restoration networks, and the experimental results are shown in Tables 10 and 11.

[0154] Table 10. Average PSNR of pseudo-reference image and distorted image

[0155]

[0156] Table 11 Average SSIM of pseudo-reference image and distorted image

[0157]

[0158] Figure 8 This image compares single-task and multi-task pseudo-reference images. The figure clearly shows that the multi-task pseudo-reference image demonstrates a significant advantage in visual perception quality. Compared to the single-task pseudo-reference image and the distorted image (corresponding to the image being evaluated), its image quality is closer to the reference image. This indicates that within the framework of multi-task learning, the generation of pseudo-reference images can better recover the visual quality and perceptual details of the image, thereby improving the quality and usability of the pseudo-reference image.

[0159] The English terms used in this invention can be found in the appendix. It is understood that the above embodiments only illustrate preferred embodiments of the invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that those skilled in the art can freely combine the above technical features and make various modifications and improvements without departing from the concept of the invention; all of these fall within the scope of protection of the invention. Therefore, any equivalent transformations and modifications made within the scope of the claims of this invention should be covered by the claims of this invention.

[0160] Appendix: English-Chinese Glossary of Proper Nouns

[0161]

Claims

1. A method for evaluating the quality of a referenceless image, characterized in that, Includes the following steps: S1. Based on the image to be evaluated, obtain several sample image blocks. For each sample image block, obtain the downsampling result of the sample image block based on a preset encoder, and obtain the upsampling result of the sample image block based on a preset decoder, so as to obtain the pseudo reference image corresponding to the sample image block and the image restoration feature corresponding to the pseudo reference image based on the upsampling result. S2. Obtain the high-level features of the sample image block and the high-level features of the pseudo-reference image respectively through a preset feature extractor, so as to obtain the difference features between the sample image block and the pseudo-reference image based on the high-level features of the sample image block and the high-level features of the pseudo-reference image; S3. Obtain the multi-scale content features of the sample image block through the preset feature extractor, and fuse the multi-scale content features based on the image restoration features to obtain the fused multi-scale features; S4. Perform global average pooling on the high-level features, the fused multi-scale features, and the differential features of the sample image patch to obtain a stitched feature map, and perform fully connected neural network mapping on the stitched feature map to obtain the predicted quality score of the sample image patch. S5. Obtain the quality evaluation result of the image to be evaluated based on the preset quality scores of all the sample image blocks.

2. The no-reference image quality assessment method according to claim 1, characterized in that, In step S1, the encoder includes a first single-layer convolutional layer, a first residual block, and four second residual blocks, as well as four residual connection modules corresponding one-to-one with the four second residual blocks; wherein, The first end of the four second residual blocks after being cascaded is connected to the output end of the first single-layer convolutional layer via the first residual block, and the output end of the first single-layer convolutional layer is used to input the sample image block. The second end of the four second residual blocks cascaded together is connected to the input end of the preset decoder; The residual connection module is used to connect the input and output of the corresponding second residual block respectively.

3. The no-reference image quality assessment method according to claim 1, characterized in that, In step S1, the decoder includes four first deconvolutional layers and one second deconvolutional layer; wherein, The first end of the cascaded connection of the four first deconvolution layers is connected to the second deconvolution layer, and the pseudo reference image corresponding to the sample image block is output through the second deconvolution layer. The second end of the cascaded connection of the four first deconvolution layers is connected to the preset encoder.

4. The no-reference image quality assessment method according to claim 1, characterized in that, In step S2, the preset feature extractor includes ResNet-50 generated based on ImageNet pre-training; In step S2, the high-level features of the sample image patch are the output of Conv5_9 in ResNet-50; In step S4, the multi-scale content features of the sample image patch include the outputs of Conv2_10, Conv3_12 and Conv4_18 in ResNet-50.

5. The no-reference image quality assessment method according to claim 1, characterized in that, In step S3, fusing the multi-scale content features based on the image restoration features to obtain the fused multi-scale features includes the following steps: S31. Obtain the convolutional features of the image restoration features and the convolutional features of the multi-scale content features based on multi-scale convolution; S32. Max pooling and average pooling are performed on the convolutional features of the image restoration features and the convolutional features of the multi-scale content features, respectively, to obtain the max pooling feature map and average pooling feature map corresponding to the image restoration features and the max pooling feature map and average pooling feature map corresponding to the multi-scale content features. S33. The max pooling feature map and average pooling feature map corresponding to the image restoration feature are concatenated and the spatial attention weight of the image restoration feature is obtained through 5×5 convolution; the max pooling feature map and average pooling feature map corresponding to the multi-scale content feature are concatenated and the weights in the space are learned through 5×5 convolution to obtain the spatial attention weight of the multi-scale content feature. S34. Based on the spatial attention weights of the image restoration features and the spatial attention weights of the multi-scale content features, the image restoration features and the multi-scale content features are concatenated to obtain a concatenated spatial attention feature map; S35. Perform global average pooling on the spliced ​​spatial attention feature map to obtain a one-dimensional vector, and obtain the weight of each channel in the one-dimensional vector according to the fully connected layer to obtain the weight vector corresponding to the one-dimensional vector. S36. Based on the spliced ​​spatial attention feature map and the weight vector, the fused multi-scale features are obtained.

6. The no-reference image quality assessment method according to claim 3, characterized in that, The method further includes: S11. Obtain a structural similarity map based on the Waterloo synthetic distortion dataset and the initial encoder and initial decoder, and train the initial encoder and initial decoder according to the structural similarity map to obtain the preset encoder and the preset decoder.

7. The method for evaluating the quality of a referenceless image according to claim 6, characterized in that, The decoder also includes a third deconvolution layer. The first end of the four first deconvolution layers is connected to the third deconvolution layer, and the third deconvolution layer outputs the structural similarity map corresponding to the sample image block.

8. A referenceless image quality assessment system, characterized in that, include: The first acquisition unit is used to acquire a number of sample image blocks based on the image to be evaluated, acquire downsampling results of the sample image block based on a preset encoder for each sample image block, and acquire upsampling results of the sample image block based on a preset decoder, so as to acquire the pseudo reference image corresponding to the sample image block and the image restoration features corresponding to the pseudo reference image based on the upsampling results. The second acquisition unit is used to acquire the high-level features of the sample image block and the high-level features of the pseudo-reference image respectively through a preset feature extractor, so as to acquire the difference features between the sample image block and the pseudo-reference image based on the high-level features of the sample image block and the high-level features of the pseudo-reference image; The third acquisition unit is used to acquire the multi-scale content features of the sample image block through the preset feature extractor, and fuse the multi-scale content features based on the image restoration features to obtain the fused multi-scale features; The fourth acquisition unit is used to perform global average pooling on the high-level features, the fused multi-scale features and the difference features of the sample image patch to obtain a stitched feature map, and to perform fully connected neural network mapping on the stitched feature map to obtain the predicted quality score of the sample image patch. The result output unit is used to obtain the quality evaluation result of the image to be evaluated based on the preset quality scores of all the sample image blocks.

9. A computer storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the no-reference image quality assessment method as described in any one of claims 1-7.

10. An electronic device, characterized in that, Including memory and processor; The memory is used to store computer programs; The processor is used to execute the computer program to implement the no-reference image quality assessment method as described in any one of claims 1-7.

Images