A method, system, electronic device, and storage medium for evaluating image quality without reference

Abstract

This invention relates to a referenceless image quality assessment method, system, electronic device, and storage medium, comprising: S1, acquiring several sample image blocks and their corresponding contrast sensitivity weighted gradient images based on the image to be evaluated; S2, acquiring multi-scale content features of the sample image blocks and multi-scale location features of the contrast sensitivity weighted gradient images based on a preset improved ResNet-50, and then concatenating and fusing them to obtain multi-scale features of the sample image blocks; S3, acquiring high-level content features of the sample image blocks and high-level location features of the contrast sensitivity weighted gradient images, concatenating the multi-scale features, high-level content features, and high-level location features, and then performing fully connected neural network mapping to obtain a predicted quality score for the sample image blocks; S4, obtaining the quality assessment result of the image to be evaluated based on the predicted quality scores of all sample image blocks. Implementing this invention enables accurate prediction of image quality scores.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and more specifically, to a method, system, electronic device, and storage medium for evaluating the quality of a referenceless image. Background Technology

[0002] Referenceless image quality assessment (NR-IQA) methods do not require reference image information during the evaluation process; instead, they evaluate image quality by analyzing the characteristics of the distorted image itself. NR-IQA methods are more practical because, in real-world applications, obtaining the original or reference image is often impossible. However, it also presents the greatest challenge, requiring consideration of the complex characteristics of distorted images and the visual characteristics of the human eye. Therefore, NR-IQA has become a research hotspot in objective image quality assessment methods, possessing significant theoretical and practical implications.

[0003] Due to the lack of reference image information, many traditional NR-IQA methods focus on specific distortion types in distorted images and propose corresponding evaluation algorithms based on these distortion types. For example, filtering-based methods are used to estimate noise in images, and sharpness and blur estimation algorithms are used to evaluate the quality of blurred images. If the image distortion process or the type of distortion in the image can be known, this method can achieve higher accuracy.

[0004] In addition, some NR-IQA methods do not target specific types of distortion, but instead extract general quality features that can describe multiple distortion types to achieve quality assessment of images with various distortions. The key and challenge of this approach lies in selecting which features to measure the degree of distortion. Traditional methods typically extract these features manually using Natural Scene Statistics (NSS), while deep learning-based methods can automatically learn them through deep learning network models. The following sections introduce NR-IQA methods based on manually extracted features and those based on deep learning.

[0005] Currently, methods based on manual feature extraction include BIQI, BRISQUE, NIQE, and IL-NIQE, while deep learning-based methods include IQA-CNN, DIQaM-NR, HyperIQA, DB-CNN, and TS-CNN.

[0006] The BIQI method uses a generalized Gaussian distribution (GGD) to fit the wavelet decomposition coefficients of the image, uses the parameters of the GGD model as features, and uses a support vector machine to classify the probabilities of five distortion types. Then, support vector regression is used to calculate the quality score for each of the five distortion types, and finally, the scores of each type are weighted and summed to obtain the overall image quality score.

[0007] The BRISQUE method also adopts the two-stage framework of BIQI. First, it calculates the multi-scale mean-demeaned contrast normalization (MSCN) coefficients of the distorted image. Then, it uses an asymmetric generalized Gaussian distribution (AGGD) to fit the MSCN coefficients and their correlation coefficients in different directions. Finally, it uses the AGGD coefficients as features to predict the quality score.

[0008] The NIQE method uses the same features as BRISQUE, and then fits the features using a multivariate Gaussian distribution (MVG). MVG fitting of NSS features is performed on both the distorted and natural images, and the distance between the MVG models of the two images is defined as the quality of the distorted image.

[0009] The IL-NIQE method uses MSCN coefficients, gradient statistics, Log-Gabor filter response, and color statistics as quality-perceived features, and then uses MVG for modeling, taking the distance between MVGs as image quality.

[0010] IQA-CNN is the first model to use a convolutional neural network (CNN) for no-reference image quality assessment. The model consists of one convolutional layer, a max-min pooling layer, and two fully connected layers. It takes non-overlapping image patches as input to the CNN to augment the data volume and takes the average quality score of all image patches as the quality score of the distorted image.

[0011] The DIQaM-NR method performs feature extraction and score prediction by constructing 10 convolutional layers and 2 fully connected layers. It also takes image patches as input, but outputs the quality scores and weights of the image patches. Finally, it performs weighted summation to obtain the quality score of the entire image.

[0012] The HyperIQA method assesses the quality of real-world distorted images. It uses ResNet-50 to extract content features from distorted images, generates adaptive parameters for different images through a hypernetwork to predict quality scores, and employs multi-scale features to enhance the model's ability to perceive detailed features.

[0013] The DB-CNN method is applicable to both synthetic and real-world distorted images. It uses a VGG-16 pre-trained on ImageNet to extract real-world distortion features and a CNN pre-trained on the Waterloo Exploration dataset and PASCAL VOC 2012 to extract synthetic-world distortion features. Finally, it fuses the two features using bilinear pooling and maps them to a quality score through a fully connected layer.

[0014] The TS-CNN method also has a two-branch structure. The two branches of this method take the distorted image and the gradient image as inputs, respectively, and classify the input image patches into different distortion categories and regions through a region-based fully convolutional neural network.

[0015] However, these methods ignore the influence of the human visual system's HVS characteristics on the perception of image distortion, and therefore may not achieve the same effect as human eye perception. Summary of the Invention

[0016] The technical problem to be solved by the present invention is to provide a method, system, electronic device and storage medium for evaluating the quality of a referenceless image.

[0017] The technical solution adopted by this invention to solve its technical problem is: constructing a referenceless image quality assessment method, including the following steps:

[0018] S1. Obtain several sample image blocks based on the image to be evaluated, and obtain the contrast sensitivity weighted gradient image corresponding to each sample image block;

[0019] S2. Obtain the multi-scale content features of the sample image block and the multi-scale position features of the contrast sensitivity weighted gradient image based on the preset improved ResNet-50, and then stitch and fuse the multi-scale content features and the multi-scale position features to obtain the multi-scale features of the sample image block.

[0020] S3. Obtain the high-level content features of the sample image block and the high-level location features of the contrast sensitivity weighted gradient image respectively. Perform feature concatenation on the multi-scale features, the high-level content features and the high-level location features to obtain a concatenated feature map. Perform fully connected neural network mapping on the concatenated feature map to obtain the predicted quality score of the sample image block.

[0021] S4. Obtain the quality evaluation result of the image to be evaluated based on the predicted quality scores of all the sample image blocks.

[0022] Preferably, in the referenceless image quality assessment method of the present invention,

[0023] In step S2, the process of obtaining multi-scale content features of the sample image patch and multi-scale location features of the contrast sensitivity weighted gradient image based on a preset improved ResNet-50, and then concatenating and fusing the multi-scale content features and the multi-scale location features to obtain the multi-scale features of the sample image patch, includes the following steps:

[0024] S21. Obtain the first feature map of the sample image block based on the preset structure layer in the preset improved ResNet-50, and the second feature map of the contrast sensitivity weighted gradient image based on the preset structure layer;

[0025] S22. The first feature map and the second feature map are spliced ​​together to obtain the primary spliced ​​feature map corresponding to the preset structure layer;

[0026] S23. The importance of different channel features is redistributed in the primary spliced ​​feature map and global average pooling is performed to obtain a one-dimensional vector corresponding to the primary spliced ​​feature map;

[0027] S24. Generate the weight vector of each channel in the one-dimensional vector based on the first preset fully connected layer and the second preset fully connected layer;

[0028] S25. The primary spliced ​​feature map is fused based on the weight vector to obtain a fused feature map;

[0029] S26. Perform global average pooling on the fused feature map to obtain the scale features of the sample image block, so as to obtain the multi-scale features of the sample image block based on all the scale features.

[0030] Preferably, in the referenceless image quality assessment method of the present invention,

[0031] In step S22, the preset structure layer includes Conv2_10 structure layer, Conv3_12 structure layer and Conv4_18 structure layer.

[0032] Preferably, the referenceless image quality assessment method of the present invention further includes:

[0033] S261. Reduce the number of channels in the fused feature map by 1×1 convolution, and perform global average pooling on the fused feature map with reduced channel number to obtain the multi-scale features of the sample image block.

[0034] Preferably, in the referenceless image quality assessment method of the present invention,

[0035] In step S3, the fully connected neural network comprises a three-layer fully connected neural network:

[0036] The step of performing a fully connected neural network mapping on the stitched feature map to obtain the predicted quality score of the sample image patch includes: obtaining the predicted quality score of the sample image patch based on the following formula:

[0037]

[0038] in, The predicted quality score for sample image patch i. , and These are the parameters of the three fully connected neural network layers. , and These are the bias terms of the three fully connected neural network layers, and F is the concatenated feature. This is the ReLU activation function.

[0039] Preferably, in the referenceless image quality assessment method of the present invention,

[0040] In step S4, obtaining the quality assessment result of the image to be evaluated based on the predicted quality scores of all the sample image patches includes:

[0041] The average of the predicted quality scores of all the sample image blocks is taken as the quality evaluation result of the image to be evaluated.

[0042] Preferably, the referenceless image quality assessment method of the present invention further includes a method based on using... The loss function is trained on the process of obtaining the predicted quality score of the sample image patch:

[0043]

[0044] in For the first The target quality score for each image patch To obtain the first Quality score of each image patch represent Regularization.

[0045] The present invention also constructs a referenceless image quality assessment system, comprising:

[0046] The first acquisition unit is used to acquire a number of sample image blocks based on the image to be evaluated, and to acquire the contrast sensitivity weighted gradient image corresponding to each of the sample image blocks;

[0047] The second acquisition unit acquires the multi-scale content features of the sample image block and the multi-scale position features of the contrast sensitivity weighted gradient image based on a preset improved ResNet-50, and splices and fuses the multi-scale content features and the multi-scale position features to obtain the multi-scale features of the sample image block.

[0048] The third acquisition unit is used to acquire the high-level content features of the sample image block and the high-level location features of the contrast sensitivity weighted gradient image, respectively, to perform feature concatenation on the multi-scale features, the high-level content features and the high-level location features to obtain a concatenated feature map, and to perform fully connected neural network mapping on the concatenated feature map to obtain the predicted quality score of the sample image block.

[0049] The result output unit is used to obtain the quality evaluation result of the image to be evaluated based on the predicted quality scores of all the sample image blocks.

[0050] The present invention also constructs a computer storage medium storing a computer program thereon, which, when executed by a processor, implements the referenceless image quality assessment method as described in any of the above.

[0051] The present invention also provides an electronic device comprising a memory and a processor;

[0052] The memory is used to store computer programs;

[0053] The processor is used to execute the computer program to implement the no-reference image quality assessment method as described in any of the above.

[0054] The referenceless image quality assessment method, system, electronic device, and storage medium of the present invention have the following beneficial effects: by extracting the content features and positional features of the image through a dual-branch convolutional neural network, accurate prediction of the image quality score can be achieved. Attached Figure Description

[0055] The present invention will be further described below with reference to the accompanying drawings and embodiments. In the accompanying drawings:

[0056] Figure 1 This is a flowchart of an embodiment of a referenceless image quality assessment method according to the present invention;

[0057] Figure 2 This is a logic block diagram of an embodiment of a no-reference image quality assessment method of the present invention;

[0058] Figure 3 This is a schematic diagram of an embodiment of the mask structure in a no-reference image quality evaluation method of the present invention;

[0059] Figure 4 This is a schematic diagram comparing gradient images and contrast sensitivity-weighted gradient images in a no-reference image quality assessment method of the present invention;

[0060] Figure 5 This is a flowchart of another embodiment of the referenceless image quality assessment method of the present invention;

[0061] Figure 6 This is a logic block diagram of another embodiment of the no-reference image quality assessment method of the present invention;

[0062] Figure 7 This is a logic block diagram of an embodiment of a referenceless image quality assessment system of the present invention;

[0063] Figure 8 This is a schematic diagram showing the results of an embodiment of the no-reference image quality assessment method of the present invention;

[0064] Figure 9 This is a schematic diagram showing the results of another embodiment of the no-reference image quality evaluation method of the present invention. Detailed Implementation

[0065] To provide a clearer understanding of the technical features, objectives, and effects of the present invention, specific embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0066] like Figure 1 and Figure 2 As shown, in the first embodiment of the referenceless image quality assessment method of the present invention, it includes: S1, acquiring a plurality of sample image blocks based on the image to be evaluated, and acquiring a contrast sensitivity-weighted gradient image corresponding to each of the sample image blocks. Specifically, the image to be evaluated can be sampled to obtain a plurality of sample image blocks of a required size. It can be understood that the quality score of each sampled image block inherits the quality score of the original image, i.e., the image to be evaluated. At the same time, a contrast sensitivity-weighted gradient image is acquired for each sample image block. Gradient images can provide rich structural and contour information, and the human visual system is also very sensitive to this information. Therefore, using gradient images can extract object shape information in distorted images based on positional branches, which is more in line with the back-side path's perception of global shape. In one embodiment, the Scharr operator is used as the gradient operator, and its mask structure is as follows. Figure 3 As shown, a gradient image is obtained. By weighting the obtained gradient image with contrast sensitivity, the frequency information that the human eye is sensitive to is increased, making the image evaluation process highly consistent with HVS perception. The specific process can be as follows:

[0067] First, the contrast sensitivity of each pixel in the obtained sample image block is calculated according to the following contrast sensitivity function to obtain the contrast sensitivity map.

[0068]

[0069] in For the contrast sensitivity of pixels, The spatial frequency of pixels in the sample image block. Obtain it using the following formula:

[0070]

[0071]

[0072] in and These are the spatial frequencies in the horizontal and vertical directions, respectively. For the pixels in the sample image patch ( The grayscale value of ).

[0073] The contrast sensitivity map and the gradient image are then weighted and fused using the following formula to obtain the contrast sensitivity weighted gradient map:

[0074]

[0075] in A contrast sensitivity-weighted gradient image for sample image patches. This is a contrast sensitivity map of a sample image patch. The gradient map of the sample image patch. , and The constant is taken as follows: In one embodiment, the constant is taken as follows: = =0.5, =0.

[0076] Gradient images and contrast-sensitivity-weighted gradient images have the following effects: Figure 4 As shown in the figure, the contrast sensitivity-weighted gradient map better highlights regions of interest to the human eye compared to the gradient image, such as patterns around the eyes, the beak of a bird, and the edges of its body. This is because the contrast sensitivity-weighted gradient map assigns different weights to different regions of the image, thus better capturing the image details that the human eye focuses on. Furthermore, the contrast sensitivity-weighted gradient map can also capture distortion structure information in the image, such as the blocky distortion structure caused by JPEG compression and image noise caused by Gaussian noise, which can significantly impact image quality.

[0077] S2. Multi-scale content features of the sample image patch and multi-scale location features of the contrast sensitivity weighted gradient image are obtained based on a pre-defined improved ResNet-50, respectively. The multi-scale content features and multi-scale location features are then concatenated and fused to obtain the multi-scale features of the sample image patch. Specifically, the sample image patch is used as the content branch, and the contrast sensitivity weighted gradient image obtained based on the sample image patch is used as the location branch. Feature extraction is performed separately using a ResNet-50 pre-trained on ImageNet as the backbone network for feature extraction to obtain multi-scale content features and multi-scale location features. The obtained scale feature maps are concatenated and fused through the multi-scale feature module, allowing the fusion process to simultaneously focus on both global and local detail features of the image. Pre-training refers to training the model on large-scale data to extract general feature representations, and then fine-tuning it for specific tasks to make it suitable for downstream tasks. The ResNet-50 used here can be pre-trained on the large image classification dataset ImageNet. Using this improved ResNet-50 for feature extraction can reduce training time and avoid the expensive computational cost of training a model from scratch.

[0078] S3. Obtain the high-level content features of the sample image patch and the high-level location features of the contrast sensitivity weighted gradient image, respectively. Concatenate the multi-scale features, the high-level content features, and the high-level location features to obtain a concatenated feature map. Apply a fully connected neural network mapping to the concatenated feature map to obtain the predicted quality score of the sample image patch. Specifically, the high-level features of the content branch and the location branch (i.e., high-level content features and high-level location features) can be obtained separately, concatenated with the obtained multi-scale features, and the concatenated feature map is then applied to a fully connected neural network to obtain the preset quality score of the sample image patch.

[0079] S4. Obtain the quality assessment result of the image to be evaluated based on the predicted quality scores of all the sample image blocks. Specifically, the final quality score of the image to be evaluated, corresponding to its quality assessment result, is obtained by performing data calculations based on the predicted quality scores of all the sample image blocks.

[0080] Optional, such as Figure 5 and Figure 6As shown, in step S2, the step of obtaining multi-scale content features of the sample image patch and multi-scale location features of the contrast sensitivity weighted gradient image based on a preset improved ResNet-50, and then concatenating and fusing the multi-scale content features and the multi-scale location features to obtain the multi-scale features of the sample image patch, includes the following steps: S21, obtaining the first feature map of the sample image patch based on a preset structural layer in the preset improved ResNet-50, and the second feature map of the contrast sensitivity weighted gradient image based on the preset structural layer; S22, concatenating... The first feature map and the second feature map are used to obtain the primary stitched feature map corresponding to the preset structural layer; S23, the importance of different channel features in the primary stitched feature map is redistributed and global average pooling is performed to obtain the one-dimensional vector corresponding to the primary stitched feature map; S24, the weight vector of each channel in the one-dimensional vector is generated based on the first preset fully connected layer and the second preset fully connected layer; S25, the primary stitched feature map is fused based on the weight vector to obtain the fused feature map; S26, the fused feature map is global average pooled to obtain the multi-scale features of the sample image patch. Specifically, in the content branch, the first feature map corresponding to its sample image patch can be obtained based on the preset structural layer in the preset improved ResNet-50, wherein the multi-scale content features can be composed of multiple first feature maps. In the location branch, the second feature map corresponding to the contrast sensitivity weighted gradient image can be obtained based on the preset structural layer in the preset improved ResNet-50, wherein the multi-scale location features can be composed of multiple second feature maps. The first and second feature maps are concatenated to obtain the primary concatenated feature map corresponding to the preset structure layer. Then, a channel attention mechanism is used to redistribute the importance of different channel features in the concatenated feature map. The concatenated feature map is then reduced to a one-dimensional vector by global average pooling. Finally, weight vectors for each channel are generated through fully connected layers FC1 (corresponding to the first preset fully connected layer) and FC2 (corresponding to the second preset fully connected layer). This assigns a corresponding weight to each channel. Finally, this weight vector is multiplied by the concatenated feature map to further fuse the feature maps of the content path and the location path (corresponding to...). Figure 2 (MS-Module process). Finally, convolutions can be performed sequentially to reduce the number of channels in the fused feature map, and global average pooling can be used to obtain multi-scale feature vectors.

[0081] Optionally, based on the above, step S261 can be included: reducing the number of channels in the fused feature map through 1×1 convolution, and performing global average pooling on the fused feature map with reduced channel count to obtain the multi-scale features of the sample image patch. This process is shown in the following equation:

[0082]

[0083] in and These are the parameters for the two fully connected layers, respectively. and These are the Sigmoid and ReLU activation functions, respectively. For global average pooling, This is a 1×1 convolution operation. For multi-scale content features, For multi-scale location features, Features after splicing Features after passing through the channel attention mechanism This results in multi-scale features after fusion. Specifically, a 1×1 convolution can be used to reduce the number of channels in the fused feature map by half to reduce computational cost, followed by global average pooling to obtain the multi-scale feature vector.

[0084] Optionally, the preset structural layers include Conv2_10 structural layers, Conv3_12 structural layers, and Conv4_18 structural layers. Specifically, the preset improved ResNet-50 is obtained by removing the last average pooling layer and fully connected layer from the original ResNet-50, and its specific structural layers can be referred to Table 1 below:

[0085] Table 1 ResNet-50 Network Structure

[0086]

[0087] Among them, the Conv2_10, Conv3_12 and Conv4_18 structural layers can be used as preset structural layers to obtain the corresponding scale features respectively, and the multi-scale features of the sample image patch can be obtained according to all the scale features.

[0088] Optionally, in step S3, the fully connected neural network includes a three-layer fully connected neural network:

[0089] The step of performing a fully connected neural network mapping on the stitched feature map to obtain the predicted quality score of the sample image patch includes: obtaining the predicted quality score of the sample image patch based on the following formula:

[0090]

[0091] in, The predicted quality score for sample image patch i. , and These are the parameters of the three fully connected neural network layers. , and These are the bias terms of the three fully connected neural network layers, and F is the concatenated feature. It is the ReLU activation function. Specifically,

[0092] The activation function can satisfy...

[0093]

[0094] The following function can be used to perform feature concatenation of the multi-scale features, the high-level content features, and the high-level location features.

[0095]

[0096] in For advanced content features, For multi-scale features, For high-level location features, This is for splicing operations.

[0097] Based on the Conv2_10 structural layer, the Conv3_12 structural layer and the Conv4_18 structural layer, three scale features are obtained, which correspond to multi-scale features.

[0098] Optionally, in step S4, obtaining the quality evaluation result of the image to be evaluated based on the predicted quality scores of all the sample image blocks includes: obtaining the average of the predicted quality scores of all the sample image blocks as the quality evaluation result of the image to be evaluated. Specifically, five 224×224 pixel image blocks can be randomly sampled from the image to be evaluated. Then, predictions are made on these five image blocks, and the average of the prediction results is taken to obtain the quality score of the test image. The specific calculation formula is shown below:

[0099]

[0100] in The number of image patches sampled from the test image. The model predicts the first The quality score of each image patch. The number and size of the sampled image patches can be adjusted as needed.

[0101] Optionally, the method of the present invention further includes using The loss function is trained on the process of obtaining the predicted quality score of the sample image patch:

[0102]

[0103] in For the first The target quality score for each image patch To obtain the first Quality score of each image patch represent Regularization; the training process employs the Adam optimizer. Specifically, the image to be evaluated can be randomly horizontally flipped to obtain a flipped image. Based on this flipped image, step S1 and subsequent actions described above are performed to obtain the predicted quality score of the sample image patch. And based on... The loss function calculates the difference between the predicted and true values. A smaller loss function value indicates higher model prediction accuracy. Therefore, optimizing the loss function optimizes this process. The specific process can be understood as follows: Each training iteration resamples the images. The training process is as follows: Divide the dataset into training and testing sets, using images from the training set to train the model; extract a certain number of images (e.g., 48) from the training set, then randomly flip each extracted image horizontally and sample 5 image patches, treating these 240 image patches as independent training samples; obtain the contrast sensitivity-weighted gradient images of the 240 image patches, and input them into the model to obtain the predicted quality score for each image patch; calculate the difference between the predicted and true values ​​using the loss function, and update the parameters of each function in the above process using backpropagation. Repeat the above steps until all images in the training set have participated in the training, thus completing one round of training. Set the total number of training rounds to 50.

[0104] In one embodiment, the Adam optimizer can be used to update the parameters in the above process. The parameters optimized by the Adam optimizer may include the parameters in two preset improved ResNet-50, and the parameters in the fusion process of the three scale features (such as...). Figure 2 The MS-Module process shown above, along with the parameters in the last three fully connected neural networks (corresponding to three fully connected layers), minimizes the loss function. The training process of a neural network can be divided into forward propagation and backward propagation. Forward propagation refers to the process by which the model infers its output based on the input. Backward propagation first calculates the difference between the model's output and the true value based on the loss function, then calculates the gradient of this difference with respect to each parameter in the output layer, and propagates this gradient layer by layer to the input layer, thereby calculating the gradient of each parameter in the entire model. Finally, gradient descent is used to update the parameters. In one embodiment, the Adam optimizer can be configured to satisfy the following during optimization: a weight decay rate of 5 × 10⁻⁴, 50 training epochs, a batch size of 48, an initial learning rate of 5 × 10⁻⁵, and a learning rate halved every 10 training epochs.

[0105] Additionally, such as Figure 7 As shown, a referenceless image quality assessment system of the present invention includes:

[0106] The first acquisition unit 110 is used to acquire a number of sample image blocks based on the image to be evaluated, and to acquire the contrast sensitivity weighted gradient image corresponding to each of the sample image blocks.

[0107] The second acquisition unit 120 acquires the multi-scale content features of the sample image block and the multi-scale position features of the contrast sensitivity weighted gradient image based on a preset improved ResNet-50, and splices and fuses the multi-scale content features and the multi-scale position features to obtain the multi-scale features of the sample image block.

[0108] The third acquisition unit 130 is used to acquire the high-level content features of the sample image block and the high-level location features of the contrast sensitivity weighted gradient image, respectively, to perform feature concatenation on the multi-scale features, the high-level content features and the high-level location features to obtain a concatenated feature map, and to perform fully connected neural network mapping on the concatenated feature map to obtain the predicted quality score of the sample image block.

[0109] The result output unit 140 is used to obtain the quality evaluation result of the image to be evaluated based on the predicted quality scores of all the sample image blocks.

[0110] Specifically, the specific operational process of the various units in the no-reference image quality assessment system can be referred to the no-reference image quality assessment method described above, and will not be repeated here.

[0111] Furthermore, an electronic device according to the present invention includes a memory and a processor; the memory is used to store a computer program; the processor is used to execute the computer program to implement any of the referenceless image quality assessment methods described above. Specifically, according to embodiments of the present invention, the processes described in the flowcharts above can be implemented as computer software programs. For example, embodiments of the present invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, when the computer program is downloaded, installed, and executed by an electronic device, it performs the functions defined above in the methods of the embodiments of the present invention. The electronic device in the present invention can be a terminal such as a laptop, desktop computer, tablet computer, or smartphone, or it can be a server.

[0112] Furthermore, this invention provides a computer storage medium storing a computer program, which, when executed by a processor, implements any of the above-mentioned methods for evaluating image quality without reference. Specifically, it should be noted that the computer-readable medium described above can be a computer-readable signal medium, a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, system, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. In this invention, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. The transmitted data signal can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, system, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0113] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0114] A specific embodiment will be used for illustration.

[0115] To avoid image content duplication between the training and test sets, the synthetic distortion dataset was divided into training and test sets at 80% and 20% based on the original images. For the real distortion dataset, the training and test sets were directly divided at the 80% and 20% ratio. To reduce the impact of random error, each dataset was randomly divided 10 times according to the above rules, and the median of the 10 experimental results was taken as the final result. This experimental design can comprehensively evaluate the performance of the above process on different datasets and avoid the problem of image content duplication between the training and test sets. The final result has higher reliability.

[0116] In this embodiment, the Spearman Rank Correlation Coefficient (SROCC) and the Pearson Correlation Coefficient (PLCC) are mainly used for performance comparison. SROCC evaluates the monotonicity of the prediction, while PLCC evaluates the linear correlation between the predicted score and the subjective quality score. Both range from -1 to 1, with larger absolute values ​​indicating better model performance. Their calculation methods are as follows:

[0117]

[0118]

[0119] in For the number of images, and The first The subjective evaluation score and predicted score of the image. and These are the mean of subjective evaluation scores and the mean of predicted scores, respectively. For the first The difference between the subjective evaluation score and the predicted score ranking of the image.

[0120] Furthermore, on the large synthetic distortion dataset Waterloo, due to the lack of subjective quality scores, performance comparisons were conducted using the original / distorted image discrimination ability test (D-Test), ranking consistency test (L-Test), and image pair preference consistency test (P-Test). The calculation methods are shown in the following formulas:

[0121]

[0122] in and These are the reference images and the distorted images in the dataset, respectively. and This is the set of reference and distorted images for model prediction.

[0123]

[0124] in For reference image quantity, The number of distortion types. and These represent the actual distortion order and the predicted distortion order, respectively.

[0125]

[0126] in For the total number of image pairs, The number of image pairs that the model predicts in the correct order.

[0127] To comprehensively test the performance of this embodiment, experiments were conducted on synthetic and real distortion datasets, and comparisons were made with mainstream methods. The synthetic distortion datasets used in the experiments included LIVE, CSIQ, TID2013, KADID-10k, and Waterloo, while the real distortion datasets included LIVEC and KonIQ-10k. We compared the results with three traditional methods—PSNR, SSIM, and BRISQUE—and seven deep learning-based methods—CNN, BIECON, MEON, DIQaM-NR, HyperIQA, DB-CNN, and TS-CNN. DB-CNN and TS-CNN, similar to this embodiment, both employ a dual-branch structure. The experimental results of this embodiment on a single dataset are shown in Tables 2 and 3, with the top two performance values ​​(SROCC and PLCC) highlighted in bold.

[0128] Based on the combined results of SROCC and PLCC, this embodiment performs well on six commonly used datasets. Compared to DB-CNN and TS-CNN, which also have a dual-branch structure, this embodiment maintains a leading position on most datasets, especially compared to TS-CNN, where it maintains a significant performance gap on real-world datasets.

[0129] Table 2 SROCC results on a single dataset

[0130]

[0131] Table 3 PLCC Results on a Single Dataset

[0132]

[0133] In addition, to test the performance of this embodiment on specific distortion types, experiments were conducted on three datasets: LIVE, CSIQ, and TID2013. For each dataset, the model was trained on the entire dataset and then tested on a specific distortion type. The experimental results are shown in Tables 4, 5, and 6.

[0134] Table 4. SROCC results for a single distortion type on the LIVE dataset.

[0135]

[0136] Table 5. SROCC results for a single distortion type on the CSIQ dataset.

[0137]

[0138] Table 6. SROCC results for a single distortion type on the TID2013 dataset.

[0139]

[0140] As can be seen from the table, this embodiment achieved the best performance in all four distortion types on LIVE and CSIQ, outperforming other methods. On the TID2013 dataset, this embodiment achieved top-two performance in 17 out of 24 distortion types, second only to HyperIQA's 19 types. Figure 8 Line graphs show the SROCC performance of CNN, MEON, DIQA, HyperIQA, DBCNN, and this embodiment across various distortion types in TID2013. The graphs clearly demonstrate that this embodiment exhibits stable and excellent performance across all distortion types in TID2013.

[0141] Based on the experimental results of individual distortion types on the three datasets, this embodiment also demonstrates excellent performance in terms of specific distortion types.

[0142] In addition, cross-dataset experiments were conducted on four datasets: LIVE, CSIQ, TID2013, and LIVEC. Specifically, the model was trained on one of the datasets and then tested on the others. For example, the model was trained on the LIVE dataset and then tested on the CSIQ, TID2013, and LIVEC datasets, and so on. This process allows for a more comprehensive test of the model's generalization performance. The SROCC results of the experiments are shown in Table 7.

[0143] The experimental results show that this embodiment achieved the best performance in 8 trials, outperforming DB-CNN in 4 trials. The SROCC line graph for the cross-dataset experiments is shown below. Figure 9 As shown in the figure, this embodiment maintains stable and excellent performance in the cross-dataset test, proving that this embodiment has good generalization performance.

[0144] Table 7. SROCC Results of the Cross-Dataset Experiment

[0145]

[0146] To further test the generalization performance of this embodiment on a large-scale dataset, it was trained on the entire LIVE dataset and tested on the Waterloo dataset. D-Test, P-Test, and L-Test metrics were calculated, and the experimental results are shown in Table 8. As can be seen from the table, this embodiment achieved top-two performance in both the D-Test and L-Test metrics, and was also very competitive in the L-Test metric, further demonstrating the excellent generalization performance of this embodiment.

[0147] Table 1 Results of D-Test, L-Test and P-Test

[0148]

[0149] To verify the effectiveness of each module in this embodiment, ablation experiments were conducted on the LIVE, CSIQ, and LIVEC datasets. Using the content branch with only distorted images as input as the baseline model, a location branch with gradient images as input, a location branch with contrast-sensitive weighted gradient images as input, and a dual-branch multi-scale feature module were progressively added. The experimental results are shown in Table 9.

[0150] As shown in Table 9, adding a location branch and using the gradient image of the distorted image as input allows the model to extract rich structural information, effectively improving performance. The performance improvement is even more significant when using a contrast-sensitive weighted gradient image as input to the location branch, indicating that using a contrast-sensitive weighted gradient image as input explicitly guides the model to focus more on HVS-sensitive parts, thus making the model highly consistent with human visual perception. Finally, introducing a bi-branch multi-scale feature module into the model improves performance regardless of whether gradient images or contrast-sensitive weighted gradient images are used as input to the location branch, with the most significant improvement on the real-distortion dataset LIVEC. This is because the image distortion level is unevenly distributed in LIVEC, and the bi-branch multi-scale feature module allows the model to simultaneously focus on global features and local details, thereby improving model performance.

[0151] Table 9. SROCC values ​​from ablation experiments

[0152]

[0153] The English terms used in this invention can be found in the appendix. It is understood that the above embodiments only illustrate preferred embodiments of the invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that those skilled in the art can freely combine the above technical features and make various modifications and improvements without departing from the concept of the invention; all of these fall within the scope of protection of the invention. Therefore, any equivalent transformations and modifications made within the scope of the claims of this invention should be covered by the claims of this invention.

[0154] Appendix: English-Chinese Glossary of Proper Nouns

[0155]

Claims

1. A method for evaluating the quality of a referenceless image, characterized in that, Includes the following steps: S1. Based on the image to be evaluated, obtain several sample image blocks and obtain the gradient image corresponding to each sample image block. Calculate the corresponding contrast sensitivity for each pixel in the obtained sample image block according to the contrast sensitivity function to obtain a contrast sensitivity map. Then, perform weighted fusion of the contrast sensitivity map and the gradient image to obtain a contrast sensitivity weighted gradient image. S2. Take the sample image block as the content branch and the contrast sensitivity weighted gradient image obtained based on the sample image block as the location branch. Obtain the multi-scale content features of the sample image block and the multi-scale location features of the contrast sensitivity weighted gradient image based on the preset improved ResNet-50, and then stitch and fuse the multi-scale content features and the multi-scale location features to obtain the multi-scale features of the sample image block. S3. Obtain the high-level content features of the content branch and the high-level position features of the position branch respectively. Perform feature concatenation on the multi-scale features, the high-level content features and the high-level position features to obtain a concatenated feature map. Perform fully connected neural network mapping on the concatenated feature map to obtain the predicted quality score of the sample image patch. S4. Obtain the quality evaluation result of the image to be evaluated based on the predicted quality scores of all the sample image blocks.

2. The no-reference image quality assessment method according to claim 1, characterized in that, In step S2, the process of obtaining multi-scale content features of the sample image patch and multi-scale location features of the contrast sensitivity weighted gradient image based on a preset improved ResNet-50, and then concatenating and fusing the multi-scale content features and the multi-scale location features to obtain the multi-scale features of the sample image patch, includes the following steps: S21. Obtain the first feature map of the sample image block based on the preset structure layer in the preset improved ResNet-50, and the second feature map of the contrast sensitivity weighted gradient image based on the preset structure layer; S22. The first feature map and the second feature map are spliced ​​together to obtain the primary spliced ​​feature map corresponding to the preset structure layer; S23. The importance of different channel features is redistributed in the primary spliced ​​feature map and global average pooling is performed to obtain a one-dimensional vector corresponding to the primary spliced ​​feature map; S24. Generate the weight vector of each channel in the one-dimensional vector based on the first preset fully connected layer and the second preset fully connected layer; S25. The primary spliced ​​feature map is fused based on the weight vector to obtain a fused feature map; S26. Perform global average pooling on the fused feature map to obtain the scale features of the sample image block, so as to obtain the multi-scale features of the sample image block based on all the scale features.

3. The no-reference image quality assessment method according to claim 2, characterized in that, In step S22, the preset structure layer includes Conv2_10 structure layer, Conv3_12 structure layer and Conv4_18 structure layer.

4. The no-reference image quality assessment method according to claim 2, characterized in that, The method further includes: S261. Reduce the number of channels in the fused feature map by 1×1 convolution, and perform global average pooling on the fused feature map with reduced channel number to obtain the multi-scale features of the sample image block.

5. The method for evaluating the quality of a referenceless image according to claim 3, characterized in that, In step S3, the fully connected neural network; Includes a three-layer fully connected neural network; The step of performing a fully connected neural network mapping on the stitched feature map to obtain the predicted quality score of the sample image patch includes: obtaining the predicted quality score of the sample image patch based on the following formula: in, The predicted quality score for sample image patch i. , and These are the parameters of the three fully connected neural network layers. , and These are the bias terms of the three fully connected neural network layers, and F is the concatenated feature. This is the ReLU activation function.

6. The no-reference image quality assessment method according to claim 1, characterized in that, In step S4, obtaining the quality assessment result of the image to be evaluated based on the predicted quality scores of all the sample image patches includes: The average of the predicted quality scores of all the sample image blocks is taken as the quality evaluation result of the image to be evaluated.

7. The method for evaluating the quality of a referenceless image according to claim 1, characterized in that, The method also includes using The loss function is trained on the process of obtaining the predicted quality score of the sample image patch: in For the first The target quality score for each image patch To obtain the first Quality score of each image patch represent Regularization.

8. A referenceless image quality assessment system, characterized in that, include: The first acquisition unit is used to acquire a number of sample image blocks based on the image to be evaluated, and acquire the contrast sensitivity weighted gradient image corresponding to each sample image block. The contrast sensitivity is calculated for each pixel in the obtained sample image block according to the contrast sensitivity function to obtain a contrast sensitivity map. The contrast sensitivity map and the gradient image are then weighted and fused to obtain a contrast sensitivity weighted gradient image. The second acquisition unit is used to take the sample image block as the content branch and the contrast sensitivity weighted gradient image obtained based on the sample image block as the location branch. It acquires the multi-scale content features of the sample image block and the multi-scale location features of the contrast sensitivity weighted gradient image based on a preset improved ResNet-50, and splices and fuses the multi-scale content features and the multi-scale location features to obtain the multi-scale features of the sample image block. The third acquisition unit is used to acquire the high-level content features of the content branch and the high-level position features of the position branch respectively, perform feature concatenation on the multi-scale features, the high-level content features and the high-level position features to obtain a concatenated feature map, and perform fully connected neural network mapping on the concatenated feature map to obtain the predicted quality score of the sample image patch. The result output unit is used to obtain the quality evaluation result of the image to be evaluated based on the predicted quality scores of all the sample image blocks.

9. A computer storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the no-reference image quality assessment method as described in any one of claims 1-7.

10. An electronic device, characterized in that, Including memory and processor; The memory is used to store computer programs; The processor is used to execute the computer program to implement the no-reference image quality assessment method as described in any one of claims 1-7.

Images