Real scene image quality assessment method based on multi-scale spatial pyramid pooling

Abstract

This invention provides a real-world image quality assessment method based on multi-scale spatial pyramid pooling. This method constructs a blind reference image quality assessment model for real-world image distortion, which includes a multi-scale semantic feature extraction module, an adaptive content understanding module, and a quality scoring module. By inputting the original image to be tested into this image quality assessment model, the multi-scale semantic feature extraction module first extracts multi-scale semantic features from the image and fixes the size of these features using a spatial pyramid structure. Then, the adaptive content understanding module uses the multi-scale semantic features to generate the scoring rules required by the quality scoring module, i.e., weights and biases. Finally, the quality scoring module fuses the multi-scale semantic features and the scoring rules to generate a quality score for the image to be tested. The image quality assessment model of this invention features high evaluation accuracy, good generalization performance, fast training speed, and high robustness.

Description

Technical Field

[0001] This invention belongs to the fields of computer vision and image quality assessment technology, and relates to a method for assessing the quality of real-world scene images based on multi-scale spatial pyramid pooling. Background Technology

[0002] The widespread adoption of smartphones has led to an explosive growth in the generation of digital images. Various types of distortion are introduced at each stage of a visual communication system, such as acquisition, compression, transmission, and display. Therefore, it is necessary to evaluate the quality of actually captured images to optimize system parameters and performance, ensuring the quality of visual content delivered to the end user.

[0003] Image quality assessment is divided into subjective and objective categories. Subjective assessment is usually the most reliable and accurate because the Human Visual System (HVS) is the final receiver of visual signals in most visual communication systems. However, it is time-consuming and costly, and cannot be directly embedded into practical systems as an optimization metric. Objective quality assessment methods are usually designed or trained using subjective assessment data. They can automatically predict visual quality and are suitable for real-time system performance evaluation and optimization. The goal of objective image quality assessment (IQA) is to achieve a higher degree of consistency between computer-predicted image quality scores and human subjective scores. Objective image quality assessment can be divided into three categories according to the amount of information provided by the original reference image: Full-Reference (FR), Reduced-Reference (RR), and No-Reference (NR), where No-Reference is also called Blind IQA (BIQA). Full-Reference assessment methods are relatively accurate and more suitable as an evaluation metric to comprehensively measure the performance of various image processing algorithms. Partial-Reference assessment methods can be embedded into image processing systems to optimize algorithms and parameter settings. However, in reality, it is often impossible or difficult to obtain a distortion-free reference image. Therefore, the no-reference quality assessment method is considered to be the most promising method in terms of practical application.

[0004] No-reference quality assessment can be categorized based on the source of image distortion into no-reference assessment based on artificially synthesized distortion and no-reference assessment based on real-world distortion. For artificially synthesized distortion assessment methods, based on different feature extraction and quality prediction methods, they can be divided into three main categories: natural scene statistics models, handcrafted feature learning models, and deep learning models. No-reference image quality assessment algorithms based on natural scene statistics (NSS) utilize the impact of distortion on the original statistical properties of natural images to construct assessment models, such as those based on the spatial domain (BRISQUE and NIQE), wavelet domain (BIQI and DIIVINE), DCT domain (BLIINDS-I and BLIINDS-II), and shearlet IQM. Models based on handcrafted feature learning aim to extract effective visual features (information entropy and image moment features such as mean, variance, kurtosis, and skewness), and learn a mapping model from visual features to image visual quality through traditional machine learning methods such as support vector regression. Deep learning-based methods primarily construct deep neural networks to learn the visual features of images to build image quality assessment models, or directly learn a function representing the visual quality of distorted images from an end-to-end perspective. However, artificially synthesized distorted images are characterized by a single type of distortion and uniform distortion distribution, while real-world images undergo mixed distortion during acquisition, storage, and processing, with uneven distortion distribution. Therefore, existing quality assessment methods based on artificially synthesized distorted images cannot effectively calculate the visual quality of real-world distorted images. Currently, image quality assessment methods for real-world distortion are still in their early stages.

[0005] Patent application number 202110222548.0 discloses an image quality assessment method, apparatus, and electronic device. The method first inputs the image to be evaluated into feature extraction modules of multiple quality assessment models to obtain feature data corresponding to each model. Then, based on preset feature selection parameters, target feature data is determined from the feature data corresponding to each quality assessment model. These preset feature selection parameters indicate the dimension of the target feature data. Finally, based on the target feature data, the quality assessment result of the image to be evaluated is determined. This method uses feature extraction modules from multiple quality assessment models to extract feature data from the image to be evaluated, ensuring that the feature data includes image features extracted by multiple quality assessment models. The feature selection parameters can identify the higher-quality feature data from the feature data corresponding to each quality assessment model as the target feature data. While this improves the generalization performance of the model to some extent, the time cost during the training phase is high, which is not conducive to real-time quality assessment.

[0006] Patent application number 201911319997.6 discloses an image quality evaluation method and apparatus. The method first obtains multiple heatmaps corresponding to the image to be evaluated. Each heatmap is used to indicate the salience of the feature parts of the object in the image to be evaluated. Different heatmaps are used for different feature parts of the object. Based on the heatmap corresponding to each feature part, a salience value reflecting the salience of each feature part contained in the image to be evaluated is obtained. Then, based on the obtained salience value, an evaluation result for the image to be evaluated is generated. This method focuses on HVS characteristics and incorporates image region salience detection methods. However, the problem is that existing salience detection-based methods are not designed for image quality evaluation, and the models are relatively complex, computationally intensive, and time-consuming.

[0007] Patent application number 201710582739.1 discloses an effective image quality evaluation system. This method includes an image acquisition module, an image processing module, a feature extraction module, and an image quality evaluation module. First, the image acquisition module acquires the image data of the original image to be evaluated. Then, the image processing module processes the acquired original image to obtain a pseudo-reference image. Next, the feature extraction module extracts image features from the grayscale image X and the pseudo-reference image Y. Finally, the image quality evaluation module performs a comprehensive evaluation of the image quality based on the image features extracted by the feature extraction module. This method has a relatively simple calculation process and cleverly transforms no-reference-image quality evaluation into full-reference-image quality evaluation. However, it cannot guarantee that the generated pseudo-reference image has effective reference value, resulting in unsatisfactory scoring accuracy.

[0008] The summary revealed that existing image quality assessment methods applied to real-world scenarios have the following drawbacks:

[0009] Most existing deep neural networks were originally designed for tasks such as image classification or object recognition, rather than IQA (Information Quality Assurance). Therefore, these models only learn global features of images rather than local features, while image distortion in real-world scenarios exists more in local regions. Thus, existing network models cannot be directly used for feature extraction.

[0010] The IQA method, which uses convolutional neural networks to extract semantic features from images, requires cropping, stretching, and padding of the original image before inputting it into the convolutional network due to the limitations of the fully connected layers on image size (e.g., 224×224, 227×227). This indirectly introduces new distortions to some extent, destroying the original overall structure and content of the image. As a result, the image to be tested in the input network model has a distortion deviation from the original image, ultimately reducing the accuracy of the quality score. Summary of the Invention

[0011] To address the aforementioned problems, this invention provides a fast and accurate method for evaluating the quality of images acquired from real-world scenes. The technical solution adopted in this invention is as follows:

[0012] This invention provides a real-world image quality assessment method based on multi-scale spatial pyramid pooling, characterized by the following steps: Step S1, constructing an image quality assessment model, which includes a multi-scale semantic feature extraction module, an adaptive content understanding module, and a quality scoring module; Step S2, inputting the image to be tested into the image quality assessment model, whereby the multi-scale semantic feature extraction module extracts multi-scale semantic features from the image to be tested, and fixes the size of the multi-scale semantic features using a spatial pyramid structure; Step S3, the adaptive content understanding module uses the multi-scale semantic features to generate the scoring rules required by the quality scoring module, namely weights and biases; Step S4, the quality scoring module fuses the multi-scale semantic features and the scoring rules to generate a quality score for the image to be tested.

[0013] The real-scene image quality evaluation method based on multi-scale spatial pyramid pooling provided by this invention may also have the following technical features: the multi-scale semantic feature extraction module uses a fine-tuned ResNet50 as the backbone network, has an attention extraction sub-module, and is fine-tuned to initialize the original ResNet50 with parameters pre-trained on the ImageNet dataset, and removes the last two layers of the original ResNet50 network, namely the average pooling layer and the fully connected layer.

[0014] The real-scene image quality evaluation method based on multi-scale spatial pyramid pooling provided by this invention may also have the following technical features, wherein step S2 includes the following sub-steps: Step S2-1, the image to be tested is input into ResNet50 and after the first layer of convolution, it is input into the attention extraction submodule for attention feature extraction; Step S2-2, the attention features extracted in step S2-1 are input into Stage1, Stage2 and Stage3 respectively to obtain semantic feature streams of different scales, and 1x1 convolution is used to reduce the dimensionality to enhance feature expression and optimize calculation; Step S2-3, the three semantic feature streams obtained in step S2-2 and the high-level semantic feature stream obtained in Stage4 are pooled using the spatial pyramid pooling structure to obtain four fixed-length semantic feature vectors, and these four fixed-length semantic feature vectors are merged to obtain multi-scale semantic features.

[0015] The real-scene image quality evaluation method based on multi-scale spatial pyramid pooling provided by this invention may also have the following technical features, wherein, in steps S2-3, three pooling operations are adopted respectively: the first pooling operation is to pool all regions of the feature map, the second divides the region of the feature map into 2×2=4 parts and pools each part, and the third divides the region of the feature map into 3×3=9 parts and pools each part. The fixed-length semantic feature vector is obtained by concatenating the results obtained by the three pooling methods, and the fixed length is 14×256.

[0016] The real-scene image quality evaluation method based on multi-scale spatial pyramid pooling provided by this invention may also have the following technical features: the adaptive content understanding module includes an adaptive pooling layer, three 1x1 convolutional layers and four parallel weight and bias generation branches. The adaptive pooling layer is used to output a tensor of a given size after adaptive pooling of any input feature map, and the number of channels does not change.

[0017] The real-scene image quality evaluation method based on multi-scale spatial pyramid pooling provided by this invention may also have the following technical features, wherein the quality scoring module uses four fully connected layers as basic target network components.

[0018] The real-scene image quality assessment method based on multi-scale spatial pyramid pooling provided in this invention may also have the following technical features, wherein the image quality assessment model follows the HVS characteristics and customizes different scoring rules according to the content of the image to be tested I to output a quality score:

[0019] s=P(I,θ I )

[0020] In the formula, θ I θ represents the weight parameters of the network. I =U(C(I),λ), where C(I) is the multi-scale semantic feature of the image I to be tested, and λ is the network parameter.

[0021] Invention Function and Effect

[0022] According to the present invention, a real-scene image quality assessment method based on multi-scale spatial pyramid pooling constructs a blind reference image quality assessment model for real-scene distortion, which features high assessment accuracy, good generalization performance, fast training speed, and high robustness. By inputting the original image to be tested into this image quality assessment model, a fixed-length multi-scale semantic feature extraction module first extracts multi-scale semantic features. Then, an adaptive content understanding module generates the weights and biases required by the quality scoring module based on the multi-scale semantic features. Finally, the quality scoring module uses the multi-scale semantic feature vector as input, and through a fully connected layer, fuses the weights and biases generated by the adaptive content understanding module to generate the final quality score of the image to be tested.

[0023] Among them, since the multi-scale semantic feature extraction module uses spatial pyramid pooling to perform fixed-length pooling operation on the input feature map, it can perform quality assessment on the original whole image of any size directly input, avoiding the adverse effects of operations such as cropping on the final score, and has high reliability and efficiency.

[0024] The real-scene image quality assessment method based on multi-scale spatial pyramid pooling of the present invention can meet the requirements of real-time quality assessment in practical applications and has important significance in fields such as image or video acquisition, compression coding, video transmission, security monitoring, and biomedicine. Attached Figure Description

[0025] Figure 1 This is a flowchart of a real-scene image quality evaluation method based on multi-scale spatial pyramid pooling in an embodiment of the present invention;

[0026] Figure 2 This is a schematic diagram of the image quality evaluation model in an embodiment of the present invention;

[0027] Figure 3 This is a schematic diagram of the spatial pyramid pooled CSPP structure in an embodiment of the present invention. Detailed Implementation

[0028] To make the technical means, creative features, objectives and effects of this invention easy to understand, the following describes in detail the real scene image quality evaluation method based on multi-scale spatial pyramid pooling of this invention with reference to embodiments and accompanying drawings.

[0029] <Example>

[0030] Figure 1 This is a flowchart of a real-scene image quality assessment method based on multi-scale spatial pyramid pooling in an embodiment of the present invention. Figure 2 This is a schematic diagram of the image quality evaluation model in an embodiment of the present invention.

[0031] like Figure 1 As shown, the real-world image quality assessment method based on multi-scale spatial pyramid pooling includes the following steps:

[0032] Step S1: Construct an image quality assessment model, which includes a multi-scale semantic feature extraction module, an adaptive content understanding module, and a quality scoring module (such as...). Figure 2 (As shown).

[0033] Step S2: Input the image to be tested into the image quality evaluation model. The multi-scale semantic feature extraction module extracts the multi-scale semantic features of the image to be tested, and fixes the size of the multi-scale semantic features through a spatial pyramid structure.

[0034] In this embodiment, the original ResNet50 is initialized using parameters pre-trained on the ImageNet dataset, and the last two layers of the original ResNet50 network, namely the average pooling layer and the fully connected layer, are removed, resulting in a multi-scale semantic feature extraction module with the fine-tuned ResNet50 as the backbone network. This module also includes an attention extraction submodule, CBAM.

[0035] The specific steps for extracting multi-scale semantic features in step S2 are as follows:

[0036] Step S2-1: Input the image to be tested into ResNet50, pass it through the first convolutional layer, and then input it into the attention extraction submodule CBAM for attention feature extraction;

[0037] Step S2-2: Input the attention features extracted in step S2-1 into Stage1, Stage2 and Stage3 respectively to obtain semantic feature streams of different scales, and use 1x1 convolution to reduce the dimensionality to enhance feature representation and optimize computation.

[0038] Step S2-3: The three semantic feature streams obtained in step S2-2 and the high-level semantic feature stream obtained in Stage 4 are pooled using a spatial pyramid pooling structure to obtain four fixed-length semantic feature vectors. These four fixed-length semantic feature vectors are then merged to obtain the multi-scale semantic feature C(I).

[0039] Figure 3 This is a schematic diagram of the spatial pyramid pooled CSPP structure in an embodiment of the present invention.

[0040] In this embodiment, as Figure 3 As shown, a spatial pyramid is used to pool each region of the feature map using three pooling methods. Specifically:

[0041] The first pooling operation pools all regions of the feature map; the second divides the feature map into 2×2=4 parts and pools each of these 4 parts separately; the third divides the feature map into 3×3=9 parts and pools each of these 9 parts separately.

[0042] Finally, the results obtained from the three pooling methods are concatenated. Thus, regardless of the size of the input feature map, the final semantic feature vector has a fixed length of 14×256.

[0043] Step S3: The adaptive content understanding module uses multi-scale semantic features to generate the scoring rules required by the quality scoring module, namely weights and biases.

[0044] In this embodiment, the adaptive content understanding module includes an adaptive pooling layer, three 1×1 convolutional layers, and four parallel weight and bias generation branches. Its main function is to understand the image content and generate scoring rules to guide the quality scoring module. Since the quality scoring module uses fully connected layers as the basic target network component, this module only needs to generate the weights and biases of the fully connected layers in the quality scoring module. Considering that the number of parameters required to generate bias is much smaller, the weights of the fully connected layers are generated through convolution, followed by a deformation operation, while the bias is generated only through global average pooling and fully connected layers.

[0045] Since the number of output channels in convolutional and fully connected layers is determined based on the dimensions of the corresponding layers in the quality scoring network, and the image sizes input to the network differ during training and testing, adaptive pooling is required before inputting high-level semantic features into the adaptive content understanding module to ensure feature map size matching. The special characteristic of adaptive pooling functions is that for any input feature map size, the size of the output tensor after adaptive pooling is a given size, and its number of channels remains unchanged. The traditional formulas for max or average pooling are:

[0046]

[0047] Among them, S out For pooling output size, S in S is the pooled input size, and padding is the fill size. kernel is the pooling kernel size, and stride is the pooling step size.

[0048] In this embodiment, when using adaptive pooling, since the input and output dimensions are given, the adaptive pooling operation can be completed by calculating and setting the pooling kernel size and stride. Thus, the resulting weights and biases can be understood as constraints on image perceptual quality, used to guide the quality scoring module in predicting quality.

[0049]

[0050] S kernel =S in -(S out -1)·stride

[0051] Traditional deep learning-based quality prediction models receive an input image I and directly map it to a quality score:

[0052] s = P(I,θ)

[0053] Here, θ represents the network's weight parameters, which are fixed after training. This means that the prediction model uses the same feature extraction method to predict the quality of different images. However, in reality, as the image content changes, using the same rules to predict the quality of images with changing content cannot completely cover their original structure.

[0054] Therefore, to solve the above problems, the image quality assessment model of the present invention strictly follows the HVS characteristics, first performing content understanding on the image, and then customizing different evaluation rules according to the content of different images:

[0055] s=P(I,θ I )

[0056] Wherein, network parameter θ I It depends on the content of the image being tested, rather than being fixed for all input images. θ I Generates using weighted and biased branches:

[0057] θ I =U(C(I),λ)

[0058] In the formula, λ represents the network parameters.

[0059] In step S4, the quality scoring module fuses multi-scale semantic features and scoring rules to generate a quality score for the image under test.

[0060] In this embodiment, the quality scoring module contains four fully connected layers. It takes a fixed-length multi-scale semantic feature vector and the weights and biases of each fully connected layer as inputs, and finally outputs the quality score of the image to be tested I: s=P(I,U(C(I),λ)).

[0061] Table 1 compares the performance metrics of this proposed model with other existing models, and Table 2 compares the training time of this model with other existing models.

[0062] In this embodiment, the performance indicators of the image quality evaluation model and existing image quality evaluation methods are experimentally evaluated. The experimental environment is as follows: Windows 10 system, 16G RAM, 11G VRAM, 2080Ti graphics card, and PyTorch encoding framework.

[0063]

[0064] Table 1

[0065] Algorithm Hours / h SROCC PLCC SFA 84 0.685 0.872 HyperIQA 127 0.906 0.917 proposed 26 0.905 0.917

[0066] Table 2

[0067] The evaluation results are shown in Tables 1 and 2 above. The image quality evaluation model of the present invention has better evaluation accuracy and training speed.

[0068] Functions and effects of the embodiments

[0069] According to the real scene image quality assessment method based on multi-scale spatial pyramid pooling provided in this embodiment, the method constructs a blind reference image quality assessment model for real scene distortion. First, the original image to be tested is input into the image quality assessment model. The multi-scale semantic feature extraction module extracts fixed-length multi-scale semantic features. Then, the adaptive content understanding module generates the weights and biases required by the quality scoring module based on the multi-scale semantic features. Finally, the quality scoring module takes the semantic feature vector under multi-scale as input, and through a fully connected layer, fuses the weights and biases generated by the adaptive content understanding module to generate the final quality score of the image to be tested.

[0070] The real-world image quality assessment model based on multi-scale spatial pyramid pooling constructed in this embodiment features high assessment accuracy, good generalization performance, fast training speed, and high robustness. In practical applications, it can meet real-time quality assessment requirements and has significant implications in fields such as image or video acquisition, compression coding, video transmission, security monitoring, and biomedicine.

[0071] In this embodiment, the multi-scale semantic feature extraction module employs spatial pyramid pooling, which can pool each region of the feature map in three ways. Regardless of the size of the input feature map, a fixed-length feature vector is obtained by concatenating the results of the three pooling operations. Therefore, it is suitable for directly inputting original test images of multiple scales, enabling quality assessment of the entire original image and avoiding the adverse effects of operations such as cropping on the final score, thus exhibiting high reliability and efficiency.

[0072] In this embodiment, by replacing and fine-tuning different backbone network models, the system can adapt to different needs of image quality assessment in different scenarios, such as prioritizing speed while ensuring a certain level of accuracy, or prioritizing accuracy while ensuring a certain level of speed, thus exhibiting high applicability.

[0073] The above embodiments are only used to illustrate specific implementations of the present invention, and the present invention is not limited to the scope of the description of the above embodiments.

Claims

1. A method for evaluating the quality of real-world scene images based on multi-scale spatial pyramid pooling, characterized in that, Includes the following steps: Step S1: Construct an image quality assessment model, which includes a multi-scale semantic feature extraction module, an adaptive content understanding module, and a quality scoring module. Step S2: Input the image to be tested into the image quality evaluation model, and the multi-scale semantic feature extraction module extracts the multi-scale semantic features of the image to be tested, and fixes the size of the multi-scale semantic features through a spatial pyramid structure; Step S3: The adaptive content understanding module uses the multi-scale semantic features to generate the scoring rules required by the quality scoring module, namely weights and biases. Step S4: The quality scoring module fuses the multi-scale semantic features and the scoring rules to generate a quality score for the image under test. Step S2 includes the following sub-steps: Step S2-1: The image to be tested is input into ResNet50 and after passing through the first convolutional layer, it is input into the attention extraction submodule for attention feature extraction; Step S2-2: Input the attention features extracted in step S2-1 into Stage1, Stage2 and Stage3 respectively to obtain semantic feature streams of different scales, and use 1x1 convolution to reduce the dimensionality to enhance feature representation and optimize computation. Step S2-3: The three semantic feature streams obtained in step S2-2 and the high-level semantic feature stream obtained in Stage 4 are pooled using a spatial pyramid pooling structure to obtain four fixed-length semantic feature vectors. These four fixed-length semantic feature vectors are then merged to obtain the multi-scale semantic features. In steps S2-3, three pooling operations are performed respectively: The first pooling operation pools all regions of the feature map. The second method divides the feature map region into 2×2=4 parts, and performs pooling on each part. The third method divides the feature map region into 3×3=9 parts, and pools each part. The fixed-length semantic feature vector is obtained by concatenating the results of three pooling methods, and the fixed length is 14×256. The adaptive content understanding module includes an adaptive pooling layer, three 1x1 convolutional layers, and four parallel weight and bias generation branches. The adaptive pooling layer is used to adaptively pool feature maps of any size and output a tensor of a given size without changing the number of channels.

2. The real-scene image quality assessment method based on multi-scale spatial pyramid pooling according to claim 1, characterized in that: in, The multi-scale semantic feature extraction module uses a fine-tuned ResNet50 as the backbone network and includes an attention extraction submodule. The fine-tuning involves initializing the original ResNet50 with parameters pre-trained on the ImageNet dataset and removing the last two layers of the original ResNet50 network: the average pooling layer and the fully connected layer.

3. The real-scene image quality assessment method based on multi-scale spatial pyramid pooling according to claim 1, characterized in that: in, The quality scoring module uses four fully connected layers as the basic target network components.

4. The real-scene image quality assessment method based on multi-scale spatial pyramid pooling according to claim 1, characterized in that: in, The image quality assessment model follows the HVS characteristics and customizes different scoring rules based on the content of the image to be tested (I) to output a quality score: s=P(I,θ I ) In the formula, θ I θ represents the weight parameters of the network. I =U(C(I),λ), where C(I) is the multi-scale semantic feature of the image I to be tested, and λ is the network parameter.

Images