Image quality evaluation method and system based on distortion information
Through the image quality evaluation method based on distorted information, the local quality score and attention weight of the image are extracted using convolutional neural network and normalized layer, the problem of information redundancy in the deep learning method is solved, and the accuracy of image quality evaluation is achieved Improved sex and efficiency, in line with human eye perception.
Patent Information
- Application Number
- CN202311577129.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-11-22
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2043-11-22
AI Technical Summary
The existing deep learning-based image quality evaluation method has information redundancy, which leads to low feature extraction efficiency and difficult to reflect the true distortion of the image, affecting the accuracy and efficiency of image quality evaluation.
A method of image quality evaluation based on distortion information is proposed. The local mass fraction and attention weight are extracted respectively through the distortion extraction module and the feature extraction module, and the overall mass fraction of the image is obtained by weighting summing, using convolutional neural network and normalization. To reduce information redundancy and improve model generalization performance.
Accurate evaluation of image quality is achieved, training difficulty is reduced, feature utilization efficiency is improved, evaluation results are more in line with the subjective perception of the human eye, information redundancy is reduced, and the objectivity and accuracy of the model are improved.
Smart Images

Figure CN117611548B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of image quality processing, and in particular to an image quality evaluation method and system based on distortion information. Background Art
[0002] The advent of the era of converged media has brought about an explosive growth in information such as images and videos. The differences in the way these information is obtained have led to large differences in image quality (Reference 1). In this context, how to accurately evaluate image quality becomes particularly important. Through better methods and strategies for image quality evaluation, we can better screen out higher-quality images and present a more attractive visual experience for information in the era of converged media.
[0003] According to the availability of reference image information, image quality assessment can be divided into full reference (FR), partial reference (RR) and no reference (NR) (Reference 2). FR-IQA needs to be evaluated by comparing the difference between the image to be tested and the reference image, while RR-IQA needs to be evaluated by partial information of the reference image. In actual applications, we often cannot obtain reference information, so NR-IQA has become particularly widespread in actual applications.
[0004] With the development of machine learning, shallow machine learning and deep machine learning have been introduced, and general image quality assessment methods have made great progress. Most of these methods use CNN feature extraction to predict the type of image distortion and regress the quality score based on the distortion type.
[0005] Considering the complexity of distortion types, most of the subsequent deep learning-based image quality assessment methods have removed the prediction of distortion types and instead directly obtained image quality scores through feature extraction and linear regression. For example, WaDIQaM (Reference 3) trains a full-reference image quality assessment model and shares the parameters with a no-reference image quality assessment model, and further transfers learning to obtain no-reference quality assessment. In addition, methods such as WaDIQaM also introduce an attention mechanism, which simulates the difference in human eye perception of different parts of the image by assigning different attention weights to different parts of the image, so that the quality assessment provided by the model can better meet the subjective feelings of the human eye.
[0006] Although CNN can extract various features of images, current research generally believes that there is a lot of information redundancy in CNN feature extraction (Reference 4). Therefore, how to better implement feature screening and reuse and enhance the interpretability of CNN feature extraction is an important research topic in current image quality assessment research.
[0007] [1]Wang Z.Applications of objective image quality assessment methods[applications corner][J].IEEE signal processing magazine,2011,28(6):137-142.
[0008] [2]Wang Z,Bovik AC,Sheikh HR,et al.Image quality assessment:fromerror visibility to structural similarity[J].IEEE transactions on imageprocessing,2004,13(4):600-612.
[0009] [3] Bosse S, Maniry D, Müller KR, et al.Deep neural networks for no-reference and full-reference image quality assessment[J]. IEEE Transactions onimage processing, 2017, 27(1):206-219.
[0010] [4]Zhang Q, Jiang Z, Lu Q, et al. Split to be slim: An overlooked redundancy in vanilla convolution[J]. arXiv preprint arXiv:2006.12085,2020. Summary of the invention
[0011] In order to filter and reuse the image features extracted by CNN, the present invention proposes an image quality evaluation method and system based on distortion information. This method realizes that only feature extraction of image distortion information is sufficient to reflect the quality of the image through an image quality evaluation model based on distortion information, which provides a new direction for image quality evaluation research. At the same time, a low-cost training method is also proposed, which greatly improves the utilization efficiency of features.
[0012] The main technical solution of the present invention is: an image quality evaluation method based on distortion information, comprising the following steps:
[0013] Step 1: preprocess the input image and divide it into several image blocks;
[0014] Step 2, inputting the preprocessed image blocks into an image evaluation model based on distortion information to obtain an overall image quality score;
[0015] The image evaluation model based on distortion information includes a distortion extraction module, a feature extraction module and a weighted summation module;
[0016] The feature extraction module uses a convolutional neural network S and a normalization layer, whose input is the preprocessed image and output is the local attention weight;
[0017] The distortion extraction module comprises a distortion extraction network and a convolutional neural network W, whose input is the preprocessed image and output is the local quality score;
[0018] The weighted summation module obtains the overall quality score of the image by weighted summing the local quality score and the local attention weight.
[0019] Furthermore, in step 1, the input image is preprocessed, including randomly cropping it into a number of image blocks, and then normalizing the image blocks. The specific normalization function is:
[0020]
[0021] Where x is the input image block, x n is the normalized input image patch.
[0022] Furthermore, in step 2, the convolutional neural network S includes 5 convolution blocks and a linear layer to output distortion information with the same width, height and dimension as the input; the number of input channels of the linear layer is the feature dimension, and the number of output channels is the number of image blocks.
[0023] Furthermore, the distortion extraction network consists of three fully convolutional networks: downsampling, convolution, and upsampling. The downsampling part includes three convolutional layers, each of which contains three 3×3 convolution kernels and three maximum pooling layers. Skip connections are used between downsampling and upsampling. The convolutional neural network W has the same architecture as the convolutional neural network S in the feature extraction module.
[0024] Furthermore, the image evaluation model in step 2 is a trained multi-scale convolutional neural network model; the training process includes the following sub-steps:
[0025] Step S1, importing an image quality assessment dataset;
[0026] Step S2, preprocessing the data;
[0027] Step S3, input the preprocessed data into the image evaluation model, optimize the model parameters by gradient descent and minimizing the loss function, so that the image evaluation model can output the quality score of the input image more accurately.
[0028] Furthermore, in step S1, the data set includes a synthetic distortion data set and a real distortion data set.
[0029] Furthermore, the preprocessing in step S2 includes random cropping and normalization processing.
[0030] Furthermore, the specific expression of the loss function in step S3 is as follows:
[0031]
[0032] in and are two input vectors, which respectively refer to the predicted value given by the image evaluation model and the true quality score of the image. N is the total number of input samples, and i is the index of multiple sample values contained in each sample.
[0033] In a second aspect, the present invention further provides an image quality evaluation system based on distortion information, comprising the following units:
[0034] A preprocessing unit, used for preprocessing the input image and dividing it into a number of image blocks;
[0035] A quality evaluation unit, used for inputting the preprocessed image blocks into an image evaluation model based on distortion information to obtain an overall image quality score;
[0036] The image evaluation model based on distortion information includes a distortion extraction module, a feature extraction module and a weighted summation module;
[0037] The feature extraction module uses a convolutional neural network S and a normalization layer, whose input is the preprocessed image and output is the local attention weight;
[0038] The distortion extraction module comprises a distortion extraction network and a convolutional neural network W, whose input is the preprocessed image and output is the local quality score;
[0039] The weighted summation module obtains the overall quality score of the image by weighted summing the local quality score and the local attention weight.
[0040] In a third aspect, the present invention further provides an image quality evaluation device based on distortion information, comprising:
[0041] one or more processors;
[0042] A storage device is used to store one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement an image quality assessment method based on distortion information as described in the above scheme.
[0043] The present invention uses a distortion-based image quality evaluation model to evaluate the image quality, obtains the local quality score and the local attention weight through the distortion extraction module and the feature extraction module respectively, and finally, performs weighted summation of the local quality score and the local attention weight to obtain the overall quality score of the input image. Compared with the traditional image quality evaluation method based on deep learning, the present invention only evaluates the image quality from the distortion information, reduces information redundancy, reduces the difficulty of training, and enables the model to more objectively and accurately reflect the degree of image distortion, and is consistent with the subjective perception of the human eye. BRIEF DESCRIPTION OF THE DRAWINGS
[0044] The technical solution of this invention is further described below using embodiments and specific implementation methods. In addition, some drawings are also used in the process of describing the technical solution. For those skilled in the art, other drawings and the intention of the invention can be obtained based on these drawings without creative work.
[0045] Figure 1 is a flow chart of a method according to an embodiment of the present invention;
[0046] Figure 2 This is a structural diagram of a multi-level feature multiplexing image quality assessment model according to an embodiment of the present invention;
[0047] Figure 3 This is a training flowchart of a multi-level feature multiplexing image quality assessment model according to an embodiment of the present invention. DETAILED DESCRIPTION
[0048] In order to facilitate ordinary technicians in the field to understand and implement the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the implementation examples described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.
[0049] This embodiment takes a given image data set to be tested as an example to further illustrate the present invention. Figure 1 , this embodiment provides a distortion-based image quality evaluation method, comprising the following steps:
[0050] Step 1: Preprocess the input image and divide it into several image blocks.
[0051] In one implementation, the input image is preprocessed, including randomly cropping it into 32 32*32 image blocks, and then normalizing the image. The specific normalization function is:
[0052]
[0053] Where x is the input image block, x n is the normalized input image patch.
[0054] This normalization strategy greatly reduces the diversity of data distribution, thereby improving the generalization performance of the model.
[0055] Step 2: Input several preprocessed image blocks into the image evaluation model based on distortion information to obtain local quality scores and local attention weights; after obtaining the local quality scores and local attention weights respectively, the overall image quality score is obtained by weighted summation;
[0056] Please see Figure 2 , the image quality assessment model based on distortion information includes a distortion extraction module, a feature extraction module and a weighted summation module;
[0057] In one embodiment, the feature extraction module uses a convolutional neural network S and a normalization layer. The convolutional neural network S extracts features from the original image to generate attention weights. The normalization layer greatly reduces the data distribution differences, thereby improving the generalization performance of the model. The convolutional neural network S includes 5 convolution blocks and a linear layer, wherein the first two convolutional layers include two 3×3 convolutional kernels and a maximum pooling layer; the last three convolutional layers include two 3×3 convolutional kernels and a 1×1 convolutional kernel and a maximum pooling layer to output distortion information with the same width, height and dimension as the input; the number of input channels of the linear layer is the feature dimension, and the number of output channels is the number of image blocks; its input is the preprocessed image, and the output is the local attention weight.
[0058] The distortion extraction module includes a distortion extraction network and a convolutional neural network W. The distortion extraction network is used to extract the distortion information of the image. The convolutional neural network W performs feature extraction on the distortion information to obtain a local quality score. The distortion extraction network consists of three full convolutional networks: downsampling, convolution, and upsampling. The downsampling part includes three convolutional layers, each of which contains three 3×3 convolution kernels and three maximum pooling layers. Jump connections are used between downsampling and upsampling. The convolutional neural network W has the same architecture as the convolutional neural network S in the feature extraction module; its input is the preprocessed image, and its output is the local quality score.
[0059] The weighted summation module obtains the overall quality score of the image by weighted summation of the local quality score and the local attention weight;
[0060] Please see Figure 3 In one embodiment, the diffusion-guided image quality assessment model used is a trained multi-scale convolutional neural network model; the training process includes the following sub-steps:
[0061] Step S1: importing image quality assessment dataset;
[0062] In one implementation, the dataset includes both synthetic distortion datasets (eg, LIVE, CSIQ, TID2013) and real distortion datasets (eg, CLIVE, KonIQ), and these datasets all include a subjective evaluation score for each image.
[0063] Step S2: preprocessing the data;
[0064] In one implementation, the input image is preprocessed, including randomly cropping it into 32 32*32 image blocks, and then normalizing the image. The normalization formula is consistent with 2, so that the data in different data sets are unified and the model performance is improved.
[0065] Step S3: Input the preprocessed data into the multi-scale convolutional neural network model, and optimize the parameters of the model by gradient descent and minimizing the loss function, so that the model can output the quality score of the input image more accurately.
[0066] In the parameter updating process of one implementation, the efficiency of parameter updating is improved by reducing the loss function. The loss function uses the absolute error, which can be described as:
[0067]
[0068] in and are two input vectors, which respectively refer to the predicted value given by the model and the true quality score of the image. N is the total number of input samples, and i is the index of multiple sample values contained in each sample.
[0069] In order to verify the performance of the algorithm, the present invention firstly conducts experiments on four image databases, namely LIVE, CSIQ, TID2013, CLIVE, and KonIQ datasets. These datasets cover most synthetic distortions and natural distortions. The LIVE database contains 29 reference images and 779 distorted images. The distortion types include JPEG2000, JPEG compression, white noise, Gaussian blur, and Rayleigh fading. The database provides the differential mean opinion score (DMOS) of each image. A smaller DMOS value indicates a higher image quality. The CSIQ database contains 30 reference images, each of which has 6 types of distortion, and each distortion has 4-5 degrees. The TID2013 database is a widely used dataset for image quality evaluation, which contains 25 reference images and 3,000 distorted images. These distorted images cover a variety of types, including Gaussian noise, blur, compression, etc. Each image is subjectively scored, and the score value is used to quantify the perceived distortion degree of the image. The CLIVE database is a comprehensive database for image quality evaluation, containing 1,100 images, including both reference images and corresponding distorted images. Distortion types include compression, blur, etc., and each image is assigned a subjective quality score. The KonIQ database focuses on multidimensional distortion in image quality evaluation, containing 1,200 images, covering a variety of distortion types such as blur, compression, noise, etc. Each image is equipped with a corresponding subjective quality score, allowing researchers to conduct in-depth research on the impact of different distortions on the perceived quality of the image.
[0070] Table 1 shows the performance of the image quality assessment algorithm based on distortion information and other related algorithms on the LIVE, CSIQ, TID2013, CLIVE, and KonIQ datasets. It can be seen that the IQA index has very good performance on all four databases, which reflects the superior performance of the image quality assessment algorithm based on distortion information.
[0071] Table 1. Comparison of model performance on different datasets
[0072]
[0073] The highest and second highest SROCC and PLCC are indicated in bold and underlined, respectively.
[0074] The embodiment of the present invention further provides an image quality evaluation system based on distortion information, comprising the following units:
[0075] A preprocessing unit, used for preprocessing the input image and dividing it into a number of image blocks;
[0076] A quality evaluation unit, used for inputting the preprocessed image blocks into an image evaluation model based on distortion information to obtain an overall image quality score;
[0077] The image evaluation model based on distortion information includes a distortion extraction module, a feature extraction module and a weighted summation module;
[0078] The feature extraction module uses a convolutional neural network S and a normalization layer, whose input is the preprocessed image and output is the local attention weight;
[0079] The distortion extraction module comprises a distortion extraction network and a convolutional neural network W, whose input is the preprocessed image and output is the local quality score;
[0080] The weighted summation module obtains the overall quality score of the image by weighted summing the local quality score and the local attention weight.
[0081] The specific implementation method of each unit is the same as each step and will not be described in detail in the present invention.
[0082] The embodiment of the present invention further provides an image quality evaluation device based on distortion information, comprising:
[0083] one or more processors;
[0084] A storage device is used to store one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement an image quality assessment method based on distortion information as described in the above scheme.
[0085] The present invention can realize accurate quality evaluation of the image to be tested, and the evaluation result can reflect the objective distortion degree of the image and conform to the subjective perception of the human eye. It can also be applied to image processing tasks such as denoising, restoration, and super-resolution restoration of guided images, and has a good prospect for promotion and application.
[0086] It should be understood that the above description of the preferred embodiment is relatively detailed and cannot be regarded as limiting the scope of patent protection of the present invention. Under the enlightenment of the present invention, ordinary technicians in this field can also make substitutions or modifications without departing from the scope of protection of the claims of the present invention, which all fall within the scope of protection of the present invention. The scope of protection requested for the present invention shall be based on the attached claims.
Claims
1. A method for evaluating image quality based on distortion information, characterized in that: The following steps are involved: Step 1: preprocess the input image and divide it into several image blocks; Step 2, inputting the preprocessed image blocks into an image evaluation model based on distortion information to obtain an overall image quality score; The image evaluation model based on distortion information includes a distortion extraction module, a feature extraction module and a weighted summation module; The feature extraction module uses a convolutional neural network S and a normalization layer, whose input is the preprocessed image and output is the local attention weight; The convolutional neural network S consists of 5 convolutional blocks and a linear layer to output distortion information with the same width, height and dimensions as the input; the number of input channels of the linear layer is the feature dimension, and the number of output channels is the number of image blocks; The distortion extraction module comprises a distortion extraction network and a convolutional neural network W, whose input is the preprocessed image and output is the local quality score; The distortion extraction network consists of three fully convolutional networks: downsampling, convolution, and upsampling. The downsampling part includes three convolutional layers, each of which contains three 3×3 convolution kernels and three maximum pooling layers. Skip connections are used between downsampling and upsampling. The convolutional neural network W has the same architecture as the convolutional neural network S in the feature extraction module. The weighted summation module obtains the overall quality score of the image by weighted summing the local quality score and the local attention weight.
2. The image quality assessment method based on distortion information according to claim 1, characterized in that: In step 1, the input image is preprocessed, including randomly cropping it into several image blocks, and then normalizing the image blocks. The specific normalization function is: Where x is the input image block, x n is the normalized input image patch.
3. The image quality assessment method based on distortion information according to claim 1, characterized in that: The image evaluation model in step 2 is a trained multi-scale convolutional neural network model; The training process consists of the following sub-steps: Step S1, importing an image quality assessment dataset; Step S2, preprocessing the data; Step S3, input the preprocessed data into the image evaluation model, optimize the model parameters by gradient descent and minimizing the loss function, so that the image evaluation model can output the quality score of the input image more accurately.
4. The image quality assessment method based on distortion information according to claim 3, characterized in that: The data set in step S1 includes a synthetic distortion data set and a real distortion data set.
5. The image quality assessment method based on distortion information according to claim 3, characterized in that: The preprocessing in step S2 includes random cropping and normalization.
6. The image quality assessment method based on distortion information according to claim 3, characterized in that: The specific expression of the loss function in step S3 is as follows: in and are two input vectors, which respectively refer to the predicted value given by the image evaluation model and the true quality score of the image. N is the total number of input samples, and i is the index of multiple sample values contained in each sample.
7. An image quality evaluation system based on distortion information, characterized in that: The following units are included: A preprocessing unit, used for preprocessing the input image and dividing it into a number of image blocks; A quality evaluation unit, used for inputting the preprocessed image blocks into an image evaluation model based on distortion information to obtain an overall image quality score; The image evaluation model based on distortion information includes a distortion extraction module, a feature extraction module and a weighted summation module; The convolutional neural network S consists of 5 convolutional blocks and a linear layer to output distortion information with the same width, height and dimensions as the input; the number of input channels of the linear layer is the feature dimension, and the number of output channels is the number of image blocks; The distortion extraction module comprises a distortion extraction network and a convolutional neural network W, whose input is the preprocessed image and output is the local quality score; The distortion extraction network consists of three fully convolutional networks: downsampling, convolution, and upsampling. The downsampling part includes three convolutional layers, each of which contains three 3×3 convolution kernels and three maximum pooling layers. Skip connections are used between downsampling and upsampling. The convolutional neural network W has the same architecture as the convolutional neural network S in the feature extraction module. The weighted summation module obtains the overall quality score of the image by weighted summing the local quality score and the local attention weight.
8. An image quality evaluation device based on distortion information, characterized in that: include: one or more processors; A storage device for storing one or more programs, which, when executed by the one or more processors, enables the one or more processors to implement an image quality assessment method based on distortion information as described in any one of claims 1 to 6.
