Image quality evaluation method

Abstract

A novel FR-IQA normal form is presented, in which flexible reference selection is involved. It is directed to generating reference features by finding an optimal interpretation of distorted features in an equal mass space constructed based on a given original feature. Quality regression loss, interference maximization loss, and content loss are employed to optimize pseudo reference feature learning without real references for distorted images of various distortion types. Experimental results on five IQA reference databases show that performance can be improved by combining FLRE with an existing FR-IQA model based on depth features.

Description

Technical Field

[0001] The present invention relates to digital image quality assessment, particularly the quality assessment of distorted images when compared with the original images. Background Art

[0002] Objective Image Quality Assessment (IQA) aims to automatically quantify the degree of image distortion, providing quality monitoring criteria or optimization objectives for numerous vision-centered systems [1]-[8]. Generally speaking, according to the availability of reference images, existing IQA methods can be divided into three categories: Full Reference IQA (FR-IQA) [9]-

[11] , Reduced Reference IQA (RR-IQA)

[12] -

[14] , and No Reference IQA (NR-IQA)

[15] -

[22] .

[0003] Existing FR-IQA methods can be divided into two categories according to top-down and bottom-up design philosophies. The former attempts to simulate the overall function of the Human Visual System (HVS) through certain assumptions, while the latter aims to simulate the processing stages in the HVS visual pathway. A mild assumption of FR-IQA is that the distorted image is generated from a reference image of original quality, so the perceptual fidelity or similarity can be quantified. Therefore, it is natural to use the given original quality image as a reference when generating a quantitative score for quantifying fidelity / similarity.

[0004] References

[0005] The following references mentioned in this specification are denoted by numbered brackets, and their entire contents are incorporated herein by reference.

[0006] [1] L.-H. Chen, C.G. Bampis, Z. Li, A. Norkin, and A.C. Bovik, “Proxiqa: A proxy approach to perceptual optimization of learned image compression,” IEEE Trans. Image Process., vol. 30, pp. 360-373, 2021.

[0007] [2] Q. Xu, J. Xiong, X. Cao, and Y. Yao, “Parsimonious mixed-effects hodgerank for crowdsourced preference aggregation,” in Proc. ACM Int. Conf. Multimedia, 2016, pp. 841-850.

[0008] [3]Y. Zhang, L. Zhu, G. Jiang, S. Kwong, and C.-C. J. Kuo, “A survey on perceptually optimized video coding,” ACM Comput. Surv., vol. 55, no. 12, pp. 1-37, 2023.

[0009] [4]G. Zhai and X. Min, “Perceptual image quality assessment: a survey,” Sci. China Inf. Sci., vol. 63, pp. 1-52, 2020.

[0010] [5]X. Min, K. Gu, G. Zhai, X. Yang, W. Zhang, P. Le Callet, and C. W. Chen, “Screen content quality assessment: overview, benchmark, and beyond,” ACM Comput. Surv., vol. 54, no. 9, pp. 1-36, 2021.

[0011] [6]Q. Xu, J. Xiong, Q. Huang, and Y. Yao, “Online hodgerank on random graphs for crowdsourceable qoe evaluation,” IEEE Trans. Multimedia, vol. 16, no. 2, pp. 373-386, 2014.

[0012] [7]Q. Xu, Q. Huang, T. Jiang, B. Yan, W. Lin, and Y. Yao, “Hodgerank on random graphs for subjective video quality assessment,” IEEE Trans. Multimedia, vol. 14, no. 3, pp. 844-857, 2012.

[0013] [8]Q. Xu, T. Jiang, Y. Yao, Q. Huang, B. Yan, and W. Lin, “Random partial paired comparison for subjective video quality assessment via hodgerank,” in Proc. ACM Int. Conf. Multimedia, no. 10, 2011, pp. 393 - 402.

[0014] [9]Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, “Image quality assessment: from error visibility to structural similarity,” IEEE Trans. Image Process., vol. 13, no. 4, pp. 600 - 612, 2004.

[0015]

[10] X. Min, G. Zhai, K. Gu, X. Yang, and X. Guan, “Objective quality evaluation of dehazed images,” IEEE Trans. Intell. Transp. Syst., vol. 20, no. 8, pp. 2879 - 2892, 2019.

[0016]

[11] X. Liu, Y. Zhang, S. Hu, S. Kwong, C.-C. J. Kuo, and Q. Peng, “Subjective and objective video quality assessment of 3d synthesized views with texture / depth compression distortion,” IEEE Trans. Image Process., vol. 24, no. 12, pp. 4847 - 4861, 2015.

[0017]

[12] A. Rehman and Z. Wang, “Reduced-reference image quality assessment by structure similarity estimation,” IEEE Trans. Image Process., vol. 21, no. 8, pp. 3378 - 3389, 2012.

[0018]

[13] S.Golestaneh and L.J.Karam,“Reduced-reference quality assessmentbased on the entropy of dwt coefficients of locally weighted gradientmagnitudes,”IEEE Trans.Image Process.,vol.25,no.11,pp.5293-5303,2016.

[0019]

[14] J.Wu,W.Lin,G.Shi,L.Li,and Y.Fang,“Orientation selectivity basedvisual pattern for reduced-reference image quality assessment,”Inf.Sci.,vol.351,pp.18-29,2016.

[0020]

[15] Z.Pan,F.Yuan,J.Lei,Y.Fang,X.Shao,and S.Kwong,“Vcrnet:Visualcompensation restoration network for no-reference image quality assessment,”IEEE Trans.Image Process.,vol.31,pp.1613-1627,2022.

[0021]

[16] X.Liu,J.van de Weijer,and A.D.Bagdanov,“Rankiqa:Learning fromrankings for no-reference image quality assessment,”inProc.Int.Conf.Comput.Vis.,2017.

[0022]

[17] B. Chen, H. Li, H. Fan, and S. Wang, “No-reference screen content image quality assessment with unsupervised domain adaptation,” IEEE Trans. Image Process., vol. 30, pp. 5463 - 5476, 2021.

[0024]

[18] X. Wang, J. Xiong, and W. Lin, “Visual interaction perceptual network for blind image quality assessment,” IEEE Trans. Multimedia, pp. 1 - 13, 2023.

[0025]

[19] L. Shen, B. Zhao, Z. Pan, B. Peng, S. Kwong, and J. Lei, “Channel recombination and projection network for blind image quality measurement,” IEEE Trans. Instrum. Meas., vol. 71, pp. 1 - 12, 2022.

[0026]

[20] Z. Pan, H. Zhang, J. Lei, Y. Fang, X. Shao, N. Ling, and S. Kwong, “Dacnn: Blind image quality assessment via a distortion - aware convolutional neural network,” IEEE Trans. Circuits Syst. Video Technol., vol. 32, no. 11, pp. 7518 - 7531, 2022.

[0027]

[21] Z. Pan, F. Yuan, X. Wang, L. Xu, X. Shao, and S. Kwong, “No-reference image quality assessment via multibranch convolutional neural networks,” IEEE Trans. Artif. Intell., vol. 4, no. 1, pp. 148-160, 2023.

[0028]

[22] X. Min, K. Ma, K. Gu, G. Zhai, Z. Wang, and W. Lin, “Unified blind quality assessment of compressed natural, graphic, and screen content images,” IEEE Trans. Image Process., vol. 26, no. 11, pp. 5462-5474, 2017.

[0029]

[23] L. Jin, J. Y. Lin, S. Hu, H. Wang, P. Wang, I. Katsavounidis, A. Aaron, and C.-C. J. Kuo, “Statistical study on perceived jpeg image quality via mcl-jci dataset construction and analysis,” Electron. Imag., vol. 2016, no. 13, pp. 1-9, 2016.

[0030]

[24] D. M. Chandler and S. S. Hemami, “Effects of natural images on the detectability of simple and compound wavelet subband quantization distortions,” J. Opt. Soc. Am. A, vol. 20, no. 7, pp. 1164-1180, 2003.

[0031]

[25] W. Lin and G. Ghinea, “Progress and opportunities in modelling just-noticeable difference (jnd) for multimedia,” IEEE Trans. Multimedia, vol. 24, pp. 3706 - 3721, 2021.

[0032]

[26] Q. Jiang, Z. Liu, S. Wang, F. Shao, and W. Lin, “Toward top-down justnoticeable difference estimation of natural images,” IEEE Trans. Image Process., vol. 31, pp. 3697 - 3712, 2022.

[0033]

[27] X. Min, G. Zhai, J. Zhou, M. C. Q. Farias, and A. C. Bovik, “Study ofsubjective and objective quality assessment of audio-visual signals,” IEEETrans. Image Process., vol. 29, pp. 6054 - 6068, 2020.

[0034]

[28] Q. Xu, J. Xiong, Q. Huang, and Y. Yao, “Robust evaluation for quality ofexperience in crowdsourcing,” in Proc. ACM Int. Conf. Multimedia, 2013, pp. 43 - 52.

[0035]

[29] Q. Xu, Q. Huang, and Y. Yao, “Online crowdsourcing subjective imagequality assessment,” in Proc. ACM Int. Conf. Multimedia, 2012, pp. 359 - 368.

[0036]

[30] K. Ding, K. Ma, S. Wang, and E. P. Simoncelli, “Comparison of full-reference image quality models for optimization of image processing systems,” Int. J. Comput. Vis., vol. 129, pp. 1258 - 1281, 2021.

[0037]

[31] J. G. Robson, “Spatial and temporal contrast-sensitivity functions of the visual system,” Josa, vol. 56, no. 8, pp. 1141 - 1142, 1966.

[0038]

[32] Weber’s Law of Just Noticeable Difference, 2017. [Online]. Available: http: / / apps.usd.edu / coglab / WebersLaw.html

[0039]

[33] G. E. Legge and J. M. Foley, “Contrast masking in human vision,” Josa, vol. 70, no. 12, pp. 1458 - 1471, 1980.

[0040]

[34] S. J. Daly, “Visible differences predictor: an algorithm for the assessment of image fidelity,” Human Vis., Visual Process., Digital Display III, vol. 1666, pp. 2 - 15, 1992.

[0041]

[35] P. C. Teo and D. J. Heeger, “Perceptual image distortion,” in Proc. Int. Conf. Image Process., vol. 2, 1994, pp. 982 - 986.

[0042]

[36] A.B. Watson, G.Y. Yang, J.A. Solomon, and J. Villasenor, “Visibility of wavelet quantization noise,” IEEE Trans. Image Process., vol. 6, no. 8, pp. 1164 - 1175, 1997.

[0043]

[37] V. Laparra, J. Balle′, A. Berardino, and E.P. Simoncelii, “Perceptual image quality assessment using a normalized laplacian pyramid,” Human Vis. Electron. Imag., pp. 43 - 48, 2016.

[0044]

[38] L. Zhang, L. Zhang, X. Mou, and D. Zhang, “Fsim: A feature similarity index for image quality assessment,” IEEE Trans. Image Process., vol. 20, no. 8, pp. 2378 - 2386, 2011.

[0045]

[39] W. Xue, L. Zhang, X. Mou, and A.C. Bovik, “Gradient magnitude similarity deviation: A highly efficient perceptual image quality index,” IEEE Trans. Image Process., vol. 23, no. 2, pp. 684 - 695, 2013.

[0046]

[40] H.R. Sheikh and A.C. Bovik, “Image information and visual quality,” IEEE Trans. Image Process., vol. 15, no. 2, pp. 430 - 444, 2006.

[0047]

[41] S. Bosse, D. Maniry, K.-R. T. Wiegand, and W. Samek, “Deep neural networks for no-reference and full-reference image quality assessment,” IEEE Trans. Image Process., vol. 27, no. 1, pp. 206 - 219, 2017.

[0048]

[42] E. Prashnani, H. Cai, Y. Mostofi, and P. Sen, “Pie app: Perceptual image-error assessment through pairwise preference,” in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2018, pp. 1808 - 1817.

[0049]

[43] R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang, “The unreasonable effectiveness of deep features as a perceptual metric,” in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2018, pp. 586 - 595.

[0050]

[44] K. Ding, K. Ma, S. Wang, and E. P. Simoncelli, “Image quality assessment: Unifying structure and texture similarity,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 44, no. 5, pp. 2567 - 2581, 2022.

[0051]

[45] X. Liao, B. Chen, H. Zhu, S. Wang, M. Zhou, and S. Kwong, “Deepwsd: Projecting degradations in perceptual space to wasserstein distance in deep feature space,” in Proc. ACM Int. Conf. Multimedia, 2022, pp. 970 - 978.

[0052]

[46] M. Zhou, X. Wei, S. Kwong, W. Jia, and B. Fang, “Just noticeable distortion-based perceptual rate control in hevc,” IEEE Trans. Image Process., vol. 29, pp. 7603-7614, 2020.

[0053]

[47] C.-H. Chou and Y.-C. Li, “A perceptually tuned subband image coder based on the measure of just-noticeable-distortion profile,” IEEE Trans. Circuits Syst. Video Technol., vol. 5, no. 6, pp. 467-476, 1995.

[0054]

[48] X. Zhang, W. Lin, and P. Xue, “Just-noticeable difference estimation with pixels in images,” J. Vis. Commun. Image Represent., vol. 19, no. 1, pp. 30-41, 2008.

[0055]

[49] A. Liu, W. Lin, M. Paul, C. Deng, and F. Zhang, “Just noticeable difference for images with decomposition model for separating edge and textured regions,” IEEE Trans. Circuits Syst. Video Technol., vol. 20, no. 11, pp. 1648-1652, 2010.

[0056]

[50] J.Wu,G.Shi,W.Lin,A.Liu,and F.Qi,“Just noticeable differenceestimation for images with free-energy principle,”IEEE Trans.Multimedia,vol.15,no.7,pp.1705-1710,2013.

[0057]

[51] J.Wu,L.Li,W.Dong,G.Shi,W.Lin,and C.-C.J.Kuo,“Enhanced justnoticeable difference model for images with pattern complexity,”IEEETrans.Image Process.,vol.26,no.6,pp.2682-2693,2017.

[0058]

[52] Z.Wei and K.N.Ngan,“Spatio-temporal just noticeable distortionprofile for grey scale image / video in dct domain,”IEEE Trans.CircuitsSyst.Video Technol.,vol.19,no.3,pp.337-346,2009.

[0059]

[53] L.Ma,K.N.Ngan,F.Zhang,and S.Li,“Adaptive block-size trans-formbased just-noticeable difference model for images / videos,”SignalProcess.Image Commun.,vol.26,no.3,pp.162-174,2011.

[0060]

[54] S.-H. Bae and M. Kim, “A novel generalized dct-based jnd profile based on an elaborate cm-jnd model for variable block-sized transforms in monochrome images,” IEEE Trans. Image Process., vol. 23, no. 8, pp. 3227-3240, 2014.

[0061]

[55] X. Shen, Z. Ni, W. Yang, X. Zhang, S. Wang, and S. Kwong, “Just noticeable distortion profile inference: A patch-level structural visibility learning approach,” IEEE Trans. Image Process., vol. 30, pp. 26-38, 2020.

[0062]

[56] H. Lin, G. Chen, M. Jenadeleh, V. Hosu, U.-D. Reips, R. Hamzaoui, and D. Saupe, “Large-scale crowdsourced subjective assessment of picturewise just noticeable difference,” IEEE Trans. Circuits Syst. Video Technol., vol. 32, no. 9, pp. 5859-5873, 2022.

[0063]

[57] X. Min, K. Gu, G. Zhai, J. Liu, X. Yang, and C. W. Chen, “Blind quality assessment based on pseudo-reference image,” IEEE Trans. Multimedia, vol. 20, no. 8, pp. 2049-2062, 2018.

[0064]

[58] X.Min, G. Zhai, K. Gu, Y. Liu, and X. Yang, “Blind image quality estimation via distortion aggravation,” IEEE Trans. Image Process., vol. 64, no. 2, pp. 508 - 517, 2018.

[0065]

[59] J. Ma, J. Wu, L. Li, W. Dong, X. Xie, G. Shi, and W. Lin, “Blind image quality assessment with active inference,” IEEE Trans. Image Process., vol. 30, pp. 3650 - 3663, 2021.

[0066]

[60] B. Chen, L. Zhu, C. Kong, H. Zhu, S. Wang, and Z. Li, “No - reference image quality assessment by hallucinating pristine features,” IEEE Trans. Image Process., vol. 31, pp. 6139 - 6151, 2022.

[0067]

[61] K. Simonyan and A. Zisserman, “Very deep convolutional networks for large - scale image recognition,” in Proc. Int. Conf. Learn. Represent., 2015, pp. 1 - 14.

[0068]

[62] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” Commun. ACM, vol. 60, no. 6, pp. 84 - 90, 2017.

[0069]

[63] N.J. Priebe and D. Ferster, “Inhibition, spike threshold, and stimulus selectivity in primary visual cortex,” Neuron, vol. 57, no. 4, pp. 482 - 497, 2008.

[0070]

[64] K.D. Harris and T.D. Mrsic - Flogel, “Cortical connectivity and sensory coding,” Nature, vol. 503, no. 7474, pp. 51 - 58, 2013.

[0071]

[65] T. Park, M.-Y. Liu, T.-C. Wang, and J.-Y. Zhu, “Semantic image synthesis with spatially - adaptive normalization,” in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2019, pp. 2337 - 2346.

[0072]

[66] Y. Shi, X. Liu, Y. Wei, Z. Wu, and W. Zuo, “Retrieval - based spatially - adaptive normalization for semantic image synthesis,” in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2022, pp. 11224 - 11233.

[0073]

[67] D. Ulyanov, A. Vedaldi, and V. Lempitsky, “Instance normalization: The missing ingredient for fast stylization,” arXiv preprint arXiv:1607.08022, 2016.

[0074]

[68] D.C. Knill and A. Pouget, “The bayesian brain: the role of uncertainty in neural coding and computation,” Trends Neurosci., vol. 27, no. 12, pp. 712 - 719, 2004.

[0075]

[69] K. Friston, J. Kilner, and L. Harrison, “A free energy principle for the brain,” J. Physiol., vol. 100, no. 1 - 3, pp. 70 - 87, 2006.

[0076]

[70] G. Zhai, X. Min, and N. Liu, “Free - energy principle inspired visual quality assessment: An overview,” Digital Signal Process., vol. 91, pp. 11 - 20, 2019.

[0077]

[71] K. Gu, G. Zhai, X. Yang, and W. Zhang, “Using free energy principle for blind image quality assessment,” IEEE Trans. Multimedia, vol. 17, no. 1, pp. 50 - 63, 2015.

[0078]

[72] G. Zhai, Y. Zhu, and X. Min, “Comparative perceptual assessment of visual signals using free energy features,” IEEE Trans. Multimedia, vol. 23, pp. 3700 - 3713, 2021.

[0079]

[73] H. Lin, V. Hosu, and D. Saupe, “Kadid - 10k: A large - scale artificially distorted iqa database,” in Proc. Int. Conf. Quality Multimedia Exper., 2019, pp. 1 - 3.

[0080]

[74] H.R.Sheikh, M.F.Sabir, and A.C.Bovik, “A statistical evaluation of recent full reference image quality assessment algorithms,” IEEE Trans. Image Process., vol.15, no.11, pp.3440 - 3451, 2006.

[0081]

[75] N.Ponomarenko, L.Jin, O.Ieremeiev, V.Lukin, K.Egiazarian, J.Astola, B.Vozel, K.Chehdi, M.Carli, F.Battisti et al., “Image database tid2013: Peculiarities, results and perspectives,” Signal Process., Image Commun., vol.30, pp.57 - 77, 2015.

[0082]

[76] E.C.Larson and D.M.Chandler, “Most apparent distortion: full - reference image quality assessment and the role of strategy,” J.Electron. Imag., vol.19, no.1, pp.011 006 - 011 006, 2010.

[0083]

[77] F.Zhou, R.Yao, B.Liu, and G.Qiu, “Visual quality assessment for super - resolved images: Database and method,” IEEE Trans. Image Process., vol.28, no.7, pp.3528 - 3541, 2019.

[0084]

[78] X.Min, G. Zhai, K. Gu, Y. Zhu, J. Zhou, G. Guo, X. Yang, X. Guan and W. Zhang, “Quality evaluation of image dehazing methods using synthetic hazy images,” IEEE Trans. Multimedia, vol. 21, no. 9, pp. 2319 - 2333, 2019.

[0085]

[79] G. Jinjin, C. Haoming, C. Haoyu, Y. Xiaoxing, J. S. Ren, and D. Chao, “Pipal: a large-scale image quality assessment dataset for perceptual image restoration,” in Proc. Eur. Conf. Comput. Vis., 2020, pp. 633 - 651.

[0086]

[80] L. A. Gatys, A. S. Ecker, and M. Bethge, “Image style transfer using convolutional neural networks,” in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2016, pp. 2414 - 2423.

[0087]

[81] Y. Jing, Y. Yang, Z. Feng, J. Ye, Y. Yu, and M. Song, “Neural style transfer: A review,” IEEE Trans. Vis. Comput. Graph., vol. 26, no. 11, pp. 3365 - 3385, 2019.

[0088]

[82] Z. Wang, E. P. Simoncelli, and A. C. Bovik, “Multiscale structural similarity for image quality assessment,” in Proc. IEEE Asilomar Conf. Signals, Syst. Comput., vol. 2, 2003, pp. 1398 - 1402.

[0089]

[83] L.Zhang,Y.Shen,and H.Li,“Vsi:A visual saliency-induced index forperceptual image quality assessment,”IEEE Trans.Image Process.,vol.23,no.10,pp.4270-4281,2014.

[0090]

[84] V.Laparra,A.B.J.Balle,and E.P.Simoncelli,“Perceptual imagequality assessment using a normalized laplacian pyramid,”Electron.Imag.,vol.2016,no.16,pp.43-48,2016.

[0091]

[85] A.Paszke,S.Gross,F.Massa,A.Lerer,J.Bradbury,G.Chanan,T.Killeen,and Z.L.et al.,“Pytorch:An imperative style,high-performance deep learninglibrary,”Adv.Neural Inform.Process.Syst.,vol.32,pp.8026-8037,2019.

[0092]

[86] D.P.Kingma and J.Ba,“Adam:a method for stochastic optimization,”in Proc.Int.Conf.Learn.Represent.,2015.

[0093]

[87] V.Q.E.Group,“Final report from the video quality experts group onthe validation of objective models of video quality assessment,”VQEG meeting,2020.

[0094]

[88] B.Bross,J.Chen,J.-R.Ohm,G.J.Sullivan,and Y.-K.Wang,“Developments in international video coding standardization after avc,with an overview of versatile video coding(vvc),”IEEE,vol.109,no.9,pp.1463-1493,2021. Summary of the Invention

[0095] An aspect of the present invention provides an image quality assessment (IQA) method, comprising the following steps: providing an original image and a distorted image related to the original image; constructing an isometric quality space of the original image at the feature level; finding the best reference of the distortion features of the distorted image within the isometric quality space; and constructing pseudo-reference features of the distortion features.

[0096] Preferably, the step of constructing the isometric quality space includes: estimating a near-threshold map of the features extracted from the original image, and constructing the isometric quality space under the guidance of the near-threshold map.

[0097] More preferably, the step of estimating the near-threshold map includes predicting the near-threshold map according to a global spatial correlation map and a local spatial correlation map.

[0098] Most preferably, the above method further includes, before the step of predicting the near-threshold map, the following steps: calculating the global standard deviation of the features extracted from the original image, calculating the local standard deviation of the features extracted from the original image; and generating a global spatial correlation map and a local spatial correlation map according to the global standard deviation and the local standard deviation.

[0099] In a variant of the preferred embodiment, the step of finding the best reference includes locating the best reference of the distortion features within the isometric quality space in a per-element minimum distance search manner.

[0100] In another variant of the preferred embodiment, the above method further includes using at least one of a quality regression loss, an interference maximization loss, and a content loss to optimize the constructed isometric quality space.

[0101] Preferably, the step of optimizing the constructed isometric quality space uses all of the quality regression loss, the interference maximization loss, and the content loss.

[0102] In another variant of the preferred embodiment, in the step of constructing the isometric quality space, a pre-trained artificial neural network is used to construct the isometric quality space.

[0103] In yet another variation of the preferred embodiment, the step of finding the best reference is performed in each layer of the artificial neural network.

[0104] In yet another variation of the preferred embodiment, the method further includes the step of predicting a quality score based on the distortion feature and the pseudo-reference feature.

[0105] In another aspect of the present invention, there is provided a non-transitory computer-readable storage medium having computer instructions recorded thereon. When the computer instructions are executed on one or more processors, the one or more processors may be caused to perform operations according to the various methods described above.

[0106] In another aspect of the present invention, there is provided a computing system including one or more processors and a memory containing instructions. When executed by the one or more processors, the instructions cause the computing system to perform operations according to the various methods described above.

[0107] It can be seen that the embodiments of the present invention provide FR-IQA methods related to flexible reference selection. These methods are dedicated to generating reference features by finding the best explanation of the distortion feature in the iso-quality space constructed based on the given original features. Even without the ground-truth reference of distorted images of various distortion types, the learning of pseudo-reference features can be optimized.

[0108] The above summary of the invention is neither intended to define the invention of the present application (which is defined by the claims) nor to limit the scope of the present invention in any way. BRIEF DESCRIPTION OF THE DRAWINGS

[0109] The above and further features of the present invention will be clearly illustrated in the following description of the embodiments, which are provided by way of example only in conjunction with the following Figure 1 drawings:

[0110] Figure 1 The FLRE paradigm and its working principle are illustrated according to the first embodiment of the present invention.

[0111] Figure 2 Shows the Figure 1 exemplary iso-quality space constructed by FLRE.

[0112] Figure 3a Shows the Figure 2 original quality image given in the iso-quality space.

[0113] Figure 3b Shows an Figure 3a image with the same image quality as that in, which is sampled from the MCL JCL dataset

[23] .

[0114] Figure 3c is Figure 3a and Figure 3b the difference map of the images in

[0115] Figure 4 shows the comparison between traditional FR-IQA and Figure 1 the FLRE in

[0116] Figure 5 shows Figure 1 the detailed structure of the PNTE module in the FLRE in

[0117] Figure 6 is Figure 1 the illustration of the PRS strategy in the FLRE in

[0118] Figure 7 shows a table presenting the performance comparison of three traditional IQA databases.

[0119] Figure 8 shows a table presenting the performance comparison of three image restoration databases.

[0120] Figure 9 presents the visualization of the regression results of different FR-IQA models on the TID2013 and PIPAL datasets.

[0121] Figure 10 describes the visualization of the given reference features and pseudo-reference features generated by FLRE+LPIPS. Detailed implementation manner

[0122] Figure 1 shows the first embodiment of the present invention, namely the FR-IQA paradigm, which is called FLRE in the present invention. The function of FLRE is to generate a feature-level reference for the distorted image by selecting its corresponding best explanation within the equal-quality space of quality assessment. Traditional FR-IQA methods use the given original quality image as a reference to predict the perceived quality of the distorted image. However, visual perception close to the threshold indicates that there may be many indistinguishable original quality images in the scene, and the so-called original image used as a reference in FR-IQA is just one of them. Although many FR-IQA methods for evaluating perceptual similarity have been proposed, there are few methods for determining the best reference for perceptual similarity measurement. For example, both the traditional bottom-up and top-down FR-IQA methods directly apply the given reference image to quantify perceptual similarity, but they ignore whether the reference image is the best image in the sense of perceptual similarity comparison.

[0123] In contrast, the FLRE paradigm in the first embodiment of the present invention is developed in the feature space, which obtains the feature-level reference of the distorted image by selecting the corresponding best explanation in the equal-quality space, thereby providing degrees of freedom for the selection of the reference image of the distorted image. To this end, Figure 1 the FLRE in Figure 1 includes a pre-trained VGG16 network 20, a PNTE (Perceptual Near-Threshold Estimation) module 22 connected to the VGG16 network 20, and a PRS (Pseudo-Reference Search) module 24 connected to the PNTE module 22. Note that the VGG16 network 20 is also connected to the PRS module 24. For ease of understanding, in Figure 1 the same VGG16 network 20 is shown twice. The VGG16 network 20 is used to extract features from the input image. In ref the example shown, the original image I dist are both input into the VGG16 network 20 for feature extraction.

[0124] The PNTE module 22 and the PRS module 24 are configured in the FLRE to execute the PNTE and PRS strategies respectively. In particular, the PNTE module predicts the equal-quality map for the given original quality features to form the equal-quality space. Subsequently, the PRS strategy is used to locate the reference of the distorted features in the equal-quality space in a way of element-wise minimum distance search. Due to the lack of the ground-truth reference (i.e., the best explanation) for each distorted image, the pseudo-reference feature learning is optimized under three constraints, namely the quality regression loss, the interference maximization loss, and the content loss. The FLRE is implemented as a plug-in module before the deterministic FR-IQA process. The experimental results described later show that combining the FLRE with the existing deep feature-based FR-IQA models can significantly improve the quality prediction performance, far exceeding the existing state-of-the-art methods. The details of the workflow of the FLRE will be discussed below.

[0125] As Figure 1 shown, given the original image I ref and the distorted image I dist , first, their hierarchical representations, i.e., F ref and F dist , are extracted using the pre-trained VGG16 network 20. Then, the PNTE module 22 is used to predict the near-threshold map of F ref to construct the equal-quality space. Subsequently, for the distorted features at each scale, the PRS module 24 locates the reference features of the distorted features in an element-wise search manner. Finally, the quality score is predicted based on the distorted features and their pseudo-reference features.

[0126] Figure 2 Shows a schematic diagram of an exemplary isometric quality space constructed by FLRE. When centered on the given original quality image as Figure 3a shown, many isometric quality images can be generated by perturbing it in various directions and amplitudes within the visual perception threshold range, thus forming an isometric quality space. In the isometric quality space, all images have the same perceived quality as the given original quality image. Figure 3a Shows Figure 2 the given original quality image in the isometric quality space of Figure 3b , which is an isometric quality image sampled from the MCL JCL dataset. Figure 3c Shows Figure 3a and Figure 3b the difference map between. Figure 3a and Figure 3b The PSNR value between is 35.44 dB. Figure 4 Shows a comparison between traditional FR-IQA and FLRE. I A and I C are the original quality and distorted images respectively. h is the perceived quality metric in the given FR-IQA. For the traditional FR-IQA paradigm, the quality score is directly calculated based on I A and I C , that is, h(I A , I C ). For FLRE, the reference of I C is its best explanation I B within the isometric quality space, so h(I B , I C ) is calculated.

[0127] Next, the design process of FLRE will be described. As mentioned above, Figure 1 the FR-IQA paradigm in

[0128] A. Isometric Quality Space Construction

[0129] According to the near-threshold characteristics of the HVS, the PNTE module 22 is designed to estimate the near-threshold map of the features extracted from the given original quality image, so as to construct the isometric quality space under the guidance of the near-threshold map. As Figure 1 shown, first, the pre-trained VGG16 network 20

[61] is used on the ImageNet

[62] database to extract the feature representation of the given original quality image (denoted as I ref ), as follows:

[0130]

[0131] Among them, represents the feature representation of I in the s-th layer of the VGG space ref . C s and H s ×W s respectively represent the number of channels and the spatial dimension of the feature representation in the s-th layer. S is the total number of layers. In this embodiment, the feature representations of conv1_2, conv2_2, conv3_3, conv4_3, and conv5_3 in the VGG16 network 20 are adopted. The visual perception threshold is determined by the interaction or interference between stimuli. The spatial correlation in the feature map is considered here. Specifically, calculate 's global and local standard deviations to obtain the global and local spatial correlation maps (denoted as and ) respectively. Among them, in the calculation of the local standard deviation, the size of the sliding window is 3×3 and the stride is 1.

[0132] Subsequently, the near-threshold map is predicted with the help of the global and local spatial correlation maps. As Figure 5 shown, first, is normalized by the instance normalization layer

[67] (denoted as Norm) as follows:

[0133]

[0134] where μ c and σ c are calculated independently for each channel in the spatial dimension:

[0135]

[0136]

[0137] Then, the local and global spatial correlation maps are respectively input into two four-layer convolutional networks to generate the modulation parameters based on local perception (where γ l (1) represents the scale factor, and represents the bias), and the modulation parameters based on global perception (denoted as and ) respectively. Adding the two sets of parameters can obtain the final modulation parameters for (denoted as and ) respectively:

[0138]

[0139] where α γ and αβ and are learnable weight parameters. By according to and will unnormalized to generate the modulated feature

[0140]

[0141] The modulated feature is fed into a 3×3 convolution with a stride of 1 to generate a near-threshold map that is just at the critical point of perceptual equivalence Subsequently, when the input of conv2_2 changes from is replaced by , the feature representations are collected from conv2_2, conv3_3, conv4_3, and conv5_3 of the pre-trained VGG16 network 20. The hierarchical near-threshold feature maps (denoted as

[0142] ) can be expressed as:

[0143]

[0144] where, represents the near-threshold mapping of the s-th layer in the VGG space. S is the total number of layers. For , its perceptual threshold mapping (denoted as ) can be calculated as follows:

[0145]

[0146] where, |·| is the absolute value operation. Therefore, by changing the features of the given original quality image according to T (s) , an isometric quality space can be constructed, and many isometric quality features can be generated. The isometric quality mapping on the boundary of the isometric quality space can be expressed as:

[0147]

[0148] where, is the isometric quality map on the upper bound, is the isometric quality map on the lower bound.

[0149] B. Pseudo-reference Search

[0150] The human brain can actively infer the best explanation (i.e., the ideal reference) for a distorted image

[68] ,

[69] ,

[70] ,

[71] ,

[72] . Based on the assumption that the best explanation for a distorted feature is the feature with the smallest distance from the distorted feature among all features of equal quality, the PRS strategy was thus developed to locate the reference of the distorted feature in the space of equal quality by means of an element-by-element minimum distance search. Let denote the feature representation of the distorted image I dist at the s-th layer in the VGG space. Figure 6 shows examples of using the PRS strategy under different conditions. In Figure 6 , A, B, and C are respectively and while D, E, and F are different examples for . The PRS strategy is to find the feature with the smallest distance from among all features of equal quality in the space of equal quality in an element-by-element manner. More specifically, for , its reference (denoted as ) can be described as:

[0151]

[0152] where and are the distances from and to respectively. They can be calculated as follows:

[0153]

[0154] During quality prediction, the quality score Q of the distorted image is calculated according to F r ′ ef and F dist :

[0155] Q = f(F′ ref , F dist ), (16)

[0156] where and f is the FR-IQA algorithm based on deep features.

[0157] C. Objective Function

[0158] The objective function consists of three loss functions, namely the quality regression loss, the interference maximization loss, and the content loss, which are denoted by and respectively. In particular, the interference maximization loss focuses on maximizing and The difference between them enables the learned equal-quality space to contain as many equal-quality features as possible. The interference maximization loss is described as follows:

[0159]

[0160] where ∈ is a small positive constant to avoid numerical instability when the denominator approaches zero. In this embodiment, ∈ is set to 1×10 -6 . The content loss is also used to ensure that the main content represented by is the same as the main content represented by F ref in the same scene. The content loss is expressed as follows:

[0161]

[0162] where l c represents the set of VGG16 layers used to calculate the content loss. In this embodiment, l c = {conv4_2}.

[0163] To maintain the quality perception of the pseudo-reference features, a quality regression loss is adopted to minimize the mean square error between the ground-truth quality score and the predicted quality score:

[0164]

[0165] where Y is the ground-truth quality score of I dist . The objective function is defined as follows:

[0166]

[0167] where λ1 and λ2 are the weighting factors of and respectively.

[0168] In the next section, the implementation details of FLRE in Figure 1 are provided first. Then, FLRE is compared with some existing FR-IQA models to predict traditional and algorithm-related distortions. Finally, a series of ablation studies are conducted to investigate the contribution of each component in FLRE and the impact of pseudo-reference features at different layers.

[0169] A. Experimental Setup

[0170] FLRE was trained on the KADID-10k dataset

[73] and tested on three traditional IQA datasets (LIVE

[74] , CSIQ

[76] , and TID2013

[75] ) and three image restoration datasets with human-annotated scores (QADS

[77] , SHQR

[78] , and PIPAL

[79] ). More details can be seen in Table I below.

[0171] Table I IQA Database Description

[0172]

[0173] FLRE was implemented by PyTorch

[85] . The Adam

[86] optimizer was used, with an initial learning rate of 1×10 -4 , and a weight decay of 5×10 -4 . After every 5 epochs, the learning rate was reduced by 5 times. FLRE was trained on the KADID-10k dataset for 50 epochs to converge, and the batch size was set to 12. In Equation (16), existing deep feature-based methods can be used for distance measurement as f. In this experiment, DISTS

[44] , LPIPS

[43] , and DeepWSD

[45] were used, where DISTS and DeepWSD are pre-trained models on the KADID-10k dataset. For LPIPS, it was re-trained on the KADID-10k dataset for fair comparison. The input size was 256×256×3. During the training of FLRE, the parameters of the selected FR-IQA methods were fixed. When f is LPIPS, the weight parameters and in Equation (20) were set to 2 and 15 respectively. When is set to DISTS trained on the KADID-10k dataset, the weight parameters λ1 and λ2 and in Equation (20) were set to 1 and 10 respectively. When f is set to DeepWSD trained on the KADID-10k dataset, the weight parameters and in Equation (20) were set to 5 and 10 respectively.

[0174] Regarding the evaluation criteria, there are three commonly used criteria, namely the Spearman rank correlation coefficient (SRCC), the Pearson linear correlation coefficient (PLCC), and the Kendall rank correlation coefficient (KRCC), where SRCC and KRCC are used to measure the monotonicity of the prediction, and PLCC reflects the prediction accuracy. The higher the values of PLCC, KRCC, and SRCC, the more consistent the IQA model is with HVS perception. Before calculating these correlation coefficients, a non-linear logistic mapping function was used to evaluate the performance of various IQA methods in the common space. The mapping score of the prediction score Q obtained by the FR-IQA method can be calculated as:

[0175]

[0176] Among them, ξ1, ξ2, ξ3, ξ4 and ξ5 will be determined during the curve fitting process.

[0177] B. Quality prediction performance

[0178] 1) Correlation: This experiment aims to explore the consistency between the proposed model and the HVS in terms of quality prediction. To conduct performance comparison, several traditional FR-IQA models are adopted, including PNSR, SSIM[9], MS-SSIM

[82] , FSIM

[38] , VSI

[83] , VIF

[40] , NLPD

[84] , GMSD

[39] , MAD

[76] , DeepIQA

[41] , PieAPP

[42] , LPIPS

[43] , DIST

[44] and DeepWSD

[45] . To ensure the fairness of the comparison, the source codes of all participating models from their respective authors are obtained, except for DeepIQA and LPIPS. The DeepIQA and LPIPS used in this experiment are their corresponding re-trained versions on the KADID-10k dataset. The comparison results are as Figure 7 shown in Table II and Figure 8 shown in Table III, where FLRE+LPIPS, FLRE+DISTS and FLRE+DeepWSD respectively represent LPIPS, DISTS and DeepWSD of the pseudo-reference features predicted by FLRE. In Figures 7-8 , larger PLCC, SRCC and KRCC values indicate that the IQA model is more consistent with the HVS perception. The best, sub-optimal and third-best results are highlighted in bold + underline, bold and underline respectively.

[0179] As can be seen from the experimental results, compared with traditional models, the present invention combines FLRE with existing deep feature-based methods, and can achieve competitive results on LIVE, TID2013, CSIQ, QADS, and PIPAL. In addition, it can be found that FLRE can improve the performance of the original FR-IQA method. In terms of SRCC, FLRE+LPIPS on LIVE, TID2013, CSIQ, QADS, and PIPAL improved by approximately 0.23%, 1.39%, 0.05%, 1.80%, and 3.70% compared to LPIPS, respectively. FLRE+DISTS on LIVE, TID2013, CSIQ, QADS, and PIPAL improved by approximately 0.19%, 2.39%, 0.34%, 1.09%, and 0.75% compared to DISTS, respectively. FLRE+DeepWSD on LIVE, TID2013, CSIQ, QADS, and PIPAL databases improved by approximately 0.15%, 0.31%, 0.37%, 5.67%, and 10.99% compared to DeepWSD, respectively. It is worth noting that although the PLCC values of FLRE+LPIPS and FLRE+DISTS on the LIVE database are lower than those of LPIPS and DISTS, they still perform well in correctly ranking the relative image quality. For the SHRQ dataset, a slight decrease in the performance of FLRE can be observed.

[0180] Algorithms based on generative adversarial networks (GANs) have shown excellent visual performance in the field of image inpainting, but they pose a major challenge to IQAs. Different from synthetic distortions, the distortions introduced by algorithms based on generative adversarial networks are more difficult to simulate. In Table IV below, the results of the performance evaluation of different FR-IQA models on the PIPAL dataset based on GAN-based distortions are provided. The experimental results show that LPIPS, DISTS, and DeepWSD combined with FLRE can further improve the prediction performance of IQAs generated by GANs, even though they are not retrained on any images generated by image restoration algorithms. In summary, the results in Table II and Table IV demonstrate the effectiveness of FLRE, which is attributed to FLRE's ability to flexibly select reference features of distortion features among numerous equal-quality features, thus providing a more accurate reference benchmark for FR-IQA rather than the given original quality features.

[0181] Table IV

[0182] SRCC Results of GAN-Based Distortions on the PIPAL Dataset

[0183] The best two results are highlighted in bold

[0184]

[0185] Scatter Plot: To further visually demonstrate the performance of the FR-IQA model, Figure 9 shows the scatter plot of the subjective scores and the objective scores predicted by some representative IQA models on the TID2013 and PIPAL datasets. In Figure 9 , the x-axis is the fitted score Q after mapping the original objective scores generated by each FR-IQA model through formula (21), and the y-axis is the subjective score Y. The R score reflects the goodness of the fit. The higher the R score, the closer the relationship between the subjective score and the predicted score. In each subplot, the closer the fitted score is to the red line, the closer the prediction result of the FR-IQA model is to the human perceptual judgment. Here, the goodness of fit of the fitted R is used to quantify the relationship between Y and . The goodness of fit of the fitted R can be expressed as:

[0186]

[0187] where K is the total number of distorted images in the dataset. and Y k are the fitted objective score and the subjective score of the k-th distorted image, respectively. is the average of the subjective scores of all distorted images in the dataset. It can be seen from the Figure 10 results that on the TID2013 dataset and the PIPAL dataset, the FR-IQA model with FLRE improves the goodness of fit of the original model without FLRE.

[0188] C. Pseudo-Reference Generation Performance

[0189] To verify Figure 1 the accuracy of the reference predicted by the IQA method in , this method was tested on two JND datasets, namely the VVC_JND dataset

[55] and the MCL_JCL dataset

[23] . The VVC_JND dataset contains 202 original quality images and their JND versions generated by the VVC video coding reference platform

[88] . The MCL_JCL dataset contains 50 original quality images and their corresponding 50 JND images with JPEG compression distortion. When the distorted image is a JND image (i.e., the distorted image is in the equal-quality space), the reference of the distorted image is itself. In this experiment, to conduct quality assessment, the MSE value between the JND features and the reference features used by the FR-IQA model was calculated. Ideally, the reference features should be as close as possible to the JND features. Therefore, the smaller the MSE value, the better. Table V below reports the results of the average MSE (denoted by ) for the entire database.

[0190] Table V. Average MSE Comparison between True JND Features and Reference Features Used by Different FR-IQA Models for Quality Assessment

[0191]

[0192] Compared with the original FR-IQA models, it can be found that on the VVC_JND dataset, the values of FLRE+LIPIS, FLRE+DISTS, and FLRE+DeepWSD decreased by 55.45%, 70.40%, and 87.95% respectively, decreased by 55.62%, 66.13%, and 87.88% respectively on the VVC_JND dataset, and decreased by 55.62%, 66.13%, and 87.88% respectively on the MCL_JCL dataset. These results indicate that the pseudo-reference features predicted by FLRE can better explain the distorted features compared with the given original quality features.

[0193] In Figure 10 , a clear comparison is provided by visualizing the given reference features used by LPIPS and the pseudo-reference features used by FLRE+LPIPS. In Figure 10 , I is the given original quality image, and I JND is the JND version of the JPEG compression distortion. Figure 10 The sub-images (a)-(e) in Figure 10 are the feature maps of I at different scales. JND The sub-images (f)-(e) in Figure 10 are the feature maps of I Figure 10 The sub-images (k)-(o) in JND are the predicted pseudo-reference feature maps. Figure 10 The sub-images (p)-(t) in JND are the difference maps between the feature maps of I and I

[0194] It should be noted that for better visualization, for the features at each scale, the channel dimension is reduced by performing average pooling operation along its channel axis. It can be observed that the pseudo-reference features generated by the IQA method are closer to the JND features. In addition, by comparing Figure 10 the sub-images (p) and (u) in

[0195] D. Ablation Study

[0196] To study the contributions of different modules and loss functions in FLRE, ablation experiments were conducted using FLRE+DeepWSD as an example. The corresponding results are listed in Table VI below, where Experiment 1 and Experiment 6 are DeepWSD and FLRE+DeepWSD respectively. In particular, in the second experiment, the PRS of FLRE+DeepWSD was ablated, and only was used to optimize the learnable feature space of the PNTE module. When evaluating the quality of distorted images, the second variant directly used the output of the PNTE module to perform IQA without using PSR. The SRCC results in the second experiment showed that the performance of the second variant decreased significantly on all five datasets, indicating that the effectiveness of FLRE does not attribute to the increase in the number of convolutional layers. In addition, by comparing the results of the second and fourth experiments, it can be found that the model with PRS has higher performance, indicating that PRS can effectively select pseudo-reference features for IQA. To verify the effectiveness of the PNTE module, the PNTE module was replaced with two 3×3 convolutions with a stride of 1, resulting in the third experimental setting. The SRCC results showed that the PNTE module can better learn equal-quality features under the guidance of spatial correlation. By comparing the SRCC results of the fourth and fifth variants, it can be found that in the image restoration database containing GAN- and CNN-based distorted images, the model performance is better after optimizing the learned equal-quality space. Then, was further added to enhance the consistency between the main content of images in the learned equal-quality space and the given original quality image. The results showed that when and were used simultaneously, the IQA method obtained a higher SRCC value.

[0197] Table VI

[0198] Ablation study of the network architecture and loss function of FLRE+DEEPWSD. The best two results are highlighted in bold

[0199]

[0200]

[0201] In addition, FLRE+DeepWSD was compared with eight variants using pseudo-reference features of different layers:

[0202] (1)-(5) Use pseudo-reference features of a single layer; (6)-(8) Use pseudo-reference features of multiple layers. In particular, when using pseudo-reference features of a single layer, the model only executes the PNTE module and PRS of a specific layer. When using pseudo-reference features from multiple layers, first predict the near-threshold map based on and input it into VGG16 to construct an equal-quality space at each layer. Subsequently, use PRS to locate the reference features of the distortion features at each layer. Finally, obtain the reference features of different layers for IQA. Table VII below lists the SRCC comparison results on five datasets. One can find that when the original quality features given for each layer are replaced by their corresponding pseudo-reference features, the performance of the IQA model reaches the best. This phenomenon indicates that the features of different layers jointly determine the image quality. When the given original quality features are replaced by the predicted pseudo-reference features, which can better explain the distorted images, the FR-IQA model can make more accurate predictions.

[0203] Table VII

[0204] Performance comparison of FLRE+DEEPWSD using pseudo-reference features at different layers. The best two results are shown in bold.

[0205] The default settings of FLRE+DEEPWSD are shown in bold.

[0206]

[0207]

[0208] In summary, it can be seen that in the above embodiments, a new FR-IQA paradigm, namely FLRE, is proposed. This paradigm / method first generates an equal-quality space of the original quality image by identifying near-threshold distortions. Subsequently, based on the widely accepted view that the image quality is determined by intrinsic and perceptual features, a feature-level pseudo-reference of the distorted image is constructed. The main features of this embodiment can be summarized as follows:

[0209] · A new FR-IQA paradigm FLRE is proposed, which is developed at the feature level to enable the distorted image to flexibly select the best reference within the equal-quality space.

[0210] · A PNTE module is designed to construct an equal-quality space of the given original quality image at the feature level.

[0211] · A pseudo-reference search (PRS) strategy is developed, which adopts an element-level minimum distance search method. The basic principle of this strategy is that the reference feature of the distortion feature should be the feature that can best explain the distortion feature within the equal-quality space.

[0212] Accordingly, the above-described exemplary embodiments of the present invention have been fully described. Although specific embodiments have been mentioned in the description, those skilled in the art will clearly recognize that the present invention can be implemented in variations of these specific details. Therefore, the present invention should not be construed as being limited to the embodiments set forth herein.

[0213] Although the embodiments have been described in detail in the drawings and the foregoing description, they should be regarded as illustrative rather than restrictive, since only exemplary embodiments have been shown and the scope of the present invention is not limited in any way. It is understood that any feature described herein can be used in combination with any embodiment. The illustrative embodiments are not mutually exclusive of each other, nor are they exclusive of other embodiments not described in this specification. Therefore, the present invention also provides embodiments that include combinations of one or more of the above-described illustrative embodiments. Modifications and variations can be made to the invention described in this specification without departing from the spirit and scope of the present invention, and therefore, only the limitations set forth in the appended claims should be imposed.

[0214] The functional units and modules of the systems and methods according to the embodiments disclosed herein can be implemented using a computing device, a computer processor, or an electronic circuit, including but not limited to an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), and other programmable logic devices configured or programmed according to the present disclosure. The computer instructions or software code running on the computer device, computer processor, or programmable logic device can be easily written by a skilled software or electronics practitioner based on the content of the present disclosure.

[0215] All or part of the methods in accordance with the embodiments can be executed in one or more computing devices, including server computers, personal computers, laptops, and mobile computing devices, such as smartphones and tablet computers.

[0216] The embodiments include computer storage media, transient and non-transient memory devices, in which computer instructions or software code are stored and can be used to program a computer or a microprocessor to execute any process of the present invention. The storage media, transient and non-transient computer-readable storage media can include but are not limited to floppy disks, optical disks, Blu-ray discs, DVDs, CD-ROMs, magneto-optical discs, ROMs, RAMs, flash memory devices, or any type of medium or device suitable for storing instructions, code, and / or data.

[0217] Each functional unit and module in accordance with the various embodiments can also be implemented in a distributed computing environment and / or a cloud computing environment, in which all or part of the machine instructions are executed in a distributed manner by one or more processing devices interconnected by a communication network (e.g., an intranet, a wide area network, a local area network, the Internet, and other forms of data transmission media).

Claims

1. A method for evaluating image quality, comprising the following steps: Providing an original image and a distorted image related to the original image; Constructing an iso-quality space of the original image at a feature level; finding an optimal reference of the distortion feature of the distorted image in the equal-quality space; and A pseudo reference feature of the distorted feature is constructed.

2. The image quality assessment method according to claim 1, wherein the step of constructing the equal quality space comprises: estimating a near-threshold map of features extracted from the original image; as well as The equal-mass space is constructed under the guidance of the near-threshold map.

3. The image quality assessment method according to claim 2, wherein the step of estimating the near-threshold map comprises: The near-threshold map is predicted based on the global spatial correlation map and the local spatial correlation map.

4. The image quality assessment method according to claim 3, further comprising the following steps before the step of predicting the near-threshold map: Calculating a global standard deviation of the features extracted from the original image; Calculating a local standard deviation of the feature extracted from the original image; as well as The global spatial correlation map and the local spatial correlation map are generated according to the global standard deviation and the local standard deviation.

5. The image quality assessment method according to claim 1, wherein the step of finding the best reference comprises locating the best reference of the distortion feature in the equal quality space in an element-by-element minimum distance search manner. 6 . The image quality assessment method according to claim 1 , further comprising optimizing the constructed equal-quality space using at least one of a quality regression loss, an interference maximization loss, and a content loss.

7. The image quality assessment method according to claim 5, wherein the step of optimizing the constructed equal-quality space uses all of quality regression loss, interference maximization loss and content loss. 8 . The image quality assessment method according to claim 1 , wherein in the step of constructing the equal-quality space, a pre-trained artificial neural network is used to construct the equal-quality space.

9. The image quality assessment method according to claim 8, wherein the step of finding the optimal reference is performed in each layer of the artificial neural network.

10. The image quality assessment method according to claim 1, further comprising the step of predicting a quality score based on the distortion feature and the pseudo reference feature.

11. A non-transitory computer-readable storage medium having computer instructions recorded thereon, which, when executed on one or more processors, can cause the one or more processors to perform operations according to the method of claim 1.

12. A computing system comprising: one or more processors; as well as A memory containing instructions that, when executed by one or more processors, cause the computing system to perform operations according to the method of claim 1.

Images