A method for evaluating the quality of panoramic images with no-reference comprehensive perception

Abstract

A method for evaluating the quality of a panoramic image without reference, comprising: analyzing the global and local structures of the panoramic image to obtain structural information; calculating a color residual map to obtain the color information of the panoramic image; statistically analyzing the dependence of pixels in the spatial domain and frequency domain to obtain the statistical information of the image; inputting all the above information as features into a regression model for training and prediction, and finally obtaining the quality score of the panoramic image. The present invention studies the characteristics of panoramic images from multiple aspects, can comprehensively perceive the quality of panoramic images and make predictions. Compared with other no-reference image quality evaluation methods, it has more reliable accuracy.

Description

Technical Field

[0001] The present invention relates to the technical field of image quality evaluation. Specifically, it relates to a method for evaluating the quality of panoramic images with no-reference and comprehensive perception. Background Art

[0002] As a method for capturing panoramic images, virtual reality (VR) technology concretizes the process of people's perception of the world by simulating the human eye mechanism, and can provide viewers with views in all directions at the same time. Therefore, it has attracted more and more attention from viewers and researchers. Different from ordinary images, panoramic images capture and display information in the form of a sphere, with a broader field of view and higher resolution. Therefore, distortion will inevitably be introduced during the processes of processing, encoding, and compression, affecting the viewer's experience.

[0003] Image quality evaluation methods can generally be divided into full-reference, semi-reference, and no-reference methods. Full-reference metrics require all reference information, semi-reference only requires partial reference information, while no-reference methods predict the quality of an image without any reference information.

[0004] In existing research, many metrics widely used in two-dimensional images have been established, such as structural similarity (SSIM), peak signal-to-noise ratio (PSNR), visual information fidelity (VIF), etc. However, when directly applying traditional image quality evaluation models to evaluate the quality of panoramic images, good performance cannot be achieved. Therefore, later, a full-reference panoramic image quality evaluation method was proposed. However, in actual situations, reference images are usually difficult to obtain. Therefore, panoramic image quality evaluation models that do not require any reference information have attracted more and more attention from researchers.

[0005] However, current research methods do not comprehensively consider the characteristics of panoramic images and do not achieve good performance, resulting in the inability to form a reliable and accurate method for evaluating the quality of panoramic images with no-reference perception. Summary of the Invention

[0006] To make up for the deficiencies of the existing technology, the present invention proposes a method for evaluating the quality of panoramic images with no-reference and comprehensive perception, which considers the characteristics of panoramic images from multiple aspects and obtains better performance and more reliable accuracy.

[0007] The object of the present invention is achieved by the following technical solutions: A method for evaluating the quality of panoramic images with no-reference and comprehensive perception, the steps of which are:

[0008] S1: Analyze the global and local structures of the panoramic image to obtain structure information: For the input panoramic image, calculate its global gradient map and local texture binary pattern (TLBP) map respectively. Use the probability density following the two-parameter Weibull distribution to reflect its global structure characteristics; for the TLBP map, calculate its gray-level co-occurrence matrix (GLCM) from four directions, and calculate three statistics for the GLCM to extract the local structure characteristics of the image.

[0009] S2: Calculate the color residual map to obtain the color information of the panoramic image: Convert the image in the color space into multiple color channels representing different image information;

[0010] Perform a convolution operation on the channel image that can express color information based on the spatial rich model (SRM) to calculate its color residual map, and use the GLCM to extract its color characteristics for the color residual map;

[0011] S3: Statistically analyze the dependence of pixels in the spatial domain and frequency domain to obtain the statistical information of the image: Use the natural scene statistics (NSS) method in the spatial domain to capture the basic statistical characteristics of the panoramic image; use image entropy in the frequency domain to statistically analyze the dependence relationship between pixels.

[0012] S4: Use all the above information as features and input them into the SVR regression model for training and prediction. Select the RBF as the kernel function to finally obtain the quality score of the panoramic image.

[0013] In the above-mentioned S1, the four directions are: horizontal, vertical, main diagonal, and secondary diagonal; use three statistics (I e , I c , I m ) to summarize the local structure characteristics, and its formula is as follows:

[0014]

[0015]

[0016]

[0017] Among them, i and j are the horizontal and vertical coordinates of the pixel respectively, N represents the order of the GLCM; d and θ are the distance parameter and direction parameter of the GLCM respectively, and pd,θ(i,j) is calculated according to the following formula:

[0018]

[0019] Among them, G(i,j) is the pixel value of the gray-level co-occurrence matrix at (i,j).

[0020] In the above-mentioned S2, the conversion of the color space is calculated according to the following formula:

[0021]

[0022] Among them, O1, O2, and O3 are the transformed color spaces, and R’, G’, and B’ are calculated according to the following formulas:

[0023]

[0024] Among them, R, G, and B are different color channels, and <> represents the mean operation.

[0025] The beneficial effects of the present invention are as follows:

[0026] The evaluation method for non-reference comprehensive perception of panoramic image quality provided by the invention extracts structural features, effectively describes global and local structural features by using the complementary relationship between the gradient map and the TLBP map, and the GLCM is used to reveal local structural features, which can well reflect the rich quality perception information hidden in adjacent pixels in the image.

[0027] The evaluation method for non-reference comprehensive perception of panoramic image quality provided by the invention proposes a novel color descriptor based on the color residual image and the GLCM, which can effectively reflect the image quality of some distortions related to color information.

[0028] The evaluation method for non-reference comprehensive perception of panoramic image quality provided by the invention extracts statistical information from the spatial domain NSS features and the image entropy, and accurately describes the unnatural degree of the panoramic image in the multi-scale representation.

[0029] The evaluation method for non-reference comprehensive perception of panoramic image quality provided by the invention comprehensively perceives the quality of the panoramic image by calculating the structural features, color features, and statistical features of the panoramic image. The performance of the model, including the Pearson linear correlation coefficient, Spearman rank correlation coefficient, Kendall rank correlation coefficient, and root mean square error, is better than other existing technologies. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 is a flowchart of the evaluation method for non-reference surface perception of panoramic image quality according to an embodiment of the present invention;

[0031] Figure 2 is the relationship between the MOS and the predicted score of the model test image of the present invention;

[0032] Figure 3 is a framework diagram for implementing the evaluation method for non-reference surface perception of panoramic image quality according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0033] The present invention system and method will be described in detail below in conjunction with the accompanying drawings and specific embodiments to further understand its purpose, solution and effect.

[0034] A method for evaluating the quality of a panoramic image without reference and with comprehensive perception, the steps of which are as follows:

[0035] S1: Analyze the global and local structures of the panoramic image to obtain structure information: For the input panoramic image, calculate its global gradient map and local texture binary pattern TLBP map respectively, and use the probability density following the two-parameter Weibull distribution to reflect its global structure characteristics; For the TLBP map, calculate its gray-level co-occurrence matrix GLCM from four directions, and calculate three statistics for the GLCM to extract the local structure characteristics of the image;

[0036] In the above S1, the four directions are: horizontal, vertical, main diagonal, and secondary diagonal; Use three statistics (I e , I c , I m ) to summarize the local structure characteristics, and the formula is as follows:

[0037]

[0038]

[0039]

[0040] Among them, i and j are the horizontal and vertical coordinates of the pixel respectively, N represents the order of the GLCM; d and θ are the distance parameter and direction parameter of the GLCM respectively, and pd,θ(i,j) is calculated according to the following formula:

[0041]

[0042] Among them, G(i,j) is the pixel value of the gray-level co-occurrence matrix at (i,j).

[0043] S2: Calculate the color residual map to obtain the color information of the panoramic image: Convert the image in the color space into multiple color channels representing different image information;

[0044] The conversion of the color space is calculated according to the following formula:

[0045]

[0046] Among them, O1, O2, O3 are the converted color spaces, and R’, G’, B’ are calculated according to the following formula:

[0047]

[0048] Among them, R, G, and B are different color channels, and <> represents the mean operation.

[0049] Perform a convolution operation on the channel image that can express color information based on the Spatial Rich Model (SRM), calculate its color residual map, and use the Gray-Level Co-Occurrence Matrix (GLCM) to extract its color features from the color residual map.

[0050] S3: Statistically analyze the dependence of pixels in the spatial domain and frequency domain to obtain the statistical information of the image: Use the Natural Scene Statistics (NSS) method in the spatial domain to capture the basic statistical features of the panoramic image; use image entropy in the frequency domain to statistically analyze the dependence relationship between pixels.

[0051] S4: Use all the above information as features to input into the Support Vector Regression (SVR) model for training and prediction, select the Radial Basis Function (RBF) as the kernel function, and finally obtain the quality score of the panoramic image.

[0052] Embodiment 1:

[0053] As Figure 1 shown is the flowchart of the evaluation method for the quality of a reference-free panoramic image according to an embodiment of the present invention.

[0054] Refer to Figure 1 , the evaluation method for the quality of the reference-free panoramic image in this embodiment includes:

[0055] S1: Analyze the global and local structures of the panoramic image to obtain the structural information.

[0056] In the preferred embodiment, the further analysis of the global structure of the panoramic image in S1 includes: The input image undergoes a convolution operation to calculate its gradient map.

[0057] Furthermore, calculate the Weibull distribution of the gradient map. The probability density function of the Weibull distribution is as follows:

[0058]

[0059] where x is the image gradient response, and α and β represent its shape parameter and scale parameter.

[0060] In the preferred embodiment, the two parameters of the Weibull distribution are used to reflect the global structure features of the panoramic image.

[0061] In the preferred embodiment, the further analysis of the local structure of the panoramic image in S1 includes: Calculate its TLBP map for the input panoramic image.

[0062] Furthermore, use the GLCM in the TLBP map to summarize the local structure features.

[0063] In the preferred embodiment, GLCM is calculated from four directions: horizontal, vertical, main diagonal, and secondary diagonal. Three statistics (I e , I c , I m ) summarizes the local structural characteristics, and its formula is as follows:

[0064]

[0065]

[0066]

[0067] Among them, i and j are the horizontal and vertical coordinates of the pixel, respectively, and N represents the order of GLCM. d and θ are the distance parameter and direction parameter of GLCM, respectively. pd,θ(i,j) is calculated as follows:

[0068]

[0069] Among them, G(i,j) is the pixel value of the gray-level co-occurrence matrix at (i,j).

[0070] S2: Calculate the color residual map to obtain the color information of the panoramic image;

[0071] In a preferred embodiment, S2 further includes: converting the image into a color space into a plurality of color channels representing different image information, performing a convolution operation on the channel image capable of expressing color information based on a spatial rich model (SRM), and calculating its color residual map;

[0072] Furthermore, GLCM is used in the color residual map to summarize its color features.

[0073] In the preferred embodiment, the conversion of the color space is calculated according to the following formula:

[0074]

[0075] Among them, O1, O2, O3 are the converted color spaces. R', G', B' are calculated according to the following formula:

[0076]

[0077] Among them, R, G, and B are different color channels, indicating the mean operation.

[0078] S3: Count the dependencies of pixels in the spatial domain and frequency domain to obtain statistical information of the image;

[0079] In the preferred embodiment, the statistics of the dependence of pixels in the spatial domain and the frequency domain described in S3, and obtaining the statistical information of the image further includes: capturing the basic statistical features of the panoramic image by using the natural scene statistics (NSS) method in the spatial domain; and using the image entropy to statistically analyze the dependence relationship between pixels in the frequency domain.

[0080] In the preferred embodiment, the basic statistical information of the panoramic image is effectively expressed by using the coefficients of MSCN, and is modeled by the following formula:

[0081]

[0082] where x represents the coefficient of MSCN, and α and σ 2 are the parameters of GGD. μ is calculated by the following formula:

[0083]

[0084] where γ() is the gamma function, and is calculated according to the following formula:

[0085]

[0086]

[0087]

[0088] where α, τ, β, and δ are the parameters of AGGD.

[0089] In the preferred embodiment, the basic statistical features are expressed by using the parameters of the above GGD and AGGD; and the dependence relationship between statistical pixels is expressed by using the image entropy histogram features.

[0090] S4: Input all the above information as features into the regression model for training and prediction, and finally obtain the quality score of the panoramic image.

[0091] In the preferred embodiment, the regression model in S4 is further SVR, and RBF is selected as the kernel function.

[0092] Add the meanings of the parameters in all formulas.

[0093] To verify the feasibility of the method of this embodiment, the present invention uses the publicly reliable panoramic image dataset OIQA to evaluate the model proposed by the present invention. Among them, the OIQA dataset contains 16 reference panoramic images and 320 distorted panoramic images, and the distorted images are obtained from 16 original images through JPEG, JPEG2000, Gaussian noise GN, and Gaussian blur GB compression types and combined with five compression levels. The experimental environment of this experiment is the win10 system and the matlab2018a experimental platform.

[0094] The evaluation metrics used in this invention are PLCC, SRCC, KRCC, and RMSE, and the median values of PLCC, SRCC, KRCC, and RMSE are used as the final evaluation metrics. The higher the values of PLCC, SRCC, and KRCC, and the lower the value of RMSE, the better the performance of the model.

[0095] In this experiment, each database is divided into two non-overlapping parts: 80% as the training set and 20% as the test set. Then, the extracted features and the corresponding subjective scores are used to train the regression model. To avoid contingency in the calculation process, the test and training processes are iterated 1000 times.

[0096] The following Table 1 is a comparison table of the overall performance of the algorithm of this invention and other full-reference / reference quality evaluation algorithms on the OIQA database.

[0097] Table 1 Comparison of the overall performance of quality evaluation algorithms on the OIQA dataset

[0098]

[0099] As Figure 2 shown is the relationship between the MOS and the predicted score of the test images of the model of this invention, Figure 2 which is the relationship between the MOS and the predicted score of the test images of the model on the OIQA database. Figure 2 In it, the horizontal axis represents the objective score predicted by the model, and the vertical axis represents the subjective value provided by the dataset. It can be seen that the objective score predicted by the model of this invention has a good fitting degree with the provided subjective score.

[0100] From Table 1 described above and Figure 2 it can be seen that, compared with other quality evaluation algorithms, the model proposed in this invention can better perceive the subjective evaluation of humans on images. Therefore, the image quality evaluation is more accurate and reliable.

[0101] As Figure 3 shown is the framework diagram of the method for evaluating the quality of a reference-free surface-aware panoramic image for implementing an embodiment of this invention.

[0102] Please refer to Figure 3 , in this embodiment, first, the distorted panoramic image 1 is obtained, and then the perceptual features 2 of the image are extracted, that is, in S1, the global and local structures of the panoramic image are analyzed to obtain the structural information; in S2, the color residual map is calculated to obtain the color information of the panoramic image; and in S3, the dependence of pixels is statistically analyzed in the spatial domain and the frequency domain to obtain the statistical information of the image. After obtaining the multi-faceted perceptual features 3, all the above information is used as features to be input into the regression model for training and prediction, and finally the quality score 4 of the panoramic image is obtained.

[0103] The above are only the preferred embodiments of the present invention. The present specification selects and specifically describes these embodiments to better explain the principle of the present invention, rather than limiting the present invention. For those of ordinary skill in the art, it can be understood that substitutions, changes, modifications, and variations made without departing from the principle and scope of the present invention should all fall within the scope protected by the present invention.

Claims

1. A method for evaluating the quality of panoramic images with full perception without reference, characterized in that, the steps are as follows: S1: Analyze the global and local structures of the panoramic image to obtain structure information: For the input panoramic image, calculate its global gradient map and local texture binary pattern (TLBP) map respectively, and use the probability density following the two-parameter Weibull distribution to reflect its global structure characteristics; For the TLBP map, calculate its gray-level co-occurrence matrix (GLCM) from four directions, and calculate three statistics of the GLCM to extract the local structure characteristics of the image; S2: Calculate the color residual map to obtain the color information of the panoramic image: Convert the image in the color space into multiple color channels representing different image information; Perform convolution operations on the channel images that can express color information based on the spatial rich model (SRM), calculate its color residual map, and use the GLCM to extract its color characteristics from the color residual map; S3: Statistically analyze the dependence of pixels in the spatial domain and frequency domain to obtain the statistical information of the image: Use the natural scene statistics (NSS) method in the spatial domain to capture the basic statistical characteristics of the panoramic image; Use image entropy in the frequency domain to statistically analyze the dependence relationship between pixels; S4: Input all the above information as features into the SVR regression model for training and prediction, select RBF as the kernel function, and finally obtain the quality score of the panoramic image.

2. The method for evaluating the quality of panoramic images with full perception without reference according to claim 1, characterized in that, In S1, the four directions are: horizontal, vertical, main diagonal, and secondary diagonal; three statistics (I e , I c , I m ) are used to summarize the local structural features, and the formula is as follows: wherein, i and j are the horizontal and vertical coordinates of the pixel respectively, N represents the order of the GLCM; d and θ are the distance parameter and direction parameter of the GLCM respectively, and pd,θ(i,j) is calculated according to the following formula: where G(i,j) is the pixel value of the gray-level co-occurrence matrix at (i,j).

3. The method for evaluating the quality of panoramic images with full perception without reference according to claim 1, characterized in that, in the said S2, the conversion of the color space is calculated according to the following formula: where O1, O2, O3 are the converted color spaces, and R’, G’, B’ are calculated according to the following formula: where R, G, B are different color channels, and < > represents the mean operation.

Images