No-reference panoramic image quality evaluation method based on visual characteristics

Abstract

The invention provides a no-reference panoramic image quality evaluation method based on visual characteristics, and the method comprises the steps: carrying out the saliency detection of a panoramic image in an ERP projection format, and obtaining a saliency image; performing coordinate transformation on the saliency image to obtain a three-dimensional spherical saliency image; performing sliding window saliency value calculation on the three-dimensional spherical saliency image to obtain a viewport image; performing hexahedral projection transformation on the panoramic image to obtain a hexahedral projection image; and feature extraction operation is carried out on the viewport image, the hexahedron projection image and the panoramic image to obtain viewport features, hexahedron projection features and global features, feature splicing operation is carried out to obtain spliced features, and a quality evaluation result is obtained through a random forest regression device. According to the method, the problems that distortion is caused by deformation stretching of the panoramic image, feature selection is inaccurate, and an evaluation result does not tend to subjective feeling of human eyes are solved.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a method for evaluating the quality of a panoramic image without reference based on visual characteristics. Background Art

[0002] In previous studies, according to the different feature expression spaces, the reference-free panoramic image quality evaluation models can be divided into three categories. The first category is the model of the panoramic image expression space. Since the panoramic space image is in a two-dimensional plane form, this type of method is also relatively simple and easy to understand, but it has obvious stretching deformation, which will introduce new distortion and affect the accuracy of the results; the second category is the model based on other projection expression spaces. This type of model is mainly to overcome the problem that panoramic images will have obvious stretching deformation at the poles. They first project the VR image to other expression spaces. Compared with the panoramic expression space, these spaces solve the problem of stretching deformation distortion introduced by panoramic mapping in the polar regions. This method also has some other mapping distortions and is still not equivalent to the spherical VR image experienced by users in reality; the third category is the model based on the actual viewing space. This type of model mainly simulates the real process or characteristics of humans watching VR content in reality to obtain objective quality evaluation results that are closer to human subjective evaluation.

[0003] The purpose of the viewport-based method is to simulate the characteristic that the human eye has a limited viewing range at a certain moment. This type of method first restores the viewport image in the actual viewing space from the panoramic image, and then calculates the quality score of the entire VR image by measuring the quality of the viewport image. Compared with the first two types, this type of model is more in line with the actual viewing characteristics of users, but there are still problems such as the inconsistency between the selected viewport image and the viewport image selected by the user in the real viewing situation and the lack of consideration of the correlation between the viewports. Therefore, it is very necessary to design a reference-free panoramic image quality evaluation method based on visual characteristics. Summary of the invention

[0004] The purpose of the present invention is to provide a reference-free panoramic image quality evaluation method based on visual characteristics, so as to alleviate the problem that the evaluation result is not close enough to the subjective perception of the human eye due to the projection deformation of the panoramic image.

[0005] To achieve the above object, the present invention provides the following solutions:

[0006] A method for evaluating the quality of a panoramic image without reference based on visual characteristics comprises the following steps:

[0007] Perform saliency detection on the panoramic image in ERP projection format to obtain a saliency image;

[0008] Perform coordinate transformation on the saliency image to obtain a three-dimensional spherical saliency image;

[0009] Calculate the sliding window saliency value of the three-dimensional spherical saliency image to obtain a viewport image;

[0010] Performing hexahedral projection transformation on the panoramic image to obtain a hexahedral projection image;

[0011] Perform feature extraction operations on the viewport image, the hexahedral projection image and the panoramic image respectively to obtain viewport features, hexahedral projection features and global features;

[0012] Perform feature splicing operation on viewport features, hexahedral projection features and global features to obtain splicing features;

[0013] A random forest regression operation is performed on the stitching features to obtain a quality score, which is used as the quality evaluation result of the panoramic image.

[0014] Optionally, saliency detection is processed using the SalGAN algorithm.

[0015] Optionally, the specific steps of performing coordinate transformation on the saliency image are:

[0016] The plane coordinates of the saliency image are converted from the sampling point coordinate system to the projection plane coordinate system. The conversion formula is: u = (m + 0.5) / W, 0 ≤ m ≤ W; v = (n + 0.5) / H, 0 ≤ n ≤ H; where (m, n) is the sampling plane coordinate, (u, v) is the projection plane coordinate, and W and H are the width and height of the projection plane respectively;

[0017] The longitude and latitude coordinates are calculated according to the projection plane coordinates. The calculation formula is: φ = (u-0.5)*(2*π); θ = (0.5-v)*π; where φ is the abscissa of longitude and latitude, and θ is the ordinate of longitude and latitude.

[0018] Optionally, the specific steps of calculating the sliding window saliency value of the three-dimensional spherical saliency image are: performing a sliding window operation on the three-dimensional spherical saliency image, if the median brightness of the window is in the top 30% of the sorting, then cutting out an image of the sliding window size on the three-dimensional spherical saliency image and saving it; if the median brightness of the window is not in the top 30% of the sorting, then continue to slide down.

[0019] Optionally, the viewport feature, the hexahedral projection feature and the global feature all include: a spatial gradient feature, a spatial gradient amplitude feature, a brightness information feature, a NIQE feature and a CORNIA feature.

[0020] Optionally, a random forest regressor is used to perform a random forest regression on the concatenated features, and the parameter n_estimators of the random forest regressor is set to 8.

[0021] According to the specific embodiment provided by the present invention, the present invention discloses the following technical effects: the method for evaluating the quality of a panoramic image without reference based on visual characteristics provided by the present invention comprises: performing saliency detection on a panoramic image in an ERP projection format to obtain a saliency image; performing coordinate transformation on the saliency image to obtain a three-dimensional spherical saliency image; performing sliding window saliency value calculation on the three-dimensional spherical saliency image to obtain a viewport image; performing hexahedral projection transformation on the panoramic image to obtain a hexahedral projection image; performing feature extraction operations on the viewport image, the hexahedral projection image and the panoramic image respectively to obtain viewport features, hexahedral projection features and global features; performing feature splicing operations on the viewport features, the hexahedral projection features and the global features to obtain splicing features; performing random forest regression operations on the splicing features to obtain a quality score, and using the quality score as the quality evaluation result of the panoramic image. The method alleviates the problem that the evaluation results caused by the projection deformation of the panoramic image are not close enough to the subjective feelings of the human eye. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0023] Figure 1 The flowchart of the method for evaluating the quality of a panoramic image without reference based on visual characteristics is shown in FIG. DETAILED DESCRIPTION

[0024] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0025] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and understandable, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0026] like Figure 1 As shown, an embodiment of the present invention provides a method for evaluating the quality of a panoramic image without reference based on visual characteristics, comprising the following steps:

[0027] Step 100: performing saliency detection on the panoramic image in the ERP projection format to obtain a saliency image;

[0028] Specifically, saliency detection is processed using the SalGAN algorithm, and the specific steps are as follows:

[0029] The entire panoramic image is input into the generator network for processing; the generator consists of a multi-layer neural network, each layer is responsible for extracting different visual features, including low-level edge and texture features and high-level structural information;

[0030] Through multiple layers of nonlinear transformations, the generator finally outputs a saliency map, which is the predicted visual attention distribution map. This process simulates how the human eye selects focus points in complex scenes.

[0031] The loss function is calculated for the saliency map, and the composite loss function KL divergence (KL_div), Pearson correlation coefficient (PLCC) and normalized scan path score (NSS) are used to jointly optimize the calculation process; in order to reduce the computational complexity, the predicted saliency map and the true saliency map (if available) are pooled to reduce their resolution;

[0032] A pair of image-saliency map combinations are input into a discriminator network, and the discriminator is trained to push the generator to produce a saliency map that is more difficult to distinguish; the image-saliency map combination includes a combination of a real saliency map and a panoramic image, and a combination of a saliency map generated by the generator and a panoramic image; this process is used to distinguish between real and fake saliency maps;

[0033] The generator and discriminator perform alternating cycles and update their weights according to their respective loss functions to improve the ability to distinguish between real salient maps and generated salient maps;

[0034] Finally, the trained predicted saliency map is obtained.

[0035] Step 200: performing coordinate transformation on the saliency image to obtain a three-dimensional spherical saliency image;

[0036] Specifically, the specific steps for coordinate transformation of the saliency image are:

[0037] The plane coordinates of the entire panoramic image and the trained predicted saliency map are converted to spherical latitude and longitude coordinates. The conversion formula is:

[0038] u=(m+0.5) / W, 0≤m≤W;

[0039] v=(n+0.5) / H, 0≤n≤H;

[0040] Where (m, n) is the sampling plane coordinate, (u, v) is the projection plane coordinate, W and H are the width and height of the projection plane respectively;

[0041] And calculate the longitude and latitude coordinates (φ, θ) based on the projection plane coordinates. The calculation formula is:

[0042] φ=(u-0.5)*(2*π);

[0043] θ = (0.5-v)*π;

[0044] Step 300: Calculate the sliding window saliency value of the three-dimensional spherical saliency image to obtain a viewport image; the specific steps are: slide the window on the predicted saliency map, the sliding window size is 480*480, the step size is 1, the threshold for selecting the viewport image is the top 30% of the median saliency of all sliding windows, if the median brightness of the sliding window is in the top 30%, then cut out the image of the sliding window size at the corresponding position of the three-dimensional spherical saliency image and save it; if the median brightness of the sliding window is not in the top 30%, continue to slide down.

[0045] Step 400: performing hexahedral projection transformation on the panoramic image to obtain a hexahedral projection image;

[0046] Specifically, the steps of the hexahedral projection transformation are: converting the panoramic image in the ERP projection format to a spherical surface, re-projecting it onto a cube model, and unfolding it to each face so that each face obtains a projection, and each face saves its own projection image separately.

[0047] Step 500: performing feature extraction operations on the viewport image, the hexahedron projection image and the panoramic image respectively to obtain viewport features, hexahedron projection features and global features;

[0048] Specifically, the viewport features include: spatial gradient features, spatial gradient amplitude features, brightness information features, NIQE features and CORNIA features. The feature categories and specific steps for extracting features from hexahedral projection images and ERP projection images are the same as the viewport image feature extraction steps.

[0049] Step 600: performing a feature splicing operation on the viewport feature, the hexahedral projection feature and the global feature to obtain a splicing feature;

[0050] Step 700: Perform a random forest regression operation on the stitching features to obtain a quality score, and use the quality score as a quality evaluation result of the panoramic image.

[0051] Specifically, a random forest regressor is used to perform a random forest regression operation on the concatenated features, and the parameter n_estimators of the random forest regressor is set to 8.

[0052] The beneficial effects of the present invention are as follows:

[0053] 1) It effectively alleviates the problem that the evaluation results are not close enough to the subjective perception of the human eye due to the projection deformation of the panoramic image;

[0054] 2) More in line with human visual characteristics;

[0055] 3) It has a quality evaluation effect that is more consistent with subjective data;

[0056] 4) It does not require a large amount of training data and high computing performance, which reduces costs.

[0057] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0058] The present invention uses specific examples to illustrate the principles and implementation methods of the present invention. The above examples are only used to help understand the method and core ideas of the present invention. At the same time, for those skilled in the art, according to the ideas of the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A no-reference panoramic image quality assessment method based on visual characteristics, It is characterized in that The steps include: Perform saliency detection on the panoramic image in ERP projection format to obtain a saliency image; Performing coordinate transformation on the saliency image to obtain a three-dimensional spherical saliency image; Performing sliding window saliency value calculation on the three-dimensional spherical saliency image to obtain a viewport image; Performing hexahedral projection transformation on the panoramic image to obtain a hexahedral projection image; Performing feature extraction operations on the viewport image, the hexahedron projection image and the panoramic image respectively to obtain viewport features, hexahedron projection features and global features; Performing a feature splicing operation on the viewport feature, the hexahedral projection feature and the global feature to obtain a splicing feature; A random forest regression operation is performed on the stitching features to obtain a quality score, and the quality score is used as a quality evaluation result of the panoramic image.

2. The method for evaluating the quality of panoramic images without reference based on visual characteristics according to claim 1, It is characterized in that The saliency detection is processed using the SalGAN algorithm.

3. The method for evaluating the quality of panoramic images without reference based on visual characteristics according to claim 1, It is characterized in that The specific steps of performing coordinate transformation on the saliency image are: The plane coordinates of the saliency image are converted from the sampling point coordinate system to the projection plane coordinate system, and the conversion formula is: u = (m + 0.5) / W, 0 ≤ m ≤ W; v = (n + 0.5) / H, 0 ≤ n ≤ H; where (m, n) is the sampling plane coordinate, (u, v) is the projection plane coordinate, and W and H are the width and height of the projection plane respectively; The longitude and latitude coordinates are calculated according to the projection plane coordinates, and the calculation formula is: φ=(u-0.5)*(2*π); θ=(0.5-v)*π; wherein φ is the abscissa of the longitude and latitude, and θ is the ordinate of the longitude and latitude.

4. The method for evaluating the quality of panoramic images without reference based on visual characteristics according to claim 1, It is characterized in that The specific steps of calculating the saliency value of the three-dimensional spherical saliency image are as follows: performing a sliding window operation on the three-dimensional spherical saliency image, if the median brightness of the window is in the top 30% of the sorting, then cutting out an image of the sliding window size on the three-dimensional spherical saliency image and saving it; if the median brightness of the window is not in the top 30% of the sorting, then continue to slide down.

5. The method for evaluating the quality of panoramic images without reference based on visual characteristics according to claim 1, It is characterized in that The viewport feature, the hexahedral projection feature and the global feature all include: spatial gradient feature, spatial gradient amplitude feature, brightness information feature, NIQE feature and CORNIA feature.

6. The method for evaluating the quality of panoramic images without reference based on visual characteristics according to claim 1, It is characterized in that A random forest regressor is used to perform a random forest regression operation on the concatenated features, and the parameter n_estimators of the random forest regressor is set to 8.

Images