A saliency-weighted method for assessing the quality of no-reference panoramic images.

Abstract

A saliency-weighted, no-reference panoramic image quality assessment method is proposed, comprising dataset selection, image preprocessing, construction of a Swing-Pigment Quality Assurance (SW-PIQA) network, training the SW-PIQA network, and testing the SW-PIQA network. The panoramic image is projected into six viewport images, resolving the stretching and distortion problem caused by the equirectangular projection format and conforming to human subjective perception. Saliency detection methods and convolutional neural networks are used to obtain the saliency and image features of each viewport image, respectively. The extracted image features are adaptively weighted and fused, and regression analysis is performed on the weighted features to obtain the image quality prediction score. Simulation comparison experiments demonstrate that the proposed method achieves good quality prediction results on different datasets, matches human subjective ratings, and is applicable to different distortion types.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, specifically relating to a method for evaluating the quality of non-reference panoramic images. Background Technology

[0002] Image quality assessment uses computers to simulate how the human eye perceives and evaluates the quality of an image. Building network models based on the habits and characteristics of human image observation often yields results that are highly consistent with subjective perception. Reasonably considering the characteristics of the human visual system is a crucial breakthrough in image quality assessment.

[0003] When viewing panoramic images using virtual reality devices, the visual content is first represented by a sphere in 3D spherical coordinates, and then rendered as a planar segment tangent to the sphere; this planar segment is called the viewport. The generation of the viewport simulates the human eye's observation behavior in virtual reality. Compared to the default projection format of panoramic images, such as rectangular projection, the viewport better matches the human eye's perceptual process. Since the human eye cannot focus on all viewport images simultaneously, it is essential to process them differently based on their attractiveness to the human eye. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to overcome the shortcomings of the prior art and provide a saliency-weighted method for evaluating the quality of non-reference panoramic images that conforms to the characteristics of the human visual system, is consistent with subjective evaluation, and is highly efficient.

[0005] The technical solution adopted to solve the above technical problems consists of the following steps:

[0006] (1) Select dataset

[0007] The panoramic images in the panoramic image quality evaluation dataset are divided into training and test sets in an 8:2 ratio, with no overlap between the training and test sets.

[0008] (2) Image preprocessing

[0009] By using cube projection, the panoramic images of the training and test sets are mapped to obtain six viewport images: front, back, left, right, top, and bottom.

[0010] (3) Constructing the SW-PIQA network

[0011] The SW-PIQA network consists of a feature extraction module 1 connected in parallel with a significance weighting module 2, and then connected in series with a quality regression module 3.

[0012] The significance weighting module 2 is composed of a significance module and a weight allocation module connected in series.

[0013] The quality regression module 3 is composed of a weighting module and a regression module connected in series.

[0014] (4) Training the SW-PIQA network

[0015] 1) Constructing the loss function

[0016] The loss function Loss is constructed as follows:

[0017]

[0018] Where M is the number of test sets, sp j For the predicted quality score of the panoramic image, st j This represents the true quality score of the panoramic image.

[0019] 2) Training the SW-PIQA network

[0020] The software requirements were CentOS 7.5 operating system, based on PyTorch 1.7.0 framework. The training set was input into the SW-PIQA network for training. The training parameters were a learning rate of 0.0001, a smoothing coefficient alpha of 0.9, a batch size of 16, and 25 training epochs, and training continued until the loss function converged.

[0021] (5) Testing the SW-PIQA network

[0022] The test set is input into the trained SW-PIQA network to obtain the predicted image quality score for the panoramic image.

[0023] In step (3) of this invention, the SW-PIQA network is constructed. The feature extraction module 1 is composed of six ResNet34 convolutional neural networks connected in parallel. The six ResNet34 convolutional neural networks extract the features Σ of the front, back, left, right, top, and bottom viewport images, respectively. n as follows:

[0024]

[0025] The feature dimension is 512, and ε n It is a 7×7 matrix, n∈{1,2,…,6}.

[0026] In step (3) of the present invention, the SW-PIQA network is constructed. The regression module is formed by connecting fully connected layer fc1, fully connected layer fc2, fully connected layer fc3, fully connected layer fc4, fully connected layer fc5, and fully connected layer fc6 in parallel, and then connecting it in series with fully connected layer fc7. The input of fully connected layer fc1, fully connected layer fc2, fully connected layer fc3, fully connected layer fc4, fully connected layer fc5, and fully connected layer fc6 is 512, and the output is 10; the input of fully connected layer fc7 is 60, and the output is 1.

[0027] In step (3) of this invention, the method for constructing the saliency module in building the SW-PIQA network is as follows:

[0028] The saliency value s of each pixel in the viewport image is determined by the following formula. k :

[0029]

[0030] Where N1 is the number of pixels in the sub-region of pixel k, N2 is the number of pixels in the neighborhood of the sub-region of pixel k, and N1 and N2 are finite positive integers, v p v is the feature vector corresponding to the pixel in the sub-region. q Let D[·] be the feature vector corresponding to the pixel in the neighborhood of the sub-region, and let D[·] be the distance metric between the average feature vector of the sub-region and the average feature vector of the neighborhood of the sub-region.

[0031] In step (3) of this invention, the method for constructing the saliency module in building the SW-PIQA network is as follows:

[0032] The saliency value s of each pixel in the viewport image is determined by the following formula. k :

[0033]

[0034] Among them, I k I is the pixel value of pixel k in the viewport image. i Let be the pixel value of pixel i in the viewport image, N be the number of pixels in the viewport image, N be a finite positive integer, and ||·|| be the Euclidean distance.

[0035] In step (3) of this invention, the method for constructing the saliency module in building the SW-PIQA network is as follows:

[0036] The saliency value s of each pixel in the viewport image is determined by the following formula. k :

[0037] s k =||I μ -I ωk ||

[0038] Among them, I μ I is the global average pixel value of the viewport image. ωk Let be the pixel value of pixel k in the viewport image after Gaussian filtering.

[0039] In step (3) of this invention, the method for constructing the saliency module in building the SW-PIQA network is as follows:

[0040] The saliency value s of each pixel in the viewport image is determined by the following formula. k :

[0041]

[0042] Where c is the color value of pixel k, r is the total number of colors, and f l For color c l The probability of appearing in the image, DC[c,c l ] represents the color values ​​c and c in the image. l The color distance between them is measured.

[0043] In step (3) of this invention, which involves constructing the SW-PIQA network, the method for constructing the weight allocation module is as follows:

[0044] The weights w of the six viewport images are determined by the following formula. n :

[0045]

[0046]

[0047] Among them, S n Let represent the total saliency value of the nth viewport image, The output weight w represents the significance value of pixel k in the nth viewport image; n This is used to construct a weight allocation module.

[0048] In step (3) of this invention, which involves constructing the SW-PIQA network, the method for constructing the weighting module is as follows:

[0049] The weighted feature E of the viewport image is determined by the following formula. n :

[0050] E n =Σ n ⊙w n

[0051] Here, ⊙ represents dot product, which is used to construct a weighted module.

[0052] The beneficial effects of this invention are:

[0053] This quality assessment method converts a single panoramic image into six viewport images. A saliency detection method is used to extract a saliency map, and weights are adaptively assigned to each viewport image based on the saliency map. The viewport images are then fed into a parallel convolutional neural network to extract features from each of the six viewport images. The features are multiplied by their corresponding weights to obtain a weighted feature vector. Regression of these six weighted feature vectors yields the predicted quality score for the panoramic image. Simulation comparison experiments show that this method can more accurately predict the perceived quality level of distorted images, achieving good quality prediction results on different datasets and exhibiting high consistency with subjective quality assessments. This invention can be used for the quality assessment of panoramic images. Attached Figure Description

[0054] Figure 1 This is a flowchart of Embodiment 1 of the present invention.

[0055] Figure 2 This is a schematic diagram of the SW-PIQA network structure.

[0056] Figure 3 yes Figure 2 A schematic diagram of the structure of the significance weighting module 2.

[0057] Figure 4 yes Figure 2 A schematic diagram of the structure of the quality regression module 3.

[0058] Figure 5 yes Figure 4 A schematic diagram of the regression module. Specific implementation methods

[0059] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments, but the present invention is not limited to the following embodiments.

[0060] Example 1

[0061] exist Figure 1 In this embodiment, the saliency-weighted no-reference panoramic image quality assessment method consists of the following steps:

[0062] (1) Select dataset

[0063] The panoramic images in the panoramic image quality evaluation dataset are divided into training and test sets in an 8:2 ratio, with no overlap between the training and test sets.

[0064] (2) Image preprocessing

[0065] By using cube projection, the panoramic images of the training and test sets are mapped to obtain six viewport images: front, back, left, right, top, and bottom.

[0066] (3) Constructing the SW-PIQA network

[0067] Figure 2 A schematic diagram of the SW-PIQA network structure is given. Figure 2 In this embodiment, the SW-PIQA network is composed of a feature extraction module 1 and a saliency weighting module 2 connected in parallel and then connected in series with a quality regression module 3.

[0068] In this embodiment, the feature extraction module 1 consists of six ResNet34 convolutional neural networks connected in parallel. The six ResNet34 convolutional neural networks extract the features Σ from the front, back, left, right, top, and bottom viewport images, respectively. n as follows:

[0069]

[0070] The feature dimension is 512, and ε n It is a 7×7 matrix, n∈{1,2,…,6}.

[0071] Figure 3 Given Figure 2 A schematic diagram of the structure of the significance weighting module 2. Figure 3 In this embodiment, the saliency weighting module 2 is composed of a saliency module and a weight allocation module connected in series.

[0072] The saliency module is constructed as follows:

[0073] The saliency value s of each pixel in the viewport image is determined by the following formula. k :

[0074]

[0075] Where N1 is the number of pixels in the sub-region of pixel k, N2 is the number of pixels in the neighborhood of the sub-region of pixel k, and N1 and N2 are finite positive integers, v p v is the feature vector corresponding to the pixel in the sub-region. q Let D[·] be the feature vector corresponding to the pixel in the neighborhood of the sub-region, and let D[·] be the distance metric between the average feature vector of the sub-region and the average feature vector of the neighborhood of the sub-region.

[0076] The construction method of the weight allocation module is as follows:

[0077] The weights w of the six viewport images are determined by the following formula. n :

[0078]

[0079]

[0080] Among them, S nLet represent the total saliency value of the nth viewport image, The output weight w represents the significance value of pixel k in the nth viewport image; n This is used to construct a weight allocation module.

[0081] Figure 4 Given Figure 2 A schematic diagram of the structure of the quality regression module 3. Figure 4 In this embodiment, the quality regression module 3 is composed of a weighting module and a regression module connected in series.

[0082] The method for constructing the weighted module is as follows:

[0083] The weighted feature E of the viewport image is determined by the following formula. n :

[0084] E n =E n ⊙w n

[0085] Here, ⊙ represents dot product, which is used to construct a weighted module.

[0086] Figure 5 Given Figure 2 A schematic diagram of the regression module. Figure 5 In this embodiment, the regression module is composed of fully connected layers fc1, fc2, fc3, fc4, fc5, and fc6 connected in parallel, and then connected in series with fully connected layer fc7. The inputs of fully connected layers fc1, fc2, fc3, fc4, fc5, and fc6 are 512, and the outputs are 10; the input of fully connected layer fc7 is 60, and the output is 1.

[0087] (4) Training the SW-PIQA network

[0088] 1) Constructing the loss function

[0089] The loss function Loss is constructed as follows:

[0090]

[0091] Where M is the number of test sets, sp j For the predicted quality score of the panoramic image, st j This represents the true quality score of the panoramic image.

[0092] 2) Training the SW-PIQA network

[0093] The software requirements were CentOS 7.5 operating system, based on PyTorch 1.7.0 framework. The training set was input into the SW-PIQA network for training. The training parameters were a learning rate of 0.0001, a smoothing coefficient alpha of 0.9, a batch size of 16, and 25 training epochs, and training continued until the loss function converged.

[0094] (5) Testing the SW-PIQA network

[0095] The test set is input into the trained SW-PIQA network to obtain the predicted image quality score for the panoramic image.

[0096] A saliency-weighted method for evaluating the quality of no-reference panoramic images was developed.

[0097] Example 2

[0098] The saliency-weighted no-reference panoramic image quality assessment method in this embodiment consists of the following steps:

[0099] (1) Select dataset

[0100] The steps are the same as in Example 1.

[0101] (2) Image preprocessing

[0102] The steps are the same as in Example 1.

[0103] (3) Constructing the SW-PIQA network

[0104] The SW-PIQA network in this embodiment is composed of a feature extraction module 1 and a saliency weighting module 2 connected in parallel and then connected in series with a quality regression module 3.

[0105] In this embodiment, the feature extraction module 1 consists of six ResNet34 convolutional neural networks connected in parallel. The six ResNet34 convolutional neural networks extract the features Σ from the front, back, left, right, top, and bottom viewport images, respectively. n as follows:

[0106]

[0107] The feature dimension is 512, and ε n It is a 7×7 matrix, n∈{1,2,…,6}.

[0108] The saliency weighting module 2 in this embodiment is composed of a saliency module and a weight allocation module connected in series.

[0109] The saliency module is constructed as follows:

[0110] The saliency value s of each pixel in the viewport image is determined by the following formula. k :

[0111] s k =||I μ -I ωk ||

[0112] Among them, I μ I is the global average pixel value of the viewport image. ωk Let be the pixel value of pixel k in the viewport image after Gaussian filtering.

[0113] The other steps are the same as in Example 1. This completes the saliency-weighted method for evaluating the quality of no-reference panoramic images.

[0114] Example 3

[0115] The saliency-weighted no-reference panoramic image quality assessment method in this embodiment consists of the following steps:

[0116] (1) Select dataset

[0117] The steps are the same as in Example 1.

[0118] (2) Image preprocessing

[0119] The steps are the same as in Example 1.

[0120] (3) Constructing the SW-PIQA network

[0121] The SW-PIQA network in this embodiment is composed of a feature extraction module 1 and a saliency weighting module 2 connected in parallel and then connected in series with a quality regression module 3.

[0122] In this embodiment, the feature extraction module 1 consists of six ResNet34 convolutional neural networks connected in parallel. The six ResNet34 convolutional neural networks extract the features Σ from the front, back, left, right, top, and bottom viewport images, respectively. n as follows:

[0123]

[0124] The feature dimension is 512, and ε n It is a 7×7 matrix, n∈{1,2,…,6}.

[0125] The saliency weighting module 2 in this embodiment is composed of a saliency module and a weight allocation module connected in series.

[0126] The saliency module is constructed as follows:

[0127] The saliency value s of each pixel in the viewport image is determined by the following formula. k :

[0128]

[0129] Where c is the color value of pixel k, r is the total number of colors, and f l For color c l The probability of appearing in the image, DC[c,c l ] represents the color values ​​c and c in the image. l The color distance between them is measured.

[0130] The other steps are the same as in Example 1. This completes the saliency-weighted method for evaluating the quality of no-reference panoramic images.

[0131] Example 4

[0132] The saliency-weighted no-reference panoramic image quality assessment method in this embodiment consists of the following steps:

[0133] (1) Select dataset

[0134] The steps are the same as in Example 1.

[0135] (2) Image preprocessing

[0136] The steps are the same as in Example 1.

[0137] (3) Constructing the SW-PIQA network

[0138] The SW-PIQA network in this embodiment is composed of a feature extraction module 1 and a saliency weighting module 2 connected in parallel and then connected in series with a quality regression module 3.

[0139] In this embodiment, the feature extraction module 1 consists of six ResNet34 convolutional neural networks connected in parallel. The six ResNet34 convolutional neural networks extract the features Σ from the front, back, left, right, top, and bottom viewport images, respectively. n as follows:

[0140]

[0141] The feature dimension is 512, and ε n It is a 7×7 matrix, n∈{1,2,…,6}.

[0142] The saliency weighting module 2 in this embodiment is composed of a saliency module and a weight allocation module connected in series.

[0143] The saliency module is constructed as follows:

[0144] The saliency value s of each pixel in the viewport image is determined by the following formula. k :

[0145]

[0146] Among them, I k I is the pixel value of pixel k in the viewport image. i Let be the pixel value of pixel i in the viewport image, N be the number of pixels in the viewport image, N be a finite positive integer, and ||·|| be the Euclidean distance.

[0147] The other steps are the same as in Example 1. This completes the saliency-weighted method for evaluating the quality of no-reference panoramic images.

[0148] To verify the beneficial effects of the present invention, the inventors selected 544 and 315 images from the CVIQ and MVAQD panoramic image quality assessment datasets, respectively, and conducted comparative simulation experiments using the method of Embodiment 1 of the present invention against Peak Signal-to-Noise Ratio (PSNR), Spherical Peak Signal-to-Noise Ratio (S-PSNR), Structural Similarity (SSIM), Spherically Weighted Structural Similarity (WS-SSIM), Blind / referenceless image spatial quality evaluator (BRISQUE), Segmented spherical projection based blind omnidirectional image quality assessment (SSP-BOIQA), Multi-channel convolution neural network for blind 360-degree image quality assessment (MC360IQA), and Multi-angle projection based blind omnidirectional image quality assessment (MP-BOIQA).

[0149] In the simulation experiment, the Pearson linear correlation coefficient (PLCC), the Spearman rank correlation coefficient (SRCC), and the root mean square error (RMSE) are determined by the following formula:

[0150]

[0151]

[0152]

[0153] Where M represents the number of test set images, st j sp is the true score of the j-th image. j It is the prediction score of the j-th image. It is the average of the true scores of the images in the test set. R is the average of the predicted scores of the test set images. tj R represents the order of the true scores of the j-th image in the test set. pj This represents the ranking of the prediction scores for the j-th image in the test set. Experimental and computational results are shown in Table 1.

[0154] Table 1. Simulation results of the method and comparative experimental method in Example 1.

[0155]

[0156] As shown in Table 1, the Pearson linear correlation coefficient, Spearman rank correlation coefficient, and root mean square error of the method of this invention tested on the CVIQ dataset are 0.9461, 0.9614, and 3.8680, respectively, and on the MVAQD dataset are 0.9336, 0.9440, and 0.3551, respectively. These results outperform existing panoramic image quality assessment methods, indicating that the model achieves good quality assessment results on different datasets.

[0157] By considering the characteristics of the human visual system, the features extracted from the viewport image are weighted and fused based on the visual saliency of the image, and the final evaluation result is highly similar to human subjective perception.

Claims

1. A method for evaluating the quality of referenceless panoramic images based on saliency weighting, characterized in that... It consists of the following steps: (1) Selecting a dataset The panoramic images in the panoramic image quality assessment dataset are divided into training and test sets in an 8:2 ratio, with no overlap between the training and test sets; (2) Image preprocessing By using cube projection, the panoramic images of the training and test sets are mapped to obtain six viewport images: front, back, left, right, top, and bottom. (3) Construct the SW-PIQA network The SW-PIQA network consists of a feature extraction module 1 connected in parallel with a significance weighting module 2, and then connected in series with a quality regression module 3. The significance weighting module 2 is composed of a significance module and a weight allocation module connected in series; The quality regression module 3 is composed of a weighting module and a regression module connected in series; (4) Training the SW-PIQA network 1) Constructing the loss function Construct the loss function as follows : in, The number of test sets, The predicted quality score for the panoramic image. The true quality score of the panoramic image; 2) Training the SW-PIQA network The software requirements are CentOS 7.5 operating system, based on PyTorch 1.7.0 framework. The training set is input into the SW-PIQA network for training. The training parameters are: learning rate of 0.0001, smoothing coefficient alpha of 0.9, batch size of 16, training epochs of 25, and training until the loss function converges. (5) Test the SW-PIQA network The test set is input into the trained SW-PIQA network to obtain the predicted image quality score for the panoramic image.

2. The method for evaluating the quality of no-reference panoramic images based on saliency weighting according to claim 1, characterized in that: In step (3) of constructing the SW-PIQA network, the feature extraction module 1 is composed of six ResNet34 convolutional neural networks connected in parallel. The six ResNet34 convolutional neural networks extract features from the front, back, left, right, top, and bottom viewport images, respectively. as follows: The feature dimension is 512. It is a 7×7 matrix. .

3. The method for evaluating the quality of referenceless panoramic images based on saliency weighting according to claim 1, characterized in that... In step (3) of constructing the SW-PIQA network, the regression module is composed of fully connected layers fc1, fc2, fc3, fc4, fc5, and fc6 connected in parallel, and then connected in series with fully connected layer fc7. The input of fully connected layers fc1, fc2, fc3, fc4, fc5, and fc6 is 512, and the output is 10; the input of fully connected layer fc7 is 60, and the output is 1.

4. The method for evaluating the quality of referenceless panoramic images based on saliency weighting according to claim 1, characterized in that... In step (3) of constructing the SW-PIQA network, the method for constructing the saliency module is as follows: The saliency value of each pixel in the viewport image is determined by the following formula. : in, For pixels The number of pixels in the sub-region For pixels The number of pixels in the neighborhood of the sub-region. and For finite positive integers, This represents the feature vector corresponding to the pixel in the sub-region. This represents the feature vector corresponding to the pixel in the neighborhood of the sub-region. It represents the distance metric between the average eigenvector of a sub-region and the average eigenvector of its neighborhood.

5. The method for evaluating the quality of referenceless panoramic images based on saliency weighting according to claim 1, characterized in that... In step (3) of constructing the SW-PIQA network, the method for constructing the saliency module is as follows: The saliency value of each pixel in the viewport image is determined by the following formula. : in, For pixels in the viewport image pixel values, For pixels in the viewport image pixel values, This represents the number of pixels in the viewport image. For finite positive integers, It is a Euclidean distance.

6. The method for evaluating the quality of no-reference panoramic images based on saliency weighting according to claim 1, characterized in that... In step (3) of constructing the SW-PIQA network, the method for constructing the saliency module is as follows: The saliency value of each pixel in the viewport image is determined by the following formula. : in, This represents the global average pixel value of the viewport image. Pixels in the viewport image after Gaussian filtering The pixel value.

7. The method for evaluating the quality of referenceless panoramic images based on saliency weighting according to claim 1, characterized in that... In step (3) of constructing the SW-PIQA network, the method for constructing the saliency module is as follows: The saliency value of each pixel in the viewport image is determined by the following formula. : in, For pixels color value, The total number of colors, For color The probability of appearing in an image. Represents the color values ​​in the image and The color distance between them is measured.

8. The method for evaluating the quality of no-reference panoramic images based on saliency weighting according to claim 1, characterized in that... In step (3) of constructing the SW-PIQA network, the method for constructing the weight allocation module is as follows: The weights of the six viewport images are determined by the following formula. : in, They represent the first The total significance value of the viewport image. Indicates the first Pixels in a wide-view image Significance value; output weight This is used to construct a weight allocation module.

9. The method for evaluating the quality of referenceless panoramic images based on saliency weighting according to claim 1, characterized in that... In step (3) of constructing the SW-PIQA network, the method for constructing the weighting module is as follows: The weighted features of the viewport image are determined by the following formula. : The feature dimension is 512. It is a 7×7 matrix. ; This represents the dot product. The weights are represented and constructed into a weighted module.

Images