Visual prototype based panoramic image quality assessment method

Abstract

The present application relates to the field of multimedia digital image processing, and particularly relates to a panoramic image quality evaluation method based on visual prototypes. The method first constructs samples containing multiple quality levels, pre-trains the backbone network for quality perception, and extracts sample features based on the pre-trained backbone network to construct visual prototypes covering different quality levels. For the panoramic image to be measured, the features of the viewport are extracted using the backbone network, and finally the matching weights of the viewport features and each visual prototype are calculated, the quality scores corresponding to each prototype are weighted and fused to realize the quality prediction of the viewport. The quality score of the panoramic image is obtained by averaging the quality prediction results of all viewports. The method of the present application does not need to rely on artificial subjective quality scoring for supervised training, the evaluation result has high subjective consistency, and has low computational complexity, which can meet the demand of efficient panoramic image quality evaluation in actual application scenarios, and has good practical application value.

Description

Technical Field

[0001] This invention relates to the field of multimedia digital image processing, and in particular to a panoramic image quality evaluation method based on visual prototypes. Background Technology

[0002] In recent years, with the rapid development of immersive media technologies such as virtual reality, panoramic images, as an important carrier of immersive visual information, have been widely used. However, panoramic images are inevitably affected by various distortions during transmission and storage, leading to a decline in quality and consequently impacting the user's viewing experience. Therefore, accurately evaluating the quality of panoramic images is of great significance for improving user experience.

[0003] Existing panoramic image quality assessment methods typically rely on supervised training of models using subjective quality ratings with human annotations to learn the mapping relationship from panoramic images to quality scores. Specifically, during training, human subjective quality ratings are used as supervisory labels, and network parameters are optimized by minimizing the difference between the predicted quality score and the label, enabling the model to learn the mapping process from panoramic images to human subjective quality ratings. However, human annotation of subjective quality ratings usually requires significant time and manpower, and supervised training models are prone to overfitting to specific dataset content and subjective quality rating distributions, resulting in insufficient generalization ability. In contrast, subjective opinion-independent image quality assessment methods are an effective way to alleviate problems such as insufficient model generalization.

[0004] Currently, opinion-agnostic image quality assessment methods primarily target traditional 2D images. Early methods established statistical distribution models of high-quality images and predicted quality based on the difference between the image to be tested and this statistical distribution. With the development of visual language models, some works have utilized quality description text to construct quality references, achieving quality prediction by calculating the semantic similarity between the image and the text. However, existing opinion-agnostic image quality assessment methods mainly rely on the statistical distribution of high-quality images or text descriptions as quality references, resulting in relatively limited modeling of visual features at different quality levels and a lack of visual quality references covering different quality levels. Furthermore, there are modal differences between text and images; text descriptions focus more on semantic content than visual quality factors such as texture details, thus making it difficult to provide accurate visual quality references. In addition, panoramic images differ significantly from traditional 2D images in terms of acquisition and processing, making these 2D-oriented methods difficult to directly apply to panoramic image quality assessment tasks. Summary of the Invention

[0005] To address the aforementioned technical problems, this invention proposes a panoramic image quality evaluation method based on visual prototypes. By introducing visual prototypes as quality references, it enables effective evaluation of the visual quality of panoramic images.

[0006] Visual prototyping is a method that constructs representative feature representations using feature aggregation. Its core idea is to form a prototype by aggregating the representations of similar samples in the feature space, and then perform predictions based on the matching relationship between the test sample and the prototype. Existing visual prototyping methods are mainly used for tasks such as semantic category recognition. This invention, however, targets panoramic image quality assessment, constructing visual prototypes corresponding to different quality levels. By establishing a correspondence between quality levels and visual features, it provides a visual quality reference for panoramic image quality assessment.

[0007] The panoramic image quality assessment method based on visual prototypes proposed in this invention, such as... Figure 1 The diagram includes the following steps: Step 1: Construction of Distorted Samples and Quality-Aware Pre-training. Distorted samples covering multiple quality levels are constructed, and the backbone network is pre-trained based on these constructed samples to enhance its quality-awareness capabilities. Step 2, Visual Prototype Construction. Features of the distorted samples from Step 1 are extracted using a pre-trained backbone network. Samples of different quality levels are then filtered, and features of samples of the same quality level after filtration are aggregated to construct the corresponding visual prototype. Step 3: Viewport Feature Extraction. The panoramic image to be tested is projected into multiple viewports, and features of each viewport are extracted using a pre-trained backbone network. Step 4: Quality Prediction. Matching weights are calculated based on the distances between viewport features and each prototype. The quality scores corresponding to each prototype are then weighted and fused to predict the quality of the viewport. The average quality prediction results for all viewports are used to obtain the quality score of the panoramic image under test. The specific implementation framework for steps 2, 3, and 4 is as follows: Figure 2 As shown.

[0008] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. This invention constructs a visual prototype covering multiple quality levels to characterize the visual features corresponding to different quality levels, enabling the model to model visual information at different quality levels in panoramic images and improving the ability to perceive changes in image quality. 2. This invention utilizes visual prototypes as quality references and predicts quality based on the matching relationship between the features of the image under test and prototypes of different quality levels, thereby achieving a high degree of consistency between the evaluation results and subjective quality evaluation results. 3. This invention completes quality prediction solely through the matching relationship between viewport features and visual prototypes, resulting in low computational complexity and facilitating rapid deployment in practical application scenarios. Attached Figure Description

[0009] Figure 1This is a flowchart of the panoramic image quality evaluation method based on visual prototypes proposed in this invention. Figure 2 This is a framework diagram of the panoramic image quality evaluation method based on visual prototypes proposed in this invention. Detailed Implementation

[0010] The technical solution of the present invention will be described in detail below with reference to embodiments. Obviously, the described embodiments are merely some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the disclosed embodiments of the present invention without creative effort are within the scope of protection of the present invention. Figure 1 As shown, the method of the present invention comprises four steps, wherein the specific implementation framework of steps 2, 3, and 4 is as follows: Figure 2 As shown.

[0011] Step 1: Construction of distorted samples and quality-aware pre-training. Considering that existing visual prototype-based methods typically employ backbone networks trained on tasks such as image classification, primarily learning semantic information in images but with limited ability to represent perceived quality, this paper first constructs a set of distortion samples containing multiple quality levels. Based on these distortion samples, the backbone network is trained to enhance its quality perception capability. Firstly, continuous image quality is divided into multiple quality levels. Referring to the subjective quality assessment criteria in Recommendation BT.500-11 of the International Telecommunication Union Radiocommunication Sector, image quality is divided into 5 levels, employing 5 common distortion types: Gaussian blur, Gaussian noise, JPEG, overexposure, and brightness reduction. For each distortion type, 5 different distortion intensity ranges are set to correspond to the 5 quality levels. Considering that the perceived quality of images within the same quality level can vary, unlike existing methods, a distortion intensity range with a certain range is used, and distortion images are generated by randomly selecting distortion intensities within this range. This strategy can cover possible visual quality variations at the same quality level, ensuring that images generated by different distortion types have roughly the same perceptual quality at the same quality level. It also enhances sample diversity, which is beneficial for the backbone network to learn representative features related to quality levels. Since this invention uses the viewport of a panoramic image as model input, which has a similar representation to traditional two-dimensional images, training samples are constructed based on a two-dimensional image dataset. Considering the rich scene content of the DIV2K dataset, all 800 training images and 100 validation images from this dataset are used to generate distortion samples. Specifically, five distortion types and their corresponding five distortion intensities are added to all training images, generating a total of 800×5×5=20000 distortion images, which serve as training samples for the backbone network. 100×5×5=2500 distortion images are generated on all validation images using the same method as reference samples for subsequent visual prototype construction, providing quality reference information to the model. After the distorted samples are constructed, Swing Transformer V2 is used as the backbone network, and the quality level of the training samples is used as the label to perform quality-aware pre-training on the backbone network, so that the model learns discriminative quality representations of different quality levels, thereby enhancing its quality-awareness ability.

[0012] Step 2: Visual prototype construction. Based on the concept of visual prototypes, a visual prototype is constructed using reference samples for each quality level to represent the representative features of the same quality level, providing a quality reference for quality prediction. First, a pre-trained backbone network is used to extract features from the reference samples: ; in, For the first The first quality level One reference sample, , , This represents the pre-trained backbone network. Although using distortion intensity ranges aims to maintain consistent perceptual quality among samples of the same quality level, differences in image content and distortion characteristics may still cause some reference samples to deviate from the set quality level. These samples introduce additional noise, thus reducing the representativeness of the visual prototype. Therefore, a sample selection strategy is designed to remove noisy samples from the reference samples. Specifically, the backbone network is used to predict... Quality level The sample is then compared to its preset quality level. If the two are consistent, the perceived quality of the sample is considered to match the preset quality level, and the sample is retained for visual prototype construction; if there is a difference, the sample is considered noise and discarded. ; in, for The filtering mask value. Visual prototypes are typically used to represent feature centers of similar samples; therefore, this invention constructs corresponding visual prototypes by clustering sample features of the same quality level: ; in, For the first The number of reference samples retained for each quality level. The visual prototype to be constructed.

[0013] Step 3: Viewport feature extraction. First, use cubespan projection to project the panoramic image into 6 viewports. Then, a pre-trained backbone network is used to extract quality features for each viewport: ; in, For the first Features of a viewport .

[0014] Step 4, Quality Prediction. Since the degree of matching between the test sample and each prototype is usually measured based on the distance between them, the common 1-norm is first used to calculate the distance between the viewport features and each visual prototype: ; in, The absolute value operation is used to obtain a more stable feature representation. For the first The first viewport and the second The distance between each visual quality feature reflects the degree of matching between the viewport and each visual prototype; a smaller distance indicates a higher degree of matching. A lower degree of matching indicates a significant quality difference between the corresponding visual prototype and the viewport, suggesting that the quality reference provided by the visual prototype may be redundant information, thus affecting the quality prediction results. Therefore, the distance results between the viewport and all visual prototypes are selected, retaining only the smallest distances. The smallest possible result: ; in, This is the set of distance results between the viewport and all visual prototypes. For the selected set, The selection process for the results is as follows: ; in, for middle The set of minimum results is then used to normalize the selected distance results into matching weights. : ; in, for The first in The results are then used to weight and sum the quality scores corresponding to each visual prototype to obtain the viewport quality prediction result. : ; in, A corresponding quality score is assigned to each visual prototype. Finally, the quality prediction results of all viewports are averaged to obtain the quality score of the panoramic image. : . Repeat steps 3 and 4 to obtain the quality prediction results for all the panoramic images to be tested.

[0015] To verify the applicability of the method of this invention to panoramic images with different distortion types, tests were conducted on three publicly available panoramic image quality assessment databases: MVAQD, OIQA, and IQA-ODI. The MVAQD database contains 315 panoramic images with distortion types including JPEG, JP2K, HEVC, white noise, and Gaussian blur; the OIQA database contains 320 panoramic images with distortion types including JPEG, JP2K, white noise, and Gaussian blur; and the IQA-ODI database contains 1080 JPEG-distorted panoramic images generated under isometric cylindrical projection, cube projection, cylindrical parabolic projection, icosahedral projection, and octahedral projection formats. Six image quality assessment methods independent of subjective opinions were selected as comparison methods. The performance of each method was compared using the performance indicators of the quality assessment model: Spearman Rank Correlation Coefficient (SROCC), Pearson Linear Correlation Coefficient (PLCC), and Root Mean Square Error (RMSE). Larger values ​​for SROCC and PLCC indicate better model performance, while smaller values ​​for RMSE also indicate better model performance. Experimental results are shown in Table 1. The results demonstrate that, compared to other methods, the prediction results of this invention show higher consistency with subjective quality assessment results across the three databases. Table 1 Performance comparison of different methods

[0016] Table 2 shows the time required for each method to process the same panoramic image on a device equipped with an i7 3.4GHz CPU and 32GB RAM. It can be seen that, except for CLIP-IQA, the method of this invention requires the least computational time, indicating that the method achieves a good balance between prediction accuracy and inference efficiency, and has good practical application value. Table 2 Calculation Time of the Method

Claims

1. A panoramic image quality assessment method based on visual prototypes, characterized in that: The specific steps are as follows: Step 1: Construction of Distorted Samples and Quality-Aware Pre-training. Distorted samples covering multiple quality levels are constructed, and the backbone network is pre-trained based on these constructed samples to enhance its quality-awareness capabilities. Step 2, Visual Prototype Construction. Features of the distorted samples from Step 1 are extracted using a pre-trained backbone network. Samples of different quality levels are then filtered, and features of samples of the same quality level after filtration are aggregated to construct the corresponding visual prototype. Step 3: Viewport Feature Extraction. The panoramic image to be tested is projected into multiple viewports, and features of each viewport are extracted using a pre-trained backbone network. Step 4: Quality Prediction. Matching weights are calculated based on the distances between viewport features and each prototype. The quality scores corresponding to each prototype are then weighted and fused to predict the quality of the viewport. The average quality prediction results for all viewports are used to obtain the quality score of the panoramic image under test.

2. The panoramic image quality assessment method based on visual prototypes according to claim 1, characterized in that: Step 2 first involves filtering all reference samples and then using the features of the filtered samples to construct a visual prototype. Specifically, Indicates the first The first quality level Using reference samples, a pre-trained backbone network is used to predict... Quality level The sample is then compared to its preset quality level. If a difference exists, the sample is considered noise and discarded. ； in, for The filter mask value is set, with 1 and 0 indicating that the sample is kept and discarded, respectively. This mask value is then used in conjunction with the features of the reference sample to construct a visual prototype. ； in, For the first The number of reference samples retained for each quality level. The visual prototype to be constructed.

3. The panoramic image quality assessment method based on visual prototypes according to claim 1, characterized in that: In step 4, quality prediction is performed based on the matching degree between the viewport features of the panoramic image under test and each visual prototype. Specifically, firstly, based on existing visual prototype matching methods, the distance between the viewport features of the panoramic image and each visual prototype is calculated: ； in, The first panoramic image to be tested Features of a viewport , For the absolute value operation, This yields the corresponding distance results. Then, the distance results between the viewport and all visual prototypes are selected, retaining only the results from the selected sources. The smallest possible result: ； in, This is the set of distance results between the viewport and all visual prototypes. For the selected set, The selection process for the results is as follows: ； in, for middle The set of minimum results is then used to normalize the selected distance results into matching weights. : ； in, for The first in The results are then used to weight and sum the quality scores corresponding to each visual prototype to obtain the viewport quality prediction result. : ； in, A corresponding quality score is assigned to each visual prototype. Finally, the quality prediction results of all viewports are averaged to obtain the quality score of the panoramic image. : 。

Images