Panoramic image quality evaluation method and system

Abstract

The invention relates to a panoramic image quality evaluation method and system. The method comprises the following steps: acquiring a to-be-evaluated panoramic image, and generating a panoramic depth map and a panoramic saliency map according to the to-be-evaluated panoramic image; sampling according to the panoramic saliency map to obtain a plurality of sampling points, and obtaining a plurality of groups of physical attributes and a series of viewport images according to the plurality of sampling points, the panoramic saliency map and the panoramic depth map; performing feature extraction to obtain multi-scale features; performing multi-scale fusion according to the physical attributes of the viewport and the multi-scale features to obtain fused features; and according to the physical attributes of the viewport and the fused features, carrying out adaptive weight fusion of the quality hierarchy, and outputting a final quality prediction score to complete quality evaluation. According to the method, the physical attributes of the viewport are obtained through calculation, and feature extraction, fusion and adaptive fusion are performed by using the physical attributes of the viewport, so that adaptive perception of distortion visibility along with changes of view field conditions is realized.

Description

Technical Field

[0001] This invention relates to the field of computer vision, and more specifically, to a method and system for evaluating the quality of panoramic images. Background Technology

[0002] Panoramic images offer a 360-degree omnidirectional visual experience and are widely used in virtual reality, street view maps, and other fields. However, image distortion is inevitably introduced during the transmission, compression, and processing of panoramic images. To automatically evaluate the quality of these images, current mainstream methods typically employ a "viewport sampling" strategy: projecting a panoramic image in equidistant cylindrical projection (ERP) format onto a sphere. Several fixed locations on the sphere are selected (e.g., uniformly distributed at the equator, the center of a regular hexahedron, etc.) or determined based on a saliency map, and planar images with a fixed field of view (FOV, typically 90 degrees) are taken as viewports. These viewports are then input into a convolutional neural network (CNN) to extract features and regress quality scores.

[0003] When the human eye observes the world, the lens has an automatic accommodation function. For near objects (small depth) or objects of high interest (high salience), the human eye tends to focus on details, which is equivalent to zooming in to obtain higher angular resolution. For distant backgrounds (large depth) or areas of low interest, the human eye tends to observe the overall outline, which is equivalent to zooming out. Current technology forces the use of a fixed FOV (such as 90°) for all areas, resulting in insufficient ability to capture near details with distortion, or unnecessary oversampling of distant backgrounds, which is seriously inconsistent with the actual perception process of the human eye.

[0004] Existing deep learning networks (such as the Swin Transformer) typically scale input viewport images to a uniform size (e.g., 224×224 pixels). The network cannot determine whether the "texture detail distortion" in this viewport image originates from the original image or is caused by post-scaling, or whether the loss of global semantics is due to fixed FOV sampling. This lack of physical scale makes it difficult for the network to distinguish between "texture detail distortion" and "global structural distortion." Existing methods often simply average the scores across all viewports. However, the human eye's overall perception of image quality is often determined by the few most prominent and closest regions (the opposite of the "weakest link" effect). Simple average pooling masks severe distortion in critical regions. Existing Transformer-based evaluation methods (such as the Swin Transformer) typically use the output of the last layer of the network directly as a feature representation, or simply concatenate the features from four layers. However, different stages of a deep neural network contain information at different frequencies: shallow layers (Stages 1-2) contain rich high-frequency texture and noise information, while deep layers (Stages 3-4) contain low-frequency semantic and structural information. When we capture viewports of different FOVs to simulate human eye zoom, small FOV viewports (close-ups) are primarily used to examine pixel-level texture details (such as blur and noise), and should ideally utilize more of the network's shallow features; while large FOV viewports (wide-angle) are primarily used to examine overall geometric structures (such as seams and geometric distortions), and should ideally utilize more of the network's deep features. Existing technologies ignore this correspondence between "viewport physical properties" and "network feature levels," leading to mixed evaluation features and reduced accuracy. Summary of the Invention

[0005] Therefore, it is necessary to provide a more accurate panoramic image quality evaluation method and system to address the aforementioned technical problems.

[0006] One method for evaluating the quality of panoramic images includes: S1: Obtain the panoramic image to be evaluated, and generate a panoramic depth map and a panoramic saliency map based on the panoramic image to be evaluated. S2: Multiple sampling points are obtained by sampling based on the panoramic saliency map. Multiple physical attributes are obtained based on the multiple sampling points, the panoramic saliency map, and the panoramic depth map, and a series of viewport images are obtained. S3: Extract features based on the physical properties of the viewport and the viewport image to obtain multi-scale features; S4: Perform multi-scale fusion based on the physical properties of the viewport and multi-scale features to obtain the fused features; S5: Perform adaptive weighted fusion of quality levels based on the physical properties of the viewport and the fused features, and output the final quality prediction score to complete the quality evaluation.

[0007] Further, in step S1, specifically: constructing a monocular depth estimation network and a saliency detection network; using the monocular depth estimation network to generate a panoramic depth map, and using the saliency detection network to generate a panoramic saliency map.

[0008] Further, in step S2, multiple sampling points are obtained by sampling based on the panoramic saliency map, specifically as follows: The Non-Maximum Suppression (NMS) algorithm is applied to the panoramic saliency map, selecting N saliency peak points as sampling points, and the coordinates of the sampling points are... ,in .

[0009] Further, in step S2, obtaining multiple sets of physical attributes and a series of viewport images based on multiple sampling points specifically involves using the sampling points as the viewport center; For each viewport center, obtain the normalized depth value corresponding to the viewport center based on the panoramic saliency map and the panoramic depth map. and normalized significance value ; Field of view The calculation is as follows:

[0010] Reference field of view; : Indicates a significant focusing effect; the higher the significance, The smaller; : Indicates the wide-angle effect; the larger the depth value, The larger; , As the adjustment coefficient, it must satisfy the calculated... Within a reasonable range; Based on coordinates in the viewport and the calculated field of view Bilinear interpolation is used to project a spherical image into a fixed-resolution planar viewport image. .

[0011] Further, in step S3, specifically: the SwinTransformer Tiny pre-trained on the ImageNet dataset is used as the backbone network for feature extraction.

[0012] Further, in step S3, specifically: For each viewport, extract the field of view (FOV). Average normalized depth Significance of average normalization depth and pre-enhancement layer index And concatenate them into a physical vector:

[0013] in This represents vector concatenation, where the vector is sequentially input into two fully connected layers and a non-linear activation function to model the non-linear relationship between physical properties and visual sensitivity. Based on the physical vector, four independent prior feature vectors are output:

[0014] The four prior feature vectors correspond to the features of the four stages of the Swin Transformer. This is used to enhance the network's perception of prior physical information about the viewport; Perform sensory enhancement: Given stage characteristics First, global average pooling is performed:

[0015] This vector preserves the global semantic statistics of the features at this scale; Channel statistics Compared with the physical prior enhancement features obtained in the previous step By fusing, we obtain:

[0016] in This represents vector concatenation. The concatenated vector is projected onto the input feature channel dimension using a fully connected layer. The fusion result is then normalized using Softmax to obtain the enhancement weights.

[0017] The enhanced weights are then reinjected into the original features:

[0018] in This indicates element-wise multiplication.

[0019] Further, in step S4, specifically: For each viewport, first extract its field of view and average normalized depth. Significance of average normalization depth Concatenate them into a single physical vector:

[0020] The four-stage features are added pixel by pixel:

[0021] All features are upsampled to the highest resolution before being added together; Then on Perform global average pooling (GAP) to obtain a global feature vector:

[0022] global feature vectors With physical prior vector Perform splicing and cascading:

[0023] splicing vectors Input a two-layer MLP:

[0024] in:

[0025] Physical meaning: Smaller field of view → Increase the weight of lower-level features

[0026] Larger field of view → Increase the weight of high-rise features

[0027] Saliency and depth together modulate the scope of attention for texture / structural features; The generated weights are used to perform a weighted summation of the aligned feature vectors to obtain the final representation vector of the viewport. :

[0028] By combining fusion features with physical priors: .

[0029] Further, in step S5, specifically: the fused features are divided into a detail-aware group and a structure-aware group according to the physical properties of the viewport; Prior information is obtained based on the physical properties of the viewport. The N fused features are divided into two groups, and a gating factor is formed by passing them through a two-layer fully connected network and a Sigmoid activation function:

[0030] in Used to divide viewport features into: Detail-aware group: Suitable for areas with rich textures and prominent local details; Structure-aware groups: suitable for areas dominated by geometric structure and shape contours; Two sets of features passed:

[0031] To achieve information-guided grouping based on physical priors; Adaptive weighted fusion is performed on the detail-aware group and the structure-aware group respectively.

[0032] Furthermore, the adaptive weighted fusion of the detail-aware group and the structure-aware group is specifically performed as follows: Construct detail-aware branches, which employ a small receptive field structure with strong scale adaptability:

[0033] in This represents a non-linear activation function; the final enhanced feature is the element-wise summation of two convolutional layers:

[0034] Construct a structure-aware branch, which uses a 5×5 convolutional structure with a large receptive field:

[0035] Similarly, merge element by element:

[0036] The enhanced two types of features are then used to generate global representations through global pooling:

[0037] The weights of the output branches of the two MLPs and Softmax with shared parameters are as follows:

[0038] This is then used to multiply back into the two enhanced branches, achieving dynamic branch-level control:

[0039] Detail-aware branches use global max pooling to highlight significant high-frequency regions:

[0040] The structure-aware branch uses global average pooling to obtain an overall balanced representation:

[0041] Using prior vectors Generate fusion weights:

[0042] The final viewport quality score is: .

[0043] Furthermore, this invention provides a panoramic image quality evaluation system, comprising: Acquisition module: Acquires the panoramic image to be evaluated, and generates a panoramic depth map and a panoramic saliency map based on the panoramic image to be evaluated; Sampling viewport module: Multiple sampling points are obtained by sampling based on the panoramic saliency map. Based on the multiple sampling points, the panoramic saliency map, and the panoramic depth map, multiple sets of physical attributes are obtained, and a series of viewport images are obtained. Feature extraction module: Extracts features based on the physical properties of the viewport and the viewport image to obtain multi-scale features; Feature fusion module: Performs multi-scale fusion based on the physical properties of the viewport and multi-scale features to obtain fused features; Weighted fusion module: Based on the physical properties of the viewport and the fused features, it performs adaptive weighted fusion of quality levels and outputs the final quality prediction score to complete the quality evaluation.

[0044] This invention calculates the physical properties of the viewport and uses these properties for feature extraction, fusion, and adaptive fusion to achieve adaptive perception of distorted visibility as the field of view changes. Attached Figure Description

[0045] Figure 1 Here is a flowchart of a panoramic image quality evaluation method in one embodiment; Figure 2 This is a block diagram of a panoramic image quality assessment system in one embodiment; Detailed Implementation To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0046] Example 1: This embodiment provides, as follows: Figure 1 A panoramic image quality assessment method is shown, comprising: S1: Obtain the panoramic image to be evaluated, and generate a panoramic depth map and a panoramic saliency map based on the panoramic image to be evaluated. S2: Multiple sampling points are obtained by sampling based on the panoramic saliency map. Multiple physical attributes are obtained based on the multiple sampling points, the panoramic saliency map, and the panoramic depth map, and a series of viewport images are obtained. S3: Extract features based on the physical properties of the viewport and the viewport image to obtain multi-scale features; S4: Perform multi-scale fusion based on the physical properties of the viewport and multi-scale features to obtain the fused features; S5: Perform adaptive weighted fusion of quality levels based on the physical properties of the viewport and the fused features, and output the final quality prediction score to complete the quality evaluation.

[0047] This embodiment calculates the physical properties of the viewport and uses these properties for feature extraction, fusion, and adaptive fusion to achieve adaptive perception of distorted visibility as the field of view changes.

[0048] Example 2: This embodiment further discloses information based on Embodiment 1: This embodiment implements steps S1, S2, S3, and S4 through modules one, two, three, and four, respectively. Module 1: Lens Simulation Sampling Module with Deep-Signal Coupling This module aims to simulate the process by which the human eye's lens automatically adjusts its focus based on the distance to an object and the degree of interest.

[0049] 1. Sampling Center Determination: Apply the Non-Maximum Suppression (NMS) algorithm to the panoramic saliency map and select N saliency peak points as the viewport center coordinates. ,in .

[0050] 2. Adaptive Field of View (FOV) Calculation: For each viewport center, obtain its corresponding normalized depth value. (0 represents very close, 1 represents very far) and normalized significance value (0 indicates not significant, 1 indicates highly significant).

[0051] Define the field of view The calculation model is as follows:

[0052] Reference field of view (e.g., 90 degrees).

[0053] : Indicates a significant focusing effect. The higher the significance, The smaller the size (simulating the human eye's "gazing" behavior, magnifying the details).

[0054] : Indicates the wide-angle effect. The larger the depth value (the farther away the object), the greater the effect. The larger (simulating the human eye's "glance" behavior towards distant scenery).

[0055] , As the adjustment coefficient, it must satisfy the calculated... Within a reasonable range (e.g., between 50 and 110 degrees Celsius).

[0056] 3. Reprojection generation: based on center coordinates and the calculated field of view Bilinear interpolation is used to project a spherical image into a planar viewport image with a fixed resolution (e.g., 224 × 224). .

[0057] Module 2: Feature Enhancement Module for Field of View Perception This module is based on Swin Transformer (a hierarchical visual transformation network based on shift windows). By introducing physical conditions, it adaptively enhances features at different scales, enabling the network to perceive whether the input viewport image is "macro" or "wide-angle".

[0058] 1. Basic architecture: The Swin Transformer Tiny pre-trained on the ImageNet dataset is used as the backbone network for feature extraction.

[0059] 2. Physics-prior-driven feature generation: For each viewport, we first extract its field of view (FOV). Average normalized depth Significance of average normalization depth and pre-enhancement layer index And concatenate them into a physical vector:

[0060] in This represents vector concatenation, where the vector is sequentially input into two fully connected layers and a non-linear activation function to model the non-linear relationship between physical properties and visual sensitivity. The network outputs four independent prior feature vectors:

[0061] The four prior feature vectors correspond to the features of the four stages of the Swin Transformer.

[0062] This is used to enhance the network's perception of prior physical information about the viewport.

[0063] 3. Perception Enhancement Submodule Each scale is equipped with an independent "perception enhancement module". Its structure includes the following steps: Given stage characteristics First, global average pooling is performed:

[0064] This vector preserves the global semantic statistics of features at this scale, which is an important foundation for visual sensitivity modeling.

[0065] Channel statistics Compared with the physical prior enhancement features obtained in the previous step By fusing, we obtain:

[0066] in This represents vector concatenation. The concatenated vector is projected onto the input feature channel dimension using a fully connected layer. The fusion result is then normalized using Softmax to obtain the enhancement weights.

[0067] This design ensures that the relative importance of different channels can be dynamically adjusted based on physical prior enhancement features and viewport features. Ultimately, the enhancement weights are injected back into the original features:

[0068] in This indicates element-wise multiplication. Features enhanced at different scales will be used for subsequent quality regression. This design allows the network to add physical prior information at different feature levels, addressing the key issue of "the same distortion having different visibility at different fields of view / depths".

[0069] 4. Technical Effects: By introducing Viewport features integrate prior information on field of view size, saliency, and depth. For example, when At very low magnification (high amplification), the network pays more attention to high-frequency texture noise; when When the angle is very large (wide angle), the network will pay more attention to geometric distortion.

[0070] Module 3: Dynamic Scale Hierarchical Feature Fusion Module The function of this module is to build a "controller" that determines "which layer's features to prioritize" based on the physical properties of the viewport.

[0071] 1. Construction of physical prior vectors: For each viewport, first extract its field of view (FOV). Average normalized depth Significance of average normalization depth Concatenate them into a single physical vector:

[0072] 2. Gating weight generation To obtain global semantic cues for weight inference, we sum the four-stage features pixel by pixel:

[0073] In this process, all features are upsampled to the highest resolution stage before being added together to ensure scale alignment.

[0074] Then on Perform global average pooling (GAP) to obtain a global feature vector:

[0075] global feature vectors With physical prior vector Perform splicing and cascading:

[0076] This vector serves as a crucial input to the weight generation network, enabling the model to automatically adjust its preference for features at different stages based on the physical imaging conditions of the viewport. (Concatenated vector) Input a two-layer MLP:

[0077] in:

[0078] Physical meaning: Smaller field of view (zoom-in) → Increase the weight of lower-level features

[0079] Larger field of view (zoom-out) → Increase the weight of high-level features

[0080] Saliency and depth jointly modulate the scope of attention for texture / structural features. 3. Feature-weighted fusion The generated weights are used to perform a weighted summation of the aligned feature vectors to obtain the final representation vector of the viewport. :

[0081] To further preserve the physical perception control signal, the fused features are concatenated with the physical priors again before the final output:

[0082] In this way, "on-demand material retrieval" is achieved from the four stages of the Swing Transformer.

[0083] Module 4: Dual-stream Scale Focusing Feature Aggregation Module This module simulates the human eye's brain's weighted processing mechanism for different visual signals, namely, "the clearer and more focused the area, the greater its impact on the overall quality assessment." To fully utilize the detail sensitivity and structural robustness presented by the viewport under different distortion types, a dual-stream scale focusing feature aggregation is designed. A learnable prior control vector is constructed using the viewport's physical properties (field of view, depth, saliency). This vector controls the dynamic enhancement process of two types of perceptual features (detail perception and structural perception), and further performs adaptive weighted fusion at the output level based on quality hierarchy.

[0084] 1. Viewport branch gating: Based on prior information The N viewport feature vectors are divided into two groups, and these vectors are input into a two-layer fully connected network and a sigmoid activation function to form a gating factor.

[0085] in Used to divide viewport features into: Detail Group: Suitable for areas with rich textures and prominent local details.

[0086] Structure Group: Applicable to areas dominated by geometry and shape contours.

[0087] Two sets of features passed:

[0088] To achieve information-guided assignment based on physical priors.

[0089] 2. Viewport-aware interaction: To enhance the distortion of fine textures, the detail branches adopt a "small receptive field" structure with strong scale adaptability:

[0090] in This represents the non-linear activation function (ReLU). The final enhanced feature is the element-wise summation of two convolutional layers:

[0091] To emphasize the overall shape and geometry, the structural branches use a 5×5 convolutional structure with a large receptive field:

[0092] Similarly, merge element by element:

[0093] The enhanced two types of features are then used to generate global representations through global pooling:

[0094] The weights of the output branches of the two MLPs and Softmax with shared parameters are as follows:

[0095] This is then used to multiply back into the two enhanced branches, achieving dynamic branch-level control:

[0096] 3. Branch output quality calculation and final fusion The detailed quality branch uses Global Max Pooling (GMP) to highlight significantly high-frequency regions:

[0097] The structural quality branch uses global average pooling (GAP) to achieve an overall balanced representation:

[0098] To allow physical properties to play a direct role in the final mass prediction, the prior vector is utilized again. Generate fusion weights:

[0099] The final viewport quality score is:

[0100] This embodiment has the following beneficial effects: Multi-stage feature gating mechanism based on physical properties An external feature selection architecture deployed outside the pre-trained Swin Transformer backbone network is proposed. This architecture uses the physical properties of the viewport—including field of view (FOV), saliency, and depth information—as prior conditions. A lightweight gating sub-network generates dynamic weights for the outputs of the four stages of the Swin Transformer, enabling adaptive selection of features across scale levels. This method effectively suppresses redundant responses at non-critical scales, allowing the model to automatically focus on the feature levels that contribute the most to quality perception based on imaging conditions.

[0101] Multi-stage physical perception feature enhancement strategy By constructing a physical prior vector with field of view, depth, and saliency as its core, and injecting channel enhancement weights based on physical properties into multi-scale features, an adaptive perception of distorted visibility as field of view conditions is achieved. This is a lightweight and generalizable physical perception feature enhancement strategy.

[0102] Bionic "zoom sensing" collaborative mechanism: bidirectional linkage between sampling end and feature end A collaborative process for panoramic image quality assessment that simulates human visual behavior is proposed: Sampling end (similar to a lens): The field of view (FOV) changes with depth and saliency, achieving biomimetic zoom sampling.

[0103] Feature layer (similar to the visual cortex): The gating network controls the scope of attention at the backbone level based on the field of view (FOV), saliency, and depth information.

[0104] This "dual-end coupling" mechanism simulates the complete visual process of the human eye from focusing on local details to understanding the macroscopic world, making the evaluation of panoramic image quality more in line with human perception mechanisms.

[0105] Biomimetic lens sampling strategy based on physical properties This paper proposes a sampling method that constructs an inverse proportional function relationship between field of view (FOV), depth, and saliency. This method implements a dynamic scaling rule that "the smaller the depth and the higher the saliency, the smaller the FOV," simulating the zooming process of the human eye's lens. This strategy enables adaptive detail sampling during the viewport generation stage, allowing the model to naturally maintain high-resolution focus on near-field regions and salient targets.

[0106] Quality prediction mechanism based on focus grouping and dual-stream fusion A dual-path quality prediction architecture is proposed, which divides the viewport into a detail stream and a structure stream based on the field of view threshold and physical properties.

[0107] Detail flow focuses on local textures: enhancing the model's ability to perceive severe local distortions.

[0108] Structure flow focuses on global shape: avoiding the overall structure being misled by local noise.

[0109] Finally, by fusing the two stream results based on a weighted strategy of focus degree (generated by physical prior), the masking of local strong distortion regions by the traditional average pooling method is alleviated.

[0110] By simulating the physiological characteristics of the human eye—"seeing details up close and outlines from afar"—the evaluation algorithm becomes more sensitive to near-field texture distortion, which is crucial in VR experiences, and the evaluation results have a higher Pearson correlation coefficient (PLCC) with human subjective ratings (MOS).

[0111] The distribution of gating weights directly reflects the network's attention patterns to different content (e.g., close-up viewport weights are concentrated in shallow layers, while wide-angle viewport weights are concentrated in deep layers), making the decision-making process of deep learning models more consistent with human visual physiology. It also improves the ability to distinguish different types of distortion: through adaptive selection, the algorithm can accurately capture compressed noise (high frequency) using shallow features and accurately capture geometric stretching (low frequency) using deep features, significantly improving the accuracy of overall quality assessment.

[0112] Adaptive FOV sampling avoids overly high-resolution sampling of meaningless background areas (such as the sky and ground), concentrates computational resources on the visual areas that the human eye is most interested in, and improves inference efficiency.

[0113] Example 3: This embodiment provides, as follows: Figure 2 The panoramic image quality assessment system shown includes: Acquisition module: Acquires the panoramic image to be evaluated, and generates a panoramic depth map and a panoramic saliency map based on the panoramic image to be evaluated; Sampling viewport module: Multiple sampling points are obtained by sampling based on the panoramic saliency map. Based on the multiple sampling points, the panoramic saliency map, and the panoramic depth map, multiple sets of physical attributes are obtained, and a series of viewport images are obtained. Feature extraction module: Extracts features based on the physical properties of the viewport and the viewport image to obtain multi-scale features; Feature fusion module: Performs multi-scale fusion based on the physical properties of the viewport and multi-scale features to obtain fused features; Weighted fusion module: Based on the physical properties of the viewport and the fused features, it performs adaptive weighted fusion of quality levels and outputs the final quality prediction score to complete the quality evaluation.

[0114] This embodiment calculates the physical properties of the viewport and uses these properties for feature extraction, fusion, and adaptive fusion to achieve adaptive perception of distorted visibility as the field of view changes.

[0115] Example 4: This embodiment is implemented using the Python programming language and the PyTorch deep learning framework. The hardware environment is an NVIDIA GeForce RTX 4090 GPU.

[0116] Specific implementation steps: Step S1: Data Preparation and Preprocessing Obtain a panoramic image of the ERP system to be evaluated, and normalize the resolution to [resolution value missing]. .

[0117] Extract panoramic depth maps using a pre-trained UniFuse network. , and then normalize to the [0, 1] interval.

[0118] Panoramic saliency maps were extracted using a pre-trained SalNet360 network. Normalized to [0, 1].

[0119] Step S2: Run the sampling module Set the number of sampling viewports .

[0120] exist The above selects 20 local maximum points as centers. .

[0121] Set reference field of view adjustment coefficient .

[0122] For a point with a significance of 0.9 (very interesting) and a depth of 0.1 (very close), calculate its FOV:

[0123] (At this point, the field of view becomes smaller, which is equivalent to magnifying the image and focusing on the details.)

[0124] For a point with a saliency of 0.1 and a depth of 0.8 (distant background), calculate its FOV:

[0125] (At this point, the field of view widens, which is equivalent to wide-angle shooting, providing a panoramic view.)

[0126] Reproject and scale these regions to The pixel tensor is then input into the pre-trained SwinTransformer Tiny network.

[0127] Step S3: Swing Transformer Feature-Aware Enhancement Construct a pre-trained Swing Transformer Tiny network.

[0128] The prior information of each viewport and the index of the pre-enhancement layer are concatenated and input into an MLP network, which outputs a vector of dimension 128. Based on this vector, a channel-level attention weight is obtained to enhance the feature channel dimension of each layer, highlighting the most important information of that layer. The four stages of Swin are processed separately, and finally the output channels are unified to 128 enhanced features.

[0129] Step S4: Multi-level Feature Selection and Fusion For each viewport, its relevant physical attributes are first extracted, including the field of view size, average depth value, and average saliency value. These three physical attributes are combined into a vector in a fixed order, serving as a "physical prior vector" describing the imaging conditions of that viewport. This vector is used to express whether the viewport belongs to different states such as "magnified detail," "close-up observation," or "wide-angle overview," providing a basis for subsequent feature selection.

[0130] After the feature maps of the four stages of the backbone network are uniformly adjusted to the same spatial resolution, they are summed pixel by pixel to obtain a comprehensive feature. Then, a global averaging operation is performed on this comprehensive feature to obtain a global feature vector that reflects the overall semantic distribution. This global feature vector is concatenated with the previously constructed physical prior vector to obtain a joint representation containing both visual semantics and physical information. This joint representation is then input into a weight prediction module consisting of a two-layer fully connected network. This module outputs four weight values, corresponding to the feature importance of the four stages of the backbone network.

[0131] When the field of view is small (i.e., the viewport is in "zoom-in" mode), the weight of lower-level features is automatically increased to emphasize texture details. When the field of view is large (i.e., the viewport is in "wide-angle" mode), the weight of higher-level structural features increases accordingly.

[0132] Depth and saliency attributes jointly influence the model's emphasis on detailed and structural regions, making it more consistent with the human eye's attention mechanism at different viewing distances. These weights constitute a "dynamic gating signal." Based on the generated four weights, the features from the four stages are weighted and combined to obtain a comprehensive feature after hierarchical fusion. This feature contains both low-level detailed information and high-level semantic information, but the proportion is automatically determined by the dynamic gating mechanism of the previous stage.

[0133] To further enhance the physical interpretability of the model, the fused features are concatenated with the initial physical prior vectors to form the final viewport feature representation. This representation contains both visual image features and physical properties, and can be used for subsequent quality prediction or other tasks.

[0134] Step S5: Focus on Aggregation and Score Output A learnable network groups viewport features into "detail-aware group" and "structure-aware group" based on viewport prior information.

[0135] Simultaneously, the focus weight of each viewport is calculated using viewport features from both the detail-aware group and the structure-aware group, and the features of each viewport are weighted.

[0136] Finally, a multilayer perceptron with shared parameters is used to regress the detail perception score and the structure perception score. The detail perception score and the structure perception score are weighted according to the viewport prior information and then summed to obtain the final score.

Claims

1. A panoramic image quality evaluation method characterized by comprising: include: S1: Obtain the panoramic image to be evaluated, and generate a panoramic depth map and a panoramic saliency map based on the panoramic image to be evaluated. S2: Multiple sampling points are obtained by sampling based on the panoramic saliency map. Multiple physical attributes are obtained based on the multiple sampling points, the panoramic saliency map, and the panoramic depth map, and a series of viewport images are obtained. S3: Extract features based on the physical properties of the viewport and the viewport image to obtain multi-scale features; S4: Perform multi-scale fusion based on the physical properties of the viewport and multi-scale features to obtain the fused features; S5: Perform adaptive weighted fusion of quality levels based on the physical properties of the viewport and the fused features, and output the final quality prediction score to complete the quality evaluation.

2. The panoramic image quality evaluation method according to claim 1, characterized in that, In step S1, specifically: a monocular depth estimation network and a saliency detection network are constructed; a panoramic depth map is generated using the monocular depth estimation network, and a panoramic saliency map is generated using the saliency detection network.

3. The panoramic image quality evaluation method according to claim 1, characterized in that, In step S2, multiple sampling points are obtained by sampling based on the panoramic saliency map, specifically as follows: Non-maximum suppression (NMS) algorithm is applied on the panoramic saliency map, and N saliency peak points are selected as sampling points, and the coordinates of the sampling points are wherein .

4. The panoramic image quality evaluation method according to claim 2, characterized in that, In step S2, obtaining multiple sets of physical attributes and a series of viewport images based on multiple sampling points specifically involves using the sampling points as the viewport center. For each viewport center, according to the panorama saliency map and the panorama depth map, a normalized depth value corresponding to the viewport center is obtained and a normalized saliency value ; Field of view angle The calculation is as follows: : reference field angle; : indicates the saliency focusing effect; the higher the saliency, the smaller the size; : indicates the distance wide-angle effect; the greater the depth value, the greater; , is a tuning coefficient, the calculated within a reasonable range; According to the coordinates in the viewport and the calculated field of view angle , the spherical image is projected into a fixed resolution plane viewport image by bilinear interpolation .

5. The panoramic image quality evaluation method according to claim 1, characterized in that, In step S3, specifically: the Swin Transformer Tiny pre-trained on the ImageNet dataset is used as the backbone network for feature extraction.

6. The panoramic image quality evaluation method according to claim 4, characterized in that, In step S3, specifically: For each viewport, extract the field of view (FOV). Average normalized depth Significance of average normalization depth and pre-enhancement layer index And concatenate them into a physical vector: in This represents vector concatenation, where the vector is sequentially input into two fully connected layers and a non-linear activation function to model the non-linear relationship between physical properties and visual sensitivity. Based on the physical vector, four independent prior feature vectors are output: The four prior feature vectors correspond to the features of the four stages of the Swin Transformer. This is used to enhance the network's perception of prior physical information about the viewport; Perform sensory enhancement: Given stage characteristics First, perform global average pooling: This vector preserves the global semantic statistics of the features at this scale; Channel statistics Compared with the physical prior enhancement features obtained in the previous step By fusing, we obtain: in This represents vector concatenation. The concatenated vector is projected onto the input feature channel dimension using a fully connected layer. The fusion result is then normalized using Softmax to obtain the enhancement weights. The enhanced weights are then reinjected into the original features: in This indicates element-wise multiplication.

7. The panoramic image quality evaluation method according to claim 6, characterized in that, In step S4, specifically: For each viewport, first extract its field of view and average normalized depth. Significance of average normalization depth Concatenate them into a single physical vector: The four-stage features are added pixel by pixel: All features are upsampled to the highest resolution before being added together; Subsequently Perform global average pooling (GAP) to obtain a global feature vector: global feature vectors With physical prior vector Perform splicing and cascading: splicing vectors Input a two-layer MLP: in: Physical meaning: Smaller field of view → Increase the weight of lower-level features Larger field of view → Increase the weight of high-rise features Saliency and depth together modulate the scope of attention for texture / structural features; The generated weights are used to perform a weighted summation of the aligned feature vectors to obtain the final representation vector for the viewport. : By combining fusion features with physical priors: 。 8. The panoramic image quality evaluation method according to claim 7, characterized in that, In step S5, specifically: the fused features are divided into a detail-aware group and a structure-aware group according to the physical properties of the viewport; Prior information is obtained based on the physical properties of the viewport. The N fused features are divided into two groups, and a gating factor is formed by passing them through a two-layer fully connected network and a Sigmoid activation function: in Used to divide viewport features into: Detail-aware group: Suitable for areas with rich textures and prominent local details; Structure-aware groups: suitable for areas dominated by geometric structure and shape contours; Two sets of features passed: To achieve information-guided grouping based on physical priors; Adaptive weighted fusion is performed on the detail-aware group and the structure-aware group respectively.

9. A panoramic image quality evaluation method according to claim 8, characterized in that, The adaptive weighted fusion of the detail-aware group and the structure-aware group is performed separately as follows: Construct detail-aware branches, which employ a small receptive field structure with strong scale adaptability: in This represents a non-linear activation function; the final enhanced feature is the element-wise summation of two convolutional layers: Construct a structure-aware branch, which uses a 5×5 convolutional structure with a large receptive field: Similarly, merge element by element: The enhanced two types of features are then used to generate global representations through global pooling: The weights of the output branches of the two MLPs and Softmax with shared parameters are as follows: This is then used to multiply back into the two enhanced branches, achieving dynamic branch-level control: Detail-aware branches use global max pooling to highlight significant high-frequency regions: The structure-aware branch uses global average pooling to obtain an overall balanced representation: Using prior vectors Generate fusion weights: The final viewport quality score is: 。 10. A panoramic image quality evaluation system, characterized in that, include: Acquisition module: Acquires the panoramic image to be evaluated, and generates a panoramic depth map and a panoramic saliency map based on the panoramic image to be evaluated; Sampling viewport module: Multiple sampling points are obtained by sampling based on the panoramic saliency map. Based on the multiple sampling points, the panoramic saliency map, and the panoramic depth map, multiple sets of physical attributes are obtained, and a series of viewport images are obtained. Feature extraction module: Extracts features based on the physical properties of the viewport and the viewport image to obtain multi-scale features; Feature fusion module: Performs multi-scale fusion based on the physical properties of the viewport and multi-scale features to obtain fused features; Weighted fusion module: Based on the physical properties of the viewport and the fused features, it performs adaptive weighted fusion of quality levels and outputs the final quality prediction score to complete the quality evaluation.

Images