A method and system for no-reference panoramic video quality evaluation based on spatio-temporal feature fusion

Abstract

The disclosure provides a kind of based on spatiotemporal feature fusion's no reference panoramic video quality evaluation method and system, wherein based on spatiotemporal feature fusion's no reference panoramic video quality evaluation method, it includes: local space quality feature extraction processing is carried out to distorted panoramic video, determines the local space quality feature of distorted panoramic video;Distorted panoramic video is carried out time domain quality feature extraction processing, determines the time domain quality feature of distorted panoramic video;Distorted panoramic video is input into the preset spherical convolution network, determines the global space quality feature of distorted panoramic video;Local space quality feature, time domain quality feature and global space quality feature are carried out feature fusion processing, determine fusion feature;According to quality evaluation processing of fusion feature, determine the quality score of distorted panoramic video.The disclosure realizes the quality evaluation of multi-angle, multi-dimension to panoramic video, improves the accuracy of panoramic video quality evaluation task.

Description

Technical Field

[0001] This disclosure relates to the field of video quality assessment technology, and more specifically, to a method and system for assessing the quality of referenceless panoramic video based on spatiotemporal feature fusion. Background Technology

[0002] With the development of virtual reality (VR) technology, 360° panoramic videos have been widely used in entertainment, education, and healthcare. However, the viewing experience of panoramic videos is affected by compression and distortion introduced during transmission. Therefore, how to effectively evaluate the quality of panoramic videos has become an important research topic.

[0003] In full-reference quality assessment tasks, using a high-quality image / video as a reference and accurately evaluating the quality of distorted images / videos by calculating and learning the fidelity and quality degradation between the distorted image / video and the reference image / video is crucial. However, obtaining distortion-free reference images / videos is a challenging task in practical applications. Compared to full-reference quality assessment methods, non-reference quality assessment methods typically exhibit poorer effectiveness, stability, and consistency, heavily relying on feature extraction from distorted images / videos and being highly sensitive to content. Furthermore, many current quality assessment algorithms neglect the immersive, omnidirectional, and continuous characteristics of panoramic videos, making it difficult for existing methods to achieve high accuracy in panoramic video quality assessment tasks.

[0004] Therefore, the development of a no-reference panoramic video quality assessment algorithm is one of the important technical issues in this field. Summary of the Invention

[0005] To address the shortcomings of existing technologies, the purpose of this disclosure is to provide a method and system for evaluating the quality of no-reference panoramic video based on spatiotemporal feature fusion.

[0006] According to a first aspect of this disclosure, a method and system for evaluating the quality of referenceless panoramic video based on spatiotemporal feature fusion are provided, comprising:

[0007] The local spatial quality features of the distorted panoramic video are extracted and processed to determine the local spatial quality features of the distorted panoramic video.

[0008] Temporal quality feature extraction processing is performed on the distorted panoramic video to determine its temporal quality features;

[0009] The distorted panoramic video is input into a preset spherical convolutional network to determine the global spatial quality features of the distorted panoramic video.

[0010] The local spatial quality features, temporal quality features, and global spatial quality features of the distorted panoramic video are fused to determine the fused features of the distorted panoramic video.

[0011] The quality score of the distorted panoramic video is determined by performing quality evaluation processing based on the fusion characteristics of the distorted panoramic video.

[0012] Optionally, the step of extracting local spatial quality features from the distorted panoramic video to determine the local spatial quality features of the distorted panoramic video includes:

[0013] The saliency-guided viewport extraction process is performed on the distorted panoramic video to determine the optimal viewport of the distorted panoramic video.

[0014] The distorted panoramic video is subjected to quality enhancement processing to determine the pseudo-reference sequence of the distorted panoramic video;

[0015] The saliency-guided viewport extraction process is performed on the pseudo-reference sequence to determine the optimal viewport for the pseudo-reference sequence;

[0016] The optimal viewport of the distorted panoramic video and the optimal viewport of the pseudo-reference sequence are input into a preset attention-based feature extraction network to determine the local spatial quality features of the distorted panoramic video.

[0017] Optionally, the saliency-guided viewport extraction process for the distorted panoramic video, determining the optimal viewport for the distorted panoramic video, includes:

[0018] The distorted panoramic video is subjected to a first sampling process to determine a sequence of multiple first sampling consecutive frames;

[0019] The multiple first sampled consecutive frame sequences are input into a preset saliency prediction network to determine the saliency map of the video frames in an equal rectangular projection format;

[0020] Map the viewport center point on the saliency map of the video frame in the equirectangular projection format to the spherical domain, and determine the spherical domain coordinates of the viewport center point;

[0021] Based on the spherical domain coordinates of the viewport center point, a preset number of candidate viewport center points are determined;

[0022] Based on the preset number of candidate viewport center points, the candidate viewport center point with the largest significance value is determined as the optimal viewport for the distorted panoramic video.

[0023] Optionally, the step of performing quality enhancement processing on the distorted panoramic video and determining the pseudo-reference sequence of the distorted panoramic video includes:

[0024] Based on the distorted panoramic video, extract the continuous frame sequence corresponding to the distorted video frame in the distorted panoramic video. The continuous frame sequence corresponding to the distorted video frame includes the distorted video frame, T frames before the distorted video frame, and T frames after the distorted video frame.

[0025] The continuous frame sequence corresponding to the distorted video frame is input into a network based on the U-net architecture to determine the feature image of the continuous frame sequence corresponding to the distorted video frame.

[0026] The feature image of the continuous frame sequence corresponding to the distorted video frame is input into a deformable convolution to determine the offset field of each frame of the feature image;

[0027] The offset field of each frame of the feature image is input into a spatiotemporal deformable convolution to optimize the sampling offset and output a spatiotemporal feature image.

[0028] The spatiotemporal feature images are fused to determine the pseudo-reference sequence of the distorted panoramic video.

[0029] Optionally, the step of inputting the optimal viewport of the distorted panoramic video and the optimal viewport of the pseudo-reference sequence into a preset attention-based feature extraction network to determine the local spatial quality features of the distorted panoramic video includes:

[0030] Determine the viewport residual map based on the optimal viewport of the distorted panoramic video and the optimal viewport of the pseudo-reference sequence;

[0031] The optimal viewport and the viewport residual map of the distorted panoramic video are input into a preset quality evaluation network to determine the quality score of the distorted panoramic video.

[0032] The optimal viewport of the distorted panoramic video and the optimal viewport of the pseudo-reference sequence are downsampled to determine the feature images of the distorted panoramic video and the pseudo-reference sequence.

[0033] Spatial feature selection weights are assigned to the feature images of the distorted panoramic video and the feature images of the pseudo-reference sequence to determine the weighted spatial feature image of the distorted panoramic video and the weighted spatial feature image of the pseudo-reference sequence.

[0034] The weighted spatial feature image of the distorted panoramic video and the weighted spatial feature image of the pseudo-reference sequence are spliced ​​and concatenated along the channel dimension and optimized for feature extraction to determine the optimized feature image;

[0035] The local spatial quality features of the distorted panoramic video are determined by performing element-level multiplication on the viewport residual map and the optimized feature image.

[0036] Optionally, the step of extracting temporal quality features from the distorted panoramic video to determine its temporal quality features includes:

[0037] The distorted panoramic video is subjected to a second sampling process to determine the second sampled video frame sequence;

[0038] The second sampled video frame sequence is subjected to saliency-guided viewport extraction processing to determine the optimal viewport for each video frame of the second sampled video frame sequence.

[0039] The optimal viewport of each video frame in the second sampled video frame sequence is input into a preset temporal quality feature extraction network based on a convolutional neural network to determine the temporal quality features of the distorted panoramic video.

[0040] Optionally, the step of performing feature fusion processing on the local spatial quality features, temporal quality features, and global spatial quality features of the distorted panoramic video to determine the fused features of the distorted panoramic video includes:

[0041] The local spatial quality features, temporal quality features, and global spatial quality features of the distorted panoramic video are dimensionality-reduced using a global pooling layer and a fully connected layer to determine the local spatial quality features, temporal quality features, and global spatial quality features that are in the same dimension.

[0042] The local spatial quality features, temporal quality features, and global spatial quality features that are in the same dimension are cascaded to determine the fusion features of the distorted panoramic video.

[0043] According to a second aspect of this disclosure, a no-reference panoramic video quality assessment system based on spatiotemporal feature fusion is provided, comprising:

[0044] The local spatial quality feature extraction module is used to extract local spatial quality features from the distorted panoramic video and determine the local spatial quality features of the distorted panoramic video.

[0045] The temporal quality feature extraction module is used to perform temporal quality feature extraction processing on the distorted panoramic video to determine the temporal quality features of the distorted panoramic video.

[0046] A global spatial quality feature extraction module is used to input the distorted panoramic video into a preset spherical convolutional network to determine the global spatial quality features of the distorted panoramic video.

[0047] The feature fusion module is used to perform feature fusion processing on the local spatial quality features, the temporal quality features and the global spatial quality features of the distorted panoramic video to determine the fused features of the distorted panoramic video.

[0048] The quality evaluation module is used to perform quality evaluation processing based on the fusion characteristics of the distorted panoramic video and determine the quality score of the distorted panoramic video.

[0049] According to a third aspect of this disclosure, a non-transitory computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the steps of the method provided in the first aspect of this disclosure.

[0050] According to a fourth aspect of this disclosure, an electronic device is provided, comprising:

[0051] A memory on which computer programs are stored;

[0052] A processor for executing the computer program in the memory to implement the steps of the method provided in the first aspect of this disclosure.

[0053] Compared with the prior art, the embodiments disclosed herein have at least one of the following beneficial effects:

[0054] The above technical solution extracts local spatial quality features, temporal quality features, and global spatial quality features from distorted panoramic videos. These features are then fused, and the fused features are subjected to quality evaluation processing. This enables multi-angle and multi-dimensional quality evaluation of distorted panoramic videos, obtaining objective quality scores without the need for additional reference images, thus improving the accuracy of panoramic video quality evaluation tasks. Attached Figure Description

[0055] Other features, objects, and advantages of this disclosure will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0056] Figure 1 This is a flowchart illustrating a no-reference panoramic video quality assessment method based on spatiotemporal feature fusion, according to an exemplary embodiment.

[0057] Figure 2 This is a flowchart illustrating another method for evaluating the quality of non-reference panoramic video based on spatiotemporal feature fusion, according to an exemplary embodiment.

[0058] Figure 3 This is a schematic diagram illustrating a preset number of alternative viewport center points according to an exemplary embodiment.

[0059] Figure 4is a block diagram of a no-reference panoramic video quality assessment system based on spatiotemporal feature fusion, shown according to an exemplary embodiment. Detailed Description of the Embodiments

[0060] The present disclosure is described in detail below in conjunction with specific embodiments. The following embodiments help those skilled in the art to further understand the present disclosure, but do not limit the present disclosure in any form. It should be noted that several variations and improvements can be made by those of ordinary skill in the art without departing from the concept of the present disclosure. All of these fall within the protection scope of the present disclosure.

[0061] Figure 1 is a flowchart of a no-reference panoramic video quality assessment method based on spatiotemporal feature fusion, shown according to an exemplary embodiment. Figure 2 is a flowchart of another no-reference panoramic video quality assessment method based on spatiotemporal feature fusion, shown according to an exemplary embodiment.

[0062] As shown in Figure 1 , the present disclosure provides a no-reference panoramic video quality assessment method based on spatiotemporal feature fusion, comprising S11 to S15.

[0063] In S11, local spatial quality feature extraction processing is performed on the distorted panoramic video to determine the local spatial quality feature of the distorted panoramic video.

[0064] In a possible embodiment, S11 may include S21 to S24.

[0065] In S21, saliency-guided viewport extraction processing is performed on the distorted panoramic video to determine the optimal viewport of the distorted panoramic video.

[0066] First sampling processing is performed on the distorted panoramic video to determine a plurality of first sampled consecutive frame sequences.

[0067] Wherein, the first sampling processing refers to performing equal-interval sampling of 3 consecutive frames based on a first preset interval on the distorted panoramic video as the first sampled consecutive frame sequence, a total of n (0<n≤N–2) first sampled consecutive frame sequences are sampled, the total number of frames of the distorted panoramic video is N frames, and each first sampled consecutive frame sequence includes a center frame I t and the center frame I t adjacent frames {I t-1 , I t+1}.

[0068] The plurality of first sampled consecutive frame sequences are input into a preset saliency prediction network to determine the video frame saliency map in the equirectangular projection format.

[0069] Following the example above, input 3n frames of images into a preset saliency prediction network to generate a video frame saliency map in the format of equal rectangular projection, i.e., a video frame saliency map in ERP format.

[0070] Set both the horizontal and vertical field of view of each viewport on the video frame saliency map to 90°.

[0071] Map the viewport center point on the saliency map of the video frame in the equirectangular projection format to the spherical domain to determine the spherical domain coordinates of the viewport center point.

[0072] Following the example above, referring to the methods of cube mapping and sphere center mapping, the viewport center point in the saliency map of the ERP format is... Mapping to the spherical domain determines the spherical domain coordinates of the viewport center point. The conversion formula between the ERP domain and the spherical domain is as follows:

[0073]

[0074] Where φ represents the longitude of the spherical region, θ represents the latitude of the spherical region, u represents the x-coordinate in the rectangular projection format, v represents the y-coordinate in the rectangular projection format, W represents the length of the video frame in the rectangular projection format, and H represents the width of the video frame in the rectangular projection format.

[0075] The conversion formula between spherical coordinates (φ, θ) and tangent plane spatial coordinates Π0 is:

[0076]

[0077] Where x represents the abscissa of the spatial coordinates of the tangent plane, and y represents the ordinate of the spatial coordinates of the tangent plane. This represents the longitude of the spherical domain coordinates corresponding to the point of tangency of the tangent plane, that is, the longitude of the viewport center point in the spherical domain coordinates. This represents the latitude of the spherical domain coordinates corresponding to the tangent point of the tangent plane, that is, the latitude of the spherical domain coordinates corresponding to the center point of the viewport.

[0078] Figure 3 This is a schematic diagram illustrating a preset number of alternative viewport center points according to an exemplary embodiment.

[0079] Based on the spherical domain coordinates of the viewport center point, determine a preset number of alternative viewport center points;

[0080] Following the example above, such as Figure 3 As shown, 30 candidate viewport center points are selected on the saliency mapping sphere of the saliency map. The candidate viewport center points are evenly distributed at the equator, 30°N, 60°N, 30°S, 60°S, as well as the North and South Poles of the saliency mapping sphere.

[0081] Based on a preset number of candidate viewport center points, the candidate viewport center point with the largest significance value is determined as the optimal viewport for the distorted panoramic video.

[0082] Following the example above, the saliency value of the center point of each candidate viewport represents the sum of pixel values ​​in the saliency map corresponding to that viewport. The candidate viewport with the largest saliency value is denoted as Max{S1,S2,S3,…,S…}. 30 The candidate viewport with the largest significance value is selected as the optimal viewport for the distorted panoramic video. Wherein, S... i This represents the saliency value of the i-th candidate viewport center point, which is the sum of pixel values ​​in the saliency map corresponding to the i-th candidate viewport center point.

[0083] like Figure 2 As shown, as another example, the distorted 360° video frame sequence is input into the saliency prediction module to generate saliency frames, and then the saliency frames are input into the optimal viewport extraction module to select the optimal viewport for the distorted 360° video frame sequence.

[0084] S22, perform quality enhancement processing on the distorted panoramic video and determine the pseudo-reference sequence of the distorted panoramic video.

[0085] Extract the continuous frame sequence corresponding to the distorted video frames from the distorted panoramic video.

[0086] The consecutive frame sequence corresponding to the distorted video frame includes the distorted video frame, the T-frame before the distorted video frame, and the T-frame after the distorted video frame.

[0087] As an example, the sequence of consecutive frames corresponding to the distorted video frames is a sequence of consecutive frames with a length of 2T+1. in Indicates a distorted video frame. This represents the T-th video frame preceding the distorted video frame. This represents the T-th video frame following the distorted video frame.

[0088] The continuous frame sequence corresponding to the distorted video frame is input into a network based on the U-net architecture to determine the feature image of the continuous frame sequence corresponding to the distorted video frame.

[0089] Among them, the network based on the U-net architecture is an encoder-decoder architecture and a network with skip connections.

[0090] The feature image of the continuous frame sequence corresponding to the distorted video frame is input into a deformable convolution to determine the offset field of each frame of the feature image.

[0091] Deformable convolution captures motion information between frames by determining the offset field of each frame of the feature image.

[0092] The offset field of each frame of the feature image is input into a spatiotemporally deformable convolution to optimize the sampling offset and output a spatiotemporal feature image.

[0093] Spatiotemporal feature images contain spatiotemporal deformation information of video frame sequences, used to represent the motion and changes between video frames.

[0094] Spatiotemporally deformable convolution uses learnable offsets to optimize traditional regular sampling offsets. The offset optimization formula is expressed as:

[0095] p k ←p k +δ (t,p),k

[0096] Where, p k δ represents the k-th sampling position of the convolution kernel on the input feature image. (t,p),k This represents the offset assigned to the k-th convolutional window centered at a specific spatiotemporal location (t, p).

[0097] The spatiotemporal feature images are fused to determine the pseudo-reference sequence of the distorted panoramic video.

[0098] Specifically, the spatiotemporal feature map is used to mine and learn complementary information in the features to form a fusion feature map F. The fusion feature map F includes spatiotemporal information between different frames, which is a feature map containing complete spatiotemporal dynamic characteristics, thereby generating an enhanced target frame and outputting a pseudo-reference sequence of distorted panoramic video.

[0099] The enhanced target frame, i.e., the pseudo-reference sequence of the distorted panoramic video, is generated using the following formula:

[0100]

[0101] in, Indicates the enhanced target frame. This represents the enhanced residual predicted through nonlinear mapping. This represents the distorted video frame in a distorted panoramic video.

[0102] like Figure 2 As shown, as an example, a distorted 360° video frame sequence is input as a reference sequence generation module, and a pseudo-reference sequence is generated through quality enhancement.

[0103] S23, perform saliency-guided viewport extraction processing on the pseudo-reference sequence to determine the optimal viewport for the pseudo-reference sequence.

[0104] The method for saliency-guided viewport extraction of the pseudo-reference sequence is the same as the method for saliency-guided viewport extraction of the distorted panoramic video in step S21 of this disclosure. The optimal viewport of the pseudo-reference sequence can be determined by referring to step S21, and will not be repeated here.

[0105] S24, input the optimal viewport of the distorted panoramic video and the optimal viewport of the pseudo-reference sequence into a preset attention-based feature extraction network to determine the local spatial quality features of the distorted panoramic video.

[0106] Determine the viewport residual map based on the optimal viewport of the distorted panoramic video and the optimal viewport of the pseudo-reference sequence.

[0107] The formula for determining the viewport residual plot is as follows:

[0108]

[0109] in, Represents the viewport residual diagram. The best viewport for representing the pseudo-reference sequence, The optimal viewport for representing distorted panoramic video.

[0110] For the pixels corresponding to the best viewport of the distorted panoramic video and the best viewport of the pseudo-reference sequence, bilinear interpolation is used for alignment, and the residual value between the best viewport of the distorted panoramic video and the best viewport of the pseudo-reference sequence is normalized to [0,1].

[0111] The optimal viewport and viewport residual map of the distorted panoramic video are input into a preset quality evaluation network to determine the quality score of the distorted panoramic video.

[0112] The quality score of distorted panoramic video is used to reflect the local spatial quality characteristics of distorted panoramic video.

[0113] Downsampling is performed on the optimal viewport of the distorted panoramic video and the optimal viewport of the pseudo-reference sequence to determine the feature images of the distorted panoramic video and the pseudo-reference sequence.

[0114] Among them, the feature images of the distorted panoramic video and the feature images of the pseudo-reference sequence are low-resolution feature images, which retain the main feature information of the original video frames while reducing the amount of data and reducing computational complexity.

[0115] As an example, two 3D convolutional layers can be used to downsample the optimal viewport of the distorted panoramic video and the optimal viewport of the pseudo-reference sequence for spatiotemporal feature learning.

[0116] Spatial feature selection weights are assigned to the feature images of the distorted panoramic video and the feature images of the pseudo-reference sequence to determine the weighted spatial feature image of the distorted panoramic video and the weighted spatial feature image of the pseudo-reference sequence.

[0117] Among them, a spatial attention module (SA) can be used to assign spatial feature selection weights to the feature images of the distorted panoramic video and the feature images of the pseudo-reference sequence, highlighting important spatial information for subsequent feature extraction optimization, improving the network's ability to learn spatial information, and enabling the network to focus on key regions.

[0118] The spatial attention module is represented as:

[0119]

[0120] Where F represents the feature image after downsampling by the 3D convolutional layer, F' represents the weighted spatial feature image after weighting by the spatial attention module, and M... s (F) represents the spatial feature selection weights output by the spatial attention module, σ represents the sigmoid function, and f 7×7 This indicates that a 7×7 filter is performing a convolution operation. Indicates the dot product method. Indicates the cascading method.

[0121] The weighted spatial feature image of the distorted panoramic video and the weighted spatial feature image of the pseudo-reference sequence are spliced ​​and concatenated along the channel dimension and optimized for feature extraction to determine the optimized feature image.

[0122] In this process, after concatenating two feature maps along the channel dimension, four 3D convolutional layers and corresponding channel attention (CA) modules are introduced to optimize feature extraction.

[0123] The channel attention module is represented as:

[0124]

[0125] M c (F c )=σ(ML(AvgPool(F c ))+ML(MaxPool(F c )))

[0126] Where F” represents the optimized feature image, that is, the feature image after being weighted by the channel attention module. c M represents the concatenated feature image. c (F c) represents the channel feature selection weights output by the channel attention module, and ML represents the multilayer perceptron.

[0127] The extracted optimized features include optimized temporal and spatial features, which can accurately reflect the quality information of video frames.

[0128] The multilayer perceptron (ML) disclosed herein includes two convolutional layers and one ReLU layer.

[0129] The local spatial quality features of the distorted panoramic video are determined by performing element-level multiplication on the viewport residual map and the optimized feature image.

[0130] The degree of perceptual distortion is quantified by performing element-wise multiplication of the viewport residual map with the optimized feature image.

[0131] The above technical solution can extract local spatial quality features of distorted panoramic videos to reflect the local spatial quality of the video and quantify the quality differences between distorted video frames.

[0132] like Figure 2 As shown, as an example, local spatial quality features are determined based on the optimal viewport and pseudo-reference sequence of a distorted 360° video frame sequence.

[0133] S12, perform temporal quality feature extraction processing on the distorted panoramic video to determine the temporal quality features of the distorted panoramic video.

[0134] In one possible embodiment, S12 may include S31 to S33.

[0135] S21, perform a second sampling process on the distorted panoramic video to determine the second sampled video frame sequence.

[0136] The second sampling process involves sampling 8 frames of the distorted panoramic video at equal intervals based on a second preset interval as the center frame, and extracting the 8 frames before and after each center frame to form a 64-frame second sampling video frame sequence.

[0137] S22, perform saliency-guided viewport extraction processing on the second sampled video frame sequence to determine the optimal viewport for each video frame of the second sampled video frame sequence.

[0138] The method for saliency-guided viewport extraction processing of the second sampled video frame sequence is the same as the method for saliency-guided viewport extraction processing of the distorted panoramic video in step S21 of this disclosure. The optimal viewport for each video frame of the second sampled video frame sequence can be determined by referring to step S21, and will not be repeated here.

[0139] S23, input the best viewport of each video frame in the second sampled video frame sequence into a preset temporal quality feature extraction network based on a convolutional neural network to determine the temporal quality features of the distorted panoramic video.

[0140] The preset convolutional neural network-based time quality feature extraction network is similar in structure to the Fast-path network. The backbone network of the convolutional neural network-based time quality feature extraction network disclosed herein adopts ResNet-50 and is pre-trained using the ImageNet dataset.

[0141] like Figure 2 As shown, as an example, a distorted 360° video frame sequence is input into the temporal quality feature extraction module to determine the high frame rate sampling to generate the viewport sequence. Then, a Fast-path network or a ResNet-50 network is used to capture fast motion information to determine the temporal feature quality of the distorted panoramic video.

[0142] S13, input the distorted panoramic video into a preset spherical convolutional network to determine the global spatial quality features of the distorted panoramic video.

[0143] The preset spherical convolutional network includes four spherical convolutional blocks, each of which includes a spherical convolutional layer, a batch normalization layer, and a spherical pooling layer.

[0144] The sampling grid position of the spherical convolution kernel of the spherical convolution block will be distorted, just like the tangent plane of a sphere is mapped to the ERP format at different latitudes. The changes of the spherical convolution kernel are obtained by using sphere center mapping and inverse sphere center mapping.

[0145] The sphere center mapping formula, namely the conversion formula between the ERP domain and the spherical domain mentioned above:

[0146]

[0147] Where φ represents the longitude of the spherical region, θ represents the latitude of the spherical region, u represents the x-coordinate in the rectangular projection format, v represents the y-coordinate in the rectangular projection format, W represents the length of the video frame in the rectangular projection format, and H represents the width of the video frame in the rectangular projection format.

[0148] Inverse sphere center mapping formula:

[0149]

[0150] in,

[0151] Where x represents the abscissa of the spatial coordinates of the tangent plane, and y represents the ordinate of the spatial coordinates of the tangent plane. This represents the longitude of the spherical domain coordinates corresponding to the point of tangency of the tangent plane. φ represents the latitude of the spherical domain coordinates corresponding to the tangent point of the tangent plane, φ represents the longitude of the spherical domain coordinates corresponding to the spatial coordinates of the tangent plane, and θ represents the latitude of the spherical domain coordinates corresponding to the spatial coordinates of the tangent plane.

[0152] The inverse center mapping represents the process of mapping the points of the tangent plane of the sphere back to the sphere.

[0153] The preset spherical convolutional network outputs global spatial quality features of the distorted panoramic video.

[0154] like Figure 2 As shown, as another example, a sequence of distorted 360° video frames is input into a global spatial quality feature extraction module, which outputs distorted video frames. The distorted video frames are then input into a spherical convolutional network to extract global spatial information and determine the global spatial quality features of the distorted 360°.

[0155] S14. The local spatial quality features, temporal quality features and global spatial quality features of the distorted panoramic video are fused to determine the fusion features of the distorted panoramic video.

[0156] In one possible embodiment, S14 may include S41 to S42.

[0157] S41 uses a global pooling layer and a fully connected layer to perform dimensionality reduction on the local spatial quality features, temporal quality features, and global spatial quality features of the distorted panoramic video, thereby determining the local spatial quality features, temporal quality features, and global spatial quality features that are in the same dimension.

[0158] The fully connected layer is composed of linear layers.

[0159] S42 cascades local spatial quality features, temporal quality features, and global spatial quality features in the same dimension to determine the fusion features of the distorted panoramic video.

[0160] S15. Based on the fusion characteristics of the distorted panoramic video, perform quality evaluation processing to determine the quality score of the distorted panoramic video.

[0161] Following the example above, the fusion features of the distorted panoramic video are input into a preset quality evaluation network, which outputs the quality score of the distorted panoramic video. The specific formula is as follows:

[0162]

[0163] Where q represents the quality score of the distorted panoramic video, and F local F represents the local spatial quality characteristic. glob F represents the global spatial quality characteristic. fast This represents the time-domain quality characteristics.

[0164] Its preset quality assessment network loss is:

[0165]

[0166] Where K represents the total number of distorted panoramic videos in the training set, x n Let y represent the nth input video. n Let δ represent the subjective score corresponding to the nth input video, λ represent the training parameters, λ represent the hyperparameters, and L2 represent the regularization term.

[0167] like Figure 2 As shown, as another example, the local spatial quality features, temporal quality features, and global spatial quality features of the distorted 360° video frame sequence are fused, and then dimensionality reduction and fusion are performed through a global pooling layer and a fully connected layer to obtain the objective quality score of the distorted 360° video frame sequence.

[0168] The above technical solution extracts local spatial quality features, temporal quality features, and global spatial quality features from distorted panoramic videos. These features are then fused, and the fused features are subjected to quality evaluation processing. This enables multi-angle and multi-dimensional quality evaluation of distorted panoramic videos, obtaining objective quality scores without the need for additional reference images, thus improving the accuracy of panoramic video quality evaluation tasks.

[0169] In one possible embodiment, the no-reference panoramic video quality assessment method based on spatiotemporal feature fusion provided in this disclosure was compared and analyzed with other methods S-PSNR, WS-PSNR, CPP-PSNR, V-MEON, TLVQM, SimpleVQA, MC360IQA, VGCN, NR-OVQA, and ProVQA on the ODV-VQA dataset, as shown in Table 1:

[0170]

[0171] Table 1

[0172] As shown in Table 1, the no-reference panoramic video quality assessment method based on spatiotemporal feature fusion proposed in this disclosure achieves the best results in the evaluation metrics PLCC, SROCC (Spearman Rank Order Correlation Coefficient), and RMSE (Root Mean Square Error). Compared with the ProVQA method, this disclosure achieves performance gains of 0.59%, 0.57%, and 8.7% in PLCC, SROCC, and RMSE, respectively. In terms of the KROCC (Kendall Rank Order Correlation Coefficient), the no-reference panoramic video quality assessment method proposed in this disclosure is second only to the ProVQA method.

[0173] Therefore, the referenceless panoramic video quality assessment method based on spatiotemporal feature fusion proposed in this disclosure performs excellently in panoramic video quality assessment tasks, achieving remarkable results while considering temporal information and mapping distortion.

[0174] Figure 4 This is a block diagram illustrating a no-reference panoramic video quality assessment system based on spatiotemporal feature fusion, according to an exemplary embodiment.

[0175] Based on the same concept, this disclosure also provides a referenceless panoramic video quality assessment system based on spatiotemporal feature fusion, such as... Figure 4 As shown, it includes: a local spatial quality feature extraction module 110, a temporal quality feature extraction module 120, a global spatial quality feature extraction module 130, a feature fusion module 140, and a quality evaluation module 150.

[0176] The local spatial quality feature extraction module 110 is used to perform local spatial quality feature extraction processing on the distorted panoramic video to determine the local spatial quality features of the distorted panoramic video.

[0177] The temporal quality feature extraction module 120 is used to perform temporal quality feature extraction processing on the distorted panoramic video to determine the temporal quality features of the distorted panoramic video.

[0178] The global spatial quality feature extraction module 130 is used to input the distorted panoramic video into a preset spherical convolutional network to determine the global spatial quality features of the distorted panoramic video.

[0179] The feature fusion module 140 is used to perform feature fusion processing on the local spatial quality features, the temporal quality features and the global spatial quality features of the distorted panoramic video to determine the fused features of the distorted panoramic video.

[0180] The quality evaluation module 150 is used to perform quality evaluation processing based on the fusion characteristics of the distorted panoramic video and determine the quality score of the distorted panoramic video.

[0181] The above technical solution extracts local spatial quality features, temporal quality features, and global spatial quality features from distorted panoramic videos. These features are then fused, and the fused features are subjected to quality evaluation processing. This enables multi-angle and multi-dimensional quality evaluation of distorted panoramic videos, obtaining objective quality scores without the need for additional reference images, thus improving the accuracy of panoramic video quality evaluation tasks.

[0182] Regarding the embodiments of the above system, the specific ways in which each module performs operations have been described in detail in the embodiments of the method, and will not be elaborated here.

[0183] Based on the same concept described above, in another embodiment of this disclosure, an electronic device is also provided, including a memory, a processor, and a computer program stored in the memory and capable of running on the processor, wherein the processor executes the program to perform a no-reference panoramic video quality evaluation method based on spatiotemporal feature fusion.

[0184] Optionally, the memory is used to store programs; the memory may include volatile memory, such as random-access memory (RAM), such as static random-access memory (SRAM), double data rate synchronous dynamic random-access memory (DDRSDRAM), etc.; the memory may also include non-volatile memory, such as flash memory. The memory is used to store computer programs (such as application programs, functional modules, etc. that implement the above methods), computer instructions, etc., and the aforementioned computer programs, computer instructions, etc., can be partitioned and stored in one or more memories. Furthermore, the aforementioned computer programs, computer instructions, data, etc., can be accessed by the processor.

[0185] The aforementioned computer programs, computer instructions, etc., can be stored in partitions within one or more memory locations. Furthermore, the aforementioned computer programs, computer instructions, data, etc., can be accessed by a processor.

[0186] A processor is used to execute a computer program stored in memory to implement the various steps of the methods involved in the above embodiments. For details, please refer to the relevant descriptions in the preceding method embodiments.

[0187] The processor and memory can be separate structures or integrated structures. When the processor and memory are separate structures, they can be coupled together via a bus.

[0188] In this embodiment of the disclosure, a non-transitory computer-readable storage medium is also provided, on which a computer program is stored. When the computer program is executed by a processor, it implements the steps of a no-reference panoramic video quality evaluation method based on spatiotemporal feature fusion in any of the above embodiments.

[0189] Those skilled in the art will understand that embodiments of this disclosure can be provided as methods, systems, or computer program products. Therefore, this disclosure can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this disclosure can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0190] This disclosure is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0191] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0192] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0193] Although preferred embodiments of this disclosure have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this disclosure.

[0194] Obviously, those skilled in the art can make various modifications and variations to this disclosure without departing from its spirit and scope. Therefore, if such modifications and variations fall within the scope of the claims of this disclosure and their equivalents, this disclosure is also intended to include such modifications and variations.

Claims

1. A method for evaluating the quality of referenceless panoramic video based on spatiotemporal feature fusion, characterized in that, include: The local spatial quality features of the distorted panoramic video are extracted and processed to determine the local spatial quality features of the distorted panoramic video. Temporal quality feature extraction processing is performed on the distorted panoramic video to determine its temporal quality features; The distorted panoramic video is input into a preset spherical convolutional network to determine the global spatial quality features of the distorted panoramic video. The local spatial quality features, temporal quality features, and global spatial quality features of the distorted panoramic video are fused to determine the fused features of the distorted panoramic video. The quality score of the distorted panoramic video is determined by performing quality evaluation processing based on the fusion characteristics of the distorted panoramic video. The step of extracting local spatial quality features from the distorted panoramic video to determine its local spatial quality features includes: The saliency-guided viewport extraction process is performed on the distorted panoramic video to determine the optimal viewport of the distorted panoramic video. The distorted panoramic video is subjected to quality enhancement processing to determine the pseudo-reference sequence of the distorted panoramic video; The saliency-guided viewport extraction process is performed on the pseudo-reference sequence to determine the optimal viewport for the pseudo-reference sequence; The optimal viewport of the distorted panoramic video and the optimal viewport of the pseudo-reference sequence are input into a preset attention-based feature extraction network to determine the local spatial quality features of the distorted panoramic video. The saliency-guided viewport extraction process for the distorted panoramic video, determining the optimal viewport for the distorted panoramic video, includes: The distorted panoramic video is subjected to a first sampling process to determine a sequence of multiple first sampling consecutive frames; The multiple first sampled consecutive frame sequences are input into a preset saliency prediction network to determine the saliency map of the video frames in an equal rectangular projection format; Map the viewport center point on the saliency map of the video frame in the equirectangular projection format to the spherical domain, and determine the spherical domain coordinates of the viewport center point; Based on the spherical domain coordinates of the viewport center point, a preset number of candidate viewport center points are determined; Based on the preset number of candidate viewport center points, the candidate viewport center point with the largest significance value is determined as the optimal viewport for the distorted panoramic video.

2. The method according to claim 1, characterized in that, The process of performing quality enhancement processing on the distorted panoramic video and determining the pseudo-reference sequence of the distorted panoramic video includes: Based on the distorted panoramic video, extract the continuous frame sequence corresponding to the distorted video frame in the distorted panoramic video. The continuous frame sequence corresponding to the distorted video frame includes the distorted video frame, T frames before the distorted video frame, and T frames after the distorted video frame. The continuous frame sequence corresponding to the distorted video frame is input into a network based on the U-net architecture to determine the feature image of the continuous frame sequence corresponding to the distorted video frame. The feature image of the continuous frame sequence corresponding to the distorted video frame is input into a deformable convolution to determine the offset field of each frame of the feature image; The offset field of each frame of the feature image is input into a spatiotemporal deformable convolution to optimize the sampling offset and output a spatiotemporal feature image. The spatiotemporal feature images are fused to determine the pseudo-reference sequence of the distorted panoramic video.

3. The method according to claim 1, characterized in that, The step of inputting the optimal viewport of the distorted panoramic video and the optimal viewport of the pseudo-reference sequence into a preset attention-based feature extraction network to determine the local spatial quality features of the distorted panoramic video includes: Determine the viewport residual map based on the optimal viewport of the distorted panoramic video and the optimal viewport of the pseudo-reference sequence; The optimal viewport and the viewport residual map of the distorted panoramic video are input into a preset quality evaluation network to determine the quality score of the distorted panoramic video. The optimal viewport of the distorted panoramic video and the optimal viewport of the pseudo-reference sequence are downsampled to determine the feature images of the distorted panoramic video and the pseudo-reference sequence. Spatial feature selection weights are assigned to the feature images of the distorted panoramic video and the feature images of the pseudo-reference sequence to determine the weighted spatial feature image of the distorted panoramic video and the weighted spatial feature image of the pseudo-reference sequence. The weighted spatial feature image of the distorted panoramic video and the weighted spatial feature image of the pseudo-reference sequence are spliced ​​and concatenated along the channel dimension and optimized for feature extraction to determine the optimized feature image; The local spatial quality features of the distorted panoramic video are determined by performing element-level multiplication on the viewport residual map and the optimized feature image.

4. The method according to claim 1, characterized in that, The step of extracting temporal quality features from the distorted panoramic video to determine its temporal quality features includes: The distorted panoramic video is subjected to a second sampling process to determine the second sampled video frame sequence; The second sampled video frame sequence is subjected to saliency-guided viewport extraction processing to determine the optimal viewport for each video frame of the second sampled video frame sequence. The optimal viewport of each video frame in the second sampled video frame sequence is input into a preset temporal quality feature extraction network based on a convolutional neural network to determine the temporal quality features of the distorted panoramic video.

5. The method according to claim 1, characterized in that, The step of performing feature fusion processing on the local spatial quality features, temporal quality features, and global spatial quality features of the distorted panoramic video to determine the fused features of the distorted panoramic video includes: The local spatial quality features, temporal quality features, and global spatial quality features of the distorted panoramic video are dimensionality-reduced using a global pooling layer and a fully connected layer to determine the local spatial quality features, temporal quality features, and global spatial quality features that are in the same dimension. The local spatial quality features, temporal quality features, and global spatial quality features that are in the same dimension are cascaded to determine the fusion features of the distorted panoramic video.

6. A referenceless panoramic video quality assessment system based on spatiotemporal feature fusion, used to implement the referenceless panoramic video quality assessment method based on spatiotemporal feature fusion as described in any one of claims 1-5, characterized in that, include: The local spatial quality feature extraction module is used to extract local spatial quality features from the distorted panoramic video and determine the local spatial quality features of the distorted panoramic video. The temporal quality feature extraction module is used to perform temporal quality feature extraction processing on the distorted panoramic video to determine the temporal quality features of the distorted panoramic video. A global spatial quality feature extraction module is used to input the distorted panoramic video into a preset spherical convolutional network to determine the global spatial quality features of the distorted panoramic video. The feature fusion module is used to perform feature fusion processing on the local spatial quality features, the temporal quality features and the global spatial quality features of the distorted panoramic video to determine the fused features of the distorted panoramic video. The quality evaluation module is used to perform quality evaluation processing based on the fusion characteristics of the distorted panoramic video and determine the quality score of the distorted panoramic video.

7. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the steps of the method described in any one of claims 1-5.

8. An electronic device, characterized in that, include: A memory on which computer programs are stored; A processor for executing the computer program in the memory to implement the steps of the method according to any one of claims 1-5.

Images