An objective quality evaluation method for windowed six-degree-of-freedom composite videos

Abstract

The present invention relates to the technical field of video quality evaluation, and specifically to an objective quality evaluation method for a windowed six-degree-of-freedom composite video, comprising: obtaining a windowed six-degree-of-freedom composite video to be evaluated as a target composite video; decomposing the target composite video into a plurality of time-space domains; respectively evaluating the rendering distortion, viewpoint path inappropriateness, and compression distortion of the target composite video in the corresponding time-space domains; performing weighted calculations on the rendering distortion, viewpoint path inappropriateness, and compression distortion of the target composite video to obtain a corresponding objective quality evaluation score. The present invention comprehensively considers various distortions of a windowed six-degree-of-freedom video, and can effectively evaluate the visual discomfort caused by multi-dimensional viewpoint switching of a windowed six-degree-of-freedom video, thereby improving the comprehensiveness and effect of the objective quality evaluation of the windowed six-degree-of-freedom composite video, and maintaining a high degree of consistency with the subjective perception quality of the human eye.

Description

Technical Field

[0001] The present invention relates to the technical field of video quality evaluation, and in particular to an objective quality evaluation method for a window six-degree-of-freedom synthetic video. Background Art

[0002] Free viewpoint video allows users to freely switch viewpoints in the horizontal direction, improving the scene experience. With the rapid development of software and hardware technologies and users' pursuit of higher viewing experience, immersive video has emerged. The Motion Picture Expert Group has formulated immersive video standards and is committed to promoting the development of immersive video from three degrees of freedom to six degrees of freedom. Among them, windowed six-degree-of-freedom video allows users to pan the viewpoint in the horizontal and vertical directions and change the viewing angle in the front and back directions, providing users with a freer and more novel immersive visual experience, which is the future direction of the development of immersive video technology.

[0003] In order to provide users with a comfortable and smooth viewing experience of seamless viewpoint switching, immersive video systems need to use viewpoint color plus depth image based rendering (DIBR) technology to synthesize virtual viewpoints to achieve the coherence of user interactive browsing scenes. However, DIBR introduces geometric distortion, namely rendering distortion. Unlike the global distortion of ordinary videos, rendering distortion will cause local flickering during video playback, which seriously affects the perceived quality. Therefore, it is necessary to design a quality evaluation method for windowed 6-DOF synthetic videos to feedback and optimize each link of the windowed 6-DOF video system and promote its promotion and application in various fields. Existing quality evaluation methods are mainly divided into two categories: full reference (FR) and no reference (NR). The advantage of the FR evaluation model is high performance, but the distortion-free reference video of the windowed 6-DOF synthetic video often does not exist. Therefore, the quality evaluation of the windowed 6-DOF synthetic video belongs to NR quality evaluation.

[0004] In recent years, many scholars have proposed corresponding objective quality evaluation models for rendering distortion in free viewpoint video systems. For example, Peng Zongju et al. disclosed a free viewpoint video quality evaluation method in the patent with publication number CN109218710B, which can effectively evaluate the quality of free viewpoint videos. However, the applicant found that the existing method is an image-based evaluation method, that is, the average value of the image quality scores of each frame in the video is used as the evaluation score of the entire video, so it is impossible to measure the impact of rendering distortion on quality. At the same time, since the existing method is proposed for the free viewpoint video system, it can only switch the viewpoint in the horizontal direction, but cannot translate the viewpoint in the vertical direction or change the viewing angle in the front and back directions, which makes it difficult to evaluate the visual discomfort caused by multi-dimensional viewpoint switching in the window six-degree-of-freedom synthetic video. In addition, the existing other virtual viewpoint NR objective quality evaluation only considers the impact of rendering distortion, and does not consider the impact of the fusion of traditional distortion, rendering distortion, etc. on video quality, resulting in the objective quality evaluation results being not comprehensive enough. Therefore, how to design an objective quality evaluation method that fully considers various distortions of windowed six-degree-of-freedom videos and can effectively evaluate the visual discomfort caused by multi-dimensional viewpoint switching is a technical problem that needs to be solved urgently. Summary of the invention

[0005] In view of the deficiencies of the above-mentioned prior art, the technical problem to be solved by the present invention is: how to provide an objective quality evaluation method for a windowed six-degree-of-freedom composite video, so as to comprehensively consider various distortions of the windowed six-degree-of-freedom video, and effectively evaluate the visual discomfort caused by multi-dimensional viewpoint switching of the windowed six-degree-of-freedom video, thereby improving the comprehensiveness and effectiveness of the objective quality evaluation of the windowed six-degree-of-freedom composite video, and maintaining a high degree of consistency with the subjective perception quality of the human eye.

[0006] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0007] An objective quality evaluation method for a window six-degree-of-freedom composite video comprises the following steps:

[0008] S1: Obtain a six-degree-of-freedom synthetic video of the window to be evaluated as a target synthetic video;

[0009] S2: Decompose the target synthetic video into several spatiotemporal domains;

[0010] S3: Evaluate the rendering distortion, viewpoint path inappropriateness and compression distortion of the target synthetic video in the corresponding spatiotemporal domains respectively;

[0011] S4: Perform weighted calculation on the rendering distortion, viewpoint path inappropriateness and compression distortion of the target synthetic video to obtain the corresponding objective quality evaluation score.

[0012] Preferably, in step S2, the target synthetic video is first converted into a three-dimensional array V consisting of width W, height H and number of frames T, where the indexes of width W, height H and number of frames T are x, y and t respectively; then, based on different combinations of x, y and t, three corresponding spatiotemporal domains are formed: XY-T domain, XT-Y domain and TY-X domain;

[0013] Among them, the three-dimensional array V in the XY-T domain T It includes T images of size W×H; a three-dimensional array V in the XT-Y domain Y It consists of H images of size W×T; a three-dimensional array V in the TY-X domain X Contains W images of size T×H.

[0014] Preferably, in step S3, high-frequency information evaluation, edge clarity evaluation, brightness change feature evaluation and semantic integrity evaluation are performed in the XT-Y domain and the TY-X domain, and the rendering distortion of the target synthetic video is calculated based on the evaluation results.

[0015] Preferably, in step S3, before calculating the rendering distortion, firstly, the depth map E of the interval frame is estimated by the two-frame optical flow method; then, the maximum intensity values ​​of the depth map E in the x direction and the y direction are calculated, and used as the decomposition fixed points in the TY-X domain and the XT-Y domain respectively; finally, the image set Ω1 in the TY-X domain and the image set Ω2 in the XT-Y domain are decomposed to obtain; the number of images in the image set Ω1 is W k , the number of images in the image set Ω2 is H k ;

[0016]

[0017] Where: I1(·) and I k (·) represents the first image and the kth image in the XY-T domain; h (·) and l (·) represents the horizontal and vertical components of the optical flow field; x=0,1,...W, y=0,1,...H; η=5 represents the regularization parameter; ρ D and ρ S represents the penalty function; N x,y represents the 5×5 neighboring pixels around (x, y); η N =1 indicates the weight parameter.

[0018] Preferably, step S3 specifically includes the following steps:

[0019] S301: First, the three-dimensional array V in the TY-X domain XConvert to grayscale video X, perform two-dimensional discrete Fourier transform on each image in X, and calculate the amplitude spectrum of each coordinate of the image; then calculate the amplitude spectrum average of the corresponding image based on the amplitude spectrum of each coordinate of the image; finally calculate the high-frequency information in the TY-X domain based on the amplitude spectrum average of each image;

[0020] A(u,v;x)=α×ln|F(u,v;x)|;

[0021] Where: A(u,v;x) represents the amplitude spectrum of the x-th image at the (u,v) coordinate in the TY-X domain; F(u,v;x) represents the Fourier transform value of the x-th image at the (u,v) coordinate in the TY-X domain; α = 20;

[0022]

[0023] Where: represents the average amplitude spectrum of the xth image in the TY-X domain;

[0024]

[0025] Where: Represents high-frequency information in the TY-X domain; Ω1 and W k Represents the image set and number of images in the TY-X domain;

[0026] S302: First, the (n+m)-order Zernike moment of each image in the TY-X domain is calculated; then, the corresponding reconstructed image is obtained by inverse transformation of the Zernike moment of the image; then, the structural similarity between the original grayscale image and the reconstructed image is calculated to obtain the edge clarity of the corresponding image; finally, the edge clarity in the TY-X domain is calculated based on the edge clarity of each image;

[0027]

[0028]

[0029] V n,m (ρ,φ; x) = R n,m (ρ; x)e jm(φ；x) ;

[0030]

[0031] Where: Z n,m (x) represents the (n+m)th order Zernike moment of the xth image in the TY-X domain; X′(t,y;x) represents the pixel value of the xth image in the TY-X domain, t and y are pixel indices, t=0,1,...T, y=0,1,...H; V n,m(ρ, φ; x) represents V n,m (t, y; x) is mapped to the Zernike polynomial in the unit circle; λ N Represents the number of pixels mapped from the normalization factor to the unit circle; s represents the index of the order; n-|m| is an even number, n≥m;

[0032]

[0033] Where: X″(x) represents the reconstructed image of the xth image in the TY-X domain; n max represents the maximum Zernike order;

[0034]

[0035] Where: represents the edge clarity of the x-th image in the TY-X domain; ε represents a constant; N is set to 256;

[0036]

[0037] Where: represents the edge sharpness in the TY-X domain;

[0038] S303: firstly, calculating the brightness change feature of each image in the TY-X domain; then, evaluating the brightness change feature in the TY-X domain based on the brightness change feature of each image;

[0039]

[0040] Where: represents the brightness change characteristics of the xth image in the TY-X domain; μ y；x represents the average value of the yth row of the xth image in the TY-X domain;

[0041]

[0042] Where: Indicates the brightness variation characteristics in the TY-X domain;

[0043] S304: firstly, calculating the semantic integrity of each image in the TY-X domain; then calculating the semantic integrity in the TY-X domain based on the semantic integrity of each image;

[0044]

[0045] Where: Indicates the semantic integrity of the x-th image in the TY-X domain; hole(x) indicates the number of pixels in the x-th image in the TY-X domain when the pixel values ​​of the RGB channels are all 0;

[0046]

[0047] Where: Indicates semantic completeness in the TY-X domain;

[0048] S305: Execute steps S301 to S304 in the XT-Y domain accordingly, and calculate and obtain high-frequency information, edge clarity, brightness change characteristics, and semantic integrity in the XT-Y domain;

[0049] S306: Calculate the rendering distortion of the target synthetic video based on the high frequency information, edge clarity, brightness change characteristics and semantic integrity in the TY-X domain and the XT-Y domain;

[0050]

[0051] Where: S ren_dis Represents the drawing distortion of the target synthetic video.

[0052] Preferably, in step S3, the distribution of feature points of the image is first analyzed in the XT-Y domain to reflect the path trend generated by different viewpoint switching; then the viewpoint path incompatibility under different viewpoint switching paths is evaluated as the viewpoint path incompatibility of the target synthetic video.

[0053] Preferably, step S3 specifically includes the following steps:

[0054] S311: Change the V in the XT-Y domain Y Video B after conversion to binary;

[0055] S312: define a Hessian matrix; and convert the Hessian matrix of each pixel into an approximate value det[H appr ox(y)], used to find feature points;

[0056]

[0057] Where: c represents the pixel point in the yth image in the XT-Y domain; σ B Indicates the image scale, with a value of 1.2; L xx , L xt and L tt represents the convolution of the second derivative of Gaussian;

[0058] det[H approx (y)]=D xx (y)D tt (y)-[0.9D xt (y)] 2 ;

[0059] Where: Second-order derivative D xx(y)=[g(x+1,t;y)-g(x,t;y)]-[g(x,t;y)-g(x-1,t;y)], second derivative D xt and D tt (y) Similarly, g(x, t; y) represents the Gaussian convolution image of the yth image in the XT-Y domain;

[0060] S313: First, the most stable feature points are screened out using the non-maximum suppression and three-dimensional linear interpolation methods of the 3×3×3 neighborhood, and 100 feature points are selected as the final feature point set P; then, the coordinates of the feature point set P are normalized, and the statistical laws of the feature points are reflected using the 2D Gaussian distribution;

[0061]

[0062] Where: μ x,y , μ t,y and σ x,y , σ t,y They represent the mean and standard deviation of the x-direction and t-direction of the y-th image in the XT-Y domain, respectively; r represents the correlation coefficient, which is 0.5;

[0063] S314: Calculating the viewpoint path incompatibility caused by viewpoint path switching of each image;

[0064]

[0065] Where: represents the viewpoint path inappropriateness of the yth image in the XT-Y domain;

[0066] S315: calculating the viewpoint path unsuitability in the XT-Y domain based on the viewpoint path unsuitability of each image as the viewpoint path unsuitability of the target synthetic video;

[0067]

[0068] Where: S vp_dis represents the viewpoint path incompatibility in the XT-Y domain; Ω2 and H k Represents the image set and number of images in the XT-Y domain.

[0069] Preferably, in step S3, high-frequency information evaluation and edge clarity evaluation are performed in the XY-T domain, and compression distortion of the target synthetic video is calculated based on the evaluation results.

[0070] Preferably, step S3 specifically includes the following steps:

[0071] S321: calculating the amplitude spectrum average value of each image in the XY-T domain, and calculating the high-frequency information in the XY-T domain based on the amplitude spectrum average value of each image;

[0072]

[0073] Where: represents the average amplitude spectrum of the t-th image in the XY-T domain; represents the high-frequency information in the XY-T domain; Ω3 and T represent the image set and the number of images in the XY-T domain;

[0074] S322: calculating edge definition of each image in the XY-T domain, and calculating edge definition in the XY-T domain based on the edge definition of each image;

[0075]

[0076] Where: Indicates the edge clarity of the t-th image in the XY-T domain; Indicates edge definition in the XY-T domain;

[0077] S323: Calculating compression distortion in the XY-T domain based on high frequency information and edge definition in the XY-T domain;

[0078]

[0079] Where: S com_dis Represents compression distortion in the XY-T domain; Represents high-frequency information in the XY-T domain; Indicates edge sharpness in the XY-T domain.

[0080] Preferably, in step S4, the objective quality evaluation score is calculated by the following formula:

[0081] Q w-6DoF =ω1R(S com_dis )+ω2R(S ren_dis )+ω3R(S vp_dis );

[0082] Where: Q w-6DoF represents the objective quality evaluation score; S com_dis Indicates compression distortion; S ren_dis Indicates drawing distortion; S vp_dis represents the viewpoint path inappropriateness; ω1, ω2, and ω3 represent the weights of compression distortion, rendering distortion, and viewpoint path inappropriateness, respectively; R(·) represents the regression function.

[0083] Compared with the prior art, the objective quality evaluation method in the present invention has the following beneficial effects:

[0084] The present invention weightedly calculates the objective quality evaluation score by evaluating the rendering distortion, viewpoint path inappropriateness and compression distortion of the target synthetic video. On the one hand, the present invention considers the characteristics of the multidimensional distribution of the window six-degree-of-freedom synthetic video by evaluating the rendering distortion; explores the correlation with human visual perception by evaluating the viewpoint path inappropriateness, so that the evaluation result is more in line with the subjective perception of the human eye; considers the influence of spatial distortion on the quality of the distorted video by evaluating the compression distortion, so that the application scope and robustness of the evaluation model are better. Therefore, the present invention comprehensively considers the various distortions of the window six-degree-of-freedom synthetic video, and can effectively evaluate the visual discomfort caused by the multi-dimensional viewpoint switching of the window six-degree-of-freedom video, thereby improving the accuracy and effect of the objective quality evaluation of the window six-degree-of-freedom synthetic video, and maintaining a high degree of consistency with the subjective perception quality of the human eye. On the other hand, the present invention can reflect the different degrees of influence of each distortion on the video quality by weighted calculation of the rendering distortion, viewpoint path inappropriateness and compression distortion, thereby improving the accuracy of the evaluation model.

[0085] The present invention evaluates distortion in multiple spatiotemporal domains, so that the evaluation of drawing distortion, viewpoint path inappropriateness and compression distortion can be completed correspondingly based on different spatiotemporal distortion characteristics, so that various types of distortion of the windowed six-degree-of-freedom video can be well reflected in their respective spatiotemporal layers, ensuring the comprehensiveness of distortion detection, that is, being able to comprehensively detect the impact of various spatiotemporal domain distortions on video quality, thereby further improving the accuracy and effect of objective quality evaluation of the windowed six-degree-of-freedom synthetic video. BRIEF DESCRIPTION OF THE DRAWINGS

[0086] In order to make the purpose, technical solution and advantages of the invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings, in which:

[0087] Figure 1 It is a logic block diagram of an objective quality evaluation method for a windowed six-degree-of-freedom synthetic video;

[0088] Figure 2 The six-degree-of-freedom composite videos of the window obtained under different virtual viewpoint drawing schemes: Figure 2 (a) is the synthetic video generated under the S1 rendering scheme. Figure 2 (b) is the synthetic video generated under the S4 rendering scheme;

[0089] Figure 3 Switch paths for both viewpoints: Figure 3 (a) is a fixed path, Figure 3 (b) is a significant path;

[0090] Figure 4 The three spatiotemporal domains after video decomposition: Figure 4 (a), 4(b), and 4(c) represent the XY-T domain, XT-Y domain, and TY-X domain, respectively;

[0091] Figure 5 Schematic diagram for accurately positioning images in the XT-Y and TY-X domains using the estimated depth map;

[0092] Figure 6 Distortion visualization and feature extraction results of four virtual viewpoint rendering schemes in the TY-X domain (the solid line box is the visualization of high-frequency information and edge clarity; the dotted line is the visualization of brightness changes; the dotted line box is the visualization of semantic integrity);

[0093] Figure 7 (a) and 7(b) are the movement trends of the fixed path and the significant path in the XT-Y domain, respectively; Figure 7 (c) and Figure 7 (d) is the binary graph of the two paths and their feature point distribution; Figure 7 (e) and Figure 7 (f) is the 2D Gaussian fitting result of the two path feature points. DETAILED DESCRIPTION

[0094] The following is further described in detail through specific implementation methods:

[0095] Example:

[0096] This embodiment discloses an objective quality evaluation method for a window six-degree-of-freedom composite video.

[0097] like Figure 1 As shown, the objective quality evaluation method of the window six-degree-of-freedom synthetic video includes the following steps:

[0098] S1: Obtain a six-degree-of-freedom synthetic video of the window to be evaluated as a target synthetic video;

[0099] S2: Decompose the target synthetic video into several spatiotemporal domains;

[0100] S3: Evaluate the rendering distortion, viewpoint path inappropriateness and compression distortion of the target synthetic video in the corresponding spatiotemporal domains respectively;

[0101] S4: Perform weighted calculation on the rendering distortion, viewpoint path inappropriateness and compression distortion of the target synthetic video to obtain the corresponding objective quality evaluation score.

[0102] The present invention weightedly calculates the objective quality evaluation score by evaluating the rendering distortion, viewpoint path inappropriateness and compression distortion of the target synthetic video. On the one hand, the present invention considers the characteristics of the multidimensional distribution of the window six-degree-of-freedom synthetic video by evaluating the rendering distortion; explores the correlation with human visual perception by evaluating the viewpoint path inappropriateness, so that the evaluation result is more in line with the subjective perception of the human eye; considers the influence of spatial distortion on the quality of the distorted video by evaluating the compression distortion, so that the application scope and robustness of the evaluation model are better. Therefore, the present invention comprehensively considers the various distortions of the window six-degree-of-freedom synthetic video, and can effectively evaluate the visual discomfort caused by the multi-dimensional viewpoint switching of the window six-degree-of-freedom video, thereby improving the accuracy and effect of the objective quality evaluation of the window six-degree-of-freedom synthetic video, and maintaining a high degree of consistency with the subjective perception quality of the human eye. On the other hand, the present invention can reflect the different degrees of influence of each distortion on the video quality by weighted calculation of the rendering distortion, viewpoint path inappropriateness and compression distortion, thereby improving the accuracy of the evaluation model.

[0103] In addition, the present invention evaluates distortion in multiple spatiotemporal domains, making it possible to correspondingly complete the evaluation of drawing distortion, viewpoint path inappropriateness and compression distortion based on different spatiotemporal distortion characteristics, so that various types of distortion of the windowed six-degree-of-freedom video can be well reflected in their respective spatiotemporal layers, ensuring the comprehensiveness of distortion detection, that is, being able to comprehensively detect the impact of various spatiotemporal domain distortions on video quality, thereby further improving the accuracy and effectiveness of objective quality evaluation of windowed six-degree-of-freedom synthetic videos.

[0104] In this embodiment, the window six-degree-of-freedom synthetic video is obtained by combining texture / depth videos of different quality levels, different virtual viewpoint drawing methods and different viewpoint switching paths. Among them, the four window six-degree-of-freedom video sequences are ETRIChef, TechnicolorPainter, OrangeKitchen, and OrangeShaman; the four quality levels of texture / depth videos are the original texture / depth videos without compression distortion and the three quality levels of texture / depth videos compressed by quantization parameter pairs (25,34), (35,42) and (45,48) respectively; the four drawing methods are drawing one virtual viewpoint (S1) with one reference viewpoint, drawing one virtual viewpoint (S2) with two reference viewpoints, drawing one virtual viewpoint (S3) with three reference viewpoints and drawing one virtual viewpoint (S4) with four reference viewpoints; the two viewpoint switching paths are fixed path and significant path.

[0105] The applicant discovered in the study that the distortion of the windowed six-degree-of-freedom synthetic video is mainly composed of three parts: rendering distortion in the time domain, viewpoint path inappropriateness in the time domain, and compression distortion in the space domain.

[0106] Drawing distortion in the time domain is also called flicker distortion. Figure 2 (a) and Figure 2 (b) shows examples of flicker distortion in the windowed six-degree-of-freedom synthesis sequence ETRIChef under the S1 and S4 rendering schemes, and the compression quantization parameters and viewpoint switching paths selected for the two synthetic videos are consistent. Figure 2 (a) and Figure 2 (b) As can be seen from the left sub-image, the flickering of the six-degree-of-freedom window usually occurs in the depth mutation area of ​​the video, that is, the edge of the video and the junction of the foreground and background.

[0107] Different from the free viewpoint video where the flicker distortion is only distributed in the horizontal direction of the target, the flicker distortion of the window six-degree-of-freedom video is arbitrarily distributed in the horizontal and vertical directions of the target, which is more complex. Figure 2 (a) and Figure 2 (b) The right sub-figure shows the pixel intensity change of the same edge position in consecutive frames. The pixel intensity of the target edge drawn by S1 changes dramatically with the frame playback, and shows severe flickering visible to the naked eye; while the pixel intensity of the target edge drawn by S4 does not change significantly in consecutive frames, and the flickering phenomenon is almost unnoticeable. It can be seen that in the windowed six-degree-of-freedom synthetic video, the scheme of using more reference viewpoints for virtual viewpoint drawing contains less flicker distortion and has higher visual perception quality.

[0108] Figure 3 Two schematic diagrams of viewpoint switching paths are given to analyze the visual discomfort caused by viewpoint switching paths. Figure 3 (a) is an “S-shaped” fixed viewing path that is the same for all sequences, Figure 3 (b) is a salient viewing path that gradually guides the viewpoint to the salient area of ​​the video. Compared with the salient path, the fixed path does not follow the human visual attention mechanism and has more abrupt perspective conversion and more frequent viewpoint switching operations, which will cause the video to show scene jitter and motion blur during playback, thereby aggravating the user's dizziness and eye fatigue. It can be seen that compared with the fixed path, the salient path has higher visual comfort and video perception quality.

[0109] Compression artifacts are a common spatial distortion that is distributed in each frame of the video and exhibits global characteristics. Generally, the higher the compression quantization parameter value is set, the more blurred the video will be after decoding.

[0110] In the specific implementation process, the target synthetic video is first converted into a three-dimensional array V consisting of width W, height H and number of frames T, where the indexes of width W, height H and number of frames T are x, y and t respectively; then, based on different combinations of x, y and t, three corresponding spatiotemporal domains are formed: XY-T domain, XT-Y domain and TY-X domain;

[0111] Figure 4Three spatiotemporal representations of the windowed six-DOF video sequence ETRIChef are shown. Among them, the three-dimensional array V in the XY-T domain T Includes T images of size W×H (such as Figure 4 (a) ); three-dimensional array V in the XT-Y domain Y Includes H images of size W×T such as Figure 4 (b) ); three-dimensional array V in the TY-X domain X Includes W images of size T×H such as Figure 4 (c)).

[0112] The present invention decomposes the target synthetic video into three spatiotemporal domains of XY-T, XT-Y and TY-X according to the width, height and number of frames of the target synthetic video, so that the temporal distortion and spatial distortion in the window six-degree-of-freedom synthetic video can be fully reflected, and then the evaluation of drawing distortion, viewpoint path inappropriateness and compression distortion can be correspondingly completed based on different spatiotemporal distortion characteristics, so that various distortions of the window six-degree-of-freedom video can be well reflected in their respective spatiotemporal layers, ensuring the comprehensiveness of distortion detection, so as to fully detect the influence of various spatiotemporal distortions on video quality.

[0113] In the specific implementation process, Figure 5 As shown, before calculating the rendering distortion, the XY-T domain is first divided into multiple frame segments {I1…I k},{I k+1 …I 2k},…, and estimate the depth map E of the interval frame by the two-frame optical flow method; then calculate the maximum intensity value of the depth map E in the x direction and the y direction, and use them as the decomposition fixed points in the TY-X domain and the XT-Y domain respectively; finally, decompose to obtain the image set Ω1 in the TY-X domain and the image set Ω2 in the XT-Y domain; the number of images in the image set Ω1 is W k , the number of images in the image set Ω2 is H k ;

[0114]

[0115] Where: I1(·) and I k (·) represents the first image and the kth image in the XY-T domain; h (·) and l (·) represents the horizontal and vertical components of the optical flow field; x=0,1,...W, y=0,1,...H; η=5 represents the regularization parameter; ρ D and ρ S represents the penalty function; N x,y represents the 5×5 neighboring pixels around (x, y); η N =1 indicates the weight parameter.

[0116] Since rendering distortions are usually locally distributed in depth mutation areas, if all images in the three spatiotemporal domains are decomposed, a lot of computational redundancy will inevitably occur.

[0117] Therefore, the present invention estimates the depth map of the interval frame through the two-frame optical flow method, so that the images in the TY-X domain and the XT-Y domain can be accurately located, and thus the unnecessary image storage and calculation redundancy can be reduced while effectively reflecting the drawing distortion and viewpoint switching path, thereby improving the efficiency of objective quality evaluation.

[0118] During the specific implementation process, high-frequency information evaluation, edge clarity evaluation, brightness change feature evaluation and semantic integrity evaluation are performed in the XT-Y domain and the TY-X domain, and the drawing distortion of the target synthetic video is calculated based on the evaluation results.

[0119] The applicant found that compared with the flicker distortion in the free viewpoint video which is only distributed in the horizontal direction of the target, the flicker distortion in the windowed six-degree-of-freedom synthetic video is randomly distributed in the horizontal and vertical directions of the target, which has higher complexity. Therefore, in combination with the multi-dimensional distribution characteristics of rendering distortion, the present invention proposes to extract rendering distortion in both the XT-Y and TY-X domains. The manifestation of rendering distortion in the two domains is essentially the change of high-frequency components, edge definition, video brightness and semantic information of the hole area, such as Figure 6 As shown in the solid box in , the high-frequency components in S1 to S4 gradually decrease, and the edge sharpness gradually decreases; Figure 6 The dotted line in the figure shows that the brightness changes from S1 to S4 videos gradually stabilize; Figure 6 The dotted box in shows that the semantic information of videos from S1 to S4 is gradually complete.

[0120] Therefore, the present invention performs high-frequency information evaluation, edge definition evaluation, brightness change feature evaluation and semantic integrity evaluation on the XT-Y and TY-X domains to comprehensively evaluate the rendering distortion. Specifically, the following steps are included:

[0121] S301: First, the three-dimensional array V in the TY-X domain X Convert to grayscale video X, perform two-dimensional discrete Fourier transform on each image in X, and calculate the amplitude spectrum of each coordinate of the image; then calculate the amplitude spectrum average of the corresponding image based on the amplitude spectrum of each coordinate of the image; finally calculate the high-frequency information in the TY-X domain based on the amplitude spectrum average of each image;

[0122] A(u,v;x)=α×ln|F(u,v;x)|;

[0123] Where: A(u,v;x) represents the amplitude spectrum of the x-th image at the (u,v) coordinate in the TY-X domain; F(u,v;x) represents the Fourier transform value of the x-th image at the (u,v) coordinate in the TY-X domain; α = 20;

[0124]

[0125] Where: represents the average amplitude spectrum of the xth image in the TY-X domain;

[0126]

[0127] Where: Represents high-frequency information in the TY-X domain; Ω1 and W k Represents the image set and number of images in the TY-X domain;

[0128] For the evaluation of high-frequency information, Field and Brady demonstrated in their visual nerve research as early as 1997 that the amplitude spectrum can simulate the perceptual ability of visual neurons.

[0129] S302: First, the (n+m)-order Zernike moment of each image in the TY-X domain is calculated; then, the corresponding reconstructed image is obtained by inverse transformation of the Zernike moment of the image; then, the structural similarity between the original grayscale image and the reconstructed image is calculated to obtain the edge clarity of the corresponding image; finally, the edge clarity in the TY-X domain is calculated based on the edge clarity of each image;

[0130]

[0131]

[0132] V n,m (ρ,φ; x) = R n,m (ρ; x)e jm(φ；x) ;

[0133]

[0134] Where: Z n,m (x) represents the (n+m)th order Zernike moment of the xth image in the TY-X domain; X′(t,y;x) represents the pixel value of the xth image in the TY-X domain, t and y are pixel indices, t=0,1,...T, y=0,1,...H; V n,m (ρ, φ; x) represents V n,m (t, y; x) is mapped to the Zernike polynomial in the unit circle; λ N Represents the number of pixels mapped from the normalization factor to the unit circle; s represents the index of the order; n-|m| is an even number, n≥m;

[0135]

[0136] Where: X″(x) represents the reconstructed image of the xth image in the TY-X domain; n max represents the maximum Zernike order;

[0137]

[0138] Where: represents the edge sharpness of the x-th image in the TY-X domain; ε represents a constant (ε is a small constant to avoid the denominator being zero); N is set to 256; X″ is composed of n max =10 when the reconstruction is obtained;

[0139]

[0140] Where: represents the edge sharpness in the TY-X domain;

[0141] For edge clarity assessment, Lyvers et al. have verified that moments perform well in describing edge sharpness. In particular, due to the simplicity of inverse transformation of orthogonal Zernike moments, they are widely used in shape feature description and image reconstruction.

[0142] S303: firstly, calculating the brightness change feature of each image in the TY-X domain; then, evaluating the brightness change feature in the TY-X domain based on the brightness change feature of each image;

[0143]

[0144] Where: represents the brightness change characteristics of the xth image in the TY-X domain; μ y；x represents the average value of the yth row of the xth image in the TY-X domain;

[0145]

[0146] Where: Indicates the brightness variation characteristics in the TY-X domain;

[0147] The present invention calculates the standard deviation of the horizontal pixel intensity in the grayscale video X as the brightness change feature.

[0148] S304: firstly, calculating the semantic integrity of each image in the TY-X domain; then calculating the semantic integrity in the TY-X domain based on the semantic integrity of each image;

[0149]

[0150] Where: Indicates the semantic integrity of the x-th image in the TY-X domain; hole(x) indicates the number of pixels in the x-th image in the TY-X domain when the pixel values ​​of the RGB channels are all 0;

[0151]

[0152] Where: Indicates semantic completeness in the TY-X domain;

[0153] The present invention evaluates the semantic integrity of a video by calculating the proportion of the hole area in the video.

[0154] S305: Execute steps S301 to S304 in the XT-Y domain to calculate the high-frequency information, edge clarity, brightness change characteristics and semantic integrity in the XT-Y domain; since rendering distortion also occurs in the XT-Y domain, the four steps of high-frequency information evaluation, edge clarity evaluation, brightness change characteristics evaluation and semantic integrity evaluation are also applied to the XT-Y domain. Among them, the specific evaluation method, steps and formula (the corresponding parameters in the formula need to be replaced) are consistent with those in the TY-X domain.

[0155] S306: Calculate the rendering distortion of the target synthetic video based on the high frequency information, edge clarity, brightness change characteristics and semantic integrity in the TY-X domain and the XT-Y domain;

[0156]

[0157] Where: S ren_dis Represents the drawing distortion of the target synthetic video.

[0158] The present invention evaluates rendering distortion in the XT-Y domain and the TY-X domain, so that the change of rendering distortion in the time domain can be reflected and the characteristics of the multi-dimensional distribution of rendering distortion can be fully considered, thereby improving the accuracy of rendering distortion evaluation. At the same time, the present invention can analyze the influence of high-frequency information, edge clarity, brightness change characteristics and semantic integrity on rendering distortion by performing high-frequency information evaluation, edge clarity evaluation, brightness change characteristics evaluation and semantic integrity evaluation, so that the comprehensiveness of rendering distortion evaluation is better, thereby being able to evaluate the robustness of the model.

[0159] In the specific implementation process, the distribution of feature points of the image is first analyzed in the XT-Y domain to reflect the path trend generated by different viewpoint switching; then the viewpoint path incompatibility under different viewpoint switching paths is evaluated as the viewpoint path incompatibility of the target synthetic video.

[0160] Since there is currently no quality evaluation method proposed for the viewpoint switching problem in windowed six-degree-of-freedom synthetic videos, the present invention analyzes the visual discomfort distortion caused by the viewpoint switching path for the first time. Figure 7 (a) and 7(b) are the XT-Y domain images under the fixed path and the salient path, respectively. Figure 7 It can be seen that the image in the XT-Y domain can well reflect the motion trend of different viewpoint switching paths.

[0161] Therefore, by analyzing the motion distribution law of the image in the XT-Y domain, the visual discomfort caused by the viewpoint switching path can be evaluated. The specific steps include:

[0162] S311: Binarization of V in the XT-Y domain by an adaptive threshold method Y The video B after conversion to binary; Figure 7 (c) and 7(d).

[0163] S312: define a Hessian matrix; and convert the Hessian matrix of each pixel into an approximate value det[H approx (y)] is used to find feature points; the motion patterns of viewpoints in the time domain are counted through feature points to speed up feature description.

[0164]

[0165] Where: c represents the pixel point in the yth image in the XT-Y domain; σ B Indicates the image scale, with a value of 1.2; L xx , L xt and L tt represents the convolution of the second derivative of Gaussian;

[0166] det[H approx (y)]=D xx (y)D tt (y)-[0.9D xt (y)] 2 ;

[0167] Where: Second-order derivative D xx (y)=[g(x+1,t;y)-g(x,t;y)]-[g(x,t;y)-g(x-1,t;y)], second derivative D xt and D tt (y) Similarly, g(x, t; y) represents the Gaussian convolution image of the yth image in the XT-Y domain;

[0168] S313: First, the most stable feature points are screened out using the non-maximum suppression and three-dimensional linear interpolation methods of the 3×3×3 neighborhood, and 100 feature points are selected as the final feature point set P; then, the coordinates of the feature point set P are normalized, and the statistical laws of the feature points are reflected using the 2D Gaussian distribution;

[0169]

[0170] Where: μ x,y , μ t,y and σ x,y , σ t,y Respectively represent the mean and standard deviation of the x-direction and t-direction of the y-th image in the XT-Y domain; r represents the correlation coefficient, which is 0.5; the 2D Gaussian distribution results of the feature points are as follows Figure 7 (e) and 7(f). It can be seen that the 2D Gaussian distribution graph presents a hill shape. The mean and standard deviation in the x and t directions determine the shape of the hill. The closer the mean is to the center of the image and the smaller the standard deviation is, the closer the hill is to the center of the image and the steeper the shape is. This further shows that the distribution of P is more concentrated and more in line with the visual perception characteristics of the human eye.

[0171] S314: Calculating the viewpoint path incompatibility caused by viewpoint path switching of each image;

[0172]

[0173] Where: represents the viewpoint path inappropriateness of the yth image in the XT-Y domain;

[0174] S315: calculating the viewpoint path unsuitability in the XT-Y domain based on the viewpoint path unsuitability of each image as the viewpoint path unsuitability of the target synthetic video;

[0175]

[0176] Where: S vp_dis represents the viewpoint path incompatibility in the XT-Y domain; Ω2 and H k Represents the image set and number of images in the XT-Y domain.

[0177] The present invention detects visual discomfort under different viewpoint switching paths by using the statistics of feature point distribution rules in the XT-Y domain, that is, evaluates the viewpoint path discomfort. On the one hand, the XT-Y domain can well reflect the movement trend of the viewpoint switching path in the time domain, and can improve the accuracy of the viewpoint path discomfort evaluation. On the other hand, by reflecting the changing trend of the viewpoint path through the statistical distribution rules of feature points, it is possible to reduce computational redundancy, thereby improving the efficiency of the viewpoint path discomfort evaluation. In addition, the present invention preliminarily explores the correlation between the computational model and human visual perception by performing 2D Gaussian fitting on the feature points to extract visual discomfort features, so that the evaluation results are more in line with the subjective perception of the human eye.

[0178] During the specific implementation, high-frequency information evaluation and edge clarity evaluation are performed in the XY-T domain, and the compression distortion of the target synthetic video is calculated based on the evaluation results.

[0179] The applicant found that compression distortion is a common spatial distortion, but existing synthetic video objective quality evaluation methods often ignore its impact on the quality of distorted videos. Therefore, the present invention evaluates compression distortion to improve the application scope and robustness of the model. In the windowed six-degree-of-freedom synthetic video, unlike the rendering distortion, the compression distortion is globally distributed in each frame and presents a blurred appearance.

[0180] Since compression distortion generally does not change with time domain, the blur caused by it shows different high-frequency energy and edge sharpness in the XY-T domain. Specifically, the following steps are included:

[0181] S321: Calculate the amplitude spectrum average value of each image in the XY-T domain, and calculate the high-frequency information in the XY-T domain based on the amplitude spectrum average value of each image; refer to step S301 for the specific process.

[0182]

[0183] Where: represents the average amplitude spectrum of the t-th image in the XY-T domain; represents the high-frequency information in the XY-T domain; Ω3 and T represent the image set and the number of images in the XY-T domain;

[0184] S322: Calculate the edge clarity of each image in the XY-T domain, and calculate the edge clarity in the XY-T domain based on the edge clarity of each image; for the specific process, refer to step S302.

[0185]

[0186] Where: Indicates the edge clarity of the t-th image in the XY-T domain; Indicates edge definition in the XY-T domain;

[0187] S323: Calculating compression distortion in the XY-T domain based on high frequency information and edge definition in the XY-T domain;

[0188]

[0189] Where: S com_dis Represents compression distortion in the XY-T domain; Represents high-frequency information in the XY-T domain; Indicates edge sharpness in the XY-T domain.

[0190] The present invention can improve the accuracy of compression distortion by evaluating compression distortion in the XY-T domain. At the same time, the present invention calculates compression distortion through the evaluation results of high-frequency information and edge clarity, and can comprehensively detect the impact of various time-space domain distortions on video quality, thereby improving the robustness of the model. In addition, compared with the existing virtual viewpoint video objective quality evaluation method, the distortion types considered by the present invention are more diverse and can be applied to a variety of distortion environments. The robustness and application scope of the evaluation model are greatly improved.

[0191] In the specific implementation process, the objective quality evaluation score is calculated by the following formula:

[0192] Q w-6DoF =ω1R(S com_dis )+ω2R(S ren_dis )+ω3R(S vp_dis );

[0193] Where: Q w-6DoF represents the objective quality evaluation score; S com_dis Indicates compression distortion; S ren_dis Indicates drawing distortion; S vp_dis represents the viewpoint path inappropriateness; ω1, ω2, and ω3 represent the weights of compression distortion, rendering distortion, and viewpoint path inappropriateness, respectively; R(·) represents the regression function.

[0194] The present invention performs weighted calculation on rendering distortion, viewpoint path inappropriateness and compression distortion, and can reflect the different influences of each distortion on video quality, thereby improving the accuracy of the evaluation model.

[0195] In order to better illustrate the advantages of the present invention, the following experiments are disclosed in this embodiment.

[0196] In order to verify the effect of the technical solution of the present invention, the present invention experimentally analyzes the key parameters of each step; and compares the model proposed by the present invention with the existing natural image quality assessment (Image Quality Assessment, IQA) method, natural video quality assessment (Video Quality Assessment, VQA) method, virtual viewpoint IQA method and virtual viewpoint VQA method. The experiment will be verified on three virtual viewpoint video databases. For all quality assessment methods, the present invention adopts a unified model testing scheme. That is, 80% of the videos in the subjective database are used for training, and 20% are used for testing. The process of training and testing is performed 1000 times, and the median of 1000 trainings is taken as the final result.

[0197] The three performance evaluation indicators used in this experiment are Pearson Linear Correlation Coefficient (PLCC), Spearman Rank Order Correlation Coefficient (SROCC) and Root Mean Square Error (RMSE). Among them, PLCC and RMSE are used to evaluate the accuracy between the predicted scores and subjective scores of the objective quality assessment model, and SROCC is used to measure the monotonicity of the predicted scores and subjective scores. The higher the PLCC and SROCC values, the lower the RMSE value, indicating that the model performance is better.

[0198] Table 1 lists the SSIM (Structure Similarity) values ​​of the original image and its reconstructed image when the maximum order nmax of the Zernike moment in the window six-degree-of-freedom synthetic video database is 0 to 90 (interval of 10).

[0199] The results show that: 1) Under the same rendering method and compression quantization parameter, the SSIM value increases with n max 2) At the same n max When n is , the SSIM value increases with the increase of the reference viewpoint and the quantization parameter, which effectively verifies that the reconstruction similarity of the Zernike moment can well reflect the characteristics of the edge clarity in the spatiotemporal domain. In addition, since the SSIM value is different at different n max Therefore, we only need to select the n with the largest SSIM range difference. max As a representative of the characteristic experiment, further observation of the SSIM values ​​of the TY-X and XT-Y domains shows that when n max When it is equal to 10, the range difference is the largest. maxWhen n is 50, the maximum range difference is obtained. Considering that the computation time of reconstructed image increases with n max The improvement is exponentially increasing, and the present invention finally selects n max Equal to 10 is taken as the optimal solution to balance the efficiency and practical feasibility of the model.

[0200] Table 1 SSIM values ​​between the original image and the reconstructed image with different maximum Zernike moment orders

[0201]

[0202] Table 2 lists the number of images in the TY-X domain and XT-Y domain of the windowed six-degree-of-freedom synthetic video (W k and H k ) takes a value from 1 to 10. It can be seen that as the number of images increases, the images contain more distortion details, and the model performance shows an overall upward trend. However, the increase in the number of images is accompanied by a linear increase in the model calculation time. When balancing accuracy and calculation time, the present invention observes that when the number of images reaches 4, the model performance presents a relatively saturated state, and the calculation time at this time is also within an acceptable range. Therefore, it is finally determined that the number of images in both the TY-X and XT-Y domains is set to 4, that is, W k =H k =4.

[0203] Table 2 Model performance under different numbers of images in window six-degree-of-freedom synthetic video TY-X and XT-Y domains

[0204]

[0205] Table 3 shows the model performance of the window six-degree-of-freedom synthetic video using two regression models and four weighting strategies. Among them, the best performance is marked in bold. The two regression models are Random Forest (RF) regression and Support Vector Regression (SVR). It can be seen from the experimental results that under the same weighting strategy, the performance of the model when using SVR is slightly lower than that using the RF regression method. Therefore, the present invention uses RF regression to generate the final objective quality score.

[0206] At the same time, the present invention further analyzes the impact of different weight strategies on model performance. Through subjective analysis, it can be roughly estimated that the impact of the three main distortions on video quality is: compression distortion ≥ drawing distortion ≥ viewing path discomfort. Therefore, the three weight parameters can be initially sorted as ω1 ≥ ω2 ≥ ω3. On this basis, we experimented with the remaining four combinations of weight parameters, namely (ω1, ω2, ω3) set to (0.5, 0.4, 0.1), (0.5, 0.3, 0.2), (0.4, 0.4, 0.2) and (0.4, 0.3, 0.3). It can be seen from Table 3 that all listed weighting strategies have achieved good performance, among which the best effect is achieved when the weight parameters are (0.4, 0.4, 0.2). In addition, the performance of the weight parameters (0.4, 0.4, 0.2) in both RF regression and SVR is better than other combinations, which also verifies the robustness of the weighting strategy. Therefore, in the present invention, the weight parameters (ω1, ω2, ω3) are set to (0.4, 0.4, 0.2).

[0207] Table 3 Model performance of window six-degree-of-freedom synthetic video using different regression models and weighting strategies

[0208]

[0209] Table 4 conducts feature ablation experiments to analyze the impact of each distortion component of the windowed 6-DOF synthetic video on the model performance, where the best performance is marked in bold. The distortion components of the experiment mainly include: S com_dis , S in the XT-Y domain ren_dis , S in the TY-X domain ren_dis and XT-Y domain S vp_dis Through the analysis of the results, the following two conclusions can be drawn.

[0210] Index 1-4 indicates that the model consists of one distortion component, each of which performs well, but S in the XY-T domain com_dis , S in the XT-Y domain ren_dis and S in the TY-X domain ren_dis The contribution to the model performance is higher than that of S in the XT-Y domain vp_dis For example, the PLCC values ​​of the first three distortion components are all higher than 0.7, while the PLCC value of the latter is only 0.6350. This means that the effect of viewing path discomfort on the quality of windowed 6DOF synthetic video is relatively weak.

[0211] Index 5-10 indicates that the model consists of two distortion components, and index 11-14 indicates that the model consists of three distortion components. By comparing these two categories, we find that the model performance is better and more stable as the distortion components increase. For example, the PLCC range of the model based on the distortion component is 0.7635-0.8256, while the PLCC range of the model based on the three distortion components reaches 0.8310-0.8632. Finally, the PLCC value of the model based on the four distortion components (index 15) proposed in the present invention reaches 0.8876, showing the best performance effect.

[0212] Table 4 Effects of various distortion components of the window six-degree-of-freedom synthetic video on model performance

[0213]

[0214] Table 5 lists the performance comparison results of the model of the present invention with the natural IQA method, the natural VQA method and the virtual viewpoint IQA method on the window six-degree-of-freedom synthetic video subjective database. All methods in Table 5 are NR quality evaluation models. Among them, the natural IQA methods include: BIQI, NIQE, BLIINDS-II, SSEQ and Li. The natural VQA methods include: VideoBLIINDS, VIIDEO and Dendi. The virtual viewpoint IQA methods include: APT, MNSS, NIQSV, NIQSV+ and Wang.

[0215] The experimental results show that both the natural IQA / VQA method and the virtual viewpoint IQA method have limited performance in measuring the distortion of the six degrees of freedom of the window. The reason why the natural IQA / VQA method performs poorly is that they mainly evaluate global distortions such as blur and noise, while the rendering distortion is generally only distributed in the local part of the image. The reason why the virtual viewpoint IQA method performs poorly is that they ignore the impact of time domain distortion on video quality. Therefore, even the best performing Video BLIINDS method among them, with a PLCC of 0.8652, is still not as good as the method of the present invention (PLCC reaches 0.8876).

[0216] In summary, compared with the existing classic and latest methods, the present invention obtains the highest PLCC, SROCC values ​​and the lowest RMSE value on the subjective database of window six-degree-of-freedom synthetic videos, which proves that the objective quality evaluation method proposed in the present invention can comprehensively evaluate the temporal and spatial distortion of window six-degree-of-freedom synthetic videos and has irreplaceable superiority.

[0217] Table 5 Performance comparison results of the proposed model and other models on the subjective database of window six-degree-of-freedom synthetic video

[0218]

[0219] Table 6 lists the performance comparison results of the proposed model and the virtual viewpoint VQA method on the mainstream IRCCyN / IVC DIBR video quality database and the SIAT synthetic video quality database. Among the compared virtual viewpoint VQA methods, CQM, PSPTNR, Liu, SVQM, EM-VQA and SR-3DVQA are FR quality evaluation models, and CTI, FDI, NR_MWT and STD are NR quality evaluation models. By comparing the experimental data, we can see that in the IRCCyN / IVC DIBR video quality database, the proposed method obtains the highest PLCC, SROCC and the lowest RMSE, which shows that the performance of the proposed method is better than the most advanced synthetic VQA method. In the SIAT video quality database, the proposed method also achieves the second best performance, which is only slightly worse than the SR-3DVQA method. However, the SR-3DVQA metric is dependent on the reference video, which leads to certain limitations in practical applications. Therefore, the proposed method is highly competitive.

[0220] Table 6 Performance comparison results of the proposed model and other models on the IRCCyN / IVC DIBR video database and SIAT synthetic video database

[0221]

[0222] In general, the objective quality evaluation method of the present invention has achieved satisfactory experimental results on the self-built window six-degree-of-freedom synthetic video, mainstream IRCCyN / IVC DIBR video and SIAT synthetic video quality database. This result proves that the objective quality evaluation method proposed in the present invention has strong versatility and robustness when evaluating variable synthetic scenes, especially window six-degree-of-freedom synthetic scenes.

[0223] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit the technical solution. Those skilled in the art should understand that those modifications or equivalent substitutions of the technical solution of the present invention that do not depart from the purpose and scope of the technical solution should be included in the scope of the claims of the present invention.

Claims

1. An objective quality evaluation method for a windowed six-degree-of-freedom synthetic video, characterized in that: The following steps are involved: S1: Obtain a six-degree-of-freedom synthetic video of the window to be evaluated as a target synthetic video; S2: Decompose the target synthetic video into several spatiotemporal domains; In step S2, the target synthetic video is first converted into a three-dimensional array V consisting of width W, height H and number of frames T, where the indexes of width W, height H and number of frames T are x, y and t respectively; then, based on different combinations of x, y and t, three corresponding spatiotemporal domains are formed: XY-T domain, XT-Y domain and TY-X domain; Among them, the three-dimensional array V in the XY-T domain T It includes T images of size W×H; a three-dimensional array V in the XT-Y domain Y It consists of H images of size W×T; a three-dimensional array V in the TY-X domain X Includes W images of size T×H; S3: Evaluate the rendering distortion, viewpoint path inappropriateness and compression distortion of the target synthetic video in the corresponding spatiotemporal domains respectively; S4: Perform weighted calculation on the rendering distortion, viewpoint path inappropriateness and compression distortion of the target synthetic video to obtain the corresponding objective quality evaluation score; In step S4, the objective quality evaluation score is calculated by the following formula: Q w-6DoF =ω1R(S com_dis )+ω2R(S ren_dis )+ω3R(S vp_dis ); Where: Q w-6DoF represents the objective quality evaluation score; S com_dis Indicates compression distortion; S ren_dis Indicates drawing distortion; S vp_dis represents the viewpoint path inappropriateness; ω1, ω2, and ω3 represent the weights of compression distortion, rendering distortion, and viewpoint path inappropriateness, respectively; R(·) represents the regression function.

2. The objective quality evaluation method of a window six-degree-of-freedom composite video according to claim 1, characterized in that: In step S3, high-frequency information evaluation, edge clarity evaluation, brightness change feature evaluation, and semantic integrity evaluation are performed in the XT-Y domain and the TY-X domain, and the rendering distortion of the target synthetic video is calculated based on the evaluation results.

3. The objective quality evaluation method of a window six-degree-of-freedom composite video according to claim 2, characterized in that: In step S3, before calculating the rendering distortion, the depth map E of the interval frame is first estimated by the two-frame optical flow method; then the maximum intensity value of the depth map E in the x direction and the y direction is calculated, and used as the decomposition fixed points in the TY-X domain and the XT-Y domain respectively; finally, the image set Ω1 in the TY-X domain and the image set Ω2 in the XT-Y domain are decomposed to obtain; the number of images in the image set Ω1 is W k , the number of images in the image set Ω2 is H k ; Where: I1(·) and I k (·) represents the first image and the kth image in the XY-T domain; h (·) and l (·) represents the horizontal and vertical components of the optical flow field; x=0,1,...W, y=0,1,...H; η=5 represents the regularization parameter; ρ D and ρ S represents the penalty function; N x,y represents the 5×5 neighboring pixels around (x, y); η N =1 indicates the weight parameter.

4. The objective quality evaluation method of a window six-degree-of-freedom composite video according to claim 2, characterized in that: Step S3 specifically includes the following steps: S301: First, the three-dimensional array V in the TY-X domain X Convert to grayscale video X, perform two-dimensional discrete Fourier transform on each image in X, and calculate the amplitude spectrum of each coordinate of the image; then calculate the amplitude spectrum average of the corresponding image based on the amplitude spectrum of each coordinate of the image; finally calculate the high-frequency information in the TY-X domain based on the amplitude spectrum average of each image; A(u,v;x)=α×ln|F(u,v;x)|; Where: A(u,v;x) represents the amplitude spectrum of the x-th image at the (u,v) coordinate in the TY-X domain; F(u,v;x) represents the Fourier transform value of the x-th image at the (u,v) coordinate in the TY-X domain; α = 20; Where: represents the average amplitude spectrum of the xth image in the TY-X domain; Where: Represents high-frequency information in the TY-X domain; Ω1 and W k represents the image set and number of images in the TY-X domain; S302: First, the (n+m)-order Zernike moment of each image in the TY-X domain is calculated; then, the corresponding reconstructed image is obtained by inverse transformation of the Zernike moment of the image; then, the structural similarity between the original grayscale image and the reconstructed image is calculated to obtain the edge clarity of the corresponding image; finally, the edge clarity in the TY-X domain is calculated based on the edge clarity of each image; Where: Z n,m (x) represents the (n+m)th order Zernike moment of the xth image in the TY-X domain; X′(t,y;x) represents the pixel value of the xth image in the TY-X domain, t and y are pixel indices, t=0,1,...T, y=0,1,...H; V n,m (ρ, φ; x) represents V n,m (t, y; x) is mapped to the Zernike polynomial in the unit circle; λ N Represents the number of pixels mapped from the normalization factor to the unit circle; s represents the index of the order; n-|m| is an even number, n≥m; Where: X″(x) represents the reconstructed image of the xth image in the TY-X domain; n max represents the maximum Zernike order; Where: represents the edge clarity of the x-th image in the TY-X domain; ε represents a constant; N is set to 256; Where: represents the edge sharpness in the TY-X domain; S303: firstly, calculating the brightness change feature of each image in the TY-X domain; then, evaluating the brightness change feature in the TY-X domain based on the brightness change feature of each image; Where: represents the brightness change characteristics of the xth image in the TY-X domain; μ y；x represents the average value of the yth row of the xth image in the TY-X domain; Where: Indicates the brightness variation characteristics in the TY-X domain; S304: firstly, calculating the semantic integrity of each image in the TY-X domain; then calculating the semantic integrity in the TY-X domain based on the semantic integrity of each image; Where: Indicates the semantic integrity of the x-th image in the TY-X domain; hole(x) indicates the number of pixels in the x-th image in the TY-X domain when the pixel values ​​of the RGB channels are all 0; Where: Indicates semantic completeness in the TY-X domain; S305: Execute steps S301 to S304 in the XT-Y domain accordingly, and calculate and obtain high-frequency information, edge clarity, brightness change characteristics, and semantic integrity in the XT-Y domain; S306: Calculate the rendering distortion of the target synthetic video based on the high frequency information, edge clarity, brightness change characteristics and semantic integrity in the TY-X domain and the XT-Y domain; Where: S ren_dis Represents the drawing distortion of the target synthetic video.

5. The objective quality evaluation method of a windowed six-degree-of-freedom composite video according to claim 1, characterized in that: In step S3, the distribution of feature points of the image is first analyzed in the XT-Y domain to reflect the path trend generated by different viewpoint switching; then the viewpoint path incompatibility under different viewpoint switching paths is evaluated as the viewpoint path incompatibility of the target synthetic video.

6. The objective quality evaluation method of a window six-degree-of-freedom composite video according to claim 5, characterized in that: Step S3 specifically includes the following steps: S311: Change the V in the XT-Y domain Y Video B after conversion to binary; S312: define a Hessian matrix; and convert the Hessian matrix of each pixel into an approximate value det[H approx (y)], used to find feature points; Where: c represents the pixel point in the yth image in the XT-Y domain; σ B Indicates the image scale, with a value of 1.2; L xx , L xt and L tt represents the convolution of the second derivative of Gaussian; det [H approx (y)]=D xx (the) D tt (y)-[0.9D xt (the)] 2 ; Where: Second-order derivative D xx (y)=[g(x+1,t;y)-g(x,t;y)]-[g(x,t;y)-g(x-1,t;y)], second derivative D xt and D tt (y) Similarly, g(x, t; y) represents the Gaussian convolution image of the yth image in the XT-Y domain; S313: First, the most stable feature points are screened out using the non-maximum suppression and three-dimensional linear interpolation methods of the 3×3×3 neighborhood, and 100 feature points are selected as the final feature point set P; then, the coordinates of the feature point set P are normalized, and the statistical laws of the feature points are reflected using the 2D Gaussian distribution; Where: μ x,y , μ t,y and σ x,y , σ t,y They represent the mean and standard deviation of the x-direction and t-direction of the y-th image in the XT-Y domain, respectively; r represents the correlation coefficient, which is 0.5; S314: Calculating the viewpoint path incompatibility caused by viewpoint path switching of each image; Where: represents the viewpoint path inappropriateness of the yth image in the XT-Y domain; S315: calculating the viewpoint path unsuitability in the XT-Y domain based on the viewpoint path unsuitability of each image as the viewpoint path unsuitability of the target synthetic video; Where: S vp_dis represents the viewpoint path incompatibility in the XT-Y domain; Ω2 and H k Represents the image set and number of images in the XT-Y domain.

7. The objective quality evaluation method of a windowed six-degree-of-freedom composite video according to claim 1, characterized in that: In step S3, high-frequency information evaluation and edge clarity evaluation are performed in the XY-T domain, and the compression distortion of the target synthetic video is calculated based on the evaluation results.

8. The objective quality evaluation method of a windowed six-degree-of-freedom composite video according to claim 7, characterized in that: Step S3 specifically includes the following steps: S321: calculating the amplitude spectrum average of each image in the XY-T domain, and calculating the high-frequency information in the XY-T domain based on the amplitude spectrum average of each image; Where: represents the average amplitude spectrum of the t-th image in the XY-T domain; represents the high-frequency information in the XY-T domain; Ω3 and T represent the image set and the number of images in the XY-T domain; S322: calculating edge definition of each image in the XY-T domain, and calculating edge definition in the XY-T domain based on the edge definition of each image; Where: Indicates the edge clarity of the t-th image in the XY-T domain; Indicates edge definition in the XY-T domain; S323: Calculating compression distortion in the XY-T domain based on high frequency information and edge definition in the XY-T domain; in, Where: S com_dis Represents compression distortion in the XY-T domain; Represents high-frequency information in the XY-T domain; Indicates edge sharpness in the XY-T domain.

Images