A Deep Learning-Based Method and System for No-Reference Rendering Video Quality Assessment

Abstract

This invention provides a deep learning-based method and system for evaluating the quality of no-reference rendered videos. The method includes: Step 1: Segmenting video data and randomly extracting image frames to construct feature sequences and obtain image quality scores; Step 2: Segmenting video data and performing motion estimation to obtain a second video subset; Step 3: Performing image difference calculation on the second video subset and then inputting it into a pre-trained image difference detector and multilayer perceptron to obtain a quality score reflecting the temporal stability of the video; Step 4: Comprehensively evaluating the quality scores to obtain a final evaluation score. This invention comprehensively evaluates video quality in both the temporal and spatial domains. In the absence of a reference video, it performs additional optimizations for common distortion types in rendered videos, helping to judge the quality of the rendered image and assisting users in balancing image quality and rendering overhead, thereby optimizing rendering settings on different platforms.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence, and specifically to a method and system for evaluating the quality of no-reference rendered videos based on deep learning. Background Technology

[0002] With the development of rendering technology and the popularity of real-time rendering applications, rendered video is becoming an important part of people's daily media. Quality assessment of these videos helps optimize rendering settings and improve rendering quality. Subjective video quality assessment relying on the human eye can yield accurate results, but this method is costly and inefficient.

[0003] In many applications, automatic, objective video quality assessment metrics are more practical. These objective video quality assessment metrics can be further divided into reference-based and referenceless video quality assessments. Reference-based video quality assessment requires a high-quality, lossless video as a reference object, and the target video is compared with the reference video to obtain a relative video quality score. However, in rendering applications, it is often difficult to obtain suitable reference videos for comparison due to the difficulty in reproducing perfectly consistent camera and object motion. In this case, referenceless video quality assessment metrics are more suitable as quality assessment standards because they do not require video as a reference object input.

[0004] Traditional no-reference video quality assessment methods use image features such as local contrast, brightness, and chroma to measure video quality. However, these hand-designed features are difficult to assess the quality of complex video content accurately. With the introduction of deep learning into video quality assessment, new methods have delegated the task of video feature extraction to deep neural networks, improving the accuracy of the assessment. Furthermore, some methods have also evaluated temporal factors affecting video quality.

[0005] For example, Reference 1 (Quality assessment of in-the-wild videos[C], Proceedings of the 27th ACM international conference on multimedia, 2019: 2351-2359) provides a video quality assessment algorithm, VSFA, which introduces a gated recurrent unit to simulate the human eye's perception of changes in video quality, extending the dimension of video quality assessment from the spatial domain to the temporal domain. However, this method extracts information from a single image frame and then further assesses the video quality.

[0006] For example, invention application CN118741086A discloses an image quality inspection method, device, vehicle, and storage medium for vehicle-mounted video transmission. Based on a convolutional neural network, it acquires raw video stream data from the sending end in real time and extracts each raw video frame image from the raw video stream data; acquires multiple second video frame images from the receiving end in real time; performs identification recognition on the multiple second video frame images to obtain second identification information corresponding to each second video frame image and multiple de-identified third video frame images; determines the first image transmission quality based on the differences between each first identification information and the corresponding second identification information; inputs each raw video frame image and each third video frame image into a preset image quality evaluation model to obtain the second image transmission quality; and determines the target image transmission quality based on the first image transmission quality and the second image transmission quality.

[0007] The above video quality assessment metrics based on deep neural networks have achieved high accuracy on camera-captured video datasets. However, rendered videos differ significantly from real-world videos. Past video quality assessment methods have focused more on distortions caused by camera capture, such as blur, overexposure, underexposure, and noise. For rendered videos, distortions caused by insufficient sampling rates, such as jagged edges, moiré patterns, and flicker, require additional attention. These are neglected by past video quality assessment methods, making them difficult to directly apply to the rendering field. Summary of the Invention

[0008] The purpose of this invention is to provide a deep learning-based no-reference rendering video quality assessment method and system. This method comprehensively assesses video quality from both temporal and spatial domains, focusing on local losses in video quality assessment. It aims to solve distortion problems such as jagged edges, moiré patterns, and flicker caused by insufficient sampling rate in rendering, and to provide a more accurate reference for subsequent rendering.

[0009] To achieve the above-mentioned objectives, an embodiment provides a deep learning-based method for evaluating the quality of no-reference rendered video, comprising the following steps:

[0010] Step 1: Divide the video data into several video segments with a preset step size. Randomly select one frame from each video segment to form the first data set. Construct a feature sequence based on the first data set to obtain the image quality score of the first data set.

[0011] Step 2: Divide the video data into several first video subsets on an average basis. Randomly extract consecutive image frames from the first video subsets and perform motion estimation to obtain the motion vectors of the first video subsets. The first video subsets are then processed by inverse deformation and occlusion removal to obtain the second video subsets.

[0012] Step 3: Perform image difference calculation on the motion vectors of the second video subset, and input the difference calculation result into the pre-trained image difference detector and the first multilayer perceptron to obtain the quality score of video temporal stability;

[0013] Step 4: Input the image quality score and video temporal stability quality score of the first dataset into the second multilayer perceptron to obtain the final evaluation score of the video data. The number of neurons in the second multilayer perceptron is less than the number of neurons in the first multilayer perceptron.

[0014] This invention comprehensively evaluates video quality from both temporal and spatial domains, obtains a video quality score in the absence of a reference video, and makes additional optimizations for the types of distortion that are prone to occur in rendered videos. The final video quality score helps to judge the quality of the rendered image and assists users in balancing image quality and rendering overhead, thereby optimizing the rendering settings on different platforms.

[0015] In one embodiment, dividing the video data into several video segments with a preset step size includes: dividing the total number of image frames of the video data by the frame rate of the video data to set the corresponding segmentation step size.

[0016] In one embodiment, the step of constructing a feature sequence based on a first data set to obtain an image quality score for the first data set includes: extracting features from each data in the first data set using a pre-trained Swing Transformer, concatenating the extracted features to construct a corresponding feature sequence, and inputting the feature sequence into a nonlinear layer to output the image quality score for the first data set.

[0017] In one embodiment, the step of randomly extracting consecutive image frames from the first video subset and performing motion estimation to obtain the motion vector of the first video subset includes: randomly selecting N consecutive frames of images from each first video subset and cropping them to a uniform size; performing motion estimation using a dense optical flow tracing algorithm on the first N-1 frames of images in each first video subset to obtain the motion vector relative to the Nth frame; and masking the pixels that do not have mapped motion vectors.

[0018] In one embodiment, the reverse deformation includes performing reverse deformation and occlusion removal processing on the first N-1 frames of the image after mask annotation:

[0019] F′ k→(N) =Warping(F k M k )·D

[0020] Among them, F′ k→(N) This represents the image frame after inverse warping. Warping(·) is the inverse warping operation.k M represents the k-th image frame. k This represents the motion vector estimate of the first N-1 frames relative to the Nth frame, and D represents the overlapping image in the occlusion region marking of the N-1 frames.

[0021] In one embodiment, the pre-trained image difference detector and the first multilayer perceptron are optimized for parameters using a loss function, which is a weighted combination of the Pearson linear correlation coefficient and the ranking loss function.

[0022] In one embodiment, the Pearson linear correlation coefficient L PLCC :

[0023]

[0024] Where s represents the number of videos, Let represent the final evaluation score obtained by the prediction of the i-th video, qi represent the true score of the human annotation of the i-th video, a represent the mean of the final evaluation scores obtained by the prediction of all videos, and b represent the mean of the true scores of the human annotation of all videos.

[0025] In one embodiment, the ranking loss function L ranking :

[0026]

[0027] Where s represents the number of videos, Let q represent the final evaluation score obtained from the prediction of the j-th video. j Let represent the actual score of the j-th video, which is manually annotated, and sgn(·) represent the sign function.

[0028] To clearly demonstrate the deep learning-based no-reference rendering video quality assessment method, the present invention also provides a deep learning-based no-reference rendering video quality assessment system, which includes an image quality assessment unit, a temporal stability assessment unit, and a comprehensive video quality assessment unit.

[0029] The image quality evaluation unit is used to randomly select image frames from the input video data to output the corresponding image quality score;

[0030] The temporal stability evaluation unit is used to perform reverse deformation and occlusion removal processing on the input video data, and to perform image difference calculation and feature regression on the processing results to output the corresponding video temporal stability quality score.

[0031] The video quality comprehensive evaluation unit performs a comprehensive evaluation based on the image quality score and the video temporal stability quality score, and outputs the corresponding final evaluation score.

[0032] Compared with the prior art, the beneficial effects of the present invention include at least the following:

[0033] (1) By segmenting video data, randomly extracting image frames from the video data and constructing feature sequences to obtain image quality scores, we can better consider the changes of each pixel in the temporal domain and focus on the local quality loss of the rendered video.

[0034] (2) Motion estimation and image difference calculation are performed on the video. These difference images can reflect the temporal continuity of video segments at different time frequencies, which helps to detect distortion in the rendered video.

[0035] (3) The video quality is comprehensively evaluated from the temporal and spatial domains to obtain the final evaluation score. In the absence of reference video, additional optimization is performed on the types of distortion that are prone to occur in the rendered video. This helps to judge the quality of the rendered image and assists users in weighing the image quality and rendering overhead, thereby optimizing the rendering settings on different platforms. Attached Figure Description

[0036] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0037] Figure 1 This embodiment provides a flowchart illustrating a deep learning-based no-reference rendering video quality assessment method.

[0038] Figure 2 This is a schematic diagram illustrating the process of obtaining image quality score and video temporal stability quality score.

[0039] Figure 3 This is a schematic diagram of the image difference detector and multilayer perceptron provided in this embodiment;

[0040] Figure 4 This is a schematic diagram of the structure of a deep learning-based no-reference rendering video quality assessment system provided in this embodiment. Detailed Implementation

[0041] To make the objectives, technical solutions, and advantages of this invention clearer, the following description is provided in conjunction with the accompanying drawings and...

[0042] The embodiments further illustrate the present invention in detail. It should be understood that the specific embodiments described herein are merely illustrative of the invention and do not limit the scope of protection of the invention.

[0043] To address distortion issues such as jagged edges, moiré patterns, and flickering caused by insufficient sampling rate in rendering, this embodiment provides a deep learning-based no-reference rendering video quality assessment method. The following section combines... Figure 1 and Figure 2 This invention specifically introduces a deep learning-based no-reference rendering video quality assessment method, comprising the following steps:

[0044] S1. Divide the video data into several video segments with a preset step size. Randomly select one frame from each video segment to form a first data set. Construct a feature sequence based on the first data set to obtain the image quality score of the first data set.

[0045] In the embodiment, the input is video data V = {F1, F2, ..., F} containing z frames and with a frame rate of r. z The video data V is uniformly divided into... A segment {C1, C2, ..., C} N Each segment lasts for 1 second, where F represents an image frame and C represents a uniformly divided set of image frames;

[0046] Next, one frame is randomly selected from each segment to form the first data set {C′1, C′2, ..., C′}. N These image frames are then fed into a pre-trained Swing Transformer to extract features;

[0047] Finally, the extracted features are concatenated to construct the corresponding feature sequence, and the feature sequence is input into a nonlinear layer to output the image quality score q of the first dataset. a .

[0048] S2. Divide the video data into several first video subsets, randomly extract continuous image frames from the first video subsets and perform motion estimation to obtain the motion vectors of the first video subsets, and obtain the second video subsets by inverse deformation and occlusion removal processing of the first video subsets.

[0049] In the embodiment, the video data V = {F1, F2, ..., F...} z The video is divided into 10 first video subsets containing 5 consecutive frames, forming a first video subset set {K1, K2, ..., K}. 10 For each first video subset K i A cropping coordinate is randomly generated. Based on this coordinate, each image frame in the first video subset is cropped into a segment {F′} with a height of 480 pixels and a width of 800 pixels.j F′ j+1 F′ j+2 F′ j+3 F′ j+4}, where F′ represents the cropped image frame, and j is a random number.

[0050] It's important to note that the five frames within the same first video subset still use the same cropping coordinates to maintain their temporal continuity. Next, for each first video subset K... i Motion estimation is performed on the first four frames to obtain the motion vector estimate M of the first four frames relative to the fifth frame. k Then, by marking M k For pixels without mapped motion vectors, an image D marking the occluded region can be obtained.

[0051] Next, use M k Perform reverse deformation on the first 4 frames:

[0052] F′ k→(j+4) =Warping(F k M k )·D (1)

[0053] Where k∈{j, j+1, j+2, j+3}, Warping(·) is the inverse warping operation, and D is four occlusion region marker maps {D j D j+1 D j+2 D j+3 The overlapping parts in}

[0054] After the deformation is completed, 5 aligned frames are obtained, denoted as the second video subset K′. i ={F′ j→(j+4) F′ (j+1)→(j+4) F′ (j+2)→(j+4) F′ (j+3)→(j+4) F′ j+4 Construct a second subset of videos {K′1, K′2, ..., K′}. 10}

[0055] S3. Perform image difference calculation on the motion vectors of the second video subset, and input the difference calculation result into the pre-trained image difference detector and the first multilayer perceptron to obtain the quality score of video temporal stability.

[0056] In the embodiments, such as Figure 3 As shown, the second video subset set {K′1, K′2, ..., K′} constructed based on S2 10For each second video subset, image difference calculation is performed. Image difference operation is performed on each pair of adjacent frame images, as well as each pair of frame images with an interval of 1 frame, 2 frames, and 3 frames, thereby obtaining 10 image difference results. These difference images reflect the temporal continuity of video segments at different time frequencies, which helps to detect distortions such as flickering and jagged edges that are prone to occur in rendering.

[0057] The difference results of these 10 images are then input into a neural network based on depthwise separable convolution to obtain a feature vector of length 84.

[0058] Finally, feature regression is performed on the features of all subsets using average pooling and a first-level multilayer perceptron (MLP) to obtain a quality score q that reflects the temporal stability of the video. b .

[0059] S4. Input the image quality score and video temporal stability quality score of the first dataset into the second multilayer perceptron to obtain the final evaluation score of the video data, wherein the number of neurons in the second multilayer perceptron is less than the number of neurons in the first multilayer perceptron.

[0060] In the embodiment, the image quality score q obtained based on S1 a The time stability mass score q obtained from S3 b The contribution of the video data to the final evaluation score is non-linear. Therefore, S4 uses a second multilayer perceptron (MLP) to map this non-linear state. The structure of the second multilayer perceptron (MLP) in S4 is roughly the same as that of the first multilayer perceptron (MLP) in S3. The only difference is that the second multilayer perceptron (MLP) only uses one-quarter of the number of neurons of the first multilayer perceptron (MLP) to prevent overfitting.

[0061] During training, the parameters of the pre-trained image difference detector and the first multilayer perceptron (MLP) are optimized using a loss function. The loss function uses the Pearson linear correlation coefficient L. PLCC and sorting loss function L ranking The final evaluation score obtained from predicting multiple videos in S4 is represented as follows: The true scores of multiple manually labeled videos are represented as Q = {q1, q2, ..., q...} s},in, Let q represent the final evaluation score obtained from the prediction of the s-th video. s This represents the actual score of the s-th video, which was manually annotated.

[0062] The Pearson linear correlation coefficient L used PLCC as follows:

[0063]

[0064] Where s represents the number of videos, Let represent the final evaluation score obtained by the prediction of the i-th video, gi represent the true score of the human annotation of the i-th video, a represent the mean of the final evaluation scores obtained by the prediction of all videos, and b represent the mean of the true scores of the human annotation of all videos.

[0065] The Pearson linear correlation coefficient L used ranking as follows:

[0066]

[0067] Where s represents the number of videos, This represents the final evaluation score obtained from the prediction of the i-th video. Let q represent the final evaluation score obtained from the prediction of the i-th video. i Let represent the true score of the human-annotated video i, qj represent the true score of the human-annotated video j, and sgn(·) represent the sign function;

[0068] The final loss function LOSS is determined by the Pearson linear correlation coefficient L. PLCC and sorting loss function L ranking Composed of weighted combinations:

[0069] LOSS = L PLCC +α·L ranking (4)

[0070] Where α is the weight, set to 0.3.

[0071] To test the actual effectiveness of the technical solution provided in this embodiment, it will be evaluated on the ReVQ-2k rendered video dataset and other types of video datasets. When the dataset provides temporal stability quality scores as labels, training can be performed based on these scores to achieve better results. Methods based on temporal stability quality scores are marked "Ours-" in Table 1. Methods trained only based on the overall quality score are marked "Ours" in Table 1. The effectiveness of the quality assessment is measured using two metrics: SRCC and PLCC. The performance of this invention on the ReVQ-2k rendered video dataset is shown in Table 1 (higher values ​​are better).

[0072] Table 1

[0073]

[0074] In addition, the present invention has also achieved good results on non-rendered video datasets, as shown in Table 2 (the larger the value, the better), which demonstrates the robustness of the deep learning-based no-reference rendered video quality assessment method provided by the present invention.

[0075] Table 2

[0076]

[0077]

[0078] To clearly demonstrate the deep learning-based no-reference rendering video quality assessment method, this embodiment also provides a deep learning-based no-reference rendering video quality assessment system for implementing the aforementioned deep learning-based no-reference rendering video quality assessment method, such as... Figure 4 As shown, the no-reference rendering video quality assessment system includes an image quality assessment unit, a temporal stability assessment unit, and a comprehensive video quality assessment unit.

[0079] The image quality evaluation unit is used to randomly select image frames from the input video data to output the corresponding image quality score;

[0080] The temporal stability evaluation unit is used to perform reverse deformation and occlusion removal processing on the input video data, and to perform image difference calculation and feature regression on the processing results to output the corresponding video temporal stability quality score.

[0081] The video quality comprehensive evaluation unit performs a comprehensive evaluation based on the image quality score and the video temporal stability quality score, and outputs the corresponding final evaluation score.

[0082] By using the above system, video quality can be comprehensively evaluated in both the temporal and spatial domains. In the absence of a reference video, additional optimizations can be made for the types of distortion that are prone to occur in the rendered video. This helps to judge the quality of the rendered image and assists users in balancing image quality and rendering overhead. As a result, the rendering settings on different platforms can be optimized, and a better implementation of a deep learning-based no-reference rendering video quality evaluation method can be achieved.

[0083] Furthermore, it should be understood that after reading the above description of the present invention, those skilled in the art can make various alterations or modifications to the present invention, and these equivalent forms also fall within the scope defined by the appended claims.

Claims

1. A deep learning-based method for evaluating the quality of no-reference rendered video, characterized in that, Includes the following steps: Step 1: Divide the video data into several video segments with a preset step size. Randomly select one frame from each video segment to form the first data set. Construct a feature sequence based on the first data set to obtain the image quality score of the first data set. Step 2: Divide the video data into several first video subsets, randomly extract consecutive image frames from the first video subsets and perform motion estimation to obtain the motion vectors of the first video subsets, and obtain the second video subsets by inverse deformation and occlusion removal processing of the first video subsets; Step 3: Perform image difference calculation on the motion vectors of the second video subset, and input the difference calculation result into the pre-trained image difference detector and the first multilayer perceptron to obtain the quality score of video temporal stability; Step 4: Input the image quality score and video temporal stability quality score of the first dataset into the second multilayer perceptron to obtain the final evaluation score of the video data. The number of neurons in the second multilayer perceptron is less than the number of neurons in the first multilayer perceptron.

2. The no-reference rendered video quality assessment method according to claim 1, characterized in that, The method of dividing video data into several video segments with a preset step size includes: dividing the total number of image frames of the video data by the frame rate of the video data to set the corresponding segmentation step size.

3. The method for evaluating the quality of no-reference rendered video according to claim 1, characterized in that, The method of constructing a feature sequence based on a first data set to obtain an image quality score for the first data set includes: extracting features from each data in the first data set using a pre-trained SwinTransformer, concatenating the extracted features to construct a corresponding feature sequence, and inputting the feature sequence into a nonlinear layer to output the image quality score for the first data set.

4. The method for evaluating the quality of no-reference rendered video according to claim 1, characterized in that, The step of randomly extracting consecutive image frames from the first video subset and performing motion estimation to obtain the motion vector of the first video subset includes: randomly selecting N consecutive frames of images from each first video subset and cropping them to a uniform size; performing motion estimation using a dense optical flow tracing algorithm for the first N-1 frames of images in each first video subset to obtain the motion vector relative to the Nth frame; and masking the pixels that do not have mapped motion vectors.

5. The method for evaluating the quality of no-reference rendered video according to claim 1, characterized in that, The reverse warp and occlusion removal process includes performing reverse warp and occlusion removal on the first N-1 frames of the image after masking: F′ k→(N) =Warping(F k ,M k )·D Among them, F′ k→(N) This represents the image frame after inverse warping. Warping(·) is the inverse warping operation. k M represents the k-th image frame. k This represents the motion vector estimate of the first N-1 frames relative to the Nth frame, and D represents the overlapping image in the occlusion region marking of the N-1 frames.

6. The method for evaluating the quality of no-reference rendered video according to claim 1, characterized in that, The pre-trained image difference detector and the first multilayer perceptron are optimized for parameters through a loss function, which is a weighted combination of the Pearson linear correlation coefficient and the ranking loss function.

7. The method for evaluating the quality of no-reference rendered video according to claim 6, characterized in that, The Pearson linear correlation coefficient L mentioned above PLCC : Where s represents the number of videos, Let q represent the final evaluation score obtained from the prediction of the i-th video. i Let represent the true score of the human-annotated i-th video, a represent the mean of the final evaluation scores obtained from predictions of all videos, and b represent the mean of the true scores of the human-annotated i-th videos.

8. The method for evaluating the quality of no-reference rendered video according to claim 6, characterized in that, The ranking loss function L ranking : Where s represents the number of videos, This represents the final evaluation score obtained from the prediction of the j-th video. Let q represent the final evaluation score obtained from the prediction of the i-th video. i Let q represent the actual score of the i-th video, which is manually labeled. j Let represent the actual score of the j-th video, which is manually annotated, and sgn(·) represent the sign function.

9. A deep learning-based no-reference rendering video quality assessment system, characterized in that, The system is used to implement the deep learning-based no-reference rendering video quality assessment method according to any one of claims 1 to 8, wherein the no-reference rendering video quality assessment system includes an image quality assessment unit, a temporal stability assessment unit, and a comprehensive video quality assessment unit. The image quality evaluation unit is used to randomly select image frames from the input video data to output the corresponding image quality score; The temporal stability evaluation unit is used to perform reverse deformation and occlusion removal processing on the input video data, and to perform image difference calculation and feature regression on the processing results to output the corresponding video temporal stability quality score. The video quality comprehensive evaluation unit performs a comprehensive evaluation based on the image quality score and the video temporal stability quality score, and outputs the corresponding final evaluation score.

Images