An Objective Evaluation Method for Full-Reference Ultra-High Definition Video Quality Based on Multi-Feature Fusion

Abstract

The present invention discloses an objective evaluation method for the quality of full-reference ultra-high-definition videos based on multi-feature fusion. First, the ultra-high-definition source video and the distorted video are framed extracted and the Y, U, and V components are calculated. Then, the luminance gradient similarity feature, the visual perception feature, and the chrominance similarity feature are calculated respectively. Next, the features are combined into a one-dimensional feature vector, and a support vector regression model is used to fuse the features and perform regression to obtain the quality scores of each frame. Finally, the average value of the quality scores of each frame is calculated through average pooling to obtain the quality score of the distorted video. The present invention uses a support vector regression model to fuse multiple features and complete the image quality prediction task, improving the prediction accuracy of the video quality score while reducing the algorithm complexity. Experiments show that the performance of this method is superior to most of the existing full-reference video quality evaluation methods.

Description

Technical Field

[0001] The present invention belongs to the technical field of digital image and digital video processing, and particularly relates to an objective evaluation method for full-reference ultra-high definition video quality based on multi-feature fusion. Background Art

[0002] Ultra-high definition video, as a complex source of visual information, contains a large amount of valuable information. In recent years, with the gradual development of ultra-high definition video services, the demand for ultra-high definition video quality evaluation technology has become increasingly urgent. Different types and degrees of distortion will be introduced after ultra-high definition video goes through processing links such as acquisition, compression, storage, transmission, and display, resulting in a decline in video quality. An efficient and accurate video quality evaluation method is of great significance for the quality monitoring of ultra-high definition video services and the research and development of related systems or devices.

[0003] Video quality evaluation methods can be divided into subjective evaluation methods and objective evaluation methods. Subjective evaluation is to subjectively score the video quality by observers. Although the scoring results conform to people's subjective feelings, it has disadvantages such as large workload and long time consumption. The objective evaluation method is to calculate the quality index of the video by the computer according to a certain algorithm. Compared with the subjective evaluation method, it has more application scenarios and broader application requirements. According to whether a reference video is required during evaluation, the objective evaluation method can be divided into three types: full-reference, semi-reference, and no-reference evaluation methods.

[0004] (1) The full-reference video quality evaluation method refers to comparing the difference between the video under test and the reference video when a distortion-free video is given as the reference video, analyzing the distortion degree of the video under test, and thus obtaining the quality evaluation result of the video under test.

[0005] (2) The semi-reference video quality evaluation method refers to extracting partial feature information of the reference video as a reference, comparing and analyzing the video under test, and thus obtaining the quality evaluation result of the video.

[0006] (3) The no-reference video quality evaluation method refers to evaluating the quality of the video under test without a reference video.

[0007] Since the full-reference method can make full use of all the information in the reference video, its evaluation performance is better than that of the other two types of objective evaluation methods and can replace the subjective evaluation method to complete the video quality evaluation task in some application scenarios. Common full-reference methods include: video quality evaluation based on pixel statistics (mainly peak signal-to-noise ratio PSNR, mean square error MSE, etc.), video quality evaluation based on feature extraction, video quality evaluation based on deep learning, etc. Among them, the method based on pixel statistics has less computational complexity and is easy to implement, but the correlation between its results and subjective evaluation results is relatively low; the method based on deep learning can achieve good performance, but the model training cost is relatively high, and the generalization performance depends greatly on the training set. Relatively speaking, the method based on feature extraction has relatively low computational complexity and good generalization ability. Summary of the Invention

[0008] Combined with the characteristics of ultra-high-definition videos, the present invention proposes a full-reference evaluation method based on feature extraction, that is, an objective evaluation method for the quality of ultra-high-definition videos based on multi-feature fusion. This method first extracts frames from the ultra-high-definition source video (i.e., the reference video) and the distorted video and calculates the Y, U, and V components, then calculates the luminance gradient similarity feature, visual perception feature, chrominance similarity feature, etc. respectively. Next, the features are combined into a one-dimensional feature vector, and the support vector regression (SVR) model is used to fuse the features and perform regression to obtain the quality score of each frame. Finally, the average value of the quality scores of each frame is calculated through average pooling to obtain the quality score of the distorted video. The specific steps are as follows:

[0009] Step 1: Select an ultra-high-definition video quality evaluation database.

[0010] The database consists of source videos (i.e., reference videos) and distorted videos. The source videos are distortion-free ultra-high-definition videos, with no less than 20 segments, each segment having a duration of no less than 10 seconds and a frame rate f F not less than 50 Hz. The distorted videos are obtained by compressing, adding noise, etc. to the source videos, and each distorted video must have a subjective evaluation MOS value. The video content should cover typical scenes such as indoor, outdoor, buildings, people, natural scenery, sports competitions, art performances, large-scale mass activities, etc. as much as possible.

[0011] Step 2: Extract frames from the source videos and the distorted videos.

[0012] Extract frames from each source video and its corresponding distorted video, and the frame extraction rate f S is not less than 1:50 to obtain the frame-extracted image sequences of the source videos and the corresponding distorted videos.

[0013] Step 3: Calculate the Y, U, and V components.

[0014] Calculate the Y, U, and V components of each frame in the source video frame extraction image sequence and the corresponding distorted video frame extraction image sequence. The calculation method is shown in Equation (1) (Note: If the video itself is in YUV format, this step is not required).

[0015]

[0016] Step 4: Calculate the luminance gradient similarity feature.

[0017] Step 4.1: Calculate the luminance gradient magnitude GM(x, y) of each frame in the source video frame extraction image sequence and the distorted video frame extraction image sequence. This value represents the contrast information of the image and is calculated using the Scharr operator. The calculation method is shown in Equations (2) and (3).

[0018]

[0019]

[0020] In Equation (3), G x (x, y) and G y (x, y) are the horizontal and vertical gradients of the image; Y(x, y) is the luminance matrix of the image, i.e., the Y matrix.

[0021] Step 4.2: Calculate the luminance gradient similarity feature S GM between each frame in the source video frame extraction image sequence and the corresponding frame in the distorted video frame extraction image sequence. The calculation method is shown in Equation (4).

[0022]

[0023] In Equation (4), GM S (x, y) and GM D (x, y) represent the luminance gradient magnitudes of the frame extraction images in the source video and the distorted video at (x, y), respectively; T1 is a constant with a value of 160; L and N are the number of horizontal and vertical pixels of the image, respectively.

[0024] Step 5: Calculate the visual perception feature.

[0025] Step 5.1: Calculate the visual perception feature I S of the source video. This value represents the mutual information between each frame image in the source video frame extraction image sequence and the frame image perceived by the human eye. The calculation method is shown in Equation (5).

[0026]

[0027] In Equation (5), is the visual noise variance with a value of 2. The calculation method of D S (x, y) is shown in Equation (6).

[0028]

[0029] In formula (6), Y S (x, y) represents the luminance matrix of the frame-extracted images of the source video; G 17*17 is a 17×17 Gaussian filter template when the variance is 2.56.

[0030] Step 5.2, calculate the visual perception feature I D of the distorted video, which represents the mutual information between each frame image in the frame-extracted image sequence of the distorted video and the frame image perceived by the human eye. The calculation method is shown in formula (7).

[0031]

[0032] In formula (7), the calculation method of M is shown in formula (8), and the calculation method of the distortion variance is shown in formula (9).

[0033]

[0034]

[0035] In formulas (8) and (9), the calculation methods of C S，D (x, y) and D D (x, y) are shown in formulas (10) and (11) respectively, where Y D (x, y) is the luminance matrix of the frame-extracted images of the distorted video.

[0036] C S，D (x, y) = G 17*17 * [Y S (x, y) · Y D (x, y)] - [G 17*17 * Y S (x, y)] · [G 17*17 * Y D (x, y)] (10)

[0037]

[0038] Step 5.3, calculate the visual perception feature ratio I, that is, the ratio I D of the visual perception feature I S of each frame in the frame-extracted image sequence of the distorted video to the visual perception feature I D / I S of the corresponding frame in the frame-extracted image sequence of the source video. The calculation method is shown in formula (12).

[0039]

[0040] Step 6, calculate the chromaticity similarity feature.

[0041] Step 6.1, calculate the similarity features S U (x, y) and S V (x, y) between each frame in the source video frame extraction image sequence and the corresponding frame in the distorted video frame extraction image sequence respectively. The calculation method is shown in Formulas (13) and (14).

[0042]

[0043]

[0044] In Formulas (13) and (14), U S (x, y), V S (x, y) are the U - component and V - component matrices of the source video frame extraction image respectively, and U D (x, y), V D (x, y) are the U - component and V - component matrices of the distorted video frame extraction image respectively; T2 and T3 are constants, and their values are 200.

[0045] Step 6.2, calculate the chromaticity similarity feature S C , and the calculation method is shown in Formula (15).

[0046]

[0047] Step 7, perform feature fusion and score regression.

[0048] Step 7.1, generate a feature vector. Combine the 4 features S GM , I S , I, S C obtained in Steps 4 to 6 for each frame into a one - dimensional feature vector F, that is, F = {S GM , I S , I, S C}.

[0049] Step 7.2, use the SVR model for feature fusion and score regression. Train the SVR model with the feature vector F of each frame and the corresponding MOS value of the distorted video. Use SVR to fuse and regress the 4 features in the feature vector F to obtain the score of each frame. The kernel function of the SVR model is RBF (Radial Basis Function), the gamma value is 0.05, and the penalty coefficient C is 8.

[0050] Step 8, perform full - reference video quality assessment on the ultra - high - definition video to be measured.

[0051] Step 8.1: According to Steps 2 to 7, frame extraction, feature calculation, and feature combination are performed on the UHD video under test and its source video to obtain the feature vector F. The feature vector F of each frame is fed into the trained SVR model, and the quality score of each frame is predicted by the SVR model.

[0052] Step 8.2: Average pooling is performed on the quality scores of each frame, that is, the average value of the scores of each frame is calculated to obtain the final video quality score. In actual use, the video quality score can be converted into a percentage system according to needs.

[0053] Compared with the prior art, the present invention has the following advantages:

[0054] (1) Starting from the perspective of feature extraction, the present invention screens out features that can comprehensively meet visual sensitivity, algorithm accuracy, and algorithm speed among numerous features, and uses these features as the basis for judging the video quality score. While improving the prediction accuracy of the video quality score, the algorithm complexity is reduced.

[0055] (2) This method combines feature extraction with machine learning, uses the support vector regression model to fuse multiple features and complete the image quality prediction task, and fully utilizes the advantages of feature extraction and machine learning. Experiments show that the performance of this method is better than most of the existing full-reference video quality assessment methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 is a flowchart of the specific implementation manner of the present invention;

[0057] Figure 2 is a structural diagram of the multi-feature fusion algorithm of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0058] Embodiment.

[0059] The flowchart of the embodiment is as Figure 1 shown and includes the following steps:

[0060] Step S10: Select the UHD video quality assessment database

[0061] Step S20: Extract frames from the source video and the distorted video.

[0062] Step S30: Calculate the Y, U, and V components

[0063] Step S40: Calculate the luminance gradient similarity feature.

[0064] Step S50: Calculate the visual perception feature.

[0065] Step S60: Calculate the chrominance similarity feature.

[0066] Step S70, perform feature fusion and score regression.

[0067] Step S80, perform full-reference video quality assessment on the ultra-high-definition video to be measured.

[0068] The step S10 of selecting the ultra-high-definition video quality assessment database in the embodiment further includes the following steps:

[0069] Step S100, the database consists of source videos (i.e., reference videos) and distorted videos. The source videos are distortion-free ultra-high-definition videos, not less than 20 segments, each segment with a duration of not less than 10 seconds, and the frame rate f F is not less than 50 Hz. The distorted videos are obtained by processing the source videos such as compression and adding noise, and each distorted video must have a subjective evaluation MOS value. The video content should preferably include typical scenes such as indoor, outdoor, buildings, people, natural scenery, sports competitions, art performances, large-scale mass activities, etc.

[0070] The step S20 of extracting frames from the source videos and distorted videos in the embodiment further includes the following steps:

[0071] Step S200, extract frames from each source video and its corresponding distorted video, and the frame extraction rate f S is not less than 1:50, to obtain the frame extraction image sequences of the source videos and the corresponding distorted videos.

[0072] The step S30 of calculating the Y, U, V components in the embodiment further includes the following steps:

[0073] Step S300, calculate the Y, U, V components of each frame in the frame extraction image sequences of the source videos and the corresponding distorted videos. The calculation method is shown in formula (1) (Note: If the video itself is in YUV format, this step is not required).

[0074] The step S40 of calculating the luminance gradient similarity feature in the embodiment further includes the following steps:

[0075] Step S400, calculate the luminance gradient magnitude GM(x, y) of each frame in the frame extraction image sequences of the source videos and the distorted videos. This value represents the contrast information of the image and is calculated using the Scharr operator. The calculation method is shown in formula (2)(3).

[0076] Step S410, calculate the luminance gradient similarity feature S between each frame in the frame extraction image sequence of the source video and the corresponding frame in the frame extraction image sequence of the distorted video GM , and the calculation method is shown in formula (4).

[0077] The step S50 of calculating the visual perception feature in the embodiment further includes the following steps:

[0078] Step S500, calculate the visual perception feature I of the source video S , that is, the mutual information between each frame image in the frame extraction image sequence of the source video and the frame image perceived by the human eye. The calculation method is shown in formula (5).

[0079] Step S510, calculate the visual perception feature I of the distorted video D , that is, the mutual information between each frame image in the frame extraction image sequence of the distorted video and the frame image perceived by the human eye. The calculation method is shown in formula (7).

[0080] Step S520, calculate the visual perception feature ratio I, that is, the visual perception feature I of each frame in the frame extraction image sequence of the distorted video D and the visual perception feature I of the corresponding frame in the frame extraction image sequence of the source video S ratio I D / I S , and the calculation method is shown in formula (12).

[0081] The steps of calculating the chromaticity similarity feature in the embodiment further include the following steps:

[0082] Step S600, calculate the similarity features S U (x, y) and S V (x, y) of the U component and the V component between each frame in the frame extraction image sequence of the source video and the corresponding frame in the frame extraction image sequence of the distorted video respectively. The calculation methods are shown in formulas (13) and (14).

[0083] Step S610, calculate the chromaticity similarity feature S C , and the calculation method is shown in formula (15).

[0084] The steps of performing feature fusion and score regression in the embodiment further include the following steps:

[0085] Step S700, generate an eigenvalue vector. Combine the 4 features S of each frame obtained in steps S40 to S60 GM 、I S 、I、S C into a one-dimensional feature vector F, that is, F = {S GM 、I S 、I、S C}.

[0086] Step S710: Perform feature fusion and score regression using the SVR model. The SVR model is trained with the feature value vectors F of each frame and the corresponding MOS values of the distorted videos. The SVR model is used to fuse and regress the 4 features in the feature vector F to obtain the scores of each frame. The kernel function of the SVR model is RBF (Radial Basis Function), the gamma value is 0.05, and the penalty coefficient C is 8.

[0087] The full-reference video quality evaluation step S80 of the embodiment further includes the following steps:

[0088] Step S800: According to steps S20 to S70, extract frames, calculate features, and combine features for the measured ultra-high-definition video and its source video to obtain the feature vector F. The feature vector F of each frame is fed into the trained SVR model, and the SVR model predicts the quality scores of each frame.

[0089] Step S810: Perform average pooling on the quality scores of each frame, that is, calculate the average value of each frame to obtain the final video quality score. In actual use, the video quality score can be converted into a percentage system according to needs.

[0090] The experimental results of applying this method are given below.

[0091] The measured videos used in this experiment consist of 250 4K ultra-high-definition distorted videos with subjective evaluation MOS values. Each video has a duration of 10 seconds and a frame rate of 50 Hz. These 250 distorted videos are obtained by compressing and decoding 50 distortion-free 4K ultra-high-definition source videos with different degrees of H.264 or HEVC (each source video is compressed and decoded with 5 different degrees of H.264 or HEVC to obtain 5 distorted videos). The video scene content covers indoor, outdoor, buildings, people, natural scenery, sports competitions, art performances, large-scale mass activities and other scenes.

[0092] In this experiment, the 250 measured videos are randomly divided into a training set and a test set according to an 8:2 ratio. The training set contains 200 measured videos (corresponding to 40 source videos), and the test set contains 50 measured videos (corresponding to 10 source videos). The network model of this method is trained with the training set, and the trained network model is tested with the test set. To ensure the robustness of the model, the cross-validation method is used to repeat the experiment 1000 times, and the average value of the 1000 experimental results is taken as the final result.

[0093] The video frame extraction rate f in the experiment SIt is 1:50, and 10 frames are extracted from each video. Three general evaluation metrics, namely the Spearman rank correlation coefficient (SRCC), the Pearson linear correlation coefficient (PLCC), and the root mean square error (RMSE), are used to measure the performance of this method. Table 1 presents the experimental results. For comparison, Table 1 also presents the test results of several other commonly used full-reference image quality assessment methods.

[0094] As can be seen from Table 1, the correlation coefficients between the objective evaluation results and the subjective evaluation results of this method are SRCC = 0.9341, PLCC = 0.9408, and the root mean square error RMSE = 1.0173. Except that the SRCC is slightly lower than that of VIF, it is better than other methods, indicating the effectiveness of this method for the objective evaluation of ultra-high-definition video quality.

[0095] Table 1 Performance comparison between this method and several other commonly used methods

[0096] model SRCC PLCC RMSE this method 0.9341 0.9408 1.0173 PSNR 0.7489 0.7363 27.2731 SSIM 0.8721 0.6878 5.2158 FSIM 0.9023 0.8932 1.4006 VMAF 0.9164 0.9151 75.7106 VIF 0.9396 0.8985 5.2891

Claims

1. An objective evaluation method for full-reference ultra-high definition video quality based on multi-feature fusion, characterized in that: The method includes the following steps: Step 1, select a database for ultra-high-definition video quality evaluation; The database consists of source videos, i.e., reference videos, and distorted videos; the source videos are distortion-free ultra-high-definition videos, with no less than 20 segments, each segment having a duration of no less than 10 seconds and a frame rate f F not less than 50 Hz; the distorted videos are obtained by compressing and adding noise to the source videos, and each distorted video must have a subjective evaluation MOS value; the video content includes indoor, outdoor, buildings, people, natural scenery, sports competitions, art performances, and large-scale mass activity scenes; Step 2, extract frames from the source video and the distorted video; Frame extraction is performed on each source video and its corresponding distorted video, and the frame extraction rate f S is not less than 1:50 to obtain the frame extraction image sequences of the source video and the corresponding distorted video; Step 3, calculate the Y, U, and V components; Calculate the Y, U, and V components of each frame in the source video frame extraction image sequence and the corresponding distorted video frame extraction image sequence. The calculation method is shown in formula (1); if the source video is in YUV format, there is no need to calculate the Y, U, and V components; Step 4, calculate the luminance gradient similarity feature; Step 5, calculate the visual perception feature; Step 6, calculate the chrominance similarity feature; Step 7, perform feature fusion and score regression; Step 8, perform full-reference video quality evaluation on the ultra-high-definition video to be measured; Calculate the visual perception feature. The steps are as follows: Step 5.1, calculate the visual perception feature I of the source video S , which represents the mutual information between each frame image in the frame extraction image sequence of the source video and the frame image perceived by the human eye. The calculation method is shown in formula (5); In formula (5), is the visual noise variance, with a value of 2; D S The calculation method of (x, y) is shown in formula (6); In formula (6), Y S (x, y) represents the luminance matrix of the source video frame image; G 17*17 is a 17*17 Gaussian filter template when the variance is 2.56; Step 5.2, calculate the visual perception feature I of the distorted video D , the visual perception feature I of the distorted video D represents the mutual information between each frame image in the frame extraction image sequence of the distorted video and the frame image perceived by the human eye. The calculation method is shown in formula (7); In formula (7), the calculation method of M is shown in formula (8), and the calculation method of the distortion variance is shown in formula (9); In Formulas (8) and (9), C S，D (x, y) and D D (x, y) are calculated according to Formulas (10) and (11) respectively, where Y D (x, y) is the luminance matrix of the frame-extracted image of the distorted video; C S, D(x,y) = G 17*17 *[Y S (x,y)·Y D (x,y)] - [G 17*17 *Y S (x,y)]·[G 17*17 *Y D (x,y)] (10) Step 5.3, calculate the visual perception feature ratio I, that is, the visual perception feature I of each frame in the frame extraction image sequence of the distorted video D and the visual perception feature I of the corresponding frame in the frame extraction image sequence of the source video S The ratio I D / I S , and the calculation method is shown in formula (12); 。 2. The objective evaluation method for full-reference ultra-high definition video quality based on multi-feature fusion according to claim 1, characterized in that: Calculate the luminance gradient similarity feature. The steps are as follows: Step 4.1, calculate the luminance gradient magnitude GM(x, y) of each frame in the source video frame extraction image sequence and the distorted video frame extraction image sequence. This value represents the contrast information of the image and is calculated using the Scharr operator. The calculation method is shown in formulas (2) and (3); In formula (3), G x (x, y) and G y (x, y) are the horizontal and vertical gradients of the image, and Y(x, y) is the luminance matrix of the image, i.e., the Y matrix; Step 4.2, calculate the luminance gradient similarity feature S between each frame in the source video frame extraction image sequence and the corresponding frame in the distorted video frame extraction image sequence, and the calculation method is shown in formula (4); GM , and the calculation method is shown in formula (4); In formula (4), GM s (x, y) and GM D (x, y) respectively represent the luminance gradient magnitudes of the frame-extracted images in the source video and the distorted video. T1 is a constant with a value of 160; L and N are the number of horizontal and vertical pixels of the image respectively.

3. An objective evaluation method for full-reference ultra-high definition video quality based on multi-feature fusion according to claim 1, characterized in that: Calculate the chrominance similarity feature. The steps are as follows: Step 6.1, calculate the similarity features S U (x,y) and S V (x,y) between each frame in the source video frame extraction image sequence and the corresponding frame in the distorted video frame extraction image sequence. The calculation method is shown in formulas (13) and (14); In Formulas (13) and (14), U S (x, y), V S (x, y) are respectively the U - component and V - component matrices of the source video frame - extracted image, and U D (x, y), V D (x, y) are respectively the U - component and V - component matrices of the distorted video frame - extracted image; T2 and T3 are constants, and their values are 200; Step 6.2, calculate the chromaticity similarity feature S C , and the calculation method is shown in formula (15); 。 4. An objective evaluation method for full-reference ultra-high definition video quality based on multi-feature fusion according to claim 1, characterized in that: Perform feature fusion and score regression. The steps are as follows: Step 7.1, generate a feature vector; combine the 4 features S of each frame obtained in Steps 4 to 6 GM 、I S 、I、S G into a one-dimensional feature vector F, that is, F = {S GM 、I S 、I、S C}; Step 7.2, use the SVR model for feature fusion and score regression; train the SVR model with the feature vector F of each frame and the corresponding MOS value of the distorted video. Use SVR to fuse and regress the 4 features in the feature vector F to obtain the score of each frame; the kernel function of the SVR model is RBF, the gamma value is 0.05, and the penalty coefficient C is 8.

5. An objective evaluation method for full-reference ultra-high definition video quality based on multi-feature fusion according to claim 1, characterized in that: Perform full-reference video quality evaluation on the ultra-high-definition video to be measured. The steps are as follows: Step 8.1, extract frames, calculate features, and merge features for the ultra-high-definition video to be measured and its source video to obtain the feature vector F; send the feature vector F of each frame into the trained SVR model, and the SVR model predicts the quality score of each frame; Step 8.2, perform average pooling on the quality scores of each frame, that is, calculate the average value of each frame to obtain the final video quality score; in actual use, convert the video quality score into a percentage system according to needs.

Images