A full-reference ultra-high-definition video quality objective evaluation method based on deep learning
Through a full reference ultra-high-definition video quality evaluation method based on deep learning, using ResNet18 network and frame extraction, image block cropping, airspace and time domain pooling operations, the low efficiency of ultra-high-definition video quality evaluation and human eye visual characteristics in the existing technology are solved. Unconsidered issues have achieved efficient and accurate video quality evaluation.
Patent Information
- Application Number
- CN202210239824.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-03-12
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2042-03-12
AI Technical Summary
The existing full reference video quality evaluation method is difficult to be applicable to high-resolution and high frame rate ultra-high-definition videos, with low computing efficiency and insufficient consideration of human eye visual characteristics.
Using a full reference ultra-high-definition video quality evaluation method based on deep learning, the ResNet18 network is trained by establishing an image quality evaluation dataset, combining frame extraction, image block cropping, airspace and time domain pooling operations, and simulate subjective evaluation results for video quality evaluation. .
It improves the efficiency and accuracy of video quality evaluation, is better than existing methods, can effectively process ultra-high-definition video, improves computing efficiency and takes into account the visual characteristics of the human eye.
Smart Images

Figure CN114598864B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of digital image and digital video processing, and in particular to a full-reference ultra-high-definition video quality objective evaluation method based on deep learning. Background Art
[0002] Ultra-high-definition video will introduce different types and degrees of distortion after being processed through acquisition, compression, storage, transmission, and display, which will lead to a decrease in video quality. Therefore, studying efficient and accurate video quality evaluation methods is of great significance for the quality monitoring of ultra-high-definition video services and the research and development of related systems or equipment.
[0003] Video quality evaluation is divided into two types: subjective evaluation and objective evaluation. Subjective evaluation is performed by observers directly scoring the video quality, so it can truly reflect the perceived quality of the video image and the evaluation results are reliable. However, subjective evaluation needs to be carried out in a specific laboratory environment and requires a large number of observers to conduct multiple repeated experiments, which is time-consuming and labor-intensive and difficult to meet the needs of actual application scenarios. Objective evaluation is to establish an evaluation model based on the visual characteristics of the human eye, and the model automatically completes the video quality evaluation task. Compared with subjective evaluation, objective evaluation has more application scenarios and wider application needs.
[0004] According to whether the original distortion-free reference video information is used during evaluation, objective video quality evaluation methods can be divided into three types: full reference, semi-reference, and no reference. The full reference method means that all the information of the reference video can be used during evaluation, the semi-reference method means that only part of the information of the reference video can be used during evaluation, and the no reference method means that no information of the reference video is used during evaluation. Since the full reference method can make full use of all the information of the reference video, its evaluation performance is better than the other two methods. In some applications, it can replace the subjective evaluation method to complete the video quality evaluation task.
[0005] Among the existing full-reference video quality objective evaluation methods, deep learning-based methods show good performance for general video quality evaluation. However, it is difficult to achieve good results if these methods are used to evaluate the quality of ultra-high-definition videos. The main reason is that ultra-high-definition videos have the characteristics of high resolution and high frame rate. When using existing methods to process ultra-high-definition videos, the large amount of data will cause the computational efficiency of the quality evaluation model to drop significantly. In addition, most of the existing methods are based on the quality evaluation of single-frame images, and do not fully consider the visual characteristics of the human eye in terms of time domain fusion. Therefore, it is very necessary to propose an efficient and accurate full-reference ultra-high-definition video quality evaluation method. Summary of the invention
[0006] Aiming at the problem that the existing full-reference video quality evaluation algorithm cannot be well applied to ultra-high-definition videos with high resolution and high frame rate, the present invention proposes a full-reference ultra-high-definition video quality objective evaluation method based on deep learning. By training the deep learning network, it can simulate the subjective evaluation results to evaluate the quality of ultra-high-definition videos, including the following steps:
[0007] Step 1: Establish an image quality evaluation dataset.
[0008] An ultra-high-definition video quality evaluation database is selected, and each distorted video in the database and its corresponding reference video (i.e., the source video without distortion) are subjected to frame extraction and image block cropping. The subjective evaluation MOS value of each distorted video is assigned to the image block obtained by frame extraction and image block cropping of the video, that is, the MOS value of each distorted video image block is the MOS value of the distorted video to which the image block belongs.
[0009] Each distorted video image block and its MOS value together with the corresponding reference video image block constitute an image quality assessment dataset, which is used to train a full-reference image quality assessment network based on deep learning.
[0010] Step 1.1, select an ultra-high-definition video quality evaluation database. The database should contain distortion-free ultra-high-definition videos as reference videos, as well as distorted videos that have been compressed, denoised, and processed by the reference videos. Each distorted video must have a subjective evaluation MOS value. There should be no less than 20 reference videos, each of which should be no less than 10 seconds long, and the frame rate fF should be no less than 50 Hz. Each source video should generate no less than 5 distorted videos with different degrees of damage. The video content should include typical scenes such as indoors, outdoors, buildings, people, natural scenery, sports competitions, cultural performances, and large-scale mass activities as much as possible;
[0011] Step 1.2, extract frames for each reference video and its corresponding distorted video, with a frame extraction rate fS not less than 1:50, to obtain a frame extraction image sequence of the reference video and the corresponding distorted video;
[0012] Step 1.3, perform image block cropping on each frame in the frame image sequence, cropping 5 image blocks from each frame, located at the center of the frame image, and the upper left, lower left, upper right, and lower right regions adjacent to the central image block. The size of each image block is 1 / 16 of the frame image, that is, the number of pixels of the image block in the horizontal and vertical directions is 1 / 4 of the frame image;
[0013] Step 1.4, assign the subjective evaluation MOS value of each distorted video to the image block obtained by frame extraction and image block cropping of the video, that is, the MOS value of each distorted video image block is the MOS value of the distorted video to which the image block belongs. Each distorted video image block and its MOS value together with the corresponding reference video image block constitute an image quality evaluation data set for training an image quality evaluation network based on deep learning.
[0014] Step 2: Train the image quality assessment network based on deep learning.
[0015] The distorted video image blocks and their MOS values in the image quality assessment dataset and the corresponding reference video image blocks are used as training data to train the image quality assessment network based on deep learning. The training steps are as follows:
[0016] Step 2.1, build an image quality assessment network based on ResNet18, which consists of a feature extraction module and a quality regression module. The feature extraction module consists of two independent ResNet18 networks and a feature merging layer (Concat layer). The input of each ResNet18 network is a color image with three channels of R, G, and B, and the output is the corresponding feature vector. The feature merging layer merges the feature vectors output by the two ResNet18 networks into one feature vector. The quality regression module is composed of two cascaded fully connected layers, with the input as a feature vector and the output as the corresponding quality score;
[0017] In step 2.2, the distorted video image blocks and the corresponding reference video image blocks in the image quality assessment dataset are input into the image quality assessment network in pairs, the corresponding subjective evaluation MOS values are used as labels, and the network is trained using the Adam update strategy and L1 loss function.
[0018] Step 3: Perform full reference video quality evaluation on the tested ultra-high definition video.
[0019] The tested video and its reference video are subjected to frame extraction and image block cropping, and the cropped tested video image blocks and the corresponding reference video image blocks are input into the trained image quality evaluation network in pairs to obtain the objective quality score of each tested video image block. Finally, the objective quality score of each image block is subjected to spatial and temporal pooling to obtain the objective quality score of the tested video.
[0020] Step 3.1, extract frames of the tested ultra-high-definition video and the reference video according to step 1.2 to obtain a frame-extracted image sequence of the tested video and the reference video, assuming that the number of frames of the frame-extracted image sequence is N;
[0021] Step 3.2, performing image block cropping on each frame of the extracted frame image sequence according to step 1.3 to obtain a tested video image block sequence and a corresponding reference video image block sequence;
[0022] Step 3.3: Input the image blocks in the tested video image block sequence and the corresponding image blocks in the reference video image block sequence into the trained image quality evaluation network in pairs to obtain the quality score q of each tested video image block. i (k), where i represents the frame number of the frame image sequence of the tested video (i=1, 2, ..., N), k represents the image block number in a frame image, and k=0, 1, 2, 3, 4 represent the center block, upper left block, upper right block, lower left block, and lower right block respectively;
[0023] Step 3.4: perform spatial domain pooling on the quality scores of the five image blocks belonging to the same frame image, that is, perform weighted average on the quality scores of the five image blocks to obtain the quality score q of the frame image. i The specific calculation method is shown in formula (1);
[0024]
[0025] Step 3.5, perform temporal pooling on the quality score of each frame, including forward minimum pooling and backward average pooling. Forward minimum pooling is to set the quality score of the current frame to the minimum value of the quality scores of the L frames before the current frame (excluding the current frame), and backward average pooling is to set the quality score of the current frame to the average value of the quality scores of the current frame and the L frames after the current frame. L is the pooling length, and the size of L is determined by the frame rate f of the video. F And the frame rate f S Determine, the specific calculation method is shown in formula (2). Forward minimum pooling value and backward average pooling value The calculation method of is shown in formula (3) and formula (4).
[0026] L=(1~3)f F ·f S (2)
[0027]
[0028]
[0029] Calculate the forward minimum pooling value and backward average pooling value The average of the current frame is the final quality score q′ i , as shown in formula (5).
[0030]
[0031] Step 3.6, based on step 3.5, perform time-domain global average pooling on the quality scores of all frames in the tested video frame image sequence, that is, calculate the average of the image quality scores of each frame in the tested video frame image sequence to obtain the objective quality score Q of the tested video, as shown in formula (6).
[0032]
[0033] Compared with the prior art, the present invention has the following advantages:
[0034] (1) This method uses frame extraction and image block cropping operations to effectively reduce the amount of data processed by the network while retaining the main information of the ultra-high-definition video, thereby improving the efficiency of the algorithm;
[0035] (2) This method uses ResNet18 as the core module of the image quality evaluation network, which effectively improves the training effect of the network and enables the network to effectively extract image features;
[0036] (3) This method combines the visual characteristics of the human eye and performs spatial and temporal pooling operations on the image quality score. Experiments show that the performance of this method is better than most of the existing full-reference video quality assessment methods. BRIEF DESCRIPTION OF THE DRAWINGS
[0037] Figure 1 A flowchart of a specific embodiment of the present invention;
[0038] Figure 2 A schematic diagram of image block cropping positions for ultra-high-definition video frame images;
[0039] Figure 3 This is a structural diagram of the image quality assessment network based on deep learning in the present invention.
[0040] Figure 4 It is a flow chart of the overall implementation of the present invention. DETAILED DESCRIPTION
[0041] Implementation method.
[0042] The flowchart of the implementation method is as follows Figure 1 As shown, the following steps are included:
[0043] Step S10, establishing an image quality evaluation data set;
[0044] Step S20, training an image quality assessment network based on deep learning;
[0045] Step S30, performing full reference video quality evaluation on the ultra-high definition video under test.
[0046] The step S10 of establishing the image quality assessment data set in the implementation manner further includes the following steps:
[0047] Step S100, select an ultra-high-definition video quality evaluation database. The database should contain distortion-free ultra-high-definition videos as reference videos, as well as distorted videos that have been compressed, denoised, and processed by the reference videos. Each distorted video must have a subjective evaluation MOS value. There should be no less than 20 reference videos, each of which should be no less than 10 seconds long and have a frame rate of no less than 50 Hz. Each source video should generate no less than 5 distorted videos with different degrees of damage. The video content should include typical scenes such as indoors, outdoors, buildings, people, natural scenery, sports competitions, cultural performances, and large-scale mass activities as much as possible;
[0048] Step S110, extracting frames for each reference video and its corresponding distorted video, with a frame extraction rate of not less than 1:50, to obtain a frame extraction image sequence of the reference video and its corresponding distorted video;
[0049] Step S120, perform image block cropping on each frame of the image sequence, cropping 5 image blocks from each frame, which are located at the center of the frame image, and in the upper left, lower left, upper right, and lower right regions adjacent to the central image block. The size of each image block is 1 / 16 of the frame image, that is, the number of pixels of the image block in the horizontal and vertical directions is 1 / 4 of the frame image;
[0050] Step S130, assign the subjective evaluation MOS value of each distorted video to the image block obtained by frame extraction and image block cropping of the video, that is, the MOS value of each distorted video image block is the MOS value of the distorted video to which the image block belongs. Each distorted video image block and its MOS value and the corresponding reference video image block together constitute an image quality evaluation data set for training an image quality evaluation network based on deep learning.
[0051] The step S20 of training the image quality assessment network based on deep learning in the implementation manner further includes the following steps:
[0052] Step S200, build an image quality assessment network based on ResNet18, which consists of a feature extraction module and a quality regression module. The feature extraction module consists of two independent ResNet18 networks and a feature merging layer (Concat layer). The input of each ResNet18 network is a color image with three channels of R, G, and B, and the output is the corresponding feature vector. The feature merging layer merges the feature vectors output by the two ResNet18 networks into one feature vector. The quality regression module is composed of two fully connected layers in cascade, with the input being the feature vector and the output being the corresponding quality score;
[0053] Step S210, the distorted video image blocks and the corresponding reference video image blocks in the image quality assessment data set are input into the image quality assessment network in pairs, the corresponding subjective evaluation MOS values are used as labels, and the network is trained using the Adam update strategy and the L1 loss function.
[0054] The step S30 of performing full reference video quality evaluation on the tested ultra-high-definition video in the implementation manner further includes the following steps:
[0055] Step S300, extracting frames of the tested ultra-high-definition video and the reference video according to step S110 to obtain a frame-extracted image sequence of the tested video and the reference video, and assuming that the number of frames of the frame-extracted image sequence is N;
[0056] Step S310, performing image block cropping on each frame of the extracted frame image sequence according to step S120 to obtain a tested video image block sequence and a corresponding reference video image block sequence;
[0057] Step S320: input the image blocks in the tested video image block sequence and the corresponding image blocks in the reference video image block sequence into the trained image quality evaluation network in pairs to obtain the quality score q of each tested video image block. i (k), where i represents the frame number of the frame image sequence of the tested video (i=1, 2, ..., N), k represents the image block number in a frame image, and k=0, 1, 2, 3, 4 represent the center block, upper left block, upper right block, lower left block, and lower right block respectively;
[0058] Step S330, spatially pooling the quality scores of the five image blocks belonging to the same frame image, that is, weighted averaging the quality scores of the five image blocks to obtain the quality score q of the frame image. i , the specific calculation method is shown in formula (1);
[0059] Step S340, perform local temporal pooling operations on the quality scores of each frame of the image, that is, perform forward minimum pooling and backward average pooling. The forward minimum pooling is to set the quality score of the current frame to the minimum value of the quality scores of the L frames before the current frame (excluding the current frame), and the backward average pooling is to set the quality score of the current frame to the average value of the quality scores of the current frame and the L frames after the current frame. L is the pooling length, and the size of L is determined by the frame rate f of the video. F And the frame rate f S Determine, the specific calculation method is shown in formula (2). Forward minimum pooling value and backward average pooling value The calculation method of is shown in formula (3) and formula (4). Calculate the forward minimum pooling value and backward average pooling value The average of the current frame is the final quality score q′i , as shown in formula (5);
[0060] Step S350, based on step S340, performs time-domain global average pooling on the quality scores of all frames in the tested video frame image sequence, that is, calculates the average of the image quality scores of each frame in the tested video frame image sequence to obtain the objective quality score Q of the tested video, as shown in formula (6).
[0061] The experimental results of applying the present invention are given below.
[0062] The videos used in this experiment consist of 250 4K ultra-high-definition distorted videos with subjective evaluation MOS values, each of which is 10 seconds long and has a frame rate of 50 Hz. These 250 distorted videos are obtained by 50 distortion-free 4K ultra-high-definition source videos after H.264 or HEVC compression and decoding to varying degrees (each source video is compressed and decoded to 5 different degrees of H.264 or HEVC to obtain 5 distorted videos). The video scene content covers indoor and outdoor scenes, buildings, people, natural scenery, sports competitions, cultural performances, large-scale mass activities and other scenes.
[0063] In this experiment, 250 videos under test are divided into a training set and a test set according to a ratio of 8:2. The training set contains 200 videos under test (corresponding to 40 source videos), and the test set contains 50 videos under test (corresponding to 10 source videos). The network model of this method is trained with the training set, and the trained network model is tested with the test set. Video frame rate f S The ratio of the two images is 1:10, and the length of the local pooling in the time domain is L=12. The performance of this method is measured using three common evaluation indicators: the Spearman rank correlation coefficient (SRCC), the Pearson linear correlation coefficient (PLCC), and the root mean square error (RMSE). Table 1 gives the experimental results. For comparison, Table 1 also gives the test results of several other commonly used full-reference image quality evaluation methods.
[0064] As shown in Table 1, the correlation coefficients between the objective evaluation results and the subjective evaluation results of this method are SRCC=0.9442 and PLCC=0.9466, which are better than the commonly used methods such as PSNR, SSIM, VMAF, and VIF. The RMSE is 1.4525, which is slightly higher than FSIM and also better than several other methods. The experimental results show the effectiveness of this method for objective evaluation of ultra-high-definition video quality.
[0065] Table 1 Performance comparison of this method with several other commonly used methods
[0066]
[0067]
Claims
1. A full-reference ultra-high-definition video quality objective evaluation method based on deep learning, characterized by: The method comprises the following steps: Step 1, establish an image quality evaluation dataset; An ultra-high-definition video quality evaluation database is selected, and each distorted video in the ultra-high-definition video quality evaluation database and its corresponding reference video, i.e., a distortion-free source video, are subjected to frame extraction and image block cropping; the subjective evaluation MOS value of each distorted video is assigned to an image block obtained by frame extraction and image block cropping of the distorted video, i.e., the MOS value of each distorted video image block is the MOS value of the distorted video to which the image block belongs; each distorted video image block and its MOS value and the corresponding reference video image block together constitute an image quality evaluation data set for training a full reference image quality evaluation network based on deep learning; Step 2: training an image quality assessment network based on deep learning; The distorted video image blocks and their MOS values in the image quality assessment dataset and the corresponding reference video image blocks are used as training data to train the image quality assessment network based on deep learning. Step 3, performing full reference video quality evaluation on the tested ultra-high definition video; The tested video and its reference video are subjected to frame extraction and image block cropping, and the cropped tested video image blocks and the corresponding reference video image blocks are input into the trained image quality evaluation network in pairs to obtain the objective quality score of each tested video image block; finally, the objective quality score of each image block is subjected to spatial and temporal pooling to obtain the objective quality score of the tested video; To establish an image quality assessment dataset, the steps are as follows: Step 1.1, select an ultra-high-definition video quality evaluation database; the database should contain distortion-free ultra-high-definition videos as reference videos, as well as distorted videos that are compressed and noised from the reference videos; each distorted video must have a subjective evaluation MOS value; the reference videos should be no less than 20 segments, each segment should be no less than 10 seconds long, and the video content should include typical scenes of indoors, outdoors, buildings, people, natural scenery, sports competitions, cultural performances, and large-scale mass activities as much as possible; Step 1.2, performing frame extraction processing on each reference video and its corresponding distorted video, with a frame extraction rate of not less than 1:50, to obtain a frame extraction image sequence of the reference video and its corresponding distorted video; Step 1.3, perform image block cropping on each frame of the image sequence, cropping 5 image blocks from each frame, which are located at the center of the frame image, and the upper left, lower left, upper right, and lower right regions adjacent to the central image block. The size of each image block is 1 / 16 of the frame image, that is, the number of pixels of the image block in the horizontal and vertical directions is 1 / 4 of the frame image; Step 1.4, assigning the subjective evaluation MOS value of each distorted video to the image block obtained by frame extraction and image block cropping of the distorted video, that is, the MOS value of each distorted video image block is the MOS value of the distorted video to which the image block belongs; each distorted video image block and its MOS value and the corresponding reference video image block together constitute an image quality evaluation data set for training an image quality evaluation network based on deep learning; The full reference video quality evaluation is performed on the tested ultra-high-definition video. The steps are as follows: Step 3.1, extracting frames of the tested ultra-high-definition video and the reference video according to step 1.2 to obtain a frame-extracted image sequence of the tested video and the reference video; Step 3.2, performing image block cropping on each frame of the extracted frame image sequence according to step 1.3 to obtain a tested video image block sequence and a corresponding reference video image block sequence; Step 3.3, inputting the image blocks in the tested video image block sequence and the image blocks in the corresponding reference video image block sequence into the trained image quality evaluation network in pairs, and obtaining the quality score of each tested video image block; Step 3.4: perform spatial domain pooling on the quality scores of the five image blocks belonging to the same frame image, that is, perform weighted average on the quality scores of the five image blocks to obtain the quality score q of the frame image. i ; The specific calculation method is shown in formula (1); Where i represents the frame number of the image sequence of the video under test, k represents the image block number in a frame, and k=0, 1, 2, 3, 4 represent the center block, upper left block, upper right block, lower left block, and lower right block respectively; Step 3.5, perform time domain pooling operation on the quality score of each frame image, including forward minimum pooling and backward average pooling; the forward minimum pooling is to set the quality score of the current frame to the minimum value of the quality scores of the L frames before the current frame, and the backward average pooling is to set the quality score of the current frame to the average value of the quality scores of the current frame and the L frames after the current frame; L is the pooling length, and the size of L is determined by the frame rate f of the video. F And the frame rate f S Determine, the specific calculation method is shown in formula (2); forward minimum pooling value and backward average pooling value The calculation method of is shown in formula (3) and formula (4), where N is the total number of frames of the measured video frame image sequence; L=(1~3)f F ·f S (2) Calculate the forward minimum pooling value and backward average pooling value The average of the current frame is the final quality score q′ i , as shown in formula (5); Step 3.6: Based on step 3.5, perform temporal global average pooling on the quality scores of each frame image, that is, calculate the average value of the quality scores of each frame image in the frame image sequence of the tested video. The average value is the objective quality score of the tested video, as shown in formula (6); 2. The method for evaluating the quality of ultra-high-definition video based on deep learning according to claim 1, characterized in that: To train a deep learning-based image quality assessment network, the steps are as follows: Step 2.1, build an image quality assessment network based on ResNet18, which consists of a feature extraction module and a quality regression module; the feature extraction module consists of two independent ResNet18 networks and a Concat layer, the input of each ResNet18 network is a color image with three channels of R, G, and B, and the output is the corresponding feature vector; the Concat layer merges the feature vectors output by the two ResNet18 networks into one feature vector; the quality regression module is composed of two fully connected layers in cascade, the input is the feature vector, and the output is the corresponding quality score; In step 2.2, the distorted video image blocks and the corresponding reference video image blocks in the image quality assessment dataset are input into the image quality assessment network in pairs, the corresponding subjective evaluation MOS values are used as labels, and the network is trained using the Adam update strategy and L1 loss function.
