A method for evaluating the reducibility of video super-resolution images

Abstract

The present invention provides a method for evaluating the image reduction of video super-resolution, including: S1. Convert the original video and the super-resolved video into video YUV data respectively; S2. Read the data of one frame of image from the YUV data of the original video and the super-resolved video respectively; S3. Scale the YUV data of the super-resolved image read in S2 according to the image scaling algorithm to the same resolution as the original video; S4. Perform edge detection algorithms on the scaled image and the original video image to obtain the main data of the scaled image and the main data of the original video image; S5. Calculate the structural similarity between the original video frame image and the scaled image as the reduction score of this frame of data; S6. Repeat steps S1-S5 until all video frames are completed. The method of the present invention evaluates the image quality for the video main body, improves the evaluation efficiency, and at the same time reduces the influence of unimportant backgrounds on the evaluation results.

Description

Technical Field

[0001] The present invention relates to the field of image quality evaluation, particularly to the field of image quality evaluation for video super-resolution. Specifically, it relates to a method for evaluating the reducibility of video super-resolution images. Background Art

[0002] Video super-resolution, generally abbreviated as video super-res, is a technology that transforms low-resolution videos into higher-resolution videos through various algorithms. With the development of the video industry, the resolution of televisions is getting higher and higher, 4K TVs are gradually popularized, and 8K TVs are also starting to develop. People's requirements for the resolution of video viewing are getting higher and higher. However, many existing video resources are only standard definition or high definition, and ultra-high definition video resources are extremely scarce. To convert the original standard definition videos into high definition or even ultra-high definition, it is necessary to rely on video super-resolution technology. Currently, the market demand for video super-resolution is gradually increasing. Many classic film and television works need to be super-resolved to improve the resolution to meet the audience's requirements for higher clarity.

[0003] With the development of video super-resolution technology and the growth of market demand, the requirements for video super-resolution quality evaluation are also getting higher and higher. However, evaluating the quality of video super-resolution images is a difficult point. Since the purpose of video super-resolution is to process low-resolution videos into higher-resolution videos through technical means, but this process adds pixels based on speculation on the original video, which may change the original image due to incorrect prediction. The commonly used image quality evaluation method PSNR (Peak Signal-to-Noise Ratio) is mainly used to evaluate the overall quality of images. In the super-resolution scenario, it is possible that the PSNR score is relatively high, but there are significant differences between the actually super-resolved image and the original image, and it is impossible to evaluate whether the images before and after super-resolution are consistent. Therefore, a method is needed to evaluate the consistency of the images before and after super-resolution. Video content generally has a main body and a background. It is important that the main body is clear, and the background is blurred. When viewers watch a video, their visual focus is also concentrated on the main body, and they are not sensitive to the background image. Blurred backgrounds are likely to cause large errors in image quality evaluation, but this part of the background data is not important. Based on this, a method for comparing the consistency of the main body of the image is proposed to evaluate the reducibility of video super-resolution images, which can evaluate the quality of video super-resolution from the aspect of reducibility and can complement the PSNR evaluation method to make a more accurate evaluation of the super-resolution quality. Summary of the Invention

[0004] The present invention provides a method for evaluating the reducibility of video super-resolution images, which evaluates the image quality of the video main body, improves the evaluation efficiency, and at the same time reduces the influence of unimportant backgrounds on the evaluation results.

[0005] The technical solution of the present invention is as follows:

[0006] The method for evaluating the restoration of super-resolution video images of the present invention includes the following steps: S1. Convert the original video and the super-resolved video into video YUV data respectively; S2. Read the data of one frame of image from the YUV data of the original video and the super-resolved video respectively; S3. Scale the YUV data of the super-resolved image read in S2 according to the image scaling algorithm to the same resolution as the original video; S4. Perform edge detection on the scaled image and the original video image to obtain the main data of the scaled image and the main data of the original video image; S5. Calculate the structural similarity between the original video frame image and the scaled image as the restoration score of this frame of data; S6. Repeat steps S1 - S5 until all video frames are completed, and calculate the average value of the SSIM scores for all video frames, which is the final restoration score of super-resolution.

[0007] Optionally, in the method for evaluating the restoration of super-resolution video images described above, in step S1, extract the ES video stream data from the original video and the super-resolved video through a video processing tool, then decode the ES data to generate video data in YUV format and save it as a corresponding file.

[0008] Optionally, in the method for evaluating the restoration of super-resolution video images described above, in step S2, read the YUV format file of the super-resolved video generated in S1, obtain the YUV sampling format of the video, and read the data of one frame of image from the YUV format file of the super-resolved video according to the sampling format; in the same way, read the data of one frame of image from the YUV format file of the original video.

[0009] Optionally, in the method for evaluating the restoration of super-resolution video images described above, in step S3, for the YUV data of the super-resolved image read in step S2, use the Lanczos image scaling algorithm for scaling to obtain the scaled frame image data.

[0010] Optionally, in the method for evaluating the restoration of super-resolution video images described above, in step S4, perform edge detection on the scaled image generated in S3 using the canny edge detection algorithm to obtain the main data of the scaled image; for the original video frame image, perform edge detection using the same algorithm to obtain the main data of the original video image.

[0011] Optionally, in the method for evaluating the restoration of super-resolution video images described above, for the two settings of the high threshold and the low threshold used in the edge detection algorithm, use a high threshold of 80 and a low threshold of 60.

[0012] Optionally, in the above method for evaluating the reducibility of video super-resolution images, in step S5, for the main data of the two frames of images generated in S4, calculate the structural similarity of these two frames of images: If the structural similarity of the two frames of images is lower than 0.75, it is considered that the difference is too large and the super-resolution quality is poor, and the quality evaluation ends. If the structural similarity is not lower than 0.75, record the structural similarity value SSIM of these two frames of images i , and continue to evaluate other image frames.

[0013] Optionally, in the above method for evaluating the reducibility of video super-resolution images, in step S6, repeat the evaluation of steps S1 - S5 until the evaluation of all video frames is completed, or end prematurely in step S5. If the evaluation of all video frames is completed, calculate the overall average value of structural similarity through the structural similarity of each frame obtained in step S5, which is the reducibility score of the final super-resolution.

[0014] According to the technical solution of the present invention, the beneficial effects are:

[0015] The evaluation of the reducibility of the video super-resolution image of the present invention is to scale the super-resolved video to obtain an image with the same resolution as the original video, then obtain the main data of the image through edge detection, and then calculate the structural similarity for the main data, and finally obtain the overall average value of structural similarity of the entire video. It mainly utilizes the fact that in general, the main body and the background of a video image are relatively distinct, the main body is clear while the background is blurred, and the focus of the audience is also concentrated on the main body when watching. Therefore, only evaluating the main data can reduce the computational workload of the evaluation on the one hand, and on the other hand, it can also reduce the influence of the background on the evaluation result, because the background itself is relatively blurred and is likely to cause a large error in the evaluation result.

[0016] To better understand and illustrate the concept, working principle and invention effect of the present invention, the present invention will be described in detail below with reference to the accompanying drawings and through specific embodiments: Description of the Drawings

[0017] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following will briefly introduce the accompanying drawings required for the description of the specific embodiments or the prior art.

[0018] Figure 1 is a flowchart of the method for evaluating the reducibility of the video super-resolution image of the present invention. Detailed Embodiments

[0019] To make the purpose, technical method and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and specific examples. These examples are merely illustrative and not limiting to the present invention.

[0020] As Figure 1 shown, the method for evaluating the reduction of video super-resolution images of the present invention includes the following steps:

[0021] S1. Convert the original video and the super-resolved video into video YUV data respectively.

[0022] In this step, extract the elementary stream (ES) video stream data from the original video and the super-resolved video respectively, and then decode it into data in YUV (a color encoding method) format. Specifically, use the video processing tool ffmpeg to extract the ES video stream data from the original video and the super-resolved video, and then use the ffmepg tool to decode the ES video stream data to generate video data in YUV format, and save it as corresponding files, namely the YUV format file of the original video (original video.yuv) and the YUV format file of the super-resolved video (super-resolved video.yuv).

[0023] S2. Read the data of one frame of image from the YUV data of the original video and the super-resolved video respectively.

[0024] In this step, read the YUV format file (super-resolved video.yuv) of the super-resolved video generated in S1, obtain the YUV sampling format (YUV420, YUV422, YUV444) of the video, and read the data Frame of one frame of image from the YUV format file (super-resolved video.yuv) of the super-resolved video according to the sampling format. SR . In the same way, read the data of one frame of image from the YUV format file (original video.yuv) of the original video.

[0025] S3. Scale the super-resolved image (the YUV data of the super-resolved image read in S2) according to the image scaling algorithm to the same resolution as the original video.

[0026] In this step, for the YUV data Frame of the super-resolved image read in S2 SR , use the Lanczos image scaling algorithm for scaling to obtain the scaled frame image data Frame scale . The scaling ratio is based on the super-resolution ratio. If the video is super-resolved by N times, then scale it by 1 / N. In this way, the resolution after scaling is the same as that of the original video.

[0027] S4. Perform edge detection on the scaled image and the original video image to obtain the main body data of the scaled image and the main body data of the original video image.

[0028] Apply an edge detection algorithm to the image generated in step S3 to obtain the main data of this frame of the image, and use the same method to obtain the main data of the original video image.

[0029] In this step, perform edge detection on the scaled frame image Frame generated in S3 scale Use the canny edge detection algorithm to obtain the main data Main of the scaled image scale . For the two settings of the high threshold and the low threshold used in the algorithm, after repeated tests, a high threshold of 80 and a low threshold of 60 are adopted, and the effect of edge detection is relatively good, which can better highlight the main body and remove the background. For the original video image Frame orig , perform edge detection using the same algorithm to obtain the main data Main of the original video image orig ;

[0030] S5. Calculate the structural similarity (SSIM) of the main data of the original video frame image and the scaled image. SSIM is the reduction score of this frame of data.

[0031] In this step, for the main data Main scale and Main orig of the two frames of images generated in S4, calculate the structural similarity SSIM i of these two frames of images. If the structural similarity of the two frames of images is lower than 0.75, it is considered that the difference is too large and the super-resolution quality is poor, and the quality evaluation ends. If the structural similarity is not lower than 0.75, record the structural similarity value SSIM i of these two frames of images and continue with the evaluation of other image frames.

[0032] S6. Repeat steps S1 - S5 until all video frames are completed. Calculate the average value of the SSIM scores for all video frames, which is the reduction score of the final super-resolution.

[0033] In this step, repeat the evaluation of steps S1 - S5 until all video frames are evaluated, or end prematurely in step S5. If all video frames are evaluated, calculate the overall average structural similarity SSIM i using the SSIM avg of each frame obtained in step S5. This value is the reduction score of the final super-resolution.

[0034] The method for evaluating the reducibility of video super-resolution images of the present invention is mainly used to evaluate whether the super-resolved images are reducible to the original video and whether obvious distortion occurs. First, ES video stream data is extracted from the original video and the super-resolved video files, and the ES video stream data is decoded to obtain a YUV format file of the video. One frame of image data is read from the YUV data according to the sampling format. Since image quality comparison needs to be carried out at the same resolution, the data of the super-resolved image is scaled according to the Lanczos algorithm to the same resolution as the original video, and then edge detection is performed on the scaled image to obtain the main data of this frame of image. The main data of the original video is obtained in the same way. The reducibility of the super-resolved image is determined by judging the structural similarity between the two frames of images. The closer the structural similarity is to 1, the higher the reducibility of the super-resolved image.

[0035] According to the test experiment, the test scores of the present invention are significantly positively correlated with the subjective evaluation scores, indicating that the present invention can relatively correctly reflect the subjective experience of viewers.

[0036] The test results are shown in Table 1:

[0037] Table 1

[0038]

[0039] The above description is the best embodiment based on the concept and working principle of the invention. The above embodiments should not be construed as limiting the protection scope of the present claims. Combinations of other implementation manners and implementation methods according to the concept of the present invention all belong to the protection scope of the present invention.

Claims

1. A method for evaluating the restoration of video super-resolution images, characterized in that, It includes the following steps: S1. Convert the original video and the super-resolved video into video YUV data respectively; S2. Read the data of one frame of image from the YUV data of the original video and the super-resolved video respectively; S3. Scale the YUV data of the super-resolved image read in S2 according to the image scaling algorithm to the same resolution as the original video; S4. Perform edge detection on the scaled image and the original video image to obtain the main body data of the scaled image and the main body data of the original video image; S5. Calculate the structural similarity between the original video frame image and the main body data of the scaled image as the reduction score of this frame of data; S6. Repeat steps S1 - S5 until all video frames are completed. Calculate the average value of the SSIM scores for all video frames, which is the reduction score of the final super-resolution.

2. The method for evaluating the reduction of a video super-resolution image according to claim 1, wherein In step S1, extract the ES video stream data from the original video and the super-resolved video through a video processing tool, then decode the ES data to generate video data in YUV format and save it as a corresponding file.

3. The method for evaluating the reduction of a video super-resolution image according to claim 1, wherein In step S2, read the YUV format file of the super-resolved video generated in S1, obtain the YUV sampling format of the video, and read the data of one frame of image from the YUV format file of the super-resolved video according to the sampling format; Read the data of one frame of image from the YUV format file of the original video in the same way.

4. The method for evaluating the reduction of a video super-resolution image according to claim 1, characterized in that, In step S3, for the YUV data of the super-resolved image read in step S2, use the Lanczos image scaling algorithm for scaling to obtain the scaled frame image data.

5. The method for evaluating the reduction of a video super-resolution image according to claim 1, wherein In step S4, perform edge detection on the scaled image generated in S3 using the canny edge detection algorithm to obtain the main body data of the scaled image; for the original video frame image, perform edge detection using the same algorithm to obtain the main body data of the original video image.

6. The method for evaluating the restoration of a super-resolution video image according to claim 5, characterized in that, For the two settings of the high threshold and the low threshold used in the edge detection algorithm, use a high threshold of 80 and a low threshold of 60.

7. The method for evaluating the reduction of a video super-resolution image according to claim 1, wherein In step S5, for the main data of the two frames of images generated in S4, calculate the structural similarity of these two frames of images: If the structural similarity of the two frames of images is lower than 0.75, it is considered that the difference is too large and the super-resolution quality is poor, and the quality evaluation ends. If the structural similarity is not lower than 0.75, record the structural similarity value SSIM of these two frames of images i , and continue to evaluate other image frames.

8. The method for evaluating the restoration of a video super-resolution image according to claim 1, characterized in that In step S6, repeat the evaluation of steps S1 - S5 until the evaluation of all video frames is completed, or end prematurely in step S5. If the evaluation of all video frames is completed, calculate the overall structural similarity average value through the structural similarity of each frame obtained in step S5, which is the reduction score of the final super-resolution.

Images