Guided optimization of a video quality metric

Abstract

Methods and systems are described to optimize a computation of a video quality metric. In one embodiment, a device receives a video clip, where the video clip includes a compressed video clip and an uncompressed video clip. The device may divide the video clip into a plurality of windows, wherein each of the windows includes a plurality of video frames. The device further may, for each of the plurality of windows, compute an estimated video quality error using a set of video frames selected from that window. Furthermore, if the estimated video quality error is greater than a threshold, the device may divide that window into a set of smaller windows and add the set of smaller windows to the plurality of windows, else the device may compute an estimated video quality score for that window.

Description

[0001] GUIDED OPTIMIZATION OF A VIDEO QUALITY METRIC

[0002] CROSS REFERENCES TO RELATED APPLICATIONS

[0003] [1] This application claims the benefit of priority from U.S. Provisional Application No. 63 / 654,335, filed on May 31, 2024, and European Patent Application No. 24193200.3, filed on August 6, 2024, each of which is incorporated by reference herein in its entirety.

[0004] TECHNOLOGY

[0005] [2] The present invention relates generally to video processing. More particularly, an embodiment of the present invention relates to using a guided optimization of determining a video quality metric.

[0006] BACKGROUND

[0007] [3] When video content is encoded for streaming or distribution, the video is compressed to a certain bitrate so that the encoded video can fit within a given bandwidth or storage medium limit. Due to the compression, the encoded video can suffer from loss of quality, which can range from negligible or not noticeable to the viewer to yielding visible artifacts that detract from the viewer experience. A video metric is a way to measure this loss of quality and adjust to the video encoding ensure the best viewer experience.

[0008] [4] However, the computation of video quality metric can be quite expensive. For example, one video quality metric, Video Multimethod Assessment Fusion (VMAF), is computed on a frame-by-frame basis. For videos at high resolutions such as 4K, VMAF can take much longer than encoding, and even longer than the duration of the video in real-time. For streaming services that need to encode large amounts of video content, this barrier can be unacceptable.

[0009] BRIEF DESCRIPTION OF THE DRAWINGS

[0010] [5] The present invention is illustrated by way of example and not limitation in the figures of the accompanying drawings in which like references indicate similar elements. [6] Figure 1 shows an example of a system that can be used in one or more embodiments of this invention.

[0011] [7] Figure 2 shows an example of a system that computes a video quality metric overall error with an explicitly computed video quality metric.

[0012] [8] Figure 3 shows, in a flow diagram, an example of a process for computing a video quality metric error using guided optimization that can be used with one or more embodiments of the invention.

[0013] [9] Figure 4A shows, in a flow diagram, an example of a process for computing an estimated midpoint error for B-Frames that can be used with one or more embodiments of the invention.

[0014]

[0010] Figure 4B shows, in a flow diagram, an example of a process for computing an estimated midpoint error for P-Frames that can be used with one or more embodiments of the invention.

[0015]

[0011] Figure 5 shows an example of computing a video quality metric using a window over multiple video frames.

[0016]

[0012] Figure 6 shows an example of computing a video quality metric using a window and an estimated midpoint error over multiple video frames.

[0017]

[0013] Figure 7 shows an example comparing an estimated VMAF score versus an actual VMAF score for a window size of 64 that can be used with one or more embodiments of the invention.

[0018]

[0014] Figure 8 shows an example comparing an estimated VMAF score versus an actual VMAF score for a window size of 100 that can be used with one or more embodiments of the invention.

[0019]

[0015] Figure 9 shows an example comparing an estimated VMAF score versus an actual VMAF score for a window size of 300 that can be used with one or more embodiments of the invention.

[0020]

[0016] Figure 10 shows an example of a data processing system that can be used to perform or implement one or more embodiments of the invention. DETAILED DESCRIPTION

[0021]

[0017] Various embodiments and aspects will be described with reference to details discussed below, and the accompanying drawings will illustrate the various embodiments. The following description and drawings are illustrative and are not to be construed as limiting. Numerous specific details are described to provide a thorough understanding of various embodiments. However, in certain instances, well-known or conventional details are not described in order to provide a concise discussion of embodiments.

[0022]

[0018] Reference in the specification to "one embodiment" or "an embodiment" means that a particular feature, structure, or characteristic described in conjunction with the embodiment can be included in at least one embodiment. The appearances of the phrase "in one embodiment" in various places in the specification do not necessarily all refer to the same embodiment. The processes depicted in the figures that follow are performed by processing logic that comprises hardware (e.g., circuitry, dedicated logic, etc.), software, or a combination of both. Although the processes are described below in terms of some sequential operations, it should be appreciated that some of the operations described may be performed in a different order. Moreover, some operations may be performed in parallel rather than sequentially.

[0023]

[0019] The embodiments described herein can be used in apparatuses which include one or more processors in a processing system, and which include memory and which are configured to perform any one of the methods described herein. Moreover, the embodiments described herein can be implemented using non-transitory machine-readable storage media storing executable computer program instructions which when executed by a machine cause the machine to perform any one of the methods described herein.

[0024]

[0020] The embodiments described herein can be used to optimize a computation of a video quality metric. In one embodiment, when video content is encoded for streaming or distribution, the video is compressed to a certain bitrate so that it can fit within a given bandwidth or storage medium. Due to the compression, the encoded video can suffer from loss of quality, which can be negligible or not noticeable to the viewer or ranger to yielding visible artifacts that detract from the viewer experience. Video metrics are a way to measure this loss of quality and adjust to ensure the best viewer experience. As used herein the terms ‘encoded video’ and ‘compressed video’ are used interchangeably to refer to a video or video clip (i.e. part of a video) that has been encoded for streaming or distribution. Similarly, the terms ‘original video’, ‘original studio quality video’ and ‘uncompressed video’ can be used interchangeably herein to refer to videos and video clips before being encoded according to a desired bitrate for streaming or distribution.

[0025]

[0021] There are several ways to compute a video quality metric that measures a loss of quality due to various processing of the video (e.g., different video codecs, encoders, encoding settings, transmission variants, and / or other processing effects). In one embodiment, a video quality metric is a mathematical model that objectively evaluates a degradation of quality on a video due to processing of the video. In one embodiment, a video quality metric can be a full reference model, which compares each frame of the original video with the corresponding frame in the processed video. Video Multimethod Assessment Fusion (VMAF) is a full reference video quality metric that uses multiple elementary metrics and machine learning to produce a metric that predicts video quality as perceived by an average viewer. As such, VMAF captures the subjectivity of the human visual system and is widely used. VMAF is run on a frame-by-frame basis, generating a score from 0 to 100 for each frame of the encoded video as compared to the original reference video. The VMAF score for an entire clip is the average of the VMAF scores for each frame. Since VMAF uses the computation of multiple elementary metrics and its internal machine learning model, VMAF can be slow to run frame-by-frame for entire video clips. For videos at high resolutions such as 4K, VMAF can take much longer than encoding, and even longer than the duration of the video in real-time. For streaming services that need to encode large amounts of video content, this barrier is unacceptable.

[0026]

[0022] In one embodiment, a device that reduces the time it takes to compute video quality metric for an encoded video clip is described. The device uses a guided temporal subsampling that aims to choose different windows of video frames to compute the video quality metric that gives an estimate of the shape of the VMAF signal for the video. In this embodiment, the device uses the following inputs: the original reference video clip, the encoded reduced-bitrate video clip, the window size parameter, and the frame type for each frame in the encoded video.

[0027]

[0023] In one embodiment, the encode quality of a frame depends on its frame type, where B-frames typically have lower video quality metric scores than P-frames (and the occasional I-frame is handled with the P-frames). Thus, the device uses a list of frame types in the encoded video so as to handle the B-frames and P-frames separately. The device also takes as input a window size, measured in number of frames. The entire video clip is initially split into windows of this size, and each window is processed as described below.

[0028]

[0024] Figure 1 shows an example of a system 100 that can be used in one or more embodiments the invention. In one embodiment, the system 100 includes an encoder 106 that outputs encoded video 108 from an original studio quality video 102 at a desired bitrate 104. In this embodiment, the original studio quality video 102 can be a digital video, such as High Definition, 4K, and / or another type of digital video. The video can be one or more video clips. The encoder 106 encodes the original studio quality video 102 at the desired bitrate 104. For example, and in one embodiment, the encoded video can be an H.264 / Advanced Video Coding (AVC), a High Efficiency Video Coding (HEVC), or another type of video coding.

[0029]

[0025] A problem can occur because when the video 102 is encoded for streaming or distribution, the video 102 is compressed to the desired bitrate so that it can fit within a given bandwidth or storage medium limit. Due to the compression, the encoded video suffers from loss of quality. To determine a loss of quality in the encoded video 108, a video metric is computed. Thus, the video metrics are a way to measure this loss of quality and adjust to ensure the best viewer experience.

[0030]

[0026] A problem with computing a video quality metric is that computing the metric can be computationally intensive. For example, and in one embodiment, VMAF is a full reference video quality metric that uses multiple elementary metrics and machine learning to produce a metric that predicts video quality as perceived by an average viewer. VMAF, however, is run on a frame-by-frame basis, generating a score from 0 to 100 for each frame of the encoded video as compared to the original reference video. The VMAF score for an entire clip is the average of the VMAF scores for each frame. Since VMAF requires the computation of multiple elementary metrics and the internal machine learning model, computing a VMAF score for a clip can be slow to run frame-by-frame for entire video clips. For videos at high resolutions such as 4K, VMAF can take much longer than encoding, and even longer than the duration of the video in real time.

[0027] In one embodiment, a metric device 118 computes an estimated video quality metric more efficiently than a frame-by-frame video quality metric computation. In this embodiment, the metric device 110 computes the estimated video quality metric using a guided subsampling 110 using windows of video frames to determine an average video quality metric score for each of the video frame windows. In this embodiment, the metric device 118 uses the window size 116 to determine the multiple windows. While in one embodiment, the windows are initially the same size, in alternate embodiments, the initial windows can be different sizes. In one embodiment, the window size 1 16 is an input to the video quality metric calculation. As described further herein, the inputs to the metric device include both an original video and a corresponding encoded video. It should be understood that the video quality metrics are computed for frames of the encoded video in the sense that the quality of the encoded video is what is being assessed. However, the quality metrics may be computed based on both the encoded video and the original video.

[0031]

[0028] The metric device may compute, for each window, an estimated video quality error using a selected set of video frames, and if the estimated video quality error is greater than a threshold, divide that window into a set of smaller windows and add the set of smaller windows to the multiple windows. If the estimated video quality is less than the threshold, the metric device may compute an estimated video quality score for that window. The metric device may then determine a video quality score for the video clip based on the plurality of estimated video quality scores determined for the plurality of windows. The estimated video quality error may be computed as an error of an estimated video quality score for at least one of the selected video frames. For example, where the selected frames include a first frame, a last frame and a midpoint frame, an estimated video quality score for the midpoint frame might be determined by interpolating between computed video quality scores of first and last frames of the window, and this estimated midpoint video quality score could be compared to a computed midpoint video quality score for the midpoint frame to determine the estimated video quality error for the window. The estimated video quality score for the window may be computed based on an average video quality metric score over estimated video quality metric scores obtained by interpolation between the selected video frames of the window (e.g. estimated video quality score metrics may be computed for each frame of the window by interpolating between the first and last video frames of the window).

[0029] In a further embodiment, the metric device 118 separates B-frames and P-frames for each window (using the encoded frame types 114), estimates the video quality metric scores for each set separately and combines the results to arrive at a video quality metric score for the encoded video. In this embodiment, the metric device 118 performs a guided subsampling by selecting the first and last B-frame in the window and computes their VMAF scores. The metric device 118 finds the B-frame nearest to the midpoint of the window and computes its VMAF score. This midpoint score is compared against the average of the first and last B-frame scores. If the difference falls within a configurable error threshold, the VMAF scores of the remaining B-frames in the window can be linearly interpolated between the first and last B-frame scores. Otherwise, if the midpoint difference exceeds the error threshold, then the VMAF signal has too much variability to reasonably capture with linear interpolation, so the window is split into two halves and the process is repeated. Once window boundaries are determined, the interpolated B-frame VMAF scores are averaged to produce a B-estimate for the window. While in one embodiment, the metric device 118 computes a VMAF score as the video quality metric, in alternate embodiments, the metric device 118 computes another type of video quality metric (e.g., Peak signal-to-noise ratio (PSNR), Peak signal-to-noise ratio for human visual system (PSNR-HVS), Structural similarity index measure (SSIM), and / or another type of video quality metric).

[0032]

[0030] In another embodiment, a similar process is repeated for the P-frames to produce a P-estimate of the video quality metric for that window, which is combined with the B- estimate using a weighted average based on the proportion of B-frames and P-frames in the window. This yields an estimate of the VMAF score for this window. Each window requires 6 VMAF computations (start, midpoint, and end for B-frames and for P-frames), instead of computing VMAF on every frame. If the window is, for example, 100 video frames, then six video quality metric computations are made for the 100 video frames in the window.

[0033]

[0031] After all windows are processed, the window scores are combined using a weighted average based on the number of frames in each window. This yields an estimate of the VMAF score for the entire clip. Computing the estimated video quality metric is further described in Fig. 3 below.

[0032] In one embodiment, the encoder 106 can be a server, personal computer, laptop, camera, smartphone, or another device that can process video clips. In one embodiment, the metric device 118 can be a server, cluster, personal computer, laptop, camera, smartphone, or another device that can store video clips.

[0034]

[0033] In order to assess the metric device’s 118 computation of the video quality metric, an explicit video quality metric score can be computed by the metric device 118 or another device and compared with the estimated video quality metric score. In addition, the metric device can compute the estimated video quality score using different initial window sizes so as to determine beneficial windows size(s). Figure 2 shows an example of a system 200 that computes a video quality metric overall error with an explicitly computed video quality metric. Similar to Fig. 1 above, the system 200 includes an encoder 206 that outputs encoded video 208 from an original studio quality video 202 at a desired bitrate 204. The video can be one or more video clips. The encoder 206 encodes the original studio quality video 202 at the desired bitrate 204. In addition, the metric device 224 computes the estimated video quality metric 216 using guided subsampling 214 for the inputted video clip using the original studio quality video 202, encoded video 208, encoded frame types 220, and window size input 218. In a further embodiment, the metric device 224 computes the frame-by-frame video quality metric 210 using the original studio quality video 202 and encoded video 208 to arrive at the actual video quality metric 212. With the estimated metric video quality score 216 and the actual metric video quality score 212, the metric device can compute a metric video quality error 222. By computing different window sizes for a video clip, the metric device 224 can determine a range of metric video quality score errors.

[0035]

[0034] For example, and in one embodiment, a window size of 64 frames yields a speedup of about 7x with an error of up to 0.3 VMAF points across the entire clip, while a window size of 300 frames yields a speedup of about 20x with an error of up to 0.7 VMAF points. A window size in between, such as 100 frames, can find a compromise between these points. In this embodiment, this way of estimating the video quality metric, speeds up the efficiency of the computer as the computing of a video quality metric is greatly sped up with a low error rate (e.g., less than 1%). Example computations of an estimated metric video quality score are illustrated in Figures 7-9 below.

[0035] Figure 3 shows, in a flow diagram, an example of a process 300 for computing a video quality metric error using guided optimization that can be used with one or more embodiments of the invention. In one embodiment, process 300 begins by receiving the input at block 302. In this embodiment, the input is the original studio quality video, encoded video, encoded frame types, and windows size. In this embodiment, the original studio quality video, encoded video, encoded frame types, and windows size is as described in Figure 1 above. At block 304, process 300 generates an initial list of windows, where each of the windows is a window of video frames of the inputted video. In one embodiment, the initial list of windows is a list of windows with a size determined by the inputted window size. For example, and in one embodiment, if the input video includes 380 frames and the window size is 100, then three windows of size 100 and a window of size 80 is generated. In another embodiment, the initial list of windows can be different sized windows.

[0036]

[0036] Process 300 performs a processing loop (blocks 306-320) to compute a video quality metric error for each window. At block 308, process 300 computes an estimated midpoint error. In one embodiment, process 300 computes the estimated midpoint error by determining a midpoint video frame and computing an actual video quality metric score and determining an estimated midpoint video quality metric score (e.g., by interpolation). With these two scores, process 300 can determine an error. In one embodiment, process 300 determines separate midpoint errors for B-frames and P-frames in that window. In one embodiment, the separate midpoint errors for B-frames and P-frames in that window are combined in a weighted average using the relative proportion of the number of B- frames and P-frames in that window. Alternatively, the midpoint B-frames and / or P- frames are used separately to determine whether the window should be split. Determining the estimated midpoint error is further described in Figures 4A-B below.

[0037]

[0037] At block 310, process 300 determines if the estimated midpoint error is larger than an error threshold. In one embodiment, the error threshold is a numerical value set by the code, input by the user, or set in another way. A larger error threshold would be less "sensitive" (splits fewer windows, which is faster but less accurate), and a smaller one would be more sensitive (and result in more window splits). In one embodiment, the error threshold varies depending on the type of video quality metric that is used. In one embodiment, a value of 3.0 can be used as an error threshold for a VMAF video quality metric.

[0038]

[0038] If so, process 300 determines if the size of the current window is greater than (or greater than or equal to) two times a minimum window size. If the current window size is greater than (or greater than or equal to) two times a minimum window size, process 300 divides the window into two or more smaller windows and reprocesses the new windows. In one embodiment, process 300 adds the new smaller windows to the list of windows and execution proceeds to block 306. If the estimated midpoint error is less than (or less than or equal to) an error threshold at block 310 or the current window size is less than (or less than or equal to) two times a minimum window size at block 312, execution proceeds to block 316, where process 300 keeps the window at block 316. At block 318, process 300 computes the estimated video quality metric for this window. In one embodiment, process 300 computes the video quality for the metric by computing an average video quality metric score for the B-frames and the P-frames. In one embodiment, process 300 interpolates the B-frames in the window using the initial video quality metric scores from the can be linearly interpolated between the first and last B -frame video quality metric scores. With these estimated scores for the B-frames, process 300 averages these scores based on the number of B-Frames in the window. In a further embodiment, process 300 performs a similar process to determine a P-frame average video quality metric score. With the averaged B-frame and P-frame video quality metric scores, process 300 computes a weighted average of the B-frame and P-frame video quality metric scores (weighted using the number of B-frames and P-frames) to arrive at the estimated video quality metric. The processing loop ends at block 320.

[0039]

[0039] At block 322 process 300 combines video quality metrics from each window into an estimated video quality metric. In one embodiment, process 300 combines the window scores using a weighted average based on the number of frames in each window. This yields an estimate of the estimated video quality metric score for the inputted video. Process 300 returns the estimated video quality metric at block 324.

[0040]

[0040] As described above, process 300 computes a midpoint error to determine if the window should be split up. In one embodiment, the midpoint error is separately computed for the B-frames and P-frames. Figure 4A shows, in a flow diagram, an example of a process 400 for computing an estimated midpoint error for B-Frames that can be used with one or more embodiments of the invention. Figure 4A begins by getting the indices for the anchor B-frames at block 402. In one embodiment, process 400 determines the first B-frame, the last B-frames, and the midpoint B-frame. In one embodiment, the first and last B-frame can be the first and last B-frame in the window, although not necessarily the first and last frames in the window (e.g., the first and / or last frames are P-frames). At block 404, process 400 computes the video quality metric at each of the anchor frames.

[0041]

[0041] Process 400 estimates a midpoint video quality metric by interpolation. In one embodiment, process 400 interpolates between the video quality metric scores of the first B-frame and the last B-frame in the window. In this embodiment, process 400 uses linear interpolation to estimate the midpoint video quality metric, although other types of interpolations can be used. At block 408, process 400 calculates an estimate midpoint error by taking the difference of the calculated midpoint video quality metric (computed in block 404) and the estimated midpoint video quality metric score (computed at block 406).

[0042]

[0042] Figure 4B shows, in a flow diagram, an example of a process 450 for computing an estimated midpoint error for P-Frames that can be used with one or more embodiments of the invention. Figure 4B begins by getting the indices for the anchor P-frames at block 452. In one embodiment, process 450 determines the first P-frame, the last P-frames, and the midpoint P-frame. In one embodiment, the first and last B-frame can be the first and last P-frame in the window, although not necessarily the first and last frames in the window (e.g., the first and / or last frames are B-frames). At block 454, process 450 computes the video quality metric at each of the anchor frames.

[0043]

[0043] Process 450 estimates a midpoint video quality metric by interpolation. In one embodiment, process 450 interpolates between the video quality metric scores of the first P-frame and the last P-frame in the window. In this embodiment, process 450 uses linear interpolation to estimate the midpoint video quality metric, although other types of interpolations can be used. At block 458, process 450 calculates an estimate midpoint error by taking the difference of the calculated midpoint video quality metric (computed in block 454) and the estimated midpoint video quality metric score (computed at block 456).

[0044] Figure 5 shows an illustration 500 of computing a video quality metric 534 using a window and an estimated midpoint error over multiple video frames. In Figure 5, a plot of metric scores 502 versus video frames 504 is illustrated. The metric score is plotted (506). A window 508 of video frames is shown in Figure 5, where the first B-frame 516, last B- frame 518, first P-frame 520, are last P-frame 522 are identified for this window. Using the initial and last B-frames, the B-frame at midpoint 512 can be determined. Similarly, using the first and last P-frames, the midpoint P-frame 510 is determined. With the determined midpoint B-frame 512 and midpoint P-frames 510, the metric scores for each of these frames (510 and 512) can be explicitly computed. In addition, the estimated midpoint B-frame metric score 530 and the estimated midpoint P-frame metric score 532 can be computed using the metric scores of the first B-frame 516, last B-frame 518, first P-frame 520, are last P-frame 522. With these metric scores, the midpoint error can be computed for the B-frames 526 and P-frames 528. These midpoint errors (526 & 528) can be used to determine whether to split the window as described above in Figures 1 and 3.

[0044]

[0045] Figure 6 shows an illustration 600 of computing a video quality metric 622 using a window of multiple video frames. In Figure 6, a plot 606 of metric scores 602 versus video frames 604 is illustrated with a window 608 of video frames. In the window 608, there is a set of B-frames and a set of P-frames. The B-frame video quality metric scores tend to be lower than the P-frame video quality metric scores, so these sets of metric scores can be treated separately. If the window is of a sufficient size, the B-frame metric scores can be interpolated from the first 612 and last 618 B-frame video quality metric scores (614). Similarly, the P-frame metric scores can be interpolated from first 610 and last 616 B-frame video quality metric scores (614). The B-frame and P-frame metric scores can be used to compute the video quality metric score for the window as described above in Figures 1 and 3.

[0045]

[0046] Figure 7 shows an example comparing (700) an estimated VMAF score (708A-B) versus an actual VMAF score (706A-B) for a window size of 64 that can be used with one or more embodiments of the invention. In Figure 7, the P-frames and interpolation 710A plot and the B-frames and interpolation 710B illustrate the plot of the individual metric values (e.g., VMAF metric) (702A-B) versus frame number 704. These plots illustrate an estimated VMAF score that has an error of 0.0482 VMAF points, which is an error of less than 1%. A total of 204 samples was computed for 1499 frames in the inputted video. This resulted in a speedup of the computing the estimated VMAF score of 7.35 times.

[0046]

[0047] Figure 8 shows an example comparing (800) an estimated VMAF score (808A-B) versus an actual VMAF score (806 A-B) for a window size of 100 that can be used with one or more embodiments of the invention. In Figure 8, the P-frames and interpolation 810A plot and the B-frames and interpolation 810B illustrate the plot of the individual metric values (e.g., VMAF metric) (802A-B) versus frame number 804. These plots illustrate an estimated VMAF score that has an error of 0.2465 VMAF points, which is an error of less than 1 %. A total of 132 samples were computed for 1499 frames in the inputted video. This resulted in a speedup of the computing the estimated VMAF score of 1 1.36 times.

[0047]

[0048] Figure 9 shows an example comparing (900) an estimated VMAF score (908A-B) versus an actual VMAF score (906A-B) for a window size of 300 that can be used with one or more embodiments of the invention. In Figure 9, the P-frames and interpolation 910A plot and the B-frames and interpolation 910B illustrate the plot of the individual metric values (e.g., VMAF metric) (902A-B) versus frame number 904. These plots illustrate an estimated VMAF score that has an error of 0.6649 VMAF points, which is also an error of less than 1%. A total of 66 samples were computed for 1499 frames in the inputted video. This resulted in a speedup of the computing the estimated VMAF score of 22.71 times.

[0048]

[0049] Figure 10 shows an example of a data processing system 1000 that can be used by or in a camera or other device to provide one or more embodiments described herein. The systems and methods described herein can be implemented in a variety of different data processing systems and devices, including general-purpose computer systems, special purpose computer systems, or a hybrid of general purpose and special purpose computer systems. Data processing systems that can use any one of the methods described herein include a camera, a smartphone, a set top box, a computer, such as a laptop or tablet computer, embedded devices, game systems, and consumer electronic devices, etc., or other electronic devices.

[0049]

[0050] Figure 10 is a block diagram of data processing system 1000 hardware according to an embodiment. Note that while Figure 10 illustrates the various components of a data processing system, it is not intended to represent any particular architecture or manner of interconnecting the components as such details are not germane to the present invention. It will also be appreciated that other types of data processing systems that have fewer components than shown or more components than shown in Figure 10 can also be used with one or more embodiments of the present invention.

[0050]

[0051] As shown in Figure 10, the data processing system 1000 includes one or more buses 1009 that serve to interconnect the various components of the system. The system in Figure 10 can include a camera or be coupled to a camera. One or more processing devices 1003 are coupled to the one or more buses 1009 as is known in the art. Memory 1005 may be DRAM or non-volatile RAM or may be flash memory or other types of memory or a combination of such memory devices. This memory is coupled to the one or more buses 1009 using techniques known in the art. The data processing system can also include non-volatile memory 1007, which may be a hard disk drive or a flash memory or a magnetic optical drive or magnetic memory or an optical drive or other types of memory systems that maintain data even after power is removed from the system. The non-volatile memory 1007 and the memory 1005 are both coupled to the one or more buses 1009 using known interfaces and connection techniques. A display controller 1021 is coupled to the one or more buses 1009 in order to receive display data to be displayed on a display device which can be one of displays. The data processing system 1000 can also include one or more input / output (1 / 0) controllers 1015 which provide interfaces for one or more 1 / 0 devices, such as one or more cameras, touch screens, ambient light sensors, and other input devices including those known in the art and output devices (e.g., speakers). The input / output devices 1017 are coupled through one or more 1 / 0 controllers 1015 as is known in the art. The ambient light sensors can be integrated into the system in Figure 10.

[0051]

[0052] While Figure 10 shows that the non-volatile memory 1007 and the memory 1005 are coupled to the one or more buses directly rather than through a network interface, it will be appreciated that the present invention can utilize non-volatile memory that is remote from the system, such as a network storage device which is coupled to the data processing system through a network interface such as a modem or Ethernet interface. The buses 1009 can be connected to each other through various bridges, controllers and / or adapters as is well known in the art. In one embodiment the 1 / 0 controller 1010 includes one or more of a USB (Universal Serial Bus) adapter for controlling USB peripherals, an IEEE 1394 controller for IEEE 1394 compliant peripherals, or a Thunderbolt controller for controlling Thunderbolt peripherals. In one embodiment, one or more network device(s) 1025 can be coupled to the bus(es) 1009. The network device(s) 1025 can be wired network devices (e.g., Ethernet) or wireless network devices (e.g., Wi-Fi, Bluetooth) that receive images from a camera, etc.

[0052]

[0053] Although separate embodiments are enumerated below, it will be appreciated that these embodiments can be combined or modified, in whole or in part, into various different combinations. The combinations of these embodiments can be any one of all possible combinations of the separate embodiments.

[0053]

[0054] Embodiment 1 is a method comprising: receiving a video clip, wherein the video clip includes a compressed video clip and an uncompressed video clip; dividing the video clip into a plurality of windows, wherein each of the windows includes a plurality of video frames; for each of the plurality of windows, computing an estimated video quality error using a set of video frames selected from that window, if the estimated video quality error is greater than a threshold, dividing that window into a set of smaller windows and adding the set of smaller windows to the plurality of windows, and if the estimated video quality error is less than the threshold, computing an estimated video quality score for that window; determining a video quality score for the video clip from the plurality of estimated video quality scores.

[0054]

[0055] Embodiment 2 is a method of embodiment 1 wherein the plurality of windows are sequential temporal windows.

[0056] Embodiment 3 is a method of embodiment 1 or 2, wherein the plurality of windows is divided into initial equal sized windows.

[0055]

[0057] Embodiment 4 is a method of any previous embodiment, further comprising: determining a set of frame types for the video.

[0056]

[0058] Embodiment 5 is a method of embodiment 4, further comprising: determining a first video frame, midpoint video frame, and last video frame for each of the windows and each of the set of frame types.

[0057]

[0059] Embodiment 6 is a method of embodiment 5, wherein the computing of the estimated video quality error for a window comprises: computing a video quality metric at the first video frame, midpoint video frame, and last video frame for each of the set of frame types; estimating a midpoint video quality metric via interpolation; and calculating the estimated video quality error of each of the frame types using computed midpoint video frame video quality metric and the estimated midpoint video quality metric; and calculating an estimated video quality.

[0058]

[0060] Embodiment 7 is a method of any previous embodiment, wherein the video quality score is a weighted average of the plurality of estimated video quality scores.

[0059]

[0061] Embodiment 8 is a method of any preceding embodiment, wherein the video quality metric is one of Video Multimethod Assessment Fusion (VMAF), Peak signal-to- noise ratio (PSNR), Peak signal-to-noise ratio for human visual system (PSNR-HVS), or Structural similarity index measure (SSIM).

[0060]

[0062] Embodiment 9 is a method of any preceding embodiment, wherein the dividing of the window comprises: splitting the window into two sub-widows.

[0063] Embodiment 10 is a method of embodiment 9, wherein the two sub- windows have equal size.

[0061]

[0064] Embodiment 11 is a method of any preceding embodiment, wherein computing the estimated video quality score for the window comprises: interpolating a video quality metric score for each frame in the window; computing an estimated video quality metric score for each type of frame in the windows; and determining the estimated video quality score for the window from the estimated video quality metric score for each type of frame.

[0062]

[0065] Embodiment 12 is a method of embodiment 11, wherein the determining comprises: computing a weighted average using the estimated video quality metric score for each type of frame.

[0063]

[0066] Embodiment 13 is an apparatus comprising a processing system and memory and configured to perform any one of the methods in embodiments 1-12.

[0064]

[0067] Embodiment 14 is a non- transitory machine-readable storage storing executable program instructions which when executed by a machine cause the machine to perform any one of the methods of embodiments 1-12.

[0065]

[0068] It will be apparent from this description that one or more embodiments of the present invention may be embodied, at least in part, in software. That is, the techniques may be carried out in a data processing system in response to its one or more processors executing a sequence of instructions contained in a storage medium, such as a non- transitory machine- readable storage medium (e.g., DRAM or flash memory). In various embodiments, hardwired circuitry may be used in combination with software instructions to implement the present invention. Thus, the techniques are not limited to any specific combination of hardware circuitry and software, or to any particular source for the instructions executed by the data processing system.

[0069] In the foregoing specification, specific exemplary embodiments have been described. It will be evident that various modifications may be made to those embodiments without departing from the broader spirit and scope set forth in the following claims. The specification and drawings are, accordingly, to be regarded in an illustrative sense rather than a restrictive sense.

Claims

CLAIMS1. A method comprising: receiving a video clip, wherein the video clip includes a compressed video clip and an uncompressed video clip; dividing the video clip into a plurality of windows, wherein each of the windows includes a plurality of video frames; for each of the plurality of windows, computing an estimated video quality error using a set of video frames selected from that window, if the estimated video quality error is greater than a threshold, dividing that window into a set of smaller windows and adding the set of smaller windows to the plurality of windows, and if the estimated video quality error is less than the threshold, computing an estimated video quality score for that window; determining a video quality score for the video clip from the plurality of estimated video quality scores.

2. The method of claim 1 , wherein the plurality of windows are sequential temporal windows.

3. The method of claim 1 or 2, wherein the plurality of windows is divided into initial equal sized windows.

4. The method of any preceding claim, further comprising: determining a set of frame types for the video.

5. The method of claim 4, further comprising:determining a first video frame, midpoint video frame, and last video frame for each of the windows and each of the set of frame types.

6. The method of claim 5, wherein the computing of the estimated video quality error for a window comprises: computing a video quality metric at the first video frame, midpoint video frame, and last video frame for each of the set of frame types; estimating a midpoint video quality metric via interpolation between the first video frame and the last video frame; and calculating the estimated video quality error of each of the frame types using computed midpoint video frame video quality metric and the estimated midpoint video quality metric; and calculating an estimated video quality.

7. The method of any preceding claim, wherein the computing of the estimated video quality error for a window comprises: computing the estimated video quality error for B-frames and P-frames.

8. The method of any preceding claim, wherein the video quality score is a weighted average of the plurality of estimated video quality scores.

9. The method of any preceding claim, wherein the video quality metric is one of Video Multimethod Assessment Fusion (VMAF), Peak signal-to-noise ratio (PSNR), Peak signal-to-noise ratio for human visual system (PSNR-HVS), or Structural similarity index measure (SSIM).

10. The method of any preceding claim, wherein the dividing of the window comprises: splitting the window into two sub-windows.

11. The method of claim 10, wherein the two sub-windows have equal size.

12. The method of any preceding claim, wherein computing the estimated video quality score for the window comprises: interpolating a video quality metric score for each frame in the window; computing an estimated video quality metric score for each type of frame in the windows; and determining the estimated video quality score for the window from the estimated video quality metric score for each type of frame.

13. The method of claim 12, wherein the determining comprises: computing a weighted average using the estimated video quality metric score for each type of frame.

14. The method of any preceding claim, wherein the selected frames are frames of the compressed video clip; wherein the estimated video quality error is computed as an error of an estimated video quality score for at least one of the selected video frames; and wherein the estimated video quality score for the window is computed based on the set of video frames selected from that window.

15. The method of claim 14, wherein computing the estimated video quality error for a window comprises: computing a video quality metric score at each of a first anchor video frame, a second anchor video frame and a midpoint video frame between the first and second anchor video frames; estimating a midpoint video quality metric via interpolation between the first anchor video frame and the second anchor video frame;calculating the estimated video quality error of the window using the computed video quality metric at the midpoint video frame and the estimated midpoint video quality metric.

16. The method of claim 15, wherein the estimated video quality score for the window is computed as an average video quality metric score based on estimated video quality metric scores obtained for a set of video frames of the window by interpolating between the first anchor video frame and the second anchor video frame.

17. An apparatus comprising a processing system and memory and configured to perform any one of the methods in claims 1-16.

18. A non-transitory machine-readable storage storing executable program instructions which when executed by a machine cause the machine to perform any one of the methods of claims 1-16.

Images