Enhanced real-time visual quality metric generation for video coding

Abstract

The invention relates to enhanced real-time visual quality metric generation for video coding. This disclosure describes systems, methods, and devices related to generating visual quality metrics for encoded video frames. A method includes generating respective first visual quality metrics for pixels of an encoded video frame; generating respective second visual quality metrics for the pixels, the respective first visual quality metrics and the respective second visual quality metrics indicating an estimated human perception of the encoded video frame; generating a pixel block-based weight for each of the first visual quality metrics; generating a frame-based weight for each of the second visual quality metrics; and generating a human visual score indicative of the visual quality of the encoded video frame based on the respective first visual quality metrics, based on the weights of the pixel blocks, the respective second visual quality metrics, and based on the weights of the frames.

Description

TECHNICAL FIELD

[0001] The present disclosure relates generally to systems and methods for video encoding, and more particularly to real-time visual quality metrics for video encoding. BACKGROUND

[0002] Video encoding can be a lossy process that sometimes results in a decrease in quality when compared to the original source video. Video encoding standards are being developed to improve video quality. SUMMARY

[0003] According to an aspect of the present disclosure, an apparatus for generating a real-time visual quality metric for video encoding is provided, the apparatus comprising a storage device coupled with a graphics processing circuit, the graphics processing circuit configured to generate individual first visual quality metrics for pixels of a first encoded video frame, generate individual second visual quality metrics for the pixels, the individual first visual quality metrics and the individual second visual quality metrics indicative of an estimated human perception of the first encoded video frame, generate pixel block-based weights for the individual first visual quality metrics, generate frame-based weights for the individual second visual quality metrics, and generate a human visual score indicative of a visual quality of the first encoded video frame based on the individual first visual quality metrics, the pixel block-based weights, the individual second visual quality metrics, the frame-based weights, and a visual metric proportional to a temporal redundancy between the first encoded video frame and a second encoded video frame.

[0004] According to an aspect of the present disclosure, a non-transitory computer readable medium storing computer executable instructions that, when executed by one or more graphics processors, cause performance of operations comprising generating individual first visual quality metrics for pixels of a first encoded video frame, generating individual second visual quality metrics for the pixels, the individual first visual quality metrics and the individual second visual quality metrics indicative of an estimated human perception of the first encoded video frame, generating pixel block-based weights for the individual first visual quality metrics, generating frame-based weights for the individual second visual quality metrics, and generating a human visual score indicative of a visual quality of the first encoded video frame based on the individual first visual quality metrics, the pixel block-based weights, the individual second visual quality metrics, the frame-based weights, and a visual metric proportional to a temporal redundancy between the first encoded video frame and a second encoded video frame.

[0005] According to an aspect of the disclosure, a method for generating real-time visual quality metrics for video encoding is provided, including generating, by a graphics processing circuit of a device, individual first visual quality metrics for pixels of a first encoded video frame, generating, by the graphics processing circuit, individual second visual quality metrics for the pixels, the individual first visual quality metrics and the individual second visual quality metrics indicative of an estimated human perception of the first encoded video frame, generating, by the graphics processing circuit, pixel block-based weights for the individual first visual quality metrics, generating, by the graphics processing circuit, frame-based weights for the individual second visual quality metrics, and generating, by the graphics processing circuit, a human visual score indicative of a visual quality of the first encoded video frame based on the individual first visual quality metrics, the pixel block-based weights, the individual second visual quality metrics, the frame-based weights, and a visual metric proportional to a temporal redundancy between the first encoded video frame and a second encoded video frame. BRIEF DESCRIPTION OF DRAWINGS

[0006] Figure 1 An example system illustrating components of an encoding and decoding device is depicted in accordance with some example embodiments of the disclosure.

[0007] Figure 2 Exemplary components of a video box for video encoding are depicted in accordance with one or more example embodiments of the disclosure.

[0008] Figure 3 Exemplary components of a video box of Figure 2 are depicted in accordance with one or more example embodiments of the disclosure.

[0009] Figure 4 An example histogram for a structural similarity index measurement is shown in accordance with one or more example embodiments of the disclosure.

[0010] Figure 5 A "diagnostic" view of a frame using per-pixel block analysis in multiple layers is shown in accordance with one or more example embodiments of the disclosure.

[0011] Figure 6 A change chart for a human visual metric for evaluating encoded video frames is shown in accordance with one or more example embodiments of the disclosure.

[0012] Figure 7 Multiple layers of a machine learning model of Figure 1 are shown in accordance with one or more example embodiments of the disclosure.

[0013] Figure 8 Exemplary components of an encoder are depicted in accordance with one or more example embodiments of the disclosure.

[0014] Figure 9Exemplary components of an encoder for downscaling are depicted in accordance with one or more example embodiments of the present disclosure.

[0015] Figure 10 A flowchart of an illustrative process for enhanced real-time visual quality metric generation for video encoding is illustrated in accordance with one or more example embodiments of the present disclosure.

[0016] Figure 11 An embodiment of an exemplary system is illustrated in accordance with one or more example embodiments of the present disclosure. DETAILED DESCRIPTION

[0017] The following description and drawings are illustrative of specific embodiments and are not intended to limit the scope of the disclosure. Other embodiments can include structural, logical, electrical, process, and other changes. Some embodiments can be included in, or alternative embodiments can be part of, one or more other embodiments. The embodiments described in the claims are not intended to be limited to the specific embodiments described herein, but encompass any embodiments including personal, social, economic, utilitarian, and / or other dimensions.

[0018] Visual quality assessment is critical for graphics and video applications. Video coding is interested in the ability to score the perceived human response to video that has been encoded with lossy compression. For example, the way an automated system assesses the quality of an encoded video frame can reflect the way a human viewer can perceive the quality of the video frame. Some existing techniques that better link visual quality assessment to the human visual system improve video coding, but have significant limitations and are not very efficient.

[0019] Peak signal to noise ratio (PSNR) and structural similarity index measurement (SSIM) are two quality metrics that evaluate visual impairments (e.g., caused by video compression) of encoded video frames. PSNR does not attempt to model any particular type of visual impairment, but rather provides a simple mathematical model based on mean squared error (difference) of video images. SSIM improves upon PSNR by considering luminance, contrast, and structure as independent impairment types, and combines each together for a composite score. Multi-scale SSIM (MS-SSIM) improves upon SSIM by computing SSIM metrics at multiple down-scaled resolutions (e.g., encoding layers), and combining them as a weighted product to mimic that the human eye cannot see artifacts at full resolution. However, artifacts that remain even after down-scaling can be more perceptible to humans than to computers. Recent techniques, such as video multimethod assessment fusion (VMAF), utilize supervised machine learning to combine multiple metrics together.

[0020] However, some existing methods for metrics that score perceptual human responses are complex and consume large software cycles, thus representing a performance overhead, which either limits them to offline video encoding, or reduces real-time resolution to meet real-time requirements. Furthermore, the way that automated systems evaluate the quality of encoded video frames can not reflect the way that a human viewer can perceive frame quality. For example, a human viewer can notice poor quality of a single pixel, while some automated systems and the visual quality metrics that they use can determine that a block of pixels with a single poor pixel is a high quality block of pixels (e.g., due to averaging the visual quality metric of the pixels in a given block of pixels).

[0021] While PSNR calculation is straightforward and often roughly correlates with subjective visual scores, many different types of impairment can result in the same PSNR score, yet each type of impairment can produce a different subjective human score. SSIM can identify different types of impairment that a user can observe, improving its ability to correlate with user scores, but it requires an order of magnitude more computation to generate than PSNR and performs only slightly better than PSNR alone. SSIM tends to overemphasize fine details imperceptible to humans. MS-SSIM uses approximately two to three times more computation than SSIM because it computes SSIM on multiple levels of downscaled video and further amplifies its correlation with subjective video. A drawback of MS-SSIM is the computational overhead required for its generation, and MS-SSIM is often computed by software running in a central processing unit, which cannot keep up with real-time hardware encoding of high-definition video and higher resolutions. Furthermore, while MS-SSIM is more accurate than both SSIM and PSNR, it still falls short in measuring impairment scores in the human visual system.

[0022] Therefore, there is a need to generate visual quality metrics associated with subjective scores more efficiently than existing methods alone, without the overhead of software post-processing of the encoded video to evaluate the video by generating scores during hardware video encoding (e.g., in parallel with encoding). A control feedback loop, such as bitrate control (BRC), running rapidly in nearby firmware, can directly compare the number of bits spent encoding a frame with the viewer's approximate visual impairment and determine whether the user's requirements are met without subsequent re-encoding.

[0023] In one or more embodiments, the VMAF method significantly improves the accuracy of perceived human response video ratings (e.g., Human Visual System—HVS—ratings) compared to conventional methods because VMAF is trained using scores from human viewers. The encoding engine of the graphics processing unit can compute metrics at the per-pixel level in parallel with video frame encoding and use these metrics as intermediate aggregations to detect the range and distribution of visual quality of frames, unlike relying on an arithmetic mean of metrics and unlike generating metrics in a central processing unit. For example, some pixels in a frame may have a high PSNR, while some pixels in another frame may have a low PSNR (or another visual quality metric). The way the arithmetic mean of the PSNR (or other visual quality metrics) of a frame is weighted may differ from how a human observer assigns weight to the lower PSNR portions of a frame. In this way, the distribution of visual quality of frames based on intermediate per-pixel metrics can enhance the quality metric analysis of encoded video frames. The pixel-level metric chosen by VMAF has a computational complexity on par with MS-SSIM. If the metric is determined in software, this limits performance during real-time encoding operations, just as MS-SSIM is limited to offline applications or those with high software computational overhead. These techniques also increase latency, which may be infeasible for ultra-low latency scenarios (e.g., low-latency applications).

[0024] In one or more embodiments, a fixed-function encoder may be able to access the original, unmodified source video and the resulting encoded output images. This disclosure allows for the addition of dedicated hardware logic (e.g., added to a graphics processing unit) to compute visual impairment metrics on the fly without additional memory reads or increased latency for the user. Furthermore, numerous different metrics can be computed, allowing for post-processing to blend these metrics in a manner similar to VMAF performance. Accurate scoring of the video's subjective quality unlocks further compression, which can be used to produce smaller videos of the same subjective quality, or videos of higher subjective quality at the same size. This disclosure provides this enhancement with negligible overhead in terms of power, latency, or performance by computing metrics within the encoder itself during encoding (e.g., rather than externally). The hardware can aggregate metric data in novel ways that leverage insights possessed by the encoder that are not always observable through external quality calculations. Such combinations of hardware metrics and their aggregation have been shown to have accuracy similar to VMAF.

[0025] In one or more embodiments, advanced quality metrics (AQM) for video boxes (VDBOX) can be unified across codecs that support low-power coding within the VDBOX (e.g., the low-power coding path of the encoder's VDBOX, referred to as VDENC) and / or packetization (PAK) (e.g., quantization, entropy coding, pixel reconstruction, and motion compensation), as well as on-the-fly / online metric generation of key objective quality metrics used during encoding. The VDBOX AQM (VDDAQM) can reside within the VDBOX of the graphics processing unit, and all PAKs (i.e., all past, present, and future hardware codecs) can contribute source pixels and reconstructed pixels to the VDDAQM. Furthermore, this disclosure provides a "standalone" mode for independent access to the VDAQM, bypassing PAKs, to support image comparisons outside of VDENC+PAK usage. These metrics can be part of a feedback loop to BRC and advanced constant quantization parameter (CQP) and / or quality-defined variable bitrate (QVBR) kernels to adjust encoder settings in real time. Furthermore, these metrics can be used in silicon-based quality parameter training consistent with specific content types. Overall, these metrics enable machine learning-optimized encoding, as there are ways to minimize bits, but may require quantizing visual quality to optimize them together. Reported metrics can be frame-based summaries (e.g., totals), region / category-based, or block-based surfaces. VDAQM can support PSNR, SSIM, and MS-SSIM, and can include many more metrics.

[0026] In one or more embodiments, VDAQM can be codec-agnostic, avoiding the need for additional separate gates for encoding. VDAQM can use high-level metrics, such as MS-SSIM, and can aggregate the analysis results of metrics using frame-based reporting, category-based reporting, and / or mapping.

[0027] In one or more embodiments, machine learning can utilize models with multiple layers, such as multilayer perceptrons, support vector machines (SVMs), random forests, etc. (e.g., linear regression with machine learning). The machine learning model can receive visual features generated by VDAQM (e.g., as shown in Table 1 below) and can use multiple layers to generate scores for visual quality metrics (e.g., HVS scores) (e.g., motion scores). These layers can be trained using human responses as training data. For example, training data may include human viewer scores representing visual quality metrics, such as PSNR, SSIM, etc. Based on the metrics generated by VDAQM and the human training data, the machine learning model can generate scores that will be used by the encoder / decoder (e.g., for the selection of encoding parameters).

[0028] In one or more embodiments, VDAQM may include: 1) unifying all encoder quality metrics in a single unit that can be shared across all PAKs, and generating metrics "on the fly" without round trips to memory or slowing down VDENC or PAKs; 2) extending beyond PSNR metrics, starting with SSIM and MS-SSIM, to support metrics that more closely track the human visual system (HVS); and 3) aggregating statistics in a useful manner to reduce computational overhead by summarizing quality data with per-frame SSIM histograms, per-class minimum and average SSIMs (e.g., where each class can be generally defined based on application needs), and per-block reports, allowing for targeted quantization parameter (QP) tuning at the coding unit (CU) or macroblock (MB) level.

[0029] In one or more embodiments, dedicated encoder hardware can be used to encode video and compute coding metrics in parallel (e.g., using VDENC). The dedicated encoder hardware can receive source video and encode it into a bitstream. The inputs to the dedicated encoder hardware may include the source video and a decoded view (e.g., a view of the encoded video that the decoder will see). In this way, the encoder can include VDENC, VDAQM, and a high-efficiency video coding (HEVC) / H.265 controller (HuC) engine on the same hardware, resulting in a noticeable performance degradation because the metrics are generated internally to the encoder rather than remotely (e.g., away from the encoder).

[0030] In one or more embodiments, VDAQM can be an engine (e.g., a computer program) that determines PSNR, SSIM, and MS-SSIM metrics for reconstructed images (e.g., video frames). The VDAQM engine can operate in parallel with other encoder engines, such as multi-format codecs (MFX), HCP, AVP, VDENC, and HuC pipelines, and operate on real-time reconstructed pixels for AVC, HEVC, AV1, and other codecs. The VDAQM engine can operate in a standalone mode, allowing it to operate even when other PAK engines are disabled.

[0031] In one or more embodiments, the VDAQM engine can aggregate coding metrics from a neural network to generate a score (e.g., an HVS score) that is not simply an average per-pixel score for a single frame of video. Specifically, the score can use intermediate per-pixel data instead of an average score for a single frame. Machine learning from the neural network can identify relationships between metrics of inter-coded and intra-coded frames (e.g., inter-coded metrics may be more important than intra-coded metrics, or vice versa). For example, machine learning can use feature regression or neural network visual analysis. The VDAQM engine can provide a feedback loop that writes to memory as encoding occurs. The VDAQM engine's reporting of metrics can include region- or category-based reports, or mapping (e.g., heatmap) reports.

[0032] In one or more embodiments, the VDAQM engine can use coding metrics in an enhanced manner. For example, coding metrics may include averages, such as the PSNR and sum of squared errors (SSE) of the Y′UV color model (e.g., Y′ is luminance, and U and V are chrominance values), an SSIM histogram, minimum and average values ​​for luminance and chrominance per category, block occurrence counts per category, and minimum SSIM for each block. MS-SSIM can be determined by aggregating the SSIMs of the original layer and the downscaled layer. SSIM may include luminance (L), contrast (C), and structure (S) components. The exponents used in MS-SSIM may not be supported by the hardware, so software can combine SSIM results to generate MS-SSIM values. Hardware can capture intermediate terms that allow for post-processing to determine the overall MS-SSIM using the following formula (1):

[0033]

[0034] Where I and J are the two images to be compared, M is the total number of layers, and i, which is equal to 1, is the original image resolution. The SSIM of the i-th layer is defined as L. i *C i *Si β, γ, and α are constants that can vary, and examples of β, γ, and α are further provided in this paper. M c i and s i The terms refer to the L, C, and S terms of the SSIM metric. i and s i The product is determined per layer i. β, γ, and α represent weighting factors for each layer, where the weighting factors indicate the importance of the SSIM value per layer. For example, machine learning can be used to adjust the weights based on whether a human viewer is more likely or less likely to notice artifacts in lower or higher coding layers.

[0035] The above description is for illustrative purposes only and is not intended to be limiting. Many other examples, configurations, processes, algorithms, etc., may exist, some of which are described in more detail below. Example embodiments will now be described with reference to the accompanying drawings.

[0036] Figure 1 This is an example system 100 illustrating components of an encoding and decoding device according to some example embodiments of the present disclosure.

[0037] refer to Figure 1System 100 may include device 102 having encoder and / or decoder components. As shown, device 102 may include content source 103 (e.g., camera or other image capture device, stored images / videos, etc.) providing video and / or audio content. Content source 103 may provide media (e.g., video and / or audio) to partitioner 104, which may prepare frames of the content for encoding. Subtractor 106 may generate residuals, as further illustrated herein. Transformer and quantizer 108 may generate and quantize transform units to facilitate encoding performed by encoder 110 (e.g., entropy encoder). Transformed and quantized data may be inversely transformed and inversely quantized by inverse transformer and quantizer 112. Adder 114 may compare the inversely transformed and inversely quantized data with prediction blocks generated by prediction unit 116 to obtain reconstructed frames. Filter 118 (e.g., an in-loop filter for resizing / cropping, color conversion, deinterleaving, compositing / blending, etc.) can modify the reconstructed frame from adder 114 and can store the reconstructed frame in image buffer 120 for use by prediction unit 116. Control 121 can manage many coding aspects (e.g., parameters), including at least the setting of quantization parameters (QP), but may also include setting bit rate, rate distortion or scene characteristics, prediction and / or transform partition or block size, available prediction mode types, and optimal mode selection parameters, for example, at least in part based on data from prediction unit 116. Utilizing the coding aspects, transform and quantizer 108 can generate and quantize transform units to facilitate encoding performed by encoder 110, which can generate transmissible encoded data 122 (e.g., an encoded bitstream).

[0038] Still referencing Figure 1 Device 102 can receive encoded data (e.g., encoded data 122) from a bitstream, and decoder 130 can decode the encoded data to extract quantized residual coefficients and context data. Inverse transform and quantizer 132 can reconstruct pixel data based on the quantized residual coefficients and context data. Adder 134 can add the residual pixel data to the prediction block generated by prediction unit 136. Filter 138 can filter the result data from adder 134. The filtered data can be output by media output 140 or stored as a reconstructed frame in image buffer 142 for use by prediction unit 136.

[0039] refer to Figure 1System 100 performs the intra-prediction methods disclosed herein and is configured to perform at least one or more implementations of the methods described herein, including intra-block copying. In various implementations, system 100 may be configured to perform video coding and / or implement video codecs according to one or more standards. Additionally, in various forms, video coding system 100 may be implemented as part of an image processor, video processor, and / or media processor and perform inter-prediction, intra-prediction, predictive coding, and residual prediction. In various implementations, system 100 may perform video compression and decompression and / or implement video codecs according to one or more standards or specifications, such as H.264 (Advanced Video Coding, or AVC), VP8, H.265 (High Efficiency Video Coding, or HEVC) and its SCC extension, VP9, ​​Alliance OpenMedia Version 1 (AV1), H.266 (Universal Video Coding, or VVC), DASH (Dynamic Adaptive Streaming over HTTP), and others. While this document may describe System 100 and / or other systems, schemes or processes, this disclosure is not necessarily always limited to any particular video coding standard or specification or its extensions, except for the IBC prediction mode operation mentioned herein.

[0040] Still referencing Figure 1 System 100 may include a machine learning model 150 for evaluating a visual quality metric 152 generated by device 102. The machine learning model may receive the visual quality metric 152 and human training data 154 as input and may generate an HVS score (e.g., a motion score 156) based on the input. In one or more embodiments, the machine learning model 150 may be a multilayer perceptron neural network model, as described above. Figure 7 Further description: Device 102 can use a score of 156 to select encoding parameters.

[0041] As used herein, the term "encoder" can refer to both an encoder and / or a decoder. Similarly, as used herein, the term "encoding" can refer to encoding via an encoder and / or decoding via a decoder. An encoder or decoder may have components of both an encoder and a decoder. An encoder may have a decoder loop as described below.

[0042] For example, system 100 may be an encoder in which current video information, in the form of data relating to a sequence of video frames, can be received for compression. In one form, the video sequence (e.g., from content source 103) is formed from input frames of synthesized screen content, such as content from or relating to business applications like word processors, presentations, or spreadsheets, computers, video games, virtual reality images, etc. In another form, the images may be formed from a combination of synthesized screen content and images captured by a natural camera. In yet another form, the video sequence may simply be video captured by a natural camera. Partitioner 104 may partition each frame into smaller, more manageable units and then compare the frames to compute a prediction. If a difference or residual is determined between the original block and the prediction, the resulting residual is transformed and quantized, then entropy-encoded and sent out as a bitstream along with the reconstructed frames to a decoder or storage device. To perform these operations, system 100 may receive input frames from content source 103. The input frames may be frames that have been sufficiently preprocessed for encoding.

[0043] System 100 can also manage many coding aspects, including at least the setting of quantization parameters (QP), but may also include, for example, setting bit rate, rate distortion or scene characteristics, prediction and / or transformation partition or block size, available prediction mode types, and optimal mode selection parameters.

[0044] The output of transform and quantizer 108 can be fed to inverse transform and quantizer 112 to generate the same reference or reconstructed block, frame, or other unit that would be generated at a decoder such as decoder 130. Thus, prediction unit 116 can use inverse transform and quantizer 112, adder 114, and filter 118 to reconstruct frames.

[0045] Prediction unit 116 can perform inter-prediction including motion estimation and motion compensation, intra-prediction as described herein, and / or combined inter-intra-prediction. Prediction unit 116 can select the optimal prediction mode (including intra-prediction mode) for a given block, typically based on bit cost and other factors. Prediction unit 116 can select intra-prediction and / or inter-prediction modes when multiple such modes are available for each. The prediction output of prediction unit 116, taken in block form, is then provided to subtractor 106 to generate residuals, and also in the decoding loop to add the predictions to the reconstructed residuals from the inverse transform to reconstruct frames.

[0046] Partitioner 104 or other initial units (not shown) can sequentially place frames for encoding and assign them classifications, such as I-frames, B-frames, P-frames, etc., where I-frames are intra-predicted. Alternatively, frames can be divided into slices (e.g., I-slices), where each slice can be predicted differently. Thus, for HEVC or AV1 encoding of an entire I-frame or I-slice, spatial or intra-prediction is used, and in one form, only from data within the frame itself.

[0047] In various implementations, prediction unit 116 can perform intra-block copy (IBC) prediction mode, and non-IBC mode can operate any other available intra-prediction mode, such as neighbor-level, diagonal, or direct coding (DC) prediction mode, palette mode, orientation or angle mode, and any other available intra-prediction mode. Other video coding standards, such as HEVC or VP9, ​​may have different sub-block sizes, but the IBC search disclosed herein can still be used. However, it should be noted that the above are merely example partition sizes and shapes, and this disclosure is not limited to any particular partition and partition shape and / or size, unless such a limitation is mentioned or the context implies such a limitation, such as mentioning an optional maximum efficiency size. It should be noted that multiple alternative partitions can be provided as prediction candidates for the same image region, as described below.

[0048] Prediction unit 116 can select previously decoded reference blocks. A comparison can then be performed to determine if any reference block matches the current block being reconstructed. This may involve hash matching, SAD search, or other comparisons of image data, etc. Once a match is found for a reference block, prediction unit 116 can use image data from one or more matching reference blocks to select a prediction mode. In one form, previously reconstructed image data of the reference block is provided as the prediction, but alternatively, the original pixel image data of the reference block can be provided instead. Either selection can be used regardless of the type of image data previously used to match the block.

[0049] The predicted block can then be subtracted from the current block of the original image data at subtractor 106, and the resulting residual can be partitioned into one or more transform blocks (TUs) so that transform and quantizer 108 can, for example, use a discrete cosine transform (DCT) to transform the partitioned residual data into transform coefficients. Using the quantization parameters (QP) set by system 100, transform and quantizer 108 then performs lossy resampling or quantization on the coefficients. Frames and residuals, as well as support or context data block sizes and intra-shift vectors, etc., can be entropy-encoded by encoder 110 and transmitted to decoder.

[0050] In one or more embodiments, system 100 may have a decoder or may be a decoder and may receive encoded video data in the form of a bitstream and having image data (chroma and luminance pixel values) and context data, which includes residuals in the form of quantized transform coefficients and the identity of a reference block, including, for example, at least the size of the reference block. The context may also include prediction patterns for individual blocks or other partitions such as slices, inter-predictive motion vectors, partitions, quantization parameters, filter information, etc. System 100 may use entropy decoder 130 to process the bitstream to extract the quantized residual coefficients and context data. System 100 may then use inverse transform and quantizer 132 to reconstruct the residual pixel data.

[0051] System 100 can then use adder 134 (and assembler, not shown) to add the residual to the prediction block. System 100 can also decode the obtained data using decoding techniques depending on the encoding mode indicated in the bitstream syntax, and either a first path including prediction unit 136 or a second path including filter 138. Prediction unit 136 performs intra-prediction using a reference block size and intra-displacement or motion vector extracted from the bitstream and previously established at the encoder. Prediction unit 136 can reconstruct the prediction block using the reconstructed frames and inter-prediction motion vectors from the bitstream. Prediction unit 136 can set the correct prediction mode for each block, wherein the prediction mode can be extracted and decompressed from the compressed bitstream.

[0052] In one or more embodiments, the encoded data 122 may include both video and audio data. In this way, the system 100 can encode and decode both audio and video.

[0053] In one or more embodiments, while encoder 110 generates encoded data 122, system 100 may generate a coding quality metric indicating visual quality (e.g., post-processing of encoded data 122 is not required to evaluate visual quality). Evaluating the coding quality metric in parallel with the encoding performed by encoder 110 may allow control feedback, such as BRC (e.g., facilitated by control 121), to compare the number of bits spent encoding a frame with the coding quality metric. When one or more coding quality metrics indicate poor quality (e.g., failure to meet a threshold), recoding (e.g., using adjusted parameters) may be required. Coding quality metrics indicating visual quality may include PSNR, SSIM, MS-SSIM, VMAF, etc. Coding quality metrics may be based on a comparison between encoded video and source video. System 100 may compare a decoded version of encoded image data with a pre-encoded version of the image data. Using the CU or MB of the encoded image data and the pre-encoded version of the image data, system 100 may generate a coding quality metric, which can be used as metadata for the corresponding video frame. System 100 can use coding quality metrics to adjust coding parameters, for example, based on perceived human responses to the coded video. For instance, a lower SSIM might indicate more visible artifacts, which could result in less compression in subsequent coding parameters.

[0054] The above description is for illustrative purposes and is not intended to be restrictive.

[0055] Figure 2 Exemplary components of a video box (VDBOX) 202 for video encoding are depicted according to one or more example embodiments of this disclosure.

[0056] refer to Figure 2 VDBOX 202 can be referred to as a multi-format codec (MFX). This component may also include HuC 204 (e.g., representing...). Figure 1 The control unit 121 can receive user control 205 as input (e.g., inter-frame or intra-frame type, quantization parameters, frame header, reference picture selection, etc.) and can allow the graphics processing unit (e.g., see [link to relevant documentation]). Figure 11 Instead of a central processing unit (CPU) handling functions such as bit rate control and header parsing, this component can handle other tasks. These components may include the VDENC 206 pipeline, which can represent low-power coding paths, and dedicated hardware for computation and searching. For example, VDENC 206 can use motion search. This component may include the VDAQM 208 pipeline—a newly added feature for enhanced quality analysis—and support metrics such as PSNR, SSIM, and MS-SSIM. The HuC 204 can select coding parameters based on metric 210.

[0057] In one or more embodiments, the generated bitstream 212 can be enhanced due to metric 210 from VDAQM 208. For example, when any metric 210 is above or below its respective threshold, this can indicate whether the encoding parameters used in the frame from which the evaluation is the source of metric 210 should be adjusted (e.g., resulting in recoding using the adjusted parameters) and / or whether subsequent frames should be encoded using the adjusted parameters. Specifically, metric 210 generated by VDAQM 208 can be fed back to HuC 204, which can determine whether a frame should be recoded based on metric 210. In contrast, existing techniques may rely on metadata indicating whether a target frame size has been achieved, but such a determination does not indicate whether visual quality has been achieved, which is where VDAQM 208 and metric 210 improve upon existing techniques. In one or more embodiments, metric 210 can be fed into another machine learning model for further analysis (e.g., per-pixel analysis of the entire frame).

[0058] In one or more embodiments, the VDBOX 202 can perform bitstream decoding, intraprediction, motion estimation, quantization, entropy coding, pixel reconstruction, and motion compensation. In this way, the VDBOX 202 can represent information about... Figure 1 Several components are shown and described.

[0059] In one or more embodiments, metric 210 may be fed back to HuC 204 (e.g., for encoding parameter decision to be implemented at VDENC 206) and / or may be payload-transferred (e.g., Figure 5 (The diagnostic view shown).

[0060] In one or more embodiments, HuC 204 may represent an application-specific integrated circuit (ASIC), which allows, for example, logic gates to be used instead of software to determine and evaluate metrics. In this way, the generation and use of metric 210 can be performed "on-chip" for the graphics processor, rather than requiring the central processing unit to receive metric 210 and perform actions based on metric 210.

[0061] Figure 3 One or more exemplary embodiments of this disclosure are described. Figure 2 An exemplary component of the VDBOX 202 for encoder 200.

[0062] refer to Figure 3The encoder 200's VDBOX 202 may include a video command streamer (VCS) 302 for acquiring, decoding, and manipulating data. The VDBOX 202 may also include a VRT router 304. Figure 2 The VDENC 206 pipeline, HEVC codec pipeline (HCP) 308, AVP codec pipeline 310, MFX codec pipeline 312, Figure 2 The diagram shows the HuC 204 and VDAQM 208. As shown, VDAQM 208 can run in parallel with other pipelines and can operate on real-time reconstructed pixels for AVC, HEVC, AV1, and other codecs. VDAQM 208 can even operate when other pipelines / engines are not enabled. The illustrated pipelines / engines can communicate with memory 320 to share encoded metrics. In this way, pipelines / engines can share and generate metrics in real time without round trips to memory 320. Multiplexer 322 can be used to communicate metrics from pipelines / engines of VDBOX 202, and metrics can be fed back to VRT router 304. The VDENC 206 pipeline can generate encoded bitstreams and reconstructed images, which can be fed back (e.g., to VAQM 208) for use by other codec pipelines. VDAQM 208 can generate visual quality metrics based on coded bitstreams generated by the codec pipeline (including those generated by VDENC 206) and reconstructed images, and can write the visual quality metrics to memory 320. In one or more embodiments, the visual quality metrics can be generated by... Figure 1 The metric 152 is represented. VDAQM 208 can generate visual quality metrics in parallel with encoding.

[0063] In one or more embodiments, the video coding metrics of the VDBOX 202 pipeline may include at least some of the metrics shown in Table 1 below.

[0064] Table 1: Video Coding Metrics

[0065]

[0066]

[0067] The term "blk" can refer to a block of pixels.

[0068] As shown in Table 1, the metrics generated by VDAQM 208 (e.g., Figure 2The metrics (210) may include the PSNR for the entire frame, the SSIM for each layer, the SSIM histogram for the entire frame, the minimum SSIM for each category, the mean, and the pixel block occurrence count (e.g., "blk" in Table 1 could refer to a 4x4 pixel block or another size), and the minimum SSIM for each pixel block. These metrics can be generated based on a comparison between the coded frame and a reconstructed version of the frame. Since SSIM has five layers, the metrics can be used to determine which layers to retain or exclude. To save costs, some SSIM calculations may not include all five layers (e.g., layers 1-4 could be evaluated instead). For example, the mean (LY*CY*SY), mean (LU*CU*SY), mean (LV*CV*SV), and mean (CY*SY) in Table 1 could be intermediate SSIM values ​​on a layer-by-layer basis, and the aggregated SSIM could be determined based on these intermediate values. Because the PSNR is determined for layer 0, it is advantageous (e.g., a 4x computational relief) that SSIM for layer 0 may not be necessary, as SSIM for layer 0 is computationally expensive. In this way, SSIM can be used for some layers, and PSNR can be used to supplement some layers.

[0069] Referring to the histogram in Table 1, the histogram (e.g., in...) Figure 4 (As shown in the image) The SSIM can be determined for each pixel block. The lowest pixel SSIM of a block (e.g., a value between 0 and 1, where 0 is lower quality and 1 is higher quality) can represent the quality of that block (e.g., relative to the average SSIM of the block). A histogram counts the number of blocks falling into different histogram bins within a frame.

[0070] Referring to the metric for each category in Table 1, intra-coded blocks can be aggregated together, and inter-coded blocks can be aggregated together. Instead of using the average SSIM of all inter-coded blocks and the average SSIM of all intra-coded blocks, the mean and worst (e.g., lowest SSIM) blocks are used. For example, the average SSIM might be low, but the worst SSIM might not be very low, so there might not be any significant outliers with poor quality. Alternatively, the average SSIM might be average, but there might be low worst-case SSIM values ​​that the averaging will ignore.

[0071] Referring to Min(blkYL*C*S) in Table 1, this metric represents the value of each SSIM block (e.g., diagnostic view), such as... Figure 5 As shown.

[0072] In one or more embodiments, VDAQM 208 can compute metrics at the per-pixel level and use these metrics as intermediate aggregations to detect the range and distribution of visual quality of frames, in contrast to relying on the arithmetic mean of metrics. For example, half of the pixels in a frame may have a high PSNR, and the other half may have a low PSNR (or another metric). The way the arithmetic mean of the PSNR (or other metric) of a frame is weighted may differ from how a human observer weights the lower PSNR portion of a frame. In this way, the distribution of visual quality of frames based on intermediate per-pixel metrics can enhance the quality metric analysis of coded video frames and provide improvement over using the arithmetic mean of the entire frame.

[0073] In one or more embodiments, VDAQM 208 can be codec-agnostic, avoiding the need for additional separate gates for encoding. VDAQM 208 can use high-level metrics, such as MS-SSIM, and VDAQM 208 can aggregate the analysis results of metrics using frame-based reports, category-based reports, and / or mappings. Metric 210 can be... Figure 2 The HuC 204 evaluation is likely low-powered, rather than requiring a supercomputer or some other more complex processing unit for evaluation.

[0074] Figure 4 An example histogram 400 for measuring structural similarity index is shown according to one or more example embodiments of this disclosure.

[0075] refer to Figure 4 Histogram 400 can represent the SSIM histogram mentioned in Table 1 above. The histogram is generated at the global (e.g., frame) level based on the lowest SSIM value of any pixel in a pixel block, and histogram 400 shows how frame-level SSIM changes over time. For example, a lower SSIM might indicate more visible artifacts in a video frame. The per-frame SSIM histogram is important for providing a distribution of visual quality over a given frame. In contrast, relying on the average SSIM can be misleading because a small portion of a frame might be the part from which a human might judge the overall quality of the entire frame. Instead of simply showing a time-based map of the overall SSIM, histogram 400 shows the distribution of quality over time in more detail (e.g., SSIM for each compartment over time).

[0076] Still referencing Figure 4Histogram 400 illustrates multiple SSIM compartments: SSIM compartment 401, SSIM compartment 402, SSIM compartment 404, SSIM compartment 406, SSIM compartment 408, SSIM compartment 410, and SSIM compartment 412. The number of SSIM compartments is illustrative and not intended to be restrictive. As shown, the majority of a frame falls within SSIM compartment 401 (e.g., a certain percentage of the frame falls within SSIM compartment 401). For example, SSIM compartment 401 can be used for pixel blocks whose lowest SSIM value is above a maximum threshold. SSIM compartment 402 can be used for pixel blocks whose lowest SSIM value is above the next maximum threshold. SSIM compartment 404 can be used for pixel blocks whose lowest SSIM value is above the next maximum threshold. SSIM compartment 406 can be used for pixel blocks whose lowest SSIM value is above the next maximum threshold. SSIM compartment 408 can be used for pixel blocks whose lowest SSIM value is above the next maximum threshold. SSIM bins 410 can be used for pixel blocks whose lowest SSIM value is higher than the next highest threshold. SSIM bins 412 can be used for pixel blocks whose lowest SSIM value is higher than the next highest threshold. In this way, SSIM bins 410 and 412 can indicate pixel blocks with the lowest SSIM values, which represent a small portion of the frame, while the majority of pixel blocks in the frame fall within the higher-quality SSIM bins 401. This technique can be used to identify the worst pixel blocks by using the lowest SSIM pixel value of the pixel block instead of the average SSIM value of the pixel block. Therefore, histogram 400 represents a simplified way of reporting the visual quality distribution on the encoded frame. Histogram 400 can be used to generate scalar values ​​that are fed into the neural network (e.g., fed back to...). Figure 1 The metric 152 of the machine learning model 150. Histogram 400 represents the SSIM value of a single layer of a frame, while other histograms can be generated for other layers based on the respective SSIM values ​​of these layers.

[0077] Figure 5According to one or more example embodiments of this disclosure, “diagnostic” views (e.g., “X-ray” or “quality” views) of frames using per-pixel block analysis across multiple layers are illustrated. The diagnostic views include a diagnostic view 500 from layer 0 and a diagnostic view 550 from layer 2, and the block size can be 16x16 or some other size. Data for the diagnostic views 500 and 550 can be generated for the different individual layers based on the Min(blkYL*C*S) metric in Table 1, meaning that diagnostic views 500 and 550 can be based on the “worst” pixel of each pixel block (e.g., the pixel with the lowest metric for any of the various human visual metrics in a block can be considered representative of that block). The brighter the area in the diagnostic view, the stronger the visual artifacts in the frame. The layer 2 view may be based on downscaling from layer 0 twice (e.g., downscaling from layer 0 to layer 1, and then downscaling from layer 1 to layer 2). This downscaling is why some of the stronger artifacts in diagnostic view 500 may be less pronounced in diagnostic view 550. Diagnostic views 500 and 550 can be used as part of a preprocessing stage (e.g., to feed into a model for evaluating metrics) and can be processed internally (e.g., using graphics processing hardware) or "load-shifted" (e.g., to a central processing unit) to allow human feedback on how human viewers rate the frames. The model (e.g., Figure 1 The machine learning model 150 (implemented remotely or within the graphics processing circuitry) can generate pixel weights for individual metrics at a block-based or frame-based level (e.g., see Table 1). In this way, the weights of different metrics used at different blocks, layers, or the entire frame can vary based on weights that can be adjusted on the fly during encoding based on the generated metrics and human feedback.

[0078] MS-SSIM can be determined by aggregating the SSIM values ​​of the original layer and the four downscaling layers. SSIM uses L, C, and S terms, and the MS-SSIM formula (1) above uses an exponent that the encoding hardware may not support. Therefore, software can combine the results to generate the MS-SSIM value.

[0079] PSNR is relatively consistent with the presence or absence of subjective video enhancement, while VMAF is higher when subjective video enhancement is present. In contrast, VMAF and MS-SSIM are more consistent with each other (e.g., higher VMAF maps to higher MS-SSIM), which is one reason for relying on MS-SSIM data, as MS-SSIM may provide a better HVS score before more advanced metrics such as VMAF need to be included in the encoder hardware.

[0080] Figure 6A graph 600 is shown, illustrating the variation of human visual metrics for evaluating encoded video frames, according to one or more example embodiments of this disclosure.

[0081] refer to Figure 6 Chart 600 shows the changes in R 2 Tests (e.g., relative to human visual scores). As shown in the figure, VMAF 602 metric alone (e.g., with a mean of 603) is more relevant to human perception than MS-SSIM 604 metric alone (e.g., with a mean of 605) or PSNR 606 metric alone (e.g., with a mean of 607) (e.g., with a higher R-value). 2 (Value). However, combining MS-SSIM and PSNR metrics (e.g., combined metric 608 with a mean of 609) may provide higher correlation with human visual ratings than VMAF in some cases, and is at least comparable to the range of VMAF metric 602 alone. In this way, the combined metrics in Table 1 allow for a strong correlation with human viewer ratings of encoded video frames, comparable to high-quality VMAF metrics.

[0082] Figure 7 One or more exemplary embodiments of this disclosure are shown Figure 1 The machine learning model has 150 layers.

[0083] refer to Figure 7 Feature 702 (e.g., by Figure 2 The VDAQM 208-generated metric 210, and / or other non-VDAQM human visual quality metrics, such as motion indicators indicating the difference between the current and previous images, sum of co-location absolute differences, sum of absolute differences in motion estimates, inter- and intra-percentages, etc., representing a metric proportional to the temporal redundancy between frames (this may be a metric generated by VDENC 206 or elsewhere), can be input to the first layer 704 (e.g., a layer of an MLP, where the machine learning model 150 can represent the MLP). For example, VDAQM features 702 may include up to 16 features, or some other number (e.g., a 1x16 feature array), such as frame YUV layer 0PSNR, SSIM Y LCS layers 1-4, SSIM Y CS layers 1-4, SSIM histogram compartment counts, as shown in Table 1. Furthermore, features may include an overall temporal relevance score (e.g., a motion score metric proportional to the temporal redundancy between frames, generated by VDAQM 206 or elsewhere). Figure 2The VDENC 206 generation). The first layer 704 can have eight neurons and can use matrix multiplication (e.g., GEMM—general matrix multiplication) to apply matrices (e.g., matrix B <8x16> for 16 features based on user response scores, and matrix C). <8> (For eight neurons). The output of the first layer 704 (e.g., the GEMM of matrices B and C in the first layer 704) can be input to a rectified linear unit (ReLU) 706, which outputs one input value when the input value is greater than zero, and zero for input values ​​less than or equal to zero. The output of ReLU 706 can be input to a second layer 708, which may have eight neurons, and can use matrix multiplication (e.g., GEMM) to apply matrices (e.g., matrix B <8x8> for 16 features based on user response scores, and matrix C...). <8> (For eight neurons). The output of the second layer 708 (e.g., the GEMM of matrices B and C in the second layer 708) can be input to ReLU 710, which outputs one input value when the input value is greater than zero, and zero for input values ​​less than or equal to zero. The output of ReLU 710 can be input to the third layer 712, which may have one neuron, and matrix multiplication (e.g., GEMM) can be used to apply matrices (e.g., matrix B<1x8> for 16 features based on user response scores, and matrix C...). <1> (For a single neuron). The output 714 of the third layer 712 and the machine learning model 150 can be a score (e.g., the HVS score).

[0084] In one or more embodiments, tests have shown that R outputs 714 2 The (determination coefficient) is comparable to VMAF and better than MS-SSIM and PSNR.

[0085] In one or more embodiments, the machine learning model 150 can be trained as follows: The machine learning model 150 can be a fully connected neural network MLP with ReLU. The parameters of the machine learning model 150 can be initialized based on a given rate. Stochastic gradient descent (SGD) with a fixed learning rate can optimize the machine learning model 150, and the mean absolute loss can be used by the machine learning model 150 to determine the error. The machine learning model 150 can be trained for a finite number of epochs, and a patience hyperparameter can be used to stop early. Data can be normalized between 0 and 1 and fed into the machine learning model 150 for training. The performance of the machine learning model 150 can be evaluated based on the sum of absolute errors (SAE). The parameters and the machine learning model 150 can be simple enough to run on a HuC with or without vectorization and can be instantiated in the coded hardware.

[0086] In one or more embodiments, the machine learning model 150 may be at least partially derived from... Figure 1 The circuitry on device 102 is implemented. For example, as... Figure 11 As shown, the artificial intelligence (AI) accelerator 1167 provides machine learning capabilities to evaluate VDAQM features 702 and determine output 714 for evaluating whether to re-encode the coded frame.

[0087] Figure 8 An exemplary component 800 of an encoder is depicted according to one or more example embodiments of this disclosure.

[0088] For example, component 800 can represent Figure 1 and Figure 3 Some components.

[0089] refer to Figure 8 , Figure 3 The VRT router 304 can communicate with the advanced quality metric setup (AQS) 804. The AQS 804 can receive pixel data from multiple codecs, such as MFX 806, HCP 808, and AVP 810 (e.g., with...). Figure 3(Similarly). Using pixel data from multiple codecs (e.g., metrics based on comparing coded frames with reconstructed frames), AQS 804 can determine SSIM 820 and MS-SSIM 822 in parallel, and both SSIM 820 and MS-SSIM 822 can be provided to an advanced quality metrics controller (AQX) 824, which can write the metrics to memory 826 (e.g., similar to...). Figure 3 ).

[0090] Figure 9 An exemplary component 900 for a downscaling encoder is depicted according to one or more example embodiments of this disclosure.

[0091] For example, component 900 can represent Figure 1 and Figure 3 Some components.

[0092] refer to Figure 9 The source video 902 can be of a first size (e.g., 8x8), and a first 2x downscaler 904 can downscale the source video 902 to a second size 906 (e.g., 4x4). A second 2x downscaler 908 can downscale the video of the second size 906 to a third size 910 (e.g., 2x2). A third 2x downscaler 912 can downscale the video of the third size 910 to a fourth size 914 (e.g., 1x1). A fourth 2x downscaler 916 can downscale the video of the fourth size 914 to a fifth size 918. The downscaled video can be sent to a multiplexer 920 and can represent various layers of the video. The output of the multiplexed video layer can be input to an SSIM pipeline 922 (e.g., similar to...). Figure 8 (SSIM820) to determine the SSIM 924 of the video.

[0093] Figure 10 A flowchart illustrating an illustrative process 1000 for generating enhanced real-time visual quality metrics for video encoding is shown according to one or more example embodiments of this disclosure.

[0094] In block 1002, the device (e.g., Figure 11The graphics card (1165) can determine individual first visual quality metrics for pixels of encoded video frames. For example, the first visual quality metric can be any metric shown in Table 1, and can be determined on a per-pixel basis at the pixel block level or the frame level, as shown in Table 1. The first visual quality metric can be for one or more coding layers, and may or may not include each layer (e.g., PSNR is included for layer 0, but not for layers 1-4). The device can determine individual first visual quality metrics for pixels of multiple pixel blocks in one or more video frames.

[0095] In block 1004, the device can determine various second visual quality metrics for pixels. The various first and second visual quality metrics indicate an estimated human perception for the coded video frame. The second visual quality metrics may be specific to one or more coding layers and may or may not include each layer (e.g., SSIM may be included for layers 1-4, but not for layer 0). The device can determine various second visual quality metrics for pixels in multiple pixel blocks within one or more video frames. Other visual quality metrics for pixels (e.g., third, fourth, etc.) can be determined and aggregated using block-based and / or frame-based aggregation (e.g., according to various weights described further below).

[0096] In block 1006, the device can generate first weights for each first visual quality metric. In block 1008, the device can generate second weights for each second visual quality metric. The first and second weights can be any combination of block-based or frame-based weights, as shown in Table 1. For example, one of the visual quality metrics can be a histogram using pixel block-based weights, per-class (e.g., inter-coded or intra-coded classes), or per-block metric (e.g., SSIM value). One of the visual quality metrics can be the mean of the entire frame (e.g., PSNR or SSIM) (e.g., using frame-based weights). Weights can be generated based on which metrics are most likely to correspond to how a human viewer views the frame. For example, training data and / or additional feedback data from human viewers may indicate that some visual artifacts are more noticeable than others and affect their human quality score for the frame. One visual quality metric may be more indicative of the human score than another. For example, frame-based PSNR or SSIM can provide a higher quality score for a frame than block-based scores, which depend on the minimum pixel metric of the pixel block, and the human viewer score of a frame may be lower than frame-based PSNR or SSIM (or closer to block-based metric), so block-based weights can be set higher than frame-based weights.

[0097] In block 1010, the device can determine a human visual score (e.g., HVS score) indicating the visual quality of the encoded video frame based on various first visual quality metrics, first weights (e.g., applied to each first visual quality metric), various second visual quality metrics, and second weights (e.g., applied to each second visual quality metric). For example, the score may include a weighted sum or weighted average of the various visual quality metrics. Machine learning can be used to determine the score locally or remotely. Machine learning may include, for example... Figure 7 The MLP shown. The score can be based on weights indicating the importance of certain visual quality metrics. For example, based on human training data indicating human perception of the visual quality of encoded video frames, the weight of a first visual quality metric can be higher or lower than the weight of a second visual quality metric. When one or more visual quality metrics for one or more pixel blocks are higher or lower than a frame threshold, this may indicate that the frame is of strong or poor quality. For example, a higher SSIM may indicate higher quality, so the SSIM threshold could be 0.5, where the lowest SSIM of the pixels in the frame can be compared to the threshold to determine whether the frame should be recoded. Similar thresholds can be used for other visual quality metrics. When one or more visual quality metrics, based on threshold comparisons, indicate that one or more pixels in a frame are likely to be perceived as having poor visual quality by a human viewer, the device can facilitate frame recoding. For example, when one or more metrics, based on threshold comparisons, indicate higher quality, the human visual score may be higher. Because weights can be at the per-block and / or per-frame level, the aggregation of visual metrics used to generate the human visual score can be enhanced. For example, some technologies may aggregate metrics only at the per-block or per-frame level, while this device can aggregate metrics at multiple levels, combining weighted scores from different levels to generate a score that uses combined metrics and is more consistent with human viewer perception (e.g., such as...). Figure 6 As shown). Human visual scores can also reflect overall temporal relevance scores (e.g., motion score measures proportional to the temporal redundancy between frames, as indicated by...). Figure 2 (Generated using VDENC 206). In this way, the human visual score can be based on a combination of VDAQM 208 metrics and at least one metric proportional to the temporal redundancy between frames. For example, the greater the motion between frames, the lower the human visual score (e.g., because humans may perceive greater motion as blurry, etc.).

[0098] In block 1012, optionally, the device can select encoding parameters for encoding (e.g., if the frame will be encoded using certain encoding parameters when the evaluation of blocks 1002-1010 is part of a look-ahead process for predicting scores), or for recoding the frame when a comparison of the human visual score with a threshold indicates that the visual quality of the frame is too low. For example, when the human visual score is below a score threshold (e.g., because one or more SSIM values ​​and / or other metrics are below the respective thresholds), the device can recode the frame with less lossy compression to reduce the likelihood that visual impairment is apparent to a human viewer.

[0099] In block 1014, optionally, the device may encode the frame (once or subsequently) using selected encoding parameters.

[0100] In block 1016, optionally, the device can report human visual scores, for example, reported to... Figure 2 The HuC 204 can use human visual scores to select encoding parameters for the same or other video frames. In this way, the generation of metrics, human visual scores, and selection of encoding parameters can be performed "locally" on the graphics processing hardware, without requiring metrics to be generated or evaluated elsewhere (e.g., in a central processing unit separate from the graphics processor), thus allowing metrics to be generated on the fly during the encoding process.

[0101] In block 1018, optionally, the device can train the score generation process of block 1010 by repeating blocks 1002-1010 to generate one or more additional human visual scores based on different encoding parameters applied to the frame. For example, the various visual quality metrics of blocks 1002 and 1004 can be based on if the frame is encoded using a first parameter. If the frame is encoded using different encoding parameters, the device can determine the visual quality metric for the same frame and can generate human visual scores for any set of encoding parameters applied to the video frame. Based on the various human visual scores for different encoding parameters, the device can continue to evaluate the human visual scores of frames until the optimal or other satisfactory encoding parameters are identified for encoding (e.g., until the human visual scores meet a score threshold). In this way, because the steps of process 1000 can be performed within the graphics processing circuitry, the device can evaluate multiple different encoding parameters by generating human visual scores for any encoding parameter, and can do so on-the-fly during the encoding process without having to offload the metric generation and evaluation load to other hardware (e.g., Figure 11 Processors 1110 and / or 1130).

[0102] The above description is for illustrative purposes and is not intended to be restrictive.

[0103] Figure 11 An embodiment of the exemplary system 1100 is illustrated according to one or more exemplary embodiments of this disclosure.

[0104] In various embodiments, system 1100 may include an electronic device or may be implemented as part of an electronic device.

[0105] In some embodiments, system 1100 may represent, for example, implementing Figure 1 A computer system consisting of one or more components.

[0106] The embodiments are not limited to this context. More generally, system 1100 is configured to implement all the logic, systems, processes, logical flows, methods, formulas, devices, and functions described herein with reference to the accompanying drawings.

[0107] System 1100 may be a computer system with multiple processor cores, such as a distributed computing system, supercomputer, high-performance computing system, computing cluster, mainframe computer, microcomputer, client-server system, personal computer (PC), workstation, server, portable computer, laptop computer, tablet computer, handheld device (such as personal digital assistant (PDA)), or other device for processing, displaying, or transmitting information. Similar embodiments may include, for example, entertainment devices such as portable music players or portable video players, smartphones or other cellular phones, telephones, digital video cameras, digital still cameras, external storage devices, etc. Other embodiments implement larger-scale server configurations. In other embodiments, system 1100 may have a single processor with one core or more processors. Note that the term "processor" refers to a processor with a single core or a processor package with multiple processor cores.

[0108] In at least one embodiment, the computing system 1100 represents Figure 1 One or more components. More generally, computing system 1100 is configured to implement all the logic, systems, processes, logical flows, methods, apparatuses, and functions described herein with reference to the above figures.

[0109] When used in this application, the terms "system," "component," and "module" are intended to refer to a computer-related entity, or hardware, a combination of hardware and software, software, or software in execution, examples of which are provided by exemplary system 1100. For example, a component may be, but is not limited to, a process running on a processor, a processor, a hard disk drive, multiple storage devices (optical and / or magnetic storage media), an object, an executable file, an executing thread, a program, and / or a computer.

[0110] For illustration, both the application running on the server and the server itself can be components. One or more components may reside within a process and / or a thread of execution, and components may be confined to a single computer and / or distributed across two or more computers. Furthermore, components can communicatively couple to each other to coordinate operation via various types of communication media. This coordination may involve one-way or two-way exchange of information. For example, components may convey information in the form of signals relayed through a communication medium. Signals can be implemented as signals assigned to various signal lines. In this assignment, each message is a signal. However, alternative embodiments may employ data messages. Such data messages can be sent over various connections. Exemplary connections include parallel interfaces, serial interfaces, and bus interfaces.

[0111] As shown in the figure, system 1100 includes a motherboard 1105 for mounting platform components. Motherboard 1105 is a point-to-point (PP) interconnect platform that includes a processor 1110, a processor 1130 coupled via a PP interconnect / interface as an Ultra Path Interconnect (UPI), and a device 1119. In other embodiments, system 1100 may be another bus architecture, such as a multipoint branch bus. Furthermore, each of processors 1110 and 1130 may be a processor package with multiple processor cores. As an example, processors 1110 and 1130 are shown as including one or more processor cores 1120 and 1140, respectively. While system 1100 is an example of a dual-socket (2S) platform, other embodiments may include more than two sockets or one socket. For example, some embodiments may include a quad-socket (4S) platform or an octa-socket (8S) platform. Each socket is a bay for the processor and may have a socket identifier. Note that the term "platform" refers to a motherboard that has certain components installed, such as the processor 1110 and chipset 1160. Some platforms may include additional components, and some platforms may only include sockets for installing the processor and / or chipset.

[0112] Processors 1110 and 1130 can be any of a variety of commercially available processors, including but not limited to... Core(2) and processor; and processor; Application, embedded, and security processors; and and Processors; IBM and Cell processors; and similar processors. Dual microprocessors, multi-core processors, and other multiprocessor architectures can also be used as processors 1110 and 1130.

[0113] Processor 1110 includes an integrated memory controller (IMC) 1114 and PP interconnects / interfaces 1118 and 1152. Similarly, processor 1130 includes an IMC 1134 and PP interconnects / interfaces 1138 and 1154. IMCs 1114 and 1134 couple processors 1110 and 1130 to their respective memories, namely memories 1112 and 1132. Memories 1112 and 1132 may be part of the platform’s main memory (e.g., dynamic random-access memory, DRAM), such as double data rate type 3 (DDR3) or type 4 (DDR4) synchronous DRAM (SDRAM). In this embodiment, memories 1112 and 1132 are locally attached to their respective processors 1110 and 1130.

[0114] In addition to processors 1110 and 1130, system 1100 may also include device 1119. Device 1119 may be connected to chipset 1160 via PP interconnect / interfaces 1129 and 1169. Device 1119 may also be connected to memory 1139. In some embodiments, device 1119 may be connected to at least one of processors 1110 and 1130. In other embodiments, memories 1112, 1132, and 1139 may be coupled to processors 1110 and 1130 and device 1119 via a bus and a shared memory hub.

[0115] System 1100 includes a chipset 1160 coupled to processors 1110 and 1130. Furthermore, chipset 1160 may be coupled to storage medium 1103, for example, via an interface (I / F) 1166. I / F 1166 may be, for example, a Peripheral Component Interconnect-enhanced (PCI-e). Processors 1110, 1130, and device 1119 can access storage medium 1103 through chipset 1160.

[0116] Storage medium 1103 may include any non-transitory computer-readable or machine-readable storage medium, such as an optical storage medium, a magnetic storage medium, or a semiconductor storage medium. In various embodiments, storage medium 1103 may include an article of manufacture. In some embodiments, storage medium 1103 may store computer-executable instructions, such as those for implementing one or more processes or operations described herein (e.g., Figure 10 The process 1000) contains computer-executable instructions 1102. Storage medium 1103 may store computer-executable instructions for any of the above formulas. Storage medium 1103 may also store computer-executable instructions for the models and / or networks (e.g., neural networks, etc.) described herein. Examples of computer-readable or machine-readable storage media may include any tangible medium capable of storing electronic data, including volatile or non-volatile memory, removable or non-removable memory, erasable or non-erasable memory, writable or rewritable memory, etc. Examples of computer-executable instructions may include any suitable type of code, such as source code, compiled code, interpreted code, executable code, static code, dynamic code, object-oriented code, visual code, etc. It should be understood that the embodiments are not limited to this context.

[0117] Processor 1110 is coupled to chipset 1160 via PP interconnects / interfaces 1152 and 1162, and processor 1130 is coupled to chipset 1160 via PP interconnects / interfaces 1154 and 1164. Direct Media Interfaces (DMIs) may be coupled to PP interconnects / interfaces 1152 and 1162, and PP interconnects / interfaces 1154 and 1164, respectively. The DMI may be a high-speed interconnect facilitating, for example, Gigabit Transfers per Second (GT / s), such as DMI 3.0. In other embodiments, processors 1110 and 1130 may be interconnected via a bus.

[0118] Chipset 1160 may include a controller hub, such as a platform controller hub (PCH). Chipset 1160 may include a system clock to perform clocking functions and includes interfaces for I / O buses, such as Universal Serial Bus (USB), Peripheral Component Interconnect (PCI), Serial Peripheral Interconnect (SPI), Integrated Interconnect (I2C), etc., to facilitate connectivity of peripheral devices on the platform. In other embodiments, chipset 1160 may include more than one controller hub, such as a chipset having a memory controller hub, a graphics controller hub, and an input / output (I / O) controller hub.

[0119] In this embodiment, chipset 1160 is coupled to trusted platform module (TPM) 1172 and UEFI, BIOS, and flash memory components 1174 via interface (I / F) 1170. TPM 1172 is a dedicated microcontroller designed to protect hardware by integrating encryption keys into the device. UEFI, BIOS, and flash memory components 1174 can provide pre-boot code.

[0120] Furthermore, chipset 1160 includes I / F 1166 to couple chipset 1160 to a high-performance graphics engine, graphics card 1165. Graphics card 1165 can implement one or more of the processes or operations described herein (e.g., Figure 10 The process 1000), and may include Figures 1-3 Components (e.g.) Figure 1 Partitioner 104 Figure 1 Subtractor 106 Figure 1 Transformers and quantizers 108 Figure 1 Encoder 110, Figure 1 Inverse transform and quantizer 112 Figure 1 Adder 114 Figure 1 Prediction unit 116 Figure 1 Control 121, Figure 2 and Figure 3(e.g., VDBOX 202, etc.). Due to the enhancements to graphics card 1165 described herein, graphics card 1165 can generate human visual quality metrics for encoded video frames without having to offload the metric generation load, and optimal encoding parameters can be identified and selected within graphics card 1165. In other embodiments, system 1100 may include a flexible display interface (FDI) between processors 1110 and 1130 and chipset 1160. The FDI interconnects the graphics processing unit core in the processor with chipset 1160.

[0121] Various I / O devices 1192 are coupled to bus 1181, as well as a bus bridge 1180 coupling bus 1181 to a second bus 1191 and an I / F 1168 connecting bus 1181 to chipset 1160. In one embodiment, the second bus 1191 may be a low pin count (LPC) bus. Various devices may be coupled to the second bus 1191, including, for example, a keyboard 1182, a mouse 1184, a communication device 1186, a storage medium 1101, and audio I / O 1190.

[0122] Artificial intelligence (AI) accelerator 1167 can be circuitry configured to perform AI-related computations. AI accelerator 1167 can be connected to storage medium 1101 and chipset 1160. AI accelerator 1167 can provide the processing power and energy efficiency required to perform rich data computations. AI accelerator 1167 is a specialized hardware accelerator or computer system designed to accelerate artificial intelligence and machine learning applications, including artificial neural networks and machine vision. AI accelerator 1167 is suitable for algorithms used in robotics, the Internet of Things, and other data-intensive and / or sensor-driven tasks.

[0123] Many I / O devices 1192, communication devices 1186, and storage media 1101 may reside on the motherboard 1105, while the keyboard 1182 and mouse 1184 may be additional peripherals. In other embodiments, some or all of the I / O devices 1192, communication devices 1186, and storage media 1101 may be additional peripherals and may not reside on the motherboard 1105.

[0124] Examples may be described using the phrase "in one example" or "example" and its derivatives. These terms mean that a particular feature, structure, or property described in connection with the example is included in at least one example. The phrase "in one example" appearing in various places in this specification does not necessarily refer to the same example.

[0125] Examples can be described using the terms “coupling” and “connection” and their derivatives. These terms are not necessarily intended to be synonyms. For example, a description using the terms “connection” and / or “coupling” may indicate that two or more elements are in direct physical or electrical contact with each other. However, the term “coupling” may also refer to two or more elements that are not in direct contact with each other, yet still cooperate or interact with each other.

[0126] Furthermore, in the aforementioned “Detailed Description” section, various features are grouped together in a single example to simplify the disclosure. This method of disclosure should not be interpreted as reflecting an intention to claim more features than are expressly recited in each claim. Rather, as reflected in the appended claims, the inventive subject matter resides in fewer than all features of a single disclosed example. Therefore, the appended claims are incorporated herein into the “Detailed Description” section, with each claim standing alone as a separate example. In the appended claims, the terms “comprising” and “therein” are used as concise English equivalents of the corresponding terms “including” and “wherein”, respectively. Additionally, the terms “first,” “second,” and “third,” etc., are used merely as labels and are not intended to impose numerical requirements on their objects.

[0127] Although the subject matter is described in language that is specific to structural features and / or methodological actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are disclosed as examples of implementing the claims.

[0128] A data processing system suitable for storing and / or executing program code will include at least one processor directly or indirectly coupled to memory elements via a system bus. Memory elements may include local memory used during the actual execution of the program code, mass storage devices, and cache memory that provides temporary storage for at least some of the program code to reduce the number of times code must be retrieved from mass storage devices during execution. The term "code" covers a wide range of software components and constructs, including applications, drivers, processes, routines, methods, modules, firmware, microcode, and subroutines. Thus, the term "code" can be used to refer to any set of instructions that, when executed by a processing system, perform one or more desired operations.

[0129] The logic circuits, devices, and interfaces described herein can perform functions implemented in hardware and using code that executes on one or more processors. A logic circuit refers to hardware or hardware and code that implements one or more logical functions. A circuit is hardware and can refer to one or more circuits. Each circuit performs a specific function. The circuitry of a circuit system may include discrete electrical components interconnected by one or more conductors, integrated circuits, chip packages, chipsets, memories, etc. Integrated circuits include circuits created on a substrate such as a silicon wafer and may include components. Integrated circuits, processor packages, chip packages, and chipsets may include one or more processors.

[0130] A processor can receive signals, such as instructions and / or data, at one or more inputs and process these signals to generate at least one output. When executing code, the code alters the physical states and characteristics of the transistors that make up the processor pipeline. The physical states of the transistors are translated into logical bits of one and zero stored in registers within the processor. The processor can transfer the physical states of transistors into registers and to another storage medium.

[0131] A processor may include circuitry that performs one or more sub-functions, which are implemented to perform the overall function of the processor. An example of a processor is a state machine or application-specific integrated circuit (ASIC) that includes at least one input and at least one output. A state machine can manipulate at least one input to generate at least one output by performing a predetermined series of serial and / or parallel manipulations or transformations on at least one input.

[0132] The logic described above can be part of the design for an integrated circuit chip. Chip designs are created using a graphical computer programming language and stored on computer storage media or data storage media (e.g., disks, magnetic tapes, physical hard drives, or virtual hard drives, such as those stored in a storage access network). If the designer does not manufacture the chip or the photomask used to manufacture it, the designer transfers the obtained design directly or indirectly to these entities by physical means (e.g., by providing a copy of the storage medium) or electronic means (e.g., via the Internet). The stored design is then converted to an appropriate format (e.g., GDSII) for fabrication.

[0133] The resulting integrated circuit chip may be distributed by the manufacturer as a raw wafer (i.e., as a single wafer with multiple unpackaged chips), as a bare die, or in a package. In the latter case, the chip is mounted in a single-chip package (e.g., a plastic carrier with leads for attachment to a motherboard or other higher-level carrier) or in a multi-chip package (e.g., a ceramic carrier with either or both surface-mount or buried interconnects). In any case, the chip is subsequently integrated with other chips, discrete circuit elements, and / or other signal processing devices as part of: (a) an intermediate product, such as a processor board, server platform, or motherboard, or (b) a final product.

[0134] The term “exemplary” is used herein to mean “serving as an example, instance, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as being more preferred or advantageous than other embodiments. As used herein, the terms “computing device,” “user equipment,” “communication station,” “station,” “handheld device,” “mobile device,” “wireless device,” and “user equipment” (UE) refer to wireless communication devices such as cellular phones, smartphones, tablets, netbooks, wireless terminals, laptops, femtocells, high data rate (HDR) subscriber stations, access points, printers, point-of-sale equipment, access terminals, or other personal communication system (PCS) devices. Devices can be mobile or fixed.

[0135] As used herein, the term "communication" is intended to include sending, or receiving, or both. This may be particularly useful in claims when describing the organization of data sent by one device and received by another device, but only the functionality of one of these devices is required to constitute infringement of the claims. Similarly, when only the functionality of one of two devices is claimed, the bidirectional exchange of data between the two devices (both devices sending and receiving during the exchange) can be described as "communication." The term "communication" as used herein for wireless communication signals includes sending and / or receiving wireless communication signals. For example, a wireless communication unit capable of wireless communication signal communication may include a wireless transmitter that sends wireless communication signals to at least one other wireless communication unit, and / or a wireless communication receiver that receives wireless communication signals from at least one other wireless communication unit.

[0136] As used herein, unless otherwise specified, the use of ordinal adjectives such as “first,” “second,” “third,” etc., to describe common objects merely indicates that different instances of similar objects are mentioned, and is not intended to imply that the objects described in this way must be in a given sequence in time, space, ranking, or any other way.

[0137] Various devices and systems can be used in combination with some implementation examples, such as personal computers (PCs), desktop computers, mobile computers, laptops, notebook computers, tablet computers, server computers, handheld computers, handheld devices, personal digital assistant (PDA) devices, handheld PDA devices, onboard devices, offboard devices, hybrid devices, in-vehicle devices, non-in-vehicle devices, mobile or portable devices, consumer devices, non-mobile or non-portable devices, wireless communication stations, wireless communication devices, wireless access points (APs), wired or wireless routers, wired or wireless modems, video devices, audio devices, audio-video (A / V) devices, wired or wireless networks, wireless local area networks, wireless video area networks (WVANs), local area networks (LANs), wireless LANs (WLANs), personal area networks (PANs), wireless PANs (WPANs), and so on.

[0138] Embodiments of this disclosure are disclosed, particularly in the appended claims relating to a method, a storage medium, an apparatus, and a computer program product, wherein any feature mentioned in one class of claims (e.g., method) may also be claimed in another class of claims (e.g., system). Dependent or backreferences in the appended claims are chosen solely for formal reasons. However, any subject matter arising from the deliberate backreference of any preceding claim (especially multiple dependents) may also be claimed, so that any combination of claims and their features is disclosed and claimable regardless of the dependents chosen in the appended claims. Claimable subject matter includes not only combinations of features recited in the appended claims but also any other combination of features in the claims, wherein each feature mentioned in a claim may be combined with any other feature or combination of other features in the claims. Furthermore, any embodiments and features described or depicted herein may be claimed in a separate claim and / or in any combination with any embodiments or features described or depicted herein or with any features of the appended claims.

[0139] The foregoing description of one or more implementations provides illustrations and descriptions, but is not intended to be exhaustive or to limit the scope of the embodiments to the precise forms disclosed. Modifications and variations are possible in accordance with the above teachings, or may be obtained by implementing various embodiments.

[0140] Embodiments of this disclosure are disclosed, particularly in the appended claims relating to a method, a storage medium, an apparatus, and a computer program product, wherein any feature mentioned in one class of claims (e.g., method) may also be claimed in another class of claims (e.g., system). Dependent or backreferences in the appended claims are chosen solely for formal reasons. However, any subject matter arising from the deliberate backreference of any preceding claim (especially multiple dependents) may also be claimed, so that any combination of claims and their features is disclosed and claimable regardless of the dependents chosen in the appended claims. Claimable subject matter includes not only combinations of features recited in the appended claims but also any other combination of features in the claims, wherein each feature mentioned in a claim may be combined with any other feature or combination of other features in the claims. Furthermore, any embodiments and features described or depicted herein may be claimed in a separate claim and / or in any combination with any embodiments or features described or depicted herein or with any features of the appended claims.

[0141] The foregoing description of one or more implementations provides illustrations and descriptions, but is not intended to be exhaustive or to limit the scope of the embodiments to the precise forms disclosed. Modifications and variations are possible in accordance with the above teachings, or may be obtained by implementing various embodiments.

[0142] The foregoing description of block diagrams and flowcharts of systems, methods, apparatuses, and / or computer program products according to various implementations has described certain aspects of this disclosure. It will be understood that one or more blocks of the block diagrams and flowcharts, as well as combinations of blocks in the block diagrams and flowcharts, can be implemented by computer-executable program instructions. Similarly, depending on some implementations, some blocks of the block diagrams and flowcharts may not necessarily need to be executed in the order presented, or may not need to be executed at all.

[0143] These computer-executable program instructions can be loaded onto a special-purpose computer or other specific machine, processor, or other programmable data processing apparatus to produce a particular machine, such that the instructions, which execute on the computer, processor, or other programmable data processing apparatus, create means for implementing one or more functions specified in one or more blocks of the flowchart. These computer program instructions can also be stored in a computer-readable storage medium or memory, which can direct the computer or other programmable data processing apparatus to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means for implementing one or more functions specified in one or more blocks of the flowchart. As an example, some implementations may provide a computer program product including a computer-readable storage medium in which computer-readable program code or program instructions are implemented, the computer-readable program code being adapted to be executed to implement one or more functions specified in one or more blocks of the flowchart. Computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational elements or steps to be executed on the computer or other programmable apparatus to produce a computer-implemented process, such that the instructions, which execute on the computer or other programmable apparatus, provide elements or steps for implementing the functions specified in one or more blocks of the flowchart.

[0144] Therefore, the blocks in block diagrams and flowcharts support combinations of means for performing a specified function, combinations of elements or steps for performing a specified function, and program instruction means for performing a specified function. It should also be understood that each block in a block diagram and flowchart, and combinations of blocks in block diagrams and flowcharts, can be implemented using a dedicated hardware-based computer system that performs the specified function, element, or step, or using a combination of dedicated hardware and computer instructions.

[0145] Conditional language such as "can," "may," "may," or "can" generally aims to express, unless otherwise specifically stated or understood in the context of its use, that certain implementations may include certain features, elements, and / or operations, while others may not. Therefore, such conditional language generally does not intend to imply that features, elements, and / or operations are required in any way by one or more implementations, or that one or more implementations necessarily include logic for determining whether these features, elements, and / or operations are included in any particular implementation or whether they should be performed in any particular implementation, with or without user input or prompts.

[0146] Many modifications and other implementations of the present disclosure described herein will be apparent from the teachings given in the foregoing description and the associated accompanying drawings. Therefore, it is to be understood that the present disclosure is not limited to the specific implementations disclosed, and that modifications and other implementations are intended to be included within the scope of the appended claims. Although specific terminology is used herein, it is used only in a general descriptive sense and not for limitation.

Claims

1. A graphics card, comprising: a memory; a codec circuit to generate an encoded frame and store data related to the encoded frame in the memory; and a graphics processing circuit to: access the data in the memory; based on the data, compute a first quality metric related to the encoded frame, a second quality metric related to the encoded frame, and a third quality metric related to the encoded frame; based on the first quality metric, the second quality metric, and the third quality metric, execute a machine learning model to generate a quality score related to the encoded frame; and output the quality score. The machine learning model is a support vector machine (SVM). The graphics processing circuit to:

2. The graphics card of claim 1, wherein, based on the data, compute a peak signal-to-noise ratio (PSNR) of the encoded frame; and 3. The graphics card of claim 1, wherein, output the PSNR. The data includes decoded data related to the encoded frame. The graphics processing circuit to:

4. The graphics card of claim 1, wherein, generate the quality score during encoding performed by the codec circuit.

5. The graphics card of claim 1, wherein, The graphics processing circuit to: operate in parallel with the codec circuit.

6. The graphics card of claim 1, wherein, The codec circuit to: based on the quality score, re-encode the encoded frame.

7. The graphics card of claim 1, wherein, 8. A computing method, comprising: generating an encoded frame and storing data related to the encoded frame; accessing the data; based on the data, computing a first quality metric related to the encoded frame, a second quality metric related to the encoded frame, and a third quality metric related to the encoded frame; based on the first quality metric, the second quality metric, and the third quality metric, executing a machine learning model to generate a quality score related to the encoded frame; and outputting the quality score. The machine learning model is a support vector machine (SVM).

10. The method of claim 8, further comprising: based on the data, computing a peak signal-to-noise ratio (PSNR) of the encoded frame; and 9. The method of claim 8, wherein, outputting the PSNR. The data includes decoded data related to the encoded frame.

12. The method of claim 8, further comprising: generating the quality score during encoding.

13. The method of claim 8, further comprising:

11. The method of claim 8, wherein, based on the quality score, re-encoding the encoded frame.

14. One or more computer-readable media storing instructions that, in response to execution by one or more processors, cause the one or more processors to perform the method of any one of claims 8-13.

15. A computing system, comprising: instructions; at least one programmable circuit; and a graphics card programmed based on one or more of the at least one programmable circuit and the instructions to perform the method of any one of claims 8-13.

16. An apparatus of computing, comprising means for performing the method of any one of claims 8-13. ​ ​ ​ ​ ​ 17. A computer program product comprising instructions that, in response to execution by one or more processors, cause the one or more processors to perform the method of any one of claims 8 to 13.

Images