Application-based rate distortion optimization in video compression

Abstract

The invention relates to application-based rate distortion optimization in video compression. Systems and methods herein for a video encoder associated with an interface for receiving, from an application, at least one metric associated with a quality preference of video compression to be performed by the video encoder, a weight map is provided for the video encoder and for providing the weight map to enable the video encoder to perform rate distortion optimization (RDO) on frames received from the application using the weight map to weight one or more first blocks associated with a single one of the frames larger than one or more second blocks associated with the single one of the frames.

Description

Technical Field

[0001] At least one embodiment is directed to a video encoder for performing rate-distortion optimization (RDO).For example, an application provides at least one quality preference to use different weights for different blocks in a video frame to implement RDO for a received frame. Background Art

[0002] Rate-distortion optimization (RDO) can be used to support decisions related to selecting a coding mode from different available modes in video coding (e.g., the H.264 / AVC standard). Such modes can provide different distortion reduction or trade-off methods. For example, rate constraints can be used to perform mode selection to address distortion issues. For example, in the H.264 / AVC video coding standard, RDO can be provided by allowing selection from intra-mode and inter-mode, but can also be extended to select skip mode, where each of such modes supports different video data compression methods. RDO can be used to determine the motion vector or prediction direction to be used with certain modes (including inter-mode or intra-mode), and can be used to perform various aspects of video coding. These aspects may include spatial domain transforms, quantization, and entropy coding. From the perspective of execution, it can be selected to perform inter-frame or intra-frame mode coding to fully handle video coding requirements, based in part on the minimum rate-distortion cost (RD-cost) of at least one macroblock in the frame. RDO can also be used with other standards, including MPEG2 and MPEG4. However, even with RDO, such video coding standards (including AVC, HEVC, VP9, ​​AV1, and VVC) are still based on limitations of the human eye's sensitivity to lower frequencies, sensitivity to luma relative to chroma, and sensitivity to temporal changes. BRIEF DESCRIPTION OF THE DRAWINGS

[0003] Figure 1 A system subject to an embodiment of application-based rate-distortion optimization in video compression is shown;

[0004] Figure 2 Aspects of a system related to application-based rate-distortion optimization in video compression according to at least one embodiment are illustrated;

[0005] Figure 3 Further aspects of a system related to application-based rate-distortion optimization in video compression according to at least one embodiment are shown;

[0006] Figure 4 Computer and processor aspects of a system related to application-based rate-distortion optimization in video compression according to at least one embodiment are shown;

[0007] Figure 5A process flow of a system related to application-based rate-distortion optimization in video compression is shown in accordance with at least one embodiment;

[0008] Figure 6 Yet another process flow of a system related to application-based rate-distortion optimization in video compression according to at least one embodiment is shown; and

[0009] Figure 7 Yet another process flow of a system related to application-based rate-distortion optimization in video compression in accordance with at least one embodiment is shown. DETAILED DESCRIPTION

[0010] Figure 1 A system 100 is shown that is subject to an embodiment of application-based rate-distortion optimization (RDO) in video compression, as described in detail herein. The system 100 includes an encoder 104, such as a video encoder, which can receive an input frame 102 associated with a video stream and can provide an output frame 122 that is a compressed or altered version of the input frame 102. In at least one embodiment, the encoder 104 can be based in part on one of the H.264 standard, the MPEG2 standard, the AVC standard, the HEVC standard, the VP9 standard, the AV1 standard, or the VVC standard. However, the encoder 104 can be any encoder standard that allows weighted input, such as by using a quantization parameter (QP) for mode selection.

[0011] In terms of video encoding, a choice can be made to perform inter-frame or intra-frame mode encoding. This choice can determine how many bits the encoder is willing to sacrifice to hide and / or eliminate distortion. The trade-offs related to distortion may vary from encoder to encoder. For example, the trade-offs may be between different user presets, between different target bit rates (e.g., which may affect the bit budget), and between different frames in a group of frames (GOP) to be encoded. In another example, the trade-off may include the possibility of a certain distortion occurring within a frame, and that the distortion is not a reference to other frames, so the distortion may not be propagated further in the encoding.

[0012] Video compression can be computationally intensive, with current state-of-the-art compression ratios providing 1 / 200 to 1 / 1000 compression, but requiring more compute resources to perform this compression. However, with artificial intelligence and machine learning (AI / ML) workloads using large amounts of images and video, autonomous vehicles generating large amounts of video in each vehicle, applications such as smart cities requiring more video data, content created for entertainment requiring higher video resolutions and more bit depth, and today’s remote work video conferencing technology, it is recognized that video compression must be performed in a more efficient manner. In addition, it is recognized that the limitations of the human eye, as well as the use of color space conversion and separation of luminance (brightness) and chrominance provide aggressive quantization or extraction of features that can limit the ability to provide high-quality video compression.

[0013] For example, a Fourier transform or other related transform may be performed on the blocks within each frame to convert the data therein into the frequency domain and allow information to be quantized or discarded based on selected frequencies. In doing so, lower frequency transform coefficients may be quantized less aggressively than higher frequency transform coefficients. Additionally, motion estimation may be used to capture and encode motion across video frames. While all of these aspects attempt to improve video compression, they all serve a similar goal of allowing the encoder to compress video into a smaller bitstream by eliminating noise, artifacts, allowing at least denser motion estimation, and exploiting temporal and spatial redundancy.

[0014] In view of all these benefits, encoders may differ based in part on the selection of appropriate tools to enable aspects thereof to provide bit economy, such as the ability to select input frames 102 and select regions within those frames (e.g., regions provided by macroblocks (MBs) 102A), and other such methods that may be defined within the encoder as different modes that may require more bits to ensure the desired quality. RDO may be provided within the mode selection module 116 of the encoder 104 to meet these requirements by determining the cost associated with each selection made using an RDO metric (e.g., sum of squared errors (SSE) or sum of transform differences (SATD)) and allowing selection based on cost. However, such selection may have a low correlation with subjective perceived quality. For example, the encoder 104 may provide multiple modes to add more complex compression tools. Such modes may affect the ability of the RDO metric to predict the quality expected using the selection. In one instance, while the visual quality metric may correspond to an SSE RDO metric using a peak signal-to-noise ratio (PSNR), this may be a poor predictor of quality assessment.

[0015] Additional RDO metrics allow further mode selection, including Video Multi-Method Assessment Fusion (VMAF), Structural Similarity Index (SSIM), Multi-Scale SSIM (MS-SSIM), and the aforementioned PSNR. However, for the encoder 104, temporal effects still need to be addressed, as it may be done only at the frame level using such additional RDO metrics. Distortion can be determined as the difference from the original image. However, in some applications, the focus of the application may be to avoid objectionable visual artifacts rather than minimizing such distortion. For such applications, the user may not be a human observer, but rather an AI / ML model, for example. AI / ML models may not require RDO metrics based on the human visual system, so many such RDO metrics may not be appropriate or relevant.

[0016] In at least one embodiment, the system 100 for application-based RDO in video compression enables an application that intends to use the encoder 104 to provide a metric associated with the application, and the encoder 104 will perform video compression based on the metric. For example, the encoder 104 (also referred to herein as a video encoder) can receive transform coefficients or parameters (e.g., QP) from an interface 140, which in turn uses the metric provided by the application to determine weights to be attached to certain blocks associated with a video frame and enables weights of blocks that are different from other blocks in the encoder 104. Interface 140 is used interchangeably with application-aware interface in this article. Interface 140 is application-aware because it is suitable for notifying an application of capabilities associated with it (or with an encoder 104 associated therewith) and is suitable for initiating the receipt of at least one metric from the application. Interface 140 provides a weight map of such weights to the encoder 104. Therefore, application-based RDO preserves the video quality of portions of a video frame that are associated with the application.

[0017] In at least one embodiment, interface 140 is part of a processor or an execution unit within a processor. In addition, encoder 104 is also part of the same or different processor or part of the same or different execution unit within a processor. Interface 140 is used to receive metrics from an application. The metric can be associated with the application's preference for its video compression requirements. In addition, in at least one embodiment, interface 140 can initially provide its capabilities to the application to enable it to receive metrics. Interface 140 is used to generate a weight map based on the metrics. Interface 140 has application-level awareness and performs weight calculations for intra-frame blocks based at least in part on the metrics provided by the application. For example, the metric is a quality metric of the application's preference. In one example, an application for a classification ML model may need or prefer to give higher weights to certain blocks associated with a video frame having an object therein relative to other blocks associated with a video frame having a background therein.

[0018] In at least one embodiment, the weighted blocks may be subject to one or more further noise estimation algorithms in time or space. This enables, in addition to the weights associated with the weight map, the weights of individual blocks affected by stronger noise to be reduced. In addition, the weights of the weight map may be determined in part based on the edges within the frame with the blocks. Further, the weights of the weight map may be derived from the output of a detection or classification model run on the image, which is used to detect or classify aspects within the image, such as objects relative to the background. Further, the weights of the weight map may be determined at least in part based on a visual quality metric that is part of the metric from the application. For example, the visual quality metric may include an extended perceptual weighted PSNR (XPSNR) or a perceptual weighted PSNR (WPSNR).

[0019] In at least one embodiment, the information or output of interface 140 can be used with an encoder 104 that is agnostic to the metric. The weights provided by interface 140 can be provided as a map of weights in the input to encoder 104 or any execution unit of the processor that performs RDO. This allows the result of RDO to select a video compression mode that is ultimately beneficial to the application. Therefore, these aspects of system 100 and the methods it supports divide RDO into two parts. In one part, system 100 and the methods it supports allow the application to control the weights associated with RDO. In the second part, system 100 and the methods it supports allow encoder 104 to remain agnostic, which can be executed entirely by the execution unit of the processor or the processor itself. In this way, the result of encoder 104 is substantially free of performance loss.

[0020] In at least one embodiment, utilizing the software aspects of the encoder 104 herein, the system 100 and the methods it supports are also able to perform video compression using weight maps without sharing secrets associated with the application. For example, some applications may need to notify during video compression that specific blocks are to be treated differently from other blocks. An example is that a security camera application may need special treatment of the video of faces or areas that provide more information than other areas of the security camera application. The video can be classified in the display. The secret can be the scaling and weighting in the notification pixel domain, which does not need to be shared in the system 100 and the methods it supports herein. In at least one embodiment, an application programming interface (API) can have a weight map for each frame. Classification of edges, face detection, or other object detection can be supported. The API receives parameters for the sequence and sequence frames. The weight map provides weights for one or more blocks of each frame of the video compression to be performed by the encoder 104. Therefore, the bitstream from the encoder 104 may be lower in bandwidth. In addition, artificial intelligence (AI) or machine learning (ML) applications can benefit from this approach because the underlying data is not compromised by application-based RDO.

[0021] In at least one embodiment, the RDO may be determined using any classical or weighted metric. However, the RDO may be determined using a reference to a previously encoded and reconstructed frame. This approach would allow generative AI to be used with application-based RDO. In at least one embodiment, application-based RDO may also address bit allocation and intelligent bit savings through aspects that occur in a rate controller that is part of or supports the encoder 104. For example, if a block of a frame is found to be similar to a co-located block of that frame, then the energy of the residual is low. Then, regardless of the quantization performed, the quantizer will have no impact and there is no need to allocate bits for such information.

[0022] In at least one embodiment, application-based RDO during video compression allows an application that intends to use encoder 104 to provide an application-related metric, wherein encoder 104 uses at least one weight map associated with the metric to weight different blocks associated with a video frame as part of mode selection or to perform video compression. Specifically, a video encoder associated with interface 140 can receive a metric associated with the quality preference of the application from the application. For example, the metric can be received in interface 140 or in a rate controller aspect of encoder 104.

[0023] The encoder 104 may support RDO performed on at least one of the frames received from the application based in part on a metric, wherein a weight map is used to weight one or more first blocks associated with a single frame in the frame more than one or more second blocks associated with the single frame in the frame. In addition, one or more features of the RDO, interface 140, and encoder 104 may be implemented in hardware (e.g., individual processors or individual execution units of a processor) without performance loss associated with video quality. Finally, the application may be able to determine that the video encoder has the ability to perform application-based RDO and may provide the metric to the encoder 104 or interface 140 before providing the video stream.

[0024] Application-based RDO can address bit allocation and intelligent bit savings during video compression performed in blocks of a frame, which can be performed in a rate controller based on discoveries within certain blocks. However, in order to also address the issue that blocks that may have content similar to the collated blocks or objects within the blocks result in lower residual energy and reduce the effect of the quantization performed when such discoveries are made, the system 100 and its related methods enable the application to provide at least one metric for performing weighting in RDO. Thus, applications such as video conferencing, AI / ML, facial recognition, etc. can provide at least one metric related to their respective needs, which may belong to a specific image standard, such as VMAF, SSIM, MS-SSIM, PSNR, etc. This metric can be the basis for weighting of video frames for the application in RDO.

[0025] In at least one embodiment, Figure 1 An encoder 104 is provided that is subject to H.264 encoding. The encoder 104 includes hardware or software modules such as a prediction module 112, a transform and quantization (T and Q) module 108, and an entropy coding module 110. There may also be additional modules such as an inverse module 114, a filter module 120, a motion processing module 118 (for supporting motion estimation and related aspects), and a previous or reference frame module 106. The application-based RDO herein has no effect on the decoding process of the bitstream provided from the encoder 104, including the output frame 122. For example, the decoding process may be in accordance with H.264 decoding or other decoding related to the encoding format used in the encoder 104 (particularly in the entropy coding module 110).

[0026] The bitstream of the frame to be compressed may include an input frame 102, which may be segmented into units of MBs 102A. In at least one embodiment, the application-based RDO may support MBs of different sizes, including but not limited to 8x8, 8x16, 16x8, 4x4, and 16x16. The MBs may correspond to display pixel data obtained at the location of the block. The prediction module 112 may generate a prediction MB that may be used to generate residual data reflecting the quantized data as part of the application-based RDO and for video compression. There may be multiple prediction options associated with the prediction module 112, including intra-frame prediction associated with previously encoded data from a current frame (e.g., input frame 102). Another option associated with the prediction module 112 includes inter-frame prediction using encoded data from other previously encoded frames (i.e., reference frames, such as from the reference frame module 106). These reference frames may appear before or after the current frame in display order and may be associated with motion compensation, such as a motion processing module 118 using previously encoded frames (e.g., frames provided by the previous or reference frame module 106).

[0027] Another option associated with the prediction module 112 includes using different prediction block sizes that can be used for both intra prediction and inter prediction options. Using different prediction block sizes for MB 102A can change the accuracy associated with the prediction. Another option associated with the prediction module 112 includes using multiple frames during prediction, which is available in the inter prediction option to provide better prediction accuracy. Another option is to skip the MB data or residual data so that the encoder 104 itself performs the inference of the MB data based in part on the predicted MB.

[0028] The intra prediction may be based at least in part on spatial data within the input frame 102, where an MB is generated as part of the intra prediction that is different from the MB 102A of the input frame 102. The residual data may be a residual MB generated by subtracting the predicted MB from the current MB. The residual MB may be transformed, quantized, and entropy encoded in the provided modules 108, 110, depending on the mode selected, for example, by the mode selection module 116 that performs RDO. In addition, in the encoder 104, the quantized data may be rescaled and inverse transformed in the inverse module 114. The output of the inverse module 114 may be filtered and combined with the predicted MB in the prediction module 112. Motion estimation from the motion processing module 118 may also be included. The result may be a reconstructed MB, which is provided to the previous or reference frame module 106 for further prediction.

[0029] According to at least one embodiment, Figure 2 Aspects 200 of a system related to application-based rate-distortion optimization in video compression are shown. Figure 2Further details are described regarding aspects of application 202 interfacing with encoder 104 via interface 140 to perform application-based RDO. Figure 1 104) are available for video compression, but one or more of these options may be combined as part of a mode that may be selected using RDO. Additionally, in addition to the prediction module 112 supporting mode selection, the mode selection module 116 allows the encoder 104 to adjust the QP within a mode (e.g., within an inter mode). In at least one embodiment, a rate controller may be external or internal to the encoder 104 to provide a QP associated with rate control of a bitstream generated from the encoder 104. This may be performed to return a compressed video bitstream that provides the maximum decoding quality but is also minimal in terms of encoding bitrate.

[0030] In at least one embodiment, to enable the application 202 to provide specific input associated with the application-based RDO, the capability 204A of the encoder 104 to support the application-based RDO can be communicated to the application 202 before or after the application 202 issues a request. For example, an indication can be provided to the application 202 that the encoder 204 can receive the weight map 206 for the RDO based in part on the metrics provided by the application 202. In another example, the application 202 provides the metrics to the interface 140 regardless of the encoder's capability 204A, and one or more weight processes 220 of the interface 140 can process the metrics to provide the weight map 206 to the encoder 104. In addition, although shown as being closer to the encoder 104 or a portion of the encoder 104, the interface 140 can be performed by software, hardware, firmware, API, driver, or other callable or interface aspects that can be associated with the application 202.

[0031] In at least one embodiment, a weight map 206 is provided to enable the encoder 104 to use the weights therein to change or affect the weights used in at least one transform and quantization module 108. For example, a transform function may be used to transform a block of residual data. In a non-limiting example, the transform may be a 4x4 or 8x8 transform function, which provides coefficients that may be weights in a standard basis mode. In at least one embodiment, some codecs may support transform sizes larger than 4x4 or 8x8 transform functions. These weights may be modified using weights from the weight map 206. For example, the coefficients may be quantized so that each coefficient is quantized by an integer. Quantization may modify the effect of only certain coefficients, such as weights from the weight map 206, based in part on the QP, rather than modifying the entire frame.

[0032] In one instance, higher values ​​of QP associated with certain weights in weight map 206 can modify certain coefficients that are lower in value and less relevant to an application (e.g., an AI / ML application). The output of this modification may be higher compression in certain areas, but lower image quality in these areas. In addition, higher values ​​of QP for certain areas of the frame are associated with certain weights from weight map 206, which can enable modification of some coefficients that may be non-zero to remain after quantization. This can provide better image quality and lower compression for certain areas of the frame on the decoding side. In addition, the modifications in the standard basis mode can be used to perform inverse operations on the residual data.

[0033] In at least one embodiment, there may be multiple weight processes 220 associated with the interface 140. The weight processes 220 may be provided in different or the same execution units, or may be different application programming interfaces (APIs), all of which have one or more weight maps 206 specified for the metric. However, a single API may be an application-aware API with different classes associated with different weight maps 206 to be provided for different metrics. Furthermore, the weight map 206 may be for each frame or set of frames and may be provided to the encoder 104 in association with the metric. Thus, in at least one embodiment, an input frame 102 may be received in the interface 140 to allow determination of a weight map 206 to be used with the input frame and one or more subsequent frames.

[0034] In at least one embodiment, a processed frame 214 (which may be a downsampled or filtered frame of the input frame 102) may be provided to the interface 140 instead of the input frame 102 to allow determination of a weight map 206 to be used with the input frame. In one non-limiting example, the processed frame 214 may be processed as a color format conversion. In at least one embodiment, the weight map 206 may be generated from at least one retained weight map 206 by changing the weights therein based in part on the received metrics. In addition, the systems and methods herein support dynamic weighting, wherein determination of the weight map 206 is dynamically updated for subsequent input frames instead of a single input frame 102.

[0035] In at least one embodiment, the application 202 may define the metric as part of the preferences 204B external to the encoder 104. Thus, the encoder 104 does not play any role in the metric received from the application 202. In addition, the metric may be a metric that is unfamiliar to the encoder 104, unlike the RDO metrics described elsewhere herein, which may include PSNR, SSIM, and MS-SSIM. However, the application 202 may need to assign specific weights to different pixels represented in the MB. In at least one embodiment, the application 202 may include a table 202A that includes aspects that are determined to be secrets related to its preferences. The application may include: a manifest file with keys that are used to ensure that secure calls can be made to the interface 140, and metrics 204B provided for the weight map 206.

[0036] In at least one embodiment, the application 202 does not communicate the blocks of frames that need to be processed in any special way. For example, a security camera application may have special requirements for certain parts of the frame, requiring their quality to be maintained better than other parts. The classification of the video frame may be part of the secret, including scaling and weighting information in the pixel domain, which may not be shared. However, an API (such as a weight process 220) can be provided to map the weights of each frame. In at least one embodiment, the parameters of the sequence and frame from the application 202 can be used to perform classification, edge detection, face detection, or other object detection in the interface 140. The generated weight map 206 can include a block of weights for each frame. Since the weights are specific to the region of the frame, the final bitstream from the encoder 104 will require fewer bits for some applications, but more bits may be allocated for intensive applications. For example, in the case of AI / ML applications, the region of interest (ROI) may be specifically enabled by the metrics of the sequence and frame from the application. A weight table for each MB can be generated, such as an 8x8 pixel block associated with a particular MB.

[0037] In at least one embodiment, the flag parameter 216 allows for further adjustment of the weight map associated with the weight process 220. For example, the flag parameter 216 can be used by the application to adjust frequency; sensitivity to brightness, sensitivity to chrominance, sensitivity to time changes, or other aspects that are preferred by the application 202. The flag parameter 216 can be part of one or more of the capabilities 204A communicated to the application 202 and the metrics 204B communicated to the interface 140 to allow the application to select at least one value associated with the flag parameter 216. In at least one embodiment, the encoder 104 can perform a noise estimation algorithm, which is one or more of temporal, spatial, or a combination thereof, to provide noise information and weighting to be performed on the input frame 102. The encoder 104 can determine that one or more second blocks associated with a single frame in the frame are subject to more noise in the noise information than one or more first blocks associated with the single frame in the frame. The encoder 104 can perform additional weighting on the one or more first blocks and the one or more second blocks based at least in part on the noise.

[0038] In at least one embodiment, the weight map 206 of the interface 140 includes weights based at least in part on one or more edges within a single one of the input frames 102 or based at least in part on a visual quality metric of at least one metric. In addition, the weight map 206 of the interface 140 includes weights based at least in part on an output of a detection or classification model applied to a single one of the input frames 102. In addition, the weight map 206 of the interface 140 includes weights based at least in part on a visual quality metric, which can be part of the at least one metric 204B. The output bitstream is transmitted 208 to a decoder 210 at the receiver side, where it is decoded for an application 212. No modifications to the decoder 210 are required to provide the application-based RDO herein.

[0039] For example, RDO may be performed in a mode selection module 116 of the encoder 104 and may include cycling through different prediction modes to select a prediction mode for a block of a frame, a frame itself, or multiple frames. Thus, for each prediction mode, a prediction block (P) may be generated for a macroblock (MB), a residual may be determined, a transform and quantization (T and Q) may be performed (including using rate control or external influences on quantization, such as using the QP from the weight map 206), a number of bits associated with encoding the residual may be determined, the MB may be reconstructed, and a distortion and cost of using the original and reconstructed MB may be determined. Then, based at least in part on the cost of all modes cycled, a mode selection may be performed so that at least the prediction and residual are used with entropy encoding of the original frame and at least some subsequent frames to generate an output frame 122 for transmission 208 to a decoder 210.

[0040] Figure 3 Further aspects of a system related to application-based rate-distortion optimization (RDO) in video compression according to at least one embodiment are shown 300. To use an application-provided metric to influence the RDO process, the system herein includes one or more processing units to enable RDO to be performed on an input frame 102 having MBs 102A from an application 202 based in part on a weight map 206 associated with at least one metric of the application's quality preference 204B. The weight map 206 provides a first weight 310 to be associated with one or more first blocks 304 of the provided MBs 102A of a single one of the input frames 102. The weight map 206 also provides a second weight 312 that is less than the first weight 310 to one or more second blocks 306 of the provided MBs 102A of a single one of the input frames 102.

[0041] In at least one embodiment, even though the weights are described with reference to blocks 304, 306, the weights may be applied in the T and Q phases and may be applied to cause changes in the distortion 302. For example, the coefficients 324 of the transform portion of the T and Q phases may include the distortion 302, which may be further adjusted by the weights applied from the weight map 206. For example, for each prediction mode 320, the transform outputs coefficients 324, which are weighted values ​​associated with a base mode, such as provided by the inverse module 114. Once weighted with the weights from the weight map 206, the base mode may be used to reconstruct a block of residual 322. The reconstruction is represented as an image block 314 with a weighted base mode, where different affected blocks 316, 318 belong to different weights 310, 312 of the weight map 206. For example, the coefficients may be quantized by division or other changes caused by the weights 310, 312. Quantization changes the coefficients 324 according to the weights 310, 312 that provide a QP. In at least one embodiment, the output from encoder 104 is a compressed bitstream that includes the values ​​of the quantized transform coefficients and information used by the decoder to recreate the predicted MB, as well as information about the compression structure and frame sequence. Prior to transmission 208, these values ​​are converted to binary code using, for example, variable length coding (VLC).

[0042] In at least one embodiment, one or more processing units further execute a noise estimation algorithm to provide noise information, which algorithm is one or more of time or space or a combination thereof. It can be determined that one or more second blocks 306 associated with a single input frame in the input frames 102 are subject to more noise in the noise information than one or more first blocks 304 associated with a single input frame in the input frames 102. In addition, additional weighting can be performed on the one or more first blocks 304 and the one or more second blocks 306 based at least in part on the noise.

[0043] In at least one embodiment, one or more processing units adapted for application-based RDO also determine the first weight and the second weight of the weight map 206 based at least in part on one or more edges within a single frame in the frame or based at least in part on a visual quality metric of at least one metric. The one or more processing units are also used to perform a detection or classification model on a single frame. Then, the first weight and the second weight determined for the weight map 206 can be based at least in part on the output of the detection or classification model. Therefore, at least one of the weight processes 220 can perform a detection or classification model to determine aspects of the input frame 102 to be weighted differently based in part on the metrics from the application. For example, metrics from sequences and frames of the application can guide the detection or classification model to determine specific objects or features to be classified and enable the selection of blocks to be weighted differently as part of the RDO. In addition, the at least one metric can include a visual quality metric, and the one or more processing units are also used to determine the first weight and the second weight based at least in part on the visual quality metric of the at least one metric.

[0044] Figure 4 A computer and processor aspect 400 of a system associated with application-based rate-distortion optimization in video compression according to at least one embodiment is shown. For example, one or more processors 402 may include one or more processing or execution units 408 that may execute an application 202 associated with video data. In addition, one or more processing units 408 may execute aspects of the encoder 104 and the interface 140. The one or more execution units 408 may determine a feature associated with the encoder 104 that is associated with at least one metric of the quality preference 204B of the application 202. The one or more execution units 408 may generate a weight map 206 associated with the metric to enable the video encoder to perform rate-distortion optimization (RDO) on a frame received from the application using the weight map. For example, the application-based RDO herein may use the weight map to give a greater weight to one or more first blocks associated with a single frame in a frame than to one or more second blocks associated with a single frame in a frame.

[0045] In at least one embodiment, the one or more execution units 408 are further configured to execute a noise estimation algorithm to provide noise information, the algorithm being one or more of time or space or a combination thereof. The one or more execution units 408 may determine that one or more second blocks associated with a single frame are subject to greater noise in the noise information than one or more first blocks associated with a single frame. The one or more execution units 408 may also perform additional weighting on the one or more first blocks and the one or more second blocks based at least in part on the noise.

[0046] In at least one embodiment, the one or more execution units 408 are further configured to determine the weights in the weight map based at least in part on one or more edges within a single frame of the frames or based at least in part on a visual quality metric of the at least one metric; but the weights in the weight map may also be determined based at least in part on an output of a detection or classification model performed on a single frame of the frames. In at least one embodiment, when the at least one metric includes a visual quality metric, the one or more execution units 408 may perform further determination of the weights in the weight map based at least in part on the visual quality metric.

[0047] In accordance with at least one embodiment, the computer and processor aspect 400 may be executed by one or more processors 402 that include a system on a chip (SOC) or some combination of systems on a chip (SOC) formed by a processor that may include an execution unit to execute instructions. In at least one embodiment, the computer and processor aspect 400 may include, but is not limited to, components, such as the processor 402, to execute algorithms for processing data in accordance with the present disclosure using an execution unit 408 that includes logic, such as in the embodiments described herein. In at least one embodiment, the computer and processor aspect 400 may include a processor, such as Processor series, Xeon TM , XScale TM and / or StrongARM TM , Core TM or Nervana TM The computer and processor aspects 400 may be implemented as a microprocessor, available from Intel Corporation of Santa Clara, California, although other systems (including PCs with other microprocessors, engineering workstations, set-top boxes, etc.) may also be used. In at least one embodiment, the computer and processor aspects 400 may execute a version of the WINDOWS operating system available from Microsoft Corporation of Redmond, Washington, although other operating systems (e.g., UNIX and Linux), embedded software, and / or graphical user interfaces may also be used.

[0048] Embodiments may be used in other devices, such as handheld devices and embedded applications. Some examples of handheld devices include cellular phones, Internet Protocol devices, digital cameras, personal digital assistants ("PDAs"), and handheld PCs. In at least one embodiment, the embedded application may include a microcontroller, a digital signal processor ("DSP"), a system on a chip, a network computer ("NetPC"), a set-top box, a network hub, a wide area network ("WAN") switch, or any other system that can execute one or more instructions according to at least one embodiment.

[0049] In at least one embodiment, the computer and processor aspect 400 may include, but is not limited to, a processor 402, which may include, but is not limited to, one or more execution units 408 for executing the instructions described herein. Figure 1-3 and Figure 5-7 In at least one embodiment, the computer and processor aspect 400 is a single processor desktop or server system, but in another embodiment, the computer and processor aspect 400 can be a multi-processor system.

[0050] In at least one embodiment, the processor 402 may include, but is not limited to, a complex instruction set computer ("CISC") microprocessor, a reduced instruction set computing ("RISC") microprocessor, a very long instruction word ("VLIW") microprocessor, a processor implementing a combination of instruction sets, or any other processor device, such as, for example, a digital signal processor. In at least one embodiment, the processor 402 may be coupled to a processor bus 410 that may transmit data signals between the processor 402 and other components in the computer and processor aspect 400.

[0051] In at least one embodiment, processor 402 may include, but is not limited to, level 1 ("L1") internal cache memory ("cache") 404. In at least one embodiment, processor 402 may have a single internal cache or multiple levels of internal cache. In at least one embodiment, the cache memory may be external to processor 402. Other embodiments may also include a combination of internal and external caches, depending on the particular implementation and requirements. In at least one embodiment, register file 406 may store different types of data in various registers, including, but not limited to, integer registers, floating point registers, status registers, and instruction pointer registers.

[0052] In at least one embodiment, execution unit 408 (including but not limited to logic to perform integer and floating point operations) also resides in processor 402. In at least one embodiment, processor 402 may also include a microcode ("ucode") read-only memory ("ROM") that stores microcode for certain macroinstructions. In at least one embodiment, execution unit 408 may include logic to handle packed instruction set 409.

[0053] In at least one embodiment, by including packed instruction set 409 and associated circuitry for executing the instructions in the instruction set of a general purpose processor, operations used by many multimedia applications may be performed using packed data in processor 402. In at least one embodiment, many multimedia applications may be executed faster and more efficiently by using the entire width of the processor's data bus to perform operations on packed data, which may eliminate the need to transfer smaller units of data over the processor's data bus to perform one or more operations on one data element at a time.

[0054] In at least one embodiment, execution unit 408 may also be used in microcontrollers, embedded processors, graphics devices, DSPs, and other types of logic circuits. In at least one embodiment, computer and processor aspects 400 may include, but are not limited to, memory 420. In at least one embodiment, memory 420 may be a dynamic random access memory ("DRAM") device, a static random access memory ("SRAM") device, a flash memory device, or other memory device. In at least one embodiment, memory 420 may store instructions 419 and / or data 421 represented by data signals that may be executed by processor 402.

[0055] In at least one embodiment, the system logic chip may be coupled to the processor bus 410 and the memory 420. In at least one embodiment, the system logic chip may include, but is not limited to, a memory controller hub ("MCH") 416, and the processor 402 may communicate with the MCH 416 via the processor bus 410. In at least one embodiment, the MCH 416 may provide a high bandwidth memory path 418 to the memory 420 for instruction and data storage and for storage of graphics commands, data, and textures. In at least one embodiment, the MCH 416 may direct data signals between the processor 402, the memory 420, and other components in the computer and processor aspect 400, and bridge data signals between the processor bus 410, the memory 420, and the system I / O interface 422. In at least one embodiment, the system logic chip may provide a graphics port for coupling to a graphics controller. In at least one embodiment, the MCH 416 may be coupled to the memory 420 via a high bandwidth memory path 418, and the graphics / video card 412 may be coupled to the MCH 416 via an accelerated graphics port ("AGP") interconnect 414.

[0056] In at least one embodiment, the computer and processor aspects 400 may use the system I / O interface 422 as a proprietary hub interface bus to couple the MCH 416 to an I / O controller hub ("ICH") 430. In at least one embodiment, the ICH 430 may provide direct connection to certain I / O devices via a local I / O bus. In at least one embodiment, the local I / O bus may include, but is not limited to, a high-speed I / O bus used to connect peripheral devices to the memory 420, chipset, and processor 402. Examples may include, but are not limited to, an audio controller 429, a firmware hub ("flash BIOS") 428, a wireless transceiver 426, a data storage device 424, a traditional I / O controller 423 including a user input and keyboard interface 425, a serial expansion port 427 (e.g., a universal serial bus ("USB") port), and a network controller 434. In at least one embodiment, the data storage device 424 may include a hard drive, a floppy drive, a CD-ROM device, a flash memory device, or other mass storage device.

[0057] In at least one embodiment, Figure 4 A computer and processor aspect 400 is shown, which includes interconnected hardware devices or "chips", while in other embodiments, Figure 4 An exemplary SoC may be shown. In at least one embodiment, Figure 4 The devices shown in can be interconnected with a proprietary interconnect, a standardized interconnect (eg, PCIe), or some combination thereof. In at least one embodiment, one or more components of computer and processor aspect 400 are interconnected using a Compute Express Link (CXL) interconnect.

[0058] Figure 5 A process flow or method 500 of a system related to application-based rate-distortion optimization in video compression according to at least one embodiment is shown. The method 500 includes enabling an application to provide a quality preference for video data 502. For example, the application may include a video conferencing application, an AI / ML application, a facial recognition application, etc., each of which may have different video data quality preferences specific to how the application uses the video data. In at least one embodiment, the method 500 includes informing an encoder or processor that is to process the video data that application-based RDO is enabled before the application provides its quality preference. For example, this can be achieved by a driver of the encoder or processor communicating with the application or by using an API of an interface.

[0059] The method 500 includes determining or verifying 504 that at least one communication received from an application is associated with video data. The method 500 includes determining 506 a quality preference from the communication, wherein the quality preference includes at least one metric associated with the application. The at least one metric is associated with video compression to be performed by a video encoder on video data having a plurality of frames. The method 500 includes generating 508 a weight map associated with the metric, such as by using at least one weight process of an interface. The method 500 also includes performing 510 RDO on a single frame of the frames of the video data from the application based in part on using the weight map, wherein performing 510 step uses the weight map to weight one or more first blocks associated with the single frame in the frame more than one or more second blocks associated with the single frame in the frame.

[0060] Figure 6 Yet another process or method 600 of a system related to application-based rate-distortion optimization in video compression in accordance with at least one embodiment is shown. Figure 6 Can support Figure 5 In at least one embodiment, method 600 may include a step or sub-step of executing 602 a noise estimation algorithm to provide noise information, the algorithm being one or more of time or space. Step 602 may support Figure 5 The method 600 includes determining 604 noise from noise information of one or more first blocks and one or more second blocks associated with a single frame in the frame. The method 600 includes verifying 606 that the one or more second blocks are subject to more noise in the noise information than the one or more first blocks. The method 600 includes performing additional weighting 608 on the one or more first blocks and the one or more second blocks based at least in part on the noise. The step 608 may be Figure 5 The weight map of step 510 in method 500 is supplemented.

[0061] Figure 7 A further process flow of a system related to application-based rate-distortion optimization in video compression is shown in accordance with at least one embodiment. Figure 7 Can support Figure 5700 includes executing 702 a detection or classification model on a single frame of the frame. For example, the detection or classification model is used to perform classification on edges or to perform detection on faces or other objects, together forming features of the model. Verification 704 may be performed so that the detection or classification model produces a feature output. For example, multiple features may need to select a feature as a dominant feature. In at least one embodiment, method 700 includes determining 706 a weight as part of a weight map of step 508 based at least in part on the output of the detection or classification model. Method 700 may include verifying 708 that the weight map is complete before providing 710 the weight map to perform the RDO in step 510.

[0062] Other variations are within the spirit and scope of the present disclosure. Thus, while the disclosed technology is susceptible to various modifications and alternative constructions, certain illustrative embodiments thereof are shown in the drawings and have been described in detail above. However, it should be understood that there is no intention to limit the disclosure to the particular forms disclosed, but on the contrary, it is intended to cover all modifications, alternative constructions, and equivalents that fall within the spirit and scope of the disclosure as defined in the appended claims.

[0063] In the context of describing the disclosed embodiments (especially in the context of the following claims), the use of the terms "a", "an", and "the" and similar referents should be interpreted as covering the singular and the plural, unless otherwise specified herein or clearly contradicted by the context, and are not used as definitions of the terms. The terms "include", "have", "include", and "contain" should be interpreted as open-ended terms (meaning "including but not limited to") unless otherwise specified. "Connected" when unmodified and referring to a physical connection should be interpreted as partially or completely contained, attached to, or connected together, even if there is something in between. The recitation of ranges of values ​​herein is intended only as a shorthand method to individually reference each individual value falling within the range, unless otherwise specified herein, and each individual value is incorporated into the specification as if it were individually recited herein. In at least one embodiment, the use of the term "set" (e.g., "a set of items") or "subset" should be interpreted as a non-empty set containing one or more members, unless otherwise specified or contradicted by the context. Furthermore, unless otherwise stated or contradicted by context, the term "subset" of a corresponding set does not necessarily mean a true subset of the corresponding set, but a subset and a corresponding set may be equal.

[0064] Unless expressly stated otherwise or clearly contradicted by context, connective language (e.g., phrases of the form "at least one of A, B, and C" or "at least one of A, B, and C") should be understood together with the context as generally used to indicate that an item, term, etc., may be A or B or C, or any non-empty subset of the set of A, B, and C. For example, in the illustrative example of a set with three members, the connective phrases "at least one of A, B, and C" and "at least one of A, B, and C" refer to any of the following sets: {A}, {B}, {C}, {A, B}, {A, C}, {B, C}, {A, B, C}. Thus, such connective language is generally not intended to imply that certain embodiments require at least one of A, at least one of B, and at least one of C to be present. Furthermore, unless expressly stated otherwise or contradicted by context, the term "plurality" refers to a plural state (e.g., "a plurality of items" refers to a plurality of items). In at least one embodiment, the number of the plurality of items is at least two, but may be more when expressly or by context indicates. Further, the phrase "based on" means "based at least in part on" rather than "based solely on" unless otherwise specified or clear from context.

[0065] Unless otherwise specified herein or clearly contradicted by context, the operations of the processes described herein may be performed in any suitable order. In at least one embodiment, processes such as the processes described herein (or variations and / or combinations thereof) are performed under the control of one or more computer systems configured with executable instructions and implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) that is jointly executed by hardware or a combination thereof on one or more processors. In at least one embodiment, the code is stored on a computer-readable storage medium, for example, in the form of a computer program that includes a plurality of instructions that can be executed by one or more processors.

[0066] In at least one embodiment, the computer-readable storage medium is a non-transitory computer-readable storage medium that does not include a transient signal (e.g., a propagating transient electrical or electromagnetic transmission), but includes a non-transitory data storage circuit (e.g., a buffer, cache, and queue) within a transient signal transceiver. In at least one embodiment, the code (e.g., executable code or source code) is stored on a set of one or more non-transitory computer-readable storage media that have executable instructions stored thereon (or other memory for storing executable instructions) that, when executed (i.e., as a result of execution) by one or more processors of a computer system, cause the computer system to perform the operations described herein. In at least one embodiment, the set of non-transitory computer-readable storage media includes a plurality of non-transitory computer-readable storage media, and one or more individual non-transitory storage media of the plurality of non-transitory computer-readable storage media lack all of the code, while the plurality of non-transitory computer-readable storage media collectively store all of the code. In at least one embodiment, the executable instructions are executed so that different instructions are executed by different processors - for example, a non-transitory computer-readable storage medium stores the instructions, and a main central processing unit ("CPU") executes some instructions while a graphics processing unit ("GPU") executes other instructions. In at least one embodiment, different components of the computer system have separate processors, and different processors execute different subsets of instructions.

[0067] In at least one embodiment, the arithmetic logic unit is a group of combinational logic circuits that use one or more inputs to produce a result. In at least one embodiment, the processor uses the arithmetic logic unit to implement mathematical operations, such as addition, subtraction, or multiplication. In at least one embodiment, the arithmetic logic unit is used to implement logical operations, such as logical and / or or exclusive or. In at least one embodiment, the arithmetic logic unit is stateless and is made of physical switch components (such as semiconductor transistors arranged to form logic gates). In at least one embodiment, the arithmetic logic unit can be operated internally as a state logic circuit with an associated clock. In at least one embodiment, the arithmetic logic unit can be constructed as an asynchronous logic circuit whose internal state is not maintained in an associated register group. In at least one embodiment, the processor uses the arithmetic logic unit to combine operands stored in one or more registers of the processor and generate outputs that can be stored by the processor in another register or memory location.

[0068] In at least one embodiment, as a result of processing instructions retrieved by the processor, the processor presents one or more inputs or operands to the arithmetic logic unit so that the arithmetic logic unit produces a result based at least in part on the instruction code provided to the input of the arithmetic logic unit. In at least one embodiment, the instruction code provided by the processor to the ALU is based at least in part on the instructions executed by the processor. In at least one embodiment, the combinational logic in the ALU processes the inputs and produces outputs placed on a bus within the processor. In at least one embodiment, the processor selects a target register, memory location, output device, or output storage location on an output bus so that the processor is clocked so that the result produced by the ALU is sent to the desired location.

[0069] Thus, in at least one embodiment, a computer system is configured to implement one or more services that individually or collectively perform the operations of the processes described herein, and such computer system is configured with applicable hardware and / or software that allows the operations to be performed. In addition, the computer system implementing at least one embodiment of the present disclosure is a single device, and in another embodiment is a distributed computer system that includes multiple devices that operate in different ways, so that the distributed computer system performs the operations described herein, and so that a single device does not perform all operations.

[0070] The use of any and all examples or exemplary language (e.g., "such as") provided herein is intended only to better illustrate embodiments of the present disclosure and does not limit the scope of the present disclosure unless otherwise stated. No language in the specification should be construed as indicating that any non-claimed element is essential to the practice of the disclosure.

[0071] In the specification and claims, the terms "coupled" and "connected" and their derivatives may be used. It should be understood that these terms may not be synonymous with each other. On the contrary, in specific examples, "connected" or "coupled" may be used to indicate that two or more elements are in direct or indirect physical or electrical contact with each other. "Coupled" may also mean that two or more elements are not in direct contact with each other, but still cooperate or interact with each other.

[0072] Unless expressly stated otherwise, it is to be understood that throughout the specification, terms such as “process,” “compute,” “calculate,” “determine,” etc. refer to the actions and / or processes of a computer or computing system or similar electronic computing device that manipulates and / or transforms data represented as physical (e.g., electronic) quantities in the computing system's registers and / or memories into other data similarly represented as physical quantities in the computing system's memories, registers, or other such information storage, transmission, or display devices.

[0073] Similarly, the term "processor" may refer to any device or part of a device that processes electronic data from registers and / or memory and converts the electronic data into other electronic data that can be stored in registers and / or memory. As a non-limiting example, a "processor" may be a CPU or a GPU. A "computing platform" may include one or more processors. As used herein, a "software" process may include, for example, software and / or hardware entities that perform work over time, such as tasks, threads, and intelligent agents. In addition, each process may refer to multiple processes for executing instructions sequentially or in parallel, continuously or intermittently. In at least one embodiment, the terms "system" and "method" are used interchangeably herein, as long as a system can embody one or more methods and a method can be considered a system.

[0074] In this document, reference may be made to obtaining, acquiring, receiving or inputting analog or digital data into a subsystem, a computer system or a computer-implemented machine. In at least one embodiment, the process of obtaining, acquiring, receiving or inputting analog and digital data may be accomplished in a variety of ways, such as by receiving data as a parameter of a function call or a call to an application programming interface. In at least one embodiment, the process of obtaining, acquiring, receiving or inputting analog or digital data may be accomplished by transmitting data via a serial or parallel interface. In at least one embodiment, the process of obtaining, acquiring, receiving or inputting analog or digital data may be accomplished by transmitting data from a providing entity to an acquiring entity via a computer network. Reference may also be made to providing, outputting, transmitting, sending or presenting analog or digital data. In at least one embodiment, the process of providing, outputting, transmitting, sending or presenting analog or digital data may be accomplished by transmitting data as an input or output parameter of a function call, a parameter of an application programming interface or an interprocess communication mechanism.

[0075] Although the description herein sets forth an example implementation of the technology, other architectures may be used to implement the functionality and are intended to be within the scope of the present disclosure. In addition, although specific allocations of responsibilities may be defined above for descriptive purposes, the various functions and responsibilities may be allocated and divided in different ways depending on the circumstances.

[0076] Furthermore, although the subject matter has been described in language specific to structural features and / or methodological acts, it should be understood that the subject matter claimed in the appended claims is not necessarily limited to the specific features or acts described. Rather, the specific features and acts are disclosed as exemplary forms of implementing the claims.

Claims

1. A video encoder for use in association with an interface for receiving from an application at least one metric associated with a quality preference for video compression to be performed by the video encoder, and for providing a weight map to enable the video encoder to perform rate-distortion optimization (RDO) on frames received from the application using the weight map to weight one or more first blocks associated with a single one of the frames to be larger than one or more second blocks associated with the single one of the frames.

2. The video encoder of claim 1, wherein the interface is an application-aware interface adapted to notify the application of capabilities associated therewith and to enable receipt of the at least one metric from the application.

3. The video encoder according to claim 1, further comprising: At least one execution unit associated with the video encoder is configured to execute the RDO or include the interface.

4. The video encoder according to claim 1, further comprising an execution unit, wherein the execution unit is configured to: executing a noise estimation algorithm to provide noise information, the noise estimation algorithm being temporal, spatial, or a combination thereof; determining that the one or more second blocks associated with the single one of the frames are more affected by noise in the noise information than the one or more first blocks associated with the single one of the frames; and Additional weighting is performed on the one or more first blocks and the one or more second blocks based at least in part on the noise.

5. The video encoder of claim 1, wherein the weight map comprises weights based at least in part on one or more edges within the single one of the frames or based at least in part on a visual quality metric of the at least one metric.

6. The video encoder of claim 1, wherein the weight map comprises weights based at least in part on an output of a detection or classification model applied to the single one of the frames.

7. The video encoder of claim 1, wherein the at least one metric comprises a visual quality metric, and wherein the weight map comprises weights based at least in part on the visual quality metric.

8. A system comprising: One or more processing units for enabling rate-distortion optimization (RDO) to be performed on frames received from an application based in part on a weight map associated with at least one metric of a quality preference of the application, the weight map being configured to provide a first weight to one or more first blocks associated with a single one of the frames, and a second weight less than the first weight to one or more second blocks associated with the single one of the frames.

9. The system of claim 8, wherein the one or more processing units are further configured to: executing a noise estimation algorithm to provide noise information, the noise estimation algorithm being temporal, spatial, or a combination thereof; determining that the one or more second blocks associated with the single one of the frames are more affected by noise in the noise information than the one or more first blocks associated with the single one of the frames; and Additional weighting is performed on the one or more first blocks and the one or more second blocks based at least in part on the noise.

10. The system of claim 8, wherein the one or more processing units are further configured to: The first weight and the second weight are determined based at least in part on one or more edges within the single one of the frames or based at least in part on a visual quality metric of the at least one metric.

11. The system of claim 8, wherein the one or more processing units are further configured to: executing a detection or classification model on the single one of the frames; and The first weight and the second weight are determined based at least in part on an output of the detection or classification model.

12. The system of claim 8, wherein the at least one metric comprises a visual quality metric, and the one or more processing units are further configured to: The first weight and the second weight are determined based at least in part on a visual quality metric of the at least one metric.

13. A system comprising: One or more processing units for: executing an application associated with the video data; Determining a feature associated with a video encoder, the feature associated with at least one metric of a quality preference of the application; and generating a weight map associated with the metric to enable the video encoder to perform rate-distortion optimization (RDO) on frames received from the application using the weight map to weight one or more first blocks associated with a single frame in the frame more than one or more second blocks associated with the single frame in the frame.

14. The system of claim 13, wherein the one or more processing units are further configured to: executing a noise estimation algorithm to provide noise information, the noise estimation algorithm being temporal, spatial, or a combination thereof; determining that the one or more second blocks associated with the single one of the frames are more affected by noise in the noise information than the one or more first blocks associated with the single one of the frames; and Additional weighting is performed on the one or more first blocks and the one or more second blocks based at least in part on the noise.

15. The system of claim 13, wherein the one or more processing units are further configured to: The weights in the weight map are determined based at least in part on one or more edges within the single one of the frames or based at least in part on a visual quality metric of the at least one metric.

16. The system of claim 13, wherein the one or more processing units are further configured to: executing a detection or classification model on the single one of the frames; The weights in the weight map are determined based at least in part on an output of the detection or classification model.

17. A method for a video encoder, comprising: receiving at least one metric associated with a quality preference of an application associated with video compression to be performed by the video encoder; as well as Rate-distortion optimization (RDO) is performed on frames received from the application based at least in part on a weight map associated with the at least one metric, the weight map weighting one or more first blocks associated with a single one of the frames more than one or more second blocks associated with the single one of the frames.

18. The method according to claim 17, further comprising: executing a noise estimation algorithm to provide noise information, the noise estimation algorithm being temporal, spatial, or a combination thereof; determining that the one or more second blocks associated with the single one of the frames are more affected by noise in the noise information than the one or more first blocks associated with the single one of the frames; as well as Additional weighting is performed on the one or more first blocks and the one or more second blocks based at least in part on the noise.

19. The method according to claim 17, further comprising: The weights as part of the weight map are determined based at least in part on one or more edges within the single one of the frames or based at least in part on a visual quality metric of the at least one metric.

20. The method of claim 17, further comprising: executing a detection or classification model on the single one of the frames; as well as Weights as part of the weight map are determined based at least in part on outputs of the detection or classification model.

Images