Codec Rate-Distortion Compensation Downsampler

Abstract

A system includes a video downsampler based on a machine learning (ML) model, the video downsampler configured to receive an input video sequence having a first display resolution and map the input video sequence to a lower-resolution video sequence having a second display resolution, the second display resolution being lower than the first display resolution. The system also includes a neural network-based (NN-based) proxy video codec configured to transform the lower-resolution video sequence into a decoded proxy bitstream. Additionally, the system includes an upsampler configured to generate an output video sequence using the decoded proxy bitstream.

Description

Background Art

[0001] Downsampling is an operation in content streaming systems to produce different representations in terms of bitrate and resolution that are usable by different types of client devices. In modern streaming systems, the streaming server provides different encoded representations in terms of resolution and bitrate so that the client device can dynamically download the representation that best matches its playback context (e.g., display size and network conditions). To provide such representations, the streaming server needs to downsample the source video to different resolutions before encoding. Downsampling can be performed with filters that are not perceptually optimal. BRIEF DESCRIPTION OF THE DRAWINGS

[0002] Figure 1 A diagram illustrating an exemplary video processing system including a codec rate distortion compensated downsampler based on a trained machine learning (ML) model according to one embodiment is shown;

[0003] Figure 2 A method for training according to one embodiment is shown. Figure 1 Figure 1 shows a training pipeline system for a codec rate-distortion compensation downsampler based on an ML model;

[0004] Figure 3 Describing various embodiments Figure 1 An alternative exemplary architecture for an ML model-based codec rate-distortion compensation downsampler as shown in ;

[0005] Figure 4 A flow chart outlining an exemplary method for training an ML model-based codec rate-distortion compensation downsampler according to one embodiment is shown;

[0006] Figure 5A FIG. 1 shows an embodiment of the present invention according to an aspect of the present invention. Figure 1 a diagram of an exemplary conditional downsampling network implemented by a video processing system of FIG; and

[0007] Figure 5B Another aspect of the present invention is shown. Figure 1 Diagram of an exemplary conditional downsampling network implemented by a video processing system. DETAILED DESCRIPTION

[0008] The following description contains specific information related to the embodiments of the present disclosure. Those skilled in the art will recognize that the present disclosure can be implemented in a manner different from that specifically discussed herein. The drawings and detailed descriptions thereof in this application are for exemplary embodiments only. Unless otherwise indicated, the same or corresponding elements in the drawings may be represented by the same or corresponding reference numerals. In addition, the drawings and illustrations in this application are generally not depicted to scale and are not intended to correspond to actual relative sizes.

[0009] The present application discloses systems and methods for training and utilizing codec rate-distortion compensating downsamplers based on machine learning (ML) models, which overcome the shortcomings and deficiencies in the prior art. Filters such as bilinear, cubic, or Lanczos filters are not perceptually optimal and do not take into account that the source content will be encoded after downsampling. To this end, the present application discloses a perceptually optimized downsampling method comprising 1) a learned downsampler, 2) a proxy video codec that emulates a standard-compliant image or video codec, 3) a time-aware loss function, 4) an adjustment mechanism that allows inference time interpolation between different loss functions, and 5) a mechanism for arbitrary scaling. The downsampling solutions disclosed in the present application are applied only to the source content. Therefore, these solutions are compatible with existing image or video encoding pipelines and do not require any changes on the client side, while providing improved rate-distortion performance in terms of perceptual quality metrics. In addition, in some embodiments, the codec rate-distortion compensating downsampling solution of the present invention can be implemented as an automated process.

[0010] It is worth noting that, as defined in this application, the terms "automation," "automated," and "automating" refer to systems and processes that do not require the involvement of a human user, such as a human editor or system administrator. For example, while in some embodiments a human system administrator may review the performance of the systems and methods disclosed herein, or in the case of conditional downsampling discussed below, may provide user-defined perceptual loss function weights, human involvement is optional. Thus, in some embodiments, the processes described in this application may be performed under the control of hardware processing components of the disclosed systems.

[0011] It should also be noted that, as defined in this application, the expression "machine learning model" or "ML model" may refer to a mathematical model for making future predictions based on patterns learned from data samples or "training data". Various learning algorithms can be used to map the correlations between input data and output data. These correlations form a mathematical model that can be used to make future predictions for new input data. Such a prediction model may include one or more logistic regression models, Bayesian models, or neural networks (NNs). Furthermore, in the context of deep learning, a "deep neural network" may refer to a NN that utilizes multiple hidden layers between the input layer and the output layer, which may allow learning based on features that are not clearly defined in the original data. In various embodiments, a NN may be trained as a classifier and may be used to perform image processing or natural language processing.

[0012] It should also be noted that, as defined in this application, the feature "proxy video codec" refers to a NN or other ML model-based software module that has been pre-trained to replicate the rate-distortion characteristics of a standard codec (e.g., H.264 or AV1). In contrast, existing deep learning-based codecs are not trained to replicate the distortion characteristics of standard codecs. As a result, the downsampling artifacts introduced by existing deep learning-based codecs are very different from the downsampling artifacts produced by the standard codec and replicated by the proxy video codec disclosed herein. That is, by introducing a knowledge distillation-based proxy video codec that aims to reproduce the output of a traditional image / video codec, the method disclosed in this application advantageously facilitates the optimization of downsampler performance for the actual distortion produced by the standard codec.

[0013] Figure 1 FIG. 1 is a diagram illustrating an exemplary video processing system 100 including a trained ML model-based codec rate-distortion compensated video downsampler 112 (hereinafter referred to as “ML model-based video downsampler 112”) according to one embodiment. Figure 1 As shown, the video processing system 100 includes a computing platform 102 having processing hardware 104 and a system memory 106 implemented as a computer-readable non-transitory storage medium. According to the present exemplary embodiment, the system memory 106 stores a trained ML model-based video downsampler 112, a video codec 114, and a simulation module 120.

[0014] like Figure 1As further shown, the video processing system 100 is implemented in a use environment that includes a video source 134 providing a video sequence 138 (hereinafter referred to as "input video sequence 138"), a communication network 130, and a media content consumer 108 utilizing a client device 140 including a display 148. In addition, Figure 1 A network communication link 132 is shown communicatively coupling a video source 134 and a client device 140 with the video processing system 100 via the communication network 130 . Figure 1 Also shown in FIG. 1 are a lower resolution video sequence 152 and a decoded bitstream 146 corresponding to the input video sequence 138 .

[0015] It should be noted that while the video processing system 100 can receive an input video sequence 138 from a video source 134 via the communication network 130 and the network communication link 132, in some embodiments, the video source 134 can take the form of a content source integrated with the computing platform 102, or can communicate directly with the video processing system 100, as indicated by the dashed communication link 136. It should also be noted that in some embodiments, the video processing system 100 can omit the simulation module 120. Therefore, in some embodiments, the system memory 106 can store the trained ML model-based video downsampler 112 and video codec 114 instead of the simulation module 120.

[0016] The input video sequence 138 may include audio-video (AV) content in the form of, for example, a video game, a movie, or episodic program content, including streamed or broadcast episodic content. The input video sequence 138 may include a high-definition (HD) or ultra-high-definition (UHD) baseband video signal with embedded audio, subtitles, timecode, and other auxiliary metadata (e.g., ratings and parental guidance). In some implementations, the input video sequence 138 may be provided by a video source 134, such as a TV broadcast network or other media distribution entity (e.g., a movie studio, a streaming platform, etc.), using, for example, a supplementary audio program (SAP) or a descriptive video service (DVS).

[0017] about Figure 1, it should be noted that although the trained ML model-based video downsampler 112, video codec 114, and simulation module 120 are depicted as being stored in the system memory 106 for conceptual clarity, more generally, the system memory 106 can take the form of any computer-readable non-transitory storage medium. As used in this application, the expression "computer-readable non-transitory storage medium" refers to any medium other than a carrier wave or other transient signal that provides instructions to the processing hardware of the computing platform (such as the processing hardware 104 of the computing platform 102). Therefore, the computer-readable non-transitory storage medium can correspond to various types of media, such as volatile media and non-volatile media. Volatile media can include dynamic memory, such as dynamic random access memory (dynamic RAM), while non-volatile memory can include optical, magnetic, or electrostatic storage devices. Common forms of computer-readable non-transitory storage media include, for example, optical disks, RAM, programmable read-only memory (PROM), erasable PROM (EPROM), and FLASH memory.

[0018] It should also be noted that although Figure 1 The trained ML model-based video downsampler 112, video codec 114, and simulation module 120 are depicted as being co-located with one another in the system memory 106, but this representation is provided merely for conceptual clarity. More generally, the video processing system 100 may include one or more computing platforms, such as computer servers, which may be co-located or may form an interlinked but distributed system, such as a cloud-based system. As a result, the processing hardware 104 and the system memory 106 may correspond to distributed processor and memory resources within the video processing system 100. Therefore, it should be understood that the trained ML model-based video downsampler 112, video codec 114, and simulation module 120 may be stored remotely from one another within the distributed memory resources of the video processing system 100. Furthermore, and as described above, in some embodiments, the simulation module 120 may be omitted from the system memory 106.

[0019] The processing hardware 104 may include multiple hardware processing units, such as one or more central processing units, one or more graphics processing units, and one or more tensor processing units, one or more field programmable gate arrays (FPGAs), custom hardware for machine learning training or reasoning, and an application programming interface (API) server. As a definition, as used in this application, the terms "central processing unit" (CPU), "graphics processing unit" (GPU), and "tensor processing unit" (TPU) have their customary meanings in the art. That is, the CPU includes an arithmetic logic unit (ALU) for performing arithmetic and logical operations of the computing platform 102 and a control unit (CU) for retrieving programs from the system memory 106, while the GPU can be implemented to reduce the processing overhead of the CPU by performing computationally intensive graphics or other processing tasks. The TPU is a dedicated integrated circuit (ASIC) configured specifically for artificial intelligence (AI) applications (e.g., machine learning modeling).

[0020] In some embodiments, computing platform 102 may correspond to one or more web servers accessible, for example, via a packet-switched network such as the Internet. Alternatively, computing platform 102 may correspond to one or more computer servers supporting a dedicated wide area network (WAN), a local area network (LAN), or included in another type of limited distribution or dedicated network. As yet another alternative, in some embodiments, video processing system 100 may be implemented virtually, such as in a data center. For example, in some embodiments, video processing system 100 may be implemented in software or as a virtual machine.

[0021] Although the client device 140 Figure 1 140, but this representation is provided by way of example only. More generally, client device 140 may be any suitable mobile or fixed computing device or system that implements data processing capabilities sufficient to provide a user interface, support connection to communication network 130, and implement the functionality attributed herein to client device 140. For example, in some embodiments, client device 140 may take the form of a desktop computer, a laptop computer, a tablet computer, a smartphone, a digital media player, a gaming platform, or a smart wearable device such as a smartwatch.

[0022] Regarding the display 148 of the client device 140, the display 148 can be physically integrated with the client device 140, or can be communicatively coupled to the client device 140 but physically separate therefrom. For example, if the client device 140 is implemented as a smart TV, a smart phone, a laptop computer, a tablet computer, or a smartwatch, the display 148 can be integrated with the client device 140. Conversely, if the client device 140 is implemented as a desktop computer or a gaming platform, the display 148 can take the form of a monitor separate from the client device 140, taking the form of a computer tower or a gaming console, respectively. Furthermore, the display 148 can be implemented as a liquid crystal display (LCD), a light emitting diode (LED) display, an organic light emitting diode (OLED) display, a quantum dot (QD) display, or any other suitable display screen that performs a physical transformation of a signal into light.

[0023] As an overview of the runtime or inference time performance of the video processing system 100, it is noteworthy that the processing hardware 104 of the computing platform 102 may receive an input video sequence 138 having a first display resolution and may use the trained ML model-based video downsampler 112 to map the input video sequence 138 to a lower resolution video sequence 152 having a second display resolution that is lower than the first display resolution. Figure 1 ), is used to train the trained ML model-based video downsampler 112, which is a proxy video codec that has been pre-trained to replicate the rate-distortion characteristics of the video codec 114 in the form of a standard video codec.

[0024] Due to the training of the trained ML model-based video downsampler 112, the downsampling of the input video sequence 138 performed by the ML model-based video downsampler 112 causes the lower resolution video sequence 152 to be substantially optimized for processing by the video codec 114. The processing hardware 140 may then transform the lower resolution video sequence 152 into a decoded bitstream 146 using the video codec 114 and may output the decoded bitstream 146. For example, Figure 1 As shown, in some embodiments, the video processing system 100 can output the decoded bitstream 146 to the client device 140 via the communication network 130 and the network communication link 132 for display to the media content consumer 108 on the display 148 as an output video sequence corresponding to the input video sequence 138.

[0025] Figure 2 A method for training according to one embodiment is shown. Figure 1 FIG. 2 is a diagram of a training pipeline system 210 for a video downsampler 112 based on a trained ML model. Figure 2 As shown, the training pipeline system 210 includes an ML model-based video downsampler 211 , an emulation module 220 including an NN-based proxy video codec 222 , an upsampler 226 , and an optimization block 260 . Figure 2 Also shown are a training input video sequence 227, a lower resolution video sequence 250 downsampled from the training input video sequence 227, a decoded proxy bitstream 224 output by the NN-based proxy video codec 222, and a training output video sequence 228 corresponding to the training input video sequence 227.

[0026] Before training the trained ML model-based video downsampler 112 using the training pipeline system 210, the ML model-based video downsampler 211 generally corresponds to Figure 1 Thus, the ML model-based video downsampler 211 may share any architectural features attributed to the trained ML model-based video downsampler 112 by the present disclosure, and vice versa. Figure 2 The simulation module 220 in the embodiment generally corresponds to Figure 1 2. Thus, the simulation module 120 may share any characteristics attributed by the present disclosure to the simulation module 220, and vice versa. That is, the simulation module 120 may include features corresponding to one or more of the NN-based proxy video codec 222, the upsampler 226, and the optimization block 260, respectively.

[0027] Thus, the training pipeline system 210 includes an ML model-based video downsampler 211 configured to receive a training input video sequence 227 having a first display resolution and map the training input video sequence 227 to a lower resolution video sequence 252 having a second display resolution lower than the first display resolution. In addition, the training pipeline system 210 also includes an NN-based proxy codec 222 configured to transform the lower resolution video sequence 250 into a decoded proxy bitstream 224, and an upsampler 226 configured to receive the decoded proxy bitstream 224 and generate an output training video sequence 228 corresponding to the input training video sequence 227 and having a higher second display resolution.

[0028] Referring to the training pipeline system 210, given a high-resolution input video sequence with L frames:

[0029]

[0030] and a target downsampling ratio 0<s<1, the video downsampler 112 based on the trained ML model, denoted herein by the symbol “D”, performs the operation of generating a downsampled version:

[0031]

[0032] therefore:

[0033]

[0034] Different learning strategies can be used to implement a trained ML model-based video downsampler D. For example, possible examples of D may include a feature extraction module, followed by a downsampling module, and finally another filtering or refinement module that predicts the downsampled result, fixes the downsampled residual, or a kernel that allows upsampling the desired lower resolution result. To this end, Figure 3 Four optional embodiments of an ML model-based video downsampler architecture are shown, which are suitable for being adopted by the ML model-based video downsampler 211 or the trained ML model-based video downsampler 112, which use convolutional neural networks (CNNs) to implement downsampling of features extracted from convolutional layers in different ways.

[0035] For example, reference Figure 3 , the ML model based video downsampler architecture 312a uses strided convolution (“StridedConvolution”), while the ML model based video downsampler architecture 312b is characterized by utilizing traditional filters such as linear, bicubic or Lanczos filters for resampling, for example, the ML model based video downsampler architecture 312c is similar to the ML model based video downsampler architecture 312b, but additionally uses accurate sub-pixel offset and scale information as additional inputs to the convolutional layers. The ML model based video downsampler architecture 312d is designed to predict a sub-pixel aware kernel that can then be applied to the original To generate Given the nature of strided convolution, the ML model-based video downsampler architecture 312a only supports integer downsampling or reduction factors, while the ML model-based video downsampler architectures 312b, 312c, and 312d can also advantageously support arbitrary scaling factors. Therefore, in various embodiments, the trained ML model-based video downsampler 112 can include a CNN. Furthermore, in some embodiments, the trained ML model-based video downsampler 112 can advantageously be configured to support arbitrary, i.e., non-integer as well as integer, scaling factors.

[0036] For video-based downsampling, temporal correlations between frames are also considered, and a perceptually correct loss function is employed, as described in more detail below. However, it is easy to understand that when L=1, the trained ML-based video downsampler 112 reduces to the case of an image-only downsampler operating on a single frame input. When L>1, the input to the downsampler is a sequence of video frames, and Figure 2 The exemplary training pipeline system 210 in can utilize two-dimensional (2D) or three-dimensional (3D) convolutions, thereby considering the time frame sequence information during downsampling.

[0037] NN-based proxy video codec 222:

[0038] One challenge in video downsamplers 211 based on end-to-end trained ML models is that current compression standards are based on complex heuristics and are therefore non-differentiable. To address and overcome this limitation, this application discloses a novel and inventive knowledge distillation method based on a NN-based proxy video codec 222, which is denoted by Representation. The NN-based proxy video codec 222 is pre-trained with data generated from a specific video codec of a specific standard (e.g., H.264, HEVC / H.265, MPEG-4, or AV1) to predict the rate-distortion characteristics of the specific standard video codec in a differentiable manner. That is, the NN-based proxy video codec 222 is pre-trained to replicate the rate-distortion characteristics of the standard video codec, which rate-distortion characteristics may be unique to the specific standard video codec. Furthermore, the NN-based proxy video codec 222 is advantageously differentiable.

[0039] During pre-training, the learning goal of the NN-based proxy video codec 222 is to learn to predict the artifacts that a standard video codec would produce. Receive video And output distorted video according to specific codec and its estimated rate Instead of being trained only for a specific target rate, the NN-based proxy video codec 222 can be conditioned on additional encoding parameters, including, for example, a target bitrate, in order to predict distortion for a variety of different settings. The NN-based proxy video codec 222 can predict the final frame and the distortion, or just the distortion itself. As described above, the NN-based proxy video codec 222 can be pre-trained offline and can be used in the training pipeline system 210 to train the ML model-based video downsampler 211. Importantly, according to the codec rate-distortion compensation solution disclosed in this application, the effect of compression is included in the loss function used to train the NN-based proxy video codec 222 so that the NN-based proxy video codec 222 can replicate the rate-distortion characteristics of the standard codec. Notably, during inference, i.e., at runtime, the real video codec 114 is used in conjunction with the trained ML model-based video downsampler 112 to produce a backward-compatible compressed bitstream Loss function:

[0040] The choice of loss function is nontrivial, especially in the constrained setting where only the downsampling operation is optimized and the upsampling is fixed. In particular, it may be insightful to consider the following example: Suppose Figure 1 The fixed-scale upscaling on the client device 140 in [ 14 ] is performed using a nearest neighbor / box filter technique for interpolation. In this case, regardless of the downsampling technique used, the output generated on the display 148 will appear pixelated, and averaging / box filtering will provide optimal downscaling with respect to the L2 loss. The specific downsampler architecture is also irrelevant as long as it can compute the average, and the improvement can only be generated by the loss function. Therefore, in order to achieve a more perceptually accurate downsampling result, it is necessary to optimize the loss function for a better representation of human vision.

[0041] However, because different losses may best represent different features of the human visual system, the solution disclosed in this paper adopts an objective function that combines multiple different perceptual loss functions while also considering the following rates:

[0042]

[0043] Here, the weights are the unit decomposition (∑ω i =1), and These are different perceptual losses such as L1, L2 losses based on Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index metric (SSIM), Learned Perceptual Patch Similarity (LPIPS) metric, Deep Image Structure and Texture Similarity (DISTS) metric, or even Regenerative Adversarial Network (GAN) based losses. is based on An estimated rate of the lower resolution video sequence 250.

[0044] Thus, the optimization block 260 of the training pipeline system 210 can train the ML model-based video downsampler 211 using the training input video sequence 227, the training output video sequence 228, and an objective function based on the estimated rate of the lower resolution video sequence 250 and the plurality of perceptual loss functions. Furthermore, in some embodiments, as specifically shown in Equation 4 above, the objective function for training the ML model-based video downsampler 211 to provide the trained ML model-based video downsampler 212 includes a combination of the estimated rate of the lower resolution video sequence 250 and a weighted sum of the plurality of perceptual loss functions.

[0045] It is worth noting that 2. The upsampler 226 represents an upsampler that upsamples the lower-resolution video sequence 250 to a training output video sequence 228 having a display resolution greater than the display resolution of the lower-resolution video sequence 250, and which may have a display resolution that substantially matches the display resolution of the training input video sequence 227. It should also be noted that in various embodiments, the upsampler 226 may be implemented as a fixed upsampler or an ML model-based learning upsampler. Furthermore, in embodiments where the upsampler 226 is in the form of an ML model-based upsampler, the ML model-based upsampler 226 and the ML model-based video downsampler 211 may be trained simultaneously.

[0046] Temporal coherence:

[0047] As described above, when L>1, the input / output of the trained ML model-based video downsampler 112 is a frame sequence, which can be jointly considered in the objective function of Equation 4. This approach advantageously enables temporal stability to be enforced on the lower resolution video sequence 152 provided as the output of the ML model-based video downsampler 112. As an example, the frame sequence can be considered together to optimize the temporal contour, which facilitates a temporally coherent downscaling result.

[0048] The function of the training pipeline system 210 will refer to Figure 4 Further description, Figure 4 A flowchart 470 is shown representing an exemplary method used by the video processing system 100 to train the ML model-based video downsampler 211 according to one embodiment. Figure 4 It is noteworthy that certain details and features are omitted from flowchart 470 so as not to obscure the discussion of the inventive features of the present application.

[0049] Now combine Figure 2 refer to Figure 4 , the flowchart 470 may begin by providing (act 471) an input video sequence having a first display resolution (ie, the training input video sequence 227) to the ML model-based video downsampler 211. Further reference is made to Figure 1 , in action 471 , the training input video sequence 227 may be provided by the processing hardware 104 of the computing platform 102 to the ML model-based video downsampler 211 .

[0050] Flowchart 470 also includes mapping, using the ML model-based video downsampler 211, the training input video sequence 227 to a lower resolution video sequence 250 having a second display resolution that is lower than the first display resolution of the training input video sequence 227 (act 472). Act 472 effectively maps the high resolution video sequence to the low resolution video sequence and is performed by the ML model-based video downsampler 211 under the control of the processing hardware 104 of the computing platform 102.

[0051] Flowchart 470 also includes transforming the lower-resolution video sequence 250 into a decoded proxy bitstream 224 using the NN-based proxy video codec 222 (act 473). As described above, the NN-based proxy video codec 224 is pre-trained to replicate the rate-distortion characteristics of the video codec 114. Act 473 may be performed by the processing hardware 104 of the computing platform 102 using the NN-based proxy video codec 222 of the emulation module 220.

[0052] Flowchart 470 also includes generating, using upsampler 226 that receives decoded proxy bitstream 224, an output video sequence (i.e., training output video sequence 228) that corresponds to training input video sequence 227 and has a display resolution that is higher than the second display resolution, i.e., higher than the display resolution of lower-resolution video sequence 250 (act 474). In some embodiments, the display resolution of training output video sequence 228 can be a display resolution that is substantially the same as the first display resolution of training input video sequence 227. However, in other embodiments, the display resolution of training output video sequence 228, while greater than the second display resolution of lower-resolution video sequence 240, can be lower than the first display resolution of training input video sequence 227. Act 474 can be performed by processing hardware 104 of computing platform 102 using upsampler 226 of emulation module 220.

[0053] Flowchart 470 also includes training the ML model-based video downsampler 211 using the training input video sequence 227, the training output video sequence 228, and an objective function based on the estimated rate of the lower resolution video sequence and the plurality of perceptual loss functions (act 475). Act 475 may be performed using the optimization block 260 of the simulation module 220 under the control of the processing hardware 104 of the computing platform 102 in the manner described above with reference to Equation 4.

[0054] As described above, the objective function expressed in Equation 4 includes a combination of the estimated rate of the lower resolution video sequence 250 and a weighted sum of multiple perceptual loss functions. In some embodiments, the training of the ML model-based video downsampler 211 can be further based on the corresponding weighting factor (ω) applied to each perceptual loss function. i ). These weighting factors can be calculated based on an initial optimization of the objective function represented by Equation 4, can be selected by an administrator of the training pipeline system 210, or can include both calculated weighting factors and administrator-selected weighting factors. In some embodiments, the method outlined by flowchart 470 can also include providing these weighting factors as another input to the ML model-based video downsampler 211, and also using these weighting factor inputs to train the ML model-based downsampler 211. Therefore, in some embodiments, the ML model-based video downsampler 211 is also configured to receive the weighting factors included in the weighted sum of the multiple perceptual loss functions included in Equation 4.

[0055] With respect to the method outlined by flowchart 470 and described above, it is noted that in some implementations, actions 471 , 472 , 473 , 474 , and 475 may be performed in an automated process that may omit human involvement.

[0056] Figure 5A An embodiment of the present invention is shown. Figure 1 FIGURE 1 is a diagram of an exemplary conditional downsampling network implemented by the video processing system 100. Figure 5A As shown, the conditional downsampling network 500A includes a trained ML model-based video downsampler 512 , a video codec 514 , and a simulation module 520A including an upsampler 526 and an optimization block 560 . Figure 5A Also shown are samples 540 of the input video sequence 138 to the video processing system 100, lower resolution samples 552 downsampled from samples 540, a decoded bitstream 554A output by the video codec 514, output samples 556A corresponding to the input samples 540, and weighting factors 558.

[0057] The trained ML model-based video downsampler 512, video codec 514, and simulation module 520A generally correspond to Figure 1 14 and emulation module 120 in the trained ML model-based video downsampler 112, video codec 114, and emulation module 120. Thus, the trained ML model-based video downsampler 512, video codec 514, and emulation module 520A may share any characteristics attributed to the corresponding trained ML model-based video downsampler 112, video codec 114, and emulation module 120 according to the present disclosure, and vice versa. Additionally, the upsampler 526 generally corresponds to Figure 2 4, and may share any characteristics attributed to the corresponding features described above. That is, the upsampler 526 may be implemented as a fixed upsampler or as a learned upsampler based on an ML model. In addition, the weighting factors 558 correspond to the weighting factors applied to each perceptual loss function included in Equation 4, which may be calculated based on the initial optimization of the objective function of Equation 4, may be selected by an administrator of the video processing system 100, or may include both calculated weighting factors and administrator-selected weighting factors.

[0058] refer to Figure 5B , Figure 5B A description showing another aspect of the inventive concept is given by Figure 1 FIGURE 1 is a diagram of an exemplary conditional downsampling network implemented by the video processing system 100. Figure 5B As shown, the conditional downsampling network 500B includes a trained ML model-based video downsampler 512 and a simulation module 520B including a NN-based proxy video codec 522, an upsampler 526, and an optimization block 560. Figure 5B Also shown are samples 540 of the input video sequence 138 to the video processing system 100, lower resolution samples 552 downsampled from samples 540, a decoded bitstream 554B output by the proxy video codec 522, output samples 556B corresponding to the input samples 540, and weighting factors 558.

[0059] It is worth noting that in Figure 5B Zhong You and Figure 5A Any features identified by the same reference numerals as used in the examples correspond to those previously identified features, respectively, and may share any of the properties attributed to them above. Thus, the trained ML model-based video downsampler 512, video codec 514, and emulation module 520A generally correspond to Figure 1The trained ML model-based video downsampler 112, video codec 114, and simulation module 120 in FIG, and the weighting factors 558 correspond to the weighting factors applied to each perceptual loss function included in Equation 4. In addition, the NN-based proxy video codec 522 generally corresponds to Figure 2 The NN-based proxy video codec 222 in FIG. 5 and may share any characteristics attributed to the corresponding features described above. That is, the NN-based proxy video codec 522 may be configured to replicate the rate-distortion characteristics of the video codec 114 / 514 .

[0060] like Figure 5A and Figure 5B As shown, in various embodiments, the emulation module 120 / 520A or the emulation module 120 / 520B may include an upsampler 526 or both the NN-based proxy video codec 522 and the upsampler 526 .

[0061] Regarding the conditional downsampling networks 500A and 500B, it is noteworthy that these networks are trained to interpolate between loss functions and enable the administrator of the video processing system 100 to control the level of detail included in the downsampled video. In this case, Figure 5A and Figure 5B As shown, the trained ML model-based video downsampler 512 may also receive the weighting factor 558. That is, in this embodiment, the downsampling algorithm used by the ML model-based video downsampler 512 is the weighting factor 558 and And the function of s:

[0062]

[0063] In order to work correctly in this conditional implementation, the trained ML model-based video downsampler 512 should have a wide range of different loss weighting factors during training, such as randomly sampled loss weighting factors. The conditional downsampling network trained with random loss weighting factors can successfully interpolate between perceptual loss functions during inference with administrator-controlled parameters. As will be discussed with reference to some of the specific limitations described below, the same parameter approach can also be used with a single network trained for different upsamplers, different video codecs, different quantization parameters, etc.

[0064] In some embodiments, it may be advantageous or desirable for the trained ML model-based video downsampler 112 / 512 of the video processing system 100 to adaptively respond to the type of content included in the input video sequence 138. Thus, in some embodiments, before the input video sequence 138 is mapped to the lower resolution video sequence 152 by the trained ML model-based video downsampler 112 / 512, the processing hardware 104 may extract samples 540 of the input video sequence 138 and may use the trained ML model-based video downsampler 112 / 512 to map the samples 540 to the lower resolution samples 552. The processing hardware 104 may also use, for example, Figure 5A The video codec 114 / 514 shown or using Figure 5B The proxy video codec 522 shown transforms the lower resolution samples 552 into a decoded bitstream 554A or 554B and uses the upsampler 526 and the decoded sample bitstream 554A or 554B to predict output samples 556A or 556B corresponding to the extracted samples 540. The processing hardware 104 can then modify one or more parameters of the trained ML model-based video downsampler 112 / 512 based on the predicted output samples 556A or 556B, thereby advantageously rendering the trained ML model-based video downsampler 112 / 512 content adaptively.

[0065] In addition to the perceptually optimized downsampling framework discussed above, there are numerous specializations and extensions supported by novel and creative approaches, including:

[0066] Image-only downsampling: As described above, as a specific use case, when using an image-only proxy codec and L=1, the present solution also enables perceptually optimized downsampling for the image codec.

[0067] Single image / video downsampling: As a specific use case, the trained ML model-based video downsampler 112 / 512 can be further optimized on a per-content basis by training a different network for each of the different types of content.

[0068] Identity Proxy Codec: By using an identity proxy, perceptually optimized downsampling can be supported without the need to train the codec in the pipeline system 210.

[0069] Multiple output resolutions: The solution can be easily adapted to support multiple output resolutions, which can, for example, match the resolution required by a streaming service.

[0070] Optimal downsampling for learning super-resolution: As described above, refer to Figure 2In some implementations, the upsampler 226 may be a learned super-resolution upsampler. In this case, the super-resolution upsampler may also be trained end-to-end using the ML model-based video downsampler 211.

[0071] Multiple quality levels and upsamplers: The framework of this solution is flexible. On the one hand, downsampling can be optimized for a specific codec setting and upsampler. Alternatively, the upsampling algorithm can be trained for multiple different codec settings and upsamplers, resulting in a single downsampling algorithm that is compatible with a wide variety of upsamplers and codec settings.

[0072] Parametric Upsampling: Similar to the conditional downsampling discussed above, the method disclosed in this application can also be extended to support multiple upsampling kernels.

[0073] Parametric Codecs: It is also possible to train a single downsampler and a single proxy codec network conditioned on different standard codecs.

[0074] Chroma Subsampling: To avoid unnecessary conversions and overhead, the video processing solution disclosed herein can also operate directly on YUV420p or other content with chroma subsampling.

[0075] Pre-processing: Besides downsampling, there may be other pre-processing tasks where optimization would be beneficial. Note that if the downsampling factor = 1, the present solution is optimized for mitigating compression artifacts.

[0076] Thus, the present application discloses systems and methods for training and utilizing a codec rate-distortion compensated downsampler based on a machine learning model that overcomes the shortcomings and deficiencies in the prior art. The present solution improves upon the prior art in several ways. For example, compared to non-learning techniques, the present solution utilizes a data-driven approach to identify optimal downsampling based on multiple perceptual loss functions. Compared to existing learning methods (none of which consider the codec during training), the present solution uses a pre-trained proxy video codec to replicate the rate-distortion characteristics of a standard video codec. Compared to existing learning methods, the perceptually optimized downsampler disclosed herein advantageously 1) supports more perceptual loss functions (e.g., LPIPS, DISTS, and temporal loss), 2) can provide sub-pixel position-aware convolutions, 3) integrates the codec in the training loop, and 4) supports system administrator-controllable parameters that enable fine-grained control based on content type.

[0077] Furthermore, the pre-trained proxy codec used in the present solution differs significantly from existing deep learning-based codecs, which generate artifacts that are very different from those produced by standard codecs. Therefore, because the present solution utilizes a pre-trained proxy codec to reproduce the output of a traditional codec, the method disclosed herein is advantageously able to better compensate for the distortion produced by such standard codecs.

[0078] From the above description, it is apparent that various techniques can be used to realize the concepts described in this application without departing from the scope of these concepts. In addition, although concepts have been described with specific reference to certain embodiments, those of ordinary skill in the art will recognize that, without departing from the scope of these concepts, changes can be made in form and detail. Therefore, the described embodiments are considered to be illustrative and non-restrictive in all aspects. It should also be understood that the application is not limited to the specific embodiments described herein, but many rearrangements, modifications and replacements are possible without departing from the scope of this disclosure.

Claims

1. A system comprising: (a) Video downsampler based on machine learning (ML) model, which is configured as follows: receiving an input video sequence comprising a plurality of input images, the input video sequence having a first display resolution; as well as mapping the input video sequence to a lower resolution video sequence comprising a plurality of lower resolution images, the lower resolution video sequence having a second display resolution lower than the first display resolution; (b) a neural network-based (NN-based) proxy video codec configured to transform the lower-resolution video sequence into a decoded proxy bitstream, wherein the NN-based proxy video codec is pre-trained to replicate the rate-distortion characteristics of a standard video codec in a differentiable manner; as well as (c) An upsampler configured to generate an output video sequence using the decoded proxy bitstream.

2. The system of claim 1 , wherein generating the output video sequence comprises: receiving the decoded proxy bitstream; And generating an output video sequence corresponding to the input video sequence, the output video sequence having a display resolution higher than a second display resolution.

3. The system of claim 1 , wherein the machine learning model-based video downsampler is trained using the input video sequence, the output video sequence, and an objective function based on an estimated rate of the lower resolution video sequence and a plurality of perceptual loss functions.

4. The system of claim 3, wherein the objective function comprises a combination of an estimated rate of the lower resolution video sequence and a weighted sum of the plurality of perceptual loss functions.

5. A system according to claim 4, wherein the video downsampler based on the machine learning model is further configured to receive a plurality of weighting factors included in the weighted sum of the plurality of perceptual loss functions, and wherein the video downsampler based on the machine learning model is also trained using the plurality of weighting factors.

6. The system according to claim 1, wherein: The neural network based proxy video codec and the upsampler are part of a simulation module.

7. The system according to claim 1, wherein: The upsampler includes an upsampler based on a machine learning model.

8. The system of claim 7, wherein the machine learning model-based upsampler and the machine learning model-based video downsampler are trained simultaneously.

9. A method for training a video downsampler based on a machine learning (ML) model, the method comprising: Providing an input video sequence comprising a plurality of input images to the machine learning model-based video downsampler, the input video sequence having a first display resolution; mapping the input video sequence to a lower resolution video sequence comprising a plurality of lower resolution images using the machine learning model-based video downsampler, the lower resolution video sequence having a second display resolution lower than the first display resolution; transforming the lower-resolution video sequence into a decoded bitstream using a neural network-based (NN-based) proxy video codec, wherein the NN-based proxy video codec is pre-trained to replicate rate-distortion characteristics of a standard video codec in a differentiable manner; generating, using an upsampler receiving the decoded bitstream, an output video sequence comprising a plurality of output pictures corresponding to the input video sequence and having a display resolution higher than the second display resolution; as well as The machine learning model-based video downsampler is trained using the input video sequence, the output video sequence, and an objective function based on an estimated rate of the lower resolution video sequence performed by the NN-based proxy video codec and a plurality of perceptual loss functions.

10. The method according to claim 9, wherein: The neural network based proxy video codec and the upsampler are part of a simulation module. The method of claim 9 , wherein the upsampler comprises an upsampler based on a machine learning model.

12. The method of claim 11, further comprising simultaneously training the machine learning model-based upsampler and the machine model-based video downsampler.

13. The method of claim 9, wherein the objective function comprises a combination of an estimated rate of the lower resolution video sequence and a weighted sum of the plurality of perceptual loss functions.

14. The method according to claim 13 further includes providing a plurality of weighting factors included in the weighted sum of the plurality of perceptual loss functions to the video downsampler based on the machine learning model, wherein training the video downsampler based on the machine learning model is also performed using the plurality of weighting factors.

15. A video processing system comprising: processing hardware and system memory that stores a video codec and a trained machine learning model-based video downsampler that has been trained using a neural network-based (NN-based) proxy video codec; The processing hardware configuration is: receiving an input video sequence comprising a plurality of input images, the input video sequence having a first display resolution; mapping the input video sequence to a lower-resolution video sequence comprising a plurality of lower-resolution images using the trained machine learning model-based video downsampler, the lower-resolution video sequence having a second display resolution lower than the first display resolution; transforming the lower resolution video sequence into a decoded bitstream using the video codec, wherein the NN-based proxy video codec is pre-trained to replicate rate-distortion characteristics of a standard video codec in a differentiable manner; as well as The decoded bitstream is output. The video processing system according to claim 15 , wherein: The neural network based proxy video codec and upsampler are part of the simulation module.

17. The video processing system of claim 15, wherein the trained machine learning model-based video downsampler is configured to support arbitrary scaling factors.

18. The video processing system of claim 15, further comprising a simulation module comprising the neural network-based proxy video codec or upsampler.

19. The video processing system of claim 18, wherein the processing hardware is further configured to: Before mapping the input video sequence to the lower resolution video sequence: extracting a content sample of the input video sequence; mapping the content samples to lower resolution samples using the trained machine learning model-based video downsampler; transforming the lower resolution samples into a decoded sample bitstream using one of the video codec or the neural network-based proxy video codec; predicting output samples corresponding to the content samples using the upsampler and the decoded sample bitstream; as well as One or more parameters of the trained machine learning model-based video downsampler are modified based on the predicted output samples, thereby adaptively rendering the trained machine learning model-based video downsampler content.

Images