Video compression based on machine learning models

Abstract

A system processing hardware executes a video compression encoder based on a machine learning (ML) model to receive uncompressed video content and corresponding motion compensated video content, compare the uncompressed and motion compensated video content to identify an image space residual, transform the image space residual into a latent space representation of the uncompressed video content, and transform the motion compensated video content into a latent space representation of the motion compensated video content using a trained image compression ML model. The video compression encoder based on the ML model further encodes the latent space representation of the image space residual to produce an encoded latent residual, encodes the latent space representation of the motion compensated video content using the trained image compression ML model to produce an encoded latent video content, and produces compressed video content corresponding to the uncompressed video content using the encoded latent residual and the encoded latent video content.

Description

[0001] Cross-references to related applications

[0002] This application claims the benefit and priority of U.S. Provisional Patent Application Serial No. 63 / 172,315, filed April 8, 2021, entitled “Neural Network-Based Video Codec”, and U.S. Provisional Patent Application Serial No. 63 / 255,280, filed October 13, 2021, entitled “Micro-dose Delivery for Low Bitrate Video Compression”, the entire contents of which are incorporated herein by reference. Background Technology

[0003] Video content represents a significant portion of total internet traffic, and this is expected to increase further with rising spatial resolution, frame rates, and video color depth, as more users adopt streaming services. While existing codecs have already achieved impressive performance, they are designed to the point where adding further minor improvements is unlikely to meet future demands. Therefore, exploring fundamentally different ways of performing video encoding could advantageously yield new video codecs with improved performance and flexibility.

[0004] For example, one advantage of using a trained machine learning (ML) model (such as a neural network (NN)) in the form of a generative adversarial network (GAN) to perform video compression is that it enables the ML model to infer visual details that are expensive to obtain in terms of data transmission. However, training ML models (such as GANs) is often challenging because training alternates between minimization and maximization steps to converge to a saddle point of the loss function. The task becomes even more challenging when the temporal domain is taken into account and the increased complexity it introduces. Attached Figure Description

[0005] Figure 1 A diagram is shown based on an exemplary system for performing video compression based on a machine learning (ML) model.

[0006] Figure 2A A diagram illustrating an exemplary ML model-based video codec architecture based on one implementation is shown.

[0007] Figure 2B A diagram illustrating an exemplary ML-based video codec architecture according to another implementation is shown;

[0008] Figure 2C A diagram illustrating an exemplary ML model-based video codec architecture according to yet another implementation is shown;

[0009] Figure 3a flowchart outlining an example method for performing ML model based video compression according to one implementation is shown; and

[0010] Figure 4 a flowchart outlining an example method for performing ML model based video compression according to another implementation is shown. DETAILED DESCRIPTION

[0011] The following description contains specific information pertaining to implementations in the present disclosure. Those skilled in the art will recognize that the present disclosure can be practiced with different implementations than those specifically discussed in this document. The drawings in this application and their accompanying detailed description are directed to exemplary implementations. Unless noted otherwise, like or corresponding elements in the drawings can be indicated by like or corresponding reference numerals. Furthermore, the drawings and illustrations in this application are generally not to scale, and are not intended to correspond to actual relative dimensions.

[0012] As noted above, video content represents a large portion of total internet traffic, and is expected to increase even more as spatial resolution frame rates and video color depths increase and more users adopt streaming services. While existing codecs have achieved impressive performance, they have been designed to the point where adding further small improvements is unlikely to meet future demands. Thus, exploring fundamentally different ways to perform video encoding can advantageously result in new video codecs with improved performance and flexibility.

[0013] For example, and as further described above, one advantage of using a trained machine learning (ML) model, such as a neural network (NN), in the form of a generative adversarial network (GAN), for example, to perform video compression is that it enables the ML model to infer visual details that would otherwise be expensive to obtain in terms of data transfer. However, training ML models, such as GANs, is generally challenging because training alternates between minimization steps and maximization steps to converge to a saddle point of a loss function. The task becomes more challenging when considering the time domain and increased complexity introduced by simply increasing the data.

[0014] The present application discloses a framework based on knowledge distillation and latent space residual to construct a video compression codec with similar hallucination capabilities as trained GANs using any adversarially trained image compression ML model as a base, which is particularly important when targeting low bitrate video compression. The images resulting from the ML model based video compression solution of the present application are visually pleasing without requiring high bitrates. When using the ML model based video codec, some image details of the synthesis can look realistic but slightly deviate from the ground truth. However, the ML model based video compression solution of the present application is able to provide image quality that would not be possible with conventional approaches using the same amount of transmitted data. Moreover, in some implementations, the ML model based video compression solution of the present application can be implemented as a substantially automated system and method.

[0015] It should be noted that, as used in the present application, the terms “automation,” “automated,” and “automating” refer to systems and processes that do not require human user involvement, such as a human editor or system administrator. While in some implementations, a human system administrator can check the performance of an automated system operating according to the automated processes described herein, human involvement is optional. Thus, the processes described in the present application can be performed under the control of the hardware processing components of the disclosed systems.

[0016] It should also be noted that, as defined in the present application, the expression “machine learning model” (hereinafter “ML model”) refers to a mathematical model that makes future predictions based on patterns learned from data samples obtained from a set of known matches and known mismatches (referred to as training data). Various learning algorithms can be used to map the correlations between input data and output data. These correlations form a mathematical model that can be used to make future predictions on new input data. For example, such predictive models can include one or more logistic regression models, Bayesian models, or neural networks (NNs). Moreover, machine learning models can be designed to improve their performance on a particular task step by step.

[0017] In the context of deep learning, a "deep neural network" (deep NN) can refer to an NN that utilizes multiple hidden layers between the input and output layers, allowing learning based on features not explicitly defined in the original data. As used in this application, features labeled as NNs refer to deep neural networks. In various implementations, NNs can be used to perform image processing or natural language processing. While the novelty and inventiveness of this application are described below with reference to an exemplary NN class called GAN, this description is provided merely for conceptual clarity. More generally, the ML model-based video compression solution of this application can be implemented using other types of ML models and may be particularly advantageous when used with ML models that are cumbersome, expensive, or time-consuming to train.

[0018] Figure 1 An exemplary system for performing ML model-based video compression, according to an implementation, is shown. Figure 1 As shown, system 100 includes computing platform 102 having processing hardware 104 and system memory 106 implemented as a computer-readable non-transitory storage medium. According to this exemplary implementation, system memory 106 stores uncompressed video content 116 and ML model-based codec software resources 130.

[0019] like Figure 1 As further shown, system 100 is implemented in an environment that includes a communication network 110 and a user system 120 configured for use by user 114 and including a display 122. Furthermore, Figure 1 A network communication link 112 is shown that connects user system 120 and system 100 via communication network 110. Figure 1 Compressed video content 117 output by system 100 is also shown, corresponding to uncompressed video content 116.

[0020] Although for clarity of concept, this application refers to the ML model-based codec software resource 130 as stored in system memory 106, more generally, system memory 106 can take the form of any computer-readable non-transitory storage medium. As used herein, the expression "computer-readable non-transitory storage medium" refers to any medium excluding carrier waves or other transient signals that provide instructions to the processing hardware 104 of computing platform 102. Therefore, computer-readable non-transitory storage medium can correspond to various types of media, such as volatile and non-volatile media. Volatile media can include dynamic memory, such as dynamic random access memory (dynamic RAM), while non-volatile memory can include optical, magnetic, or electrostatic storage devices. Common forms of computer-readable non-transitory storage media include, for example, optical discs, RAM, programmable read-only memory (PROM), erasable PROM (EPROM), and flash memory.

[0021] In addition, although Figure 1 The ML model-based codec software resource 130 is depicted as being stored monolithically in system memory 106, but this representation is provided only to aid in conceptual clarity. More generally, system 100 may include one or more computing platforms 102, such as computer servers, which may be located in the same location, or may form interactively linked but distributed systems, such as cloud-based systems. Consequently, processing hardware 104 and system memory 106 may correspond to distributed processor and memory resources within system 100. Therefore, in some implementations, one or more of the features of the ML model-based codec software resource 130 described below may be stored remotely to each other on the distributed memory resources of system 100.

[0022] Processing hardware 104 may include multiple hardware processing units, such as one or more central processing units, one or more graphics processing units and one or more tensor processing units, one or more field-programmable gate arrays (FPGAs), custom hardware for machine learning training or inference, and, for example, application programming interface (API) servers. By definition, as used herein, the terms “central processing unit (CPU),” “graphics processing unit (GPU),” and “tensor processing unit (TPU)” have their conventional meanings in the art. That is, the CPU includes an arithmetic logic unit (ALU) for performing arithmetic and logical operations of computing platform 102, and a control unit (CU) for retrieving programs (such as ML model-based codec software resources 130) from system memory 106, while the GPU may be implemented to reduce the processing overhead of the CPU by performing computationally intensive graphics or other processing tasks. The TPU is an application-specific integrated circuit (ASIC) specifically configured for artificial intelligence (AI) processes (such as machine learning).

[0023] In some implementations, the computing platform 102 can correspond to one or more web servers, for example, that are accessible through a communication network 110 in the form of a packet-switched network such as the Internet. Further, in some implementations, the communication network 110 can be a high-speed network suitable for high-performance computing (HPC), such as a 10 GigE network or an Infiniband network. In some implementations, the computing platform 102 can correspond to one or more computer servers that support a private wide-area network (WAN), a local-area network (LAN), or that are included in another type of limited distribution or private network. As yet another alternative, in some implementations, the system 100 can be implemented virtually, such as in a data center. For example, in some implementations, the system 100 can be implemented in software, or as a virtual machine.

[0024] According to Figure 1 The illustrated implementation, a user 114 can utilize a user system 120 to interact with the system 100 through the communication network 110. The user system 120 and the communication network 110 enable the user 114 to obtain compressed video content 117 corresponding to uncompressed video content 116 from the system 100. Although the user system 120 is shown in Figure 1 as a desktop computer, this representation is provided by way of example only. More generally, the user system 120 can be any suitable mobile or fixed computing device or system that implements sufficient data processing capabilities to provide a user interface, support a connection to the communication network 110, and implement the functionality attributed herein to the user system 120. For example, in some implementations, the user system 120 can take the form of a laptop computer, a tablet computer, a smart phone, or a game console, for example. In other implementations, however, the user system 120 can be a "dumb terminal" peripheral component of the system 100 that enables the user 114 to provide input via a keyboard or other input device and to provide video content via a display 122. In those implementations, the user system 120 and the display 122 can be controlled by the processing hardware 104 of the system 100.

[0025] With respect to the display 122 of the user system 120, the display 122 can be physically integrated with the user system 120, or can be communicatively coupled to the user system 120, but physically separate from the user system 120. For example, where the user system 120 is implemented as a smartphone, laptop computer, or tablet computer, the display 122 will typically be integrated with the user system 120. By contrast, where the user system 120 is implemented as a desktop computer, the display 122 can take the form of a monitor separate from the user system 120 in the form of a computer tower. Moreover, the display 122 can take the form of a liquid crystal display (LCD), a light-emitting diode (LED) display, an organic light-emitting diode (OLED) display, a quantum dot (QD) display, or a display using any other suitable display technology that performs a physical transformation of an electrical signal to light.

[0026] The ML model-based codec software resource 130 is further described below with reference to Figure 2A , 2B and 2C. Reference is made to Figure 2A , Figure 2A A diagram of an example ML model-based video codec architecture 224 is shown in accordance with one implementation. As shown in Figure 2A , the ML model-based video codec architecture 224 includes a trained image compression ML model 232 that acts as an encoder and an ML model-based decoder 238. Figure 2A Also shown in Figure 2A are an uncompressed input image 216, a latent space representation 234 of the input image 216, a quantized latent 236, and a decoded image 218 corresponding to the uncompressed input image 216. Note that Figure 1 The features of the ML model-based video codec architecture 224 shown in may be included in the ML model-based codec software resource 130 in

[0027] . Figure 2A According to the example implementation shown in Figure 2A , the mapping from image space to latent space is implemented with the ML model-based video codec architecture 224, where the bottleneck value constitutes the latent representation. The function g indicates the mapping from image space to latent space performed by the trained image compression ML model 232, and g -1 indicates the inverse mapping. First, an image x is mapped to its latent representation y = g(x). As shown in , after quantization, the resulting quantized latent is losslessly encoded into a bitstream that can be decoded into an image

[0028] Image compression can be formally stated as the minimization of the expected length of the bitstream and the expected distortion of the reconstructed image compared to the original image, formulated as the optimization of the following rate-distortion objective function:

[0029]

[0030] where: is the rate term, and is the distortion term.

[0031] It is noted that in the notation used in equation 1, the g parameter includes g -1 . Here d indicates a distortion measure and can include a combination of l2, structural similarity index measure (SSIM), learned perceptual image patch similarity (LPIPS), etc. The rate corresponds to the length of the bitstream needed to encode the quantized representation using an entropy model based on learning of the unknown distribution of natural images

[0032] According to one implementation of the novel and inventive concept of the present application, the above neural image compression formulation can be augmented with a ML model in the form of a conditional GAN. In this adversarial training, D is represented as a discriminator neural network that learns to distinguish between the ground truth x and the decoded image

[0033]

[0034] The training of the discriminator is interleaved with the training of the image compression ML model 232, in which case the rate-distortion objective augmented with an adversarial loss is optimized:

[0035]

[0036] where: and keep the rate and distortion terms respectively, while is the adversarial loss.

[0037] To exploit temporal redundancy in video coding, video compression relies on information transfer by motion compensation. More precisely, a subsequent frame x t may be predicted from its preceding frame x t+1 (identified by reference number 217 in Figure 2B and 2C ). As defined in the present application, the expression “motion compensation” refers to the whole process of computing and encoding motion vectors, and any post-processing that can occur. In Figure 2B ​​and 2C In particular, motion compensation is denoted by reference number 240 as MC, and results in the generation of a motion compensated frame 219. For simplicity, it is assumed that motion compensation has been completed, and that the result is an estimate of the image and the motion vector field .

[0038] However, it should be noted that the video compression techniques described below with reference to Figure 2B and 2C do not require a single preceding reference frame x t to be used for motion estimation. In various use cases, one or more frames can be used for motion compensation. Furthermore, that frame or those frames used for motion compensation need not be located in the causal region of the video. As an example, bi-prediction can be performed such that a particular frame can extract motion estimates from future frames as well as from previous frames, effectively decoupling the playback order and the encoding order.

[0039] Two strategies for compressing the residual information needed to correct the current estimate using a trained image compression ML model are described below. The first strategy is referred to as “knowledge distillation with latent space residuals,” while the second strategy is referred to as “knowledge distillation with image space residuals.”

[0040] Knowledge distillation with latent space residuals:

[0041] Figure 2B A diagram showing an example ML model based video codec architecture 226 that utilizes a latent space representation 234a (y t+1 ) of the original frame 217 (x t+1 ) and a latent space representation 234b (y ) of the motion compensated frame 219 As shown in Figure 2B , the ML model based video codec architecture 226 includes an ML model based video compression encoder 233 and an image compression ML model 232. The residual mapping performed by the ML model based video compression encoder 233 and its inverse mapping are denoted as h and h -1 , respectively. The residual mapping h extracts residual information r from the two latent representations y t+1 and This information r is referred to as residual because the goal is to encode as little information as possible to accomplish and recover It should be noted that the features of the ML model based video codec architecture 226 shown in Figure 2B may be included in Figure 1between the ML model based codec software resources 130 in the system 100. That is, in some implementations, the ML model based codec software resources 130 can include a ML model based video compression encoder 233 and an image compression ML model 232.

[0042] Since the residual mapping function h and its inverse h -1 are run at a single time instant, time can be omitted from the following notation. The following definitions are established: and It is noted that, since is obtained via motion compensation, it is available at both encoding and decoding time. This solution is designed to exploit the image compression ML model g trained with adversarial loss. The parameters of the image compression ML model g remain unchanged. To achieve this residual compression, the parameters of the residual mapping h (including its inverse h -1 ) and the parameters of the probability model need to be trained. This can be done by optimizing the following rate-distortion loss:

[0043]

[0044] where: is the rate term, and is the distortion term.

[0045] It is noted that the target frame is no longer the ground truth x, but is now the output of the image compression ML model g. This enables to perform knowledge distillation and preserves the details hallucination capability of the adversarially trained image compression model. The residual mapping itself can be implemented as a combination of several techniques, as described in more detail in the accompanying paper entitled “Knowledge Distillation for GAN Based Video Codec”, which is hereby fully incorporated by reference into the present application.

[0046] Knowledge distillation with image space residuals:

[0047] A different approach to exploit the available trained image compression ML model g is to process the image space residuals, as represented in the exemplary ML model based video codec architecture 228 in Figure 2C . Similar to the latent residual case, it is assumed that motion compensation has already been done and only the difference between the current frame x t+1 and the motion compensated frame needs to be encoded, and is identified by reference number 219 in Figure 2C . As Figure 2CAs shown, the ML model-based video codec architecture 228 includes a ML model-based video compression encoder 235 and an image compression ML model 232. It should be noted that Figure 2C The features of the ML model-based video codec architecture 228 shown can be included in the ML model-based codec software resource 130 in Figure 1 . That is, in some implementations, the ML model-based codec software resource 130 can include the ML model-based video compression encoder 235 and the image compression ML model 232.

[0048] The method depicted by the ML model-based video codec architecture 228 differs in that, in the implementation shown in Figure 2C , the image space residual 237 corresponding to the uncompressed frame 217 and identified as frame z t+1 is encoded as:

[0049]

[0050] The neural encoder and neural decoder functions are denoted as h and h -1 , respectively. They can be implemented as neural network layers as described above with reference to Figure 2A . The function h maps the image residual 237 to its latent space representation 239 denoted as r, whereas the decoder function h -1 decodes the image directly. In line with the disclosed method utilizing features from the trained image compression ML model g, a merge function is designed. The goal of this function is to integrate features from the image compression ML model 232. In its simplest form, this merge function can be implemented as a masking mechanism that takes binary values 0, 1 to indicate which features to use, and this selection can be done exhaustively after extensive experimentation with possible combinations. In a more complex design, the mask can be learned during training and optionally conditioned on the image itself.

[0051] The training loss can be expressed as:

[0052]

[0053] where, is the rate term, is the distortion term, and is the adversarial loss. It should be noted that the training loss expressed by equation 6 includes the adversarial loss of the image compression ML model g. This corresponds to compressing the image into a single frame utilizing the trained image compression ML model g.

[0054] Temporal instability, such as flicker, hallucinated details, etc., can arise in both of the above knowledge distillation processes. To maintain temporally stable results, a temporal smoothing component can be added. More formally, given a previously decoded frame and motion vectors The goal is to process the frame to remove any temporal artifacts:

[0055]

[0056] The superscript indicates a temporally processed frame. W is an image warping function that uses the motion field to warp the previous frame to match the current frame Note that in some implementations, may represent multiple frames, not a single previously decoded frame. In implementations that utilize multiple previously decoded frames, each motion vector can be treated as a pair of data points that includes a displacement and a reference frame index.

[0057] The correction function F can be implemented as a NN and can be trained using the following loss:

[0058]

[0059] where is a temporal term and is an optional adversarial loss term, d is a distortion error that penalizes the deviation between the appearance of two consecutive frames to enforce temporal stability (e.g., it can be an l1 loss). M is a merge function that can be implemented as a binary mask that indicates where the motion vector is valid to limit the penalty to the regions where the motion is correctly estimated. Finally, as an option, an adversarial loss can be added to avoid over-smoothing the final output.

[0060] The knowledge distillation approach that utilizes image space residuals will be further described with reference to Figure 3 corresponds to the example ML model based video codec architecture 228 in Figure 3 Figure 3 A flowchart 350 is shown that presents an example method for performing ML model based video compression according to one implementation. With respect to the method outlined in Figure 3 , it should be noted that certain details and features have been left out of the flowchart 350 in order to not obscure the discussion of the inventive features in the present application.

[0061] Reference is now made to Figure 1 in conjunction with Figure 2C and 2C ​, flowchart 350 includes receiving uncompressed video content (e.g., uncompressed frames 217) and motion compensated video content (e.g., motion compensated frames 219) corresponding to the uncompressed video content (act 351). As shown, the uncompressed frames 217 and the motion compensated frames 219 can be received in act 351 by the ML model-based video compression encoder 235. Further, and as further described above with reference to Figure 1 the ML model-based codec software resources 130 and stored in the system memory 106. Thus, the uncompressed frames 217 and the motion compensated frames 219 can be received in act 351 by the ML model-based video compression encoder 235 executed by the processing hardware 104 of the system 100. Figure 1

[0062] Flowchart 350 also includes comparing the uncompressed video content to the motion compensated video content to identify image space residuals 237 corresponding to the uncompressed video content (act 352). Continuing with reference to Figure 2C and Figure 2C , in act 352 of the ML model-based video compression encoder 235 executed by the processing hardware 104 of the system 100, the uncompressed frames 217 and the motion compensated frames 219 can be compared and image space residuals identified.

[0063] Flowchart 350 also includes transforming the image space residuals 237 into latent space representations 239 of the image space residuals 237 (act 353). The image space residuals 237 can be transformed into latent space representations 239 of the image space residuals 237 in act 353 by the ML model-based video compression encoder 235 executed by the processing hardware 104 of the system 100 and using a neural encoder function h.

[0064] Flowchart 350 also includes receiving the motion compensated video content (e.g., motion compensated frames 219) using the trained image compression ML model 232 (act 354). As described above, the trained image compression ML model 232 can include a trained NN, e.g., such as a trained GAN. Further, and as described above, in some implementations, the trained image compression ML model 232 can include a NN trained using an objective function that includes an adversarial loss. Act 354 can be performed by the ML model-based video compression encoder 235 executed by the processing hardware 104 of the system 100.

[0065] Flowchart 350 also includes transforming the motion compensated video content represented by the motion compensated frames 219 into latent space representations 234 of the motion compensated video content using the trained image compression ML model 232 (act 355). As Figure 1 ​As shown, the ML model-based video compression encoder 235 can transform the motion compensated frame 219 into its latent space representation 234 using the trained image compression ML model 232, i.e., the processing hardware 104 of the system 100 can execute the ML model-based video compression encoder 235 using the image compression ML model 232 to perform action 355.

[0066] It should be noted that while the flowchart 350 depicts actions 354 and 355 after actions 351, 352, and 353, this representation is provided by way of example only. In some other implementations, actions 354 and 355 can be performed in sequence, but in parallel with (i.e., substantially simultaneously as) actions 351, 352, and 353. In other implementations, action 354 or actions 354 and 355 can precede one or more of actions 351, 352, and 353.

[0067] The flowchart 350 also includes encoding the latent space representation 239 of the image space residual 237 to produce an encoded latent residual (action 356). The latent space representation 239 of the image space residual 237 can be encoded in action 356 by the ML model-based video compression encoder 235 executed by the processing hardware 104 of the system 100 to produce the encoded latent residual.

[0068] The flowchart 350 also includes encoding the latent space representation 234 of the motion compensated frame 219 using the trained image compression ML model 232 to produce an encoded latent video content (action 357). The latent space representation 234 of the motion compensated frame 219 can be encoded in action 357 by the ML model-based video compression encoder 235 executed by the processing hardware 104 of the system 100 and using the trained image compression ML model 232 to produce the encoded latent video content.

[0069] It should be noted that while the flowchart 350 depicts action 357 as following action 356, this representation is provided by way of example only. The only constraint imposed on the timing of action 357 is that it follows action 355, while the only constraint imposed on the timing of action 356 is that it follows action 353. Thus, in various implementations, action 357 can follow action 356, can precede action 356, or can be performed in parallel with (i.e., substantially simultaneously as) action 356. That is, in some implementations, the encoded latent residual produced in action 356 and the encoded latent video content produced in action 357 can be produced in parallel.

[0070] In conjunction with Figure 3 and Figure 2CFlowchart 350 also includes generating compressed video content 117 corresponding to uncompressed video content 116 (Action 358) using the encoded potential residual generated in Action 356 and the encoded potential video content generated in Action 357. In some implementations, the compressed video content 117 corresponding to uncompressed video content 116 can be generated based on the difference between the encoded potential residual generated in Action 356 and the encoded potential video content generated in Action 357. Further reference Figure 4 The compressed video content 117 can be generated in action 358 by a video compression encoder 235 based on an ML model executed by the processing hardware 104 of system 100.

[0071] Reference Figure 2B Further, the knowledge distillation method utilizing latent spatial residuals is described, which corresponds to Figure 4 The exemplary ML model-based video codec architecture 226 is shown in the figure. Figure 4 A flowchart 460, based on another implementation, illustrates an exemplary method for performing ML model-based video compression. (About...) Figure 1 The method outlined in the flowchart 460 should be noted that certain details and features have been omitted from the flowchart so as not to obscure the discussion of the inventive features in this application.

[0072] Now combine Figure 2B and Figure 4 Reference Figure 2B Flowchart 460 includes receiving uncompressed video content (e.g., uncompressed frame 217) and motion-compensated video content (e.g., motion-compensated frame 219) corresponding to the uncompressed video content (action 461) using a trained image compression ML model 232. Figure 1 As shown, in action 461, an ML-based video compression encoder 233 may receive uncompressed frame 217 and motion-compensated frame 219 using a trained image compression ML model 232. As described above, the trained image compression ML model 232 may include a trained neural network, such as a trained GAN. Furthermore, and as described above, in some implementations, the trained image compression ML model 232 may include a neural network trained using an objective function that includes an adversarial loss. Additionally, and further referenced above... Figure 2B The ML model-based video compression encoder 233 may be included in the ML model-based codec software resource 130 and stored in the system memory 106. Therefore, the ML model-based video compression encoder 233, which can be executed by the processing hardware 104 of the system 100 in action 461, receives uncompressed frames 217 and motion-compensated frames 219.

[0073] Flowchart 460 also includes transforming, using the trained image compression ML model 232, the uncompressed video content represented by the uncompressed frames 217 into a first latent space representation 234a of the uncompressed video content (act 462). As shown, Figure 2B

[0074] Flowchart 460 also includes transforming, using the trained image compression ML model 232, the uncompressed video content represented by the motion compensated frames 219 into a second latent space representation 234b of the uncompressed video content (act 463). As shown, Figure 3

[0075] It should be noted that while flowchart 460 depicts act 463 as following act 462, this representation is provided by way of example only. In various implementations, act 463 can be performed after act 462, can be performed before act 462, or can be performed in parallel with (i.e., substantially simultaneously with) act 462. That is, in some implementations, the transforming of the uncompressed video content into the first latent space representation 234a and the transforming of the motion compensated video content into the second latent space representation 234b can be performed in parallel.

[0076] Flowchart 460 also includes generating, based on the first latent space representation 234a and the second latent space representation 234b, a bitstream for transmission of compressed video content 117 corresponding to the uncompressed video content 116 (act 464). In some implementations, act 464 can include determining a latent space residual using the first latent space representation 234a and the second latent space representation 234b. For example, such a latent space residual can be based on a difference between the first latent space representation 234a and the second latent space representation 234b. In implementations in which a latent space residual is determined as part of act 464, the latent space residual can be used to generate the bitstream for transmission of compressed video content 117 corresponding to the uncompressed video content 116. In act 464, the generation of the bitstream for transmission of compressed video content 117 can be performed by the ML model-based video compression encoder 233 executed by the processing hardware 104 of system 100. ​​

[0077] With respect to ​ And 4 With respect to the actions denoted in

[0078] Accordingly, the present application discloses a framework comprising an ML model based video compression solution based on knowledge distillation and latent space residual to enable the use of a video compression codec with hallucination capabilities similar to trained GANs, which is particularly important when targeting low bitrate video compression. The ML model based video compression solution of the present application advances the state of the art by providing visually pleasing images without requiring high bitrate. When using an ML model based video codec, some image details of the synthesis can look very realistic but slightly deviate from the ground truth. However, the ML model based video compression solution of the present application advantageously enables to provide image quality which would not be possible with the same amount of transmitted data in conventional approaches.

[0079] From the above description, it is clear that the concepts described herein can be implemented in various ways and that the application should not be limited to the specific implementations described above. Furthermore, while the concepts have been described in connection with certain implementations, persons skilled in the art will recognize immediately upon study of the foregoing that changes, modifications and alternatives can be made without departing from the scope of the concepts. Accordingly, the described implementations are to be considered illustrative and not restrictive, and the application is not to be limited to the details given herein, but can be implemented with various other hardware or software modules.

Claims

1. A system comprising: A computing platform, the computing platform including processing hardware and system memory, the system memory storing a video compression encoder based on a machine learning (ML) model, the ML-based video compression encoder including a trained image compression ML model; The processing hardware is configured to execute a video compression encoder based on an ML model, to: Receive uncompressed video content and motion-compensated video content corresponding to the uncompressed video content; The uncompressed video content is compared with the motion-compensated video content to identify the image spatial residual corresponding to the uncompressed video content; Transform the image space residual into a latent space representation of the image space residual; The trained image compression ML model is used to receive the motion-compensated video content; The trained image compression ML model is used to transform the motion-compensated video content into a latent spatial representation of the motion-compensated video content; The latent spatial representation of the image spatial residual is encoded to generate an encoded latent residual; The latent spatial representation of the motion-compensated video content is encoded using the trained image compression ML model to produce encoded latent video content. and Compressed video content corresponding to the uncompressed video content is generated using the encoded potential residual and the encoded potential video content.

2. The system according to claim 1, wherein, The encoded potential residual and the encoded potential video content are generated in parallel.

3. The system according to claim 1, wherein, The processing hardware is configured to execute the ML model-based video compression encoder to generate compressed video content corresponding to the uncompressed video content based on the difference between the encoded potential residual and the encoded potential video content.

4. The system according to claim 1, wherein, The trained image compression ML model includes a trained artificial neural network (NN).

5. The system according to claim 4, wherein, The trained neural network is trained using an objective function that includes adversarial loss.

6. The system according to claim 4, wherein, The trained neural network includes a generative adversarial network (GAN).

7. A method used by a system including a computing platform having processing hardware and system memory storing a video compression encoder based on a machine learning (ML) model, the ML-based video compression encoder comprising a trained image compression ML model, the method comprising: Uncompressed video content and motion-compensated video content corresponding to the uncompressed video content are received by a video compression encoder based on an ML model executed by the processing hardware. The uncompressed video content is compared with the motion-compensated video content by a video compression encoder based on an ML model executed by the processing hardware, thereby identifying the image spatial residual corresponding to the uncompressed video content; The image spatial residual is transformed into a latent spatial representation of the image spatial residual by a video compression encoder based on an ML model executed by the processing hardware; The motion-compensated video content is received through a trained image compression ML model executed by the processing hardware; The motion-compensated video content is transformed into a latent spatial representation of the motion-compensated video content by a trained image compression ML model executed by the processing hardware. The latent spatial representation of the image spatial residual is encoded by a video compression encoder based on an ML model, which is executed by the processing hardware, to produce an encoded latent residual; Encoded video content is produced by encoding the latent spatial representation of the motion-compensated video content using a trained image compression ML model executed by the processing hardware. and Compressed video content corresponding to the uncompressed video content is generated by a video compression encoder based on an ML model executed by the processing hardware, using encoded latent residuals and encoded latent video content.

8. The method according to claim 7, wherein, The encoded potential residual and the encoded potential video content are generated in parallel.

9. The method according to claim 7, wherein, Based on the difference between the encoded potential residual and the encoded potential video content, compressed video content corresponding to the uncompressed video content is generated.

10. The method of claim 7, wherein the trained image compression ML model comprises a trained artificial neural network (NN).

11. The method according to claim 10, wherein, The trained neural network is trained using an objective function that includes adversarial loss.

12. The method according to claim 7, wherein, The trained neural network includes a generative adversarial network (GAN).

13. A system comprising: A computing platform, the computing platform including processing hardware and system memory, the system memory storing a video compression encoder based on a machine learning (ML) model, the ML-based video compression encoder including a trained image compression ML model; The processing hardware is configured to execute a video compression encoder based on an ML model, to: The trained image compression ML model is used to receive uncompressed video content and motion-compensated video content corresponding to the uncompressed video content; The uncompressed video content is transformed into a first latent space representation of the uncompressed video content using the trained image compression ML model; The trained image compression ML model is used to transform the motion-compensated video content into a second latent space representation of the uncompressed video content; and A bitstream is generated based on the first latent spatial representation and the second latent spatial representation for transmitting compressed video content corresponding to the uncompressed video content.

14. The system according to claim 13, wherein, The processing hardware is also configured to execute the ML model-based video compression encoder to: The first latent space representation and the second latent space representation are used to determine the latent space residual.

15. The system according to claim 14, wherein, The processing hardware is also configured to execute the ML model-based video compression encoder to: The latent spatial residual is used to generate a bitstream for transmitting the compressed video content corresponding to the uncompressed video content.

16. The system according to claim 14, wherein, The latent space residual is based on the difference between the first latent space representation and the second latent space representation.

17. The system according to claim 13, wherein, The transformation of the uncompressed video content to the first latent space representation and the transformation of the motion-compensated video content to the second latent space representation are performed in parallel.

18. The system according to claim 13, wherein, The trained image compression ML model includes a trained artificial neural network (NN).

19. The system according to claim 13, wherein, The trained neural network is trained using an objective function that includes adversarial loss.

20. The system according to claim 13, wherein, The trained neural network includes a generative adversarial network (GAN).

Images