Microprocessing for Low Bit-Rate Video Compression

Abstract

A system includes: a machine learning (ML) model-based video encoder configured to: receive an uncompressed video sequence including a plurality of video frames; determine a first subset of video frames and a second subset of video frames from the plurality of video frames; encode the first subset of video frames to produce a first subset of compressed video frames; and identify first decompression data for the first subset of compressed video frames. The ML model-based video encoder is further configured to: encode the second subset of video frames to produce a second subset of compressed video frames; and identify second decompression data for the second subset of compressed video frames. The first decompression data is dedicated to decoding the first subset of compressed video frames and not dedicated to the second subset of compressed video frames, and the second decompression data is dedicated to decoding the second subset of compressed video frames and not dedicated to the first subset of compressed video frames.

Description

[0001] Cross - reference to related applications

[0002] This application claims the benefit and priority of U.S. Provisional Patent Application No. 63 / 172,315, titled "Neural Network Based Video Codecs", filed on April 8, 2021, and U.S. Provisional Patent Application No. 63 / 255,280, titled "Microdosing For Low Bitrate Video Compression", filed on October 13, 2021, the entire contents of which are incorporated herein by reference. Background of the Invention

[0003] Video content represents a large portion of the total Internet traffic and is expected to increase even more as the spatial resolution, frame rate, color depth of videos increase, and more users adopt streaming services. Although existing codecs have achieved impressive performance, they have been designed to such an extent that further adding small improvements is unlikely to meet future requirements. Therefore, exploring fundamentally different ways to perform video coding may advantageously result in new classes of video codecs with improved performance and flexibility.

[0004] For example, one advantage of using a trained machine learning (ML) model (such as a neural network (NN)), in the form of, for example, a generative adversarial network (GAN), to perform video compression is that it enables the ML model to infer visual details that would otherwise be costly in terms of data transmission to obtain. However, model size remains a significant issue in state - of - the - art proposals, and existing solutions require significant computational power on the decoding side. That is, a significant drawback of existing GAN - based compression frameworks is that they typically require large decoder models that are sometimes trained on private datasets. Therefore, it is often impossible to retrain these models to their original performance, and even when training data is available, retraining the models would be complex and time - consuming. In addition, the memory and inference - time requirements of existing large decoder models make them less practical, especially in the context of video coding. Brief Description of the Drawings

[0005] Figure 1 A diagram showing an exemplary system that provides a video codec based on a machine learning (ML) model, according to one embodiment;

[0006] Figure 2 A diagram showing a comparison of existing methods of neural compression with an exemplary micro - compression method, according to one implementation;

[0007] Figure 3Shows a conventional high - performance codec including a large decoder according to one embodiment;

[0008] Figure 4A Depicts the training of a small student decoder using a teacher decoder according to an exemplary embodiment of the present inventive concept;

[0009] Figure 4B Depicts an exemplary ML - model - based decoding network suitable for use in the small student decoder shown in Figure 4A ;

[0010] Figure 5 Shows a video compression pipeline for implementing a knowledge distillation (KD) technique according to one embodiment, the knowledge distillation (KD) technique including micro - processing for low - bitrate video compression;

[0011] Figure 6 Shows a flowchart outlining an exemplary method for performing micro - processing for low - bitrate video compression according to one embodiment; and

[0012] Figure 7 Shows a flowchart according to one embodiment depicting additional actions for extending the method outlined in Figure 6 ; Detailed Description

[0013] The following description contains specific information related to embodiments in the present disclosure. Those skilled in the art should recognize that the present disclosure may be implemented in ways different from those specifically discussed herein. The accompanying drawings and their detailed descriptions in this application are only directed to exemplary embodiments. Unless otherwise indicated, the same or corresponding elements in the drawings may be denoted by the same or corresponding reference numerals. Additionally, the drawings and illustrations in this application are generally not to scale and are not intended to correspond to actual relative sizes.

[0014] This application relates to systems and methods for providing a video codec based on a machine learning (ML) model. Additionally, this application discloses a knowledge distillation (KD) method that is capable of maintaining good perceptual image quality while reducing the decoder size. According to this novel and creative principle, the goal of KD is to transfer the learned knowledge of a teacher network to a smaller student network, which still has competitive performance relative to the teacher network. By requiring less storage and computational power than the initial teacher network, the student network can operate, for example, on lower-power devices such as mobile phones or dedicated devices. As disclosed herein, the ability to compress the generator network or decoder in an autoencoder setting is advantageous both in terms of memory requirements and computational efficiency. This is particularly important for image and video compression, where most of the computations should preferably be performed on the sender (encoder) side, while decoding should be simple. Especially in the context of video streaming, something useful will typically be encoded once for distribution, but may be decoded millions of times.

[0015] One advantage of using a trained machine learning model, such as an artificial neural network (NN), etc., to perform video compression is that it enables the machine learning model to infer visual details that would otherwise be costly in terms of the data transmission to be acquired. Thus, the resulting images are typically visually pleasing without requiring a high bitrate. Some of the image details synthesized when using a video codec based on a machine learning model may appear realistic, yet slightly deviate from the true values. However, the machine learning model-based video compression scheme of the present invention can provide image quality that is not possible to obtain with the same amount of transmitted data in conventional methods. Additionally, in some embodiments, the machine learning model-based scheme of the present invention can be implemented as a largely automated system and method.

[0016] Note that, as used in this application, the terms "automation", "automated", and "automate" refer to systems and processes that do not require the involvement of a human user such as a human editor or system administrator. Although in some embodiments, a human system administrator may check the performance of an automated system operating according to the automated processes described herein, such human involvement is optional. Thus, the processes described in this application may be performed under the control of the hardware processing components of the disclosed system.

[0017] It is also noted that, as defined in the present application, the expression "machine learning model" (hereinafter referred to as "ML model") refers to a mathematical model used to make future predictions based on patterns learned from data samples obtained from a set of reliable known matches and known mismatches (referred to as training data). Various learning algorithms can be used to map the correlation between input data and output data. These correlations form a mathematical model that can be used to make future predictions for new input data. For example, such a prediction model may include one or more logistic regression models, Bayesian models, or NNs. In addition, machine learning models may be designed to gradually improve their performance for a specific task.

[0018] In the context of deep learning, a "deep neural network" (deep NN) may refer to an NN that utilizes multiple hidden layers between an input layer and an output layer, which may allow for learning based on features not explicitly defined in the original data. As used in the present application, the feature labeled as NN refers to a deep neural network. In various embodiments, an NN may be used to perform image processing or natural language processing. Although the present novel and creative principles are described below by reference to an exemplary class of NNs known as generative adversarial networks (GANs), this description is provided only for the sake of conceptual clarity.

[0019] Figure 1 An exemplary system for performing video compression based on a machine learning (ML) model according to one embodiment is shown. As Figure 1 shown, system 100 includes a computing platform 102 having processing hardware 104 and a system memory 106 implemented as a computer-readable non-transitory storage medium. According to this exemplary embodiment, the system memory 106 stores an uncompressed video sequence 116 and a video encoder 108 based on an ML model.

[0020] As Figure 1 further shown, system 100 is implemented in a usage environment including a communication network 110 and a user system 120 configured to be used by a user 114. The user system 120 includes a display 122, user system processing hardware 124, and a user system memory 126 implemented as a computer-readable non-transitory storage medium storing a video decoder 128 based on an ML model. In addition, Figure 1 a network communication link 112 that interactively connects the user system 120 and the system 100 through the communication network 110 is shown, as well as a compressed video bitstream 118 output by the video encoder 108 based on the ML model and corresponding to the uncompressed video sequence 116.

[0021] Although, for clarity of concepts, the ML model-based video encoder 108 is referred to as being stored in the system memory 106 in the present application, more generally, the system memory 106 may take the form of any computer-readable non-transitory storage medium. As used in the present application, the expression "computer-readable non-transitory storage medium" refers to any medium other than a carrier wave or other transitory signal that provides instructions to the processing hardware 104 of the computing platform 102. Thus, the computer-readable non-transitory storage medium may correspond to various types of media, such as volatile media and non-volatile media. Volatile media may include dynamic memories, such as dynamic random access memory (dynamic RAM), while non-volatile memories may include optical, magnetic, or electrostatic storage devices. Common forms of computer-readable non-transitory storage media include, for example, optical discs, RAM, programmable read-only memory (PROM), erasable PROM (EPROM), and FLASH memory.

[0022] In addition, although Figure 1 the ML model-based video encoder 108 is depicted as being stored entirely in the system memory 106, this representation is also provided only to aid in conceptual clarity. More generally, the system 100 may include one or more computing platforms 102, for example, such as computer servers, which may be co-located, or may form an interactively linked but distributed system, such as a cloud-based system. As a result, the processing hardware 104 and the system memory 106 may correspond to distributed processor and memory resources within the system 100. Thus, in some embodiments, one or more features of the ML model-based video encoder 108 may be stored remotely from each other on the distributed memory resources of the system 100.

[0023] Processing hardware 104 may include multiple hardware processing units. For example, such as one or more central processing units, one or more graphics processing units, and one or more tensor processing units, one or more field-programmable gate arrays (FPGAs), custom hardware for machine learning training or inference, and application programming interface (API) servers. By definition, as used in this application, the terms "central processing unit" (CPU), "graphics processing unit" (GPU), and "tensor processing unit" (TPU) have their customary meanings in the art. That is, the CPU includes an arithmetic logic unit (ALU) for performing arithmetic and logical operations of the computing platform 102, and a control unit (CU) for retrieving programs such as the ML model-based video encoder 108 from the system memory 106, while the GPU may be implemented to reduce the processing overhead of the CPU by performing compute-intensive graphics or other processing tasks. The TPU is an application-specific integrated circuit (ASIC) specifically configured for artificial intelligence (AI) processes such as machine learning.

[0024] In some embodiments, the computing platform 102 may correspond to one or more web servers, which are accessible via a communication network 110 in the form of a packet-switched network, such as the Internet for example. Additionally, in some embodiments, the communication network 110 may be a high-speed network suitable for high-performance computing (HPC), such as a 10GigE network or an Infiniband network. In some embodiments, the computing platform 102 may correspond to one or more computer servers that support a private wide area network (WAN), a local area network (LAN), or are included in another type of limited distribution or private network. As yet another alternative, in some embodiments, the system 100 may be implemented virtually, such as in a data center. For example, in some embodiments, the system 100 may be implemented in software or as a virtual machine.

[0025] According to Figure 1In the illustrated embodiment, user 114 may utilize user system 120 to interact with system 100 via communication network 110. User system 120 and communication network 110 enable user 114 to obtain from system 100 a compressed video bitstream 118 corresponding to uncompressed video sequence 116.

[0026] Although user system 120 is shown in Figure 1 as a desktop computer, this representation is provided only as an example. More generally, user system 120 may be any suitable mobile or fixed computing device or system that implements data processing capabilities sufficient to provide a user interface, support a connection to communication network 110, and implement the functions ascribed to user system 120 herein. For example, in some embodiments, user system 120 may take the form of, for example, a laptop computer, a tablet computer, a smart phone, or a gaming console. However, in other embodiments, user system 120 may be a "dumb terminal" peripheral component of system 100 that enables user 114 to provide input via a keyboard or other input device and to receive video content via display 122. In those embodiments, user system 120 and display 122 may be controlled by the processing hardware 104 of system 100.

[0027] Regarding display 122 of user system 120, display 122 may be physically integrated with user system 120 or may be communicatively coupled to user system 120 but physically separated therefrom. For example, in the case where user system 120 is implemented as a smart phone, a laptop computer, or a tablet computer, display 122 will typically be integrated with user system 120. In contrast, in the case where user system 120 is implemented as a desktop computer, display 122 may take the form of a monitor separate from user system 120 in the form of a computer tower. Additionally, display 122 may take the form of a liquid crystal display (LCD), a light emitting diode (LED) display, an organic light emitting diode (OLED) display, a quantum dot (QD) display, or any other suitable display technology that performs a physical transformation of a signal to light.

[0028] By background art, the mapping from the image space to the latent space may be implemented using an ML model-based video encoder 108, where the bottleneck value constitutes the latent representation. The function g represents the mapping from the image space to the latent space performed by the ML model-based video encoder 108, and g -1 represents the inverse mapping. The uncompressed original image x is first mapped to its latent representation y = g(x). After quantization, the resulting quantized latents Losslessly encoded into a compressed video bitstream 118 that can be decoded into an uncompressed image corresponding to the original image x

[0029] Image compression can be formally expressed as minimizing the expected length of the bitstream and the expected distortion of the reconstructed image compared to the original, equating to optimizing the following rate-distortion objective function:

[0030]

[0031] Where: is the rate term, and is the distortion term.

[0032] It should be noted that in the notation used in Equation 1, the parameter g includes g -1 . Here d indicates the distortion measure and can include a combination of the structural similarity index measure (SSIM), the learned perceptual image patch similarity (LPIPS), etc. The rate corresponds to the length of the bitstream required to encode the quantized representation x based on an entropy model learned on the unknown distribution of natural images p . By reducing the weight λ, better compression can be achieved at the cost of greater distortion on the reconstructed image. By reducing the weight λ, better compression can be achieved at the cost of greater distortion on the reconstructed image.

[0033] According to one embodiment of this novel and creative concept, the above ML model-based image compression equation can be enhanced with an ML model in the form of a conditional GAN. In this case of adversarial training, D is represented as a discriminator neural network that learns to distinguish between the ground truth x and the decoded image conditioned on the latent representation :

[0034]

[0035] The training of the discriminator is alternated with the training of the image compression ML model 232, in which case the rate-distortion objective augmented with the adversarial loss is optimized:

[0036]

[0037] Where: and are still the rate term and the distortion term, while is the adversarial loss.

[0038] To exploit temporal redundancy in video coding, video compression relies on information transfer through motion compensation. More precisely, by considering motion information, a subsequent frame x can be predicted from its previous frame x t t+1 . As defined in the present application, the expression "motion compensation" refers to the complete process of calculating and encoding motion vectors, and any post-processing that may occur. For simplicity, it is assumed that motion compensation has been completed, and the result is an estimate of the image and the motion vector field .

[0039] Microprocessing for low-bitrate compression:

[0040] Figure 2 FIG. 230 shows a block diagram comparing the existing method 232 of neural compression with the microprocessing compression method 234 introduced in the present application. This microprocessing compression method 234 is based on: 1) training a simplified student decoder with data generated from a large decoder, 2) overfitting the simplified student decoder model to a specific image or set of images, and 3) sending the dedicated decoder weights together with the decompressed data 236 and the image latent. To demonstrate the feasibility of this microprocessing compression method, its incorporation into the state-of-the-art model for neural image and video compression targeting low-bitrate settings is described.

[0041] First, replace the high-fidelity compression (HiFiC) or other high-performance decoders with a much smaller student decoder. Note that HiFiC currently provides the state of the art in low-bitrate neural image compression (i.e., approximately 0.15 bits per pixel) and produces highly competitive results at the cost of a relatively large (i.e., approximately 156 million parameters) decoder network. However, although it is shown that the HiFiC architecture will be used in the existing method 232, this representation is merely exemplary. In other embodiments, the HiFiC encoder-decoder network of the existing method 232 may generally be replaced by any GAN-trained network with a similar architecture, based on residual blocks.

[0042] Compared with the existing method 232, the microprocessing compression method 234 disclosed in the present application advantageously allows a much smaller decoder (e.g., approximately 8 million parameters) and a fifty (50%) faster decoding time, while producing output images that are visually similar to those provided by HiFiC. Second, the application of this microprocessing KD strategy in a neural video compression framework based on latent residuals is described. In this case, the simplified student decoder is overfitted to the sequence in order to be able to provide a sequence-specific decoder.

[0043] As Figure 2 ​As shown, according to the existing method 232 of neural compression, the encoder-decoder pair is trained on a large dataset to obtain good overall performance for various different contents. Once the autoencoder is fully trained, the decoder is deployed and sent to the receiver. Then, the large decoder can decode any type of content.

[0044] In contrast, according to the present microprocessing method 234, the ML model-based video encoder 208 is configured to divide the uncompressed video sequence 216 data into subsets and learn content-specific decoders with the corresponding information of each subset This specialization enables the training of the ML model-based video decoder 238, which advantageously requires fewer parameters, less memory footprint, and uses less computation. Note that the ML model-based video encoder 208, the uncompressed video sequence 216, and the ML model-based video decoder 238 generally correspond to Figure 1 the ML model-based video encoder 108, the uncompressed video sequence 116, and the ML model-based video decoder 138 in. Thus, the ML model-based video encoder 108, the uncompressed video sequence 116, and the ML model-based video decoder 138 may share any features attributed to the corresponding ML model-based video encoder 208, uncompressed video sequence 216, and ML model-based video decoder 238 of this disclosure, and vice versa.

[0045] Once the ML model-based video decoder 238 is fully trained and the reconstruction quality requirements of the ML model-based video encoder 108 for the subset are met, the content-specific information (e.g., decompressed data 236) may be stored together with the subset. If the ML model-based video decoder 238 wants to decode an image in the form of weights then the specific decompressed data of the subset must be sent only once for each subset. The process of applying the present microprocessing KD method to image compression with GAN and its extension to video compression using latent space residuals is discussed below.

[0046] Figure 3Shows a traditional HiFiC architecture 340. Its decoder 348 can be divided into three subnets: the head 342 includes approximately two million (2M) parameters, the residual network (res_blocks) 344 includes approximately 149M parameters, and the tail 346 includes approximately 5.5M parameters. It should be noted that the rough information of the images processed using the HiFiC architecture 340 is saved in the latent space, and the hallucination of the texture is generated by the res_block 344 of the decoder 348. In particular, the size of the res_block 344 is due to the fact that the model has been trained on a large (private) dataset, and thus such a large size is required to capture all the textures seen during training.

[0047] However, if it is known in advance which images should be compressed (e.g., the frames of a video sequence with similar features), then it is possible to overfit to that data during encoding and only send the necessary weights to correctly decode those images (i.e., decompress the data 236). This is implemented using the NN architectures disclosed in this application and referenced Figure 4A and 4B described.

[0048] According to Figure 4A and 4B In the exemplary embodiment shown, by training a smaller subnetwork, the micro-residual network (micro-RN) 460, the size of the student decoder 438 is significantly reduced relative to the size of the teacher decoder 452. This subnetwork, the micro-residual network, mimics Figure 3 the behavior of the res_block 344 for a specific subset of images in Figure 1 and 2 thereby scaling down the hallucination ability of the student decoder 438. As seen above, as an alternative to the HiFiC architecture, in some embodiments, based on the residual block, this novel and creative principle may generally be applied to any GAN training network with a similar architecture. In such a GAN training network embodiment, the residual block portion of the GAN training network decoder can be replaced by the Micro-RN 460. It should also be noted that the student decoder 438 generally corresponds to

[0049] According to Figure 4BIn the exemplary embodiment shown, as is well known in the art, Micro-RN 460 is based on a degradation-aware (DA) block. However, while existing methods using DA blocks typically utilize a kernel prediction network to control weights based on a degradation vector, in accordance with this embodiment each subset weight set is different. Micro-RN460 is defined by two parameters: C h , the number of hidden channels, and B, the number of DA blocks. In one embodiment, 3×3 convolutions may be used. Referring to Figure 4B , DConv represents depth convolution. Micro-RN 460 is trained with the Figure 4A shown teacher-student architecture, and the head 456 and tail 458 of the student decoder 438 (hereinafter referred to as "ML model-based video decoder 438") are pre-trained and borrowed from the teacher-decoder 452.

[0050] Let be the image of subset and be the image compressed by the teacher network. In accordance with this concept, the following loss function is optimized:

[0051]

[0052] where, is the output of the student network, MSE (mean squared error) and d p is the distortion loss, and k M and k p are their respective weights. The perceptual loss d p = LPIPS. As a result, the loss forces the ML model-based video decoder 438 to generate images that look similar to those generated by the teacher decoder 452 and further reduces the perceptual loss to the ground truth. It should be noted that the encoder and entropy model using hyperprior modeling may be frozen. Thus, the compressed data 236 only contains the weights of Micro-RN 460. This advantageously makes full use of the powerful encoder and the hyperprior of HiFiC as well as the model's knowledge of the private training dataset.

[0053] To illustrate the application of KD in the neural video compression scenario, a network such as Figure 5 the network 570 in Figure 5As shown, the network 570 includes two parts: a frame prediction network (FPN) 572 and a latent residual network (LRN) 574. Given a sequence of frames (groups of pictures, or GOPs) to be encoded x0,...,x GOP , where x0 is a key frame (I frame) and x1,...,x GOP are predicted frames (P frames), compression of the sequence may work as follows:

[0054] First, it is possible to compress the I-frame using a neural image compression network to generate an encoded latent quantity y0. Let Represents the potential quantity from quantization Then, for each P frame, x t+1 ,1≤t+1≤GOP: (1) Using FPN 692, reconstructed from the previous frame Generate x t+1 Time prediction FPN 692 is accomplished by first calculating the t+1 and The optical flow between t+1 (2) Use a neural motion compression network to generate encoding and f t+1 The quantified potential (3) Using the decompressed stream right transform, and then motion compensate it to generate a temporal prediction

[0055] To compute the residual between the temporal prediction and the P frame, LRN 574 is used to: (4) Optimize the prediction using EI 576 (pre-trained image compression encoder) and x t+1 Both are encoded, and (5) the potential residual between the P frame and the predicted frame is calculated It was then quantified and analyzed using EM I 578 is entropy coded. Then, the final compressed bitstream of GOP is composed of That is, the potential for the I-frame, and the compressed flow field and the potential residual (both quantized and entropy coded) for each of the P-frames.

[0056] In low bitrate settings, HiFiC seems to be a suitable choice for a neural image compression architecture that can be used with the latent space residual framework described above. However, as seen above, the size of the HiFiC decoder is a limiting factor. Additionally, in video, inference time can be critical, where a decoding frame rate of approximately thirty frames per second (30 fps) is often required. The microprocessing scheme disclosed in this application advantageously advances the state of the art by increasing computational efficiency while reducing inference time. During encoding, the scheme is overfitted to a specific sequence such that for all frames of that sequence only one transmission is required. Then, this novel and creative decoding process continues by receiving the Micro-RN weights for a specific sequence and loading them onto the ML model-based video decoder 438, which weights are then fixed during decoding of the sequence. Due to the small computational overhead imposed by this microprocessing scheme, the decoding time can be advantageously reduced by 50% while achieving visuals similar to those of larger and slower existing decoders.

[0057] The above reference Figure 2 、 3 、4A, 4B and 5 describes knowledge distillation with a microprocessing method that will be further described with reference to Figure 6 and 7 will be further described, Figure 6 shows a flowchart 680 that illustrates an exemplary method of performing microprocessing for low bitrate video compression according to one embodiment, while Figure 7 shows a flowchart 790 that describes additional actions for extending the Figure 6 method outlined in Figure 6 and 7 Regarding the actions described in

[0058] Note that certain details and features are omitted from flowcharts 680 and 790 so as not to obscure the discussion of the inventive features in this application.

[0058] Now in connection with Figure 1 and 2 Referring to Figure 6 , flowchart 680 includes receiving an uncompressed video sequence 116 / 216 that includes a plurality of video frames (action 681). As Figure 2 shown, the uncompressed video sequence 116 / 216 may be received by the ML model-based video encoder 108 / 208 in action 681. Additionally, as described above with reference to Figure 1 , the ML model-based video encoder 108 / 208 may be stored in the system memory 106. Thus, the uncompressed video sequence 116 / 216 may be received by the ML model-based video encoder 108 / 208 executed by the processing hardware 104 of the system 100 in action 681.

[0059] The flow chart 680 further includes determining a first subset of video frames and a second subset of video frames from the plurality of video frames (action 682). In some embodiments, the determination of the first subset of video frames and the second subset of video frames in action 682 may be based on similarities and dissimilarities among the video frames included in the uncompressed video sequence 116 / 219. In other words, in some embodiments, the first subset of video frames determined in action 682 may include video frames that are visually similar to each other, while the second subset of video frames may include other video frames that are more visually similar to each other compared to the video frames of the first subset of video frames. Continuing with reference Figure 1 and Figure 2 , the determination of the first subset of video frames and the second subset of video frames in action 682 may be implemented by the ML model-based video encoder 108 / 208, which is executed by the processing hardware 104 of the system 100.

[0060] The flow chart 680 also includes encoding the first subset of video frames determined in action 682 to produce a first subset of compressed video frames (action 683). As described above, in action 683, the first subset of video frames may be encoded by the ML model-based video encoder 108 / 208 to produce a first subset of compressed video frames which is executed by the processing hardware 104 of the system 100.

[0061] The flow chart 680 further includes identifying first decompression data for the first subset of compressed video frames (action 684). In some embodiments, identifying the first decompression data includes, in action 683, overfitting the first decompression data during the encoding of the first subset of video frames That is, in some embodiments, the identification of the first decompression data in action 684 may be performed in parallel with, i.e., substantially simultaneously with, the encoding of the first subset of video frames in action 682 to produce the first subset of compressed video frames . As described above, in action 684, the first decompression data for the first subset of compressed video frames may be identified by the ML model-based video encoder 108 / 208 which is executed by the processing hardware 104 of the system 100. The flow chart 680 also includes encoding the second subset of video frames determined in action 682 to produce a second subset of compressed video frames which is executed by the processing hardware 104 of the system 100.

[0062] The flow chart 680 also includes encoding the second subset of video frames determined in action 682 to produce a second subset of compressed video frames (Action 685). As described above, in Action 685, the second subset of video frames may be encoded by the ML model-based video encoder 108 / 208 to produce the second subset of compressed video frames The decoder is executed by the processing hardware 104 of the system 100.

[0063] Flowchart 680 further includes identifying second decompressed data for the second subset of compressed video frames of (Action 686). In some embodiments, identifying the second decompressed data includes overfitting the second decompressed data during the encoding of the second subset of video frames in Action 685 That is, in some embodiments, the second decompressed data in Action 686 may be identified in parallel with encoding the second subset of video frames in Action 685 to produce the second subset of compressed video frames, i.e., substantially simultaneously. As described above, in Action 686, the first decompressed data for the second subset of compressed video frames 2 may be identified by the ML model-based video encoder 108 / 208 The decoder is executed by the processing hardware 104 of the system 100. The decoder is executed by the processing hardware 104 of the system 100.

[0064] Regarding the first decompressed data and the second decompressed data It should be noted that these data are specific to the respective subsets of compressed video frames with which they are associated. Thus, the first decompressed data is specific to decoding the first subset of compressed video frames and not the second subset of compressed video frames and the second decompressed data is specific to decoding the second subset of compressed video frames and not the first subset of compressed video frames

[0065] It should also be noted that although Flowchart 680 depicts Actions 685 and 686 as occurring after Actions 683 and 684, this representation is provided only as an example. In some other embodiments, Actions 683 and 685 may be performed in parallel before Actions 684 and 686. In some embodiments, it may also be performed in parallel. Thus, in some embodiments, Actions 683 and 684 may be performed in parallel with Actions 685 and 686.

[0066] In some embodiments, the method outlined by Flowchart 680 may end with Action 686. However, in other embodiments, the method may continue by Figure 7One or more action extensions described by the flowchart 790 in Figure 1 and 2 Referring to Figure 7 , the flowchart 790 includes sending a first subset of compressed video frames a second subset of compressed video frames first decompressed data and second decompressed data to the ML model-based video decoder 138 / 238 / 438 (action 791). As Figure 2 shown, in action 791, the first subset of compressed video frames the second subset of compressed video frames the first decompressed data and the second decompressed data may be sent by the ML model-based video encoder 108 / 208 via the communication network 110 and the network communication link 112 to the ML model-based video decoder 138 / 238 / 438, which is executed by the processing hardware 104 of the system 102.

[0067] The flowchart 790 further includes receiving the first subset of compressed video frames the second subset of compressed video frames the first decompressed data and the second decompressed data (action 792). As Figure 2 shown, the first subset of compressed video frames the second subset of compressed video frames the first decompressed data and the second decompressed data may be received by the ML model-based video decoder 138 / 238 / 438 in action 792. In some embodiments, the ML model-based video decoder may be executed by the user system processing hardware 124. However, and as noted above with reference to Figure 1 , in some embodiments, the user system 120 may be a dumb terminal peripheral component of the system 100. In those latter embodiments, the ML model-based video decoder 138 / 238 / 438 is included as a feature of the system 100 and may be executed to implement action 792 by the processing hardware 104 of the system 100.

[0068] The flowchart 790 further includes using the first decompressed data to decode the first subset of compressed video frames (action 793). As described above, in action 793, the first decompressed data Decode the first subset of compressed video frames This decoder is executed by the user system processing hardware 124 or by the processing hardware 104 of the system 100.

[0069] In some embodiments, as noted above, the ML model-based video decoder 138 / 238 / 438 may include an NN, for example, such as a MicroRN. In embodiments where the ML model-based video decoder 138 / 238 / 438 includes a MicroRN, the first decompressed data may only contain the weights of the MicroRN for decoding the first subset of compressed video frames In addition, in some embodiments, the first decompressed data may be received once and only once for overall decoding of the first subset of compressed video frames

[0070] Flowchart 790 further includes using the second decompressed data to decode the second subset of compressed video frames (Action 794). As described above, in Action 794, the second decompressed data may be used by the ML model-based video decoder 138 / 238 / 438 to decode the second subset of compressed video frames This decoder is executed by the user system processing hardware 124 or by the processing hardware 104 of the system 100.

[0071] In embodiments where the ML model-based video decoder 138 / 238 / 438 includes a MicroRN, the second decompressed data may only contain the weights of the MicroRN for decoding the second subset of compressed video frames In addition, in some embodiments, the second decompressed data may be received once and only once for overall decoding of the second subset of compressed video frames

[0072] It should be noted that although flowchart 790 depicts Action 794 as after Action 793, this representation is provided only as an example. In some embodiments, using the first decompressed data in Action 793 to decode the first subset of compressed video frames and using the second decompressed data in Action 794 to decode the second subset of compressed video frames may be executed in parallel, that is, substantially simultaneously.

[0073] Regarding the combination of the ML model-based video encoder 108 / 208 and the ML model-based video decoder 138 / 238 / 438, it should be noted that the ML model-based video encoder 108 / 208 may be implemented as a HiFiC encoder, while the ML model-based video decoder 138 / 238 / 438 is configured to have fewer parameters. For example, it may have, for instance, ten times fewer parameters than a HiFiC decoder (i.e., a large decoder that does not use the first decompressed data and the second decompressed data . In addition, the ML model-based video decoder 138 / 238 / 438 may be configured to facilitate a faster decoding time, e.g., 50% faster than a HiFiC decoder (i.e., a large decoder that does not use the first decompressed data and the second decompressed data ).

[0074] Regarding Figure 6 and Figure 7 the actions described in

[0075] note that in various embodiments, the actions 681, 682, 683, 684, 685, and 686 of flowchart 680 (hereinafter "actions 681-686"), or actions 681-686, and action 791 of flowchart 790, or actions 681-686 / 791 / 792 / 793, and 794 may be implemented as an automated process that omits human participation.

[0076] From the above description, it is obvious that various techniques can be used to implement the concepts described in this application without departing from the scope of these concepts. In addition, although the concepts have been described with reference to certain embodiments, those of ordinary skill in the art should recognize that changes can be made in form and detail without departing from the scope of these concepts. Therefore, the described embodiments are considered illustrative rather than restrictive in all respects. It should also be understood that this application is not limited to the specific embodiments described herein, but that many rearrangements, modifications, and substitutions are possible without departing from the scope of the present disclosure.

Claims

1. A system, comprising: A video encoder based on a machine learning ML model; And A video decoder based on an ML model, including a micro-residual network MicroRN based on a degradation-aware DA block, the MicroRN being defined by several hidden channels of the MicroRN and several degradation-aware DA blocks; the MicroRN serves as a residual network in a decoder trained by a generative adversarial network GAN; The video encoder based on the ML model is configured to: Receive an uncompressed video sequence including a plurality of video frames; Determine a first subset of video frames and a second subset of video frames from the plurality of video frames; Encode the first subset of video frames to produce a first subset of compressed video frames; Identify first decompression data for the first subset of compressed video frames; Encode the second subset of video frames to produce a second subset of compressed video frames; Identify second decompression data for the second subset of compressed video frames; And Send the first subset of compressed video frames, the second subset of compressed video frames, the first decompression data, and the second decompression data to the video decoder based on the ML model; and The video decoder based on the ML model is configured to: Receive the first subset of compressed video frames, the second subset of compressed video frames, the first decompression data, and the second decompression data; Decode the first subset of compressed video frames using the first decompression data; And Decode the second subset of compressed video frames using the second decompression data.

2. The system according to claim 1, wherein, Identifying the first decompression data includes overfitting the first decompression data during the encoding of the first subset of video frames, and wherein identifying the second decompression data includes overfitting the second decompression data during the encoding of the second subset of video frames.

3. The system according to claim 1, wherein The first decompression data is dedicated to decoding the first subset of compressed video frames and not to the second subset of compressed video frames, and the second decompression data is dedicated to decoding the second subset of compressed video frames and not to the first subset of compressed video frames.

4. The system according to claim 1, wherein The first decompression data and the second decompression data only contain the weights of the micro-residual network.

5. The system according to claim 1, wherein The video encoder based on the ML model includes a high-fidelity compression HiFiC encoder, and wherein the video decoder based on the ML model has at least ten times fewer parameters compared to a HiFiC decoder that does not use the first decompression data and the second decompression data.

6. The system according to claim 1, wherein The video encoder based on the ML model includes a HiFiC encoder, and wherein the video decoder based on the ML model is configured to be 50% faster compared to a HiFiC decoder that does not use the first decompression data and the second decompression data.

7. The system according to claim 1, wherein The first decompression data is only received once for decoding the first subset of compressed video frames, and wherein the second decompression data is only received once for decoding the second subset of compressed video frames.

8. A method used by a system including a machine learning (ML)-based video encoder and an ML-based video decoder, the video decoder including a micro-residual network (MicroRN) based on degradation-aware blocks, the MicroRN being defined by a number of hidden channels and a number of degradation-aware blocks of the MicroRN; the MicroRN being used as a residual network in a decoder trained by a generative adversarial network (GAN); The method includes: Receiving, by the ML-based video encoder, an uncompressed video sequence including a plurality of video frames; Determining, by the ML-based video encoder, a first subset of video frames and a second subset of video frames from among the plurality of video frames; Encoding, by the ML-based video encoder, the first subset of video frames to produce a first subset of compressed video frames; Identifying, by the ML-based video encoder, first decompression data for the first subset of compressed video frames; Encoding, by the ML-based video encoder, the second subset of video frames to produce a second subset of compressed video frames; Identifying, by the ML-based video encoder, second decompression data for the second subset of compressed video frames; And Sending, by the ML-based video encoder, the first subset of compressed video frames, the second subset of compressed video frames, the first decompression data, and the second decompression data to the ML-based video decoder; Receiving, by the ML-based video decoder, the first subset of compressed video frames, the second subset of compressed video frames, the first decompression data, and the second decompression data; Decoding, by the ML-based video decoder, the first subset of compressed video frames using the first decompression data; And Decoding, by the ML-based video decoder, the second subset of compressed video frames using the second decompression data.

9. The method according to claim 8, wherein Identifying the first decompression data includes overfitting the first decompression data during the encoding of the first subset of video frames, and wherein identifying the second decompression data includes overfitting the second decompression data during the encoding of the second subset of video frames.

10. The method according to claim 8, wherein The first decompression data is dedicated to decoding the first subset of compressed video frames and not dedicated to the second subset of compressed video frames, and the second decompression data is dedicated to decoding the second subset of compressed video frames and not dedicated to the first subset of compressed video frames.

11. The method according to claim 8, wherein, The first decompression data and the second decompression data contain only the weights of the micro-residual network.

12. The method according to claim 8, wherein, The ML-based video encoder includes a high-fidelity compression (HiFiC) encoder, and wherein the ML-based video decoder has at least ten times fewer parameters compared to a HiFiC decoder that does not use the first decompression data and the second decompression data.

13. The method according to claim 8, wherein, The ML model-based video encoder includes a HiFiC encoder, and wherein the ML model-based video decoder is configured to be 50% faster than a HiFiC decoder that does not use the first decompressed data and the second decompressed data.

14. The method according to claim 8, wherein, The first decompressed data is received only once for decoding the first subset of compressed video frames, and wherein the second decompressed data is received only once for decoding the second subset of compressed video frames.

Images