System and method for encoding and decoding image data using generative adversarial model

Abstract

A video encoder and decoder system employing a compression decision generative adversarial model ("CDGAM") has an encoder with a compression network and a decoder with a task network. The compression network is a generator part of the pre-trained CDGAM, and the task network is a discriminator part of the pre-trained CDGAM. A compression network and a task network are trained using common image data, where the compression network is trained to generate a minimum compression representation that produces a loss substantially similar to the original image at the discriminator.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of priority to U.S. Provisional Application No. 63 / 317,590, filed on March 8, 2022, entitled “Systems and Methods for Generative Adversarial Models For Image and Video Compression,” the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present invention relates generally to the field of video encoding and decoding, and more particularly to systems and methods for encoding and decoding image data using a general adversarial model. Background Art

[0004] A video codec may include electronic circuits or software that compresses or decompresses digital video. It can convert uncompressed video to a compressed format and vice versa. In the context of video compression, a device that compresses video (and / or performs some of its functions) may generally be referred to as an encoder, and a device that decompresses video (and / or performs some of its functions) may be referred to as a decoder.

[0005] The format of the compressed data may conform to a standard video compression specification. The compression may be lossy in that the compressed video lacks some information present in the original video. The result of this may include that the decompressed video may be of lower quality compared to the original uncompressed video because there is not enough information to accurately reconstruct the original video.

[0006] There may be a complex relationship between video quality, the amount of data used to represent the video (e.g., determined by the bitrate), the complexity of the encoding and decoding algorithms, sensitivity to data loss and errors, ease of editing, random access, end-to-end delay (e.g., latency), etc.

[0007] Motion compensation may include methods of predicting a video frame or a portion thereof, such as a previous frame and / or a future frame, given a reference frame by taking into account the motion of the camera and / or objects in the video. It may be employed in the encoding and decoding of video data for video compression, such as in the encoding and decoding of the Advanced Video Coding (AVC) standard (also known as H.264) of the Motion Picture Experts Group (MPEG). Motion compensation may describe a picture in terms of a transformation from a reference picture to a current picture. The reference picture may be previous in time compared to the current picture, and may be from the future compared to the current picture. Compression efficiency may be improved when images may be accurately synthesized from previously transmitted and / or stored images.

[0008] While video content is typically considered for human consumption, there is an increasing need for video in industrial settings and other environments where contention is evaluated by machines rather than humans.

[0009] Recent trends in robotics, surveillance, monitoring, the Internet of Things, etc. have introduced use cases where a large portion of all images and videos recorded in the field are consumed only by machines and do not reach human eyes. These machines process images and videos with the goal of completing tasks such as object detection, object tracking, segmentation, event detection, etc. Recognizing that this trend is universal and will only accelerate in the future, international standardization bodies have made efforts to standardize image and video encoding optimized primarily for machine consumption. For example, in addition to already established standards such as compact descriptors for visual search and compact descriptors for video analysis, standards such as JPEG AI and video coding for machines have been initiated. Therefore, it is increasingly important in the art to further improve the encoding and decoding of videos for machine consumption and in hybrid systems where videos are consumed by both human viewers and machines.

[0010] In many applications, such as surveillance systems with multiple cameras, smart transportation, smart city applications, and / or smart industry applications, traditional video encoding may need to compress large amounts of video from cameras and transmit them over the network for machine and human consumption. Subsequently, at the machine site, feature extraction algorithms can be applied, including object detection, event action recognition, pose estimation, etc., typically using convolutional neural networks or deep learning techniques. Summary of the invention

[0011] The present disclosure generally describes a framework for image and video compression that can be used in systems where the end user is a machine or a human observer or both, such as Video Coding for Machines (VCM), Internet-of-things (IoT), etc. The framework combines a generative adversarial model (GAM) that is pre-trained and distributed to the encoder and decoder sides of the system. The encoder implements a neural network that compresses the input image / video and is trained as a generator. The decoder implements a neural network that uses the compressed image / video as input and outputs a task-dependent decision and is trained as a discriminator. The encoder and decoder can be updated at any time by sending messages between each other or receiving messages from a third party.

[0012] The encoded images / video and model updates are preferably transferred between the encoder and decoder using a bitstream.The proposed framework can support systems for hybrid use cases where the end-users are machines or humans or both.

[0013] In one embodiment, a video encoder is provided that preferably includes a preprocessor that receives an input video signal to be encoded. A compression network is coupled to the preprocessor. The compression network is a pre-trained generator portion of a compression-decision generative adversarial model ("CDGAM"). A network update encoder is operably coupled to the compression network and generates a network update substream. The video encoder is also coupled to the compression network and generates an encoded image substream. A multiplexer receives the network update substream and the encoded image substream and generates a unified encoded bitstream.

[0014] Preferably, the compression network is trained with the discriminator portion of the CDGAM, which provides a generator loss function that is back-propagated to the generator. The compression network can be trained to compress the input image to a minimal compressed representation that produces the same or similar loss as the original image at the discriminator. In some embodiments, the compression network can be trained using the following loss function:

[0015] L CD =E x [logD(x)]+E x [log(1-D(C(x)))]

[0016] Where C(x) is the output of the compression network and D(x) is the output of the discriminator.

[0017] The unified coded bitstream preferably includes a stream level header, a video substream having a video substream header and a video substream payload, and a network update substream having a network update substream header and a network update payload.

[0018] Implementations of a video decoder are also provided. The decoder may include a demultiplexer that receives an input bitstream and parses the bitstream into at least one of a network update substream and a video substream. The network update decoder is coupled to the demultiplexer and receives the network update substream from it. The decoder also includes a video decoder that is coupled to the demultiplexer and receives the video substream from it. The task network is coupled to the network update decoder and the video decoder and receives the network update substream and the video substream. The task network is a pre-trained discriminator portion from a compression decision generative adversarial model, wherein the generator portion is used in the encoder to generate the received bitstream.

[0019] The task network can be trained with image data and a generator part that provides compressed image data. During training, the task network outputs a discriminator loss function, which is back-propagated to the task network to train the discriminator.

[0020] The task network is also used to pre-train a compression network to compress the input image into a minimal compressed representation that produces the same or similar loss as the original image at the discriminator.

[0021] The task network in the decoder can be trained using the following loss function:

[0022] L CD =E x [logD(x)]+E x [log(1-D(C(x)))]

[0023] Where C(x) is the output of the compression network and D(x) is the output of the discriminator.

[0024] These and other aspects and features of non-limiting embodiments of the present invention will become apparent to those skilled in the art from the following description of non-limiting specific embodiments of the present invention read in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] For the purpose of illustrating the present invention, the accompanying drawings show various aspects of one or more embodiments of the present invention. However, it should be understood that the present invention is not limited to the precise arrangements and instrumentalities shown in the accompanying drawings, in which:

[0026] Figure 1 is a block diagram illustrating an exemplary embodiment of a video encoding system;

[0027] Figure 2is a block diagram illustrating an exemplary implementation of video encoding for a machine system;

[0028] Figure 3 is a block diagram illustrating an exemplary embodiment of a system for adaptive neural network optimization for multi-task feature encoding;

[0029] Figure 4 is a block diagram illustrating an exemplary implementation of a machine process having a decoder output as an input for task completion;

[0030] Figure 5 is a block diagram of an exemplary embodiment of a bitstream structure;

[0031] Figure 6 is a schematic diagram illustrating an exemplary implementation of a pruned neural network;

[0032] Figure 7 is a schematic diagram illustrating an exemplary implementation of a complete neural network;

[0033] Figure 8 is a schematic diagram showing two exemplary embodiments of the NNB-NNH split point;

[0034] Fig. 9 is a block diagram illustrating an exemplary implementation of a machine learning module;

[0035] Fig.10 is a schematic diagram illustrating an exemplary implementation of a neural network;

[0036] Fig.11 is a schematic diagram illustrating an exemplary implementation of a node of a neural network;

[0037] Fig.12 is a simplified block diagram of an exemplary generative adversarial model (“GAM”).

[0038] Fig.13A and Fig. 13B are shown respectively as Fig.12 A simplified block diagram of an exemplary training process for the discriminator and generator of the GAM shown.

[0039] Fig.14 is a simplified block diagram of an exemplary implementation of a GAM framework according to the present disclosure.

[0040] Fig.15 is a simplified block diagram illustrating a video encoding and decoding system (such as a VCM framework) employing a compression decision GAM ("CDGAM") according to the present disclosure.

[0041] Fig.16 is a schematic diagram illustrating an exemplary bitstream construction according to an embodiment of the present disclosure.

[0042] Fig.17 is a block diagram illustrating an exemplary embodiment of a video decoder;

[0043] Fig.18 is a block diagram illustrating an exemplary implementation of a video encoder; and

[0044] Fig.19 is a block diagram of a computing system that can be used to implement any one or more of the methods disclosed herein, or any one or more portions thereof.

[0045] The drawings are not necessarily drawn to scale and may be illustrated by phantom lines, diagrammatic representations, and partial views. In certain instances, details that are not necessary for understanding the embodiments or that render other details difficult to perceive may have been omitted. DETAILED DESCRIPTION

[0046] Figure 1 An exemplary embodiment of an encoding / decoding system including a channel applied to a machine is shown. Such a system typically complies with one or more video compression standards, such as HEVC, AV1, VVC, etc. However, conventional methods may require a large amount of video transmission from multiple cameras, which may take a lot of time to make efficient and fast real-time analysis and decisions. In some embodiments, a video coding (VCM) method for a machine can solve this problem by encoding the video and extracting some features at the transmitter site, and then sending the resulting encoded bitstream to a VCM decoder. At the decoder site, the video can be decoded for human vision, and the features can be decoded for the machine. As used herein, the term "VCM" refers broadly to video encoding and decoding for machine consumption, and although the disclosed systems and methods may be standard-compliant, the present disclosure is not limited to a specific proposed protocol or standard.

[0047] As used in the present disclosure, "features" are specific structures and / or content attributes of data. Examples of features may include SIFT, audio features, color histograms, motion histograms, speech levels, loudness levels, etc. Features may be timestamped. Each feature may be associated with a single frame in a set of frames. Features may include high-level content features, such as timestamps, labels of people and objects in the video, coordinates of objects and / or regions of interest, frame masks for region-based quantization, and / or any other features that may occur to a person skilled in the art when reading the entire contents of the present disclosure. As another non-limiting example, features may include features describing the spatial and / or temporal characteristics of a frame or group of frames. Examples of features describing spatial and / or temporal characteristics may include motion, texture, color, brightness, edge count, blur, block effects, etc. When in offline mode, all machine models as described in further detail below may be stored at the encoder and / or in a memory of the encoder and / or in a memory accessible to the encoder. Examples of such models may include, but are not limited to, whole or partial convolutional neural networks, key point extractors, edge detectors, saliency map constructors, etc. When in online mode, one or more models may be transmitted to the feature extractor 220 by a remote machine in real time or at some point in time prior to extraction.

[0048] At the decoder site, it should be understood that the video can be decoded for human vision and the features can be decoded for machines. Systems that provide video for both human vision and machine consumption are sometimes referred to as hybrid systems. The systems and methods disclosed herein are intended to be applied to machine-based systems as well as hybrid systems.

[0049] Figure 11 is a high-level block diagram of a system for encoding and decoding video in a hybrid system that includes both human viewer consumption of video content and machine consumption. A source video is received by a video encoder 105, which provides a compressed bitstream for transmission over a channel to a video decoder 110. The video encoder can encode the video for human consumption and encode the video for machine consumption. The video decoder 110 provides complementary processing of the compressed bitstream to extract the video for human vision 115 and task analysis and feature extraction 120 for machine consumption. Feature extraction can be classified as any computer vision task, such as edge detection, line detection, object detection, or more recent techniques, such as convolutional neural networks, where the output of feature extraction can be spatially mapped back to the pixel space of the input video. Video encoding can include any standard video encoder and / or encoding technology, such as, for example, Advanced Video Codec (AVC), Versatile Video Coding (VVC), or High Efficiency Video Coding (HEVC).

[0050] Reference now Figure 2 , an exemplary embodiment of an encoder for machine video coding (VCM) is shown. VCM encoder 202 can be implemented using any circuit, including but not limited to digital and / or analog circuits; VCM encoder 202 can be configured using hardware configuration, software configuration, firmware configuration and / or any combination thereof. VCM encoder 202 can be implemented as a computing device and / or a component of a computing device, which can include but is not limited to any computing device as described below. In an embodiment, VCM encoder 202 can be configured to receive input video 204 and generate output bitstream 208. Receiving input video 204 can be done in any manner described below. Bitstream can include but is not limited to any bitstream as described below.

[0051] The VCM encoder 202 may include, but is not limited to, a preprocessor 206, a video encoder 210, a feature extractor 215, an optimizer 220, a feature encoder 225, and / or a multiplexer 230. The preprocessor 206 may receive a stream of input video 204 and parse out video, audio, and metadata substreams of the stream. The preprocessor 206 may include and / or communicate with a decoder, as described in further detail below; in other words, the preprocessor 206 may have the ability to decode the input stream. In a non-limiting example, this may allow decoding of the input video 204, which may enable downstream pixel domain analysis.

[0052] Further references Figure 2, VCM encoder 202 can operate in a hybrid mode and / or a video mode; when in the hybrid mode, VCM encoder 200 can be configured to encode visual signals intended for human consumers and to encode feature signals intended for machine consumers; machine consumers can include, but are not limited to, any device and / or component, including, but not limited to, a computing device as described in further detail below. For example, when in the hybrid mode, the input signal can pass through preprocessor 206.

[0053] Still reference Figure 2 , the video encoder 210 may include, but is not limited to, any video encoder 210 as described in further detail below. When the VCM encoder 202 is in hybrid mode, the VCM encoder 202 may send an unmodified input video 204 to the video encoder 210 and send a copy of the same input video 204 and / or an input video 204 that has been modified in some manner to the feature extractor 215. The modification to the input video 204 may include any scaling, transformation, or other modification that would occur to a person of ordinary skill in the art upon reading the entirety of this disclosure. For example, but not limited to, the input video 204 may be resized to a smaller resolution, a certain number of pictures in a sequence of pictures in the input video 204 may be discarded, the frame rate of the input video 204 may be reduced, color information may be modified, for example, but not limited to, by converting an RGB video that may be converted to a grayscale video, and the like.

[0054] Still reference Figure 2 , the video encoder 210 and the feature extractor 215 are connected and may exchange useful information in both directions. For example, but not limited to, the video encoder 210 may transmit motion estimation information to the feature extractor 220, and vice versa. The video encoder 210 may provide the feature extractor 215 with a quantization map and / or data description of a region of interest (ROI) that the video encoder 210 and / or the feature extractor 215 may identify, and vice versa. The video encoder 210 may provide the feature extractor 215 with data describing one or more segmentation decisions based on features present and / or identified in the input video 204, the input signal, and / or any frame and / or subframe thereof; the feature extractor 218 may provide the video encoder 210 with data describing one or more segmentation decisions based on features present and / or identified in the input video 204, the input signal, and / or any frame and / or subframe thereof. The video encoder 210 and / or the feature extractor 215 may share and / or transmit temporal information for optimal group of pictures (GOP) decisions with each other. Each of these techniques and / or processes may be performed, but are not limited to, as described in further detail below.

[0055] Continue to refer Figure 2, the feature extractor 220 can operate in an offline mode or an online mode. The feature extractor 220 can identify and / or otherwise act on and / or manipulate features. As used in the present disclosure, "features" are specific structures and / or content attributes of data. Examples of features may include SIFT, audio features, color histograms, motion histograms, speech levels, loudness levels, etc. Features may be timestamped. Each feature may be associated with a single frame in a set of frames. Features may include high-level content features such as timestamps, labels of people and objects in the video, coordinates of objects and / or regions of interest, frame masks for region-based quantization, and / or any other features that may occur to a person skilled in the art when reading the entire contents of this disclosure. As another non-limiting example, features may include features that describe spatial and / or temporal characteristics of a frame or group of frames. Examples of features that describe spatial and / or temporal characteristics may include motion, texture, color, brightness, edge count, blur, block effects, etc. When in offline mode, all machine models as described in further detail below may be stored at the encoder and / or in a memory of the encoder and / or in a memory accessible to the encoder. Examples of such models may include, but are not limited to, whole or partial convolutional neural networks, keypoint extractors, edge detectors, saliency map constructors, etc. When in online mode, one or more models may be transmitted to the feature extractor 220 by a remote machine in real time or at some point prior to extraction.

[0056] Still reference Figure 2 , the feature encoder 225 is configured to encode, for example, but not limited to, a feature signal generated by the feature extractor 220. In an embodiment, after extraction, the feature extractor 220 may pass the extracted features to the feature encoder 225. The feature encoder 225 may use entropy coding and / or similar techniques (for example, but not limited to, as described below) to produce a feature stream, which may be passed to the multiplexer 230. The video encoder 210 and / or the feature encoder 225 may be connected via the optimizer 220; the optimizer 220 may exchange useful information between the video encoder 210 and the feature encoder 225. For example, but not limited to, information related to codeword construction and / or length for entropy coding may be exchanged and reused via the optimizer 220 for optimal compression.

[0057] In the embodiments, and with continued reference to Figure 2, the video encoder 210 can generate a video stream; the video stream can be passed to the multiplexer 230. The multiplexer 230 can multiplex the video stream with the feature stream generated by the feature encoder 225; alternatively or additionally, the video bitstream and the feature bitstream can be transmitted to different devices through different channels, different networks, and / or transmitted at different times or time intervals (time multiplexing). Each of the video stream and the feature stream can be implemented in any manner suitable for implementing any bitstream as described in the present disclosure. In an embodiment, the multiplexed video stream and feature stream can generate a mixed bitstream, which can be sent as described in further detail below.

[0058] Still reference Figure 2 , in the case where the VCM encoder 200 is in video mode, the VCM encoder 200 may use the video encoder 210 for both video encoding and feature encoding. The feature extractor 220 may send features to the video encoder 210; the video encoder 210 may encode the features into a video stream that may be decoded by a corresponding video decoder 250. It should be noted that the VCM encoder 200 may use a single video encoder 210 for both video encoding and feature encoding, in which case the VCM encoder 200 may use different sets of parameters for video and features; alternatively, the VCM encoder 200 may be two separate video encoders 210 that may operate in parallel.

[0059] Still reference Figure 2 , system 200 may include and / or communicate with a VCM decoder 240. VCM decoder 240 and / or its elements may be implemented using any circuit and / or type of configuration suitable for the configuration of VCM encoder 200 as described above. VCM decoder 240 may include, but is not limited to, a demultiplexer 245. If multiplexed as described above, demultiplexer 245 may operate to demultiplex the bitstream. For example, but not limited to, demultiplexer 245 may separate a multiplexed bitstream containing one or more video bitstreams and one or more feature bitstreams into separate video bitstreams and feature bitstreams.

[0060] Continue to refer Figure 2 , VCM decoder 240 may include a video decoder 250. Video decoder 250 may be implemented in any manner suitable for a decoder, as described in further detail below, but not limited to. In one embodiment, but not limited to, video decoder 250 may generate an output video that may be viewed by a human or other living being and / or a device having visual perception capabilities.

[0061] Still reference Figure 2, the VCM decoder 240 may include a feature decoder 255. In one embodiment, and without limitation, the feature decoder 255 may be configured to provide one or more decoded data to a machine. A machine may include, but is not limited to, any computing device as described below, including but not limited to any microcontroller, processor, embedded system, system on chip, network node, etc. The machine may operate, store, train, receive input from a machine model, generate output for a machine model, and / or otherwise interact with a machine model, as described in further detail below. The machine may be included in the Internet of Things (IoT), which is defined as a network of objects with processing and communication components, some of which may not be conventional computing devices such as desktop computers, laptops, and / or mobile devices. Objects in IoT may include, but are not limited to, any device having an embedded microprocessor and / or microcontroller and one or more components for interfacing with a local area network (LAN) and / or a wide area network (WAN); one or more components may include, but are not limited to, for example, a wireless transceiver communicating in the 2.4-2.485 GHz range, such as a BLUETOOTH transceiver following a protocol published by Bluetooth SIG, Inc. of Kirkland, Washington, and / or a network communication component operating according to the MODBUS protocol published by Schneider Electric SE of Rueil-Malmaison, France and / or the ZIGBEE specification of the IEEE 802.15.4 standard published by the Institute of Electrical and Electronics Engineers (IEEE). A person skilled in the art will recognize various alternative or additional communication protocols and devices supporting such protocols that may be adopted in accordance with the present disclosure upon reading the entire contents of the present disclosure, each of which is considered to be within the scope of the present disclosure.

[0062] The machine model 265 may be included and integrated within the decoder 240, or as a separate component separate from the decoder 240 at a remote location. The machine model is preferably coupled to both the feature decoder 255 in the decoder and the feature extractor 215 in the encoder. Examples of such machine models may include, but are not limited to, a whole or part of a convolutional neural network, a keypoint extractor, an edge detector, a saliency map constructor, and the like.

[0063] Continue to refer Figure 2Each of the VCM encoder 202 and / or the VCM decoder 240 may be designed and / or configured to perform any method, method step, or sequence of method steps in any embodiment described in the present disclosure in any order and to any degree of repetition. For example, each of the VCM encoder 202 and / or the VCM decoder 240 may be configured to repeatedly perform a single step or sequence until a desired or commanded result is achieved; may iteratively and / or recursively perform repetitions of a step or sequence of steps using the output of a previous repetition as input to a subsequent repetition, aggregate the input and / or output of a repetition to produce an aggregated result, reduce or decrement one or more variables (such as global variables), and / or divide a larger processing task into a set of iteratively addressed smaller processing tasks. Each of the VCM encoder 202 and / or the VCM decoder 240 may perform any step or sequence of steps as described in the present disclosure in parallel, such as performing a step two or more times simultaneously and / or substantially simultaneously using two or more parallel threads, processor cores, etc.; the division of tasks between parallel threads and / or processes may be performed according to any protocol suitable for the division of tasks between iterations. Those skilled in the art, upon reading this disclosure in its entirety, will recognize the various ways in which steps, sequences of steps, processing tasks and / or data may be subdivided, shared, or otherwise processed using iteration, recursion, and / or parallel processing.

[0064] The present system and method are based on a machine learning architecture that supports multiple tasks for end users. Most common machine learning architectures used today are neural networks. One of the disadvantages of simple single-task neural networks is the time complexity and computational cost of training. In order to achieve high performance, it is usually necessary to use very large data sets with hundreds of thousands and sometimes millions of samples (such as images and videos) to train the neural network. Training a separate network every time a new use case appears may be highly redundant and wasteful of resources. Therefore, a method has been developed to reuse the trained parts of the neural network for multiple tasks. By training a part of the network to support multiple tasks, users can save storage space, computing power and reduce energy consumption. In the context of a neural network, the backbone is the part of the network that includes most layers and nodes and is primarily responsible for a large number of calculations. On the other hand, the head is the part of the network responsible for making the final prediction or decision based on the output of the backbone. As used herein, those neural network parts that can be reused and are substantially unchanged for different tasks are referred to as "neural network backbones" (trunks). The rest of the neural network modified based on the target task is referred to as "neural network heads" (heads).

[0065] Figure 3is a block diagram illustrating an exemplary embodiment of a system for adaptive neural network optimization for multi-task feature coding; the system can be used in a method for adaptive neural network optimization, which can be used to implement multi-task feature coding in a system such as a standard for machine video coding (VCM), or for implementing multi-task feature coding in any other system for neural networks performing image and video processing by machines such as the Internet of Things (IoT).

[0066] Typically, an input image or video is passed through an encoder 300 that extracts relevant features. The encoder 300 includes a preprocessor 305 coupled to a neural network backbone (neural network backbone) 310, the output of which is a set of features that can be further compressed using a feature / video encoder 315. The neural network backbone 310 can also receive input and / or training data from a machine model 360. The output of the video encoder 315 is a feature substream. In addition to the features, a description of the neural network backbone 310 can be compressed, and the description substream can be combined with the feature substream in a multiplexer 320 from the neural network description encoder 325 to form a bitstream 330.

[0067] The bitstream 330 is sent through a channel to a decoder 335 which decodes / decompresses the two substreams and produces a neural network backbone description as well as a feature set. The bitstream 330 is received by a demultiplexer 340 which parses the bitstream into a neural network backbone description substream applied to a neural network description decoder 345 and a feature set substream applied to a feature / video decoder 350. The end user (machine) 355 receives the feature set and passes it as input to a neural network head (NNH) which produces at output one or more task-related decisions to be performed by the machine. This is possible if the complete neural network (NN) is pre-trained and the parameters of the neural network backbone are known and unchanged.

[0068] Figure 4 4 is a block diagram illustrating an exemplary embodiment of a machine process, wherein the output of a decoder 435 is used as an input to a task completion process. In the case where the neural network backbone 415 has not been trained by the machine and the parameters of the neural network backbone 415 are unknown, the machine receives the neural network backbone description 405 from the decoder 435 and reconstructs the complete neural network backbone 415, and optionally fine-tunes the neural network head, either to align with the neural network backbone, or to support additional tasks that were not included in the original neural network backbone training. Here, neural network retraining (NNH fine-tuning) is an optional step 420.

[0069] In the case where the machine 355 knows the complete neural network, such as by knowing all neural network backbone parameters, the neural network backbone description does not need to be sent to the decoder. In such a case, the bitstream may contain only the feature substream 410.

[0070] To achieve flexibility on the encoder side, in case the task needs to be changed or the parameters of the current task need to be changed, the machine can send the neural network backbone description to the encoder side aligned with the relevant machine model.

[0071] On the encoder side, the feature / video encoder can be implemented using existing video and feature standards, such as Versatile Video Coding (VVC), High Efficiency Video Coding (HEVC), AOMedia Video 1 (AV1), Compact Descriptors for Video Analysis (CDVA), etc.

[0072] The neural network description encoder can be implemented using simple techniques such as Variable Length Coding (VLC) or more advanced standards such as Neural Network Compression and Representation (NNR).

[0073] Figure 5 5 is a block diagram of an exemplary embodiment of a bitstream structure suitable for use with the present encoding and decoding system. The bitstream includes a stream-level header 505, which contains high-level syntax describing the presence of substreams and contains parameters of such substreams (e.g., length, duration, format, etc.). The demultiplexer in the decoder uses this information to extract the substreams.

[0074] The bitstream may also include a feature substream 510 including a feature stream header 515 including syntax elements to describe the feature stream payload 520 in terms of length, format, and other relevant parameters. The feature stream header 515 may be used by a feature decoder to extract and decode the feature stream payload 520.

[0075] The neural network description substream 525 typically contains a neural network description header 530, which contains parameters describing the length, format, type, etc. of the neural network description. Alternatively, instead of a complete description of all neural network parameters, the encoder may signal the index of the neural network used in a predetermined and agreed list or lookup table. The list may be maintained by a central registry that updates the list and signals the update to the end user. The feature decoder uses the neural network description header to extract and decode the neural network description payload.

[0076] The split point between the neural network head and the neural network backbone can be adaptively chosen based on the level of abstraction and cross-task generalization. Furthermore, when training neural networks for multiple tasks and use cases, some parts of the trained neural network may be primarily or entirely task-related, while other parts may be primarily or entirely redundant for the specific task. In the following paragraphs, we propose two techniques for adaptive optimization: vertical optimization (task-specific ablation) and horizontal optimization (computation offloading).

[0077] Figure 6 600 is a schematic diagram showing an exemplary embodiment of a pruned neural network; such an embodiment can implement vertical optimization and / or task-specific ablation. The techniques of ablation or pruning are well established in the machine learning literature, with a particular focus on neural networks. The idea is that parts of a neural network can be removed when training for a specific task. By pruning neurons or connections (synapses), a single neural network 600 can be optimized to support multiple tasks. Figure 6 An example of pruning is given in , which shows a complete neural network 600, a neural network with pruned synapses 605, and a neural network with pruned neurons 610. By removing unimportant weights from the network, several improvements can be expected: better generalization, fewer required training examples, and improved learning and / or classification speed. The basic idea is to use second-order derivative information to make a trade-off between network complexity and training set error.

[0078] As an example, one part of a neural network can be trained for object detection, while another part can be trained for object tracking or segmentation. At a lower level, one part of the network can be trained to detect one class of objects (e.g., people), while another part can be trained to detect another class of objects (e.g., cars).

[0079] Figure 7is a schematic diagram showing an exemplary embodiment of a complete neural network. The computational complexity of the encoder generally depends linearly on the size of the neural network backbone. Although vertical optimization can reduce the amount of computation in the neural network backbone in some cases, the split point between the neural network backbone and the neural network head is the primary tool for computational offloading. The split point is preferably selected based on the criteria of energy consumption, computational power, and bandwidth between the encoder and decoder.

[0080] Figure 7 is a schematic diagram showing an exemplary embodiment of a neural network as described in the present disclosure, showing a combined convolutional neural network-deep neural network (CNN-DNN) architecture for machine learning of image and video data. An input image or video passes through a convolutional neural network (CNN) 705, which uses a series of processing steps including a convolution layer 710 and a pooling layer 715. Each layer in the network uses filters to process the input and outputs a so-called feature map, which is a two-dimensional image containing some level of detail related to the entire task. A CNN can have two or three to dozens of such layers. The lower layers filter small details of the input image, such as edges and lines. As the CNN is moved to a higher layer, the output more closely represents the interesting features of the task. For example, the highest layer in a CNN for face detection will produce a feature map containing features such as eye contours, nose contours, etc. The output of the highest layer is "flattened" and passed through a deep neural network (DNN) 720, which outputs the probability that the image contains a certain object (e.g., "car", "cat") with or without location details.

[0081] In addition to the object detection task, the same architecture with minor modifications can be used for object tracking, segmentation, event detection and other tasks. As we have seen, more general features are represented at the lower layers of CNN 705. These features are shared between different objects and sometimes even between different tasks. This fact is used in the design of multi-task architectures. The part of the neural network that can be reused and does not change for different tasks is called the "neural network backbone" (trunk). The rest of the neural network that is modified based on the target task is called the "neural network head" (head).

[0082] refer to Figure 8, showing two exemplary variants of the backbone-head separation of the multi-task architecture. The training process can be common to all backbone-head architectures and variants, for example, the backbone can be trained only once, and all parameters (such as connection weights and activation function thresholds) are typically not changed. Using this backbone, the head is added to it, for example by connecting the last layer of the backbone to the first layer of the head. The entire network is then put through training, keeping the backbone parameters constant and only changing the parameters of the head until satisfactory performance is achieved on the training and / or validation datasets.

[0083] exist Figure 8 In Option 1, the entire CNN 805 is designated as the backbone 825, while the DNN 820 is designated as the head 830. This means that the CNN 805 can be trained only once, while the DNN 820 can be trained multiple times, each time for a separate task, or alternatively several DNNs 820 can be trained, each for a separate task. This variant is best suited for tasks that span different modalities, such as object detection and event detection. For example, the backbone can be trained to detect people, while the first head can be trained to recognize people, and the second head can be trained to detect a certain number of people.

[0084] Still reference Figure 8 In Option 2, only the first portion of CNN 805, e.g., the first convolutional layer 810 and the first pooling layer 815, is designated as the backbone 835, while the rest of CNN 805 together with DNN 820 is designated as the head 840. Figure 8 An example is shown where only the first layer is included in the backbone, but the number of lower layers of the CNN 805 that may optionally be included in the backbone may vary and may be pre-specified or determined during the training process. Typically, all lower layers that represent the most general features with respect to the task are included in the backbone. Figure 8 This variant of is best suited for more general task support. The backbone can be reused in different object detection tasks. For example, a backbone that detects outlines and edges of people and animals can be used to implement detection and tracking of moving living objects.

[0085] Figure 8 The presented variants of Option 1 and Option 2 in are chosen for illustrative purposes only, as the cut-off point for backbone-head separation can be implemented at any layer of a CNN or DNN. Figure 8In one extreme case depicted in , the backbone contains only the first layer of the CNN. This implementation greatly reduces the computational load on the edge device and offloads most of the computation to the central server. In most cases, it will also produce a larger bitstream. At the other extreme, the head can contain only the last layer of the DNN, and in this case, almost the entire computation is performed at the edge device.

[0086] To determine the right cut-off point, the system architect needs to decide which trade-offs are appropriate for a given use case. The trade-offs to consider are power consumption, computational constraints, bitstream bandwidth, etc. These decisions depend largely on the type of device that plays the role of encoder, decoder, and the communication channel that is available.

[0087] Reference now Fig. 9 , showing an exemplary embodiment of a machine learning module 900 that can perform one or more machine learning processes as described in the present disclosure. The machine learning module can use machine learning processes to perform the determination, classification and / or analysis steps, methods, processes, etc. described in the present disclosure. A "machine learning process" as used in the present disclosure is a process that automatically uses training data 904 to generate an algorithm that will be executed by a computing device / module to produce an output 908 given data provided as input 912; this is in contrast to non-machine learning software programs in which the commands to be executed are predetermined by the user and written in a programming language.

[0088] Still reference Fig. 9As used herein, "training data" is data containing correlations that a machine learning process can use to model relationships between data elements of two or more categories. For example, but not limited to, training data 904 may include multiple data entries, each entry representing a group of data elements that are recorded, received, and / or generated together; data elements may be related by shared presence in a given data entry, by proximity in a given data entry, and the like. Multiple data entries in training data 904 may indicate one or more trends of correlations between categories of data elements; for example, but not limited to, higher values ​​of a first data element belonging to a first category of data elements may tend to be associated with higher values ​​of a second data element belonging to a second category of data elements, indicating a possible ratio or other mathematical relationship that links the values ​​belonging to the two categories. Data elements of multiple categories may be related in training data 904 according to various correlations; correlations may indicate causal and / or predictive connections between categories of data elements, which can be modeled by a machine learning process as relationships such as mathematical relationships, as described in further detail below. Training data 904 may be formatted and / or organized by categories of data elements, such as by associating data elements with one or more descriptors corresponding to the categories of data elements. As a non-limiting example, the training data 904 may include data entered by a person or process in a standardized form, such that the input of a given data element in a given field in the form can be mapped to one or more descriptors of a category. The elements in the training data 904 may be linked to the descriptors of the category by tags, tokens, or other data elements; for example, but not limited to, the training data 904 may be provided in a fixed-length format, a format that links the location of the data to the category (such as a comma-separated value (CSV) format), and / or a self-describing format (such as an extensible markup language (XML), JavaScript object notation (JSON), etc.), so that a process or device can detect the data category.

[0089] Alternatively or additionally, and with continued reference to Fig. 9, the training data 904 may include one or more elements that are not classified; that is, the training data 904 may not be formatted or contain descriptors for some data elements. The machine learning algorithm and / or other process may use, for example, natural language processing algorithms, tokenization, detection of relevant values ​​in the raw data, etc. to sort the training data 904 according to one or more classifications; the categories may be generated using correlation and / or other processing algorithms. As a non-limiting example, in a text corpus, phrases that constitute "n" compound words (such as nouns modified by other nouns) may be identified based on the statistically significant prevalence of n-grams containing such words in a particular order; such n-grams may be classified as language elements, such as "words" that are tracked similar to single words, thereby generating new categories as a result of statistical analysis. Similarly, in a data entry that includes some text data, a person's name may be identified by reference to a list, dictionary, or other compendium of terms, thereby allowing self-organizing classification by a machine learning algorithm, and / or automatically associating the data in the data entry with descriptors or automatically associating the data in the data entry into a given format. The ability to automatically classify data entries can enable the same training data 904 to be applicable to two or more different machine learning algorithms, as described in further detail below. The training data 904 used by the machine learning module 900 can associate any input data described in the present disclosure with any output data described in the present disclosure. As a non-limiting illustrative example

[0090] Further references Fig. 9, one or more supervised and / or unsupervised machine learning processes and / or models as described in further detail below may be used to filter, sort and / or select training data; such models may include, but are not limited to, training data classifiers 916. Training data classifiers 916 may include a "classifier," as used in the present disclosure, a "classifier" is a machine learning model as defined below, such as a mathematical model, a neural network, or a program generated by a machine learning algorithm referred to as a "classification algorithm," which, as described in further detail below, classifies inputs into data categories or bins, outputs data categories or bins and / or labels associated therewith. A classifier may be configured to at least output data that labels or otherwise identifies data sets that are clustered together, found to be close under a distance metric as described below, etc. The machine learning module 900 may generate a classifier using a classification algorithm, which is defined as a process by which a computing device and / or any module and / or component running thereon derives a classifier from the training data 904. Classification can be performed using, but not limited to, the following classifiers: linear classifiers (such as, but not limited to, logistic regression and / or naive Bayes classifiers), nearest neighbor classifiers (such as k-nearest neighbor classifiers), support vector machines, least squares support vector machines, fisher's linear discriminant, quadratic classifiers, decision trees, boosted trees, random forest classifiers, learning vector quantization, and / or neural network-based classifiers.

[0091] Still reference Fig. 9 , the machine learning module 900 can be configured to perform a lazy learning process 920 and / or protocol, which can be alternatively referred to as a "lazy loading" or "call on demand" process and / or protocol, which can be a process of performing machine learning by combining the input and the training set to derive an algorithm to be used to produce the output on demand when receiving the input to be converted to the output. For example, an initial set of simulations can be performed to cover the initial heuristics and / or "first guesses" at the output and / or relationship. As a non-limiting example, the initial heuristic can include a ranking of the associations between the input and the elements of the training data 904. Heuristics can include selecting a certain number of the highest ranking associations and / or training data 904 elements. Lazy learning can implement any suitable lazy learning algorithm, including but not limited to the K nearest neighbor algorithm, the lazy naive Bayes algorithm, etc.; those skilled in the art will be aware of the various lazy learning algorithms that can be applied to generate outputs as described in the present disclosure when reading the entire content of the present disclosure, including but not limited to the lazy learning application of the machine learning algorithm as described in further detail below.

[0092] Alternatively or additionally, and with continued reference to Fig. 9, the machine learning process as described in the present disclosure can be used to generate a machine learning model 924. The "machine learning model" used in the present disclosure is a mathematical and / or algorithmic representation of the relationship between input and output generated using any machine learning process (including but not limited to any process described above) and stored in a memory; the input is submitted to the machine learning model 924 created in the past, and the machine learning model 924 generates an output based on the derived relationship. For example, but not limited to, a linear regression model generated using a linear regression algorithm can use coefficients derived during the machine learning process to calculate linear combinations of input data to calculate output data. As another non-limiting example, the machine learning model 924 can be generated by creating an artificial neural network, such as a convolutional neural network including an input layer of nodes, one or more intermediate layers, and an output layer of nodes. Connections between nodes can be created via a process of "training" the network, in which elements from a set of training data 904 are applied to input nodes, and then a suitable training algorithm (such as Levenberg-Marquardt, conjugate gradient, simulated annealing, or other algorithms) is used to adjust the connections and weights between nodes in adjacent layers of the neural network to produce expected values ​​at output nodes. This process is sometimes referred to as deep learning.

[0093] Still reference Fig. 9 , the machine learning algorithm may include at least a supervised machine learning process 928. As defined herein, the supervised machine learning process 928 includes at least an algorithm that receives a training set that associates multiple inputs with multiple outputs, and seeks to find one or more mathematical relationships that associate the inputs with the outputs, wherein each of the one or more mathematical relationships is optimal according to a specific criterion specified by the algorithm using some scoring functions. For example, a supervised learning algorithm may include inputs and outputs as described above in the present disclosure, and a scoring function that represents the form of the expected relationship to be detected between the inputs and the outputs; the scoring function may, for example, seek to maximize the probability that a given input and / or a combination of element inputs is associated with a given output to minimize the probability that a given input is not associated with a given output. The scoring function may be expressed as a risk function that represents the "expected loss" of the algorithm that associates the inputs with the outputs, wherein the loss is calculated as an error function that represents the degree to which the prediction generated by the relationship is incorrect when compared to a given input-output pair provided in the training data 904. Those skilled in the art will recognize various possible variations of at least a supervised machine learning process 928 that can be used to determine the relationship between inputs and outputs when reading the entire contents of the present disclosure. The supervised machine learning process may include a classification algorithm as defined above.

[0094] Further references Fig. 9, the machine learning process may include at least an unsupervised machine learning process 932. As used herein, an unsupervised machine learning process is a process of deriving inferences in a data set without regard to labels; thus, an unsupervised machine learning process can conveniently discover any structure, relationship, and / or correlation provided in the data. An unsupervised process may not require a response variable; an unsupervised process may be used to find interesting patterns and / or inferences between variables, to determine the degree of correlation between two or more variables, etc.

[0095] Still reference Fig. 9 , the machine learning module 900 can be designed and configured to create a machine learning model 924 using techniques for developing a linear regression model. The linear regression model can include an ordinary least squares regression, which is intended to minimize the square of such a difference according to an appropriate norm (e.g., a vector space distance norm) for measuring the difference between the predicted result and the actual result; the coefficients of the resulting linear equation can be modified to improve minimization. The linear regression model can include a ridge regression method, in which the function to be minimized includes a least squares function plus a term that multiplies the square of each coefficient by a scalar to penalize large coefficients. The linear regression model can include a least absolute shrinkage and selection operator (LASSO) model, in which ridge regression is combined with multiplying the least squares term by a factor of 1 divided by twice the number of samples. The linear regression model can include a multi-task LASSO model, in which the norm applied in the least squares term of the LASSO model is a Frobenius norm equal to the square root of the sum of the squares of all terms. The linear regression model may include an elastic net model, a multi-task elastic net model, a minimum angle regression model, a LARS lasso model, an orthogonal matching pursuit model, a Bayesian regression model, a logistic regression model, a stochastic gradient descent model, a perceptron model, a passive aggressive algorithm, a robust regression model, a Huber regression model, or any other suitable model that a person skilled in the art may think of when reading the entire content of this disclosure. The linear regression model may be generalized to a polynomial regression model in an embodiment, thereby seeking a polynomial equation (e.g., a quadratic, cubic, or higher order equation) that provides the best predicted output / actual output fit; a method similar to the above method may be applied to minimize the error function, which is apparent to a person skilled in the art after reading the entire content of this disclosure.

[0096] Continue to refer Fig. 9, the machine learning algorithm may include, but is not limited to, linear discriminant analysis. The machine learning algorithm may include quadratic discriminant analysis. The machine learning algorithm may include kernel ridge regression. The machine learning algorithm may include support vector machines, including but not limited to support vector regression processes based on classification. The machine learning algorithm may include a stochastic gradient descent algorithm, including a classification and regression algorithm based on stochastic gradient descent. The machine learning algorithm may include a nearest neighbor algorithm. The machine learning algorithm may include various forms of latent space regularization, such as variational regularization. The machine learning algorithm may include a Gaussian process, such as a Gaussian process regression. The machine learning algorithm may include a cross decomposition algorithm, including partial least squares and / or canonical correlation analysis. The machine learning algorithm may include a naive Bayes method. The machine learning algorithm may include a decision tree-based algorithm, such as a decision tree classification or regression algorithm. The machine learning algorithm may include an ensemble method, such as a packing meta-estimator, a random tree forest, an adaptive boost (AdaBoost), a gradient tree boost, and / or a voting classifier method. The machine learning algorithm may include a neural network algorithm, including a convolutional neural network process.

[0097] Reference now Fig.10 , showing an exemplary implementation of a neural network 1000. Neural network 1000, also known as an artificial neural network, is a network of "nodes" or data structures having one or more inputs, one or more outputs, and functions that determine the outputs based on the inputs. Such nodes can be organized in a network, such as, but not limited to, a convolutional neural network, including an input layer of nodes, one or more intermediate layers, and an output layer of nodes. Connections between nodes can be created via a process of "training" the network, in which elements from a training data set are applied to the input nodes, and then a suitable training algorithm (such as Levenberg-Marquardt, conjugate gradient, simulated annealing, or other algorithms) is used to adjust the connections and weights between nodes in adjacent layers of the neural network to produce expected values ​​at the output nodes. This process is sometimes referred to as deep learning. Connections can run only from input nodes to output nodes in a "feedforward" network, or the output of one layer can be fed back to the input of the same or different layer in a "recurrent network."

[0098] Reference now Fig.11 , shows an exemplary implementation of a node of a neural network. The node may include but is not limited to multiple inputs x i , the multiple input x i A node can receive values ​​from the inputs of the neural network containing the node and / or from other nodes. A node can multiply its corresponding input x using i The weight w iTo perform a weighted sum of the inputs. Additionally or alternatively, a bias b can be added to the weighted sum of the inputs so that an offset that is independent of the layer's inputs is added to each unit in the neural network layer. The weighted sum can then be input to the function In this function Can generate one or more outputs y. Applied to input x i The weight w i It can indicate whether an input is "excitatory", indicating that it has a strong influence on one or more outputs y, for example by having a corresponding weight with a large value, and / or "inhibitory", indicating that it has a weak influence on one or more inputs y, for example by having a corresponding weight with a small value. i The value of may be determined by training a neural network using the training data, which may be performed using any suitable process as described above.

[0099] Still reference Fig.11 As used in this disclosure, a "convolutional neural network" is a neural network in which at least one hidden layer is a convolutional layer that convolves the input of the layer with a subset of the input called a "kernel", and one or more additional layers (such as pooling layers, fully connected layers, etc.). CNNs may include, but are not limited to, deep neural network (DNN) extensions, where a DNN is defined as a neural network with two or more hidden layers.

[0100] Generative Adversarial Models

[0101] Generative Adversarial Models (GAMs) are based on the idea of ​​Generative Adversarial Networks (GANs). GANs are a special case of GAMs where the generator and discriminator components are implemented as neural networks (NNs).

[0102] Fig.12 is a simplified example of the GAM framework. The main components of a GAM are a generator 1210 (G) and a discriminator 1220 (D). The generator 1210 learns to generate credible data. The generated instances become negative training examples for the discriminator 1220. The discriminator 1220 learns to distinguish the generator's "fake" sample data 1215 from the real sample data 1218. The discriminator 1220 penalizes the generator 1210 for producing untrustworthy results. The output of the discriminator 1220 is compared with the real input data 1218 to produce two loss functions, the discriminator loss 1230 and the generator loss 1240, which are used to adjust the parameters of the discriminator 1220 and the generator 1210, respectively.

[0103] The discriminator 1220 is a classifier that attempts to distinguish real data 1218 from "fake" data 1215 created by the generator 1210. The discriminator 1220 can use any network architecture suitable for the type of data being classified. The training data for the discriminator typically comes from two sources. Real data instances 1218, such as real pictures 1250 of objects of interest (such as people, cars, buildings, etc.). The discriminator uses these instances as positive examples during training. Fake data instances 1215 are created by the generator 1210 based on random input data 1260. The discriminator 1220 uses these instances of fake data from the generator 1210 as negative examples during training. The training process involving two interleaved stages is further depicted in Figure 13.

[0104] The discriminator is connected to two loss functions 1230 and 1240. Fig.13A , during the discriminator training phase, the discriminator 1220 ignores the generator loss 1240 and only uses the discriminator loss 1230. During the discriminator training, the discriminator 1220 classifies both real data and fake data from the generator 1210. The discriminator loss penalizes the discriminator 1220 for misclassifying real instances as fake or misclassifying fake instances as real. If implemented as a neural network, the discriminator 1220 updates its weights according to the discriminator loss 1230 passing through the discriminator 1220 by back propagation 1310.

[0105] refer to Fig. 13B , the generator 1210 learns to create fake data by incorporating feedback 1320 from the discriminator 1220. Through this feedback, the generator 1210 learns to generate data such that the discriminator 1220 classifies its output as real. Generator training generally requires tighter integration between G and D than required for discriminator training. The portion of the GAM that trains the generator 1210 includes a source of random input data, a generator 1210 that converts the random input into instances of fake data, a discriminator 1220 that classifies the generated data, and a generator loss function 1240 that penalizes the generator for failing to deceive the discriminator.

[0106] In one example, you can use Fig. 13B The generator training phase is performed by the following steps schematically shown in FIG: sampling random noise, generating a generator output from the sampled random noise, obtaining a discriminator classification of the generator output, calculating a loss function from the discriminator classification, and back-propagating 1320 through the discriminator 1220 and the generator 1210 to obtain the gradient. If a neural network is used, the gradient is used to change only the generator weights.

[0107] The entire GAM training is preferably done by alternating discriminator and generator training phases. For example, the discriminator can be trained for one or more epochs, the generator can be trained for one or more epochs, and these phases can be repeated.

[0108] Preferably, the generator 1210 is kept substantially constant during the discriminator 1220 training phase. Since the discriminator is trained to distinguish real data from fake data, it can also learn how to recognize the flaws of the generator. This is a different problem for a thoroughly trained generator than for an untrained generator that produces random output. Similarly, it is preferred to keep the discriminator 1220 substantially constant during the generator 1210 training phase. Otherwise, the generator 1210 may be in a "trying to hit a moving target" environment, which may lead to convergence problems.

[0109] To reach the end of training, a unified loss function shared between the discriminator 1220 and the generator 1210 is adopted, which combines the two losses that need to be minimized.

[0110] This function can have the following forms:

[0111] L DG =E x [logD(x)]+E z [log(1-D(G(z)))],

[0112] in:

[0113] D(x) is the discriminator’s estimate of the probability that a real data instance x is real,

[0114] E x is the expected value of all real data instances,

[0115] G(z) is the output of the generator given the noise z,

[0116] D(G(z)) is the discriminator’s estimate of the probability that a fake instance is real, and

[0117] Ez is the expected value of all random inputs to the generator (actually, the expected value of all generated fake instances G(z)).

[0118] This formula is derived from the cross entropy between the true distribution and the generated distribution. In this case, the discriminator tries to make L DG maximize, while the generator tries to make L DG minimize.

[0119] Compressed Decision GAM

[0120] For the purposes of current video encoding / decoding frameworks, the GAM paradigm can be implemented such that the role of the generator is delegated to a neural network for image / video compression such as High-fidelity Generative Image Compression ("HiFiC") or similar AI-based compression systems such as JPEG AI, while the role of the discriminator is delegated to a decision network that accomplishes tasks such as object detection, object tracking, segmentation, etc. Fig.14 , the compressor (generator) 1410 and the task network (discriminator) 1420 can be trained in a similar manner to the conventional generator and discriminator in GAM, with the changes specified below.

[0121] The components of the frame are Fig.14 . This model is called a compression-decision GAM ("CDGAM") because the roles of the generator 1410 and the discriminator 1420 are now compression and decision, respectively. This framework involves many modifications to the GAM described above. For example, the input to the generator 1410 is now the same input image data 1405 that is passed to the discriminator 1420. Instead of trying to generate a similar representation of the input image from a random signal, the compressor 1410 tries to compress the input image to a minimal compressed representation that produces the same or similar loss as the original image. In this way, the discriminator 1420 receives samples of actual image data 1418 and samples of compressed image data 1415 for comparison.

[0122] Additionally, the training objective includes a compression optimizer, since the goal is to find the smallest compressed representation of the input image that still "looks" the same to the discriminator. To do this, the loss function can be modified as follows:

[0123] L CD =E x [logD(x)]+E x [log(1-D(C(x)))]

[0124] Limited by R(C(x)) <R T

[0125] Here, C(x) is the output of the compression network (generator), R(C(x)) is the bit rate of the compressed representation, and R T is the rate threshold.

[0126] Hybrid System with CDGAM

[0127] The pre-trained components of CDGAM can be distributed as part of a hybrid system for multi-task use cases or image / video encoding for specific tasks. The end user of the system can be a machine or a human, or both. The implementation of the proposed model can be performed using a neural network architecture as described in this paper.

[0128] Fig.15 1 is a simplified block diagram showing an example of a hybrid framework for video encoding and decoding using CDGAM. Pre-trained generator 1515 is implemented on the encoder side. Encoder 1505 includes a preprocessor 1510, which receives input image / video and converts it into a format that can be accepted as input of compression network 1515. Preprocessor 1510 implements algorithms such as resolution scaling, frame rate sampling, color conversion, etc. In some cases, preprocessor 1510 can be configured to implement more complex algorithms, such as edge detection, key point detection, morphological operations, etc. Utilizing more complex algorithms, preprocessor 1510 can only pass the part of the input image / video containing the region of interest. In some cases, such operations can result in lower overall complexity / running time of the encoder and less energy use.

[0129] As described above, the compression network (generator) 1515 receives the preprocessed image / video and transforms it into a compressed representation according to a pre-trained model. Based on the characteristics of the preprocessed image / video, such as spatial or temporal complexity represented by high spatial or temporal frequencies, the current state of the encoder energy supply (such as battery power) and network conditions, the compression network 1515 may implement low-complexity inference, where some parts of the network are changed or removed (pruned). Such changes are then sent as the output of the compression network 1515.

[0130] The image / video encoder 1520 may be provided as an optional component present at the encoder for two purposes - to further compress the compressed images / videos, or to compress the pre-processed images / videos in case the compression network 1515 cannot compress them. This may occur in power saving mode, or in case the output of the compression network 1515 is corrupted.

[0131] With additional compression, the compressed output of the compression network 1515 is passed to an image / video encoder 1520, which implements an efficient video coding standard, such as, but not limited to, Versatile Video coding (VVC). The video encoder 1520 and the compression network 1515 are connected to a feedback loop that allows the parameters of the compression network 1515 to be adjusted to produce an input to the encoder 1520 that provides optimal compression. A joint optimization process that can be implemented is described in the inventor's international patent application PCT / US22 / 40722, entitled "Systems and Methods for Joint Optimization Training and Encoder Side Downsampling," filed on August 18, 2022, the entire contents of which are incorporated herein by reference.

[0132] Still reference Fig.15 , a network update encoder 1525 is coupled to the compression network 1515 and is active when changes occur in the compression network 1515. Any changes to the compression network parameters may be encoded using symbol statistics compression and passed to a multiplexer 1530. The multiplexer 1530 multiplexes the two substreams (i.e., compressed images / video and network updates) into a unified bitstream 1535, which is sent to a decoder 1540 via a channel.

[0133] At the decoder 1540, the bitstream 1535 is received at the demultiplexer 1545, which parses the high-level syntax of the bitstream and splits the received unified bitstream into a network update substream and an image / video substream.

[0134] The network update decoder 1550 receives the network update substream (if present) and parses and decodes it using the inverse operations performed by the network update encoder 1525 in a manner well known in the art.

[0135] An image / video decoder 1555 may be optionally provided and may be active in those cases where the encoder side 1505 encodes image data using the image / video encoder 1520. The image / video decoder (if active) parses and decodes the image / video substream and produces a decoded image / video as input to the final task network 1560.

[0136] The task network (discriminator) 1560 is the second part of the CDGAM, which is pre-trained with the compression network 1515 and is used to process the compressed image / video and produce output suitable for task completion. If the task network 1560 receives non-empty input from the network update decoder 1550 and updates from the compression network (generator) 1515, it updates its own parameters accordingly. In some cases, this requires retraining the CDGAM and can be done offline outside the decoder 1540.

[0137] The machine model 1565 contains the current state of the CDGAM parameters and potential updates, including updates to the pre-processor 1510. It may optionally be stored at the decoder 1540, or it may be located at a third party location from which updates are sent to both the encoder 1505 and the decoder 1540 using available communication channels.

[0138] In applications where humans are the end users, such as entertainment and surveillance, the system may encode images / video using only the image / video encoder 1520, or may alternatively use a CDGAM that is trained in such a way that the compressed images are perceptually as close to the input images as possible, for example using a common objective quality metric (e.g., PSNR, SSIM, etc.) as its loss function.

[0139] Another application of the proposed framework for machine use may require the system to encode only the features, rather than the entire image / video. In this case, the CDGAM can be trained using a generator 1515 that transforms the input image into features, such as an intermediate feature map of a convolutional neural network or any other lower dimensional representation of the input image. The discriminator 1560 takes the features and produces an output decision. The generator 1515 attempts to compress the features into a smaller representation, and the loss function is a standard task-related loss function, such as cross entropy. The encoder 1505 can additionally use an image / video encoder 1520 to further compress the compressed features.

[0140] The encoder 1505 may also implement multiple instances of the compression network 1515, and the decoder 1540 may correspondingly implement multiple instances of the task network 1560 to support multi-task use cases. In this case, the network selection may be orchestrated from the machine model 1565 and implemented by the preprocessor 1510.

[0141] BitTorrent

[0142] The bitstream 1535 produced by the encoder 1505 preferably contains an image / video substream and a network update substream. Fig.16 An exemplary bitstream structure of the present system and method is depicted in .

[0143] The bitstream 1535 preferably includes a stream level header 1605 as well as an image / video substream 1610 and a network update substream 1625. Any of the substreams may be empty. The existence of a substream and the length and type of the current substream are usually described by parameters in the stream level header 1605.

[0144] The image / video substream 1610 includes an image / video stream header 1615 describing elements of the image / video stream and an image / video stream payload 1620. The image / video stream header 1615 typically includes parameters describing the type and length of coding units of the image / video stream payload 1620.

[0145] The network update substream 1625 preferably includes a network update stream header 1630 describing the elements of the network update substream 1625 and a network update substream payload 1635 containing a binary representation of the modifications to the compressed network 1515. The network update stream header 1630 contains parameters describing the type and length of the network update elements.

[0146] If the decoder 1540 does not have a complete description of the pre-trained generator 1515, the encoder 1505 can signal the description to the decoder 1540, such as by using a standard index table entry, and in the case of a neural network, can signal the network description, such as the Open Neural Network Exchange (ONNX), to the decoder 1540. This can save bandwidth and reduce or eliminate the need to send a full network description as part of the bitstream.

[0147] Fig.17 1700 is a system block diagram illustrating an example decoder capable of adaptive cropping. The decoder 1700 may include an entropy decoder processor 1704, an inverse quantization and inverse transform processor 1708, a deblocking filter 1712, a frame buffer 1716, a motion compensation processor 1720, and / or an intra prediction processor 1724.

[0148] in operation, and still refer to Fig.17, the bitstream 1728 may be received by the decoder 1700 and input to the entropy decoder processor 1704, which may entropy decode portions of the bitstream into quantized coefficients. The quantized coefficients may be provided to an inverse quantization and inverse transform processor 1708, which may perform inverse quantization and inverse transform to create a residual signal, which may be added to the output of the motion compensation processor 1720 or the intra-prediction processor 1724, depending on the processing mode. The output of the motion compensation processor 1720 and the intra-prediction processor 1724 may include a block prediction based on a previously decoded block. The sum of the prediction and the residual may be processed by the deblocking filter 1712 and stored in the frame buffer 1716.

[0149] In the implementation, and still referring to Fig.17 , the decoder 1700 may include circuits configured to implement any of the operations described above in any embodiment described above in any order and at any degree of repetition. For example, the decoder 1700 may be configured to repeatedly perform a single step or sequence until the desired or commanded result is achieved; the output of the previous repetition may be used as the input of the subsequent repetition to iteratively and / or recursively perform the repetition of the step or sequence of steps, aggregate the input and / or output of the repetition to produce an aggregated result, reduce or decrement one or more variables (such as global variables), and / or divide a larger processing task into a set of iteratively addressed smaller processing tasks. The decoder may perform any step or sequence of steps as described in the present disclosure in parallel, such as using two or more parallel threads, processor cores, etc. to perform the steps twice or more simultaneously and / or substantially simultaneously; the task division between parallel threads and / or processes may be performed according to any protocol suitable for dividing tasks between iterations. Those skilled in the art will recognize various ways in which steps, sequence of steps, processing tasks and / or data may be subdivided, shared or otherwise processed using iteration, recursion and / or parallel processing when reading the entire contents of the present disclosure.

[0150] Fig.18 is a system block diagram illustrating an example video encoder 1800 capable of adaptive cropping. The example video encoder 1800 may receive an input video 1804, which may be initially partitioned or separated according to a processing scheme such as a tree-structured macroblock partitioning scheme (e.g., a quadtree plus a binary tree). An example of a tree-structured macroblock partitioning scheme may include partitioning a picture frame into large block elements called coding tree units (CTUs). In some embodiments, each CTU may be further partitioned one or more times into multiple sub-blocks called coding units (CUs). The final result of this partitioning may include groups of sub-blocks that may be referred to as prediction units (PUs). Transform units (TUs) may also be utilized.

[0151] Still reference Fig.18 , the example video encoder 1800 may include an intra prediction processor 1808, a motion estimation / compensation processor 1812 (which may also be referred to as an intra prediction processor) capable of constructing a motion vector candidate list (including adding a global motion vector candidate to the motion vector candidate list), a transform / quantization processor 1816, an inverse quantization / inverse transform processor 1820, an in-loop filter 1824, a decoded picture buffer 1828, and / or an entropy encoding processor 1832. The bitstream parameters may be input to the entropy encoding processor 1832 to be included in an output bitstream 1836.

[0152] In operation, and continue to refer to Fig.18 , for each block of a frame of the input video, it may be determined whether to process the block via intra picture prediction or using motion estimation / compensation. The block may be provided to an intra prediction processor 1808 or a motion estimation / compensation processor 1812. If the block is to be processed via intra prediction, the intra prediction processor 1808 may perform processing to output a prediction result (predictor). If the block is to be processed via motion estimation / compensation, the motion estimation / compensation processor 1812 may perform processing including constructing a motion vector candidate list, including adding a global motion vector candidate to the motion vector candidate list (if applicable).

[0153] Further references Fig.18 , a residual can be formed by subtracting the prediction result from the input video. The residual can be received by a transform / quantization processor 1816, which can perform a transform process (e.g., a discrete cosine transform (DCT)) to produce coefficients that can be quantized. The quantized coefficients and any associated signaling information can be provided to an entropy coding processor 1832 for entropy coding and included in an output bitstream 1836. The entropy coding processor 1832 can support the encoding of signaling information related to encoding the current block. In addition, the quantized coefficients can be provided to an inverse quantization / inverse transform processor 1820, which can regenerate pixels, which can be combined with the prediction result and processed by an in-loop filter 1824, the output of which can be stored in a decoded picture buffer 1828 for use by a motion estimation / compensation processor 1812 that can construct a motion vector candidate list (including adding a global motion vector candidate to the motion vector candidate list).

[0154] Continue to refer Fig.18Although some variations have been described in detail above, other modifications or additions are possible. For example, in some embodiments, the current block may include any symmetric block (8x8, 16x16, 32x32, 64x64, 128x128, etc.) and any asymmetric block (8x4, 16x8, etc.).

[0155] In some embodiments, and still with reference to Fig.18 , a quadtree plus binary decision tree (QTBT) can be implemented. In QTBT, at the coding tree unit level, the partitioning parameters of QTBT can be dynamically derived to adapt to local characteristics without sending any overhead. Subsequently, at the coding unit level, the joint classifier decision tree structure can eliminate unnecessary iterations and control the risk of misprediction. In some embodiments, the LTR frame block update mode can be used as an additional option that can be used at each leaf node of the QTBT.

[0156] In some embodiments, and still with reference to Fig.18 , additional syntax elements may be signaled at different hierarchical levels of the bitstream. For example, a flag may be enabled for the entire sequence by including an enable flag encoded in a sequence parameter set (SPS). In addition, a CTU flag may be encoded at the coding tree unit (CTU) level.

[0157] Some embodiments may include a non-transitory computer program product (ie, a physically embodied computer program product) storing instructions that, when executed by one or more data processors of one or more computing systems, cause at least one data processor to perform the operations described herein.

[0158] Still reference Fig.18, the encoder 1800 may include circuits configured to implement any operation as described above in any order and with any degree of repetition. For example, the encoder 1800 may be configured to repeatedly perform a single step or sequence until the desired or commanded result is achieved; the output of the previous repetition may be used as the input of the subsequent repetition to iteratively and / or recursively perform the repetition of the step or sequence of steps, aggregate the input and / or output of the repetition to produce an aggregated result, reduce or decrement one or more variables (such as global variables), and / or divide a larger processing task into a set of iteratively addressed smaller processing tasks. The encoder 1800 may perform any step or sequence of steps as described in the present disclosure in parallel, such as using two or more parallel threads, processor cores, etc. to perform the step twice or more simultaneously and / or substantially simultaneously; the task division between parallel threads and / or processes may be performed according to any protocol suitable for dividing tasks between iterations. Those skilled in the art will recognize various ways in which steps, sequence of steps, processing tasks, and / or data may be subdivided, shared, or otherwise processed using iteration, recursion, and / or parallel processing when reading the entire contents of the present disclosure.

[0159] Continue to refer Fig.18 , a non-transitory computer program product (i.e., a physically embodied computer program product) may store instructions that, when executed by one or more data processors of one or more computing systems, cause at least one data processor to perform the operations and / or steps thereof described in the present disclosure, including but not limited to any of the operations described above and / or any operations that the decoder 900 and / or encoder 1800 may be configured to perform. Similarly, a computer system that may include one or more data processors and a memory coupled to the one or more data processors is also described. The memory may temporarily or permanently store instructions that cause at least one processor to perform one or more operations described herein. In addition, the method may be implemented by one or more data processors within a single computing system or distributed between two or more computing systems. Such computing systems may be connected via one or more connections (including connections through a network (e.g., the Internet, a wireless wide area network, a local area network, a wide area network, a wired network, etc.), via a direct connection between one or more of a plurality of computing systems, etc., and may exchange data and / or commands or other instructions, etc.

[0160] It should be noted that any one or more aspects and embodiments described herein can be conveniently implemented using one or more machines programmed according to the teachings of this specification (e.g., one or more computing devices used as user computing devices for electronic documents, one or more server devices, such as document servers, etc.), which is clear to those of ordinary skill in the computer field. Based on the teachings of this disclosure, a skilled programmer can easily prepare appropriate software coding, which is clear to those of ordinary skill in the software field. The aspects and implementations using software and / or software modules discussed above may also include appropriate hardware for assisting in implementing the machine executable instructions of the software and / or software modules.

[0161] Such software can be a computer program product using a machine-readable storage medium. A machine-readable storage medium can be any medium capable of storing and / or encoding a sequence of instructions executed by a machine (e.g., a computing device) and causing the machine to perform any of the methods and / or embodiments described herein. Examples of machine-readable storage media include, but are not limited to, disks, optical disks (e.g., CDs, CD-Rs, DVDs, DVD-Rs, etc.), magneto-optical disks, read-only memory "ROM" devices, random access memory "RAM" devices, magnetic cards, optical cards, solid-state memory devices, EPROMs, EEPROMs, and any combination thereof. As used herein, machine-readable media is intended to include a single medium and a collection of physically separated media, such as, for example, a collection of optical disks or one or more hard disk drives combined with a computer memory. As used herein, a machine-readable storage medium does not include signal transmission in transient form.

[0162] Such software may also include information (e.g., data) carried as a data signal on a data carrier (such as a carrier wave). For example, machine executable information may be included as a data-bearing signal embodied in a data carrier, where the signal encodes a sequence of instructions or a portion thereof executed by a machine (e.g., a computing device) and any related information (e.g., data structures and data) that causes the machine to perform any of the methods and / or embodiments described herein.

[0163] Examples of computing devices include, but are not limited to, electronic book reading devices, computer workstations, terminal computers, server computers, handheld devices (e.g., tablet computers, smart phones, etc.), network devices, network routers, network switches, bridges, any machine capable of executing a sequence of instructions specifying actions to be taken by the machine, and any combination thereof. In one example, the computing device may include and / or be included in an all-in-one machine.

[0164] Fig.19A graphical representation of one embodiment of a computing device in the exemplary form of a computer system 1900 is shown within which a set of instructions for causing a control system to perform any one or more aspects and / or methods of the present disclosure may be executed. It is also contemplated that a plurality of computing devices may be utilized to implement a specifically configured set of instructions for causing one or more devices to perform any one or more aspects and / or methods of the present disclosure. The computer system 1900 includes a processor 1904 and a memory 1908 that communicate with each other and with other components via a bus 1912. The bus 1912 may include any of several types of bus structures using any of a variety of bus architectures, including, but not limited to, a memory bus, a memory controller, a peripheral bus, a local bus, and any combination thereof.

[0165] Processor 1904 may include any suitable processor, such as but not limited to a processor containing logic circuits for performing arithmetic and logic operations, such as an arithmetic and logic unit (ALU), which may be regulated by a state machine and guided by operational inputs from memory and / or sensors; processor 1904 may be organized according to Von Neumann and / or Harvard architectures as non-limiting examples. Processor 1904 may include, incorporate and / or be incorporated into but not limited to a microcontroller, a microprocessor, a digital signal processor (DSP), a field programmable gate array (FPGA), a complex programmable logic device (CPLD), a graphics processing unit (GPU), a general-purpose GPU, a tensor processing unit (TPU), an analog or mixed signal processor, a trusted platform module (TPM), a floating point unit (FPU) and / or a system on a chip (SoC).

[0166] The memory 1908 may include various components (e.g., machine-readable media), including, but not limited to, random access memory components, read-only components, and any combination thereof. In one example, a basic input / output system 1916 (BIOS), including basic routines such as those that help transfer information between elements within the computer system 1900 during startup, may be stored in the memory 1908. The memory 1908 may also include (e.g., stored on one or more machine-readable media) instructions (e.g., software) 1920 that embody any one or more aspects and / or methods of the present disclosure. In another example, the memory 1908 may also include any number of program modules, including, but not limited to, an operating system, one or more application programs, other program modules, program data, and any combination thereof.

[0167] The computer system 1900 may also include a storage device 1924. Examples of storage devices (e.g., storage device 1924) include, but are not limited to, hard disk drives, disk drives, optical disk drives combined with optical media, solid-state memory devices, and any combination thereof. The storage device 1924 may be connected to the bus 1912 via an appropriate interface (not shown). Example interfaces include, but are not limited to, SCSI, Advanced Technology Attachment (ATA), Serial ATA, Universal Serial Bus (USB), IEEE1394 (FIREWIRE, FireWire), and any combination thereof. In one example, the storage device 1924 (or one or more of its components) may be removably interfaced with the computer system 1900 (e.g., via an external port connector (not shown)). In particular, the storage device 1924 and the associated machine-readable medium 1928 may provide non-volatile and / or volatile storage of machine-readable instructions, data structures, program modules, and / or other data for the computer system 1900. In one example, the software 1920 may reside in whole or in part in the machine-readable medium 1928. In another example, the software 1920 may reside completely or partially within the processor 1904 .

[0168] The computer system 1900 may also include an input device 1932. In one example, a user of the computer system 1900 may input commands and / or other information into the computer system 1900 via the input device 1932. Examples of the input device 1932 include, but are not limited to, an alphanumeric input device (e.g., a keyboard), a pointing device, a joystick, a game controller, an audio input device (e.g., a microphone, a voice response system, etc.), a cursor control device (e.g., a mouse), a touch pad, an optical scanner, a video capture device (e.g., a still camera, a video camera), a touch screen, and any combination thereof. The input device 1932 may be connected to the bus 1912 via any of a variety of interfaces (not shown), including, but not limited to, a serial interface, a parallel interface, a game port, a USB interface, a FireWire interface, a direct interface to the bus 1912, and any combination thereof. The input device 1932 may include a touch screen interface, which may be part of or separate from the display 1936, discussed further below. The input device 1932 may be used as a user selection device for selecting one or more graphical representations in the graphical interface as described above.

[0169] The user may also input commands and / or other information to the computer system 1900 via the storage device 1924 (e.g., a removable disk drive, a flash drive, etc.) and / or the network interface device 1940. A network interface device (such as the network interface device 1940) may be used to connect the computer system 1900 to one or more of various networks (such as the network 1944) and one or more remote devices 1948 connected to one or more of the various networks. Examples of network interface devices include, but are not limited to, network interface cards (e.g., mobile network interface cards, LAN cards), modems, and any combination thereof. Examples of networks include, but are not limited to, wide area networks (e.g., the Internet, enterprise networks), local area networks (e.g., networks associated with offices, buildings, campuses, or other relatively small geographic spaces), telephone networks, data networks associated with telephone / voice providers (e.g., mobile communications provider data and / or voice networks), direct connections between two computing devices, and any combination thereof. Networks such as the network 1944 may employ wired and / or wireless communication modes. In general, any network topology may be used. Information (eg, data, software 1920 , etc.) may be transferred to and / or from computer system 1900 via network interface device 1940 .

[0170] The computer system 1900 may also include a video display adapter 1952 for transmitting images capable of display to a display device, such as a display device 1936. Examples of display devices include, but are not limited to, liquid crystal displays (LCDs), cathode ray tubes (CRTs), plasma displays, light emitting diode (LED) displays, and any combination thereof. Display adapter 1952 and display device 1936 may be used in conjunction with processor 1904 to provide graphical representations of various aspects of the present disclosure. In addition to the display device, the computer system 1900 may include one or more other peripheral output devices, including, but not limited to, audio speakers, printers, and any combination thereof. Such peripheral output devices may be connected to bus 1912 via a peripheral interface 1956. Examples of peripheral interfaces include, but are not limited to, serial ports, USB connections, FireWire connections, parallel connections, and any combination thereof.

[0171] The foregoing is a detailed description of an illustrative embodiment of the present invention. Various modifications and additions may be made without departing from the spirit and scope of the present invention. The features of each of the various embodiments described above may be appropriately combined with the features of the other described embodiments so as to provide a variety of feature combinations in associated new embodiments. In addition, although a plurality of separate embodiments have been described above, what has been described herein is merely an illustration of the application of the principles of the present invention. In addition, although the specific methods herein may be shown and / or described as being performed in a particular order, the order is highly variable within the ordinary technician to implement the methods, systems and software according to the present disclosure. Therefore, this description is intended to be used as an example only, and not to limit the scope of the present invention in other ways.

[0172] Exemplary embodiments have been disclosed above and shown in the accompanying drawings. It will be appreciated by those skilled in the art that various changes, omissions and additions may be made to the specific disclosure herein without departing from the spirit and scope of the present invention.

Claims

1. A video encoder, comprising: a preprocessor, the preprocessor receiving an input signal; a compression network coupled to the preprocessor, the compression network comprising a pre-trained generator in a compression decision generative adversarial model; a network update encoder operatively coupled to the compression network and generating a network update substream; a video encoder coupled to the compression network and generating an encoded image substream; as well as A multiplexer receives the network update substream and the coded image substream and generates a unified bitstream.

2. The video encoder according to claim 1, wherein: The compression network is trained with a discriminator that provides a generator loss function that is back-propagated to the generator.

3. The video encoder according to claim 2, wherein: The compression network is trained to compress an input image into a minimal compressed representation that produces the same or similar loss as the original image at the discriminator.

4. The video encoder according to claim 3, wherein: The compression network is trained using the following loss function: L CD =E x [logD(x)]+E x [log(1-D(C(x)))] Where C(x) is the output of the compression network and D(x) is the output of the discriminator.

5. The video encoder according to claim 1, wherein: The unified bit stream includes: stream-level headers; a video substream having a video substream header and a video substream payload; and A network update sub-stream has a network update sub-stream header and a network update payload.

6. A video decoder comprising: a demultiplexer that receives an input bitstream and parses the bitstream into at least one of a network update substream and a video substream; a network update decoder coupled to the demultiplexer and receiving the network update substream from the demultiplexer; a video decoder coupled to the demultiplexer and receiving the video substream from the demultiplexer; as well as A task network is coupled to the network update decoder and the video decoder and receives the network update substream and the video substream, the task network including a pre-trained discriminator from a compression decision generative adversarial model.

7. The video decoder according to claim 6, wherein: The task network is trained using the image data and a generator that provides compressed image data, during which the task network output is back-propagated to a discriminator loss function of the task network.

8. The video decoder according to claim 7, wherein: The task network is used to train a compression network to compress an input image into a minimum compressed representation that produces the same or similar loss as the original image at the discriminator.

9. The video decoder according to claim 8, wherein: The task network is trained using the following loss function: L CD =E x [logD(x)]+E x [log(1-D(C(x)))] Where C(x) is the output of the compression network and D(x) is the output of the discriminator.

10. The video decoder according to claim 6, wherein: The received bitstream includes a stream level header, a video substream having a video substream header and a video substream payload, and a network update substream having a network update substream header and a network update payload.

Images