Systems and methods for layer reconstruction and encoder-driven feature information extraction in video coding for machines

Abstract

Systems and methods are provided for encoding and decoding video for machine consumption. A video encoder includes a neural network front end which receives image data and generates a plurality of feature layers. A layer analysis module evaluates features in the image data, identifies layers not significant to a machine task at a decoder site, and provides output data signaling unneeded layers. A neural network (NN) based feature reduction module receives the plurality of layers from the NN front end and generates a bottleneck image including the plurality of layers. These bottleneck images are packed and compressed and provided with the data from the layer analysis module in an encoded bitstream.

Description

[0001] SYSTEMS AND METHODS FOR LAYER RECONSTRUCTION AND ENCODER- DRIVEN FEATURE INFORMATION EXTRACTION IN VIDEO CODING FOR MACHINES

[0002] Statement of Related Cases

[0003] The present application claims the benefit of priority to U.S. provisional application serial number 63 / 624,855, filed on January 25, 2024, and entitled "SUBSET OF LAYERS RECONSTRUCTION IN VIDEO CODING FOR MACHINES," and U.S. provisional application serial number 63 / 624,888, filed on January 25, 2024, and entitled ‘"Encoder- Driven Feature Information Extraction and Enhancement for Decoder Efficiency Improvement,’’ the disclosures of which are hereby incorporated by reference in its entirety.

[0004] Field of the Disclosure

[0005] The present application generally relates to the field of video encoding and decoding. In particular, the present invention is directed to systems and methods for reducing bandwidth in a video coding for machines system by selecting removing or modifying selected feature layers from an encoded bitstream

[0006] Background

[0007] As the number and scale of deployed video sensors / devices increases, an increasing amount of video is expected to be processed by machines. A system or solution employing thousands of cameras would produce huge amounts of video that cannot be monitored by humans in a cost-effective manner. Machines or computational systems that ingest and analyze video offer effective solutions to enable decision support systems and analytics engines. Machines designed to perform an analysis task are typically not as sensitive to the quality7and resolution of video as human operators are. Video Coding for Machines (VCM) addresses this opportunity to transform and represent video to minimize the computing, storage, and streaming of video data while ensuring that a machine task is performed with high operational efficiency.

[0008] Feature Coding based Video Coding for Machines (FCVCM) is based on the observation that video analysis based on convolutional neural networks (CNNs) are the most prominent solutions but require significant computational resources. FCVCM methods attempt to compress and transmit features of a video extracted from a CNN and on the receiver side, use the decompressed features to continue processing on a CNN that performs a machine task.

[0009] A typical CNN to perform object detection is shown in Figure 1. A CNN trained using training data such as video, images, audio, LiDAR, thermal imagery, or even text is used to perform a task and provide information about data input to the network. An input image is processed first by the Feature Pyramid Network (FPN) 105 that produces feature maps at layers P2, P3, P4, and P5. The feature maps from layers P2, P3, P4, and P5 are further processed by a Region Proposal Network (RPN) 110 and a Box Head 115, which may include a fast region based convolutional network (Fast R-CNN) ConvFCHead 120 and a second Fast R-CNN for output layers 125 that determine the regions of interest (ROI) and detected labels for the ROIs. In such architectures, FPN 105, RPN 110, and the Box Head 115 represent computationally complex processes.

[0010] System complexity can be reduced, and the amount of data usage can also be reduced by splitting the computations performed in a CNN between a transmi tter / camera and receiver side. In such architectures, a camera or video source end of a system would consist of a first portion, or front end. of the CNN such as the FPN 105. The output of the FPN, feature maps P2. P3. P4. and P5 are then compressed and transmitted to a receiver where the received and decompressed feature maps are input to the second portion of the network comprising components such as RPN 110 and Box Head 115 to complete the execution of the network and produce the network output. Such CNNs may be trained to perform tasks such as object detection, segmentation, action detection, and object tracking.

[0011] Figure 2 shows a ty pi cal approach to processing such a split network. The NN part 1 205 represents the camera / video source where input video or image is process by a part 1 of the neural network to produce a plurality of feature maps, such as four feature maps P2, P3, P4, and P5. The number and structure of the feature maps depend on the neural network. For example, some networks may only have three feature maps rather than four. The size or dimension of each of the feature maps also depends on the neural network. The feature maps are prepared for compression by packing and quantizing the feature maps.

[0012] Fig. 3 further illustrates NN Part 1 305. In this stage, features are extracted from the Neural Network such as a Res-Net in FPN 105. These features can be composed of four layers P2 (310a). P3 (310b), P4 (310c), and P5 (31 Od), each with 256 channels. Each channel in each layer represents a convolution with a kernel and represents input image features. Typically, all channels of a given layer have the same dimensions. For example, as illustrated in Fig. 4, in layer P2, each channel has two dimensions P2cW and P2cH. These two dimensions, representing the width (W) and height (H) of the channel (c), depend on the input image and the neural network. Similarly, the channel dimensions in the other three layers of this example network can be represented with the corresponding channel width and height as P3cW X P3cH. P4cW X P4cH, and P5cW X P5cH.

[0013] In the case of a Feature Pyramid Network 105, each layer is smaller in size than the previous layer due to information loss from the neural networks pooling functionality that reduces the spatial size of the information to a more abstract representation. For example, each layer may be half the size of the previous layer, with p2 being the largest and p5 being the smallest layer.

[0014] Pixels of an image input to the network pass through the network. The extracted feature maps are the transformed input pixels in each channel of each of the feature maps. Features in each of the extracted channels are essentially floating-point values that represent input image features. Compressing or coding feature maps therefore requires compressing all values that represent input image features.

[0015] Referring back to Fig. 2, Packing 210 and quantization 215 are performed to convert the floating-point feature values of the neural network into an image representation, as well as augmenting data for the encoder, such as by using 10-bit quantization. It will be appreciated, however, that other quantization levels may be used.

[0016] For the packing and tiling phase 210, features are converted into image format by mapping channel data to pixels and assembling them by layers to form a full image. For example, consider a P2 layer with 256 channels and each channel with a width and height dimension of P2cW X P2cH. Channel data in such layers can be packed into tiles of size P2cW X P2cH that are laid as a rectangular grid, as illustrated in FIG. 4.

[0017] The feature map data from each of the 256 channels can be tiled into a grid with N x M tiles, each with size P2cW X P2cH. This results in an equivalent P2 layer feature map of width P2W = P2cW x N and height P2H = P2cH x M. The raw floating point feature map data of the P2 layer has a dimension of P2w x P2H. Such two-dimensional feature map data is normalized to a 10-bit representation resulting in a 10-bit image that corresponds to the P2 layer. Similarly, 10-bit layer images of layers P3, P4, and P5 are produced by tiling and normalization. Such normalization is also referred to as uniform 1 O-bit quantization as the floating-point feature map data is mapped to 1024 levels in a 10-bit representation of the data.

[0018] In the example of an R-CNN neural network, the size of each channel in the P3 layer is half that of the channel size in the P2 layer in each dimension. Similarly, the channel dimension in P4 is half that of P3, and the channel dimension in P5 is half that of P4. The amount of layer and channel data in P5, therefore, is significantly less than that of P2.

[0019] P2 layer:

[0020] Width, P2W = P2cW x N and

[0021] Height, P2H = P2cH x M

[0022] M x N = 256;

[0023] P3 layer:

[0024] P3cW = P2cW / 2; P3cH = P2cH / 2;

[0025] Width, P3W = P3cW x 2N and

[0026] Height, P3H = P3cH x M / 2

[0027] P4 layer:

[0028] P4cW = P3cW / 2 = P2cW / 4; P4cH = P3cH / 2 = P2cH / 4;

[0029] Width, P4W = P4cW x 4N and

[0030] Height, P4H = P4cH x M / 4

[0031] P5 layer:

[0032] P5cW = P4cW / 2 = P2cW / 8; P5cH = P4cH / 2 = P2cH / 8;

[0033] Width, P5W = P5cW x 8N and

[0034] Height, P5H = P5cH x M / 8

[0035] Channel data from all the four layers is packed into a single image and sent to the input of an encoder. Typically, the channels are tiled such that the channels of the p2 layer are arranged in a 16x16 grid in a predetermined channel order. The channels of the subsequent layers are tiled to fit the width of the P2 channel arrangement. An example of tiled channel data from P2, P3, P4 and P5 layers is shown in FIG. 5.

[0036] With continued reference to Fig. 2, the tiled and quantized images are compressed using image and video compression 220 such as Versatile Video Coding (VVC) protocol or any other image or video compression methods, such as HEVC, AVI and the like. The compressed data is transmitted to the receiver over communication networks. The compressed channel image data is received and decompressed using an appropriate decompression system 225 to produce a decoded tiled image. If lossy compression is used, the decoded image may have distortion caused by the lossy compression.

[0037] The received image is dequantized (de-normalized) 230 and converted to floating point feature map data which is then unpacked 235 into the corresponding channels of the original network features that can be fed into the second half of the neural network. In these operations, a reversal of the process in steps Packing (Tiling) 210 and Quantization 215, which may take the form of 10-bit uniform quantization, is completed.

[0038] Once the raw layer information is reconstructed from the decompressed data, it can be directly fed into the second part of the neural network (e.g., RPN and Box Head in the R- CNN example) NN Part 2 240 that will output the machine task output (e.g., object detection or segmentation information).

[0039] The machine task performance (e.g., accuracy of object detection or segmentation information) would be impacted by the methods used to compress the feature map data. A lossless compression would faithfully reconstruct feature maps and will not impact machine task performance at the receiver. Any lossy compression (e.g., using video or image compression tools) is likely to impact the machine task performance. Higher degradation in machine task performance is expected for higher compression levels that introduces larger distortion in the reconstructed feature maps.

[0040] Summary of the Disclosure

[0041] [Complete when claims drafted]

[0042] In one embodiment of the present disclosure, a video encoder in a system for video coding for machines is provided. The encoder includes a neural network (NN) front end which receives input image data, such as video content, and generates a plurality of feature layers. A layer analysis module is provided which evaluates features in the image data, identifies feature layers which are not significant to a machine task at a decoder site, and provides output data signaling the unneeded layers. A neural network (NN) based feature reduction module receives the plurality of layers from the NN front end and generates a bottleneck image including the plurality of layers. A quantization module receives the bottleneck image and provides a quantized output. A packing module receives the quantized botleneck images from the quantization module and packs a number of quantized botleneck images into a video frame. A compression module receives the packed video frame from the packing module and generates a compressed video signal to which the output data from the layer analysis module is appended to provide a bitstream to a decoder site.

[0043] In some embodiments of the present disclosure, the output data signaling unneeded layers further comprises a layer ID for each layer and an associated flag indicating if the layer should be reconstructed by a decoder.

[0044] In certain embodiments, the NN front end is a multi-scale feature network and the plurality of feature layers are of different scales that retain different levels of information. Preferably, the NN front end is part of a feature pyramid network which is completed by a second part located at the decoder site.

[0045] The layer analysis module can determine significance of a layer based on a range of criteria. In some embodiments, the layer analysis module determines the significance of a layer based on the size of the object relevant to a machine task.

[0046] Also in accordance with the present disclosure, a decoder in a system for video coding for machines is provided. The decoder includes a decompression module configured to receive a compressed bitstream and reverse a compression process which is applied at an encoder. For example, if VTM-12 compression is applied at the encoder, a complimentary VTM12 decompression is applied at the decoder. An unpacking module is provided which receives the decompressed bitstream including at least one frame having a plurality of quantized botleneck images arranged therein and extracts the plurality of botleneck images. A dequantization module receives the plurality of botleneck images and is configured to substantially reverse a quantization process applied by an encoder to reconstruct a plurality of botleneck images. The neural network (NN) based feature restoration module receiving the reconstructed botleneck images and receiving signaling information in the bitstream from a layer analysis operation in an encoder indicating one or more layers to exclude from processing, the NN-based feature restoration module reconstructs the set of layers embedded in the botleneck images less any layers signaled for exclusion. A layer fill module receives the subset of layers from the NN-based feature restoration module and generates fill layers for any excluded layer. A neural network (NN) back end is provided and receives the reconstructed and filled layers and performs a desired machine task. In some embodiments, the signaling information comprises a layer ID for each layer and an associated flag indicating if the layer should be reconstructed by the NN-based feature restoration module.

[0047] In certain embodiments, the layer fill module inserts null data, such as all zeros, for any excluded layer.

[0048] The NN back end can be formed as a second part of a split feature pyramid network, where the first part of the split feature pyramid network is located at an encoder providing the compressed bitstream.

[0049] In another embodiment, a video encoder in a system for video coding for machines provided which includes a neural network (NN) front end receiving image data, such as video, and generates a plurality of feature layers. A neural network (NN) information inference module evaluates the features in the image data to identify features that may enhance decoder side operations and provides decoder side usage data for inclusion in a bitstream. A quantization module receives the feature layers from the NN front end and provides a quantized output. A packing module receives the quantized feature layers from the quantization module and packing a plurality of feature layers into a video frame. A compression module receives the packed video frame from the packing module and generating a compressed video signal and appends the output of the NN information coding module to provide a bitstream to a decoder site. The decoder side usage data may be appended before or after compression. Compression can take the form of any conventional compression methods known in the art, such as VTM12.

[0050] These and other aspects and features of non-limiting embodiments of the present invention will become apparent to those skilled in the art upon review of the following description of specific non-limiting embodiments of the invention in conjunction with the accompanying drawings.

[0051] Brief Description of the Figures

[0052] For the purpose of illustrating the invention, the drawings show aspects of one or more embodiments of the invention. However, it should be understood that the present invention is not limited to the precise arrangements and instrumentalities shown in the drawings, wherein: FIG. 1 is a block diagram of a generalized R CNN known in the art;

[0053] FIG. 2 is a block diagram of a typical FCVCM system;

[0054] FIG. 3 is a block diagram further illustrating exemplary layers in the NN Part 1 of Fig. 2;

[0055] FIG. 4 is a pictorial representation of tiling of feature maps in accordance with the present disclosure;

[0056] FIG. 5 is a pictorial representation of a tiled image in accordance with the present disclosure;

[0057] FIG. 6 is a block diagram of an example of an FCVCM system in accordance with an embodiment of the present disclosure;

[0058] FIG. 7 is a block diagram of an example of an FCVCM system in accordance with an embodiment of the present disclosure

[0059] FIG. 8 is a block diagram illustrating an example of an architecture in which a neural network (NN) based feature reduction module combines feature map layers into bottleneck images.

[0060] FIG. 9 is a block diagram illustrating an example of an system architecture, complimentary to that in FIG. 8, in which an NN-based feature restoration module reconstructs a full set of feature map layers from received bottleneck images.

[0061] FIG. 10 is a block diagram in accordance with a further embodiment of the present disclosure illustrating an example of a system architecture, complimentary to that in FIG. 8, in which an NN-based feature restoration module reconstructs a partial set of feature map layers from received bottleneck images.

[0062] FIG. 11 is a block diagram of a decoder in accordance with the present disclosure including an architecture with an NN-based feature restoration module which is capable of reconstructing a partial set of feature map layers from received bottleneck images.

[0063] FIG. 12 is a block diagram of an example of an FCVCM system in accordance with an embodiment of the present disclosure;

[0064] FIG. 13 is a block diagram of an example of an FCVCM system in accordance with an embodiment of the present disclosure FIG. 14 is a block diagram illustrating an exemplary embodiment of a video decoder;

[0065] FIG. 15 is a block diagram illustrating an exemplary embodiment of a video encoder; and

[0066] FIG. 16 is a block diagram of a computing system that can be used to implement any one or more of the methodologies disclosed herein and any one or more portions thereof.

[0067] The drawings are not necessarily to scale and may be illustrated by phantom lines, diagrammatic representations and fragmentary views. In certain instances, details that are not necessary for an understanding of the embodiments or that render other details difficult to perceive may have been omitted.

[0068] Exemplary Embodiment of the Present Disclosure

[0069] Referring to FIG. 2, image data is applied to the input of the encoder system and the layers extracted from NN Part 1 205 of the neural network are packed 210, quantized 215, compressed 220 and transmitted to a receiver at a decoder site. As used herein, the term image data is intended to be construed broadly, including video, still images, LiDAR, RADAR, thermal imagery; and other similar machine data that can be processed in a similar manner. At the decoder site, the bitstream is received where the layer data is decoded, reconstructed into a format accepted by NN Part 2 240, and then input to NN Part 2 240 for completing the machine task. This includes decompression 225, dequantization 230 and unpacking 235 processes which are complimentary to those used at the encoder.

[0070] As discussed above, multi-scale feature networks use layers of different scales that retain different levels of information. Generally, there are larger layers that retain more information and smaller layers, depending on the network architectures, generally retain less information but allow the network to process larger more complex objects by focusing on the dominant features of that layer.

[0071] In a typical FCM (Feature Compression for Machine) scheme, such as exemplified in Figs. 1 and 2, all of the extracted layers (e.g., p2, p3, p4, p5) from the feature pyramid network (FPN) will be compressed at the encoder side and are then transmitted to the decoder to reconstruct all of the originally extracted layers to perform a machine task, such as machine vision, using the NN part 2. However, depending on the applicable machine task, all of the FPN layers might not be needed to perform the particular machine task. FPNs are typically designed such that the smallest resolution of the feature map will have a large receptive field to capture the necessary information for large objects and the largest resolution of the feature maps will have a small receptive field to capture the information for small objects. This may allow for the selective use of layers for a certain machine task without a loss of accuracy in the task. For instance, if an image / video only consists of large objects, then using only the p5 layer (smallest resolution of the feature maps), it's possible to perform the end task without losing accuracy. To exploit this characteristic of the FPN, the present disclosure provides systems and methods to signal necessary layer(s) as a parameter in the bitstream so that the decoder need only reconstruct the subset of the original layers set to perform the end machine vision task using the NN part 2. Since these systems and methods do not reconstruct all the layers on the decoder side, the present systems may significantly decrease the complexity of the decoder and the inference time of the NN part 2. An example of an embodiment such a system in accordance with the present disclosure is depicted in Fig. 6.

[0072] Figure 6 is a simplified block diagram of an encoder and decoder in an FCM system in accordance with the present disclosure. Similar to the system of Fig. 2. the FCM system of Fig. 6 includes an encoder including NN Part 1 605, a quantization module 610. packing module 615, and compression module 620, generating a compressed bitstream provided over a channel to a decoder side system. At the decoder side, the system includes decompression module 625, unpacking module 630, dequantization module 635 and NN Part 2 640. each substantially as described in connection with Fig. 2. Referring to Fig. 6, as compared to the FCM system of Fig. 2, the present embodiment further includes a layer truncation module 608 which is interposed between NN Part 1 605 and the quantization module 610. On the decoder side, a layer fill module 638 is interposed between the dequantization module 635 and NN Part 2 640. The layer truncation module 608 will preferably provide an output to the quantization module 610 comprising a subset of the layers output from NN part 1 605 and provides information regarding the truncation, such as signaling layer truncation and the total number of layers for inclusion in the bitstream as well as providing the name of the necessary layer for the end task. This is typically based on a priori knowledge of the machine task at the decoder site and the split neural network architecture in which the NN Part 1 605 at the encoder site and the NN Part 2 at the decoder site are both part of the same neural network. On the decoder side, after reconstructing the subset of the original layers, the layer fill module 638 will fill the truncated laycr(s), such as by using zeros, and the NN part 2 640 will run the inference based on the reconstructed set of layers.

[0073] Another variant of the present embodiments is illustrated in the simplified block diagram of Fig. 7. In contrast to the embodiment of Fig. 6 in which layers are truncated prior to encoding, in Fig. 7 a neural network-based feature reduction and restoration method has been used to reduce the number of feature maps and layers into small bottleneck images for compressing and transmitting to the decoder side and a layer analysis module 707 is used to determine which layers may be excluded at the decoder and append that information to the bitstream. Similar to the system of Fig. 2, the system of Fig. 7 includes an encoder including NN Part 1, 705, Quantization module 710, packing module 715, and compression module 720, generating a compressed bitstream provided over a channel to a decoder side system. At the decoder side, the system includes decompression module 725, unpacking module 730, dequantization module 735 and NN Part 2 740, each substantially as described in connection with Fig. 2.

[0074] Referring to Fig. 7. at the encoder, a layer analysis module 707 is now provided which is coupled to the output of NN Part 1 705. Layer analysis module 707 provides data for the compressed bitstream and has an output coupled to aNN-based feature reduction module 709. Layer analysis module 707 and NN-based Feature Reduction module 709 will generally receive all the layers output from NN Part 1 705 as input and will provide an output reflecting the subset of layers useful for a specific end task. At the decoder side, a NN-based feature restoration module 737 is coupled to the output of dequantization module 735. The feature restoration module 737 receives the information from the bitstream appended by layer analysis module 707. Applying this information only the layers needed for the machine task are provided by the feature restoration module 737 to a layer fill module 739. The layer fill module 739 will preferably reconstruct the unneeded layers, such as by filing those layers with zeros, instead of providing all of the layers to the NN Part 2. For example, if the layer analysis module 707 indicated that only P4 and P5 layers were required for a machine task, the P2 and P3 layers need not be reconstructed at the decoder. Even though the information is present in the bottleneck images for the unneeded layers, omitting these layers from processing simplifies decoding and machine task processing.

[0075] In some applications, it is possible to run only the partial of the NN-based feature restoration process according to the layer parameter signaled in the bitstream.

[0076] Figure 8 illustrates an example of an architecture in which an NN-based feature reduction module 800 combines all the feature map layers, e.g.. P2 840, P3 830, P4 820, P5 840 into bottleneck images (or latent code) 850. Subsequently, these bottleneck images are compressed using different traditional codecs, such as HEVC, VVC, or learned codecs, and then transmitted to the receiver. The bottleneck images are reduced feature maps which have the same dimensions as the smallest layer, which in this example is the P5 layer. The feature reduction module 800 takes in all the feature maps of the various sizes and compresses them into outputs that are of the smallest input size, e.g., the P5 layer. While the bottleneck images have the same dimensions as the P5 layer, they do not contain the same content as the P5 layer.

[0077] The feature reduction module 800 is preferably a trained NN that takes “large” feature maps as inputs and produces reduced size feature maps as output. The size of the output is preferably configured to be same as the smallest input, which is P5 in the present examples. Layer data may be generally reduced to bottleneck data and compressed using a data compression method. In a typical scenario, the compressed data is received by a receiver.

[0078] Referring to Fig. 9. at a decoder site, the reconstructed bottleneck images 950 are used to reconstruct all the feature layers 910, 920, 930, 940 using the NN-based feature restoration module 900. These feature restoration models usually involve computationally heavy operations, such as convolution and up sampling, to reconstruct all the original feature layers from the bottleneck image(s), even though not all the layers may be necessary for the machine vision task.

[0079] Figure 10 and Figure 11 illustrate an example system of the exemplary disclosed embodiments being applied. In the embodiments of Figs 10 and 11, rather than reconstructing all the original layers from the bottleneck image(s), these embodiments only a subset of the necessary’ layers for the machine task are reconstructed. This approach has the potential to substantially reduce decoder complexity and energy consumption. In certain cases, especially when there is no GPU available on the receiver side, this described embodiment could significantly accelerate the end machine tasks, such as machine vision task. This layer reduction is typically performed on the encoder side. For example, when the object size of a particular machine task is such that the P4 and P5 layers are sufficient to represent the desired objects, the P2 and P3 layers are not needed and can be omitted from the encoded bitstream. This process may be determined implicitly by the inference loop on the encoder.

[0080] Exemplary Bitstream Syntax:

[0081] The subset of the layer reconstruction method of Figures 10 and 11 may vary from frame to frame in video. As the content in a video changes, some layer reconstruction information in a fc-VCM bitstream may also change. Such layer reconstruction information may remain unchanged for a group of successive frames in a video. In some applications, layer reconstruction information may not change at all (e.g., a fixed camera monitoring a parking lot to count cars or people). A flexible bitstream syntax in accordance with the present disclosure provides for the ability to signal different possibilities. Layer reconstruction information signaling can be included in the picture header to signal the layers in the picture, in the group of pictures (GOP) header when the layer information is the same for all the frames in a GOP, or in the sequence header or session header when layer reconstruction information remains unchanged for all frames in the video sequence. Other means of signaling layer information are possible.

[0082] An example of a bitstream syntax:

[0083] Iri num layers = Total Number of Layers (3-bit). max number of layers = 8

[0084] Iri layer id = The ID of the layers (3-bit) is_reconstruct = Signal whether the layer should be reconstructed or not on the decoder side (1-bit)

[0085] Information extracted from the neural network on the encoder side of the process, including feature maps, holds valuable data that can enhance decoder processes. However, performing this analysis to extract information on the decoder side of the system can be computationally expensive. Decoders often need to operate efficiently, especially when running on edge devices. Therefore, it is beneficial for the encoder side of the system to shoulder the computational burden of both extracting and potentially drawing conclusions from this neural network information and then transmitting the pertinent data to the decoder through the bitstream. This approach can lead to improvement in system performance by utilizing intermediate feature information without hindering decoder performance.

[0086] Fig. 12 is a block diagram of an encoding and decoding system in accordance with the present disclosure. Similar to Figure 2, Fig. 12 shows an approach to processing a split neural network. On the encoder side, the system includes a neural network front end 1205, a packing and tiling module 1210, quantization module 1215 and compression module 1220. In addition, the neural network part 1 1205 provides an additional output to a neural network information inference module 1250 which is coupled to a NN information coding module 1260 which in turn is coupled to the output of the quantization module 1215 prior to compression of the bitstream.

[0087] The neural network infonnation inference module 1250 extracts intermediate feature information from the neural network and is utilized to draw conclusions that may help encoder or decoder processes. Infonnation Inference may include passing feature information through a region of interest (“RO1”) pooler to extract object size data, feature map analysis, or other methods of inferencing the intennediate features. For example, inference module 1250 receives as input either feature maps (from NN Part 1) or features p2-p5. Using this input, the inference module 1250 either calculates statistics, such as the dynamic range of the features, variance of values (per channel, or per layer), etc. The inference module 1250 can also be implemented using a NN that is trained to detect features that are important for a particular machine task.

[0088] The NN information coding module 1260 is coupled to the information inference module 1250 and is employed to compress the output of the inference module for the bitstream. This can be performed, for example, using entropy coding. The compressed output from the NN information coding module 1260 may be provided to an adder at the input of compression block 1220 or an adder at the output of compression block 1220 such that the data from NN information coding module 1260 is appended to the bitstream to be provided over a channel to a decoder site.

[0089] Still referring to Fig. 12, at the decoder side, the system includes decompression module 1225, dequantization 1230, unpacking module 1235 and NN Part 2 1240, which are described above in connection with Fig. 2. The decoder side system further includes an extraction method module 1270 and a decoder side usage module 1280.

[0090] The extraction method module 1270 receives the decompressed bitstream from decompression module 1225: The extraction method module 1280 parses the bitstream to extract the relevant encoder side inference data passed from the encoder.

[0091] Decoder side usage module 1280 receives the inference data from the extraction method module 1270. The application of the encoder side inference data may be used to increase efficiency or performance of decoder side processes or the machine task. For example, the inference data may indicate that one or more features or feature layers are not relevant to the machine process and can be deemphasized or eliminated in subsequent processing. Decoder side usage module 1270 decodes the information that was coded by NN coding module 1260, which is in turn the output of inference module 1250. The usage module 1280 takes this infonnation and applies pre- or post-processing to the decompressed features. For example, reconstruction refinement may be applied, based on the range of values in the input. In another example, the usage module 1280 can instruct the decoder to extract only certain layers, or only certain channels within layers.

[0092] Fig. 13 is a block diagram of an additional embodiment of an encoding and decoding system in accordance with the present disclosure. On the encoder side, as described in connection with Fig. 2, the system includes a Neural Network front end 1305, a packing and tiling module 1310, a quantization module 1315 and a compression module 1320. This embodiment further includes an extraction module 1350 coupled to the output of the NN part 1 1305 and an additional packing module 1360 the output of which is summed with output of the quantization module 1315 prior to compression 1320. In the extraction module 1350, information extracted from the bitstream preferably includes input object sizes, network-layer layout, nullified layers, e.g., layers derived post inference, potentially problematic input image characteristics, and the like.

[0093] In processing block 1380, encoder side parameters from the bitstream are applied to the decoding process to improve both decoding and system performance. This may result in increased decoder efficiency as well as the ability to avoid processing input images that are corrupted, of low quality7, or are otherwise likely to present problems with decoding.

[0094] Object sizes are used to decide which layers are sufficient for successful detection. Network Layer quantity (number) is used to instruct the decoder which layers it needs to use in the NN part 2 to be able take as input decompressed features. Input image characteristics, such as color space, resolution and number of small object in the image can also be used to adjust the parameters and hyperparameters of the NN Part 2. For example, a separate NN Part 2 can be used to process features that are coming from the grayscale image, as compared to the full color image. A corrupt image flag can be used to signal to the decoder that the features which are produced from such an image could be discarded, or at least considered with a lower priority when performing detection. Corruption of the image can be detected using a pre-processor on the encoder side that detects blur, point noise, missing blocks in the image or similar image irregularities.

[0095] Bitstream Addition Example fli num layers: Total number of network layers fli_layer_id: Identification for layer, 3-bit value, up to 8 layers can be signaled, fli layer truncated: Bool if layer has been truncated by prior processing. CorruptFlag: Bool to signal if prior processing detected corrupt input image. ColorSpaceFlag: Bool to signal if prior processing detected distorted coloring. NoiseFlag: Bool to signal if prior processing detected high noise. BlurFlag: Bool to signal if prior processing detected high blur. ResolutionFlag: Bool to signal if prior processing detected low resolution, num small objects: Number of small objects in input image. num_medium_objects: Number of medium objects in input image. num_large_objects: Number of large objects in input image. FIG. 14 is a system block diagram illustrating an example of a decoder 1400 capable of practicing the present methods. Decoder 1400 may include an entropy decoder processor 1404, an inverse quantization and inverse transformation processor 1408, a deblocking filter 1412, a frame buffer 1416, a motion compensation processor 1420 and / or an intra prediction processor 1424.

[0096] In operation, and still referring to FIG. 14, bit stream 1428 may be received by decoder 1400 and input to entropy decoder processor 1404. which may entropy decode portions of bit stream into quantized coefficients. Quantized coefficients may be provided to inverse quantization and inverse transformation processor 1408, which may perform inverse quantization and inverse transformation to create a residual signal, which may be added to an output of motion compensation processor 1420 or intra prediction processor 1424 according to a processing mode. An output of the motion compensation processor 1420 and intra prediction processor 1424 may include a block prediction based on a previously decoded block. A sum of prediction and residual may be processed by deblocking filter 1412 and stored in a frame buffer 1416.

[0097] In an embodiment, and still referring to FIG. 14. decoder 1400 may include circuitry configured to implement any operations as described above in any embodiment as described above, in any order and with any degree of repetition. For instance, decoder 1400 may be configured to perform a single step or sequence repeatedly until a desired or commanded outcome is achieved; repetition of a step or a sequence of steps may be performed iteratively and / or recursively using outputs of previous repetitions as inputs to subsequent repetitions, aggregating inputs and / or outputs of repetitions to produce an aggregate result, reduction or decrement of one or more variables such as global variables, and / or division of a larger processing task into a set of iteratively addressed smaller processing tasks. Decoder may perform any step or sequence of steps as described in this disclosure in parallel, such as simultaneously and / or substantially simultaneously performing a step two or more times using two or more parallel threads, processor cores, or the like; division of tasks between parallel threads and / or processes may be performed according to any protocol suitable for division of tasks between iterations. Persons skilled in the art, upon reviewing the entirety of this disclosure, will be aware of various ways in which steps, sequences of steps, processing tasks, and / or data may be subdivided, shared, or otherwise dealt with using iteration, recursion, and / or parallel processing.

[0098] FIG. 15 is a system block diagram illustrating an example video encoder 1500. Example video encoder 1500 may receive an input video 1504, which may be initially segmented or dividing according to a processing scheme, such as a tree-structured macro block partitioning scheme (e.g., quad-tree plus binary tree). An example of a tree-structured macro block partitioning scheme may include partitioning a picture frame into large block elements called coding tree units (CTU). In some implementations, each CTU may be further partitioned one or more times into a number of sub-blocks called coding units (CU). A final result of this portioning may include a group of sub-blocks that may be called predictive units (PU). Transform units (TU) may also be utilized.

[0099] Still referring to FIG. 15. example video encoder 1500 may include an intra prediction processor 1508, a motion estimation / compensation processor 1512, which may also be referred to as an inter prediction processor, capable of constructing a motion vector candidate list including adding a global motion vector candidate to the motion vector candidate list, a transform / quantization processor 1516, an inverse quantization / inverse transform processor 1520, an in-loop filter 1524, a decoded picture buffer 1528. and / or an entropy coding processor 1532. Bit stream parameters may be input to the entropy coding processor 1532 for inclusion in the output bit stream 1536.

[0100] In operation, and with continued reference to FIG. 15, for each block of a frame of input video, whether to process block via intra picture prediction or using motion estimation / compensation may be determined. Block may be provided to intra prediction processor 1508 or motion estimation / compensation processor 1512. If block is to be processed via intra prediction, intra prediction processor 1508 may perform processing to output a predictor. If block is to be processed via motion estimation / compensation, motion estimation / compensation processor 1512 may perform processing including constructing a motion vector candidate list including adding a global motion vector candidate to the motion vector candidate list, if applicable.

[0101] Further referring to FIG. 15, a residual may be formed by subtracting a predictor from input video. Residual may be received by transform / quantization processor 1516, which may perform transformation processing (e.g., discrete cosine transform (DCT)) to produce coefficients, which may be quantized. Quantized coefficients and any associated signaling information may be provided to entropy coding processor 1532 for entropy encoding and inclusion in output bit stream 1536. Entropy encoding processor 1532 may support encoding of signaling information related to encoding a current block. In addition, quantized coefficients may be provided to inverse quantization / inverse transformation processor 1520, which may reproduce pixels, which may be combined with a predictor and processed by in loop filter 1524, an output of which may be stored in decoded picture buffer 1528 for use by motion estimation / compensation processor 1512 that is capable of constructing a motion vector candidate list including adding a global motion vector candidate to the motion vector candidate list.

[0102] With continued reference to FIG. 15, although a few' variations have been described in detail above, other modifications or additions are possible. For example, in some implementations, current blocks may include any symmetric blocks (8x8, 16x16. 32x32. 64x64, 128 x 128, and the like) as w ell as any asymmetric block (8x4, 16x8, and the like).

[0103] In some implementations, and still referring to FIG. 15, a quadtree plus binary decision tree (QTBT) may be implemented. In QTBT, at a Coding Tree Unit level, partition parameters of QTBT may be dynamically derived to adapt to local characteristics without transmitting any overhead. Subsequently, at a Coding Unit level, a joint-classifier decision tree structure may eliminate unnecessary iterations and control the risk of false prediction. In some implementations, LTR frame block update mode may be available as an additional option available at every leaf node of QTBT.

[0104] In some implementations, and still referring to FIG. 15, additional syntax elements may be signaled at different hierarchy levels of bitstream. For example, a flag may be enabled for an entire sequence by including an enable flag coded in a Sequence Parameter Set (SPS). Further, a CTU flag may be coded at a coding tree unit (CTU) level.

[0105] Some embodiments may include non-transitory computer program products (i.e., physically embodied computer program products) that store instructions, which when executed by one or more data processors of one or more computing systems, cause at least one data processor to perform operations herein.

[0106] Still referring to FIG. 15, encoder 1500 may include circuitry configured to implement any operations as described above in any embodiment, in any order and with any degree of repetition. For instance, encoder 1500 may be configured to perform a single step or sequence repeatedly until a desired or commanded outcome is achieved; repetition of a step or a sequence of steps may be performed iteratively and / or recursively using outputs of previous repetitions as inputs to subsequent repetitions, aggregating inputs and / or outputs of repetitions to produce an aggregate result, reduction or decrement of one or more variables such as global variables, and / or division of a larger processing task into a set of iteratively addressed smaller processing tasks. Encoder 1500 may perform any step or sequence of steps as described in this disclosure in parallel, such as simultaneously and / or substantially simultaneously performing a step two or more times using two or more parallel threads, processor cores, or the like; division of tasks between parallel threads and / or processes may be performed according to any protocol suitable for division of tasks between iterations. Persons skilled in the art, upon reviewing the entirety of this disclosure, will be aware of various ways in which steps, sequences of steps, processing tasks, and / or data may be subdivided, shared, or otherwise dealt with using iteration, recursion, and / or parallel processing.

[0107] With continued reference to FIG. 15, non-transitory computer program products (i.e., phy sically embodied computer program products) may store instructions, w hich when executed by one or more data processors of one or more computing systems, causes at least one data processor to perform operations, and / or steps thereof described in this disclosure, including without limitation any operations described above and / or any operations decoder 1200 and / or encoder 1500 may be configured to perform. Similarly, computer systems are also described that may include one or more data processors and memory coupled to the one or more data processors. The memory may temporarily or permanently store instructions that cause at least one processor to perform one or more of the operations described herein. In addition, methods can be implemented by one or more data processors either within a single computing system or distributed among two or more computing systems. Such computing systems can be connected and can exchange data and / or commands or other instructions or the like via one or more connections, including a connection over a network (e g. the Internet, a wireless wide area network, a local area network, a wide area network, a wired network, or the like), via a direct connection between one or more of the multiple computing systems, or the like.

[0108] It is to be noted that any one or more of the aspects and embodiments described herein may be conveniently implemented using one or more machines (e g., one or more computing devices that are utilized as a user computing device for an electronic document, one or more sen- er devices, such as a document server, etc.) programmed according to the teachings of the present specification, as will be apparent to those of ordinary skill in the computer art. Appropriate software coding can readily be prepared by skilled programmers based on the teachings of the present disclosure, as will be apparent to those of ordinary skill in the software art. Aspects and implementations discussed above employing software and / or software modules may also include appropriate hardware for assisting in the implementation of the machine executable instructions of the software and / or software module.

[0109] Such software may be a computer program product that employs a machine-readable storage medium. A machine-readable storage medium may be any medium that is capable of storing and / or encoding a sequence of instructions for execution by a machine (e.g., a computing device) and that causes the machine to perform any one of the methodologies and / or embodiments described herein. Examples of a machine-readable storage medium include, but are not limited to. a magnetic disk, an optical disc (e.g., CD, CD-R, DVD. DVD- R, etc.), a magneto-optical disk, a read-only memory “ROM” device, a random-access memory' “RAM” device, a magnetic card, an optical card, a solid-state memory device, an EPROM, an EEPROM, and any combinations thereof. A machine-readable medium, as used herein, is intended to include a single medium as well as a collection of physically separate media, such as, for example, a collection of compact discs or one or more hard disk drives in combination with a computer memory'. As used herein, a machine-readable storage medium does not include transitory forms of signal transmission.

[0110] Such software may also include information (e.g., data) carried as a data signal on a data carrier, such as a carrier wave. For example, machine-executable information may be included as a data-carrying signal embodied in a data carrier in which the signal encodes a sequence of instruction, or portion thereof, for execution by a machine (e.g., a computing device) and any related information (e.g., data structures and data) that causes the machine to perform any one of the methodologies and / or embodiments described herein.

[0111] Examples of a computing device include, but are not limited to. an electronic book reading device, a computer workstation, a terminal computer, a server computer, a handheld device (e.g., a tablet computer, a smartphone, etc.), a web appliance, a network router, a network switch, a network bridge, any machine capable of executing a sequence of instructions that specify an action to be taken by that machine, and any combinations thereof. In one example, a computing device may include and / or be included in a kiosk. FIG. 16 shows a diagrammatic representation of one embodiment of a computing device in the exemplary form of a computer system 1600 within which a set of instructions for causing a control system to perform any one or more of the aspects and / or methodologies of the present disclosure may be executed. It is also contemplated that multiple computing devices may be utilized to implement a specially configured set of instructions for causing one or more of the devices to perform any one or more of the aspects and / or methodologies of the present disclosure. Computer system 1600 includes a processor 1604 and a memory 1608 that communicate with each other, and with other components, via a bus 1612. Bus 1612 may include any of several types of bus structures including, but not limited to, a memory bus, a memory controller, a peripheral bus, a local bus, and any combinations thereof, using any of a variety of bus architectures.

[0112] Processor 1604 may include any suitable processor, such as without limitation a processor incorporating logical circuitry for performing arithmetic and logical operations, such as an arithmetic and logic unit (ALU), which may be regulated with a state machine and directed by operational inputs from memory and / or sensors; processor 1604 may be organized according to Von Neumann and / or Harvard architecture as a non-limiting example. Processor 1604 may include, incorporate, and / or be incorporated in, without limitation, a microcontroller, microprocessor, digital signal processor (DSP), Field Programmable Gate Array (FPGA), Complex Programmable Logic Device (CPLD), Graphical Processing Unit (GPU), general purpose GPU, Tensor Processing Unit (TPU). analog or mixed signal processor, Trusted Platform Module (TPM), a floating-point unit (FPU), and / or system on a chip (SoC)

[0113] Memory 1608 may include various components (e.g., machine-readable media) including, but not limited to, a random-access memory component, a read only component, and any combinations thereof. In one example, a basic input / output system 1616 (BIOS), including basic routines that help to transfer information between elements within computer system 1600, such as during start-up, may be stored in memory 1608. Memory 1608 may also include (e.g., stored on one or more machine-readable media) instructions (e.g., software) 1620 embodying any one or more of the aspects and / or methodologies of the present disclosure. In another example, memory 1608 may further include any number of program modules including, but not limited to, an operating system, one or more application programs, other program modules, program data, and any combinations thereof. Computer system 1600 may also include a storage device 1624. Examples of a storage device (e.g., storage device 1624) include, but are not limited to, a hard disk drive, a magnetic disk drive, an optical disc drive in combination with an optical medium, a solid-state memory device, and any combinations thereof. Storage device 1624 may be connected to bus 1612 by an appropriate interface (not shown). Example interfaces include, but are not limited to, SCSI, advanced technology attachment (ATA), serial ATA, universal serial bus (USB), IEEE 1394 (FIREWIRE), and any combinations thereof. In one example, storage device 1624 (or one or more components thereof) may be removably interfaced with computer system 1600 (e.g., via an external port connector (not shown)). Particularly, storage device 1624 and an associated machine-readable medium 1628 may provide nonvolatile and / or volatile storage of machine-readable instructions, data structures, program modules, and / or other data for computer system 1600. In one example, software 1620 may reside, completely or partially, within machine-readable medium 1628. In another example, software 1620 may reside, completely or partially, within processor 1604.

[0114] Computer system 1600 may also include an input device 1632. In one example, a user of computer system 1600 may enter commands and / or other information into computer system 1600 via input device 1632. Examples of an input device 1632 include, but are not limited to, an alpha-numeric input device (e.g., a keyboard), a pointing device, ajoystick, a gamepad, an audio input device (e.g., a microphone, a voice response system, etc.), a cursor control device (e.g., a mouse), a touchpad, an optical scanner, a video capture device (e.g., a still camera, a video camera), a touchscreen, and any combinations thereof. Input device 1632 may be interfaced to bus 1612 via any of a variety of interfaces (not shown) including, but not limited to, a serial interface, a parallel interface, a game port, a USB interface, a FIREWIRE interface, a direct interface to bus 1612, and any combinations thereof. Input device 1632 may include a touch screen interface that may be a part of or separate from display 1636, discussed further below. Input device 1632 may be utilized as a user selection device for selecting one or more graphical representations in a graphical interface as described above.

[0115] A user may also input commands and / or other information to computer system 1600 via storage device 1624 (e.g., a removable disk drive, a flash drive, etc.) and / or network interface device 1640. A network interface device, such as netw ork interface device 1640, may be utilized for connecting computer system 1600 to one or more of a variety of networks, such as netw ork 1644, and one or more remote devices 1648 connected thereto. Examples of a network interface device include, but are not limited to, a network interface card (e.g., a mobile network interface card, a LAN card), a modem, and any combination thereof. Examples of a network include, but are not limited to, a wide area network (e.g., the Internet, an enterprise network), a local area network (e.g., a network associated with an office, a building, a campus or other relatively small geographic space), a telephone network, a data network associated with a telephone / voice provider (e.g.. a mobile communications provider data and / or voice network), a direct connection between two computing devices, and any combinations thereof. A network, such as network 1644, may employ a wired and / or a wireless mode of communication. In general, any network topology may be used. Information (e.g., data, softw are 1620, etc.) may be communicated to and / or from computer system 1600 via network interface device 1640.

[0116] Computer system 1600 may further include a video display adapter 1652 for communicating a displayable image to a display device, such as display device 1636. Examples of a display device include, but are not limited to, a liquid cr stal display (LCD), a cathode ray tube (CRT), a plasma display, a light emitting diode (LED) display, and any combinations thereof. Display adapter 1652 and display device 1636 may be utilized in combination with processor 1604 to provide graphical representations of aspects of the present disclosure. In addition to a display device, computer system 1600 may include one or more other peripheral output devices including, but not limited to, an audio speaker, a printer, and any combinations thereof. Such peripheral output devices may be connected to bus 1612 via a peripheral interface 1656. Examples of a peripheral interface include, but are not limited to, a serial port, a USB connection, a FIREWIRE connection, a parallel connection, and any combinations thereof.

[0117] The foregoing has been a detailed description of illustrative embodiments of the invention. Various modifications and additions can be made without departing from the spirit and scope of this invention. Features of each of the various embodiments described above may be combined with features of other described embodiments as appropriate in order to provide a multiplicity7of feature combinations in associated new7embodiments. Furthermore, while the foregoing describes a number of separate embodiments, what has been described herein is merely illustrative of the application of the principles of the present invention. Additionally, although particular methods herein may be illustrated and / or described as being performed in a specific order, the ordering is highly variable within ordinary skill to achieve methods, systems, and software according to the present disclosure. Accordingly, this description is meant to be taken only by way of example, and not to otherwise limit the scope of this invention.

[0118] Exemplary embodiments have been disclosed above and illustrated in the accompanying drawings. It will be understood by those skilled in the art that various changes, omissions, and additions may be made to that which is specifically disclosed herein without departing from the spirit and scope of the present invention.

Claims

What is Claimed:

1. A video encoder in a system for video coding for machines, comprising: a neural network (NN) front end, the NN front end receiving image data and generating a plurality of feature layers; layer analysis module evaluating features in the image data, identifying layers not significant to a machine task at a decoder site, and providing output data signaling unneeded layers; a neural network (NN) based feature reduction module receiving the plurality of layers from the NN front end generating a bottleneck image including the plurality of layers; a quantization module receiving the bottleneck image and providing a quantized output; a packing module receiving the quantized bottleneck images from the quantization module and packing a plurality of quantized bottleneck images into a video frame; a compression module, the compression module receiving the packed video frame from the packing module and generating a compressed video signal and appending the output of the layer analysis module to provide a bitstream to a decoder site.

2. The encoder of claim 1, wherein the providing output data signaling unneeded layers further comprises a layer ID for each layer and an associated flag indicating if the layer should be reconstructed by a decoder.

3. The encoder of claim 1, wherein the NN front end is a multi-scale feature network and the plurality of feature layers are of different scales that retain different levels of information.

4. The encoder of claim 3, wherein the NN front end is part of a feature pyramid netw ork which is completed by a second part located at the decoder site.

5. The encoder of claim 1, wherein the layer analysis module determines significance of a layer based on the size of the object relevant to a machine task.

6. The encoder of claim 1, wherein the NN front end generates four feature layers.

7. A decoder in a system for video coding for machines, comprising: a decompression module configured to receive a compressed bitstream and reverse a compression process applied at an encoder; an unpacking module, the unpacking module receiving the decompressed bitstream including at least one frame having a plurality of quantized bottleneck images arranged therein, the unpacking module extracting the plurality of bottleneck images; a dequantization module receiving the plurality of bottleneck images and configured to substantially reverse a quantization process applied by an encoder to reconstruct a plurality of bottleneck images; a neural network (NN) based feature restoration module receiving the reconstructed bottleneck images and receiving signaling information in the bitstream from a layer analysis operation in an encoder indicating one or more layers to exclude from processing, the NN-based feature restoration module reconstructing the set of layers embedded in the bottleneck images less any layers signaled for exclusion; a layer fill module receiving the subset of layers from the NN-based feature restoration module and generating fill layers for any excluded layer; a neural network (NN) back end, the neural network back end receiving the reconstructed and filled layers and performing a machine task.

8. The decoder of claim 7 wherein the signaling information comprises a layer ID for each layer and an associated flag indicating if the layer should be reconstructed by the NN-based feature restoration module.

9. The decoder of claim 8, wherein the layer fill module inserts null data for any- excluded layer.

10. The decoder of claim 7, wherein the NN back end is a second part of a split feature pyramid network, where the first part of the split feature pyramid network is located at an encoder providing the compressed bitstream.

11. A video encoder in a system for video coding for machines, comprising: a neural network (NN) front end, the NN front end receiving image data and generating a plurality7of feature layers; a neural network (NN) information inference module evaluating features in the image data to enhance decoder side operations and providing decoder side usage data for inclusion in a bitstream; a quantization module receiving the feature layers from the NN front end and providing a quantized output; a packing module receiving the quantized feature layers from the quantization module and packing a plurality- of feature layers into a video frame; a compression module, the compression module receiving the packed video frame from the packing module and generating a compressed video signal and appending the output of NN information coding module to provide a bitstream to a decoder site.

12. A decoder in a system for video coding for machines, comprising: a decompression module configured to receive a compressed bitstream and reverse a compression process applied at an encoder;an unpacking module, the unpacking module receiving the decompressed bitstream including at least one frame having a plurality of quantized bottleneck images arranged therein; a dequantization module receiving the decompressed image data and configured to substantially reverse a quantization process applied by an encoder; an extraction module coupled to the decompression module and extracting decoder side usage data encoded in the bitstream by an encoder; a neural network (NN) back end. the neural network back end receiving at least a portion of the reconstructed image data and decoder side usage data and processing a subset of the image data based on the decoder side usage data to perform a machine task.

Images