Method, apparatus, and system for encoding and decoding tensors

Abstract

A system and method for encoding at least a plurality of tensors forming a hierarchical representation for a single frame into a bitstream, the method including deriving a first unit of information from a plurality of tensors forming the hierarchical representation, the plurality of tensors including at least a first tensor and a second tensor, wherein a feature map of the first tensor has greater spatial resolution than a feature map of the second tensor, and encoding at least the first unit of information into the bitstream in a first mode. In a second mode, the method also includes deriving a second unit of information derived from the first tensor and encoding the second unit of information and the first unit of information into the bitstream.

Description

[Technical Field]

[0001] Reference to related applications This application claims the benefit under 35 U.S.C. § 119 of the filing date of Australian Patent Application No. 2022252784, filed October 13, 2022, the entire contents of which are incorporated herein by reference as if fully set forth herein.

[0002] The present invention relates generally to digital video signal processing, and more particularly to a method, apparatus, and system for encoding and decoding tensors from convolutional neural networks. The present invention also relates to a computer program product including a computer-readable medium having recorded thereon a computer program for encoding and decoding tensors from convolutional neural networks using video compression techniques. [Background technology]

[0003] Convolutional neural networks (CNNs) are an emerging technology that addresses use cases involving machine vision, such as object detection, instance segmentation, object tracking, human pose estimation, and action recognition, among others. Applications for CNNs can involve the use of "edge devices" with sensors and some processing capabilities, coupled to application servers as part of a "cloud." CNNs can require relatively high computational complexity than can typically be afforded by edge devices, either in terms of computing power or power consumption. Running CNNs in a distributed manner has emerged as one solution for running cutting-edge networks using edge devices with limited capabilities. In other words, distributed processing allows legacy edge devices to still provide the capabilities of cutting-edge CNNs by distributing processing between the edge devices and external processing means, such as cloud servers. Such distributed network architectures are sometimes called "collaborative intelligence" and offer benefits such as reusing partial results from a first part of a network with several different second parts, perhaps each optimized for a different task. Collaborative intelligence architectures introduce the need for efficient compression of tensor data for transmission over networks such as WANs.

[0004] CNNs typically contain many layers, such as convolutional and fully connected layers, and data is passed from one layer to the next in the form of "tensors." Partitioning a network across different devices creates the need to compress the intermediate tensor data passing from one layer to the next within a CNN. Such compression can be called "feature compression," as the intermediate tensor data are often called "features" or "feature maps" and represent partially processed forms of input, such as image frames or video frames. The International Organization for Standardization / International Electrotechnical Commission Joint Technical Committee 1 / Subcommittee 29 / Working Group 2-8 (ISO / IEC JTC1 / SC29 / WG2-8), also known as the "Moving Picture Experts Group" (MPEG), is tasked with researching compression techniques in various contexts, often related to video. WG2's "MPEG Technical Requirements" established the "Video Compression for Machines" (VCM) ad hoc group, tasked with researching compression for machine consumption and feature compression. The feature compression mandate is in an exploratory phase with a 'Call for Evidence' (CfE) issued soliciting techniques that can significantly outperform feature compression results achieved using state-of-the-art standardized techniques.

[0005] CNNs typically require that the weights for each layer be predetermined during the training phase, during which a very large amount of training data is passed through the CNN and the results determined by the network being trained are compared to the ground truth associated with the training data. The discrepancy between the obtained result and the desired result is expressed as a "loss" and measured using a "loss function." Using the determined loss, a process for updating the network weights, such as stochastic gradient descent (SGD), is performed. Network weight updates typically involve backpropagation of a "gradient" indicating the delta to be applied to the network weights, starting at the network's output layer and ending at the network's input layer, covering all intermediate, or "hidden," layers of the network. The speed of weight updates is scaled by a "learning rate" hyperparameter, which is typically set to facilitate the training process in finding a global minimum in terms of loss (i.e., the highest possible task performance for the network architecture and training data) while avoiding the training process from getting "stuck" in a local minimum. Getting stuck in a local minimum corresponds to obtaining suboptimal task performance for the network architecture and being unable to find new weight values ​​that could lead to higher task performance. The network weights are iteratively updated by feeding input data and ground truth data organized into "batches," iteratively improving network performance until no further improvements can be achieved. Each iteration through the training dataset forms an "epoch" of training, and training typically requires multiple epochs to achieve a high level of performance for a task. The trained network is then available for deployment, operating in a mode in which the weights are fixed and gradients for weight update are omitted. The process of running the pre-trained CNN with inputs and progressively transforming the inputs into outputs according to the CNN's topology is commonly referred to as "inference."

[0006] Tensors generally have four dimensions: batch, channel, height, and width. The first dimension, "batch," is typically sized at 1 when inferring video data, indicating that one frame is passed through the CNN as a single batch. When training a network, the value of the batch dimension can be increased according to a predetermined "batch size," so that multiple frames are passed through the network in each batch before the network weights are updated. Multi-frame videos can be passed through a single tensor with the batch dimension increased in size according to the number of frames in a given video. However, due to practical considerations regarding memory consumption and access, inference on video data is typically performed frame-by-frame. The "channel" dimension indicates the number of simultaneous "feature maps" for a given tensor, while the height and width dimensions indicate the size of the feature maps at a particular stage of the CNN. The channel count varies through the CNN according to the network architecture. Feature map size also varies depending on the subsampling that occurs at a particular network layer.

[0007] The overall complexity of CNNs tends to be relatively high, with a relatively large number of MAC (multiply-accumulate) operations performed and a large number of intermediate tensors written to and read from memory along with the read weights for each layer of the CNN execution. Therefore, dividing a neural network into parts enables such implementation of more complex networks even on less capable edge devices.

[0008] Feature compression can benefit from existing video compression standards, such as Versatile Video Coding (VVC), developed by the Joint Video Experts Team (JVET). VVC is particularly expected to address the continuing demand for higher compression performance as video formats increase in capacity (e.g., at higher resolutions and higher frame rates) and to address growing market demand for service delivery over WANs, where bandwidth costs are relatively high. VVC is implementable in modern silicon processes and offers an acceptable tradeoff between achieved performance and implementation cost. Implementation cost may be considered, for example, in terms of one or more of silicon area, CPU processor load, memory utilization, and bandwidth. Part of the versatility of the VVC standard lies in the wide selection of tools available for compressing video data and the wide range of applications for which VVC is suitable. Other video compression standards, such as High Efficiency Video Coding (HEVC) and AV-1, may also be used for feature compression applications.

[0009] Video data includes a sequence of frames of image data, each containing one or more color channels. When feature map data is represented in packed frames, monochrome frames, which have only luma and no color channels, are generally appropriate. When only luma samples are present, the resulting monochrome frames are said to use a "4:0:0 chroma format."

[0010] The VVC standard defines a "block-based" architecture, where a frame is first divided into an array of square regions known as "coding tree units" (CTUs). In VVC, a CTU typically occupies 128x128 luma samples. Other possible CTU sizes when using the VVC standard are 32x32 and 64x64. However, CTUs at the right and bottom edges of each frame may be smaller in area, and implicit division occurs to ensure that coding blocks remain within the frame. Each CTU is associated with a "coding tree," which defines the decomposition of the CTU's area into a set of blocks, also called "coding units" (CUs). Blocks applicable only to the luma channel or only to the chroma channels are called "coding blocks" (CBs). A prediction of the coding block's content is held in a "prediction block" (PB) or "prediction unit" (PU), and the residual block, which defines the array of sample values ​​additively combined with the PB or PU, is called a "transform block" (TB) or "transform unit" (TU) due to its typical use in the transform process to generate the TB or TU.

[0011] Despite the above distinction between "units" and "blocks", the term "block" may be used as a generic term for an area or region of a frame where an operation is applied to all color channels.

[0012] For each CU, a prediction unit (PU) is generated for the contents (sample values) of the corresponding region of the frame data ("prediction unit"). Furthermore, a representation of the difference (or "spatial domain" residual) between the contents of the region seen at the input to the encoder and the prediction is formed. The differences in each color channel can be transformed and coded as a sequence of residual coefficients, forming one or more TUs for a given CU. The applied transform can be a discrete cosine transform (DCT) or other transform applied to each block of residual values. The transform is applied in isolation (i.e., a two-dimensional transform is performed in two passes, one horizontally and one vertically). The block is first transformed by applying a one-dimensional transform to each row of samples in the block. The partial results are then transformed by applying a one-dimensional transform to each column of the partial results, producing a final block of transform coefficients that substantially decorrelates the residual samples. Transforms of various sizes are supported by the VVC standard, including transforms of rectangular-shaped blocks, with each side dimension being a power of two. The transform coefficients are quantized for entropy coding into the bitstream.

[0013] A PB or PU in VVC can be generated using either an intra-frame prediction process or an inter-frame prediction process. Intra-frame prediction involves using previously processed samples within a frame to generate a prediction of the current block of data samples within the frame. Inter-frame prediction involves using a block of samples obtained from a previously decoded frame to generate a prediction of the current block of samples in a frame. The block of samples obtained from the previously decoded frame is offset from the spatial location of the current block, often according to a filtered motion vector. An intra-frame prediction block can be (i) a uniform sample value ("DC intra-prediction"), (ii) a plane with an offset and horizontal and vertical gradients ("planar intra-prediction"), (iii) a population of blocks with neighboring samples applied in a specific direction ("directional intra-prediction"), or (iv) the result of matrix multiplication using neighboring samples and selected matrix coefficients.

[0014] VVC can be used to compress intermediate feature maps from the first part (the "backbone") of a neural network separated into two parts. During compression, the feature maps from the backbone are arranged into frames and quantized from the floating-point domain to a sample domain suitable for compression as video data. To reduce the spatial domain of the feature maps, additional neural network layers can be implemented at the interface between the VVC encoder and decoder and the intermediate point within the CNN where the division occurs. Training for such additional network layers may not be suitable for the varied and unpredictable feature map data encountered. Training may not result in a CNN that is adaptable to operating points of varying quality in terms of task performance. The operating points of the encoder and decoder may also change during operation, which must support varying levels of quality for the reconstructed tensors fed to the rest of the network on the decoder side. Summary of the Invention

[0015] It is an object of the present invention to substantially overcome, or at least ameliorate, one or more disadvantages of existing arrangements.

[0016] One aspect of the present disclosure provides a method for encoding at least a plurality of tensors forming a hierarchical representation for a single frame into a bitstream, the method including: deriving a first unit of information from a plurality of tensors forming the hierarchical representation, the plurality of tensors including at least a first tensor and a second tensor, wherein a feature map of the first tensor has greater spatial resolution than a feature map of the second tensor; encoding at least the first unit of information into the bitstream in a first mode; deriving a second unit of information from at least the first tensor in a second mode; and encoding the second unit of information and the first unit of information into the bitstream in the second mode.

[0017] Another aspect of the present disclosure provides a method for decoding at least a plurality of tensors forming a hierarchical representation for a single frame from a bitstream, the method including: decoding a bitstream including at least a first unit of information from the bitstream; and, in a first mode, deriving a plurality of tensors forming the hierarchical representation from the first unit of information, the plurality of tensors including at least a first tensor and a second tensor, wherein a feature map of the first tensor has greater spatial resolution than a feature map of the second tensor; and, in a second mode, decoding a second unit of information from the bitstream; and, in the second mode, deriving a plurality of tensors forming at least a portion of the hierarchical representation from the second unit of information and at least a portion of the first unit of information, wherein the second unit of information corresponds to the first tensor.

[0018] Another aspect of the present disclosure provides a non-transitory computer-readable storage medium storing a program for executing a method for encoding at least a plurality of tensors forming a hierarchical representation for a single frame into a bitstream, the method including: deriving a first unit of information from a plurality of tensors forming the hierarchical representation, the plurality of tensors including at least a first tensor and a second tensor, wherein a feature map of the first tensor has greater spatial resolution than a feature map of the tensor; encoding at least the first unit of information into the bitstream in a first mode; deriving a second unit of information from the first tensor in a second mode; and encoding the second unit of information and the first unit of information into the bitstream in the second mode.

[0019] Another aspect of the present disclosure provides an encoder configured to encode at least a plurality of tensors forming a hierarchical representation for a frame into the bitstream by: deriving a first unit of information from the plurality of tensors forming the hierarchical representation, the plurality of tensors including at least a first tensor and a second tensor, a feature map of the first tensor having greater spatial resolution than a feature map of the second tensor; encoding at least the first unit of information into the bitstream in a first mode; deriving a second unit of information from at least the first tensor in a second mode; and encoding the second unit of information and the first unit of information into the bitstream in the second mode.

[0020] Another aspect of the present disclosure provides a system having a memory and a processor, the processor configured to execute code stored on the memory to perform a method for encoding at least a plurality of tensors forming a hierarchical representation for a single frame into a bitstream, the method including: deriving a first unit of information from a plurality of tensors forming the hierarchical representation, the plurality of tensors including at least a tensor and a tensor, and a feature map of the first tensor having greater spatial resolution than a feature map of the tensor; encoding at least the first unit of information into the bitstream in a first mode; deriving a second unit of information from the first tensor in a second mode; and encoding the second unit of information and the first unit of information into the bitstream in the second mode.

[0021] Another aspect of the present disclosure provides a non-transitory computer-readable storage medium storing a program for executing a method of decoding at least a plurality of tensors forming a hierarchical representation for a single frame from a bitstream, the method comprising: decoding a bitstream including at least a first unit of information from the bitstream; and in a first mode, deriving a plurality of tensors forming the hierarchical representation from the first unit of information, the plurality of tensors including at least a first tensor and a second tensor, wherein a feature map of the first tensor has greater spatial resolution than a feature map of the second tensor; and decoding a second unit of information from the bitstream in a second mode; and deriving a plurality of tensors forming at least a portion of the hierarchical representation from the second unit of information and at least a portion of the first unit of information, wherein the second unit of information corresponds to at least the first tensor.

[0022] Another aspect of the present disclosure provides a decoder configured to decode at least a plurality of tensors forming a hierarchical representation for a single frame from a bitstream, the bitstream including at least a first unit of information, and in a first mode, deriving a plurality of tensors forming the hierarchical representation from the first unit of information, the plurality of tensors including at least a first tensor and a second tensor, a feature map of the first tensor having greater spatial resolution than a feature map of the second tensor, and in a second mode, decoding a second unit of information from the bitstream, and deriving a plurality of tensors forming at least a portion of the hierarchical representation from the second unit of information and at least a portion of the first unit of information, the second unit of information corresponding to at least the first tensor.

[0023] Another aspect of the present disclosure provides a system having a memory and a processor configured to execute code stored on the memory to implement a method for decoding at least a plurality of tensors forming a hierarchical representation for a single frame from a bitstream, the method comprising: decoding a bitstream including at least a first unit of information from the bitstream; and, in a first mode, deriving a plurality of tensors forming the hierarchical representation from the first unit of information, the plurality of tensors including at least a first tensor and a second tensor, a feature map of the first tensor having greater spatial resolution than a feature map of the second tensor; and, in a second mode, decoding a second unit of information from the bitstream; and deriving a plurality of tensors forming at least a portion of the hierarchical representation from the second unit of information and at least a portion of the first unit of information, the second unit of information corresponding to at least the first tensor.

[0024] Other aspects are also disclosed. [Brief explanation of the drawings]

[0025] At least one embodiment of the present invention will now be described with reference to the following drawings: [Figure 1] FIG. 1 is a schematic block diagram illustrating a distributed machine task system. [Figure 2A] FIG. 2A forms a schematic block diagram of a general-purpose computer system on which the distributed machine task system of FIG. 1 may be implemented. [Figure 2B] FIG. 2B forms a schematic block diagram of a general-purpose computer system on which the distributed machine task system of FIG. 1 may be implemented. [Figure 3A] FIG. 3A is a schematic block diagram showing the functional modules of the backbone part of the CNN. [Figure 3B] FIG. 3B is a schematic block diagram illustrating the residual block of FIG. 3A. [Figure 3C] FIG. 3C is a schematic block diagram of the residual unit of FIG. 3A. [Figure 3D] FIG. 3D is a schematic block diagram illustrating the CBL module of FIG. 3A. [Figure 4] FIG. 4 is a schematic block diagram showing the functional modules of an alternative backbone portion of a CNN. [Figure 5] FIG. 5 is a schematic block diagram illustrating a cross-layer tensor bottleneck encoder for reducing tensor dimensionality before compression. [Figure 6] Figure 6 shows a method for running the first part of the CNN, compressing it using a bottleneck encoder, and encoding the resulting compressed feature maps. [Figure 7A] FIG. 7A is a schematic block diagram illustrating a packing arrangement of multiple compressed tensors. [Figure 7B] FIG. 7B is a schematic block diagram illustrating a bitstream for multiple compressed tensors. [Figure 8] FIG. 8 is a schematic block diagram illustrating the functional modules of a video encoder. [Figure 9] FIG. 9 is a schematic block diagram showing the functional modules of a video decoder. [Figure 10] FIG. 10 is a schematic block diagram illustrating a cross-layer tensor bottleneck decoder for restoring tensor dimensionality after compression. [Figure 11] FIG. 11 shows the method for decoding the bitstream, reconstructing the decorrelated feature maps, and running the second part of the CNN. [Figure 12A] FIG. 12A is a schematic block diagram showing the head section of the CNN. [Figure 12B] FIG. 12B is a schematic block diagram illustrating the upscaler module of FIG. 12A. [Figure 12C] FIG. 12C is a schematic block diagram illustrating the detection module of FIG. 12A. [Figure 13] FIG. 13 is a schematic block diagram showing an alternative head portion of a CNN. [Figure 14] FIG. 14 shows how a tensor can be encoded into a reduced dimensional form. [Figure 15] FIG. 15 shows how to decode a tensor from a reduced dimensional form back to a tensor of the original dimensionality. DETAILED DESCRIPTION OF THE INVENTION

[0026] Where reference is made in any one or more of the accompanying drawings to steps and / or features having the same reference numerals, those steps and / or features have the same function or operation for the purposes of this description, unless intended to the contrary.

[0027] A distributed machine task system can include edge devices, such as network cameras or smartphones, that generate intermediate compressed data. A distributed machine task system can also include end devices, such as server farm-based (“cloud”) applications, that operate on the intermediate compressed data to generate some task result. Additionally, edge device functionality may be embodied in the cloud, and the intermediate compressed data may be stored for later processing, potentially for multiple different tasks as needed. Examples of machine tasks include object detection and instance segmentation, both of which generate task results measured as a “mean average precision” (mAP) for detection that exceeds an intersection-over-union (IoU) threshold, such as 0.5. Another exemplary machine task is object tracking, with a mean object tracking accuracy (MOTA) score as a typical task result.

[0028] A convenient form of intermediate compressed data is the compressed video bitstream, due to the availability of high-performance compression standards and their implementations. Video compression standards typically operate on integer samples of some given bit depth, such as 8 or 10 bits per sample, arranged in a planar array.

[0029] Tensors typically have the following dimensions: batch size, channel count, height, and width. For example, a tensor of dimensions [1, 256, 76, 136] is said to contain one batch of tensors, each containing 256 feature maps of size 136 × 76. For video data, inference is typically performed one frame at a time, rather than using tensors containing multiple frames, resulting in a batch size of 1.

[0030] FIG. 1 is a schematic block diagram illustrating the functional modules of a distributed machine task system 100 that runs a neural network split into two parts, one of which may reside in an edge device and the other in a cloud server. The system 100 can be used to perform a method for decorrelating, packing, and quantizing feature maps from encoded data into planar frames for encoding and decoding the feature maps. This method can be performed so that compressed data is encoded to reduce bitrate while adapting to changing statistics encountered in the input data. Thus, the system 100 provides the ability to perform "live" training (or "refinement training") on input tensor data to generate weight updates for networks actively used in the encoder and decoder. While ground truth for the input data is required to train the task network, refinement training is applied only to the portions of the network that form the bottleneck encoder and decoder. The goal of refinement training is to preserve the data passing through with minimal degradation. Thus, the tensors at the input to the bottleneck encoder form the ground truth for the output of the bottleneck decoder. Refinement training alleviates the need for predetermined network weights to be sufficiently trained to predict all possible input data.

[0031] Figure 1 will be described with reference to Figure 6, which illustrates a method 600 for performing a first portion of the CNN, and Figure 11, which illustrates a method 1100 for performing a second portion of the CNN. Reference is also made to Figure 7A, which illustrates the packing arrangement of feature maps from compressed tensors into monochrome video frames, and Figure 7B, which illustrates the bitstream format used in encoding and decoding the tensors.

[0032] System 100 implements a "Faster RCNN" network, which is used for object detection and splits into a backbone portion and a head portion in the illustrated example at a midpoint typically described as the "P layer." Other networks, such as "Mask RCNN," can be implemented in system 100. In particular, the backbones for Faster RCNN and Mask RCNN have the same topology and dimensionality, including convolution, batch normalization, and activation functions. The heads for Faster RCNN are a subset of the heads for Mask RCNN, and Mask RCNN includes a "mask head" that is used to generate instance segmentation maps in addition to the bounding box outputs present in both Faster RCNN and Mask RCNN. The mask head includes two convolutional layers and generates a segmentation map for each "region of interest" resulting from the RoIAlign stage, as described with reference to FIG. 13. Thus, Mask RCNN can be used to perform both object detection and instance segmentation, with the additional complexity of the network head resulting from the use of the mask head.

[0033] System 100 includes a source device 110 for generating encoded tensor data from a video source 112 in the form of an encoded video bitstream 123. System 100 also includes a destination device 140 for decoding the tensor data in the form of the encoded video bitstream 123 to generate task result 153. A communication channel 130 is used to communicate the encoded video bitstream 123 from source device 110 to destination device 140. In some configurations, either or both source device 110 and destination device 140 may comprise respective mobile phone handsets (e.g., “smartphones”) or network cameras and cloud applications. Communication channel 130 may be a wired connection, such as Ethernet, or a wireless connection, such as WiFi or 5G, including a connection via a wide area network (WAN) or an ad hoc connection. Additionally, source device 110 and destination device 140 may comprise applications in which the encoded video data is captured on some computer-readable storage medium, such as a hard disk or memory in a file server.

[0034] Source device 140 operates according to method 600 stored in memory 206 and executed under the execution of processor 205 (see FIG. 2A ). Method 600 may be performed using an apparatus such as a configured FPGA, ASIC, or ASSP. Alternatively, as described below, method 600 may be executed by source device 110 under the execution of processor 205 as one or more software code modules of application program 233 (see FIG. 2A ). Software code modules of application program 233 that implement method 600 may reside, for example, on hard disk drive 210 and / or memory 206, as described in connection with FIG. 2A . Method 600 includes functionality for encoding tensors for one frame of video data and updating weights used to encode the tensors. The updated weights are used to encode subsequent frames of video data based on the performance of the currently used weights compared to an internal model, with the weights being updated as image frames are received by source device 110.

[0035] Video source 112 provides a source of captured video frame data (shown as 113), such as an image capture sensor, a previously captured video sequence stored on a non-transitory recording medium, or a video feed from a remote image capture sensor. Video source 112 may also be the output of a computer graphics card, for example, displaying the video output of an operating system and various applications running on a computing device (e.g., a tablet computer). Examples of source device 110 that may include an image capture sensor as video source 112 include smartphones, video camcorders, professional video cameras, and network video cameras.

[0036] Source device 140 begins the machine task by executing a first portion of a CNN called backbone network 114 to generate intermediate tensors, denoted as 115. To facilitate bitrate reduction of bitstream 123, source device 140 reduces the dimensionality of tensor 115 using a compression network known as a "bottleneck encoder," denoted as encoder 116. Encoder 116 operates to reduce the dimensionality of tensor 115. At destination device 140, a "bottleneck decoder," denoted as 150, restores the tensor dimensionality so that output tensor 151 corresponds to the tensor dimensionality of tensor 115. Bottleneck encoder 116 and bottleneck decoder 150 include trainable network layers, and therefore require weights to be specified.

[0037] For task networks (including the backbone network 114 and head network 152), offline training (or "pre-training") is one possible method for specifying weights. One drawback of using offline-trained weights is the need for a training process to predict a very wide range of different input data. Typical video compression standards accommodate widely varying inputs by providing some degree of data adaptability, such as by using "context adaptive binary arithmetic coding" (CABAC) to model various input data statistics. Neural networks typically use predetermined (fixed) weights and are therefore not adaptive to the input data. Tensors 115 can form multiscale representations generated by a feature pyramid network (FPN) in the backbone 114, which are then "fused" into a single tensor, for example, in the encoder 116, using an approach called "multiscale feature compression" (MSFC). MSFC is typically used to merge all FPN layers into a single tensor and reduce the spatial dimensionality of the single tensor to the spatial resolution of the smallest FPN layer tensor. Merging all FPN layers into a single tensor is done at the expense of spatial detail for the undecomposed (larger) layers of the FPN.

[0038] The operation of source device 110 will be described with reference to method 600 of Figure 6. Method 600 begins with neural network first portion execution step 610.

[0039] In step 610, the CNN backbone 114 receives one frame of video frame data 113 from the video source 112 and executes a particular initial layer of the overall CNN, such as a layer corresponding to the "backbone" of the CNN. Step 610 outputs tensors 115. The backbone layer of the CNN 114 may, for example, generate multiple tensors as output for each frame, corresponding to different spatial scales at which the backbone, including the FPN, is applied to the input image. The tensors 115 resulting from the backbone 114 with the FPN form a hierarchical representation of the frame data 113, including feature map data. Each successive layer in the hierarchical representation has half the width and half the height of the previous layer. The FPN may result in three tensors in tensor 115, corresponding to the three layers output from the backbone 114 when the "YOLOv3" network is executed by the system 100, with the tensors 115 having varying spatial resolutions and channel counts. When system 100 is running a network such as "Faster RCNN X101-FPN" or "Mask RCNN X101-FPN," tensors 115 include tensors for four layers P2-P5. The first portion of layers executed in backbone module 114 is sometimes referred to as the "backbone" of the overall task network, although the particular division of layers between source device 110 and destination device 140 need not correspond to boundary results from layers typically defined as the "backbone" in the task network. The terms "backbone" and "head" are used herein to refer to the division of the network into first and second portions, and such division can be selected based on considerations other than the machine task network architecture, such as available computational resources in source device 110 or destination device 140.

[0040] Method 600 continues under control of processor 205 from step 610 to bottleneck encoding execution step 615. In step 615, source device 110 uses bottleneck encoder 116 to reduce the number and dimensionality of tensors 115 by executing several network layers corresponding to compression operations that also use associated sets of weights in destination device 140. Step 615 receives as input at least one tensor on which a neural network for a machine task has been partially executed (i.e., on which a backbone network has been executed) and generates tensor 117. For a given frame of frame data 113, tensor 115 includes multiple tensors due to the use of FPN in backbone 114, with different spatial resolutions between the FPN layers. This "multi-scale" representation is converted into fewer tensors in bottleneck encoder 116, e.g., one ("base layer") or two ("base layer" and "enhancement layer") tensors in reduced tensor 117 for each frame of frame data 113, whose spatial dimensions have also been reduced. The cross-layer fusion and spatial reduction performed in the encoder 116 is an example of one method of deep feature compression for fusing neural network features. In the described example, cross-layer fusion is performed using a technique called “multi-scale feature compression” (MSFC). The operation of MFSC is described below with reference to FIG. 5. A variation of MSFC as described with reference to FIG. 5 is possible, which involves the application of single-scale feature fusion (SSFC) to each FPN layer without cross-layer fusion. While the application of a separate SSFC stage for each FPN layer allows for channel count reduction for each layer, it does not provide any means for spatial reduction of layers with larger resolution than the FPN layer resolution, nor does it exploit cross-layer redundancy. Step 615 operates to perform data compression for data associated with an image frame (in the described example, processed by the backbone module 114).Data compression is performed by the encoder's neural network using the currently associated or applied set of weights. The method of cross-layer fusion and spatial reduction performed in step 615 is described with reference to method 1400 of Figure 14. Control in processor 205 passes from step 615 to a compressed tensor encoding step 620.

[0041] In step 620, quantization and packing module 184 quantizes tensor 117 from the floating-point domain to integer samples, such as 10-bit samples. Module 184 also packs the feature maps for each channel of tensor 117 into frames using the packing format described with reference to FIG. 7A to generate packed frames 185. Video encoder 120 encodes packed frames 185 as compressed tensors 121 into bitstream 123. Control in processor 205 passes from step 620 to a bottleneck decoding execution step 625.

[0042] In step 625, tensor 117 is provided to bottleneck decoder 118. Bottleneck decoder 118 operates with a set of weights known to destination device 140 to generate a reconstructed tensor 119 from tensor 117. Reconstructed tensor 119 has the same tensor dimensionality as tensor 115. Tensor 119 represents a degraded version of tensor 115, with losses due to the reduced dimensionality of tensor 117 and the optimality of the weights of modules 116 and 118 relative to the input tensor statistics. Bottleneck decoder 118 is initialized with the same weights used in bottleneck decoder 150. Method 1500, described with reference to FIG. 15, illustrates one approach for generating reconstructed tensor 119 from tensor 117. Control in processor 205 passes from step 625 to a bottleneck performance measurement step 630.

[0043] In step 630, a value for evaluating data compression is determined or obtained using the weights determined in step 640. For example, a mean squared error (MSE) module 178 compares tensor 115 with tensor 119 to generate an MSE value 179 that indicates the loss due to the conversion of tensor 117 to and from the reduced dimensionality of tensor 117. Over time, the statistics of tensor 115 may change, leading to a decrease in the performance of bottleneck encoder 116 and bottleneck decoder 118, seen as a decrease in MSE 179, although short-term fluctuations in MSE may also be expected. MSE 179 provides an indication of the expected signal quality of tensor 151 in destination device 140, except that the lossy video encoding process of video encoder 120 is not included in this indication. Thus, MSE 179 provides an indication of the performance of bottleneck encoders and decoders (i.e., 116, 118, 150) in storing tensor data passed from backbone 114 to head 152. The value 179 is obtained or acquired using the tensor 119 after data compression using the weights associated with the neural network in steps 615 and 630, the tensor 115 input to the encoder 116. In other embodiments, methods other than MSE may be used to evaluate data compression. Control in the processor 205 passes from step 630 to a state training step 635.

[0044] In step 635, the processor 205 updates a training state variable based on the MSE value 179. The training state variable is stored in memory 206 and indicates whether the source device 110 is currently training a set of modules. If the MSE value 179 is below a threshold, the training state variable is set to the "TRAINING" state. The threshold may be determined in various ways, such as using a moving average of previously generated MSE values ​​179 or comparing a desired MSE value to a predetermined or configurable threshold based on the moving average of the MSE values. If, after a period of training, no weight updates have been performed, the training state variable may be set back to discontinue further training. If the training state variable indicates that training is in progress, control in the processor 205 proceeds from step 635 to a trainable bottleneck encoding / decoding execution step 640 ("TRAIN" in FIG. 6). Independent of measurements such as MSE value 179, step 635 can operate to enter the TRAINING state periodically, e.g., once a day or once a week, to provide a continuous means to determine whether an improved model can be generated even in the absence of clear indicators of performance degradation such as a decrease in measured MSE value 179. If the training state variable does not indicate that training is in progress, control in processor 205 passes from step 635 to weight update flag encoding step 660 ("NOT TRAIN" in FIG. 6).

[0045] In step 640, trainable bottleneck encoder 170 and trainable bottleneck decoder 174 operate to compress and decompress tensor 115 to generate tensor 175. Trainable bottleneck encoder 170 has the same structure as encoder 116. Trainable bottleneck decoder 174 has the same structure as decoder 150. The structural compatibility between modules 170 and 116, and between modules 174 and 150, allows weights generated in modules 170 and 174 to be forwarded to modules 116 and 150, respectively, for subsequent processing. Compressed tensor 171 is output from trainable bottleneck encoder 170 and passed to trainable bottleneck decoder 174. A forward pass through modules 170 and 174 is performed using the weights currently present in modules 170 and 174. The network topology and layer dimensionality of modules 170 and 174 correspond to those of modules 116 and 118, respectively. The operation of modules 170 and 174 is in accordance with methods 1400 and 1500, respectively, described with reference to Figures 14 and 15. Control in processor 205 passes from step 640 to a trainable bottleneck performance measurement step 645.

[0046] In step 645, values ​​for evaluating data compression are determined or obtained using the weights determined in step 640. Step 645 evaluates data compression performance using the same method as step 630. For example, mean squared error (MSE) module 180 generates measurement loss 181 by performing a mean squared error calculation on tensors 115 and 175. Measurement loss 181 value provides a measure of the ability of modules 170 and 174 to accurately restore a tensor after its dimensionality has been reduced by using reduced-dimensionality tensor 171. Control in processor 205 passes from step 645 to a backpropagation step 650.

[0047] In step 650, modules 170 and 174 perform weight updates using a process of “backpropagation,” whereby weights are updated based on a process such as stochastic gradient descent (SGD) to attempt to minimize measurement loss 181. Continuous weight updates via backpropagation allow trainable bottleneck encoder 170 and trainable bottleneck decoder 174 to adapt to such changes in the statistics of tensor 115, potentially achieving a lower MSE 181 compared to MSE 179. The rate at which weights are updated is scaled by a “learning rate.” A higher learning rate generally results in fewer backpropagation operations, i.e., a faster training process, which minimizes measurement loss, but risks instability in training due to over-adjusting weights that prevent the loss function from finding a local minimum. While a smaller learning rate may take longer to train the network, a smaller learning rate value is also less likely to over-adjust the rate. When using a small batch size, such as one image, a smaller learning rate is desirable to reduce the influence of individual frames that may be statistical outliers. A typical learning rate may be a value such as 0.01 or 0.001, scaled by the inverse of the batch size or by the inverse square root of the batch size to arrive at a final learning rate for a given batch. The learning rate can be varied over time, with larger values ​​used initially when the network weights are far from their final values ​​of "weights" and smaller values ​​used later when the network is closer to optimal or acceptable in terms of MSE. In the context of refinement training of bottleneck encoders and decoders, smaller learning rates, such as between 0.001 and 0.0001, are appropriate. However, modules 170 and 174 receive tensors one at a time, and input tensors may be grouped into batches of size greater than one. Increasing the batch size can improve training because each weight update step is influenced by different inputs.Increasing the batch size increases the memory requirements for modules 170 and 174. Furthermore, for video data, statistical variability in successive frames or successive tensors from the backbone 114 is less pronounced, reducing the benefit of using a larger batch size. Other processes for updating the weights may also be used, such as the "AdamW" optimizer, which utilizes momentum and scaling and decouples weight decay from gradient updates. The reduced tensor 171 forms the result of the bottleneck encoder and decoder (i.e., 170 and 174) that have undergone training or adaptation to the actual input data when they encounter the actual input data, i.e., frame data 113 converted to tensor 115. Control in processor 205 passes from step 650 to a weight update determination step 655.

[0048] In step 655, trigger module 182 compares MSE 179 to MSE 181. If MSE 179 is observed to be below MSE 181 for a period of time, and the difference exceeds a threshold, a weight update process is initiated. The period may correspond to a number of frames, a moving average, or a combination thereof, indicating sub-par performance of modules 116 and 118 compared to achievable performance, as indicated by modules 170 and 174. The heuristic for initiating a weight update is adapted to capture the point where the currently used weights for tensor reduction and restoration (i.e., the transformation of tensor 115 to tensor 117, and finally to tensor 119 or 151) are no longer well suited to the statistics of the received input data. The weights used in modules 170 and 174 are costly to encode in bitstream 123 and therefore are not transmitted to destination device 140 with each update operation. A less frequent transmission of updated weights from source device 110 to destination device 140 resulting from the determination of step 655, e.g., based on detection of performance degradation while using currently active weights, may be sufficient for adequate system performance. The result of step 655 is a decision whether to perform a weight update in the formation of weight update flag 750. Steps 625-655 operate to determine whether the set of weights for data compression applied in the next round of data compression should be changed. Whether a change is implemented is determined based on a comparison of MSE values ​​181 and 179. If a weight update or change is performed, the training state variables are also reset to discontinue further training on subsequent invocations of method 600. Control in processor 205 passes from step 655 to weight update flag encoding step 660.

[0049] 8 encodes the weight update flag into the bitstream 123 as a weight update flag 750. The instruction to update the weight may be included in a supplementary enhancement information (SEI) message associated with the current packed frame 185. The SEI message 744 contains the weight and may be associated with the current packed frame 185 either in a separate SEI message containing the weight and the weight update flag 750, or in a separate SEI message containing the weight. Alternatively, the presence of the SEI message containing the weight may be an indication from the source device 140 to the destination device 140 that a weight update should be performed. Control in the processor 205 passes from step 660 to a weight update flag test step 665.

[0050] In step 665, if the weight update flag 750 indicates a determination in the trigger module 182 that the weights should be changed or updated ("UPDATE" as shown in FIG. 6), control in the processor 205 proceeds to a load update weights step 667. Otherwise, if the weight update flag does not indicate a determination in the trigger module 182 to perform a weight update, control in the processor 205 proceeds from step 665 to processing the next frame of video data in a execute neural network first portion step 675 ("NO UPDATE" as shown in FIG. 6).

[0051] In step 667, the bottleneck decoder weights 176 are passed from the bottleneck decoder 174 to the bottleneck decoder 118 and to the weight encoder 186. The bottleneck encoder weights 172 from the trainable bottleneck encoder 170 are loaded or applied to the bottleneck encoder 116. The weight changes may point to a particular set of stored values ​​or may replace previously stored values. In step 667, the association of the encoder 116 is changed to the weights 172 from the set of weights used in step 615. Associating the weights 172 with the encoder 116 may involve loading the weights 172 into an area of ​​memory 206 referenced by the encoder 116 or changing a pointer to select the weights 172 for subsequent use in the encoder 116. Control in the processor 205 passes from step 667 to an update weight encoding step 670.

[0052] In step 670, weight encoder 186 encodes bottleneck decoder weights 176 to generate coded weights 187. Coded weights 187 are stored in bitstream 123 (as weights 752 in FIG. 7B) using multiplexer 122. Encoding weights 187 may involve directly coding weights 187 using variable-length codewords to compress each weight value. An arithmetic coding scheme such as context-adaptive binary arithmetic coding (CABAC) may be used to code weights 187. Alternatively, the coded information may represent deltas between weights, e.g., deltas relative to weights previously used by bottleneck decoder 118 and also known by bottleneck decoder 146, via previous weight updates and synchronized initial states. One exemplary syntax that can be used to represent the weights is standard ISO / IEC 15938-17, sometimes referred to as "Neural Network Coding" or "Neural Network Representation" or "MPEG-NNR," although other means for efficiently encoding neural network weights into a bitstream, such as the "Open Neural Network Exchange Intermediate Representation," may be used. Upon completion of step 670, the encoding of the partial task result, i.e., tensor 115, is complete for one frame from video source 112. Progression to a subsequent frame, such as the next frame, occurs. The remaining steps of method 600 relate to subsequent frames from video source 112 and indicate the use or nonuse of the updated weights determined in step 655. Control in processor 205 passes from step 670 to a second neural network first portion execution step 675.

[0053] In step 675, similar to step 610, a first portion of the neural network is executed on a subsequent frame of frame data from video source 112, e.g., frame 113a, to generate updated tensors for frame 113a, such as tensor 115a. Control in processor 205 passes from step 675 to a second bottleneck encoding execution step 680.

[0054] In step 680, similar to step 615, the bottleneck encoder 116 performs encoding of the updated tensor 115a to produce an updated tensor 117a. Step 680 performs encoding using the bottleneck encoder weights 172 received by the bottleneck encoder 116 from the trainable bottleneck encoder 172 if a weight update was performed in step 667, as determined in step 655. In other words, data compression is performed using the associated updated set of weights 176. Control in the processor 205 passes from step 680 to a second compressed tensor encoding step 685.

[0055] In step 685, similar to step 620, the updated tensor 117a, represented as a packed frame according to the format of frame 700, is encoded by video encoder 120 into video bitstream 121 as compressed frame N+1 (see 746 in FIG. 7B ). Video bitstream 121 is multiplexed into bitstream 123 by multiplexer 122. Source device 140, under the execution of processor 205, continues to encode feature maps of successive frames of video data 113 from backbone 114, with trainable layers such as those of modules 116 and 118 being updated from time to time as determined by module 182. As a result of the operation of method 600, bitstream 123 includes the encoded tensors from the first portion of the neural network, possibly including dynamic weight updates for tensor dimensionality reduction. The updates depend on the performance of bottleneck encoder 116 relative to the performance achievable by bottleneck encoder 170 and decoder 174. The training on the received data in modules 170 and 174 takes advantage of the fact that for compression tasks, the goal is to recover the input data with minimal loss. In other words, for compression tasks, the ground truth is the input data. The training performed by modules 170 and 174 is performed simultaneously with the use of the "expanded" network weights present in modules 116, 118, and 150, and uses the same input data. Therefore, the training performed by modules 170 and 174 can be said to "overfit" the current input data. However, because the network is capable of dynamic weight updates, this overfitting behavior can be considered data-driven adaptation. Support for sophisticated training alleviates the need to increase the training complexity of the bottleneck encoder and decoder and / or the complexity of the bottleneck encoder and decoder themselves to accommodate a wider statistical variety of input data.Considering the data path provided in system 100 from input frame 113 to task result 153, i.e., modules 114, 116, 150, and 152, the trainable portion of this path corresponds to modules 116 and 150, forming a subset of all CNN layers in the data path.

[0056] The operation of the destination device 140 portion of system 100 will be described with reference to method 1100 of Figure 11. Method 1100 relates to data encoded at a source device on which neural network processing for machine tasks has been performed in part (i.e., by backbone network 114). Method 1100 may be performed by software code modules of application program 233 stored on memory 206 and controlled by execution of processor 205. Method 1100 begins with a packed frame decoding step 1110.

[0057] In step 1110, demultiplexer 142 receives bitstream 123. Demultiplexer 142 extracts video bitstream 143 from bitstream 123, which corresponds to bitstream 121. Video bitstream 143 is provided to video decoder 146, which generates decoded packed frames 162. Decoded packed frames 162 are passed to unpacking and dequantization module 160, which extracts each feature map from packed frame 162, dequantizes it from the integer sample domain to the floating-point domain, and places the feature map into a tensor, such as decoded tensor 147. Control in processor 205 passes from step 1110 to a bottleneck decoding execution step 1120.

[0058] In step 1120, bottleneck decoder 150 converts tensor 147, including weights for layers such as convolutional layers, into tensor 151, which has increased dimensionality compared to tensor 147. Bottleneck decoder 150 operates to decode data associated with images (including in the case of image frames of video) for which compression has already been performed in source device 110. The decoding is performed by the neural network of bottleneck decoder 150 using the current set of associated weights. Bottleneck decoder 150 uses a decoding method associated with the encoding or compression performed by bottleneck encoder 116, for example, MFSC. Operation of bottleneck decoder 150 follows method 1500 described with reference to FIG. 15 . Control in processor 205 passes from step 1120 to neural network second portion execution step 1130.

[0059] In step 1130, head module 152 executes a second portion of the overall neural network implemented in system 100 to generate task results 153. Task results 153 are stored in task result buffer 154, which is typically implemented in memory 206. Thus, method 1100 executes the remaining portion of the neural network machine task in step 1130. Control in processor 205 passes from step 1130 to a weight update flag decoding step 1140.

[0060] 9 decodes the weight update flag 750 from the bitstream 123. The weight update flag 750 provides information indicating whether the set of weights for bottleneck decoding should be changed, i.e., whether the weights in the bottleneck decoder 150 should be updated, as determined by the trigger module 182. Control in the processor 205 passes from step 1140 to a weight update indication test step 1150.

[0061] In step 1150, application 233 determines whether the weights should be updated based on the decoded weight update flag. If the weight update flag 750 indicates a weight update or change ("UPDATE" in step 1150), control in processor 205 proceeds to step 1160, where updated weights are decoded. As described in connection with step 670, the information can relate to weight values ​​or deltas between weight values. Otherwise, control in processor 205 proceeds to step 1180, where the packed second frame is decoded ("NO UPDATE" in step 1150).

[0062] In step 1160, information indicative of the coded weights 145 is extracted from the bitstream 123 using the demultiplexer 142. The weight decoder 148 converts or decodes the coded weights 145 into decoded weights 149. Control in the processor 205 passes from step 1160 to an apply updated weights step 1170.

[0063] In step 1170, the bottleneck decoder 150 is updated to use or apply the decoded weights 149, maintaining synchronization with the weights present in the bottleneck decoder 118 in the source device 140. In other words, the neural network associations of the bottleneck decoder 150 are updated from the weights used in step 1120 to use the weights 149. The update may point to a specific set of stored values ​​or replace previously stored values. Once step 1170 is complete, the CNN tasks implemented in the backbone 114 and head 152 are performed for one frame, and progress to subsequent frames, such as the next frame. Control in the processor 205 passes from step 1170 to step 1180.

[0064] In step 1180, similar to step 1110, video decoder 146 decodes second encoded frame 746 from bitstream 123 to produce a second packed frame, e.g., frame 147a. Control in processor 205 passes from step 1180 to a second bottleneck decoding execution step 1190.

[0065] In step 1190, bottleneck decoder 150 decodes frame 147a to produce second decoded tensor 151a using weights 149, as indicated by weight update flag 750. Destination device 140 continues to run CNN head 152 using tensor 151a to produce another task result, such as result 153a, and continues to decode subsequent frames, occasionally updating the weights used in the bottleneck decoder as indicated by decoded weight update flag 750.

[0066] The contents of task results buffer 154 may be presented to a user via a graphical user interface, for example, or provided to an analytics application where some action is determined based on the task results, which may include a summary-level presentation of aggregated task results to a user. The functionality of each of source device 110 and destination device 140 may be implemented in a single device in some embodiments, examples of which include a mobile phone handset, a tablet computer, and a cloud application.

[0067] Notwithstanding the exemplary devices described above, each of the source device 110 and the destination device 140 may generally be configured within a general-purpose computing system through a combination of hardware and software components. FIG. 2A illustrates such a computer system 200, including a computer module 201, input devices such as a keyboard 202, a mouse pointer device 203, a scanner 226, a camera 227 that may be configured as a video source 112, a microphone 280, and output devices including a printer 215, a display device 214 that may be configured as a display device for presenting task results 151, and a loudspeaker 217. An external modulator-demodulator (Modem) transceiver device 216 may be used by the computer module 201 to communicate with a communications network 220 via a connection 221. The communications network 220, which may represent the communications channel 130, may be a WAN (wide area network), such as the Internet, a cellular telecommunications network, or a private WAN. If the connection 221 is a telephone line, the modem 216 may be a conventional “dial-up” modem. Alternatively, if connection 221 is a high-capacity (e.g., cable or optical) connection, modem 216 may be a broadband modem. A wireless modem may also be used for wireless connection to communication network 220. Transceiver device 216 may provide the functionality of transmitter 122 and receiver 142, and communication channel 130 may be implemented in connection 221.

[0068] The computer module 201 typically includes at least one processor unit 205 and a memory unit 206. For example, the memory unit 206 can include semiconductor random access memory (RAM) and semiconductor read-only memory (ROM). The computer module 201 also includes several input / output (I / O) interfaces, including an audio-video interface 207 that couples to a video display 214, a loudspeaker 217, and a microphone 280; an I / O interface 213 that couples to a keyboard 202, a mouse 203, a scanner 226, a camera 227, and optionally a joystick or other human interface device (not shown); and an interface 208 for an external modem 216 and a printer 215. The signal from the audio-video interface 207 to the computer monitor 214 is typically the output of a computer graphics card. In some embodiments, the modem 216 may be incorporated within the computer module 201, for example, within the interface 208. The computer module 201 also has a local network interface 211 that allows the computer system 200 to be coupled via a connection 223 to a local area communication network 222, known as a local area network (LAN). As shown in Figure 2A, the local communication network 222 can also be coupled via a connection 224 to a wide network 220, which typically includes a so-called "firewall" device or device of similar functionality. The local network interface 211 can be an Ethernet TM Circuit card, Bluetooth TM The local network interface 211 may have a wireless or IEEE 802.11 wireless configuration, although numerous other types of interfaces may be implemented for the interface 211. The local network interface 211 may also provide the functionality of the transmitter 122 and receiver 142, and the communication channel 130 may also be implemented in a local communication network 222.

[0069] The I / O interfaces 208 and 213 can provide either or both serial and parallel connectivity, the former typically being implemented according to the Universal Serial Bus (USB) standard and having a corresponding USB connector (not shown). A storage device 209 is provided and typically includes a hard disk drive (HDD) 210. Other storage devices, such as a floppy disk drive or magnetic tape drive (not shown), may also be used. An optical disk drive 212 is typically provided to serve as a non-volatile source of data. For example, such optical disks (e.g., CD-ROM, DVD, Blu-ray Disc, TM ), USB-RAM, portable, external hard drives, and floppy disks, portable memory devices may be used as suitable sources of data to computer system 200. Typically, any of HDD 210, optical drive 212, and networks 220 and 222 may be configured to operate as video source 112 or as a destination for decoded video data to be stored for playback via display 214. Source device 110 and destination device 140 of system 100 may be embodied in computer system 200.

[0070] The components 205-213 of the computer module 201 typically communicate via an interconnected bus 204 and in a manner which results in a conventional mode of operation of the computer system 200 known to those skilled in the art. For example, the processor 205 is coupled to the system bus 204 using a connection 218. Similarly, the memory 206 and optical disk drive 212 are coupled to the system bus 204 by a connection 219. Examples of computers on which the described arrangements can be practiced include IBM-PCs and compatibles, SunSPARCstations, Apple Macs, and the like. TM , or a similar computer system.

[0071] Where appropriate or desirable, source device 110 and destination device 140, as well as the methods described below, may be implemented using computer system 200. In particular, source device 110, destination device 140, and the methods described may be implemented as one or more software application programs 233 executable within computer system 200. Source device 110, destination device 140, and the steps of the methods described are performed by instructions 231 (see FIG. 2B ) in software 233 executing within computer system 200. The software instructions 231 may be formed as one or more code modules, each for performing one or more specific tasks. The software may also be divided into two separate portions, with a first portion and corresponding code modules performing the methods described and a second portion and corresponding code modules managing the user interface between the first portion and a user.

[0072] The software may be stored on a computer-readable medium, including, for example, the storage devices described below. The software is loaded from the computer-readable medium into computer system 200 and then executed by computer system 200. A computer-readable medium having such a computer program or software recorded thereon is a computer program product. Use of the computer program product in computer system 200 preferably results in advantageous apparatus for implementing source device 110 and destination device 140 and the methods described.

[0073] The software 233 is typically stored on the HDD 210 or in the memory 206. The software is loaded into the computer system 200 from a computer-readable medium and executed by the computer system 200. Thus, for example, the software 233 may be stored on an optically readable disk storage medium (e.g., a CD-ROM) 225 that is read by the optical disk drive 212.

[0074] In some examples, the application program 233 may be encoded on one or more CD-ROMs 225 and supplied to the user, and may be read via a corresponding drive 212, or may be read by the user from the network 220 or 222. Additionally, the software may be loaded into the computer system 200 from other computer-readable media. A computer-readable storage medium refers to any non-transitory, tangible storage medium that provides recorded instructions and / or data to the computer system 200 for execution and / or processing. Examples of such storage media are floppy disks, magnetic tape, CD-ROMs, DVDs, Blu-ray Discs, and the like. TM , a hard disk drive, a ROM or integrated circuit, a USB memory, a magneto-optical disk, or a computer-readable card such as a PCMCIA card, etc., whether such devices are internal or external to the computer module 201. Examples of transitory or non-tangible computer-readable transmission media that may also participate in providing software, application programs, instructions, and / or video data or encoded video data to the computer module 201 include wireless or infrared transmission channels, as well as network connections to another computer or networked device, and the Internet or intranet, including email transmissions and information stored on websites, etc.

[0075] A second portion of application program 233 and the corresponding code modules described above may be executed to implement one or more graphical user interfaces (GUIs) that are rendered or represented on display 214. A user of application and computer system 200, typically through operation of keyboard 202 and mouse 203, may manipulate the interface in a functionally adaptable manner to provide control commands and / or input to applications associated with the GUI. Other forms of functionally adaptable user interfaces may also be implemented, such as an audio interface that utilizes speech prompts output via loudspeaker 217 and user voice commands input via microphone 280.

[0076] Figure 2B is a detailed schematic block diagram of processor 205 and "memory" 234. Memory 234 represents the logical collection of all memory modules (including storage devices 209 and semiconductor memory 206) accessible by computer module 201 of Figure 2A.

[0077] When the computer module 201 is first powered on, a power-on self-test (POST) program 250 is executed. The POST program 250 is typically stored in the ROM 249 of the semiconductor memory 206 of FIG. 2A. Hardware devices, such as the ROM 249, that store software are sometimes referred to as firmware. The POST program 250 inspects the hardware within the computer module 201 to ensure proper functionality and typically checks the processor 205, memory 234 (209, 206), and a basic input / output system software (BIOS) module 251, also typically stored in the ROM 249, for correct operation. If the POST program 250 executes successfully, the BIOS 251 boots the hard disk drive 210 of FIG. 2A. The booting of the hard disk drive 210 causes a bootstrap loader program 252, resident on the hard disk drive 210, to execute via the processor 205. This loads the operating system 253 into the RAM memory 206, and the operating system 253 begins operation. Operating system 253 is a system-level application executable by processor 205 and performs a variety of high-level functions, including processor management, memory management, device management, storage management, software application interface, and general-purpose user interface.

[0078] The operating system 253 manages the memory 234 (209, 206) to ensure that each process or application running on the computer module 201 has enough memory to run without conflicting with memory allocated to another process. Furthermore, the different types of memory available in the computer system 200 of FIG. 2A need to be used appropriately so that each process can run effectively. Thus, the aggregate memory 234 is not intended to show how specific segments of memory are allocated (unless otherwise specified), but rather to provide a general view of the memory accessible by the computer system 200 and how such memory is used.

[0079] As shown in FIG. 2B, processor 205 includes several functional modules, including a control unit 239, an arithmetic logic unit (ALU) 240, and a local or internal memory 248, sometimes referred to as a cache memory. Cache memory 248 typically includes several storage registers 244-246 within a register section. One or more internal buses 241 functionally interconnect these functional modules. Processor 205 also typically has one or more interfaces 242 for communicating with external devices via system bus 204 using connection 218. Memory 234 is coupled to bus 204 using connection 219.

[0080] Application program 233 includes a set of instructions 231, which may include conditional branch and loop instructions. Program 233 may also include data 232 used in the execution of program 233. Instructions 231 and data 232 are stored in memory locations 228, 229, 230 and 235, 236, 237, respectively. Depending on the relative sizes of memory locations 228-230 and instructions 231, a particular instruction may be stored in a single memory location, as indicated by the instruction shown in memory location 230. Alternatively, an instruction may be segmented into several parts, each stored in a separate memory location, as indicated by the instruction segments shown in memory locations 228 and 229.

[0081] Generally, the processor 205 is given a set of instructions to execute therein. The processor 205 awaits a subsequent input to which the processor 205 responds by executing another set of instructions. Each input may be provided from one or more of several sources, including data generated by one or more of the input devices 202, 203, data received from an external source over one of the networks 220, 202, data retrieved from one of the storage devices 206, 209, or data retrieved from a storage medium 225 inserted into a corresponding reader 212, all shown in FIG. 2A. Execution of the set of instructions may in some cases result in the output of data. Execution may also involve storing data or variables in memory 234.

[0082] The bottleneck encoder 116, bottleneck decoder 148, and described methods may use input variables 254, which are stored in corresponding memory locations 255, 256, 257 in memory 234. The bottleneck encoder 116, bottleneck decoder 148, and described methods generate output variables 261, which are stored in corresponding memory locations 262, 263, 264 in memory 234. Intermediate variables 258 may be stored in memory locations 259, 260, 266, and 267.

[0083] 2B, registers 244, 245, 246, arithmetic logic unit (ALU) 240, and control unit 239 cooperate to perform the sequence of micro-operations required to perform a "fetch, decode, and execute" cycle for each instruction in the instruction set that makes up program 233. Each fetch, decode, and execute cycle: a fetch operation that fetches or reads instructions 231 from memory locations 228, 229, 230; a decode operation in which the control unit 239 determines which instructions have been fetched; and The control unit 239 and / or the ALU 240 execute instructions. Includes:

[0084] A further fetch, decode, and execute cycle for the next instruction may then be performed. Similarly, a store cycle may be performed by the control unit 239 storing or writing a value to the memory location 232.

[0085] Each step or sub-process in the methods of FIGS. 4-15 is associated with one or more segments of program 233 and is typically performed by register sections 244, 245, 247, ALU 240, and control unit 239 within processor 205, which cooperate to perform fetch, decode, and execute cycles for all instructions in the instruction set for the referenced segment of program 233.

[0086] 3A is a schematic block diagram showing the functional modules of a CNN backbone portion 310 that may function as a CNN backbone 114. The backbone portion 114 may be referred to as "DarkNet-53" and forms the backbone of the "YOLOv3" object detection network. Different backbones are possible, resulting in different numbers of layers and dimensionality of tensors 115 for each frame.

[0087] As shown in FIG. 3A, video data 113 is passed to a resizer module 304. The resizer module 314 resizes the frames to a resolution suitable for processing by the CNN backbone 310, generating resized frame data 312. If the resolution of the frame data 113 is already suitable for the CNN backbone 310, the operation of the resizer module 304 is not required. The resized frame data 312 is passed to a convolutional batch normalization leaky rectified linear (CBL) module 314, which generates tensors 316. The CBL 314 includes modules such as those described with reference to CBL module 360, as shown in FIG. 3D.

[0088] Referring to FIG. 3D , CBL module 360 ​​takes tensor 361 as input. Tensor 361 is passed to convolutional layer 362 to generate tensor 363. If convolutional layer 362 has a stride of 1 and padding is set to k samples, then with a convolution kernel of size 2k+1, tensor 363 has the same spatial dimensions as tensor 361. If convolutional layer 362 has a larger stride, such as 2, tensor 363 has smaller spatial dimensions compared to tensor 361, e.g., half the size for a stride of 2. Regardless of the stride, the size of the channel dimensions of tensor 363 may vary compared to the channel dimensions of tensor 361 for a particular CBL block. Tensor 363 is passed to batch normalization module 364, which outputs tensor 365. Batch normalization module 364 normalizes input tensor 363 and applies a scaling factor and offset value to generate output tensor 365. The scaling factor and offset values ​​are derived from the training process. Tensor 365 is passed to a leaky modified linear activation ("LeakyReLU") module 366, which produces tensor 367. Module 366 provides a "leaky" activation function, where positive values ​​in the tensor are passed through and negative values ​​are significantly reduced in magnitude, e.g., to 0.1X their former values.

[0089] Returning to Figure 3A, tensor 316 is passed from CBL block 314 to residual block 11 module 320. Module 320 contains a serial concatenation of three residual blocks containing 1, 2, and 8 residual units within them, respectively.

[0090] A residual block such as that present in module 320 will be described with reference to ResBlock 340, as shown in FIG. 3B. ResBlock 340 receives tensor 341. Tensor 341 is zero-padded by zero-padding module 342 to produce tensor 343. Tensor 343 is passed to CBL module 344 to produce tensor 345. Tensor 345 is passed to residual unit 346, and residual block 340 includes a series of concatenated residual units. The last residual unit in residual unit 346 outputs tensor 347.

[0091] As shown in FIG. 3C , a residual unit, such as unit 346, is described with reference to ResUnit 350. ResUnit 350 takes tensor 351 as input. Tensor 351 is passed to CBL module 352 to produce tensor 353. Tensor 353 is passed to a second CBL unit 354 to produce tensor 355. Addition module 356 sums tensor 355 with tensor 351 to produce tensor 357. Addition module 356 is sometimes referred to as a “shortcut” because input tensor 351 effectively affects output tensor 357. For untrained networks, ResUnit 350 operates on pass-through tensors. When training is performed, CBL modules 352 and 354 operate to derive tensor 357 from tensor 351 according to the training data and ground truth data.

[0092] Returning to FIG. 3A , Res11 module 320 outputs tensor 322. Tensor 322 is output from backbone module 310 as one of the layers and is also provided to Res8 module 324. Res8 module 324 is a residual block (i.e., 340) containing eight residual units (i.e., 350). Res8 module 324 generates tensor 326. Tensor 326 is passed to Res4 module 328 and output from backbone module 310 as one of the layers. Res4 module 328 is a residual block (i.e., 340) containing four residual units (i.e., 350). Res4 module 324 generates tensor 329. Tensor 329 is output from backbone module 310 as one of the layers.

[0093] Collectively, layer tensors 322, 326, and 329 are output as tensor 115. Backbone CNN 310 can take a video frame with a resolution of 1088x608 as input and generate three tensors corresponding to three layers, with dimensions of [1, 256, 76, 136], [1, 512, 38, 68], and [1, 1024, 19, 34]. Another example of three tensors corresponding to three layers could be [1, 512, 34, 19], [1, 256, 68, 38], and [1, 128, 136, 76], which are divided at the 75th, 90th, and 105th network layers in CNN 310, respectively. Each tensor can have a different resolution from the next tensor. The resolution of each tensor can double in height and width between each tensor. In forming the output tensor 115, the layer tensors 322, 326, and 329 provide a hierarchical representation of the frame data, including the feature map data, for encoding into a bitstream. The segmentation points depend on the CNN 310.

[0094] 4 is a schematic block diagram illustrating the functional modules of an alternative backbone portion 400 of a CNN, which may function as the CNN backbone 114. The backbone portion 400 implements a residual network with a feature pyramid network ("ResNet FPN") and is an alternative to the CNN backbone 114. Frame data 113 is input and passes through tensors 409, 413, 417, 425, through a stem network 408, a res2 module 412, a res3 module 416, a res4 module 420, a res5 module 424, and a max pooling module 428, which produces a P6 tensor 429 as an output.

[0095] The stem network 408 includes a 7x7 convolution with a stride of 2 and a max-pooling operation. The res2 module 412, the res3 module 416, the res4 module 420, and the res5 module 424 perform convolution operations with LeakyReLU activations. Each module 412, 416, 420, and 424 also performs half the resolution of the processed tensor via a stride setting of 2. Tensors 413, 417, 421, and 425 are passed to 1x1 horizontal convolution modules 446, 444, 442, and 440, respectively. Modules 440, 442, 444, and 446 produce tensors 441, 443, 445, and 447, respectively. Tensor 441 is passed to the 3x3 output convolution module 470, which produces output tensor P5 471. Tensor 441 is also passed to upsampler module 450 to produce upsampled tensor 451. Summation module 460 sums tensors 443 and 451 to produce tensor 461. Tensor 461 is passed to upsampler module 452 and 3x3 horizontal convolution module 472. Module 472 outputs P4 tensor 473. Upsampler module 452 produces upsampled tensor 453. Summation module 462 sums tensors 445 and 453 to produce tensor 463. Tensor 463 is passed to 3x3 horizontal convolution module 474 and upsampler module 454. Module 474 outputs P3 tensor 475. Upsampler module 454 outputs upsampled tensor 455. Sum module 464 sums tensors 447 and 455 to produce tensor 465, which is passed to a 3x3 horizontal convolution module 476. Module 476 outputs a P2 tensor 477. Upsampler modules 450, 452, and 454 use nearest neighbor interpolation for low computational complexity. Tensors 429, 471, 473, 475, and 477 form the output tensor 115 of the CNN backbone 400.In forming the output tenors 115, the FPN of tensors 429, 471, 473, 475, and 477 provides a hierarchical representation of the frame data, including the data of the feature maps, for encoding into the bitstream.

[0096] Figure 5 is a schematic block diagram illustrating one type of bottleneck encoder 500 that can function as the bottleneck encoder 116, or, if implemented with support for gradient backpropagation with weight updates, as a trainable bottleneck encoder 170. Figure 7A illustrates the packing arrangement of feature maps from compressed tensors into monochrome video frames.

[0097] 14 illustrates a method 1400 for reducing tensor dimensionality using the bottleneck encoder 500 of FIG. 5. The method 1400 may be performed using an apparatus such as a configured FPGA, ASIC, or ASSP. Alternatively, as described below, the method 1400 may be performed by the source device 110 as one or more software code modules of the application program 233 under execution of the processor 205. The software code modules of the application program 233 that implement the method 1400 may reside on the hard disk drive 210 and / or the memory 206, for example. The method 1400 is repeated for each frame of compressed data in the bitstream 123. The method 1400 may be stored on a computer-readable storage medium and / or the memory 206.

[0098] Method 1400 begins with a first FPN tensor selection step 1410. Bottleneck encoder 500 receives FPN tensor 501 corresponding to tensor 115 and operates according to method 1400 to limit the dimensionality of the received tensor to fewer layers, thereby reducing its spatial size. Applying a bottleneck encoder and decoder between the first (backbone 114) and second (head 152) portions of the separated neural network enables a spatial domain reduction in the frame of packed tensor data. The spatial domain reduction is achieved by using an interface between the bottleneck encoder and bottleneck decoder as the division point between the first and second portions (114 and 152) of the neural network. Bottleneck encoder 116 serves as an additional layer added to the first portion of the neural network, and bottleneck decoder 150 serves as an additional layer added to the beginning of the second portion of the neural network. Tensors 115 can be considered to include a first set of tensors (e.g., P2, P3, P4, and P5) and a second set of tensors (e.g., P2 and P3), where the feature maps of the second set of tensors are subsets of the first set of tensors. The feature maps of the second set of tensors include tensors with a spatial resolution greater than that of the feature maps of the first set of tensors. A first tensor (e.g., P2 or P3) that belongs to both the first set of tensors and the second set of tensors is represented in the base layer (first unit of information) and the enhancement layer (second unit of information). A second tensor (e.g., P4 or P5) that belongs to the first set of tensors but not the second set of tensors is represented in the base layer (first unit of information) but not in the enhancement layer (second unit of information). The first unit of information typically encodes all tensors in tensor 115, i.e., P2 through P5, and the second unit of information typically encodes a subset of the tensors encoded by the first unit of information (e.g., P2 and P3), the subset including tensors having greater feature map resolutions than the feature map resolutions of the P2 through P5 tensors. Other combinations of sets of tensors are possible.The second unit of information may, for example, encode only tensor P2, or may encode tensors P2-P4. The first unit of information may include only tensors P2-P4 with tensor P5 packed separately into frame 700 for encoding into bitstream 123. If the first unit of information includes tensors P2-P4, then the second unit of information may include tensor P2 or tensors P2 and P3.

[0099] The sensitivity of task results to bottlenecks depends on the nature of the task. For object detection, spatial sensitivity is lower, so spatial downsampling of larger layers of the FPN is not detrimental to the resulting mAP. Instance segmentation and the resulting segmentation maps are more sensitive to loss of spatial detail and therefore benefit from less severe spatial downsampling, especially for the spatially large tensors of the FPN. In the described configuration, the bottleneck encoder 500 operates at two scales. The first scale covers all FPN layers, and the second scale covers a subset of the FPN layers. The second scale can include larger layers, such as P2 and P3 (502 and 503), thus providing additional fidelity to these higher-resolution feature maps. The input FPN tensor 501 comprises layers P2 502, P3 503, P4 504, and P5 505. The spatial resolution of layers P2-P4 (502, 503, 504) is a power-of-two multiple of the spatial resolution of P5 505. If P5 505 has width and height (w, h), then P2-P4 (502, 503, 504) have dimensions (8w, 8h), (4w, 4h), and (2w, 2h), respectively. In other words, each tensor has a resolution that forms an exponential sequence where the width and height between successive tensors double. Layers P2-P5 each have 256 channels. While the base layer is described as required and the enhancement layer is optional (depending on circumstances such as fidelity), a configuration in which a second enhancement layer is included is possible. The second enhancement layer can be included only if an enhancement layer is included and the tensors of the second enhancement layer are a subset of the tensors of the first enhancement layer (as described above). For example, the tensors in the second subset may be for P2 only if the (first) enhancement layer relates to P2 and P3. In other words, the second enhancement layer provides further fidelity for certain tensors as already enhanced by the first enhancement layer. Cascaded enhancement layers provide further flexibility in providing incremental quality improvements by including tensors from each progressive enhancement layer in the packed frames.

[0100] In the above example, inputs P2 through P5 correspond to the hierarchical feature pyramid network outputs (P2 477, P3 475, P4 473, and P5 471) generated by the CNN backbone 400 of FIG. 4. When the CNN backbone is executed based on FIG. 3A and output tensors 329, 326, and 322, tensors 329, 326, and 322 are derived into two sets of tensors: the first group has one FPN layer 329, and the second group has two FPN layers 322 and 326. The first group with one FPN layer 329 has the lowest spatial resolution and does not require the operation of the MSFF module 510 in the bottleneck encoder 116; tensor 329 is passed directly to the SSFC encoder 550 as tensor 529. The second group is processed by MSFF 510 in bottleneck encoder 116, with tensor 326 passed as tensor 503 (input) and tensor 322 passed as tensor 502. In step 1410, bottleneck encoder 500, under execution of processor 205, selects a plurality of adjacent tensors among tensors 501 as a first plurality of tensors, such as four tensors P2 502, P3 503, P4 504, and P5 505. Tensors 501 form a hierarchical representation of frame data 113 resulting from application of FPN to frame data 113. Using a stride equal to two convolution stages in the FPN, i.e., modules 412, 416, 420, and 424, ordered according to decomposition level, e.g., P2 through P5, the spatial dimensions of the tensors between tensors 501 are half the width and height of each of the respective tensors. There is a degree of inter-layer correlation between layers P2-P5 of tensor 501, despite the layers having different spatial resolutions. Utilizing inter-layer correlation allows for a reduction in channel count for concatenation of tensors between layers, provided that the tensors are initially spatially scaled to the same resolution, e.g., the smallest resolution of the tensors being combined. Combining tensors of significantly different spatial resolutions typically results in a relatively high loss of detail in the higher-resolution tensor due to a higher ratio of downsampling operations.For example, scaling from P2 502 to P5 505 requires reducing the width and height to 1 / 8 of their previous values ​​due to the region reduction to 1 / 64 of the P2 502 region. For tasks that depend on spatial detail, such as instance segmentation, the mAP degrades. Such a reduction in mAP due to high downsampling of larger layers occurs for small object detection, where higher resolution layers are relied upon by the network head. Control in processor 205 passes from step 1410 to a first bottleneck tensor generation step 1420.

[0101] In step 1420, MSFF module 510 (see FIG. 5), under execution of processor 205, combines each of the first set of tensors, i.e., tensors 502, 503, 504, and 505, to generate combined tensor 529. Combined tensor 529 is encoded as compressed tensor 557. Combined tensor 529 forms the "base layer" representation of the FPN layer tensors. Downsample modules 522, 522a, and 522b operate on tensors with larger spatial scales, i.e., P2 502 at 8h, 8w, and 256, P3 503 at 4h, 4w, and 256, and P4 504 at 2h, 2w, and 256, respectively. Modules 522, 522a, and 522b downsample to match the spatial scale of P5 505 at a minimum tensor, i.e., h, w, 256, to generate downscaled P5 tensors 523, 523a, and 523b, respectively. Concatenation module 524 performs channel-wise concatenation of tensors 505, 523, 523a, and 523b to generate concatenated tensor 525 of dimensions h, w, 512. Concatenated tensor 525 is passed to squeeze and excitation (SE) module 526 to generate tensor 527. SE module 526 sequentially performs global pooling, a fully connected layer that reduces channel count, a rectified linear unit activation, a second fully connected layer that restores channel count, and a sigmoid activation function to generate the scaled tensor. Tensor 525 is scaled according to the scaling tensor to produce output as tensor 527. SE block 526 can be trained to adaptively change the weighting of different channels in the tensor passed through it based on the output of the first fully connected layer, which reduces each feature map for each channel to a single value. Each single value is then passed through a nonlinear activation unit (ReLU) to create a conditional representation of the unit appropriate for the weighting of the other channels, with restoration to the full channel count performed by the second fully connected layer.Thus, SE block 526 is able to extract nonlinear inter-channel correlations to a greater extent than would be possible using purely convolutional (linear) layers when generating tensor 527 from tensor 525. Because tensors 525 and 527 contain 512 channels, the decorrelation achieved by SE block 526 spans two FPN layers P5 and P4 as a result of the concatenation of the two FPN layers.

[0102] Tensor 527 is passed to convolutional layer 528, which runs one or more convolutional layers to produce a first combined tensor 529 with a reduced channel count of F channels, typically 256 channels (i.e., F=256).

[0103] The operation of the SSFC encoder 550 reduces the dimensionality of the combined tensor 529 to produce a compressed tensor 557. The combined tensor 529 is passed to a convolutional layer 552 of the encoder 550. The encoder 550 produces a tensor 553. The tensor 553 has a channel count reduced from 256 to a smaller value C', such as 64. The value 96 may also be used for C', resulting in a larger area requirement for the packed frame, as described with reference to FIG. 7A. The tensor 553 is passed to a batch normalization module 554 to produce a tensor 555. The batch normalized tensor 555 has the same dimensionality as the tensor 553. The tensor 555 is passed to a tanh layer 556. The tanh layer 556 performs a hyperbolic tangent (tanh) layer, like layer 536, to produce a compressed tensor 557. The compressed tensor 557 has the same dimensionality as the tensor 553. Step 1420 operates to derive or decode a first unit of information from tensor 501. Control in processor 205 passes from step 1420 to a second bottleneck tensor existence determination step 1430.

[0104] In step 1430, processor 205 determines whether to generate and encode a second set of encoding tensors, denoted as second bottleneck tensors 537. The decision in step 1430 may depend on at least one of the configuration of system 100 and the machine tasks to be completed in head network 152. For example, if system 100 is configured to perform a task requiring a high degree of spatial acuity, such as instance segmentation, a decision may be made to include an enhancement layer, allowing a higher mAP to be achieved by destination device 140. If system 100 is configured to perform a task requiring a lower degree of spatial acuity (or “regular quality”), such as object detection, a decision may be made to omit the enhancement layer, saving the bitrate cost of an additional layer. If system 100 is configured for “regular quality” operation, the second set of tensors is not generated, and system 100 is said to operate in 'first mode'. If the system 100 is configured for “high quality” operation, a second set of tensors is generated and the system 100 is configured to operate in a “second mode.” The decision to include the second set of tensors is made if the machine task performed by the destination device 140 requires relatively high retention of spatial detail (for example, instance segmentation). Control in the processor 205 proceeds from step 1420 to a second bottleneck tensor presence indication encoding step 1440. In some configurations of the system 100, a consumer of the task results 153, such as a human operator or an algorithm aggregating tasks from many networks, can determine the need for higher quality and signal an indication to the source device 110 to include (or omit) an enhancement layer via an out-of-band communication channel. One exemplary configuration involves the neural network head 152 of the destination device 140 performing general person detection and, upon signaling the person detection to the source device 110 to include an enhancement layer, running a more capable alternative network head as the network head 152.Alternate, more capable network heads may perform object detection tasks with higher specificity, such as identifying persons of interest.

[0105] In step 1440, entropy encoder 838, under execution of processor 205, encodes into bitstream 123 a flag indicating the decision made in step 1420 to operate in the first mode (base layer only) or the second mode (base layer and enhancement layer). The flag may be included in SEI message 744 as flag 751. Control in processor 205 passes from step 1430 to a second bottleneck tensor existence test step 1450.

[0106] In step 1450, software 233 determines whether to generate second bottleneck tensor 537. If a flag indicating that a second bottleneck tensor should be generated is present (“PRESENT” in step 1450), control of processor 205 passes from step 1450 to second FPN tensor selection step 1460. Otherwise, if the flag is not present (“ABSENT” in step 1450), method 1400 ends. The end of method 400 upon performance of step 1450 (“ABSENT” in step 1450) can be considered a first mode of operation of encoder 500. Proceeding from step 1450 to step 1460, the following steps can be considered a second mode of operation of encoder 500. Thus, step 1450 determines whether to operate in a first mode or a second mode based on at least one of a quality setting and a machine task to be completed. According to the above example, operation in the second mode is determined by determining whether the machine task to be completed is instance segmentation.

[0107] In step 1460, a second set of FPN tensors is selected. The second set of FPN tensors is a subset of the tensors selected as part of step 1410 and generally includes tensors having greater spatial resolution and adjacent spatial scales, e.g., P2 502 and P3 503. Switch 570 is activated to allocate the second set of FPN tensors to subsequent processing stages. In particular, tensor 502 is provided as 502a, and tensor 503 is provided as 503a upon closing switch 270. Control in processor 205 passes from step 1460 to a second bottleneck tensor generation step 1470.

[0108] In step 1470, the MSFF module 510, under execution by the processor 205, combines each tensor of the second tensor, i.e., 502a, 503a, as described with reference to FIG. 5, to generate a combined tensor 519. The combined tensor 519 forms a subset of the FPN layer tensors and provides an "augmented layer" representation. The downsample module 512 operates on the tensor having a larger spatial scale, i.e., P2 502 at 8h, 8w, 256, and downsamples it to match the spatial scale of the smaller tensor, i.e., P3 503 at 4h, 4w, 256, to generate a downscaled P2 tensor 513. The concatenation module 514 performs a channel-wise concatenation of tensors 503a and 513 to generate a concatenated tensor 515. The tensor 515 has dimensions 4h, 4w, 512. The concatenated tensor 515 is passed to a squeeze and excitation (SE) module 516 to generate a tensor 517. The SE module 516 operates in the same manner as described with reference to the SE module 526. The tensor 517 is passed to a convolutional layer 518. The convolutional layer 518 operates similarly to the convolutional layer 528 to generate a second concatenated tensor 519. The second concatenated tensor 519 has a channel count reduced to F channels, typically 256 channels (i.e., F=256). As a result of modules 512, 514, 516, and 518, the tensors of the two FPN layers are reduced to a single tensor with the same channel count as the input FPN layer tensor and the smaller spatial resolution of the two FPN layer tensors. Dimensionality reduction is achieved using several network layers and relies on training layers (e.g., layers 516 and 518) rather than on-the-fly determining correlations to exploit.

[0109] The SSFC encoder 530 operates to further reduce the dimensionality of the combined tensor 519 to produce a compressed tensor 537. The combined tensor 519 is passed to a convolutional layer 532 to produce a tensor 533. The tensor 533 has a channel count reduced from 256 to a smaller value C', such as 64. The tensor 533 is passed to a batch normalization module 534 to produce a tensor 535. The batch normalized tensor 535 has the same dimensionality as the tensor 533. The tensor 535 is passed to a tanh layer 536 to produce a compressed tensor 537. The compressed tensor 537 has the same dimensionality as the tensor 533. The use of a hyperbolic tangent (tanh) layer compresses the dynamic range of the values ​​in the tensor 537 to [-1, 1] and removes outliers. Layers 532, 534, and 536 operate similarly to layers 552, 554, and 556, respectively.

[0110] Compressed tensors 557 and 537 provide units of information for feature maps of frame data 113 obtained using convolution operations of (i) MSFF 510 and SSFC encoder 550 on tensors 505, 504, 503, and 502, and (if present) (ii) MSFF 510 and SSFC encoder 530 on tensors 503 and 502. Compressed tensors 557 and 537 provide a set of tensors 560 corresponding to bottleneck encoded tensors 117. When encoder 500 is operating in the first mode (“ABSENT” at 1450, switch 570 closed), tensor 557 provides tensor 560. Tensor 560 is provided to quantize and pack module 184 as tensor 117 for encoding into a bitstream by video encoder 120.

[0111] In the configuration of bottleneck encoder 500, tanh modules 536 and 556 are omitted, and tensors 535 and 555 are passed as tensors 537 and 557, respectively. In other words, in the configuration omitting 536, output 537 relates to the combined tensor 519 applied to convolutional layer 532 and batch normalization layer. In the configuration omitting 556, output 557 relates to the combined tensor 529 applied to convolutional layer 532 and batch normalization layer. Omitting the tanh modules results in the preservation of outliers or values ​​with large magnitude, which experiments found to result in a disproportionate contribution to the final task performance in head 152.

[0112] Figure 7A shows a frame 700 which is an example of a single monochrome video frame. Frame 700 corresponds to the packed and quantized feature map data 185. The nature of tanh when removing outliers results in a distribution suitable for linear quantization with respect to the bit depth of frame 700. The channels of the compressed tensor 557 are packed as feature maps of a specific size, such as the feature map 712 within region 714 of frame 700. The channels of the compressed tensor 537 (if present as determined in step 1430) are packed as feature maps of a different size, such as the feature map 710, in region 716 of frame 700. One channel of the compressed tensor 537 corresponds to one feature map represented by one rectangular region such as region 710. One channel of the compressed tensor 557 corresponds to one feature map represented by one rectangular region such as region 712. Regions 714 and 716 (if present) form the packed representations of tensors 557 and 537, which, when compressed by the video encoder 120, form the first unit of information and the second unit of information, respectively. The first and second units of information may be stored in a way that allows independent encoding and decoding, such as using separate slices, tiles, or sub-pictures of frame 700, or completely separate pictures. The video decoder 148 may simply decode region 714, i.e., the first unit of information, and discard region 716, i.e., the second unit of information, and still provide the tensor 151 to the CNN head 152 to generate a task result, although the achievable fidelity is lower than if region 716 or the second unit of information were decoded.

[0113] Figure 7B is a schematic block diagram showing bitstream 723, which may be part of bitstream 123, encoding tensor data. Compressed frame n 742 includes compressed tensors arranged as described with reference to Figure 7A. SEI message 744 includes weight update flag 750 and, if indicated by weight update flag 750, neural network weights 752. Compressed frame N+1 746 includes compressed tensors that use weights derived from neural network weights 752. The presence of a second bottleneck tensor (such as 537) may be encoded in SEI message 744 as presence flag 751.

[0114] FIG. 8 is a schematic block diagram illustrating the functional modules of video encoder 120, also referred to as a feature map encoder. Video encoder 120 encodes packed feature map frame 185, shown as frame 700 in the example of FIG. 7A, to generate video bitstream 121. Generally, data passes between functional modules within video encoder 120 in groups of samples or coefficients, such as dividing a block into fixed-size sub-blocks, or as an array. Video encoder 120 may be implemented using a general-purpose computer system 200, as shown in FIGS. 2A and 2B. The various functional modules may be implemented by dedicated hardware within computer system 200, or by software executable within computer system 200, such as one or more software code modules of a software application program 233 residing on hard disk drive 205 and controlled in its execution by processor 205. Alternatively, video encoder 120 may be implemented by a combination of dedicated hardware and software executable within computer system 200. Video encoder 120 and the described methods may alternatively be implemented with dedicated hardware, such as one or more integrated circuits performing functions or sub-functions of the described methods. Such dedicated hardware may include a graphics processing unit (GPU), a digital signal processor (DSP), an application specific standard product (ASSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or one or more microprocessors and associated memory. In particular, video encoder 120 includes modules 810-890, each of which may be implemented as one or more software code modules of software application program 233.

[0115] 8 is an example of a generic video coding (VVC) video encoding pipeline, other video coding standards or embodiments may use the processing stages of modules 810-890. The frame data 185 (and bitstream 121) may also be stored in memory 206, a hard disk drive 210, a CD-ROM, a Blu-ray disc TM or other computer-readable storage medium. Additionally, frame data 185 (and bitstream 121) may be received from (or transmitted to) an external source, such as a server connected to a communications network 220 or a radio frequency receiver. Communications network 220 may provide limited bandwidth, necessitating the use of rate control in video encoder 120 to avoid saturating the network when it is difficult to compress frame data 185. Frame data 185 may be in any chroma format and bit depth supported by the profile in use, e.g., 4:0:0, 4:2:0 for the "Main 10" profile of the VVC standard, with 8-10 bits of sample precision.

[0116] The block partitioner 710 first divides the frame data 185 into CTUs that are approximately square in shape and configured to use a specific size for the CTU. The maximum valid size of a CTU can be, for example, 32x32, 64x64, or 128x128 luma samples, configured by the "sps_log2_ctu_size_minus5" syntax element present in the "sequence parameter set." The CTU size also provides the maximum CU size, since a CTU without further division contains one CU. The block partitioner 810 further divides each CTU into one or more CBs according to the luma coding tree and the chroma coding tree. The luma channel is sometimes referred to as the primary color channel. Each chroma channel may also be referred to as a secondary color channel. CBs have various sizes and can include both square and non-square aspect ratios. However, in the VVC standard, CBs, CUs, PUs, and TUs always have side lengths that are powers of two. Thus, a current CB, denoted as 812, is output from block partitioner 810, which proceeds according to iterations over one or more blocks of the CTU according to the luma coding tree and chroma coding tree of the CTU.

[0117] The CTUs resulting from the initial partitioning of frame data 185 may be scanned in raster scan order and grouped into one or more "slices." A slice may be an "intra" (or "I") slice. An intra slice (I slice) indicates that all CUs in the slice are intra-predicted. Generally, the first picture in a coded layer video sequence (CLVS) contains only I slices and is called an "intra picture." A CLVS may include periodic intra pictures that form "random access points" (i.e., intermediate frames in a video sequence where decoding can begin). Alternatively, a slice may be uni-predictive or bi-predictive ("P" or "B" slices, respectively), indicating the additional availability of uni-prediction and bi-prediction in the slice, respectively.

[0118] Video encoder 120 encodes a sequence of pictures according to a picture structure. One picture structure is "low latency," in which pictures using inter prediction may only reference pictures that occur earlier in the sequence. Low latency allows each picture to be output as soon as it is decoded, in addition to being stored for possible reference by subsequent pictures. Another picture structure is "random access," in which the coding order of pictures differs from the display order. Random access allows inter-predicted pictures to reference other pictures that have been decoded but not yet output. A degree of picture buffering is required so that future reference pictures with respect to display order are present in the decoded picture buffer, resulting in a delay of multiple frames.

[0119] When a chroma format other than 4:0:0 is used, in an I slice, the coding tree of each CTU may diverge below the 64x64 level into two separate coding trees, one for luma and the other for chroma. The use of separate trees allows for different block structures between luma and chroma within the luma 64x64 area of ​​a CTU. For example, a large chroma CB may be co-located with multiple smaller luma CBs, and vice versa. In a P or B slice, a single coding tree for a CTU defines a common block structure for luma and chroma. The resulting blocks of the single tree may be intra-predicted or inter-predicted.

[0120] In addition to dividing a picture into slices, a picture may also be divided into "tiles." A tile is a sequence of CTUs that cover a rectangular area of ​​the picture. CTU scanning occurs in a raster-scan fashion within each tile, progressing from one tile to the next. A slice can be either an integer number of tiles or an integer number of consecutive rows of CTUs within a given tile.

[0121] For each CTU, video encoder 120 operates in two stages. In the first stage (called the “search” stage), block partitioner 810 tests various potential configurations of the coding tree. Each potential configuration of the coding tree has an associated “candidate” CB. The first stage involves testing various candidate CBs to select a CB that provides relatively high compression efficiency with relatively low distortion. The test stage generally involves Lagrangian optimization, whereby candidate CBs are evaluated based on a weighted combination of rate (i.e., coding cost) and distortion (i.e., error relative to input frame data 185). The “best” candidate CB (i.e., the CB with the lowest evaluated rate / distortion) is selected for subsequent encoding into bitstream 121. Included in the evaluation of candidate CBs are the options of using the CB for a given region, or further dividing the region according to various partitioning options and coding each of the smaller resulting regions with additional CBs, or further dividing the region. Consequently, both the coding tree and the CB itself are selected in the search stage.

[0122] For each CB, e.g., CB 812, video encoder 120 generates a predictive block (PB), indicated by arrow 820. PB 820 is a prediction of the content of the associated CB 812. Subtractor module 822 generates a difference (or "residual," which refers to the difference in the spatial domain) between PB 820 and CB 812, indicated as 824. Difference 824 is the block-sized difference between corresponding samples in PB 820 and CB 812. Difference 824 is transformed and quantized and represented as a transform block (TB), indicated by arrow 836. PB 820 and associated TB 836 are typically selected from one of many possible candidate CBs, for example, based on estimated cost or distortion.

[0123] A candidate coding block (CB) is a CB that results from one of the prediction modes available to video encoder 120 for the associated PB and the resulting residual. When combined with the predicted PB in video encoder 120, the TB 836 reduces the difference between the decoded CB and the original CB 812, at the expense of additional signaling in the bitstream.

[0124] Thus, each candidate coding block (CB), i.e., a prediction block (PB) combined with a transform block (TB), has an associated coding cost (or “rate”) and an associated differential (or “distortion”). The distortion of a CB is typically estimated as a difference between sample values, such as the sum of absolute differences (SAD), the sum of squared differences (SSD), or a Hadamard transform applied to the difference. The resulting estimate from each candidate PB can be determined by a mode selector 886 using the differentials 824 to determine a prediction mode 887. The prediction mode 887 indicates a decision to use a particular prediction mode for the current CB, e.g., intra-frame prediction or inter-frame prediction. Estimating the coding cost associated with each candidate prediction mode and the corresponding residual coding can be performed at a significantly lower cost than entropy coding of the residual. Thus, several candidate modes can be evaluated to determine the optimal mode for rate-distortion detection, even in a real-time video encoder.

[0125] Determining the optimal mode in terms of rate distortion is typically accomplished using a variation of Lagrangian optimization.

[0126] A Lagrangian or similar optimization process may be used both to select the optimal partitioning of the CTUs into CBs (by the block partitioner 810) as well as to select the best prediction mode from multiple possibilities. Through application of a Lagrangian optimization process of the candidate modes in the mode selection module 886, the intra-prediction mode with the lowest cost measure is selected as the "best" mode. The lowest-cost mode includes a selected secondary transform index 888, which is also encoded in the bitstream 121 by the entropy encoder 838.

[0127] In the second stage of operation of the video encoder 120 (called the "coding" stage), iterations over the determined coding trees for each CTU are performed in the video encoder 120. For CTUs that use separate trees, for each 64x64 luma region of the CTU, the luma coding tree is coded first, followed by the chroma coding tree. Only the luma CB is coded within the luma coding tree, and only the chroma CB is coded within the chroma coding tree. For CTUs that use a shared tree, a single tree describes the CUs (i.e., the luma CBs and chroma CBs) according to the common block structure of the shared tree.

[0128] The entropy encoder 838 supports bitwise coding of syntax elements using variable-length and fixed-length codewords, as well as arithmetic coding modes for syntax elements. Portions of the bitstream, such as "parameter sets," such as the sequence parameter set (SPS) and picture parameter set (PPS), use a combination of fixed-length and variable-length codewords. Slices, also called contiguous portions, have a slice header that uses variable-length coding, followed by slice data that uses arithmetic coding. The slice header defines parameters specific to the current slice, such as slice-level quantization parameter offsets. For a given slice, the slice data includes syntax elements for each CTU in the slice. The use of variable-length coding and arithmetic coding requires sequential parsing within each portion of the bitstream. Portions may be delimited by start codes to form "network abstraction layer units" or "NAL units." Arithmetic coding is supported using a context-adaptive binary arithmetic coding process.

[0129] Arithmetically coded syntax elements consist of a sequence of one or more "bins." Bins, like bits, have a value of "0" or "1." However, bins are not coded as discrete bits in the bitstream 121. Bins have an associated predicted (or "likely" or "most likely") value and an associated probability, known as a "context." If the actual bin to be coded matches the predicted value, a "most probable symbol" (MPS) is coded. Coding the most probable symbol is relatively inexpensive in terms of consumed bits in the bitstream 121 and involves a cost equivalent to less than one discrete bit. If the actual bin to be coded does not match a possible value, a "least probable symbol" (LPS) is coded. Coding the least probable symbol has a relatively high cost in terms of consumed bits. Bin coding techniques allow for efficient coding of bins where the probability of "0" versus "1" is skewed. For syntax elements with two possible values ​​(i.e., "flags"), a single bin is appropriate. For syntax elements with many possible values, a sequence of bins is required.

[0130] The decomposition of a syntax element's value into a sequence of one or more bins is called "binarization" of the syntax element. Binarization may involve the conditional existence of later bins on the values ​​of earlier bins, allowing for variable bin-length binarization. Furthermore, each bin may be associated with two or more contexts. The selection of a context for a bin is called "context modeling." Context modeling may depend on previous bins in the syntax element, bin values ​​of adjacent syntax elements (i.e., from adjacent blocks), etc. Each time a context-coded bin is coded, the context selected for that bin (if any) is updated in a manner that reflects the new bin values. In this way, binary arithmetic coding schemes are said to be adaptive.

[0131] Also supported by the entropy encoder 838 are bins lacking context, called "bypass bins." Bypass bins are coded with an equal probability distribution between "0" and "1." Thus, each bin has a coding cost of one bit in the bitstream 121 and is typically used when there is no statistical skew (or no easily exploited statistical skew) in the probability distribution of bin values. The lack of context saves memory and reduces complexity, and thus bypass bins are used when the distribution of values ​​for a particular bin is not skewed. One example of an entropy coder that uses context and adaptation is known in the art as CABAC (context-adaptive binary arithmetic coder), and many variations of this coder are used in video coding.

[0132] The entropy encoder 838 uses a combination of context-coded and bypass-coded bins to encode a quantization parameter 892 and, if used for the current CB, a secondary transform index 888. The quantization parameter 892 is encoded using a "delta QP" generated by the QP controller module 890. The delta QP is signaled at most once in each area known as a "quantization group." The quantization parameter 892 is applied to the residual coefficients of the luma CB. An adjusted quantization parameter is applied to the residual coefficients of the co-located chroma CB. The adjusted quantization parameter may include mapping from the luma quantization parameter 892 according to a mapping table and CU level offset selected from a list of offsets. The secondary transform index 888 is signaled if the residual associated with the transform block contains significant residual coefficients only at coefficient positions that are transformed into primary coefficients by application of a secondary transform.

[0133] The residual coefficients of each TB associated with a CB are coded using a residual syntax. The residual syntax is designed to efficiently code low-magnitude coefficients by using primarily arithmetically coded bins to indicate coefficient significance and reserving bypass bins for higher-magnitude residual coefficients, along with lower magnitude values. Thus, residual blocks containing very low magnitude values ​​and a sparse distribution of significant coefficients are efficiently compressed. Furthermore, two residual coding schemes exist. The regular residual coding scheme is optimized for TBs with significant coefficients located primarily in the upper-left corner of the TB, as occurs when a transform is applied. The transform-skip residual coding scheme is available for TBs where no transform is performed and allows for efficient coding of residual coefficients regardless of their distribution across the TB.

[0134] The multiplexer module 884 outputs the PB 820 from the intra-frame prediction module 864 according to a determined best intra-prediction mode selected from the tested prediction modes of each candidate CB. The candidate prediction modes need not include all possible prediction modes supported by the video encoder 120. Intra-prediction is divided into three types: first, "DC intra-prediction," which involves populating the PB with a single value representing the average of nearby reconstructed samples; second, "planar intra-prediction," which involves populating the PB with samples according to a plane, where the DC offset and vertical and horizontal gradients are derived from nearby reconstructed neighboring samples. The neighboring reconstructed samples typically include a row of reconstructed samples above the current PB that extends to the right of the PB and a column of reconstructed samples to the left of the current PB that extends downward beyond the PB; and third, "angular intra-prediction," which involves populating the PB with reconstructed neighboring samples that have been filtered and propagated across the PB in a specific direction (or "angle"). In VVC, 65 angles are supported, and rectangular blocks can utilize additional angles not available to square blocks, yielding a total of 87 angles.

[0135] A fourth type of intra prediction is available for chroma PBs, whereby the PB is generated from co-located luma reconstructed samples according to a "cross-component linear model" (CCLM) mode. Three different CCLM modes are available, each using a different model derived from neighboring luma and chroma samples. The derived model is used to generate a block of samples for the chroma PB from the co-located luma samples. The luma block can be intra-predicted using matrix multiplication of a reference sample using one matrix selected from a predefined set of matrices. This matrix intra prediction (MIP) achieves gains by using a matrix trained on a large set of video data, where the matrix represents a relationship between the predicted block and the reference sample that is not easily captured in angular, planar, or DC intra prediction modes.

[0136] The module 864 may also generate prediction units by copying blocks from nearby current frames using the "intra block copy" (IBC) method. The location of the reference blocks is constrained to an area equivalent to one CTU. For a 128x128 CTU, a division into 64x64 quadrants, sometimes called "Virtual Pipeline Data Units" (VPDUs), is performed. The referenceable area includes VPDUs in the current CTU with all CUs decoded and VPDUs in the previous CTU (unless the current CTU is the first in a slice, tile, or subpicture), up to a total area of ​​128x128 luma samples. This area, known as the "IBC virtual buffer," limits the IBC reference area and therefore the required storage. The IBC buffer is populated with reconstructed samples 854 (i.e., before loop filtering), and therefore requires a buffer separate from the frame buffer 872. For a 128x128 CTU size, the virtual buffer contains only samples from the CTU adjacent to the left of the current CTU. When the CTU size is 32x32 or 64x64, the virtual buffer contains up to 4 or 16 CTUs to the left of the current CTU. Regardless of the CTU size, access to neighboring CTUs to obtain samples for IBC reference blocks is constrained by boundaries such as picture, slice, or tile edges. In particular, for feature maps in FPN layers with smaller dimensions, using a CTU size such as 32x32 or 64x64 results in a more aligned reference region that covers the previous set of feature maps. When feature map arrangements are ordered based on SAD, SSE, or other difference metrics, accessing similar feature maps for IBC prediction provides coding efficiency benefits.

[0137] Residuals of predicted blocks when encoding feature map data differ from those found for natural video. Such natural video is typically captured by an image sensor or screen content, such as that commonly found in operating system user interfaces. Feature map residuals tend to contain many details, which are more suitable for skip coding than the predominantly low-frequency coefficients of various transforms. Experiments show that feature map residuals have sufficient local similarity to benefit from transform coding. However, the distribution of feature map residual coefficients is not clustered toward the DC (top-left) coefficient of the transform block. In other words, there is sufficient correlation to demonstrate a gain for transforms when encoding feature map data, and this also holds true when intra-block copying is used to generate predicted blocks of feature map data. Therefore, Hadamard cost estimation can be used when evaluating residuals resulting from candidate block vectors for intra-block copying when encoding feature map data, rather than relying solely on SAD or SSD cost estimation. SAD or SSD cost estimation tends to select block vectors with residuals more suitable for transform skip coding and may miss block vectors with residuals that are compactly coded using transforms. The Multiple Transform Selection (MTS) tool of the VVC standard can be used when encoding feature map data, so that in addition to the DCT-2 transform, a combination of DCT-7 and DCT-8 transforms are available horizontally and vertically for residual coding.

[0138] An intra-predicted luma coding block may be divided either vertically or horizontally into a set of equally sized prediction blocks, each with a minimum area of ​​16 luma samples. This intra-subpartition (ISP) approach allows separate transform blocks to contribute to the prediction block generation from one subpartition to the next in the luma coding block, improving compression efficiency.

[0139] If no previously reconstructed neighboring samples are available, e.g., at the edge of a frame, a default halftone value of half the range of samples is used. For example, for 10-bit video, a value of 512 is used. Because no previous samples are available for the CB located in the upper left position of the frame, angular and planar intra prediction modes produce the same output as DC prediction mode (i.e., a plane of samples with halftone values ​​as magnitudes).

[0140] For inter-frame prediction, the prediction block 882 is generated by the motion compensation module 880 using samples from one or two frames preceding the current frame in the coding order in the bitstream and output as the PB 820 by the multiplexer module 884. Furthermore, for inter-frame prediction, a single coding tree is typically used for both the luma and chroma channels. The order in which frames are coded in the bitstream may differ from the order of the frames when captured or displayed. If one frame is used for prediction, the block is said to be "uni-predictive" and has one associated motion vector. If two frames are used for prediction, the block is said to be "bi-predictive" and has two associated motion vectors. For P slices, each CU can be intra-predicted or uni-predicted. For B slices, each CU can be intra-predicted, uni-predicted, or bi-predicted.

[0141] Frames are typically coded using a "group of pictures" (GOP) structure, allowing for temporal hierarchy of frames. A frame may be divided into multiple slices, each of which encodes a portion of the frame. The temporal hierarchy of frames allows frames to reference previous and subsequent pictures in the order in which they are displayed. Images are coded in the order necessary to ensure that dependencies for decoding each frame are satisfied. Instead of using one or two motion vectors to select and filter reference sample blocks for a prediction unit, the prediction unit is divided into multiple smaller blocks, a motion field is generated, and an affine inter-prediction mode is available, in which each smaller block has a separate motion vector. The motion field uses motion vectors of points near the prediction unit as "control points." Affine prediction allows for motion coding that is different from transforms, with less need for deeply partitioned coding trees. The bi-prediction mode available for VVC performs a geometric blend of two reference blocks along a selected axis with an offset and angle from the center of the signaled block. This geometric partitioning mode ("GPM") allows for the use of larger coding units along the boundary between two objects, with the boundary geometry coded for the coding unit as an angle and center offset. Instead of using Cartesian (x, y) offsets, motion vector differences can be coded as direction (up / down / left / right) and distance, using a set of supported power-of-two distances. The motion vector predictor is obtained from neighboring blocks ("merge mode") as if no offset was applied. The current block shares the same motion vector with the selected neighboring block.

[0142] Samples are selected according to a motion vector 878 and a reference picture index. The motion vector 878 and reference picture index apply to all color channels; therefore, inter prediction is primarily described in terms of operation on the PU rather than the PB. The decomposition of each CTU into one or more inter prediction blocks is described using a single coding tree. Inter prediction methods may vary in the number of motion parameters and their precision. Motion parameters typically include a reference frame index indicating which reference frame from a list of reference frames should be used and a spatial transformation for each of the reference frames, but may also include more frames, dedicated frames, or complex affine parameters such as scaling and rotation. In addition, a predetermined motion refinement process may be applied to generate a dense motion estimate based on the referenced sample block.

[0143] Determining and selecting a PB 820 and subtracting it from the original sample block in a subtractor 822 results in a residual with the lowest coding cost, represented as 824, which undergoes lossy compression. The lossy compression results from the quantization process of the coefficients produced by the forward transform into residual coefficients ready to be entropy coded into a bitstream. A forward primary transform module 826 applies a forward transform to the difference 824, converting it from the spatial domain to the frequency domain and producing primary transform coefficients, represented by arrow 828. The maximum primary transform size in one dimension is either a 32-point DCT-2 or a 64-point DCT-2 transform, configured by the "sps_max_luma_transform_size_64_flag" in the sequence parameter set. If the CB being coded is larger than the maximum supported primary transform size, represented as a block size (e.g., 64x64 or 32x32), the primary transform 826 is applied in a tiled manner to transform all samples of the difference 824. When non-square CBs are used, tiling is also performed using the largest available transform size in each dimension of the CB. For example, if a maximum transform size of 32 is used, a 64x16 CB uses two 32x16 linear transforms arranged in a tiled fashion. If the CB is larger than the maximum supported transform size, the CB is filled with TBs in a tiled fashion. For example, a 128x128 CB with a 64-pt maximum transform size is filled with four 64x64 TBs arranged in a 2x2 configuration. A 64x128 CB with a 32-pt maximum transform size is filled with eight 32x32 TBs arranged in a 2x4 configuration.

[0144] Application of the transform 826 results in multiple TBs for the CB. If each application of the transform operates on a TB of the difference 824 larger than 32x32, e.g., 64x64, all resulting primary transform coefficients 828 outside the top-left 32x32 region of the TB are set to zero (i.e., discarded). The remaining primary transform coefficients 828 are passed to a quantizer module 834. The primary transform coefficients 828 are quantized according to a quantization parameter 892 associated with the CB to generate primary transform coefficients 832. In addition to the quantization parameter 892, the quantizer module 834 may also apply a "scaling list" to enable non-uniform quantization within the TB by further scaling the residual coefficients according to their spatial position within the TB. The quantization parameter 892 may be different for the luma CB versus each chroma CB. The primary transform coefficients 832 are passed to a forward secondary transform module 830, which generates transform coefficients represented by arrow 836 by performing a non-separable secondary transform (NSST) operation or by bypassing the secondary transform. The forward primary transform is typically separable, transforming a set of rows and then a set of columns for each TB. The forward primary transform module 826 uses either a Type II Discrete Cosine Transform (DCT-2) in the horizontal and vertical directions, or bypassing the horizontal and vertical transforms, or a combination of a Type VII Discrete Sine Transform (DST-7) and a Type VIII Discrete Cosine Transform (DCT-8) in the horizontal or vertical directions for luma TBs whose width and height do not exceed 16 samples. The use of a combination of DST-7 and DCT-8 is referred to in the VVC standard as a "Multiple Transform Selection Set" (MTS).

[0145] The forward secondary transform of module 830 is generally a non-separable transform, which is applied only to the residual of intra-predicted CUs and may nevertheless be bypassed. The forward secondary transform operates on either 16 samples (organized as a 4x4 sub-block at the top left of the primary transform coefficients 828) or 48 samples (organized as three 4x4 sub-blocks at the top left of the primary transform coefficients 828) to generate a set of secondary transform coefficients. The set of secondary transform coefficients may be fewer in number than the set of primary transform coefficients from which they are derived. By applying the secondary transform only to sets of coefficients that are adjacent to each other and include a DC coefficient, the secondary transform is referred to as a "low-frequency non-separable secondary transform" (LFNST). Such secondary transforms may be obtained through a training process and by virtue of their non-separable nature and trained origin, and can exploit additional redundancy in the residual signal that cannot be captured by separable transforms such as DCT and DST variants applied horizontally and vertically. Furthermore, when LFNST is applied, all remaining coefficients in TB are zero in both the primary and secondary transform domains.

[0146] The quantization parameter 892 is constant for a given TB, thus resulting in uniform scaling for the generation of residual coefficients in the primary transform domain for the TB. The quantization parameter 892 may vary periodically with the signaled "delta quantization parameter." The delta quantization parameter (delta QP) is signaled once for CUs contained within a given area, called a "quantization group." If the CU is larger than the quantization group size, the delta QP is signaled once with one of the TBs of the CU. That is, the delta QP is signaled by the entropy encoder 838 once for the first quantization group of the CU and not for any subsequent quantization groups of the CU. Non-uniform scaling is also possible by applying a "quantization matrix," whereby the scaling factor applied to each residual coefficient is derived from a combination of the quantization parameter 892 and a corresponding entry in the scaling matrix. The scaling matrix may have a size smaller than the TB size, and when applied to the TB, a nearest-neighbor approach is used to provide a scaling value for each residual coefficient from a scaling matrix of a size smaller than the TB size. The residual coefficients 836 are provided to an entropy encoder 838 for encoding in the bitstream 121. Typically, the residual coefficients of each TB having at least one significant residual coefficient of a TU are scanned to generate an ordered list of values ​​according to a scan pattern. The scan pattern generally scans the TB as a sequence of 4x4 "subblocks," providing a regular scan operation with a granularity of 4x4 sets of residual coefficients, with the arrangement of the subblocks depending on the size of the TB. The scan within each subblock and the progression from one subblock to the next typically follows a backward diagonal scan pattern. Furthermore, a quantization parameter 892 is coded into the bitstream 121 using a delta QP syntax element, and an initial slice QP for a given slice or subpicture and secondary transform index 888 is coded into the bitstream 121.

[0147] As described above, video encoder 120 needs access to a frame representation that corresponds to the decoded frame representation seen by the video decoder. Accordingly, the residual coefficients 836 pass through an inverse secondary transform module 844, operating according to secondary transform indices 888 to generate intermediate inverse transform coefficients represented by arrow 842. The intermediate inverse transform coefficients 842 are inverse quantized by an inverse quantizer module 840 according to a quantization parameter 892 to generate inverse transform coefficients represented by arrow 846. The inverse quantizer module 840 may also perform inverse non-uniform scaling of the residual coefficients using a scaling list, corresponding to the forward scaling performed in the quantizer module 834. The inverse transform coefficients 846 are passed to an inverse primary transform module 848 to generate residual samples of the TUs, represented by arrow 850. The inverse primary transform module 848 applies a DCT-2 transform horizontally and vertically, constrained by the maximum available transform size, as described with reference to the forward primary transform module 826. The type of inverse transform performed by the inverse secondary transform module 844 corresponds to the type of forward transform performed by the forward secondary transform module 830. The type of inverse transform performed by the inverse primary transform module 848 corresponds to the type of primary transform performed by the primary transform module 826. The summation module 852 sums the residual samples 850 and the PU 820 to generate reconstructed samples for the CU (indicated by arrow 854).

[0148] The reconstructed samples 854 are passed to a reference sample cache 856 and an in-loop filter module 868. The reference sample cache 856, typically implemented using static RAM on the ASIC to avoid costly off-chip memory accesses, provides the minimum sample storage required to satisfy the dependencies for generating intra-frame PBs for subsequent CUs in the frame. The minimum dependencies typically include a "line buffer" of samples along the bottom of a row of CTUs for use by the next row of CTUs, and column buffering whose extent is set by the height of the CTU. The reference sample cache 856 provides reference samples (represented by arrow 858) to a reference sample filter 860. The sample filter 860 applies a smoothing operation to generate filtered reference samples (indicated by arrow 862). The filtered reference samples 862 are used by an intra-frame prediction module 864 to generate intra-predicted blocks of samples, represented by arrow 866. For each candidate intra-prediction mode, intra-frame prediction module 864 generates a block of samples, i.e., 866. The block of samples 866 is generated by module 864 using techniques such as DC, planar, or angular intra prediction. The block of samples 866 may also be generated using a matrix multiplication approach with a matrix selected by video encoder 120 from a set of matrices and neighboring reference samples as input, where the selected matrix is ​​signaled to bitstream 121 using an index to identify which matrix from the set of matrices should be used by video decoder 144.

[0149] The in-loop filter module 868 applies several filtering stages to the reconstructed samples 854. The filtering stages include a "deblocking filter" (DBF), which applies smoothing aligned to CU boundaries to reduce artifacts due to discontinuities. Another filtering stage present in the in-loop filter module 868 is an "adaptive loop filter" (ALF), which applies a Wiener-based adaptive filter to further reduce distortion. A further filtering stage available in the in-loop filter module 868 is a "sample adaptive offset" (SAO) filter. The SAO filter operates by first classifying the reconstructed samples into one or more categories and then applying an offset at the sample level according to the assigned category.

[0150] The filtered samples, represented by arrow 870, are output from the in-loop filter module 868. The filtered samples 870 are stored in a frame buffer 872. The frame buffer 872 typically has the capacity to store several pictures (e.g., up to 16) and is therefore stored in the memory 206. The frame buffer 872 is typically not stored using on-chip memory due to the large memory consumption required. Access to the frame buffer 872 is therefore costly in terms of memory bandwidth. The frame buffer 872 provides reference frames (represented by arrow 874) to a motion estimation module 876 and a motion compensation module 880.

[0151] The motion estimation module 876 estimates several “motion vectors” (denoted as 878), each of which is a Cartesian spatial offset from the position of the current CB and references a block in one of the reference frames in the frame buffer 872. A filtered block of reference samples (denoted as 882) is generated for each motion vector. The filtered reference samples 882 form further candidate modes available for potential selection by the mode selector 886. Furthermore, for a given CU, the PU 820 may be formed using one reference block (“uni-predicted”) or two reference blocks (“bi-predicted”). For a selected motion vector, the motion compensation module 880 generates the PB 820 according to a filtering process that supports sub-pixel accuracy in motion vectors. Thus, the motion estimation module 876 (which operates on many candidate motion vectors) can perform a simplified filtering process compared to that of the motion compensation module 880 (which operates only on the selected candidate), achieving reduced computational complexity. If video encoder 120 selects inter prediction for a CU, motion vector 878 is encoded into bitstream 121. Figure 9 shows video decoder 146, also referred to as a feature map decoder. Video decoder 146 of Figure 9 is an example of a generic video coding (VVC) video decoding pipeline, although other video codecs may be used to perform the processing stages described herein. As shown in Figure 9, bitstream 143 is input to video decoder 146. Bitstream 143 may be stored in memory 206, hard disk drive 210, a CD-ROM, a Blu-ray disc, or any other storage device. TM or other non-transitory computer-readable storage medium. Alternatively, bitstream 143 may be received from an external source, such as a server connected to communications network 220 or a radio frequency receiver. Bitstream 143 includes encoded syntax elements representing captured frame data to be decoded.

[0152] The bitstream 143 is input to the entropy decoder module 920. The entropy decoder module 920 extracts syntax elements from the bitstream 143 by decoding a sequence of "bins" and passes the values ​​of the syntax elements to other modules in the video decoder 146. The entropy decoder module 920 uses variable-length and fixed-length decoding to decode the SPS, PPS, or slice header, an arithmetic decoding engine, and decodes the syntax elements of the slice data as a sequence of one or more bins. Each bin can use one or more "contexts," which describe the probability levels used to code the "1" and "0" values ​​of the bin. If multiple contexts are available for a given bin, a "context modeling" or "context selection" step is performed to select one of the available contexts for decoding the bin.

[0153] The entropy decoder module 920 applies an arithmetic coding algorithm, such as "context-adaptive binary arithmetic coding" (CABAC), to decode syntax elements from the bitstream 143. The decoded syntax elements are used to reconstruct parameters within the video decoder 146. The parameters include residual coefficients (represented by arrow 924), quantization parameters 974, secondary transform indices 970, and mode selection information such as intra-prediction modes (represented by arrow 958). The mode selection information also includes information such as motion vectors and the division of each CTU into one or more CBs. The parameters are used to generate PBs, typically combined with sample data from previously decoded CBs.

[0154] According to the secondary transform index, a secondary transform is applied, or if no operation is performed (bypass), the residual coefficients 924 are passed to an inverse secondary transform module 936. The inverse secondary transform module 936 generates reconstructed transform coefficients 932, i.e., primary transform domain coefficients, from the secondary transform domain coefficients. The reconstructed transform coefficients 932 are input to an inverse quantizer module 928. The inverse quantizer module 928 performs inverse quantization (or “scaling”) on the residual coefficients 932, i.e., in the primary transform coefficient domain, according to a quantization parameter 974 to create reconstructed intermediate transform coefficients, represented by arrow 940. The inverse quantizer module 928 may also apply a scaling matrix to provide non-uniform inverse quantization within the TB, corresponding to the operation of the inverse quantizer module 840. If the bitstream 143 indicates that a non-uniform inverse quantization matrix is ​​to be used, the video decoder 144 reads the quantization matrix from the bitstream 143 as a sequence of scaling factors and arranges the scaling factors into a matrix. Inverse scaling uses a quantization matrix in combination with the quantization parameters to create reconstructed intermediate transform coefficients 940 .

[0155] The reconstructed transform coefficients 940 are passed to an inverse linear transform module 944. Module 944 converts the coefficients 940 from the frequency domain back to the spatial domain. The inverse linear transform module 944 applies an inverse DCT-2 transform horizontally and vertically, constrained by the maximum available transform size, as described with reference to the forward linear transform module 826. The result of the operation of module 944 is a block of residual samples, represented by arrow 948. The size of the block of residual samples 948 is equal to the corresponding CB. The residual samples 948 are provided to a summation module 950.

[0156] In addition module 950, residual samples 948 are added to the decoded PB (represented as 952) to generate a block of reconstructed samples, represented by arrow 956. The reconstructed samples 956 are provided to a reconstructed sample cache 960 and an in-loop filtering module 988. The in-loop filtering module 988 generates a reconstructed block of frame samples, represented as 992. The frame samples 992 are written to a frame buffer 996.

[0157] The reconstructed sample cache 960 operates similarly to the reconstructed sample cache 856 of the video encoder 120. The reconstructed sample cache 960 provides storage for reconstructed samples needed to intra-predict subsequent CBs without the memory 206 (e.g., by using the data 232, which is typically on-chip memory, instead). Reference samples, represented by arrow 964, are retrieved from the reconstructed sample cache 960 and provided to a reference sample filter 968 to generate filtered reference samples, indicated by arrow 972. The filtered reference samples 972 are provided to an intra-frame prediction module 976. The module 976 generates blocks of intra-predicted samples, represented by 980, according to the intra-prediction mode parameters 958 signaled in the bitstream 143 and decoded by the entropy decoder 920. The intra-prediction module 976 supports the modes of the module 864, including IBC and MIP. The blocks of samples 980 are generated using modes such as DC, planar, or angular intra-prediction.

[0158] If the prediction mode of a CB is indicated to use intra prediction in the bitstream 143, the intra-predicted samples 980 form the decoded PB 952 via a multiplexer module 984. Intra prediction generates a predictive block of samples (PB), which is a block in one color component derived using "neighboring samples" in the same color component. Neighboring samples are samples that are adjacent to the current block and are preceding it in block decoding order, and therefore have already been reconstructed. When a luma block and a chroma block are co-located, the luma block and the chroma block may use different intra-prediction modes. However, the two chroma CBs share the same intra-prediction mode.

[0159] If the prediction mode of the CB is indicated as inter-prediction in the bitstream 143, the motion compensation module 934 generates a block of inter-predicted samples represented as 938. The block of inter-predicted samples 938 is generated using a motion vector decoded from the bitstream 143 by the entropy decoder 920 and a reference frame index to select and filter a block of samples 998 from the frame buffer 996. The block of samples 998 is obtained from a previously decoded frame stored in the frame buffer 996. In the bi-predictive case, two blocks of samples are generated and blended to generate samples for the decoded PB 952. The frame buffer 996 is populated with filtered block data 992 from the in-loop filtering module 988. Similar to the in-loop filtering module 868 of the video encoder 120, the in-loop filtering module 988 applies any of DBF, ALF, and SAO filtering operations. Generally, motion vectors are applied to both the luma and chroma channels, although the filtering processes for sub-sample interpolation in the luma and chroma channels are different. The frame from frame buffer 996 is output as decoded frame 162 .

[0160] FIG. 10 is a schematic block diagram illustrating a cross-layer tensor inverse bottleneck decoder 1000, corresponding to decoder 150 (and similarly decoders 118 and 174) for restoring tensor dimensionality after compression. FIG. 15 illustrates a method 1500 for restoring tensor dimensionality using bottleneck decoder 150 of FIG. 10. Method 1500 may be performed using an apparatus such as a configured FPGA, ASIC, or ASSP. Alternatively, as described below, method 1500 may be performed by destination device 140 as one or more software code modules of application program 233 under execution of processor 205. Software code modules of application program 233 implementing method 1500 may reside on hard disk drive 210 and / or memory 206, for example. Method 1500 is repeated for each frame of compressed data in bitstream 123. Method 1500 may be stored on a computer-readable storage medium and / or memory 206. Method 1500 decodes a bitstream containing a compressed representation of the entire FPN and, optionally, an additional compressed representation of a portion of the FPN, and provides a "switchable" means for combining the two representations on the FPN (the entire FPN and the additional portion) into a final decoded FPN tensor for providing to CNN head 152.

[0161] Decoder 150 receives tensor 147 generated by operation of video decoder 146 and module 160 on bitstream 143. Tensor 147 includes tensors 1011 and 1021 as first and second units of information. Tensor 1021 corresponds to a decoded version of tensor 537. Similarly, tensor 1011 corresponds to a decoded version of tensor 557. If bitstream 143 is generated by operation of encoder 500 in a first mode, at least tensor 1011 is decoded from the bitstream by video decoder 146. Flags related to the mode operation may also be decoded. If bitstream 143 is generated by operation of encoder 500 in a first mode, tensor 1021 is also decoded by video decoder 146. Method 1500 begins with a first bottleneck tensor decoding step 1510.

[0162] In step 1510, the SSFC decoder 1010 is implemented under the execution of the processor 205. The SSFC decoder executes a neural network layer to decompress the first decoded compressed tensor 1011 of the tensor 147 and generate a first decoded composite tensor 1017. The convolutional layer 1012 receives the tensor 1011 with C' = 64 channels and outputs a tensor 1013 with F = 256 channels. The tensor 1013 is passed to a batch normalization layer 1014. The batch normalization layer outputs a tensor 1015. The tensor 1015 is passed to a parameterized leaky rectified linear (PReLU) layer 1016. The PReLU layer 1016 outputs a tensor 1017. Tensor 1017 is passed to MSFR module 1030, which produces tensors 1051, 1053, 1055, and 1037, forming the "base layer" of the decoded FPN layer. The base layer provides lower fidelity than would be present if additional "enhancement layer" decoded FPN layers were included. Upsampling modules 1032, 1034, and 1036 receive tensor 1017 and perform interpolation at 2X, 4X, and 8X scales to produce tensors 1033, 1035, and 1037, respectively. For example, tensor 1033 has twice the width and height of tensor 1017.

[0163] Tensor 1037 forms one output from MSFR module 1030 and is passed to downsample module 1042. Downsample module 1042 downsamples tensor 1037 by a factor of 2 horizontally and vertically to produce tensor 1043, which has the same dimensionality as tensor 1035. Tensor 1043 is provided to convolutional layer 1048, which outputs tensor 1049. Addition module 1054 adds tensors 1035 and 1049 to produce tensor 1055 as the output of MSFR module 1030. Downsample module 1040 downsamples tensor 1035 by a factor of 2 horizontally and vertically to produce tensor 1041, which has the same dimensionality as tensor 1033. Tensor 1041 is provided to convolutional layer 1046, which outputs tensor 1047. Summation module 1052 sums tensors 1033 and 1047 to generate tensor 1053 as the output of MSFR module 1030. Downsampling module 1038 downsamples tensor 1033 by a factor of 2 horizontally and vertically to generate tensor 1039, which has the same dimensionality as tensor 1017. Tensor 1039 is provided to convolutional layer 1044, which outputs tensor 1045. Summation module 1050 sums tensors 1017 and 1045 to generate tensor 1051 as the output of MSFR module 1030.

[0164] Step 1510 generates tensors 1051, 1053, 1055, and 1037. Tensors 1051, 1053, 1055, and 1037 form a hierarchical representation of the image frame and can be thought of as including a first tensor (e.g., 1037 P'2 or 1055 P'3) and a second tensor (e.g., 1053 P'4 or 1051 P'5), where the feature maps in the first tensor have greater spatial resolution than the feature maps in the second tensor. Control in processor 205 passes from step 1160 to a second bottleneck tensor presence indication decoding step 1520.

[0165] In step 1520, entropy decoder 920, under execution of processor 205, decodes from bitstream 123 an indication of whether bitstream 123 includes a second bottleneck tensor. The presence of the second bottleneck tensor may be determined from the decoded presence flag 751 obtained from SEI message 744. Control in processor 205 passes from step 1520 to a second bottleneck tensor presence test step 1530.

[0166] In step 1530, application 233 executes to determine whether decoding of bitstream 123 in step 1520 indicated the inclusion of a second bottleneck tensor. If presence flag 751 indicates that a second bottleneck tensor was coded ("PRESENT" in step 1530), control in processor 205 passes from step 1530 to second bottleneck tensor decoding step 1540. Otherwise, if the presence of a second bottleneck tensor is not determined ("ABSENT" in step 1530), control in processor 205 passes to first and second tensor combining step 1550. Decoder 1000 can be considered to be operating in a first (base) mode of operation if it is determined that the second bottleneck tensor is not present ("ABSENT" in 1530). Decoder 1000 can be considered to be operating in a second (enhanced) mode of operation if it is determined that the second bottleneck tensor is not present ("PRESENT" in 1530).

[0167] In step 1540, the SSFC decoder 1020 executes under the execution of the processor 205. The SSFC decoder 1020 executes a neural network layer to decompress the second decoded compressed tensor 1021 of the tensor 147 and generate a second decoded composite tensor 1027.

[0168] Convolutional layer 1022 receives tensor 1021 with C'=64 channels and outputs tensor 1023 with F=256 channels. Tensor 1023 is passed to batch normalization layer 1024. Batch normalization layer 1024 outputs tensor 1025. Tensor 1025 is passed to PReLU layer 1026. PReLU layer 1026 outputs tensor 1027.

[0169] The MSFR module 1030 uses an upsampling module 1060, a downsampling module 1062, a convolutional layer 1064, and an addition module 1066 to generate decoded tensors 1061 and 1067. Tensor 1027 is passed to the upsampling module 1060, which performs interpolation to generate tensor 1061, which has twice the width and height of tensor 1027. Tensor 1061 is output from the MSFR module 1030 and passed to the downsampling module 1062, which downsamples tensor 1061 to generate tensor 1063, which has the same dimensionality as tensor 1027. Tensor 1063 is provided to a convolutional layer 1064 with a stride of 1, which outputs tensor 1065. Addition module 1066 adds tensors 1065 and 1027 to produce tensor 1067, which is output from MSFR module 1030. Control in processor 205 passes from step 1540 to step 1550.

[0170] In step 1550, decoded FPN tensors P'2-P'5 are generated, i.e., 1051, 1053, 1073, and 1077. Tensors 1051 and 1053 from the base layer portion of MSFR module 1030 are ready to be passed to CNN head 150. Tensors 1073 and 1077 are determined in step 1550 from tensors 1055 and 1037, and optionally from tensors 1071 and 1078. If it is determined not to use an enhancement layer, i.e., if it is determined to omit the second bottleneck tensor, multiplexers 1074 and 1076 output tensors 1055 and 1037 as tensors 1073 and 1077, respectively. If the second bottleneck tensor is omitted, the CNN head will have tensors for all FPN layers that allow the task to be performed, but task performance will be reduced due to lower spatial fidelity in the higher-resolution layers, e.g., P2 and P3. The reduced task performance from using only the base layer tends to limit the maximum achievable mAP for near-lossless compression achieved in the video encoder 120 and loss-minimized instance segmentation in the bottleneck encoder and decoder. The presence of an enhancement layer (in addition to an enhancement layer) can increase the maximum achievable mAP to approximately that achieved when the neural network is run as a single operation, i.e., without splitting it into two parts.

[0171] If a decision is made to include an enhancement layer ("PRESENT" in step 1030), convolutions 1070 and 1072 are performed. Convolution 1070 takes as input tensors 1055 and 1067 concatenated along the channel dimension and produces output tensor 1071. Convolution 1072 takes as input tensors 1037 and 1061 concatenated along the channel dimension and produces output tensor 1078. Multiplexers 1074 and 1076 pass along tensors 1071 and 1078 as tensors 1073 and 1077. When an enhancement layer is included, output tensors 1071 and 1073 for P'2 and P'3 increase spatial fidelity, which benefits tasks such as instance segmentation.

[0172] Thus, in the second mode, a plurality of tensors are derived that form at least a portion of a hierarchical representation of the image data. The tensors are derived from tensor 1021 and at least some of the tensors decoded in step 1510 that are associated with the first tensor (e.g., 1051 and 1053).

[0173] In an implementation of method 1500, multiplexers 1074 and 1076 are omitted, and convolutions 1072 and 1074 are initialized according to a determination of whether an enhancement layer is used. If an enhancement layer is being used, pre-trained weights are used to initialize convolutions 1072 and 1074. Thus, in the second (enhancement) mode, the convolutional layers receive tensors P'2 and P'3 (each an example of a first tensor) derived from the first and second units of information, respectively. If an enhancement layer is not being used (in the first mode), convolutions 1072 and 1074 are initialized such that the convolutional weights corresponding to input tensors 1055 and 1037 form an identity matrix, and the weights corresponding to input tensors 1061 and 1067 are zeroed out. Applying the identity matrices from convolution modules 1072 and 1074 to input tensors 1055 and 1037 results in tensors 1055 and 1037 being output as 1071 and 1078 respectively, and input tensors 1061 and 1067 do not contribute to the output as their corresponding weights are zeroed out.

[0174] Method 1500 concludes with performance of step 1550, producing a decoded FPN ready for processing by CNN head 150. Method 1500 is invoked again for each frame of video data encoded in bitstream 123.

[0175] The operation of the bottleneck encoder 116 and bottleneck decoder 150 provides quality scalability using base layer and enhancement layer representations of FPN tensors. To ensure the intended operation of the enhancement layers as "deltas" to improve the fidelity of the decoded FPN tensor 119 or 151 relative to the FPN tensor 115 emanating from the backbone 114, trainable layers in the bottleneck encoder and decoder associated with the base layer must be trained first, with the enhancement layers inactive. The SE module 526, convolution 528, SSFC encoder 550, SSFC decoder 1010, and convolutions 1044, 1046, and 1048 are trained to provide the base layer capabilities of the bottleneck encoder and decoder. To train the enhancement layers, modules associated with the base layer in the bottleneck encoder and decoder are fixed, and modules associated with the enhancement layers are set as trainable. We then learn that the enhancement layer modules (SE module 516, convolution 518, SSFC encoder 530, SSFC decoder 1020, convolution 1064) that act to merge the base layer tensors with the enhancement layer tensors, and the two convolutions 1070 and 1072, provide a "delta" improvement in performance over and above that achieved using only the base layer.

[0176] 12A is a schematic block diagram showing a head 152 of a CNN for object detection. Different networks can be used in place of the CNN head 152 depending on the task to be performed in the destination device 140. An input tensor 151 is split into tensors for each layer (i.e., tensors 1210, 1220, and 1234). The tensor 1210 is passed to a CBL module 1212 to generate a tensor 1214. The tensor 1214 is passed to a detection module 1216 and an upscaler module 1222. The detection module 1216 operates to detect a bounding box 1218. The bounding box 1218 is a form of the detection tensor. The bounding box 1218 is passed to a non-maximum suppression (NMS) module 1248. The NMS module 1248 selects one of the multiple inputs generated by the detection module to generate a detection result 153. Scaling by the width and height of the original video is performed prior to resizing the backbone portion of the network 114 to generate a bounding box that addresses coordinates in the original video data 113. The upscaler module 1222 generates an upscaled tensor 1224 that is scaled by the width and height of the original video. The upscaled tensor 1224 is passed to a CBL module 1226. The CBL module 1226 generates a tensor 1228 as output. The tensor 1228 is passed to a detection module 1230 and an upscaler module 1236. The detection module 1230 operates similarly to the detection module 1216 and generates a detection tensor 1232. The detection tensor 1232 is provided to an NMS module 1248.

[0177] Upscaler module 1236 operates similarly to module 1260 and outputs an upscaled tensor 1238. The upscaled tensor 1238 is passed to CBL module 1240. CBL module 1240 operates similarly to modules 1212 and 1226 and outputs tensor 1242 to detection module 1244. Detection module 1244 operates similarly to detection module 1216 and generates detected tensor 1246. The detected tensor 1246 is provided to NMS module 1248.

[0178] CBL modules 1212, 1226, and 1240 each include a concatenation of five CBL modules, each of which is described with reference to Figure 3D. Upscaler modules 1222 and 1236 are each instances of upscaler module 1260, as shown in Figure 12B.

[0179] Upscaler module 1260 accepts tensor 1262 and tensor 1264 as input. Tensor 1262 is passed to CBL module 1266 to generate tensor 1268. Tensor 1268 is passed to upsampler 1270 to generate upsampled tensor 1272 using nearest neighbor interpolation or various other methods. Concatenation module 1274 generates tensor 1276 by concatenating upsampled tensor 1272 with input tensor 1264.

[0180] Detection modules 1216, 1230, and 1244 are instances of detection module 1280 as shown in FIG. 12C. Detection module 1260 receives tensor 1282, which is passed to CBL module 1284 to generate tensor 1286. Tensor 1286 is passed to convolution module 1288, which implements the detection kernel. A 1×1 kernel is applied to generate output on the feature maps in the three layers. The detection kernel is 1×1×(B×(5+C)), where B is the number of bounding boxes a particular cell can predict, typically 3, and C is the number of classes, which can be 80, resulting in a kernel size of 255 detection attributes. Module 1288 outputs tensor 1290. The constant “5” represents four bounding box attributes (box center x, y and size scale x, y) and one object confidence level (“objectness”). The result of the detection kernel has the same spatial dimension as the input feature map, but the depth of the output corresponds to the detected attribute. A detection kernel is applied to each layer, typically three layers, resulting in a large number of candidate bounding boxes. A process of non-maximum suppression is applied to the resulting bounding boxes by the NMS module 1048 to discard redundant boxes, such as overlapping predictions at similar scales, resulting in a final set of bounding boxes as output for object detection.

[0181] 13 is a schematic block diagram illustrating an alternative CNN head portion 1300 that can be implemented for module 152. Head portion 1300 forms part of an overall network known as "Fast-RCNN" and includes a feature network (i.e., backbone portion 400), a region proposal network, and a detection network. The input to head portion 1300 is tensor 151. Tensor 151 includes P2-P6 layer tensors 1310, 1312, 1314, 1316, and 1318, respectively. The P2-P6 tensors 1310, 1312, 1314, 1316, and 1318 are input to a region proposal network (RPN) head module 1320. The RPN head module 1320 performs convolutions on the input tensors to generate intermediate tensors. The intermediate tensors are fed to two subsequent sibling layers in module 1320, one for classification and one for bounding box, or "region of interest" (ROI), regression, producing an output of classification and bounding boxes 1322. The classification and bounding boxes 1322 are passed to NMS module 1324, which prunes redundant bounding boxes by removing duplicate boxes with lower scores to produce pruned bounding boxes 1326.

[0182] The bounding box 1326 is passed to a region of interest (ROI) alignment module 1328, i.e., "RoIAlign" stage. The ROI alignment module 1328 also receives tensors P2-P6 and generates a fixed-size feature map from the various input size maps using a bilinear interpolation operation. In the operation performed by the ROI alignment module 1328, subsampling results from bilinear interpolation of multiple subregions within the received region of interest as 3x3 subregions to generate an output region of interest as a single output value within an output tensor.

[0183] In the configuration of CNN backbone 400 and CNN head 1300, the "P6" layer tensor 429 is omitted from the output tensor 115, and in CNN head 1300, the P6 input tensor 1318 is generated by performing a "Maxpool" operation with a stride equal to 2 on the P5 tensor 1316. Because the P6 layer can be reconstructed from the P5 layer, there is no need to separately encode and decode the P6 layer as an explicit FPN layer of either the first set of tensors or the second set of tensors.

[0184] The inputs to the ROI alignment module 1328 are the P2-P5 feature maps 1310, 1312, 1314, and 1316 (corresponding to 1077, 1073, 1053, and 1051 in FIG. 10, respectively), as well as the region of interest proposal 1326. Each proposal (ROI) from 1326 is associated with a portion of the feature maps (1310-1316) to generate a fixed-size map. The fixed-size map is sized independently of the underlying portion of the feature maps 1310-1316. One of the feature maps 1310-1316 is selected so that the resulting cropped map has sufficient detail, for example, according to the following rule: floor(4 + log2(sqrt(box_area) / 224)), where 224 is the canonical box size. Thus, the ROI alignment module 1328 crops the incoming feature map according to the proposal 1326, generating a tensor 1330. The tensors 1330 are fed to a fully connected (FC) neural network head 1332. The FC head 1332 runs two fully connected layers to generate class scores and bounding box predictor delta tensors 1334. The class scores are typically 80-element tensors, with each element corresponding to a predicted score for the corresponding object category. The bounding box predictor delta tensor is an 80x4=320-element tensor containing the bounding box for the corresponding object category. Final processing is performed by an output layer module 1336, which receives the tensors 1334 and performs filtering operations to generate filtered tensors 1338. The final processing encodes one or more bounding boxes for each location in each FPN layer's tensor with an indication of the intended classification and a confidence that the bounding box corresponds to an object (an "object-likeness" value). Low-scoring (low-classified) objects are eliminated from further consideration. The non-maximum suppression module 1340 receives the filtered tensor 1334 and removes duplicate bounding boxes encoded in the received tensor by removing duplicate boxes with lower classification scores, resulting in an inference output tensor 151.

[0185] In the configuration of source device 110 and destination device 140, backbone 114 and head 152 are omitted, and the bottleneck encoder and decoder, i.e., 170, 174, 116, 118, and 150, operate as an end-to-end learning image compression and decompression neural network, taking image frames as input to the encoding stage and outputting decoded image frames from the decoding stage. The end-to-end learning image compression network is trained on the fly to adapt to changes in the input frame data as needed. This allows for the use of potentially smaller networks that are dynamically updated to match the specific video or image being compressed, rather than relying on pre-trained networks that need to be trained on a wide variety of source material to achieve consistent performance. Examples of different types of source material for which adaptation may be required include screen content, camera-captured content (under various lighting and other conditions), rendered content, etc. Metrics for measuring the performance of the bottleneck encoder and decoder may be different from MSE; for example, MS-SSIM may be generated in steps 630 and 645. The performance metrics of MS-SSIM may be useful if the system 100 is operable to provide trainable end-to-end learning image compression.

[0186] The initial weights for the bottleneck encoders and decoders, i.e., 170, 174, 116, 118, and 150, may be derived by performing training using a dataset and ground truth appropriate for the original task network, i.e., the network formed by the backbone 114 and head 152. Training using such a dataset may be performed with fixed network weights for the backbone 114 and head 152 and network weights for the inserted bottleneck encoders and decoders, and allowed to be updated. Such initial weights may be used in the source device 110 and the destination device 140 before any refinement training. Such initial weights, when measured per channel during training, may indicate the tendency for MSE to vary between channels. The variance of MSE between channels under a loss function corresponding to the final task performance indicates the relative contribution each channel makes to the final task result. In the configuration of the source device 110, modules 178 and 180 generate per-channel MSE weights so that channels that contribute less to the final task performance are "mitigated" or reduced in their contribution to the final MSE. The per-channel (or "per-channel") scaling of MSE based on predetermined weights allows the refinement training to adapt without over-allocating importance to preserving channels that make a relatively small contribution to the final task score.

[0187] In the bottleneck encoder and decoder configuration, modules 116, 120, 170, 174, and 150 operate to merge tensors from all FPN layers into a single tensor with set dimensions according to the minimum resolution tensor of the FPN tensors for one image. In other words, referring to Figures 5 and 10, one MSFF module 510 fuses together the tensors for all FPN layers (e.g., 502, 503, 504, and 505) into a single tensor, and MSFR module 1030 reconstructs the tensors for all FPN layers (e.g., 1077, 1073, 1053, and 1051). Furthermore, if necessary, switch 570 can be closed to encode additional data for the P2 and P3 tensors as described in connection with Figure 5 and decoded in connection with the MSFR 1030 and the SSFC decoder 1020 as described in Figure 10.

[0188] In a configuration of source device 110, multiple sets of trained weights are available, e.g., weights optimized for screen content, camera-captured content, and rendered content. Source device 110 is operable to select an optimal set of weights from among the available predetermined weights and signal the selected weights to destination device 140. Source device 110 can "test" each set of weights in modules 170 and 174 to determine which set should be used. A change in the content type of frame data 113 can cause a degradation in performance as measured by module 182, prompting a re-evaluation of which set of predetermined weights should be used.

[0189] In a configuration of source device 110, modules 170 and 174 are operable to train weights associated with the enhancement layer (second unit of information) and convolutions 1070 and 1072, but not the base layer (first unit of information). Signaling related to weight updates in bitstream 123 supports indicating that only the enhancement layer should be updated when a decision is made to update weights.

[0190] In another configuration of source device 110, modules 170 and 174 are operable to train the base layer and enhancement layer as separate training stages. When the base layer is trained, the enhancement layer is disabled, allowing optimal base layer weights to be derived. Once updated base layer weights are determined at source device 110 and communicated to destination device 140, the enhancement layer requires retraining before it can be enabled. Retention is necessary because the enhancement layer operates in combination with the retrained base layer. Once the enhancement layer weights are trained for operation on the new base layer, they must be communicated to destination device 140 before the enhancement layer can be re-enabled for compression of tensors 115.

[0191] In configurations where the base layer and enhancement layers are trained separately, an additional weight update flag (i.e., a flag in addition to weight update flag 750) is used to indicate for which layer the weight update should be performed. Weights 752 contains the weights of the indicated layer.

[0192] In the configuration of system 100, each feature map in the enhancement layer 537 is represented as a set of coefficients that can be applied to a set of basis vectors. The basis vectors are derived from the enhancement layer using a principal component analysis (PCA) method, such as singular value decomposition (SVD). When the PCA method is used, region 716 of frame 700 contains basis vectors and coefficients, with one coefficient per basis vector per feature map in the enhancement layer 537. At the source device 110, the transformation from the enhancement layer 537 to the coefficients is performed using dot products with the basis vectors. At the destination device 140, the transformation from the coefficients to the reconstructed enhancement layer 1021 is performed using dot products with the coefficients and the basis vectors. The PCA method can be applied to both the base layer and the enhancement layer, resulting in two sets of basis vectors and two sets of coefficients. The operation of the PCA encoder and PCA decoder is described in detail in the manuscript '[VCM Track 1] Tensor Compression Using VVC', ISO / IEC JTC 1 / SC 29 / WG 2 manuscript m59591. The PCA method can be applied to both the base layer and the enhancement layer, with separate basis vectors and coefficients generated for each layer. The PCA method can be applied only to the enhancement layer, with basis vectors and coefficients generated for this layer, while all feature maps of the base layer are packed directly into frame 700.

[0193] If separate bottleneck encoders are applied to different, non-overlapping sets of tensors in the FPN layer, the PCA method can be applied independently to any of the resulting compressed tensors. If a PCA encoder is used, the input tensors to the PCA encoder are received from the output of the SSFC encoders 550 and 530, i.e., from the output of the tanh modules 556 and 536 (if present) or from the tensors 557 and 537 that are the output of the batch normalization modules 554 and 534. The output basis vectors, coefficients, and mean feature maps are forwarded to the frame 700 for quantization and packing. If a PCA decoder is used, the input basis vectors, coefficients, and mean feature maps are taken from the packed frame 700 and fed to each PCA decoder, which outputs tensors 1011 and 1021 that are used by the SSFC decoders 1010 and 1020, respectively. In the configuration of system 100, when PCA encoders and decoders are used, batch normalization 554 and 534 is deferred until after the PCA decoder, i.e., modules 554 and 534 are omitted from the SSFC encoders 550 and 530 and performed after the PCA decoder in the SSFC decoders 1010 and 1020.

[0194] In the configuration of system 100, source device 140, regardless of whether the task it performs is object detection or instance segmentation, runs a Mask RCNN backbone and is initialized with pre-trained weights for the Mask RCNN network. When performing object detection, destination device 140 runs a Faster RCNN head at 152 and is initialized with pre-trained weights for the Faster RCNN network. When the Faster RCNN head is executed, a bottleneck decoder 150 is required to act as an interface between the feature maps generated from the Mask RCNN backbone but fed to the Faster RCNN head. Because bottleneck encoder 116 may not know which head network will be executed when performing encoding, it remains optimized with respect to the Mask RCNN backbone and head training. To prepare initial weights for bottleneck decoder 150, a "hybrid" training process may be performed. The hybrid training process involves instantiating a Faster RCNN network with a backbone initialized with the Mask RCNN weights while the head is initialized with the Faster RCNN weights. The MSFC encoder 116 is initialized with weights corresponding to the Mask RCNN training of the MSFC (bottleneck encoder and decoder). Then, only the bottleneck decoder 150 is set to trainable, while all other network layers are set fixed. A training operation is performed that not only decodes the compressed FPN tensor but also trains the bottleneck decoder to adapt the resulting feature maps to match the expected input to the Faster RCNN head, thereby minimizing loss. As a result of the training, the bottleneck decoder 150 provides adaptation between the Mask RCNN backbone and the Faster RCNN head, while the bottleneck encoder 16 remains optimized for the more capable network (i.e., Mask RCNN).If bottleneck decoder 150 is initialized with weights trained for a "hybrid" operating system (Mask RCNN backbone and Faster RCNN head), the resulting bitstreams from source device 110 are suitable for both object detection and instance segmentation at the time of their generation; i.e., the same bitstream can be used later for both tasks without any transcoding or other operations. If a single bitstream generated by source device 110 can serve to provide tensors to different neural network heads (if the different neural networks corresponding to each neural network head share the same backbone topology and dimensionality), source device 110 is said to support a "shared backbone" operating mode.

[0195] In another configuration of the system 100, the C' channel counts of the SSFC encoder 530 and SSFC decoder 1020, i.e., the enhancement layers, are reduced compared to the C' channel counts of the SSFC encoder 550 and SSFC decoder 1010, i.e., the base layer, respectively. The enhancement layers may use a C' value of 32. Such a configuration has larger feature maps, i.e., fewer 710 packed into a frame 700. The ability to retrain the bottleneck encoder and bottleneck decoder applied to the enhancement layers allows fewer channels to be used to encode the enhancement details present in the P2 and P3 layers, adapting to changing statistics encountered across the full channel count of the applicable tensors (i.e., across the 256 channels of the P2 and P3 FPN layers).

[0196] Industrial Applicability The described arrangements are applicable to the computer and data processing industry, and in particular to digital signal processing for encoding and decoding of signals such as video and image signals, achieving high compression efficiency.

[0197] The configuration described in connection with the weight encoding of FIG. 1 provides a system that can adapt to dynamically changing statistics of input video data by undergoing a refinement training process from time to time, as necessitated by continuous monitoring of the performance of the bottleneck encoder and decoder in use. Upon determining refined weights that provide improved performance, the weights actively used in the encoder and decoder are updated to use the refined weights and maintain operation during the training process. The use of continuous performance monitoring allows training of the MFSC unit 116 and the MFSC decoding unit 150 to be performed based on changes in data input or on specific data type input during inference operations. Thus, feature compression operations can be adjusted or updated without requiring separate off-system training for specific image types (e.g., natural images or computer-generated images) or scenarios.

[0198] The configurations described in connection with Figures 5 and 10 allow for flexible operation between high performance and low performance requirements, as described in connection with Figures 14 and 15. A different architecture is not required for each system, but rather flexibility is provided without the need to load different networks.

[0199] The above describes only some embodiments of the present invention, modifications and / or variations may be made without departing from the scope and spirit of the present invention, and the embodiments are illustrative and not limiting.

Claims

1. A method for decoding one or more tensors for a frame from a bitstream, Decoding the first information from the bitstream, Based on the aforementioned first information, one generation process is executed from among a plurality of generation processes, including the first generation process and the second generation process. It has, The aforementioned plurality of generation processes each have at least one convolutional layer, In the first generation process, a first tensor is generated based on the bitstream, and a second tensor different from the first tensor is generated based on the bitstream. In the second generation process, the first tensor is generated based on the bitstream, and the second tensor is not generated. A method characterized by the following:

2. The method according to claim 1, characterized in that the size of the second tensor is different from the size of the first tensor.

3. Further comprising generating a plurality of tensors based on the first tensor, The method according to claim 1, characterized in that the plurality of tensors have different sizes from each other.

4. The method according to claim 2, characterized in that the size of the second tensor is greater than the size of the first tensor.

5. A non-temporary computer-readable storage medium for storing a program for performing a method of decoding one or more tensors for a frame from a bitstream, wherein the method is: Decoding the first information from the bitstream, Based on the aforementioned first information, one generation process is executed from among a plurality of generation processes, including the first generation process and the second generation process. It has, The aforementioned plurality of generation processes each have at least one convolutional layer, In the first generation process, a first tensor is generated based on the bitstream, and a second tensor different from the first tensor is generated based on the bitstream. In the second generation process, the first tensor is generated based on the bitstream, and the second tensor is not generated. A non-temporary computer-readable storage medium characterized by the following features.

6. A decoding device for decoding one or more tensors for a frame from a bitstream, Means for decoding the first information from the bitstream, A means for executing one generation process from among a plurality of generation processes, including a first generation process and a second generation process, based on the first information mentioned above. It has, The aforementioned plurality of generation processes each have at least one convolutional layer, In the first generation process, a first tensor is generated based on the bitstream, and a second tensor different from the first tensor is generated based on the bitstream. In the second generation process, the first tensor is generated based on the bitstream, and the second tensor is not generated. A decoding device characterized by the following features.

7. Memory and The system comprises a processor, the processor being configured to execute code stored in memory to implement a method for decoding one or more tensors for a frame from a bitstream, and the method is Decoding the first information from the bitstream, Based on the aforementioned first information, one generation process is executed from among a plurality of generation processes, including the first generation process and the second generation process. It has, The aforementioned plurality of generation processes each have at least one convolutional layer, In the first generation process, a first tensor is generated based on the bitstream, and a second tensor different from the first tensor is generated based on the bitstream. In the second generation process, the first tensor is generated based on the bitstream, and the second tensor is not generated. A system characterized by the following features.

Images