Method, apparatus, and system for encoding and decoding tensors

Abstract

A method for decoding a plurality of tensors forming a hierarchical representation of a frame from a bitstream and a method for encoding a plurality of tensors forming a hierarchical representation of a frame into a bitstream includes decoding coefficients from the bitstream for a first tensor and generating a second tensor from the first tensor and a set of basis vectors, the second tensor having a higher channel count and the same spatial size as the first tensor. The decoding method also includes generating a plurality of decoded tensors from the second tensor using one or more trained convolutional layers.

Description

[Technical Field]

[0001] This application claims the benefit under 35 U.S.C. § 119 of the filing date of Australian Patent Application No. 2023200117, filed January 10, 2023, the entirety of which is incorporated herein by reference as if fully set forth herein.

[0002] The present invention relates generally to digital video signal processing, and more particularly to a method, apparatus, and system for encoding and decoding tensors from convolutional neural networks. The present invention also relates to a computer program product including a computer-readable medium having recorded thereon a computer program for encoding and decoding tensors from convolutional neural networks using video compression techniques. [Background technology]

[0003] Convolutional neural networks (CNNs) are an emerging technology that addresses use cases involving machine vision, such as object detection, instance segmentation, object tracking, human pose estimation, and action recognition, among others. Applications for CNNs can involve the use of "edge devices" with sensors and some processing capabilities, coupled to application servers as part of a "cloud." CNNs can require relatively high computational complexity than can typically be afforded by edge devices, either in terms of computing power or power consumption. Running CNNs in a distributed manner has emerged as one solution for running cutting-edge networks using edge devices with limited capabilities without pushing all computational complexity onto cloud servers. In other words, distributed processing allows legacy edge devices to still provide the capabilities of cutting-edge CNNs by distributing processing between edge devices and external processing means, such as cloud servers. Such distributed network architectures are sometimes called "collaborative intelligence" (CI) and offer benefits such as reusing partial results from a first part of a network with several different second parts, each optimized for a different task. The CI architecture introduces the need for efficient compression of tensor data for transmission over networks such as WANs.

[0004] CNNs typically contain many layers, such as convolutional and fully connected layers, and data is passed from one layer to the next in the form of "tensors." Partitioning the network across different devices introduces the need to compress the intermediate multidimensional tensor data passing from one layer to the next within the CNN. Compressing such tensors is sometimes called "feature compression," since the intermediate tensor data are often called "features" or "feature maps" (generally a collection of 2D "feature maps" forming a tensor, with each feature map corresponding to one "channel") and represent a partially processed form of the input, such as an image frame or video frame. The International Organization for Standardization / International Electrotechnical Commission Joint Technical Committee 1 / Subcommittee 29 / Working Group 2-8 (ISO / IEC JTC1 / SC29 / WG2-8), also known as the "Moving Pictures Experts Group" (MPEG), is tasked with researching compression techniques in various contexts, often related to video. WG2 'MPEG Technical Requirements' established the 'Video Coding for Machines' (VCM) ad hoc group, tasked with researching compression for machine consumption and feature compression. The feature compression mandate is in an exploratory phase with a "Call for Evidence" (CfE) issued soliciting techniques that can significantly outperform feature compression results achieved using state-of-the-art standardized techniques.

[0005] CNNs typically require that the weights for each layer be predetermined during the training stage, during which a large amount of training data is passed through the CNN and the results determined by the network being trained are compared to the ground truth associated with the training data. The discrepancy between the obtained result and the desired result is expressed as a "loss" and measured using a "loss function." Using the determined loss, a process is performed to update the network weights, such as stochastic gradient descent (SGD). Network weight updates typically involve backpropagating a "gradient" that indicates the delta to be applied to the network weights, starting at the network's output layer, ending at the network's input layer, and covering the network's intermediate, or "hidden," layers. The speed of weight updates is scaled by a "learning rate" hyperparameter, which is typically set to facilitate the training process in finding a global minimum in terms of loss (i.e., the highest possible task performance for the network architecture and training data) while avoiding the training process from getting "stuck" in a local minimum. Getting stuck in a local minimum corresponds to obtaining suboptimal task performance for the network architecture and being unable to find new weight values ​​that could lead to higher task performance. The network weights are iteratively updated by feeding input data and ground truth data organized into "batches," iteratively improving network performance until no further improvements can be achieved. Each iteration through the training dataset forms an "epoch" of training, and training typically requires multiple epochs to achieve a high level of performance for a task. The trained network is then available for deployment, operating in a mode in which the weights are fixed and gradients for weight update are omitted. The process of running the pre-trained CNN with inputs and progressively transforming the inputs into outputs according to the CNN's topology is commonly referred to as "inference."

[0006] Generally, tensors have four dimensions: batch, channel, height, and width. The first dimension, "batch," is typically sized at 1 when inferring video data, indicating that one frame is passed through the CNN as a batch. When training a network, the value of the batch dimension can be increased according to a predetermined "batch size," so that multiple frames are passed through the network in each batch before the network weights are updated. Multi-frame videos can be passed through a single tensor with the batch dimension increased in size according to the number of frames in a given video. However, due to practical considerations regarding memory consumption and access, inference on video data is typically performed frame-by-frame. The "channel" dimension indicates the number of simultaneous "feature maps" for a given tensor, while the height and width dimensions indicate the size of the feature maps at a particular stage of the CNN. The channel count varies through the layers of a CNN according to the network architecture. Feature map size also varies depending on the subsampling or upsampling that occurs at a particular network layer.

[0007] The overall complexity of CNNs tends to be relatively high, with a relatively large number of multiply-accumulate (MAC) operations performed and a large number of intermediate tensors written to and read from memory along with the read weights for each layer of the CNN execution. Therefore, dividing a neural network into parts enables such implementation of more complex networks even on less capable edge devices.

[0008] Feature compression can benefit from existing video compression standards, such as Versatile Video Coding (VVC) developed by the Joint Video Experts Team (JVET). VVC is expected to address the continuing demand for higher compression performance, particularly as video formats increase in capacity (e.g., at higher resolutions and higher frame rates), and to address growing market demand for service delivery over WANs where bandwidth costs are relatively high. VVC is implementable in modern silicon processes and offers an acceptable tradeoff between achieved performance and implementation cost. Implementation cost may be considered, for example, in terms of one or more of silicon area, CPU processor load, memory usage, and bandwidth. Other video compression standards, such as High Efficiency Video Coding (HEVC) or AV-1, may also be used for feature compression applications.

[0009] Video data includes a sequence of frames of image data, each containing one or more color channels. When feature map data is represented in packed frames, monochrome frames, which have only luma and no chroma channels, are generally appropriate. When only luma samples are present, the resulting monochrome frames are said to use a "4:0:0 chroma format."

[0010] The VVC standard defines a "block-based" architecture, where a frame is first divided into an array of square regions known as "coding tree units" (CTUs). In VVC, a CTU typically occupies 128x128 luma samples. Other possible CTU sizes when using the VVC standard are 32x32 and 64x64. However, CTUs at the right and bottom edges of each frame may be smaller in area, and implicit division occurs to ensure that coding blocks remain within the frame. Each CTU is associated with a "coding tree," which defines the decomposition of the CTU's area into a set of blocks, also called "coding units" (CUs). Blocks applicable only to the luma or chroma channels are called "coding blocks" (CBs). A prediction of the coding block's contents is held in a "prediction block" (PB) or "prediction unit" (PU), and the residual block, which defines the array of sample values ​​additively combined with the PB or PU, is called a "transform block" (TB) or "transform unit" (TU) due to its typical use in transform processing to generate the TB or TU.

[0011] Despite the above distinction between "units" and "blocks", the term "block" may be used as a generic term for an area or region of a frame where an operation is applied to all color channels.

[0012] For each CU, a prediction unit (PU) is generated for the contents (sample values) of the corresponding region of the frame data ("prediction unit"). Furthermore, a representation of the difference (or "spatial domain" residual) between the contents of the region seen at the input to the encoder and the prediction is formed. The differences in each color channel can be transformed and encoded as a sequence of residual coefficients, forming one or more TUs for a given CU. The applied transform can be a discrete cosine transform (DCT) or other transform applied to each block of residual values. The transform is applied in isolation (i.e., a two-dimensional transform is performed in two passes, one horizontally and one vertically). The block is first transformed by applying a one-dimensional transform to each row of samples in the block. The partial results are then transformed by applying a one-dimensional transform to each column of the partial results, producing a final block of transform coefficients that substantially decorrelates the residual samples. Transforms of various sizes are supported by the VVC standard, including transforms of rectangular-shaped blocks, with each side dimension being a power of two. The transform coefficients are quantized for entropy coding into the bitstream.

[0013] A PB or PU in VVC can be generated using either intra-frame or inter-frame prediction processes. Intra-frame prediction involves using previously processed samples within a frame to generate a prediction of the current block of data samples within the frame. Inter-frame prediction involves using blocks of samples obtained from one or two previously decoded frames to generate a prediction of the current block of samples in a frame. The blocks of samples obtained from previously decoded frames are offset from the spatial location of the current block, often according to a filtered motion vector. An intra-frame prediction block can be (i) a uniform sample value ("DC intra-prediction"), (ii) a plane with an offset and horizontal and vertical gradients ("planar intra-prediction"), (iii) a population of blocks with neighboring samples applied in a specific direction ("directional intra-prediction"), or (iv) the result of matrix multiplication using neighboring samples and selected matrix coefficients.

[0014] VVC can be used to compress intermediate feature maps from the first part (the "backbone") of a neural network separated into two parts. During compression, feature maps from the backbone are arranged into frames and quantized from the floating-point domain to a sample domain suitable for compression as video data. To reduce the spatial domain of the feature maps, additional neural network layers can be implemented at the interface between the VVC encoder and decoder and the intermediate point within the CNN where the division occurs. Training for such additional network layers may not be suitable for the varied and unpredictable feature map data encountered. Training may not result in a CNN that is adaptable to operating points of varying quality in terms of task performance. The operating points of the encoder and decoder may also change during operation, and must support varying levels of quality for the reconstructed tensors fed to the rest of the network on the decoder side. Summary of the Invention

[0015] SUMMARY OF THE INVENTION It is an object of the present invention to substantially overcome, or at least ameliorate, one or more problems of existing arrangements.

[0016] One aspect of the present disclosure provides a method for decoding multiple tensors forming a hierarchical representation of a frame from a bitstream, the method including: decoding coefficients from the bitstream for a first tensor; generating a second tensor from the first tensor and a set of basis vectors, the second tensor having a higher channel count and the same spatial size as the first tensor; and generating multiple decoded tensors from the second tensor using one or more trained convolutional layers.

[0017] Another aspect of the present disclosure provides a method for encoding a plurality of tensors forming a hierarchical representation of a frame into a bitstream, the method including: generating a first tensor from the plurality of tensors using one or more downsampling filters; generating a second tensor from the first tensor by scaling the first tensor according to a scaling tensor; generating coefficients of a third tensor using the second tensor and a set of basis vectors, the third tensor having fewer channels and the same spatial size as the second tensor; and encoding the coefficients of the third tensor for the frame into a bitstream.

[0018] Another aspect of the present disclosure provides a decoder for decoding multiple tensors forming a hierarchical representation of a frame from a bitstream, the decoder being configured to: decode coefficients from the bitstream for a first tensor; generate a second tensor from the first tensor and a set of basis vectors, the second tensor having a higher channel count and the same spatial size as the first tensor; and generate multiple decoded tensors from the second tensor using one or more trained convolutional layers.

[0019] Another aspect of the present disclosure provides a non-transitory computer-readable storage medium having stored thereon a program for performing a method for decoding a plurality of tensors forming a hierarchical representation of a frame from a bitstream, the method including: decoding coefficients from the bitstream for a first tensor; generating a second tensor from the first tensor and a set of basis vectors, the second tensor having a higher channel count and the same spatial size as the first tensor; and generating a plurality of decoded tensors from the second tensor using one or more trained convolutional layers.

[0020] Another aspect of the present disclosure provides a system having a memory and a processor configured to execute code stored in the memory for performing a method for decoding a plurality of tensors forming a hierarchical representation of a frame from a bitstream, the method including: decoding coefficients from the bitstream for a first tensor; generating a second tensor from the first tensor and a set of basis vectors, the second tensor having a higher channel count and the same spatial size as the first tensor; and generating a plurality of decoded tensors from the second tensor using one or more trained convolutional layers.

[0021] Another aspect of the present disclosure provides an encoder for encoding a plurality of tensors forming a hierarchical representation of a frame into a bitstream, the encoder being configured to generate a first tensor from the plurality of tensors using one or more downsampling filters, generate a second tensor from the first tensor by scaling the first tensor according to a scaling tensor, generate coefficients of a third tensor using the second tensor and a set of basis vectors, the third tensor having fewer channels and the same spatial size as the second tensor, and encode the coefficients of the third tensor for the frame into the bitstream.

[0022] Another aspect of the present disclosure provides a non-transitory computer-readable storage medium having stored thereon a program for executing a method for encoding a plurality of tensors forming a hierarchical representation of a frame into a bitstream, the method including: generating a first tensor from the plurality of tensors using one or more downsampling filters; generating a second tensor from the first tensor by scaling the first tensor according to a scaling tensor; generating coefficients of a third tensor using the second tensor and a set of basis vectors, the third tensor having fewer channels and the same spatial size as the second tensor; and encoding the coefficients of the third tensor into the bitstream for the frame.

[0023] Another aspect of the present disclosure provides a system having a memory and a processor configured to execute code stored in the memory for performing a method for encoding a plurality of tensors forming a hierarchical representation of a frame into a bitstream, the method including generating a first tensor from the plurality of tensors using one or more downsampling filters; generating a second tensor from the first tensor by scaling the first tensor according to a scaling tensor; generating coefficients of a third tensor using the second tensor and a set of basis vectors, the third tensor having fewer channels and the same spatial size as the second tensor; and encoding the coefficients of the third tensor into the bitstream for the frame.

[0024] Other aspects are also disclosed. [Brief explanation of the drawings]

[0025] At least one embodiment of the present invention will now be described with reference to the following drawings and appendices. [Figure 1] FIG. 1 is a schematic block diagram illustrating a distributed machine task system. [Figure 2A] FIG. 2A forms a schematic block diagram of a general-purpose computer system on which the distributed machine task system of FIG. 1 may be implemented. [Figure 2B] FIG. 2B forms a schematic block diagram of a general-purpose computer system on which the distributed machine task system of FIG. 1 may be implemented. [Figure 3A] FIG. 3A is a schematic block diagram showing the functional modules of the backbone part of the CNN. [Figure 3B] FIG. 3B is a schematic block diagram illustrating the residual block of FIG. 3A. [Figure 3C] FIG. 3C is a schematic block diagram of the residual unit of FIG. 3A. [Figure 3D] FIG. 3D is a schematic block diagram illustrating the CBL module of FIG. 3A. [Figure 4] FIG. 4 is a schematic block diagram showing the functional modules of an alternative backbone portion of a CNN. [Figure 5] FIG. 5 is a schematic block diagram illustrating the multi-scale feature fusion stage. [Figure 6] FIG. 6 is a schematic block diagram illustrating an inter-channel decorrelation based tensor encoder as part of a distributed machine task system. [Figure 7] FIG. 7 is a schematic block diagram showing a feature map packer, a packed frame encoder, and an unpacker (or "subpicture encoder") as part of an inter-channel decorrelation based tensor encoder. [Figure 8] FIG. 8 is a schematic block diagram illustrating the functional modules of a video encoder. [Figure 9] FIG. 9 is a schematic block diagram illustrating an inter-channel decorrelation based tensor decoder as part of a distributed machine task system. [Figure 10] FIG. 10 is a schematic block diagram showing the functional modules of a video decoder. [Figure 11A] FIG. 11A is a schematic block diagram illustrating an embodiment of a multi-scale feature reconstruction stage. [Figure 11B] FIG. 11B is a schematic block diagram illustrating an embodiment of a multi-scale feature reconstruction stage. [Figure 12A]FIG. 12A is a schematic block diagram showing the head unit of a CNN. [Figure 12B] FIG. 12B is a schematic block diagram illustrating the upscaler module of FIG. 10A. [Figure 12C] FIG. 12C is a schematic block diagram illustrating the detection module of FIG. 10A. [Figure 13] FIG. 13 is a schematic block diagram showing an alternative head portion of a CNN. [Figure 14A] FIG. 14A is a schematic block diagram illustrating the arrangement of sub-pictures for holding inter-channel decorrelated tensor data. [Figure 14B] FIG. 14B is a schematic block diagram illustrating the arrangement of sub-pictures for holding inter-channel decorrelated tensor data. [Figure 15] FIG. 15 is a schematic block diagram illustrating a bitstream carrying encoded inter-channel decorrelation feature maps and associated metadata. [Figure 16] FIG. 16 shows a method for running the first part of the CNN, decorrelating, and encoding the resulting feature maps. [Figure 17] FIG. 17 shows the method for decoding the bitstream, reconstructing the decorrelated feature maps, and running the second part of the CNN. DETAILED DESCRIPTION OF THE INVENTION

[0026] Where reference is made in any one or more of the accompanying drawings to steps and / or features having the same reference numerals, those steps and / or features have the same function or operation for the purposes of this description, unless intended to the contrary.

[0027] A distributed machine task system can include edge devices, such as network cameras or smartphones, that generate intermediate compressed data. The distributed machine task system can also include end devices, such as server farm-based ("cloud") applications, that operate on the intermediate compressed data to generate task results. In addition, edge device functions may be embodied in the cloud, and the intermediate compressed data may be stored for later processing, potentially for multiple different tasks, as needed.

[0028] A convenient form of intermediate compressed data is the compressed video bitstream, due to the availability of high-performance compression standards and their implementations. Video compression standards typically operate on integer samples of some given bit depth, such as 10 bits, arranged in a planar array. Color video has three planar arrays, corresponding to the color components Y, Cb, Cr, or R, G, B, depending on the application. CNNs typically operate on floating-point data in the form of tensors. Tensors generally have a relatively small spatial dimension compared to the input video data on which the CNN operates, while having more channels than the three channels typical of color video data, e.g., 128, 256, or 512 channels.

[0029] Tensors typically have the following dimensions: frames, channels, height, and width. For example, a tensor with dimensions [1, 256, 76, 136] is said to contain one frame of data, each containing 256 feature maps (channels) of size 136x76. For video data, inference is typically performed one frame at a time (frame value 1) rather than using tensors containing multiple frames.

[0030] VVC supports the division of a picture into multiple subpictures, each of which can be independently coded and decoded. In one approach, each subpicture is coded as a "slice," or a contiguous sequence of coded control units (CTUs). A "tiling" mechanism is also available to divide a picture into several independently decodable regions. Subpictures can be specified in a somewhat flexible way, with different rectangular sets of CTUs coded as each subpicture. The flexible definition of subpicture dimensions allows for efficient storage of types of data that require different areas within a picture, avoiding large "unused" areas, i.e., areas of the frame that are not used for the reconstruction of tensor data.

[0031] FIG. 1 is a schematic block diagram illustrating the functional modules of a distributed machine task system 100 capable of distributing and executing a machine task network. Splitting a particular neural network into two parts requires the specification of a "split point" within the network. The layers in the network from the input layer to the split point are executed on a first device, and the resulting intermediate tensor(s) are compressed. The layers from the split point to the last layer in the network are executed using the decompressed tensor from the first device as input to the layer immediately following the split point. At the split point, there may be one or more tensors that need to be compressed for transport over a communication channel with limited bandwidth compared to the bandwidth requirements for transmitting uncompressed tensors. When a "feature pyramid network" (FPN) is used, it is common for the layers in the FPN to be related in width and height, such that a given layer is half the width and height of the adjacent layer between them. The FPN architecture may also define width and height halvings that occur on every alternating layer. In some architectures, multiple tensors of the same width and height are found. The network can be partitioned within the FPN of the machine task network, facilitating the performance of various machine task networks where layers up to the partition point are common between them ("shared backbone" architecture). Therefore, a compression method applicable to the various network topologies used in modern CNNs would be beneficial for application to a wide range of scenarios.

[0032] System 100 can be used to implement methods for decorrelating, packing, and quantizing feature maps into planar frames for encoding and decoding the feature maps from the encoded data. System 100 can be used in some embodiments to be rate-controllable, with the bit rate of the compressed tensor varying based on a selected rate, with the quality of the reconstructed tensor varying. System 100 can also be used in some embodiments to prevent quantized representations of tensors from unnecessarily consuming bits that do not provide a commensurate benefit in terms of task performance.

[0033] System 100 includes a source device 110 for generating coded tensor data 115a from a CNN backbone 114 in the formation of coded video bitstream 121. System 100 also includes a destination device 140 for decoding the tensor data in the formation of coded video bitstream 143. A communication channel 130 is used to communicate coded video bitstream 121 from source device 110 to destination device 140. In some configurations, either or both of source device 110 and destination device 140 may include respective mobile phone handsets (e.g., “smartphones”) or network cameras and cloud applications. Communication channel 130 may be a wired connection, such as Ethernet, or a wireless connection, such as WiFi or 5G, including a connection via a wide area network (WAN) or an ad hoc connection. Additionally, source device 110 and destination device 140 may include applications in which coded video data is captured on some computer-readable storage medium, such as a file server or a hard disk drive in memory.

[0034] As shown in FIG. 1, source device 110 includes a video source 112, a CNN backbone 114, a tensor combiner 162, a principal component analysis (PCA) encoder 160, and a transmitter 122. Video source 112 typically comprises a source of captured video frame data (shown as 113), such as an image capture sensor, a previously captured video sequence stored on a non-transitory recording medium, or video input from a remote image capture sensor. Video source 112 may also be the output of a computer graphics card, e.g., displaying the video output of an operating system and various applications running on a computing device (e.g., a tablet computer). Examples of source device 110 that may include an image capture sensor as video source 112 include smartphones, video camcorders, professional video cameras, and network video cameras. Video source 112 may generate independent images or may generate time-sequential images, i.e., video.

[0035] The CNN backbone 114 receives the video frame data 113, executes a particular layer of the overall CNN, such as a layer corresponding to the "backbone" of the CNN, and outputs tensor 115a. The CNN backbone layer may generate multiple tensors as output, corresponding to different spatial scales of the input image represented by the video frame data 113, for example, when splitting the network within an FPN. For example, when a "YOLOv3" network is run by the system 100 and the spatial resolution and channel count vary, the FPN may result in three tensors corresponding to the three layers output from the backbone 114 as tensor 115a. When the system 100 is running a network such as a "Faster RCNN X101-FPN" or a "Mask RCNN X101-FPN," tensor 115a may include tensors for four layers (P2-P5). The use of an FPN, as described below, results in multiple tensors forming a hierarchical representation of a frame that is encoded into (and decoded from) a bitstream when a network split occurs within the FPN. To implement a trained downsampling stage, the tensor combiner 162 may combine multiple layers by performing convolutions with a stride greater than 1, such as a stride of 2, resulting in a tensor with the same dimensions as another spatially smaller tensor in the multiple tensors 115a. The resulting set of tensors, all with the same spatial dimensions, may be concatenated along the channel dimension and further processed by additional network layers to generate the combined tensor 115. The operation of the tensor combiner is described with reference to FIG. 5. The PCA encoder 160 receives the combined tensor 115 output from the tensor combiner 162. Combining layers is appropriate when there is sufficient inter-layer correlation so that the combined layers can be represented using fewer basis vectors than the layers to be separately decorrelated. The degree of inter-layer correlation is a characteristic of the network itself and the input data provided.The degree of inter-layer correlation allows for a reduction in the total number of basis vectors for the combined tensor compared to the sum of the number of basis vectors for the separately decorrelated tensors of each layer. For example, if 25 basis vectors are typically used per layer and two layers are concatenated, the number of basis vectors required by two concatenated layers can be set to less than 50, and the number of basis vectors required by four concatenated layers can be set to less than 100.

[0036] PCA encoder 160 operates to encode joint tensor 115 to generate bitstream 121, and is described with reference to Figure 6. Bitstream 121 is provided to transmitter 122 for transmission over communication channel 130, or bitstream 121 is written to storage device 132 for later use.

[0037] Source device 110 supports a particular network for CNN backbone 114. However, destination device 140 may use one of several networks for head CNN 150. When using one of several networks for head CNN 150, partially processed data in the form of packed feature maps may be stored for later use in performing various tasks without having to perform the operations of CNN backbone 114 again.

[0038] Bitstream 121 is transmitted by transmitter 122 as coded video data (or "coded video information") over communication channel 130. Bitstream 121 may, in some embodiments, be stored in storage memory 132, which is a non-transitory storage device such as "flash" memory or a hard disk drive, until later transmitted over (or instead of) communication channel 130. For example, coded video data may be provided on demand to a customer over a wide area network (WAN) for video analytics applications.

[0039] The destination device 140 includes a receiver 142, a PCA decoder 170, a tensor separator 172, a CNN head 150, and a CNN task result buffer 152. The receiver 142 receives encoded video data from the communication channel 130 and passes the video bitstream 143 to the PCA decoder 170. The PCA decoder 170 outputs a decoded combined tensor 149a, which is provided to the tensor separator 172. The tensor separator 172 performs the inverse operation of the tensor combiner 162 to generate the extracted tensor 149. The extracted tensor 149 is passed to the CNN head 150. At a split point where the network splits at a stage where one tensor combination in a layer is unnecessary, the tensor separator 172 does nothing, and the tensor 149 corresponds to the tensor 149a. The CNN head 150 executes the layer after the task initiated in the CNN backbone 114 to generate the task result 151. Task results 151 are stored in task result buffer 152. The contents of task result buffer 152 may be presented to a user via a graphical user interface, for example, or provided to an analysis application where some action is determined based on the task results, which may include a summary-level presentation of aggregated task results to the user. The functionality of each of source device 110 and destination device 140 may also be embodied in a single device, examples of which include a mobile phone handset, a tablet computer, and a cloud application. While the examples described herein relate to PCA, other decomposition analysis or inter-channel decorrelation-based methods may be used for encoder 160 and decoder 170.

[0040] Notwithstanding the exemplary devices described above, each of source device 110 and destination device 140 may generally be configured within a general-purpose computing system through a combination of hardware and software components. FIG. 2A illustrates such a computer system 200, including a computer module 201, a keyboard 202, a mouse pointer device 203, a scanner 226, a camera 227, which may be configured as a video source 112, input devices such as a microphone 280, and output devices including a printer 215, a display device 214, and a loudspeaker 217. An external modulator-demodulator (Modem) transceiver device 216 may be used by computer module 201 to communicate with a communications network 220 via connection 221. Communications network 220, which may represent communications channel 130, may be a WAN, such as the Internet, a cellular telecommunications network, or a private WAN. In the case where connection 221 is a telephone line, modem 216 may be a conventional “dial-up” modem. Alternatively, if connection 221 is a high-capacity (e.g., cable or optical) connection, modem 216 may be a broadband modem. A wireless modem may also be used for wireless connection to communication network 220. Transceiver device 216 may provide the functionality of transmitter 122 and receiver 142, and communication channel 130 may be implemented in connection 221.

[0041] The computer module 201 typically includes at least one processor unit 205 and a memory unit 206. For example, the memory unit 206 can include semiconductor random access memory (RAM) and semiconductor read-only memory (ROM). The computer module 201 also includes several input / output (I / O) interfaces, including an audio-video interface 207 that couples to a video display 214, a loudspeaker 217, and a microphone 280; an I / O interface 213 that couples to a keyboard 202, a mouse 203, a scanner 226, a camera 227, and optionally a joystick or other human interface device (not shown); and an interface 208 for an external modem 216 and a printer 215. The signal from the audio-video interface 207 to the computer monitor 214 is typically the output of a computer graphics card. In some embodiments, the modem 216 may be incorporated within the computer module 201, for example, within the interface 208. The computer module 201 also has a local network interface 211 that allows the computer system 200 to be coupled via a connection 223 to a local area communication network 222, known as a local area network (LAN). As shown in Figure 2A, the local communication network 222 can also be coupled via a connection 224 to a wide network 220, which typically includes a so-called "firewall" device or device of similar functionality. The local network interface 211 can be an Ethernet TM Circuit card, Bluetooth TM The local network interface 211 may have a wireless or IEEE 802.11 wireless configuration, although many other types of interfaces may be implemented for the interface 211. The local network interface 211 may also provide the functionality of the transmitter 122 and receiver 142, and the communication channel 130 may also be implemented in the local communication network 222.

[0042] The I / O interfaces 208 and 213 can provide either or both serial and parallel connectivity, the former typically being implemented according to the Universal Serial Bus (USB) standard and having a corresponding USB connector (not shown). A storage device 209 is provided and typically includes a hard disk drive (HDD) 210. Other storage devices, such as a floppy disk drive or magnetic tape drive (not shown), may also be used. An optical disk drive 212 is typically provided to serve as a non-volatile source of data. For example, such optical disks (e.g., CD-ROM, DVD, Blu-ray Disc, TM ), USB-RAM, portable, external hard drives, and floppy disks can be used as suitable sources of data to computer system 200. Typically, any of HDD 210, optical drive 212, and networks 220 and 222 may be configured to operate as video source 112 or as a destination for decoded video data to be stored for playback via display 214. Source device 110 and destination device 140 of system 100 may be embodied in computer system 200.

[0043] The components 205-213 of the computer module 201 typically communicate via an interconnected bus 204 and in a manner which results in a conventional mode of operation of the computer system 200 known to those skilled in the art. For example, the processor 205 is coupled to the system bus 204 using a connection 218. Similarly, the memory 206 and optical disk drive 212 are coupled to the system bus 204 by a connection 219. Examples of computers on which the described arrangements can be practiced include IBM-PCs and compatibles, SunSPARCstations, Apple Macs, and the like. TM , or a similar computer system.

[0044] If desired, PCA encoder 160 and PCA decoder 170, as well as the methods described below, may be implemented using computer system 200. Specifically, PCA encoder 160, PCA decoder 170, and the methods described may be implemented as one or more software application programs 233 executable within computer system 200. In particular, PCA encoder 160, PCA decoder 170, and the steps of the methods described are performed by instructions 231 (see FIG. 2B) in software 233 executing within computer system 200. The software instructions 231 may be formed as one or more code modules, each for performing one or more specific tasks. The software may also be divided into two separate parts, with a first part and corresponding code modules performing the methods described and a second part and corresponding code modules managing the user interface between the first part and a user.

[0045] The software may be stored on a computer-readable medium, including, for example, the storage devices described below. The software is loaded from the computer-readable medium into computer system 200 and then executed by computer system 200. A computer-readable medium having such a computer program or software recorded thereon is a computer program product. Use of the computer program product in computer system 200 preferably results in an advantageous apparatus for implementing source device 110 and destination device 140 and the methods described.

[0046] The software 233 is typically stored on the HDD 210 or memory 206. The software is loaded into the computer system 200 from a computer-readable medium and executed by the computer system 200. Thus, for example, the software 233 may be stored on an optically readable disk storage medium (e.g., a CD-ROM) 225 that is read by the optical disk drive 212.

[0047] In some examples, the application program 233 may be encoded on one or more CD-ROMs 225 and supplied to the user, and may be read via a corresponding drive 212, or may be read by the user from the network 220 or 222. Additionally, the software may be loaded into the computer system 200 from other computer-readable media. A computer-readable storage medium refers to any non-transitory, tangible storage medium that provides recorded instructions and / or data to the computer system 200 for execution and / or processing. Examples of such storage media are floppy disks, magnetic tape, CD-ROMs, DVDs, Blu-ray Discs, and the like. TM , hard disk drives, ROMs or integrated circuits, USB memory, magneto-optical disks, or computer-readable cards such as PCMCIA cards, etc., whether such devices are internal or external to the computer module 201. Examples of transitory or non-tangible computer-readable transmission media that may also participate in providing software, application programs, instructions, and / or video data or encoded video data to the computer module 201 include wireless or infrared transmission channels, as well as network connections to other computers or networked devices, and the Internet or intranets, including email transmissions and information stored on websites, etc.

[0048] A second portion of application program 233 and the corresponding code modules described above may be executed to implement one or more graphical user interfaces (GUIs) that are rendered or represented on display 214. A user of application and computer system 200, typically through operation of keyboard 202 and mouse 203, may manipulate the interface in a functionally adaptable manner to provide control commands and / or input to applications associated with the GUI. Other forms of functionally adaptable user interfaces may also be implemented, such as an audio interface that utilizes speech prompts output via loudspeaker 217 and user voice commands input via microphone 280.

[0049] Figure 2B is a detailed schematic block diagram of processor 205 and "memory" 234. Memory 234 represents the logical collection of all memory modules (including storage devices 209 and semiconductor memory 206) accessible by computer module 201 of Figure 2A.

[0050] When the computer module 201 is first powered on, a power-on self-test (POST) program 250 is executed. The POST program 250 is typically stored in the ROM 249 of the semiconductor memory 206 of FIG. 2A. Hardware devices, such as the ROM 249, that store software are sometimes referred to as firmware. The POST program 250 inspects the hardware within the computer module 201 to ensure proper functionality and typically checks the processor 205, memory 234 (209, 206), and a basic input / output system software (BIOS) module 251, also typically stored in the ROM 249, for correct operation. If the POST program 250 executes successfully, the BIOS 251 boots the hard disk drive 210 of FIG. 2A. The booting of the hard disk drive 210 causes a bootstrap loader program 252, resident on the hard disk drive 210, to execute via the processor 205. This loads the operating system 253 into the RAM memory 206, and the operating system 253 begins operation. Operating system 253 is a system-level application executable by processor 205 and performs a variety of high-level functions, including processor management, memory management, device management, storage management, software application interface, and general-purpose user interface.

[0051] The operating system 253 manages the memory 234 (209, 206) to ensure that each process or application running on the computer module 201 has enough memory to execute without conflicting with memory allocated to another process. Furthermore, the different types of memory available in the computer system 200 of FIG. 2A must be used appropriately so that each process can execute effectively. Thus, the aggregate memory 234 is not intended to indicate how specific segments of memory are allocated (unless otherwise specified), but rather is intended to provide a general view of the memory accessible by the computer system 200 and how such memory is used.

[0052] As shown in FIG. 2B, processor 205 includes several functional modules, including a control unit 239, an arithmetic logic unit (ALU) 240, and a local or internal memory 248, sometimes referred to as a cache memory. Cache memory 248 typically includes several storage registers 244-246 within a register section. One or more internal buses 241 functionally interconnect these functional modules. Processor 205 also typically has one or more interfaces 242 for communicating with external devices via system bus 204 using connection 218. Memory 234 is coupled to bus 204 using connection 219.

[0053] Application program 233 includes a set of instructions 231, which may include conditional branch and loop instructions. Program 233 may also include data 232 used in the execution of program 233. Instructions 231 and data 232 are stored in memory locations 228, 229, 230 and 235, 236, 237, respectively. Depending on the relative sizes of memory locations 228-230 and instructions 231, a particular instruction may be stored in a single memory location, as indicated by the instruction shown in memory location 230. Alternatively, an instruction may be segmented into several parts, each stored in a separate memory location, as indicated by the instruction segments shown in memory locations 228 and 229.

[0054] Generally, the processor 205 is given a set of instructions to execute therein. The processor 205 awaits a subsequent input to which the processor 205 responds by executing another set of instructions. Each input may be provided from one or more of several sources, including data generated by one or more of the input devices 202, 203, data received from an external source over one of the networks 220, 202, data retrieved from one of the storage devices 206, 209, or data retrieved from a storage medium 225 inserted into a corresponding reader 212, all shown in FIG. 2A. Execution of the set of instructions may in some cases result in the output of data. Execution may also involve storing data or variables in memory 234.

[0055] The PCA encoder 160, the PCA decoder 170, and the described methods may use input variables 254, which are stored in the memory 234 in corresponding memory locations 255, 256, and 257. The PCA encoder 160, the PCA decoder 170, and the described methods generate output variables 261, which are stored in the memory 234 in corresponding memory locations 262, 263, and 264. Intermediate variables 258 may be stored in memory locations 259, 260, 266, and 267.

[0056] 2B, registers 244, 245, 246, arithmetic logic unit (ALU) 240, and control unit 239 cooperate to perform the sequence of micro-operations required to perform a "fetch, decode, and execute" cycle for each instruction in the instruction set that makes up program 233. Each fetch, decode, and execute cycle: a fetch operation that fetches or reads instructions 231 from memory locations 228, 229, 230; a decode operation in which the control unit 239 determines which instructions have been fetched; and The control unit 239 and / or the ALU 240 execute instructions. Includes.

[0057] A further fetch, decode, and execute cycle for the next instruction may then be performed. Similarly, a store cycle may be performed by the control unit 239 storing or writing a value to the memory location 232.

[0058] Each step or sub-process in the described methods of FIGS. 16 and 17 is associated with one or more segments of program 233 and is typically performed by register sections 244, 245, 246, ALU 240, and control unit 239 within processor 205, which cooperate to perform fetch, decode, and execute cycles for all instructions in the instruction set for the referenced segment of program 233.

[0059] 3A is a schematic block diagram 300 illustrating the functional modules of a CNN backbone portion 310, which may serve as an implementation of the CNN backbone 114. The backbone portion 114 is sometimes referred to as "DarkNet-53," although different backbones are possible, resulting in different numbers of layers and dimensions of the tensor 115 for each frame.

[0060] As shown in Figure 3A, video data 113 is passed to resizer module 304. Resizer module 304 resizes each frame of video data 113 to a resolution suitable for processing by CNN backbone 310, generating resized frame data 312. If the resolution of video data 113 is already suitable for CNN backbone 310, operation of resizer module 304 is not required. Resized frame data 312 is passed to convolution batch normalized leaky rectified linear (CBL) module 314, generating tensors 316. CBL 314 includes modules such as those described with reference to CBL module 360, as shown in Figure 3D.

[0061] The CBL module 360 ​​takes tensor 361 of the resized frame data 312 as input. Tensor 361 is passed to a convolutional layer 362 to generate tensor 363. If convolutional layer 362 has a stride of 1, tensor 363 has the same spatial dimensions as tensor 361. If convolutional layer 362 has a larger stride, such as 2, tensor 363 has smaller spatial dimensions compared to tensor 361, e.g., the width and height of a stride of 2 are halved. Regardless of the stride, the size of the channel dimensions of tensor 363 may vary compared to the channel dimensions of tensor 361 for a particular CBL block. Tensor 363 is passed to a batch normalization module 364, which outputs tensor 365. The batch normalization module 364 normalizes the input tensor 363 and applies a scaling factor and an offset value to generate the output tensor 365. The scaling factor and offset value are derived from a training process. Tensor 365 is passed to leaky modified linear activation (“LeakyReLU”) module 366, which produces tensor 367. Module 366 provides a “leaky” activation function, where positive values ​​in the tensor are passed through and negative values ​​are significantly reduced in magnitude, e.g., to 0.1X their previous value.

[0062] Returning to FIG. 3A, tensor 316 is passed from CBL block 314 to a residual block module 320, such as a 1+2+8 module (also called an 11 module), which internally includes one residual unit, two residual units, and a concatenation of eight residual units.

[0063] The residual block is described with reference to ResBlock 340 shown in FIG. 3B. ResBlock 340 receives tensor 341. Tensor 341 is zero-padded by zero-padding module 342 to generate tensor 343. Tensor 343 is passed to CBL module 344 to generate tensor 345. Tensor 345 is passed to residual unit 346. Residual unit 346 includes a series of concatenated residual units based on the number of residual blocks (e.g., 11 units for block 320). The last residual unit in residual unit 346 outputs tensor 347.

[0064] As shown in FIG. 3C , the residual unit is described with reference to ResUnit 350. ResUnit 350 takes tensor 351 as input. Tensor 351 is passed to CBL module 352 to produce tensor 353. Tensor 353 is passed to a second CBL unit 354 to produce tensor 355. Addition module 356 sums tensor 355 with tensor 351 to produce tensor 357. Addition module 356 is sometimes referred to as a “shortcut” because input tensor 351 effectively affects output tensor 357. For untrained networks, ResUnit 350 operates on pass-through tensors. When training is performed, CBL modules 352 and 354 operate to separate tensor 357 from tensor 351 according to the training data and ground truth data.

[0065] Returning to FIG. 3A , Res11 module 320 outputs tensor 322. Tensor 322 is output from backbone module 310 as one of the layers and is also provided to Res8 module 324. Res8 module 324 is a residual block (i.e., 340) containing eight residual units (i.e., 350). Res8 module 324 generates tensor 326. Tensor 326 is passed to Res4 module 328 and output from backbone module 310 as one of the layers. Res4 module is a residual block (i.e., 340) containing four residual units (i.e., 350). Res4 module 328 generates tensor 329. Tensor 329 is output from backbone module 310 as one of the layers. Collectively, layer tensors 322, 326, and 329 are output as tensor 115a. The backbone CNN 310 can take a video frame with a resolution of 1088x608 as input and generate three tensors corresponding to three layers, with dimensions of [1, 256, 76, 136], [1, 512, 38, 68], and [1, 1024, 19, 34]. Another example of the three tensors 115 corresponding to the three layers may be [1, 512, 34, 19], [1, 256, 68, 38], and [1, 128, 136, 76], which are split at the 75th, 90th, and 105th feature maps in the CNN 310, respectively. The split points depend on the CNN 310.

[0066] Res11 320, Res8 324, and Res4 328 each operate similarly to ResBlock 340. CBL 314, CBL 344, and CBL 354 each operate similarly to CBL 360.

[0067] 4 is a schematic block diagram illustrating the functional modules of an alternative backbone portion 400 of a CNN that may serve as an implementation of the CNN backbone 114. The backbone portion 400 implements a residual network with a feature pyramid network ("ResNet FPN"), used in networks such as Faster RCNN and Mask RCNN. Frame data 113 is input and passes through a stem network 408, a res2 module 412, a res3 module 416, a res4 module 420, and a res5 module 424 via tensors 409, 413, 417, 421, and 425, respectively.

[0068] The stem network 408 includes a 7x7 convolution with a stride of 2 and a max-pooling operation. The res2 module 412, the res3 module 416, the res4 module 420, and the res5 module 424 perform convolution operations such as LeakyReLU activation. Each module 412, 416, 420, and 424 also divides the width and height of the processed tensor in half via a stride setting of 2. Each of the tensors 413, 417, 421, and 425 is passed to one of the 1x1 horizontal convolution modules 446, 444, 442, and 440, respectively. The modules 446, 444, 442, and 440 produce tensors 447, 445, 443, and 441, respectively. The tensor 441 is passed to the 3x3 output convolution module 470, which produces the output tensor P5 471.

[0069] Tensor 441 is also passed to upsampler module 450 to produce upsampled tensor 451. Summation module 460 sums tensors 443 and 451 to produce tensor 461. Tensor 461 is passed to upsampler module 452 and 3x3 horizontal convolution module 472. Module 472 outputs P4 tensor 473. Upsampler module 452 produces upsampled tensor 453. Summation module 462 sums tensors 445 and 453 to produce tensor 463. Tensor 463 is passed to 3x3 horizontal convolution module 474 and upsampler module 454. Module 474 outputs P3 tensor 475. Upsampler module 454 outputs upsampled tensor 455. Sum module 464 sums tensors 447 and 455 to produce tensor 465, which is passed to 3x3 horizontal convolution module 476. Module 476 outputs P2 tensor 477. Upsampler modules 450, 452, and 454 use nearest neighbor interpolation for low computational complexity. Tensors 429, 471, 473, 475, and 477 form output tensor 115a of CNN backbone 400. While Figure 4 shows a specific backbone portion of the Faster RCNN network architecture (the "P-layer split point"), different splits into backbone and head are possible. Splitting the network at tensor 409 is called the "stem" split point. Splitting the network at tensors 447, 445, 443, and 441 is called the "C-layer" split point.

[0070] Figure 5 is a schematic block diagram 500 illustrating one type of multi-scale feature fusion module 500 that may function as tensor combiner 162. Figure 6 is a schematic block diagram 600 illustrating an inter-channel decorrelation based tensor encoder. Figure 7 is a schematic block diagram 700 illustrating a sub-picture encoder. Encoders 600 and 700 form PCA encoder 160 of system 100. Figures 5-7 are described with reference to Figure 16.

[0071] FIG. 16 illustrates a method 1600 for executing a first portion of a CNN and encoding the resulting feature map for a frame of video data. In encoding the feature map, tensors are coded into a bitstream. The method 1600 may be implemented using an apparatus such as a configured FPGA, ASIC, or ASSP. Alternatively, as described below, the method 1600 may be implemented by the source device 110 as one or more software code modules of the application program 233 under execution by the processor 205. The software code modules of the application program 233 that implement the method 1600 may reside on the hard disk drive 210 and / or the memory 206, for example. The method 1600 is executed for each frame of video data generated by the video source 112. The method 1600 may be stored on a computer-readable storage medium and / or the memory 206. The method 1600 begins with a CNN first portion execution step 1610.

[0072] In step 1610, the CNN backbone 114, under execution by the processor 205, executes a subset of layers of a particular CNN to convert the input frame 113 into intermediate tensors 115a. The intermediate tensors 115a may be stored, for example, in the memory 206 and / or the hard disk drive 210. An example of a CNN is "Faster R-CNN" or "Mask R-CNN," as shown in FIG. 4, where the subset of layers corresponds to all layers up to the "P-layer" split point. For example, if multiple tensors are extracted from the CNN backbone 114 due to the use of an FPN, the tensors 115a include one tensor for each FPN layer. Control in the processor 205 passes from step 1610 to a tensor reduction execution step 1615.

[0073] In step 1615, the tensor combiner 162, operating under the execution of the processor 205, resamples the individual tensors in the tensors 115a and generates a single combined tensor 115 as an output. The multi-scale feature fusion (MSFF) module 500 includes an MSFF block 510, shown in FIG. 5, which generates a single tensor from the multiple tensors 115a using one or more downsampling filters. The MSFF block 510, operating under the execution of the processor 205, combines the tensors of the first set of tensors, i.e., 502, 503, 504, and 505, to generate the combined tensor 115a. The combined tensor 115a forms a representation of the FPN layer tensors. Downsample modules 522a, 522b, and 522c operate on tensors with larger spatial scales, i.e., P4 504 at 2h, 2w, 256, P3 503 at 4h, 4w, 256, and P2 502 at 8h, 8w, 256, respectively. Modules 522a, 522b, and 522c perform downsampling to match the spatial scale of the smallest tensor, i.e., P5 505 at h, w, 256, to generate downscaled P5 tensors 523a, 523b, 523c, respectively. Concatenation module 524 performs channel-wise concatenation of tensors 505, 523a, 523b, and 523c to generate concatenated tensor 525 of dimensions h, w, 1024. Concatenated tensor 525 is passed to aperture and excitation (SE) module 526 to generate tensor 527. The SE module 526 sequentially performs global pooling, a fully connected layer that reduces channel counts, a normalized linear unit activation unit, a second fully connected layer that restores channel counts, and a sigmoid activation function to generate a scaling tensor. The tensor 525 is scaled according to the scaling tensor to generate an output as tensor 527. The SE block 526 can be trained to adaptively change the weights of different channels in the tensor passed through it based on the output of the first fully connected layer. The output of the first fully connected layer reduces each feature map for each channel to a single value.Each single value is passed through a nonlinear activation unit (ReLU) to create a conditional representation of the unit appropriate for weighting the other channels, with restoration to the full channel count performed by a second fully connected layer. Thus, in generating tensor 527 from tensor 525, SE block 526 is able to extract nonlinear inter-channel correlations to a greater extent than would be possible using purely convolutional (linear) layers. Tensor 527 is passed to convolutional layer 528, which performs one or more convolutional layers to generate combined tensor 115 with a reduced channel count of F channels, typically 256 channels (i.e., F=256). Control in processor 205 passes from step 1615 to an average channel determination step 1620.

[0074] In alternative embodiments, convolutional layer 528 may be omitted from module 500, as indicated by the dashed line. In embodiments that omit convolutional layer 528, tensor 527 is output as tensor 115. In embodiments that effectively exclude 528, tensor 115 is generated using the output of a tensor summation involving the results of activation layers in SE block 526. The number of channels in tensor 115 is not reduced to F channels when layer 528 is omitted. For layers P2-P5, each with 256 channels, the channel count in tensor 115 when convolutional layer 528 is omitted is 1024. Applying convolutional layer 528 to scaled tensor 527 reduces the channel count, and therefore the dimensionality, of the tensor data fed to PCA encoder 160 and restored by PCA decoder 170, as described above.

[0075] In a mean channel determination step 1620, the tensor 115 is averaged. Referring to Figure 6, module 610, under execution of processor 205, performs an average operation on the tensor 115 across the spatial dimensions to generate per-channel averages 611. The mean channels 611 are quantized by quantizer 612, under execution of processor 205, to generate an integer (quantized) mean channel list 613 in step 1620, with the quantization range indicating the floating-point range required to hold the mean channels 611. Control in processor 205 passes from step 1620 to a mean channel encoding step 1630.

[0076] In a mean channel encoding step 1630, the subpicture encoder 614, under execution of the processor 205, packs the integer mean channels 613 into a subpicture (e.g., as subpicture 1410 shown in FIG. 14) and encodes the subpicture using the subpicture encoder 614 to generate bitstream portion 615. Because the mean channels 613 contain one value per channel, the channels 613 may be treated as feature maps of height 1 and width 256, for example, if they represent the mean of a tensor with 256 channels.

[0077] The sub-picture encoder 614 implements the architecture 700. The architecture 700 includes a feature map packer 710, a video frame encoder 714, and an unpacker 720. The sub-picture encoders 614, 636, 654a, 654b, and 654c are implemented as instances of the architecture 700. The packer 710 receives a tensor 708 having a given channel count, width, and height dimensions, and containing integer values, i.e., already quantized. For example, the sub-picture encoder 614 receives the integer average channel 613. The packer 710 packs the received tensor into a 2D planar array of samples. Generally, in the described configuration, the feature maps of each channel of the tensor 708 are stored as a sub-picture frame 712 in a left-to-right and top-to-bottom manner. The sub-picture frame 712 must be large enough to hold the channels of the tensor 708, including allowing for gaps in packing due to mismatches between the feature map size and the dimensions of the sub-picture frame 712. Referring to FIG. 8, the operation of a video frame encoder 714, typically implemented as a VVC encoder, is described. The encoder 714 generates coded bitstream portions 716 corresponding to each sub-picture. For example, the sub-picture encoder 614 outputs coded bitstream portion 615. The encoder 714 also outputs a reconstructed frame 718 corresponding to a lossy version that is reconstructed upon decoding the bitstream portion 716. The reconstructed frame 718 represents a reconstruction of the average channel 613, and loss caused by the encoding is modeled or represented. The loss reflects coding loss, such as that incurred by the particular encoding method used by the encoder 714. In the example described in FIGS. 7 and 8, the encoder 714 is a VVC encoder. The reconstructed frame 718 is passed to an unpacker 720. The unpacker 720 extracts the feature maps to generate a reconstructed tensor 722 having the same dimensionality as the tensor 708, and forms the tensor by performing a channel-wise concatenation of the unpacked feature maps.

[0078] As a result of the operation of module 700, PCA encoder 160 can use a version of the feature map (or coefficients) that corresponds to the version seen by PCA decoder 170. Thus, PCA encoder 160 can operate with higher fidelity than if the effects of lossy encoding were not considered. Subpicture encoders 614, 636, 654a, 654b, and 654c are configured to disable loop filtering both internally and across subpicture boundaries, since loop filtering is generally optimized for human consumption of the decoded picture. Furthermore, motion compensation is inhibited from accessing samples across subpicture boundaries via the "sps_subpic_treated_as_pic_flag" flag, activated for each subpicture. System 100 uses video compression to relatively efficiently represent data resulting from dimensionality reduction performed on intermediate tensor data that needs to be propagated from CNN backbone 114 to CNN head 150.

[0079] Returning to FIG. 16, control in processor 205 passes from step 1630 to a reconstructed mean channel recovery step 1640 .

[0080] In a reconstructed mean channel recovery step 1640, the subpicture encoder 614, under execution of the processor 205, outputs a reconstructed picture, e.g., 716, in the implementation of the encoder 700, which corresponds to a lossy version of the subpicture input for video compression, e.g., 712. The reconstructed picture is unpacked and output as an integer tensor 616. Control in the processor 205 passes from step 1640 to a zero mean tensor step 1645.

[0081] In step 1645, the inverse quantizer module 620 and the subtraction module 622, under execution by the processor 205, generate a zero-mean tensor 623. The inverse quantizer 620 takes the integer tensor 616 and uses the quantization range determined in the quantizer module 612 to output a reconstructed average channel 621. The average reconstructed channel 621 is a list of values, one value per channel in the tensor 115, corresponding to the detected DC offset in each channel. For each channel in the combined tensor 115, a DC shift is performed by subtracting a constant value for the feature map from each spatial location in the feature map. The subtracted values ​​are the respective values ​​in the average reconstructed channel 621. The result of the subtraction module 622 is a zero-centered tensor 623, where the DC component detected in each feature map has been removed from the respective feature map. Control in the processor 205 passes from step 1645 to a basis vector determination step 1650.

[0082] In a basis vector determination step 1650, the decomposition module 630 operates under the execution of the processor 205 to generate a set of basis vectors for the combined tensor 115. The decomposition module 630 receives the zero-centered tensor 623 as input and generates a set of basis vectors 631 by performing a principal component analysis method, such as singular value decomposition (SVD). One basis vector maps all channels to a single value; therefore, with 256 channels in the tensor 115, one basis vector has dimensions of 256x1. If the decomposition module generates the first N basis vectors, such as 25, the resulting basis vectors have dimensions of 256xN or 256x25. The basis vectors are relative to the origin; therefore, a zero-mean tensor 623 needs to be used to ensure that an orthonormal basis can be found. Each basis vector is a vector that associates all channels with a reduced set of channels. Thus, the basis vectors collectively allow the tensor data across all channels to be represented in a smaller set of basis vectors. Each basis vector is derived using all samples in each feature map for a given channel under consideration. Vector 631 contains fewer basis vectors than channels in tensor 115, corresponding to a reduction in the dimensionality of tensor 115. Basis vectors 631 represent tensor 115 in the subspace that explains the greatest amount of variance in tensor 115 relative to the number of components in basis vector 631. The basis vectors are ordered from vectors with the greatest explained variance to vectors with the least explained variance. In effect, basis vectors 631 allow representation of tensor 115 with minimal loss of quality for a given number of components, where the components are the first N ranked basis vectors. Control in processor 205 proceeds from step 1465 to a basis vector encoding step 1660.

[0083] In a basis vector encoding step 1660, the quantizer module 632, under execution of the processor 205, operates to quantize the basis vectors 631 into the integer domain. The resulting integer basis vectors 634 are packed into sub-pictures and encoded to generate bitstream portions 637 by a sub-picture encoder 636, under execution of the processor 205. Control in the processor 205 passes from step 1660 to a reconstructed basis vector recovery step 1670.

[0084] In step 1670, the subpicture encoder 636, under execution of the processor 205, operates to generate a reconstructed integer tensor 638 to obtain a reconstructed version of the subpicture 1412, e.g., 638. The basis vectors 638 are unpacked from the reconstructed subpicture and dequantized to the floating-point domain (as reconstructed basis vectors 640) by the inverse quantizer 660, under execution of the processor 205. Control in the processor 205 passes from step 1670 to a coefficient determination step 1680.

[0085] In step 1680, the dot product module 642, under execution of the processor 205, performs a dot product of each channel in the tensor 623 against each vector in the reconstruction basis vectors 640 to generate a coefficient tensor 644. The coefficients form a tensor having the same width and height as the tensor 115, but with a channel count corresponding to the number of components (or basis vectors) generated by the decomposition module 630, which is one less than the number of channels in the tensor 115. The coefficient tensor 644 represents the contribution of each basis vector in reproducing each value in each feature map. Control in the processor 205 passes from step 1680 to a coefficient quantization step 1690.

[0086] In step 1690, the quantizer module 646, under execution of the processor 205, quantizes the coefficient tensor 644 to generate integer coefficients 648, a tensor with the same dimensionality as the coefficient tensor 644, i.e., c channels, w width, and h height, where c corresponds to the number of basis vectors. The quantization range is determined from the extreme values ​​among the coefficient tensor 644. Control in the processor 205 passes from step 1690 to step 16100, which assigns coefficients to groups.

[0087] In performing either of steps 1680 and 1690, method 1600 operates to generate coefficients for a tensor using the (reduced) tensor and set of basis vectors generated in step 1615. The tensor for which coefficients are generated has the same spatial size and fewer channels as the tensor generated in step 1615.

[0088] In step 16100, the group module 650, under execution of the processor 205, determines a set of groups for the coefficient tensor 644. Each group contains coefficients for the entire feature map and one or more channels that are contiguous along the channel dimension. The first group always contains coefficients that use the first basis vector. In other words, the coefficient tensor 644 is divided along the channel dimension to form a set 652 of groups of coefficients that collectively have c channels, w width, and h height. Each group contains a contiguous range of channels and is contiguous with each other. Collectively, the groups contain all channels. The context of the coefficient tensor 644 is such that the channel dimension corresponds to the number of basis vectors, not the channel count of the combined tensor 115. The boundaries between groups are established when encoding the first frame of video data, which corresponds to establishing a sub-picture layout in the sequence parameter set (SPS) associated with an instantaneous decoding refresh (IDR) picture in the bitstream 121. Each group can be coded as a unit of coefficient information that is either included in the bitstream 121 or omitted. Thus, including the omission of groups of coefficients forms a means for rate control or quality scalability of system 100. Using larger groups incurs less coding overhead due to the use of fewer subpictures, but at the expense of coarser granularity of rate control. Groups containing coefficients of the first basis vector, i.e., the basis vector with the largest amount of explained variance, are relatively large and can always be coded, providing a fixed minimum quality level. Boundaries between slices whose corresponding basis vectors correspond to smaller amounts of explained variance can be set closer together, resulting in smaller groups—and therefore more groups—being required until all coefficients of all basis vectors can be coded in bitstream 121. Regardless of the number of groups, the total number of coefficients remains constant, as it is set by the number of basis vectors in joint tensor 115 and the feature map size.Apart from gaps resulting from packing coefficients within each group into separate sub-pictures (as described below in connection with Figures 14A and 14B), the coded area for the resulting packed frame data is substantially unaffected by dividing the coefficients into groups. Generally, the first group contains a relatively large number of coefficients necessary to provide the minimum fidelity of the reconstructed tensor that provides an adequate level of performance. Subsequent sub-pictures are then smaller in size, providing fewer coefficients per sub-picture and thus finer granularity for rate control. The final sub-picture is larger in size because it must occupy the remaining area within the rectangular picture 1400.

[0089] In effect, step 1650 operates to generate coefficients for a tensor using multiple basis vectors, with the coefficients forming one feature map for each basis vector of the multiple basis vectors. Upon determining the set of groups, step 16100 operates to generate assignments that group the coefficients according to their respective basis vectors. As described below, the assignments may be adjusted. The assignments may be based at least in part on the fidelity of the reconstructed tensor, i.e., using coefficients 670. Coefficients 670 are based on reconstructed feature frames generated in video encoders 614, 636, 654a, 654b, and 654c, where "reconstructed" indicates that the feature frames include artifacts resulting from the quantization and forward transform stages. Because entropy coding is lossless, it is not necessary to generate coefficients 670 by entropy decoding bitstream 121. Control in processor 205 passes from step 16100 to step 16110, which involves encoding groups of coefficients as subpictures.

[0090] In step 16110, each resulting group of grouped coefficients 652 is provided to a sub-picture encoder (e.g., one each of encoders 654a, 654b, and 654c) so that each group is processed by a separate sub-picture encoder, with enough instances of the sub-picture encoder to divide coefficients 648 into groups 652. Each sub-picture encoder generates a bitstream portion, resulting in a set of bitstream portions 656. Encoder 600 must select which bitstream portions within portions 656 to include in bitstream 121. The selection can be based on testing the fidelity of the reconstructed tensor with various numbers of bitstream portions or groups, as described in the remaining steps of method 1600. Control in processor 205 proceeds from step 16120, which is the step of selecting the first N groups of coefficients.

[0091] In step 16120, processor 205 selects the first N groups from groups 652, including the group with the basis vector with the largest explained variance, which, when grouped based on basis vector (and therefore explained variance) order, form a contiguous set of groups. Initially, the value N may be set as 1, i.e., to select the first group. Alternatively, the value N may be set to the value ultimately selected when processing the previous frame in a previous invocation of method 1600. Control in processor 205 passes from step 16120 to a reconstruct tensor step 16130.

[0092] In step 16130, under execution of the processor 205, the merge group module 664, the inverse quantizer module 668, and the dot product module 672 operate to generate a reconstructed tensor 674. The reconstructed tensor 674 is a version of the tensor 637 that takes into account the loss resulting from quantization from the floating-point domain to the integer sample domain (i.e., from modules 612, 632, and 646), the loss from the forward transform modules 826 and 830 in the video encoder 800, and the sample-domain quantization from the quantizer module 834. Because the operation of the entropy encoder 838 is lossless, there is no further mismatch between the reconstructed tensor 674 and the tensor 942 derived in the PCA decoder 170. The reconstructed tensor 674 is generated using the reconstructed coefficients 662 for the group of coefficients selected in step 16120. The reconstructed tensor 674 is sometimes referred to as a "projection" of the basis vectors with the coefficients because the tensor 674 occupies a subspace of higher dimensionality than the subspace occupied by the coefficients. The merge group module 664 merges each group of coefficients, resulting in a single tensor of coefficients. In other words, the merge group module 664 performs a concatenation of groups of coefficients along the channel dimension corresponding to the basis vector in the transformed domain to generate reconstructed integer coefficients 666. The reconstructed coefficients 666 are dequantized from the integer domain to the floating-point domain using the quantization range determined by the quantizer module 646 to generate reconstructed coefficients 670. The dot product module 672 is used to generate the reconstructed tensor 674 by performing a dot product operation on the reconstructed coefficients 670 and the reconstructed basis vectors 640. The results of the dot product module 672 are added between groups of coefficients. When adding one group of coefficients, a delta of the reconstruction tensor 674 may be generated by performing a dot product 672 using only the coefficients and basis vectors in the additional group, and the delta is added to the previously determined reconstruction tensor 674 to generate an updated reconstruction tensor 674. Control in the processor 205 passes from step 16130 to an error measurement step 16140.

[0093] In step 16140, mean squared error (MSE) module 676 generates an MSE result that takes as input the zero-centered tensor 623 and the reconstructed tensor 674. The MSE result provides an indication of the fidelity of the reconstructed tensor 674. Higher fidelity of the reconstructed tensor 674 reduces the loss introduced by feature compression in system 100 compared to the performance achieved by a network embodied in system 100 without being split into portions, but requires more bitstream portions to be coded, and therefore higher bandwidth in communication channel 130. Control in processor 205 passes from step 16140 to an error threshold test step 16150.

[0094] In step 16150, processor 205 compares the MSE result from step 16140 to a threshold. The threshold may be a predetermined value or a running average of previously encountered MSE results. If the MSE result is below the threshold and the number of currently selected groups is less than all available groups, step 1650 returns "YES" and control in processor 205 passes from step 16150 to adjust N step 16160. Otherwise, step 1650 returns "NO" and control in processor 205 passes from step 16150 to select N subpictures step 16170.

[0095] Steps 16130-16150 operate to encode and decode coefficients such that each group of coefficients is encoded and decoded independently. The encoding and decoding is used in step 16150 to determine whether the assignments determined in step 16100 should be adjusted. As described in connection with modules 646-668, the assignment or grouping of coefficients may be based on a distance metric (delta), such as the mean squared error (MSE) or sum of absolute differences (SAD), between tensor 674 and tensor 623 resulting from a dot product operation on the decoded or reconstructed coefficients.

[0096] In step 16160, the number of selected groups is incremented by one, resulting in the selection of one additional group, the next group containing coefficients with more explained variance among the currently non-selected groups. Steps 16100 through 16160 can therefore be thought of as first generating an assignment that groups coefficients according to their respective basis vectors, and then selecting which subset of the groups of coded coefficients should be included in bitstream 121. Control in processor 205 passes from step 16160 to step 16130.

[0097] In step 16170, bitstream portions of bitstream portion 656 are selected by gating module 658 such that only bitstream portions corresponding to the first N groups of subpictures determined in step 16120 are selected. The selected bitstream portions are propagated by gating module 658 to form selected bitstream portion 660. For subpictures that are not selected, either a "gap" or uncoded region in picture 1400 may result, or the subpicture may be replaced with a bitstream portion having content that is compactly represented and consumes negligible bitrate, such as DC midtone values. Control in processor 205 passes from step 16170 to coefficient group coding step 16180.

[0098] In step 16180, the processor 205 encodes signaling into the bitstream indicating the coefficient grouping and coding. In other words, the coefficients of each tensor are coded into the bitstream for each frame. When coding an IDR picture, signaling is coded to indicate the division of the coefficients into groups. For every frame, signaling is coded to indicate which groups are present in the bitstream 121. For example, a value N indicating the use of the first N groups is coded into the bitstream, or a bitmap indicating the use of a particular group is coded into the bitstream.

[0099] Step 16170 operates to encode the subpicture arrangement described by the allocations generated by steps 16100-16160, while step 16180 operates to encode the allocations into a bitstream. Once a change in allocation is determined, a new subpicture structure is initiated, as described in connection with Figure 15. Control in processor 205 passes from step 16180 to a bitstream write step 16190.

[0100] In step 16190, the sub-picture bitstream combiner 680 combines (or "writes") the bitstream portions 615, 637, and 660 to generate (or "write") the single bitstream 121. Once step 16190 is complete, the method 1600 is finished for the current frame 113 from the video source 112.

[0101] 8 is a schematic block diagram 800 illustrating the functional modules of the video encoder 714. The video encoder 714 encodes one subpicture among a set of subpictures that comprise an overall picture. While it is possible for all subpictures to be encoded in one encoding pass, using one encoding pass prevents a lossy decoded version of the data in a given subpicture from being used as input when generating data to be encoded in another subpicture; therefore, the PCA encoder 160 cannot consider lossy encoding in the pipeline. Generally, data passes between functional modules in the video encoder 714 in groups of samples or coefficients, such as dividing a block into fixed-size subblocks, or as an array. 2A and 2B , the video encoder 714 may be implemented using a general-purpose computer system 200, with various functional modules implemented by dedicated hardware within the computer system 200 or by software executable within the computer system 200, such as one or more software code modules of a software application program 233 resident on a hard disk drive 205 and controlled in its execution by the processor 205. Alternatively, the video encoder 714 may be implemented by a combination of dedicated hardware and software executable within the computer system 200. The video encoder 714 and the described methods may alternatively be implemented with dedicated hardware, such as one or more integrated circuits that perform the functions or sub-functions of the described methods. Such dedicated hardware may include a graphics processing unit (GPU), a digital signal processor (DSP), an application specific standard product (ASSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or one or more microprocessors and associated memories. In particular, the video encoder 714 comprises modules 810-890, each of which may be implemented as one or more software code modules of the software application program 233.

[0102] While the video encoder 714 in FIG. 7 is an example of a generic video coding (VVC) video encoder, other video codecs may be used to perform the processing stages described herein. For example, HEVC may be used. The described example generates a bitstream of encoded data. If other codecs are used, some embodiments may pack the data into a different format, such as a frame format. The video encoder 714 receives subpicture frame data 712, such as a series of frames of a subpicture, where each frame includes one or more color channels. The frame data 712 may be in any chroma format and bit depth supported by the profile in use, e.g., 4:0:0, 4:2:0 for the "Main 10" profile of the VVC standard, with 8-10 bits of sample precision. The block partitioner 810 first divides the frame data 712 into CTUs that are roughly square in shape and configured to use a specific size for the CTU. The maximum valid size of a CTU can be, for example, 32x32, 64x64, or 128x128 luma samples, as configured by the "sps_log2_ctu_size_minus5" syntax element present in the "sequence parameter set." The CTU size also provides the maximum CU size, since a CTU without further division contains one CU. The block partitioner 810 further divides each CTU into one or more CBs according to the luma coding tree and the chroma coding tree. The luma channel may also be referred to as the primary color channel. Each chroma channel may also be referred to as the secondary color channel. CBs have various sizes and can include both squared and non-squared aspect ratios. However, in the VVC standard, CBs, CUs, PUs, and TUs always have side lengths that are powers of two. Thus, a current CB, denoted as 812, is output from the block partitioner 810, which proceeds according to the iteration over one or more blocks of the CTU according to the luma coding tree and the chroma coding tree of the CTU.

[0103] The CTUs resulting from the initial partitioning of the frame data 712 may be scanned in raster scan order and grouped into one or more "slices." A slice may be an "intra" (or "I") slice. An intra slice (I slice) indicates that all CUs in the slice are intra predicted. Generally, the first picture in a coding layer video sequence (CLVS) contains only I slices and is called an "intra picture." A CLVS may include periodic intra pictures that form "random access points" (i.e., intermediate frames in the video sequence where decoding can begin). Alternatively, a slice may be uni-predictive or bi-predictive ("P" or "B" slice, respectively), indicating the additional availability of uni-prediction and bi-prediction in the slice, respectively.

[0104] The video encoder 714 encodes a sequence of pictures according to a picture structure. One picture structure is "low latency," in which pictures using inter prediction may only reference pictures that occur earlier in the sequence. Low latency allows each picture to be output as soon as it is decoded, in addition to being stored for possible reference by subsequent pictures. Another picture structure is "random access," in which the coding order of pictures differs from the display order. Random access allows inter-predicted pictures to reference other pictures that have been decoded but not yet output. A degree of picture buffering is required so that future reference pictures with respect to display order are present in the decoded picture buffer, resulting in a delay of multiple frames.

[0105] If a chroma format other than 4:0:0 is used, in an I slice, the coding tree of each CTU may diverge below the 64x64 level into two separate coding trees, one for luma and one for chroma. The use of separate trees allows for different block structures between luma and chroma within the luma 64x64 area of ​​a CTU. For example, a large chroma CB may be co-located with multiple smaller luma CBs, and vice versa. In a P or B slice, a single coding tree for a CTU defines a common block structure for luma and chroma. The resulting blocks of the single tree may be intra-predicted or inter-predicted.

[0106] In addition to dividing a picture into slices, a picture may also be divided into "tiles." A tile is a sequence of CTUs that cover a rectangular area of ​​the picture. CTU scanning occurs in a raster-scan fashion within each tile, progressing from one tile to the next. A slice can be either an integer number of tiles or an integer number of consecutive rows of CTUs within a given tile.

[0107] For each CTU, the video encoder 714 operates in two stages. In the first stage (called the “search” stage), the block partitioner 810 tests various potential configurations of the coding tree. Each potential configuration of the coding tree has an associated “candidate” CB. The first stage involves testing various candidate CBs to select a CB that provides relatively high compression efficiency with relatively low distortion. The testing generally involves Lagrangian optimization, whereby the candidate CBs are evaluated based on a weighted combination of rate (i.e., coding cost) and distortion (i.e., error relative to the input frame data 712). The “best” candidate CB (i.e., the CB with the lowest evaluated rate / distortion) is selected for subsequent encoding into the bitstream portion 716. Included in the evaluation of the candidate CBs are the options of using the CB for a given area, or further dividing the area according to various partitioning options and encoding each of the smaller resulting areas with additional CBs, or further dividing the area. As a result, both the coding tree and the CB itself are selected in the search stage.

[0108] For each CB, e.g., CB 812, the video encoder 714 generates a predictive block (PB), indicated by arrow 820. The PB 820 is a prediction of the content of the associated CB 712. A subtractor module 822 generates a difference (or a "residual," which refers to the difference in the spatial domain) between the PB 820 and the CB 812, indicated as 824. The difference 824 is a block-sized difference between corresponding samples in the PB 820 and the CB 812. The difference 824 is transformed and quantized and represented as a transform block (TB), indicated by arrow 836. The PB 820 and the associated TB 836 are typically selected from one of many possible candidate CBs, for example, based on estimated cost or distortion.

[0109] A candidate coding block (CB) is a CB that results from one of the prediction modes available to the video encoder 714 for the associated PB and the resulting residual. The TB 836, when combined with the predicted PB in the video encoder 714, reduces the difference between the decoded CB and the original CB 812, at the expense of additional signaling in the bitstream.

[0110] Thus, each candidate coding block (CB), i.e., a prediction block (PB) combined with a transform block (TB), has an associated coding cost (or “rate”) and an associated differential (or “distortion”). The distortion of a CB is typically estimated as a difference in sample values, such as the sum of absolute differences (SAD), the sum of squared differences (SSD), or a Hadamard transform applied to the difference. The resulting estimate from each candidate PB can be determined by a mode selector 886 using the differentials 824 to determine a prediction mode 887. The prediction mode 887 indicates a particular prediction mode for the current CB, e.g., a decision to use intra-frame prediction or inter-frame prediction. Estimating the coding cost associated with each candidate prediction mode and the corresponding residual coding can be performed at a significantly lower cost than entropy coding of the residual. Thus, several candidate modes can be evaluated to determine the optimal mode in a rate-distortion sense, even in a real-time video encoder.

[0111] Determining the optimal mode in terms of rate distortion is typically accomplished using a variation of Lagrangian optimization. A Lagrangian or similar optimization process may be used both to select the optimal partitioning of the CTUs into CBs (by the block partitioner 810) as well as to select the best prediction mode from multiple possibilities. Through application of the Lagrangian optimization process to the candidate modes in the mode selector module 886, the intra-prediction mode with the lowest cost measure is selected as the "best" mode. The lowest-cost mode includes the selected secondary transform index 888, which is also encoded in the bitstream 716 by the entropy encoder 838.

[0112] In the second stage of operation of the video encoder 714 (called the "encoding" stage), iterations over the determined coding trees for each CTU are performed in the video encoder 714. For CTUs that use separate trees, for each 64x64 luma region of the CTU, the luma coding tree is coded first, followed by the chroma coding tree. Within the luma coding tree, only the luma CB is coded, and within the chroma coding tree, only the chroma CB is coded. For CTUs that use a shared tree, a single tree describes the CUs (i.e., the luma CBs and chroma CBs) according to the common block structure of the shared tree.

[0113] The entropy encoder 838 supports bitwise coding of syntax elements using variable-length and fixed-length codewords, as well as arithmetic coding modes for syntax elements. Portions of the bitstream, such as "parameter sets," such as the sequence parameter set (SPS) and picture parameter set (PPS), use a combination of fixed-length and variable-length codewords. Slices, also called contiguous portions, have a slice header that uses variable-length coding, followed by slice data that uses arithmetic coding. The slice header defines parameters specific to the current slice, such as slice-level quantization parameter offsets. The slice data contains syntax elements for each CTU in the slice. The use of variable-length coding and arithmetic coding requires sequential parsing within each portion of the bitstream. Portions can be delineated with start codes to form "network abstraction layer units" or "NAL units." Arithmetic coding is supported using a context-adaptive binary arithmetic coding process.

[0114] Arithmetically coded syntax elements consist of a sequence of one or more "bins." Bins, like bits, have a value of "0" or "1." However, bins are not coded as separate bits in the bitstream portion 716. Bins have an associated predicted (or "likely" or "most likely") value and an associated probability, known as a "context." If the actual bin to be coded matches the predicted value, a "most probable symbol" (MPS) is coded. Coding the most probable symbol is relatively inexpensive in terms of consumed bits in the bitstream portion 716 and involves a cost equivalent to less than one separate bit. If the actual bin to be coded does not match a possible value, a "least probable symbol" (LPS) is coded. Coding the least probable symbol has a relatively high cost in terms of consumed bits. Bin coding techniques allow for efficient coding of bins where the probability of "0" versus "1" is skewed. For syntax elements with two possible values ​​(i.e., "flags"), a single bin is appropriate. For syntax elements with many possible values, a sequence of bins is required.

[0115] The presence of a later bin in the sequence may be determined based on the value of a previous bin in the sequence. Furthermore, each bin may be associated with two or more contexts. The selection of a particular context may depend on previous bins in the syntax element, bin values ​​of adjacent syntax elements (i.e., from adjacent blocks), etc. Each time a context-coded bin is coded, the context selected for that bin (if any) is updated in a manner that reflects the new bin value. In this way, binary arithmetic coding schemes are said to be adaptive.

[0116] Also supported by the entropy encoder 838 are bins that lack context, called "bypass bins." Bypass bins are coded assuming an equal probability distribution between "0" and "1." Thus, each bin has a coding cost of 1 bit in the bitstream portion 716. The lack of context saves memory and reduces complexity, and thus bypass bins are used when the distribution of values ​​for a particular bin is skewed. One example of an entropy coder that uses context and adaptation is known in the art as CABAC (Context-Adaptive Binary Arithmetic Coder), and many variations of this coder are used in video coding.

[0117] The QP controller 890 determines a quantization parameter 892 that is used to establish the quantization step size for use by the quantizer 834 and the inverse quantizer 840. A larger quantization step size causes the primary transform coefficients 828 to be quantized to smaller values, reducing the bit rate of the bitstream portion 716 at the expense of reducing the fidelity of the inverse transform coefficients 846.

[0118] The entropy encoder 838 uses a combination of context-coded and bypass-coded bins to encode a quantization parameter 892 and, if used for the current CB, an LFNST index 888. The quantization parameter 892 is coded at the beginning of each slice, and changes in the quantization parameter 892 within a slice are coded using a "delta QP" syntax element. The delta QP syntax element is signaled at most once in each area known as a "quantization group." The quantization parameter 892 is applied to the residual coefficients of the luma CB. An adjusted quantization parameter is applied to the residual coefficients of the co-located chroma CB. The adjusted quantization parameter may include mapping from the luma quantization parameter 892 according to a CU-level offset selected from a list of offsets and a mapping table. The secondary transform index 888 is signaled when the residual associated with the transform block contains significant residual coefficients only at coefficient positions that are transformed into primary coefficients by application of a secondary transform.

[0119] The residual coefficients of each TB associated with a CB are coded using a residual syntax. The residual syntax is designed to efficiently code low-magnitude coefficients by using primarily arithmetically coded bins to indicate coefficient significance, along with lower magnitude values, and reserving bypass bins for higher-magnitude residual coefficients. Thus, residual blocks containing very small magnitude values ​​and a sparse distribution of significant coefficients are efficiently compressed. Furthermore, two residual coding schemes exist. The regular residual coding scheme is optimized for TBs with significant coefficients located primarily in the upper-left corner of the TB, as occurs when a transform is applied. The transform-skip residual coding scheme is available for TBs where no transform is performed and can efficiently code residual coefficients regardless of their distribution across the TB.

[0120] The multiplexer module 884 outputs the PB 820 from the intra-frame prediction module 864 according to the determined best intra-prediction mode selected from the tested prediction modes of each candidate CB. The candidate prediction modes need not include all possible prediction modes supported by the video encoder 714. Intra-prediction is divided into three types: first, "DC intra-prediction," which involves populating the PB with a single value representing the average of nearby reconstructed samples; second, "planar intra-prediction," which involves populating the PB with samples according to a plane, where the DC offset and vertical and horizontal gradients are derived from nearby reconstructed neighboring samples. The nearby reconstructed samples typically include a row of reconstructed samples above the current PB that extends to the right of the PB and a column of reconstructed samples to the left of the current PB that extends downward beyond the PB; and third, "angular intra-prediction," which involves populating the PB with reconstructed neighboring samples that have been filtered and propagated across the PB in a specific direction (or "angle"). In VVC, 65 angles are supported, and rectangular blocks can utilize additional angles not available to square blocks, yielding a total of 87 angles.

[0121] A fourth type of intra prediction is available for chroma PBs, whereby the PB is generated from co-located luma reconstructed samples according to a "cross-component linear model" (CCLM) mode. Three different CCLM modes are available, each using a different model derived from adjacent luma and chroma samples. The derived model is used to generate a block of samples for the chroma PB from the co-located luma samples. The luma block can be intra-predicted using matrix multiplication of a reference sample using one matrix selected from a predefined set of matrices. This matrix intra prediction (MIP) achieves gains by using a matrix trained on a large set of video data, where the matrix represents a relationship between the predicted block and the reference sample that is not easily captured in angular, planar, or DC intra prediction modes.

[0122] The module 864 may also generate prediction units by copying blocks from nearby current frames using an "intra block copy" (IBC) method. The location of the reference blocks is constrained to an area equivalent to one CTU divided into 64x64 regions known as VPDUs. The area covering the VPDUs of the previous CTU and the processed VPDU of the current CTU within each row or CTU and each slice or tile is limited to an area corresponding to one 128x128 luma sample, regardless of the configured CTU size of the bitstream. This area, known as the "IBC virtual buffer," limits the IBC reference area and therefore the required storage. The IBC buffer is populated with reconstructed samples 854 (i.e., before loop filtering), thus requiring a separate buffer to the frame buffer 872. If the CTU size is 128x128, the virtual buffer contains only samples from the CTU adjacent to and to the left of the current CTU. When the CTU size is 32x32 or 64x64, the virtual buffer contains up to 4 or 16 CTUs to the left of the current CTU. Regardless of the CTU size, access to neighboring CTUs to obtain samples for IBC reference blocks is constrained by boundaries such as picture, slice, or tile edges. In particular, for FPN layer feature maps with smaller dimensions, using a CTU size such as 32x32 or 64x64 results in a more aligned reference region that covers the previous set of feature maps. When feature map arrangements are ordered based on SAD, SSE, or other difference metrics, accessing similar feature maps for IBC prediction provides coding efficiency benefits.

[0123] The residuals of prediction blocks when encoding feature map data differ from those found for natural video. Natural video is typically captured by an image sensor or screen content, such as that commonly found in operating system user interfaces. Feature map residuals tend to contain a lot of detail. The level of detail in feature map residuals is suitable for transforming more coding skips than the primarily low-frequency coefficients of various transforms. An intra-predicted luma coding block can be divided either vertically or horizontally into a set of equally sized prediction blocks, each with a minimum area of ​​16 luma samples.

[0124] If no previously reconstructed neighboring samples are available, such as at the edge of a frame, a default halftone value of half the range of samples is used. For example, for 10-bit video, a value of 512 is used. Because no previous samples are available for the CB located in the top-left position of the frame, angular and planar intra prediction modes produce the same output as DC prediction mode (i.e., a plane of samples with halftone values ​​as magnitudes).

[0125] For inter-frame prediction, the prediction block 882 is generated by the motion compensation module 880 using samples from one or two frames preceding the current frame in the coding order in the bitstream and output as the PB 820 by the multiplexer module 884. Furthermore, for inter-frame prediction, a single coding tree is typically used for both the luma and chroma channels. The order in which frames in the bitstream are coded may differ from the order of the frames when captured or displayed. If one frame is used for prediction, the block is said to be "uni-predictive" and has one associated motion vector. If two frames are used for prediction, the block is said to be "bi-predictive" and has two associated motion vectors. For P slices, each CU can be intra-predicted or uni-predicted. For B slices, each CU can be intra-predicted, uni-predicted, or bi-predicted.

[0126] Frames are typically coded using a "group of pictures" structure, allowing for temporal hierarchy of frames. A frame may be divided into multiple slices, each of which encodes a portion of the frame. The temporal hierarchy of frames allows frames to reference previous and subsequent pictures in the order in which they are displayed. Images are coded in the order necessary to ensure that dependencies for decoding each frame are satisfied. Instead of using one or two motion vectors to select and filter reference sample blocks for a prediction unit, an affine inter-prediction mode is available. The prediction unit is divided into multiple smaller blocks, and a motion field is generated, with each smaller block having a separate motion vector. The motion field uses motion vectors of points near the prediction unit as "control points." Affine prediction allows for coding of different motions, distinct from translation, with less need for deeply split coding trees. The bi-prediction mode available for VVC performs geometric blending of two reference blocks along a selected axis with an angle and offset from the center of the signaled block. This geometric partitioning mode ("GPM") allows for the use of larger coding units along the boundary between two objects, with the boundary geometry coded for the coding unit as an angle and center offset. Instead of using Cartesian (x, y) offsets, motion vector differences can be coded as a direction (up / down / left / right) and distance, with a set of power-of-two distances supported. The motion vector predictor is obtained from neighboring blocks ("merge mode") as if no offset was applied. The current block shares the same motion vector with the selected neighboring block.

[0127] Samples are selected according to a motion vector 878 and a reference picture index. The motion vector 878 and the reference picture index apply to all color channels; therefore, inter prediction is primarily described in terms of operation on the PU, not the PB. The decomposition of each CTU into one or more inter prediction blocks is described using a single coding tree. Inter prediction methods may vary in the number of motion parameters and their precision. The motion parameters typically comprise a reference frame index indicating which reference frame from a list of reference frames should be used and a spatial transformation for each of the reference frames, but may include more frames, dedicated frames, or complex affine parameters such as scaling and rotation. In addition, a predetermined motion refinement process may be applied to generate a dense motion estimate based on the referenced sample block.

[0128] Determining and selecting a PB 820 and subtracting it from the original sample block in a subtractor 822 results in a residual with the lowest coding cost, represented as 824, which undergoes lossy compression. The lossy compression process includes the steps of transform, quantization, and entropy coding. A forward linear transform module 826 applies a forward transform to the difference 824, converting it from the spatial domain to the frequency domain and generating linear transform coefficients, represented by arrow 828. The maximum linear transform size in one dimension is either a 32-point DCT-2 or a 64-point DCT-2 transform, configured by the "sps_max_luma_transform_size_64_flag" in the sequence parameter set. If the CB being coded is larger than the maximum supported linear transform size, represented as a block size (e.g., 64x64 or 32x32), the linear transform 826 is applied in a tiled manner to transform all samples of the difference 824. If a non-square CB is used, tiling is also performed using the largest available transform size in each dimension of the CB. For example, if a maximum transform size of 32 is used, a 64x16 CB uses two 32x16 primary transforms arranged in a tiled fashion. If the CB is larger than the maximum supported transform size, the CB is filled with TBs in a tiled fashion. For example, a 128x128 CB with a 64-pt transform maximum size is filled with four 64x64 TBs arranged in a 2x2 fashion. A 64x128 CB with a 32-pt transform maximum size is filled with eight 32x32 TBs arranged in a 2x4 fashion.

[0129] Application of the transform 826 results in multiple TBs for the CB. If each application of the transform operates on a TB of the difference 824 larger than 32x32, e.g., 64x64, all resulting primary transform coefficients 828 outside the top-left 32x32 area of ​​the TB are set to zero (i.e., discarded). The remaining primary transform coefficients 828 are passed to a quantizer module 834. The primary transform coefficients 828 are quantized according to a quantization parameter 892 associated with the CB to generate primary transform coefficients 832. In addition to the quantization parameter 892, the quantizer module 834 may also apply a "scaling list" to enable non-uniform quantization within the TB by further scaling the residual coefficients according to their spatial position within the TB. The quantization parameter 892 may be different for the luma CB versus each chroma CB. The primary transform coefficients 832 are passed to a forward secondary transform module 830, which generates transform coefficients represented by arrow 836 by performing a non-separable secondary transform (NSST) operation or by bypassing the secondary transform. The forward primary transform 826 is typically separable, transforming a set of rows and then a set of columns of each TB. The forward primary transform module 826 uses either a Type II Discrete Cosine Transform (DCT-2) in the horizontal and vertical directions, or bypassing the transform in the horizontal and vertical directions, or a combination of a Type VII Discrete Sine Transform (DST-7) and a Type VIII Discrete Cosine Transform (DCT-8) in the horizontal or vertical directions for luma TBs whose width and height do not exceed 16 samples. The use of a combination of a DST-7 and a DCT-8 is referred to as a "multiple transform selection set" (MTS) in the VVC standard.

[0130] The forward secondary transform of module 830 is generally a non-separable transform, which is applied only to the residual of intra-predicted CUs and may nevertheless be bypassed. The forward secondary transform operates on either 16 samples (organized as a 4x4 sub-block at the top left of the primary transform coefficients 828) or 48 samples (organized as three 4x4 sub-blocks at the top left of the primary transform coefficients 828) to generate a set of secondary transform coefficients. The set of secondary transform coefficients may be smaller in number than the set of primary transform coefficients from which they are derived. By applying the secondary transform only to sets of coefficients that are adjacent to each other and include the DC coefficient, the secondary transform is referred to as a "low-frequency non-separable secondary transform" (LFNST). Such secondary transforms are obtainable through a training process and due to their non-separable nature and trained origin, can exploit additional redundancy in the residual signal that cannot be captured by separable transforms such as DCT and DST variants. Furthermore, when an LFNST is applied, all remaining coefficients in the TB are zero, both in the primary transform domain and the secondary transform domain.

[0131] The quantization parameter 892 is constant for a given TB, thus resulting in uniform scaling for generating residual coefficients in the primary transform domain for the TB. The quantization parameter 892 may vary periodically with a signaled "delta quantization parameter." The delta quantization parameter (delta QP) is signaled once for CUs contained within a given area, called a "quantization group." If the CU is larger than the quantization group size, the delta QP is signaled once with one of the TBs of the CU. That is, the delta QP is signaled by the entropy encoder 838 once for the first quantization group of the CU and not for any subsequent quantization groups of the CU. Non-uniform scaling is also possible by applying a "quantization matrix," whereby the scaling factor applied to each residual coefficient is derived from a combination of the quantization parameter 892 and a corresponding entry in the scaling matrix. The scaling matrix can have a size smaller than the TB size, and when applied to the TB, a nearest-neighbor approach is used to provide a scaling value for each residual coefficient from the scaling matrix, which has a size smaller than the TB size. The residual coefficients 836 are provided to an entropy encoder 838 for encoding in the bitstream portion 716. Typically, the residual coefficients of each TB having at least one significant residual coefficient of a TU are scanned to generate an ordered list of values ​​according to a scan pattern. The scan pattern generally scans the TB as a sequence of 4x4 "subblocks," providing a regular scan operation with a granularity of 4x4 sets of residual coefficients, with the arrangement of the subblocks depending on the size of the TB. The scan within each subblock and the progression from one subblock to the next typically follows a backward diagonal scan pattern. Additionally, a quantization parameter 892 is coded in the bitstream portion 716 using a delta QP syntax element, and an initial slice QP for a given slice or subpicture and secondary transform index 888 is coded in the bitstream portion 716.

[0132] As described above, the video encoder 714 needs to access a frame representation that corresponds to the decoded frame representation seen by the video decoder. Accordingly, the residual coefficients 836 pass through an inverse secondary transform module 844, operating according to secondary transform indices 888 to generate intermediate inverse transform coefficients represented by arrow 842. The intermediate inverse transform coefficients 842 are inverse quantized by an inverse quantization module 840 according to a quantization parameter 892 to generate inverse transform coefficients represented by arrow 846. The inverse quantization module 840 may also perform inverse non-uniform scaling of the residual coefficients using a scaling list, corresponding to the forward scaling performed in the quantizer module 834. The inverse transform coefficients 846 are passed to an inverse primary transform module 848 to generate residual samples of the TUs, represented by arrow 850. The inverse primary transform module 848 applies a DCT-2 transform horizontally and vertically, constrained by the maximum available transform size, as described with reference to the forward primary transform module 826. The type of inverse transform performed by the inverse secondary transform module 844 corresponds to the type of forward transform performed by the forward secondary transform module 830. The type of inverse transform performed by the inverse primary transform module 848 corresponds to the type of primary transform performed by the primary transform module 826. A summation module 852 adds the residual samples 850 and the PU 820 to generate reconstructed samples for the CU (indicated by arrow 854).

[0133] The reconstructed samples 854 are passed to a reference sample cache 856 and an in-loop filter module 868. The reference sample cache 856, typically implemented using static RAM on an ASIC to avoid costly off-chip memory accesses, provides the minimum sample storage required to satisfy dependencies for generating intra-frame PBs for subsequent CUs in the frame. The minimum dependencies typically include a "line buffer" of samples along the bottom of a row of CTUs for use by the next row of CTUs, and column buffering, the extent of which is set by the height of the CTU. The reference sample cache 856 supplies reference samples (represented by arrow 858) to a reference sample filter 860. The sample filter 860 applies a smoothing operation to generate filtered reference samples (indicated by arrow 862). The filtered reference samples 862 are used by an intra-frame prediction module 864 to generate an intra-predicted block of samples, represented by arrow 866. For each candidate intra-prediction mode, the intra-frame prediction module 864 generates a block of samples, i.e., 866. The block of samples 866 is generated by module 864 using techniques such as DC, planar, or angular intra prediction. The block of samples 866 may also be generated using a matrix multiplication approach with neighboring reference samples as input and a matrix selected by the video encoder 814 from a set of matrices, the selected matrix being signaled in the bitstream 716 using an index to identify which matrix from the set of matrices should be used by the video decoder.

[0134] The in-loop filter module 868 applies several filtering stages to the reconstructed samples 854. The filtering stages include a "deblocking filter" (DBF), which applies smoothing aligned to CU boundaries to reduce artifacts due to discontinuities. Another filtering stage present in the in-loop filter module 768 is an "adaptive loop filter" (ALF), which applies a Wiener-based adaptive filter to further reduce distortion. A further filtering stage available in the in-loop filter module 868 is a "sample adaptive offset" (SAO) filter. The SAO filter operates by first classifying the reconstructed samples into one or more categories and then applying an offset at the sample level according to the assigned category.

[0135] The filtered samples, represented by arrow 870, are output from the in-loop filter module 868. The filtered samples 870 are stored in a frame buffer 872. The frame buffer 872 typically has the capacity to store several pictures (e.g., up to 16) and is therefore stored in the memory 206. The frame buffer 872 is typically not stored using on-chip memory due to the large memory consumption required. Access to the frame buffer 872 is therefore costly in terms of memory bandwidth. The frame buffer 872 provides a reference frame (represented by arrow 874) to the motion estimation module 876 and the motion compensation module 880. The reference frame 874 is output as a reconstructed frame 718 of the corresponding subpicture encoder module 700 (614, 636, 654a, 654b, 654c) and provided to the unpacker module 720. In the example of FIG. 8, the reconstructed frame is the result of the lossy VVC encoding operation resulting from the operations of modules 810-890.

[0136] The motion estimation module 876 estimates several “motion vectors” (denoted as 878), each of which is a Cartesian spatial offset from the position of the current CB and references a block in one of the reference frames in the frame buffer 872. A filtered block of reference samples (denoted as 882) is generated for each motion vector. The filtered reference samples 882 form further candidate modes available for potential selection by the mode selector 886. Furthermore, for a given CU, the PU 820 may be formed using one reference block (“uni-predicted”) or two reference blocks (“bi-predicted”). For a selected motion vector, the motion compensation module 880 generates the PB 820 according to a filtering process that supports sub-pixel accuracy in motion vectors. Thus, the motion estimation module 876 (which operates on many candidate motion vectors) can perform a simplified filtering process compared to that of the motion compensation module 880 (which operates on only the selected candidate), achieving reduced computational complexity. If the video encoder 714 selects inter prediction for the CU, the motion vector 878 is encoded into the bitstream portion 716.

[0137] 8 is described with reference to Universal Video Coding (VVC), other video coding standards or embodiments may also employ the processing stages of modules 810-890. The frame data 712 (and bitstream 121) may also be stored in memory 206, hard disk drive 210, CD-ROM, Blu-ray disc TMor other computer-readable storage medium. Additionally, frame data 712 (and bitstream 121) may be received from (or transmitted to) an external source, such as a server connected to communications network 220 or a radio frequency receiver. Communications network 220 may provide limited bandwidth, necessitating the use of rate control in video encoder 120 to avoid saturating the network if frame data 712 is difficult to compress.

[0138] The bitstream 121 may be constructed from one or more slices representing a spatial section (a collection of CTUs) of the frame data 712, generated by one or more instances of the video encoder 714, each operating cooperatively under the control of the processor 205 to generate a bitstream portion 716. The bitstream portion 716 may also include one slice corresponding to a subpicture to be output as a collection of subpictures forming a picture, each subpicture being independently encodable and independently decodable with respect to either other slices or subpictures in the picture. The ability to independently encode and decode any subpicture within a picture allows the effects of lossy compression on packed feature maps or coefficients contained in any given subpicture to be taken into account in the PCA encoder 160 by using a lossy version of the feature map or coefficient in a later stage of tensor compression.

[0139] FIG. 9 is a schematic block diagram 900 illustrating an implementation of an inter-channel decorrelation-based tensor decoder 170. FIG. 11A is a schematic block diagram 1100 illustrating a multi-scale feature reconstruction (MSFR) module. FIG. 17 is a schematic block diagram illustrating a method 1700 for decoding tensors. The decoder-side operation of the system 100 is described with reference to the method 1700. The method 1700 may be implemented using an apparatus such as a configured FPGA, ASIC, or ASSP. Alternatively, as described below, the method 1700 may be implemented by the destination device 140 as one or more software code modules of the application program 233 under execution by the processor 205. The software code modules of the application program 233 that implement the method 1700 may reside on the hard disk drive 210 and / or the memory 206, for example. The method 1700 is repeated for each frame of compressed data in the bitstream 123. The method 1700 may be stored on a computer-readable storage medium and / or the memory 206. The method 1700 provides a means for decoding compressed representations of tensors, with scalability in quality and accurate modeling of inaccuracies introduced due to the use of lossy compression mechanisms. The video bitstream 143 is passed to a picture decoder 904, which implements a VVC video decoder and decodes the bitstream generated by the PCA encoder 160. The picture decoder 904 decodes sub-pictures present in the video bitstream 143, each sub-picture corresponding to a different data type needed to generate the tensor 149. Each decoded sub-picture provides a unit of information for the decoded tensor, which corresponds to a mean channel 611, basis vectors 631, and coefficients 648, which are coded in multiple groups. The method 1700 begins with a coefficient group decoding step 1710.

[0140] In step 1710, a picture decoder, such as a VVC decoder, decodes the bitstream 143 under the control of the processor 205. As shown in FIG. 9, the inter-channel decorrelation-based tensor decoder 170 includes a picture decoder 904. FIG. 10 shows a VVC decoder 1000 as an exemplary embodiment of the picture decoder 904. In the decoder 1000, an entropy decoder 1020 decodes coefficient groups from the bitstream 143 under the execution of the processor 205. The coefficients form a tensor with channel, width, and height dimensions. The coefficient grouping divides the coefficients into groups along the channel dimension, such that each group contains a contiguous set of coefficients when the coefficients are ordered based on the explained variance of the corresponding basis vector. There is always one group containing the 0th channel (the coefficients of the basis vector with the largest explained variance). There are no unused channels among the groups; that is, groups can be concatenated along the channel dimension to form a single tensor of coefficients for use in projecting or inversely transforming into the same subspace as the original tensor. A group contains a set of contiguous coefficients along the channel dimension, and is required to include the 0th channel, but the lowest channel is not required to be included in any group. For example, if 25 basis vectors are used (so the coefficients are listed as [0..24]), the group definitions could be [0..4], [5..9], [10..14] (omitting coefficients [15..24] from the basis vectors).

[0141] When decoding the group signaling, step 1710 operates to decode an assignment that maps the arrangement of independently coded regions of a picture to multiple groups of basis vectors. That is, the group of s subpictures coded in step 16180 is decoded. The group signaling may be implemented as a list in which each value specifies the number of coefficients along the channel dimension. Since coefficients are tensors with c, h, and w dimensions, grouping based on channel ranges results in groups containing multiple feature maps, with each feature map in the group having a width and height. Each group in the list begins with the next coefficient after the end of the previous group, and ends the list with a zero-sized group or exhaustion of available coefficients (whichever occurs first). Control in processor 205 passes from step 1710 to average channel decoding step 1720.

[0142] In step 1720, the picture decoder 904, under execution of the processor 205, outputs a mean channel sub-picture 950 which is passed to the unpacker 952. The unpacker 952 extracts the integer mean channel 954, where the C value corresponds to the number of channels in the tensor 149. The integer mean channel 956 is passed to the inverse quantizer 956, where a conversion from the sample domain to the floating-point domain is performed using an appropriate quantization range, for example, obtained from the bitstream 143. The operation of the inverse quantizer 956 results in a decoded mean channel 958. Control in the processor 205 passes from step 1720 to a basis vector decoding step 1730.

[0143] In step 1730, the picture decoder 904, under the execution of the processor 205, outputs a subpicture 930 including packed basis vectors (e.g., performing a method such as SVD) generated by the decomposition module 630 before quantization and lossy compression. In other words, the basis vectors are decoded from the bitstream in the example described in subpicture packed format. The format may vary based on the dimensionality of the basis vectors and the assignment of coefficients to groups. The subpicture 930 is passed to an unpacker 932. The unpacker 932 extracts integer basis vectors 934 as a series of non-overlapping arrays arranged in the subpicture 930. The integer basis vectors 934 are passed to an inverse quantizer 936. The inverse quantizer 936 constructs floating-point basis vectors 938 and applies the quantization ranges obtained from the bitstream 143. The decoded assignments provide an indication of which basis vectors should have corresponding coefficients in the feature frame since not all subpictures were selected in step 1670. Control in processor 205 passes from step 1730 to a coefficient decoding step 1740.

[0144] In step 1740, the picture decoder 904, under the execution of the processor 205, outputs coefficient sub-pictures 910. The coefficient sub-pictures 910 are passed to the unpacker 912. Each sub-picture includes one or more feature maps corresponding to one group of coefficients decoded in step 1710. In other words, the sub-pictures 910 include a set of coefficients, with one coefficient per sample per basis vector of the tensor 143. The sub-pictures 910 are output by the picture decoder 904 and passed to the unpacker 912. The unpacker 912 extracts each coefficient from the sub-picture 910 based on the dimensionality of the tensor 149 and the group from step 16100, and outputs an integer coefficient tensor 914. Due to the selection of a subset of groups in step 16170, the sub-picture 910 may not include coefficients corresponding to all basis vectors. When decoding the sub-pictures 910, a feature frame can be considered to be decoded from the bitstream. The feature frame includes independently coded regions (sub-pictures), which include regions with coefficients corresponding to one or more basis vectors decoded in step 1730. The regions may also encode basis vectors and mean channel values.

[0145] Control in processor 205 passes from step 1740 to a coefficient merging step 1750 .

[0146] As described above, the coefficients form a tensor with dimensions of channel, width, and height. In other words, the coefficients are decoded from the bitstream and extracted from the feature frame as feature maps (one feature map per channel) that form the coefficient tensor. In step 1750, the merge group module 916, under execution of the processor 205, concatenates the coefficient group tensors 914 along the channel dimension to generate an integer coefficient tensor 918. Control in the processor 205 passes from step 1750 to a coefficient dequantization step 1760.

[0147] In step 1760, the inverse quantizer 920, under execution of the processor 205, converts the integer coefficient tensor 918 from the integer domain to the floating-point domain according to the quantization range obtained from the bitstream 143, outputting the floating-point coefficients 922. Step 1760 operates to obtain coefficients from the decoded feature frame by an inverse quantization operation. Control in the processor 205 passes from step 1760 to a tensor generation step 1770.

[0148] In step 1770, a zero-centered tensor 942 is generated by dot product module 940 under execution of processor 205 by performing a dot product on coefficients 922 and basis vectors 938. Sum module 960 adds zero-centered tensor 942 with mean channel 958 to generate reconstructed combined tensor 149a as output from PCA decoder 170. Completion of step 1770 effectively generates tensor 149a from the set of basis vectors and coefficient tensor decoded in the operations of steps 1710-1750. Combined tensor 149a has a higher channel count and the same spatial size as the coefficient tensor. Tensor 149a can also be viewed as a projection of the coefficients and basis vectors generated using the dot product operation, for example, using module 940. Control in processor 205 passes from step 1770 to a tensor reconstruction step 1780.

[0149] Architecture 1100 includes MSFR module 1130, which operates to generate multiple tensors from tensor 149a generated by execution of step 1770 using one or more trained convolutional layers. In step 1780, reconstructed tensor 149 is generated by MSFR module 1130 from reconstructed combined tensor 149a. Upsample modules 1132, 1134, and 1136 upsample tensor 149a horizontally and vertically by factors of 2, 4, and 8, respectively, to generate tensors 1133, 1135, and 1137. Tensor 1137 forms one (P'2) output from MSFR module 1130 and is passed to downsample module 1142. Downsample module 1142 downsamples tensor 1137 horizontally and vertically by a factor of 2 to generate tensor 1143 having the same dimensionality as tensor 1135. Tensor 1143 is fed to a convolutional layer 1148, which outputs tensor 1149. A summation module 1154 adds tensors 1135 and 1149 to produce tensor 1155 as the output (P'3) of the MSFR module 1130.

[0150] Downsample module 1140 downsamples tensor 1135 by a factor of 2 horizontally and vertically to produce tensor 1141, which has the same dimensionality as tensor 1133. Tensor 1141 is fed to convolutional layer 1146, which outputs tensor 1147. Sum module 1152 adds tensors 1133 and 1147 to produce tensor 1153 as the output (P'4) of MSFR module 1130.

[0151] Downsample module 1138 downsamples tensor 1133 by a factor of 2 horizontally and vertically to produce tensor 1139, which has the same dimensionality as tensor 149a. Tensor 1139 is provided to convolutional layer 1144, which outputs tensor 1145. Summation module 1150 adds tensors 149a and 1145 to produce tensor 1151 as the output (P'5) of MSFR module 1130. Collectively, tensors 1151, 1153, 1155, and 1157 form tensor 149, providing the decoded P2-P5 layers. Control in processor 205 passes from step 1780 to a neural network second portion execute step 1790.

[0152] In an alternative embodiment of architecture 1100, tensor 149a is input to a convolutional neural network, and the output of the convolutional neural network is input to upsamplers 1132, 1134, and 1136. For example, FIG. 11B shows tensor 149a being input to a trained convolutional layer 1190 and the generated output 149a_conv. In a configuration using convolutional layer 1190, output 149a_conv is input to upsamplers 1132, 1134, and 1136. Convolutional layer 1190 has fewer output channels than input channels and is applied to tensor 149a before generating the multiple tensors by MSFR module 1130, as shown. The embodiment using convolutional layer 1190 corresponds to an embodiment in which convolutional layer 528 is excluded from encoding-side module 500. Regardless of whether layer 1190 is used, the trained convolutional layers of FIGS. 11A and 11B perform spatial resizing to restore the hierarchical representation (FPN) of the frame.

[0153] In step 1790, the CNN head 150, under execution of the processor 205, performs a second portion of the neural network task. An example of the second or head portion 150 is described with reference to Figures 12A-C and 13. The method 1700 finishes processing the current frame and is invoked again for the next received frame in the bitstream 143.

[0154] In the implementation of method 1700, subpictures of groups not selected for inclusion in the bitstream are decoded as planar regions coded with a neutral value, such as the DC midtone value. A quantization range that results in a DC midtone value corresponding to the dequantized floating-point value 0.0 is used. Method 1700 processes all subpictures, including omitted ones, by replacing them with a neutral value, and the projection from the basis vector subspace back to the tensor 115 subspace can be performed without knowing which particular group was used for a given frame. Because subpictures containing coefficients coded with neutral values ​​do not affect the final reconstructed combined tensor 149, there can be no different processing for groups of included versus omitted coefficients.

[0155] Figure 10 shows an example embodiment 1000 of a picture decoder 904, also referred to as a video decoder. The video decoder 904 of Figure 9 is an example of a generic video coding (VVC) video decoding pipeline, although other video codecs may be used to perform the processing stages described herein, such as HEVC. As shown in Figure 10, a bitstream 143 is input to the video decoder 804. The bitstream 143 may be stored in memory 206, hard disk drive 210, a CD-ROM, a Blu-ray disc, or any other suitable storage medium. TMThe bitstream 143 may be read from a network 220, a computer-readable storage medium, or other non-transitory computer-readable storage medium and provided to the embodiment 1000 as a bitstream 143. Alternatively, the bitstream 143 may be received from an external source, such as a server connected to the communications network 220 or a radio frequency receiver. The bitstream 143 includes coding syntax elements that represent captured frame data to be decoded. If the sub-pictures are to be decoded independently, the portion of the bitstream 143 corresponding to each sub-picture may be provided to a separate instance of the embodiment 1000. A separate instance of the embodiment 1000 for each sub-picture allows for parallel decoding of the sub-pictures to improve throughput.

[0156] The bitstream 143 is input to the entropy decoder module 1020. The entropy decoder module 1020 extracts syntax elements from the bitstream 1010 by decoding a sequence of "bins" and passes the values ​​of the syntax elements to other modules in the video decoder 904. The entropy decoder module 1020 uses variable-length and fixed-length decoding to decode the SPS, PPS, or slice header of the arithmetic decoding engine and decodes the slice data syntax elements as a sequence of one or more bins. Each bin can use one or more "contexts," which describe the probability levels used to encode the bin's "1" and "0" values. If multiple contexts are available for a given bin, a "context modeling" or "context selection" step is performed to select one of the available contexts for decoding the bin. The process of decoding the bins sequentially forms a feedback loop, so that each slice can be decoded in its entirety by a given instance of the entropy decoder 1020. A single (or a small number) of high-performance entropy decoder 1020 instances can decode all slices or sub-pictures of a frame or picture from the bitstream 1010, and multiple low-performance entropy decoder 1020 instances can simultaneously decode slices of a frame from the bitstream 1010.

[0157] The entropy decoder module 1020 applies an arithmetic coding algorithm, such as "context-adaptive binary arithmetic coding" (CABAC), to decode syntax elements from the bitstream 1010. The decoded syntax elements are used to reconstruct parameters within the video decoder 904. The parameters include residual coefficients (represented by arrow 1024), quantization parameters 1074, secondary transform indices 1070, and mode selection information such as intra-prediction modes (represented by arrow 1058). The mode selection information also includes information such as motion vectors and the division of each CTU into one or more CBs. The parameters are used to generate PBs, typically combined with sample data from previously decoded CBs.

[0158] The residual coefficients 1024 are passed to an inverse secondary transform module 1036, which applies a secondary transform or performs no operation (bypass) according to a secondary transform index. The inverse secondary transform module 1036 generates reconstructed transform coefficients 1032. That is, module 1036 generates primary transform domain coefficients from the secondary transform domain coefficients. The reconstructed transform coefficients 1032 are input to an inverse quantizer module 1028. The inverse quantizer module 1028 performs inverse quantization (or "scaling") on the residual coefficients 1032, i.e., in the primary transform coefficient domain, according to a quantization parameter 1074 to create reconstructed intermediate transform coefficients, represented by arrow 1040. The inverse quantizer module 1028 may also apply a scaling matrix to provide non-uniform inverse quantization within the TB, corresponding to the operation of the inverse quantizer module 840. If the bitstream 143 indicates that a non-uniform inverse quantization matrix is ​​to be used, the video decoder 904 reads the quantization matrix from the bitstream 143 as a sequence of scaling coefficients and arranges the scaling coefficients into a matrix. The inverse scaling uses the quantization matrix in combination with the quantization parameters to create reconstructed intermediate transform coefficients 1040.

[0159] The reconstructed transform coefficients 1040 are passed to an inverse linear transform module 1044. Module 1044 converts the coefficients 1040 from the frequency domain back to the spatial domain. The inverse linear transform module 1044 applies an inverse DCT-2 transform horizontally and vertically, constrained by the maximum available transform size, as described with reference to the forward linear transform module 826. The result of the operation of module 1044 is a block of residual samples, represented by arrow 1048. The size of the block of residual samples 1048 is equal to the corresponding CB. The residual samples 1048 are supplied to a summation module 1050.

[0160] In summation module 1050, residual samples 1048 are added to the decoded PB (represented as 1052) to generate a block of reconstructed samples, represented by arrow 1056. The reconstructed samples 1056 are provided to a reconstructed sample cache 1060 and an in-loop filtering module 1088. The in-loop filtering module 1088 generates a reconstructed block of frame samples, represented as 1092. The frame samples 1092 are written to a frame buffer 1096. The frame buffer 1096 outputs an image or video frame 1014, which corresponds to tensor 149 of FIG. 1.

[0161] The reconstructed sample cache 1060 operates similarly to the reconstructed sample cache 856 of the video encoder 714. The reconstructed sample cache 1060 provides storage for reconstructed samples needed to intra-predict subsequent CBs without the memory 206 (e.g., by instead using the data 232, which is typically on-chip memory). Reference samples, represented by arrow 1064, are retrieved from the reconstructed sample cache 1060 and provided to a reference sample filter 1068 to generate filtered reference samples, indicated by arrow 1072. The filtered reference samples 1072 are provided to an intra-frame prediction module 1076. The module 1076 generates blocks of intra-predicted samples, represented by arrow 1080, according to the intra-prediction mode parameters 1058 signaled in the bitstream 1010 and decoded by the entropy decoder 1020. The intra-prediction module 1076 supports modes of the encoder-side module 864, including IBC and MIP. The blocks of samples 1080 are generated using modes such as DC, planar, or angular intra prediction.

[0162] If the prediction mode of a CB is indicated to use intra prediction in the bitstream 143, the intra-predicted samples 1080 form the decoded PB 1052 via a multiplexer module 1084. Intra prediction generates a predictive block of samples (PB), which is a block in one color component derived using "neighboring samples" in the same color component. Neighboring samples are samples adjacent to the current block and have already been reconstructed by preceding them in block decoding order. When a luma block and a chroma block are co-located, the luma block and the chroma block may use different intra-prediction modes. However, the two chroma CBs share the same intra-prediction mode.

[0163] If the prediction mode of the CB is indicated as inter-prediction in the bitstream 143, the motion compensation module 1034 generates a block of inter-predicted samples, denoted as 1038. The block of inter-predicted samples 1038 is generated using a motion vector decoded from the bitstream 143 by the entropy decoder 1020 and a reference frame index to select and filter a block of samples 1098 from the frame buffer 1096. The block of samples 1098 is obtained from a previously decoded frame stored in the frame buffer 1096. In the bi-predictive case, two blocks of samples are generated and blended together to generate samples for the decoded PB 1052. The frame buffer 1096 is populated with filtered block data 1092 from the in-loop filtering module 1088. Similar to the in-loop filtering module 868 of the video encoder 714, the in-loop filtering module 1088 applies any of DBF, ALF, and SAO filtering operations. Generally, motion vectors are applied to both the luma and chroma channels, but the filtering process for sub-sample interpolation in the luma and chroma channels is different.

[0164] 8 and 10 do not show modules that pre-process the video before encoding and post-process the video after decoding to shift sample values ​​to achieve a more uniform use of the range of sample values ​​within each chroma channel. A multi-segment linear model is derived in the video encoder 714 and signaled in the bitstream for use by the video decoder 904 to undo the sample shift. The linear model chroma scaling (LMCS) tool provides compression advantages for certain color spaces and content that have some degree of non-uniformity in sample space utilization, particularly limited range utilization, that can result in higher quality loss from the application of quantization.

[0165] FIG. 12A is a schematic block diagram illustrating an example embodiment 1200 of a CNN head portion 150 for object detection, corresponding to the YOLOv3 network excluding the DarkNet-53 backbone portion. The embodiment 1200 can be used when a CNN backbone is implemented, for example, as in FIG. 3A. Depending on the task to be performed at the destination device 140, a different network can be used instead of the CNN head 150. The input tensor 149 is separated into tensors for each layer (i.e., tensors 1210, 1220, and 1234). The tensor 1210 is passed to a CBL module 1212 to generate a tensor 1214. The tensor 1214 is passed to a detection module 1216 and an upscaler module 1222. The detection module outputs a bounding box 1218 in the formation of the detection tensor. The bounding box 1218 is passed to a non-maximum suppression (NMS) module 1248.

[0166] Scaling by the original video width and height is performed in upscaler module 1222 prior to resizing the backbone portion of network 114 to generate a bounding box that addresses coordinates within the original video data 113. Upscaler module 1222 receives tensor 1214 and tensor 1220 and generates an upscaled tensor 1224 that is passed to CBL module 1226. CBL module 1226 generates tensor 1228 as output. Tensor 1228 is passed to detection module 1230 and upscaler module 1236. Detection module 1230 generates a detected tensor 1232 that is fed to NMS module 1248. Upscaler module 1236 is another example of module 1222. Upscaler module 1236 receives tensor 1228 and tensor 1234 and outputs an upscaled tensor 1238. The upscaled tensor 1238 is passed to a CBL module 1240 which outputs a tensor 1242 to a detection module 1244 .

[0167] CBL modules 1212, 1226, and 1240 each include a concatenation of five CBL modules, e.g., CBL model 360 shown in Figure 3D. Upscaler modules 1222 and 1236 are instances of upscaler module 1260, as shown in Figure 12B. Module 1248 receives tensors 1218, 1232, 1236 and outputs task result 151.

[0168] As shown in Figure 12B, upscaler module 1260 accepts tensor 1262 (e.g., tensor 1214 in Figure 12A) as input. Tensor 1262 is passed to CBL module 1266 (having the structure of module 360) to generate tensor 1268. Tensor 1268 is passed to upsampler 1270 to generate upsampled tensor 1272. Concatenation module 1274 generates tensor 1276 by concatenating upsampled tensor 1272 with a second input tensor 1264 (e.g., tensor 1220 input to upscaler 1222 in Figure 12A).

[0169] Detection modules 1216, 1230, and 1244 are instances of detection module 1280 as shown in FIG. 12C. Detection module 1280 receives tensor 1282. Tensor 1282 is input to CBL module 1284, which has the structure of module 360. CBL module 1284 generates tensor 1286. Tensor 1286 is passed to convolution module 1288, which implements the detection kernel. In some configurations, the detection kernel applies a 1x1 kernel to generate output on feature maps in each of the tensor's three layers. The detection kernel is 1x1x(Bx(5+C)), where B is the number of bounding boxes a particular cell can predict, typically 3, and C is the number of classes, which can be 80, resulting in a kernel size of 255 detection attributes (i.e., tensor 1090). The constant "5" represents four bounding box attributes (box center x, y and size scale x, y) and one object confidence level ("object-likeness"). The result of the detection kernel has the same spatial dimension as the input feature map, but the depth of the output corresponds to the detection attribute. The detection kernel is applied to each layer, typically three layers, resulting in a large number of candidate bounding boxes. A process of non-maximum suppression is applied to the resulting bounding boxes by the NMS module 1048 to discard redundant boxes, such as overlapping predictions at similar scales, resulting in a final set of bounding boxes as output for object detection.

[0170] FIG. 13 is a schematic block diagram illustrating an alternative head portion 1300 of a CNN. The head portion 1300 can be implemented as a CNN head 150, with the CNN backbone 114 implemented, for example, as backbone 400. The head portion 1300 forms part of an overall network known as "faster RCNN" and includes a feature network (i.e., backbone portion 400), a region proposal network, and a detection network. The input to the head portion 1300 is tensor 149, which includes P2-P6 layer tensors 1310, 1312, 1314, 1316, and 1318. The P2-P5 layer tensors 1310, 1312, 1314, and 1316 correspond to P2-P5 outputs 477, 475, 473, and 471 in FIG. 4. The P2-P6 tensors 1310, 1312, 1314, 1316, and 1318 are input to a region proposal network (RPN) head module 1320. The P6 tensor 1318 is generated by a max pooling module 1342 and operates on the P5 tensor 1316 to perform a 2x2 max pooling operation. The RPN head module 1320 performs convolutions on the input tensors to generate intermediate tensors. The intermediate tensors are fed to two subsequent sibling layers: (i) one for classification and (ii) one for bounding box, or "region of interest" (ROI), regression. The resulting output is a classification and bounding box 1322. The classification and bounding box 1322 is passed to an NMS module 1324. The NMS module 1324 prunes redundant bounding boxes by removing duplicate boxes with lower scores to generate pruned bounding boxes 1326. The bounding boxes 1326 are input to a region of interest (ROI) pooler 1328. The ROI pooler 1328 uses some of the layer tensors (described further below) and the bounding boxes 1326 of tensor 149 to generate fixed-size feature maps from the various input-size maps using a max-pooling operation. The max-pooling operation sub-samples and takes the maximum value within each group of input values ​​to generate a single output value in an output tensor.

[0171] The inputs to the ROI puller 1328 are the P2-P5 feature maps 1310, 1312, 1314, and 1316, as well as a region of interest proposal 1326. Each proposal (ROI) from 1326 is associated with a portion of the feature maps (1310-1316) to generate a fixed-size map. The fixed-size map is sized independently of the underlying portion of the feature maps 1310-1316. One of the feature maps 1310-1316 is selected so that the resulting cropped map has sufficient detail, for example, according to the following rule: floor(4 + log2(sqrt(box_area) / 224)), where 224 is the standard box size. The ROI puller 1328 operates to crop the input feature map according to the proposal 1326, generating a tensor 1330.

[0172] The tensor 1330 is fed to a fully connected (FC) neural network head 1332. The FC head 1332 runs two fully connected layers to generate a class score and a bounding box predictor delta tensor 1334. The class score is typically an 80-element tensor, with each element corresponding to a predicted score for the corresponding object category. The bounding box predictor delta tensor is an 80x4 = 320-element tensor and contains the bounding box for the corresponding object category. Final processing is performed by an output layer module 1336, which receives the tensor 1334 and performs a filtering operation to generate a filtered tensor 1338. Low-scoring (low-classification) objects are removed from further consideration. A non-max suppression module 1340 receives the filtered tensor 1338 and removes overlapping bounding boxes by removing overlapping boxes with lower classification scores, resulting in an inference output tensor 1340 corresponding to tensor 151.

[0173] FIG. 14A is a schematic block diagram illustrating the division of picture 1400 into sub-pictures 1410, 1412, 1416a, 1416b, and 1416c, as performed by sub-picture bitstream combiner 680, for example. Each sub-picture is packed by packer 710 of a corresponding one of encoders 614, 636, 654a, 654b, and 654c. In the example of FIGS. 14A and 14B, coefficients are shown divided into three groups and thus using three sub-pictures. However, other numbers of groups may be used in addition to the examples shown in FIGS. 14A and 14B. Each sub-picture contains information arranged in a two-dimensional array of samples. Referring to FIG. 14B, picture 1400b corresponds to picture 1400, sub-picture 1410b corresponds to 1410, and sub-picture 1412b corresponds to 1412. Subpicture 1410b holds mean channel data, such as mean channel 1420 of tensor 115. Mean channel 1410b corresponds to bitstream 615 of subpicture encoder 614 in the configuration of Figure 6. Subpicture 1412b holds basis vectors corresponding to 637 in Figure 6. In the example of Figure 14B, the basis vectors include, among other things, basis vector 1422, and the basis vectors are packed into the area of ​​subpicture 1412b in a non-overlapping manner. Areas within picture 1400 (or 1400b) not used to store any data may be occupied by sample values ​​corresponding to the value "0" after application of inverse quantization to convert from sample values ​​to the floating-point domain. Similarly, areas within sub-pictures 1410 (or 1410b), 1412 (or 1412b), 1416a, 1416b, and 1416c that are not used to store any data may be occupied by sample values ​​that correspond to the value "0" after application of inverse quantization to convert from sample values ​​to the floating-point domain. Sub-pictures 1416a, 1416b, and 1416c hold coefficient groups corresponding to 654a, 654b, and 654c in Figure 6, where the coefficients for each basis vector form a width by height feature map.For example, in the first coefficient group as represented in sub-picture 1416a with 10 basis vectors, the coefficients for one tensor are arranged as 10 feature maps (applicable to basis vectors [0..9]), such as feature map 1430, each with a width and height corresponding to that of tensor 115. Coefficient sub-picture 1416b has coefficients packed as four feature maps (applicable to basis vectors [10..14]), such as feature map 1432. Coefficient sub-picture 1416c has 10 feature maps (applicable to basis vectors [15..25]), such as feature map 1434.

[0174] Figure 15 is a schematic block diagram illustrating a bitstream 1500 that carries encoded packed feature maps and associated metadata. The bitstream 1500 corresponds to the bitstream 121 generated by the PCA encoder 160 or the bitstream 143 decoded by the PCA decoder 170. The bitstream 1500 includes groups of syntax preceded by "network abstraction layer" (NAL) unit headers. For example, the NAL unit header 1508 precedes a sequence parameter set (SPS) 1510. The SPS 1510 specifies the layout of the picture 1400, including the positions and sizes of the subpictures 1410, 1412, 1416a, 1416b, and 1416c, using subpicture information 1511. The SPS 1510 also indicates the chroma format, bit depth, and resolution of the frame data represented by the bitstream 1500. If N is adjusted in step 16160, the coded coefficient group in step 16180 is adjusted to start a new sub-picture structure.

[0175] The coefficient groups used by the PCA encoder 160 may be encoded in the SEI message 1513 using a list for dividing the coefficient tensor into groups along the channel dimension, encoded as coefficient group information 1591. The size (i.e., width and height) of the coefficient feature map is also encoded as part of the coefficient group information 1591 in the SEI message 1513. The quantization range, as determined in the quantizers 612, 632, 646, is encoded as quantization range 1592.

[0176] Picture 1514 is coded in bitstream 1500. Each picture includes one or more subpictures, such as coded subpicture 1520, coded subpicture 1410 (or 1410b), etc. For the first picture in the bitstream and for IDR pictures, intra-slices are used to avoid prediction dependencies on other access units in bitstream 1500. Coded subpicture 1522, which codes subpicture 1412, includes a slice header 1530 followed by slice data 1540. Slice data 1540 includes a sequence of CTUs that provide a coded representation of the frame data. CTUs are square, typically 128x128 in size, which is not well aligned with typical feature map sizes. Aligning feature maps to a minimum block size, such as a 4x4 grid, partially ameliorates this misalignment. The coded sub-pictures 1524, 1526, and 1528 code the sub-pictures 1416a, 1416b, and 1416c that correspond to respective groups of coefficients.

[0177] In some configurations of method 1600, basis vector determination step 1650 is performed less frequently than every frame. If a basis vector is not determined for a particular frame, a basis vector from a previous frame is used. Step 1650 may be performed infrequently, such as once at the beginning of a video sequence, or periodically, such as each time a frame is encoded using intra prediction (such as a new IDR picture or an intra picture in a random access picture structure). When step 1650 is performed, the amount of explained variance for each basis vector is also derived. The degree of explained variance may form the basis for determining the ordering of groups of coefficients and / or regions or subpictures. If step 1650 is performed and the difference in explained variance from that determined in a previous execution of step 1650 exceeds a threshold, source device 110 may derive a new grouping, and therefore a new division of picture 1400 into subpictures. If a division of the picture 1400 into sub-pictures is performed, an IDR picture needs to be sent to signal the division, and an instance of the SEI message 1513 is signaled to indicate the division of the coefficients into groups utilizing the defined sub-picture structure.

[0178] In the configuration of method 1600, when packing coefficient feature maps into the last subpicture, i.e., 1416c, a variable number of coefficient feature maps are packed, providing a fine-granularity mechanism for rate control. Because the last subpicture tends to be large, encoding all coefficient feature maps for it to occupy unused space in picture 1400 provides a coarse granularity of the summation rate for including this subpicture in bitstream 121. An iterative method of steps 16120-16170 may be performed within the last subpicture to determine how many coefficient feature maps to use. Once the number of coefficient feature maps to use is determined, the last subpicture (1416c) may be re-encoded using only the used coefficient feature maps packed into the subpicture area.

[0179] In the illustrated exemplary configuration, tensor combiner 162 is implemented as MSFF module 500, and tensor separator 172 is implemented as MSFR module 1130, both of which involve trained layers in their operation. In an alternative configuration, tensor combiner 162 may combine two layers in step 1615 by resampling the tensors for one layer and concatenating the resulting tensor with the tensor of another layer, e.g., an adjacent layer in an FPN, to generate combined tensor 115. The number and dimensionality of tensors may be stored as a layer mapping in SEI message 1513. In an alternative embodiment, tensor separator 172 performs the inverse operation of tensor combiner 162 to generate extracted tensor 149 based on the layer mapping of the decoded SEI message. If the FPN includes multiple layers, such as four layers, the layers may be treated as two sets of layers and processed by separate instances of tensor combiner 162, PCA encoder 160, PCA decoder 170, and tensor separator 172. Layers P2 and P3 may be treated as one set, and layers P4 and P5 may be treated as another set. Despite the processing as separate sets of layers, the associated mean channel tensors, basis vectors, and coefficients for each set may be packed into a single feature frame.

[0180] Regardless of the method used, tensor combiner 162 operates to fuse multiple tensors forming a hierarchical representation, typically from an FPN, into a single tensor suitable for decorrelation using PCA methods. Each tensor output in tensor 115a forms part of a split point of the network, which is split into a first (backbone 114) portion and a second (head 150) portion. If an FPN is not used, the first and second portions do not contain a hierarchical representation of the input to the first portion.

[0181] The methods presented herein enable efficient representation of tensors in a format suitable for compression using modern block-based compression standards such as VVC or HEVC. While block-based compression is not intuitively applicable to data such as coefficients for projecting basis vectors to reconstruct feature maps, it uncovers additional, unexpected redundancies within blocks, such as by using various transforms, including trained quadratic transforms. While the methods presented herein are described with reference to the "Faster RCNN" and "YOLOv3" network architectures and the specific division of these networks into "backbone" and "head" portions, the methods are applicable to any neural network that operates on multidimensional tensor data and to different divisions of such networks into "backbone" and "head" portions.

[0182] Industrial Applicability The described arrangements are applicable to the computer and data processing industry, and in particular to digital signal processing for encoding and decoding of signals such as video and image signals, achieving high compression efficiency.

[0183] Also disclosed is an arrangement for quantizing floating-point tensor data within groups of feature maps or channels and packing the resulting integer values ​​into planar frames using the logarithmic quantization domain. Quantization and dequantization methods using the logarithmic quantization domain allow for greater compression efficiency due to the lack of bits spent on encoding exact values ​​for large magnitude tensor values, where such precision does not result in further improvement in task performance for the network in use.

[0184] In some described configurations, MFSC feature compression or expansion is used in conjunction with PCA encoding or decoding, respectively. Conventional use of MFSC techniques, including the three main modules MSFF, SSFC (encoder and decoder), and MSFR, can provide performance at the expense of high training requirements, which reduce flexibility. Conventional techniques using PCA can suffer from performance issues when fewer coefficients are used, resulting in a reduced utilization of feature frame area. Combining PCA with the use of trained convolutional layers for feature compression, including MSFF, PCA, and MSFR, can achieve adequate accuracy without incurring higher training requirements. Furthermore, the use of PCA allows for scalability not possible with MFSC alone, since the number of basis vectors onto which coefficients are encoded can be changed between frames, and PCA can include error analysis. Furthermore, the basis vectors can be updated intermittently or determined less frequently for any given frame.

[0185] In other configurations, PCA is implemented per channel, so that only some groups or subpictures are selected for encoding. As mentioned above, selecting and therefore packing a variable number of coefficient feature maps can provide a fine-grained mechanism for rate control.

[0186] The above description describes several embodiments, and modifications and / or variations can be made without departing from the scope and spirit of the overcoming embodiments. That is, the present embodiments are illustrative and not limiting.

Claims

1. 1. A method for decoding a plurality of tensors forming a hierarchical representation of a frame from a bitstream, comprising: decoding coefficients from the bitstream for a first tensor; generating a second tensor from the first tensor and a set of basis vectors, the second tensor having a higher channel count and the same spatial size as the first tensor; generating the plurality of decoded tensors from the second tensor using one or more trained convolutional layers; and A method having the following.

2. The method of claim 1 , wherein the second tensor is a projection of the basis vectors and the coefficients generated using a dot product operation.

3. The method of claim 1 , wherein the basis vectors are decoded from the bitstream.

4. The method of claim 1 , wherein the trained convolutional layers perform spatial resizing to recover the hierarchical representation of the frame.

5. The method of claim 2 , wherein a channel count of the projection is equal to a channel count of each of the tensors of the plurality of tensors.

6. The method of claim 1 , wherein a convolution of fewer output channels than input channels is applied to the second tensor prior to generating the plurality of tensors.

7. 1. A method for encoding into a bitstream a plurality of tensors forming a hierarchical representation of a frame, comprising: generating a first tensor from the plurality of tensors using one or more downsampling filters; generating a second tensor from the first tensor by scaling the first tensor according to a scaling tensor; generating coefficients of a third tensor using the second tensor and a set of basis vectors, the third tensor having fewer channels and the same spatial size as the second tensor; encoding, for the one frame, the coefficients of the third tensor into the bitstream; A method having the following.

8. The method of claim 7 , wherein the second tensor is generated using the output of an addition of tensors that include the result of an activation layer.

9. 8. The method of claim 7, wherein generating the second tensor further comprises applying a convolutional layer to the scaled first tensor to reduce the channel count.

10. 1. A decoder for decoding a plurality of tensors forming a hierarchical representation of a frame from a bitstream, the decoder comprising: decoding coefficients from the bitstream for a first tensor; generating a second tensor from the first tensor and a set of basis vectors, the second tensor having a higher channel count and the same spatial size as the first tensor; generating the plurality of decoded tensors from the second tensor using one or more trained convolutional layers; A decoder configured to:

11. 1. A non-transitory computer-readable storage medium storing a program for performing a method for decoding a plurality of tensors forming a hierarchical representation of a frame from a bitstream, the method comprising: decoding coefficients from the bitstream for a first tensor; generating a second tensor from the first tensor and a set of basis vectors, the second tensor having a higher channel count and the same spatial size as the first tensor; generating the plurality of decoded tensors from the second tensor using one or more trained convolutional layers; and 1. A non-transitory computer-readable storage medium having:

12. 1. A system comprising: Memory and a processor configured to execute code stored in the memory for performing a method for decoding a plurality of tensors forming a hierarchical representation of a frame from a bitstream; and the method comprises: decoding coefficients from the bitstream for a first tensor; generating a second tensor from the first tensor and a set of basis vectors, the second tensor having a higher channel count and the same spatial size as the first tensor; generating the plurality of decoded tensors from the second tensor using one or more trained convolutional layers; and A system having:

13. 1. An encoder for encoding a plurality of tensors forming a hierarchical representation of a frame into a bitstream, comprising: generating a first tensor from the plurality of tensors using one or more downsampling filters; generating a second tensor from the first tensor by scaling the first tensor according to a scaling tensor; generating coefficients of a third tensor using the second tensor and a set of basis vectors, the third tensor having fewer channels and the same spatial size as the second tensor; For the one frame, encoding the coefficients of the third tensor into the bitstream. An encoder configured as follows.

14. 1. A non-transitory computer-readable storage medium storing a program for performing a method for encoding a plurality of tensors forming a hierarchical representation of a frame into a bitstream, the method comprising: generating a first tensor from the plurality of tensors using one or more downsampling filters; generating a second tensor from the first tensor by scaling the first tensor according to a scaling tensor; generating coefficients of a third tensor using the second tensor and a set of basis vectors, the third tensor having fewer channels and the same spatial size as the second tensor; encoding, for the one frame, the coefficients of the third tensor into the bitstream; 1. A non-transitory computer-readable storage medium having:

15. 1. A system comprising: Memory and a processor configured to execute code stored in the memory for performing a method for encoding a plurality of tensors forming a hierarchical representation of a frame into a bitstream; and the method comprises: generating a first tensor from the plurality of tensors using one or more downsampling filters; generating a second tensor from the first tensor by scaling the first tensor according to a scaling tensor; generating coefficients of a third tensor using the second tensor and a set of basis vectors, the third tensor having fewer channels and the same spatial size as the second tensor; encoding, for the one frame, the coefficients of the third tensor into the bitstream; A system having:

Images