Method, apparatus and system for encoding and decoding tensor

Abstract

A method for decoding a plurality of tensors forming a hierarchical representation of a single frame from a bitstream and a method for encoding tensors forming a hierarchical representation of a single frame in a bitstream. The decoding method includes: decoding a coefficient from a bitstream for a first tensor; and generating a second tensor from the first tensor and the set of basis vectors, the second tensor having the same spatial size and a higher number of channels than the first tensor. The decoding method further includes using one or more trained convolutional layers to generate a plurality of decoded tensors from the second tensor.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit under 35 U.S.C. §119 of the filing date of Australian patent application 2023200117, filed on January 10, 2023, which is incorporated herein by reference in its entirety as if fully set forth herein. Technical Field

[0003] The present invention generally relates to digital video signal processing, and in particular, to methods, devices, and systems for encoding and decoding tensors from convolutional neural networks. The present invention also relates to a computer program product comprising a computer-readable medium having recorded thereon a computer program for encoding and decoding tensors from a convolutional neural network using video compression techniques. Background Art

[0004] Convolutional neural networks (CNNs) are an emerging technology for addressing use cases involving machine vision, such as object detection, instance segmentation, object tracking, human pose estimation, and action recognition. Applications of CNNs can involve the use of "edge devices" with sensors and some processing power, coupled to application servers as part of the "cloud." CNNs can require relatively high computational complexity, exceeding what can typically be provided within the computational capacity or power consumption of edge devices. Executing CNNs in a distributed manner has emerged as a solution for running cutting-edge networks using limited-capacity edge devices without having to infuse all computational complexity into cloud servers. In other words, distributed processing allows traditional edge devices to still provide the capabilities of cutting-edge CNNs by distributing processing between the edge devices and external processing components, such as cloud servers. This distributed network architecture can be referred to as "collaborative intelligence (CI)" and offers benefits such as reusing partial results from one part of the network for several different second parts, potentially each optimized for different tasks. CI architectures introduce the need for efficient compression of tensor data for transmission over networks such as wide area networks (WANs).

[0005] CNNs typically include many layers, such as convolutional layers and fully connected layers, where data is passed from one layer to the next in the form of "tensors." Splitting networks across different devices introduces the need to compress the intermediate multidimensional tensor data passed from one layer to the next within a CNN. Such tensor compression can be referred to as "feature compression," as the intermediate tensor data is often referred to as "features" or "feature maps" (generally, a collection of 2D "feature maps" forms a tensor, and each feature map corresponds to a "channel") and represents a partially processed form of an input, such as an image frame or video frame. The International Organization for Standardization / International Electrotechnical Commission Joint Technical Committee 1 / Subcommittee 29 / Working Group 2-8 (ISO / IEC JTC1 / SC29 / WG2-8), also known as the "Moving Picture Experts Group" (MPEG), has been tasked with studying compression techniques in various contexts, often related to video. WG2, "MPEG Technical Requirements," has established the "Video Coding for Machines" (VCM) ad hoc group, which is tasked with studying compression for machine consumption and feature compression. The feature compression commission is in the exploratory phase with the release of a Call for Evidence (CfE) for techniques that can significantly outperform feature compression results achieved using state-of-the-art standardized techniques.

[0006] CNNs typically require that weights for each layer be predetermined during a training phase, in which a very large amount of training data is passed through the CNN and the results determined by the trained network are compared with the ground truth associated with the training data. The difference between the obtained result and the expected result is represented as a "loss" and measured using a "loss function." Using the determined loss, a process (such as stochastic gradient descent (SGD)) is performed to update the network weights. Network weight updates typically involve backpropagation of "gradients," which indicate the deltas to be applied to the network weights, starting from the network's output layer and ending at the network's input layer, and covering the network's intermediate or "hidden" layers. The rate of weight updates is scaled by a "learning rate" hyperparameter, which is typically set to facilitate the training process finding a global minimum in terms of loss (i.e., the highest possible task performance for the network architecture and training data) while avoiding the training process from getting "stuck" in local minima. Being stuck in a local minimum corresponds to obtaining suboptimal task performance for the network architecture and being unable to find new weight values that could lead to higher task performance. The network weights are repeatedly updated by feeding input data and ground truth data organized into "batches" to iteratively refine the network performance until no further improvement in accuracy can be achieved. Iterations over the entire training data set form an "epoch" of training, and training typically requires multiple epochs to achieve a high level of performance for the task. The trained network can then be used for deployment, operating in a mode where the weights are fixed and gradients for weight updates are omitted. The process of executing a pre-trained CNN with input and gradually transforming the input into output according to the topology of the CNN is generally referred to as "inference."

[0007] Typically, a tensor has four dimensions: batch, channels, height, and width. The first dimension, "batch," is typically of size 1 when performing inference on video data and indicates that one frame is passed through the CNN as a batch. When training the network, the value of the batch dimension can increase, so that multiple frames are passed through the network in each batch according to a predetermined "batch size" before the network weights are updated. Multiple frames of video can be passed through as a single tensor with a batch dimension whose size increases based on the number of frames in a given video. However, due to practical considerations related to memory consumption and access, inference on video data is typically performed frame by frame. The "channels" dimension indicates the number of concurrent "feature maps" for a given tensor, while the height and width dimensions indicate the size of the feature maps at a particular stage of the CNN. The number of channels passed through a CNN layer varies depending on the network architecture. The size of the feature maps also varies depending on the subsampling or upsampling occurring in a particular network layer.

[0008] The overall complexity of CNNs tends to be relatively high due to the relatively large number of multiply-accumulate (MAC) operations performed and the large number of intermediate tensors written to and read from memory, as well as the weights read for the performance of each layer of the CNN. Therefore, splitting the neural network into parts allows for more complex networks to be implemented even in less capable edge devices.

[0009] Feature compression can benefit from existing video compression standards, such as Versatile Video Coding (VVC) developed by the Joint Video Experts Group (JVET). It is expected that VVC will address the continued demand for even higher compression performance, and the growing market demand for service delivery over WANs (where bandwidth costs are relatively high), particularly as the capabilities of video formats increase (e.g., with higher resolutions and higher frame rates). VVC can be implemented in contemporary silicon processes and provides an acceptable tradeoff between the achieved performance and the implementation cost. The implementation cost can be considered to be one or more of, for example, silicon area, CPU processor load, memory utilization, and bandwidth. Other video compression standards, such as High Efficiency Video Coding (HEVC) or AV-1, can also be used for feature compression applications.

[0010] Video data comprises a sequence of frames of image data, each frame comprising one or more color channels. Where feature map data is to be represented in packed frames, a monochrome frame having only luma and no chroma channels is generally sufficient. When only luma samples are present, the resulting monochrome frame is said to use a "4:0:0 chroma format."

[0011] The VVC standard specifies a "block-based" architecture in which a frame is first partitioned into an array of square areas called "coding tree units" (CTUs). In VVC, a CTU typically occupies 128×128 luma samples. Other possible CTU sizes when using the VVC standard are 32×32 and 64×64. However, the CTUs at the right and lower edges of each frame may be smaller in area, where implicit splitting occurs to ensure that the coding blocks remain in the frame. Associated with each CTU is a "coding tree" (also called a "coding unit" (CU)) that defines the decomposition of the area of the CTU into a set of blocks. Blocks that apply to only the luma channel or only the chroma channel are called "coding blocks" (CBs). The prediction of the contents of the coding block is held in a "prediction block" (PB) or a "prediction unit" (PU), and the residual block that defines the array of sample values to be combined with the PB or PU is called a "transform block" (TB) or a "transform unit" (TU) due to the transform processing typically used in the generation of the TB or TU.

[0012] Despite the above distinction between "unit" and "block", the term "block" may be used as a general term for an area or region of a frame for which operations are applied to all color channels.

[0013] For each CU, a prediction unit (PU) is generated ("prediction unit") of the contents (sample values) of the corresponding region of the frame data. In addition, a representation of the difference between the prediction and the contents of the region seen at the input of the encoder (or "spatial domain" residual) is formed. The differences in each color channel can be transformed and encoded into a sequence of residual coefficients, forming one or more TUs for a given CU. The transform applied can be a discrete cosine transform (DCT) or other transform applied to each block of residual values. The transforms are applied individually (i.e., two-dimensionally in two steps, once horizontally and once vertically). The block is first transformed by applying a one-dimensional transform to each row of samples in the block. The partial result is then transformed by applying a one-dimensional transform to each column of the partial result to produce a final block of transform coefficients that substantially decorrelate the residual samples. The VVC standard supports transforms of various sizes, including transforms of rectangular blocks whose sides are powers of 2. The transform coefficients are quantized for entropy encoding into the bitstream.

[0014] A PB or PU in a VVC can be generated using either intra prediction or inter prediction processes. Intra prediction involves using previously processed samples in the frame being used to generate a prediction of the current block of data samples in that frame. Inter prediction involves using a block of samples obtained from one or two previously decoded frames to generate a prediction of the current block of samples in the frame. The blocks of samples obtained from the previously decoded frames are offset in spatial position relative to the current block according to a motion vector, which is typically filtered. The intra predicted block can be: (i) uniform sample values ("DC intra prediction"), (ii) a plane with an offset and horizontal and vertical gradients ("planar intra prediction"), (iii) a population of blocks with neighboring samples applied in a particular direction ("angular intra prediction"), or (iv) the result of a matrix multiplication using neighboring samples and selected matrix coefficients.

[0015] VVC can be used to compress intermediate feature maps from the first part (the "backbone") of a neural network that is split into two parts. In compression, the feature maps from the backbone are arranged into frames and quantized from a floating point domain to a sample domain suitable for compression as video data. In order to reduce the spatial area of the feature maps, additional neural network layers can be implemented at the interface between the VVC encoder and decoder and the intermediate point in the CNN where the split occurs. Training is performed on such additional network layers, which may not be suitable for the varying and unpredictable feature map data encountered. Training may not result in a CNN that has adaptability to operating points of various qualities in terms of task performance. The operating points of the encoder and decoder may also change during operation, where it is necessary to support varying quality levels of the reconstructed tensors to be fed to the rest of the network on the decoder side. Summary of the Invention

[0016] It is an object of the present invention to substantially overcome or at least ameliorate one or more disadvantages of existing arrangements.

[0017] One aspect of the present disclosure provides a method for decoding multiple tensors that form a hierarchical representation of a single frame from a bitstream, the method comprising: decoding coefficients from the bitstream for a first tensor; generating a second tensor from the first tensor and a set of basis vectors, the second tensor having the same spatial size and a higher number of channels than the first tensor; and using one or more trained convolutional layers to generate multiple decoded tensors from the second tensor.

[0018] Another aspect of the present disclosure provides a method for encoding multiple tensors that form a hierarchical representation of a single frame into a bitstream, the method comprising: using one or more downsampling filters to generate a first tensor from the multiple tensors; generating a second tensor from the first tensor by scaling the first tensor according to a scaling tensor; using the second tensor and a set of basis vectors to generate coefficients of a third tensor, the third tensor having the same spatial size and fewer channels than the second tensor; and encoding the coefficients of the third tensor into the bitstream for the single frame.

[0019] Another aspect of the present disclosure provides a decoder for decoding a plurality of tensors forming a hierarchical representation of a single frame from a bitstream, the decoder being configured to: decode coefficients from the bitstream for a first tensor; generate a second tensor from the first tensor and a set of basis vectors, the second tensor having the same spatial size and a higher number of channels than the first tensor; and generate a plurality of decoded tensors from the second tensor using one or more trained convolutional layers.

[0020] Another aspect of the present disclosure provides a non-transitory computer-readable storage medium storing a program for executing a method for decoding a plurality of tensors forming a hierarchical representation of a single frame from a bitstream, the method comprising: decoding coefficients from a first tensor from the bitstream; generating a second tensor from the first tensor and a set of basis vectors, the second tensor having the same spatial size and a higher number of channels than the first tensor; and generating a plurality of decoded tensors from the second tensor using one or more trained convolutional layers.

[0021] Another aspect of the present disclosure provides a system comprising: a memory; and a processor, wherein the processor is configured to execute code stored on the memory for implementing a method for decoding multiple tensors from a bitstream to form a hierarchical representation of a single frame, the method comprising: decoding coefficients from the bitstream for a first tensor; generating a second tensor from the first tensor and a set of basis vectors, the second tensor having the same spatial size and a higher number of channels than the first tensor; and using one or more trained convolutional layers to generate multiple decoded tensors from the second tensor.

[0022] Another aspect of the present disclosure provides an encoder for encoding multiple tensors that form a hierarchical representation of a single frame into a bitstream, the encoder being configured to: generate a first tensor from the multiple tensors using one or more downsampling filters; generate a second tensor from the first tensor by scaling the first tensor according to a scaling tensor; generate coefficients of a third tensor using the second tensor and a set of basis vectors, the third tensor having the same spatial size and fewer channels than the second tensor; and encode the coefficients of the third tensor into the bitstream for the single frame.

[0023] Another aspect of the present disclosure provides a non-transitory computer-readable storage medium storing a program for executing a method for encoding multiple tensors forming a hierarchical representation of a single frame into a bitstream, the method comprising: generating a first tensor from the multiple tensors using one or more downsampling filters; generating a second tensor from the first tensor by scaling the first tensor according to a scaling tensor; generating coefficients of a third tensor using the second tensor and a set of basis vectors, the third tensor having the same spatial size and fewer channels than the second tensor; and encoding the coefficients of the third tensor into the bitstream for the single frame.

[0024] Another aspect of the present disclosure provides a system comprising: a memory; and a processor, wherein the processor is configured to execute code stored on the memory for implementing a method for encoding multiple tensors forming a hierarchical representation of a single frame into a bitstream, the method comprising: using one or more downsampling filters to generate a first tensor from the multiple tensors; generating a second tensor from the first tensor by scaling the first tensor according to a scaling tensor; using the second tensor and a set of basis vectors to generate coefficients of a third tensor, the third tensor having the same spatial size and fewer channels than the second tensor; and encoding the coefficients of the third tensor into the bitstream for the single frame.

[0025] Other aspects are also disclosed. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] At least one embodiment of the present invention will now be described with reference to the following drawings and appendix, in which:

[0027] Figure 1 is a schematic block diagram illustrating a distributed machine task system;

[0028] Figure 2A and Figure 2B Form a practical Figure 1 A schematic block diagram of a general computer system for a distributed machine task system;

[0029] Figure 3A is a schematic block diagram showing the functional modules of the main body of CNN;

[0030] Figure 3B It shows Figure 3A Schematic diagram of the residual block;

[0031] Figure 3C It shows Figure 3A Schematic diagram of the residual unit;

[0032] Figure 3D It shows Figure 3A Schematic block diagram of the CBL module;

[0033] Figure 4 is a schematic block diagram illustrating the functional modules of an alternative backbone of a CNN;

[0034] Figure 5 is a schematic block diagram illustrating a multi-scale feature fusion stage;

[0035] Figure 6 is a schematic block diagram illustrating an inter-channel decorrelation-based tensor encoder as part of a distributed machine task system;

[0036] Figure 7is a schematic block diagram illustrating a feature map packer, a packed frame encoder, and an unpacker (or "sub-picture encoder") as part of an inter-channel decorrelation-based tensor encoder;

[0037] Figure 8 is a schematic block diagram illustrating functional modules of a video encoder;

[0038] Figure 9 is a schematic block diagram illustrating an inter-channel decorrelation-based tensor decoder as part of a distributed machine task system;

[0039] Figure 10 is a schematic block diagram illustrating functional modules of a video decoder;

[0040] Figure 11A and Figure 11B is a schematic block diagram illustrating an embodiment of a multi-scale feature reconstruction stage;

[0041] Figure 12A is a schematic block diagram showing the head of a CNN;

[0042] Figure 12B It shows Figure 12A A schematic block diagram of an upgrader module;

[0043] Figure 12C It shows Figure 12A A schematic block diagram of a detection module;

[0044] Figure 13 is a schematic block diagram illustrating an alternative head of a CNN;

[0045] Figure 14A and Figure 14B is a schematic block diagram illustrating an arrangement of sub-pictures for preserving inter-channel decorrelated tensor data;

[0046] Figure 15 is a schematic block diagram illustrating a bitstream holding encoded inter-channel decorrelation feature maps and associated metadata;

[0047] Figure 16 Shows a method for performing the first part of a CNN, performing decorrelation, and encoding the resulting feature maps; and

[0048] Figure 17 A method for decoding the bitstream, reconstructing the decorrelated feature map, and performing the second part of the CNN is shown. DETAILED DESCRIPTION

[0049] Where steps and / or features are referenced in any one or more of the figures with the same reference numerals, these steps and / or features have the same function(s) or operation(s) for the purposes of this specification, unless otherwise indicated.

[0050] A distributed machine task system can include edge devices, such as web cameras or smartphones, that generate intermediate compressed data. The distributed machine task system can also include end devices, such as server farm ("cloud")-based applications that operate on the intermediate compressed data to produce task results. Additionally, edge device functionality can be embodied in the cloud, and the intermediate compressed data can be stored for later processing (possibly for multiple different tasks as needed).

[0051] A convenient form of intermediate compressed data is a compressed video bitstream, due to the availability of high-performance compression standards and their implementations. Video compression standards typically operate on integer samples of some given bit depth (such as 10 bits) arranged in a planar array. Depending on the application, color video has, for example, three planar arrays corresponding to the color components Y, Cb, Cr or R, G, B. CNNs typically operate on floating-point data in the form of tensors. Tensors generally have relatively smaller spatial dimensions than the input video data on which CNNs operate, and have more channels (e.g., 128, 256, or 512 channels) than the typical three channels of color video data.

[0052] A tensor typically has the following dimensions: frame, channel, height, and width. For example, a tensor of dimension [1, 256, 76, 136] would be considered to contain data for one frame, which includes two hundred and fifty-six (256) feature maps (channels), each of size 136 × 76. For video data, inference is typically performed on one frame at a time (frame value is 1), rather than using a tensor containing multiple frames.

[0053] VVC supports partitioning a picture into multiple sub-pictures, each of which can be independently encoded and independently decoded. In one approach, each sub-picture is encoded as a "slice" or a continuous sequence of encoded CTUs. A "tile" mechanism can also be used to partition a picture into multiple independently decodable regions. Sub-pictures can be specified in a slightly flexible way, where various rectangular sets of CTUs are encoded as corresponding sub-pictures. The flexible definition of sub-picture dimensions allows data types that require different areas to be efficiently maintained in one picture, thereby avoiding large "unused" areas (i.e., areas of the frame that are not used for reconstruction of tensor data).

[0054] Figure 11 is a schematic block diagram illustrating the functional modules of a distributed machine task system 100, which is capable of executing machine task networks in a distributed manner. Splitting a particular neural network into two parts requires specifying a "split point" in the network. The layers of the network, starting from the input layer and ending at the split point, are executed in a first device, and the resulting intermediate tensor(s) are compressed. The layers of the network, starting from the split point and ending at the last layer, are executed using the decompressed tensor(s) from the first device as input to the layer(s) immediately following the split point. At the split point, there may be one or more tensors that need to be compressed to transmit over a communication channel with limited bandwidth compared to the bandwidth required to transmit uncompressed tensors. In the case of a "feature pyramid network" (FPN), it is common to have layers in the FPN correlated in width and height, such that a given layer is half the width and height of an adjacent layer. The FPN architecture may also define width and height halving to occur at each alternating layer. In some architectures, multiple tensors of the same width and height are seen. The network can be split within the FPN of the machine task network, thereby facilitating various machine task networks, where the layers up to the split point are common between these machine task networks ("shared backbone" architecture). Therefore, compression methods that are applicable to the various network topologies used in contemporary CNNs are beneficial for application to a wide range of scenarios.

[0055] The system 100 can be used to implement a method for decorrelating, packing, and quantizing feature maps into a planar frame to encode the feature maps and decode the feature maps from the encoded data. In some embodiments, the system 100 can be used so that the bit rate of the compressed tensor is rate controllable, where the quality of the reconstructed tensor varies based on the selected rate. In some embodiments, the system 100 can be used so that the quantized representation of the tensor does not unnecessarily consume bits, where the bits do not provide a commensurate benefit in terms of task performance.

[0056] System 100 includes a source device 110 for generating encoded tensor data 115a from a CNN backbone 114 in the form of an encoded video bitstream 121. System 100 also includes a destination device 140 for decoding the tensor data in the form of an encoded video bitstream 143. A communication channel 130 is used to communicate the encoded video bitstream 121 from source device 110 to destination device 140. In some arrangements, one or both of source device 110 and destination device 140 may include a respective mobile phone handset (e.g., a "smartphone") or webcam and a cloud application. Communication channel 130 may be a wired connection such as Ethernet or a wireless connection such as WiFi or 5G, including connections across a wide area network (WAN) or across ad hoc connections. In addition, source device 110 and destination device 140 may include applications that capture the encoded video data on some computer-readable storage medium, such as a hard drive or memory in a file server.

[0057] like Figure 1 As shown, source device 110 includes a video source 112, a CNN backbone 114, a tensor combiner 162, a principal component analysis (PCA) encoder 160, and a transmitter 122. Video source 112 typically includes a source of captured video frame data (denoted as 113), such as a camera sensor, a previously captured video sequence stored on a non-transitory recording medium, or a video feed from a remote camera sensor. Video source 112 can also be the output of a computer graphics card, for example, displaying the video output of an operating system and various applications executed on a computing device (e.g., a tablet computer). Examples of source devices 110 that can include a camera sensor as video source 112 include smartphones, video camcorders, professional video cameras, and webcams. Video source 112 can produce independent images, or can produce temporally sequential images (i.e., video).

[0058] The CNN backbone 114 receives the video frame data 113 and performs a specific layer of the overall CNN (such as a layer corresponding to the "backbone" of the CNN), thereby outputting a tensor 115a. The backbone layer of the CNN can produce multiple tensors as output (e.g., corresponding to different spatial scales of the input image represented by the video frame data 113 when the network is split within the FPN). For example, if the system 100 performs a "YOLOv3" network, the FPN can obtain three tensors corresponding to three layers output from the backbone 114 as tensors 115a, where the tensors 115a have different spatial resolutions and numbers of channels. When the system 100 is performing a network such as "Faster RCNN X101-FPN" or "Mask RCNN X101-FPN", the tensor 115a may include tensors for four layers (P2 to P5). As described below, when the split point of the network occurs within FPN, the use of FPN results in multiple tensors that form a hierarchical representation of a single frame to be encoded into a bitstream (and decoded from the bitstream). The tensor combiner 162 can combine multiple layers by convolution with a stride greater than 1 (such as a stride of 2, etc.) to implement a trained downsampling stage, resulting in a tensor with the same dimensions as another spatially smaller tensor in the plurality of tensors 115a. The resulting set of tensors, all of which have the same spatial dimensions, can be concatenated along the channel dimension and further processed by additional network layers to produce the combined tensor 115. Reference Figure 5 15 is a block diagram illustrating the operation of the tensor combiner. The PCA encoder 160 receives the combined tensor 115 output from the tensor combiner 162. Combining layers is appropriate when there is sufficient inter-layer correlation to represent the combined layer using fewer basis vectors than would be required in the case of individually decorrelated layers. The degree of inter-layer correlation is a property of the network itself and the input data provided. The degree of inter-layer correlation permits the total number of basis vectors for the combined tensor to be reduced compared to the sum of the number of basis vectors in the case of individually decorrelated tensors for each layer. For example, if 25 basis vectors are typically used per layer and two layers are cascaded, the number of basis vectors required for the two cascaded layers can be set to less than 50, and the number of basis vectors required for the four cascaded layers can be set to less than 100.

[0059] The PCA encoder 160 is used to encode the combined tensor 115 to generate a bit stream 121, and the reference Figure 6 The bit stream 121 is supplied to a transmitter 122 for transmission via a communication channel 130 , or is written to a storage unit 132 for later use.

[0060] The source device 110 supports a specific network for the CNN backbone 114. However, the destination device 140 may use one of several networks for the CNN head 150. When one of several networks is used for the CNN head 150, partially processed data in the form of packed feature maps may be stored for later use in performing various tasks without having to perform the CNN backbone 114 operation again.

[0061] The bitstream 121 is transmitted as encoded video data (or "encoded video information") by a transmitter 122 over a communication channel 130. In some implementations, the bitstream 121 may be stored in a storage 132 (where the storage 132 is a non-transitory storage device such as a "Flash" memory or a hard drive) until (or in lieu of) later transmission over the communication channel 130. For example, the encoded video data may be provided on demand to consumers over a wide area network (WAN) for use in video analytics applications.

[0062] Destination device 140 includes receiver 142, PCA decoder 170, tensor splitter 172, CNN head 150, and CNN task result buffer 152. Receiver 142 receives encoded video data from communication channel 130 and passes video bitstream 143 to PCA decoder 170. PCA decoder 170 outputs decoded combined tensor 149a, which is fed to tensor splitter 172. Tensor splitter 172 performs the inverse operation of tensor combiner 162 to produce extracted tensor 149. Extracted tensor 149 is passed to CNN head 150. At a split point where the network is partitioned using a single tensor stage, layer combination is not required, tensor splitter 172 performs no operation, and tensor 149 corresponds to tensor 149a. CNN head 150 performs the later layers of the task starting from CNN backbone 114 to produce task result 151. Task result 151 is stored in task result buffer 152. The contents of task result buffer 152 can be presented to a user, for example, via a graphical user interface, or provided to an analysis application that determines an action based on the task results, which may include a summary-level presentation of the aggregated task results to the user. It is also possible that the functionality of source device 110 and destination device 140 can be embodied in a single device, examples of which include mobile phones and tablet computers, as well as cloud applications. Although the examples described herein relate to PCA, other decomposition analysis or methods based on inter-channel decorrelation can be used for encoder 160 and decoder 170.

[0063] Although example devices are described above, source device 110 and destination device 140 may each typically be configured within a general-purpose computer system via a combination of hardware and software components. Figure 2AIllustrated is a computer system 200 comprising a computer module 201; input devices such as a keyboard 202, a mouse pointer device 203, a scanner 226, a camera 227 that can be configured as a video source 112, and a microphone 280; and output devices including a printer 215, a display device 214, and a speaker 217. Computer module 201 can use an external modulator-demodulator (modem) transceiver device 216 to communicate with a communications network 220 via a connection 221. Communications network 220, which can represent communications channel 130, can be a (WAN) such as the Internet, a cellular telecommunications network, or a private WAN. If connection 221 is a telephone line, modem 216 can be a conventional "dial-up" modem. Alternatively, if connection 221 is a high-capacity (e.g., cable or optical) connection, modem 216 can be a broadband modem. Wireless connections to communications network 220 can also be made using wireless modems. The transceiver device 216 may provide the functionality of the transmitter 122 and the receiver 142 , and the communication channel 130 may be embodied in the connection 221 .

[0064] Computer module 201 typically includes at least one processor unit 205 and a memory unit 206. Memory unit 206 may include, for example, semiconductor random access memory (RAM) and semiconductor read-only memory (ROM). Computer module 201 also includes a plurality of input / output (I / O) interfaces, including an audio-video interface 207 coupled to a video display 214, speakers 217, and microphone 280; an I / O interface 213 coupled to a keyboard 202, mouse 203, scanner 226, camera 227, and an optional joystick or other human interface device (not shown); and an interface 208 for an external modem 216 and printer 215. The signal from audio-video interface 207 to computer monitor 214 is typically the output of the computer's graphics card. In some implementations, modem 216 may be internal to computer module 201, for example, within interface 208. The computer module 201 also has a local network interface 211, which allows the computer system 200 to be coupled to a local area communication network 222, known as a local area network (LAN), via a connection 223. Figure 2A As shown, the local area communication network 222 may also be connected to the wide area network 220 via a connection 224, wherein the local area communication network 222 will typically include a so-called "firewall" device or a device with similar functionality. The local network interface 211 may include an Ethernet TM )Circuit card, Bluetooth TM) wireless arrangement or IEEE 802.11 wireless arrangement; however, various other types of interfaces may be implemented for the interface 211. The local network interface 211 may also provide the functionality of the transmitter 122 and the receiver 142, and the communication channel 130 may also be embodied in the local area communication network 222.

[0065] I / O interfaces 208 and 213 may provide either or both serial and parallel connectivity, with the former typically being implemented according to the Universal Serial Bus (USB) standard and having a corresponding USB connector (not shown). A storage device 209 is provided and typically includes a hard disk drive (HDD) 210. Other storage devices (not shown) such as floppy disk drives and tape drives may also be used. An optical disk drive 212 is typically provided to serve as a non-volatile source of data. Optical disks (e.g., CD-ROMs, DVDs, Blu-ray Discs, etc.) may be used, for example. TM )), portable memory devices such as USB-RAM, portable external hard drives, and floppy disks are suitable sources of data for computer system 200. Typically, any of HDD 210, optical drive 212, and networks 220 and 222 may also be configured to operate as video source 112 or as a destination for stored decoded video data to be reproduced via display 214. Source device 110 and destination device 140 of system 100 may be embodied in computer system 200.

[0066] The components 205 to 213 of the computer module 201 typically communicate via an interconnecting bus 204 and in a manner that results in conventional modes of operation of the computer system 200 known to those skilled in the relevant art. For example, the processor 205 is coupled to the system bus 204 using a connection 218. Similarly, the memory 206 and the optical drive 212 are coupled to the system bus 204 via a connection 219. Examples of computers on which the described arrangement may be practiced include IBM-PCs and compatibles, Sun SPARCstations, Apple Mac™, or similar computer systems.

[0067] Where appropriate or desired, the PCA encoder 160 and the PCA decoder 170, as well as the methods described below, may be implemented using the computer system 200. In particular, the PCA encoder 160, the PCA decoder 170, and the methods to be described may be implemented as one or more software applications 233 executable within the computer system 200. In particular, instructions 231 (see instructions 231 and 233) in the software 233 may be executed within the computer system 200. Figure 2B) to implement PCA encoder 160, PCA decoder 170, and the steps of the method. Software instructions 231 can be organized into one or more code modules, each for performing one or more specific tasks. The software can also be split into two separate parts, with a first part and corresponding code modules performing the method, and a second part and corresponding code modules managing the user interface between the first part and the user.

[0068] For example, the software may be stored in a computer-readable medium, including a storage device described below. The software is loaded from the computer-readable medium into the computer system 200 and then executed by the computer system 200. A computer-readable medium having such software or a computer program recorded on the computer-readable medium is a computer program product. Using the computer program product in the computer system 200 preferably implements an advantageous apparatus for implementing the source device 110 and the destination device 140 and the described method.

[0069] The software 233 is typically stored in the HDD 210 or the memory 206. The software is loaded into the computer system 200 from a computer readable medium and executed by the computer system 200. Thus, for example, the software 233 may be stored on an optically readable disk storage medium (e.g., a CD-ROM) 225 that is read by the optical disk drive 212.

[0070] In some examples, the application 233 is provided to the user in a manner encoded on one or more CD-ROMs 225 and read via corresponding drives 212, or alternatively, the application 233 can be read by the user from the network 220 or 222. Furthermore, the software can also be loaded into the computer system 200 from other computer-readable media. A computer-readable storage medium refers to any non-transitory tangible storage medium that provides recorded instructions and / or data to the computer system 200 for execution and / or processing. Examples of such storage media include floppy disks, magnetic tapes, CD-ROMs, DVDs, Blu-ray discs, hard drives, ROMs or integrated circuits, USB memories, magneto-optical discs, or computer-readable cards such as PCMCIA cards, regardless of whether these devices are internal or external to the computer module 201. Examples of transient or non-tangible computer-readable transmission media that may also participate in the provision of software, applications, instructions and / or video data or encoded video data to the computer module 201 include: radio or infrared transmission channels and network connections to another computer or networked device, and the Internet or an intranet including email transmissions and information recorded on websites.

[0071] The second portion of the application 233 and the corresponding code modules described above can be executed to implement one or more graphical user interfaces (GUIs) to be rendered or otherwise displayed on the display 214. Users and applications of the computer system 200 can manipulate the interface in a functionally applicable manner by typically manipulating the keyboard 202 and mouse 203 to provide control commands and / or input to the applications associated with these (one or more) GUIs. Other functionally applicable forms of user interfaces can also be implemented, such as audio interfaces that utilize voice prompts output via the speaker 217 and user voice commands input via the microphone 280.

[0072] Figure 2B is a detailed schematic block diagram of the processor 205 and the "memory" 234. The memory 234 represents Figure 2A A logical aggregation of all memory modules (including storage device 209 and semiconductor memory 206) that can be accessed by computer module 201.

[0073] When the computer module 201 is initially powered on, a power-on self-test (POST) program 250 is executed. The POST program 250 is typically stored in Figure 2A 249 of the semiconductor memory 206. Hardware devices such as the ROM 249 that store software are sometimes referred to as firmware. The POST program 250 checks the hardware within the computer module 201 to ensure proper operation and typically checks the processor 205, the memory 234 (209, 206) and the basic input-output system software (BIOS) module 251, which is also typically stored in the ROM 249, for correct operation. Once the POST program 250 runs successfully, the BIOS 251 activates Figure 2A Activating the hard disk drive 210 causes the processor 205 to execute a boot loader program 252 residing on the hard disk drive 210. This loads the operating system 253 into the RAM memory 206, where the operating system 253 begins operating. The operating system 253 is a system-level application executable by the processor 205 to implement various high-level functions including processor management, memory management, device management, storage management, software application interfaces, and general user interfaces.

[0074] The operating system 253 manages the memory 234 (209, 206) to ensure that each process or application running on the computer module 201 has sufficient memory to execute without conflicting with memory allocated to another process. Figure 2AThe aggregate memory 234 is a diagram illustrating the different types of memory available in the computer system 200 so that the various processes can run efficiently. Therefore, the aggregate memory 234 is not intended to illustrate how specific segments of memory are allocated (unless otherwise specified), but rather to provide an overview of the memory accessible to the computer system 200 and how such memory is used.

[0075] like Figure 2B As shown, the processor 205 includes multiple functional modules, including a control unit 239, an arithmetic logic unit (ALU) 240, and a local or internal memory 248, sometimes referred to as a cache. The cache 248 typically includes a plurality of storage registers 244 to 246 in a register section. One or more internal buses 241 functionally interconnect these functional modules. The processor 205 also typically has one or more interfaces 242 for communicating with external devices via the system bus 204 using a connection 218. The memory 234 is coupled to the bus 204 using a connection 219.

[0076] Application program 233 includes an instruction sequence 231 that may include conditional branch instructions and loop instructions. Program 233 may also include data 232 used when executing program 233. Instructions 231 and data 232 are stored in memory locations 228, 229, 230 and 235, 236, 237, respectively. Depending on the relative sizes of instructions 231 and memory locations 228-230, a particular instruction may be stored in a single memory location, as illustrated by the instruction shown in memory location 230. Alternatively, an instruction may be segmented into multiple portions, each stored in a separate memory location, as illustrated by the instruction segments shown in memory locations 228 and 229.

[0077] Typically, a processor 205 is given a set of instructions, which are executed within the processor 205. The processor 205 awaits subsequent input, to which the processor 205 reacts by executing another set of instructions. Each input may be provided from one or more of a plurality of sources, including data generated by one or more of the input devices 202, 203, data received from an external source across one of the networks 220, 202, data retrieved from one of the storage devices 206, 209, or data retrieved from a storage medium 225 inserted into a corresponding reader 212 (all of which are described in detail in the accompanying drawings). Figure 2A Execution of the instruction set may, in some cases, result in outputting data. Execution may also involve storing data or variables in memory 234.

[0078] The PCA encoder 160, PCA decoder 170, and the described methods may use input variables 254 stored in corresponding memory locations 255, 256, 257 within the memory 234. The PCA encoder 160, PCA decoder 170, and the described methods produce output variables 261 stored in corresponding memory locations 262, 263, 264 within the memory 234. Intermediate variables 258 may be stored in memory locations 259, 260, 266, and 267.

[0079] refer to Figure 2B The processor 205, registers 244, 245, 246, arithmetic logic unit (ALU) 240, and control unit 239 work together to perform the sequence of micro-operations required to perform a "fetch, decode, and execute" cycle for each instruction in the instruction set that makes up program 233. Each fetch, decode, and execute cycle includes:

[0080] a fetch operation for fetching or reading an instruction 231 from memory locations 228 , 229 , 230 ;

[0081] a decode operation in which the control unit 239 determines which instruction was fetched; and

[0082] An execute operation, wherein in the execute operation, the control unit 239 and / or the ALU 240 executes the instruction.

[0083] Thereafter, a further fetch, decode, and execute cycle for the next instruction may be performed.Similarly, a store cycle may be performed, whereby the control unit 239 stores or writes a value to the memory location 232 .

[0084] To be explained Figure 16 and Figure 17 Each step or sub-process in the method is associated with one or more segments of the program 233 and is typically performed by the register segments 244, 245, 246, ALU 240 and control unit 239 in the processor 205 working together to perform a fetch, decode and execute cycle for each instruction in the instruction set of the segment of the program 233.

[0085] Figure 3A is a schematic block diagram 300 illustrating the functional modules of a CNN backbone 310 that can be used as an implementation of a CNN backbone 114. Backbone 114 is sometimes referred to as "DarkNet-53," although different backbones are possible, resulting in different numbers and dimensions of layers in tensors 115 for each frame.

[0086] like Figure 3AAs shown in FIG, the video data 113 is passed to a resizer module 304. The resizer module 304 resizes each frame of the video data 113 to a resolution suitable for processing by the CNN backbone 310, thereby generating resized frame data 312. If the resolution of the video data 113 is already suitable for the CNN backbone 310, then the operation of the resizer module 304 is not required. The resized frame data 312 is passed to a convolutional batch normalization leaky rectified linear (CBL) module 314 to generate a tensor 316. The CBL 314 contains the reference Figure 3D The modules described are shown as CBL modules 360.

[0087] CBL module 360 takes as input a tensor 361 of resized frame data 312. This tensor 361 is passed to convolutional layer 362 to produce tensor 363. If convolutional layer 362 has a stride of 1, tensor 363 has the same spatial dimensions as tensor 361. If convolutional layer 362 has a larger stride (such as 2), tensor 363 has smaller spatial dimensions than tensor 361. For example, for a stride of 2, the width and height of tensor 363 are halved. Regardless of the stride, the size of the channel dimension of tensor 363 can vary compared to the channel dimension of tensor 361 for a particular CBL block. Tensor 363 is passed to batch normalization module 364, which outputs tensor 365. Batch normalization module 364 normalizes input tensor 363 and applies a scaling factor and offset value to produce output tensor 365. The scaling factor and offset value are derived from the training process. Tensor 365 is passed to a leaky rectified linear activation ("LeakyReLU") module 366 to produce tensor 367. Module 366 provides a "leaky" activation function whereby positive values in the tensor are passed through and negative values are severely reduced in magnitude, e.g., to 0.1 times their previous value.

[0088] Return to Figure 3A , the tensor 316 is passed from the CBL block 314 to a residual block module 320, such as a 1+2+8 module (also called an 11 module) that internally contains a cascade of 1 residual unit, 2 residual units, and 8 residual units.

[0089] References Figure 3BResBlock 340 is shown to illustrate a residual block. ResBlock 340 receives a tensor 341. Tensor 341 is zero-padded by a zero-padding module 342 to produce a tensor 343. Tensor 343 is passed to a CBL module 344 to produce a tensor 345. Tensor 345 is passed to a residual unit 346. Residual unit 346 includes a series of cascaded residual units based on the number of residual blocks (e.g., 11 units for block 320). The last residual unit in residual unit 346 outputs a tensor 347.

[0090] References Figure 3C 350 is shown to illustrate a residual unit. ResUnit 350 takes tensor 351 as input. Tensor 351 is passed to a CBL module 352 to produce tensor 353. Tensor 353 is passed to a second CBL unit 354 to produce tensor 355. An addition module 356 sums tensor 355 with tensor 351 to produce tensor 357. Addition module 356 can also be called a "shortcut" because input tensor 351 actually affects output tensor 357. For an untrained network, ResUnit 350 acts to pass through the tensor. When training, CBL modules 352 and 354 act to deviate tensor 357 from tensor 351 based on training data and ground truth data.

[0091] Return to Figure 3A , the Res11 module 320 outputs a tensor 322. Tensor 322 is output from the backbone module 310 as one of the layers and is also provided to the Res8 module 324. The Res8 module 324 is a residual block (i.e., 340) including eight residual units (i.e., 350). The Res8 module 324 generates a tensor 326. The tensor 326 is passed to the Res4 module 328 and is output from the backbone module 310 as one of the layers. The Res4 module is a residual block (i.e., 340) including four residual units (i.e., 350). The Res4 module 328 generates a tensor 329. The tensor 329 is output from the backbone module 310 as one of the layers. In general, the layer tensors 322, 326, and 329 are output as tensor 115a. The CNN backbone 310 may take a video frame of resolution 1088×608 as input and generate three tensors corresponding to three layers, with the following dimensions: [1, 256, 76, 136], [1, 512, 38, 68], and [1, 1024, 19, 34]. Another example of the three tensors 115 corresponding to the three layers may be [1, 512, 34, 19], [1, 256, 68, 38], and [1, 128, 136, 76], separated at the 75th, 90th, and 105th feature maps in the CNN 310, respectively. The separation point depends on the CNN 310.

[0092] Each of Res11 320, Res8 324, and Res4 328 operates in a similar manner to ResBlock 340. Each of CBL 314, CBL 344, and CBL 354 operates in a similar manner to CBL 360.

[0093] Figure 4 1 is a schematic block diagram illustrating the functional modules of an alternative CNN backbone 400 that can be used as an implementation of the CNN backbone 114. The backbone 400 implements a residual network with a feature pyramid network ("ResNet FPN") and is used in networks such as FasterRCNN and MaskRCNN. Frame data 113 is input and passed through the stem network 408, res2 module 412, res3 module 416, res4 module 420, and res5 module 424 via tensors 409, 413, 417, 421, and 425, respectively.

[0094] The stem network 408 includes 7×7 convolutions and max pooling operations with a stride of two (2). The res2 module 412, the res3 module 416, the res4 module 420, and the res5 module 424 perform convolution operations such as LeakyReLU activations. Each module 412, 416, 420, and 424 also halves the width and height of the processed tensor once via a stride of 2. Each of the tensors 413, 417, 421, and 425 is passed to one of the 1×1 horizontal convolution modules 446, 444, 442, and 440, respectively. Modules 446, 444, 442, and 440 produce tensors 447, 445, 443, and 441, respectively. Tensor 441 is passed to the 3×3 output convolution module 470, which produces the output tensor P5 471.

[0095] Tensor 441 is also passed to upsampler module 450 to produce upsampled tensor 451. Summation module 460 sums tensors 443 and 451 to produce tensor 461. Tensor 461 is passed to upsampler module 452 and 3×3 horizontal convolution module 472. Module 472 outputs P4 tensor 473. Upsampler module 452 produces upsampled tensor 453. Summation module 462 sums tensors 445 and 453 to produce tensor 463. Tensor 463 is passed to 3×3 horizontal convolution module 474 and upsampler module 454. Module 474 outputs P3 tensor 475. Upsampler module 454 outputs upsampled tensor 455. Summation module 464 sums tensors 447 and 455 to produce tensor 465, which is passed to 3×3 horizontal convolution module 476. Module 476 outputs P2 tensor 477. Upsampler modules 450, 452, and 454 use nearest neighbor interpolation to reduce computational complexity. Tensors 429, 471, 473, 475, and 477 form the output tensor 115a of CNN backbone 400. Although Figure 4 A specific trunk ("P-layer" split point) of the Faster RCNN network architecture is shown, but different splits into the trunk and head are possible. Splitting the network at tensor 409 is called a "stem" split point. Splitting the network at tensors 447, 445, 443, and 441 is called a "C-layer" split point.

[0096] Figure 5 is a schematic block diagram 500 illustrating one type of multi-scale feature fusion module 500 that may be used as the tensor combiner 162 . Figure 6 is a schematic block diagram 600 illustrating a tensor encoder based on inter-channel decorrelation. Figure 7 7 is a schematic block diagram illustrating a sub-picture encoder. Encoders 600 and 700 form the PCA encoder 160 of the system 100. Figure 16 To illustrate Figures 5 to 7 .

[0097] Figure 16A method 1600 is shown for performing the first portion of a CNN and encoding the resulting feature maps for a frame of video data. When encoding the feature maps, tensors are encoded into a bitstream. Method 1600 can be implemented using a device such as a configured FPGA, ASIC, or ASSP. Alternatively, as described below, method 1600 can be implemented by source device 110 as one or more software code modules of application 233 under execution by processor 205. The software code modules of application 233 implementing method 1600 can reside, for example, in hard drive 210 and / or memory 206. Method 1600 is performed for each frame of video data generated by video source 112. Method 1600 can be stored on a computer-readable storage medium and / or in memory 206. Method 1600 begins with step 1610 of performing the first portion of a CNN.

[0098] At step 1610, the CNN backbone 114, under execution of the processor 205, performs a subset of the layers of a specific CNN to convert the input frame 113 into an intermediate tensor 115a. The intermediate tensor 115a may be stored in the memory 206 and / or the hard drive 210, for example. Figure 4 As shown, the example CNN is "Faster R-CNN" or "Mask R-CNN," and the subset of layers corresponds to all layers up to the "P-layer" split point. When multiple tensors are extracted from CNN backbone 114, for example due to the use of FPN, tensor 115a contains one tensor for each FPN layer. From step 1610, control in processor 205 proceeds to a tensor reduction step 1615.

[0099] At step 1615, the tensor combiner 162 operates under the execution of the processor 205 to resample the individual tensors in the tensor 115a to produce a combined tensor 115 as output. The multi-scale feature fusion (MSFF) module 500 includes Figure 5As shown, the MSFF block 510 uses one or more downsampling filters to generate a single tensor from multiple tensors 115a. Under execution by the processor 205, the MSFF block 510 combines the tensors of the first set of tensors (i.e., 502, 503, 504, 505) to generate a combined tensor 115a. The combined tensor 115a forms a representation of the FPN layer tensor. Downsampling modules 522a, 522b, and 522c operate on tensors with larger spatial scales (i.e., P4 504 at 2h, 2w, 256, P3 503 at 4h, 4w, 256, and P2 502 at 8h, 8w, 256, respectively). Modules 522a, 522b, and 522c perform downsampling to match the spatial scale of the smallest tensor (i.e., P5 505 at h, w, 256), thereby producing downscaled P5 tensors 523a, 523b, and 523c, respectively. Cascade module 524 performs channel-by-channel concatenation of tensors 505, 523a, 523b, and 523c to produce a concatenated tensor 525 of dimension h, w, 1024. Concatenated tensor 525 is passed to squeeze and excite (SE) module 526 to produce tensor 527. SE module 526 sequentially performs global pooling, a fully connected layer with reduced channel count, a rectified linear unit activation unit, a second fully connected layer to restore the channel count, and a sigmoid activation function to produce a scaled tensor. Tensor 525 is scaled according to the scaled tensor to produce an output as tensor 527. The SE block 526 can be trained to adaptively change the weighting of different channels in the passing tensor based on the output of the first fully connected layer. The output of the first fully connected layer reduces each feature map for each channel to a single value. Each single value is passed through a nonlinear activation unit (ReLU) to create a conditional representation of the unit suitable for the weighting of other channels, where the second fully connected layer performs the restoration to the full number of channels. Thus, the SE block 526 is able to extract nonlinear inter-channel correlation when generating tensor 527 from tensor 525 to a greater extent than is possible using a pure convolutional (linear) layer. Tensor 527 is passed to the convolutional layer 528. The convolutional layer 528 implements one or more convolutional layers to produce a combined tensor 115, where the number of channels is reduced to F channels (typically 256 channels) (i.e., F=256). Control in the processor 205 enters the determine mean channel step 1620 from step 1615.

[0100] In an alternative implementation, as shown in dashed lines, convolution layer 528 can be omitted from module 500. In an implementation in which convolution layer 528 is omitted, tensor 527 is output as tensor 115. Effectively, in implementations other than 528, tensor 115 is generated using the output of the addition of tensors including the results of the activation layers in SE block 526. If layer 528 is omitted, the number of channels at tensor 115 is not reduced to F channels. In the case where layers P2 through P5 each have 256 channels, the number of channels at tensor 115 when convolution layer 528 is omitted would be 1024. As described above, applying convolution layer 528 to scaled tensor 527 reduces the number of channels and, therefore, the dimensionality of the tensor data fed to PCA encoder 160 and recovered by PCA decoder 170.

[0101] At determine mean channel step 1620, the tensor 115 is averaged. Figure 6 , module 610, under execution by processor 205, performs an average operation on tensor 115 across spatial dimensions to produce per-channel means 611. At step 1620, quantizer 612, under execution by processor 205, quantizes mean channels 611 to produce integer (quantized) mean channel lists 613 and quantization ranges indicating the floating point range required to maintain mean channels 611. From step 1620, control in processor 205 passes to an encode mean channel step 1630.

[0102] At an encode mean channel step 1630, the sub-picture encoder 614, under execution by the processor 205, packs the integer mean channels 613 into a sub-picture (e.g., as the sub-picture 1410 shown in FIG. 14 ), and encodes the sub-picture using the sub-picture encoder 614 to produce a bitstream portion 615. Because the mean channel 613 contains one value per channel, the channel 613 can be viewed as a feature map of, for example, height 1 and width 256 when representing the mean of a tensor having 256 channels.

[0103] The sub-picture encoder 614 implements the architecture 700. The architecture 700 includes a feature map packer 710, a video frame encoder 714, and an unpacker 720. The sub-picture encoders 614, 636, 654a, 654b, and 654c are implemented as instances of the architecture 700. The packer 710 receives a tensor 708 having a given number of channels, width, and height dimensions and containing integer values (i.e., that has been quantized). For example, the sub-picture encoder 614 receives an integer mean channel 613. The packer 710 packs the received tensor into a 2D planar array of samples. Typically, in the described arrangement, the feature maps of the individual channels of the tensor 708 are stored as sub-picture frames 712 in a left-to-right and top-to-bottom manner. The sub-picture frames 712 need to be of sufficient size to hold the channels of the tensor 708, including gaps to allow for packing due to mismatches between the feature map size and the sub-picture frame 712 dimensions. Reference Figure 8 714. The operation of a video frame encoder 714, which is typically implemented as a VVC encoder, is illustrated. The encoder 714 produces an encoded bitstream portion 716 corresponding to a corresponding picture. For example, the sub-picture encoder 614 outputs an encoded bitstream portion 615. The encoder 714 also outputs a reconstructed frame 718 corresponding to a lossy version reproduced when the bitstream portion 716 is decoded. The reconstructed frame 718 represents a reconstruction of the mean channel 613, in which the losses invoked by the encoding are modeled or represented. These losses reflect encoding losses, such as encoding losses caused by the particular encoding method used at the encoder 714. Figure 7 and Figure 8 In the example depicted, the encoder 714 is a VVC encoder. The reconstructed frame 718 is passed to the unpacker 720. The unpacker 720 extracts the feature maps to produce a reconstructed tensor 722 having the same dimensions as the tensor 708, thereby forming the tensor by performing a channel-by-channel concatenation of the unpacked feature maps.

[0104] As a result of the operation of module 700, the PCA encoder 160 is able to use a version of the feature map (or coefficients) that corresponds to the version seen in the PCA decoder 170. Therefore, the PCA encoder 160 can operate at a higher level of fidelity than if the effects of lossy encoding were not taken into account. The sub-picture encoders 614, 636, 654a, 654b and 654c are configured to disable loop filtering both within and across sub-picture boundaries, as loop filtering is typically optimized for human consumption of decoded pictures. In addition, motion compensation is prohibited from accessing samples across sub-picture boundaries via an activated "sps_subpic_treated_as_pic_flag" flag for each sub-picture. The system 100 uses video compression to relatively efficiently represent the data resulting from the dimensionality reduction performed on the intermediate tensor data that needs to be propagated from the CNN backbone 114 to the CNN head 150.

[0105] return Figure 16 , the control in the processor 205 enters the step 1640 of restoring and reconstructing the mean channel from step 1630 .

[0106] At a Recover Reconstructed Mean Channel step 1640, the sub-picture encoder 614, under execution by the processor 205, outputs a reconstructed picture (e.g., 716) corresponding to the lossy version of the sub-picture input for video compression (e.g., 712) within the implementation of the encoder 700. The reconstructed picture is unpacked and output as an integer tensor 616. From step 1640, control in the processor 205 passes to a Zero Mean Tensor step 1645.

[0107] At step 1645, the inverse quantizer module 620 and the subtraction module 622, under execution by the processor 205, generate a zero-mean tensor 623. The inverse quantizer 620 takes the integer tensor 616 and outputs a reconstructed mean channel 621 using the quantization range determined in the quantizer module 612. The reconstructed mean channel 621 is a list of values corresponding to the detected DC offset in the corresponding channel (one value per channel in the tensor 115). For each channel in the combined tensor 115, a DC shift is performed by subtracting a constant value for the feature map from each spatial location in the feature map. The subtracted value is the corresponding value in the reconstructed mean channel 621. As a result of the subtraction module 622, a zero-centered tensor 623 is output, in which the DC component found in each feature map is removed from the corresponding feature map. Control in the processor 205 passes from step 1645 to a determine basis vectors step 1650.

[0108] At the determine basis vector step 1650 , the decomposition module 630 operates under the execution of the processor 205 to generate a set of basis vectors for the combined tensor 115 . The decomposition module 630 receives the zero-centered tensor 623 as input and generates a set of basis vectors 631 by performing a principal component analysis method (such as singular value decomposition (SVD)). A basis vector maps all channels to a single value, so if there are 256 channels in the tensor 115, a basis vector has dimensions 256×1. If the decomposition module generates the first N basis vectors (such as 25), the resulting basis vectors have dimensions 256×N or 256×25. The basis vectors are relative to the origin, so the use of a zero-mean tensor 623 is necessary to ensure that an orthogonal basis can be found. Each basis vector is a vector that relates all channels to a reduced set of channels. In this way, the basis vectors collectively enable the tensor data across all channels to be represented as a smaller set of basis vectors. Each basis vector is derived by considering all samples in each feature map for a given channel. Vector 631 contains fewer basis vectors than channels present in tensor 115, corresponding to a dimensionality reduction of tensor 115. The basis vectors in 631 represent tensor 115 in a subspace that accounts for or "explains" the greatest amount of variance in tensor 115 for the number of components in basis vectors 631. Basis vectors are ordered from vectors with the greatest explained variance down to vectors with the least explained variance. In other words, basis vectors 631 enable tensor 115 to be represented with minimal quality degradation for a given number of components, where the components are the top N ranked basis vectors. From step 1465, control in processor 205 proceeds to an encode basis vector step 1660.

[0109] At an Encode Basis Vectors step 1660, the quantizer module 632 operates under execution of the processor 205 to quantize the basis vectors 631 into the integer domain. The resulting integer basis vectors 634 are packed into sub-pictures and encoded by the sub-picture encoder 636 under execution of the processor 205 to produce the bitstream portion 637. From step 1660, control in the processor 205 passes to a Recover Reconstructed Basis Vectors step 1670.

[0110] At step 1670, the sub-picture encoder 636, under execution by the processor 205, generates a reconstructed integer tensor 638 and operates to obtain a reconstructed version (e.g., 638) of the sub-picture 1412. The basis vectors 638 are unpacked from the reconstructed sub-picture and inverse quantized back to the floating-point domain (as reconstructed basis vectors 640) by the inverse quantizer 660, under execution by the processor 205. From step 1670, control in the processor 205 passes to a determine coefficients step 1680.

[0111] At step 1680, dot product module 642, under execution by processor 205, performs a dot product of each channel in tensor 623 with each vector in reconstructed basis vectors 640 to generate coefficient tensor 644. The coefficients form a tensor having the same width and height as tensor 115, but having a number of channels corresponding to the number of components (or basis vectors) generated by decomposition module 630, which is less than the number of channels in tensor 115. Coefficient tensor 644 represents the contribution of each basis vector in reproducing each value in each feature map. From step 1680, control in processor 205 passes to quantize coefficients step 1690.

[0112] At step 1690, the quantizer module 646, under execution by the processor 205, quantizes the coefficient tensor 644 to produce integer coefficients 648 (a tensor having the same dimensions as the coefficient tensor 644 (that is, having c channels, a width of w, and a height of h, where c corresponds to the number of basis vectors). The quantization range is determined based on the extreme values in the coefficient tensor 644. From step 1690, control in the processor 205 passes to an assign coefficients to groups step 16100.

[0113] When performing either of steps 1680 and 1690, method 1600 operates to generate coefficients for a tensor using the (reduced) tensor and the set of basis vectors generated at step 1615. The tensor for which the coefficients are generated has the same spatial size and fewer channels than the tensor generated at step 1615.

[0114] At step 16100, the group module 650, under execution by the processor 205, determines a set of groups for the coefficient tensor 644. Each group contains coefficients for the entire feature map and one or more channels, contiguous along the channel dimension. The first group always contains coefficients using the first basis vector. In other words, the coefficient tensor 644 is sliced along the channel dimension to form a set of coefficient groups 652 having a total of c channels, a width of w, and a height of h. Each group includes a contiguous range of channels, and the groups are adjacent. In general, a group includes all channels. In the context of the coefficient tensor 644, the channel dimension corresponds to the number of basis vectors, not the number of channels of the combined tensor 115. The boundaries between groups are established when encoding the first frame of video data, which corresponds to establishing a sub-picture layout in the sequence parameter set (SPS) associated with an instantaneous decoder refresh (IDR) picture in the bitstream 121. Each group can be encoded as a unit of coefficient information to be included in or omitted from the bitstream 121. In this way, the inclusion or omission of coefficient groups forms a means of rate control or quality scalability for the system 100. The use of larger groups results in less coding overhead due to the use of fewer sub-pictures, but at the expense of coarser granularity of the rate control. The group containing the coefficients of the first basis vectors (i.e. the basis vectors with the largest amount of explained variance) may be relatively large and always encoded, thereby providing a fixed minimum quality level. The boundaries between slices corresponding to basis vectors with smaller amounts of explained variance may be set closer together, resulting in smaller groups, and therefore more groups are required, until all coefficients of all basis vectors can be encoded in the bitstream 121. Regardless of the number of groups, the total number of coefficients does not change, as this is set by the number of basis vectors within the combined tensor 115 and the feature map size. In addition to creating gaps for packing the coefficients in each group into separate sub-pictures (as discussed below with respect to Figure 14A and Figure 14B (Aside from the above, the coding area for the resulting packed frame data is substantially unaffected by the division of the coefficients into groups. Typically, the first group contains the relatively large number of coefficients necessary to provide the minimum fidelity for reconstructing the tensor to give a suitable level of performance. Subsequent sub-pictures are then small in size, resulting in a small number of coefficients per sub-picture and, therefore, fine granularity in rate control. The final sub-picture size is larger due to the need to occupy the remaining area in the rectangular picture 1400.

[0115] Effectively, step 1650 operates to generate coefficients for a tensor using a plurality of basis vectors, the coefficients forming a feature map for each of the plurality of basis vectors. Upon determining the set of groups, step 16100 operates to generate an allocation that groups the coefficients according to the corresponding basis vectors. As described below, the allocation can be adjusted. The allocation can be based at least in part on the fidelity of the reconstructed tensor (i.e., using coefficients 670). The coefficients 670 are based on the reconstructed feature frames generated in the video encoders 614, 636, 654a, 654b, and 654c. Here, "reconstructed" indicates that the feature frames include artifacts resulting from the quantization stage and the forward transform stage. Since entropy coding is lossless, the coefficients 670 do not need to be generated by entropy decoding the bitstream 121. From step 16100, control in the processor 205 passes to step 16110 of encoding the coefficient groups into sub-pictures.

[0116] At step 16110, each group of resulting grouped coefficients 652 is provided to a sub-picture encoder (e.g., a corresponding one of encoders 654a, 654b, and 654c) such that each group is processed by a separate sub-picture encoder, where sufficient instances of the sub-picture encoder are used to partition the coefficients 648 into groups 652. Each sub-picture encoder generates a bitstream portion, resulting in a set of bitstream portions 656. The encoder 600 needs to select which bitstream portions within the portion 656 to include in the bitstream 121. As described in the remaining steps of method 1600, this selection can be based on testing the fidelity of reconstructed tensors having varying numbers of bitstream portions or groups. From this step, control in the processor 205 passes to a select top N groups of coefficients step 16120.

[0117] At step 16120, the processor 205 selects the first N groups in the group 652 so that the group with the basis vector having the largest explained variance is included and these groups, when grouped sequentially based on the basis vector (and therefore explained variance), form a continuous set of groups. Initially, the value N can be set to 1, i.e., the first group is selected. Alternatively, the value N can be set to the value last selected when processing the previous frame in the previous call of method 1600. From step 16120, control in the processor 205 enters a reconstruct tensor step 16130.

[0118] At step 16130, the merge group module 664, the inverse quantizer module 668, and the dot product module 672, under execution by the processor 205, operate to generate a reconstructed tensor 674. The reconstructed tensor 674 is a version of the tensor 637 that accounts for the losses caused by quantization from the floating point domain to the integer sample domain (i.e., from modules 612, 632, and 646), the losses from the forward transform modules 826 and 830 in the video encoder 800, and the sample domain quantization from the quantizer module 834. Since the operation of the entropy encoder 838 is lossless, there is no further difference between the reconstructed tensor 674 and the tensor 942 derived in the PCA decoder 170. The reconstructed tensor 674 is generated for the coefficient group selected at step 16120 using the reconstructed coefficients 662. The reconstructed tensor 674 can be referred to as a "projection" of the basis vectors and the coefficients because the tensor 674 occupies a higher dimensional subspace than the subspace occupied by the coefficients. Merge group module 664 merges the coefficient groups to produce a single coefficient tensor. In other words, merge group module 664 concatenates the coefficient groups along the channel dimension, which correspond to basis vectors in the transform domain, to produce reconstructed integer coefficients 666. Reconstructed coefficients 666 are inversely quantized from the integer domain to the floating-point domain using the quantization range determined by quantizer module 646 to produce reconstructed coefficients 670. Dot product module 672 is used to generate a reconstructed tensor 674 by performing a dot product operation on the reconstructed coefficients 670 and the reconstructed basis vectors 640. The results of dot product module 672 are summed between coefficient groups. When a coefficient group is added, a difference in the reconstructed tensor 674 can be generated by performing a dot product 672 using only the coefficients in the additional group and the basis vectors, and the difference is added to the previously determined reconstructed tensor 674 to produce an updated reconstructed tensor 674. From step 16130, control in processor 205 proceeds to measure error step 16140.

[0119] At step 16140, the mean square error (MSE) module 676 generates an MSE result using the zero-centered tensor 623 and the reconstructed tensor 674 as inputs. The MSE result provides an indication of the fidelity of the reconstructed tensor 674. The higher fidelity of the reconstructed tensor 674 results in a lower loss introduced by feature compression in the system 100 relative to the performance achieved by a network embodied in the system 100 that does not require splitting into parts, but requires encoding more bitstream parts and, therefore, requires higher bandwidth in the communication channel 130. From step 16140, control in the processor 205 passes to an error threshold test step 16150.

[0120] At step 16150, the processor 205 compares the MSE result from step 16140 to a threshold. The threshold may be a predetermined value or a running average of previously encountered MSE results. If the MSE result is below the threshold and the number of currently selected groups is less than all available groups, step 1650 returns "yes" and control in the processor 205 passes from step 16150 to an adjust N step 16160. Otherwise, step 1650 returns "no" and control in the processor 205 passes from step 16150 to a select N sub-pictures step 16170.

[0121] Steps 16130 to 16150 operate to encode and decode the coefficients such that each group of coefficients is independently encoded and decoded. The encoding and decoding are used at step 16150 to determine whether to adjust the allocation determined at step 16100. As described with respect to modules 646 to 668, the allocation or grouping of coefficients can be based on a distance metric (difference) between tensor 623 and tensor 674 (such as the sum of absolute differences (SAD) or mean squared error (MSE)), which is generated from a dot product operation on the decoded or reconstructed coefficients.

[0122] At step 16160, the number of selected groups is increased by one so that an additional group is selected, the next group of the currently unselected groups containing coefficients having a greater explained variance. Thus, steps 16100 to 16160 can be considered as first generating an assignment that groups the coefficients according to their respective basis vectors, and then selecting which subset of the encoded coefficient groups to include in bitstream 121. From step 16160, control in processor 205 proceeds to step 16130.

[0123] At step 16170, bitstream portions of bitstream portions 656 are selected by gate module 658 so that only bitstream portions corresponding to the first N groups of sub-pictures as determined at step 16120 are selected. The selected bitstream portions are propagated by gate module 658 to form selected bitstream portions 660. For sub-pictures that are not selected, "gaps" or uncoded areas in picture 1400 may result, or the sub-pictures may be replaced with bitstream portions having content that is compactly represented and consumes negligible bit rate (such as DC midtone values). From step 16170, control in processor 205 passes to a coded coefficient grouping step 16180.

[0124] At step 16180, processor 205 encodes into the bitstream signaling indicating the grouping and encoding of the coefficients. In other words, the coefficients of each tensor are encoded into the bitstream for each frame. When encoding an IDR picture, signaling indicating the division of the coefficients into groups is encoded. For all frames, signaling indicating which groups are present in the bitstream 121 is encoded. For example, a value N indicating the use of the first N groups is encoded into the bitstream, or a bitmap indicating the use of specific groups is encoded into the bitstream.

[0125] Step 16170 operates to encode the sub-picture arrangement described by the allocation produced by steps 16100 to 16160, and step 16180 operates to encode the allocation into a bitstream. Figure 15 As described above, when the change in allocation has been determined, a new sub-picture structure is started. From step 16180 , control in processor 205 passes to a write bitstream step 16190 .

[0126] At step 16190, sub-picture bitstream combiner 680 combines bitstream portions 615, 637, and 660 to produce (or "write") a single bitstream 121. Upon completion of step 16190, method 1600 terminates for the current frame 113 from video source 112.

[0127] Figure 8 8 is a schematic block diagram 800 illustrating the functional modules of the video encoder 714. The video encoder 714 encodes one sub-picture from a set of sub-pictures that comprise the overall picture. Although all sub-pictures can be encoded in a single pass, using a single pass prevents a lossy decoded version of the data within a given sub-picture from being used as input when generating data to be encoded in another sub-picture, and therefore the PCA encoder 160 cannot account for lossy encoding in the pipeline. Typically, data is passed between functional modules within the video encoder 714 in groups of samples or coefficients (such as partitioning of a block into fixed-size sub-blocks, etc.) or as arrays. Figure 2A and Figure 2BAs shown, the video encoder 714 can be implemented using a general-purpose computer system 200, wherein various functional modules can be implemented using dedicated hardware within the computer system 200, using software executable within the computer system 200 (such as one or more software code modules of the software application 233 residing on the hard drive 205 and whose execution is controlled by the processor 205), etc. Alternatively, the video encoder 714 can be implemented using a combination of dedicated hardware and software executable within the computer system 200. The video encoder 714 and the described method can alternatively be implemented in dedicated hardware, such as one or more integrated circuits that perform the functions or sub-functions of the described method. Such dedicated hardware can include a graphics processing unit (GPU), a digital signal processor (DSP), an application-specific standard product (ASSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or one or more microprocessors and associated memory. In particular, the video encoder 714 includes modules 810 to 890, wherein each of these modules can be implemented as one or more software code modules of the software application 233.

[0128] although Figure 7The video encoder 714 is an example of a Versatile Video Coding (VVC) video encoder, but other video codecs may also be used to perform the processing stages described herein. For example, HEVC may be used. The described example generates a bitstream of encoded data. If other codecs are used, some implementations may package the data into a different format, such as a frame format. The video encoder 714 receives sub-picture frame data 712, such as a series of frames of a sub-picture, each frame including one or more color channels. The frame data 712 can be in any chroma format and bit depth supported by the profile being used, for example, 4:0:0, 4:2:0 of the "Main 10" profile of the VVC standard, with a sample precision of eight (8) to ten (10) bits. The block partitioner 810 first partitions the frame data 712 into CTUs, which are typically square in shape and configured so that a specific size of CTU is used. The maximum effective size of a CTU can be, for example, 32×32, 64×64, or 128×128 luma samples, configured by the "sps_log2_ctu_size_minus5" syntax element present in the "sequence parameter set". The CTU size also provides the maximum CU size, since a CTU without further splitting will contain one CU. The block partitioner 810 also partitions each CTU into one or more CBs based on the luma coding tree and the chroma coding tree. The luma channel may also be referred to as the primary color channel. The chroma channels may also be referred to as secondary color channels. CBs have various sizes and may include both square and non-square aspect ratios. However, in the VVC standard, CBs, CUs, PUs, and TUs always have side lengths that are powers of two. Therefore, based on the luma coding tree and the chroma coding tree of the CTU, the current CB denoted as 812 is output from the block partitioner 810 (advancing based on iterations over one or more blocks of the CTU).

[0129] The CTUs resulting from the first partitioning of the frame data 712 may be scanned in raster scan order and may be grouped into one or more "slices". A slice may be an "intra" (or "I") slice. An intra slice (I slice) indicates that each CU in the slice is intra predicted. Typically, the first picture in a coding layer video sequence (CLVS) contains only I slices and is referred to as an "intra picture". A CLVS may contain periodic intra pictures that form "random access points" (i.e., intermediate frames in a video sequence from which decoding may begin). Alternatively, a slice may be uni-predicted or bi-predicted ("P" or "B" slices, respectively), indicating the additional availability of uni-prediction and bi-prediction, respectively, in a slice.

[0130] The video encoder 714 encodes a sequence of pictures according to a picture structure. One picture structure is "low delay," in which pictures using inter-frame prediction may only reference pictures that appear earlier in the sequence. Low delay allows each picture to be output as soon as it is decoded and also stored for possible reference by subsequent pictures. Another picture structure is "random access," in which the order in which pictures are encoded differs from the order in which they are displayed. Random access allows inter-frame predicted pictures to reference other pictures that have been decoded but not yet output. Some degree of picture buffering is required so that future reference pictures in display order are present in the decoded picture buffer, resulting in a multi-frame delay.

[0131] When using a chroma format other than 4:0:0, in an I slice, the coding tree for each CTU can diverge below the 64×64 level into two separate coding trees, one for luma and the other for chroma. The use of separate trees allows different block structures to exist between luma and chroma within the luma 64×64 region of the CTU. For example, a large chroma CB can be collocated with many smaller luma CBs, and vice versa. In a P or B slice, a single coding tree for a CTU defines a block structure that is common to both luma and chroma. The resulting blocks of a single tree can be intra-predicted or inter-predicted.

[0132] In addition to partitioning pictures into slices, pictures can also be partitioned into "tiles." A tile is a sequence of CTUs covering a rectangular area of the picture. CTU scanning is performed within each tile in a raster scan, progressing from one tile to the next. A slice can be an integer number of tiles, or an integer number of consecutive CTU rows within a given tile.

[0133] For each CTU, the video encoder 714 operates in two phases. In the first phase (referred to as the "search" phase), the block partitioner 810 tests various potential configurations of the coding tree. Each potential configuration of the coding tree has an associated "candidate" coding block (CB). The first phase involves testing various candidate CBs to select a CB that provides relatively high compression efficiency and relatively low distortion. Testing typically involves Lagrangian optimization, whereby candidate CBs are evaluated based on a weighted combination of rate (i.e., coding cost) and distortion (i.e., error relative to the input frame data 712). The "best" candidate CB (i.e., the CB with the lowest estimated rate / distortion) is selected for subsequent encoding into the bitstream portion 716. Included in the evaluation of candidate CBs is the option of using a CB for a given region, further splitting the region according to various splitting options and encoding each smaller resulting region with further CBs, or even further splitting the region. Thus, both the coding tree and the CBs themselves are selected in the search phase.

[0134] Video encoder 714 generates a prediction block (PB) indicated by arrow 820 for each CB (e.g., CB 812). PB 820 is a prediction of the contents of the associated CB 712. Subtractor module 822 generates a difference (or "residual," meaning the difference is in the spatial domain) between PB 820 and CB 812, indicated as 824. Difference 824 is the block-sized difference between corresponding samples in PB 820 and CB 812. Difference 824 is transformed, quantized, and represented as a transform block (TB) indicated by arrow 836. PB 820 and associated TB 836 are typically selected from one of a plurality of possible candidate CBs, for example, based on an assessed cost or distortion.

[0135] A candidate coding block (CB) is a CB obtained from one of the prediction modes available to the video encoder 714 for the associated PB and the resulting residual. When combined with the predicted PB in the video encoder 714, the TB 836 reduces the difference between the decoded CB and the original CB 812 at the expense of additional signaling in the bitstream.

[0136] Thus, each candidate coding block (CB) (i.e., a prediction block (PB) in combination with a transform block (TB)) has an associated coding cost (or "rate") and an associated difference (or "distortion"). The distortion of the CB is typically estimated as a difference in sample values, such as the sum of absolute differences (SAD), the sum of squared differences (SSD), or a Hadamard transform applied to the difference. The mode selector 886 uses the difference 824 to determine the estimates obtained from each candidate PB to determine a prediction mode 887. The prediction mode 887 indicates the decision to use a particular prediction mode (e.g., intra prediction or inter prediction) for the current CB. The estimation of the coding cost associated with each candidate prediction mode and the corresponding residual encoding can be performed at a significantly lower cost than entropy encoding of the residual. Thus, even in a real-time video encoder, multiple candidate modes can be evaluated to determine the best mode in terms of rate-distortion.

[0137] Determining the best mode based on rate-distortion is typically accomplished using a variation of Lagrangian optimization. Lagrangian or similar optimization processes can be employed to select both the optimal partitioning of the CTU into CBs (using the block partitioner 810) and the selection of the best prediction mode from multiple possibilities. By applying a Lagrangian optimization process of the candidate modes in the mode selector module 886, the intra-prediction mode with the lowest cost measure is selected as the "best" mode. The lowest cost mode includes the selected secondary transform index 888, which is also encoded into the bitstream 716 by the entropy encoder 838.

[0138] In the second phase of the operation of the video encoder 714 (referred to as the "encoding" phase), the determined coding tree(s) for each CTU are iterated in the video encoder 714. For CTUs using separate trees, the luma coding tree is encoded first, followed by the chroma coding tree, for each 64x64 luma region of the CTU. Within the luma coding tree, only the luma CBs are encoded, and within the chroma coding tree, only the chroma CBs are encoded. For CTUs using shared trees, a single tree describes the CU (i.e., luma and chroma CBs) according to the common block structure of the shared tree.

[0139] The entropy encoder 838 supports bitwise encoding of syntax elements using variable length and fixed length codewords, as well as arithmetic coding modes of syntax elements. Portions of the bitstream, such as "parameter sets" (e.g., sequence parameter sets (SPS) and picture parameter sets (PPS)), use a combination of fixed length codewords and variable length codewords. Slices (also called continuous portions) have a slice header that uses variable length encoding, followed by slice data that uses arithmetic encoding. The slice header defines parameters specific to the current slice, such as slice-level quantization parameter offsets. The slice data includes syntax elements for each CTU in the slice. The use of variable length coding and arithmetic coding requires sequential parsing within each portion of the bitstream. These portions can be described with start codes to form "network abstraction layer units" or "NAL units." Arithmetic coding is supported using a context-adaptive binary arithmetic coding process.

[0140] The arithmetically coded syntax elements consist of a sequence of one or more "bins" (binary files). Like bits, bin values are "0" or "1." However, bins are not encoded as discrete bits in the bitstream portion 716. Bins have associated predicted (or "likely" or "maximum probability") values and associated probabilities (known as "context"). When the actual bin to be encoded matches the predicted value, a "maximum probability symbol" (MPS) is encoded. Encoding the maximum probability symbol is relatively cheap in terms of consumed bits in the bitstream portion 716, including a total cost of less than one discrete bit. When the actual bin to be encoded does not match the possible value, a "minimum probability symbol" (LPS) is encoded. Encoding the minimum probability symbol has a relatively high cost in terms of consumed bits. Bin encoding techniques enable efficient encoding of bins that skew the probabilities of "0" vs. "1." For syntax elements with two possible values (i.e., "flags"), a single bin is sufficient. For syntax elements with many possible values, a sequence of bins is required.

[0141] The presence of a later bin in a sequence can be determined based on the value of an earlier bin in the sequence. In addition, each bin can be associated with more than one context. The selection of a particular context can depend on, among other things, the bin values of earlier bins in the syntax element, as well as those of neighboring syntax elements (i.e., neighboring syntax elements from neighboring blocks). Each time a context coding bin is encoded, the context selected for that bin (if any) is updated to reflect the new bin value. Therefore, the binary arithmetic coding scheme is said to be adaptive.

[0142] The entropy encoder 838 also supports bins that lack context (referred to as "bypass bins"). Bypass bins are encoded assuming an equal probability distribution between "0" and "1". Thus, each bin has an encoding cost of one bit in the bitstream portion 716. The absence of context saves memory and reduces complexity, so bypass bins are used where the distribution of values for a particular bin is not skewed. One example of an entropy encoder that employs context and adaptation is known in the art as CABAC (Context Adaptive Binary Arithmetic Coder), and many variations of this encoder have been employed in video coding.

[0143] The QP controller 890 determines a quantization parameter 892 that is used to establish a quantization step size for use by the quantizer 834 and the dequantizer 840. A larger quantization step size causes the main transform coefficients 828 to be quantized to smaller values, thereby reducing the bit rate of the bitstream portion 716 at the expense of reducing the fidelity of the inverse transform coefficients 846.

[0144] The entropy encoder 838 encodes a quantization parameter 892 and, if applicable, an LFNST index 888 for the current CB, using a combination of context-encoded and bypass-encoded bins. The quantization parameter 892 is encoded at the beginning of each slice, and changes to the quantization parameter 892 within a slice are encoded using the "delta QP" syntax element. The delta QP syntax element is signaled at most once in each region known as a "quantization group." The quantization parameter 892 is applied to the residual coefficients of the luma CB. An adjusted quantization parameter is applied to the residual coefficients of the collocated chroma CBs. The adjusted quantization parameter may include a mapping from the luma quantization parameter 892 according to a mapping table and a CU-level offset selected from an offset list. The secondary transform index 888 is signaled when the residual associated with the transform block includes significant residual coefficients only in those coefficient positions that were transformed into primary coefficients by applying the secondary transform.

[0145] The residual coefficients of each TB associated with a CB are encoded using a residual syntax. The residual syntax is designed to efficiently encode coefficients with low amplitudes, using primarily arithmetic coded bins to indicate the significance of the coefficient and the amplitude for lower values, and reserving bypass bins for residual coefficients with higher amplitudes. As a result, residual blocks consisting of very low amplitude values and sparsely placed significant coefficients are compressed efficiently. In addition, there are two residual coding schemes. As seen when the transform is applied, the conventional residual coding scheme is optimized for TBs where the significant coefficients are primarily located in the upper left corner of the TB. The transform-skipped residual coding scheme can be used for TBs that are not transformed and is able to efficiently encode residual coefficients regardless of their distribution throughout the TB.

[0146] A multiplexer module 884 outputs a PB 820 from the intra prediction module 864 based on the determined best intra prediction mode selected from the test prediction modes for each candidate CB. The candidate prediction modes need not include every conceivable prediction mode supported by the video encoder 714. Intra prediction is of three types: first, "DC intra prediction," which involves filling the PB with a single value representing the average of nearby reconstructed samples; second, "planar intra prediction," which involves filling the PB with samples based on a plane, using a DC offset and vertical and horizontal gradients derived from nearby reconstructed neighboring samples. The nearby reconstructed samples typically include a row of reconstructed samples above the current PB (extending to a certain extent to the right of the PB) and a column of reconstructed samples to the left of the current PB (extending to a certain extent downward beyond the PB); and third, "angular intra prediction," which involves filling the PB with reconstructed neighboring samples filtered and propagated across the PB in a particular direction (or "angle"). In VVC, sixty-five (65) angles are supported, with rectangular blocks being able to utilize additional angles not available to square blocks, for a total of eighty-seven (87) angles.

[0147] A fourth type of intra prediction can be used for chroma PBs, whereby the PBs are generated from collocated luma reconstructed samples according to a "cross-component linear model" (CCLM) mode. Three different CCLM modes are available, each using a different model derived from neighboring luma and chroma samples. The derived model is used to generate a block of samples for the chroma PBs from the collocated luma samples. Matrix multiplication of reference samples can be used to intra predict luma blocks using a matrix selected from a predefined set of matrices. This matrix intra prediction (MIP) achieves gains by using matrices trained on a large set of video data, where the matrices represent the relationship between reference samples and the predicted block that is not easily captured in angular, planar, or DC intra prediction modes.

[0148] Module 864 can also generate prediction units by copying blocks from nearby the current frame using an "intra-block copy" (IBC) method. The location of the reference block is constrained to an area equivalent to one CTU, which is divided into 64×64 regions called VPDUs. Regardless of the configured CTU size for the bitstream, this area covers the processed VPDU of the current CTU and the VPDUs of one or more previous CTUs within each row or CTU and each slice or tile, up to the limit of an area corresponding to one 128×128 luma sample. This area is called the "IBC virtual buffer" and limits the IBC reference area, thereby limiting the required storage. The IBC buffer is filled with reconstructed samples 854 (i.e., before loop filtering), requiring a separate buffer from the frame buffer 872. When the CTU size is 128×128, the virtual buffer includes samples only from the CTU adjacent to the left of the current CTU. When the CTU size is 32×32 or 64×64, the virtual buffer includes CTUs from up to four or sixteen CTUs to the left of the current CTU. Regardless of the CTU size, accessing neighboring CTUs to obtain samples of the IBC reference block is constrained by boundaries (such as edges of pictures, slices, or tiles). In particular, for feature maps of FPN layers with smaller sizes, using CTU sizes such as 32×32 or 64×64 results in more aligned reference areas to cover the set of previous feature maps. Accessing similar feature maps for IBC prediction provides the advantage of efficient coding when feature map placement is sorted based on SAD, SSE, or other difference metrics.

[0149] The residuals of the prediction blocks when encoding feature map data are different from the residuals seen for natural video. Natural video is typically captured by an image sensor or screen content, as commonly seen in operating system user interfaces, etc. Feature map residuals tend to contain a lot of detail. The level of detail in feature map residuals is more suitable for transform skip coding than the significant low frequency coefficients of various transforms. The intra-predicted luma coding block can be partitioned vertically or horizontally into a set of equally sized prediction blocks, each having a minimum area of sixteen (16) luma samples.

[0150] In cases where previously reconstructed neighboring samples are not available, such as at the edges of a frame, a default halftone value of half the sample range is used. For example, for 10-bit video, a value of five hundred and twelve (512) is used. Since no previous samples are available for the CB located at the top left of the frame, the angular and planar intra prediction modes produce the same output as the DC prediction mode (i.e., a flat surface of samples with halftone values as amplitudes).

[0151] For inter prediction, a prediction block 882 is generated by the motion compensation module 880 using samples from one or two frames preceding the current frame in the coding order frames in the bitstream, and the prediction block 882 is output by the multiplexer module 884 as PB820. In addition, for inter prediction, a single coding tree is typically used for both the luma channel and the chroma channel. The order of the coded frames in the bitstream may be different from the order of the frames when captured or displayed. When one frame is used for prediction, the block is called "uni-predicted" and has one associated motion vector. When two frames are used for prediction, the block is called "bi-predicted" and has two associated motion vectors. For P slices, each CU can be intra-predicted or uni-predicted. For B slices, each CU can be intra-predicted, uni-predicted, or bi-predicted.

[0152] Frames are typically encoded using a "group of pictures" structure, enabling a temporal hierarchy. Frames can be partitioned into multiple slices, each encoding a portion of the frame. This temporal hierarchy allows frames to reference previous and subsequent pictures in the order in which they are displayed. Images are encoded in the necessary order to ensure that dependencies are met for decoding each frame. An affine inter-frame prediction mode is available. Instead of using one or two motion vectors to select and filter the reference sample blocks for a prediction unit (PU), the PU is partitioned into multiple smaller blocks, and a motion field is generated, with each smaller block having a different motion vector. The motion field uses the motion vectors of nearby points in the PU as "control points." Affine prediction allows encoding of motion other than translation, reducing the need for depth-split coding trees. The bi-prediction mode available with VVC performs a geometric blend of two reference blocks along a selected axis, signaling the angle and offset relative to the block center. This geometric partitioning mode ("GPM") allows the use of larger CUs along the boundary between two objects, with the geometry of the boundary being used for encoding the CU as the angle and center offset. Motion vector differences can be encoded as direction (up / down / left / right) and distance (a set of power-of-two distances is supported) instead of using Cartesian (x,y) offsets. Motion vector predictors are obtained from neighboring blocks ("merge mode") as if no offset was applied. The current block will share the same motion vector as the selected neighboring block.

[0153] Samples are selected based on the motion vector 878 and the reference picture index. The motion vector 878 and the reference picture index apply to all color channels and therefore describe inter prediction primarily in terms of operations on PUs rather than PBs. A single coding tree is used to describe the decomposition of each CTU into one or more inter prediction blocks. Inter prediction methods can vary in the number of motion parameters and their precision. The motion parameters typically include a reference frame index that indicates which reference frame(s) from a reference frame list will be used plus a spatial translation of each reference frame, but can include more frames, specific frames, or complex affine parameters such as scaling and rotation. In addition, a predetermined motion refinement process can be applied to generate a dense motion estimate based on the reference sample block.

[0154] A PB 820 has been determined and selected, and is subtracted from the original sample block at a subtractor 822 to obtain a residual with the lowest coding cost (denoted as 824), which is then lossily compressed. The lossy compression process includes the steps of transform, quantization, and entropy coding. A forward main transform module 826 applies a forward transform to the difference 824, converting the difference 824 from the spatial domain to the frequency domain and producing the main transform coefficients indicated by arrow 828. The maximum main transform size in one dimension is a 32-point DCT-2 or 64-point DCT-2 transform, as configured by the "sps_max_luma_transform_size_64_flag" in the sequence parameter set. If the CB being encoded is larger than the maximum supported main transform size (e.g., 64×64 or 32×32) indicated as the block size, the main transform 826 is applied in a tiled manner to transform all samples of the difference 824. In the case of a non-square CB, tiling is also performed using the maximum available transform size in each dimension of the CB. For example, when using a maximum transform size of thirty-two (32), a 64x16 CB uses two 32x16 primary transforms arranged in a tiled manner. When the size of the CB is larger than the maximum supported transform size, the CB is padded with TBs in a tiled manner. For example, a 128x128 CB with a 64-pt transform maximum size is padded with four 64x64 TBs in a 2x2 arrangement. A 64x128 CB with a 32-pt maximum size is padded with eight 32x32 TBs in a 2x4 arrangement.

[0155] The application of transform 826 results in multiple TBs for the CB. When each application of the transform operates on a TB greater than 32×32 (e.g., 64×64) in size, all resulting main transform coefficients 828 outside the upper left 32×32 region of the TB are set to zero (i.e., discarded). The remaining main transform coefficients 828 are passed to a quantizer module 834. The main transform coefficients 828 are quantized according to quantization parameters 892 associated with the CB to produce main transform coefficients 832. In addition to quantization parameters 892, quantizer module 834 may also apply a "scaling list" to allow for non-uniform quantization within the TB by further scaling the residual coefficients according to their spatial location within the TB. Quantization parameters 892 may be different for the luma CB and each chroma CB. The main transform coefficients 832 are passed to a forward secondary transform module 830 to produce transform coefficients indicated by arrow 836 by performing a non-separable secondary transform (NSST) operation or bypassing the secondary transform. The forward main transform 826 is typically separable, transforming a set of rows and then a set of columns of each TB. For luma TBs with a width and height not exceeding 16 samples, the forward main transform module 826 uses a type II discrete cosine transform (DCT-2) in the horizontal and vertical directions, or a bypass of the transform in the horizontal and vertical directions, or a combination of a type VII discrete sine transform (DST-7) and a type VIII discrete cosine transform (DCT-8) in the horizontal or vertical directions. In the VVC standard, the use of a combination of DST-7 and DCT-8 is called a "Multiple Transform Selection Set" (MTS).

[0156] The forward secondary transform of module 830 is typically a non-separable transform that is applied only to the residual of the intra-frame predicted CU and can nevertheless be bypassed. The forward secondary transform operates on sixteen (16) samples (arranged as the upper left 4×4 sub-block of the main transform coefficients 828) or forty-eight (48) samples (arranged as three 4×4 sub-blocks of the upper left 8×8 coefficients of the main transform coefficients 828) to produce a set of secondary transform coefficients. The set of secondary transform coefficients can be fewer in number than the set of main transform coefficients from which they are derived. Since the secondary transform is only applied to a set of coefficients that are adjacent to each other and include the DC coefficient, the secondary transform is called a "low frequency non-separable secondary transform" (LFNST). This secondary transform can be obtained through a training process and, due to its non-separable nature and trained origin, can exploit additional redundancy in the residual signal that cannot be captured by separable transforms (such as variants of DCT and DST). In addition, when LFNST is applied, all remaining coefficients in the TB are zero in both the primary transform domain and the secondary transform domain.

[0157] Quantization parameter 892 is constant for a given TB and therefore results in uniform scaling of the residual coefficients generated in the main transform domain for that TB. Quantization parameter 892 can be periodically varied with a signaled "delta quantization parameter." The delta quantization parameter (delta QP) is signaled once for each CU contained within a given region (referred to as a "quantization group"). If the CU is larger than the quantization group size, the delta QP is signaled once using one of the CU's TBs. That is, the delta QP is signaled once by entropy encoder 838 for the first quantization group of the CU and not for any subsequent quantization groups of the CU. Non-uniform scaling can also be achieved by applying a "quantization matrix," whereby the scaling factor applied to each residual coefficient is derived from a combination of quantization parameter 892 and the corresponding entry in the scaling matrix. The scaling matrix can have a size smaller than the size of the TB, and when applied to a TB, a nearest neighbor approach is used to provide a scaled value for each residual coefficient from a scaling matrix smaller than the TB size. The residual coefficients 836 are supplied to an entropy encoder 838 for encoding in the bitstream portion 716. Typically, the residual coefficients of each TB of a TU having at least one significant residual coefficient are scanned according to a scan pattern to produce an ordered list of values. The scan pattern typically scans the TB into a sequence of 4x4 "sub-blocks," providing a conventional scan operation with a granularity of 4x4 sets of residual coefficients, where the arrangement of the sub-blocks depends on the size of the TB. The scanning within each sub-block and the progression from one sub-block to the next typically follows a reverse diagonal scan pattern. Additionally, a quantization parameter 892 is encoded in the bitstream portion 716 using a delta QP syntax element, and a slice QP and a secondary transform index 888 for an initial value in a given slice or sub-picture are encoded in the bitstream portion 716.

[0158] As described above, the video encoder 714 requires access to a frame representation corresponding to the decoded frame representation seen by the video decoder. Therefore, the residual coefficients 836 are passed through an inverse secondary transform module 844, which operates according to a secondary transform index 888 to produce intermediate inverse transform coefficients, indicated by arrow 842. The intermediate inverse transform coefficients 842 are inversely quantized by a dequantizer module 840 according to a quantization parameter 892 to produce inverse transform coefficients, indicated by arrow 846. The dequantizer module 840 may also perform inverse non-uniform scaling of the residual coefficients using a scaling list, corresponding to the forward scaling performed in the quantizer module 834. The inverse transform coefficients 846 are passed to an inverse main transform module 848 to produce residual samples for the TU (indicated by arrow 850). The inverse main transform module 848 applies a DCT-2 transform horizontally and vertically, subject to the constraints of the maximum available transform size described with reference to the forward main transform module 826. The type of inverse transform performed by inverse secondary transform module 844 corresponds to the type of forward transform performed by forward secondary transform module 830. The type of inverse transform performed by inverse main transform module 848 corresponds to the type of main transform performed by main transform module 826. Summation module 852 adds residual samples 850 and PU 820 to produce reconstructed samples of the CU (indicated by arrow 854).

[0159] Reconstructed samples 854 are passed to a reference sample cache 856 and an in-loop filter module 868. The reference sample cache 856, typically implemented using static RAM on an ASIC to avoid expensive off-chip memory accesses, provides the minimum sample storage required to satisfy the dependencies used to generate intra PBs for subsequent CUs in the frame. The minimum dependencies typically include a "line buffer" of samples along the bottom of a row of CTUs for use by the next row of CTUs, as well as a column buffer (whose extent is set by the CTU height). The reference sample cache 856 feeds reference samples (indicated by arrow 858) to a reference sample filter 860. The sample filter 860 applies a smoothing operation to produce filtered reference samples (indicated by arrow 862). The filtered reference samples 862 are used by the intra prediction module 864 to generate an intra-prediction block of samples, indicated by arrow 866. For each candidate intra prediction mode, the intra prediction module 864 generates a sample block (i.e., 866). The sample block 866 is generated by the module 864 using techniques such as DC, planar, or angular intra prediction. A matrix multiplication approach may also be used to generate the sample block 866, with neighboring reference samples as input and a matrix selected by the video encoder 814 from a set of matrices, with the selected matrix signaled in the bitstream 716 using an index to identify which matrix in the set of matrices is to be used by the video decoder.

[0160] The in-loop filter module 868 applies several filtering stages to the reconstructed samples 854. The filtering stages include a "deblocking filter" (DBF), which applies smoothing aligned with CU boundaries to reduce artifacts caused by discontinuities. Another filtering stage present in the in-loop filter module 768 is an "adaptive loop filter" (ALF), which applies a Wiener-based adaptive filter to further reduce distortion. Another available filtering stage in the in-loop filter module 868 is a "sample adaptive offset" (SAO) filter. The SAO filter operates by first classifying the reconstructed samples into one or more categories and applying an offset at the sample level according to the assigned category.

[0161] Filtered samples, represented by arrow 870, are output from the in-loop filter module 868. The filtered samples 870 are stored in a frame buffer 872. The frame buffer 872 typically has the capacity to store several (e.g., up to sixteen (16)) pictures and is therefore stored in the memory 206. Due to the large memory consumption required, on-chip memory is typically not used to store the frame buffer 872. As such, access to the frame buffer 872 is expensive in terms of memory bandwidth. The frame buffer 872 provides a reference frame, represented by arrow 874, to the motion estimation module 876 and the motion compensation module 880. The reference frame 874 is output as a reconstructed frame 718 for the corresponding sub-picture encoder module 700 (614, 636, 654a, 654b, 654c) and is provided to the unpacker module 720. In Figure 8 In the example of , the reconstructed frame is a result of the operation of lossy VVC encoding due to the operations of modules 810 to 890.

[0162] Motion estimation module 876 estimates multiple "motion vectors" (designated 878), each of which is a Cartesian spatial offset relative to the position of the current CB, referencing a block in one of the reference frames in frame buffer 872. A filtered block of reference samples (designated 882) is generated for each motion vector. Filtered reference samples 882 form a further candidate mode for potential selection by mode selector 886. Furthermore, for a given CU, PB 820 can be formed using one reference block ("uni-prediction") or two reference blocks ("bi-prediction"). For the selected motion vector, motion compensation module 880 generates PB 820 based on a filtering process that supports sub-pixel precision in the motion vector. Thus, motion estimation module 876 (which operates on many candidate motion vectors) can perform a simplified filtering process compared to motion compensation module 880 (which operates only on the selected candidate), resulting in reduced computational complexity. When video encoder 714 selects inter prediction for a CU, motion vector 878 is encoded in bitstream portion 716.

[0163] Although the reference to Versatile Video Coding (VVC) describes Figure 8 The video encoder 714 is used for encoding, but other video encoding standards or implementations may also use the processing stages of modules 810 to 890. The video encoder 714 may also be used to encode the video from the memory 206, the hard drive 210, the CD-ROM, the Blu-ray Disc TM ) or other computer-readable storage medium (or write to memory 206, hard drive 210, CD-ROM, Blu-ray disc, or other computer-readable storage medium). In addition, frame data 712 (and bitstream 121) can be received from (or sent to) an external source (such as a server or radio frequency receiver connected to communication network 220). Communication network 220 may provide limited bandwidth, thereby necessitating the use of rate control in video encoder 714 to avoid saturating the network when frame data 712 is difficult to compress.

[0164] The bitstream 121 may be constructed from one or more slices representing spatial portions (sets of CTUs) of the frame data 712 produced by one or more instances of the video encoder 714, each of which produces a bitstream portion 716 and operates in a coordinated manner under the control of the processor 205. The bitstream portion 716 may also include a slice corresponding to a picture to be output as a set of sub-pictures forming a picture (each sub-picture is independently encodable and independently decodable with respect to any of the other slices or sub-pictures in the picture). The ability to independently encode and decode any sub-picture in a picture allows the PCA encoder 160 to take into account the effects of lossy compression on the packed feature maps or coefficients contained in any given sub-picture by using lossy versions of the feature maps or coefficients in the later stages of tensor compression.

[0165] Figure 9 is a schematic block diagram 900 illustrating an implementation of the tensor decoder 170 based on inter-channel decorrelation. Figure 11A is a schematic block diagram 1100 illustrating a multi-scale feature reconstruction (MSFR) module. Figure 171700 is a schematic block diagram illustrating a method 1700 for decoding a tensor. The operation of the decoder side of system 100 is described with reference to method 1700. Method 1700 may be implemented using a device such as a configured FPGA, ASIC, or ASSP. Alternatively, as described below, method 1700 may be implemented by destination device 140 as one or more software code modules of application 233 under execution by processor 205. The software code modules of application 233 implementing method 1700 may reside, for example, in hard drive 210 and / or memory 206. Method 1700 is repeated for each frame of compressed data in bitstream 123. Method 1700 may be stored on a computer-readable storage medium and / or in memory 206. Method 1700 provides a means for decoding compressed representations of tensors with quality scalability and accurate modeling of inaccuracies introduced by the use of lossy compression mechanisms. The video bitstream 143 is passed to the picture decoder 904, which implements the VVC video decoder and decodes the bitstream generated by the PCA encoder 160. The picture decoder 904 decodes the sub-pictures present in the video bitstream 143, each sub-picture corresponding to a different data type required to generate the tensor 149. Each decoded sub-picture provides information elements of the decoded tensor, corresponding to the mean channel 611, the basis vector 631, and the coefficients 648, where the coefficients 648 are encoded in multiple groups. The method 1700 begins with a decode coefficient grouping step 1710.

[0166] At step 1710, a picture decoder such as a VVC decoder decodes the bitstream 143 under the control of the processor 205. Figure 9 As shown, the inter-channel decorrelation based tensor decoder 170 includes a picture decoder 904 . Figure 10A VVC decoder 1000 is shown as an example implementation of the picture decoder 904. In the decoder 1000, the entropy decoder 1020, under execution by the processor 205, decodes the coefficient groups from the bitstream 143. The coefficients form a tensor having dimensions of channel, width, and height. The coefficient grouping divides the coefficients into groups along the channel dimension so that when the coefficients are sorted based on the explained variance of the corresponding basis vectors, each group contains a continuous set of coefficients. There is always one group that contains the zeroth channel (the coefficient of the basis vector with the largest explained variance). Between groups, there are no unused channels in the coefficients, that is, the groups can be concatenated along the channel dimension to form a single tensor of coefficients for projection or inverse transformation back into the same subspace as the original tensor. Although the groups are required to contain a continuous set of coefficients along the channel dimension and include the zeroth channel, the least significant channel is not required to be included in any group. For example, if 25 basis vectors are to be used (so the coefficients are enumerated as [0..24]), the group definition can be [0..4], [5..9], [10..14] (omitting the coefficients [15..24] for the basis vectors).

[0167] When decoding the group signaling, step 1710 operates to decode the allocation of mapping the arrangement of independently coded regions of the picture to multiple groups of basis vectors. That is, the grouped s sub-pictures encoded at step 16180 are decoded. Group signaling can be implemented as a list where each value specifies the number of coefficients along the channel dimension. Since the coefficients are tensors with dimensions c, h, w, the channel-wide grouping results in a group containing multiple feature maps, thereby having a width and height for each feature map in the group. Each group in the list starts with the next coefficient after the previous group ends and terminates the list with a zero-sized group or the exhaustion of available coefficients (whichever occurs first). Control in the processor 205 enters the decode mean channel step 1720 from step 1710.

[0168] At step 1720, picture decoder 904, under execution by processor 205, outputs mean channel sub-picture 950, which is passed to unpacker 952. Unpacker 952 extracts integer mean channels 954, where the value C corresponds to the number of channels in tensor 149. Integer mean channels 956 are passed to inverse quantizer 956, where a conversion from the sample domain to the floating-point domain is performed using, for example, an appropriate quantization range obtained from bitstream 143. Operation of inverse quantizer 956 results in decoded mean channels 958. From step 1720, control in processor 205 proceeds to decode basis vectors step 1730.

[0169] At step 1730, the picture decoder 904, under execution by the processor 205, outputs a sub-picture 930 containing packed basis vectors generated by the decomposition module 630 (e.g., by performing a method such as SVD) prior to quantization and lossy compression. In other words, in the example described with a packed sub-picture format, the basis vectors are decoded from the bitstream. This format can vary based on the dimensionality of the basis vectors and the allocation of coefficients into groups. The sub-picture 930 is passed to an unpacker 932. The unpacker 932 extracts the integer basis vectors 934 as a series of arrays placed in a non-overlapping manner within the sub-picture 930. The integer basis vectors 934 are passed to an inverse quantizer 936. The inverse quantizer 936 constructs floating-point basis vectors 938, applying the quantization range obtained from the bitstream 143. The decoded allocation provides an indication of which basis vectors will have corresponding coefficients in the feature frame, as not all sub-pictures may have been selected at step 1670. From step 1730 , control in processor 205 passes to a decode coefficients step 1740 .

[0170] At step 1740, picture decoder 904, under execution by processor 205, outputs coefficient sub-picture 910. Coefficient sub-picture 910 is passed to unpacker 912. Each sub-picture contains one or more feature maps corresponding to a group of coefficients decoded at step 1710. In other words, sub-picture 910 contains coefficients, one coefficient per sample per basis vector of tensor 143. Sub-picture 910 is output by picture decoder 904 and passed to unpacker 912. Unpacker 912 extracts each coefficient from sub-picture 910 based on the dimensions of tensor 149 and the grouping from step 16100, thereby outputting integer coefficient tensor 914. Due to the selection of a subset of groups at step 16170, sub-picture 910 may not contain coefficients corresponding to all basis vectors. When decoding sub-picture 910, it can be considered that a feature frame is being decoded from the bitstream. The feature frame contains independently coded regions (sub-pictures) including regions having coefficients corresponding to one or more basis vectors decoded at step 1730. The regions may also encode basis vectors and mean channel values.

[0171] From step 1740 , control in processor 205 proceeds to a combine coefficients step 1750 .

[0172] As described above, the coefficients form a tensor with dimensions channel, width, and height. In other words, the coefficients are decoded from the bitstream and extracted from the feature frame as feature maps (one feature map for each channel) that form a coefficient tensor. At step 1750, the merge group module 916, under execution by the processor 205, concatenates the array tensor 914 along the channel dimension to produce the integer coefficient tensor 918. From step 1750, control in the processor 205 passes to the inverse quantize coefficients step 1760.

[0173] At step 1760, the inverse quantizer 920, under execution by the processor 205, converts the integer coefficient tensor 918 from the integer domain to the floating-point domain according to the quantization range obtained from the bitstream 143, thereby outputting floating-point coefficients 922. Step 1760 operates to obtain coefficients from the decoded feature frame through the inverse quantization operation. From step 1760, control in the processor 205 passes to a generate tensor step 1770.

[0174] At step 1770, dot product module 940, under execution by processor 205, generates zero-centered tensor 942 by performing a dot product on coefficients 922 and basis vectors 938. Summation module 960 adds zero-centered tensor 942 to mean channel 958 to generate reconstructed combined tensor 149a as output from PCA decoder 170. Completion of step 1770 effectively generates tensor 149a from the decoded coefficient tensor and set of basis vectors at the operations of steps 1710 to 1750. Combined tensor 149a has the same spatial size and a higher number of channels than the coefficient tensor. Tensor 149a can also be considered a projection of the coefficients and basis vectors generated using a dot product operation (e.g., using module 940). From step 1770, control in processor 205 passes to reconstruct tensor step 1780.

[0175] The architecture 1100 includes an MSFR module 1130. The MSFR module operates to generate a plurality of tensors from the tensor 149a generated by performing step 1770 using one or more trained convolutional layers. At step 1780, the MSFR module 1130 generates a reconstructed tensor 149 from the reconstructed combined tensor 149a. Upsampling modules 1132, 1134, and 1136 upsample the tensor 149a horizontally and vertically by factors of 2, 4, and 8, respectively, to generate tensors 1133, 1135, and 1137. Tensor 1137 forms a (P'2) output from the MSFR module 1130 and is passed to a downsampling module 1142. The downsampling module 1142 downsamples the tensor 1137 horizontally and vertically by a factor of 2 to generate a tensor 1143 having the same dimensions as tensor 1135. Tensor 1143 is provided to convolutional layer 1148, which outputs tensor 1149. Summation module 1154 adds tensors 1135 and 1149 to produce tensor 1155 as the output (P'3) of MSFR module 1130.

[0176] Downsampling module 1140 downsamples tensor 1135 horizontally and vertically by a factor of 2 to produce tensor 1141 having the same dimensions as tensor 1133. Tensor 1141 is provided to convolutional layer 1146, which outputs tensor 1147. Summation module 1152 adds tensors 1133 and 1147 to produce tensor 1153 as the output (P'4) of MSFR module 1130.

[0177] Downsampling module 1138 downsamples tensor 1133 horizontally and vertically by a factor of 2 to produce tensor 1139 having the same dimensions as tensor 149a. Tensor 1139 is provided to convolutional layer 1144, which outputs tensor 1145. Summation module 1150 adds tensors 149a and 1145 to produce tensor 1151 as the output (P'5) of MSFR module 1130. Collectively, tensors 1151, 1153, 1155, and 1157 form tensor 149 and provide the decoded P2 through P5 layers. From step 1780, control in processor 205 proceeds to step 1790, which performs the second portion of the neural network.

[0178] In an alternative embodiment of architecture 1100, tensor 149a is input to a convolutional neural network, and the output of the convolutional neural network is input to upsamplers 1132, 1134, and 1136. For example, Figure 11B Tensor 149a is shown being input to a trained convolutional layer 1190 and producing output 149a_conv. In an arrangement using convolutional layer 1190, output 149a_conv is input to upsamplers 1132, 1134, and 1136. Convolutional layer 1190 has fewer output channels than input channels and, as shown, is applied to tensor 149a before multiple tensors are produced by MSFR module 1130. An implementation using convolutional layer 1190 corresponds to an implementation in which convolutional layer 528 is excluded from module 500 on the encoding side. Regardless of whether layer 1190 is used, Figure 11A and Figure 11B The trained convolutional layers implement spatial resizing to recover the hierarchical representation of frames (FPN).

[0179] At step 1790, the CNN head 150 performs the second part of the neural network task under the execution of the processor 205. 12A to 12C and Figure 13 143 to illustrate an example second or header portion 150. The method 1700 terminates processing for the current frame and is invoked again for the next received frame in the bitstream 143.

[0180] In the arrangement of method 1700, sub-pictures of groups not selected for inclusion in the bitstream are decoded as flat regions encoded with neutral values, such as DC midtone values. A quantization range is used that results in a DC midtone value corresponding to an inverse quantized floating point value of 0.0. Method 1700 can process all sub-pictures (including those omitted by replacing them with neutral values) and perform a projection from the basis vector subspace back to the subspace of tensor 115 without knowing which specific groups are used for a given frame. Since sub-pictures containing coefficients encoded with neutral values do not affect the final reconstructed combined tensor 149, it is possible that there is no different treatment for included coefficient groups vs. omitted coefficient groups.

[0181] exist Figure 10 An example implementation 1000 of a picture decoder 904 (also referred to as a video decoder) is shown in FIG. Figure 9 The video decoder 904 is an example of a Versatile Video Coding (VVC) video decoding pipeline, but other video codecs can also be used to perform the processing stages described herein (eg, HEVC, etc.). Figure 10 As shown, the bit stream 143 is input to the video decoder 804. The bit stream 143 can be obtained from the memory 206, the hard drive 210, the CD-ROM, the Blu-ray disk TM ) or other non-transitory computer-readable storage medium and provided to implementation 1000 as a bitstream 143. Alternatively, bitstream 143 can be received from an external source (such as a server connected to communication network 220 or a radio frequency receiver). Bitstream 143 contains encoded syntax elements representing the captured frame data to be decoded. In the case where sub-pictures are decoded independently, the portion of bitstream 143 corresponding to each sub-picture can be provided to a separate instance of implementation 1000. A separate instance of implementation 1000 for each sub-picture allows for parallel decoding of the sub-pictures to increase throughput.

[0182] The bitstream 143 is input to the entropy decoder module 1020. The entropy decoder module 1020 extracts syntax elements from the bitstream 1010 by decoding a sequence of "bins" and passes the values of the syntax elements to other modules in the video decoder 904. The entropy decoder module 1020 uses variable-length and fixed-length decoding to decode the SPS, PPS, or slice header, using an arithmetic decoding engine to decode the syntax elements of the slice data into a sequence of one or more bins. Each bin can use one or more "contexts," where the context describes the probability level of "one" and "zero" values to be used to encode the bin. In the case where multiple contexts are available for a given bin, a "context modeling" or "context selection" step is performed to select one of the available contexts for decoding the bin. The process of decoding the bins forms a sequential feedback loop, whereby each slice can be decoded by a given entropy decoder 1020 instance within the entire slice. A single (or a few) high-performance entropy decoder 1020 instances can decode all slices or sub-pictures of a frame or picture from the bitstream 1010 , and multiple lower-performance entropy decoder 1020 instances can concurrently decode slices of a frame from the bitstream 1010 .

[0183] The entropy decoder module 1020 applies an arithmetic coding algorithm, such as "Context Adaptive Binary Arithmetic Coding" (CABAC), to decode syntax elements from the bitstream 1010. The decoded syntax elements are used to reconstruct parameters within the video decoder 904. The parameters include residual coefficients (represented by arrow 1024), quantization parameters 1074, secondary transform indices 1070, and mode selection information (represented by arrow 1058) such as intra prediction modes. The mode selection information also includes information such as motion vectors and partitioning of each CTU into one or more CBs. The parameters are typically combined with sample data from previously decoded CBs to generate PBs.

[0184] The residual coefficients 1024 are passed to an inverse secondary transform module 1036, where a secondary transform is applied or not applied (bypassed) depending on the secondary transform index. The inverse secondary transform module 1036 produces reconstructed transform coefficients 1032. That is, module 1036 generates primary transform domain coefficients from the secondary transform domain coefficients. The reconstructed transform coefficients 1032 are input to a dequantizer module 1028. The dequantizer module 1028 inverse quantizes (or "scales") the residual coefficients 1032 (i.e., in the primary transform coefficient domain) to create reconstructed intermediate transform coefficients, represented by arrow 1040, according to quantization parameters 1074. The dequantizer module 1028 can also apply a scaling matrix to provide non-uniform dequantization within the TB, corresponding to the operation of the dequantizer module 840. If the use of a non-uniform inverse quantization matrix is indicated in the bitstream 143, the video decoder 904 reads the quantization matrix from the bitstream 143 as a sequence of scale factors and arranges the scale factors into a matrix. Inverse scaling uses the quantization matrix in combination with the quantization parameters to create reconstructed intermediate transform coefficients 1040 .

[0185] The reconstructed transform coefficients 1040 are passed to an inverse main transform module 1044. Module 1044 transforms the coefficients 1040 from the frequency domain back to the spatial domain. Inverse main transform module 1044 applies an inverse DCT-2 transform horizontally and vertically, subject to the constraints of the maximum available transform size as described with reference to forward main transform module 826. The result of the operation of module 1044 is a block of residual samples, indicated by arrow 1048. Residual sample block 1048 is equal in size to the corresponding CB. Residual samples 1048 are supplied to summation module 1050.

[0186] At summation module 1050, residual samples 1048 are added to the decoded PB (denoted as 1052) to produce a block of reconstructed samples represented by arrow 1056. Reconstructed samples 1056 are fed to a reconstructed sample cache 1060 and an in-loop filtering module 1088. In-loop filtering module 1088 produces a reconstructed block of frame samples denoted as 1092. Frame samples 1092 are written to a frame buffer 1096. Frame buffer 1096 outputs the same Figure 1 The tensor 149 corresponds to the image or video frame 1014.

[0187] Reconstructed sample cache 1060 operates in a manner similar to reconstructed sample cache 856 of video encoder 714. Reconstructed sample cache 1060 provides storage for reconstructed samples needed for intra prediction of subsequent CBs without memory 206 (e.g., by using data 232, typically on-chip memory, instead). Reference samples, represented by arrow 1064, are obtained from reconstructed sample cache 1060 and supplied to reference sample filter 1068 to generate filtered reference samples, represented by arrow 1072. Filtered reference samples 1072 are supplied to intra prediction module 1076. Module 1076 generates a block of intra-predicted samples, represented by arrow 1080, based on intra-prediction mode parameters 1058 signaled in bitstream 1010 and decoded by entropy decoder 1020. Intra prediction module 1076 supports the modes of encoder-side module 864, including IBC and MIP. Sample block 1080 is generated using modes such as DC, planar, or angular intra prediction.

[0188] When the prediction mode of a CB is indicated in the bitstream 143 to use intra prediction, intra-prediction samples 1080 are passed through a multiplexer module 1084 to form a decoded PB 1052. Intra prediction produces a prediction block (PB) of samples, which is a block in a color component that is derived using "neighboring samples" in the same color component. Neighboring samples are samples that are adjacent to the current block and have already been reconstructed because they are at the front in the block decoding order. In the case of juxtaposition of luma and chroma blocks, the luma and chroma blocks can use different intra prediction modes. However, the two chroma CBs share the same intra prediction mode.

[0189] When the prediction mode for the CB is indicated as inter prediction in the bitstream 143, the motion compensation module 1034 generates a block of inter-predicted samples, designated 1038. Block 1038 of inter-predicted samples is generated by selecting and filtering a block of samples from a frame buffer 1096 using the motion vector and reference frame index decoded from the bitstream 143 by the entropy decoder 1020. Block 1098 of samples is obtained from a previously decoded frame stored in the frame buffer 1096. To perform bi-prediction, two blocks of samples are generated and blended together to produce samples for the decoded PB 1052. Frame buffer 1096 is populated with filtered block data 1092 from the in-loop filtering module 1088. Like the in-loop filtering module 868 of the video encoder 714, the in-loop filtering module 1088 applies any of DBF, ALF, and SAO filtering operations. Generally, motion vectors are applied to both the luma and chroma channels, although the filtering process for subsample interpolation in the luma and chroma channels differs.

[0190] exist Figure 8 and Figure 10Not shown are modules for pre-processing the video before encoding and post-processing the video after decoding to shift sample values so that more uniform use of the sample value range within each chroma channel is achieved. A multi-segment linear model is derived in the video encoder 714 and signaled in the bitstream for use by the video decoder 904 to undo the sample shifting. The Linear Model Chroma Scaling (LMCS) tool provides compression benefits for color spaces and content that have some non-uniformity in their use of sample space (especially limited range use) that may result in higher quality loss from the application of quantization.

[0191] Figure 12A is a schematic block diagram illustrating an example implementation 1200 of the head 150 of a CNN for object detection (corresponding to the portion of the “YOLOv3” network excluding the “DarkNet-53” backbone). Figure 3A When implementing a CNN backbone in this manner, implementation 1200 can be used. Depending on the task to be performed in the destination device 140, the CNN head 150 can be replaced with a different network. The input tensor 149 is separated into tensors for each layer (i.e., tensors 1210, 1220, and 1234). Tensor 1210 is passed to the CBL module 1212 to produce tensor 1214. Tensor 1214 is passed to the detection module 1216 and the upscaling module 1222. The detection module outputs a bounding box 1218 in the form of a detection tensor. Bounding box 1218 is passed to the non-maximum suppression (NMS) module 1248.

[0192] To generate a bounding box addressing coordinates in the original video data 113, scaling to the original video width and height is performed at an upscaling module 1222 before resizing for the backbone of the network 114. Upscaling module 1222 receives tensor 124 and tensor 1220 and generates an upscaled tensor 1224, which is passed to a CBL module 1226. CBL module 1226 generates tensor 1228 as output. Tensor 1228 is passed to a detection module 1230 and an upscaling module 1236. Detection module 1230 generates a detection tensor 1232, which is fed to an NMS module 1248. Upscaling module 1236 is another instance of module 1222. Upscaling module 1236 receives tensor 1228 and tensor 1234 and outputs an upscaled tensor 1238. The upgraded tensor 1238 is passed to the CBL module 1240 , which outputs a tensor 1242 to the detection module 1244 .

[0193] CBL modules 1212, 1226, and 1240 each include five CBL modules (e.g., Figure 3DUpgrader modules 1222 and 1236 are cascaded. Figure 12B Instances of updater module 1260 are shown. Module 1248 receives tensors 1218 , 1232 , and 1236 and outputs task result 151 .

[0194] like Figure 12B As shown, the upgrader module 1260 accepts a tensor 1262 (e.g., Figure 12A 1214) as input. Tensor 1262 is passed to CBL module 1266 (having the structure of module 360) to produce tensor 1268. Tensor 1268 is passed to upsampler 1270 to produce upsampled tensor 1272. Cascade module 1274 concatenates upsampled tensor 1272 with second input tensor 1264 (e.g., input to Figure 12A The tensor 1220 of the upgrader 1222 in is cascaded to produce tensor 1276.

[0195] Detection modules 1216, 1230, and 1244 are as follows Figure 12C 1090 ). The result of the detection kernel has the same spatial dimensions as the input feature map, but the depth of the output corresponds to the detection attribute. The detection kernel is applied at various layers, typically three layers, which results in a large number of candidate bounding boxes. The NMS module 1048 applies non-maximum suppression to the resulting bounding boxes to discard redundant boxes, such as overlapping predictions of similar scales, thereby obtaining the final set of bounding boxes as the output of object detection.

[0196] Figure 1313 is a schematic block diagram showing an alternative head 1300 of a CNN. The head 1300 may be implemented as a CNN head 150, wherein the CNN backbone 114 is implemented as the backbone 400, for example. The head 1300 forms part of an overall network known as "Faster RCNN" and includes a feature network (i.e., the backbone portion 400), a region proposal network, and a detection network. The input to the head 1300 is a tensor 149, which includes P2 to P6 layer tensors 1310, 1312, 1314, 1316, and 1318. The P2 to P5 layer tensors 1310, 1312, 1314, and 1316 are connected to Figure 4 , corresponding to the P2 to P5 outputs 477, 475, 473, and 471. The P2 to P6 tensors 1310, 1312, 1314, 1316, and 1318 are input to the Region Proposal Network (RPN) head module 1320. The P6 tensor 1318 is produced by the max pooling module 1342, which operates on the P5 tensor 1316 to perform a 2×2 max pooling operation. The RPN head module 1320 performs a convolution on the input tensor to produce an intermediate tensor. The intermediate tensor is fed into two subsequent sibling layers: (i) one for classification and (ii) one for bounding box or "region of interest" (ROI) regression. The resulting output is the classification and bounding box 1322. The classification and bounding box 1322 are passed to the NMS module 1324. The NMS module 1324 prunes redundant bounding boxes by removing overlapping boxes with low scores to produce pruned bounding boxes 1326. The bounding box 1326 is input to a region of interest (ROI) pooler 1328. The ROI pooler 1328 uses some of the layer tensors of the tensor 149 (described further below) and the bounding box 1326 to produce a fixed-size feature map from various input size maps using a max pooling operation. In a max pooling operation, the subsampling takes the maximum value from each set of input values to produce a single output value in the output tensor.

[0197] The inputs to the ROI pooler 1328 are the P2 to P5 feature maps 1310, 1312, 1314, and 1316 and the region of interest proposal 1326. Each proposal (ROI) from 1326 is associated with a portion of the feature map (1310 to 1316) to produce a fixed-size map. The size of the fixed-size map is independent of the underlying portion of the feature maps 1310 to 1316. For example, one of the feature maps 1310 to 1316 is selected so that the resulting crop has sufficient detail according to the following rule: floor(4+log2(sqrt(box_area) / 224)), where 224 is a typical box size. The ROI pooler 1328 operates to crop the incoming feature map according to the proposal 1326, resulting in a tensor 1330.

[0198] Tensor 1330 is fed into a fully connected (FC) neural network head 1332. The FC head 1332 performs two fully connected layers to produce class score and bounding box prediction difference tensors 1334. The class score is typically a tensor of 80 elements, each element corresponding to the predicted score of the corresponding object class. The bounding box prediction difference tensor is a tensor of 80×4=320 elements, containing the bounding boxes of the corresponding object class. Final processing is performed by the output layer module 1336, which receives tensor 1334 and performs a filtering operation to produce a filtered tensor 1338. Low-scoring (low-classification) objects are no longer considered further. The non-maximum suppression module 1340 receives the filtered tensor 1338 and removes overlapping bounding boxes by removing overlapping boxes with lower classification scores, resulting in an inference output tensor 1340 corresponding to tensor 151.

[0199] Figure 14A is a schematic block diagram illustrating partitioning of picture 1400 into sub-pictures 1410, 1412, and 1416a, 1416b, and 1416c as implemented, for example, by sub-picture bitstream combiner 680. The sub-pictures are packed by packer 710 of a respective one of encoders 614, 636, 654a, 654b, and 654c. Figure 14A and Figure 14B The example of shows the coefficients of three groups and thus uses three sub-pictures. However, in other implementations, a different number of groups is possible. Each sub-picture includes information arranged in a two-dimensional array of samples. Figure 14B , picture 1400b corresponds to picture 1400, sub-picture 1410b corresponds to 1410, and sub-picture 1412b corresponds to 1412. Sub-picture 1410b holds mean channel data, such as mean channel 1420 for tensor 115. Mean channel 1410b corresponds to Figure 6 The bitstream 615 of the sub-picture encoder 614 in the arrangement of corresponds to the bitstream 615 of the sub-picture encoder 614 in the arrangement of . Figure 6 The 637 corresponding basis vectors. Figure 14B In the example of , the basis vectors include basis vector 1422 and others, where the basis vectors are packed into the area of sub-picture 1412b in a non-overlapping manner. The area in picture 1400 (or 1400b) that is not used to store any data can be occupied by sample values corresponding to the value "0" after inverse quantization is applied to convert from sample values back to the floating-point domain. Similarly, the area in sub-pictures 1410 (or 1410b), 1412 (or 1412b), 1416a, 1416b and 1416c that is not used to store any data can be occupied by sample values corresponding to the value "0" after inverse quantization is applied to convert from sample values back to the floating-point domain. Sub-pictures 1416a, 1416b and 1416c remain the same Figure 6..9], each having a width and height corresponding to the width and height of tensor 115. Coefficient sub-picture 1416b has coefficients packed into four feature maps such as feature map 1432 (applicable to basis vectors [10..14]). Coefficient sub-picture 1416c has 10 feature maps such as feature map 1434 (applicable to basis vectors [15..25]).

[0200] Figure 15 1616a, 1416b, and 1416c. The bitstream 1500 corresponds to the bitstream 121 generated by the PCA encoder 160 or the bitstream 143 decoded by the PCA decoder 170. The bitstream 1500 includes a syntax group beginning with a "Network Abstraction Layer" (NAL) unit header. For example, the NAL unit header 1508 precedes the sequence parameter set (SPS) 1510. The SPS 1510 uses sub-picture information 1511 to specify the layout of the picture 1400, including the location and size of sub-pictures 1410, 1412, 1416a, 1416b, and 1416c. The SPS 1510 also indicates the chroma format, bit depth, and resolution of the frame data represented by the bitstream 1500. If N is adjusted at step 16160, the coefficient grouping encoded at step 16180 is adjusted to begin a new sub-picture structure.

[0201] The coefficient grouping used by the PCA encoder 160 may be encoded in the SEI message 1513, using a list encoded as coefficient grouping information 1591 to split the coefficient tensor into groups along the channel dimension. The size (i.e., width and height) of the coefficient feature map is also encoded in the SEI message 1513 as part of the coefficient grouping information 1591. The quantization range as determined in the quantizers 612, 632, 646 is encoded as the quantization range 1592.

[0202] Picture 1514 is encoded in the bitstream 1500. Each picture includes one or more sub-pictures, such as encoded sub-picture 1520, thereby encoding sub-picture 1410 (or 1410b). For the first picture of the bitstream and for IDR pictures, intra slicing is used to avoid any prediction dependencies on other access units in the bitstream 1500. Encoded sub-picture 1522, which encodes sub-picture 1412, includes a slice header 1530 followed by slice data 1540. Slice data 1540 includes a sequence of CTUs, providing an encoded representation of the frame data. CTUs are square and typically 128×128 in size, which does not align well with typical feature map sizes. Alignment of feature maps with the smallest block size, such as a 4×4 grid, partially improves this misalignment. The encoded sub-pictures 1524, 1526, and 1528 encode the sub-pictures 1416a, 1416b, and 1416c corresponding to the respective coefficient groups.

[0203] In an arrangement of method 1600, determine basis vectors step 1650 is performed less frequently than for each frame. When basis vectors are not determined for a particular frame, basis vectors from an earlier frame are used. Step 1650 may be performed infrequently, such as once at the beginning of a video sequence, or periodically, such as each time a frame is to be encoded using intra prediction (such as a new IDR picture or an intra picture in a random access picture structure). When performing step 1650, for each basis vector, the amount of explained variance of the corresponding basis vector is also derived. The degree of explained variance can form the basis for determining the grouping of coefficients and / or for sorting regions or sub-pictures. When performing step 1650 and the difference in explained variance relative to the variance determined in the previous performance of step 1650 exceeds a threshold, source device 110 may derive a new grouping and, therefore, a new partitioning of picture 1400 into sub-pictures. When partitioning of the picture 1400 into sub-pictures is performed, an IDR picture needs to be sent to signal the partitioning, and an instance of the SEI message 1513 is signaled to indicate the partitioning of coefficients into groups utilizing the defined sub-picture structure.

[0204] In the arrangement of method 1600, when packing coefficient feature maps into the last sub-picture (i.e., 1416c), a variable number of coefficient feature maps are packed, thereby providing a fine-grained mechanism for rate control. Since the last sub-picture is often large in size so as to occupy unused areas in picture 1400, encoding all coefficient feature maps for this purpose will result in a coarse granularity in the addition rate used to include the sub-picture in the bitstream 121. The iterative method of steps 16120 to 16170 can be performed within the last sub-picture to determine how many coefficient feature maps to use. Once the number of coefficient feature maps to be used is determined, the last sub-picture (1416c) can be re-encoded while only the used coefficient feature maps are packed into the sub-picture area.

[0205] In the described example arrangement, tensor combiner 162 is implemented as MSFF module 500 and tensor separator 172 is implemented as MSFR module 1130, both of which involve trained layers in their operations. In an alternative arrangement, tensor combiner 162 can combine two layers by resampling the tensor of one layer and concatenating the resulting tensor with the tensor of another layer (e.g., an adjacent layer in an FPN), thereby producing combined tensor 115 at step 1615. The number and dimensions of the tensors can be stored as a layer map in SEI message 1513. In an alternative implementation, tensor separator 172 performs the inverse operation of tensor combiner 162 to produce extraction tensor 149 based on the layer map of the decoded SEI message. In the case of an FPN including multiple layers (such as four layers), these layers can be treated as two layer sets and processed by separate instances of tensor combiner 162, PCA encoder 160, PCA decoder 170, and tensor separator 172. Layers P2 and P3 can be viewed as one set, and layers P4 and P5 can be viewed as another set. Although viewed as separate layer sets, the associated mean channel tensors, basis vectors, and coefficients of each set can be packed into a single feature frame.

[0206] Regardless of the method used, the tensor combiner 162 operates to fuse together multiple tensors that typically form a hierarchical representation from the FPN into a single tensor suitable for decorrelation using a PCA method. Each tensor output in the tensor 115a forms part of the split point of the network, which is split into a first (trunk 114) part and a second (head 150) part. If FPN is not used, the first and second parts do not include a hierarchical representation of the input to the first part.

[0207] The methods proposed in this paper enable efficient representation of tensors in a format suitable for compression using contemporary block-based compression standards such as VVC or HEVC. Block-based compression, while not intuitively applicable to data such as the coefficients used to project basis vectors to reconstruct feature maps, reveals additional unintended redundancy in blocks, such as by using various transforms including trained quadratic transforms. Although the methods proposed in this paper are illustrated with reference to the "Faster RCNN" and "YOLOv3" network architectures and the specific partitioning of these networks into "backbone" and "head" parts, these methods are applicable to any neural network operating on multi-dimensional tensor data, and to different partitionings of such networks into "backbone" and "head" parts.

[0208] Industrial applicability

[0209] The described arrangement is suitable for use in the computer and data processing industry, and in particular for use in digital signal processing for encoding and decoding signals such as video and image signals, thereby achieving high compression efficiency.

[0210] Also disclosed is an arrangement for quantizing floating-point tensor data in a channel group or feature map and packing the resulting integer values into a planar frame using a logarithmic quantization domain. The quantization and inverse quantization methods employing a logarithmic quantization domain enable greater compression efficiency due to the absence of bits spent encoding the exact values of large-magnitude tensor values, where such precision does not result in additional improvement in the task performance of the network being used.

[0211] In some of the described arrangements, MFSC feature compression or expansion is used in conjunction with PCA encoding or decoding, respectively. Conventional use of MFSC techniques, which include three main modules: MSFF, SSFC (encoder and decoder), and MSFR, can offer performance at the expense of high training requirements and reduced flexibility due to these training requirements. Conventional techniques using PCA can underperform when using fewer coefficients, resulting in a reduced utilization of the feature frame area. Combining PCA with trained convolutional layers for feature compression including MSFF, PCA, and MSFR allows for suitable accuracy without incurring higher training requirements. Furthermore, because the number of basis vectors whose coefficients are encoded can vary between frames, and because PCA can include error analysis, the use of PCA allows for scalability not possible with MFSC alone. Furthermore, the basis vectors can be updated intermittently or determined less frequently for any given frame.

[0212] In other arrangements, PCA is implemented in a channel-by-channel manner so that only some groups or sub-pictures are selected for encoding. As described above, selecting and thus packing a variable number of coefficient feature maps can provide a fine-grained mechanism for rate control.

[0213] The foregoing describes only some embodiments of the present invention, and modifications and / or changes may be made thereto without departing from the scope and spirit of the invention, the embodiments being illustrative and not restrictive.

Claims

1. A method for decoding a plurality of tensors forming a hierarchical representation of a single frame from a bitstream, the method comprising: decoding coefficients from the bitstream for a first tensor; generating a second tensor from the first tensor and a set of basis vectors, the second tensor having the same spatial size and a higher number of channels than the first tensor; as well as One or more trained convolutional layers are used to generate a plurality of decoded tensors from the second tensor.

2. The method according to claim 1, wherein The second tensor is a projection of the coefficients and the basis vectors generated using a dot product operation.

3. The method according to claim 1, wherein The basis vectors are decoded from the bitstream.

4. The method according to claim 1, wherein The trained convolutional layers implement spatial resizing to recover a hierarchical representation of the frame.

5. The method according to claim 2, wherein: The number of channels of the projection is equal to the number of channels of each tensor in the plurality of tensors.

6. The method according to claim 1, wherein Before generating the plurality of tensors, a convolution having fewer output channels than input channels is applied to the second tensor.

7. A method for encoding a plurality of tensors forming a hierarchical representation of a single frame into a bitstream, the method comprising: generating a first tensor from the plurality of tensors using one or more downsampling filters; generating a second tensor from the first tensor by scaling the first tensor according to a scale tensor; generating coefficients of a third tensor using the second tensor and the set of basis vectors, the third tensor having the same spatial size and fewer channels than the second tensor; as well as Coefficients of the third tensor are encoded into the bitstream for the single frame.

8. The method according to claim 7, wherein: The second tensor is produced using the output of the addition of the tensors including the results of the activation layers.

9. The method according to claim 7, wherein: Generating the second tensor also includes applying a convolutional layer to the scaled first tensor to reduce the number of channels.

10. A decoder for decoding a plurality of tensors forming a hierarchical representation of a single frame from a bitstream, the decoder being configured to: decoding coefficients from the bitstream for a first tensor; generating a second tensor from the first tensor and a set of basis vectors, the second tensor having the same spatial size and a higher number of channels than the first tensor; as well as One or more trained convolutional layers are used to generate a plurality of decoded tensors from the second tensor.

11. A non-transitory computer-readable storage medium storing a program for executing a method for decoding a plurality of tensors forming a hierarchical representation of a single frame from a bitstream, the method comprising: decoding coefficients from the bitstream for a first tensor; generating a second tensor from the first tensor and a set of basis vectors, the second tensor having the same spatial size and a higher number of channels than the first tensor; as well as One or more trained convolutional layers are used to generate a plurality of decoded tensors from the second tensor.

12. A system comprising: Memory; as well as A processor, wherein the processor is configured to execute code stored on the memory for implementing a method for decoding a plurality of tensors forming a hierarchical representation of a single frame from a bitstream, the method comprising: decoding coefficients from the bitstream for a first tensor; generating a second tensor from the first tensor and a set of basis vectors, the second tensor having the same spatial size and a higher number of channels than the first tensor; and One or more trained convolutional layers are used to generate a plurality of decoded tensors from the second tensor.

13. An encoder for encoding a plurality of tensors forming a hierarchical representation of a single frame into a bitstream, the encoder being configured to: generating a first tensor from the plurality of tensors using one or more downsampling filters; generating a second tensor from the first tensor by scaling the first tensor according to a scale tensor; generating coefficients of a third tensor using the second tensor and the set of basis vectors, the third tensor having the same spatial size and fewer channels than the second tensor; as well as Coefficients of the third tensor are encoded into the bitstream for the single frame.

14. A non-transitory computer-readable storage medium storing a program for executing a method for encoding a plurality of tensors forming a hierarchical representation of a single frame into a bitstream, the method comprising: generating a first tensor from the plurality of tensors using one or more downsampling filters; generating a second tensor from the first tensor by scaling the first tensor according to a scale tensor; generating coefficients of a third tensor using the second tensor and the set of basis vectors, the third tensor having the same spatial size and fewer channels than the second tensor; as well as Coefficients of the third tensor are encoded into the bitstream for the single frame.

15. A system comprising: Memory; as well as a processor, wherein the processor is configured to execute code stored on the memory for implementing a method for encoding a plurality of tensors forming a hierarchical representation of a single frame into a bitstream, the method comprising: generating a first tensor from the plurality of tensors using one or more downsampling filters; generating a second tensor from the first tensor by scaling the first tensor according to a scale tensor; generating coefficients of a third tensor using the second tensor and a set of basis vectors, the third tensor having the same spatial size and fewer channels than the second tensor; and Coefficients of the third tensor are encoded into the bitstream for the single frame.

Images