Methods, apparatus, and systems for encoding and decoding tensors

Abstract

A method for decoding a video frame tensor from a bitstream. The method includes decoding a first mean and a plurality of first basis vectors from the bitstream, and generating a plurality of first tensors for a first video frame using the first mean and the plurality of first basis vectors. The method further includes decoding a plurality of second basis vectors from the bitstream, and generating a plurality of second tensors for a second video frame using the first mean and the plurality of second basis vectors, wherein the second video frame is different from the first video frame.

Description

Technical Field

[0001] This application claims priority under 35 U.S.C. § 119 based on the filing date of Australian Patent Application No. 2023202312, filed on April 14, 2023, which is hereby incorporated by reference in its entirety as if fully set forth herein.

[0002] The present invention generally relates to digital video signal processing, and more particularly to methods, apparatuses, and systems for encoding and decoding tensors from convolutional neural networks. The present invention also relates to a computer program product including a computer-readable medium having recorded thereon a computer program for encoding and decoding tensors from convolutional neural networks using video compression techniques.

Background Art

[0003] Convolutional neural networks (CNNs) are an emerging technology that addresses machine vision-related use cases such as object detection, instance segmentation, object tracking, human pose estimation, and motion recognition. CNN applications may involve the use of "edge devices" with sensors and a certain level of processing power, connected to application servers as part of the "cloud." CNNs can require relatively high computational complexity, which can exceed the computing power and power consumption typically provided by edge devices. Distributed CNN execution is emerging as a solution to run state-of-the-art networks on edge devices with limited capabilities without shifting all the computational complexity to cloud servers. In other words, distributed processing allows legacy edge devices to continue providing the capabilities of state-of-the-art CNNs by distributing processing between edge devices and other processing means such as cloud servers. Such distributed network architectures are called "Collaborative Intelligence (CI)" and offer advantages such as reusing partial results obtained in the first part of the network in multiple different second parts, each optimized for different tasks. CI architectures require efficient compression of tensor data for transmission over networks such as WANs.

[0004] CNNs typically contain numerous layers, such as convolutional and fully connected layers, and data is transmitted between layers in the form of "tensors." When a network is divided among different devices, compression of the intermediate multidimensional tensor data transmitted between layers within the CNN becomes necessary. This type of tensor compression is called "feature compression," and the intermediate tensor data is often referred to as "features" or "feature maps." A feature or feature map is usually a collection of 2D "feature maps," where each feature map forms a tensor corresponding to one "channel" of the tensor. Intermediate tensor data represents a partially processed form of input data, such as image frames or video frames. The International Organization for Standardization / International Electrotechnical Commission Joint Technical Committee 1 / Section 29 / Working Group 2-8 (ISO / IEC JTC1 / SC29 / WG2-8), also known as the "Video Expert Group" (MPEG), is responsible for studying compression techniques in various contexts, particularly those related to video. WG2 "MPEG Technical Requirements" established the "Feature Compression for Machine Video Coding (FCVCM)" ad hoc group, tasked with studying feature compression. FCVCM AHG reviewed the issued Call for Evidence (CfE) and demonstrated a technique that significantly outperforms feature compression results achieved using state-of-the-art standardization techniques directly applied to tensors. FCVCM AHG is currently refining its Call for Proposals (CfP) for feature compression techniques.

[0005] In CNNs, the weights of each layer typically need to be pre-determined during the training phase. During this phase, a vast amount of training data passes through the CNN, and the results derived by the training network are compared to the true values ​​associated with the training data. The difference between the obtained results and the desired results is expressed as "loss" and measured using a "loss function." The determined loss is then used to perform a network weight update process, such as stochastic gradient descent (SGD). Network weight updates typically involve backpropagation of the "gradient" (indicating the delta to be applied to the network weights), starting from the network's output layer and ending at the input layer, covering the network's intermediate (hidden) layers. The speed of weight updates is adjusted by a hyperparameter called the "learning rate," which is usually set to facilitate the search for a global minimum in the loss (the best possible task performance for the network architecture and training data) while avoiding the training process becoming "stuck" to a local minimum. Falling into a local minimum corresponds to obtaining task performance that is not optimal for the network architecture, and to being unable to find new weight values ​​that would lead to higher task performance. By supplying input data and ground truth data in batches, the network weights are repeatedly updated, and network performance is iteratively improved until no further accuracy improvements can be achieved. The iteration of the entire training dataset is called an "epoch," and multiple epochs of training are usually required to improve the performance of a task. The trained network then moves to a "deployment" mode in which it operates with fixed weights and gradient calculations for weight updates are omitted. The process of providing input to a pre-trained CNN and transforming the input into output step by step according to the CNN's topology is generally called "inference."

[0006] Generally, a tensor consists of four dimensions: batch, channel, height, and width. The first dimension, "batch," is typically 1 in size when performing inference on video data, indicating that one frame passes through the CNN as one batch. When training a network, the batch dimension value may be increased according to a predetermined "batch size" so that multiple frames pass through the network in each batch before the network weights are updated. Multi-frame video may also be processed as a single tensor with the batch dimension expanded according to the number of frames in the video. However, for practical reasons regarding memory consumption and access, inference on video data is usually performed on a frame-by-frame basis. The "channel" dimension indicates the number of "feature maps" being processed concurrently in a particular tensor, and the height and width dimensions indicate the size of the feature maps at a particular stage of the CNN. The number of channels varies in each layer of the CNN depending on the network architecture. The size of the feature maps also changes due to subsampling or upsampling that occurs in a particular network layer.

[0007] Convolutional Neural Networks (CNNs) tend to have relatively high overall complexity, involving a large number of multiplication and integration (MAC) operations, numerous intermediate tensors being written to and read from memory, and weights being loaded for each layer of the CNN to run. Therefore, dividing the neural network allows for the implementation of more complex networks even in systems that include low-performance edge devices.

[0008] Feature compression can benefit from existing video compression standards such as Universal Video Coding (VVC), developed by the Joint Video Expert Team (JVET). VVC is expected to address the growing demand for increased compression performance, particularly with advancements in video format capabilities (higher resolution, higher frame rates, etc.), and the increasing market demand for service delivery over WANs where bandwidth costs are relatively high. VVC is implementable with modern silicon processes and offers an acceptable trade-off between performance and implementation cost. Implementation cost is considered in terms of one or more of the following: silicon area, CPU processor load, memory usage, and bandwidth. Other video compression standards such as High Efficiency Video Coding (HEVC) and AV-1 can also be used for feature compression applications.

[0009] Video data consists of a frame sequence of image data, with each frame containing one or more color channels. When representing feature map data with packed frames, monochrome frames, which generally contain only luminance and no chroma channels, are usually sufficient. If only luminance samples are present, the resulting monochrome frame is said to use a "4:0:0 chroma format."

[0010] The VVC standard defines a "block-based" architecture, where a frame is first divided into a square array of regions called "encoded tree units (CTUs)." In VVC, a CTU typically occupies 128 x 128 lumens. Other CTU sizes available in the VVC standard are 32 x 32 and 64 x 64. However, CTUs at the rightmost and bottommost edges of each frame may have smaller areas, and implicit division occurs to ensure that the encoded blocks fit within the frame. Each CTU is associated with an "encoded tree," which defines how the CTU region is decomposed into a series of blocks (also called encoded units: CUs). Blocks that apply only to lumens or only to chroma channels are called "encoded blocks" (CBs). Information predicting the contents of an encoded block is held in a "prediction block" (PB) or "prediction unit" (PU), and residual blocks that define an array of sample values ​​that are additively combined with the PB or PU are called "transformation blocks" (TBs) or "transformation units" (TUs) because transformation processes are usually used to generate TBs or TUs.

[0011] Regardless of the above distinction between "unit" and "block," the term "block" may be used as a general term for a region or group of regions within a frame to which an operation is applied across all color channels.

[0012] For each CU, a prediction unit (PU) is generated for the content (sample value) of the corresponding region of the frame data ("prediction unit"). Furthermore, a value is formed that represents the difference (spatial domain residual) between the predicted value observed at the time of encoding input and the measured value of the region. The difference for each color channel is transformed and encoded as a sequence of residual coefficients, forming one or more TUs for a given CU. The transformation applied is a discrete cosine transform (DCT) or other transformation, and is applied to each block of residual values. This transformation is applied separately (i.e., a two-dimensional transformation is performed in two passes, horizontal and vertical). The blocks are first transformed by applying a one-dimensional transformation to each row of samples within the block. Next, a one-dimensional transformation is applied to each column of the partial result to generate a final transformed coefficient block with the correlation of residual samples effectively removed. The VVC standard supports transformations of various sizes, including rectangular blocks where the length of each side is a power of 2. The transformed coefficients are quantized for entropy encoding to the bitstream.

[0013] In VVC, the PB or PU is generated using either an intra-frame prediction or an inter-frame prediction process. In intra-frame prediction uses processed samples within a frame to generate a prediction for the current data sample block within that frame. Inter-frame prediction uses sample blocks obtained from one or two pre-decoded frames to generate a prediction for the current sample block within a frame. Sample blocks obtained from pre-decoded frames are offset from the spatial position of the current block according to an action vector (often with filtering applied). The intra-frame prediction block can be (i) a uniform sample value ("DC intra-frame prediction"), (ii) a plane with offsets and horizontal and vertical slopes ("planar intra-frame prediction"), (iii) an aggregation of blocks with adjacent samples applied in a specific direction ("angle intra-frame prediction"), or (iv) the result of matrix multiplication using adjacent samples and selected matrix coefficients.

[0014] VVC is used to compress intermediate feature maps obtained from the first part ("backbone") of a neural network that has been separated into two parts. During compression, the feature maps from the backbone are placed in frames and quantized from the floating-point domain to a sample domain suitable for compression as video data. Techniques related to principal component analysis (PCA) are used to apply tensor dimensionality reduction before compression using video compression standards such as VVC. Tensor dimensionality reduction reduces the amount of data to be compressed, improving compression efficiency and shortening the execution time of the VVC coding and decoding stages. However, dimensionality reduction introduces complexity, which offsets the time-saving effect seen in VVC coding. There is a need to address, or at least improve, the increased complexity caused by applying PCA to tensor dimensionality reduction. [Overview of the project]

[0015] The object of the present invention is to substantially overcome, or at least improve upon, one or more drawbacks of existing configurations.

[0016] One aspect of the present disclosure provides a method for decoding a video frame tensor from a bitstream, the method comprising: decoding a first mean and a plurality of first basis vectors from the bitstream; generating a plurality of first tensors for a first video frame using the first mean and the plurality of first basis vectors; decoding a plurality of second basis vectors from the bitstream; and generating a plurality of second tensors for a second video frame using the first mean and the plurality of second basis vectors, wherein the second video frame is different from the first video frame.

[0017] Another aspect of the present disclosure provides a method for encoding a video frame tensor into a bitstream, the method comprising: generating a first mean and a plurality of first basis vectors for a plurality of tensors of a first video frame; encoding the first mean and the plurality of first basis vectors into the bitstream; generating a plurality of second basis vectors for a plurality of tensors of a second video frame using the first mean; and encoding the plurality of second basis vectors into the bitstream, wherein the second video frame is different from the first video frame.

[0018] Another aspect of the present disclosure provides a non-temporary computer-readable storage medium for storing a program for performing a method of decoding a video frame tensor from a bitstream, the method comprising: decoding a first mean and a plurality of first basis vectors from the bitstream; generating a plurality of first tensors for a first video frame using the first mean and the plurality of first basis vectors; decoding a plurality of second basis vectors from the bitstream; and generating a plurality of second tensors for a second video frame using the first mean and the plurality of second basis vectors, wherein the second video frame is different from the first video frame.

[0019] Another aspect of the present disclosure provides a video decoder for decoding a video frame tensor from a bitstream, the video decoder is configured to decode a first mean and a plurality of first basis vectors from the bitstream, generate a plurality of first tensors for a first video frame using the first mean and the plurality of first basis vectors, decode a plurality of second basis vectors from the bitstream, and generate a plurality of second tensors for a second video frame using the first mean and the plurality of second basis vectors, wherein the second video frame is different from the first video frame.

[0020] Another aspect of this disclosure provides a system which includes memory and

[0021] A processor configured to execute code stored in memory for performing a method of decoding a video frame tensor from a bitstream, the method comprising: decoding a first mean and a plurality of first basis vectors from the bitstream; generating a plurality of first tensors for a first video frame using the first mean and the plurality of first basis vectors; decoding a plurality of second basis vectors from the bitstream; and generating a plurality of second tensors for a second video frame using the first mean and the plurality of second basis vectors, wherein the second video frame is different from the first video frame.

[0022] Another aspect of the present disclosure provides a non-temporary computer-readable storage medium for storing a program for performing a method of encoding a video frame tensor into a bitstream, the method comprising: generating a first mean and a plurality of first basis vectors for a plurality of tensors of a first video frame; encoding the first mean and the plurality of first basis vectors into the bitstream; generating a plurality of second basis vectors for a plurality of tensors of a second video frame using the first mean; and encoding the plurality of second basis vectors into the bitstream, wherein the second video frame is different from the first video frame.

[0023] Another aspect of the present disclosure provides a video encoder for encoding a tensor of a video frame into a bitstream. The video encoder generates a first average for a plurality of tensors of a first video frame and a plurality of first basis vectors, encodes the first average and the plurality of first basis vectors into the bitstream, uses the first average to generate a plurality of second basis vectors for a plurality of tensors of a second video frame, and encodes the plurality of second basis vectors into the bitstream, wherein the second video frame is different from the first video frame.

[0024] Also, another aspect of the present disclosure provides a system, the system includes a memory and

[0025] a processor configured to execute code stored in the memory for implementing a method of encoding a tensor of a video frame into a bitstream. The method includes generating a first average for a plurality of tensors of a first video frame and a plurality of first basis vectors, encoding the first average and the plurality of first basis vectors into the bitstream, using the first average to generate a plurality of second basis vectors for a plurality of tensors of a second video frame, and encoding the plurality of second basis vectors into the bitstream, wherein the second video frame is different from the first video frame.

[0026] Other aspects are also disclosed.

Brief Description of the Drawings

[0027] At least one embodiment of the present invention will be described with reference to the following drawings and appendices.

[0028] [Figure 1] FIG. 1 is a schematic block diagram showing a distributed machine task system;

[0029] [Figure 2A]Figure 2A forms a schematic block diagram of a general-purpose computer system capable of executing the distributed machine task system shown in Figure 1; [Figure 2B] Figure 2B forms a schematic block diagram of a general-purpose computer system capable of executing the distributed machine task system of Figure 1;

[0030] [Figure 3A] Figure 3A is a schematic block diagram showing the core functional modules of a CNN;

[0031] [Figure 3B] Figure 3B is a schematic block diagram showing the residual block of Figure 3A;

[0032] [Figure 3C] Figure 3C is a schematic block diagram showing the residual unit in Figure 3A;

[0033] [Figure 3D] Figure 3D is a schematic block diagram showing the CBL module in Figure 3A;

[0034] [Figure 4] Figure 4 is a schematic block diagram showing the functional modules of the alternative backbone portion of the CNN;

[0035] [Figure 5] Figure 5 is a schematic block diagram showing the multiscale feature fusion stage;

[0036] [Figure 6] Figure 6 is a schematic block diagram of a tensor encoder based on inter-channel discorrelation as part of a distributed machine learning task system;

[0037] [Figure 7] Figure 7 is a schematic block diagram showing the feature map packer, packed frame encoder, and unpacker (or “subpicture encoder”) as part of an inter-channel decorrelation-based tensor encoder;

[0038] [Figure 8] Figure 8 is a schematic block diagram showing the functional modules of the video encoder;

[0039] [Figure 9] Figure 9 is a schematic block diagram showing a tensor decoder based on inter-channel correlation removal as part of a distributed machine learning system;

[0040] [Figure 10] Figure 10 is a schematic block diagram showing the functional module of the video decoder.

[0041] [Figure 11] Figure 11 is a schematic block diagram showing an embodiment of the multiscale feature reconstruction stage.

[0042] [Figure 12A] Figure 12A is a schematic block diagram showing the head of a CNN;

[0043] [Figure 12B] Figure 12B is a schematic block diagram showing the upscaler module of Figure 10A;

[0044] [Figure 12C] Figure 12C is a schematic block diagram showing the detection module in Figure 10A;

[0045] [Figure 13] Figure 13 is a schematic block diagram showing the alternative head of the CNN;

[0046] [Figure 14A] Figure 14A is a schematic block diagram showing the arrangement of subpictures for holding inter-channel uncorrelated tensor data; [Figure 14B] Figure 14B is a schematic block diagram showing the arrangement of subpictures for holding inter-channel uncorrelated tensor data;

[0047] [Figure 15] Figure 15 is a schematic block diagram showing the bitstream holding the encoded inter-channel decorrelation feature map and associated metadata;

[0048] [Figure 16] Figure 16 shows how to run the first part of the CNN, perform decorrelation, and encode the resulting feature map;

[0049] [Figure 17] Figure 17 shows how to decode the bitstream, reconstruct the decorrelated feature map, and run the second part of the CNN;

[0050] [Figure 18] Figure 18 is a schematic block diagram showing a spatially subsampled tensor. [Modes for carrying out the invention]

[0051] Where steps and / or features are shown in the attached drawings with the same reference numerals, in this specification, those steps and / or features shall have the same function or operation unless otherwise explicitly stated.

[0052] A distributed machine task system can include edge devices such as network cameras and smartphones that generate intermediate compressed data. The distributed machine task system can also include end devices such as server farm-based ("cloud") applications that process the intermediate compressed data and generate task results. Furthermore, the functionality of the edge devices can be implemented in the cloud, and the intermediate compressed data can be stored for subsequent processing and used for multiple different tasks as needed.

[0053] A convenient format for intermediate compressed data is the compressed video bitstream, for which high-performance compression standards and their implementations are available. Video compression standards typically process integer samples with a given bit depth, such as 10 bits, in a planar array. Color video has three planar arrays, corresponding to color components such as Y, Cb, Cr, or R, G, B, depending on the application. Convolutional Neural Networks (CNNs) typically operate with floating-point data in tensor form. Tensors have a relatively small spatial dimension compared to the input video data that CNNs process and have more channels than the typical three channels of color video data (e.g., 128, 256, or 512 channels).

[0054] Tensors typically have dimensions of frames, channels, height, and width. For example, a tensor with dimensions [1,256,76,136] represents one frame of data consisting of 256 feature maps (channels), with each map having a size of 136 x 76. In the case of video data, inference is usually performed frame by frame (frame value 1) rather than using a tensor containing multiple frames.

[0055] VVC supports the ability to divide a single picture into multiple subpictures, each of which can be independently encoded and decoded. In one approach, each subpicture is encoded as a single "slice," i.e., a continuous sequence of encoded CTUs. A "tiling" mechanism is also available, allowing a single picture to be divided into numerous independently decodeable regions. Subpictures can be specified with relative flexibility, allowing various sets of rectangular CTUs to be encoded as corresponding subpictures. By flexibly defining the dimensions of subpictures, data types requiring different regions within a single picture can be efficiently preserved, avoiding large "unused" regions (frame regions not used to reconstruct tensor data).

[0056] Figure 1 is a schematic block diagram showing the functional modules of a distributed machine task system 100 capable of executing a distributed machine task network in a distributed manner. To divide a particular neural network into two parts, it is necessary to specify a "split point" within the network. The network layers from the input layer to the split point are processed by a first device, and the resulting intermediate tensors are compressed. The layers from the split point to the final layer of the network are executed using the decompressed tensors from the first device as input to the layer immediately following the split point. At the split point, there may be one or more tensors that need to be compressed in order to be transmitted over a communication channel with limited bandwidth compared to the bandwidth required to transmit the uncompressed tensors. When a "feature pyramid network" (FPN) is used, layers within the FPN are related by width and height, and it is common for a particular layer to have half the width and height within an adjacent group of layers. In FPN architectures, the halving of width and height may be defined to occur alternately between layers. In some architectures, there may be multiple tensors with the same width and height. When FPNs appear in the relatively early stages of a neural network topology, a partition point within the FPN is necessary to effectively distribute the network load between edge devices and the cloud. When partitioning occurs within the FPN of a machine learning task network, the layers up to the partition point are shared across the machine learning task networks ("shared backbone" architecture), enabling the performance of diverse machine learning task networks. When partition points occur within an FPN, tensor compression techniques can leverage redundancy between FPN layers to improve compression performance. Therefore, compression techniques applicable to various network topologies used in modern CNNs are beneficial for a wide range of scenarios.

[0057] System 100 can be used to implement a method for decorrelating, packing, and quantizing feature maps into planar frames, encoding them, and decoding the feature maps from the encoded data. Decorrelating (or "decomposing" or "reducing dimensionality") tensor data for compression is computationally intensive. System 100 can be implemented in the configuration described to reduce the computational load associated with tensor data dimensionality reduction.

[0058] System 100 includes a transmitting device 110 that generates encoded tensor data 115a in the form of an encoded video bitstream 121 from a CNN backbone 114. System 100 also includes a receiving device 140 that decodes the tensor data in the form of an encoded video bitstream 143. A communication channel 130 is used to transmit the encoded video bitstream 121 from the source device 110 to the destination device 140. In some configurations, the source device 110 and the destination device 140, or both, may consist of a corresponding mobile phone terminal (e.g., a "smartphone") or a network camera and cloud application. The communication channel 130 may be a wired connection such as Ethernet, or a wireless connection such as WiFi or 5G (including wide area networks (WANs) and ad-hoc connections). Furthermore, the transmitting device 110 and the receiving device 140 may consist of an application on which the encoded video data is stored in a computer-readable storage medium such as a hard disk drive or memory in a file server.

[0059] As shown in Figure 1, the source device 110 includes a video source 112, a CNN backbone 114, a tensor synthesizer 162, a principal component analysis (PCA) encoder 160, and a transmitter 122. The video source 112 typically consists of a source (e.g., an imaging sensor, a pre-imaged video sequence stored on a non-temporary recording medium, or a video feed from a remote imaging sensor) of captured video frame data (shown as 113). The video source 112 may also include the output of a computer graphics card that displays the video output of an operating system and various applications running on a computing device (e.g., a tablet computer). Examples of source devices 110 that may include an imaging sensor as the video source 112 include smartphones, video cameras, professional video cameras, and network video cameras. The video source 112 can generate independent images or temporally continuous images, i.e., video.

[0060] The CNN backbone 114 receives video frame data 113, executes a specific layer of the entire CNN (for example, the layer corresponding to the CNN's "backbone"), and outputs tensor 115a. The CNN backbone layer may output multiple tensors, such as those corresponding to different spatial scales of the input image represented by the video frame data 113, when the network is divided within the FPN. For example, if system 100 is running the "YOLOv3" network, the FPN may output three tensors corresponding to three layers with different spatial resolutions and channel counts as tensor 115a from backbone 114. If system 100 is running a network such as "Faster RCNN X101-FPN" or "Mask RCNN X101-FPN", tensor 115a may contain tensors corresponding to four layers (P2-P5). When network division points occur within the FPN due to the use of the FPN, as described later, multiple tensors forming a hierarchical representation of a single frame are generated and encoded (and decoded) into a bitstream. The tensor synthesizer 162 synthesizes multiple layers by performing convolutions with a stride greater than 1 (e.g., stride 2), implementing a trained downsampling stage, thereby generating a tensor among the multiple tensors 115a that has the same dimension as another spatially smaller tensor. The resulting group of tensors all have the same spatial dimension, are concatenated along the channel dimension, and are processed by an additional network layer to generate the synthesized tensor 115. The operation of the tensor synthesizer 162 is explained with reference to Figure 5. The PCA encoder 160 receives the synthesized tensor 115, which is the output from the tensor synthesizer 162. Layer synthesis is suitable when there is sufficient inter-layer correlation so that the synthesized layer can be represented with fewer basis vectors than would be required if the layers were decorrelated individually. The degree of inter-layer correlation is a characteristic of the network itself and the provided input data. Depending on the degree of inter-layer correlation, the total number of basis vectors in the synthesized tensor can be reduced compared to the sum of the number of basis vectors required if the tensors of each layer were decorrelated individually.For example, if 25 basis vectors are typically used in each layer, then when two layers are linked, the number of basis vectors required for the linked two layers can be set to less than 50, and the number of basis vectors required for the linked four layers can be set to less than 100.

[0061] The PCA encoder 160 is responsible for encoding the composite tensor 115 to generate the bitstream 121, as will be explained with reference to Figure 6. The bitstream 121 is either supplied to the transmitter 122 for transmission via the communication channel 130 or written to the storage device 132 for later use.

[0062] The transmitting device 110 supports a specific network for the CNN backbone 114. Meanwhile, the receiving device 140 can use any of several networks for the head CNN 150. When using any of several networks for the head CNN 150, partially processed data in the form of packed feature maps is saved and made available for use in subsequent tasks without having to re-run the processing of the CNN backbone 114.

[0063] Bitstream 121 is transmitted by transmitter 122 via communication channel 130 as encoded video data (or "encoded video information"). In some implementations, bitstream 121 can be stored in storage memory 132, which is a non-temporary storage device such as "flash" memory or a hard disk drive, until it is later transmitted via communication channel 130 (or instead of transmission via communication channel 130). For example, the encoded video data may be provided on demand to customers over a wide area network (WAN) for video analytics applications.

[0064] The destination device 140 includes a receiver 142, a PCA decoder 170, a tensor separator 172, a CNN head 150, and a CNN task result buffer 152. The receiver 142 receives encoded video data from the communication channel 130 and passes the video bitstream 143 to the PCA decoder 170. The PCA decoder 170 outputs a decoded composite tensor 149a, which is fed to the tensor separator 172. The tensor separator 172 performs the inverse operation of the tensor combiner 162 to produce an extracted tensor 149. The extracted tensor 149 is passed to the CNN head 150. At the split points where the network is divided, and at stages where a single tensor synthesis is not required, the tensor separator 172 performs no operation, and tensor 149 corresponds to tensor 149a. The CNN head 150 receives tensor 149 and executes subsequent layers of the task initiated in the CNN backbone 114 to produce a task result 151. The task result 151 is stored in the task result buffer 152. The contents of the task result buffer 152 are presented to the user, for example, via a graphical user interface, or provided to an analysis application where some action is determined based on the task result (which may include a summary-level presentation of the aggregated task result). Furthermore, the functions of the transmitting device 110 and the receiving device 140 can be implemented in a single device, such as a mobile phone terminal, tablet computer, or cloud application. While the examples described herein relate to PCA, other decomposition analysis methods and methods based on inter-channel decorrelation can also be used for the encoder 160 and decoder 170.

[0065] Regardless of the example devices described above, the transmitting device 110 and the receiving device 140 may be configured within a general-purpose computing system through a combination of hardware and software components. Figure 2A shows such a computer system 200, which includes: a computer module 201; input devices (such as a keyboard 202, a mouse pointer device 203, a scanner 226, a camera 227 (configurable as a video source 112), a microphone 280, etc.); and output devices (such as a printer 215, a display device 214, a speaker 217, etc.). The computer module 201 may use an external modulator / demodulator (modem) transceiver device 216 to communicate with the communication network 220 via connection 221. The communication network 220 (which may represent communication channel 130) may be the Internet, a cellular communication network, or a wide area network (WAN) such as a private WAN. If connection 221 is a telephone line, the modem 216 may be a conventional “dial-up” modem. On the other hand, if connection 221 is a high-capacity connection (e.g., cable or fiber optic), modem 216 may be a broadband modem. A wireless modem can also be used for wireless connection to wireless communication network 220. Transceiver device 216 combines the functions of transmitter 122 and receiver 142, and communication channel 130 may be embodied in connection 221.

[0066] The computer module 201 typically includes at least one processor unit 205 and a memory unit 206. For example, the memory unit 206 may have semiconductor random access memory (RAM) and semiconductor read-only memory (ROM). The computer module 201 further includes the following input / output (I / O) interfaces: an audio-video interface 207 for connecting to a video display 214, a speaker 217, and a microphone 280; an I / O interface 213 for connecting to a keyboard 202, a mouse 203, a scanner 226, a camera 227, and optionally a joystick or other human interface device (not shown); and an interface 208 for an external modem 216 and a printer 215. The signal from the audio-video interface 207 to the computer monitor 214 generally refers to the output of a computer graphics card. In some embodiments, the modem 216 may be integrated into the computer module 201, for example, located within interface 208. The computer module 201 also includes a local network interface 211, which allows the computer system 200 to be connected to a local area network (LAN), or local area communication network 222, via connection 223. As shown in Figure 2A, the local communication network 222 can also be connected to a wide area network 220 via connection 224, which typically includes a so-called "firewall" device or a device with equivalent functionality. The local network interface 211 may consist of an Ethernet® circuit card, a Bluetooth® radio device, or an IEEE 802.11 radio device, but many other types of interfaces can be employed for interface 211. The local network interface 211 may also provide the functionality of a transmitter 122 and a receiver 142, and a communication channel 130 may be implemented within the local communication network 222.

[0067] I / O interfaces 208 and 213 may provide either serial or parallel connectivity, or both, the former typically implemented according to the Universal Serial Bus (USB) standard and equipped with a corresponding USB connector (not shown). A storage device 209 is provided, typically including a hard disk drive (HDD) 210. Other storage devices such as floppy disk drives and magnetic tape drives (not shown) may also be used. An optical disk drive 212 is typically provided to function as a non-volatile data source. For example, portable memory devices (optical disks (e.g., CD-ROM, DVD, Blu-ray Disc®), USB-RAM, portable external hard drives, floppy disks, etc.) can be used as suitable data sources to the computer system 200. Typically, the HDD 210, optical drive 212, and networks 220 and 222 can also be configured to operate as a video source 112 or as a storage location for decoded video data saved for playback via the display 214. The source device 110 and destination device 140 of system 100 are embodied in computer system 200.

[0068] The components 205 to 213 of the computer module 201 typically communicate via an interconnected bus 204 and operate in a manner that realizes conventional operating modes of computer systems 200 known to experts in the relevant technical field. For example, the processor 205 is connected to the system bus 204 using connection 218. Similarly, the memory 206 and the optical disc drive 212 are connected to the system bus 204 via connection 219. Examples of computers capable of implementing this configuration include IBM-PC and compatible machines, Sun's SPARCstation, Apple's Mac®, or equivalent computer systems.

[0069] Where appropriate or necessary, the PCA encoder 160 and PCA decoder 170, as well as the method described later, can be implemented using a computer system 200. In particular, the PCA encoder 160, PCA decoder 170, and the method described later can be implemented as one or more software application programs 233 that can be executed within the computer system 200. Specifically, the steps of the PCA encoder 160, PCA decoder 170, and the method described are implemented by instructions 231 (see Figure 2B) in the software 233 executed within the computer system 200. The software instructions 231 may be formed as one or more code modules, each performing one or more specific tasks. Alternatively, the software may be divided into two independent parts, with the first part and its corresponding code module executing the method described, and the second part and its corresponding code module managing the interface between the first part and the user.

[0070] The software may be stored on a computer-readable medium, including, for example, a storage device described later. The software is loaded from the computer-readable medium into the computer system 200 and then executed by the computer system 200. Such a computer-readable medium on which software or a computer program is recorded is a computer program product. The use of the computer program product in the computer system 200 preferably results in a source device 110 and a destination device 140, as well as an advantageous device for implementing the described method.

[0071] The software 233 is typically stored on the HDD 210 or in memory 206. The software is loaded from a computer-readable medium into the computer system 200 and executed by the computer system 200. Therefore, for example, the software 233 may be stored on an optical disk storage medium (e.g., a CD-ROM) 225 that is read by an optical disk drive 212.

[0072] In some cases, the application program 233 may be encoded on one or more CD-ROMs 225 and provided to the user, and loaded via the corresponding drive 212, or loaded by the user from the network 220 or 222. Furthermore, the software may also be loaded into the computer system 200 from other computer-readable media. Computer-readable storage media refers to any non-temporary, tangible storage medium that provides the computer system 200 with recorded instructions and / or data for execution and / or processing. Examples of such storage media include floppy disks, magnetic tapes, CD-ROMs, DVDs, Blu-ray discs (trademarks), hard disk drives, ROMs or integrated circuits, USB memory sticks, magneto-optical disks, or computer-readable cards such as PCMCIA cards, whether these devices are inside or outside the computer module 201. Examples of temporary or intangible computer-readable transmission media that may be used to provide software, application programs, instructions, and / or video data or encoded video data to the computer module 201 include wireless or infrared transmission channels, network connections to other computers or network-connected devices, and the internet or intranets, including email transmissions and information recorded on websites.

[0073] The second part of the application program 233 and the corresponding code modules described above are executable for implementing one or more graphical user interfaces (GUIs) that are rendered or otherwise represented on the display 214. Users of the computer system 200 and the application can operate the interface in a functionally adaptable manner, typically by operating the keyboard 202 and mouse 203, and provide control commands and / or inputs to the application associated with the GUI. Other forms of functionally adaptable user interfaces can also be implemented, such as a voice interface that outputs voice prompts via the speaker 217 and takes user voice commands via the microphone 280.

[0074] Figure 2B is a detailed schematic block diagram of the processor 205 and “memory” 234. Memory 234 represents the logical collection of all memory modules (including storage device 209 and semiconductor memory 206) accessible by the computer module 201 in Figure 2A.

[0075] When the computer module 201 is first powered on, the Power-On Self-Test (POST) program 250 is executed. The POST program 250 is typically stored in the ROM 249 within the semiconductor memory 206 shown in Figure 2A. Hardware devices like the ROM 249 that store software are sometimes called firmware. The POST program 250 checks the hardware within the computer module 201 to ensure it is functioning correctly, typically checking the correct operation of the processor 205, memory 234 (209, 206), and the Basic Input / Output System Software (BIOS) module 251 (also typically stored in the ROM 249). If the POST program 250 executes successfully, the BIOS 251 starts the hard disk drive 210 shown in Figure 2A. The startup of the hard disk drive 210 causes the bootstrap loader program 252, which resides on the hard disk drive 210, to be executed via the processor 205. This loads the operating system 253 into the RAM memory 206, and the operating system 253 begins to operate. Operating System 253 is a system-level application runnable by Processor 205, performing various high-level functions including processor management, memory management, device management, storage management, software application interface, and general-purpose user interface.

[0076] The operating system 253 manages memory 234 (209, 206) to ensure that each process or application running on the computer module 201 has enough memory to run without conflicting with memory allocated to other processes. Furthermore, the different types of memory available in the computer system 200 in Figure 2A must be used appropriately so that each process can run effectively. Therefore, integrated memory 234 is not intended to show how specific memory segments are allocated (unless otherwise noted), but rather to provide a general overview of the memory accessible to the computer system 200 and how it is used.

[0077] As shown in Figure 2B, the processor 205 comprises several functional modules, including a control unit 239, an arithmetic unit (ALU) 240, and local or internal memory 248 (sometimes called cache memory). The cache memory 248 typically includes several storage registers 244-246 in a register section. One or more internal buses 241 functionally interconnect these functional modules. The processor 205 also typically includes one or more interfaces 242 for communicating with external devices via the system bus 204 through connections 218. Memory 234 is connected to the bus 204 using connection 219.

[0078] The application program 233 includes a set of instructions 231, which may include conditional branch instructions and loop instructions. The program 233 may also include data 232 used to execute the program 233. The instructions 231 and data 232 are stored in memory locations 228, 229, 230 and 235, 236, and 237, respectively. Depending on the relative size of the instructions 231 and memory locations 228-230, a particular instruction may be stored in a single memory location, as indicated by memory location 230. Alternatively, the instruction may be divided into multiple parts, each part stored in a separate memory location, as indicated by the instruction segments shown in memory locations 228 and 229.

[0079] Generally, the processor 205 receives a set of instructions to be executed internally. The processor 205 waits for the next input and, in response to that input, executes another set of instructions. Each input may come from one or more of a number of sources, including data generated by one or more input devices 202, 203, data received from an external source via either network 220, 202, data obtained from either storage device 206, 209, and data obtained from a storage medium 225 inserted into the corresponding reader 212, as shown in Figure 2A. Execution of an instruction set may, in some cases, result in the output of data. Execution may also involve saving data or variables to memory 234.

[0080] The PCA encoder 160, the PCA decoder 170, and the described method may use input variables 254 stored in corresponding memory locations 255, 256, and 257 in memory 234. The PCA encoder 160, the PCA decoder 170, and the described method generate output variables 261, which are stored in corresponding memory locations 262, 263, and 264 in memory 234. Intermediate variables 258 may be stored in memory locations 259, 260, 266, and 267.

[0081] Referring to processor 205 in Figure 2B, registers 244, 245, 246, arithmetic logic unit (ALU) 240, and control unit 239 work together to perform a series of microoperations necessary to execute a “fetch, decode, execute” cycle for each instruction in the instruction set that makes up program 233. Each fetch, decode, execute cycle includes: A fetch operation to fetch or read instruction 231 from memory locations 228, 229, and 230; Decoding operation in which the control unit 239 determines which instruction has been fetched; An execution operation in which the control unit 239 and / or ALU 240 execute instructions.

[0082] Subsequently, further fetch, decrypt, and execute cycles for the next instruction may be performed. Similarly, a store cycle may be performed in which the control unit 239 stores or writes a value to memory location 232.

[0083] Each step or subprocess of the method shown in Figures 16 and 17 is associated with one or more segments of program 233, and typically register sections 244, 245, 246, ALU 240, and control unit 239 in processor 205 work together to perform a fetch, decode, and execute cycle for all instructions in the instruction set for that segment of program 233.

[0084] Figure 3A is a schematic block diagram 300 showing the functional modules of the CNN backbone 310, which can function as an implementation of the CNN backbone 114. The backbone 114 is sometimes called "DarkNet-53," but different backbones are possible, resulting in different numbers of layers and dimensionality of tensor 115 in each frame.

[0085] As shown in Figure 3A, the video data 113 is passed to the resizer module 304. The resizer module 304 resizes each frame of the video data 113 to a resolution suitable for processing by the CNN backbone 310, generating resized frame data 312. If the resolution of the video data 113 is already suitable for the CNN backbone 310, the operation of the resizer module 304 is unnecessary. The resized frame data 312 is passed to the convolutional batch normalized leaky rectified linear (CBL) module 314, which generates a tensor 316. The CBL module 314 includes the module described with respect to the CBL module 360 ​​shown in Figure 3D.

[0086] The CBL module 360 ​​takes a tensor 361 of the resized frame data 312 as input. Tensor 361 is passed to the convolutional layer 362 to generate tensor 363. When the stride of the convolutional layer 362 is 1, tensor 363 has the same spatial dimensions as tensor 361. When the stride of the convolutional layer 362 is large, such as 2, the spatial dimensions of tensor 363 become smaller than those of tensor 361; for example, with a stride of 2, the width and height are halved. Regardless of the stride, the size of the channel dimensions of tensor 363 may differ from the channel dimensions of tensor 361 in a particular CBL block. Tensor 363 is passed to the batch normalization module 364 to output tensor 365. The batch normalization module 364 normalizes the input tensor 363 and applies scaling coefficients and offset values ​​to generate output tensor 365. The scaling coefficients and offset values ​​are derived from the learning process. Tensor 365 is passed to the leaky rectified linear activation ("LeakyReLU") module 366, which generates tensor 367. Module 366 provides a "leaky" activation function, where positive values ​​in the tensor pass through unchanged, while negative values ​​are significantly reduced in magnitude (e.g., to 0.1 times their original value).

[0087] Returning to Figure 3A, tensor 316 is passed from CBL block 314 to residual block module 320 (res1+2+8 module, also known as res11 module), which is a contiguous arrangement of three residual blocks, each containing 1 residual unit, 2 residual units, and 8 residual units, respectively. Within each residual block, in CBL block 344, which will be explained with reference to Figure 3B, the spatial resolution of the tensor is reduced by half in both the horizontal and vertical directions by a stride 2 convolution.

[0088] The residual block is described with reference to ResBlock340 shown in Figure 3B. ResBlock340 receives tensor 341. Tensor 341 is zero-padding by the zero-padding module 342 to produce tensor 343. Tensor 343 is passed to the CBL module 344 to produce tensor 345. The CBL module 344 includes a convolution (e.g., 362) with a stride parameter set to 2, resulting in tensor 345 having half the width and height of tensor 343. Tensor 345 is passed to the residual unit 346. The residual unit 346 contains a series of residual units concatenated based on the number of residual blocks (e.g., 11 units for block 320). The last unit of residual unit 346 outputs tensor 347.

[0089] The residual unit is described with reference to the ResUnit350 shown in Figure 3C. The ResUnit350 takes tensor 351 as input. Tensor 351 is passed to the CBL module 352 to generate tensor 353. Tensor 353 is passed to a second CBL unit 354 to generate tensor 355. The adder module 356 adds tensor 355 and tensor 351 to generate tensor 357. The adder module 356 is also called a “shortcut” because the input tensor 351 substantially influences the output tensor 357. In an untrained network, the ResUnit350 acts as a pass-through for the tensor. As training takes place, CBL modules 352 and 354 work to deviate tensor 357 from tensor 351 based on the training data and true data.

[0090] Returning to Figure 3A, the Res11 module 320 outputs tensor 322. Tensor 322 is output from the backbone module 310 as one of the layers and simultaneously supplied to the Res8 module 324. The Res8 module 324 is a residual block (i.e., 340) and contains eight residual units (i.e., 350). The Res8 module 324 generates tensor 326. Tensor 326 is passed to the Res4 module 328 and output from the backbone module 310 as one of the layers. The Res4 module is a residual block (i.e., 340) and contains four residual units (i.e., 350). The Res4 module 328 generates tensor 329. Tensor 329 is output from the backbone module 310 as one of the layers. The layer tensors 322, 326, and 329 are collectively output as tensor 115a and may be referred to as layers 0-2 or L0, L1, and L2, respectively. The backbone CNN310 takes a video frame with a resolution of 1088×608 as input and generates three tensors corresponding to three layers, with the following dimensions: [1,256,76,136], [1,512,38,68], and [1,1024,19,34]. Another example of the three tensors 115a corresponding to the three layers is [1,512,34,19], [1,256,68,38], and [1,128,136,76], which are separated at layer indices 75, 90, and 105 respectively, when the layers are numbered according to the YOLOv3 software implementation of the backbone 300 and head 1200.

[0091] Res11 320, Res8 324, and Res4 328 each perform the same function as ResBlock340. CBL314, CBL344, and CBL354 each perform the same function as CBL360.

[0092] Figure 4 is a schematic block diagram showing a functional module of an alternative backbone portion 400 of a CNN that can function as an implementation of the CNN backbone 114. The backbone portion 400 implements a residual network ('ResNetFPN') with a feature pyramid network, which is used in networks such as FasterRCNN and MaskRCNN. Frame data 113 is input and passes through tensors 409, 413, 417, 421, and 425, respectively, to the stem network 408, res2 module 412, res3 module 416, res4 module 420, and res5 module 424.

[0093] The stem network 408 includes a 7x7 convolution with a stride of 2 and a maximum pooling operation. The res2 module 412, res3 module 416, res4 module 420, and res5 module 424 perform convolution operations such as LeakyReLU activation. Each module 412, 416, 420, and 424 also performs a single operation to halve the width and height of the processed tensor via a stride setting of 2. Tensors 413, 417, 421, and 425 are passed to one of the 1x1 transverse convolution modules 446, 444, 442, and 440, respectively. Modules 446, 444, 442, and 440 produce tensors 447, 445, 443, and 441, respectively. Tensor 441 is passed to the 3x3 output convolution module 470 to produce the output tensor P5 471.

[0094] Tensor 441 is also passed to the upsampling module 450, which generates the upsampled tensor 451. The summing module 460 adds tensors 443 and 451 to generate tensor 461. Tensor 461 is passed to the upsampling module 452 and the 3x3 horizontal convolution module 472. Module 472 outputs the P4 tensor 473. The upsampling module 452 generates the upsampled tensor 453. The addition module 462 adds tensors 445 and 453 to generate tensor 463. Tensor 463 is passed to the 3x3 horizontal convolution module 474 and the upsampling module 454. Module 474 outputs the P3 tensor 475. The upsampling module 454 outputs the upsampled tensor 455. The additive module 464 adds tensors 447 and 455 to generate tensor 465, which is then passed to the 3x3 horizontal convolution module 476. Module 476 outputs the P2 tensor 477. The upsampling modules 450, 452, and 454 use nearest neighbor interpolation to reduce computational complexity. Tensors 471, 473, 475, and 477 form the output tensor 115a of the CNN backbone 400. Figure 4 shows a specific backbone portion ("P-layer partition point") of the FasterRCNN network architecture, but different partitioning methods for the backbone and head are possible. Partitioning the network at tensor 409 is called the "stem" partition point. Partitioning the network at tensors 447, 445, 443, and 441 is called the "C-layer" partition point.

[0095] Figure 5 is a schematic block diagram 500 showing one type of multiscale feature fusion (MSFC) module 500 that can function as a tensor synthesizer 162. Figure 6 is a schematic block diagram 600 showing a tensor encoder based on inter-channel uncorrelatedness. Figure 7 is a schematic block diagram 700 showing a subpicture encoder. Encoders 600 and 700 constitute the PCA encoder 160 of system 100. Figures 5 to 7 are explained with reference to Figure 16.

[0096] Various methods exist to find low-dimensional spaces for encoding tensors, and these consist of sets of the tensor's "principal components." Existing principal component analysis (PCA) methods are complex. On the other hand, a method called "eigenvalue decomposition" (EVD) is available, but this is only suitable for square matrix input data. To reveal channel-specific redundancy within a tensor, the tensor is first reconstructed to be presented in [sample, channel] dimensions, and PCA is performed. One of the challenges that arise when performing PCA is the applicability of the basis vectors and the need to generate new basis vectors over time as new data becomes available. In a sense, the most true basis vectors are those derived from the entire dataset. In video sequences, it is usually not practical to buffer the entire sequence tensor before further compression. An "incremental PCA" method has been derived that allows the basis vectors to be updated to reflect new data. It should be noted that since the basis vectors are referenced to the origin zero, new data also needs to be zero-centered, which avoids bias in the basis vectors caused by non-zero offsets in "incremental PCA." The average may fluctuate with the arrival of new data. There are methods to incrementally update the eigenvectors obtained from the EVD, which corrects the moving average.

[0097] In video sequences, the inventors' experiments have shown that the channel-by-channel mean of the input tensor is remarkably stable across many frames in a given sequence. This stability of the tensor mean over a wide time range suggests that the mean shift in new tensors is relatively small and therefore negligible, thus avoiding the need for mean correction or periodic mean updates. The BCHW (batch, channel, height, width) dimension tensor, set up for channel-by-channel PCA, is reshaped to AC (area, i.e., width × height × channel) dimension. The AC dimension is non-square and suitable for techniques such as Singular Value Decomposition (SVD). While incremental SVD methods exist, such as the Lanczos method, some require a fixed mean for new data, or significantly increase complexity to accommodate changes in the data mean. However, instead of introducing incremental decomposition methods that obtain basis vectors from multiple frames, it is possible to use non-incremental SVD methods while retaining the mean calculated from different tensors of the initial frames. The use of mean-preserving non-incremental SVD differs from the fixed-mean approach in that the basis vectors in the second SVD run are derived by assuming that the mean of the second tensor is substantially similar to the mean of the first SVD run. Except for reusing the mean of the first SVD run from the first tensor, the first and second SVD runs are independent and operate on different tensors from different video frames. Modern SVD algorithms generate a three-diagonal matrix (a square matrix with non-zero values ​​only on the diagonals and adjacent cells) from the input matrix using numerically stable Householder transforms, and the Lanczos method is rarely used due to instability issues.

[0098] Figure 16 shows a method 1600 for executing the first part of a CNN and encoding the resulting feature map of a video data frame. In encoding the feature map, tensors are encoded into a bitstream. Method 1600 can be implemented using devices such as configured FPGAs, ASICs, and ASSPs. Alternatively, as described later, method 1600 may be implemented by the source device 110 as one or more software code modules of an application program 233 under the execution of a processor 205. The software code modules of the application program 233 implementing method 1600 may reside, for example, on a hard disk drive 210 and / or memory 206. Method 1600 is executed for each frame of video data generated by the video source 112. Method 1600 may be stored on a computer-readable storage medium and / or in memory 206. Method 1600 begins at step 1605 of the execution of the first part of the CNN.

[0099] In step 1605, the CNN backbone 114, under the execution of processor 205, executes a subset of the layers of a particular CNN, converting the input frame 113 into an intermediate tensor 115a. The intermediate tensor 115a is stored, for example, in memory 206 and / or hard disk drive 210. For example, the CNN is "FasterR-CNN" or "MaskR-CNN," which is a subset of layers corresponding to all layers up to the "P layer" split point, as shown in Figure 4. If multiple tensors are extracted from the CNN backbone 114, for example, by using an FPN, tensor 115a will contain one tensor for each layer of the FPN. Control of processor 205 proceeds from step 1605 to step 1610, which is the execution of tensor reduction.

[0100] In step 1610, the tensor synthesizer 162 operates under the execution of the processor 205, resampling individual tensors from the group of tensors 115a to produce a single synthesized tensor 115 as output. The multiscale feature fusion (MSFF) module 500 includes the MSFF block 510 shown in Figure 5, which generates a single tensor from multiple tensors 115a using one or more downsampling filters. Under the execution of the processor 205, the MSFF block 510 synthesizes each tensor of the first group of tensors (i.e., 502, 503, 504, 505) to produce a synthesized tensor 115a. The synthesized tensor 115a forms a representation of the FPN layer tensor. The downsampling modules 522a, 522b, and 522c process the data using tensors with larger spatial scales, namely P4 504 with dimensions "2h, 2w, 256", P3 503 with dimensions "4h, 4w, 256", and P2 502 with dimensions "8h, 8w, 256", respectively. Modules 522a, 522b, and 522c perform downsampling to match the spatial scale of the smallest tensor (i.e., P5 505 with dimensions "h, w, 256"), generating downscaled P5 tensors 523a, 523b, and 523c, respectively. The concatenation module 524 concatenates tensors 505, 523a, 523b, and 523c on a channel-by-channel basis to generate a concatenated tensor 525 with dimensions "h, w, 1024". The connected tensor 525 is passed to the squeeze exercise (SE) module 526, which generates tensor 527. The SE module 526 sequentially executes global pooling, a fully connected layer that reduces the number of channels, a rectifying linear activation unit, a second fully connected layer that restores the number of channels, and a sigmoid activation function to generate a scaling tensor. Tensor 525 is scaled according to the scaling tensor, producing tensor 527 as the output. The SE block 526 can learn to adaptively change the weighting of different channels in the tensor it passes through, based on the output of the first fully connected layer. The output of the first fully connected layer reduces the feature map of each channel to a single value.Each single value passes through a nonlinear activation unit (ReLU) to generate a conditional representation suitable for weighting other channels, and the reconstruction of the total number of channels is performed by a second fully connected layer. Thus, SE block 526 can extract nonlinear inter-channel correlations in the process of generating tensor 527 from tensor 525 to a degree that would not be possible with convolutional (linear) layers alone. Tensor 527 is passed to convolutional layer 528. Convolutional layer 528 implements one or more convolutional layers and generates a composite tensor 115 with the number of channels reduced to F channels (typically 256 channels, i.e., F=256). Control within processor 205 proceeds from step 1610 to the mean update decision step 1620.

[0101] In step 1620, the source device 110, under the execution of processor 205, determines whether a new channel mean should be determined for tensor 115. The channel mean provides a per-channel offset to generate a version of tensor 115 that is centered at zero along the channel axes. Removing the mean from the tensor is necessary to use a zero-centered tensor in PCA in the dimension where dimensionality reduction is performed. For example, dimensionality reduction along the channel dimension reduces the number of channels in the input tensor (e.g., 256) to a smaller number of channels (e.g., 64). In other words, dimensionality reduction is the process of generating a set of vectors ("components" or "basis vectors") smaller than the set of input vectors, which can be used most efficiently to reconstruct the original vectors. The input tensor is represented irreversibly as a weighted sum of the basis vectors for each channel. Each basis vector is selected to encode the amount of variance remaining in the set of input vectors to the greatest extent possible. The basis vectors generated from PCA are implicitly zero-centered. Therefore, when a non-zero-centered input is fed to PCA, the dimensionality reduction operation cannot sequentially and appropriately extract a vector representing the maximum residual explanatory variance of the input tensor.

[0102] Experiments by the inventors confirmed that the mean of tensor 115 is extremely stable across different frames of a video sequence (i.e., video data 113). Therefore, the mean of a given frame can be reused for at least one subsequent frame. Across frames of different video sequences, the inter-frame stability of the mean is not as pronounced, suggesting that the mean is partly data-driven and partly a network characteristic. Therefore, in longer video sequences, the mean is expected to change over time based on the statistics of the input tensor 115. The mean update decision step 1620 may result in calculating a new channel mean at a lower frequency than the frequency of performing the decomposition process to determine the new basis vectors. One example of reducing the mean update frequency is to perform decomposition for each intra-frame in a random access picture structure, for example, performing decomposition at a frequency of about once per second and updating the mean every Nth (N>1) intra-frame. Another example is to calculate the mean only for the first frame and reuse that mean for the rest of the sequence. Performing average updates only on the first frame is the minimum amount of averaging calculation and is more suitable for short video sequences where the statistics of the tensor 115 between frames do not change significantly throughout the duration of the video sequence. The processor 205 controls the process from step 1620 to step 1630, which involves encoding the average update flag.

[0103] In step 1630, the entropy encoder 838 (see Figure 8), under the execution of processor 205, encodes the average update flag 1591 (see Figure 15). Flag 1591 indicates what was decided in step 1620, namely whether an average exists for decoding (e.g., video frames). In other words, step 1630 encodes the decision made in step 1620—whether to update (e.g., set the average update flag value to 1) or not to update (e.g., set the average update flag value to 0)—into bitstream 121. Control of processor 205 proceeds from step 1630 to the average update test step 1640.

[0104] Step 1640 determines whether an average update is necessary. If it is determined in step 1620 that an average update should be performed, step 1640 returns YES, and control of the processor 205 proceeds from step 1640 to the average channel determination step 1650. On the other hand, if it is determined in step 1620 that an average update should not be performed, step 1640 returns NO, and control of the processor 205 proceeds from step 1640 to the average acquisition step 1690.

[0105] In the average channel determination step 1650, the tensor 115 is averaged. Referring to Figure 6, module 610, under the control of processor 205, performs an averaging operation on tensor 115 across the entire spatial dimension to generate the average 611 per channel. The average channels 611 are quantized by quantizer 612, under the control of processor 205, to generate an integer (quantized) average channel list 613 in step 1650 according to a quantization range that indicates the floating-point range required to hold the average channels 611. The quantization range indicates the extreme minimum and maximum values ​​between the average channels 611 and the integer average channel list 613 that form linear quantized values ​​across the quantization range. Control of processor 205 proceeds from step 1650 to the average channel coding step 1660.

[0106] In the average channel coding step 1660, the subpicture encoder 614, under the execution of the processor 205, packs the integer average channels 613 into a subpicture (for example, as subpicture 1410 shown in Figure 14), and uses the subpicture encoder 614 to encode the subpicture and generate the bitstream portion 615. Since the average channels 613 contain one value per channel, when representing the average of a tensor of 256 channels, it can be treated as a feature map, for example, with a height of 1 and a width of 256. In the execution of step 1660, the average channels 613 are reshaped to a size suitable for block coding. For example, the 256 channels are coded as a 16x16 or 64x4 array of averages and stored as 1420 as shown in Figure 14, so that the VVC coding tool can code the average channels 613 more efficiently by utilizing the sample array aligned to a selectable block structure supported by VVC. The average channels 1420 exist in the bitstream 121 only when the average update flag 1591 indicates an average update.

[0107] The subpicture encoder 614 implements architecture 700, which includes a feature map packer 710, a video frame encoder 714, and an unpacker 720. Subpicture encoders 614, 636, and 654 are implemented as instances of architecture 700. The packer 710 accepts a tensor 708 having a given number of channels, width, and height dimensions, and containing integer values ​​(i.e., already quantized). For example, subpicture encoder 614 accepts an integer mean channel 613. The packer 710 packs the received tensor into a two-dimensional planar array of samples. Generally, in the configuration described, the feature maps for each channel of the tensor 708 are stored as subpicture frames 712, in left-to-right, top-to-bottom order. The subpicture frame 712 must have sufficient area to accommodate the channels of the tensor 708, taking into account the width and height of the feature map and any packing gaps due to mismatches with the dimensions of the subpicture frame 712. The operation of the video frame encoder 714 (generally implemented as a VVC encoder) is described with reference to Figure 8.

[0108] The encoder 714 generates an encoded bitstream portion 716 corresponding to the corresponding subpicture. For example, the subpicture encoder 614 outputs an encoded bitstream portion 615. The encoder 714 also outputs a reconstructed frame 718 corresponding to the lossy version of the bitstream portion 716 that is reconstructed during decoding. The reconstructed frame 718 represents the reconstruction of the average channel 613 and models or represents the loss caused by encoding. These losses reflect the encoding loss that would occur due to the specific encoding scheme used by the encoder 714. In the example described in Figures 7 and 8, the encoder 714 is a VVC encoder. The reconstructed frame 718 is passed to the unpacker 720. The unpacker 720 extracts the feature maps and generates a reconstructed tensor 722 with the same dimensions as tensor 708, which is formed by concatenating the unpacked feature maps channel by channel. For example, the subpicture encoder 614 outputs tensor 616.

[0109] As a result of the operation of module 700, PCA encoder 160 can utilize a version of the feature map (or coefficients) that corresponds to the version observed by PCA decoder 170. Therefore, PCA encoder 160 can operate at a higher fidelity level than without considering the effects of lossy encoding. Subpicture encoders 614, 636, and 654 are configured to disable loop filtering internally and between subpicture boundaries, because loop filtering is generally optimized for human viewing of decoded pictures. Furthermore, motion compensation intended for accessing samples across subpicture boundaries is prohibited via the "sps_subpic_treated_as_pic_flag" flag activated for each subpicture. System 100 uses video compression to relatively efficiently represent the data resulting from dimensionality reduction performed on intermediate tensor data that needs to be propagated from the CNN backbone 114 to the CNN head 150.

[0110] Returning to Figure 16, the control of the processor 205 proceeds from step 1660 to the reconstruction average channel restoration step 1670.

[0111] In the reconstructed mean channel recovery step 1670, the subpicture encoder 614, under the execution of processor 205, outputs a reconstructed picture (e.g., 718) as a lossy version corresponding to the subpicture input (e.g., 722) for video compression within the implementation of encoder 700. The reconstructed picture 718 is decompressed and output as an integer tensor 616. Control of processor 205 proceeds from step 1670 to mean preservation step 1680.

[0112] In the mean preservation step 1680, the processor 205 executes to store the mean 616 calculated in step 1670 in the mean preservation module 680. The processor 205 then proceeds from step 1680 to the mean removal step 16100.

[0113] In step 1690, the average stored in the average memory module 680 is retrieved from the memory device 206 as the acquired average 681 and passed to the inverse quantizer 622 via the multiplexer 682 as the acquired average 683. The processor 205 continues control from step 1690 to the mean removal step 16100.

[0114] In step 16100, the inverse quantization module 620 and subtraction module 622 operate under the execution of processor 205 to generate a mean zero tensor 623. The inverse quantizer 620 receives an integer tensor 616 via the multiplexer 682 and outputs a reconstructed mean channel 621 using the quantization range determined by the quantization module 612. The reconstructed mean channel 621 is a list of values ​​corresponding to the detected DC offsets corresponding to each channel of tensor 115, with one value for each channel in the tensor. For each channel of the composite tensor 115, a DC shift is performed by subtracting a constant value common to the feature map from each spatial position in the feature map. The value to be subtracted is the corresponding value in the mean reconstructed channel 621. As a result of the subtraction module 622, a center zero tensor 623 is output, with the DC components in each feature map removed. Control of processor 205 proceeds from step 16100 to the feature map subsampling step 16110.

[0115] In step 16110, the subsampling module 630 generates a subsampled tensor 623a from tensor 623 under the execution of processor 205. Spatial subsampling is performed by module 630, reducing the number of samples in the width and height directions. However, subsampling is not performed in the channel dimension direction (see Figure 18). By sampling alternately in the horizontal and vertical directions, a tensor 623a can be generated in which the height and width of tensor 623 are halved. For example, sampling every two feature maps of the tensor in the horizontal and vertical directions can reduce the number of input samples to one-quarter. Other subsampling rates can also be used. When the encoder 600 processes multiple tensors with different spatial resolutions (e.g., tensor 115a to which tensor synthesizer 162 is not applied), applying a larger subsampling rate (e.g., one sample every four samples in both the horizontal and vertical directions) can reduce the amount of data to be decomposed to one-sixteenth. Performing spatial subsampling on tensor 115 before channel-wise decomposition yields basis vectors that are substantially equivalent to those obtained without spatial subsampling. In other words, the accuracy of the basis vectors is somewhat tolerant of the spatial subsampling rate of tensor 115. In some cases, it is possible to subsample different tensors at different rates, and in a group of tensors, one tensor can be sampled at a different rate than others. For example, a larger feature map (e.g., P2) may be sampled less frequently than a smaller feature map (e.g., P3-P5), such as every four samples horizontally and vertically. The tolerance to spatial subsampling comes from the channel-dimensional decomposition, which reveals cross-channel redundancy mainly due to (spatial) convolution and other operations that generate tensor 115, rather than revealing spatial features.

[0116] Control of processor 205 proceeds from step 16110 to basis vector determination step 16120. In basis vector determination step 16120, decomposition module 633 operates under the execution of processor 205 to generate a set of basis vectors for the composite tensor 115. Decomposition module 633 takes a zero-centered subsampled tensor 623a as input and generates a set of basis vectors 631 by performing principal component analysis techniques such as singular value decomposition (SVD). Because tensor 623a is subsampled, the basis vectors are generated using fewer samples than the total number of samples in tensor 623a of the current video frame. Since one basis vector maps all channels to a single value using an inner product (dot) operation, if the number of channels in tensor 115 is 256, the dimension of the basis vectors is 256 × 1. If the decomposition module 633 generates the first N basis vectors (e.g., 25), the dimension of the resulting basis vectors will be 256 × N or 256 × 25. Since the basis vectors are relative to the origin, it is necessary to use the mean-zero tensor 623 so that an orthonormal basis can be found. Each basis vector is a vector that associates all channels with a reduced set of channels. Thus, the set of basis vectors makes it possible to represent tensor data spanning all channels with a smaller set of basis vectors. Each basis vector is derived by considering all samples of each feature map in a given channel. Vector 631 contains fewer basis vectors than the number of channels in tensor 115 and corresponds to the dimensionality reduction of tensor 115. The basis vectors of 631 represent tensor 115 in a subspace that explains (explains) the variance of tensor 115 to the greatest extent possible in terms of the number of components of basis vector 631. The basis vectors are ordered from the vector with the largest explained variance to the vector with the smallest explained variance. In other words, basis vector 631 makes it possible to represent tensor 115 with minimal quality degradation for a given number of components (where a component refers to the first N basis vectors). Control of processor 205 proceeds from step 16120 to basis vector coding step 16130.

[0117] In the basis vector coding step 16130, the quantization module 632 operates under the control of the processor 205 to quantize the basis vector 631 into the integer domain. The resulting integer basis vector 634 is packed into a subpicture by the subpicture encoder 636 and encoded under the control of the processor 205 to generate the bitstream portion 637. Control of the processor 205 then proceeds from step 16130 to the reconstruction basis vector restoration step 16140.

[0118] In step 16140, the subpicture encoder 636 generates a reconstructed integer tensor 638 under the operation of the processor 205, thereby obtaining a reconstructed version (e.g., 638) of the subpicture 1412. The basis vector 638 is unpacked from the reconstructed subpicture by the inverse quantizer 660 under the operation of the processor 205, inversely quantized, and returned to the floating-point region (as the reconstructed basis vector 640). Control of the processor 205 proceeds from step 16140 to the coefficient determination step 16150.

[0119] In step 16150, under the control of processor 205, dot product module 642 performs a dot product between each channel of tensor 623 and each vector of the reconstructed basis vector 640 to generate coefficient tensor 644. The coefficients form tensor 644, which has the same width and height as tensor 115, but has a number of channels corresponding to the number of components (or basis vectors) generated by decomposition module 630 (which is less than the number of channels in tensor 115). Each value of coefficient tensor 644 represents the degree to which each basis vector contributes to reproducing each value of each feature map. Control of processor 205 proceeds from step 16150 to coefficient quantization step 16160.

[0120] In step 16160, the quantization module 646, under the execution of processor 205, quantizes the coefficient tensor 644 to produce integer coefficients 662, which are tensors with the same dimensionality as the coefficient tensor 644, i.e., c channels, w width, and h height, where c corresponds to the number of basis vectors. The quantization range is determined from the extrema of the coefficient tensor 644. Control of processor 205 proceeds from step 16160 to step 16170, which assigns the coefficients to groups.

[0121] In the execution of steps 16110 to 16170, method 1600 operates to generate coefficients for a tensor using the set of (reduced) tensors and basis vectors generated in step 1610. The tensor from which the coefficients are generated has the same spatial size as the tensor generated in step 1610, but with fewer channels. In step 16170, the resulting coefficients 662 are provided to the subpicture encoder 654 to generate the bitstream portion 652. Control of the processor 205 proceeds from step 16170 to the bitstream writing step 16180.

[0122] In step 16180, bitstream portion 652 is written to bitstream 121 via subpicture combiner 690.

[0123] Upon completion of step 16180, method 1600 terminates for the current frame 113 from video source 112.

[0124] In one embodiment of Method 1600, the subsampling feature map processing step 16110 is omitted, as indicated by the dashed line. In the embodiment where step 16110 is omitted, Method 1600 proceeds directly from step 16100 to step 16120. When the subsampling operation of step 16110 is omitted, the tensor with the full feature map spatial resolution (i.e., 623) is passed directly to the decomposition module 633 as tensor 623a. Performing decomposition on tensor 623 yields basis vectors with slightly improved fidelity and increased complexity compared to performing decomposition on a subsampled tensor.

[0125] Figure 8 is a schematic block diagram 800 showing the functional modules of the video encoder 714. The video encoder 714 encodes one sub-picture from a set of sub-pictures that make up the whole picture. Although it is possible to encode all sub-pictures in a single encoding pass, the PCA encoder 160 cannot consider lossy encoding in the pipeline if a lossy decoded version of the data in a particular sub-picture is generated in a single encoding pass, as this would make it unusable as input when generating data to be encoded in another sub-picture. Generally, between functional modules in the video encoder 714, data is passed as samples or groups of coefficients (e.g., a block divided into fixed-size sub-blocks) or as an array. The video encoder 714 may be implemented using a general-purpose computer system 200, as shown in Figures 2A and 2B, in which case the various functional modules may be implemented by dedicated hardware within the computer system 200, or by software executable within the computer system 200, such as one or more software code modules of a software application program 233 that resides on a hard disk drive 205 and whose execution is controlled by the processor 205. Alternatively, the video encoder 714 may be implemented by a combination of dedicated hardware and software executable within the computer system 200. The video encoder 714 and the described method may also be implemented by dedicated hardware, such as one or more integrated circuits, that perform the functions or subfunctions of the described method. Such dedicated hardware may include graphics processing units (GPUs), digital signal processors (DSPs), application-specific standard products (ASSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or one or more microprocessors and associated memory. In particular, the video encoder 714 includes modules 810-890, which may each be implemented as one or more software code modules of the software application program 233.

[0126] The video encoder 800 in Figure 8 is an example of a general-purpose video encoding (VVC) video encoder, but other video codecs can also be used to perform the processing steps described here. For example, HEVC may be used. The described example generates a bitstream of encoded data. If other codecs are used, some implementations may pack the data into a different format, such as a frame format. The video encoder 800 receives subpicture frame data 712 (e.g., a sequence of frames of subpictures, each containing one or more color channels). The frame data 712 can be any chroma format and bit depth supported by the profile in use (e.g., 4:0:0, 4:2:0 in the "Main10" profile of the VVC standard), and the sample precision may range from 8 bits to 10 bits. The block divider 810 first divides the frame data 712 into CTUs, which are generally square and configured to use a specific size. The maximum effective size of a CTU is, for example, 32x32, 64x64, or 128x128 luma samples, which is set by the "sps_log2_ctu_size_minus5" syntax element in the "sequence parameter set". The CTU size also specifies the maximum CU size, since a CTU that is not further divided contains one CU. The block divider 810 further divides each CTU into one or more CBs according to the luma coding tree and chroma coding tree. Luma channels are also called primary color channels. Each chroma channel is also called a secondary color channel. CBs can be of various sizes and may include square and non-square aspect ratios. However, in the VVC standard, the side lengths of CBs, CUs, PUs, and TUs are always powers of 2. Thus, according to the luma coding tree and chroma coding tree of the CTU, the current CB (represented by 812) is output from the block divider 810 along an iteration over one or more blocks within the CTU.

[0127] The CTUs obtained from the first split of frame data 712 may be scanned in raster scan order and grouped into one or more “slice.” A slice may be an “intra” (or “I”) slice. An intra slice (I slice) indicates that all CUs in the slice are intra-predicted. Generally, the first picture of a coded layer video sequence (CLVS) contains only I slices and is called an “intra picture.” A CLVS may contain periodic intra pictures, which form “random access points” (i.e., intermediate frames in the video sequence from which decoding can begin). Alternatively, a slice may be unidirectional or bidirectional predictor (a “P” slice or a “B” slice, respectively), indicating that unidirectional and bidirectional predictors are additionally available within the slice.

[0128] The video encoder 714 encodes the picture sequence according to a picture structure. One picture structure is "low latency," in which case a picture using interpretation can only reference pictures that have occurred previously in the sequence. Low latency means that each picture is saved for the possibility of being referenced by subsequent pictures, and is output immediately after decoding. Another picture structure is "random access," where the encoding order of pictures is different from the display order. In random access, interpretation-predicted pictures can reference other pictures that have been decoded but not yet output. Some picture buffering is required so that future referenced pictures in the display order are present in the decoded picture buffer, which results in a delay of several frames.

[0129] When using chroma formats other than 4:0:0, in an I-slice, the coding tree of each CTU branches at a level of 64x64 or less, resulting in two independent coding trees: one for luma and one for chroma. The use of independent trees allows for different block structures to exist between luma and chroma within the 64x64 luma region of a CTU. For example, a large chroma CB may be co-located with many small luma CBs, and vice versa. In P or B-slices, a single coding tree of a CTU defines a common block structure for luma and chroma. Blocks generated from this single tree can be either intra-predictive or inter-predictive.

[0130] In addition to splitting a picture into slices, it is also possible to split a picture into "tiles." A tile refers to a continuous sequence of CTUs that cover a rectangular area of ​​the picture. CTU scanning is performed within each tile using a raster scanning method, progressing sequentially from tile to tile. Slices are either integer multiples of a tile, or integer multiples of the continuous row of CTUs within a particular tile.

[0131] For each CTU, the video encoder 714(800) operates in two stages. In the first stage (called the “search” stage), the block partitioner 810 tests various latent configurations of the coding tree. Each latent configuration of the coding tree has an associated “candidate” CB. In the first stage, various candidate CBs are tested and a CB that provides relatively high compression efficiency and relatively low distortion is selected. The tests generally involve Lagrangian optimization, and candidate CBs are evaluated based on a weighted combination of rate (i.e., coding cost) and distortion (i.e., error relative to the input frame data 712). The “best” candidate CB (i.e., the CB with the smallest evaluation rate / distortion) is selected for encoding to the subsequent bitstream portion 716. The evaluation of candidate CBs includes the choice of using a single CB for a particular region, further dividing the region according to various partitioning options and encoding each resulting subregion with a different CB, or further subdividing the region. As a result, both the coding tree and the CBs themselves are selected in the search stage.

[0132] The video encoder 714 generates a prediction block (PB) for each CB (e.g., CB812), indicated by arrow 820. PB820 is a prediction of the content of the associated CB712. The subtraction module 822 generates a difference (also called a "residual" as it is a difference in the spatial domain) 824 between PB820 and CB812. The difference 824 is the difference in block size between the corresponding samples of PB820 and CB812. The difference 824 is transformed and quantized and represented as a transformation block (TB), indicated by arrow 836. PB820 and its associated TB836 are typically selected from many candidate CBs based on evaluated costs and distortions.

[0133] A candidate coded block (CB) is a CB obtained from one of the prediction modes available to the video encoder 714 for the associated PB and the resulting residual. Combined with the predicted PB within the video encoder 714, the TB836 reduces the difference between the decoded CB and the original CB812 at the expense of additional signaling in the bitstream.

[0134] Therefore, each candidate coding block (CB), i.e., a combination of a prediction block (PB) and a transform block (TB), has an associated coding cost (or "rate") and an associated difference (or "distortion"). The distortion of a CB is typically estimated as a difference in sample values, such as the sum of absolute differences (SAD), the sum of squared differences (SSD), or the Hadamard transform on the difference. The estimate obtained from each candidate PB is determined by a mode selector 886 that uses the difference 824 to determine the prediction mode 887. The prediction mode 887 indicates the decision to use a particular prediction mode (e.g., intra-frame prediction or inter-frame prediction) for the current CB. Estimating the coding cost associated with each candidate prediction mode and the corresponding residual coding can be performed at a significantly lower cost than entropy coding of the residuals. Therefore, even in real-time video encoders, it is possible to evaluate multiple candidate modes and determine the optimal mode in terms of rate distortion.

[0135] To determine the optimal mode in terms of rate distortion, a variant of Lagrangian optimization is typically used. Lagrangian optimization or a similar optimization process can be applied to both the selection of the optimal partition (by the block partitioner 810) for dividing the CTU into CBs, and the selection of the best prediction mode from several possibilities. By applying the Lagrangian optimization process to the candidate modes in the mode selection module 886, the intra-prediction mode with the smallest cost measurement is selected as the "optimal" mode. This smallest-cost mode includes the selected quadratic transformation index 888, which is also encoded into bitstream 716 by the entropy encoder 838.

[0136] In the second stage of operation of the video encoder 714 (called the "encoding" stage), iterative processing is performed within the video encoder 714 on the determined encoding tree for each CTU. For CTUs using separate trees, for each 64x64 luma region within the CTU, the luma encoding tree is encoded first, followed by the chroma encoding tree. Only luma code blocks (CBs) are encoded within the luma encoding tree, and only chroma code blocks are encoded within the chroma encoding tree. For CTUs using a shared tree, a single tree describes Cus (luma CBs and chroma CBs) according to the common block structure of the shared tree.

[0137] The entropy encoder 838 supports bitwise coding of syntactic elements using variable-length and fixed-length codewords, as well as arithmetic coding modes for syntactic elements. Parts of the bitstream (e.g., "parameter sets" such as sequence parameter sets (SPS), picture parameter sets (PPS), and picture headers (PH)) use a combination of fixed-length and variable-length codewords. A slice (also called a contiguous region) consists of a slice header using variable-length coding and slice data using arithmetic coding. The slice header defines parameters specific to the current slice, such as slice-level quantization parameter offsets, and may contain instances of PHs. The slice data contains the syntactic elements of each CTU within the slice. The use of variable-length and arithmetic coding requires sequential parsing within each part of the bitstream. Each part is delimited by a start code, forming a "Network Abstraction Layer Unit (NAL unit)". Arithmetic coding is supported using a context-adaptive binary arithmetic coding process.

[0138] Arithmetic-encoded syntactic elements consist of a sequence of one or more "bins." A bin, like a bit, has a value of either "0" or "1," except that bins are not encoded as discrete bits in bitstream portion 716. Each bin is associated with a predicted value (or "most likely value," "highest probability value") and its associated probability (called context). If the actual bin to be encoded matches the predicted value, the "highest probability symbol" (MPS) is encoded. Encoding the highest probability symbol is relatively low-cost in terms of the number of bits consumed in bitstream portion 716 (including the cost of less than one discrete bit). If the actual bin to be encoded does not match the predicted value, the "lowest probability symbol" (LPS) is encoded. Encoding the lowest probability symbol is relatively high in terms of the number of bits consumed. Bin coding techniques enable efficient encoding of bins with biased probabilities of "0" and "1." Syntactic elements that take only two values ​​(i.e., "flags") require a single bin. Syntactic elements that take many values ​​require a series of bins. The convention of converting the values ​​of syntactic elements into a set of bins is called "binarization." When the values ​​"0" and "1" in a given bin are equally (or nearly equally) likely to occur, it is possible to omit the use of context and assume an equally probable distribution. Bins with context are called "context-coded bins," and bins without context are called "bypass-coded bins." In the binarization of one or more bins of a syntactic element, a combination of context-coded and bypass-coded bins may occur. Unlike direct 1-bit coding to a bitstream, bypass-coded bins utilize an arithmetic coding engine, allowing for a mix of context-coded and bypass-coded bins in the binarization of syntactic elements.

[0139] In certain binary encoding schemes, the presence of subsequent bins in a sequence may be determined based on the values ​​of preceding bins in the sequence, resulting in variable-length binary encoding. Furthermore, each bin is associated with multiple contexts, and one context is selected to use when encoding a particular instance of a bin. The selection of a particular context may depend on preceding bins within the syntactic element, the decoded values ​​of adjacent syntactic elements (i.e., those from adjacent blocks), etc. Each time a context-encoded bin is encoded, the context selected for that bin (if any) is updated to reflect the new bin value. For this reason, this binary arithmetic encoding scheme is said to be adaptive.

[0140] By omitting the context of bypass coding bins saves memory and reduces complexity, so bypass bins are used when the value distribution of a particular bin is not skewed. CABAC (Context Adaptive Binary Arithmetic Encoder) is a known example of an entropy encoder that employs context and adaptation, and many variations of this encoder are used for video coding.

[0141] The QP control unit 890 determines the quantization parameter 892 used to set the quantization step size used by the quantizer 834 and the inverse quantizer 840. A larger quantization step size results in the principal transformation coefficient 828 being quantized to a smaller value, reducing the bitrate of the bitstream portion 716 at the cost of a decrease in the fidelity of the inverse transformation coefficient 846.

[0142] The entropy encoder 838 encodes the quantization parameter 892 and (if used in the current CB) the LFNST index 888 using a bin that combines context coding and bypass coding. The quantization parameter 892 is encoded at the beginning of each slice, and changes in the quantization parameter 892 within a slice are encoded using a "delta-QP" syntactic element. The delta-QP syntactic element is signaled at most once within each region (quantization group). The quantization parameter 892 is applied to the residual coefficients of the luma CB. The adjusted quantization parameter is applied to the residual coefficients of the chroma CB at the same location. The adjusted quantization parameter may include mappings from the luma quantization parameter 892 based on a mapping table and CU level offsets selected from an offset list. The quadratic transformation index 888 is signaled if the residuals associated with the transformation block contain significant residual coefficients only at coefficient locations where the quadratic transformation is converted to linear coefficients.

[0143] The residual coefficients of the CB associated with each TB are encoded using residual syntax. Residual syntax is designed to efficiently encode low absolute coefficients by indicating the importance of coefficients primarily using arithmetic encoded bins, with low absolute values, and reserving bypass bins for high absolute residual coefficients. Thus, residual blocks consisting of very low absolute values ​​and a sparse distribution of importance coefficients are efficiently compressed. Furthermore, there are two residual coding schemes. The normal residual coding scheme is optimized for TBs where significance coefficients are concentrated in the upper left corner of the TB, as seen when a transformation is applied. The transformation-skip residual coding scheme is provided for TBs where no transformation is performed and can efficiently encode residual coefficients regardless of their distribution across the entire TB.

[0144] The multiplexing module 884 outputs the PB820 from the intra-frame prediction module 864 based on the optimal intra-prediction mode selected from the tested prediction modes of each candidate CB. The candidate prediction modes do not need to include all prediction modes supported by the video encoder 714. Intra-predictions are classified into three types: the first, "DC intra-prediction," fills the PB with a single value representing the mean of neighboring reconstructed samples; the second, "planar intra-prediction," fills the PB with samples on a plane using the DC offset and vertical and horizontal gradients derived from neighboring reconstructed samples. Neighboring reconstructed samples typically include a row of reconstructed samples located above the current PB (extending to a certain range to the right of the PB) and a column of reconstructed samples located to the left of the current PB (extending to a certain range to the bottom of the PB); and the third, "in-angle prediction," fills the PB with neighboring reconstructed samples filtered and propagated across the entire PB at a specific angle (or "angle"). VVC supports 65 angles, and rectangular blocks can utilize additional angles not available with square blocks, generating a total of 87 angles.

[0145] A fourth intra-prediction method is available for chroma PBs, which follows a "cross-component linear model" (CCLM) mode to generate PBs from identically positioned luma reconstructed samples. Three different CCLM modes are available, each using a different model derived from adjacent luma and chroma samples. The derived models are used to generate sample blocks for chroma PBs from identically positioned luma samples. Luma blocks can be intra-predicted by matrix multiplication of a reference sample using a matrix selected from a predefined set of matrices. This matrix intra-prediction (MIP) achieves its gain by using matrices trained on large video datasets, and these matrices represent relationships between reference samples and predicted blocks that are not readily captured by angular, plane, or DC intra-prediction modes.

[0146] Module 864 can also generate prediction units by copying blocks from the vicinity of the current frame using the "Intra-Block Copy" (IBC) method. The location of the reference block is limited to an area equivalent to one CTU (divided into multiple 64x64 areas called VPDUs) within each row or CTU and within each slice or tile, up to the area limit corresponding to 128x128 luma samples, regardless of the CTU size set in the bitstream. This area is called the "IBC virtual buffer" and limits the required memory capacity by limiting the IBC reference area. The IBC buffer stores the reconstructed sample 854 (before loop filtering), thus requiring a separate buffer from the frame buffer 872. If the CTU size is 128x128, the virtual buffer contains only samples from the CTU adjacent to the left of the current CTU. If the CTU size is 32x32 or 64x64, the virtual buffer contains up to four or sixteen CTUs to the left of the current CTU. Regardless of CTU size, access to adjacent CTUs for sampling IBC reference blocks is constrained by picture, slice, and tile boundaries. In particular, for feature maps in smaller-dimensional FPN layers, using CTU sizes such as 32x32 or 64x64 allows for better alignment of the reference region to cover a set of previous feature maps. If the feature map arrangement is ordered based on SAD, SSE, or other difference metrics, accessing similar feature maps for IBC prediction offers advantages in terms of coding efficiency.

[0147] The residuals of prediction blocks when encoding feature map data differ from those seen in natural video. Natural video is typically captured by image sensors or screen content (commonly seen in OS user interfaces, etc.). Feature map residuals tend to contain a lot of detailed information. The level of detail in feature map residuals is more suitable for transformation-skip coding than the low-frequency coefficients in various transformations. Intra-predictive luma coding blocks can be divided into groups of prediction blocks of equal size vertically or horizontally, with each block having a minimum of 16 luma sample regions.

[0148] When previously reconstructed adjacent samples are unavailable, such as at the edge of a frame, a default halftone value equivalent to half the sample range is used. For example, 512 is used for 10-bit video. Since there are no preceding samples at the top-left corner of the frame (CB), the angular and planar intra-prediction modes produce an output equivalent to the orthogonal component prediction mode (i.e., a flat sample plane with the halftone value as the absolute value).

[0149] In interframe prediction, the motion compensation module 880 generates a prediction block 882 using samples from one or two frames prior to the current frame in the encoded order frames of the bitstream, and outputs it as PB820 by the multiplexing module 884. Furthermore, in interframe prediction, a single encoded tree is typically used for both the luminous and chroma channels. The encoding order of frames in the bitstream may differ from the frame order at the time of acquisition or display. When one frame is used for prediction, the block is called a "uni-prediction" and has one associated motion vector. When two frames are used for prediction, the block is called a "bi-prediction" and has two associated motion vectors. In P-slice, each CU can be either an intra-prediction or a uni-prediction. In B-slice, each CU can be either an intra-prediction, a uni-prediction, or a bi-prediction.

[0150] Frames are typically encoded in a "group of pictures" structure, allowing for temporal layering of frames. Frames are divided into multiple slices, each encoding a portion of the frame. Temporal layering allows frames to reference preceding and succeeding pictures in display order. Images are encoded in an order that satisfies the dependencies required for decoding each frame. In affine mutual prediction mode, instead of using one or two motion vectors to select and filter the reference sample block of the prediction unit, the prediction unit is divided into multiple smaller blocks, generating a motion field where each block has its own motion vector. This motion field uses the motion vectors of points adjacent to the prediction unit as "control points." Affine prediction allows for encoding motion other than translation with less use of deeply divided coding trees. The bi-prediction mode available in VVC geometrically blends two reference blocks along a selected axis, signaling angles and offsets from the block centers. This geometric division mode (GPM) allows for the use of larger coding units along the boundary between two objects, where the boundary geometry is encoded as angles and center offsets for the coding unit. Motion vector differences can be encoded as direction (up / down / left / right) and distance instead of orthogonal coordinate (x,y) offsets, with powers of 2 distances supported. The motion vector predictor is obtained from adjacent blocks as if no offset were applied ("merge mode"). The current block shares the same motion vector as the selected adjacent block.

[0151] Samples are selected based on motion vector 878 and reference picture index. Since motion vector 878 and reference picture index are applied to all color channels, interpretation is primarily described as an operation on the PU rather than the PB. The method for decomposing each CTU into one or more interpretation blocks is described using a single coding tree. Interpretation methods may differ in the number and precision of motion parameters. Motion parameters typically consist of a reference frame index indicating the reference frame to be used from the reference frame list, and spatial transformations for each reference frame, but may also include more frames, special frames, or complex affine parameters such as scaling and rotation. Furthermore, pre-determined motion refinement processes may be applied to generate high-density motion estimates based on the referenced sample blocks.

[0152] PB820 is determined and selected, and after subtracting PB820 from the original sample block in subtractor 822, the residual (represented by 824) with the minimum coding cost is obtained, and lossy compression is applied. This lossy compression process includes the steps of transformation, quantization, and entropy coding. The forward linear transformation module 826 applies a forward transformation to the difference 824, transforming the difference 824 from the spatial domain to the frequency domain and generating linear transformation coefficients represented by arrows 828. The maximum size of the linear transformation is either 32-point DCT-2 or 64-point DCT-2, which is set by "sps_max_luma_transform_size_64_flag" in the sequence parameter set. If the CB to be encoded exceeds the maximum supported linear transformation size expressed as the block size (e.g., 64×64 or 32×32), the linear transformation 826 is applied tile-wise, transforming all samples of the difference 824. When using non-square CBs, tiling is also performed using the maximum transformation size available in each dimension of the CB. For example, if the maximum transformation size is 32, a 64x16 CB will use two 32x16 linear transformations tiled together. If the CB size exceeds the maximum supported transformation size, the CB will be tiled with TBs. For example, a 128x128 CB with a maximum transformation size of 64 points will be tiled with four 64x64 TBs in a 2x2 arrangement. A 64x128 CB with a maximum transformation size of 32 points will be tiled with eight 32x32 TBs in a 2x4 arrangement.

[0153] The application of transformation 826 generates multiple TBs for a CB. Each transformation application processes TBs with a difference of 824 greater than 32×32 (e.g., 64×64), and all linear transformation coefficients 828 outside the upper-left 32×32 region of the TB are set to zero (i.e., discarded). The remaining linear transformation coefficients 828 are passed to the quantization module 834. These linear transformation coefficients 828 are quantized according to the quantization parameter 892 associated with the CB, generating linear transformation coefficients 832. In addition to the quantization parameter 892, the quantization module 834 applies a "scaling list" to further scale the residual coefficients within the TB according to their spatial position, enabling non-uniform quantization within the TB. The quantization parameter 892 may differ between the luma CB and each chroma CB. The linear transformation coefficients 832 are passed to the forward quadratic transformation module 830, where an unseparated quadratic transformation (NSST) operation or a quadratic transformation bypass generates the transformation coefficients represented by arrow 836. The forward linear transform module 826 is typically separable and transforms the row set and subsequent column set of each TB. The forward linear transform module 826 either uses the Type II Discrete Cosine Transform (DCT-2) horizontally and vertically, bypasses the horizontal and vertical transformations, or, for Luma TBs where neither width nor height exceeds 16 samples, uses a combination of the Type VII Discrete Sine Transform (DST-7) and Type VIII Discrete Cosine Transform (DCT-8) horizontally or vertically. The use of the DST-7 and DCT-8 combination is referred to as a "Multi-Transform Selection Set" (MTS) in the VVC standard.

[0154] The forward quadratic transform of module 830 is generally an inseparable transform and applies only to the residuals of the intra-predicted CU, although it may be bypassed. The forward quadratic transform operates on 16 samples (arranged as the upper left 4x4 subblock of the linear transform coefficients 828) or 48 samples (arranged as three 4x4 subblocks within the upper left 8x8 coefficients of the linear transform coefficients 828) to generate a set of quadratic transform coefficients. The set of quadratic transform coefficients may be smaller than the set of linear transform coefficients from which they are derived. Because the quadratic transform applies only to sets of coefficients adjacent to each other (including DC component coefficients), this quadratic transform is called a "low-frequency inseparable quadratic transform" (LFNST). Such a quadratic transform is obtained through a learning process, and due to its inseparability and learning origin, it leverages additional redundancy of residual signals that cannot be captured by separable transforms such as DCT and DST. Furthermore, when an LFNST is applied, all remaining coefficients in the TB become zero in both the linear and quadratic transform domains.

[0155] The quantization parameter 892 is constant for a given TB, resulting in uniform scaling in the generation of residual coefficients in the linear transformation domain of the TB. The quantization parameter 892 may vary periodically due to a signaled "delta quantization parameter." The delta quantization parameter (delta QP) is signaled only once for CUs contained within a given region (called a quantization group). If the CU is larger than the quantization group size, the delta QP is signaled only once with one of the TBs of the CU. That is, the delta QP is signaled only once by the entropy encoder 838 for the first quantization group of the CU, and not for subsequent quantization groups of the CU. Non-uniform scaling is also possible by applying a "quantization matrix," where the scaling coefficient applied to each residual coefficient is derived from a combination of the quantization parameter 892 and the corresponding entry in the scaling matrix. The size of the scaling matrix can be set smaller than the TB size, and when applied to the TB, the nearest neighbor method is used to provide scaling values ​​for each residual coefficient from a scaling matrix smaller than the TB size. The residual coefficients 836 are fed to the entropy encoder 838 for encoding in the bitstream portion 716. Typically, the residual coefficients of each TB, where at least one residual coefficient of the TU is significant, are scanned to generate an ordered list of values ​​according to a scan pattern. The scan pattern typically scans the TB as a sequence of 4x4 "subblocks," providing a regular scan operation with a granularity of 4x4 residual coefficient sets, where the arrangement of subblocks depends on the size of the TB. Scans within each subblock and transitions between subblocks typically follow a backward diagonal scan pattern. Furthermore, the quantization parameters 892 are encoded in the bitstream portion 716 using delta-QP syntactic elements, and the initial slice QP and quadratic transformation index 888 for a particular slice or subpicture are encoded in the bitstream portion 716.

[0156] As described above, the video encoder 714 requires access to the frame representation corresponding to the decoded frame representation observed by the video decoder. Therefore, the residual coefficient 836 passes through the inverse quadratic transform module 844, which operates according to the quadratic transform index 888, to produce the intermediate inverse transform coefficient, indicated by arrow 842. The intermediate inverse transform coefficient 842 is inversely quantized by the quantization module 840 according to the quantization parameter 892, to produce the inverse transform coefficient, indicated by arrow 846. The inverse quantization module 840 can also perform inverse non-uniform scaling of the residual coefficient using a scaling list corresponding to the forward scaling performed by the quantization module 834. The inverse transform coefficient 846 is passed to the inverse linear transform module 848, which generates the residual samples of the TU (indicated by arrow 850). The inverse linear transform module 848 applies the DCT-2 transform horizontally and vertically, constrained by the maximum available transform size described in the forward linear transform module 826. The type of inverse transformation performed by the inverse quadratic transformation module 844 corresponds to the type of forward transformation performed by the forward quadratic transformation module 830. The type of inverse transformation performed by the inverse linear transformation module 848 corresponds to the type of linear transformation performed by the linear transformation module 826. The additive module 852 adds the residual sample 850 and PB 820 to generate a reconstructed sample of CU (indicated by arrow 854).

[0157] The reconstructed sample 854 is passed to the reference sample cache 856 and the in-loop filter module 868. The reference sample cache 856 is typically implemented using static RAM on the ASIC to avoid costly off-chip memory access and provides the minimum sample storage necessary to satisfy dependencies for generating intra-frame Pb for subsequent CUs in the frame. Minimal dependencies typically include a "line buffer" of samples along the bottom edge of a row of CTUs, which is used for the next row of CTUs. It also includes column buffering, which is ranged by the height of the CTUs. The reference sample cache 856 supplies the reference sample (indicated by arrow 858) to the reference sample filter 860. The sample filter 860 applies a smoothing process to generate filtered reference samples (indicated by arrow 862). The filtered reference samples 862 are used by the intra-frame prediction module 864 to generate intra-prediction block samples (indicated by arrow 866). For each candidate intra-prediction mode, the intra-frame prediction module 864 generates a sample block (i.e., 866). Sample block 866 is generated by module 864 using techniques such as DC prediction, planar intra prediction, and angle intra prediction. Sample block 866 can also be generated by a matrix multiplication approach using adjacent reference samples as input and a matrix selected by video encoder 800 from a matrix set, the selected matrix being communicated to the video decoder in bitstream 716 with an index that identifies which matrix in the matrix set will be used.

[0158] The in-loop filter module 868 applies multiple filtering stages to the reconstructed sample 854. These filtering stages include a “deblocking filter” (DBF) that applies smoothing aligned to the CU boundary to reduce artifacts caused by discontinuities. Another filtering stage present in the in-roof filter module 768 is the “adaptive loop filter” (ALF), which applies a Wiener-based adaptive filter to further reduce distortion. A further filtering stage available in the in-loop filter module 868 is the “sample-adaptive offset” (SAO) filter. The SAO filter works by first classifying the reconstructed sample into one or more categories and then applying a sample-level offset according to the assigned category.

[0159] The filtered sample (indicated by arrow 870) is output from the in-loop filter module 868. The filtered sample 870 is stored in the frame buffer 872. The frame buffer 872 is typically stored in memory 206 because it has the capacity to store multiple pictures (e.g., up to 16). The frame buffer 872 is not typically stored using on-chip memory due to its high memory consumption. Therefore, accessing the frame buffer 872 is costly in terms of memory bandwidth. The frame buffer 872 provides a reference frame (indicated by arrow 874) to the motion estimation module 876 and the motion compensation module 880. The reference frame 874 is output as a reconstructed frame 718 by the corresponding subpicture coding modules 700 (614, 636, 654) and provided to the unpacker module 720. In the example in Figure 8, the reconstructed frame is the result of lossy VVC coding (i.e., processing by modules 810-890).

[0160] The motion estimation module 876 estimates multiple “motion vectors” (indicated by 878), each of which is an orthogonal coordinate space offset from the current CB position and refers to a block of any reference frame in the frame buffer 872. For each motion vector, a filtered block of reference samples (indicated by 882) is generated. The filtered reference samples 882 form additional candidate modes that are potentially selected by the mode selector 886. Furthermore, for a particular CU, the PU 820 may be formed using one reference block ("uni-prediction") or two reference blocks ("bi-prediction"). For the selected motion vector, the motion compensation module 880 generates the PB 820 according to a filtering process that supports sub-pixel precision in the motion vector. Thus, the motion estimation module 876 (operating on multiple candidate motion vectors) can perform a simplified filtering process and achieve a reduction in computational complexity compared to the motion compensation module 880 (operating only on selected candidates). If the video encoder 714 selects inter-prediction for a CU, the motion vector 878 is encoded into the bitstream portion 716.

[0161] Although the video encoder 800 in Figure 8 is described with reference to multipurpose video coding (VVC), the processing steps of modules 810-890 may be adopted in other video coding standards and implementations. Frame data 712 (and bitstream 121) can also be read from (or written to) memory 206, hard disk drive 210, CD-ROM, Blu-ray disc (trademark), or other computer-readable storage media. Furthermore, frame data 712 (and bitstream 121) may also be received from (or transmitted to) external sources such as servers or radio frequency receivers connected to the communication network 220. The communication network 220 may have bandwidth limitations, and rate control in the video encoder 120 is necessary to avoid network saturation in situations where compressing frame data 712 is difficult.

[0162] The bitstream 121 may consist of one or more slices representing spatial sections (sets of CTUs) of frame data 712, generated by one or more instances of the video encoder 714 (each instance generates a bitstream portion 716 and works cooperatively under the control of the processor 205). The bitstream portion 716 may contain one slice corresponding to one subpicture, which is output as a set of subpictures forming a single picture, where each subpicture is independently encodeable and decodeable with respect to other slices and subpictures within the picture. The ability to independently encode and decode any subpicture within a picture allows the PCA encoder 160 to take into account the effects of lossy compression on the packed feature maps or coefficients within a particular subpicture by using a lossy version of the feature maps or coefficients contained in that subpicture in a subsequent tensor compression stage.

[0163] Figure 9 is a schematic block diagram 900 showing an embodiment of the tensor decoder 170 based on inter-channel uncorrelatedness. Figure 11 is a schematic block diagram 1100 showing a multiscale feature reconstruction (MSFR) module. Figure 17 is a schematic block diagram showing the tensor decoding method 1700. The decoding operation of system 100 will be described with reference to method 1700. Method 1700 can be implemented using devices such as configured FPGAs, ASICs, and ASSPs. Alternatively, as will be described later, method 1700 may be implemented by the destination device 140 as one or more software code modules of an application program 233 under the execution of processor 205. The software code modules of the application program 233 implementing method 1700 may reside, for example, on a hard disk drive 210 and / or memory 206. Method 1700 is repeated for each frame of compressed data in bitstream 123. Method 1700 may be stored on a computer-readable storage medium and / or memory 206.

[0164] Method 1700 provides a means for decoding a compressed representation of a tensor with quality scalability and accurate modeling of the inaccuracies resulting from the use of a lossy compression mechanism. The video bitstream 143 is passed to a picture decoder 904 that implements a VVC video decoder and decodes the bitstream generated by the PCA encoder 160. The picture decoder 904 decodes the subpictures present in the video bitstream 143, each subpicture corresponding to a different data type necessary to generate a tensor 149. Each decoded subpicture provides an information unit for the decoded tensor, these units corresponding to an average channel 611, a basis vector 631, and coefficients 652, where the coefficients 652 are encoded into multiple groups. Method 1700 begins with an average update decoding step 1710.

[0165] In step 1710, a picture decoder, such as a VVC decoder, decodes the average update flag 1591 from the bitstream 143 under the control of the processor 205. As shown in Figure 9, the inter-channel uncorrelated-based tensor decoder 170 includes a picture decoder 904. Figure 10 shows the VVC decoder 1000 as an example implementation of the picture decoder 904.

[0166] The control within processor 205 proceeds from step 1710 to the average update test step 1720.

[0167] Step 1720 determines whether an average update is necessary based on the average update flag 1591. If the decoded average update flag in step 1720 indicates that an average update should be performed, the step returns YES, and control in the processor 205 proceeds from step 1720 to the average channel decoding step 1730. Otherwise, step 1720 returns NO, and control in the processor 205 proceeds from step 1720 to the average read step 1750.

[0168] In step 1730, the picture decoder 904, under the execution of processor 205, outputs an average channel subpicture 950, which is passed to the unpacker 952. The unpacker 952 extracts an integer average channel 954 with C values ​​corresponding to the number of channels in tensor 149. The integer average channel 954 is passed to the inverse quantizer 956, which performs a conversion from the sample region to the floating-point region using an appropriate quantization range (e.g., obtained from bitstream 143 obtained from quantization range 1593 (see Figure 15)). The operation of the inverse quantizer 956 generates the decoded average channel 970. The average channel 970 is passed to the multiplexer 982 and the average storage unit 972. The multiplexer 982 outputs 970 as the average channel 958. Control of processor 205 proceeds from step 1730 to the average storage step 1740.

[0169] In step 1740, the processor 205 stores the inverse quantization-decoded average 970 in the average storage unit 972 implemented in the memory 206. The inverse quantization-decoded average 970 is obtained from the inverse quantizer 956. The control of the processor 205 proceeds from step 1740 to the basis vector decoding step 1760.

[0170] In step 1750, the processor 205 retrieves the average 974 stored from the average memory unit 972 and passes it to the adder module 960 as the channel average 958 via the multiplexer 982. Control of the processor 205 proceeds from step 1750 to the basis vector decoding step 1760.

[0171] In step 1760, the picture decoder 904 outputs a subpicture 930 containing packed basis vectors generated by the decomposition module 632 (performing techniques such as SVD) before quantization and lossy compression, under the execution of the processor 205. In other words, the basis vectors are decoded from the bitstream and described in this example in subpicture packing form. This form may differ based on the number of dimensions of the basis vectors and the grouping of the coefficients. The subpicture 930 is passed to the unpacker 932. The unpacker 932 extracts integer basis vectors 934 as a sequence of non-overlapping arrangements within the subpicture 930. The integer basis vectors 934 are passed to the inverse quantizer 936. The inverse quantizer 936 applies the quantization range of the bitstream 143 obtained from the quantization range 1593 to construct floating-point basis vectors 938. Since not all subpictures were selected in step 1670, the decoded assignment indicates which basis vectors should have coefficients corresponding to the feature frames. Processor 205 controls the process from step 1760 to coefficient decoding step 1770.

[0172] In step 1770, the picture decoder 904, under the execution of processor 205, outputs a coefficient subpicture 910. The coefficient subpicture 910 is passed to the unpacker 912. The subpicture 910 contains a feature map corresponding to the coefficients decoded in step 1710. That is, the subpicture 910 contains the coefficients, with one coefficient per sample for each basis vector of tensor 143. The subpicture 910 is output by picture decoder 904 and passed to unpacker 912. The unpacker 912 extracts each coefficient from the subpicture 910 based on the dimensions of tensor 149 to generate tensor 918. In decoding the subpicture 910, the feature frame can be considered as having been decoded from the bitstream.

[0173] The control of processor 205 proceeds from step 1770 to the inverse quantization coefficient step 1780.

[0174] In step 1780, under the execution of processor 205, the inverse quantizer 920 converts the integer coefficient tensor 918 from the integer domain to the floating-point domain based on the bitstream 143 obtained from the quantization range 1593, and outputs the floating-point coefficients 922. Step 1760 obtains the coefficients from the feature frame decoded by the inverse quantization operation. Control of processor 205 proceeds from step 1780 to tensor generation step 1790.

[0175] In step 1790, under the execution of processor 205, the inner product module 940 generates a zero-centered tensor 942 by performing an inner product on the coefficients 922 and the basis vectors 938. The sum module 960 adds the zero-centered tensor 942 and the average channel 958 to generate a composite tensor 149a, which is reconstructed as the output from PCA decoder 170. Upon completion of step 1790, tensor 149a is effectively generated from the coefficient tensor and the basis vectors decoded in steps 1710 to 1780. The composite tensor 149a has a similar spatial size to the coefficient tensor and a higher number of channels. Tensor 149a can also be considered as a projection of the coefficients and basis vectors generated by the inner product operation using module 940, for example. Control within processor 205 proceeds from step 1790 to tensor reconstruction step 17100.

[0176] Architecture 1100 includes an MSFR module 1130. The MSFR module operates to generate multiple tensors from the tensor 149a generated by the execution of step 1790 using one or more trained convolutional layers. In step 17100, the MSFR module 1130 generates a reconstructed tensor 149 from the reconstructed synthetic tensor 149a. Upsampling modules 1132, 1134, and 1136 upsample tensor 149a by a factor of 2x, 4x, and 8x horizontally and vertically, respectively, to generate tensors 1133, 1135, and 1137. Tensor 1137 forms a single (P'2,1157) output from the MSFR module 1130 and is passed to the downsampling module 1142. The downsampling module 1142 downsamples tensor 1137 by a factor of 2x horizontally and vertically to generate tensor 1143, which has the same dimensions as tensor 1135. Tensor 1143 is fed to the convolutional layer 1148, which outputs tensor 1149. The summing module 1154 adds tensors 1135 and 1149 to generate tensor 1155 as the output (P'3) of the MSFR module 1130.

[0177] The downsampling module 1140 downsamples tensor 1135 by half horizontally and vertically to generate tensor 1141, which has the same dimensions as tensor 1133. Tensor 1141 is fed to the convolutional layer 1146, which outputs tensor 1147. The summing module 1152 adds tensors 1133 and 1147 to generate tensor 1153, which becomes the output (P'4) of the MSFR module 1130.

[0178] The downsampling module 1138 downsamples tensor 1133 by half horizontally and vertically to produce tensor 1139, which has the same dimensions as tensor 149a. Tensor 1139 is fed to the convolutional layer 1144, which outputs tensor 1145. The summing module 1150 adds tensors 149a and 1145 to produce tensor 1151 as the output (P'5) of the MSFR module 1130. Tensors 1151, 1153, 1155, and 1157 collectively form tensor 149, providing the decoded P2-P5 layer. Control in the processor 205 proceeds from step 17100 to step 17110, which executes the second part of the neural network.

[0179] In step 17110, the CNN head 150, under the execution of processor 205, executes the second part of the neural network task. An example of the second part, or head part 150, is illustrated with reference to Figures 12A-C and 13. Method 1700 finishes processing the current frame and is called again for the next frame received in bitstream 143.

[0180] Methods 1600 and 1700 mean that it is not necessary to encode or decode the mean for each frame in the video sequence. For example, consider two different frames in the sequence (let's call them the first and second frames). In the first iteration of Method 1600, step 1650 generates the first mean, step 16120 generates the first basis vector for the tensor of the first video frame, and steps 1660 and 16130 encode the first mean and the first basis vector into a bitstream. In a subsequent iteration of Method 1600 (for the second frame), Method 1600 proceeds from step 1640 to step 1690 to obtain the first mean. Step 16120 uses the first mean to generate the second basis vector for the tensor of the second video frame, and step 16130 encodes the second basis vector for the bitstream.

[0181] Similarly, an example of decoding the first and second video frames from a bitstream relates to a different iteration of Method 1700. In the first iteration of Method 1700, step 1730 decodes the first mean, and step 1760 decodes the first basis vector from the bitstream. Step 1790 generates the first tensor for the first video frame using the decoded first mean and the first basis vector. In subsequent iterations of Method 1700, the steps proceed from step 1720 to step 1750, reading the initially decoded mean. Step 1760 decodes the second basis vector from the bitstream, and step 1790 generates the second tensor for the second video frame using the saved first mean and the decoded second basis vector.

[0182] For example, in the first iteration, the first average is decoded in step 1710 based on the average update flag (1591) in the bitstream. With the configuration described, the average update flag 1591 may be restored once per video frame, or once for a specific number of frames in the video sequence. In other words, the first average decoded in step 1730 is available for use with multiple basis vector sets decoded from the bitstream.

[0183] In one embodiment of Method 1700, the storage device 972 holds the decoded (integer) mean 954 before inverse quantization. Integer storage typically requires 10 bits per sample, rather than the 32 or 64 bits required for floating-point values. Each time the mean is taken from storage 972, the inverse quantizer 956 must perform inverse quantization from the integer region to the floating-point region.

[0184] In one configuration of methods 1600 and 1700, the basis vector is encoded into bitstream 121 only if an update occurs in step 16120. Step 16120 determines a new basis vector at a less frequent rate than per frame (e.g., per IDR or IRAP picture for random access picture structures, and every two frames for other picture structures). An updated basis vector flag 1592 (see Figure 15) is encoded for each frame and indicates whether the current frame contains the updated basis vector. Step 16130 is performed only if the updated basis vector indicated by the updated basis vector flag 1592 is available. If the updated basis vector is not available, the basis vector 1422 may be omitted from subpicture 1412. The destination device 140 retains the basis vector between frames to enable decoding of subsequent frames that may not contain the basis vector in bitstream 121. Step 1760 is performed if the updated basis vector is decoded from bitstream 121 and indicated by the updated basis vector flag 1592.

[0185] Figure 10 shows an example implementation 1000 of the picture decoder 904 (also called a video decoder). The video decoder 1000 in Figure 10 is an example of a multipurpose video coding (VVC) video decoding pipeline, but other video codecs such as HEVC can also be used to perform the processing steps described here. As shown in Figure 10, bitstream 143 is input to the video decoder 1000. Bitstream 143 is read from memory 206, hard disk drive 210, CD-ROM, Blu-ray disc (trademark), or other non-temporary computer-readable storage medium and provided to implementation 1000 as bitstream 143. Alternatively, bitstream 143 may be received from an external source such as a server or radio frequency receiver connected to the communication network 220. Bitstream 143 contains encoded syntactic elements representing the image frame data to be decoded. If subpictures are decoded independently, portions of bitstream 143 corresponding to each subpicture may be supplied to separate instances of implementation 1000. 1000 separate instances of the implementation for each subpicture enable parallel decoding of subpictures for improved throughput.

[0186] Bitstream 143 is input to the entropy decoder module 1020. The entropy decoder module 1020 extracts syntactic elements from bitstream 143 by decoding a series of "bins" and passes their values ​​to other modules in the video decoder 904. The entropy decoder module 1020 decodes SPS, PPS, or slice headers using variable-length and fixed-length decoding, and decodes the syntactic elements of slice data as a sequence of one or more bins using an arithmetic decoding engine. Each bin can have one or more "contexts" available, where the context describes the probability level used to encode the "1" and "0" values ​​of the bin. If multiple contexts are available for a particular bin, a "context modeling" or "context selection" step is performed to select the context to use for decoding the bin from the available contexts. The bin decoding process forms a sequential feedback loop, so that an entire particular slice is decoded by a particular instance of the entropy decoder 1020. A single (or a small number of) high-performance entropy decoder 1020 instances can decode an entire slice or sub-picture of a frame or picture from the bitstream 143, while multiple low-performance entropy decoder 1020 instances can decode slices of a frame from the bitstream 143 in parallel.

[0187] The entropy decoder module 1020 decodes syntactic elements from the bitstream 143 by applying an arithmetic coding algorithm, such as "Context-Adaptive Binary Arithmetic Coding (CABAC)". The decoded syntactic elements are used for parameter reconstruction in the video decoder 904(1000). The parameters include mode selection information such as residual coefficients (represented by arrow 1024), quantization parameters 1074, quadratic transformation index 1070, and intra-prediction mode (represented by arrow 1058). The mode selection information also includes motion vectors and information for splitting each CTU into one or more CBs. The parameters are typically used in combination with sample data from previously decoded CBs to generate PBs.

[0188] The residual coefficients 1024 are passed to the inverse quadratic transform module 1036, where the quadratic transform is applied according to the quadratic transform index, or no operation is performed (bypassed). The inverse quadratic transform module 1036 generates reconstructed transform coefficients 1032. That is, module 1036 generates linear transform region coefficients from quadratic transform region coefficients. The reconstructed transform coefficients 1032 are input to the inverse quantization module 1028. The inverse quantization module 1028 performs inverse quantization (or "scaling") on the residual coefficients 1032, i.e., in the linear transform coefficient region, according to the quantization parameter 1074, generating reconstructed intermediate transform coefficients represented by arrow 1040. The inverse quantization module 1028 may also apply a scaling matrix to achieve non-uniform inverse quantization within the TB, which corresponds to the operation of the inverse quantization module 840. If the bitstream 143 indicates the use of a non-uniform inverse quantization matrix, the video decoder 1000 reads the quantization matrix from the bitstream 143 as a column of scaling coefficients and places them in the matrix. Inverse scaling combines the quantization matrix and quantization parameters to generate a reconstructed intermediate transformation coefficient of 1040.

[0189] The reconstructed conversion coefficients 1040 are passed to the inverse linear transformation module 1044. Module 1044 performs the inverse transformation of coefficients 1040 from the frequency domain to the spatial domain. The inverse linear transformation module 1044 applies the inverse DCT-2 transformation in the horizontal and vertical directions, while being constrained by the maximum available transformation size described in the forward linear transformation module 826. The result of processing by module 1044 is a block of residual samples (indicated by arrow 1048). The size of the residual sample block 1048 is equal to the corresponding CB. The residual samples 1048 are fed to the summing module 1050.

[0190] In the addition module 1050, the residual sample 1048 is added to the decoded PB (represented by 1052) to generate a reconstructed sample block represented by arrow 1056. The reconstructed sample 1056 is supplied to the reconstructed sample cache 1060 and the in-loop filtering module 1088. The in-loop filtering module 1088 generates a reconstructed frame sample block (represented by 1092). The frame sample 1092 is written to the frame buffer 1096. The frame buffer 1096 outputs an image or video frame 1014 corresponding to the tensor 149 in Figure 1.

[0191] The reconstructed sample cache 1060 operates similarly to the reconstructed sample cache 856 of the video encoder 714. The reconstructed sample cache 1060 provides storage for the reconstructed samples necessary for intra-predicting subsequent CBs without using memory 206 (for example, by using data 232 instead, which is typically on-chip memory). A reference sample (indicated by arrow 1064) is retrieved from the reconstructed sample cache 1060 and fed to the reference sample filter 1068 to produce filtered reference samples (indicated by arrow 1072). The filtered reference samples 1072 are fed to the intra-frame prediction module 1076. Module 1076 generates blocks of intra-prediction samples (indicated by arrow 1080) according to the intra-prediction mode parameter 1058, which is signaled in the bitstream 1010 and decoded by the entropy decoder 1020. The intra-prediction module 1076 supports the modes of the encoder-side module 864, including IBC and MIP. Sample block 1080 is generated using modes such as DC, planar, or angle-in-angle prediction.

[0192] If the prediction mode of the CB in bitstream 143 is indicated to use intra-prediction, the intra-predicted sample 1080 forms the decoded PB 1052 via the multiplexer module 1084. Intra-prediction generates a predicted block (PB) of samples, which is a block of single color components derived using "adjacent samples" within the same color component. Adjacent samples refer to samples that are adjacent to the current block and have already been reconstructed because they precede it in the block decoding order. When a luminous block and a chroma block are located in the same position, the luminous block and the chroma block may use different intra-prediction modes. However, two chroma CBs share the same intra-prediction mode.

[0193] If the prediction mode of the CB in bitstream 143 is indicated as inter-prediction, the motion compensation module 1034 generates a block of inter-prediction samples (represented by 1038). The inter-prediction sample block 1038 is generated using the motion vector decoded from bitstream 143 by the entropy decoder 1020 and a reference frame index for selecting and filtering the sample block 1098 from the frame buffer 1096. The sample block 1098 is obtained from a pre-decoded frame stored in the frame buffer 1096. In the case of bi-prediction, two sample blocks are generated and blended to produce samples for the decoded PB 1052. The frame buffer 1096 stores the filtered block data 1092 from the in-roof filtering module 1088. Similar to the in-roof filtering module 868 of the video encoder 714, the in-roof filtering module 1088 applies one of the DBF, ALF, or SAO filtering operations. Generally, motion vectors are applied to both lunar and chroma channels, but the filtering processes for subsample interpolation in lunar and chroma channels differ.

[0194] Although not shown in Figures 8 and 10, there are modules for pre-encoding video processing and post-decoding video processing, which shift sample values ​​to allow for more uniform use of the sample value range within each chroma channel. A multi-segment linear model is derived in video encoder 714 and signaled within the bitstream, and is used by video decoder 904 to reverse the sample shift. Linear Model Chroma Scaling (LMCS) tools offer compression advantages for certain color spaces and content where there is non-uniformity in the use of sample space (particularly limited range) and image quality degradation due to quantization application is likely to occur.

[0195] Figure 12A is a schematic block diagram showing an implementation example 1200 of the head 150 of a CNN for object detection, which corresponds to a part of the "YOLOv3" network excluding the "DarkNet-53" backbone. Implementation example 1200 can be used when a CNN backbone is implemented, for example, as shown in Figure 3A. Depending on the task performed on the destination device 140, a different network can be substituted for the CNN head 150. The input tensor 149 is separated into tensors for each layer (i.e., tensors 1210, 1220, and 1234). Tensor 1210 is passed to the CBL module 1212 to generate tensor 1214. Tensor 1214 is passed to the detection module 1216 and the upscaler module 1222. The detection module outputs a bounding box 1218 in the form of the detection tensor. The bounding box 1218 is passed to the non-maximal suppression (NMS) module 1248.

[0196] To generate bounding boxes corresponding to the coordinates of the original video data 113, the upscaler module 1222 scales the original video by its width and height before resizing it in the backbone portion of the network 114. The upscaler module 1222 receives tensors 1214 and 1220, generates an upscaled tensor 1224, which is passed to the CBL module 1226. The CBL module 1226 generates tensor 1228 as output. Tensor 1228 is passed to the detection module 1230 and the upscaler module 1236. The detection module 1230 generates a detection tensor 1232 and supplies it to the NMS module 1248. The upscaler module 1236 is another instance of module 1222. The upscaler module 1236 receives tensors 1228 and 1234, and outputs an upscaled tensor 1238. The upscaled tensor 1238 is passed to the CBL module 1240, which outputs tensor 1242 to the detection module 1244. The detection module 1244 generates detection tensor 1246, which is supplied to the NMS module 1248.

[0197] CBL modules 1212, 1226, and 1240 each contain a concatenation of five CBL modules, such as the CBL model 360 shown in Figure 3D. Upscaler modules 1222 and 1236 are instances of upscaler module 1260 shown in Figure 12B. Module 1248 receives tensors 1218, 1232, and 1246 and outputs task result 151.

[0198] As shown in Figure 12B, the upscaler module 1260 receives tensor 1262 (for example, tensor 1214 in Figure 12A) as input. Tensor 1262 is passed to the CBL module 1266 (which has the structure of module 360) to generate tensor 1268. Tensor 1268 is passed to the upsampling module 1270 to generate the upsampled tensor 1272. The concatenation module 1274 concatenates the upsampled tensor 1272 with the second input tensor 1264 (for example, tensor 1220 which is input to the upscaler 1222 in Figure 12A) to generate tensor 1276.

[0199] Detection modules 1216, 1230, and 1244 are instances of detection module 1280 shown in Figure 12C. Detection module 1280 receives tensor 1282. Tensor 1282 is input to CBL module 1284, which has the structure of module 360. CBL module 1284 generates tensor 1286. Tensor 1286 is passed to convolution module 1288, which implements the detection kernel. Depending on the configuration, the detection kernel applies a 1x1 kernel and generates an output on the feature map in each of the three layers of the tensor. The detection kernel is represented as 1x1x(Bx(5+C)), where B is the number of predictable bounding boxes for a particular cell (usually 3) and C is the number of classes (e.g., 80), so the kernel size is 255 detection attributes (i.e., tensor 1290). The constant "5" represents four bounding box attributes (box center x,y and size scale x,y) and one object confidence level ("objectness"). The detection kernel results have the same spatial dimensions as the input feature map, but the output depth corresponds to the detection attributes. The detection kernel is applied at each layer (usually three layers), generating a large number of candidate bounding boxes. Non-maximal suppression processing by the NMS module 1048 is applied to the resulting bounding boxes to eliminate redundant boxes, such as duplicate predictions at similar scales, resulting in the final set of bounding boxes as the output of object detection.

[0200] Figure 13 is a schematic block diagram showing an alternative head 1300 of the CNN. The head 1300 can be implemented as the CNN head 150, for example, when the CNN backbone 114 is implemented as the backbone 400. The head 1300 constitutes part of the entire network known as "fasterRCNN" and includes the feature network (i.e., the backbone 400), the region proposal network, and the detection network. The input to the head 1300 is tensor 149, which includes the P2-P6 layer tensors 1310, 1312, 1314, 1316, and 1318. The P2-P5 layer tensors 1310, 1312, 1314, and 1316 correspond to the P2-P5 outputs 477, 475, 473, and 471 in Figure 4. The P2-P6 tensors 1310, 1312, 1314, 1316, and 1318 are input to the Region Proposal Network (RPN) head module 1320. The P6 tensor 1318 is generated by the Max Pool module 1342, which performs a 2x2 Max Pool operation on the P5 tensor 1316. The RPN head module 1320 performs convolution on the input tensors to generate intermediate tensors. These intermediate tensors are fed into two consecutive sibling layers: (i) for classification and (ii) for bounding box (region of interest: ROI) regression. The resulting output is the classification result and bounding box 1322. The classification result and bounding box 1322 are passed to the NMS module 1324. The NMS module 1324 prunes redundant bounding boxes by removing duplicate bounding boxes with low scores, generating pruned bounding boxes 1326. The bounding box 1326 is input to the region of interest (ROI) puller 1328. The ROI puller 1328 uses a portion of the layer tensor of tensor 149 (described later) and the bounding box 1326 to generate a fixed-size feature map from various input size maps using a max pooling operation. In the max pooling operation, the maximum value is taken from each group of input values ​​by subsampling and a single output value is generated in the output tensor.

[0201] The input to the ROI puller 1328 is the feature maps P2-P5 (1310, 1312, 1314, 1316) and the region of interest proposal 1326. Each proposal (ROI) from 1326 is associated with a portion of the feature maps (1310-1316) to generate a fixed-size map. This fixed-size map has a size independent of the portion of the underlying feature maps 1310-1316. One of the feature maps 1310-1316 is selected according to, for example, the following rule: floor(4+log2(sqrt(box_area) / 224)) (where 224 is the standard box size) so that the resulting cropped map has sufficient detail. The ROI puller 1328 crops the input feature maps according to proposal 1326 to generate tensor 1330.

[0202] Tensor 1330 is input to the fully connected (FC) neural network head 1332. The FC head 1332 runs two fully connected layers to generate a class score and a bounding box prediction delta tensor 1334. The class score is typically an 80-element tensor, where each element corresponds to the predicted score for the corresponding object category. The bounding box prediction delta tensor is an 80 × 4 = 320-element tensor containing the bounding boxes for the corresponding object categories. Final processing is performed by the output layer module 1336, which receives tensor 1334 and performs a filtering operation to generate a filtered tensor 1338. Low-scoring (low-classification) objects are excluded from further consideration. The non-maximal suppression module 1340 receives the filtered tensor 1338 and removes overlapping bounding boxes by removing the overlapping boxes with lower classification scores, thereby generating the inference output tensor 1340 corresponding to tensor 151.

[0203] Figure 14A shows a schematic block diagram of splitting picture 1400 into subpictures 1410, 1412, and 1416, as implemented, for example, by the subpicture bitstream combiner 690. Each subpicture is packed by packer 710 with corresponding encoders 614, 636, 654a, 654b, and 654c. In the examples in Figures 14A and 14B, three subpictures are used because the coefficients are split into three groups; however, other implementations may allow for a different number of groups. Each subpicture contains information arranged as a two-dimensional array of samples. Referring to Figure 14B, picture 1400b corresponds to picture 1400, subpicture 1410b corresponds to 1410, and 1412b corresponds to 1412. Subpicture 1410b holds mean channel data, such as the mean channel 1420 of tensor 115. The average channel 1410b corresponds to the bitstream 615 of the subpicture encoder 614 in the configuration of Figure 6. Subpicture 1412b holds the basis vectors corresponding to 637 in Figure 6. In the example of Figure 14B, the basis vectors include basis vector 1422, etc., and the basis vectors are stored in a non-overlapping manner within the subpicture 1412b area. Basis vector 1422 forms a CxC' array of integer quantized values, where C is the number of channels in tensor 115 and C' is the number of basis vectors obtained from the decomposition performed in step 16120. When multiple tensors are decomposed into sets of basis vectors, basis vector 1422 includes each set of basis vectors arranged in a left-aligned vertically connected manner with no overlap. Areas within picture 1400 (or 1400b) that are not used for data storage may be occupied by sample values ​​corresponding to the value "0" as a result of inverse quantization of the sample values ​​and conversion to the floating-point area. Similarly, in subpictures 1410 (or 1410b), 1412 (or 1412b), and 1416, areas not used for data storage may be occupied by sample values ​​corresponding to the value "0" after applying inverse quantization from the sample values ​​to the floating-point region. Subpicture 1416 holds the coefficients corresponding to 654 in Figure 6, and the coefficients for each basis vector form a width × height feature map.The coefficient subpicture 1416 has coefficients packed as 24 feature maps (applicable to basis vectors [0..23]), similar to feature map 1430.

[0204] Figure 15 is a schematic block diagram showing bitstream 1500, which holds the encoded packed feature map and associated metadata. Bitstream 1500 corresponds to bitstream 121, which is generated by the PCA encoder 160, or bitstream 143, which is decoded by the PCA decoder 170. Bitstream 1500 contains a syntax group, which is preceded by a "Network Abstraction Layer" (NAL) unit header. For example, the NAL unit header 1508 is preceded by a sequence parameter set (SPS) 1510. SPS 1510 specifies the layout of picture 1400, including the positions and sizes of subpictures 1410, 1412, and 1416, using subpicture information 1511. SPS 1510 also indicates the chroma format, bit depth, and resolution of the frame data represented in bitstream 1500.

[0205] The average update determined in step 1620 of the PCA encoder 160 is encoded in the SEI message 1513 as the average update flag 1591. The quantization range determined by the quantizers 612, 632, and 646 is encoded as the quantization range 1593. The placement of the average update flag in the SEI message 1513 makes it possible to decode the first average only once for multiple subpictures or frames, thereby using it to generate multiple basis vector sets. In other embodiments, the average update flag 1591 can be placed at other locations in the bitstream (e.g., SPS, PPS, PH), which may enable signal decoding on a frame-by-frame or subpicture-by-subpicture basis.

[0206] Picture 1514 is encoded into bitstream 1500. Each picture contains one or more subpictures (e.g., encoded subpicture 1520) that encode subpicture 1410 (or 1410b). For the first picture and IDR picture in the bitstream, intra-slices are used to avoid predictive dependency on other access units in bitstream 1500. Encoded subpicture 1522, which encodes subpicture 1412, contains a slice header 1530 followed by slice data 1540. Slice data 1540 contains a set of CTUs that provide an encoded representation of the frame data. CTUs are square, typically 128x128 in size, which is inconsistent with typical feature map sizes. This inconsistency is partially mitigated by aligning the feature maps to a minimum block size (e.g., 4x4 grid). Encoded subpicture 1524 encodes subpicture 1416 corresponding to the coefficients.

[0207] In one embodiment of Method 1600, the basis vector determination step 16120 is performed less frequently than every frame. If a basis vector is not determined for a particular frame, the basis vector from a preceding frame is used. Step 16120 is performed periodically and less frequently, such as at the start of a video sequence or whenever a frame is encoded using intra-prediction, such as a new IDR picture or intra-random access picture (IRAP) in a random access picture structure. When step 16120 is performed, for each basis vector, the variance explained by the corresponding basis vector is also derived. When using a low-latency picture structure, intra-pictures are encoded much less frequently and are typically applied only to the first picture in a frame sequence. However, in streaming applications, decoding may begin from frames other than the first frame in the video sequence, thus requiring accidental encoding of intra-pictures. Regardless of the difference in IDR or IRAP picture frequencies between low-latency and random-access structures, decomposition can be performed with a certain regularity (e.g., once per second), which allows adaptation to changing picture statistics, reduces the average update frequency, and enables adaptation to slowly changing picture statistics.

[0208] In the configuration of Method 1600, the decomposition is performed incrementally in step 16120. Incremental decomposition (such as incremental SVD or incremental PCA) allows for the generation of updated basis vectors by combining existing basis vectors (generated in previous calls to Method 1600) with newly partially sampled tensor 623a. When adjusting or updating basis vectors, it is possible to further subsample tensor 623a (e.g., every 4 or every 8 samples horizontally and vertically) compared to deriving new basis vectors. For the first run of PCA when no existing basis vectors to update exist, a subsampling rate using more samples than when existing basis vectors are available can be adopted. For the first run of PCA, a subsampling tensor using every 2 samples horizontally and vertically is used, while for subsequent runs where existing basis vectors are available, every 4 or every 8 samples is used. Because shifting the mean requires computationally intensive operations in decomposition step 16120, incremental SVD requires keeping the mean immutable with each basis vector update. For large matrix data that are pre-zero-centered, adjusting to the new mean requires read-modify-write operations across the entire matrix, which becomes unacceptably costly in terms of memory access. When updating basis vectors in incremental PCA, the mean update flag 1591 indicates no mean update, and even in the calculation of the current (updated) basis vector, it indicates the reuse of the previous mean generated in the previous tensor set. When updating basis vectors in full PCA, mean update may also indicate an update of the mean (e.g., via flag 1591).

[0209] In the configuration of source device 110 and destination device 140, step 1610 is not performed, and therefore the tensor combiner 162 is omitted; step 17100 is not performed, and the tensor separator 172 is omitted. In implementations where the tensor combiner 162 is omitted, method 1600 proceeds directly from step 1605 to step 1620. Similarly, if the tensor separator 172 is omitted, method 1700 proceeds directly from step 1790 to step 17110. When the tensor combiner 162 and tensor separator 172 are omitted, the PCA encoder 160 operates on each tensor from the FPN implemented in the CNN backbone 114. For example, if system 100 implements a FasterRCNN or MaskRCNN network and the split point is set at layer P, the PCA encoder 160 operates on each tensor from layers P2-P5 of the CNN backbone 400, and layers P2-P5 are reconstructed as output from the PCA decoder 170 to the CNN head 1300. If system 100 implements a YOLOv3 network and the split point is at the output of the Darknet-53 backbone, the PCA encoder 160 processes each tensor in layers L0-L2 (322, 326, 329) of the CNN backbone 300, and L0-L2 are reconstructed as output from the PCA decoder 170 to the CNN head 1200. In other words, tensor 115a is passed to the PCA encoder as tensor 115, and tensor 149a is passed to the CNN head as tensor 149.

[0210] Figure 18 is a schematic block diagram 1800 showing the tensor 1810(623) that is subject to spatial downsampling in step 16110. Tensor 1810 consists of an array of values ​​such as value 1822 and has dimensions B×C×H×W corresponding to batch (usually 1), number of channels (e.g., 256), height, and width. Spatial downsampling, which takes values ​​every two horizontally and every two vertically, selects shaded values ​​such as value 1824 to generate a subsampled tensor 623a with dimensions B×C×floor(H / 2)×floor(W / 2), where "floor()" defines the truncation operation. The illustrated subsampling takes samples from odd rows and odd columns, but it is also possible to take samples from even rows and even columns or a combination thereof, which affects the dimensionality of the subsampled tensor 623a. Other subsampling rates are also possible, such as reducing the rate to every three or every four values. This further reduces the dimensionality of the subsampled tensor 623a and reduces the complexity of decomposition step 16120 compared to decomposing tensor 623.

[0211] The methods presented herein enable the efficient representation of tensors in a format suitable for compression using modern block-based compression standards such as VVC and HEVC. Block-based compression is not intuitively applicable to data such as coefficients for projecting basis vectors and reconstructing feature maps, but the use of various transformations, including pre-trained quadratic transformations, reveals additional unexpected redundancy within blocks. The methods presented herein are described with reference to the "FasterRCNN" and "YOLOv3" network architectures, as well as specific methods for splitting these networks into "backbone" and "head" parts, but these methods are applicable to any neural network that processes multidimensional tensor data, and also to different methods for splitting such networks into "backbone" and "head" parts. [Industrial applicability]

[0212] The described configuration is applicable to the computer and data processing industries, particularly to digital signal processing for encoding and decoding signals such as video and image signals, achieving high compression efficiency.

[0213] Configurations are also disclosed in which floating-point tensor data is quantized for each channel group or feature map, and the resulting integer values ​​are packed into planar frames using a logarithmic quantization domain. Quantization and dequantization methods employing a logarithmic quantization domain achieve higher compression efficiency because there are no bits consumed to encode precise values ​​of large-amplitude tensor values ​​that would not provide additional improvements in task performance on the network in use.

[0214] The described configuration leverages the characteristics of channel-based PCA, eliminating the need to determine a new channel average for each iteration of Method 1600 while maintaining acceptable performance. Eliminating the need for averaging per iteration reduces the computational complexity associated with PCA coding implementations. As mentioned above, an average update flag can be encoded to identify when the channel average should be updated. Furthermore, the described configuration can be used in conjunction with an "incremental PCA" solution. Similarly, on the decoder side, there is no need to decode a new average channel for each iteration of Method 1700, thereby reducing computational requirements.

[0215] The above are merely examples of embodiments of the present invention, and modifications and / or changes can be made to them without departing from the scope and spirit of the invention. The embodiments are illustrative and not limiting.

Claims

1. A method for decoding a video frame tensor from a bitstream, Decoding a first mean and a plurality of first basis vectors from the bitstream, Using the first mean and the plurality of first basis vectors, a plurality of first tensors for the first video frame are generated, Decoding a plurality of second basis vectors from the bitstream, The method of generating a plurality of second tensors for a second video frame using the first mean and the plurality of second basis vectors, wherein the second video frame is different from the first video frame, Methods that include...

2. The first average is decoded based on the flag in the bitstream. The method according to claim 1.

3. The aforementioned flag is reset once per video frame. The method according to claim 2.

4. The first mean is used to generate a set of basis vectors decoded from the bitstream. The method according to claim 1.

5. The first mean is recovered once and used to generate the plurality of basis vector sets. The method according to claim 4.

6. A method for encoding a video frame tensor into a bitstream, To generate a first mean and a plurality of first basis vectors for multiple tensors of the first video frame, Encoding the first average and the plurality of first basis vectors into the bitstream, Using the first mean, a plurality of second basis vectors are generated for a plurality of tensors of the second video frame, The encoding of the plurality of second basis vectors into the bitstream, wherein the second video frame is different from the first video frame, Methods that include...

7. A flag indicating the presence of the first average for the video frame is encoded. The method according to claim 6.

8. The plurality of second basis vectors are generated using fewer samples than the total number of samples in the plurality of tensors of the second video frame. The method according to claim 6.

9. The multiple feature maps of the multiple tensors are sampled every two samples in the horizontal and vertical directions. The method according to claim 8.

10. At least one of the plurality of tensors is sampled at a different rate than the other tensors of the plurality of tensors. The method according to claim 8.

11. Larger feature maps are sampled less frequently horizontally and vertically than smaller feature maps. The method according to claim 8.

12. A non-temporary computer-readable storage medium for storing a program for performing a method of decoding a video frame tensor from a bitstream, The aforementioned method, Decoding a first mean and a plurality of first basis vectors from the bitstream, Using the first mean and the plurality of first basis vectors, a plurality of first tensors for the first video frame are generated, Decoding a plurality of second basis vectors from the bitstream, The method of generating a plurality of second tensors for a second video frame using the first mean and the plurality of second basis vectors, wherein the second video frame is different from the first video frame, Non-temporary computer-readable storage media, including [specific type of storage medium].

13. A video decoder for decoding a tensor of video frames from a bitstream, A first average and a plurality of first basis vectors are decoded from the bitstream. Using the first mean and the plurality of first basis vectors, a plurality of first tensors for the first video frame are generated, Multiple second basis vectors are decoded from the bitstream, Using the first mean and the plurality of second basis vectors, a plurality of second tensors are generated for the second video frame, and the second video frame is different from the first video frame. A video decoder configured in such a way.

14. It is a system, Memory and A processor configured to execute code stored in memory for performing a method of decoding a video frame tensor from a bitstream, Includes, The aforementioned method, Decoding a first mean and a plurality of first basis vectors from the bitstream, Using the first mean and the plurality of first basis vectors, a plurality of first tensors for the first video frame are generated, Decoding a plurality of second basis vectors from the bitstream, The method of generating a plurality of second tensors for a second video frame using the first mean and the plurality of second basis vectors, wherein the second video frame is different from the first video frame, A system that includes this.

15. A non-temporary computer-readable storage medium for storing a program for performing a method of encoding a video frame tensor into a bitstream, The aforementioned method, To generate a first mean and a plurality of first basis vectors for multiple tensors of the first video frame, Encoding the first average and the plurality of first basis vectors into the bitstream, Using the first mean, a plurality of second basis vectors are generated for a plurality of tensors of the second video frame, The encoding of the plurality of second basis vectors into the bitstream, wherein the second video frame is different from the first video frame, Non-temporary computer-readable storage media, including [specific type of storage medium].

16. A video encoder for encoding a tensor of video frames into a bitstream, Generate a first mean and a plurality of first basis vectors for multiple tensors of the first video frame. The first average and the plurality of first basis vectors are encoded into the bitstream. Using the first average, a plurality of second basis vectors are generated for a plurality of tensors of the second video frame, The plurality of second basis vectors are encoded into the bitstream, and the second video frame is different from the first video frame. A video encoder configured in such a way.

17. It is a system, Memory and A processor configured to execute code stored in memory for performing a method of encoding a video frame tensor into a bitstream, Includes, The aforementioned method, To generate a first mean and a plurality of first basis vectors for multiple tensors of the first video frame, Encoding the first average and the plurality of first basis vectors into the bitstream, Using the first mean, a plurality of second basis vectors are generated for a plurality of tensors of the second video frame, The encoding of the plurality of second basis vectors into the bitstream, wherein the second video frame is different from the first video frame, A system that includes this.

Images