Method, apparatus, and system for decoding tensors from the contents of a bitstream

Abstract

A method for decoding a tensor from the contents of a bitstream. The method involves determining the network topology from the bitstream. First information is decoded from the bitstream. The decoded tensor is generated from the first information using the determined network topology.

Description

Technical Field

[0001] Reference to Related Patent Applications This application claims priority under 35 U.S.C. § 119 based on the filing date of Australian Patent Application No. 2023203165, filed on May 19, 2023, and the entire disclosure of that application is incorporated herein by reference as if fully set forth herein. The present invention relates to digital video signal processing, and more particularly to methods, apparatuses, and systems for encoding and decoding tensors from convolutional neural networks. The present invention may also relate to a computer program product including a computer-readable medium having recorded thereon a computer program for encoding and decoding tensors from convolutional neural networks using video compression techniques.

Background Art

[0002] Convolutional neural networks (CNNs) are an emerging technology that addresses a variety of challenges, including machine vision-related use cases such as object detection, instance segmentation, object tracking, human pose estimation, and motion recognition. CNN applications often envision the use of "edge devices," which possess sensors and a certain level of processing power and are connected to application servers as part of the "cloud." CNNs can require relatively high computational complexity, sometimes exceeding the computing power and power consumption limits that a single edge device can typically provide. Running CNNs in a distributed manner is emerging as a solution to run state-of-the-art networks on edge devices with limited capabilities, without burdening all computational complexity on cloud servers, while still utilizing potentially untapped inference resources. In other words, distributed processing allows legacy edge devices to deliver state-of-the-art CNN capabilities by distributing processing between edge devices and other processing means such as cloud servers. Such distributed network architectures are called "collaborative intelligence (CI)" and offer advantages such as reusing partial results obtained in the first part of the network in multiple second parts, each optimized for different tasks. In a CI architecture, efficient compression of tensor data is necessary for transmission over networks such as WANs.

[0003] CNNs typically contain many layers, such as convolutional and fully connected layers, and data moves between layers in the form of "tensors." When a network is divided among different devices, it becomes necessary to compress the intermediate multidimensional tensor data moving between layers within the CNN to facilitate transmission over bandwidth-constrained and costly networks. This type of tensor compression is called "feature compression," and the intermediate tensor data is often referred to as "features" or "feature maps." Each feature map corresponds to one "channel" of the tensor, and features or feature maps are generally sets of two-dimensional (2D) arrays of values, which are combined into a 3D (or 4D) data structure to form a tensor. Intermediate tensor data represents a partially processed form of input data, such as image frames or video frames, encountered within a neural network. The International Organization for Standardization / International Electrotechnical Commission Joint Technical Committee 1 / Section 29 / Working Group 2 (ISO / IEC JTC1 / SC29 / WG2), also known as the "Motion Video Expert Group" (MPEG) Technical Requirements, is tasked with studying the requirements for compression techniques in various contexts, particularly those related to video. WG2 "MPEG Technical Requirements" has established an ad-hoc group for "Feature Compression for Machine Video Coding" (FCVCM) tasked with researching feature compression. The FCVCM AHG has issued a "Call for Proposals" soliciting responses that will form the basis of a standardization project on feature compression. In response to a previous "Call for Evidence (CfE)," a technology was demonstrated that significantly outperforms the feature compression results achieved using state-of-the-art standardization techniques applied directly to tensors.

[0004] In CNNs, the weights of each layer typically need to be pre-determined during the training phase. During this phase, a vast amount of training data passes through the CNN, and the results derived by the training network are compared to the true values ​​associated with the training data. The difference between the obtained results and the desired results is expressed as "loss" and measured using a "loss function." The determined loss is then used to perform a network weight update process, such as stochastic gradient descent (SGD). Network weight updates typically involve a backpropagation process of the "gradient," starting from the network's output layer and progressing backward until the input layers are updated. This process covers the network's intermediate layers, or "hidden layers." The gradient indicates the delta value to be applied to the network weights and is itself updated as part of the backpropagation process. The speed of weight updates is set by a hyperparameter called the "learning rate," which is typically set to facilitate the training process in finding a global minimum in terms of loss (the best possible task performance given the network architecture and training data), while avoiding the training process "getting stuck" in a local minimum. Falling into local minimum corresponds to obtaining suboptimal task performance for the network architecture, and refers to a state where it is no longer possible to find new weight values ​​that would lead to higher task performance. Network weights are repeatedly updated by supplying input data and ground truth data in batches, and the network performance is iteratively refined until no further improvement in accuracy can be achieved. The iterative process across the entire training dataset forms one "epoch" of training, and multiple epochs are usually required to achieve high task performance. The weights of the trained network are then available for deployment, and the network operates in a mode where the weights are fixed and gradient calculations for weight updates are omitted. The process of feeding input to a pre-trained CNN and transforming the input stepwise to output according to the CNN topology is generally called "inference".

[0005] Generally, a tensor has four dimensions: batch, channels, height, and width. The first dimension, "batch," is typically 1 in size when performing inference on video data, indicating that one frame passes through the CNN as one batch. When training a network, the batch dimension value may be increased according to a predetermined "batch size" so that multiple frames pass through the network in each batch before the network weights are updated. Multi-frame videos may be processed as a single tensor with a batch dimension size increased depending on the number of frames in a particular video. However, for practical considerations regarding memory consumption and access, inference on video data is usually performed on a frame-by-frame basis. The "channel" dimension indicates the number of "feature maps" being processed concurrently in a particular tensor, while the height and width dimensions indicate the size of the feature maps at a particular stage of the CNN. The number of channels varies in each layer of the CNN depending on the network architecture. The size of the feature maps also changes due to subsampling and upsampling that occur in a particular network layer.

[0006] Convolutional Neural Networks (CNNs) tend to be relatively complex overall, involving a large number of multiplication and integration (MAC) operations, numerous intermediate tensors being written to and read from memory, and the need to load weights for each layer of the CNN to run. Therefore, by splitting the neural network, it becomes possible to implement more complex networks even on systems including low-performance edge devices, without burdening cloud servers with the entire network execution load.

[0007] Feature compression can benefit from existing video compression standards such as General-Purpose Video Coding (VVC), developed by the Joint Video Expert Team (JVET). VVC is expected to address the ongoing demand for higher compression performance, particularly with the advancement of video formats (such as higher resolution and higher frame rates), and the growing market demand for service delivery over WANs where bandwidth costs are relatively high. VVC can be implemented using modern silicon processes and offers an acceptable trade-off between performance and implementation cost. Implementation cost is considered in terms of one or more of the following: silicon area, CPU processor load, memory usage, and bandwidth. Other video compression standards such as High Efficiency Video Coding (HEVC) and AV-1 can also be used for feature compression applications.

[0008] Video data consists of a series of image data frames, each containing one or more color channels. When representing feature map data with packed frames, monochrome frames, which generally only contain luminance and no chroma channels, are sufficient. If only luminance samples are present, the resulting monochrome frames are said to use a "4:0:0 chroma format".

[0009] The VVC standard defines a "block-based" architecture, where a frame is first divided into an array of square regions called "encoded tree units (CTUs)." In VVC, a CTU typically occupies 128x128 lumens. Other CTU sizes available in the VVC standard are 32x32 and 64x64. However, CTUs at the rightmost and bottommost edges of each frame may have smaller areas, and an implicit partitioning occurs to ensure that the encoded blocks remain within the frame. Each CTU is associated with an "encoded tree," which defines how the CTU's region is decomposed into a series of blocks (also called encoded units: CUs). Blocks that apply only to lumens or only to chroma channels are called "encoded blocks" (CBs). Predictions of the contents of an encoded block are held in "prediction blocks" (PBs) or "prediction units" (PUs), and residual blocks, which define an array of sample values ​​that are additively combined with the PB or PU, are called "transformation blocks" (TBs) or "transformation units" (TUs) because transformation processes are usually used to generate TBs or TUs.

[0010] Regardless of the above distinction between "unit" and "block," the term "block" is sometimes used as a general term to refer to an area or region of a frame to which an operation is applied across all color channels.

[0011] For each CU, a prediction unit (PU) is generated that predicts the content (sample value) of the corresponding region of the frame data ("prediction unit"). Furthermore, a representation of the difference (or "spatial region" residual) between the predicted value observed at the encoder input and the content of the region is formed. The difference in each color channel is transformed and encoded as a column of residual coefficients, forming one or more TUs for a given CU. The transformation applied is a discrete cosine transform (DCT) or other transformation, and is applied to each block of residual values. The transformations are applied separately (i.e., a two-dimensional transformation is performed in two passes, horizontal and vertical). The block is first transformed by applying a one-dimensional transformation to each row of samples within the block. Next, the partial results are transformed by applying a one-dimensional transformation to each column of the partial results, generating a final transformed coefficient block with the correlation of residual samples effectively removed. The VVC standard supports transformations of various sizes, including transformations of rectangular blocks where the length of each side is a power of 2. The transformed coefficients are quantized for entropy coding to the bitstream.

[0012] In VVC, P blocks or PUs are generated using either intra-frame prediction or inter-frame prediction processes. In-frame prediction uses samples already processed within a frame to generate a prediction for the current data sample block within that frame. Inter-frame prediction uses sample blocks obtained from one or two previously decoded frames to generate a prediction for the current sample block within a frame. Sample blocks obtained from previously decoded frames are offset from the spatial position of the current block according to a motion vector (often with filtering applied). In-frame prediction blocks are (i) uniform sample values ​​("DC In-frame Prediction"), (ii) a plane with offsets and horizontal and vertical slopes ("Planar In-frame Prediction"), (iii) a distribution of blocks with adjacent samples applied in a specific direction ("Angular In-frame Prediction"), or (iv) the result of matrix multiplication using adjacent samples and selected matrix coefficients.

[0013] VVC can be used to compress intermediate feature maps obtained from the first part ("backbone") of a neural network that has been separated into two parts. In compression, the feature maps from the backbone are placed in frames and quantized from the floating-point domain to a sample domain suitable for compression as video data. Techniques related to principal component analysis (PCA) can be applied to reduce the dimensionality of tensors before compression using video compression standards such as VVC. Tensor dimensionality reduction reduces the amount of data to be compressed, improving compression efficiency and shortening the execution time of the VVC coding and decoding stages. Dimensionality reduction introduces complexity, which offsets the time savings seen in VVC coding. There is a need to address, or at least mitigate, the increase in complexity caused by applying PCA to tensor dimensionality reduction. [Overview of the project]

[0014] The object of the present invention is to substantially overcome, or at least mitigate, one or more drawbacks of existing configurations.

[0015] According to one aspect of this disclosure, a method is provided for decoding a tensor of the contents of a bitstream, the method being Determining the network topology from the aforementioned bitstream, Decoding the first information from the bitstream, Using the determined network topology, a decoded tensor is generated from the first information, Includes.

[0016] According to another aspect of this disclosure, a decoder is provided for decoding a tensor of the contents of a bitstream, the decoder is: The network topology is determined from the aforementioned bitstream. The first information is decoded from the bitstream, Using the determined network topology, a decoded tensor is generated from the first information. It is configured in this way.

[0017] According to yet another aspect of the present disclosure, there is provided a non-transitory computer-readable storage medium storing a program for executing a method of decrypting a tensor with respect to the content of a bitstream, the method comprising: determining a network topology from the bitstream; decrypting first information from the bitstream; generating a decrypted tensor from the first information using the determined network topology; and comprising.

[0018] According to yet another aspect of the present disclosure, there is provided a system, the system comprising: a memory; a processor configured to execute code stored in the memory for implementing a method of decrypting a tensor with respect to the content of a bitstream, and comprising, the method determining a network topology from the bitstream; decrypting first information from the bitstream; generating a decrypted tensor from the first information using the determined network topology; and comprising.

[0019] Other aspects are also disclosed.

Brief Description of the Drawings

[0020] At least one embodiment of the present invention will be described below with reference to the following drawings and appendices.

[0021] [Figure 1] FIG. 1 is a schematic block diagram showing a distributed machine task system.

[0022] [Figure 2A] FIG. 2A forms a schematic block diagram of a general-purpose computer system capable of executing the distributed machine task system of FIG. 1. [Figure 2B] Figure 2B forms a schematic block diagram of a general-purpose computer system capable of executing the distributed machine task system shown in Figure 1.

[0023] [Figure 3A] Figure 3A is a schematic block diagram showing the functional modules of the CNN backbone.

[0024] [Figure 3B] Figure 3B is a schematic block diagram showing the residual block in Figure 3A.

[0025] [Figure 3C] Figure 3C is a schematic block diagram showing the residual unit in Figure 3A.

[0026] [Figure 3D] Figure 3D is a schematic block diagram showing the CBL module in Figure 3A.

[0027] [Figure 4] Figure 4 is a schematic block diagram showing the functional modules of the alternative backbone portion of the CNN.

[0028] [Figure 5] Figure 5 is a schematic block diagram of a tensor encoder using a configurable tensor compression stage.

[0029] [Figure 6] Figure 6 is a schematic block diagram showing a multiscale feature fusion stage for a tensor compressor.

[0030] [Figure 7] Figure 7 is a schematic block diagram showing a tensor compressor based on inter-channel decorrelation.

[0031] [Figure 8] Figure 8 is a schematic block diagram showing the functional modules of the video encoder.

[0032] [Figure 9A] Figure 9A is a schematic block diagram showing the arrangement of regions or subpictures for holding compressed tensor data. [Figure 9B] Figure 9B is a schematic block diagram showing the arrangement of regions or subpictures for holding compressed tensor data.

[0033] [Figure 10A] Figure 10A is a schematic block diagram showing an alternative arrangement of regions or subpictures for holding compressed tensor data. [Figure 10B] Figure 10B is a schematic block diagram showing an alternative arrangement of regions or subpictures for holding compressed tensor data.

[0034] [Figure 11] Figure 11 is a schematic block diagram showing the bitstream that holds the encoded inter-channel decorrelation feature map and associated metadata.

[0035] [Figure 12] Figure 12 is a schematic block diagram showing a tensor decoder with a configurable tensor decompressor.

[0036] [Figure 13] Figure 13 is a schematic block diagram showing the functional module of the video decoder.

[0037] [Figure 14] Figure 14 is a schematic block diagram showing an inter-channel decorrelation-based tensor decoder as part of a distributed machine task system.

[0038] [Figure 15] Figure 15 is a schematic block diagram showing an embodiment of the multiscale feature reconstruction stage.

[0039] [Figure 16A]Figure 16A is a schematic block diagram showing the head portion of a CNN.

[0040] [Figure 16B] Figure 16B is a schematic block diagram showing the upscaler module of Figure 16A.

[0041] [Figure 16C] Figure 16C is a schematic block diagram showing the detection module in Figure 16A.

[0042] [Figure 17] Figure 17 is a schematic block diagram showing the alternative head portion of the CNN.

[0043] [Figure 18] Figure 18 shows how to run the first part of the CNN, select a tensor compressor, compress the tensor using the selected tensor compressor, and encode the resulting compressed tensor.

[0044] [Figure 19] Figure 19 shows how to decode the bitstream, reconstruct the tensors according to the specified tensor decompressor, and run the second part of the CNN.

[0045] Appendix A shows a syntax table of "Supplemental Enhancement Information" (SEI) messages that hold the metadata necessary for the operation of the decoding method shown in Figure 19.

[0046] Appendix B shows a Python code snippet that implements the decoder network topology in the multiscale feature reconstruction stage shown in Figure 15.

[0047] Appendix C shows a Python code snippet that implements the decoder network topology in the tensor decoder based on inter-channel decorrelation shown in Figure 14. [Modes for carrying out the invention]

[0048] Where any steps and / or features having the same reference numerals are referred to in any one or more of the attached drawings, those steps and / or features have the same function or operation unless otherwise intended for the purposes of this specification.

[0049] A distributed machine task system may include edge devices such as network cameras and smartphones that generate intermediate compressed data. It may also include end devices, such as server farm-based ("cloud") applications, that process the intermediate compressed data and generate task results. Furthermore, the functionality of the edge devices may be implemented in the cloud, and the intermediate compressed data may be stored for subsequent processing and potentially used for multiple different tasks as needed.

[0050] A convenient format for intermediate compressed data is the compressed video bitstream, due to the availability of high-performance compression standards and their implementations. Video compression standards typically handle integer samples of a given bit depth, such as 10 bits, in a planar array. Color video has three planar arrays, corresponding to color components such as Y, Cb, Cr, or R, G, B (depending on the application). Convolutional Neural Networks (CNNs) typically operate on floating-point data in tensor form. Tensors have a relatively small spatial dimension compared to the input video data that CNNs process and have more channels than the typical three channels of color video data (e.g., 128, 256, or 512 channels).

[0051] Tensors typically have dimensions of frames, channels, height, and width. For example, a tensor with the following dimensions [1,256,76,136] can be described as containing floating-point or integer values ​​for one frame, consisting of an array of 256 feature maps (channels), each with a size of 136 x 76. In the case of video data, inference is usually performed frame by frame (frame value 1) rather than using a tensor containing multiple frames.

[0052] VVC supports dividing a single picture into multiple subpictures, each of which is encoded and decoded independently. In one method, each subpicture is encoded as a single "slice," i.e., a sequence of consecutive encoded CTUs. A "tile" mechanism is also available, which allows a picture to be divided into multiple independently decodeable regions. Subpictures can be specified with relative flexibility, and various rectangular sets of CTUs can be encoded as subpictures. Because the dimensions of subpictures can be defined flexibly, data types requiring different regions can be efficiently held in a single picture, avoiding large "unused" regions, i.e., frame regions not used for reconstructing tensor data.

[0053] Figure 1 is a schematic block diagram showing a functional module of a distributed machine task system 100 capable of executing machine task networks in a distributed manner. To divide a particular neural network into two parts, it is necessary to specify a "split point" within the network. The network layers from the input layer to the split point are processed by a first device, and the resulting intermediate tensors are compressed. The layers from the split point to the final layer of the network are processed using the decompressed tensors obtained from the first device as input to the layer immediately following the split point. At the split point, there may be one or more tensors that need to be compressed in order to be transmitted over a communication channel with limited bandwidth compared to the bandwidth required for transmitting uncompressed tensors. When using a "feature pyramid network" (FPN), the layers within the FPN share a common width-to-height relationship, and a particular layer is often half the width and height of an adjacent layer. In FPN architectures, the width and height may alternately be halved for each layer. In some architectures, multiple tensors with the same width and height are generated within the FPN. FPNs can appear relatively early in neural network topology, and to effectively distribute network load between edge devices and the cloud, it is necessary to introduce a partition point within the FPN. When partitioning occurs within an FPN in a machine task network, the layers up to the partition point are common to the machine task network ("shared backbone" architecture), enabling the performance of various machine task networks. When partitioning occurs within an FPN, tensor compression techniques can improve compression performance by leveraging redundancy between FPN layers. Therefore, compression techniques applicable to various network topologies used in modern CNNs are beneficial in a wide range of scenarios.

[0054] System 100 can be used to implement techniques for decorating, packing, and quantizing feature maps into planar frames for various neural networks, and for decoding feature maps from the encoded data. Different neural networks can be partitioned at different points, potentially generating intermediate tensors with different numbers and dimensionalities. A feature compression scheme that can adapt to different types of intermediate data and provide reconstruction results of different quality offers advantageous flexibility. Furthermore, System 100 provides flexibility in interfacing with neural networks of various architectures and with various applications that may be partitioned (e.g., for distributed execution).

[0055] System 100 includes a source device 110 that generates encoded frame data 113. The frame data 113 is passed to a CNN backbone 114 to generate a tensor 115. The tensor 115 is passed to a tensor encoder 116 to generate an encoded bitstream 121. System 100 also includes a destination device 140 that decodes the tensor data in the form of a received bitstream 143. The destination device 140 is used to decode the tensor data (or tensor) corresponding to the content of the bitstream 143 (e.g., audio data, video data, image data, text data).

[0056] Communication channel 130 is used to transmit the encoded bitstream 121 from the transmitting device 110 to the receiving device 140. In some configurations, the transmitting device 110 and the receiving device 140 may each or both consist of a mobile phone terminal (e.g., a "smartphone") or a network camera and a cloud application. Communication channel 130 can be a wired connection such as Ethernet, or a wireless connection such as WiFi or 5G, including connections over a wide area network (WAN). Communication channel 130 can also be implemented via an ad-hoc connection. Furthermore, the transmitting device 110 and the receiving device 140 may consist of applications that hold encoded video data on computer-readable storage media such as hard disk drives or memory in a file server. Although system 100 is described as including a video source 112 that provides frame data 113 to a neural network targeting a computer vision application, the appropriate neural network implemented in the CNN backbone 114 and CNN head 150 may also be input with other types of source data such as audio or text.

[0057] As shown in Figure 1, the source device 110 includes a video source 112, a CNN backbone 114, a tensor encoder 116, and a transmitter 122. The video source 112 typically consists of a source of captured video frame data (shown as 113), such as an image sensor, a pre-captured video sequence stored on a non-temporary recording medium, or a video feed from a remote image sensor. The video source 112 may also include the output of a computer graphics card that displays the video output of an operating system or various applications running on a computing device (e.g., a tablet computer). Examples of source devices 110 that include an image sensor as the video source 112 include smartphones, video cameras, professional video cameras, and network video cameras. The video source 112 can generate independent images or temporally continuous images, i.e., video.

[0058] The neural networks implemented in the CNN backbone 114 and CNN head 150 may vary depending on the application. For example, the "YOLOv3" network may be used as part of an object tracking system, while the "FasterRCNN" network may be used as an object detection system. The number and dimensions of tensor 115 depend on the specific network running in system 100 and the partition points of that particular network.

[0059] The CNN backbone 114 receives video frame data 113, executes a specific layer of the entire CNN (for example, the layer corresponding to the CNN's "backbone"), and outputs tensor 115a. When the CNN backbone layer divides the network within an FPN, it may output multiple tensors corresponding to different spatial scales of the input image represented by the video frame data 113. With the FPN, three tensors corresponding to the three layers output from the backbone 114 may be obtained as tensor 115a with different spatial resolutions and channel counts (e.g., when system 100 is running the "YOLOv3" network). When system 100 is running a network such as "FasterRCNN X101-FPN" or "MaskRCNN X101-FPN", tensor 115a may contain tensors corresponding to four layers (P2-P5). Using the FPN, when network division points occur within the FPN, as described later, multiple tensors are generated that form a hierarchical representation for encoding (and decoding) a single frame into a bitstream.

[0060] The bitstream 121 is either supplied to the transmitter 122 for transmission via the communication channel 130, or written to the storage device 132 for later use.

[0061] The transmitting device 110 supports a specific network for the CNN backbone 114. However, the receiving device 140 may use one of several networks for the head CNN 150. If one of several networks is used for the head CNN 150, partially processed data can be stored in the form of packed feature maps, which can then be used later to perform various tasks without having to re-operate the CNN backbone 114.

[0062] Bitstream 121 is transmitted by transmitter 122 as encoded data via communication channel 130. In some embodiments, bitstream 121 may be stored in a non-temporary storage device 132, such as flash memory or a hard disk drive, until it is later transmitted via communication channel 130 (or instead of being transmitted via communication channel 130). For example, the encoded video data may be provided to customers on demand via a wide area network (WAN) and used for video analysis applications.

[0063] The destination device 140 includes a receiver 142, a tensor decoder 146, a CNN head 150, and a CNN task result buffer 152. The receiver 142 receives encoded video data from the communication channel 130 and passes the bitstream 143 to the tensor decoder 146. The tensor decoder 146 outputs decoded tensors 149, which are supplied to the CNN head 150. The CNN head 150 receives the tensors 149 and runs subsequent layers of the neural network, starting from the CNN backbone 114, to generate task results 151. The task results 151 are stored in the task result buffer 152. The contents of the task result buffer 152 are presented to the user (e.g., via a graphical user interface) or provided to an analysis application, which then decides on some action based on the task results, which may include presenting an aggregated summary of the task results to the user. Furthermore, the functions of the transmitting device 110 and the receiving device 140 may also be implemented in a single device (for example, a mobile phone, tablet computer, or cloud application).

[0064] As shown in Figure 1, system 100 includes a tensor codec repository 180, which may contain network topologies encompassing various neural networks and associated partition points and reconstruction fidelity levels. Network topologies are stored in the tensor codec repository 180 for future reference (or use). The tensor codec repository 180 is accessed "out of band" or stored separately on source device 110 and destination device 140, respectively. Network topology identifiers 174 and 176 are transmitted by tensor encoder 116 and tensor decoder 146, respectively. Network topology identifiers 174 and 176 are used to determine a given network topology from bitstream 143.

[0065] As a result of a request for a predetermined network topology, network topologies 172 and 178 are returned to the tensor encoder 16 and tensor decoder 146, respectively. As described later, the information including the network topology identifier 178 is decoded by the tensor decoder 146 and used to generate a decoded tensor using the determined network topology. The tensor codec repository 180 is accessible via a public file repository or within a private network accessible to the source device 110 and the destination device 140.

[0066] Regardless of the example devices described above, the source device 110 and the destination device 140 may each be configured within a general-purpose computing system, typically through a combination of hardware and software components. Figure 2A shows such a computer system 200, which includes a computer module 201, input devices (such as a keyboard 202, a mouse pointer device 203, a scanner 226, a camera 227 (configurable as a video source 112), and a microphone 280), and output devices (including a printer 215, a display device 214, and a speaker 217). The computer module 201 may use an external modulator / demodulator (modem) transceiver device 216 to communicate with the communication network 220 via connection 221. The communication network 220 may represent a communication channel 130 and may be the Internet, a cellular communication network, or a wide area network (WAN) such as a private WAN. If connection 221 is a telephone line, the modem 216 may be a conventional "dial-up" modem. Alternatively, if connection 221 is a high-capacity (e.g., cable or fiber optic) connection, modem 216 may be a broadband modem. A wireless modem can also be used for a wireless connection to wireless communication network 220. Transceiver device 216 provides the functions of transmitter 122 and receiver 142, and communication channel 130 may be embodied in connection 221.

[0067] The computer module 201 typically includes at least one processor unit 205 and a memory unit 206. For example, the memory unit 206 may have semiconductor random access memory (RAM) and semiconductor read-only memory (ROM). The computer module 201 also includes the following input / output (I / O) interfaces: an audio-video interface 207 for connecting to a video display 214, a speaker 217, and a microphone 280; an I / O interface 213 for connecting to a keyboard 202, a mouse 203, a scanner 226, a camera 227, and optionally a joystick or other human interface devices (not shown); and an interface 208 for an external modem 216 and a printer 215. The signal from the audio-video interface 207 to the computer monitor 214 is typically the output from a computer graphics card. In some embodiments, the modem 216 may be integrated within the computer module 201 (e.g., within interface 208). The computer module 201 also has a local network interface 211, which allows the computer system 200 to connect to a local area communication network 222 (referred to as a LAN) via connection 223. As shown in Figure 2A, the local communication network 222 may also be connected to a wide area network 220 via connection 224, which typically includes a so-called "firewall" device or a device with similar functionality. The local network interface 211 may consist of an Ethernet® circuit card, a Bluetooth® radio device, or an IEEE 802.11 radio device, but many other types of interfaces may be used for interface 211. The local network interface 211 may also provide the functionality of a transmitter 122 and a receiver 142, and a communication channel 130 may be implemented within the local communication network 222.

[0068] I / O interfaces 208 and 213 may provide either serial or parallel connectivity, or both, the former typically implemented according to the Universal Serial Bus (USB) standard and equipped with a corresponding USB connector (not shown). A storage device 209 is provided, typically including a hard disk drive (HDD) 210. Other storage devices such as floppy disk drives and magnetic tape drives (not shown) may also be used. An optical disk drive 212, functioning as a non-volatile data source, is typically provided. Portable memory devices such as optical disks like CD-ROMs, DVDs, Blu-ray Discs (trademarks), USB-RAMs, portable external hard drives, and floppy disks can be used as suitable data sources for the computer system 200. Typically, the HDD 210, optical drive 212, and networks 220 and 222 can also be configured to operate as a video source 112 or as a storage location for decoded video data saved for playback via the display 214. Source devices 110 and destination devices 140 of system 100 may be implemented in the computer system 200.

[0069] The components 205 to 213 of the computer module 201 typically communicate via an interconnected bus 204, and the communication method implements a conventional operating mode of computer systems 200 known to experts in the relevant technical field. For example, the processor 205 is connected to the system bus 204 using connection 218. Similarly, the memory 206 and the optical disc drive 212 are connected to the system bus 204 by connection 219. Examples of computers capable of implementing the described configuration include IBM-PC and compatible machines, Sun's SPARCstation, Apple's Mac®, or equivalent computer systems.

[0070] The tensor encoder 116, the tensor decoder 146, and the method described later may be implemented as one or more software application programs 233 executable within a computer system 200. In particular, the steps of the tensor encoder 116, the tensor decoder 146, and the method described are implemented by instructions 231 (see Figure 2B) within the software 233 executed within the computer system 200. The software instructions 231 are formed as one or more code modules, each module performing one or more specific tasks. The software may also be divided into two distinct parts, the first part and its corresponding code module performing the method described, and the second part and its corresponding code module managing the user interface between the first part and the user.

[0071] The software is stored on a computer-readable medium, including, for example, a storage device described later. The software is loaded from the computer-readable medium into the computer system 200 and then executed by the computer system 200. Such a computer-readable medium on which software or a computer program is recorded is a computer program product. The use of the computer program product in the computer system 200 preferably provides an advantageous apparatus for implementing the source device 110 and the destination device 140 and the method described.

[0072] The software 233 is typically stored on the HDD 210 or in memory 206. The software is loaded from a computer-readable medium into the computer system 200 and executed by the computer system 200. For example, the software 233 may be stored on an optical disc storage medium (e.g., a CD-ROM) 225 that is read by the optical disc drive 212.

[0073] In some cases, the application program 233 may be encoded on one or more CD-ROMs 225 and provided to the user, and loaded via the corresponding drive 212, or loaded by the user from the network 220 or 222. Furthermore, the software may also be loaded into the computer system 200 from other computer-readable media. Computer-readable storage media refers to any non-temporary, tangible storage medium that provides the computer system 200 with recorded instructions and / or data for execution and / or processing. Examples of such storage media include floppy disks, magnetic tapes, CD-ROMs, DVDs, Blu-ray discs (trademarks), hard disk drives, ROMs or integrated circuits, USB memory sticks, magneto-optical disks, or computer-readable cards such as PCMCIA cards, whether these devices are inside or outside the computer module 201. Examples of temporary or intangible computer-readable transmission media that may be used to provide software, application programs, instructions, and / or video data or encoded video data to the computer module 201 include wireless or infrared transmission channels, network connections to other computers or network-connected devices, and the internet or intranets, including email transmissions and information recorded on websites.

[0074] The second part of the application program 233 described above and its corresponding code module are executed to implement one or more graphical user interfaces (GUIs) that are drawn or otherwise represented on the display 214. Users of the computer system 200 and the application can operate the interface in a functionally adaptable manner, typically by operating the keyboard 202 and mouse 203, and provide control commands and inputs to the application associated with the GUI. Other forms of functionally adaptable user interfaces can also be implemented, such as a voice interface that outputs voice prompts from the speaker 217 and takes user voice commands via the microphone 280.

[0075] Figure 2B is a detailed schematic block diagram of the processor 205 and “memory” 234. Memory 234 represents the logical collection of all memory modules (including storage devices 209 and semiconductor memory 206) accessible by the computer module 201 in Figure 2A.

[0076] When the computer module 201 is first powered on, the Power-On Self-Test (POST) program 250 is executed. The POST program 250 is typically stored in ROM 249 within the semiconductor memory 206 shown in Figure 2A. Hardware devices like ROM 249 that store software are sometimes called firmware. The POST program 250 inspects the hardware within the computer module 201, verifies its normal operation, and typically checks the correct operation of the processor 205, memory 234 (209, 206), and the Basic Input / Output System Software (BIOS) module 251 (also typically stored in ROM 249). If the POST program 250 executes successfully, the BIOS 251 starts the hard disk drive 210 shown in Figure 2A. The startup of the hard disk drive 210 causes the bootstrap loader program 252, which resides on the hard disk drive 210, to be executed via the processor 205. This loads the operating system 253 into RAM memory 206, and the operating system 253 begins operation. Operating System 253 is a system-level application runnable by Processor 205, performing various high-level functions including processor management, memory management, device management, storage management, software application interface, and general-purpose user interface.

[0077] The operating system 253 manages memory 234 (209, 206) to ensure that each process or application running on the computer module 201 has enough memory to run without conflicting with memory allocated to other processes. Furthermore, the different types of memory available in the computer system 200 in Figure 2A must be used appropriately so that each process can operate effectively. Therefore, the aggregated memory 234 (unless otherwise specified) does not indicate how a particular segment of memory is allocated, but rather provides a general view of the memory accessible to the computer system 200 and how that memory should be used.

[0078] As shown in Figure 2B, the processor 205 comprises several functional modules, including a control unit 239, an arithmetic unit (ALU) 240, and local or internal memory 248 (sometimes called cache memory). The cache memory 248 typically contains several storage registers 244-246 within its register section. One or more internal buses 241 functionally interconnect these functional modules. The processor 205 also has one or more interfaces 242, as it typically communicates with external devices via the system bus 204 through connection 218. Memory 234 is connected to bus 204 using connection 219.

[0079] The application program 233 includes a set of instructions 231, which may include conditional branch instructions and loop instructions. The program 233 may also include data 232 used to execute the program 233. The instructions 231 and data 232 are stored in memory locations 228, 229, 230 and 235, 236, 237, respectively. Depending on the relative size of the instructions 231 and memory locations 228-230, a particular instruction may be stored in a single memory location, as indicated by memory location 230. Alternatively, the instruction may be divided into multiple parts, each part stored in a separate memory location, as indicated by the instruction segments shown in memory locations 228 and 229.

[0080] Generally, the processor 205 receives a set of instructions to be executed internally. The processor 205 waits for the next input and, in response to that input, executes another set of instructions. Each input is provided from one of several sources, which include data generated by one of the input devices 202, 203, data received from an external source via one of the networks 220, 202, data obtained from one of the storage devices 206, 209, or data obtained from a storage medium 225 inserted into the corresponding reader 212, all of which are shown in Figure 2A. The execution of an instruction set may, in some cases, result in the output of data. It may also involve saving data or variables to memory 234.

[0081] The tensor encoder 116, tensor decoder 146, and the described method use an input variable 254, which is stored in corresponding memory locations 255, 256, and 257 in memory 234. The tensor encoder 116, tensor decoder 146, and the described method generate output variables 261, which are stored in corresponding memory locations 262, 263, and 264 in memory 234. Intermediate variables 258 may be stored in memory locations 259, 260, 266, and 267.

[0082] Referring to processor 205 in Figure 2B, registers 244, 245, 246, arithmetic logic unit (ALU) 240, and control unit 239 work together to execute a sequence of microoperations necessary to perform a "fetch, decode, execute" cycle for each instruction in the instruction set that makes up program 233. Each fetch, decode, execute cycle includes: fetch operation (fetching or reading instruction 231 from memory locations 228, 229, 230), decode operation (control unit 239 determines the fetched instruction), and execute operation (control unit 239 and / or ALU 240 execute the instruction).

[0083] Subsequently, further fetch, decrypt, and execute cycles for the next instruction may be performed. Similarly, a store cycle may be performed in which the control unit 239 stores or writes a value to memory location 232.

[0084] Each step or subprocess of the method shown in Figures 18 and 19 is associated with one or more segments of program 233, and typically the registers 244, 245, 246, ALU 240, and control unit 239 in the processor 205 work together to perform a fetch, decode, and execute cycle for each instruction in the instruction set for the corresponding segment of program 233.

[0085] Figure 3A is a schematic block diagram 300 showing the functional module of the CNN backbone portion 310, which can function as an implementation of the CNN backbone 114 when system 100 is configured to run a "YOLOv3" network. The backbone portion 114 is sometimes called "DarkNet-53," but different backbones are also available, resulting in different numbers of layers and dimensions in the tensor 115 for each frame. In one implementation, the backbone portion 310 is used as a person detector for object tracking.

[0086] As shown in Figure 3A, the video data 113 is passed to the resizer module 304. The resizer module 304 resizes each frame of the video data 113 to a resolution suitable for processing by the CNN backbone 310, generating resized frame data 312. If the resolution of the video data 113 is already suitable for the CNN backbone 310, the operation of the resizer module 304 is unnecessary. The resized frame data 312 is passed to the convolutional batch normalized leaky rectified linear (CBL) module 314, which generates a tensor 316. The CBL module 314 includes the module described with respect to the CBL module 360 ​​shown in Figure 3D.

[0087] The CBL module 360 ​​receives a tensor 361 of the resized frame data 312 as input. Tensor 361 is passed to the convolutional layer 362 to generate tensor 363. If the stride of the convolutional layer 362 is 1, tensor 363 has the same spatial dimensions as tensor 361. If the stride of the convolutional layer 362 is greater than 2, tensor 363 will have smaller spatial dimensions than tensor 361; for example, with a stride of 2, the width and height will be halved. Regardless of the stride, in a particular CBL block, the size of the channel dimensions of tensor 363 may differ from the channel dimensions of tensor 361. Tensor 363 is passed to the batch normalization module 364 to output tensor 365. The batch normalization module 364 normalizes the input tensor 363 and applies scaling coefficients and offset values ​​to generate output tensor 365. The scaling coefficients and offset values ​​are derived from the learning process. Tensor 365 is passed to the leaky rectified linear activation ("LeakyReLU") module 366, which generates tensor 367. Module 366 provides a "leaky" activation function, which allows positive values ​​of the tensor to pass through unchanged, while significantly reducing negative values ​​(for example, to about 0.1 times the original value).

[0088] Returning to Figure 3A, tensor 316 is passed from CBL block 314 to residual block module 320 (e.g., the "res1+2+8" module, also known as the res11 module), which is a contiguous set of three residual blocks, each containing one residual unit, two residual units, and eight residual units, respectively. In each residual block (see Figure 3B), the spatial resolution of the tensor is halved both horizontally and vertically by a stride 2 convolution by CBL block 344.

[0089] The residual block will be explained with reference to ResBlock340 shown in Figure 3B. ResBlock340 receives tensor 341. Tensor 341 is zero-padding by the zero-padding module 342 to produce tensor 343. Tensor 343 is passed to the CBL module 344 to produce tensor 345. The CBL module 344 includes a convolution (e.g., 362) with a stride parameter set to 2, resulting in tensor 345 having half the width and height of tensor 343. Tensor 345 is passed to the residual unit 346. The residual unit 346 contains a series of residual units concatenated based on the number of residual blocks (e.g., 11 units for block 320). The last residual unit of residual unit 346 outputs tensor 347.

[0090] The residual unit is explained with reference to the ResUnit350 shown in Figure 3C. The ResUnit350 receives tensor 351 as input. Tensor 351 is passed to the CBL module 352 to generate tensor 353. Tensor 353 is passed to a second CBL unit 354 to generate tensor 355. The additive module 356 adds tensor 355 and tensor 351 to generate tensor 357. The additive module 356 is also called a "shortcut" because the input tensor 351 effectively influences the output tensor 357. In an untrained network, the ResUnit350 simply passes the tensor through. As training takes place, CBL modules 352 and 354 work to separate tensor 357 from tensor 351 based on the training data and true data.

[0091] Returning to Figure 3A, the Res11 module 320 outputs tensor 322. Tensor 322 is output from the backbone module 310 as one of the layers and is also supplied to the Res8 module 324. The Res8 module 324 is a residual block (i.e., 340) and contains eight residual units (i.e., 350). The Res8 module 324 generates tensor 326. Tensor 326 is passed to the Res4 module 328 and is output from the backbone module 310 as one of the layers. The Res4 module is a residual block (i.e., 340) and contains four residual units (i.e., 350). The Res4 module 328 generates tensor 329. Tensor 329 is output from the backbone module 310 as one of the layers. The layer tensors 322, 326, and 329 are output collectively as tensor 115a and may be referred to as layers 0-2 or L0, L1, and L2, respectively. The backbone CNN310 takes a video frame with a resolution of 1088x608 as input and generates three tensors corresponding to three layers, with dimensions as follows: [1,256,76,136], [1,512,38,68], [1,1024,19,34]. Another example of the three tensors 115a corresponding to the three layers is [1,512,34,19], [1,256,68,38], [1,128,136,76], which are separated by layer indices 75, 90, and 105 respectively when the layers are enumerated according to the YOLOv3 software implementation of the backbone 300 and head 1200.

[0092] Res11 320, Res8 324, and Res4 328 operate similarly to ResBlock340, respectively. CBL314, CBL344, and CBL354 operate similarly to CBL360, respectively.

[0093] Figure 4 is a schematic block diagram showing the functional modules of the alternative backbone portion 400 of the CNN, which can function as an implementation of the CNN backbone 114 when system 100 is configured to run the “FasterRCNN” or “MaskRCNN” ResNet101 network. Frame data 113 is input and passes through tensors 409, 413, 417, 421, and 425 to the stem network 408, res2 module 412, res3 module 416, res4 module 420, and res5 module 424, respectively. The backbone portion 400 can be used as part of a general object detector or for instance segmentation, supporting various classes of objects.

[0094] The stem network 408 includes a 7x7 convolution with a stride of 2 and a maximum pooling operation. The res2 module 412, res3 module 416, res4 module 420, and res5 module 424 perform convolution operations such as LeakyReLU activation. Each module 412, 416, 420, and 424 also halves the width and height of the processed tensor by setting the stride to 2. Tensors 413, 417, 421, and 425 are passed to one of the 1x1 horizontal convolution modules 446, 444, 442, and 440, respectively. Modules 446, 444, 442, and 440 generate tensors 447, 445, 443, and 441, respectively. Tensor 441 is passed to the 3x3 output convolution module 470, which generates the output tensor P5 471.

[0095] Tensor 441 is then passed to the upsampling module 450, which generates the upsampled tensor 451. The summing module 460 adds tensors 443 and 451 to generate tensor 461. Tensor 461 is then passed to the upsampling module 452 and the 3x3 horizontal convolution module 472. Module 472 outputs the P4 tensor 473. The upsampling module 452 generates the upsampled tensor 453. The summing module 462 adds tensors 445 and 453 to generate tensor 463. Tensor 463 is then passed to the 3x3 horizontal convolution module 474 and the upsampling module 454. Module 474 outputs the P3 tensor 475. The upsampling module 454 outputs the upsampled tensor 455. The summing module 464 sums tensors 447 and 455 to generate tensor 465, which is passed to the 3x3 horizontal convolution module 476. Module 476 outputs the P2 tensor 477. The upsampling modules 450, 452, and 454 use nearest neighbor interpolation to reduce computational complexity. Tensors 471, 473, 475, and 477 constitute the output tensor 115a of the CNN backbone 400. Figure 4 shows a specific backbone portion ("P-layer split point") of the FasterRCNN network architecture, but the split into the backbone and head can be done in different ways. Splitting the network at tensor 409 is called the "stem" split point. Splitting the network at tensors 447, 445, 443, and 441 is called the "C-layer" split point.

[0096] Figure 5 is a schematic block diagram of a tensor encoder 116 using a configurable tensor compression stage. Figure 18 shows a method 1800 that executes the first part of a CNN, selects a tensor compressor, compresses a tensor using the selected tensor compressor, and encodes the resulting compressed tensor. The tensor encoder 116 and method 1800 can be implemented as one or more software application programs 233 that can be executed within a computer system 200. The tensor encoder 116 and method 1800 are implemented by instructions 231 (see Figure 2B) in software 233 that runs within the computer system 200. The software instructions 231 are formed as one or more code modules, each module performing one or more specific tasks. Method 1800 begins with step 1805 of selecting a set of tensor compressors / decompressors.

[0097] In step 1805, the tensor compressor selector 510 selects a set of mechanisms available for compressing and decompressing intermediate tensors under the execution of processor 205. Each mechanism in this set forms a “bottleneck” and corresponds to a structure in which the encoder network topology and the decoder network topology are coupled. The interface between the encoder network topology and the decoder network topology is the narrowest and therefore the layer with the most advanced dimensionality reduction. This interface also contains one or more tensors called “compressed tensors.” One or more tensors are generated from operations such as convolution, batch normalization, activation functions, matrix multiplication, tensor addition and / or subtraction. The dimensions of the tensors in the interface may differ with each call to method 1800 (e.g., the number of channels may vary). Supporting multiple mechanisms within a single bitstream allows for adaptation to changing network conditions and application requirements by dynamically switching between mechanisms. For example, an object-partitioned network may operate using a mechanism that provides a low bitrate at the expense of output quality (resulting in a decrease in mAP) of task results 151 from the CNN head 150.

[0098] If a user of task result 151 determines that a particular bitstream contains high-interest content (e.g., audio data, video data, image data, text data), they may request source device 110 to improve the quality (and therefore the bitrate) of bitstream 121. The dimension of the intermediate tensor (e.g., tensor 115) depends on the specific neural network executed by system 100 and whether the network is divided into a first and second part.

[0099] Each mechanism selected in step 1805 must match the dimensions of tensor 115 at the input to the encoder network topology with the dimensions of tensor 115 at the output of the decoder network topology. The compressed form of the tensor (i.e., the "bottleneck" point, or output from the encoder network topology and input to the decoder network topology) may have different numbers or dimensions. If the mechanism involves the use of learnable elements such as convolution, the tensor compressor selector 510 also determines the selected weights 516 used in the encoder network topology and decoder network topology. For a given encoder network topology and decoder network topology, multiple weights may be available, for example, different weights targeting different quality operating points. Control in processor 205 proceeds from step 1805 to the tensor compressor / decompressor selection step 1810.

[0100] In step 1810, under the execution of processor 205, tensor compressor selector 510 selects a mechanism to be used for compressing and decompressing tensor 115 and outputs the selected tensor decompressor 512 and associated metadata 520. The selection in step 1810 is made from the set determined in step 1805. This selection may be the result of a request from the destination device (device) via an out-of-band signaling mechanism to increase or decrease the decoding quality (and thus the bitrate) of bitstream 121. If the mechanism is parameterizable (e.g., the number of channels in the compressed tensor can be dynamically changed), an appropriate value is selected in step 1810.

[0101] System 100 executes a predetermined neural network, which is divided into a first part executed by the CNN backbone 114 and a second part executed by the CNN head 150. The first part of the neural network may be the Darknet-53 backbone described in Figures 3A-3D, the backbone of the FasterRCNN or MaskRCNN network described in Figure 4, or the first part of another neural network. The number and dimensionality of the tensor 115 depend on the network implemented in System 100 and the division of the network into a first part executed by the CNN backbone 114 and a second part executed by the CNN head 150. Such a mechanism includes an encoder network topology executed by the transmitting device 110 and a decoder network topology executed by the receiving device 140. The encoder and decoder network topologies may involve the use of trained layers such as convolutions, in which case weights are also required. The encoder network topology and decoder network topology form a "bottleneck" between the first network portion (i.e., the CNN backbone 114) and the second network portion (i.e., the CNN head 150), with the interface between the encoder network topology and the decoder network topology being the "narrowest" part of this bottleneck, and therefore the lowest bitrate when compressed in the form of packed video frames. By dynamically selecting, notifying, and activating different mechanisms (encoder network topology and corresponding decoder network topology), the bitrate of bitstream 121 can be adapted to network conditions or adjusted to meet quality application requirements. Mechanisms that provide higher quality generally have larger compression tensor dimensions and therefore require a larger packed frame area, resulting in a higher bitrate for bitstream 121. Control in processor 205 proceeds from step 1820 to tensor compressor instantiation step 1815.

[0102] In step 1815, the source device 110, under the execution of the processor 205, retrieves the tensor structure description 522 from the tensor codec repository 514 based on the selected tensor decompressor 512 and instantiates the tensor structure description 522 into an executable form as a tensor compressor 530. In the instantiation step 1815, the necessary memory may be declared and the data structure initialized within the memory 205 associated with the tensor structure description 522, or resources may be allocated within a reconfigurable device such as a field-programmable gate array (FPGA). The operations defined in the tensor structure description 522 may be converted into a form suitable for execution by the processor 205 as part of the instantiation step. Just-in-time compilation is one technique for obtaining a representation such as "bytecode" that the processor 205 can execute more quickly than directly interpreting the tensor structure description 522 and performing each tensor compression operation. If the instantiated tensor compressor 530 includes learnable elements such as convolutions, the tensor weight repository 518 is accessed to obtain the necessary weights 524 to be used in the learnable elements. The tensor codec repository 514 and the tensor weight repository 518 may be supplied with data from the tensor codec repository 180. The processor 205 controls the signal tensor decompression from step 1815 to step 1820.

[0103] In step 1820, the metadata encoder 544 under the execution of processor 205 encodes the selected tensor decompressor 520 into bitstream 121 as a decoder network topology 1192 in a Supplementary Enhancement Information (SEI) message 1113, as described with reference to Figure 11 and Appendix A. The selected tensor decompressor 520 may be signaled according to the “FCVCM Decoder Information SEI Message” syntax described in Appendix A. The selected tensor decompressor 520 may be signaled as an explicit network topology using a text representation (or syntax) such as the Kronos Group’s Open Neural Network Exchange ("ONNX") format or Neural Network Exchange Format ("NNEX"), or even as a short code snippet such as a Pytorch function. Such text representations can be compressed using techniques such as the “DEFLATE” or “LZMA” algorithms to reduce metadata overhead when stored in bitstream 121. Thus, the network topology is signaled in a compressed form using the aforementioned syntax. The network topology defines the operations to be performed at the destination device 140 to convert the tensor from the compressed representation back to its original numerical values ​​and dimensions, thereby supplying the uncompressed tensor to the CNN head 149. Network weights may also be signaled in SEI messages using formats such as ISO / IEC 15938-17 "Compression of Neural Networks for Multimedia Description and Analysis". When a particular network topology is signaled, it is registered at the destination device 140 and referenced and activated in subsequent bitstreams, eliminating the need to signal the network topology for each bitstream. In some configurations, system 100 can provide a predetermined network topology that does not need to be explicitly signaled within the bitstream. The predetermined network topology is activated at the destination device 140 via a reference to an identifier.A predetermined network topology is provided to the destination device 140 via external means, such as downloading from a network topology repository or registry. Network topology repositories and registries may be publicly accessible or accessible within a specific private scope, for example, by accessing a private network or secure network (e.g., VPN) that is available to an instance of the destination device 140 but not to the general public. If it is known that the destination device 140 has access rights to the desired network topology (already registered or downloadable from an external server), the source device 110 can encode a reference to identify and activate the specific network topology to be used when decoding the bitstream 121. Control of the processor 205 proceeds from step 1820 to the complexity display determination step 1830.

[0104] In step 1830, the source device 110, under the execution of the processor 205, determines a representation representing the worst-case complexity of all possible decoder network topologies that could be signaled for bitstream 121. If the source device 110 selects one of several possible decoder network topologies in step 1810, it is desirable for the destination device 140 to know whether the entire bitstream is decodeable when decoding of bitstream 121 begins. The decoder network topology initially signaled for bitstream 121 may not be the most complex topology used to decode that bitstream 121. For example, system 100 may start operating in a low bitrate mode and then increase the bitrate (and the required decoder network topology) based on some criteria. Aspects of decoder network complexity include the number of multiplication-accumulation (MAC) operations and the number of weights required to execute the decoder network topology. The decoder network complexity representation is configured to show the highest complexity among all possible decoder network topologies that the transmitting device 110 could instruct the receiving device 140 to execute. The decoder network complexity representation may be based on a decoding capability representation. In some configurations, the decoder network complexity representation may be a scalar value mapped to each aspect of network complexity. For example, the network complexity representation is a scalar value linked to aspects such as MAC counts and weight counts by referring to a lookup table, and is set to correspond to the worst-case for each aspect of the set of decompressors determined in step 1805. Control of the processor 205 proceeds from step 1830 to the coding complexity representation step 1840.

[0105] In step 1840, the metadata encoder 544 encodes the determined network complexity representation into a bitstream 121 as level information 1192 in the SEI message 1113 described in Figure 11 and Appendix A, under the execution of the processor 205. The determined network complexity representation may be encoded as a single fixed-length codeword or may take the form of multiple values, each corresponding to a different aspect of complexity, for example, a representation for the maximum MAC count and another representation for the maximum weight memory requirement. Control of the processor 205 proceeds from step 1840 to step 1850 of the first part of neural network execution.

[0106] In step 1850, the CNN backbone 114, under the control of processor 205, takes frame data 113 from video source 112 as input and executes the first part of the neural network. Control of processor 205 then proceeds from step 1850 to tensor compression execution step 1860.

[0107] In step 1860, under the execution of processor 205, tensor compressor 530 compresses tensor 115 to produce compressed tensor 532. Compressed tensor 532 has fewer elements and reduced dimensionality than tensor 115 (i.e., the number of channels and / or the width and height of the feature maps are reduced). Tensor compressor 530 implements tensor compressor 512 instantiated in pre-compiled "bytecode", machine code, or other forms suitable for direct execution by processor 205, including execution by part of processor 205 or an associated inference engine (e.g., graphics processing unit (GPU)). Control of processor 205 proceeds from step 1860 to the encoding packing form step 1870.

[0108] In step 1870, the metadata encoder 544 encodes tensor information 1195, which defines the dimensionality and feature map arrangement of the compressed tensor 532, as packing information 1194 when each tensor of the compressed tensor 532 is placed within the video frame 540. The packing information 1194 and tensor information 1195 are described with reference to Figure 11 and Appendix A. The tensor information 1195 includes the maximum number of channels and the number of channels used for each tensor. The frame region of a region must be large enough to pack the tensors within that region up to the maximum number of channels, i.e., the maximum number of feature maps. The maximum number of channels must not be changed in frames within the bitstream 121 associated with the same SPS 1110. The topology of the encoder network may generate fewer feature maps than the maximum value, and the number of channels used signals the number of channels actually used, which may vary from frame to frame. The video frame 540 is divided into regions, such as rectangular regions, and each region is used to hold feature maps from one or more tensors of the compressed tensor 532. In one configuration, method 1800 performs the step in step 1870 of determining the type of feature map (e.g., DC, coefficients, basis vectors, MSFC) required for subsequent input to tensor decoder 146. The assignment of the determined feature map type (placement in video frame 540) is also determined in step 1870, and this assignment specifies one or more tensors for each subpicture in video frame 540. Such “packing format” information is illustrated with reference to Figures 9A and 9B, and Figures 10A and 10B. Control in processor 205 proceeds from step 1870 to tensor quantization step 1880.

[0109] In step 1880, under the control of processor 205, the quantization module 534 quantizes the floating-point values ​​in each tensor of the compressed tensor 532 to produce a quantized compressed tensor 536. The quantized compressed tensor 536 has integer values ​​that take values ​​within a sample range defined by the operating bit depth of video encoder 542. For example, when encoding video with 10-bit samples, integer values ​​are allowed in the range [0, 1023]. For each tensor, the minimum floating-point value and the maximum floating-point value form the quantization range 526. Control of processor 205 proceeds from step 1880 to step 1890, which encodes the quantization range.

[0110] In step 1890, the metadata encoder 544, under the execution of processor 205, encodes the quantization range 526 for each tensor of the compressed tensor 532 as a quantization range 1196 into the bitstream 121. The quantization range is used within the bitstream 121 to enable inverse quantization to the correct range by the destination device 140. Control of processor 205 proceeds from step 1890 to tensor pack step 18100.

[0111] In step 18100, the packer module 538, under the execution of processor 205, packs the feature maps of each tensor of the compressed quantized tensor 536 into the video frame 540. The tensor feature maps are packed into the regions within the video frame 540 to which the tensor is assigned, generally from left to right, then from top to bottom. Regions are defined by SEI message 1113, but regions may correspond to "encoded layer" structures (i.e., lower layers of metadata such as SEI message 1113) such as subpictures, slices, and tiles. The encoded level structure of the frame is inaccessible outside the decoder 800 and may differ depending on the compression standard used, such as VVC, HEVC, and AV-1. Independent signaling of regions in SEI message 1113 eliminates dependence on specific functions in the encoding layer. Each region must be large enough to accommodate all the feature maps of the tensors packed into that region. Examples of region-based packing formats are illustrated with reference to Figures 9A and 9B, and Figures 10A and 10B. The control of the processor 205 proceeds from step 18100 to frame encoding step 18110.

[0112] In step 18110, the video encoder 542, under the execution of processor 205, compresses the video frame 540 and generates the encoded layer portion of the bitstream 121. The operation of video encoder 542 is described with reference to Figure 8. Method 1800 is completed and processing proceeds to the next instance of source data 113 (for example, the next frame from video source 112).

[0113] Figure 6 is a schematic block diagram showing a type of multiscale feature fusion (MSFF) module 600 that can function as a tensor compressor 530. The MSFF module 600 takes an intermediate tensor 113 and generates a compressed tensor 115 with reduced dimensionality compared to the intermediate tensor 113, thereby reducing the bitrate when encoded as part of a packed frame. The MSFF module 600 uses a trained network layer, and the corresponding module in the tensor decoder 146 needs to restore the tensor dimensions, so that the tensor 149 can be fed to the CNN head 150. The MSFF module 600 takes four tensors as input, requiring each tensor to have 256 channels, so that the MSFF module 600 is compatible with the P layer of a FasterRCNN or MaskRCNN network. However, variants of the MSFF module 600 that support different numbers of layers and channels also exist.

[0114] The MSFF module 600 outputs a single tensor with a feature map size corresponding to the P5 layer identified on the input side and 64 channels, although variations with different numbers of channels also exist. Each variation of the MSFF module 600 requires different weights for proper operation. If multiple variations of the MSFF module 600 are available in system 100 for a given network, the packing format is set to the worst-case feature map size of the compressed tensor of the decoder network topology currently in use, and the actual number of channels used is updated at runtime as part of the tensor information 1195.

[0115] The MSFC module 600 includes an MSFF block 610, shown in Figure 6, which generates a single tensor from multiple tensors 115 using one or more downsampling filters. Under the execution of the processor 205, the MSFF block 610 combines each tensor of the first group of tensors (i.e., 602, 603, 604, 605) to generate a combined tensor 629. The combined tensor 629 forms a representation of the FPN layer tensor. Downsampling modules 622a, 622b, and 622c process tensors with larger spatial scales, namely P4 604 of (2h,2w,256), P3 603 of (4h,4w,256), and P2 602 of (8h,8w,256), respectively. Modules 622a, 622b, and 622c perform downsampling to match the spatial scale of the smallest tensor (i.e., P5 605 of (h,w,256)), generating reduced P5 tensors 623a, 623b, and 623c, respectively. The concatenation module 624 concatenates tensors 605, 623a, 623b, and 623c channel by channel to generate a concatenated tensor 625 of dimension (h,w,1024). The concatenated tensor 625 is passed to the squeeze-excitation (SE) module 626 to generate tensor 627. The SE module 626 sequentially executes global pooling, a fully connected layer to reduce the number of channels, a rectifying linear activation unit, a second fully connected layer to restore the number of channels, and a sigmoid activation function to generate a scaling tensor. Tensor 625 is scaled according to the scaling tensor, generating tensor 627 as the output. SE block 626 can be learned to adaptively change the weighting of different channels in the tensor it passes through, based on the output of the first fully connected layer.

[0116] The output of the first fully connected layer reduces the feature map of each channel to a single value. Each single value passes through a nonlinear activation unit (ReLU) to generate a conditional representation of the single value suitable for weighting the other channels, and the reconstruction of the total number of channels is performed by the second fully connected layer. Thus, SE block 626 can extract nonlinear inter-channel correlations in the process of generating tensor 627 from tensor 625 to a degree that would not be possible with convolutional (linear) layers alone. Tensor 627 is passed to convolutional layer 628. Convolutional layer 628 implements one or more convolutional layers to generate a combined tensor 629, in which case the number of channels is reduced to F channels (typically 256 channels, i.e., F=256). Further reduction of the number of channels is achieved by the single-scale feature compression (SSFC) module 650.

[0117] The SSFC module 650 receives tensor 629 and applies convolution 652 to reduce the number of channels from F(256) to C'' (nominally set to 64 channels) to generate tensor 653. Tensor 653 is passed to batch normalization module 654 to generate batch normalized tensor 655, which is passed to hyperbolic tangent activation layer 656 to generate compressed tensor 532. The output of MSFC module 600 is a single tensor per frame with a fixed feature map size and a fixed number of channels.

[0118] Figure 7 is a schematic block diagram showing a channel-to-channel decorrelation-based tensor compressor 700 that can function as a tensor compressor 530. The tensor compressor 700 operates without requiring pre-trained layers and relies on extracting channel redundancy at runtime (i.e., it uses input tensor data from the CNN backbone 114 rather than relying on pre-trained weights). The tensor compressor 700 outputs three types of data: mean, basis vectors, and coefficients. The tensor compressor 700 compresses one tensor, and multiple tensors can be compressed by instantiating multiple instances of the tensor compressor 700 as tensor compressors 530. When multiple instances of the tensor compressor 700 are instantiated, tensors of the same type from each instance may be packed into the same region. For example, basis vectors from each instance may be packed into a single region. These data types are held within the tensor and packed into three separate regions within the frame. The number of basis vectors used can be changed during frame processing, and the number of coefficients used can also be changed, but the number of coefficients must not exceed the number of basis vectors. Therefore, the dimensions of the basis vectors and / or coefficients are updated as needed to inform the decoder of the contents of the decoded packed frame. As shown in Figure 7, the extracted channel averaging module 710, under the execution of processor 205, performs an averaging operation over the spatial dimensions (width and height) of tensor 115 to produce a per-channel average 712 as a one-dimensional tensor of C (number of channels in tensor 115). The extracted channel averaging module 710 may operate less frequently than for each received tensor 115, in which case the channel average 712 is updated only for a particular frame. Packed frames containing the updated channel average 712 are indicated by the number of non-zero channels in tensor 712 via tensor information 1195, or by a separate tensor update flag included in SEI message 1113.

[0119] For each channel of tensor 115, the subtraction module 714 performs a DC shift, which subtracts a constant value obtained from the average channel 712 for each feature map from each spatial position in the feature map. The subtracted value becomes the corresponding value in the average reconstructed channel 716. As a result of the subtraction module 714, a zero-centered tensor 716 is output, from which the DC component has been removed from each feature map.

[0120] The decomposition module 718 operates under the execution of processor 205 and generates a basis vector set 720 for the tensor set 115. The decomposition module 718 takes a zero-centered tensor 716 as input and generates the basis vector set 720 by performing a principal component analysis (PCA) technique such as singular value decomposition (SVD). Since each basis vector maps all channels to a single value using an inner product operation, if the tensor 115 has 256 channels, the dimension of one basis vector is 256 × 1. When the decomposition module 718 generates the first N basis vectors (e.g., 25), the dimension of the resulting basis vectors is 256 × N or 256 × 25. Because the basis vectors are relative to the origin, a zero-mean tensor 712 is used to ensure that an orthogonal basis is obtained. Each basis vector is a vector that associates all channels with a reduced group of channels. Therefore, the basis vector set makes it possible to represent tensor data spanning all channels with a smaller set of basis vectors. Each basis vector is derived by considering all samples of each feature map in a particular channel. Vector set 720 contains fewer basis vectors than the number of channels in one tensor of tensor set 115, which corresponds to a reduction in the dimensionality of the tensors of tensor set 115. Basis vector set 720 represents the tensor of tensor set 115 in a subspace that explains (explains) the maximum amount of variance that the tensor of tensor set 115 has, given the number of components of basis vector set 720. The basis vectors are ordered from the vector with the largest explanatory variable to the vector with the smallest explanatory variable. In other words, basis vector set 720 makes it possible to represent the tensor of tensor set 115 with minimal quality degradation for a given number of components, where the components refer to the top N basis vectors.

[0121] As shown in Figure 7, the inner product module 722, under the execution of the arithmetic processing unit 205, performs the inner product of each channel of the tensor 716 and each basis vector 720 to generate the coefficient tensor 724. The coefficients form the tensor 724, whose width and height are the same as the tensor of the tensor group 115, but the number of channels corresponds to the number of components (or basis vectors) generated by the decomposition module 718, and is at most less than or equal to that, which is less than the number of channels in the tensor of the tensor group 115. The tensor compressor 700 may change the number of channels of the coefficient tensor 716 in a given frame. The number of channels of the coefficient tensor 716 in a given frame is contained in the tensor information 1195 encoded in the instance of the SEI message 1113 associated with that frame. Each value of the coefficient tensor 724 represents the contribution of each basis vector in the reproduction of each value in each feature map.

[0122] Figure 8 is a schematic block diagram showing the functional modules of the video encoder 542. The video encoder 542 may be implemented using a general-purpose computer system 200, as shown in Figures 2A and 2B, in which case the various functional modules may be implemented by dedicated hardware within the computer system 200, or by software executable within the computer system 200, such as one or more software code modules of a software application program 233 that resides on a hard disk drive 205 and whose execution is controlled by the processor 205. Alternatively, the video encoder 542 may be implemented by a combination of dedicated hardware and software executable within the computer system 200. The video encoder 542 and the described method may also be implemented by dedicated hardware, such as one or more integrated circuits that perform the functions or subfunctions of the described method. Such dedicated hardware may include a graphics processing unit (GPU), a digital signal processor (DSP), an application-specific standard product (ASSP), an application-specific integrated circuit (ASIC), an FPGA, or one or more microprocessors and associated memory. In particular, the video encoder 542 includes modules 810-890, which can each be implemented as one or more software code modules of the software application program 233.

[0123] The video encoder 542 in Figure 8 is an example of a General-Purpose Video Coding (VVC) video encoder, but other video codecs may also be used to perform the processing steps described herein. For example, HEVC may be used. The example described generates a bitstream of encoded data. If other codecs are used, some implementations may pack the data into a different format, such as a frame format. The video encoder 800 receives frame data 712, each containing one or more color channels. The frame data 712 may have any chroma format and bit depth supported by the profile in use, for example, 4:0:0, 4:2:0 in the VVC standard's "Main10" profile, with sample precision ranging from 8 bits to 10 bits.

[0124] As shown in Figure 8, the block divider 810 first divides the frame data 712 into CTUs, which are generally square and configured to use a specific size. The maximum effective size of a CTU is, for example, 32×32, 64×64, or 128×128 luma samples, set by the "sps_log2_ctu_size_minus5" syntax element in the "sequence parameter set". Since any CTU that is not divided further contains one CU, the CTU size also provides the maximum CU size. The block divider 810 further divides each CTU into one or more CBs according to the luma coding tree and chroma coding tree. Luma channels are sometimes called primary color channels. Each chroma channel is sometimes called a secondary color channel. CBs can be of various sizes and may include both square and non-square aspect ratios. However, in the VVC standard, the side lengths of CBs, CUs, PUs, and TUs are always powers of 2. Therefore, the current CB (represented by 812) is output from the block partitioner 810 based on the luma-encoded tree and chroma-encoded tree of the CTU, following an iterative process on one or more blocks of the CTU.

[0125] The CTUs obtained from the first split of frame data 712 may be scanned in raster scan order and grouped into one or more “slice.” A slice may be an “intra” (or “I”) slice. An intra slice (I slice) indicates that all CUs within that slice are intra-predicted. Generally, the first picture of a coded layer video sequence (CLVS) contains only I slices and is called an “intra picture.” A CLVS may contain periodic intra pictures, which form “random access points” (i.e., intermediate frames in the video sequence from which decoding can begin). Alternatively, a slice may be uni-predictive or bi-predictive (“P” slice or “B” slice, respectively), indicating that uni-predictive and bi-predictive options are additionally available within the slice.

[0126] The video encoder 542 encodes a sequence of pictures according to a picture structure. One picture structure is "low latency," in which case a picture using interpretation can only reference pictures that have occurred previously in the sequence. Low latency means that each picture is saved for the possibility of being referenced by subsequent pictures, and is output as soon as the picture is decoded. Another picture structure is "random access," where the encoding order of pictures is different from the display order. In random access, interpretation-predicted pictures can reference other pictures that have been decoded but have not yet been output. Some picture buffering is required so that pictures that will be referenced in the future in the display order are present in the decoded picture buffer, which results in a delay of several frames.

[0127] When using a chroma format other than 4:0:0, in an I-slice, the coding tree of each CTU branches at a level of 64x64 or less, resulting in two independent coding trees: one for luma and one for chroma. Using separate trees allows for different block structures to exist between luma and chroma within the 64x64 luma region of a CTU. For example, a large chroma CB may be co-located with many small luma CBs, or vice versa. In P or B-slices, a single coding tree of a CTU defines a block structure common to both luma and chroma. Blocks generated from a single tree can be either intra-predictive or inter-predictive.

[0128] In addition to dividing a picture into slices, it can also be divided into "tiles." A tile is a sequence of CTUs that cover a rectangular area of ​​a picture. The CTU scan is performed raster-wise within each tile, progressing from tile to tile. A slice is either an integer multiple of a tile, or an integer multiple of a consecutive row of CTUs within a particular tile.

[0129] As shown in Figure 8, for each CTU, the video encoder 542 operates in two stages. In the first stage (called the “search” stage), the block partitioner 810 tests various potential configurations of the coding tree. For each potential configuration of the coding tree, there are associated “candidate” CBs. In the first stage, various candidate CBs are tested and a CB that provides relatively high compression efficiency and relatively low distortion is selected. The testing generally involves Lagrangian optimization, and candidate CBs are evaluated based on a weighted combination of rate (coding cost) and distortion (error relative to the input frame data 712). The “best” candidate CB (i.e., the CB with the lowest evaluated rate / distortion value) is selected for encoding to the subsequent bitstream portion 716. The evaluation of candidate CBs includes the choice of using a single CB for a particular region, or further dividing the region according to various partitioning options and encoding each of the resulting smaller regions with another CB, or further dividing the region. As a result, both the coding tree and the CBs themselves are selected in the search stage.

[0130] The video encoder 542 generates a prediction block (PB) for each CB (e.g., CB812), indicated by arrow 820. PB820 is a prediction of the content of the associated CB712. The subtraction module 822 generates the difference (824, also called the "residual" because it is a difference in the spatial domain) between PB820 and CB812. The difference 824 is the difference in block size between the corresponding samples of PB820 and CB812. The difference 824 is transformed, quantized, and represented as a transformed block (TB), indicated by arrow 836. The TB836 associated with PB820 is typically selected from many candidate CBs based on the evaluated cost and distortion.

[0131] A candidate coded block (CB) is a CB obtained from one of the prediction modes available to the video encoder 542 for the associated PB and the resulting residual. When combined with the predicted PB in the video encoder 542, TB836 reduces the difference between the decoded CB and the original CB812 at the expense of additional signal transmission in the bitstream.

[0132] Each candidate coding block (CB) (i.e., a combination of a prediction block (PB) and a transform block (TB)) has an associated coding cost (or "rate") and a corresponding difference (or "distortion"). The distortion of a CB is typically estimated as a difference in sample values, such as the sum of absolute differences (SAD), the sum of squared differences (SSD), or the Hadamard transform on the difference. The estimate obtained from each candidate PB is determined by a mode selector 886 that uses the difference 824 to determine the prediction mode 887. The prediction mode 887 indicates the decision to use a particular prediction mode (e.g., intra-frame prediction or inter-frame prediction) for the current CB. Estimating the coding cost associated with each candidate prediction mode and its corresponding residual coding can be done at a significantly lower cost than entropy coding of the residuals. Therefore, even in real-time video encoders, multiple candidate modes can be evaluated to determine the optimal mode in terms of rate distortion.

[0133] The determination of the preferred mode from a rate distortion perspective is typically achieved using a variant of Lagrangian optimization. Lagrangian optimization, or a similar optimization process, is used both to select the preferred partitioning method (by the block partitioner 810) for partitioning the CTU into CBs, and to select a prediction mode from several possibilities. By applying the Lagrangian optimization process to the candidate modes in the mode selection module 886, the intra-prediction mode with the lowest cost measurement is selected as the "best" mode. The lowest cost mode includes the selected quadratic transformation index 888, which is also encoded into bitstream 716 by the entropy encoder 838.

[0134] In the second stage of operation of the video encoder 542 (called the "encoding" stage), iterative processing is performed within the video encoder 542 on the determined encoding tree for each CTU. For CTUs using separate trees, for each 64x64 lumare region of the CTU, the lumare encoding tree is encoded first, followed by the chromare encoding tree. Within the lumare encoding tree, only lumare CBs are encoded, and within the chromare encoding tree, only chromare CBs are encoded. For CTUs using a shared tree, a single tree describes the CUs (i.e., lumare CBs and chromare CBs) according to the common block structure of the shared tree.

[0135] The entropy encoder 838 supports bitwise coding of syntactic elements using variable-length and fixed-length codewords, as well as an arithmetic coding mode for syntactic elements. Parts of the bitstream, such as "parameter sets" (e.g., Sequence Parameter Set (SPS), Picture Parameter Set (PPS), and Picture Header (PH)), use a combination of fixed-length and variable-length codewords. A slice (also called a contiguous region) has a slice header that uses variable-length coding, followed by slice data that uses arithmetic coding. The slice header defines parameters specific to the current slice, such as slice-level quantization parameter offsets, and may contain instances of PHs. Slice data contains syntactic elements for each CTU within the slice. The use of variable-length and arithmetic coding requires sequential parsing within each part of the bitstream. Each part is delimited by a start code and forms a "Network Abstraction Layer Unit" or "NAL Unit". Arithmetic coding is supported using a context-adaptive binary arithmetic coding process.

[0136] Arithmetic-encoded syntactic elements consist of a sequence of one or more "bins." A bin, like a bit, has a value of either "0" or "1." However, bins are not encoded as discrete bits in bitstream portion 716. Each bin is associated with a predicted value (or "most likely value," "highest probability value") and its associated probability (called context). If the bin actually to be encoded matches the predicted value, the "highest probability symbol" (MPS) is encoded. Encoding the highest probability symbol is relatively low-cost in terms of the number of bits consumed in bitstream portion 716 (including the cost of less than one discrete bit). If the bin actually to be encoded does not match the predicted value, the "lowest probability symbol" (LPS) is encoded. Encoding the lowest probability symbol is relatively high in terms of the number of bits consumed. Bin coding techniques enable efficient encoding of bins with biased probabilities of "0" and "1." Syntactic elements that take only two values ​​(e.g., flags) require a single bin. Syntactic elements with many values ​​require a series of bins. The technique of converting the values ​​of syntactic elements into a series of bins is called "binarization." When the bin values ​​"0" and "1" are equally (or nearly equally) likely, the use of context can be omitted and an equally probable distribution can be assumed. Bins with context are called "context-coded bins," and bins without context are called "bypass-coded bins." When syntactic elements are binarized into one or more bins, a combination of context-coded and bypass-coded bins may occur. Unlike directly encoding a single bit into a bitstream, bypass-coded bins utilize an arithmetic coding engine, which facilitates the mixing of context-coded and bypass-coded bins in the binarization of syntactic elements.

[0137] In certain binarization schemes, the presence of subsequent bins in a sequence may be determined based on the values ​​of preceding bins in the sequence, resulting in variable-length binarization. Furthermore, each bin is associated with multiple contexts, and a context is selected to use when encoding an instance of a particular bin. The selection of a particular context may depend on preceding bins within the syntactic element, the decoded values ​​of adjacent syntactic elements (i.e., those from adjacent blocks), etc. Each time a context-encoded bin is encoded, the context selected for that bin (if any) is updated in a way that reflects the new bin value. Thus, binary arithmetic coding schemes are said to be adaptive.

[0138] The absence of context for bypass-encoded bins saves memory and reduces complexity, so bypass bins are used when the distribution of values ​​for a particular bin is not biased. CABAC (Context Adaptive Binary Arithmetic Encoder) is an example of an entropy encoder that employs context and adaptation, and many variations of this encoder have been used for video coding.

[0139] The QP controller 890 determines the quantization parameter 892, which is used to set the quantization step size used by the quantizer 834 and dequantizer 840. As the quantization step size increases, the linear transformation coefficients 828 are quantized to smaller values, but in exchange, the fidelity of the inverse transformation coefficients 846 decreases and the bitrate of the bitstream portion 716 decreases.

[0140] The entropy encoder 838 encodes the quantization parameter 892 and the LFNST index 888, if used in the current CB, using a bin that combines context coding and bypass coding. The quantization parameter 892 is encoded at the beginning of each slice, and changes in the quantization parameter 892 within a slice are encoded using a "delta-QP" syntactic element. The delta-QP syntactic element is notified at most once within a region called a "quantization group". The quantization parameter 892 is applied to the residual coefficients of the luma CB. The adjusted quantization parameter is applied to the residual coefficients of the chroma CB at the same position. The adjusted quantization parameter may include a mapping from the luma quantization parameter 892 based on a mapping table and a CU level offset selected from an offset list. The quadratic transformation index 888 is notified if the residuals associated with the transformation block contain significant residual coefficients only at coefficient positions that are transformed into linear coefficients by the application of the quadratic transformation.

[0141] The residual coefficients of each TB associated with the CB are encoded using residual syntax. Residual syntax is designed to efficiently encode low absolute coefficients by using arithmetic-encoded bins to indicate the importance of coefficients, using low absolute values, and reserving bypass bins for high absolute residual coefficients. Thus, residual blocks consisting of very low absolute values ​​and a sparse arrangement of important coefficients are efficiently compressed. Furthermore, two residual coding schemes exist. The standard residual coding scheme is optimized for TBs where significant coefficients are mainly located in the upper-left corner of the TB, as observed when a transformation is applied. The transformation-skip residual coding scheme is available for TBs where no transformation is performed and can efficiently encode residual coefficients regardless of their distribution across the entire TB.

[0142] The multiplexer module 884 outputs the PB820 output from the intra-frame prediction module 864 based on the optimal intra-prediction mode selected from the tested prediction modes of each candidate CB. The candidate prediction modes do not need to include all prediction modes supported by the video encoder 542. Intra-predictions are classified into three types: the first, "DC intra-prediction," fills the PB with a single value representing the mean of neighboring reconstructed samples; the second, "planar intra-prediction," fills the PB by arranging samples along a plane using the DC offset and vertical and horizontal gradients derived from neighboring reconstructed samples. Neighboring reconstructed samples typically include a row of reconstructed samples located above the current PB (extending to a certain range to the right of the PB) and a column of reconstructed samples located to the left of the current PB (extending to a certain range to the bottom of the PB). The third, "angled intra-prediction," is a method of filling the PB with neighboring reconstructed samples filtered and propagated along a specific direction (or "angle"). VVC supports 65 angles, and since rectangular blocks can utilize additional angles not available to square blocks, a total of 87 angles can be generated.

[0143] A fourth intra-prediction is available for chroma PB, which follows a "cross-component linear model" (CCLM) mode and is generated from co-locating luma reconstructed samples. Three different CCLM modes are available, each using a different model derived from adjacent luma and chroma samples. The derived models are used to generate sample blocks for chroma PB from collated luma samples. Luma blocks may be intra-predicted by matrix multiplication of a reference sample using one matrix selected from a predefined set of matrices. This matrix intra-prediction (MIP) achieves gain by using matrices trained on large video datasets, where these matrices represent relationships between the reference sample and the predicted block that are not readily captured by angle-based, plane-based, or DC intra-prediction modes.

[0144] Module 864 can also generate prediction units by copying blocks from the vicinity of the current frame using the "Intra-Block Copy" (IBC) method. The location of the reference block is limited to an area corresponding to one CTU, which is divided into 64x64 areas called VPDUs. The area to be processed covers the processed VPDU of the current CTU and the VPDUs of previous CTUs within each row or CTU and each slice or tile, and this area is limited to an area upper limit corresponding to 128x128 luma samples, regardless of the bitstream's CTU size setting. This area is called the "IBC virtual buffer" and limits the required memory capacity by restricting the IBC reference area. The IBC buffer stores the reconstructed sample 854 (before loop filtering), so a separate buffer is required from the frame buffer 872. If the CTU size is 128x128, the virtual buffer contains only samples from the CTU adjacent to the left of the current CTU. If the CTU size is 32x32 or 64x64, the virtual buffer contains up to 4 or 16 CTUs to the left of the current CTU. Regardless of CTU size, access to adjacent CTUs in sampling for IBC reference blocks is constrained by picture, slice, and tile boundaries. In particular, for feature maps in smaller-dimensional FPN layers, using CTU sizes such as 32x32 or 64x64 provides a more consistent reference region covering a set of preceding feature maps. If the feature map arrangement is ordered based on SAD, SSE, or other difference metrics, accessing similar feature maps for IBC prediction offers coding efficiency advantages.

[0145] The residuals of prediction blocks when encoding feature map data differ from those seen in natural video. Natural video is typically screen content captured by image sensors or commonly seen in OS user interfaces. Feature map residuals tend to contain a lot of detailed information. The level of detail in feature map residuals is more suitable for transformation-skip coding than the low-frequency coefficients in various transformations. Intra-predictive luma coding blocks can be divided into groups of prediction blocks of equal size vertically or horizontally, with each block having an area of ​​at least 16 luma samples.

[0146] If pre-reconstructed adjacent samples are unavailable, such as at the edges of a frame, half of the sample range is used as the default halftone value. For example, in the case of 10-bit video, a value of 512 is used. Since there are no preceding samples at the CB in the upper left position of the frame, the angle intra-prediction mode and the planar intra-prediction mode produce the same output as the orthogonal component prediction mode (i.e., a flat sample plane with the halftone value as the absolute value).

[0147] In interframe prediction, the motion compensation module 880 generates a prediction block 882 using samples from one or two frames prior to the current frame in the encoded order frames of the bitstream, and outputs it as PB820 by the multiplexing module 884. Furthermore, in interframe prediction, a single encoded tree is typically used for both the luminous and chroma channels. The encoding order of the frames in the bitstream may differ from the order of the frames at the time of acquisition or display. When one frame is used for prediction, the block is called a "uni-prediction" and has one associated motion vector. When two frames are used for prediction, the block is called a "bi-prediction" and has two associated motion vectors. In P-slice, each CU can be either an intra-prediction or a uni-prediction. In B-slice, each CU can be an intra-prediction, a uni-prediction, or a bi-prediction.

[0148] Frames are typically encoded using a "group of pictures" structure, enabling a temporal hierarchy of frames. A frame is divided into multiple slices, each encoding a portion of the frame. This temporal hierarchy allows frames to reference preceding and succeeding pictures in display order. Images are encoded in the required order to satisfy the dependencies necessary for decoding each frame. In affine mutual prediction mode, instead of using one or two motion vectors to select and filter the reference sample block of the prediction unit, the prediction unit is divided into multiple smaller blocks, generating a motion field where each block has its own motion vector. This motion field uses the motion vectors of points adjacent to the prediction unit as "control points." Affine prediction can encode motion other than translation, reducing the need for deeply divided encoding trees. The bi-prediction mode available in VVC geometrically blends two reference blocks along a selected axis, signaling the angle and offset from the block centers as signals. This Geometric Division Mode (GPM) allows for the use of larger coding units along the boundary between two objects, with the boundary geometry encoded into coding units as angles and center offsets. The difference in motion vectors can be encoded as direction (up / down / left / right) and distance instead of orthogonal coordinate (x,y) offsets, with powers of 2 distances supported. The motion vector predictor is derived from adjacent blocks ("merge mode") as if no offset were applied. The current block shares the same motion vector as the selected adjacent block.

[0149] Samples are selected based on motion vector 878 and reference picture index. Since motion vector 878 and reference picture index are applied to all color channels, interpretation is primarily described as an operation on the PU rather than the PB. The method of decomposing each CTU into one or more interpretation blocks is described in a single coding tree. Interpretation methods may differ in the number and precision of motion parameters. Motion parameters typically consist of a reference frame index indicating the reference frame to be used from the reference frame list, and spatial transformations for each reference frame, but may also include more frames, special frames, or complex affine parameters such as scaling and rotation. Furthermore, pre-determined motion refinement processes may be applied to generate high-density motion estimates based on the referenced sample blocks.

[0150] PB820 is determined and selected, and after subtracting PB820 from the original sample block in subtractor 822, the residual (represented by 824) with the minimum coding cost is obtained, and lossy compression is applied to this residual. The lossy compression process consists of the steps of transformation, quantization, and entropy coding. The forward linear transformation module 826 applies a forward transformation to the difference 824, thereby transforming the difference 824 from the spatial domain to the frequency domain and generating linear transformation coefficients represented by arrows 828. The maximum one-dimensional linear transformation size is either a 32-point DCT-2 or a 64-point DCT-2 transformation, which is set by "sps_max_luma_transform_size_64_flag" in the sequence parameter set. If the CB to be encoded exceeds the maximum supported linear transformation size expressed as the block size (e.g., 64×64 or 32×32), the linear transformation 826 is applied in a tiled manner, transforming all samples of the difference 824. When using non-square CBs, tiling is also performed using the maximum transformation size available in each dimension of the CB. For example, if the maximum transformation size is 32, a 64x16 CB will have two 32x16 linear transformations tiled on it. If the size of the CB exceeds the maximum supported transformation size, the CB will be tiled with TBs. For example, a 128x128 CB with a maximum transformation size of 64pt will be filled with four 64x64 TBs in a 2x2 array. A 64x128 CB with a maximum transformation size of 32pt will be filled with eight 32x32 TBs in a 2x4 array.

[0151] The application of transformation 826 generates multiple TBs for a CB. For each transformation application, if a TB with a difference of 824 greater than 32x32 (e.g., 64x64) is processed, all resulting linear transformation coefficients 828 outside the upper left 32x32 region of the TB are set to zero (i.e., discarded). The remaining linear transformation coefficients 828 are passed to the quantization module 834. The linear transformation coefficients 828 are quantized according to the quantization parameter 892 associated with the CB to generate linear transformation coefficients 832. In addition to the quantization parameter 892, the quantization module 834 may apply a "scaling list" to further scale the residual coefficients within the TB according to their spatial position, thereby enabling non-uniform quantization within the TB. The quantization parameter 892 may differ between the luma CB and each chroma CB. The linear transformation coefficients 832 are passed to the forward quadratic transformation module 830, which performs an unseparated quadratic transformation (NSST) operation or bypasses the quadratic transformation to generate the transformation coefficients represented by arrow 836. The forward linear transform module 826 is typically separable and sequentially transforms the row and column sets of each TB. The forward linear transform module 826 either uses the Type II discrete cosine transform (DCT-2) horizontally and vertically, bypasses the horizontal and vertical transformations, or, for Luma TBs where neither width nor height exceeds 16 samples, uses a combination of the Type VII discrete sine transform (DST-7) and Type VIII discrete cosine transform (DCT-8) horizontally or vertically. The use of the DST-7 and DCT-8 combination is referred to as a "multi-transformation selection set" (MTS) in the VVC standard.

[0152] The forward quadratic transform of module 830 is generally an inseparable transform and is applied only to the residuals of the intra-predicted CU, although it may still be bypassed. The forward quadratic transform operates on 16 samples (arranged as the upper left 4x4 subblock of the linear transform coefficients 828) or 48 samples (arranged as three 4x4 subblocks within the upper left 8x8 coefficients of the linear transform coefficients 828) to generate a set of quadratic transform coefficients. The set of quadratic transform coefficients may be smaller than the set of linear transform coefficients from which they are derived. The quadratic transform is called a "low-frequency inseparable quadratic transform" (LFNST) because it applies only to sets of adjacent coefficients (including DC coefficients). Such a quadratic transform is obtained through a learning process, and its inseparability and learning origin allow it to leverage additional redundancy of residual signals that cannot be captured by separable transforms such as DCT and DST. Furthermore, when an LFNST is applied, all remaining coefficients in the TB become zero in both the linear and quadratic transform domains.

[0153] The quantization parameter 892 is constant for a given TB, resulting in uniform scaling in the generation of residual coefficients in the linear transformation domain of the TB. The quantization parameter 892 can change periodically by a signaled "delta quantization parameter." The delta quantization parameter (delta QP) is signaled only once for CUs contained within a given domain (called a "quantization group"). If a CU is larger than the quantization group size, the delta QP is signaled only once with one of the TBs of the CU. That is, the delta QP is signaled only once by the entropy encoder 838 for the first quantization group of the CU, and not for subsequent quantization groups of the CU. Non-uniform scaling is also possible by applying a "quantization matrix," in which case the scaling coefficient applied to each residual coefficient is derived from a combination of the quantization parameter 892 and the corresponding entry in the scaling matrix. The size of the scaling matrix may be smaller than the size of the TB, and when applied to a TB, the nearest neighbor method is used to provide scaling values ​​for each residual coefficient from a scaling matrix smaller than the TB size. The residual coefficients 836 are fed to the entropy encoder 838 for encoding in the bitstream portion 716. Typically, the residual coefficients of each TB, where at least one residual coefficient of the TU is significant, are scanned to generate an ordered list of values ​​according to a scan pattern. The scan pattern typically scans the TB as a sequence of 4x4 "subblocks," providing a regular scan operation with a granularity of 4x4 residual coefficient sets, where the arrangement of subblocks depends on the size of the TB. Scanning within each subblock and progress between subblocks typically follows a backward diagonal scan pattern. In addition, the quantization parameter 892 is encoded in the bitstream portion 716 using delta-QP syntactic elements, and the slice QP and quadratic transformation index 888 of the initial values ​​in a particular slice or subpicture are also encoded in the bitstream portion 716.

[0154] As described above, the video encoder 542 requires access to the frame representation corresponding to the decoded frame representation seen by the video decoder. Therefore, the residual coefficient 836 passes through the inverse quadratic transform module 844, which operates according to the quadratic transform index 888, to produce the intermediate inverse transform coefficient indicated by arrow 842. The intermediate inverse transform coefficient 842 is inversely quantized by the inverse quantization module 840 according to the quantization parameter 892, producing the inverse transform coefficient indicated by arrow 846. The inverse quantization module 840 may also perform inverse non-uniform scaling of the residual coefficient using a scaling list corresponding to the forward scaling performed by the quantization module 834. The inverse transform coefficient 846 is passed to the inverse linear transform module 848, which generates the residual samples of the TU (indicated by arrow 850). The inverse linear transform module 848 applies the DCT-2 transform horizontally and vertically, constrained by the maximum available transform size described in the forward linear transform module 826. The type of inverse transformation performed by the inverse quadratic transformation module 844 corresponds to the type of forward transformation performed by the forward quadratic transformation module 830. The type of inverse transformation performed by the inverse linear transformation module 848 corresponds to the type of linear transformation performed by the linear transformation module 826. The additive module 852 adds the residual sample 850 and PB 820 to generate a reconstructed sample of CU (indicated by arrow 854).

[0155] The reconstructed sample 854 is passed to the reference sample cache 856 and the in-loop filter module 868. The reference sample cache 856 is typically implemented using static RAM on the ASIC to avoid costly off-chip memory access, and it provides the minimum sample storage necessary to satisfy the dependency of in-frame PB generation on subsequent CUs within the frame. This minimum dependency typically includes a "line buffer" of samples along the end of a row of CTUs, which is used by the next row of CTUs, and also includes column buffering within a range set by the height of the CTUs. The reference sample cache 856 supplies the reference sample (indicated by arrow 858) to the reference sample filter 860. The sample filter 860 applies a smoothing process to produce filtered reference samples (indicated by arrow 862). The filtered reference samples 862 are used by the in-frame prediction module 864 to produce in-frame prediction blocks (indicated by arrow 866). For each candidate intra-prediction mode, the intra-frame prediction module 864 produces a sample block, i.e., 866. Sample block 866 is generated by module 864 using techniques such as DC prediction, planar intra prediction, and angle intra prediction. Sample block 866 is also generated by a matrix multiplication approach using adjacent reference samples as input, with the video encoder 800 selecting a matrix from a set of matrices. The selected matrix is ​​signaled using an index within bitstream 716 to identify which matrix from the set the video decoder should use.

[0156] The In-Loop Filter Module 868 applies multiple filtering stages to the reconstructed sample 854. These filtering stages include a "Deblocking Filter" (DBF), which applies smoothing aligned with the CU boundary to reduce artifacts caused by discontinuities. Another filtering stage present in the In-Loop Filter Module 768 is the "Adaptive Loop Filter" (ALF), which applies a Wiener-based adaptive filter to further reduce distortion. A further filtering stage available in the In-Loop Filter Module 868 is the "Sample Adaptive Offset" (SAO) filter. The SAO filter works by first classifying the reconstructed sample into one or more categories and then applying an offset at the sample level according to the assigned category.

[0157] The filtered samples are output from the in-loop filter module 868, as indicated by arrow 870. The filtered samples 870 are stored in the frame buffer 872. The frame buffer 872 typically has the capacity to store multiple (e.g., up to 16) pictures and is therefore stored in memory 206. The frame buffer 872 is not typically stored using on-chip memory due to its high memory consumption. Therefore, accessing the frame buffer 872 is costly in terms of memory bandwidth. The frame buffer 872 provides a reference frame (indicated by arrow 874) to the motion estimation module 876 and the motion compensation module 880. The reference frame 874 is output as the reconstructed frame 718 by the encoder module 542. In the example in Figure 8, the reconstructed frame is the result of lossy VVC coding (i.e., operation of modules 810 through 890).

[0158] The motion estimation module 876 estimates multiple “motion vectors” (indicated as 878), each of which is an orthogonal coordinate space offset from the current CB position and references a block of any reference frame in the frame buffer 872. For each motion vector, a filtered block of reference samples (represented as 882) is generated. The filtered reference samples 882 form further candidate modes that are potentially selected by the mode selector 886. Furthermore, for a particular CU, the PU820 may be formed using one reference block ("uni-prediction") or two reference blocks ("bi-prediction"). For the selected motion vectors, the motion compensation module 880 generates the PB820 according to a filtering process that supports sub-pixel precision in the motion vectors. Thus, the motion estimation module 876 (operating on a large number of candidate motion vectors) can perform a simplified filtering process and achieve a reduction in computational complexity compared to the motion compensation module 880 (operating only on selected candidates). If the video encoder 542 selects interpretation for CU, the motion vector 878 is encoded into the bitstream portion 716.

[0159] Although the video encoder 542 in Figure 8 is described with reference to General-Purpose Video Coding (VVC), other video coding standards and implementations may also employ the processing steps of modules 810-890. Frame data 712 (and bitstream 121) may also be read from (or written to) memory 206, hard disk drive 210, CD-ROM, Blu-ray disc (trademark), or other computer-readable storage media. Furthermore, frame data 712 (and bitstream 121) may also be received from (or transmitted to) external sources such as servers or radio frequency receivers connected to the communication network 220. The communication network 220 may have bandwidth limitations, and rate control may be required in the video encoder 120 to avoid network saturation when compressing frame data 712 is difficult.

[0160] The bitstream 121 may consist of one or more slices, each slice representing a spatial section (a set of CTUs) of frame data 712, generated by one or more instances of video encoder 542, each instance generating a bitstream portion 716, which works cooperatively under the control of processor 205. The bitstream portion 716 may contain one slice corresponding to one region, which is output as a set of subpictures forming a picture, each subpicture being independently encodeable and independently decodeable with respect to other slices and subpictures within the picture.

[0161] Figures 9A and 9B show schematic block diagrams of picture 900 divided into regions 910, 912, and 916, suitable for packing data from the PCA-based encoder network topology 700. Each tensor feature map is packed by packer 538. The examples in Figures 9A and 9B show three regions corresponding to the mean channel, basis vectors, and coefficients.

[0162] Referring to Figure 9B, picture 900b corresponds to picture 900, region 910b corresponds to 910, and 912b corresponds to 912. Region 910b holds average channel data, such as the average channel 920 for each tensor in tensor 115. In the example in Figure 9B, the basis vectors include basis vector 922, and the basis vectors are packed non-overlappingly within the region 912b. Basis vector 922 forms a C×C' array of integer quantized values, where C is the number of channels in the tensor 115 and C' is the number of basis vectors obtained from the decomposition by decomposition module 718. When multiple tensors are decomposed into sets of basis vectors, for example, when tensor 115 is composed of multiple tensors, basis vector 922 contains each set of basis vectors, which are arranged in a non-overlapping, left-aligned, vertically connected manner.

[0163] In picture 900 (or 900b), areas not used to store data may be occupied by sample values ​​corresponding to the value "0" after inverse quantization of the sample values ​​and conversion to the floating-point region. Similarly, in areas 910 (or 910b), 912 (or 912b), and 916, areas not used to store any data may be occupied by sample values ​​corresponding to the value "0" after inverse quantization of the sample values ​​and conversion to the floating-point region. Area 916 holds the coefficients corresponding to 724 in Figure 7, where the coefficients for each basis vector form a width × height feature map. The coefficient area 916 stores coefficients packed as, for example, 24 feature maps (applicable to basis vectors [0…23]), as shown in feature map 930.

[0164] Each region (910, 912, 916) is typically aligned to a 128x128 CTU grid. When specifying the position of each region in orthogonal coordinates in CTU size units, it is identified by the orthogonal CTU addresses of the upper-left and lower-right corners. For example, region 916 occupies the area from CTU (0,1), corresponding to the upper-left luma sample at position (0,128), to CTU (6,3), corresponding to the lower-right luma sample located at (6x128+127,3x128+127) or (895,511).

[0165] Figures 10A and 10B show schematic block diagrams of picture 1000 divided into a single region 1010, suitable for packing the compressed tensor 532 generated by the tensor compressor 600. Each feature map of the compressed tensor (e.g., feature map 1030) is packed into region 1010, with the feature maps being packed sequentially from left to right, moving to the next row when the available space in the current row is exhausted. The tensor compressor 600 reduces the area required to pack the compressed tensor 532 to less than the area required to pack each of the tensors 115 into picture 1000. Region 1010 is defined based on the top-left and bottom-right Cartesian coordinate system CTU addresses.

[0166] Figure 11 is a schematic block diagram showing bitstreams 121 or 143 that hold encoded packed feature maps and associated metadata. Bitstreams 121 or 143 each contain a group of syntactic elements prefixed with a "Network Abstraction Layer" (NAL) unit header. For example, the NAL unit header 1108 is placed before the Sequence Parameter Set (SPS) 1110. The SPS 1110 specifies the layout of the pictures (e.g., picture 900 or picture 1000) depending on the compressor and decompressor selected in step 1810. The picture layout includes the position and size of subpictures (e.g., 910, 912, 916) using subpicture information 1111. The SPS 1110 must be decoded before decoding a series of frames associated in the prediction structure (i.e., interprediction reference pictures) that generally have the same width, height, bit depth, and chroma format. In applications involving streaming, retransmission of the SPS 1110 may be performed to allow decoding to begin at the midpoint of the bitstream. Furthermore, if the decoder network topology changes during execution, the frame area required for the compressed tensor may change, necessitating the insertion of another SPS to indicate the updated frame dimensions. SPS 1110 also indicates the chroma format, bit depth, and resolution of the frame data represented by bitstream 121. Encoded subpicture 1122, which encodes subpicture 912, includes slice header 1130 followed by slice data 1140. Slice data 1140 includes a set of CTUs that provide an encoded representation of the frame data. Encoded subpicture 1124 encodes subpicture 916 corresponding to the coefficients. Bitstream 121 includes SEI messages 1113 written by metadata encoder 544, which have syntax compliant with or equivalent to the syntax described in Appendix A. SEI messages 1113 encode the metadata necessary to convert the decoded frame into a set of tensors usable by the CNN head 150.

[0167] Figure 12 is a schematic block diagram showing a tensor decoder 146 with a configurable tensor decompressor 1250. Figure 19 shows a method 1900 that decodes a bitstream, reconstructs the tensor according to a given tensor decoder, and performs the second part of a CNN. As previously stated, method 1900 is configured to determine the required decoder network complexity for a given decoder network topology and to determine a decoder network complexity representation for a bitstream. This representation is used to reference a given decoder network topology, and the decoder network topology is decoded from the bitstream. The bitstream is subsequently decoded using the determined decoder network topology to produce a decoded tensor based on the required network complexity and decoder network complexity representation. The decoder network complexity includes at least one of the following aspects: multiplication and integration operations, intermediate tensor values, memory access operations on the tensor, and weight number.

[0168] The tensor decoder 146 and method 1900 may be implemented as one or more software application programs 233 executable within a computer system 200. The tensor decoder 146 and method 1900 may be implemented by instructions 231 (see Figure 2B) within software 233 executed within the computer system 200. The software instructions 231 are formed as one or more code modules, each performing one or more specific tasks. Method 1900 begins with step 1910 determining a tensor decompressor.

[0169] In step 1910, the tensor decoder 146 determines the decoder network topology to use to restore the dimensions of the compressed tensor to compatible dimensions that can be used as input to the CNN head 150. The metadata parser 1208, under the execution of processor 205, decodes the decoder topology 1192 from the SEI message 1113 as the complete decoder network topology, or decodes a reference to a description of the decoder network topology, which may be pre-included in the bitstream 143 or obtained by external means (e.g., downloaded from an internet connection). The decoder network topology may be encoded in a format such as ONNX, NNEX, or Pytorch code. The decoded network topology may include a format indication showing the format in use. A format version indication may also be included, as multiple versions may be defined (or new versions may be created in the future) for a particular format. Since the format is essentially a text format, the use of an optional compression stage using techniques such as ZIP or LZMA may be indicated to minimize the storage overhead of the decoder network topology in the bitstream 143.

[0170] As shown in Figure 12, the tensor decompressor information 1230 specifies the decoder network topology by reference or structure. If the tensor decompressor information 1230 specifies the decoder network topology by reference, the structure of the decoder network topology is obtained from the tensor decompressor repository 1232 (if available), or the repository 1232 obtains the topology from the tensor codec repository 180. The decoder network topology may be retained in the repository 1232 for future use even after a different decoder network topology has been used. The decoder network topology may be retained in the repository 1232 for future use even after a subsequent SPS (e.g., 1110) indicating the start of a new bitstream has been received. If the decoder network topology requires weights, weight information 1234 is provided to the tensor weight repository, specifying the weights by reference or value. Control of the processor 205 proceeds from step 1910 to the decoding complexity display step 1920.

[0171] In step 1920, under the execution of processor 205, the metadata parser 1208 decodes one or more syntactic elements representing the worst-case complexity of the decoder network topology implemented in the tensor decompressor 1250 as a complexity representation 1191 from the SEI message 1113. The worst-case complexity includes the worst-case scenario for intermediate tensor storage within the tensor decompressor 1250, the worst-case number of MAC operations performed by the tensor decompressor 1250, and the worst-case scenario for any type of floating-point operation performed by the tensor decompressor 1250. The processor 205 then proceeds from step 1920 to step 1930, which determines the complexity of the tensor decompressor.

[0172] In step 1930, the tensor decoder 146, under the execution of processor 205, determines the complexity required to execute the determined decoder network topology. An operation count is generated by scanning the stages defined in the determined decoder network topology and counting the operations implied by each stage without executing each stage. The dimensions of the persistent tensor data (i.e., the tensor held from one call to the tensor decompressor 1250 to the next) are recorded. The amount of intermediate tensor data is recorded, and the maximum amount of intermediate tensor data to be held simultaneously in memory 206 is determined. Control of processor 205 proceeds from step 1930 to the decode packing format step 1940.

[0173] In step 1940, the metadata parser 1208, under the execution of processor 205, decodes packing information 1194 and tensor information 1195 from SEI message 1113 in bitstream 143 and determines the packing format. The packing format indicates that the frame is divided into one or more regions (e.g., rectangular regions), each used to hold feature maps corresponding to one or more tensors. If one region is used, typically the entire video frame is allocated to that region. The packing format describes the regions at an integer CTU granularity, and defines the boundaries of the regions in bitstream 143 by the orthogonal coordinates of the top-left and bottom-right CTUs, encoded in units of CTU or 128 luma samples. The regions are decoded as an ordered list, addressable by index, and allow one or more tensors to be mapped to each region. The area of ​​each region in the packing format should be sufficient to accommodate the maximum number of feature maps that the encoder network topology can generate for the tensors contained within the corresponding region. If an updated decoder network topology is notified, a new packing format must be notified to specify how to decompress the compressed tensor resulting from the new decoder network topology. Control of processor 205 proceeds from step 1940 to complexity test step 1950.

[0174] In step 1950, the tensor decoder 146 under the execution of processor 205 compares the complexity of the decoder network topology determined in step 1930 with the complexity value decoded in step 1920. If the determined complexity is less than or equal to the complexity value (OK in step 1950), control of processor 205 proceeds from step 1950 to tensor decompressor instantiation step 1970; otherwise (NOT OK in step 1950), control of processor 205 proceeds from step 1950 to error state step 1960.

[0175] In step 1960, the tensor decoder 146 enters an error state, and decoding cannot continue because the signaled decoder network topology may exceed the capabilities of the tensor decompressor 1250. Method 1900 terminates.

[0176] In step 1970, the tensor decompressor 1350 is initialized according to the decoder network topology determined in step 1910. Step 1970 is executed only if a new decoder network topology is determined in step 1910. That is, in subsequent calls to method 1900 where a new decoder network topology is not determined, the resources allocated in step 1970 can be reused. Sufficient memory is allocated to hold persistent tensors (i.e., tensors that are retained from one call to the next of method 1900), as well as memory to hold the largest intermediate tensors used simultaneously in the execution of the decoder network topology. If hardware acceleration is applied to the decoder network topology, sufficient execution units such as MACs and DSP blocks in the FPGA may also be reserved. When the decoder network topology is executed in software, sufficient execution time is ensured on available resources such as the CPU and GPU, which enables real-time operation of Method 1900 (i.e., sufficient time for repeated calls to Method 1900 when decoding incoming packetized frames without accumulating stalls, jitter, or buffering delays). Control of the processor 205 proceeds from step 1970 to step 1980, which is the decoding of the packetized frames.

[0177] In step 1980, the picture decoder 1204, under the control of processor 205, decodes one packetized frame from bitstream 143 and generates a decoded frame 1210. The operation of picture decoder 1204 is illustrated with reference to Figure 13. Control of processor 205 proceeds from step 1980 to tensor unpacking step 1990.

[0178] In step 1990, the unpacker 1214, under the execution of processor 205, reads feature maps from the decoded frame 1210 according to the packing format determined in step 1940 and described with reference to Figures 9A and 9B, and Figures 10A and 10B. For each tensor, a number of feature maps corresponding to the number of channels used, as indicated in tensor information 1195, are decoded. The channels of each tensor are unpacked as two-dimensional feature maps. The number of feature maps or channels to be decoded for a particular tensor is decoded from bitstream 143 as "number of channels". Multiple tensors, each with a channel, are arranged horizontally or vertically within a slice or subpicture in the frame. Each pre-determined type of feature map is unpacked in step 1990 by the unpacker 1214 using the feature map assignment determined for the frame (e.g., 1210) in method 1800. This assignment may be decoded from an SEI message. The unpacker 1214 outputs an integer tensor 1216, which is decoded using the decoder topology of the tensor decoder 146. Control of the processor 205 proceeds from step 1990 to the inverse quantized tensor processing step 19100.

[0179] In step 19100, under the control of processor 205, the inverse quantizer 1218 performs inverse quantization on integer tensor 1216, thereby applying the quantization range decoded from bitstream 143 to the determined number of channels of each tensor to generate the inverse quantized tensor 1220. To perform inverse quantization, metadata parser 1208 decodes the quantization range 1196 from SEI message 1113. Control of processor 205 proceeds from step 19100 to quantized tensor buffer step 19110.

[0180] In step 19110, the tensor storage module 1222 under the execution of processor 205 provides inter-frame storage of the inverse quantized tensor 1220. Of the tensors for each region in packing form, each tensor from which at least one channel or feature map has been decoded is stored in the tensor storage module 1222. The tensor storage module 1222 generates an output tensor 1224, which includes the most recent tensor for each tensor from which at least one feature map has been decoded. In other words, if a tensor is not decoded in the current frame (i.e., the number of channels is determined to be zero), the most recent value of a tensor with a non-zero number of channels that has been decoded is used. For example, when using a PCA-based method such as the encoder network topology 700, the average channels obtained from the average channel module 710 may not be updated frequently, in which case the tensor storage module 1222 holds the most recent result from the average channel module 710 for use by the tensor decompressor 1400. The control of processor 205 proceeds from step 19110 to tensor decompression step 19120.

[0181] In step 19120, the tensor decompressor 1250, under the execution of processor 205, uses tensor 1224 as input to perform the steps specified by the decoder network topology and generate the decoded tensor 149. The check performed in step 1950 ensures that the execution of the decoder network topology does not consume resources already allocated by destination device 140 for tensor decompression, and therefore successfully generates an output, using tensor 1224 as input to generate the decoded tensor 149. Control of processor 205 then proceeds from step 19120 to neural network second part execution step 19130.

[0182] In step 19130, the CNN head 150, under the execution of processor 205, executes the remaining layers of the neural network implemented by system 100, with tensor 149 as input. Method 1900 ends, and when processor 205 receives the next packetized frame in bitstream 143, method 1900 can be called again.

[0183] Figure 13 is a schematic block diagram showing the functional modules of the video decoder 1204. The video decoder 1204 may be implemented as one or more software application programs 233 that can be executed within the computer system 200. The video decoder 1204 is realized by instructions 231 (see Figure 2B) within the software 233 that runs within the computer system 200. The software instructions 231 are formed as one or more code modules, each module performing one or more specific tasks.

[0184] Bitstream 143 is input to the entropy decoder module 1320. The entropy decoder module 1320 extracts syntactic elements by decoding a sequence of "bins" from bitstream 143 and passes the values ​​of these syntactic elements to other modules in the video decoder 1204. The entropy decoder module 1320 uses variable-length and fixed-length decoding and an arithmetic decoding engine to decode the SPS, PPS, or slice header, thereby decoding the syntactic elements of the slice data as a sequence of one or more bins. Each bin can use one or more "contexts," which describe the probability levels used to encode the "1" and "0" values ​​of the bin. If multiple contexts are available for a particular bin, a "context modeling" or "context selection" step is performed to select the context to use for decoding the bin from among the available contexts. The bin decoding process forms a sequential feedback loop, and each slice is fully decoded by a specific instance of the entropy decoder 1020.

[0185] The entropy decoder module 1320 decodes syntactic elements from the bitstream 143 by applying an arithmetic coding algorithm, such as "Context-Adaptive Binary Arithmetic Coding" (CABAC). The decoded syntactic elements are used to reconstruct parameters in the video decoder 1204. The parameters include mode selection information such as residual coefficients (represented by arrow 1324), quantization parameters 1374, quadratic transformation index 1370, and intra-prediction mode (represented by arrow 1358). The mode selection information also includes motion vectors and information for splitting each CTU into one or more CBs. The parameters are typically used in combination with sample data from previously decoded CBs to generate PBs.

[0186] The residual coefficients 1324 are passed to the inverse quadratic transform module 1336, where the quadratic transform is applied according to the quadratic transform index, or no operation is performed (bypassed). The inverse quadratic transform module 1336 generates the reconstructed transform coefficients 1332. That is, module 1336 generates the linear transform region coefficients from the quadratic transform region coefficients. The reconstructed transform coefficients 1332 are input to the inverse quantization module 1328. The inverse quantization module 1328 performs inverse quantization (or "scaling") on the residual coefficients 1332, i.e., in the linear transform coefficient region, generates the reconstructed intermediate transform coefficients represented by arrow 1340 according to the quantization parameter 1374. The inverse quantization module 1328 may also apply a scaling matrix to provide non-uniform inverse quantization within the TB, which corresponds to the operation of the inverse quantization module 840. If the use of a non-uniform inverse quantization matrix is ​​indicated in the bitstream 143, the video decoder 1204 reads the quantization matrix from the bitstream 143 as a column of scaling coefficients and arranges them in a matrix. Inverse scaling uses the quantization matrix in combination with the quantization parameters to create a reconstructed intermediate transformation coefficient of 1340.

[0187] The reconstructed conversion coefficients 1340 are passed to the inverse linear transformation module 1344. Module 1344 performs the inverse transformation of coefficients 1340 from the frequency domain to the spatial domain. The inverse linear transformation module 1344 applies the inverse DCT-2 transformation in the horizontal and vertical directions, constrained by the maximum available transformation size described in the forward linear transformation module 826. The processing result of module 1344 is a block of residual samples, indicated by arrow 1348. The block of residual samples 1348 is the same size as the corresponding CB. The residual samples 1348 are fed to the summing module 1350.

[0188] In the addition module 1350, the residual sample 1348 is added to the decoded PB (represented by 1352) to generate a reconstructed sample block indicated by the arrow 1356. The reconstructed sample 1356 is supplied to the reconstructed sample cache 1360 and the in-loop filtering module 1388. The in-loop filtering module 1388 generates a reconstructed frame sample block 1392. The frame sample 1392 is written to the frame buffer 1396. The frame buffer 1396 outputs an image or video frame 1210.

[0189] The reconstructed sample cache 1360 operates similarly to the reconstructed sample cache 856 of the video encoder 542. The reconstructed sample cache 1360 provides storage for the reconstructed samples necessary for intra-predicting subsequent CBs without using memory 206 (e.g., using data 232 instead, which is typically on-chip memory). Reference samples are retrieved from the reconstructed sample cache 1360, as indicated by arrow 1364, and fed to the reference sample filter 1368, which generates filtered reference samples, as indicated by arrow 1372. The filtered reference samples 1372 are fed to the intra-frame prediction module 1376. Module 1376 generates blocks of intra-prediction samples (indicated by arrow 1380) according to the intra-prediction mode parameter 1358, which is signaled in bitstream 143 and decoded by the entropy decoder 1320. The intra-prediction module 1376 supports the modes of the encoder-side module 864, including IBC and MIP. Sample block 1380 is generated using modes such as DC, planar, or angle-in-angle prediction.

[0190] If the prediction mode of the CB in bitstream 143 is indicated to use intraprediction, the intrapredicted sample 1380 forms the decoded PB 1352 via the multiplexer module 1384. Intraprediction generates a predicted block (PB) of a sample (which is a block within one color component) derived using "adjacent samples" within the same color component. An adjacent sample is a sample that is adjacent to the current block and has already been reconstructed because it precedes it in the block decoding order. When a luma block and a chroma block are in the same position, the luma block and the chroma block may use different intraprediction modes. However, two chroma CBs share the same intraprediction mode.

[0191] If the prediction mode of the CB in bitstream 143 is indicated as inter-prediction, the motion compensation module 1334 generates a block of inter-prediction samples, which is represented as 1338. The inter-prediction sample block 1338 is generated using the motion vector decoded from bitstream 143 by the entropy decoder 1320 and a reference frame index for selecting and filtering the sample block 1398 from the frame buffer 1396. The sample block 1398 is obtained from previously decoded frames stored in the frame buffer 1396. In the case of bi-prediction, two sample blocks are generated and blended to produce samples for the decoded PB 1352. The frame buffer 1396 stores the filtered block data 1392 from the in-loop filtering module 1388. Similar to the in-loop filtering module 868 of the video encoder 542, the in-loop filtering module 1388 applies one of the DBF, ALF, or SAO filtering operations. Generally, motion vectors are applied to both lunar and chroma channels, but the filtering processes for subsample interpolation in lunar and chroma channels differ.

[0192] Figure 14 is a schematic block diagram showing an implementation of a channel-to-channel decorrelation-based tensor decoder 1400 that can function as a tensor decompressor 1250. When decoding multiple tensors, multiple instances of the tensor decoder 1400 may be instantiated as tensor decompressors 1250. Tensor 1224 is received from tensor storage 1222A and includes a coefficient tensor 1414, a basis vector tensor 1412, and a channel mean tensor 1410. Under the operation of processor 205, the inner product module 1420 performs the inner product of coefficients 1414 and basis vectors 1412 to produce a zero-centered tensor 1422. The sum module 1424 adds the zero-centered tensor 1422 and the mean channel 1410 to produce a reconstructed tensor 149 as the output from tensor decoder 1400. If an input type such as a mean does not need updating, the number of channels for that input type can be set to zero. If the number of input channels is set to zero, the most recent received value for that input may be used instead.

[0193] Figure 15 is a schematic block diagram showing a tensor decompressor 1500 that uses a multiscale feature reconstruction stage, which may be selected to be used as a tensor decompressor 1250 in step 1810. Tensor decompressor 1500 includes a single-scale feature compression (SSFC) decompressor 1510. The SSFC decompressor 1510 receives a tensor 149 with a reduced number of channels (e.g., 64 channels) and passes it to a convolutional layer 1512, which outputs a tensor 1513 with a restored number of channels (e.g., 256 channels). Tensor 1513 is passed to a batch normalization module 1514 to produce tensor 1515. Tensor 1515 is passed to a PreLU module 1516 to produce tensor 1520. Tensor decompressor 1500 includes an MSFR module 1530. The MSFR module generates multiple tensors from tensor 1520, which is produced by performing step 19100 as described with reference to Figure 19, using one or more trained convolutional layers. The upsampling modules 1532, 1534, and 1536 upsample tensor 1520 by a factor of 2x, 4x, and 8x in the horizontal and vertical directions, respectively, to generate tensors 1533, 1535, and 1537. Tensor 1537 forms one of the outputs (P"2, 1557) from MSFR module 1530 and is passed to downsampling module 1542. Downsampling module 1542 downsamples tensor 1537 by a factor of 2 horizontally and vertically to produce tensor 1543, which has the same dimensions as tensor 1535. Tensor 1543 is fed to convolutional layer 1548, which outputs tensor 1549. Totalization module 1554 adds tensors 1535 and 1549 to produce tensor 1555 as the output (P"3) of MSFR module 1530.

[0194] Figure 16A is a schematic block diagram showing an example 1600 of the head portion 150 of a CNN for object detection, which is part of the "YOLOv3" network and corresponds to the portion excluding the "DarkNet-53" backbone. The CNN head portion 150 in Figure 16A can be used when a CNN backbone is implemented, for example, as in Figure 3A. Depending on the task performed on the destination device 140, a different network may be used instead of the CNN head portion 150. The input tensor 149 is separated into tensors for each layer (i.e., tensors 1610, 1620, and 1634). Tensor 1610 is passed to the CBL module 1612 to generate tensor 1614. Tensor 1614 is passed to the detection module 1616 and the upscaler module 1622. The detection module outputs a bounding box 1618 in the form of a detection tensor. The bounding box 1618 is passed to the non-maximal suppression (NMS) module 1648.

[0195] Before resizing for the core portion of network 114, the upscaler module 1622 scales the original video by its width and height to generate bounding boxes corresponding to the coordinates of the original video data 113. The upscaler module 1622 receives tensors 1614 and 1620, generates an upscaled tensor 1624, which is passed to the CBL module 1626. The CBL module 1626 generates tensor 1628 as output. Tensor 1628 is passed to the detection module 1630 and the upscaler module 1636. The detection module 1630 generates a detection tensor 1632, which is supplied to the NMS module 1648. The upscaler module 1636 is another instance of module 1622. The upscaler module 1636 receives tensors 1628 and 1634, and outputs an upscaled tensor 1638. The upscaled tensor 1638 is passed to the CBL module 1640, which outputs tensor 1642 to the detection module 1644. The detection module 1644 generates a detection tensor 1646, which is supplied to the NMS module 1648.

[0196] CBL modules 1612, 1626, and 1640 each contain a chain of five CBL modules (e.g., CBL model 360 shown in Figure 3D). Upscaler modules 1622 and 1636 are instances of upscaler module 1660 shown in Figure 16B. Module 1648 receives tensors 1618, 1632, and 1646 and outputs task result 151.

[0197] As shown in Figure 16B, the upscaler module 1660 accepts tensor 1662 (e.g., tensor 1614 in Figure 16A) as input. Tensor 1662 is passed to the CBL module 1666 (which has the structure of module 360) to generate tensor 1668. Tensor 1668 is passed to the upsampling module 1670 to generate the upsampled tensor 1672. The concatenation module 1674 concatenates the upsampled tensor 1672 with the second input tensor 1664 (e.g., input tensor 1620 to upscaler 1622 in Figure 16A) to generate tensor 1676.

[0198] Detection modules 1616, 1630, and 1644 are instances of detection module 1680 shown in Figure 16C. Detection module 1680 receives tensor 1682. Tensor 1682 is input to CBL module 1684, which has the structure of module 360. CBL module 1684 generates tensor 1686. Tensor 1686 is passed to convolution module 1688, which implements the detection kernel. In some configurations, the detection kernel applies a 1x1 kernel, generating an output on the feature map in each of the three layers of the tensor. The detection kernel is 1x1x(Bx(5+C)), where B is the number of predictable bounding boxes for a given cell (usually 3) and C is the number of classes (e.g., 80), so the kernel size is 255 detection attributes (i.e., tensor 1290). The constant "5" represents four bounding box attributes (x, y coordinates of the box center and x, y size scale) and one object confidence level ("objectness"). The detection kernel results have the same spatial dimensions as the input feature map, but the output depth corresponds to the detection attributes. The detection kernel is applied at each layer (usually three layers), generating a large number of candidate bounding boxes. The NMS module 1648 applies non-maximal suppression to the resulting bounding boxes, discarding redundant boxes such as duplicate predictions at similar scales, and generating a final set of bounding boxes as the output for object detection.

[0199] Figure 17 is a schematic block diagram showing the head portion 1700 of the CNN. The head portion 1700 can be implemented as the CNN head portion 150, for example, if the CNN backbone 114 is implemented as the backbone 400. The head portion 1700 constitutes part of the overall network known as "FasterRCNN" and includes the feature network (i.e., the backbone portion 400), the region proposal network, and the detection network. The input to the head portion 1700 is tensor 149, which includes the P2-P6 layer tensors 1710, 1712, 1714, 1716, and 1718. The P2-P5 layer tensors 1710, 1712, 1714, and 1716 correspond to the P2-P5 outputs 477, 475, 473, and 471 in Figure 4. The P2-P6 tensors 1710, 1712, 1714, 1716, and 1718 are input to the Region Proposal Network (RPN) head module 1720. The P6 tensor 1718 is generated by the Max Pool module 1742, which performs a 2x2 Max Pool operation on the P5 tensor 1716. The RPN head module 1720 performs convolution on the input tensors to generate an intermediate tensor. This intermediate tensor is fed to two consecutive sibling layers: (i) for classification and (ii) for bounding box (region of interest: ROI) regression. The resulting output is the classification result and the bounding box 1722. The classification result and the bounding box 1722 are passed to the NMS module 1724. The NMS module 1724 prunes redundant bounding boxes by removing duplicate boxes with low scores, generating the pruned bounding box 1726. The bounding box 1726 is input to the region of interest (ROI) puller 1728. The ROI puller 1728 uses a portion of the layer tensor 149 (described later) and the bounding box 1726 to generate a fixed-size feature map from various input size maps using a max pooling operation. In the max pooling operation, the maximum value is taken from each group of input values ​​by subsampling, and a single output value is generated in the output tensor.

[0200] The inputs to the ROI puller 1728 are the P2-P5 feature maps 1710, 1712, 1714, and 1716, and the region of interest proposal 1726. Each proposal (ROI) from 1726 is associated with a portion of the feature maps (1710-1716) to generate a fixed-size map. The size of the fixed-size map is independent of the underlying portion of the feature maps 1710-1716. One of the feature maps 1710-1716 is selected so that the resulting cropped map has sufficient detail, for example, according to the following rule: floor(4+log2(sqrt(box_area) / 224)), where 224 is the standard box size. The ROI puller 1728 crops the input feature maps according to proposal 1726 to generate tensor 1730.

[0201] Tensor 1730 is input to the fully connected (FC) neural network head 1732. The FC head 1732 runs two fully connected layers to generate a class score and a bounding box prediction delta tensor 1734. The class score is typically an 80-element tensor, where each element corresponds to the predicted score for the corresponding object category. The bounding box prediction delta tensor is an 80 × 4 = 320-element tensor containing the bounding boxes for the corresponding object categories. Final processing is performed by the output layer module 1736, which receives tensor 1734 and performs a filtering operation to generate a filtered tensor 1738. Objects with low scores (low classification scores) are excluded from further consideration. The non-maximal suppression module 1740 receives the filtered tensor 1738 and removes duplicate bounding boxes with low classification scores, thereby generating an inference output tensor 1740 corresponding to tensor 151.

[0202] Weight information 1193 may contain "no_weights_flag," which indicates that the decoder network topology used does not require any weights to operate.

[0203] In a configuration of source device 110 and destination device 140, network weights are signaled in bitstream 121 as differences to another set of network weights ("base weights") that system 100 knows or can obtain from an external means such as a tensor codec repository 180. The base weights may be indicated by a reference using an identification number stored in bitstream 121. The method of signaling network weights as differences to other network weights can be implemented using a syntax such as the "MPEG incremental neural network representation" under development as part of ISO / IEC 15938-17.

[0204] The identifiers for the decoder network topology and network weights are shown as unsigned integer values ​​in Appendix A and encoded using unsigned exponential Gorloom codewords, but other representations are possible. For example, the decoder network topology and network weights can be encoded using variable-length UTF-8 strings, Universal Unique Identifier (UUID) strings, or other encoding schemes.

[0205] The metadata written by the metadata encoder 544 and metadata parser 1208 is described with reference to SEI message 1113, but the metadata can be stored using the SPS extension mechanism and / or the PPS extension mechanism. Metadata that is fixed for the duration of use of a single decoder network topology, such as packing information 1194, decoder topology 1192, and complexity representation 1191, is stored in the SPS. On the other hand, metadata that may change during the execution of a decoder network topology, such as quantization range 1196 and tensor information 1195 (e.g., number of channels), is stored in the PPS. The extension mechanisms of the SPS and PPS allow for the definition of additional syntax to notify of additional parameters beyond those specified in a particular version of the VVC (or HEVC) standard.

[0206] The methods presented herein enable the efficient representation of tensors in a format suitable for compression by modern block-based compression standards such as VVC and HEVC. Block-based compression is not intuitively applicable to data such as compressed feature maps and basis vector projection coefficients for feature map reconstruction, but the use of various transformations, including trained quadratic transformations, additionally discovers unexpected redundancy within blocks. The methods presented herein are described with reference to the "FasterRCNN" and "YOLOv3" network architectures, and specific configurations that divide these networks into "backbone" and "head" parts, but these methods are applicable to any neural network that processes multidimensional tensor data, and also to different configurations that divide such networks into "backbone" and "head" parts.

[0207] Although the source device 110 and destination device 140 are described with reference to a video source 112 consisting of video and image data, other types of content such as audio data and text data may also be supplied as input to a neural network applicable to such inputs, and the resulting intermediate feature maps may be compressed and decompressed by modules 116 and 146 with appropriate encoder and decoder network topologies. [Industrial applicability]

[0208] The described configuration is applicable to the computer and data processing industries, particularly to digital signal processing for encoding and decoding signals such as video and image signals, achieving high compression efficiency.

[0209] The above describes only some embodiments of the present invention, and modifications and / or changes can be made thereto without departing from the scope and spirit of the invention. These embodiments are illustrative and not limiting.

[0210] Appendix A The following is an example of the SEI message format and associated semantics for representing metadata related to tensor decompressor structures, tensor packing, and complexity representation within a bitstream: TIFF2026516561000002.tif219170TIFF2026516561000003.tif255170TIFF2026516561000004.tif33170 Here, u(n) refers to a fixed-length codeword of length n bits, and ue(v) refers to an unsigned exponential Gorloom variable-length codeword. FCVCM Decoder Information Semantics: If set_level_flag is 1, it indicates that the complexity level of the tensor decompressor will be displayed in this FCVCM decoder information SEI message. fcvcm_level indicates the complexity representation of any tensor decompressors executed within the decoder. The complexity representation provides a worst-case limit for the complexity of the instantiated tensor decompressors. As a bitstream compatibility requirement, the tensor decompression complexity representation must be communicated before the use of the FCVCM decoder. For example, it may be communicated in the first packetized tensor data frame within the bitstream. The following table shows the maximum allowable complexity elements for a given fcvcm_level value: If TIFF2026516561000005.tif60170update_decoder_flag is 1, it indicates that the FCVCM decoder will be updated. This update will be effective from this instance onward in the FCMVCM decoder information SEI message. If no_weights_flag is 1, it indicates that the FCVCM decoder does not contain any pre-trained elements (such as convolutions), and therefore does not require weights. If explicit_signal_decoder_flag is 1, it indicates that the FCVCM decoder architecture is explicitly notified in this instance of the FCVCM decoder information SEI message. If this value is zero, the FCVCM decoder information SEI message for this instance refers to a previously notified FCVCM decoder architecture or an FCVCM decoder architecture obtained by external means (e.g., a predefined architecture or an architecture available from a public registry). `explicit_decoder_compression_idc` specifies the compression technique (if any) to be applied to the payload containing the representation of the FCVCM decoder architecture, according to the table below: TIFF2026516561000006.tif51170explicit_decoder_format_idc specifies the format to which the FCVCM decoder architecture encodes. The following formats are supported: TIFF2026516561000007.tif60170explicit_decoder_format_version_idc specifies the version of the format to which the FCVCM decoder architecture encodes. For each supported format, a separate enumeration of explicit_decoder_format_version_idc values ​​corresponding to the version of that format is specified. `explicit_decoder_payload_len` specifies the length of the payload, in bytes, including the FCVCM decoder representation after compression (if applicable). decoder_payload[i] specifies the i-th byte of the FCVCM decoder representation. If register_decoder_idc_flag is 1, it indicates that the FCVCM decoder representation notified in this FCVCM decoder information SEI message will be registered (held) in the decoder for future reference. decoder_idc specifies an index value for addressing the FCVCM decoder representation within the registry of retained FCVCM decoder architectures. If explicit_signal_weights_flag is 1, it indicates that the weights associated with the notified FCVCM decoder representation are included in this FCVCM decoder information SEI message. explicit_weights_idc specifies the index of the weights notified in this FCVCM decoder information SEI message. explicit_weights_payload_len specifies the length of the weight payload in the FCvCM decoder information SEI message. weights_payload[i] specifies the i-th byte of the weight payload in the FCvCM decoder information SEI message. If register_weights_idc_flag is 1, it indicates that the weights notified in this FCVCM decoder information SEI message will be stored in the FCVCM decoder for future reference. registered_decoder_idc specifies an index for addressing an FCVCM decoder representation that is known to the decoder by external means or that was registered with the FCVCM decoder in a previous FCVCM decoder information SEI message. If set_region_cnt_flag is 1, this FCVCM decoder information SEI message indicates the number of regions to divide the current and subsequent pictures. `region_cnt` indicates the number of regions to divide the current and subsequent pictures into. Each region is rectangular and aligned with the CTU boundary. Each region contains feature maps from one or more tensors. If set_region_packing_flag is 1, this FCVCM decoder information SEI message specifies that the current picture should be divided into one or more rectangular regions. This division remains in effect until the next FCVCM decoder information SEI message with set_region_packing_flag set to 1 is sent. top_left_rsctuaddr[i] specifies the raster scan order address of the CTU located in the top-left position within the i-th region. top_right_rsctuaddr[i] specifies the raster scan order address of the CTU located in the upper right position of the i-th region. bottom_left_rsctuaddr[i] specifies the raster scan order address of the CTU located in the lower left position of the i-th region. bottom_right_rsctuaddr[i] specifies the raster scan order address of the CTU located in the lower right position within the i-th region. If horizontal_packing_flag[i] is 1, packing continues from left to right as packing or unpacking progresses from one tensor's feature map to the next within the i-th region. If the value is zero, when moving from one tensor's feature map to the next, the packing of the feature map progresses to the leftmost position within the current region and below any already packed feature maps. A value of 1 is used when packing multiple tensors, each containing a small number of feature maps (e.g., one). In this case, the width of the region is generally greater than the height, and the frame area of ​​the region is generally smaller. Setting set_tensor_info_flag to 1 will notify the number of tensors within the defined region and the dimensions of those tensors in this instance of the FCVCM decoder information SEI message. region_tensor_cnt[i] specifies the number of tensors to be packed into the i-th region. tensor_max_channels[i][j] specifies the maximum number of feature maps (i.e., channels) of the j-th tensor that can be packed into the i-th region. tensor_width[i][j] specifies the width of the feature map of the j-th tensor to be packed into the i-th region. tensor_height[i][j] specifies the height of the feature map of the j-th tensor to be packed into the i-th region. If update_tensor_channels_flag is 1, it indicates that a flag to update the number of packed feature maps for each tensor within each region will be notified in this instance of the FCVCM decoder information SEI message. If update_tensor_channel_flag[i][j] is 1, it indicates that the number of packings of feature maps for the j-th tensor in the i-th region is notified in this instance of the FCVCM decoder information SEI message. tensor_channel_cnt[i][j] specifies the number of packings for the feature map (i.e., channel) of the j-th tensor in the i-th region. If tensor_channel_cnt[i][j] is not notified but tensor_max_channels[i][j] is notified, its value is presumed to be equal to the corresponding tensor_max_channels[i][j]. If tensor_channel_cnt[i][j] is neither notified nor presumed in the current FCVCM decoder information SEI message, the value is carried over from the previous FCVCM decoder SEI message (if available). Otherwise, the value is presumed to be 0.

[0211] Appendix B The following code snippet shows a Python definition of MSFC-based decoder network topology 1500, which can decode one tensor and generate four decoded tensors. TIFF2026516561000008.tif204170TIFF2026516561000009.tif254170TIFF2026516561000010.tif254170TIFF2026516561000011.tif254170TIFF2026516561000012.tif254170TIFF2026516561000013.tif232170Example of complexity display: 1. Tensor information TIFF2026516561000014.tif211701.Calculation of computational complexity The number of operations can be derived by running the provided code snippet in pseudo-execution mode. In this mode, instead of performing operations such as convolution or maximum pooling, an alternative implementation simply counts the MAC (operation count) required for each operation, based on the dimensions of the input tensor, and returns a zero-value tensor of an appropriate size for later use. 2. Calculation of weight complexity The memory requirements for the weights can be calculated based on the parameters supplied to the learnable elements. For example, the number of weights in a convolution depends on the kernel size and the number of input and output channels parameters. 3. Calculation of persistent storage Variables declared and assigned to the "self" object using methods such as __init__ may contribute to the calculation of persistent storage memory.

[0212] Appendix C The following code snippet shows a Python definition (using only the numerical library "numpy") of PCA-based decoder network topology 1400 that can reconstruct one tensor and generate one decoded tensor. This code snippet assumes that the "decompress" method is implemented as part of the "Decompressor" class. This method takes three arguments, each containing one or more tensors decompressed from three images 910, 912, and 916, respectively. The input argument to the "decompress" method is the compressed tensor 1224, followed by a reference to the persistent state object "self", and the value returned by the method is tensor 149. Each input argument and return value of the "decompress" method can be a single tensor in "numpy" array format, or a list of tensors. TIFF2026516561000015.tif150170 Example of complexity display: 4. Tensor Information TIFF2026516561000016.tif41170 Each region contains one tensor with a specified dimension and usage. These input tensors generate the following internal variables: channels=256, n_components=25, weight=136, height=76, area=10336. 5. Calculation of computational complexity The computational complexity of operations like tensor "reshaping" (dimensional rearrangement) is ignored. Similarly, single multiplication (that does not manipulate arrays), assignment (including accessing parts of arrays and "slicing"), and transpose operations (e.g., the "coefs1.T" operation) are also ignored. Memory bandwidth complexity estimation may include the cost of variable substitution, including that resulting from array slicing. The operation "np.dot" performs an inner product on inputs of dimensions [10336,25] and [256,25], resulting in 25 × 256 × 10336 = 66,150,400 MAC operations. 6. Calculation of Weight Complexity This code snippet does not use PyTorch pre-trained operations such as convolution, and therefore does not use weights. 7. Calculation of persistent storage This code snippet does not refer to the "self" object and therefore does not preserve information between calls.

Claims

1. A method for decoding a tensor for the contents of a bitstream, Determining the network topology from the aforementioned bitstream, Decoding the first information from the bitstream, Using the determined network topology, a decoded tensor is generated from the first information, Methods that include...

2. The network topology is determined by decoding a second piece of information representing the network topology from the bitstream. The method according to claim 1.

3. The first information includes one or more tensors and / or one or more coefficients. The method according to claim 1.

4. At least a portion of the first information is stored in the packed frames of the video data. The method according to claim 1.

5. The decoded tensor is generated by instantiating a decoder network based on the determined network topology and executing the decoder network. The method according to claim 1.

6. The decoded tensor is generated by running the decoder network based on the determined network topology. The method according to claim 1.

7. The second information representing the network topology is signaled in a Supplementary Enhancement Information (SEI) message. The method according to claim 1.

8. The second information representing the network topology is signaled in a compressed format using syntax. The method according to claim 1.

9. The decoded network topology is retained for future reference, even after a different network topology has been used. The method according to claim 1.

10. The aforementioned network topology is indicated using an identifier. The method according to claim 1.

11. Network weights are signaled within Supplementary Enhancement Information (SEI) messages. The method according to claim 1.

12. Network weights are indicated using identifiers. The method according to claim 1.

13. The aforementioned content is one or more of audio data, video data, image data, and text data. The method according to claim 1.

14. A decoder for decoding a tensor of the contents of a bitstream, The network topology is determined from the aforementioned bitstream. The first information is decoded from the bitstream, Using the determined network topology, a decoded tensor is generated from the first information. A decoder configured in such a way.

15. A non-temporary computer-readable storage medium for storing a program for performing a method of decoding a tensor for the contents of a bitstream, The aforementioned method, Determining the network topology from the aforementioned bitstream, Decoding the first information from the bitstream, Using the determined network topology, a decoded tensor is generated from the first information, Non-temporary computer-readable storage media, including [specific type of storage medium].

16. It is a system, Memory and A processor configured to execute code stored in memory for performing a method of decoding a tensor for the contents of a bitstream, Includes, The method described above Determining the network topology from the aforementioned bitstream, Decoding the first information from the bitstream, Using the determined network topology, a decoded tensor is generated from the first information, A system that includes this.

Images