Methods, apparatus, and systems for encoding and decoding tensors

Abstract

A system and method for decoding a tensor and video from a bitstream, wherein the tensor is related to the video. The method includes decoding a video parameter set indicating the presence of multiple sub-bitstreams in the bitstream; decoding the video from a first sub-bitstream generated from the bitstream by performing a first sub-bitstream extraction process on the bitstream; and decoding a tensor for the decoded video from a second sub-bitstream obtained by performing a second sub-bitstream extraction process on the bitstream.

Description

[Technical Field]

[0001] References to related applications This application claims the benefits under 35 U.S. SC § 119 as of the filing date of Australian Patent Application No. 2023248075, filed on 10 October 2023, and is incorporated herein by reference as if it were fully described herein.

[0002] The present invention generally relates to digital video signal processing, and more particularly to methods, apparatus, and systems for encoding and decoding tensors from a convolutional neural network. The present invention also relates to a computer program product including a computer-readable medium recording a computer program for encoding and decoding tensors from a convolutional neural network using video compression techniques. [Background technology]

[0003] Convolutional neural networks (CNNs) are an emerging technology that addresses use cases involving machine vision, particularly object detection, instance segmentation, object tracking, human pose estimation, and behavior recognition. Applications for CNNs can include the use of "edge devices" with sensors and some processing power, coupled to application servers as part of the "cloud." CNNs can require relatively high computational complexity than would normally be given by edge devices in terms of either computing power or power consumption. Running CNNs in a distributed manner has emerged as one solution to run state-of-the-art networks using edge devices with limited capabilities without burdening all the computational complexity on cloud servers, while edge devices have potentially underutilized inference resources. In other words, distributed processing allows legacy edge devices to still deliver the capabilities of state-of-the-art CNNs by distributing processing between edge devices and other processing means such as cloud servers. Such distributed network architectures are sometimes called "collaborative intelligence" (CI) and offer advantages such as reusing partial results from the first part of the network in several different second parts, each part perhaps optimized for a different task. The CI architecture introduces the need for efficient compression of tensor data for transmission over networks such as WANs.

[0004] CNNs typically contain many layers, such as convolutional and fully connected layers, and data is passed from one layer to the next in the form of "tensors." Dividing a network across different devices introduces the need to compress the intermediate multidimensional tensor data passed from one layer to the next within the CNN to facilitate transmission over a network with bandwidth limitations and costs. Such tensor compression is sometimes called "feature compression," and the intermediate tensor data is often referred to as "features" or "feature maps." A feature or feature map is generally a collection of two-dimensional (2D) arrays of values ​​that combine to form a tensor in a 3D (or 4D) data structure, with each feature map corresponding to one "channel" of the tensor. Intermediate tensor data represents a partially processed form of the input, such as an image frame or video frame, that is encountered within the neural network. The International Organization for Standardization / International Electrotechnical Commission Joint Technical Committee 1 / Subcommittee 29 / Working Group 2 (ISO / IEC JTC1 / SC29 / WG2), also known as the "Moving Picture Experts Group" (MPEG) Technical Requirements, is tasked with investigating the requirements for compression techniques in various contexts, often related to video. WG2's "MPEG Technical Requirements" established the "Feature Compression for Machine Video Coding" (FCVCM) ad hoc group, tasked with investigating feature compression. The FCVCM AHG issued a "Call for Proposals" soliciting proposals to form the basis for a standardization project on feature compression. Previously, responses to a "Call for Evidence" (CfE) demonstrated techniques that could significantly surpass the feature compression results achieved using state-of-the-art standardization techniques directly applied to tensors.

[0005] CNNs typically require the weights of each layer to be pre-determined during the training phase, where a very large amount of training data passes through the CNN, and the results determined by the trained network are compared to the ground truth associated with the training data. The discrepancy between the obtained results and the desired results is expressed as "loss" and measured by the "loss function". Using the determined loss, a process is performed to update the network's weights, such as stochastic gradient descent (SGD). Updating the network's weights typically involves a process of backpropagation of the "gradient," starting at the network's output layer and working backward to cover the network's intermediate or "hidden" layers, ending when the input layer to the network is updated. The gradient represents the delta applied to the network's weights, and the gradient itself is updated as part of the backpropagation process. The rate of weight updates is set by the "learning rate" hyperparameter and is typically set to facilitate the training process finding the global minimum in terms of loss (i.e., the best possible task performance for the network architecture and training data) while avoiding the training process "falling" into a local minimum. Falling into a local minimum corresponds to achieving suboptimal task performance for the network architecture, preventing the discovery of new weight values ​​that could lead to higher task performance. Network weights are iteratively updated by feeding input data and ground truth data organized into “batches” to iteratively refine the network's performance until further improvements in accuracy cannot be achieved. A single iteration through the entire training dataset forms one “epoch” of training, and training typically requires performing multiple epochs to achieve a high level of performance for the task. The trained network weights are then available for deployment, and the network operates in a mode where the weights are fixed and gradients for weight updates are omitted.The process of running a pre-trained CNN with inputs and gradually transforming the inputs into outputs according to the CNN's topology is generally called "inference."

[0006] Generally, a tensor has four dimensions: batch, channels, height, and width. The first dimension, "batch," is typically size 1 when inferring video data, indicating that one frame passes through the CNN as one batch. When training the network, the batch dimension can be increased according to a given "batch size" so that multiple frames pass through the network in each batch before the network weights are updated. A multi-frame video can be passed through as a single tensor with a batch dimension whose size is increased according to the number of frames in a given video. However, due to practical considerations regarding memory consumption and access, inference on video data is typically performed frame by frame. The "channel" dimension indicates the number of simultaneous "feature maps" for a given tensor, and the height and width dimensions indicate the size of the feature maps at a particular stage of the CNN. The channel count changes through the layers of the CNN according to the network architecture. Also, the feature map size changes depending on whether subsampling or upsampling occurs in a particular network layer.

[0007] The overall complexity of CNNs tends to be relatively high, with a relatively large number of MAC (multiply-accumulate) operations being performed, numerous intermediate tensors being written to and read from memory, and weights also being read to execute each layer of the CNN. Therefore, dividing a neural network into parts allows for the implementation of more complex networks even on systems that include less powerful edge devices, without placing the entire burden of running the network on a cloud server.

[0008] Feature compression can benefit from existing video compression standards such as Versatile Video Coding (VVC), developed by the Joint Video Experts Team (JVET). VVC is particularly expected to address the ongoing demand for higher compression performance as video formats increase their capabilities (e.g., with higher resolution and higher frame rates) and the growing market demand for service delivery over WANs where bandwidth costs are relatively high. VVC is feasible in modern silicon processing and offers an acceptable trade-off between the performance achieved and the cost of implementation. The cost of implementation can be considered, for example, in terms of one or more of the following: silicon area, CPU processor load, memory usage, and bandwidth. Other video compression standards such as High Efficiency Video Coding (HEVC) or AV-1 can also be used for feature compression applications.

[0009] Video data consists of a sequence of frames of image data, each frame containing one or more color channels. When feature map data is represented by packed frames, monochrome frames, which generally contain only luminance and no chroma channels, are appropriate. When only luminance samples are present, the resulting monochrome frame is said to use a "4:0:0 chroma format."

[0010] The VVC standard defines a “block-based” architecture, where a frame is first divided into an array of square regions known as “coding tree units” (CTUs). In VVC, a CTU typically occupies 128x128 lumens. Other possible CTU sizes when using the VVC standard are 32x32 and 64x64. However, CTUs at the right and bottom edges of each frame may be smaller in area, resulting in implicit divisions and ensuring that coding blocks remain within the frame. Each CTU is associated with a “coding tree,” which defines the decomposition of the CTU’s area into a set of blocks also called “coding units” (CUs). Blocks applicable only to lumens or only to chroma channels are called “coding blocks” (CBs). Predictions of the content of coding blocks are held in “prediction blocks” (PBs) or “prediction units” (PUs), and residual blocks, which define an array of sample values ​​that are added together with the PB or PU, are called “transformation blocks” (TBs) or “transformation units” (TUs) due to their typical use in the transformation process in the generation of TBs or TUs.

[0011] Despite the above distinction between "units" and "blocks," the term "block" can be used as a general term to refer to an area or region of a frame in which the operation applies to all color channels.

[0012] For each CU, a prediction unit (PU) is generated for the content (sample values) of the corresponding area of ​​frame data ("prediction unit"). Furthermore, a representation of the difference (or "spatial domain" residual) between the content of the area seen in the input to the encoder and the prediction is formed. The difference in each color channel can be transformed and coded as a sequence of residual coefficients to form one or more TUs for a given CU. The transformation applied may be a discrete cosine transform (DCT) or other transformation applied to each block of residual values. Transformations are applied separately (i.e., a two-dimensional transformation is performed in two passes, one horizontal and one vertical). Blocks are first transformed by applying a one-dimensional transformation to each row of samples within the block. Sub-results are then transformed by applying a one-dimensional transformation to each column of the sub-results to produce a final block of transformation coefficients that substantially decorrelated the residual samples. Transformations of various sizes are supported by the VVC standard, including the transformation of rectangular-shaped blocks, where the dimension of each side is a power of 2. Transformation coefficients are quantized for entropy coding to the bitstream.

[0013] In VVC, the PB or PU can be generated using either intra-frame prediction or inter-frame prediction. Intra-frame prediction involves the use of previously processed samples within the frame, which are used to generate a prediction for the current block of data samples within the frame. Inter-frame prediction involves generating a prediction for the current block of samples within a frame, using a block of samples taken from one or two previously decoded frames. The block of samples taken from previously decoded frames is offset from the spatial position of the current block according to a motion vector and is often filtered. The intra-frame prediction block may be (i) a uniform sample value ("DC intra-prediction"), (ii) a plane with offsets and horizontal and vertical slopes ("plane intra-prediction"), (iii) a set of blocks with adjacent samples applied in a particular direction (population) ("angle intra-prediction"), or (iv) the result of matrix multiplication using adjacent samples and selected matrix coefficients.

[0014] VVC can be used to compress intermediate feature maps from the first part ("backbone") of a neural network that is separated into two parts. In compression, the feature maps from the backbone are placed in frames and quantized from the floating-point domain to the sample domain, which is suitable for compression as video data. CNNs represent a rapidly evolving technology, and CNNs used for machine vision tasks may be updated with different versions or replaced with new CNN architectures as the technology develops. A bitstream containing tensors from the backbone (CNN Part 1) can be decoded and fed into one or more network heads (CNN Part 2) from among many available network heads. There is a need to identify which network heads are compatible with the tensors provided from the decoded bitstream.

[0015] Machine vision tasks performed by CNNs can provide output based on the specific head network used. The results may relate to things like bounding boxes and segmentation markers. In some cases, humans may want to review or interpret the CNN results. There is a need to assist human interpretation of CNNs in several use cases. [Overview of the Initiative]

[0016] The object of the present invention is to substantially overcome, or at least improve upon, one or more problems of existing configurations.

[0017] One aspect of the present disclosure provides a method for decoding a tensor and a video from a bitstream including a first sub-bitstream and a second sub-bitstream, wherein the tensor is related to the video, and the method includes decoding a video parameter set indicating the presence of a plurality of sub-bitstreams in the bitstream; decoding the video from the first sub-bitstream obtained by performing a first sub-bitstream extraction process on the bitstream; and decoding a tensor for the decoded video from the second sub-bitstream obtained by performing a second sub-bitstream extraction process on the bitstream.

[0018] Another aspect of the Disclosure provides a decoder for decoding a tensor and video from a bitstream including a first sub-bitstream and a second sub-bitstream, wherein the tensor is related to the video, and the decoder is configured to decode a video parameter set indicating the presence of multiple sub-bitstreams in the bitstream, decode the video from the first sub-bitstream obtained by performing a first sub-bitstream extraction process on the bitstream, and decode a tensor for the decoded video from the second sub-bitstream obtained by performing a second sub-bitstream extraction process on the bitstream.

[0019] Other aspects of the present disclosure provide a non-temporary, computer-readable storage medium for storing a program for performing a method of decoding tensors and video from a bitstream including a first sub-bitstream and a second sub-bitstream, wherein the tensors are related to the video, and the method includes decoding a set of video parameters indicating the presence of a plurality of sub-bitstreams in the bitstream; decoding the video from the first sub-bitstream obtained by performing a first sub-bitstream extraction process on the bitstream; and decoding tensors for the decoded video from the second sub-bitstream obtained by performing a second sub-bitstream extraction process on the bitstream.

[0020] Other forms will also be disclosed. [Brief explanation of the drawing]

[0021] Next, at least one embodiment of the present invention will be described with reference to the following drawings and appendices. [Figure 1] Figure 1 is a schematic block diagram illustrating a distributed machine task system. [Figure 2A]FIG. 2A forms a schematic block diagram of a general-purpose computer system in which the distributed machine task system of FIG. 1 may be implemented. [Figure 2B] FIG. 2B forms a schematic block diagram of a general-purpose computer system in which the distributed machine task system of FIG. 1 may be implemented. [Figure 3A] FIG. 3A is a schematic block diagram showing functional modules of the backbone portion of a CNN. [Figure 3B] FIG. 3B is a schematic block diagram showing the residual block of FIG. 3A. [Figure 3C] FIG. 3C is a schematic block diagram showing the residual unit of FIG. 3A. [Figure 3D] FIG. 3D is a schematic block diagram showing the CBL module of FIG. 3A. [Figure 4] FIG. 4 is a schematic block diagram showing functional modules of an alternative backbone portion of a CNN. [Figure 5] FIG. 5 is a schematic block diagram of a tensor encoder using a configurable tensor compression stage. [Figure 6] FIG. 6 is a schematic block diagram showing a multi-scale feature fusion stage for a tensor compressor. [Figure 7] FIG. 7 is a schematic block diagram showing an inter-channel decorrelation-based tensor compressor. [Figure 8] FIG. 8 is a schematic block diagram showing functional modules of a video encoder. [Figure 9A] FIG. 9A is a schematic block diagram showing an arrangement of regions or sub-pictures for holding compressed tensor data. [Figure 9B] FIG. 9B is a schematic block diagram showing an arrangement of regions or sub-pictures for holding compressed tensor data. [Figure 10A] FIG. 10A is a schematic block diagram showing another arrangement of regions or sub-pictures for holding compressed tensor data. [Figure 10B]Figure 10B is a schematic block diagram showing an alternative arrangement of regions or subpictures for holding compressed tensor data. [Figure 11] Figure 11 is a schematic block diagram showing the encoded inter-channel uncorrelated feature map, compressed video, and bitstream holding the associated metadata. [Figure 12] Figure 12 is a schematic block diagram showing a tensor decoder with a configurable tensor decompressor. [Figure 13] Figure 13 is a schematic block diagram showing the functional module of the video decoder. [Figure 14] Figure 14 is a schematic block diagram showing an inter-channel uncorrelatedness-based tensor decoder as part of a distributed machine task system. [Figure 15] Figure 15 is a schematic block diagram showing an embodiment of the multiscale feature reconstruction stage. [Figure 16A] Figure 16A is a schematic block diagram showing the head portion of a CNN. [Figure 16B] Figure 16B is a schematic block diagram showing the upscaler module of Figure 16A. [Figure 16C] Figure 16C is a schematic block diagram showing the detection module shown in Figure 16A. [Figure 17] Figure 17 is a schematic block diagram showing an alternative head portion of a CNN. [Figure 18] Figure 18 shows a method for running the first part of the CNN, compressing the resulting tensor, and signaling the instructions for the first part of the CNN. [Figure 19] Figure 19 shows the method for decoding the bitstream, reconstructing the tensor, and completing the task according to the instructions for the network topology and network weights. [Figure 20] Figure 20 illustrates a method for determining network instructions for the entire network and parts thereof. [Figure 21A] Figure 21A is a schematic block diagram showing the contents of the NN instruction. [Figure 21B] Figure 21B is a schematic block diagram showing the relationships between NN instructions. Appendix A shows the syntax table of "Supplemental Extension Information" (SEI) messages that hold the metadata necessary for the operation of the decoding method in Figure 19. [Modes for carrying out the invention]

[0022] Where any step and / or feature having the same reference numeral is referenced in any of the attached drawings, those steps and / or features have the same function or operation for the purposes of this description, unless otherwise intended.

[0023] A distributed machine task system can include edge devices such as network cameras or smartphones that generate intermediate compressed data. The distributed machine task system may also include end devices, such as server farm-based ("cloud") applications, that operate on the intermediate compressed data to generate task results. In addition, edge device functionality may be embodied in the cloud, and the intermediate compressed data may potentially be stored for subsequent processing for multiple different tasks as needed. The distributed task system may perform tasks in a general form on edge devices and provide partially processed features for performing tasks in a specific form on the server side. For example, a general object detection network may run on a client (edge) device, and the results may be used on a server to conditionally run a more specific detection network using partially processed tensors generated by a first part of a CNN common to both the general task network and the specific task network. Such a first part is sometimes called a "shared backbone" because it is common to multiple task networks in both network topology and network weights.

[0024] A convenient form of intermediate compressed data is the compressed video bitstream, due to the availability of high-performance compression standards and their implementations. Video compression standards typically operate on integer samples of a given bit depth, such as 10 bits, arranged in a planar array. Color video has three planar arrays, depending on the application, corresponding to, for example, color components Y, Cb, Cr, or R, G, B. Convolutional Neural Networks (CNNs) typically operate on floating-point data in the form of tensors. Tensors generally have a relatively small spatial dimension compared to the input video data on which the CNN operates, while having more channels than the three channels typical of color video data, e.g., 128, 256, or 512 channels.

[0025] A tensor typically has the following dimensions: frames, channels, height, and width. For example, a tensor of dimensions [1, 256, 76, 136] is said to contain floating-point or integer values ​​for one frame, each containing an array of 256 feature maps (channels) of size 136x76. For video data, inference is typically performed one frame at a time (frame value 1) rather than using a tensor containing multiple frames.

[0026] VVC supports the division of a picture into multiple subpictures, each of which can be independently encoded and decoded. One approach is to encode each subpicture as a single "slice" or as a contiguous sequence of encoded CTUs. A "tiling" mechanism is also available to divide a picture into several independently decodeable regions. Subpictures can be specified in a somewhat flexible manner using various sets of rectangles of CTUs encoded as each subpicture. The flexible definition of subpicture dimensions allows for the efficient storage of types of data that require different areas within a single picture, avoiding large "unused" areas, i.e., areas of frames not used for reconstructing tensor data.

[0027] Figure 1 is a schematic block diagram showing a functional module of a distributed machine task system 100 that can execute machine task networks in a distributed manner. To divide a particular neural network into two parts, it is necessary to specify a “split point” within the network. The layers in the network from the input layer to the split point are executed on the first device (or “source device”), and the resulting intermediate tensor is compressed. The layers up to the split point are sometimes called the “backbone,” but this term can sometimes imply a specific split point. The alternative term “NN part 1” (neural network part 1, also called the first network part) does not imply the location of the split point within the network. The layers from the split point to the last layer are sometimes called the “head,” and to avoid the implied location of the split point, they may instead be called “NN part 2” (neural network part 2, also called the second network part). The first NN part 2 (also called proxy NN part 2) executes the remaining layers of the network to generate the initial task result and is included in the bitstream for later use. Based on the results obtained from the network, the second device (or "destination device") uses the decompressed tensor from the first device as input to the layer immediately following the split point, and the layers from the split point to the last layer in the alternate network are executed. Since the alternate network has a common NN part 1 with the first network, the task result 167 can be generated by executing NN part 2 using the tensor generated by NN part 1 of the first network.

[0028] At a split point, there may be one or more tensors that need to be compressed for transport over a communication channel with limited bandwidth, compared to the bandwidth requirements for transmitting uncompressed tensors. When a "feature pyramid network" (FPN) is used, layers within the FPN are typically associated with width and height such that a given layer is half the width and height of adjacent layers within the layer. FPN architectures may also involve alternating halving of width and height from one layer to the next. In some architectures, multiple tensors of the same width and height are generated within the FPN. FPNs can occur relatively early in a neural network topology, and as a result, the split point needs to occur within an FPN in order to partition network workloads in a useful way between edge devices and the cloud. When the split occurs within an FPN of a machine task network, various machine task network executions can be achieved where the layers up to the split point are common across machine task networks ("shared backbone" architecture). When the split point occurs within an FPN, tensor compression methods can leverage redundancy between FPN layers to improve compression performance. Therefore, compression methods applicable to the various network topologies used in modern CNNs are beneficial for application in a wide range of scenarios.

[0029] System 100 can be used to decorrelated, pack, and quantize feature maps into planar frames for encoding, and to implement methods for decoding feature maps from encoded data of various neural networks. Different neural networks may be partitioned at different points, potentially resulting in intermediate tensors of varying numbers and dimensions. The numerous possible neural networks and partition points create the need to determine the compatibility of the destination device 140 with the available NN Part 2 implementation repositories available in the NN Part 2 repository 160. Enumerating potential neural networks is challenging because it requires maintaining a centralized, agreed-upon repository of network names. Furthermore, reference implementations may only be available in temporary forms, such as public software repositories, which can change as commits are made. Further complexity arises from the fact that the machine learning community has not agreed on the naming of newly created networks, as seen in the "YOLO" family of networks, where multiple different researchers may claim to have published the next YOLO version of that family. Open Neural Network Exchange (ONNX) is one form that provides a device-independent way for specifying the weights and topology of neural networks. Removing the weights, the ONNX topology provides a relatively lightweight and complete representation of neural network topology. Signaling the entire topology to identify a network can become unscalable as the network topology grows, and moreover, a more abbreviated description of the network topology is sufficient to uniquely identify a network without having to refer to potentially unstable public resources. Even referring to a paper is usually insufficient, as that paper refers to a public software repository (such as one hosted on github.com). Therefore, a signaling mechanism is needed that allows for obvious compatibility decisions between NN Part 1 and one or more NN Part 2 implementations to enable interoperability between different source and destination devices.In particular, mechanisms that do not rely on centralized enumerations such as registration bodies are preferred, because establishing, maintaining, and promoting the use of such registration bodies is usually burdensome and hinders industry adoption. Instead, a more concise signaling means than transmitting network-wide weights and / or topology is a more easily deployable alternative that avoids the need for an externally maintained and recognized authoritative registry of network weights and / or topology.

[0030] In some implementations, system 100 may transport both intermediate compressed features from NN part 1 and compressed video from the video source from the source device to the destination device. Including both video and compressed features enables use cases such as overlaying task results, such as bounding boxes for specific objects, onto the video for human review. Another use case is to initiate long-term storage of videos that may be of future interest, triggered by the task results of NN part 2 on the source device or NN part 2 on the destination device. In one example, the video is retained only if a person is detected, reducing storage costs compared to storing all received videos. If it is necessary to identify a specific person of interest, NN part 2 trained to detect that specific person of interest may be executed. The destination device may reprocess the NN part 1 tensor corresponding to frames containing at least one detected person using a customized NN part 2 to further narrow down the video data to the most relevant clips, which can then be reviewed by a human. In other implementations, the video may not be retained unless a video request is made.

[0031] System 100 includes a source device 110. Source device 110 includes a video source 112 for generating unencoded frame data 113. The frame data 113 is passed to NN part 1 114 to generate a tensor 115 and to a video encoder 150 to generate a video layer bitstream 151 (first sub-bitstream). The tensor 115 is passed to a tensor encoder 116 for generating a feature layer bitstream 121 (second sub-bitstream) and to a proxy NN part 2 118 for generating a proxy task result 119. Transmitter / multiplexer 122 generates a multilayer bitstream 123. The multilayer bitstream 123 includes a sequence of access units (e.g., access unit 11105 in Figure 11) where each access unit includes one compressed frame from the video layer bitstream 151 and one compressed frame from the feature layer bitstream 121. The video layer bitstream 151 forms the "picture layer" in the multilayer bitstream 123, and the feature layer bitstream 121 forms the "auxiliary layer" in the multilayer bitstream 123. The auxiliary layer typically encodes an alpha channel for calculating transparency when rendering frames from the picture layer, or a depth map for rendering stereoscopic frames from the picture layer. Therefore, a typical use of the auxiliary layer is to encode information for the picture layer. The information encoded for the picture layer in the auxiliary layer is usually spatially closely related to the picture or video layer, for example, per sample (or nearly per sample). Packed features derived from tensor 115 from NN part 1 114 form a highly transformed version of the frame data 113. The resolution of the feature frame depends on the number of tensors in tensor 115 and the number of channels and resolution of each tensor 115, which is not related (or indirectly related) to the resolution of the frame in the frame data 113. Furthermore, there is no simple spatial relationship between the highly abstract feature map data and the 113 samples of frame data.The proxy task results 119 are encoded as metadata in the multi-layer bitstream 123, for example, using the SEI message associated with the frame used to generate each proxy task result 119.

[0032] System 100 also includes a destination device 140 for decoding tensor data in the form of the received multilayer bitstream 123. The destination device 140 may be used to decode tensor data (or tensors) for the content of the multilayer bitstream 123 (e.g., audio data, video data, image data, and text data).

[0033] Communication channel 130 is used to communicate a multi-layer bitstream 123 from source device 110 to destination device 140. In some configurations, one or both of source device 110 and destination device 140 may comprise their respective mobile phones (e.g., "smartphones") or network cameras and cloud applications. Communication channel 130 may be a wired connection such as Ethernet, or a wireless connection such as WiFi or 5G, and includes connections over a wide area network (WAN). Communication channel 130 may also be implemented via an ad-hoc connection. Furthermore, source device 110 and destination device 140 may comprise applications in which encoded video data is captured onto some computer-readable storage medium such as a hard disk drive or memory in a file server. System 100 is described as including a video source 112 that provides frame data 113 to a neural network targeting a computer vision application, but other types of source data, such as audio or text, may be input to appropriate neural networks implemented in NN part 1 114 and NN part 2 head 166.

[0034] As shown in Figure 1, the source device 110 includes a video source 112, an NN part 1 114, a tensor encoder 116, and a transmitter / multiplexer 122. The video source 112 typically comprises a source of captured video frame data (shown as 113), such as an image capture sensor, a previously captured video sequence stored on a non-temporary recording medium, or a video feed from a remote image capture sensor. The video source 112 may also be the output of a computer graphics card, for example, displaying the video output of an operating system and various applications running on a computing device (e.g., a tablet computer). Examples of source devices 110 that may include an image capture sensor as the video source 112 include smartphones, video camcorders, professional video cameras, and network video cameras. The video source 112 may produce independent images or a temporally continuous series of images, i.e., video.

[0035] System 100 implements a distributed task network, divided into two parts: NN part 1 114 and NN part 2 166. Furthermore, source device 110 provides a general-purpose, i.e., "proxy" NN part 2 118, the result of which can be used as a trigger to initiate processing at destination device 140. For example, a "YOLOv3" network may be used as part of an object tracking system, and a "FasterRCNN" network as an object detection system. The number and dimensions of tensor 115 depend on the specific network and the division point of the particular task network implemented in system 100.

[0036] NN part 1 114 receives video frame data 113, executes specific layers of the overall CNN, such as the layer corresponding to the CNN's "backbone," and outputs tensor 115. The CNN's backbone (part 1) layer may generate multiple tensors as output 115, corresponding to different spatial scales of the input image represented by the video frame data 113, for example, when the network is divided within the FPN. The FPN may produce three tensors corresponding to three layers with varying spatial resolutions and number of channels, which are output as tensor 115 from the backbone 114 (for example, if the "YOLOv3" network is being run by system 100). If system 100 is running a network such as "Faster RCNN X101-FPN" or "Mask RCNN X101-FPN," tensor 115 may contain tensors from four layers (P2-P5). As described below, when a network split point occurs within an FPN, the use of an FPN results in multiple tensors forming a hierarchical representation of a single frame that is encoded (and decoded) into a bitstream. Tensor encoder 116 receives tensor 115 and generates a feature bitstream 121 containing a compressed representation of tensor 115.

[0037] The feature bitstream 121 is supplied to the transmitter / multiplexer 122 for transmission via the communication channel 130 as part of the multilayer bitstream 123. Alternatively or additionally, the multilayer bitstream 123 may be written to storage 132 for later use.

[0038] Source device 110 supports a specific network for NN part 1 114. However, destination device 140 may use one of several networks as NN part 2 166. Selecting a compatible NN part 2 166 for use in the multi-layer bitstream 123 from an optional library of several candidate NN part 2s is generally done at destination device 140. If one of several candidate NN part 2s is used as NN part 2 166, partially processed data in the form of packed feature maps can be saved for later use in performing various tasks without having to repeat the calculations of NN part 1 114.

[0039] The multilayer bitstream 123 is transmitted as encoded data by the transmitter 122 via the communication channel 130. In some implementations, the multilayer bitstream 123 may be stored in storage memory 132, which is a non-temporary storage device such as "flash" memory or a hard disk drive, until it is later transmitted via the communication channel 130 (or instead of being transmitted via the communication channel 130). For example, the encoded video data may be provided to customers on demand via a wide area network (WAN) for video analytics applications.

[0040] The destination device 140 includes a receiver / demultiplexer 142, a tensor decoder 146, an NN Part 2 repository 160, an NN Part 2 166, a video decoder 170, and a task result renderer 168. The receiver / demultiplexer 142 receives a multilayer bitstream 123 from communication channel 130. The receiver / demultiplexer 142 includes an entropy decoder that can parse a higher level syntax applicable to the entire multilayer bitstream 123, such as an SEI message as described with reference to Figure 11. The multilayer bitstream 123 includes a primary layer that holds compressed video data and an extension (feature) layer that holds compressed tensors. Module 142 extracts the video sub-bitstream 171 (corresponding to the video layer bitstream 151) and the feature sub-bitstream 143 (corresponding to the feature layer bitstream 121) from the multi-layer bitstream 123 using a "sub-bitstream extraction process" that filters out unnecessary NAL units to decode a specific layer of the multi-layer bitstream 123. The feature sub-bitstream 143 is passed to the tensor decoder 146, and the video sub-bitstream 171 is passed to the video decoder 170. The NN part 1 network instruction (i.e., topology and weight hash) 161 is determined from the bitstream 123 and passed from module 142 to the NN part 2 repository 160. The tensor decoder 146 outputs the decoded tensor 149, which is fed to the NN part 2 166. The NN part 2 166 performs a selection network 162, which is selected from a set of NN part 2 candidates from the NN part 2 repository 160 from among candidates compatible with the NN part 1 instruction 161. Candidates for NN Part 2 are selected based on their compatibility with the specified NN Part 1 obtained from the multi-layer bitstream 123, and, if applicable, on the desired task (e.g., object tracking, image segmentation, etc.). If multiple candidates for NN Part 2 are compatible with the specified NN Part 1, the choice of which to execute is left to the discretion of the user or programmer of the destination device 140.For example, the selection may be based on user input to configure the destination device 140 for a specific concrete task, such as identifying a particular person using features from NN part 1 of source device 110. NN part 2 166 receives the tensor 149 and runs subsequent layers of the neural network, which began in NN part 1 114, to produce the task result 167.

[0041] The task result 167 is presented to the user, for example, by being visually displayed by the task result renderer 168. The task result renderer 168 draws bounding boxes or segmentation maps that highlight the detected objects or instances according to the task result 167 and overlays them on the decoded video 172 generated by the video decoder 170. The task result renderer 168 is an example of how neural network results are presented to the user. The task result renderer 168 may only be operational if the task result 168 indicates detected objects.

[0042] Furthermore, the functions of the source device 110 and the destination device 140 can also be embodied in a single device, such as a mobile phone, tablet computer, or cloud application.

[0043] Regardless of the device examples listed above, each of the source device 110 and destination device 140 can typically be configured within a general-purpose computing system through a combination of hardware and software components. Figure 2A shows such a computer system 200, including a computer module 201, a keyboard 202 and a mouse pointer device 203 and a scanner 226, a camera 227 which may be configured as a video source 112, and output devices including an input device such as a microphone 280, a printer 215, a display device 214, and a loudspeaker 217. An external modulator-demodulator (modem) transceiver device 216 may be used by the computer module 201 to communicate with a communication network 220 via connection 221. The communication network 220, which may represent communication channel 130, may be the Internet, a cellular telecommunications network, or a private WAN such as a WAN. If connection 221 is a telephone line, the modem 216 may be a conventional “dial-up” modem. Alternatively, if connection 221 is a high-capacity (e.g., cable or fiber optic) connection, modem 216 may be a broadband modem. A wireless modem may be used for wireless connection to communication network 220. Transceiver device 216 may also provide the functions of transmitter 122 and receiver 142, and communication channel 130 may be implemented in connection 221.

[0044] The computer module 201 typically includes at least one processor unit 205 and a memory unit 206. For example, the memory unit 206 may have semiconductor RAM (random access memory) and semiconductor ROM (read-only memory). The computer module 201 also includes several input / output (I / O) interfaces, including an audio-video interface 207, a loudspeaker 217, and a microphone 280 coupled to a video display 214, an I / O interface 213 coupled to a keyboard 202, a mouse 203, a scanner 226, a camera 227, and optionally a joystick or other human interface device (not shown), and an interface 208 for an external modem 216 and a printer 215. The signal from the audio-video interface 207 to the computer monitor 214 is generally the output of a computer graphics card. In some embodiments, the modem 216 may be incorporated within the computer module 201, for example, within interface 208. The computer module 201 also has a local network interface 211 that allows the computer system 200 to be connected via a connection 223 to a local area communication network 222 known as a local area network (LAN). As shown in Figure 2A, the local communication network 222 can also be connected to a wide network 220 via a connection 224, which typically includes a so-called "firewall" device or a device with similar functionality. The local network interface 211 is Ethernet TM Circuit card, Bluetooth TM It may have a wireless configuration or an IEEE 802.11 wireless configuration, but many other types of interfaces can be implemented for interface 211. The local network interface 211 can also provide the functions of a transmitter 122 and a receiver 142, and the communication channel 130 can also be implemented in the local communication network 222.

[0045] I / O interfaces 208 and 213 can provide either or both serial and parallel connectivity, the former typically implemented according to the Universal Serial Bus (USB) standard and having a corresponding USB connector (not shown). A storage device 209 is provided, typically including a hard disk drive (HDD) 210. Other storage devices such as floppy disk drives or magnetic tape drives (not shown) may also be used. An optical disk drive 212 is typically provided to function as a non-volatile source of data. For example, such optical disks (e.g., CD-ROM, DVD, Blu-ray Disc) TM Portable memory devices such as USB-RAM, portable external hard drives, and floppy disks can be used as suitable data sources to the computer system 200. Typically, the HDD 210, optical drive 212, and networks 220 and 222 may be configured to act as a video source 112 or as a destination for decoded video data stored for playback via the display 214. The source device 110 and destination device 140 of system 100 may be implemented in the computer system 200.

[0046] Components 205–213 of computer module 201 typically communicate via an interconnected bus 204 to bring about a conventional operating mode of computer system 200 known to those skilled in the art. For example, processor 205 is coupled to system bus 204 using connection 218. Similarly, memory 206 and optical disc drive 212 are coupled to system bus 204 by connection 219. Examples of computers capable of implementing the described configuration include IBM-PC and compatibles, SunSPARCstations, and AppleMac. TM , or similar computer systems.

[0047] The video encoder 150, NN part 1 114, tensor encoder 116, proxy NN part 2, modules 122 and 142, tensor decoder 146, NN part 2 166, video decoder 170 and result renderer 168, and the method described may be implemented as one or more software application programs 233 executable within a computer system 200. In particular, the modules of devices 110 and 140 and the steps of the method described are carried out by instructions 231 (see Figure 2B) within the software 233 executed within the computer system 200. The software instructions 231 may be formed as one or more code modules, each for performing one or more specific tasks. The software may also be divided into two distinct parts, the first part and its corresponding code modules for performing the method described, and the second part and its corresponding code modules for managing the user interface between the first part and the user.

[0048] The software may be stored, for example, on a computer-readable medium including a storage device as described below. The software is loaded from the computer-readable medium into the computer system 200 and then executed by the computer system 200. A computer-readable medium having such a computer program or software recorded on it is a computer program product. The use of the computer program product in the computer system 200 preferably results in an advantageous apparatus for carrying out the source device 110 and the destination device 140, and the method described.

[0049] The software 233 is typically stored on the HDD 210 or in memory 206. The software is loaded from a computer-readable medium into the computer system 200 and executed by the computer system 200. For example, the software 233 may be stored on an optically readable disk storage medium (e.g., a CD-ROM) 225 that is read by an optical disk drive 212.

[0050] In some examples, the application program 233 may be encoded on one or more CD-ROMs 225 and supplied to the user, and read via the corresponding drive 212, or read by the user from a network 220 or 222. Furthermore, the software may also be loaded into the computer system 200 from other computer-readable media. Computer-readable storage media refers to any non-temporary tangible storage medium that provides recorded instructions and / or data to the computer system 200 for execution and / or processing. Examples of such storage media include floppy disks, magnetic tapes, CD-ROMs, DVDs, and Blu-ray Discs. TM This includes, but is not limited to, hard disk drives, ROMs or integrated circuits, USB memory sticks, magneto-optical disks, or computer-readable cards such as PCMCIA cards, and whether such devices are internal to or external to the computer module 201. Examples of temporary or intangible computer-readable transmission media that may also participate in providing software, application programs, instructions, and / or video data or encoded video data to the computer module 201 include wireless or infrared transmission channels, as well as network connections to other computers or networked devices, as well as the internet or intranets, including email transmissions and information recorded on websites.

[0051] The second part of the application program 233 and the corresponding code modules described above may be implemented to implement one or more graphical user interfaces (GUIs) that are rendered or represented on the display 214. Typically, through the operation of the keyboard 202 and mouse 203, the user of the application and the computer system 200 can operate the interface in a functionally adaptable manner to provide control commands and / or inputs to the GUI-related application. Other forms of functionally adaptable user interfaces may also be implemented, such as an audio interface that utilizes speech prompts output via the loudspeaker 217 and user voice commands input via the microphone 280.

[0052] Figure 2B is a detailed schematic block diagram of the processor 205 and “memory” 234. Memory 234 represents the logical set of all memory modules (including storage devices 209 and semiconductor memory 206) accessible by the computer module 201 in Figure 2A.

[0053] When the computer module 201 is first powered on, a power-on self-test (POST) program 250 is executed. The POST program 250 is typically stored in the ROM 249 of the semiconductor memory 206 in Figure 2A. Hardware devices such as the ROM 249 that store software are sometimes called firmware. The POST program 250 checks the hardware within the computer module 201 to ensure proper functioning, and typically checks the processor 205, memory 234 (209, 206), and the Basic Input / Output System Software (BIOS) module 251, also typically stored in the ROM 249, for correct operation. If the POST program 250 executes successfully, the BIOS 251 boots the hard disk drive 210 in Figure 2A. The booting of the hard disk drive 210 causes the bootstrap loader program 252, which resides on the hard disk drive 210, to be executed via the processor 205. This loads the operating system 253 into the RAM memory 206, and the operating system 253 begins to operate. Operating System 253 is a system-level application runnable by Processor 205, performing various high-level functions including processor management, memory management, device management, storage management, software application interface, and general-purpose user interface.

[0054] The operating system 253 manages memory 234 (209, 206) to ensure that each process or application running on the computer module 201 has enough memory to run without conflicting with memory allocated to other processes. Furthermore, the different types of memory available in the computer system 200 in Figure 2A must be used appropriately so that each process can run effectively. Therefore, aggregated memory 234 is not intended to show how specific segments of memory are allocated (unless otherwise specified), but rather to provide a general view of the memory accessible by the computer system 200 and how such memory is used.

[0055] As shown in Figure 2B, the processor 205 includes several functional modules, including a control unit 239, an arithmetic logic unit (ALU) 240, and local or internal memory 248, sometimes called cache memory. The cache memory 248 typically contains several storage registers 244-246 within a register section. One or more internal buses 241 functionally interconnect these functional modules. The processor 205 also typically has one or more interfaces 242 for communicating with external devices via the system bus 204, using connection 218. Memory 234 is coupled to bus 204 using connection 219.

[0056] The application program 233 includes a set of instructions 231 that may include conditional branch and loop instructions. The program 233 may also include data 232 used for the execution of the program 233. Instructions 231 and data 232 are stored in memory locations 228, 229, 230, and 235, 236, and 237, respectively. Depending on the relative sizes of memory locations 228-230 and instruction 231, a particular instruction may be stored in a single memory location, as indicated by the instruction shown in memory location 230. Alternatively, an instruction may be segmented into several parts, each stored in a separate memory location, as indicated by the instruction segments shown in memory locations 228 and 229.

[0057] Generally, the processor 205 is given a set of instructions to be executed within it. The processor 205 awaits subsequent inputs to which it responds by executing another set of instructions. Each input may come from one or more sources, all of which are shown in Figure 2A, including data generated by one or more of the input devices 202, 203, data received from an external source across one of the networks 220, 202, data retrieved from one of the storage devices 206, 209, or data retrieved from a storage medium 225 inserted into the corresponding reader 212. Execution of a set of instructions may, in some cases, result in the output of data. Execution may also involve storing data or variables in memory 234.

[0058] The tensor encoder 116, tensor decoder 146, and the described method can use input variables 254, which are stored in memory 234 in the corresponding memory locations 255, 256, and 257. The tensor encoder 116, tensor decoder 146, and the described method generate output variables 261, which are stored in memory 234 in the corresponding memory locations 262, 263, and 264. Intermediate variables 258 may be stored in memory locations 259, 260, 266, and 267.

[0059] Referring to processor 205 in Figure 2B, registers 244, 245, 246, arithmetic logic unit (ALU) 240, and control unit 239 cooperate to execute the sequence of microoperations necessary to perform a “fetch, decode, and execute” cycle for each instruction in the instruction set that makes up program 233. Each fetch, decode, and execute cycle is: A fetch operation that fetches or reads instruction 231 from memory locations 228, 229, and 230; The control unit 239 performs a decoding operation to determine which instruction has been fetched; and The execution operation in which the control unit 239 and / or ALU240 execute the instruction It is equipped with.

[0060] Subsequently, further fetch, decrypt, and execute cycles may be performed for the next instruction. Similarly, a storage cycle may be performed by the control unit 239 storing or writing values ​​to memory location 232.

[0061] Each step or subprocess in the method described in Figures 18-20 is associated with one or more segments of program 233 and is typically executed by register sections 244, 245, 246, ALU 240, and control unit 239 in processor 205, which cooperate to perform fetch, decode, and execute cycles for all instructions in the instruction set for the mentioned segments of program 233.

[0062] Figure 3A is a schematic block diagram 300 showing a functional module of the CNN backbone portion 310, which may be provided as an implementation of NN Part 1 114 when System 100 is configured to run a “YOLOv3” network. NN Part 1 114 is sometimes called “DarkNet-53,” but different backbones are also possible, resulting in a different number and dimension of layers of tensor 115 for each frame. In one implementation, the backbone portion 310 may be used as a person detector for object tracking.

[0063] As shown in Figure 3A, the video data 113 is passed to the resizer module 304. The resizer module 304 resizes each frame of the video data 113 to a resolution suitable for processing by the CNN backbone 310, generating the resized frame data 312. If the resolution of the video data 113 is already suitable for the CNN backbone 310, the operation of the resizer module 304 is not required. The resized frame data 312 is passed to the Convolutional Batch Normalized Leaky Modified Linear (CBL) module 314, which generates a tensor 316. The CBL module 314 includes modules such as those described with reference to the CBL module 360, as shown in Figure 3D.

[0064] The CBL module 360 ​​takes a tensor 361 of the resized frame data 312 as input. Tensor 361 is passed to a convolutional layer 362 to generate tensor 363. If the convolutional layer 362 has a stride of 1, tensor 363 has the same spatial dimensions as tensor 361. If the convolutional layer 362 has a larger stride, such as 2, tensor 363 has smaller spatial dimensions compared to tensor 361; for example, with a stride of 2, the width and height are halved. Regardless of the stride, the size of the channel dimensions of tensor 363 can vary compared to the channel dimensions of tensor 361 for a given CBL block. Tensor 363 is passed to a batch normalization module 364 that outputs tensor 365. The batch normalization module 364 normalizes the input tensor 363 and applies scaling factors and offset values ​​to generate output tensor 365. The scaling factors and offset values ​​are derived from the training process. Tensor 365 is passed to the leaky modified linear activation ("LeakyReLU") module 366, which generates tensor 367. Module 366 provides a "leaky" activation function, where positive values ​​in the tensor are passed through, and the magnitude of negative values ​​is greatly reduced, for example, to 0.1x their previous values.

[0065] Returning to Figure 3A, tensor 316 is passed from CBL block 314 to residual block module 320, such as the "res1+2+8" module (also called the res11 module), which contains a concatenation of three residual blocks, each containing one residual unit, two residual units, and eight residual units, respectively. The spatial resolution of the tensor is halved horizontally and vertically in each residual block (see Figure 3B) by a convolution with a stride of 2 in CBL block 344.

[0066] The residual block will be explained with reference to ResBlock340 shown in Figure 3B. ResBlock340 receives tensor 341 (e.g., tensor 316). Tensor 341 is zero-padding by the zero-padding module 342 to produce tensor 343. Tensor 343 is passed to the CBL module 344 to produce tensor 345. The CBL module 344 includes a convolution with a stride parameter set to 2 (e.g., similar to 362), resulting in tensor 345 having half the width and half the height of tensor 343. Tensor 345 is passed to the residual unit 346. The residual unit 346 contains a series of concatenated residual units based on the number of residual blocks (e.g., 11 units for block 320). The last residual unit of residual unit 346 outputs tensor 347.

[0067] The residual unit will be explained with reference to the ResUnit350 shown in Figure 3C. ResUnit350 takes tensor 351 (e.g., tensor 345) as input. Tensor 351 is passed to the CBL module 352, which generates tensor 353. Tensor 353 is passed to the second CBL unit 354, which generates tensor 355. The summation module 356 sums tensor 355 and tensor 351 to generate tensor 357. The summation module 356 is sometimes called a "shortcut" because the input tensor 351 effectively influences the output tensor 357. In the case of an untrained network, ResUnit350 operates to pass the tensor as is. Once training is performed, CBL modules 352 and 354 operate to separate tensor 357 from tensor 351 according to the training data and ground truth data.

[0068] Returning to Figure 3A, the Res11 module 320 outputs tensor 322. Tensor 322 is output from the backbone module 310 as one of several layers and also provided to the Res8 module 324. The Res8 module 324 is a residual block (i.e., 340) containing eight residual units (i.e., 350). The Res8 module 324 generates tensor 326. Tensor 326 is passed to the Res4 module 328 and output from the backbone module 310 as one of several layers. The Res4 module is a residual block (i.e., 340) containing four residual units (i.e., 350). The Res4 module 328 generates tensor 329. Tensor 329 is output from the backbone module 310 as one of several layers. Collectively, layer tensors 322, 326, and 329 are output as tensor 115, which may be called layers 0-2, or L0, L1, and L2, respectively. Backbone CNN310 may take a video frame with a resolution of 1088x608 as input and generate three tensors corresponding to three layers, having the following dimensions: [1,256,76,136], [1,512,38,68], [1,1024,19,34]. Another example of three tensors 115 corresponding to three layers is [1,512,34,19], [1,256,68,38], [1,128,136,76], which are separated by layer indices 75, 90, and 105, respectively, when the layers are enumerated according to the YOLOv3 software implementation of backbone 300 and head 1200.

[0069] Res11 320, Res8 324, and Res4 328 each operate similarly to ResBlock340. CBL314, CBL344, and CBL354 each operate similarly to CBL360.

[0070] Figure 4 is a schematic block diagram showing a functional module of an alternative backbone portion 400 of a CNN, which may be provided as an implementation of NN part 1 114 when system 100 is configured to run a “FasterRCNN” or “MaskRCNN” ResNet 101 network. Frame data 113 is input and passes through tensors 409, 413, 417, 421, and 425 to the stem network 408, res2 module 412, res3 module 416, res4 module 420, and res5 module 424, respectively. The backbone portion 400 may be used as part of a general-purpose object detector or for instance segmentation that supports various classes of objects.

[0071] The stem network 408 includes a 7x7 convolution with a stride of 2 and a maximum pooling operation. The res2 module 412, res3 module 416, res4 module 420, and res5 module 424 perform convolution operations such as LeakyReLU activation. Each module 412, 416, 420, and 424 also halves the width and height of the processed tensor via a stride setting of 2. Each of the tensors 413, 417, 421, and 425 is passed to one of the 1x1 transverse convolution modules 446, 444, 442, and 440, respectively. Modules 446, 444, 442, and 440 produce tensors 447, 445, 443, and 441, respectively. Tensor 441 is passed to the 3x3 output convolution module 470, which produces the output tensor P5 471.

[0072] Tensor 441 is also passed to the upsampler module 450 to produce an upsampled tensor 451. The summation module 460 sums tensors 443 and 451 to produce tensor 461. Tensor 461 is passed to the upsampler module 452 and the 3x3 transverse convolution module 472. Module 472 outputs the P4 tensor 473. The upsampler module 452 produces an upsampled tensor 453. The summation module 462 sums tensors 445 and 453 to produce tensor 463. Tensor 463 is passed to the 3x3 transverse convolution module 474 and the upsampler module 454. Module 474 outputs the P3 tensor 475. The upsampler module 454 outputs the upsampled tensor 455. The summing module 464 sums tensors 447 and 455 to produce tensor 465, which is passed to the 3x3 transverse convolution module 476. Module 476 outputs the P2 tensor 477. The upsampler modules 450, 452, and 454 use nearest neighbor interpolation for low computational complexity. Tensors 471, 473, 475, and 477 form the output tensor 115 of the CNN backbone 400 (NN part 1 114). Figure 4 shows a specific backbone portion ("P-layer split point") of the Faster RCNN network architecture, although different splits into backbone and head are possible. Splitting the network at tensor 409 is called the "stem" split point. Splitting the network at tensors 447, 445, 443, and 441 is called the "C-layer" split point.

[0073] Figure 5 is a schematic block diagram 500 of a tensor encoder 116 using configurable tensor compression steps. Figure 18 shows a method 1800 for executing NN part 1, encoding the topology and weight instructions of NN part 1, and encoding the resulting compressed tensor together with the compressed video. Method 1800 may be implemented as one or more software application programs 233 executable within a computer system 200. Method 1800 may also be performed by instructions 231 (see Figure 2B) within software 233 executed within the computer system 200. The software instructions 231 may be formed as one or more code modules, each performing one or more specific tasks. Method 1800 begins with a video frame encoding step 1810.

[0074] In step 1810, the video encoder 150 encodes the frame data 113 to generate a video bitstream 151 under the execution of the processor 205. Control in the processor 205 proceeds from step 1810 to the neural network first part execution step 1820.

[0075] In step 1820, NN part 1 114 executes the first part of the neural network using frame data 113 from video source 112 as input, under the execution of processor 205, and generates a tensor 115 as output. Control in processor 205 proceeds from step 1820 to step 1830, which executes the second part of the neural network.

[0076] In step 1830, the proxy NN part 2 118 executes the second part of the neural network using the tensor 115 as input, under the execution of the processor 205. As a result of steps 1820 and 1830, an initial, i.e., "generic," or "proxy" task result 119 is produced. The proxy task result 119 can be thought of as a "low-quality" or "approximate" version of the task result that is desired to be produced by the system 100. Task result 119 is preferably a more tolerant version than the task result produced by the destination device 140. For example, task result 119 may detect any person or vehicle, whereas task result 151 may detect a specific person or vehicle. Control in the processor 205 proceeds from step 1830 to step 1840, which is the NN, NN part 1, and NN part 2 instruction decision step.

[0077] In step 1840, the source device 110, under the execution of the processor 205, determines summary values ​​that represent or identify the topology of NN part 1 114, NN part 2 118, and the combined topology of NN part 1 114 and NN part 2 118 as if they were run as a single integrated network. The network topology can be materialized in various ways, such as Python source code that executes functions in libraries such as "Pytorch," "Keras," "Caffe," or "Tensorflow." The network topology may also be represented in a compiled language such as C or C++. There are device-independent representations of CNNs that include both topology and network weights in the representation, such as ONNX. Device-independent representations like ONNX may be used in a refactored form in which arbitrary node names are replaced with an enumeration of node names. Thus, the refactored ONNX representation is immutable to insignificant variations, such as those resulting from variable naming.

[0078] A method for determining topology and weight directives using a network representation, as performed in step 1840, is illustrated with reference to Figure 20. Once a network is selected, the network topology and weight directives may be determined by the user prior to the execution of the selected network, rather than being calculated "at runtime," and retrieved from a repository. Additionally, network topology and network weight directives may be calculated or recalled for subsets of the network. If the network is divided into NN part 1 and NN part 2, separate network topology and network weight directives are determined, and the association between the two is maintained. The directives may be some summarized value, such as a hash or other algorithm for converting information into a summarized value, which can be included in the metadata of the multi-layer bitstream 123 without introducing excessive overhead. An example is to apply md5sum, sha1sum, and sha256 to a byte stream of network topology or network weights to generate the corresponding directive or hash. Hash algorithms such as md5sum, sha1sum, and sha256 have a relatively low probability of "collision," where two different hash inputs, such as network topology or network weights, produce the same hash value. These instructions are not recalculated each time they are referenced, but are retained for future reference. Instructions generated from different parts of the same network are considered "associated." The association may also be in the form of a lookup table indicating that if a tensor is decoded from a multi-layer bitstream 123 generated from NN part 1 114 with a particular network topology instruction and network weight instruction, a compatible NN part 2 (if available at the destination device 140) will be selected. If no association (lookup table entry) is found, the destination device 140 cannot further process the decoded tensor from the multi-layer bitstream 123.In step 1840, the network topology instructions and network weight instructions for NN part 1 114 and NN part 2 118, as well as the combined NN parts 1 and 2 (i.e., treating 114 and 118 as an undivided network), are determined from their respective topologies or obtained from pre-calculated results.

[0079] In essence, the “instructions” provide a way to uniquely identify a particular network or a portion of it without needing to refer to an external enumeration of network topologies. Other representations of network topologies can also be used, such as Python code using Pytorch library functions, or C or C++ code calling library functions such as the “CUDA” library functions for Nvidia GPUs or the “ROCm” device library for AMD GPUs. However, such representations require more advanced refactoring to eliminate elements that do not affect the actual network structure, such as removing code comments or whitespace. Intermediate representations, such as Python bytecode, can also be used, but such forms may change based on the development of their respective languages ​​and therefore do not form a stable representation suitable for reproducible hashes by users using different versions of the language.

[0080] The configuration may use the ONNX intermediate representation (IR) of the network as input for hashing, and multiple hashes can be associated with each other by including the ONNX IR version as part of the instruction along with the hash. Hashes represent the same network topology and / or weights, but their values ​​will differ due to updates to the ONNX protocol that affect the byte stream representation of the network topology and / or weights. Alternative ONNX formats for representing network topology and / or weights, such as "TensorRT" or "OpenVino," may also be used to generate network topology and / or weight instructions. An exemplary method for generating such instructions is illustrated with reference to Figure 20. Control in processor 205 proceeds from step 1840 to tensor compression execution step 1850.

[0081] In step 1850, the tensor encoder 116 operates to compress the tensor 115. Figure 5 shows an exemplary implementation 500 of the tensor encoder 116. In the example in Figure 5, in step 1850, the compressor 530 compresses the tensor 115 under the execution of the processor 205 to produce a compressed tensor 532. The compressed tensor 532 has the same or fewer numbers as the tensor 115 and is dimensionally reduced (i.e., the number of channels and either or both of the width and height of the feature maps are reduced). The tensor compressor may perform dimensionality reduction using a set of trained network layers as described with reference to Figure 6, or using a method based on a low-rank approximation of the tensor 115 as described with reference to Figure 7. Control in the processor 205 leads from step 1850 to the tensor information coding step 1860.

[0082] In step 1860, the metadata encoder 544 (see Figure 5) encodes tensor information 1195 (see Figure 11) that specifies the dimensions of the compressed tensor 532 and the placement of the feature maps of each tensor in the compressed tensor 532 into the bitstream 121. A “feature frame” is a video frame containing feature maps, or other values ​​that represent feature maps, such as basis vectors or coefficients. The tensor information 1195 includes the maximum number of channels and the number of channels used for each tensor. The frame area of ​​the region must be sufficient to pack the tensors within that region up to the maximum number of channels, i.e., the maximum number of feature maps. The maximum number of channels must not change for frames in the bitstream 121 associated with the same SPS1110b. The tensor compressor 530 may generate fewer feature maps than the maximum number, and the number of channels used signals the number of channels actually used, which may vary from frame to frame. The video frame 540 may be divided into regions, such as rectangular areas, each region used to hold feature maps from one or more tensors of the compressed tensor 532. In step 1860, the metadata encoder 544 also encodes the output of NN part 1 114, i.e., the dimensions of tensor 115, as NN part 1 output instruction 11102. Control in the processor 205 proceeds from step 1860 to tensor quantization step 1870.

[0083] In step 1870, referring to Figure 5, the quantizer module 534 of the encoder 116, under the execution of the processor 205, quantizes the floating-point values ​​in each tensor of the compressed tensor 532 to produce a quantized compressed tensor 536. The quantized compressed tensor 536 has integer values ​​that occupy a range within the sample range defined by the operating bit depth of the video encoder 542. For example, when encoding video using 10-bit samples, integer values ​​of interval[0,1023] are allowed. For each tensor, the minimum and maximum floating-point values ​​form the quantization range 526. Control in the processor 205 proceeds from step 1870 to the quantization range coding step 1880.

[0084] In step 1880, the metadata encoder 544 (Figure 5), under the execution of processor 205, encodes the quantization range 526 for each tensor of the compressed tensor 532 as a quantization range 1196 into a multilayer bitstream 123. The quantization range is stored in the multilayer bitstream 123 to enable inverse quantization to restore the range of floating-point values ​​generated by NN part 1 114 at the destination device 140. Control in processor 205 proceeds from step 1880 to tensor pack step 1890.

[0085] In step 1890, the packer module 538, under the execution of processor 205, packs the feature maps of each tensor of the compressed quantized tensor 536 into a video frame 540. The tensor feature maps are generally packed from left to right, then from top to bottom, into the region in the feature frame 540 to which the tensor is assigned. The region is defined by the FCVCM decoder network instruction SEI message 1113, but the region may correspond to a “coding layer” structure (i.e., beneath metadata such as SEI message 1113) such as a subpicture, slice, or tile. SEI message 1113 is encoded as a “prefix SEI” message, i.e., encoded to precede the access unit, so that the information contained therein is available as soon as each layer in the access unit is decoded. The coding-level structure of the frame may not be accessible from outside the encoder (such as encoder 800 in Figure 8) and may vary depending on the compression standard used, such as VVC, HEVC, or AV-1. By individually signaling regions in SEI message 1113, dependence on specific features in the coding layer is eliminated. Each region must be large enough to hold all feature maps of the tensor to be packed into its respective region. Examples of region-based packing formats are illustrated with reference to Figures 9A and 9B and 10A and 10B. Control in processor 205 proceeds from step 1890 to feature frame coding step 18100.

[0086] In step 18100, the video encoder 542 compresses the feature frame 540 under the execution of the processor 205 to generate the extended layer of the multilayer bitstream 123. The operation of the video encoder 542 is described with reference to Figure 8. The video encoder 542 is used to encode the extended (feature) layer of the multilayer bitstream 123. A separate instance of the video encoder 800 acts as the video encoder 150 and is used to encode the primary (video) layer of the multilayer bitstream 123. Control in the processor 205 proceeds from step 18100 to the proxy NN and NN part 1 instruction encoding step 18110.

[0087] In step 18110, the metadata encoder 544 encodes the NN and NN Part 1 instructions determined in step 1840 into a multi-layer bitstream 123 under the execution of the processor 205. For NN and NN Part 1, separate instructions or hashes are encoded for the network topology and network weights. Hashes generated by hash algorithms such as sha1sum and sha256 are typically 256-bit codes coded using fixed-length codewords. Particularly for certain widely used networks and widely used weight files, the NN network topology and network weight instructions allow the decoder to identify such widely used networks and which weights were used in those networks. The partition points used in source device 110 are identified via the NN Part 1 network topology instructions. By using instructions derived from the structure, it is not necessary to explicitly enumerate various partition points and update such enumerations every time a new network is exposed or a new partition point is used. Using structure-based instructions also avoids the need for different vendors of source device 110 and destination device 140 to agree on enumeration, i.e., a globally recognized repository of network architectures is not required. Control in processor 205 proceeds from step 18110 to proxy NN part 2 task result encoding step 18120.

[0088] In step 18120, the metadata encoder 544 encodes the proxy NN part 2 task result 119 into a multi-layer bitstream 123 under the execution of the processor 205. The result 119 may be one or more bounding boxes or image classifications and represents the result of proxy NN part 2 118 performed on the source device 110.

[0089] Method 1800 ends, and processing in processor 205 proceeds to the next instance 113 of the source data (for example, the next frame from video source 112).

[0090] Figure 6 is a schematic block diagram 600 showing one type of multiscale feature fusion (MSFF) module that can function as a tensor compressor 530. The MSFF module 600 takes a tensor 115 and produces a compressed tensor 532 with reduced dimensionality compared to tensor 115, resulting in a reduced bitrate when encoded as part of a packed frame. The MSFF module 600 uses a trained network layer and requires a module corresponding to a tensor decoder 146 to restore the tensor's dimensionality so that tensor 149 can be fed to the CNN head 150. The MSFF module 600 takes four tensors as input and requires each tensor to have 256 channels so that the MSFF module 600 is compatible with the P layer of a FasterRCNN or MaskRCNN network. However, variations of the MSFF module 600 are possible that are compatible with different numbers of layers and different numbers of channels.

[0091] The MSFF module 600 has 64 channels as output and generates a single tensor 532 with a feature map size corresponding to the P5 layer found in the input, although variations with different numbers of channels are also possible. Each variation of the MSFF module 600 requires the use of different weights for proper operation. If several variations of the MSFF module 600 are available in system 100 for a given network, the packing format may be set to the worst-case feature map number of the compressed tensor of the decoder network topology currently in use, and the actual number of channels used may be updated at runtime, i.e., during processing of the frame data 113, as part of the tensor information 1195.

[0092] The MSFC module 600 includes an MSFF block 610, shown in Figure 6, which uses one or more downsampling filters to generate a single tensor 629 from multiple tensors 115. Under the execution of the processor 205, the MSFF block 610 combines each tensor from the first set of tensors (i.e., 602, 603, 604, 605) to generate a combined tensor 629. The combined tensor 629 forms a representation of the FPN layer tensor. Downsampling modules 622a, 622b, and 622c operate on tensors with larger spatial scales, namely P4 604 of (2h,2w,256), P3 603 of (4h,4w,256), and P2 602 of (8h,8w,256), respectively. Modules 622a, 622b, and 622c perform downsampling to match the spatial scale of the smallest tensor, i.e., P5 605 of (h,w,256), producing downscaled P5 tensors 623a, 623b, and 623c, respectively. The concatenation module 624 performs channel-direction concatenation of tensors 605, 623a, 623b, and 623c, producing a concatenated tensor 625 of dimension (h,w,1024). The concatenated tensor 625 is passed to the aperture and excitation (SE) module 626 to produce tensor 627. The SE module 626 sequentially performs global pooling, a fully connected layer with reduced channel count, a rectifier linear unit activation unit, a second fully connected layer to restore the channel count, and a sigmoid activation function to produce a scaling tensor. Tensor 625 is scaled according to the scaling tensor to produce the output as tensor 627. SE block 626 can be trained to adaptively change the weighting of different channels in the tensor it passes through, based on the output of the first fully connected layer.

[0093] The output of the first fully connected layer reduces each feature map of each channel to a single value. Each single value passes through a nonlinear activation unit (ReLU) to create a conditional representation of the single value suitable for weighting the other channels, along with its reconstruction to the total number of channels performed by the second fully connected layer. Thus, the SE block 626 is able to extract nonlinear inter-channel correlations to a greater extent than possible with purely convolutional (linear) layers when generating tensor 627 from tensor 625. Tensor 627 is passed to the convolutional layer 628. The convolutional layer 628 performs one or more convolutional layers to generate a connected tensor 629 with the number of channels reduced to F channels, typically 256 channels (i.e., F=256). Further reduction in the number of channels is achieved by the single-scale feature compression (SSFC) module 650.

[0094] The SSFC module 650 receives tensor 629 and applies convolution 652 to reduce the number of channels from F (256) to C' (nominally set to 64 channels), generating tensor 653. Next, tensor 653 is passed to batch normalization module 654 to generate batch normalized tensor 655, which is then passed to hyperbolic tangent activation layer 656 to generate compressed tensor 532. The output of MSFC module 600 is one tensor per frame, with a fixed feature map size and a fixed number of channels.

[0095] Figure 7 is a schematic block diagram showing an exemplary inter-channel uncorrelatedness-based tensor compressor 700 that can function as tensor compressor 530. Tensor compressor 700 operates without requiring trained layers and relies on extracting inter-channel redundancy at runtime (i.e., using input tensor data from the CNN backbone 114, rather than relying on pre-trained weights). Tensor compressor 700 outputs three types of data: mean, basis vectors, and coefficients. Tensor compressor 700 compresses one tensor, and multiple instances of tensor compressor 700 may be instantiated as tensor compressor 530 to compress multiple tensors. If multiple instances of tensor compressor 700 are instantiated, the same types of tensors from each instance may be packed into the same region. For example, the basis vectors from each instance may be packed into a single region. These types of data are held in the tensor and packed into three separate regions within the frame. The number of basis vectors used may change during frame processing, and the number of coefficients used may also change, but the number of coefficients must not exceed the number of basis vectors. Therefore, the dimensions of the basis vectors and / or coefficients may be updated from time to time in order to signal the decoder the contents of the decoded and packed frame. As shown in Figure 7, the channel mean extraction module 710, under the execution of processor 205, performs an averaging operation across spatial dimensions (i.e., width and height) on the tensors of tensor 115 to produce a channel mean 712 as a 1D (one-dimensional) tensor of C (number of channels in the tensors of tensor 115). The channel mean extraction module 710 may operate less frequently than for each tensor 115 it receives, in which case the channel mean 712 is updated only for a particular frame. A packed frame containing the updated channel mean 712 may be indicated by the non-zero number of channels in tensor 712 via tensor information 1195, or by a separate tensor update flag included in SEI message 1113.

[0096] For each channel of the tensor 115, a DC shift is performed by the subtraction module 714. The subtraction module 714 is constant with respect to the feature map and subtracts the value obtained from the mean channel 712 from each spatial position in the feature map. The value to be subtracted is the respective value in the reconstructed mean channel 712. As a result of the subtraction module 714, a zero-centered tensor 716 is output, from which the DC component found in each feature map has been removed.

[0097] The decomposition module 718 operates under the execution of processor 205 to generate a set of basis vectors 720 for the tensor in tensor 115. The decomposition module 718 takes a zero-centered tensor 716 as input and generates the set of basis vectors 720 by performing a principal component analysis (PCA) technique such as singular value decomposition (SVD). Since one basis vector maps all channels to a single value using the dot product operation, if the tensor in tensor 115 has 256 channels, one basis vector has dimensions of 256 x 1. If the decomposition module 718 generates the first N basis vectors, e.g., 25, the resulting basis vectors have dimensions of 256 x N or 256 x 25. Because the basis vectors are relative to the origin, a zero-mean tensor 712 is used to ensure that orthogonal basis vectors are found. Each basis vector is a vector that associates all channels with a reduced set of channels. Therefore, basis vectors collectively enable the representation of tensor data across all channels into a smaller set of basis vectors. Each basis vector is derived by considering all samples in each feature map of a given channel. Vector 720 contains fewer basis vectors than the number of channels in the tensor in tensor 115, which corresponds to a reduction in the dimensionality of the tensor in tensor 115. Basis vector 720 represents the tensor in tensor 115 in a subspace that explains, i.e., "explains," the maximum amount of variance in the tensor in tensor 115 for the number of components in basis vector 720. The basis vectors are ordered from the vector with the largest explained variance to the vector with the smallest explained variance. In other words, basis vector 720 enables the representation of the tensor in tensor 115 with minimal loss of quality for a given number of components, whose components are the first N ranked basis vectors.

[0098] As shown in Figure 7, the dot product module 722, under the execution of processor 205, performs the dot product of each channel of tensor 716 with each basis vector 720 to generate the coefficient tensor 724. The coefficients have the same width and height as the tensor 115, but the number of channels in tensor 724 corresponds at most to the number of components (or basis vectors) generated by decomposition module 718, which is less than the number of channels in the tensor 115. The tensor compressor 700 may vary the number of channels in coefficient tensor 716 for a given frame. The number of channels present in coefficient tensor 716 for a given frame is contained in tensor information 1195, coded in an instance of SEI message 1113 associated with the given frame. Each value in coefficient tensor 724 represents the contribution of each basis vector in reproducing each value in each feature map. Tensors 724, 720, and 712 provide the compressed tensor 532.

[0099] Figure 8 is a schematic block diagram showing the functional modules of the video encoder 800. The video encoder 800 provides exemplary implementations of video encoders 542 and 150, respectively. The video encoder 800 may be implemented using a general-purpose computer system 200 shown in Figures 2A and 2B, where various functional modules may be implemented by dedicated hardware within the computer system 200, or by software executable within the computer system 200, such as one or more software code modules of a software application program 233 residing on a hard disk drive 205 and whose execution is controlled by a processor 205. Alternatively, the video encoder 800 may be implemented by a combination of dedicated hardware and software executable within the computer system 200. The video encoder 800 and the method described may, alternatively, be implemented in dedicated hardware, such as one or more integrated circuits performing the functions or subfunctions of the method described. Such dedicated hardware may include a graphics processing unit (GPU), a digital signal processor (DSP), an application-specific standard product (ASSP), an application-specific integrated circuit (ASIC), an FPGA, or one or more microprocessors and associated memory. In particular, the video encoder 800 comprises modules 810-890, each of which may be implemented as one or more software code modules of the software application program 233.

[0100] The video encoder 542 in Figure 8 is an example of a versatile video coding (VVC) video encoder, but other video codecs may be used to perform the processing steps described herein. For example, HEVC may be used. The example described generates a bitstream of encoded data. When other codecs are used, some implementations may pack the data into a different format, such as a frame format. The video encoder 800 receives frame data 802, each frame containing one or more color channels. The frame data 802 may be any chroma format and bit depth supported by the profile in use, for example, for the "Main10" profile of the VVC standard, it is 4:0:0, 4:2:0, with a sample precision of 8 to 10 bits. The frame data my corresponds to the video data 113 of tensor 540. The frame data 802 is also characterized by "levels" that specify aspects such as the maximum chroma sample rate, frame aspect ratio constraints, slice and tile count limits. The worst-case compression bitrate is also indicated by a combination of "level" and "tier" parameters.

[0101] As shown in Figure 8, the block partitioner 810 first divides the frame data 802 into CTUs configured to be roughly square in shape and to use a specific size for each CTU. The maximum effective size of a CTU can be 32x32, 64x64, or 128x128 lumens, for example, as defined by the "sps_log2_ctu_size_minus5" syntax element present in the "sequence parameter set". The CTU size also provides the maximum CU size, as a CTU without further division contains one CU. The block partitioner 810 further divides each CTU into one or more CBs according to the lumens coding tree and chroma coding tree. Lumen channels are sometimes called primary color channels. Each chroma channel is sometimes called a secondary color channel. CBs can be of various sizes and can include both square and non-square aspect ratios. However, in the VVC standard, CBs, CUs, PUs, and TUs always have side lengths that are powers of 2. Therefore, the current CB, represented as 812, is output from block partitioner 810, which proceeds according to the lumacoding tree and chromacoding tree of the CTU, following iterations across one or more blocks of the CTU.

[0102] The CTUs obtained from the initial split of frame data 802 may be scanned in raster scan order and grouped into one or more “slice.” A slice may be an “intra” (or “I”) slice. An intra slice (I slice) indicates that all CUs in the slice are intra-predicted. Generally, the first picture in a coding layer video sequence (CLVS) contains only I slices and is called an “intra picture.” A CLVS may contain periodic intra pictures that form “random access points” (i.e., intermediate frames in the video sequence from which decoding may begin). Alternatively, a slice may be a single-predict or a bi-predict (a “P” or “B” slice, respectively), indicating the availability of additional single-predict and bi-predictions within the slice.

[0103] The video encoder 800 encodes a sequence of pictures according to a picture structure. One picture structure is "low latency," in which a picture using interpretation can only reference pictures that have occurred previously in the sequence. Low latency allows each picture to be output as soon as it is decoded, in addition to being stored for possible references by subsequent pictures. Another picture structure is "random access," where the coding order of pictures is different from the display order. Random access allows an interpretation-predicted picture to reference other pictures that have been decoded but have not yet been output. A degree of picture buffering is required so that future reference pictures exist in the decoded picture buffer in terms of display order, resulting in a delay of several frames.

[0104] If a chroma format other than 4:0:0 is used, in an I-slice, the coding tree of each CTU may diverge below the 64x64 level into two separate coding trees, one for luma and the other for chroma. The use of separate trees allows for different block structures to exist between luma and chroma within the luma 64x64 area of ​​the CTU. For example, a large chroma CB may colcate with many smaller luma CBs, and vice versa. In a P or B-slice, a single coding tree of a CTU defines a block structure common to luma and chroma. The blocks resulting from a single tree can be intra-predicted or inter-predicted.

[0105] In addition to dividing a picture into slices, a picture can also be divided into "tiles." A tile is a sequence of CTUs covering a rectangular area of ​​a picture. CTU scanning is performed in a raster scanning manner within each tile, moving from one tile to the next. A slice can be either an integer number of tiles, or an integer number of consecutive rows of CTUs within a given tile.

[0106] For each CTU, the video encoder 800 shown in Figure 8 operates in two stages. In the first stage (called the “search” stage), the block partitioner 810 tests various potential configurations of the coding tree. Each potential configuration of the coding tree has an associated “candidate” CB. The first stage involves testing various candidate CBs to select a CB that provides relatively high compression efficiency with relatively low distortion. The tests generally involve Lagrangian optimization, thereby evaluating candidate CBs based on a weighted combination of rate (i.e., coding cost) and distortion (i.e., error relative to the input frame data 712). The “best” candidate CB (i.e., the CB with the lowest evaluated rate / distortion) is selected for subsequent encoding to the bitstream portion 816. The evaluation of candidate CBs includes the option of using the CB for a given area, or further dividing the area according to various partitioning options and coding each of the smaller resulting areas with further CBs, or further dividing the area. As a result, both the coding tree and the CBs themselves are selected in the search stage.

[0107] The video encoder 800 generates a prediction block (PB) indicated by arrow 820 for each CB, e.g., CB812. PB820 is a prediction of the contents of the associated CB812. The subtractor module 822 generates a difference (or "residual," meaning the difference is in the spatial domain) between PB820 and CB812, indicated as 824. Difference 824 is the block size difference between the corresponding samples in PB820 and CB812. Difference 824 is transformed and quantized and represented as a transformed block (TB) indicated by arrow 836. PB820 and its associated TB836 are typically selected from many possible candidate CBs, for example, based on an evaluated cost or distortion.

[0108] The candidate coding block (CB) is the CB obtained from one of the prediction modes available to the video encoder 800 for the associated PB and the resulting residual. When combined with the predicted PB in the video encoder 800, the TB836 reduces the difference between the decoded CB and the original CB812 at the expense of additional signaling in the bitstream.

[0109] Each candidate coding block (CB) (i.e., a prediction block (PB) combined with a transformation block (TB)) has an associated coding cost (or "rate") and an associated difference (or "distortion"). The distortion of a CB is typically estimated as a difference of sample values, such as an absolute difference sum (SAD), a squared difference sum (SSD), or an Hadamard transform applied to the difference. The estimate obtained from each candidate PB may be determined by a mode selector 886 using the difference 824 to determine the prediction mode 887. The prediction mode 887 indicates a decision to use a particular prediction mode for the current CB, such as intra-frame prediction or inter-frame prediction. Estimating the coding cost associated with each candidate prediction mode and the corresponding residual coding can be performed at a significantly lower cost than the entropy coding of the residuals. Therefore, even in a real-time video encoder, a large number of candidate modes may be evaluated to determine the optimal mode in terms of rate distortion.

[0110] Determining a preferred mode from the standpoint of rate distortion is typically achieved using a variation of Lagrangian optimization. Lagrangian or similar optimization processes can be employed for both selecting a preferred partition of the CTU into CBs (by block partitioner 810) and selecting a predictive mode from multiple possibilities. Through the application of Lagrangian optimization of candidate modes in the mode selector module 886, the intra-predictive mode with the lowest cost measurement is selected as the "best" mode. The lowest-cost mode includes the selected quadratic transformation index 888, which is also encoded in bitstream 816 by entropy encoder 838.

[0111] In the second stage of operation of the video encoder 800 (called the "coding" stage), the video encoder 800 performs iterations across the determined coding tree for each CTU. For CTUs using separate trees, for each 64x64 luma region of the CTU, the luma coding tree is encoded first, followed by the chroma coding tree. Within the luma coding tree, only luma CBs are encoded, and within the chroma coding tree, only chroma CBs are encoded. For CTUs using a shared tree, a single tree describes the CUs (i.e., luma CBs and chroma CBs) according to the common block structure of the shared tree.

[0112] The entropy encoder 838 supports bitwise coding of syntax elements using variable-length and fixed-length codewords, as well as arithmetic coding modes for syntax elements. Parts of the bitstream, such as "parameter sets," e.g., sequence parameter sets (SPS), picture parameter sets (PPS), and picture headers (PH), use combinations of fixed-length and variable-length codewords. Slices, also called contiguous parts, have a slice header using variable-length coding, followed by slice data using arithmetic coding. The slice header may define parameters specific to the current slice, such as slice-level quantization parameter offsets, and may contain instances of PHs. The slice data contains syntax elements for each CTU within the slice. The use of variable-length and arithmetic coding requires sequential analysis within each part of the bitstream. Each part may be delimited by a start code to form a "network abstraction layer unit" or "NAL unit." Arithmetic coding is supported using context-adaptive binary arithmetic coding processing.

[0113] Arithmetic-coded syntax elements consist of a sequence of one or more "bins." A bin, like a bit, has a value of either "0" or "1." However, bins are not encoded as discrete bits in bitstream portion 816. Each bin has an associated predicted (or "most likely" or "highest probability") value and an associated probability, known as a "context." If the actual bin being coded matches the predicted value, a "highest probability symbol" (MPS) is coded. Coding a highest probability symbol is relatively inexpensive in terms of the bits consumed in bitstream portion 816, costing less than one discrete bit. If the actual bin being coded does not match the most likely value, a "lower probability symbol" (LPS) is coded. Coding a lower probability symbol is relatively expensive in terms of the bits consumed. Bin coding techniques enable efficient coding of bins with a bias towards "0" versus "1" probabilities. For syntax elements with two possible values ​​(i.e., "flags"), a single bin is sufficient. For syntax elements with many possible values, a sequence of bins is required. The rules for converting the values ​​of syntax elements into a sequence of bins are called "binarization." When the values ​​of "0" and "1" in a bin are likely to be equal (or nearly equal), it is possible to omit the use of context and assume an equal probability distribution. Bins with context are called "context coding bins," and bins without context are called "bypass coding bins." Binarization of one or more bins of a syntax element may result in a combination of context coding bins and bypass coding bins. Unlike directly coding a single bit into a bitstream, bypass coding bins use an arithmetic coding engine, which facilitates the mixing of context coding bins and bypass coding bins in the binarization of syntax elements.

[0114] For a given binarid representation, the existence of subsequent bins in a sequence may be determined based on the values ​​of preceding bins in the sequence, resulting in a variable-length binarid representation. Furthermore, each bin may be associated with multiple contexts, and one context is selected to be used for coding an instance of a particular bin. The selection of a particular context may depend on previous bins within the syntax element, the decoded values ​​of adjacent syntax elements (i.e., those from adjacent blocks), and so on. Each time a context coding bin is coded, the context selected for that bin (if any) is updated in a way that reflects the value of the new bin. For this reason, the binary arithmetic coding scheme is said to be adaptive.

[0115] Bypass coding bins are used when the distribution of values ​​in a particular bin is not biased, as the lack of context in bypass coding bins saves memory and reduces complexity. An example of an entropy coder that employs context and adaptation is known in the art as CABAC (Context Adaptive Binary Arithmetic Coder), and many variations of this coder are used in video coding.

[0116] The QP controller 890 determines the quantization parameter 892, which is used to establish the quantization step size used by the quantizer 834 and the inverse quantizer 840. A larger quantization step size results in the first-order transformation coefficient 828 being quantized to a smaller value, which reduces the bitrate of the bitstream portion 816 at the expense of the fidelity of the inverse transformation coefficient 846.

[0117] The entropy encoder 838 encodes the quantization parameter 892 and the LFNST index 888, if used in the current CB, using a combination of context coding bins and bypass coding bins. The quantization parameter 892 is encoded at the beginning of each slice, and changes to the quantization parameter 892 within a slice are coded using the "Delta QP" syntax element. The Delta QP syntax element is signaled at most once in each area known as a "quantization group". The quantization parameter 892 is applied to the residual coefficients of the Luma CB. The adjusted quantization parameter is applied to the residual coefficients of the collated Chroma CB. The adjusted quantization parameter may include mappings from the Luma quantization parameter 892 according to CU level offsets selected from a list of mapping tables and offsets. The quadratic transformation index 888 is signaled if the residuals associated with a transformation block contain significant residual coefficients only at coefficient positions that undergo a transformation to linear coefficients by the application of the quadratic transformation.

[0118] The residual coefficients for each TB associated with a CB are coded using residual syntax. Residual syntax is designed to efficiently encode small coefficients by primarily using arithmetic coding bins to indicate the significance of coefficients, and by reserving bypass bins for larger residual coefficients along with smaller magnitudes. Therefore, residual blocks with very small magnitudes and sparsely distributed significance coefficients are efficiently compressed. Furthermore, two residual coding schemes exist. The normal residual coding scheme is optimized for TBs where significant coefficients are primarily located in the upper-left corner of the TB, as seen when a transformation is applied. The transformation-skip residual coding scheme is available for TBs where no transformation is performed and can efficiently encode residual coefficients regardless of their distribution across the TB.

[0119] The multiplexer module 884 outputs the PB820 from the intra-frame prediction module 864 according to the determined best intra-prediction mode selected from the prediction modes of each candidate CB tested. The candidate prediction modes do not need to include all possible prediction modes supported by the video encoder 800. Intra-predictions are classified into three types: firstly, “DC intra-prediction,” which involves substituting a single value representing the mean of nearby reconstructed samples into the PB; secondly, “planar intra-prediction,” which involves substituting samples into the PB according to a plane having a DC offset as well as vertical and horizontal slopes derived from nearby reconstructed neighboring samples. Nearby reconstructed samples typically include a row of reconstructed samples above the current PB, extending to some extent to the right of the PB, and a column of reconstructed samples to the left of the current PB, extending to some extent below the PB; and thirdly, “angle intra-prediction,” which involves substituting reconstructed neighboring samples into the PB that have been filtered and propagated across the PB in a specific direction (or “angle”). VVC supports 65 angles, and rectangular blocks can generate a total of 87 angles by utilizing additional angles not available to square blocks.

[0120] A fourth type of intra-prediction is available for chroma PB, where PB is generated from collated luma-reconstructed samples according to a "cross-component linear model" (CCLM) mode. Three different CCLM modes are available, each using a different model derived from adjacent luma and chroma samples. The derived models are used to generate blocks of samples for the chroma PB from collated luma samples. Luma blocks may be intra-predicted using matrix multiplication of a reference sample with one matrix selected from a predefined set of matrices. This matrix intra-prediction (MIP) achieves gain by using a matrix trained on a large set of video data, where the matrix represents a relationship between the predicted block and the reference sample that is not readily captured in angular, plane, or DC intra-prediction modes.

[0121] Module 864 may also generate prediction units by copying blocks from near the current frame using the “intrablock copy” (IBC) method. The location of the reference block is limited to an area corresponding to one CTU divided into 64x64 regions known as VPDUs, which cover the processed VPDU of the current CTU and the VPDUs of previous CTUs within each slice or tile and each row or CTU, up to an area limit corresponding to one 128x128 luma sample, regardless of the CTU size configured for the bitstream. This area is known as the “IBC virtual buffer” and limits the IBC reference area, thereby limiting the required storage. The IBC buffer is populated with reconstructed samples 854 (i.e., those before loop filtering), thus requiring a separate buffer from the frame buffer 872. If the CTU size is 128x128, the virtual buffer contains only samples from CTUs adjacent to and to the left of the current CTU. If the CTU size is 32x32 or 64x64, the virtual buffer contains up to four or sixteen CTUs to the left of the current CTU. Regardless of the CTU size, access to adjacent CTUs for sampling IBC reference blocks is constrained by boundaries such as picture, slice, or tile edges. Using CTU sizes such as 32x32 or 64x64 provides a more tailored reference area to cover a set of previous feature maps, especially for feature maps in low-dimensional FPN layers. If the feature map placement is based on SAD, SSE, or other difference metrics, access to similar feature maps for IBC prediction provides coding efficiency advantages.

[0122] The residuals of predicted blocks when encoding feature map data differ from those found in natural video. Natural video is typically screen content captured by an image sensor or commonly found in operating system user interfaces. Feature map residuals tend to contain a lot of detail. The level of detail in feature map residuals is more suitable for transform skip coding than the predominantly low-frequency coefficients of various transforms. Intra-predicted luma coding blocks may be divided vertically or horizontally into sets of equally sized predicted blocks, each block having an area of ​​at least 16 luma samples.

[0123] If previously reconstructed adjacent samples are unavailable, for example, at the edge of the frame, a default halftone value of half the sample range is used. For example, in the case of 10-bit video, a value of 512 is used. For the CB located at the top left of the frame, since no previous samples are available, the angle and plane intra-prediction modes produce the same output as the DC prediction mode (i.e., a flat plane of samples with halftone values ​​as magnitude).

[0124] In interframe prediction, the motion compensation module 880 generates a prediction block 882 using samples from one or two frames preceding the current frame in the coding order frames within the bitstream, and the multiplexer module 884 outputs it as PB820. Furthermore, in interframe prediction, a single coding tree is typically used for both the lumens and chroma channels. The coding order of frames in the bitstream may differ from the order of frames when captured or displayed. If one frame is used for prediction, the block is said to be a "single prediction" and has one associated motion vector. If two frames are used for prediction, the block is said to be a "double prediction" and has two associated motion vectors. In the case of a P slice, each CU can be intra-predicted or single-predicted. In the case of a B slice, each CU can be intra-predicted, single-predicted, or double-predicted.

[0125] Frames are typically coded using a "group of pictures" structure, enabling a temporal hierarchy of frames. A frame may be divided into multiple slices, each encoding a portion of the frame. The temporal hierarchy of frames allows frames to reference preceding and succeeding pictures in the order in which they are displayed. Images are coded in the necessary order to ensure that dependencies for decoding each frame are met. An affine interpretation mode is available, where instead of using one or two motion vectors to select and filter reference sample blocks for the prediction unit, the prediction unit is divided into multiple smaller blocks, and a motion field is generated such that each smaller block has a separate motion vector. The motion field uses the motion vectors of points near the prediction unit as "control points." Affine prediction allows for coding motions different from translation, with less need for deeply split coding trees. The biprediction mode available in VVC geometrically blends two reference blocks along a selected axis, with an angle and offset from the center of the signaled blocks. This geometric division mode ("GPM") allows the use of larger coding units along the boundary between two objects, with the geometric shape of the boundary coded for the coding unit being used as an angle and center offset. Instead of using a Cartesian (x,y) offset, the motion vector difference can be coded as direction (up / down / left / right) and distance, with a set of distances that are powers of two supported. The motion vector predictor is taken from the adjacent block ("merge mode") as if no offset were applied. The current block will share the same motion vector as the selected adjacent block.

[0126] Samples are selected according to motion vector 878 and reference picture index. Since motion vector 878 and reference picture index are applied to all color channels, interpretation is described primarily in terms of operations on Pus rather than PBs. The decomposition of each CTU into one or more interpretation blocks is described in a single coding tree. Interpretation methods may differ in the number and precision of motion parameters. Motion parameters typically include the spatial movement of each reference frame, in addition to the reference frame index indicating which reference frame from the list of reference frames is used, but may also include more frames, special frames, or complex affine parameters such as scaling and rotation. Furthermore, predetermined motion refinement processes may be applied to generate dense motion estimates based on the referenced sample blocks.

[0127] By determining and selecting PB820 and subtracting PB820 from the original sample block with subtractor 822, the residual with the lowest coding cost, represented as 824, is obtained and lossy compression is performed. The lossy compression process comprises the steps of transformation, quantization, and entropy coding. The forward linear transformation module 826 applies a forward transformation to the difference 824, transforming the difference 824 from the spatial domain to the frequency domain and generating linear transformation coefficients represented by arrows 828. The maximum linear transformation size in one dimension is either a 32-point DCT-2 or 64-point DCT-2 transformation, defined by "sps_max_luma_transform_size_64_flag" in the sequence parameter set. If the encoded CB is larger than the maximum supported linear transformation size represented as a block size (e.g., 64x64 or 32x32), the linear transformation 826 is applied in a tiling manner to transform all samples of the difference 824. When non-square CBs are used, tiling is also performed using the largest available transformation size for each dimension of the CB. For example, if a maximum transformation size of 32 is used, a 64x16 CB will use two 32x16 linear transformations arranged in a tiled manner. If a CB is larger than the maximum supported transformation size, the CB will be filled with TBs in a tiled manner. For example, a 128x128 CB with a 64-pt transformation maximum size will be filled with four 64x64 TBs in a 2x2 arrangement. A 64x128 CB with a 32-pt transformation maximum size will be filled with eight 32x32 TBs in a 2x4 arrangement.

[0128] The application of transformation 826 results in multiple TBs for the CB. If each application of the transformation operates on a TB larger than 32x32, for example, a 64x64 difference 824, then all resulting linear transformation coefficients 828 outside the top-left 32x32 area of ​​the TB are set to zero (i.e., discarded). The remaining linear transformation coefficients 828 are passed to the quantizer module 834. The linear transformation coefficients 828 are quantized according to the quantization parameter 892 associated with the CB to produce linear transformation coefficients 832. In addition to the quantization parameter 892, the quantizer module 834 may also apply a "scaling list" to enable heterogeneous quantization within the TB by further scaling the residual coefficients according to their spatial position within the TB. The quantization parameter 892 may differ for luma CBs versus each chroma CB. The linear transformation coefficients 832 are passed to the forward quadratic transformation module 830, which either performs an inseparable quadratic transformation (NSST) operation or bypasses the quadratic transformation to generate the transformation coefficients represented by arrow 836. The forward linear transformation 826 is typically separable and transforms a set of rows, then a set of columns for each TB. The forward linear transformation module 826 uses either a Type II discrete cosine transform (DCT-2) in the horizontal and vertical directions, or a bypass of the transformation in the horizontal and vertical directions, or a combination of a Type VII discrete sine transform (DST-7) and a Type VIII discrete cosine transform (DCT-8) in the horizontal or vertical directions for ruma TBs whose width and height do not exceed 16 samples. The use of the combination of DST-7 and DCT-8 is referred to as a "multi-transform selection set" (MTS) in the VVC standard.

[0129] The forward quadratic transform of module 830 is generally an inseparable transform, which is applied only to the residuals of the intra-predicted CU, and can nevertheless be bypassed. The forward quadratic transform operates on either 16 samples (configured as a 4x4 subblock above the left of the linear transform coefficients 828) or 48 samples (configured as three 4x4 subblocks above the left of the 8x8 coefficients above the linear transform coefficients 828) to generate a set of quadratic transform coefficients. The set of quadratic transform coefficients may be fewer in number than the set of linear transform coefficients from which they are derived. By applying the quadratic transform only to sets of coefficients that are adjacent to each other and contain DC coefficients, the quadratic transform is called a “low-frequency inseparable quadratic transform” (LFNST). Such quadratic transforms are obtainable through a training process and by their inseparable nature and trained origin, and can leverage additional redundancy in residual signals that cannot be captured by separable transforms such as DCT and DST variations. Furthermore, when an LFNST is applied, all remaining coefficients in the TB are zero in both the linear and quadratic transform domains.

[0130] The quantization parameter 892 is constant for a given TB, and therefore results in uniform scaling for generating residual coefficients in the linear transformation domain with respect to TB. The quantization parameter 892 can vary periodically with a signaled "delta quantization parameter". The delta quantization parameter (delta QP) is signaled once for each CU contained within a given area, called a "quantization group". If the CU is larger than the quantization group size, the delta QP is signaled once using one of the TBs of the CU. That is, the delta QP is signaled once by the entropy encoder 838 for the first quantization group of the CU and not for any subsequent quantization groups of the CU. Non-uniform scaling is also possible by applying a "quantization matrix", so that the scaling coefficient applied to each residual coefficient is derived from a combination of the quantization parameter 892 and the corresponding entry in the scaling matrix. The scaling matrix can be smaller in size than the TB size, and when applied to the TB, a nearest neighbor approach is used to provide the scaling value for each residual coefficient from a scaling matrix smaller in size than the TB size. The residual coefficients 836 are fed to the entropy encoder 838 for encoding in the bitstream portion 816. Typically, the residual coefficients of each TB having at least one significant residual coefficient of the TU are scanned according to a scan pattern to generate an ordered list of values. The scan pattern generally scans the TB as a sequence of 4x4 "subblocks," providing regular scan operation with a granularity of 4x4 sets of residual coefficients, and the arrangement of the subblocks depends on the size of the TB. Scanning within each subblock and progressing from one subblock to the next typically follows a backward oblique scan pattern. In addition, the quantization parameter 892 is encoded in the bitstream portion 716 using delta QP syntax elements, and the slice QP and quadratic transformation index 888 of the initial values ​​in a given slice or subpicture are encoded in the bitstream portion 816.

[0131] As described above, the video encoder 800 needs access to a frame representation corresponding to the decoded frame representation seen in the video decoder. Therefore, the residual coefficient 836 passes through the inverse quadratic transform module 844 and operates according to the quadratic transform index 888 to produce an intermediate inverse transform coefficient represented by arrow 842. The intermediate inverse transform coefficient 842 is inversely quantized by the inverse quantization module 840 according to the quantization parameter 892 to produce an inverse transform coefficient represented as 846. The inverse quantization module 840 may also perform inverse heterogeneous scaling of the residual coefficient using a scaling list, corresponding to the forward scaling performed in the quantizer module 834. The inverse transform coefficient 846 is passed to the inverse linear transform module 848 to produce a residual sample of TU, ​​represented by arrow 850. The inverse linear transform module 848 applies the DCT-2 transform horizontally and vertically, constrained by the maximum available transform size, as described with reference to the forward linear transform module 826. The type of inverse transformation performed by the inverse quadratic transformation module 844 corresponds to the type of forward transformation performed by the forward quadratic transformation module 830. The type of inverse transformation performed by the inverse linear transformation module 848 corresponds to the type of linear transformation performed by the linear transformation module 826. The totalization module 852 adds the residual samples 850 and PB 820 to produce a reconstructed sample of CU (indicated by arrow 854).

[0132] The reconstructed sample 854 is passed to the reference sample cache 856 and the in-loop filter module 868. The reference sample cache 856, which is typically implemented using static RAM on the ASIC to avoid costly off-chip memory access, provides the minimum sample storage required to satisfy the dependency for generating an intra-frame PB for subsequent Cus in the frame. The minimum dependency typically includes a "line buffer" of samples along the bottom of the row of CTUs for use by the next row of CTUs, and column buffering whose range is set by the height of the CTUs. The reference sample cache 856 feeds the reference sample (represented by arrow 858) to the reference sample filter 860. The sample filter 860 applies a smoothing operation to generate filtered reference samples (indicated by arrow 862). The filtered reference sample 862 is used by the intra-frame prediction module 864 to generate an intra-predicted block of samples, represented by arrow 866. For each candidate intra-prediction mode, the intra-frame prediction module 864 generates a block of samples, i.e., 866. The block of samples 866 is generated by module 864 using techniques such as DC, planar, or angular intra-prediction. The block of samples 866 can also be generated using a matrix multiplication approach with adjacent reference samples as input and a matrix selected by the video encoder 800 from a set of matrices, the selected matrix being signaled in the bitstream 816 using an index to identify which matrix from the set of matrices should be used by the video decoder.

[0133] The In-Loop Filter Module 868 applies several filtering steps to the reconstructed sample 854. These filtering steps include a “Deblocking Filter” (DBF) that applies smoothing aligned to the CU boundary to reduce artifacts caused by discontinuities. Another filtering step present in the In-Loop Filter Module 868 is the “Adaptive Loop Filter” (ALF), which applies a Wiener-based adaptive filter to further reduce distortion. A further filtering step available in the In-Loop Filter Module 868 is the “Sample Adaptive Offset” (SAO) filter. The SAO filter works by first classifying the reconstructed sample into one or more categories and then applying an offset at the sample level according to the assigned category.

[0134] The filtered sample, represented by arrow 870, is output from the in-loop filter module 868. The filtered sample 870 is stored in the frame buffer 872. The frame buffer 872 typically has the capacity to store several pictures (e.g., up to 16) and is therefore stored in memory 206. Because the frame buffer 872 requires a large amount of memory, it is not typically stored using on-chip memory. Therefore, accessing the frame buffer 872 is costly in terms of memory bandwidth. The frame buffer 872 provides a reference frame (represented by arrow 874) to the motion estimation module 876 and the motion compensation module 880. The reference frame 874 is output as the reconstructed frame 818 by the encoder module 542. In the example in Figure 8, the reconstructed frame 818 is the result of the operation of lossy VVC coding, i.e., the operation of modules 810 to 890.

[0135] The motion estimation module 876 estimates several “motion vectors” (represented as 878), each being a Cartesian space offset from the current CB position and referencing a block in one of the reference frames in the frame buffer 872. A filtered block of reference samples (represented as 882) is generated for each motion vector. The filtered reference samples 882 form further candidate modes available for potential selection by the mode selector 886. Furthermore, for a given CU, the PU820 can be formed using one reference block ("uni-predicted") or two reference blocks ("bi-predicted"). For the selected motion vector, the motion compensation module 880 generates the PB820 according to a filtering process that supports sub-pixel precision in the motion vector. Thus, the motion estimation module 876 (which operates for many candidate motion vectors) can achieve reduced computational complexity by performing a simplified filtering process compared to that of the motion compensation module 880 (which operates only for selected candidates). If the video encoder 800 selects interpretation for CU, the motion vector 878 is encoded into the bitstream portion 816.

[0136] The video encoder 800 in Figure 8 is described with reference to the Multipurpose Video Coding (VVC), but other video coding standards or implementations may also employ the processing steps of modules 810-890. The frame data 802 (and bitstream 816) is also stored in memory 206, hard disk drive 210, CD-ROM, and Blu-ray disc. TMAlternatively, it may be read from (or written to) another computer-readable storage medium. Additionally, frame data 802 (and bitstream 816) may be received from (or transmitted to) an external source such as a server or radio frequency receiver connected to the communication network 220. Bitstream 816 is related to one of bitstreams 121 and 151. Since the communication network 220 may provide limited bandwidth, rate control should be used in the video encoder 800 to avoid saturating the network when frame data 802 is difficult to compress.

[0137] The bitstream 816 may consist of one or more slices representing spatial sections (sets of CTUs) of frame data 802, generated by one or more instances of the video encoder 800, each operating in conjunction under the control of the processor 205, each generating a bitstream portion 816. The bitstream portion 816 may also include one slice corresponding to one region to be output as a set of subpictures forming a single picture, each subpicture being independently encodeable and independently decodeable to any other slice or subpicture within the picture.

[0138] Figures 9A and 9B are schematic block diagrams showing the division of a picture into regions. Figure 9A shows picture 900 divided into regions 910, 912, and 916, suitable for packing data from a cross-channel uncorrelated (PCA-based) encoder network topology 700. Each tensor feature map is packed by packer 538. The example in Figures 9A and 9B shows three regions corresponding to the mean channel, basis vectors, and coefficients.

[0139] Referring to Figure 9B, picture 900b corresponds to picture 900, region 910b corresponds to 910, and 912b corresponds to 912. Region 910b holds mean channel data, such as mean channel 920 for each tensor of tensor 115. In the example in Figure 9B, the basis vectors include, among other things, basis vector 922, which are packed so as not to overlap in the area of ​​region 912b. Basis vector 922 forms a CxC' array of integer quantized values, where C is the number of channels in tensor 115 and C' is the number of basis vectors obtained from the decomposition performed by decomposition module 718. When multiple tensors are decomposed into sets of basis vectors, for example when tensor 115 has multiple tensors, basis vector 922 contains each set of basis vectors arranged in a non-overlapping, left-aligned, vertically connected manner. The average channel 920 corresponds to channel 712, and the basis vector 922 corresponds to, for example, some of the basis vectors 720.

[0140] In picture 900 (or 900b), areas not used to store any data may be occupied by sample values ​​corresponding to the value "0" after applying inverse quantization to return the sample values ​​to the floating-point domain. Similarly, in regions 910 (or 910b), 912 (or 912b), and 916, areas not used to store any data may be occupied by sample values ​​corresponding to the value "0" after applying inverse quantization to return the sample values ​​to the floating-point domain. Region 916 holds the coefficient tensor data corresponding to 724 in Figure 7, where the coefficients of each basis vector form a width x height feature map. The coefficient region 916 has coefficients packed as 24 feature maps (applicable to basis vectors [0..23]), such as feature map 930.

[0141] Each region (910 / 910b, 912 / 912b, and 916 / 916b) is aligned to a CTU grid that is typically 128x128. The position of each region is specified as the top-left and bottom-right Cartesian CTU addresses when addressed using Cartesian coordinates in units of CTU size. For example, region 916 occupies CTUs from CTU (0,1), corresponding to the top-left luma sample at position (0,128), to CTU (6,3), corresponding to the bottom-right luma sample position at (6x128+127,3x128+127) or (895,511).

[0142] Figures 10A and 10B are schematic block diagrams showing the division of a picture into a single region. In Figure 10A, picture 1000 is divided into a single region 1010, suitable for packing the compressed tensor 532 that can be generated by the tensor compressor 600. Figure 10B shows picture 1000b, which corresponds to picture 1000, where the feature maps are packed into region 1010b, which corresponds to 1010. Each feature map of the compressed tensor, such as feature map 1030, is packed into region 1010b such that the feature maps are packed from left to right, moving to the next row when the available space in the current row is exhausted. Thanks to the tensor compressor 600, the area required to pack the compressed tensor 532 is smaller than the area required if each tensor of tensor 115 were packed into picture 1000. Region 1010 is defined based on the top-left and bottom-right Cartesian CTU addresses.

[0143] Figure 11 is a schematic block diagram showing an exemplary structure 1100 of a multi-layer bitstream 123 (including sub-bitstreams 121 or 151) that holds encoded and packed feature maps and associated metadata. The multi-layer bitstream 123 contains a group of syntax elements, each preceded by a "Network Abstraction Layer" (NAL) unit header. For example, the NAL unit header 1108 precedes the Sequence Parameter Set (SPS) 1110a. Scalability Dimension Information (SDI) SEI messages 1112 signal the characteristics of the multi-layer bitstream 123 and are present when the VVC codec is used. The SEI message 1112 is included once for the first access unit (AU) in the coded video sequence. The SEI message 1112 signals the number of layers, usually two, using the sdi_max_layers_minus1 syntax element. The first (primary) layer is the video bitstream 151. The second layer is the extension layer and is signaled using the sdi_auxiliary_info_flag set to 1 and the corresponding sdi_aux_id[i] flag (i=1 for feature layers) which is set to indicate a feature layer, e.g., feature layer 121. For sdi_aux_id[i], a value in the range of 3 to 127, e.g., 3, indicates a feature layer (AUX_FEATURE) standardized in the ITU-T Recommendation H.274|ISO / IEC23002-7 version. For sdi_aux_id[i], a value in the range of 128 to 159, e.g., 128, indicates a feature layer standardized in a different standard than ITU-T Recommendation H.274|ISO / IEC23002-7. The extension (feature) layer is associated with the primary (video) layer using the sdi_associated_primary_layer_idx syntax element.

[0144] The Video Parameter Set (VPS) 1115 signals the maximum nuh_layer_id used by the sub-bitstreams present in a multi-layer bitstream, with each nuh_layer_id value accompanied by a flag indicating which ones exist in the current VPS. If HEVC is used, both syntax elements vps_base_layer_internal_flag and vps_base_layer_available_flag are set to 1, indicating that bitstream 123 contains a base layer. If VVC is used, the syntax element vps_all_independent_layers_flag is set to 1, indicating that there are no inter-layer predictions between the video (base) layer and the feature (extension) layer. Setting ptl_multilayer_enabled_flag to 1 indicates that the "profile_level_tier" syntax is signaled in VPS1115, and separate instances of the video layer and feature layer are further indicated via the vps_num_ptls_minus1 syntax element. For additional feature layers (additional second sub-bitstreams), if present, additional instances of the profile_level_tier syntax exist. The profile_level_tier syntax provides a means of signaling the profile of the sub-bitstream along with the levels and tiers. SPS1110a includes signaling for the bit depth, chroma format, and resolution of video layer 151, also known as the “base layer”, with the associated NAL units coded in the nuh_layer_id syntax element set to 0. SPS1110b includes signaling for the bit depth, chroma format, and resolution of the feature picture (e.g., picture 900 or picture 1000) for feature layer 121, depending on the compressor and decompressor selected in step 1810. The NAL units associated with SPS1110b, i.e., associated with feature layer sub-bitstream 121, are signaled with a nuh_layer_id that is not 0, usually set to 1.If a multi-layer bitstream 123 contains multiple feature sub-bitstreams, each sub-bitstream uses a unique nuh_layer_id value and has a separate SPS. Each sub-bitstream within the multi-layer bitstream 123 is extracted by a sub-bitstream extraction process that filters bitstream 123 based on the layer identifier [nuh_layer_id] to obtain the sub-bitstream from bitstream 123. For example, the video layer 151 (first sub-bitstream) is obtained by a first sub-bitstream extraction process that filters bitstream 123 based on the first layer identifier that identifies the video layer 151 within bitstream 123 to obtain the video layer 151. Furthermore, the feature layer 121 (second sub-bitstream) is obtained by a second sub-bitstream extraction process that filters bitstream 123 based on the second layer identifier that identifies the feature layer 121 within bitstream 123 (different from the first layer identifier) ​​to obtain the feature layer 121. Each sub-bitstream can be decoded independently using independent processing. For example, as described below in relation to Figure 19, the video layer 151 (171) and the feature layer 121 (143) are filtered from the bitstream 123 using different processing methods.

[0145] When HEVC is used, the Multilayer Main profile may be used to encode and decode the video parameter set, which requires the use of a 4:2:0 chroma format with 8-bit depth for both video layer 151 (first sub-bitstream) and feature layer 121 (second sub-bitstream). For the feature layer, a neutral value of 128 is used for the chroma sample. The scalability_mask_flag indicates that the scalability dimension of the feature layer is "Auxiliary," and the "AuxId" is used to signal the type of auxiliary layer, i.e., the feature layer.

[0146] When VVC is used, the Multilayer Main 10 profile may be used to encode and decode the video parameter set, which allows the use of 4:0:0 (monochrome) for feature layer 151 (first sub-bitstream) and a bit depth in the range of 8 to 10 bits. In some implementations, a 4:0:0 chroma format may be used for feature layer 121 (second sub-bitstream). The picture layout includes the position and size of any sub-pictures (e.g., 910, 912, and 916) using sub-picture information 1111. The SPS1110 is associated by a prediction structure (i.e., inter-prediction reference picture) and generally needs to be decoded before decoding a sequence of frames having the same width, height, bit depth, and chroma format. In applications involving streaming retransmission of SDI SEI1112, FCVMC SEI1113, VPS1115, and SPS1110a and 1110b, it may be performed so that decoding can be started from a midpoint in the bitstream. Furthermore, if the decoder network topology changes during execution, the frame area required for the compressed tensor may change, necessitating the insertion of another SPS to signal the updated frame dimensions. SPS 1110a also indicates the chroma format, bit depth, and resolution of the frame data represented by bitstream 121. Coated subpicture 1122 encodes subpicture 912 and includes slice header 1130 followed by slice data 1140. Slice data 1140 contains a sequence of CTUs, providing a coded representation of the frame data. Coated subpicture 1124 encodes subpicture 916 corresponding to the coefficients. The multilayer bitstream 123 contains an FCVCM decoder network instruction SEI message 1113, written by metadata encoder 544 and conforming to the syntax described in Appendix A or an equivalent thereto. The SEI message 1113 encodes the metadata necessary to convert the decoded frame into a set of tensors suitable for use by NN Part 2 166.

[0147] Figure 12 is a schematic block diagram showing an implementation 1200 of the tensor decoder 146. The decoder 1200 has a tensor decompressor 1250. Figure 19 shows a method 1900 for executing NN Part 2 to complete the execution of a neural network, such that a multilayer bitstream 123 is decoded, the tensors are reconstructed using the tensor decompressor, and NN Part 2 is selected so as to be compatible with the tensors decoded from the multilayer bitstream 123. As described, method 1900 is configured to determine a set of compatible NN Part 2 candidates from a library of NN Part 2 implementations available at the destination device 140, and to select one of the compatible NN Part 2 candidates for execution at the destination device 140. The multilayer bitstream 123 includes a primary (video) sub-bitstream 171 and an extension (features) sub-bitstream 143, each of which is decoded to produce a video frame 172 and a decoded tensor 149.

[0148] The destination device 140 and method 1900 may be implemented as one or more software application programs 233 executable within the computer system 200. The destination device 140 and method 1900 may also be implemented by instructions 231 (see Figure 2B) within the software 233 executed within the computer system 200. The software instructions 231 may be formed as one or more code modules, each performing one or more specific tasks. Method 1900 begins with a video frame decoding step 1910.

[0149] In step 1910, the decoder (such as decoder 170 or decoder 146) decodes a video parameter set (VPS) 1115 indicating the presence of multiple sub-bitstreams in bitstream 123. The VPS 1115 is decoded based on whether the Multilayer Main profile for VVC or HEVC is being used.

[0150] Furthermore, in step 1910, the video decoder 170, under the execution of the processor 205, decodes frames from the primary (video) subbitstream 171 to generate decoded frames 172. The video decoder 160 may implement a VVC decoder or an HEVC decoder, as described with reference to Figure 13. If the bitstream 123 contains multiple subbitstreams, the video decoder 170 operates to decode the video from a first subbitstream (video subbitstream 171) obtained by performing a first subbitstream extraction process on the bitstream 123. Control in the processor 205 proceeds from step 1910 to the candidate task result decoding step 1920.

[0151] In step 1920, referring to Figure 12, the receiver / demux 142 decodes the candidate task result 11104 (Figure 3) from the multi-layer bitstream 123. The candidate task result 11104 represents the result of proxy NN part 2 1118 performed on source device 110, which may include, for example, one or more bounding boxes or image classification. Control in processor 205 proceeds from step 1920 to task result check step 1925.

[0152] In step 1925, the processor determines whether the candidate task result 11104 indicates a frame of interest. If the destination device is configured to track people in a scene, a candidate task result 11104 indicating the detection of one or more people indicates a frame of interest, and control in the processor 205 proceeds from step 1925 to the NN instruction decoding step 1930 ("YES" in step 1925). If the candidate task result 11104 does not indicate any detected people, method 1900 terminates, and the processor 205 proceeds to the next frame of data in the multi-layer bitstream 123.

[0153] In step 1930, the receiver / demux 142 decodes the NN topology instruction 1197 and NN weight instruction 1198 from the multilayer bitstream 123 under the execution of the processor 205. The NN topology instruction 1197 and NN weight instruction 1198 may also be a hash of the overall NN topology and weights, i.e., a hash of the combined network implemented in system 100, without considering the division of the network into parts for execution on source device 110 and destination device 140. The hash typically has a fixed-length value, such as a 256-bit sha1sum, sha256, sha2-384, or sha2-512 value, calculated from the byte-serialized form of the network topology and network weights. The receiver / demux 142 may use the entropy decoder 1320, as described with reference to Figure 13, to parse the bitstream data. Control in the processor 205 proceeds from step 1930 to the NN part 1 instruction decoding step 1940.

[0154] In step 1940, the receiver / demux 142 decodes the NN Part 1 topology instruction 11100 and the NN Part 1 weight instruction 11101 (collectively 161) from the multilayer bitstream 123 under the execution of the processor 205. Instructions 11101 and 11101 (161) are hashes indicating the network topology and weights that correspond exactly to NN Part 1 as performed in the NN Part 1 module 114. In step 1940, the receiver / demux 142 decodes the NN Part 1 output instruction 11102, which encodes the number and dimensions of the tensors generated by NN Part 1, i.e., before any further tensor compression performed by the tensor encoder 116. With knowledge of the dimensions of the tensors output from NN Part 1 114, the decoder can determine the dimensions of the tensors to be supplied to the NN Part 2 candidate without having to map the NN Part 1 topology instruction 1197 to the actual network topology. In other words, the destination device 140 can determine the dimensions of the tensor without having to determine the network topology on the encoder side used to generate the multilayer bitstream 123. Control in the processor 205 proceeds from step 1940 to the NN part 2 candidate selection step 1960.

[0155] In step 1960, the NN part 2 repository 160 is queried using the NN part 1 topology instruction 11100 and the NN part 1 weight instruction 11101. Based on the association between the registered NN parts 1 and 2, a set of compatible NN part 2 candidates is determined. Each NN part 2 candidate can receive the decoded tensor output from NN part 1 114 to generate the task result. As an example, MaskRCNN used for instance segmentation is a superset of the FasterRCNN network used for object detection (both in this example use the ResNet-101 architecture). Thus, the split points may divide the network into parts somewhat arbitrarily, i.e., the criteria for splitting may be driven by the desired degree of computational offload from destination device 140 to source device 110, rather than being limited by topological constraints. Meta TM Widely used weight files, such as those published by AI Research, are not compatible because MaskRCNN and FasterRCNN are trained differently at most of their layers. Therefore, tensors encoded at a partition point using one network cannot be further processed by the other. Thus, while network topology instructions indicate compatibility between these two networks, for example at a P-layer partition point, weight instructions indicate incompatibility. TMWeights other than those published by AI Research may be used, including weights trained so that the layers up to the P-layer split point are common. Thus, the multi-layer bitstream 123 can be used to perform both object detection and instance segmentation, provided that compatible weights (matching NN part 1 instructions) are used in the source device 110. The implementer in the destination device 140 may train the weights of NN part 2 for a very specific task, much more specific than the task performed by the proxy NN part 2 118. One example is the detection of specific objects, such as a specific car or person. Freezing the layers near the beginning of the network is a widely used technique in training NNs, where these layers tend to train lower-level features, reducing training time that would otherwise be considerable by training only the later layers for a specific task. Frozen early layers with flexible selection of later layers are similar to a network topology split into NN part 1 and 2, having a common "backbone" and having the same weights for NN part 1 and different weights for each NN part 2 candidate. Figure 20 illustrates a method for generating such instructions for various candidates of NN Part 2 that may be available in memory 206 of the destination device 206. In particular, since the network topology and weights are static, the instructions may be pre-computed or obtained by external means for use at runtime. Control in processor 205 proceeds from step 1960 to bitstream compatibility check test determination step 1970.

[0156] In step 1970, under the execution of the processor 205, the destination device 140 determines whether the multilayer bitstream 123 is compatible with the destination device 140, based on whether at least one candidate NN part 2 was found in step 1960. Control in the processor 205 proceeds from step 1970 to compatibility check test step 1980.

[0157] In step 1980, if step 1970 determines that no compatible NN part 2 candidate is available at the destination device 140 ("NOT OK" in step 1980), control in processor 205 proceeds to error condition step 1990. If at least one compatible NN part 2 candidate is available at the destination device 140 ("OK" in step 1990), control in processor 205 proceeds to NN part 2 selection step 19100. In essence, steps 1970 and 1980 determine whether to decode a tensor from bitstream 123. Steps 1970 and 1980 determine whether to run NN part 2 using the decoded tensor from the multilayer bitstream 123.

[0158] Steps 1970 and 1980 can be thought of as having the effect of determining whether to decode the tensor from the bitstream using the second part of the network (NN part 2). This determination is based on the availability of multiple network part 2s associated with the first part of the network (NN part 1), for example, in the NN part 2 repository 160. One of the available networks is selected for execution if it is available, i.e., if tensor decoding should be performed.

[0159] In step 1990, the destination device 140 signals to the user that, under the execution of processor 205, the feature sub-bitstream 143 in the received multilayer bitstream 123 cannot be processed. The decoded video 172 may still be presented to the user based on the results of proxy NN part 2, i.e., the video may be displayed even if more specific filtering cannot be performed within the destination device 140 if the source device 140 has detected an object of interest. Method 1900 ends, and processor 205 proceeds to the next frame.

[0160] Step 19100 is performed if steps 1970 and 1980 operate in a manner that determines to decode a tensor from the bitstream. In step 19100, processor 205 selects one of the NN part 2 candidates determined in step 1960, along with the network weights and topology obtained from the NN part 2 repository 160. Control in processor 205 then proceeds from step 19100 to instantiation step 19110 of the selected NN part 2.

[0161] In step 19110, the selected NN part 2 is instantiated or loaded to form NN part 2 166 with both the network topology and weights initialized. Control in processor 205 proceeds from step 19110 to pack frame decoding step 19120.

[0162] In step 19120, the picture decoder 1204, under the execution of the processor 205, decodes one packed frame from the feature subbitstream 143 to generate a decoded frame 1210. The operation of the picture decoder 1204 is described with reference to Figure 13. Control in the processor 205 proceeds from step 19120 to the tensor unpacking step 19130.

[0163] In step 19130, the unpacker 1214 of the decoder 146 reads feature maps from the decoded frame 1210 in accordance with the packing format described with reference to Figures 9A and 9B and 10A and 10B, under the execution of the processor 205. For each tensor, several feature maps are decoded, the number of which corresponds to the number of channels used in the tensor signaled in tensor information 1195. The channels of each tensor are unpacked as a two-dimensional feature map. The number of feature maps or channels to be decoded for a given tensor is decoded from the feature subbitstream 143 as "number of channels". Multiple tensors, each having one channel, are arranged horizontally or vertically within a slice or subpicture of the frame. Each type of feature map previously determined is unpacked by the unpacker 1214 in step 19130 using the feature map assignment determined for the frame (e.g., 1210) in method 1800. The assignment may be decoded from an SEI message. The unpacker 1214 outputs an integer tensor 1216, which is decoded using the decoder topology of the tensor decoder 146. Control in the processor 205 proceeds from step 19130 to the tensor inverse quantization step 19140.

[0164] In step 19140, the inverse quantizer 1218 of the decoder 1200 performs inverse quantization on the integer tensor 1216 by applying a quantization range, such as quantization range 1196, decoded from the multilayer bitstream 123, to the determined number of channels of each tensor, under the execution of the processor 205, thereby producing the inversely quantized tensor 1220. To perform the inverse quantization, the quantization range 1196 is decoded from the SEI message 1113 by the receiver / demux 142. Control in the processor 205 proceeds from step 19140 to the quantized tensor buffering step 19150.

[0165] In step 19150, the tensor storage module 1222 of the decoder 1200 provides inter-frame storage of the inversely quantized tensor 1220 under the execution of the processor 205. For each tensor from each region of the packing format in which at least one channel or feature map has been decoded, it is stored in the tensor storage module 1222. The tensor storage module 1222 generates an output tensor 1224 containing the most recent tensor for each tensor in which at least one feature map has been decoded. In other words, if no tensor was decoded for the current frame (i.e., it is determined that the number of channels is zero), the most recent value of a decoded tensor with a non-zero number of channels is used. For example, when using a PCA-based method such as encoder network topology 700, the average channels obtained from the average channel module 710 may be updated intermittently, in which case the tensor storage module 1222 holds the most recent result from the average channel module 710 for use by the tensor decompressor 1250. Control in processor 205 proceeds from step 19150 to tensor decompression execution step 19160.

[0166] In step 19160, the tensor decompressor 1250, under the execution of the processor 205, uses tensor 1224 as input and performs the steps specified by the decoder network topology to generate the decoded tensor 149. An exemplary structure of the tensor decompressor is described in relation to Figure 14.

[0167] If bitstream 123 contains multiple sub-bitstreams, steps 19100-19160 operate to decode the tensor for the decoded video from the second sub-bitstream (feature sub-bitstream 143) obtained by performing a second sub-bitstream extraction process on bitstream 123. Step 19160 decodes the tensor for the decoded video from the second sub-bitstream. The video layer and feature layer or bitstream can be extracted using different processes and decoded independently.

[0168] Control in processor 205 proceeds from step 19160 to the neural network second part execution step 19170, where a portion of the neural network is used to generate the task result.

[0169] In step 19170, module 166, under the execution of processor 205, executes NN Part 2, which was selected in step 19100 and executed as NN Part 2 166, using tensor 149 as input and producing task result 151 as output. Control in processor 205 proceeds from step 19170 to task result rendering step 19180. In step 19180, task result renderer 168, under the execution of processor 205, generates video frames that render the task result in association with a video layer. For example, step 168 can present both video frame 172 and a representation of task result 151. Task result 151 may be represented as a bounding box around an object of interest, a segmentation map with color tinting of the object of interest, etc. The video frame obtained from step 19180 is presented to the user of destination device 140 as the final output of system 100, for example via display 214.

[0170] Steps 19100-19180 operate to execute the second part of the network (NN part 2) to generate the neural network result when steps 1970 and 1980 decide to decode tensors from the bitstream. Method 1900 terminates, and the processor 205 may call method 1900 again when it receives the next frame in the multilayer bitstream 123.

[0171] Figure 13 is a schematic block diagram showing the functional modules of the video decoder 1300. The video decoder provides exemplary implementations of the picture decoder 1204 and the video decoder 170, respectively. The video decoder 1300 may be implemented as one or more software application programs 233 that can be executed within a computer system 200. The video decoder 1204 may be implemented by instructions 231 (see Figure 2B) within the software 233 that runs within the computer system 200. The software instructions 231 may be formed as one or more code modules, each performing one or more specific tasks. The video decoder 1204 is operable to decode both the primary (video) subbitstream 171 and the extended (features) subbitstream 143.

[0172] Either sub-bitstream 143 or 171 is input to the entropy decoder module 1320 as sub-bitstream 1301. The entropy decoder module 1320 extracts syntax elements from sub-bitstream 1301 by decoding a sequence of “bins” and passes the values ​​of those syntax elements to other modules in the video decoder 1204. The entropy decoder module 1320 decodes SPS, PPS, or slice headers using variable-length and fixed-length decoding, and decodes the syntax elements of the slice data as a sequence of one or more bins using an arithmetic decoding engine. Each bin may have one or more “contexts,” where the context describes the probability levels used to code the “1” and “0” values ​​for that bin. If multiple contexts are available for a given bin, a “context modeling” or “context selection” step is performed to select one of the available contexts for decoding the bin. The process of decoding the bins forms a sequential feedback loop in which each slice may be decoded holistically by a given instance of the entropy decoder 1320.

[0173] The entropy decoder module 1320 applies an arithmetic coding algorithm, such as "context-adaptive binary arithmetic coding" (CABAC), to decode syntax elements from sub-bitstreams 143 or 171 (1301). The decoded syntax elements are used to reconstruct parameters within the video decoder 1300. These parameters include mode selection information such as residual coefficients (represented by arrow 1324), quantization parameters 1374, quadratic transformation index 1370, and intra-prediction modes (represented by arrow 1358). The mode selection information also includes information such as motion vectors and the splitting of each CTU into one or more CBs. The parameters are typically used in combination with sample data from previously decoded CBs to generate PBs.

[0174] The residual coefficients 1324 are passed to the inverse quadratic transform module 1336, where, according to the quadratic transform index, the quadratic transform is applied or no action is performed (bypassed). The inverse quadratic transform module 1336 generates reconstructed transform coefficients 1332. That is, module 1336 generates linear transform domain coefficients from quadratic transform domain coefficients. The reconstructed transform coefficients 1332 are input to the inverse quantization module 1328. The inverse quantization module 1328 performs inverse quantization (or "scaling") on the residual coefficients 1332 in the linear transform coefficient domain to create reconstructed intermediate transform coefficients, represented by arrow 1340, according to the quantization parameter 1374. The inverse quantization module 1328 may also apply a scaling matrix to provide heterogeneous inverse quantization within the TB, corresponding to the operation of the inverse quantization module 840. If the use of a non-uniform inverse quantization matrix is ​​indicated in sub-bitstream 1301, the video decoder 1204 reads the quantization matrix from sub-bitstream 143 or 171 (1301) as a sequence of scaling coefficients and places the scaling coefficients into the matrix. Inverse scaling uses the quantization matrix in combination with the quantization parameters to create reconstructed intermediate transformation coefficients 1340.

[0175] The reconstructed conversion coefficients 1340 are passed to the inverse linear transformation module 1344. Module 1344 transforms the coefficients 1340 back from the frequency domain to the spatial domain. The inverse linear transformation module 1344 applies the inverse DCT-2 transformation horizontally and vertically, constrained by the maximum available transformation size described with reference to the forward linear transformation module 826. The result of the operation of module 1344 is a block of residual samples, indicated by arrow 1348. The block of residual samples 1348 is the same size as the corresponding CB. The residual samples 1348 are fed to the total module 1350.

[0176] In total module 1350, the residual sample 1348 is added to the decoded PB (represented as 1352) to generate a reconstructed sample block represented by arrow 1356. The reconstructed sample 1356 is supplied to the reconstructed sample cache 1360 and the in-loop filtering module 1388. The in-loop filtering module 1388 generates a reconstructed block of frame samples represented as 1392. The frame sample 1392 is written to frame buffer 1396. Frame buffer 1396 outputs an image or video frame 1397. If sub-bitstream 1301 is sub-bitstream 171, then frame 1397 corresponds to frame 172. If sub-bitstream 1301 is sub-bitstream 143, then frame 1397 corresponds to frame 1204.

[0177] The reconstructed sample cache 1360 operates similarly to the reconstructed sample cache 856 of the video encoder 800. The reconstructed sample cache 1360 provides storage for the reconstructed samples necessary for intra-predicting subsequent CBs without memory 206 (for example, by using data 232, which is normally on-chip memory instead). A reference sample, represented by arrow 1364, is retrieved from the reconstructed sample cache 1360 and fed to the reference sample filter 1368 to generate filtered reference samples, represented by arrow 1372. The filtered reference samples 1372 are fed to the intra-frame prediction module 1376. Module 1376 generates blocks of intra-prediction samples, represented by arrow 1380, according to intra-prediction mode parameters 1358, which are signaled in sub-bitstream 1301 and decoded by the entropy decoder 1320. The intra-prediction module 1376 supports the modes of encoder-side module 864, including IBC and MIP. Sample block 1380 is generated using modes such as DC, planar, or angle intra-prediction.

[0178] If the prediction mode of the CB is indicated to use intra-prediction in sub-bitstream 1301 (143 or 171), the intra-predicted sample 1380 forms the decoded PB 1352 via the multiplexer module 1384. The intra-prediction generates a predicted block (PB) of samples, which is a block in one color component and is derived using "adjacent samples" in the same color component. Adjacent samples are samples adjacent to the current block and have already been reconstructed by preceding them in the decoding order of the blocks. If the lumar and chroma blocks are collated, the lumar and chroma blocks may use different intra-prediction modes. However, two chroma CBs share the same intra-prediction mode.

[0179] If the prediction mode of the CB is indicated as inter-prediction in sub-bitstream 1301 (143 or 171), the motion compensation module 1334 generates a block of inter-prediction samples, represented as 1338. The inter-prediction sample block 1338 is generated using the motion vector decoded from sub-bitstream 143 or 171 by the entropy decoder 1320, and a reference frame index for selecting and filtering the block of samples 1398 from the frame buffer 1396. The block of samples 1398 is obtained from previously decoded frames stored in the frame buffer 1396. In the case of bi-prediction, two blocks of samples are generated, and they are blended to generate the samples for the decoded PB 1352. The frame buffer 1396 is substituted with the filtered block data 1392 from the in-loop filtering module 1388. Similar to the in-loop filtering module 868 of the video encoder 542, the in-loop filtering module 1388 applies one of the DBF, ALF, and SAO filtering operations. Generally, motion vectors are applied to both lunar and chroma channels, but the filtering processes for subsample interpolation in lunar and chroma channels differ.

[0180] Figure 14 is a schematic block diagram showing an implementation of a channel-uncorrelated tensor decoder 1400 that can function as a tensor decompressor 1250. If multiple tensors are to be decompressed, multiple instances of the tensor decoder 1400 may be instantiated as tensor decompressors 1250. Tensor 1224 is received from tensor memory 1222 and includes a coefficient tensor 1414, a basis vector tensor 1412, and a channel mean tensor 1410. The zero-centered tensor 1422 is generated by the dot product module 1420 under the execution of processor 205 by performing a dot product on coefficients 1414 and basis vectors 1412. The sum module 1424 adds the zero-centered tensor 1422 and the mean channel 1410 to produce one of the reconstructed tensors 149 as the output from the tensor decoder 1400. If one type of input, such as the mean, does not need to be updated, the number of channels for that type of input may be set to zero. If the number of input channels is set to zero, the most recently received value for that input may be used instead.

[0181] Figure 15 is a schematic block diagram showing a tensor decompressor 1500 that uses a multiscale feature reconstruction step, which can be used as a tensor decompressor 1250 in some implementations. Tensor decompressor 1500 includes a single-scale feature compression (SSFC) decompressor 1510. The SSFC decompressor 1510 takes a tensor 1224 with a reduced number of channels, such as 64 channels, and passes tensor 1224 to a convolutional layer 1512, which outputs a tensor 1513 with a restored number of channels, such as 256 channels. Tensor 1513 is passed to a batch normalization module 1514 to produce tensor 1515. Tensor 1515 is passed to a PreLU module 1516 to produce tensor 1520.

[0182] The tensor decompressor 1500 includes an MSFR module 1530. The MSFR module operates to generate multiple tensors from tensor 1520, which is produced by performing step 19160 as described with reference to Figure 19, using one or more trained convolutional layers. Upsampling modules 1532, 1534, and 1536 upsample tensor 1520 horizontally and vertically by a factor of 2, 4, and 8, respectively, to generate tensors 1533, 1535, and 1537. Tensor 1537 forms one output (P'2, 1557) from the MSFR module 1530 and is passed to the downsampling module 1542. The downsampling module 1542 downsamples tensor 1537 horizontally and vertically by a factor of 2 to generate tensor 1543, which has the same dimensions as tensor 1535. Tensor 1543 is provided to convolutional layer 1548, which outputs tensor 1549. The total module 1554 adds tensors 1535 and 1549 to generate tensor 1555 as the output (P'3) of MSFR module 1530.

[0183] Tensor 1535 is passed to the downsample module 1540. The downsample module 1540 downsamples tensor 1535 horizontally and vertically by a factor of 2 to produce tensor 1541, which has the same dimensions as tensor 1533. Tensor 1541 is provided to the convolutional layer 1546, which outputs tensor 1547. The totalization module 1552 adds tensors 1533 and 1547 together to produce tensor 1553 as the output (P'4) of the MSFR module 1530.

[0184] Tensor 1533 is passed to the downsample module 1538. The downsample module 1538 downsamples tensor 1533 horizontally and vertically by a factor of 2 to produce tensor 1539, which has the same dimensions as tensor 1520. Tensor 1539 is provided to the convolutional layer 1544, which outputs tensor 1545. The totalization module 1550 adds tensors 1520 and 1545 together to produce tensor 1551 as the output (P'5) of the MSFR module 1530. Tensors 1551, 1553, 1555, and 1557 give rise to tensor 149.

[0185] Figure 16A is a schematic block diagram showing an exemplary implementation 1600 of NN Part 2 166 of a CNN for object detection, corresponding to a portion of the "YOLOv3" network excluding the "DarkNet-53" backbone portion. NN Part 2 166 in Figure 16A can be used, for example, when the CNN backbone is implemented as shown in Figure 3A. Depending on the task performed on the destination device 140, a different network may be substituted as NN Part 2 166. The input tensor 149 is separated into tensors for each layer (i.e., tensors 1610, 1620, and 1634). Tensor 1610 is passed to the CBL module 1612 to generate tensor 1614. Tensor 1614 is passed to the detection module 1616 and the upscaler module 1622. The detection module outputs a bounding box 1618 in the form of the detection tensor. The bounding box 1618 is passed to the Non-Maximum Suppression (NMS) module 1648.

[0186] Before resizing for NN part 1 of network 114, scaling by the width and height of the original video is performed by the upscaler module 1622 to generate a bounding box that handles the coordinates in the original video data 113. The upscaler module 1622 takes tensors 1614 and tensor 1620 and produces an upscaled tensor 1624, which is passed to the CBL module 1626. The CBL module 1626 produces tensor 1628 as output. Tensor 1628 is passed to the detection module 1630 and the upscaler module 1636. The detection module 1630 produces a detection tensor 1632, which is fed to the NMS module 1648. The upscaler module 1636 is another instance of module 1622. The upscaler module 1636 takes tensors 1628 and tensor 1634 and outputs an upscaled tensor 1638. The upscaled tensor 1638 is passed to the CBL module 1640, which outputs tensor 1642 to the detection module 1644. The detection module 1644 generates detection tensor 1646, which is supplied to the NMS module 1648.

[0187] CBL modules 1612, 1626, and 1640 each contain a chain of five CBL modules (e.g., CBL model 360 shown in Figure 3D). Upscaler modules 1622 and 1636 are instances of upscaler module 1660, as shown in Figure 16B. Module 1648 receives tensors 1618, 1632, and 1646 and outputs task result 151.

[0188] As shown in Figure 16B, the upscaler module 1660 accepts tensor 1662 (e.g., tensor 1614 in Figure 16A) as input. Tensor 1662 is passed to the CBL module 1666 (which has the structure of module 360) to generate tensor 1668. Tensor 1668 is passed to the upsampler 1670 to generate the upsampled tensor 1672. The concatenation module 1674 generates tensor 1676 by concatenating the upsampled tensor 1672 with the second input tensor 1664 (e.g., tensor 1620 which is input to the upscaler 1622 in Figure 16A).

[0189] Detection modules 1616, 1630, and 1644 are instances of detection module 1680 as shown in Figure 16C. Detection module 1680 receives tensor 1682. Tensor 1682 is input to CBL module 1684, which has the structure of module 360. CBL module 1684 generates tensor 1686. Tensor 1686 is passed to convolution module 1688, which implements the detection kernel to produce result 1690. In some configurations, the detection kernel applies a 1x1 kernel to generate an output on each of the three layer feature maps of the tensor. The detection kernel is 1x1x(Bx(5+C)), where B is the number of bounding boxes that a particular cell can predict, usually 3, and C is the number of classes, which can be 80, resulting in a kernel size of 255 detection attributes (i.e., tensor 1690). The constant "5" represents four bounding box attributes (box center x,y and size scale x,y) and one object confidence level ("objectness"). The results of the detection kernel have the same spatial dimensions as the input feature map, but the depth of the output corresponds to the detection attributes. The detection kernel is applied to each layer, usually three layers, resulting in a large number of candidate bounding boxes. To discard redundant boxes, such as overlapping predictions at similar scales, a non-maximal suppression process is applied to the resulting bounding boxes by the NMS module 1648, resulting in a final set of bounding boxes as output for object detection.

[0190] Figure 17 is a schematic block diagram showing the head portion 1700 of the CNN. The head portion 1700 can be implemented as NN part 2 166, for example, when NN part 1 114 is implemented as backbone 400. The head portion 1700 forms part of the overall network known as "Faster RCNN" and includes the feature network (i.e., backbone portion 400), the region proposal network, and the detection network. The input to the head portion 1700 is tensor 149, which includes P2-P6 layer tensors 1710, 1712, 1714, 1716, and 1718. P2-P5 layer tensors 1710, 1712, 1714, and 1716 correspond to the P2-P5 outputs 477, 475, 473, and 471 in Figure 4. The P2-P6 tensors 1710, 1712, 1714, 1716, and 1718 are input to the Region Proposal Network (RPN) head module 1720. The P6 tensor 1718 is generated by the Max Pooling module 1742, which operates on the P5 tensor 1716 to perform a 2x2 Max Pooling operation. The RPN head module 1720 performs a convolution on the input tensors to generate intermediate tensors. These intermediate tensors are fed to two subsequent sibling layers: (i) for classification and (ii) for bounding boxes, or "Region of Interest" (ROI) regression. The resulting output is the classification and bounding boxes 1722. The classification and bounding boxes 1722 are passed to the NMS module 1724. The NMS module 1724 prunes redundant bounding boxes by removing duplicate boxes with lower scores to generate the pruned bounding boxes 1726. The bounding box 1726 is input to the region of interest (ROI) pooler 1728. The ROI pooler 1728 uses several layer tensors of tensor 149 (details below) and the bounding box 1726 to generate a fixed-size feature map from various input-size maps using a max pooling operation. In the max pooling operation, the maximum value of each input value group is obtained by subsampling, and a single output value is generated in the output tensor.

[0191] The inputs to the ROI puller 1728 are the P2-P5 feature maps 1710, 1712, 1714, and 1716, as well as the region of interest proposal 1726. Each proposal (ROI) from 1726 is associated with a portion of the feature maps (1710-1716) to generate a fixed-size map. The size of the fixed-size map is independent of the underlying portion of the feature maps 1710-1716. One of the feature maps 1710-1716 is selected such that the resulting cropped map has sufficient detail according to, for example, the following rule: floor(4+log2(sqrt(box_area) / 224)), where 224 is the standard box size. The ROI puller 1728 operates to crop the input feature maps according to proposal 1726, generating the tensor 1730.

[0192] Tensor 1730 is fed to a fully connected (FC) neural network head 1732. The FC head 1732 runs two fully connected layers to generate a class score and a bounding box predictor delta tensor 1734. The class score is generally an 80-element tensor, where each element corresponds to the predicted score for the corresponding object category. The bounding box predictor delta tensor is an 80 × 4 = 320-element tensor containing the bounding boxes for the corresponding object categories. Final processing is performed by the output layer module 1736, which receives tensor 1734 and performs a filtering operation to generate a filtered tensor 1738. Objects with low scores (low classifications) are excluded from further consideration. The non-maximal suppression module 1740 receives the filtered tensor 1738 and removes duplicate bounding boxes by removing the duplicate boxes with lower classification scores, resulting in an inference output tensor 1742 corresponding to tensor 151.

[0193] Figure 20 shows an exemplary method 2000 for determining network instructions for the entire network topology and weights, as well as parts thereof. Method 2000 generates hashes that can be used to identify the exemplary ONNX-form network topology and / or network weights and can be executed at runtime or prior to the execution of methods 1800 and 1900. If executed prior, the results are held in memory 206 for reference at runtime. Method 2000 is executed for each neural network supported by system 100. Furthermore, method 2000 is executed for each partition of the NN into NN Part 1 and NN Part 2 parts. In other words, method 2000 is executed for NN Part 1 and NN Part 2 obtained from each partition point of the supported NN. Neural networks such as FasterRCNN-X101-FPN and MaskRCNN-X101-FPN can be used by method 2000 to generate hashes in the partitions into NN Part 1 and 2 that occur at boundaries such as P layer or Res layer. For neural networks where the dimensions of the internal layers depend on the dimensions of the input layers, a proxy input resolution such as 1920x1080 or 3840x2160 is used as the sample input from which the internal dimensions are determined. Using the sample input size allows for a network topology byte stream independent of the actual resolution of the frame data 113. Neural networks such as YOLOv3 may also be used by Method 2000 to generate hashes, for example, using a partition at the boundary of "Darknet-53" used to divide the network into NN Part 1 and NN Part 2. Method 2000 begins with the NN subrepresentation determination step 2010.

[0194] In step 2010, the ONNX representation of the neural network or a portion thereof is read from memory 206. The ONNX representation is a byte-serialized structured description of the NN topology and weights using a format called "Protocol Buffers". Protocol Buffers provides a language-independent, platform-independent, and extensible mechanism for serializing structured data. Protocol Buffers are a set of key-value pairs with byte-level syntax that provides a much more compact representation than alternatives such as Extensible Markup Language (XML). Each key-value pair forms a "record" with a "field number", "wire type", and "payload", and can signal a variety of data types, including boolean values, integers of varying precisions, and variable-length strings. The ONNX representation works to encode the elements (or "operators") of the network topology in sequential execution order to prevent minor differences in order from affecting hashing. Network node names, operator names, and variable names are specified as strings and are all replaced with simple enumerations (e.g., "var0", "var1", "operator0", "operator1", "node0", "node1", etc.) so that insignificant naming differences do not affect the hash operation. The ONNX representation may be derived from another representation, such as Python code using libraries like Pytorch, Tensorflow, or Keras. That representation is refactored to replace items such as variable names with simple enumerations and to be configured in execution order. The intention of the refactoring is to create a representation that depends only on the network structure and, as far as possible, does not rely on programmer discretionary aspects such as variable naming. The node names and dimensions of the NN input and output are included in the resulting instructions. Control in processor 205 proceeds from step 2010 to the NN sub-topology and weight determination step 2020.

[0195] In step 2020, the processor parses the byte-serialized ONNX representation of the network or a portion thereof, removing all data values ​​corresponding to trained elements such as network weights and bias values, thereby retaining a topological subset of the ONNX representation. The ONNX representation is filtered to retain graph properties (e.g., topology, convolution dimension, activation function) but not trainable parameters; that is, it is filtered to remove information about network weights that are outside the NN structure. Additionally, the ONNX representation is filtered to generate a weight subset of the ONNX representation that contains only trained elements such as weights and bias values ​​(collectively, “weights”). Certain additional parameters that are not trainable but are data-adaptive, such as the “running mean” and “running variance” of batch normalization, are included as “weights” in this filtering process so that changes in their values ​​affect the final hash calculation. In some cases, the network weight representation may be a hash of the MPEG NNC representation of the network weights. Control in processor 205 proceeds from step 2020 to step 2030, which determines the NN sub-topology and weight instructions.

[0196] In step 2030, topology subsets and weight subsets of the ONNX representation are hashed using algorithms such as md5sum, sha1sum, sha256, or crc32 to form a compact hash of the network topology or weights with a fixed length. This results in a unique and compact identifier for the original data, which would typically occupy several hundred MB. Furthermore, the probability of collisions, i.e., the probability of two different topologies or weights yielding the same hash value, is negligibly small for hash algorithms such as those mentioned. Method 2000 is performed on the entire network and parts thereof. In other words, iterations of method 2000 are performed on each of (i) the entire CNN, (ii) the first network part (NN part 1 or backbone), and (iii) the second network part (NN part 2 or head). The operation of step 2030 is a graph-preserving filter for each network that works to determine a network topology directive, which is a hash of the filtered ONNX network representation. Additionally, the operation of step 2030 operates to determine that the network weight instruction is a filter that preserves the weight characteristics and is a hash of the filtered ONNX network representation.

[0197] If the network is divided into NN Part 1 and NN Part 2, the instructions generated for NN Part 1 and NN Part 2 include the hash of the parent (overall) network, the hash of the subsequent network (i.e., NN Part 2) relative to NN Part 1, and the hash of the preceding network (i.e., NN Part 1) relative to NN Part 2. Additional hashes associated with a given network or a portion of a network, i.e., "associations," are described with reference to Figures 21A and 21B. ONNX versions can also be generated and encoded into bitstreams, making available the associations between the first and second network portions of each ONNX version, as described in relation to Figures 21A and 21B. Figure 21A is a schematic block diagram representing NN instructions 2100, and Figure 21B is a schematic block diagram showing associations 2150 between NN instructions. The NN directive includes an informal network name 2110 as a string, an ONNX version 2112 as an integer, an NN Part 1 topology hash 2114, an NN Part 1 weight hash 2115, an NN topology hash 2116, an NN weight hash 2117, a list 2118 of inputs to the network (i.e., inputs to NN Part 1) identified by the enumerated input node names in the ONNX refactoring representation along with the dimensions of each input tensor, a list 2119 of outputs from NN Part 1 (corresponding to inputs to NN Part 2), a list 2120 of outputs from the network (i.e., outputs from NN Part 2) identified by the enumerated output node names in the ONNX refactoring representation along with the dimensions of each output tensor, and an associated hash 2122. Hashes 2114 and 2115 identify NN Part 1 used to generate the tensor encoded in the feature subbitstream 143. Hashes 2116 and 2117 identify the networks (NN Part 1 and proxy NN Part 2) used to generate the proxy task result 119. Hashes 2114-2117 are 256-bit fixed-length codewords. The associated hash 2122 contains zero or more pairs of hashes that identify the weights and topology of compatible preceding neural networks.If instruction 2100 is for NN part 2 166, it includes at least one preceding association, namely, the preceding association that identifies NN part 1 114. The association hash 2122 also includes zero or more pairs of hashes that identify the weights and topology of compatible subsequent NNs. If instruction 2100 is for NN part 1 114, it includes at least one subsequent association, typically the subsequent association that identifies the proxy NN part 2 118. The container hash, if present (not shown), includes hashes of topology and weights that identify the combined NN part 1 114 and proxy NN part 2 118. The container hash included with NN part 1 and proxy NN part 2 118 identifies the entire network used to generate the proxy task result 119 at source device 110.

[0198] The container hash included with NN Part 2 166 identifies the entire network used to generate task result 151 at destination device 140. Thus, hashes of network topology and network weights provide a mechanism to identify which network was used to generate the task result and to connect compatible network parts to each other when executed in a distributed manner. Figure 21B shows the proxy NN directives, NN Part 1 directives, and proxy NN Part 2 directives available at source device 110. The NN Part 1 directive is signaled to destination device 140, which includes NN#1-4 directives and NN Part 2#1-4 directives, each indicating compatibility (i.e., matching network topology and network weight hash values) with the signaled NN Part 1 directive. Also shown are NN#5 and NN Part 2#5, which are incompatible with the NN Part 1 directive and therefore do not indicate association with NN Part 1. As explained with reference to Figure 19, the absence of a candidate for NN Part 2 with a compatible leading hash to the NN Part 1 instruction signaled in the multi-layer bitstream 123 means that the video sub-bitstream 171 can still be used, but the tensor decoded from the feature sub-bitstream 143 cannot be used. The existence of one or more compatible candidates for NN Part 2 makes it possible to run one of the candidates as NN Part 2 150.

[0199] In one configuration of System 100, the network weights and network topology are hashed together in a single hash operation, avoiding the need to perform separate hashes on the topology and weights. When the network topology and weights are hashed together as one, it becomes impossible to identify cases where compatible topologies and incompatible weights are used, but a reduction in memory consumption and simplification of compatibility decisions are achieved.

[0200] In one configuration of source device 110 and destination device 140, steps 1850 and 19160 are omitted, and as a result, modules 116 and 146 perform video compression to feature bitstream 121 on feature maps from tensors directly obtained from NN part 1 114 and supplied to NN part 2 166, without the use of additional network layers or low-rank approximations. Omitting steps 1850 and 19160 reduces the complexity of source device 110 and destination device 140 but increases the bitrate of the multi-layer bitstream 123.

[0201] In one configuration of source device 110, the feature bitstream 121 is included in the multilayer bitstream 123 only if the proxy NN part 2 task result 119 indicates some positive result, such as object detection in the frame data 113. The inclusion of the video bitstream 151 may also be conditional on the presence of the task result 119. Conditional transmission of bitstreams 121 and 151 reduces the required bandwidth of the communication channel 130.

[0202] In one configuration of the source device 110, a proxy task result 119 is generated and supplied to the video encoder 150 to influence encoding operations such as providing high quality to the region of frame data 113 containing the detected object, or blurring the portion of frame data 113 that does not contain the object of interest. The nn_optimized_video_bitstream_flag syntax element is encoded in the SEI message to indicate whether the video bitstream was encoded using such a proxy task result as input. The signaling of the proxy task network included in the SEI message allows the decoder to determine which network was used to generate the proxy task result for use by the video encoder 150.

[0203] In one configuration of the source device 110 and the destination device 140, an existing video bitstream conforming to a standard such as the HEVC Main profile or the VVC Main10 profile retains the original compressed video as a base layer, and with a feature layer corresponding to the output of the specified NN Part 1 network added as an enhancement layer, is rewritten as a HEVC Main Multilayer profile or VVC Main10 Multilayer profile bitstream. The rewriting of the bitstream may include insertion or modification of the VPS 1115, insertion of the SDI SEI 1112, insertion of the FCVCM SEI 1113, and rewriting and / or insertion of the SPS 1110a and 1110b. Rewriting the bitstream enables the execution of NN Part 1 to be performed at a time point prior to the execution of one of the NN Part 1 candidates, and essentially reduces the execution burden of completing the execution of the task network at the time point when the calculation of the task result 167 is required.

[0204] In one configuration of the source device 110 and the destination device 140, the multi-layer bitstream 123 conforms to the HEVC Multilayer profile, and for example when the proxy NN Part 2 result 119 indicates no object detection, the base layer (i.e., the video layer) is excluded from the bitstream and from any other external means that carries the base layer from the source device 110 to the destination device 140. The VPS 1115 signals that the base layer (e.g., 1116) does not exist by setting vps_base_layer_internal_flag to 1 and vps_base_layer_available_flag to 0. Even when the base layer is excluded from the multi-layer bitstream 123, the SPS 1110b is still included and signals as if the base layer exists (e.g., 1116).

[0205] In one configuration of the source device 110 and the destination device 140, NN Part 1 and NN Part 2 are Meta TMCorresponding to the encoder and decoder portions of "vision transformer" networks such as the "Segment Anything Model" (SAM), the split point (boundary between NN parts 1 and 2) occurs at the boundary between the encoder and decoder, which is also the narrowest stage of the bottleneck architecture. When using SAM, the destination device 140 decodes a segmentation map corresponding to an object selected by one or more points or bounding boxes (collectively referred to as "prompts") provided, for example, by the user of the destination device 140. Since NN part 1 (SAM encoder) requires only frame data 113 as input, NN part 1 may be run once and its result encoded as part of a multi-layer bitstream 123 along with the video data, and NN part 2 may be run each time a different prompt is made by the user of the destination device 140.

[0206] In one configuration of source device 110 and destination device 140, the FCVCM SEI message 1113 includes network topology instructions and network weight instructions corresponding to tensor encoder 116 and / or tensor decoder 146, such as when tensor encoder 600 and tensor decoder 1500 are used. The instructions signaling tensor decoder 146 allow destination device 140 to select and instantiate an appropriate network from a repository of candidate tensor decoders to be used as tensor decoder 146.

[0207] While the configuration describes separate instructions for network topology and network weights, the configuration may also signal network topology and network weights together in a single instruction. Signaling topology and weights in a single instruction eliminates the need to filter information from the ONNX representation.

[0208] Although source device 110 and destination device 140 are described with reference to video source 112 containing video and image data, other types of content such as audio data and text data may also be provided as input to a neural network applicable to such types of input, and the resulting intermediate feature maps may be compressed and decompressed by modules 116 and 146 with appropriate encoder and decoder network topologies.

[0209] Industrial applicability The described configuration is applicable to the computer and data processing industries, and is particularly applicable to digital signal processing for encoding and decoding signals such as video and image signals, achieving high compression efficiency.

[0210] Some embodiments described herein encode instructions regarding the topology and weights of a CNN into a bitstream. Encoding information about the topology and weights provides a mechanism for identifying which network head or NN part 2 option is compatible for use with the tensors provided when decoding the bitstream, enabling implementations to adapt to changes in CNN trends and architectures.

[0211] Some embodiments described herein may include both the video layer and the feature layer in the bitstream. Including both the video layer and the feature layer in the bitstream provides a mechanism for humans to review or interpret the results of the CNN.

[0212] The above describes only some embodiments of the present invention, and modifications and / or changes can be made without departing from the scope and spirit of the invention. The embodiments are illustrative and not limiting.

[0213] Appendix A The following is an exemplary SEI message format and associated semantics for representing metadata related to the process of decoding tensors:

[0214] [Table 1]

[0215] Here, u(n) refers to a fixed-length codeword of n bits, and ue(v) refers to an unsigned exponential Golomb variable-length codeword.

[0216] FCVCM decoder information semantics: A flag of fcvcm_network_indication_cancel_flag of 1 indicates that this SEI message cancels the persistence of any previous FCVCM network indication SEI messages in output order applied to the current layer. A flag of fcvcm_network_indication_cancel_flag of 0 indicates that alpha channel information follows the bitstream.

[0217] The information contained in an FCVCM network instruction information SEI message persists for access units starting from the current picture (i.e., the picture to which the prefix SEI message is associated) and continues until a new instance of the FCVCM network instruction information SEI message, the end of a bitstream, or the start of a new coding layer video sequence (CLVS).

[0218] `network_informal_name` is a null-terminated UTF-8 string that specifies an informal name used to identify the network from which the included task results were generated. This informal name is for reference only and is not used in the decryption process.

[0219] `onnx_version` is an unsigned integer that specifies the ONNX byte format used to preserve the serialized network topology and / or weights before hashing.

[0220] A flag of nn_part1_hashes_present_flag being 1 indicates that the bitstream contains a hash that identifies NN part 1, which was used to generate the tensors contained within the feature bitstream.

[0221] nn_part1_topology_hash encodes the 256-bit sha1sum of the refactored ONNX representation of the NN part 1 topology.

[0222] nn_part1_weights_hash encodes the 256-bit sha1sum of the NN part 1 weights obtained from the refactored ONNX representation.

[0223] nn_hashes_present_flag being 1 indicates that the bitstream contains a hash that identifies the NN used to generate the proxy network result, if present in the bitstream.

[0224] nn_topology_hash encodes the 256-bit sha1sum of the refactored ONNX representation of the NN topology.

[0225] nn_weights_hash encodes the 256-bit sha1sum of the NN weights obtained from the refactored ONNX representation.

[0226] A value of nn_optimized_video_bitstream_flag of 1 indicates that the video bitstream 151 was encoded using the proxy task result 119 as a guide, while a value of 0 indicates that the video bitstream 151 was generated independently of the proxy task result 119.

[0227] nn_inputs_count specifies the number of tensors provided as input to the neural network (NN). If nn_inputs_count is set to zero, the number and dimensions of tensors provided as input to the NN are unspecified. See nn_inputs_count and its dependent syntax elements from 2118.

[0228] nn_input_name[i] specifies the node name that the i-th tensor will provide to the neural network as a null-terminated UTF-8 string.

[0229] nn_input_feature_cnt[i] specifies the number of feature maps (channels) for the i-th input tensor.

[0230] nn_input_feature_width[i] specifies the width of the feature map of the i-th input tensor.

[0231] nn_input_feature_height[i] specifies the height of the feature map of the i-th input tensor.

[0232] nn_part1_outputs_count specifies the number of tensors provided as outputs from NN part 1. If nn_part1_outputs_count is set to zero, the number and dimensions of tensors provided as outputs from NN part 1 are unspecified. nn_part1_outputs_count and its dependent syntax elements form 2119.

[0233] nn_part1_input_name[I] specifies the node name output from NN part 1 as the i-th tensor, as a null-terminated UTF-8 string.

[0234] nn_part1_input_feature_cnt[i] specifies the number of feature maps (channels) for the i-th NN part 1 output tensor.

[0235] nn_part1_input_feature_width[i] specifies the width of the feature map of the i-th NN part 1 output tensor.

[0236] nn_part1_input_feature_height[i] specifies the height of the feature map in the i-th NN part 1 output tensor.

[0237] nn_outputs_count specifies the number of tensors generated as outputs from the neural network (NN). If nn_output_count is set to zero, the number and dimensions of tensors provided as input to the NN are unspecified. nn_outputs_count and its dependent syntax elements form 2120.

[0238] nn_output_name[i] specifies the node name obtained from the neural network as the i-th tensor, as a null-terminated UTF-8 string.

[0239] nn_output_feature_cnt[i] specifies the number of feature maps (channels) for the i-th output tensor.

[0240] nn_output_feature_width[i] specifies the width of the feature map of the i-th output tensor.

[0241] nn_output_feature_height[i] specifies the height of the feature map of the i-th output tensor.

[0242] The preceding_association_flag specifies the existence of a hash indicating the NN portion (i.e., NN part 114) used to generate the tensor encoded in feature stream 121.

[0243] `preceding_association_topology_hash` specifies the sha1sum of the refactored ONNX representation of the network topology performed as NN part 1 114.

[0244] `preceding_association_weights_hash` specifies the sha1sum of the weights from the ONNX representation of NN part 1 114.

[0245] The container_association_present_flag specifies the existence of the NN (i.e., the combined NN part 1 114 and the proxy NN part 2 118) used to generate the proxy task result 119.

[0246] `container_association_topology_hash` specifies the sha1sum of the refactored ONNX representation of the network topology as performed as NN part 1 114 coupled with proxy NN part 2 118 as an unpartitioned network.

[0247] `container_association_weights_hash` specifies the sha1sum of weights from the ONNX representation of the proxy NN part 2 118 as an undivided network and the coupled NN part 1 114.

[0248] `proxy_task_result_length` specifies that if it is zero, no proxy task results are present in multilayer bitstream 123, and if it is non-zero, it specifies the length in bytes of the proxy task results contained in multilayer bitstream 123.

[0249] `proxy_task_result_byte[i]` specifies the i-th byte of the payload containing the proxy task result 119. Examples of proxy task results include a 32-bit floating-point number encoding the mAP or MOTA score, or a segmentation map compressed using, for example, run-length encoding.

Claims

1. A method for decoding a tensor and video from a bitstream including a first sub-bitstream and a second sub-bitstream, wherein the tensor is related to the video, and the method is Decoding a video parameter set indicating the presence of multiple sub-bitstreams within the bitstream, Decoding the video from the first sub-bitstream obtained by performing a first sub-bitstream extraction process on the bitstream, Decoding the tensor for the decoded video from the second sub-bitstream obtained by performing a second sub-bitstream extraction process on the bitstream, A method that includes this.

2. The method according to claim 1, wherein a HEVC Multilayer Main profile is used to decode the video parameter set in a 4:2:0 chroma format and 8-bit depth for both the first sub-bitstream and the second sub-bitstream, and a neutral value is used for the chroma sample of the second sub-bitstream, wherein the neutral value is 128.

3. The method according to claim 1, wherein a VVC Multilayer Main 10 profile is used to decode the video parameter set, the VVC Multilayer Main 10 profile having a 4:2:0 chroma format for the video layer and a 4:0:0 chroma format for the feature layer.

4. The method according to claim 1, wherein the bitstream may include a plurality of second sub-bitstreams.

5. The method according to claim 1, wherein the first sub-bitstream extraction process filters the bitstream based on a first layer identifier to obtain the first sub-bitstream from the bitstream, and the second sub-bitstream extraction process filters the bitstream based on a second layer identifier to obtain the second sub-bitstream from the bitstream, and each of the first and second sub-bitstreams can be decoded independently.

6. The method according to claim 1, further comprising using a portion of the neural network to determine the task result of the decoded tensor.

7. The method according to claim 6, further comprising rendering the task results in association with the video.

8. A decoder for decoding a tensor and video from a bitstream including a first sub-bitstream and a second sub-bitstream, wherein the tensor is related to the video, and the decoder Decode the video parameter set indicating the presence of multiple sub-bitstreams within the bitstream, The video is decoded from the first sub-bitstream obtained by performing a first sub-bitstream extraction process on the bitstream. The tensor for the decoded video is decoded from the second sub-bitstream obtained by performing a second sub-bitstream extraction process on the bitstream. A decoder configured in such a way.

9. A non-temporary, computer-readable storage medium for storing a program for performing a method of decoding tensors and video from a bitstream including a first sub-bitstream and a second sub-bitstream, wherein the tensors are related to the video, and the method is Decoding a video parameter set indicating the presence of multiple sub-bitstreams within the bitstream, Decoding the video from the first sub-bitstream obtained by performing a first sub-bitstream extraction process on the bitstream, Decoding the tensor for the decoded video from the second sub-bitstream obtained by performing a second sub-bitstream extraction process on the bitstream, A storage medium that includes this.

Images