Method, apparatus and system for encoding and decoding tensor

Abstract

Systems and methods for decoding a tensor from a bitstream for use by a second portion of a network. The second network portion is a second portion of the neural network. The method comprises: determining a network first part topology indication from a bitstream; and determining a network first partial weight indication from the bitstream. The method further includes determining whether to decode the tensor from the bitstream with the network second portion based on the network first portion topology indication and the network first portion weight indication. In case the tensor is determined to be decoded from the bitstream, the method includes decoding the tensor from the bitstream for a second portion of the network.

Description

[0001] Citation of relevant applications

[0002] This application claims the benefit of Australian Patent Application 2023248076, filed on October 10, 2023, pursuant to 35 USC §119, which is incorporated herein by reference in its entirety as set forth herein. Technical Field

[0003] This invention generally relates to digital video signal processing, and more particularly, to methods, apparatus, and systems for encoding and decoding tensors from convolutional neural networks. The invention also relates to a computer program product comprising a computer-readable medium having recorded computer programs for encoding and decoding tensors from convolutional neural networks using video compression techniques. Background Technology

[0004] Convolutional Neural Networks (CNNs) are an emerging technology for addressing use cases involving machine vision, such as object detection, instance segmentation, object tracking, human pose estimation, and action recognition. CNN applications can involve the use of "edge devices" with sensors and some processing capabilities, coupled to application servers as part of the "cloud." CNNs can require relatively high computational complexity, exceeding what is typically available in terms of the computational capacity or power consumption of edge devices. Implementing CNNs in a distributed manner has become a solution for running front-end networks using edge devices with limited capabilities, without requiring all computational complexity to be generated within cloud servers, while edge devices may have underutilized inference resources. In other words, distributed processing allows traditional edge devices to still provide the capabilities of front-end CNNs by distributing processing between the edge device and other processing components such as cloud servers. This distributed network architecture can be called "cooperative intelligence (CI)" and offers benefits such as reusing partial results from the first part of the network for several different second parts (potentially optimized for different tasks). CI architecture introduces the need for efficient compression of tensor data for transmission over networks such as WANs.

[0005] CNNs typically consist of many layers, such as convolutional layers and fully connected layers, where data is passed from one layer to the next in the form of "tensors." Splitting the network across different devices introduces the need to compress the intermediate multidimensional tensor data passed from one layer to the next within the CNN to facilitate transmission on networks with bandwidth constraints or high costs. Such tensor compression can be called "feature compression," and the intermediate tensor data is often referred to as "features" or "feature maps." Figure 1A tensor is generally a collection of two-dimensional (2D) arrays of values ​​that, when combined into a 3D (or 4D) data structure, form a tensor, where each feature map corresponds to a "channel" of the tensor. Intermediate tensor data represents a partially processed form of inputs such as image frames or video frames encountered within a neural network. The ISO / IEC JTC1 / SC29 / WG2 (also known as the Moving Picture Experts Group (MPEG)) technical requirements were tasked with researching compression techniques in various contexts, often related to video. WG2's "MPEG Technical Requirements" established an ad-hoc group for "Feature Compression in Machine Video Coding" (FCVCM), commissioned to study feature compression. The FCVCM AHG has issued a "Call for Proposals" to form the basis for standardization projects related to feature compression. Previously, responses to the "Call for Evidence" (CfE) demonstrated techniques that could significantly outperform those using state-of-the-art standardization techniques applied directly to tensors.

[0006] CNNs typically require pre-determining the weights for each layer during the training phase. During training, a very large amount of training data is passed through the CNN, and the results determined by the trained network are compared to the ground truth associated with the training data. The difference between the obtained result and the expected result is represented as the "loss" and measured using a "loss function." Using the determined loss, processes (such as stochastic gradient descent (SGD)) are performed to update the network weights. Network weight updates typically involve a backpropagation process of "gradients," which starts at the network's output layer and proceeds backward to terminate when the input layers of the network are updated, thus covering the network's intermediate or "hidden" layers. The gradient indicates the difference (delta) to be applied to the network weights and is updated itself as part of the backpropagation process. The rate of weight updates is set by a "learning rate" hyperparameter, which is typically set to allow the training process to find a global minimum in terms of loss (i.e., the highest possible task performance for the network architecture and training data), while avoiding the training process "stuck" in local minima. Getting stuck in a local minimum corresponds to achieving suboptimal task performance for the network architecture and being unable to find new weight values ​​that lead to higher task performance. The network performance is iteratively refined by repeatedly updating the network weights by feeding them into "batches" of input data and ground truth data until no further improvement in accuracy can be achieved. An "epoch" of training is formed by iterating over the entire training dataset, and training typically requires multiple epochs to achieve high-level performance for the task. The weights of the trained network are then available for deployment, and the network operates in a mode where the weights are fixed and gradient updates for the weights are omitted. The process of executing a pre-trained CNN using its inputs and progressively transforming the inputs into outputs based on the CNN's topology is often referred to as "inference."

[0007] Typically, a tensor has four dimensions: batch, channels, height, and width. The first dimension, "batch," is typically size 1 when inferring video data and indicates that one frame is passed through the CNN as a batch. As the network is trained, the value of the batch dimension can be increased so that multiple frames in each batch are passed through the network before updating the network weights, according to a predetermined "batch size." Multi-frame video can be passed through as a single tensor with a batch dimension whose size increases with the number of frames in the given video. However, for practical considerations related to memory consumption and access, inference of video data is typically performed frame by frame. The "channels" dimension indicates the number of concurrent "feature maps" for a given tensor, and the height and width dimensions indicate the size of the feature maps at a specific stage of the CNN. The channel count varies depending on the network architecture as it passes through the layers of the CNN. The size of the feature maps also varies depending on whether subsampling or upsampling occurs in a particular network layer.

[0008] CNNs tend to have relatively high overall complexity due to the large number of multiply-accumulate (MAC) operations, the large number of intermediate tensors written to and read from memory, and the reading of weights for the performance of each layer. Therefore, splitting the neural network into parts allows for the implementation of more complex networks even in systems with less capable edge devices, without requiring cloud servers to bear the full burden of the network.

[0009] Feature compression can benefit from existing video compression standards, such as Universal Video Coding (VVC) developed by the Joint Video Experts Group (JVET). VVC is expected to address the ongoing demand for even higher compression performance, particularly as video formats become more capable (e.g., with higher resolution and higher frame rates), and the growing market demand for service delivery over WANs (where bandwidth costs are relatively high). VVC can be implemented in contemporary silicon processes, offering an acceptable trade-off between the achieved performance and implementation cost. Implementation cost can be considered in terms of one or more aspects such as silicon area, CPU processor load, memory utilization, and bandwidth. Other video compression standards, such as High Efficiency Video Coding (HEVC) or AV-1, can also be used for feature compression applications.

[0010] Video data consists of a sequence of image frames, each containing one or more color channels. When representing feature map data within packed frames, monochrome frames with only luminance channels and no chrominance channels are generally sufficient. When only luminance samples exist, the resulting monochrome frames are considered to use a "4:0:0 chroma format".

[0011] The VVC standard specifies a "block-based" architecture, where frames are first divided into an array of square regions called "Code Tree Units" (CTUs). In VVC, a CTU typically occupies 128×128 luma samples. Other possible CTU sizes when using the VVC standard are 32×32 and 64×64. However, CTUs at the right and bottom edges of each frame may be smaller in area, where implicit splitting occurs to ensure that coded blocks remain within the frame. Associated with each CTU is a "code tree" (also called a "code unit" (CU)) that defines the region of the CTU as a set of blocks. Blocks applicable to luma channels only or chroma channels only are called "coded blocks" (CBs). Predictions of the contents of a coded block are maintained in a "prediction block" (PB) or "prediction unit" (PU), and the residual block defining the array of sample values ​​to be added to and combined with the PB or PU is called a "transform block" (TB) or "transform unit" (TU), due to the transform processing typically used in the generation of TBs or TUs.

[0012] Despite the aforementioned distinction between “unit” and “block”, the term “block” can be used as a general term to refer to an area or region of a frame for which the operation is applied across all color channels.

[0013] For each CU, a prediction unit (PU) (“prediction unit”) is generated for the content (sample values) of the corresponding region of the generated frame data. Furthermore, a representation of the difference (or “spatial domain” residual) between the prediction and the content of the region seen at the encoder input is formed. The differences in each color channel can be transformed and encoded into a sequence of residual coefficients, forming one or more TUs for a given CU. The applied transformation can be a Discrete Cosine Transform (DCT) or other transformation applied to the blocks of residual values. The transformation is applied individually (i.e., a two-dimensional transformation is performed twice (once horizontally and once vertically)). The block is first transformed by applying a one-dimensional transformation to each row of samples in the block. Then, the partial result is transformed by applying a one-dimensional transformation to each column of the partial result to produce a final block of transform coefficients that are substantially decorrelated to the residual samples. The VVC standard supports transforms of various sizes, including transforms of rectangular blocks where each side dimension is a power of 2. The transform coefficients are quantized for entropy encoding into the bitstream.

[0014] Intra-frame prediction or inter-frame prediction processing can be used to generate PB or PU in VVC. Intra-frame prediction involves using previously processed samples from the frame being used to generate a prediction of the current data sample block in that frame. Inter-frame prediction involves using sample blocks obtained from one or two previously decoded frames to generate a prediction of the current sample block in the frame. The sample blocks obtained from the previously decoded frames are offset relative to the spatial location of the current block according to a motion vector, which is typically filtered. Intra-frame prediction blocks can be: (i) uniform sample values ​​(“DC intra-frame prediction”), (ii) a plane with offsets and horizontal and vertical gradients (“planar intra-frame prediction”), (iii) a total of blocks with neighboring samples applied in a specific direction (“angular intra-frame prediction”), or (iv) the result of a matrix multiplication using neighboring samples and selected matrix coefficients.

[0015] VVC can be used to compress intermediate feature maps from the first part (“backbone”) of a neural network that is split into two parts. In compression, feature maps from the backbone are arranged into frames and quantized from the floating-point domain to a sample domain suitable for compression as video data. CNNs represent rapidly evolving technology, and as technology advances, CNNs for machine vision tasks can be updated with different versions or replaced with new CNN architectures. A bitstream containing tensors from the backbone (CNN first part) can be decoded and fed to one or more of the many available network heads (CNN second part). There is a need to identify which network head(s) are compatible with the tensors provided from decoding the bitstream.

[0016] Machine vision tasks performed using CNNs can provide outputs based on the specific head network used. The results can be correlated with things like bounding boxes and segmentation labels. In some instances, humans may want to review or interpret CNN results. In some use cases, there is a need to assist humans in interpreting CNNs. Summary of the Invention

[0017] The object of the present invention is to substantially overcome or at least improve one or more disadvantages of the existing arrangement.

[0018] One aspect of this disclosure provides a method for decoding a tensor from a bitstream for use by a second portion of a network, the method comprising: determining a topology indication of a first portion of the network from the bitstream; determining a weight indication of the first portion of the network from the bitstream; determining, based on the topology indication and the weight indication, whether to decode a tensor from the bitstream; and, if it is determined that a tensor should be decoded from the bitstream, decoding the tensor from the bitstream for the second portion of the network.

[0019] Another aspect of this disclosure provides a method for generating a content-oriented neural network result from a bitstream, the method comprising: determining a first part topology indication of a network from the bitstream; determining a first part weight indication of a network from the bitstream; determining whether to perform a second part of a network based on the first part topology indication and the first part weight indication; and, if it is determined that the second part of the network should be performed, performing the second part of the network using tensors decoded from the bitstream to generate the neural network result.

[0020] Another aspect of this disclosure provides a decoder for decoding tensors from a bitstream for use by a second part of a network, the decoder being configured to: determine a network first part topology indication from the bitstream; determine a network first part weight indication from the bitstream; determine whether to decode a tensor from the bitstream based on the network first part topology indication and the network first part weight indication; and, if it is determined that a tensor should be decoded from the bitstream, decode the tensor from the bitstream for the second part of the network.

[0021] Another aspect of this disclosure provides a non-transitory computer-readable storage medium storing a program for performing a method for decoding a tensor from a bit stream for use by a second portion of a network, the method comprising: determining a topology indication of a first portion of the network from the bit stream; determining a weight indication of a first portion of the network from the bit stream; determining, based on the topology indication and the weight indication, whether to decode a tensor from the bit stream; and, if it is determined that a tensor should be decoded from the bit stream, decoding the tensor from the bit stream for the second portion of the network.

[0022] Another aspect of this disclosure provides a decoder for generating a content-based neural network result from a bitstream, the decoder being configured to: determine a first part of the network topology indication from the bitstream; determine a first part of the network weight indication from the bitstream; determine whether to perform a second part of the network based on the first part of the network topology indication and the first part of the network weight indication; and, if it is determined that the second part of the network should be performed, perform the second part of the network using tensors decoded from the bitstream to generate the neural network result.

[0023] Another aspect of this disclosure provides a non-transitory computer-readable storage medium storing a program for executing a decoding method for generating a neural network result for content from a bitstream, the method comprising: determining a first part topology indication of a network from the bitstream; determining a first part weight indication of a network from the bitstream; determining whether to perform a second part of a network based on the first part topology indication and the first part weight indication; and, if it is determined that the second part of the network should be performed, performing the second part of the network using a tensor decoded from the bitstream to generate the neural network result.

[0024] Other aspects were also disclosed. Attached Figure Description

[0025] At least one embodiment of the invention will now be described with reference to the following figures and appendices, wherein:

[0026] Figure 1 This is a schematic block diagram illustrating a distributed machine task system;

[0027] Figure 2A and Figure 2B Formation that can be put into practice Figure 1 A schematic block diagram of a general-purpose computer system for distributed machine task systems;

[0028] Figure 3A This is a schematic block diagram showing the functional modules of the main body of a CNN;

[0029] Figure 3B It is shown Figure 3A A schematic diagram of the residual block;

[0030] Figure 3C It is shown Figure 3A A schematic block diagram of the residual element;

[0031] Figure 3D It is shown Figure 3A A schematic block diagram of the CBL module;

[0032] Figure 4 This is a schematic block diagram illustrating the functional modules of a CNN that can be replaced by a mainframe.

[0033] Figure 5 This is a schematic block diagram of a tensor encoder using a configurable tensor compressor stage;

[0034] Figure 6 This is a schematic block diagram illustrating the multi-scale feature fusion stage of a tensor compressor;

[0035] Figure 7 This is a schematic block diagram showing a channel-decorrelated tensor compressor;

[0036] Figure 8 This is a schematic block diagram showing the functional modules of a video encoder;

[0037] Figure 9A and Figure 9B This is a schematic block diagram showing the arrangement of regions or sub-pictures used to hold compressed tensor data;

[0038] Figure 10A and Figure 10B This is a schematic block diagram showing another arrangement for holding regions or sub-pictures of compressed tensor data;

[0039] Figure 11 This is a schematic block diagram showing the bitstream that retains the decorrelation feature maps between encoded channels, the compressed video, and the associated metadata;

[0040] Figure 12 This is a schematic block diagram showing a tensor decoder with a configurable tensor decompressor;

[0041] Figure 13 This is a schematic block diagram illustrating the functional modules of the video decoder;

[0042] Figure 14 This is a schematic block diagram illustrating an inter-channel decorrelation-based tensor decoder as part of a distributed machine task system.

[0043] Figure 15 This is a schematic block diagram illustrating an embodiment of the multi-scale feature reconstruction stage;

[0044] Figure 16A This is a schematic diagram showing the head of a CNN;

[0045] Figure 16B It is shown Figure 16A A schematic block diagram of the upgrader module;

[0046] Figure 16C It is shown Figure 16A A schematic block diagram of the detection module;

[0047] Figure 17 This is a schematic diagram showing alternative heads for CNNs;

[0048] Figure 18 This illustrates a method for performing the first part of a CNN, compressing the resulting tensors, and signaling instructions to the first part of the CNN.

[0049] Figure 19 This illustrates a method for decoding a bitstream, reconstructing a tensor, and completing a task based on network topology and network weights.

[0050] Figure 20This illustrates a method for determining network indications of the entire network and its parts;

[0051] Figure 21A It is a schematic block diagram depicting the content of the NN instruction;

[0052] Figure 21B This is a schematic block diagram illustrating the relationship between NN indicators; and

[0053] Appendix A shows the retention Figure 19 The syntax table of the "Supplemental Enhancement Information" (SEI) message, which contains the metadata required for the operation of the decoding method. Detailed Implementation

[0054] Where steps and / or features with the same reference numerals are referenced in any one or more figures, unless the contrary is intended, these steps and / or features have the same (one or more) function or (one or more) operation for the purposes of this specification.

[0055] Distributed machine task systems can include edge devices, such as webcams or smartphones that generate intermediate compressed data. Distributed machine task systems can also include end devices, such as server farm-based (“cloud”) applications that manipulate the intermediate compressed data to produce task results. Furthermore, edge device functionality can be embodied in the cloud, and the intermediate compressed data can be stored for later processing (potentially for multiple different tasks as needed). Distributed task systems can perform tasks in a general form at the edge devices and provide partially processed features on the server side for performing tasks in a specific form. For example, a general object detection network can be performed on the client (edge) device, and the results can be used on the server to conditionally perform a more specific detection network using, for example, partially processed tensors generated by a first part of a CNN common to both the general task network and the specific task network. Such a first part can be referred to as a “shared backbone” because it is common to multiple task networks in both network topology and network weights.

[0056] A convenient form of intermediate compressed data is compressed video bitstreams, thanks to the availability of high-performance compression standards and their implementations. Video compression standards typically operate on integer samples arranged in planar arrays with a given localization depth (such as 10 bits). Depending on the application, color video has, for example, three planar arrays corresponding to the color components Y, Cb, Cr or R, G, B. CNNs typically operate on floating-point data in the form of tensors. Tensors generally have a relatively smaller spatial dimension compared to the input video data on which CNNs operate, but have more channels than the typical three channels of color video data (e.g., 128, 256, or 512 channels).

[0057] Tensors typically have the following dimensions: frame, channel, height, and width. For example, a tensor of dimensions [1, 256, 76, 136] would be considered to contain a single frame of floating-point or integer values, which comprises an array of 256 (256) feature maps (channels), each of size 136 × 76. For video data, inference is typically performed one frame at a time (frame value is 1), rather than using tensors containing multiple frames.

[0058] VVC supports segmenting an image into multiple sub-images, each of which can be independently encoded and decoded. In one approach, each sub-image is encoded as a "slice" or a continuous sequence of encoded CTUs. A "tile" mechanism can also be used to segment an image into multiple independently decodeable regions. Sub-images can be specified in a slightly more flexible way, where various rectangular sets of CTUs are encoded as corresponding sub-images. The flexible definition of sub-image dimensions allows for the efficient preservation of data types requiring different regions within an image, thus avoiding large "unused" regions (i.e., regions in the frame not used for tensor data reconstruction).

[0059] Figure 1 This is a schematic block diagram illustrating the functional modules of a distributed machine task system 100 capable of performing machine task networks in a distributed manner. Dividing a particular neural network into two parts requires specifying a “split point” in the network. In a first device (or “source device”), layers in the network from the input layer up to the split point are executed, and the resulting (one or more) intermediate tensors are compressed. The layers up to the split point can be referred to as the “stem,” however, this term can sometimes imply a specific split point. The alternative term “NN part 1” (neural network part one, also called the first network part) does not imply the location of the split point in the network. The layers from the split point to the last layer can be referred to as the “head,” and to avoid any implied split point location, can alternatively be referred to as “NN part 2” (neural network part 2, also called the second network part). The first NN part 2 (also called surrogate NN part 2) executes the remaining layers of the network to produce initial task results for inclusion in a bitstream for later use. Based on the results obtained from the network, in the second device (or "destination device"), one or more decompressed tensors from the first device are used as inputs to one or more layers immediately following the split point to replace the layers in the replacement network up to the last layer. The replacement network has a common NN part 1 with the first network, and thus can produce task result 167 by using tensors generated by the NN part 1 of the first network to perform NN part 2.

[0060] At a split point, there may be one or more tensors that need to be compressed to be transmitted over a communication channel with limited bandwidth compared to the bandwidth requirements for transmitting uncompressed tensors. In the case of Feature Pyramid Networks (FPNs) in use, it is common to make the layers in the FPN correlated in width and height, such that a given layer is half the width and height of its adjacent layer. FPN architectures can also involve alternating halving of width and height from one layer to the next. In some architectures, multiple tensors of the same width and height are generated within the FPN. FPNs may occur relatively early in a neural network topology, leading to the need for split points within the FPN to achieve useful partitioning of network workloads across edge devices and the cloud. When splits occur within the FPN of a machine-task network, the performance of various machine-task networks can be achieved, where the layers up to the split point are common between these machine-task networks (“shared backbone” architecture). In the case of split points occurring within the FPN, tensor compression methods can leverage redundancy across FPN layers to improve compression performance. Therefore, compression methods applicable to the various network topologies used in contemporary CNNs are beneficial for applications in a wide range of scenarios.

[0061] System 100 can be used to implement methods for decorrelation, packing, and quantization of feature maps into planar frames to encode feature maps and decode feature maps from encoded data from various neural networks. Various neural networks can be split at different points and can produce intermediate tensors of varying numbers and dimensions. The large number of possible neural networks and split points creates a need for destination device 140 to determine compatibility with a repository of available NN Part 2 implementations available in NN Part 2 repository 160. Enumerating potential neural networks is difficult due to the need to maintain a centralized and consensus-based repository of network names. Furthermore, reference implementations may only be available in transient form (such as public software repositories, which may change with commit updates). This leads to additional complications, such as those seen in the “YOLO” network lineage, where multiple different researchers may claim to have released the next version of YOLO in that lineage, due to the machine learning community's failure to reach a consensus on naming newly created networks. Open Neural Network Exchange (ONNX) is a format that provides a device-independent way to specify the weights and topology of a neural network. After removing the weights, the ONNX topology provides a relatively lightweight complete representation of the neural network topology. Signaling the entire topology to identify network scalability may become insufficient as network topologies increase. Furthermore, a more concise indication of the network topology would be sufficient to uniquely identify the network without referencing any potentially unstable public resources. Even citing papers is often insufficient, as papers will reference public software repositories (such as those hosted on github.com). Therefore, to achieve interoperability between different source and destination devices, a signaling component with explicitly defined compatibility between NN Part 1 and one or more NN Part 2 implementations is required. In particular, mechanisms that do not rely on centralized enumeration (such as registries) are preferred, as establishing, maintaining, and facilitating the use of such registries is typically cumbersome and a barrier to industry adoption. Alternatively, a simpler signaling component compared to sending the entire network weights and / or topology is an easier-to-deploy alternative, avoiding the need for an externally maintained and accredited authoritative registry of network weights and / or topology.

[0062] In some implementations, system 100 can transmit both intermediate compressed features from NN section 1 and compressed video from a video source from a source device to a destination device. Including both video and compressed features enables use cases such as overlaying task results (e.g., bounding boxes of specific objects) onto the video for human review. Another use case is that task results from source device NN section 2 or destination device NN section 2 trigger long-term storage of videos that may be of future interest. In one example, video can be retained only when any person is detected, thus reducing storage costs compared to storing all received video. When it is necessary to identify a specific person of interest, NN section 2 can be trained to detect that specific person. The destination device can use a customized NN section 2 to reprocess the NN section 1 tensor corresponding to frames containing at least one detected person to further filter the video data to the most relevant segments, which can then be reviewed by a human. In other implementations, video may not be retained unless a request for video has been made.

[0063] System 100 includes a source device 110. Source device 110 includes a video source 112 for generating unencoded frame data 113. Frame data 113 is passed to a neural network section 1 114 to generate a tensor 115, and is passed to a video encoder 150 to generate a video layer bitstream 151 (a first sub-bitstream). Tensor 115 is passed to a tensor encoder 116 to generate a feature layer bitstream 121 (a second sub-bitstream) and a proxy neural network section 2 118 to generate proxy task results 119. A transmitter / multiplexer 122 generates a multi-layer bitstream 123. Multi-layer bitstream 123 includes a series of access units (e.g., ...). Figure 11Access units 11105, each including a compressed frame from video layer bitstream 151 and a compressed frame from feature layer bitstream 121. Video layer bitstream 151 forms a “picture layer” in multi-layer bitstream 123, and feature layer bitstream 121 forms an “auxiliary layer” in multi-layer bitstream 123. The auxiliary layer typically encodes an alpha channel used to calculate transparency when rendering frames from the picture layer or a depth map used to render stereo frames from the picture layer. Thus, the typical use of the auxiliary layer is to encode information from the picture layer. The information encoded for the picture layer in the auxiliary layer is typically spatially closely related to the picture or video layer, for example, in a sample-by-sample manner (or almost sample-by-sample manner). Packed features from tensor 115 of NN part 1114 form a height-transformed version of frame data 113. The resolution of the feature frame depends on the number of tensors in tensor 115, as well as the channel count and resolution of each tensor 115, and is unrelated (or indirectly related) to the resolution of the frames in frame data 113. Furthermore, there is no simple spatial relationship between the highly abstract feature map data and the samples in frame data 113. For example, using SEI messages associated with the frames used to generate the results 119 of each proxy task, the proxy task results 119 are encoded as metadata in multi-layer bitstream 123.

[0064] System 100 also includes a destination device 140 for decoding tensor data in the form of the received multi-layer bitstream 123. Destination device 140 can be used to decode tensor data (or tensors) of the contents of the multi-layer bitstream 123 (e.g., audio data, video data, image data, and text data).

[0065] Communication channel 130 is used to communicate multi-layer bitstream 123 from source device 110 to destination device 140. In some arrangements, one or both of source device 110 and destination device 140 may include their respective mobile phones (e.g., "smartphones") or webcams and cloud applications. Communication channel 130 may be a wired connection such as Ethernet or a wireless connection such as WiFi or 5G, including connections across wide area networks (WANs). Communication channel 130 may also be implemented across ad hoc connections. Furthermore, source device 110 and destination device 140 may include applications that capture encoded video data on some computer-readable storage medium, such as hard disk drives or memory in a file server. Although system 100 is described as including video source 112 (which will provide frame data 113 for a neural network targeting computer vision applications), other types of source data (such as audio or text) may be input to suitable neural networks implemented in NN section 1 114 and NN section 2 head 166.

[0066] like Figure 1 As shown, source device 110 includes a video source 112, an neural network section 1114, a tensor encoder 116, and a transmitter / multiplexer 122. The video source 112 typically includes a source of captured video frame data (shown as 113), such as a camera sensor, a previously captured video sequence stored on a non-transient recording medium, or a video feed from a remote camera sensor. The video source 112 can also be the output of a computer graphics card, for example, the video output of a display operating system and various applications running on a computing device (e.g., a tablet computer). Examples of source devices 110 that may include a camera sensor as the video source 112 include smartphones, video camcorders, professional cameras, and webcams. The video source 112 can produce individual images or can produce images (i.e., video) sequentially over time.

[0067] System 100 implements the task network in a distributed manner, divided into two parts: NN part 1 114 and NN part 2 166. Additionally, source device 110 provides a general or "proxy" NN part 2 118, the result of which can be used as a trigger to initiate processing in destination device 140. For example, a "YOLOv3" network can be used as part of an object tracking system, and a "Faster R-CNN" network can be used as part of an object detection system. The number and dimensions of tensors 115 depend on the specific network implemented in system 100 and the splitting point of that specific task network.

[0068] NN section 1 114 receives video frame data 113 and performs specific layers of the overall CNN (such as layers corresponding to the "backbone" of the CNN), thereby outputting tensor 115. The backbone (section 1) layer of the CNN can produce multiple tensors as output 115 (e.g., corresponding to different spatial scales of the input image represented by the video frame data 113 when the network is split within the FPN). (e.g., if system 100 is performing a "YOLOv3" network, then) the FPN can obtain three tensors as tensor 115 corresponding to the three layers output from the backbone 114, where tensor 115 has different spatial resolutions and channel counts. When system 100 is performing a network such as "Faster RCNN X101-FPN" or "Mask RCNN X101-FPN", tensor 115 can include tensors for four layers (P2 to P5). As described below, when a split point of the network occurs within the FPN, the use of the FPN yields multiple tensors that form a hierarchical representation of a single frame to be encoded into a bitstream (and decoded from the bitstream). Tensor encoder 116 receives tensor 115 and produces a feature bitstream 121 containing a compressed representation of tensor 115.

[0069] Feature bitstream 121 is supplied to transmitter / multiplexer 122 for transmission via communication channel 130 as part of multi-layer bitstream 123. Alternatively or additionally, multi-layer bitstream 123 may be written to storage unit 132 for later use.

[0070] Source device 110 supports a specific network for NN section 1 114. However, destination device 140 can use one of several networks as NN section 2 166. Generally, destination device 140 selects a compatible NN section 2 116 for use with multi-layer bitstream 123 from a library of candidate NN section 2 options. When using one of the several NN section 2 candidates for NN section 2 166, partially processed data in the form of packed feature maps can be stored for later use in various tasks without having to perform NN section 1 114 operations again.

[0071] Multi-layer bitstream 123 is transmitted as encoded data via communication channel 130 by transmitter 122. In some implementations, multi-layer bitstream 123 may be stored in storage unit 132 (where storage unit 132 is a non-transitory storage device such as flash memory or hard disk drive) until later transmission via communication channel 130 (or instead of transmission via communication channel 130). For example, encoded video data can be provided to consumers on demand via wide area network (WAN) for video analytics applications.

[0072] Destination device 140 includes a receiver / demultiplexer 142, a tensor decoder 146, a NN section 2 storage 160, a NN section 2 166, a video decoder 170, and a task result renderer 168. The receiver / demultiplexer 142 receives a multi-layer bitstream 123 from communication channel 130. The receiver / demultiplexer 142 includes an entropy decoder capable of parsing data suitable for reference. Figure 11The high-level syntax of the entire multi-layer bitstream 123 (such as SEI messages) is described. Multi-layer bitstream 123 includes a main layer holding compressed video data and an auxiliary (feature) layer holding compressed tensors. Module 142 uses a "sub-bitstream extraction process" to extract a video sub-bitstream 171 (corresponding to video layer bitstream 151) and a feature sub-bitstream 143 (corresponding to feature layer bitstream 121) from multi-layer bitstream 123. This "sub-bitstream extraction process" filters out unwanted NAL units to decode specific layers of multi-layer bitstream 123. Feature sub-bitstream 143 is passed to tensor decoder 146, and video sub-bitstream 171 is passed to video decoder 170. NN section 1 network indication (i.e., topology and weight hash) 161 is determined from bitstream 123 and passed from module 142 to NN section 2 storage 160. Tensor decoder 146 outputs decoded tensor 149, which is then fed to NN section 2 166. NN section 2 166 implements the selected network 162, which is chosen from a set of NN section 2 candidates from the NN section 2 store 160, among those compatible with NN section 1 indication 161. The NN section 2 candidate is selected based on compatibility with the indicated NN section 1 obtained from the multi-layer bitstream 123, and in some instances based on the desired task (e.g., object tracking or image segmentation). When multiple NN section 2 candidates are compatible with the indicated NN section 1, the selection of which NN section 2 candidate is made is determined by the user or programmer of the destination device 140. For example, the selection may be based on user input used to configure the destination device 140 for a specific task (such as identifying a specific person using features from the NN section 1 of the source device 110). NN section 2 166 receives tensor 149 and performs later layers of the neural network starting from NN section 1 114 to produce task result 167.

[0073] Task results 167 are presented to the user in a visual manner, such as by task results renderer 168. Task results renderer 168 draws bounding boxes or segmentation maps that highlight detected objects or instances based on task results 167 and are overlaid on the decoded video 172 generated by video decoder 170. Task results renderer 168 is an example of presenting neural network results to the user. Task results renderer 168 can operate only when task results 168 indicate detected objects.

[0074] The functionality of the source device 110 and the destination device 140 may also be embodied in a single device, examples of which include mobile phones, tablet computers, and cloud applications.

[0075] Although the example apparatus has been described above, the source apparatus 110 and the destination apparatus 140 can typically be configured within a general-purpose computer system via a combination of hardware and software components. Figure 2A Example of such a computer system 200: The computer system 200 includes: a computer module 201; input devices such as a keyboard 202, a mouse pointer device 203, a scanner 226, a camera 227 configured as a video source 112, and a microphone 280; and output devices including a printer 215, a display device 214, and a speaker 217. The computer module 201 can communicate with a communication network 220 via a connection 221 using an external modulator-demodulator (modem) transceiver device 216. The communication network 220, which may represent the communication channel 130, can be a WAN, such as the Internet, a cellular telecommunications network, or a private WAN. If the connection 221 is a telephone line, the modem 216 can be a conventional dial-up modem. Alternatively, if the connection 221 is a high-capacity (e.g., cable or optical) connection, the modem 216 can be a broadband modem. A wireless modem can also be used for wireless connection to the communication network 220. The transceiver device 216 can provide the functionality of a transmitter 122 and a receiver 142, and the communication channel 130 can be embodied in the connection part 221.

[0076] Computer module 201 typically includes at least one processor unit 205 and a memory unit 206. For example, memory unit 206 may have semiconductor random access memory (RAM) and semiconductor read-only memory (ROM). Computer module 201 also includes multiple input / output (I / O) interfaces, including: an audio-video interface 207 coupled to a video display 214, speakers 217, and a microphone 280; an I / O interface 213 coupled to a keyboard 202, a mouse 203, a scanner 226, a camera 227, and optionally a joystick or other human-machine interface device (not illustrated); and an interface 208 for external modems 216 and printers 215. Signals from the audio-video interface 207 to the computer monitor 214 are typically output from the computer's graphics card. In some implementations, modem 216 may be integrated within computer module 201, for example, within interface 208. Computer module 201 also has a local network interface 211, wherein the local network interface 211 permits computer system 200 to be coupled to a local area communication network 222, known as a local area network (LAN), via a connection 223. Figure 2AAs illustrated, the local area communication network 222 can also be coupled to the wide area network 220 via the connection 224, wherein the local area communication network 222 will typically include a so-called "firewall" device or a device with similar functionality. The local network interface 211 may include Ethernet. TM Circuit card, Bluetooth TM The interface 211 can be a wireless or IEEE 802.11 wireless interface; however, various other types of interfaces can be implemented for interface 211. The local network interface 211 can also provide the functionality of transmitter 122 and receiver 142, and the communication channel 130 can also be embodied in the local communication network 222.

[0077] I / O interfaces 208 and 213 can provide either serial connectivity or parallel connectivity, wherein the former is typically implemented according to the Universal Serial Bus (USB) standard and has a corresponding USB connector (not shown). A storage device 209 is provided, and the storage device 209 typically includes a hard disk drive (HDD) 210. Other storage devices (not shown), such as floppy disk drives and tape drives, may also be used. An optical disc drive 212 is typically provided as a non-volatile source of data. Optical discs, such as CD-ROMs, DVDs, and Blu-ray Discs, may be used. TM Portable storage devices such as USB-RAM, portable external hard disk drives, and floppy disks serve as suitable sources of data for computer system 200. Typically, any of HDD 210, optical disk drive 212, network 220, and 222 can also be configured to operate as video source 112 or as a destination for decoded video data to be stored for reproduction via display 214. The source device 110 and destination device 140 of system 100 can be embodied in computer system 200.

[0078] Components 205 to 213 of computer module 201 typically communicate via interconnect bus 204 and in a manner known to those skilled in the art to the conventional operating mode of computer system 200. For example, processor 205 is coupled to system bus 204 via connector 218. Similarly, memory 206 and optical disc drive 212 are coupled to system bus 204 via connector 219. Examples of computers in which this arrangement can be practiced include IBM-PC and compatible machines, Sun SPARCstation, and Apple Macs. TM Or a similar computer system.

[0079] The video encoder 150, NN section 1 114, tensor encoder 116, surrogate NN section 2, modules 122 and 142, tensor decoder 146, NN section 2 166, video decoder 170, and result renderer 168, along with the method to be described, can be implemented as one or more software applications 233 executable within the computer system 200. Specifically, this utilizes instructions 231 (see [link to instructions]) executed within the computer system 200 in the software 233. Figure 2B The steps of implementing the modules of devices 110 and 140 and the described methods are described. The software instructions 231 can be formed into one or more code modules, each for performing one or more specific tasks. The software can also be divided into two separate parts, wherein a first part and its corresponding code modules perform the methods, and a second part and its corresponding code modules manage the user interface between the first part and the user.

[0080] For example, the software can be stored in a computer-readable medium including the storage means described below. The software is loaded from the computer-readable medium into computer system 200 and then executed by computer system 200. The computer-readable medium having such software, or the computer program recorded on such a computer-readable medium, is a computer program product. Using the computer program product in computer system 200 preferably implements advantageous means for carrying out the source device 110 and the destination device 140, as well as the method.

[0081] Typically, software 233 is stored in HDD 210 or memory 206. The software is loaded into computer system 200 from a computer-readable medium and executed by computer system 200. Thus, for example, software 233 may be stored on an optically readable disk storage medium (e.g., CD-ROM) 225 read by optical disk drive 212.

[0082] In some instances, application 233 is supplied to the user encoded on one or more CD-ROMs 225 and read via a corresponding drive 212, or alternatively, the user may read application 233 from network 220 or 222. Furthermore, software may also be loaded into computer system 200 from other computer-readable media. Computer-readable storage media refers to any non-transitory tangible storage medium that provides recorded instructions and / or data to computer system 200 for execution and / or processing. Examples of such storage media include floppy disks, magnetic tapes, CD-ROMs, DVDs, and Blu-ray Discs. TMThis includes hard disk drives, ROMs or integrated circuits, USB storage devices, magneto-optical disks, or computer-readable cards such as PCMCIA cards, regardless of whether these devices are inside or outside the computer module 201. Examples of transient or non-tangible computer-readable transmission media that may also participate in the provision of software, applications, instructions, and / or video data or encoded video data to the computer module 201 include: radio or infrared transmission channels and network connections to another computer or networked device, as well as the Internet or intranet including email transmissions and information recorded on websites.

[0083] The second part of the aforementioned application 233 and the corresponding code modules can be executed to implement one or more graphical user interfaces (GUIs) to be rendered or otherwise represented on the display 214. By typically manipulating the keyboard 202 and mouse 203, users and applications of the computer system 200 can manipulate the interface in a functionally applicable manner to provide control commands and / or input to applications associated with these (one or more) GUIs. Other functionally applicable forms of user interfaces can also be implemented, such as audio interfaces utilizing voice prompts output via speaker 217 and user voice commands input via microphone 280.

[0084] Figure 2B This is a detailed schematic block diagram of processor 205 and "memory" 234. Memory 234 represents... Figure 2A The computer module 201 can access the logical aggregation of all memory modules (including storage device 209 and semiconductor memory 206).

[0085] Upon initial power-on of the computer module 201, a power-on self-test (POST) procedure 250 is executed. Typically, the POST procedure 250 is stored in... Figure 2A The ROM 249 of the semiconductor memory 206 is sometimes referred to as firmware. The POST program 250 checks the hardware within the computer module 201 to ensure proper operation, and typically checks the processor 205, the memories 234 (209, 206), and the Basic Input / Output System Software (BIOS) module 251, which is also typically stored in the ROM 249, for correct operation. Once the POST program 250 runs successfully, the BIOS 251 is activated. Figure 2AThe hard disk drive 210. Activating the hard disk drive 210 causes the bootstrap loader program 252 residing on the hard disk drive 210 to be executed via the processor 205. This loads the operating system 253 into the RAM memory 206, where the operating system 253 begins operation. The operating system 253 is a system-level application executable by the processor 205 to implement various advanced functions, including processor management, memory management, device management, storage management, software application interfaces, and general user interfaces.

[0086] The operating system 253 manages memory 234 (209, 206) to ensure that each process or application running on computer module 201 has sufficient memory to execute without conflicting with memory allocated to another process. Furthermore, appropriate use of memory is required. Figure 2A The computer system 200 contains different types of memory to enable efficient operation of various processes. Therefore, aggregate memory 234 is not intended to illustrate how specific partitions of memory are allocated (unless otherwise stated), but rather to provide an overview of the memory accessible to the computer system 200 and how such memory is used.

[0087] like Figure 2B As shown, processor 205 includes multiple functional modules, including a control unit 239, an arithmetic logic unit (ALU) 240, and local or internal memory 248, sometimes referred to as a cache. The cache 248 typically includes multiple storage registers 244 to 246 in a register section. One or more internal buses 241 functionally interconnect these functional modules. Processor 205 also typically has one or more interfaces 242 for communicating with external devices via system bus 204 using connection 218. Memory 234 is coupled to bus 204 using connection 219.

[0088] Application program 233 includes a sequence of instructions 231 that may contain conditional branch instructions and loop instructions. Program 233 may also include data 232 used when executing program 233. Instructions 231 and data 232 are stored in memory locations 228, 229, 230 and 235, 236, 237, respectively. Depending on the relative size of instructions 231 and memory locations 228 to 230, as described by the instructions shown in memory location 230, a particular instruction may be stored in a single memory location. Alternatively, as described by the instruction segments shown in memory locations 228 and 229, the instruction may be divided into multiple parts, each stored in a separate memory location.

[0089] Typically, an instruction set is given to processor 205, which executes the instruction set within processor 205. Processor 205 waits for subsequent input, which it responds to by executing another instruction set. Each input can be provided from one or more sources, including data generated by one or more input devices 202, 203, data received from an external source across one of networks 220, 202, data retrieved from one of storage devices 206, 209, or data retrieved from storage medium 225 inserted into the corresponding reader 212 (all of which are within...). Figure 2A (As shown in the diagram). Executing the instruction set may result in output data in some cases. Execution may also involve storing data or variables into memory 234.

[0090] The tensor encoder 116, tensor decoder 146, and the method can use the input variable 254 stored in corresponding memory locations 255, 256, and 257 within memory 234. The tensor encoder 116, tensor decoder 146, and the method generate the output variable 261 stored in corresponding memory locations 262, 263, and 264 within memory 234. Intermediate variables 258 can be stored in memory locations 259, 260, 266, and 267.

[0091] refer to Figure 2B The processor 205, registers 244, 245, 246, arithmetic logic unit (ALU) 240, and control unit 239 work together to perform micro-operation sequences required for a "fetch, decode, and execute" cycle for each instruction in the instruction set constituting program 233. Each fetch, decode, and execute cycle includes:

[0092] The fetch operation is used to fetch or read instruction 231 from memory locations 228, 229, and 230;

[0093] Decoding operation, in which control unit 239 determines which instruction was extracted; and

[0094] The operation is performed, in which the control unit 239 and / or ALU 240 executes the instruction.

[0095] Then, a further fetch, decode, and execute cycle for the next instruction can be performed. Similarly, a storage cycle can be performed, during which the control unit 239 stores or writes a value to or writes it to memory location 232.

[0096] To explain Figures 18 to 20Each step or subprocess in the method is associated with one or more segments of program 233 and is typically performed by working together in register segments 244, 245, 246, ALU 240 and control unit 239 in processor 205 to fetch, decode and execute cycles for each instruction in the instruction set of the segment of program 233.

[0097] Figure 3A This is a schematic block diagram 300 showing the functional modules of the CNN backbone 310 that can be used as the implementation of NN section 1 114 when system 100 is configured to run a "YOLOv3" network. NN section 1 114 is sometimes referred to as "DarkNet-53", although different backbones are possible, resulting in different numbers and dimensions of layers of tensors 115 for each frame. In one implementation, backbone 310 can be used as a person detector for object tracking purposes.

[0098] like Figure 3A As shown, video data 113 is passed to a resizing module 304. This resizing module 304 resizes each frame of video data 113 to a resolution suitable for processing by the CNN backbone 310, thereby producing resized frame data 312. If the resolution of video data 113 is already suitable for the CNN backbone 310, the operation of the resizing module 304 is not required. The resized frame data 312 is then passed to a convolutional batch normalization leaky rectified linear (CBL) module 314 to produce tensors 316. The CBL module 314 contains a reference... Figure 3D The CBL module 360 ​​shown is the module described.

[0099] CBL module 360 ​​takes tensor 361 of the resized frame data 312 as input. This tensor 361 is passed to convolutional layer 362 to produce tensor 363. If convolutional layer 362 has a stride of 1, tensor 363 has the same spatial dimension as tensor 361. If convolutional layer 362 has a larger stride (such as 2), tensor 363 has a smaller spatial dimension compared to tensor 361; for example, for a stride of 2, the width and height of tensor 363 are halved. Regardless of the stride, the channel dimension of tensor 363 can vary compared to the channel dimension of tensor 361 for a given CBL block. Tensor 363 is passed to batch normalization module 364, which outputs tensor 365. Batch normalization module 364 normalizes the input tensor 363 and applies a scaling factor and offset value to produce output tensor 365. The scaling factor and offset value are derived from the training process. Tensor 365 is passed to the Leaky ReLU module 366 to produce tensor 367. Module 366 provides a "leaky" activation function, whereby positive values ​​in the tensor are passed through, and negative values ​​are significantly reduced in magnitude, for example, reduced to 0.1 times their previous value.

[0100] Return to Figure 3A Tensor 316 is passed from CBL block 314 to residual block module 320, such as a cascaded “res1+2+8” module (also called res11 module) containing three residual blocks, each containing one (1) residual unit, two (2) residual units, and eight (8) residual units, respectively. In CBL block 344, the spatial resolution of the tensor is halved horizontally and vertically in each residual block by a convolution with a stride of 2 (see [link to CBL block 344]). Figure 3B ).

[0101] For reference Figure 3B The ResBlock 340 shown illustrates a residual block. ResBlock 340 receives tensor 341 (e.g., tensor 316). Tensor 341 is zero-padded by zero-padding module 342 to produce tensor 343. Tensor 343 is passed to CBL module 344 to produce tensor 345. CBL module 344 contains convolutions with a stride parameter set to 2 (e.g., similar to 362), resulting in tensor 345 having half the width and half the height of tensor 343. Tensor 345 is passed to residual unit 346. Residual unit 346 contains a series of cascaded residual units based on the number of residual blocks (e.g., eleven (11) units for block 320). The last residual unit in residual unit 346 outputs tensor 347.

[0102] For reference Figure 3CThe ResUnit 350 shown illustrates a residual unit. ResUnit 350 takes tensor 351 (e.g., tensor 345) as input. Tensor 351 is passed to CBL module 352 to produce tensor 353. Tensor 353 is passed to a second CBL unit 354 to produce tensor 355. Addition module 356 sums tensor 355 with tensor 351 to produce tensor 357. Addition module 356 can also be referred to as a "shortcut" because the input tensor 351 substantially affects the output tensor 357. For an untrained network, ResUnit 350 performs actions to pass through the tensors. During training, CBL modules 352 and 354 perform actions to deviate tensor 357 from tensor 351 based on the training data and ground truth data.

[0103] Return to Figure 3A Res11 module 320 outputs tensor 322. Tensor 322 is output as one of the layers from the backbone module 310 and is also provided to Res8 module 324. Res8 module 324 is a residual block (i.e., 340) comprising eight residual units (i.e., 350). Res8 module 324 produces tensor 326. Tensor 326 is passed to Res4 module 328 and output as one of the layers from the backbone module 310. Res4 module is a residual block (i.e., 340) comprising four residual units (i.e., 350). Res4 module 328 produces tensor 329. Tensor 329 is output as one of the layers from the backbone module 310. Overall, layer tensors 322, 326, and 329 are output as tensor 115 and can be referred to as layers 0 to 2, or L0, L1, and L2, respectively. The CNN backbone 310 can take 1088×608 video frames as input and generate three tensors corresponding to the three layers, with the following dimensions: [1,256,76,136], [1,512,38,68], and [1,1024,19,34]. Another example of the three tensors 115 corresponding to the three layers could be [1, 512, 34, 19], [1, 256, 68, 38], and [1, 128, 136, 76]. These three are separated at layer indices 75, 90, and 105, respectively, when enumerating layers according to the YOLOv3 software implementation of backbone 300 and head 1200.

[0104] Each of Res11 320, Res8 324, and Res4 328 operates in a similar manner to ResBlock 340. Each of CBL 314, CBL 344, and CBL 354 operates in a similar manner to CBL 360.

[0105] Figure 4This is a schematic block diagram illustrating a functional module of an alternative stem 400 to the CNN, which can be used as the implementation of the NN part 1114 when system 100 is configured to perform a "Faster R-CNN" or "Mask R-CNN" ResNet 101 network. Frame data 113 is input and passed through stem network 408, res2 module 412, res3 module 416, res4 module 420, and res5 module 424 via tensors 409, 413, 417, 421, and 425, respectively. Stem 400 can be used as part of a general object detector or for instance segmentation, supporting objects of various classes.

[0106] The stem network 408 includes 7×7 convolutions and max pooling operations with a stride of 2 (2). The res2 module 412, res3 module 416, res4 module 420, and res5 module 424 perform convolution operations such as LeakyReLU activation. Each of the modules 412, 416, 420, and 424 also halves the width and height of the processed tensors once via a stride setting of 2. Tensors 413, 417, 421, and 425 are passed to one of the 1×1 lateral convolution modules 446, 444, 442, and 440, respectively. Modules 446, 444, 442, and 440 produce tensors 447, 445, 443, and 441, respectively. Tensor 441 is passed to the 3×3 output convolution module 470, which produces the output tensor P5 471.

[0107] Tensor 441 is also passed to upsampler module 450 to produce upsampled tensor 451. Summation module 460 sums tensors 443 and 451 to produce tensor 461. Tensor 461 is passed to upsampler module 452 and 3×3 lateral convolution module 472. Module 472 outputs tensor P4 473. Upsampler module 452 produces upsampled tensor 453. Summation module 462 sums tensors 445 and 453 to produce tensor 463. Tensor 463 is passed to 3×3 lateral convolution module 474 and upsampler module 454. Module 474 outputs tensor P3 475. Upsampler module 454 outputs upsampled tensor 455. Summation module 464 sums tensors 447 and 455 to produce tensor 465, which is then passed to 3×3 lateral convolution module 476. Module 476 outputs tensor P2 477. Upsampling modules 450, 452, and 454 use nearest-neighbor interpolation to reduce computational complexity. Tensors 471, 473, 475, and 477 form the output tensor 115 of the CNN backbone 400 (NN part 1 114). Although Figure 4The diagram shows a specific stem (“P-layer” split point) of the Faster R-CNN network architecture, but different splits into the stem and head are possible. Splitting the network at tensor 409 is called the “stem” split point. Splitting the network at tensors 447, 445, 443, and 441 is called the “C-layer” split point.

[0108] Figure 5 This is a schematic block diagram 500 of a tensor encoder 116 using a configurable tensor compressor stage. Figure 18 A method 1800 is shown for performing NN part 1, encoding instructions for the topology and weights of NN part 1, and encoding the resulting compressed tensor together with compressed video. Method 1800 can be implemented as one or more software applications 233 executable within computer system 200. Method 1800 can be implemented via instructions 231 (see [link to instructions]) in the software 233 executing within computer system 200. Figure 2B This can be achieved by [method 1]. The software instructions 231 can be configured into one or more code modules, each for performing one or more specific tasks. Method 1800 begins with the encoding of video frames, step 1810.

[0109] At step 1810, the video encoder 150, under the execution of the processor 205, encodes the frame data 113 to generate a video bitstream 151. Control in the processor 205 then proceeds from step 1810 to step 1820, which performs the first part of the neural network.

[0110] At step 1820, under the execution of processor 205, NN part 1 114 uses frame data 113 from video source 112 as input to perform the first part of the neural network, thereby producing tensor 115 as output. Control in processor 205 then proceeds from step 1820 to step 1830 to perform the second part of the neural network.

[0111] At step 1830, the surrogate NN portion 2 118, under the execution of processor 205, uses tensor 115 as input to perform the second part of the neural network. As a result of steps 1820 and 1830, an initial or "general" or "surrogate" task result 119 is generated. The surrogate task result 119 can be considered a "low-quality" or "approximate" version of the task result expected to be generated by system 100. Task result 119 is expected to be a more lenient version of the task result generated in destination device 140. For example, task result 119 can detect any person or vehicle, while task result 151 can detect a specific person or vehicle. Control in processor 205 proceeds from step 1830 to step 1840, indicating the determination of the NN, NN portion 1, and NN portion 2.

[0112] At step 1840, source device 110, under the execution of processor 205, determines a summary value that indicates or identifies the topology of NN part 1 114, the topology of NN part 2 118, and the topology of NN part 1 114 and NN part 2 118 as a combination of a single unified network. The network topology can be represented in various ways, such as by executing functions from libraries like PyTorch, Keras, Caffe, or Tensorflow using Python source code. The network topology can also be represented using compiled languages ​​such as C or C++. Device-independent representations of CNNS exist, such as ONNX, which include both the topology and network weights in the representation. Device-independent representations such as ONNX can be used in a reconstructed form, whereby arbitrary node names are replaced with an enumeration of node names. Therefore, the reconstructed ONNX representation is invariant to insignificant changes (such as those resulting from variable naming).

[0113] refer to Figure 20This describes a method for determining topology and weight indicators using a network representation, as implemented in step 1840. When a network is selected, for example by a user, the network topology and weight indicators can be determined and obtained from a repository before the network is executed, rather than being computed "at runtime." Alternatively, the network topology and weight indicators can be computed or invoked for a subset of the network. In the case where the network is divided into NN part 1 and NN part 2, separate network topology and weight indicators are determined, and the association between them is preserved. The indicator can be a generalized value in metadata that can be included in the multi-layer bit stream 123 without introducing excessive overhead, such as a hash or other algorithm used to transform information into a generalized value. Examples include applying md5sum, sha1sum, and sha256 to the byte stream of network topology or network weight to produce the corresponding indicator or hash. Hash algorithms (such as md5sum, sha1sum, and sha256) provide a relatively low probability of "collision," meaning that the probability of two different hashed inputs (such as network topology or network weight) producing the same hash value is relatively low. These indications are retained for future reference, rather than being recalculated each time they are to be referenced. Indications generated from different parts of the same network are considered "associated". This association can take the form of a lookup table, indicating the selection of a compatible NN part 2 (if available in the destination device 140) when decoding tensors from multi-layer bitstream 123, which has specific network topology and network weight indications. If no association (lookup table entry) is found, the destination device 140 cannot further process the decoded tensors from multi-layer bitstream 123. At step 1840, the network topology and network weight indications for NN part 1 114 and NN part 2 118, as well as the combined NN parts 1 and 2 (i.e., 114 and 118 are considered unsplit networks), are determined based on the corresponding topology or obtained from pre-calculated results.

[0114] Effectively, the "indicator" provides a unique way to identify a specific network or a portion thereof without referencing an external enumeration of the network topology. Other representations of the network topology can also be used, such as Python code using PyTorch library functions, C or C++ code calling library functions (such as the "CUDA" library functions for Nvidia GPUs or the "ROCm" device library for AMD GPUs), etc. However, such representations require a greater degree of refactoring to remove elements that do not affect the actual network structure, such as code comment removal and whitespace removal. Intermediate representations such as Python bytecode can be used; however, such forms change based on the development of the corresponding language and therefore do not form a stable representation suitable for reproducible hashing by users using different versions of the language.

[0115] The deployment can use the network's ONNX intermediate representation (IR) as input for hashing, where the ONNX IR version is included along with the hash as part of the indication to provide association of multiple hashes. The hashes represent the same network topology and / or weights, but differ in value because ONNX protocol updates affect the byte stream representation of network topology and / or weights. Alternative ONNX formats for representing network topology and / or weights (such as "TensorRT" or "OpenVino") can also be used to generate network topology and / or weight indications. Reference Figure 20 This describes an example method for generating such an instruction. Control in processor 205 proceeds from step 1840 to tensor compression step 1850.

[0116] In step 1850, tensor encoder 116 operates to compress tensor 115. Figure 5 An example implementation 500 of the tensor encoder 116 is shown. Figure 5 In the example, at step 1850, compressor 530, under the execution of processor 205, compresses tensor 115 to produce compressed tensor 532. Compressed tensor 532 is the same as or fewer in number than tensor 115 and reduced in dimensionality (i.e., reduced in channel count and either or both of feature map width and height). The tensor compressor can use an ensemble of trained network layers (as referenced). Figure 6 (as described above), or using a low-rank approximation method based on tensor 115 (as referenced). Figure 7 Dimensionality reduction is performed using the method described above. Control in processor 205 proceeds from step 1850 to step 1860, which involves encoding tensor information.

[0117] At step 1860, metadata encoder 544 (see...) Figure 5The tensor information 1195 (see) specifies the dimensions and feature map of the compressed tensor 532 for the placement of each tensor in the compressed tensor 532. Figure 11 The feature map is encoded into bitstream 121. A “feature frame” is a video frame that contains a feature map or other values ​​representing the feature map (such as basis vectors and coefficients). Tensor information 1195 includes the maximum channel count and the channel count used for each tensor. The frame area of ​​the region must be large enough to pack the tensors within the region up to the maximum channel count (i.e., the maximum number of feature maps). The maximum channel count must not be changed for frames in bitstream 121 associated with the same SPS 1110b. Tensor compressor 530 can produce fewer feature maps than the maximum value, and the channel count used signals the actual number of channels used, which can vary frame by frame. Video frame 540 can be divided into regions such as rectangular regions, where each region is used to hold a feature map from one or more tensors from compressed tensor 532. At step 1860, metadata encoder 544 also encodes the dimension of tensor 115 (i.e., the output of NN part 1114) as NN part 1 output indication 11102. Control in processor 205 moves from step 1860 to step 1870 of quantization tensor.

[0118] At step 1870, refer to Figure 5 Under the execution of processor 205, the quantizer module 534 of encoder 116 quantizes the floating-point values ​​in each tensor of compressed tensor 532 to produce a quantized compressed tensor 536. The quantized compressed tensor 536 has integer values ​​and occupies a range within the sample range defined by the operating bit depth of video encoder 542. For example, when encoding video using 10-bit samples, integer values ​​in the permissible range [0, 1023]. For each tensor, the minimum and maximum floating-point values ​​form the quantization range 526. Control in processor 205 proceeds from step 1870 to the encoding quantization range step 1880.

[0119] At step 1880, metadata encoder 544 ( Figure 5 Under the execution of processor 205, the quantization range 526 is encoded as the quantization range 1196 of each tensor in the compressed tensor 532 into multi-layer bit stream 123. The quantization range is stored in multi-layer bit stream 123 so that inverse quantization can be performed in destination device 140 to restore the range of floating-point values ​​back to the range generated by NN part 1 114. Control in processor 205 proceeds from step 1880 to the packing tensor step 1890.

[0120] At step 1890, the packer module 538, executed by the processor 205, packs the feature maps of each tensor in the quantized compressed tensor 536 into the video frame 540. (Tensor features) Figure 1 Generally, regions are packed into assigned tensor regions in feature frame 540 in a left-to-right and then top-to-bottom manner. Although regions are defined in the SEI message 1113, which is indicated by the FCVCM decoder network, regions can correspond to “encoding layers” (i.e., below metadata such as SEI message 1113) such as sub-pictures, slices, or tiles. SEI message 1113 is encoded as a “prefix SEI” message (i.e., before the access unit) so that the information contained in each layer in the access unit is available once decoded. The encoding-level structure of the frame is defined in the encoder (such as... Figure 8 The encoder (e.g., 800) may not be externally accessible and may differ between compression standards used (such as VVC, HEVC, or AV-1). The individual signal notification for each region in SEI message 1113 eliminates any dependency on specific features at the coding layer. Each region needs to be large enough to hold all feature maps of the tensors to be packed into the corresponding region. (See reference...) Figure 9A and Figure 9B as well as Figure 10A and Figure 10B This describes an example of a region-based packing format. Control in processor 205 moves from step 1890 to step 18100, where the feature frame is encoded.

[0121] At step 18100, the video encoder 542, under the execution of the processor 205, compresses the feature frame 540 to generate an auxiliary layer of the multi-layer bitstream 123. (See reference...) Figure 8 The operation of video encoder 542 is described below. Video encoder 542 is used to encode the auxiliary (feature) layer of multi-layer bitstream 123. A separate instance of video encoder 800 is used as video encoder 150 to encode the main (video) layer of multi-layer bitstream 123. Control in processor 205 proceeds from step 18100 to the encoding agent NN and NN section 1 instructing step 18110.

[0122] At step 18110, the metadata encoder 544, under the execution of processor 205, encodes the NN and NN Part 1 indications as determined at step 1840 into the multi-layer bitstream 123. For the NN and NN Part 1, individual indications or hashes are encoded for network topology and network weights. Hashes generated by hash algorithms such as sha1sum or sha256 are typically 256-bit codes encoded using fixed-length codewords. In particular, for certain widely used networks and widely used weight files, the NN network topology and network weight indications allow the decoder to identify such widely used networks and which weights are used with the network. Split points used in the source device 110 are identified via the NN Part 1 network topology indication. Using structure-derived indications avoids the need to explicitly enumerate various split points and update such enumeration when a new network is published and new split points are used. Using structure-based indications also avoids the need for different vendors of the source device 110 and destination device 140 to agree on enumeration, that is, no globally recognized repository of network architecture is required. Control in processor 205 moves from step 18110 to step 18120, which is the result of the task in the NN part 2 of the encoding agent.

[0123] At step 18120, the metadata encoder 544, under the execution of the processor 205, encodes the result 119 of the surrogate NN part 2 task into a multi-layer bitstream 123. The result 119 may be one or more bounding boxes or image classifications and indicates the result of the surrogate NN part 2 118 as performed in the source device 110.

[0124] Method 1800 terminates and processing in processor 205 proceeds to the next instance of source data 113 (e.g., the next frame from video source 112).

[0125] Figure 6 This is a schematic block diagram 600 illustrating a type of multi-scale feature fusion (MSFF) module that can be used as a tensor compressor 530. The MSFF module 600 takes tensor 115 and produces a compressed tensor 532, which has a reduced dimension compared to tensor 115, resulting in a lower bit rate when encoded as part of a packed frame. The MSFF module 600 uses trained network layers and requires a corresponding module in tensor decoder 146 to recover the tensor dimensions, so tensor 149 can be fed into the CNN head 150. The MSFF module 600 takes four tensors as input and requires each of the tensors to have 200 x 56 (256) channels, making the MSFF module 600 compatible with P layers of Faster R-CNN or Mask R-CNN networks. However, variations of the MSFF module 600 compatible with different numbers of layers and different channel counts are possible.

[0126] MSFF module 600 produces a tensor 532 as an output with sixty-four (64) channels and a feature map size corresponding to the P5 layer seen at the input; however, variants with different channel counts are also possible. Each variant of MSFF module 600 requires different weights for appropriate operation. Given that several variants of MSFF module 600 can be used in system 100 for a given network, the packet format can be set to the worst-case feature map count of the compressed tensor of the currently used decoder network topology, and the actual channel count used can be updated at runtime (i.e., when processing frame data 113) as part of tensor information 1195.

[0127] MSFC module 600 includes Figure 6 The MSFF block 610 shown uses one or more downsampling filters to generate a single tensor 629 from multiple tensors 115. Under the execution of processor 205, MSFF block 610 combines the tensors (i.e., 602, 603, 604, 605) of the first tensor set to produce a combined tensor 629. The combined tensor 629 forms a representation of the FPN layer tensors. Downsampling modules 622a, 622b, and 622c operate on tensors with larger spatial scales (i.e., P4 604 at (2h, 2w, 256), P3 603 at (4h, 4w, 256), and P2 602 at (8h, 8w, 256), respectively). Modules 622a, 622b, and 622c downsample to match the spatial scale of the minimum tensor (i.e., P5 605 at (h, w, 256)), thereby generating downscaled P5 tensors 623a, 623b, and 623c, respectively. Cascading module 624 performs channel-wise cascading of tensors 605, 623a, 623b, and 623c to generate a cascaded tensor 625 of dimension (h, w, 1024). Cascaded tensor 625 is passed to squeeze and excite (SE) module 626 to generate tensor 627. SE module 626 sequentially performs global pooling, a fully connected layer with reduced channel count, a modified linear unit activation unit, a second fully connected layer with restored channel count, and a sigmoid activation function to generate a scaled tensor. Tensor 625 is scaled according to the scaled tensor to produce the output as tensor 627. SE block 626 can be trained to adaptively change the weighting of different channels in the passing tensor based on the output of the first fully connected layer.

[0128] The output of the first fully connected layer reduces the feature maps for each channel to a single value. Each single value is passed through a nonlinear activation unit (ReLU) to create a conditional representation of that single value that is weighted to the other channels, where the recovery to the full channel count is performed by the second fully connected layer. Thus, the SE block 626 is able to extract nonlinear inter-channel correlations when generating tensor 627 from tensor 625 to a greater extent than is possible using purely convolutional (linear) layers. Tensor 627 is passed to convolutional layer 628. Convolutional layer 628 implements one or more convolutional layers to produce a combined tensor 629, where the channel count is reduced to F channels (typically 256 channels) (i.e., F=256). Further reduction in the channel count is achieved by a single-scale feature compression (SSFC) module 650.

[0129] The SSFC module 650 receives tensor 629 and applies convolution 652 to reduce the channel count from F(256) to C' (nominally set to 64 channels) to produce tensor 653. Tensor 653 is then passed to the batch normalization module 654 to produce a batch normalized tensor 655, which is passed to the hyperbolic tangent activation layer 656 to produce a compressed tensor 532. The output of the MSFC module 600 is one tensor per frame with a fixed feature map size and a fixed channel count.

[0130] Figure 7 This is a schematic block diagram illustrating an example of an inter-channel decorrelation-based tensor compressor 700 that can be used as a tensor compressor 530. Tensor compressor 700 operates without requiring any trained layers and relies on extracting inter-channel redundancy at runtime (i.e., using input tensor data from the CNN backbone 114 instead of relying on pre-trained weights). Tensor compressor 700 outputs three types of data: mean, basis vectors, and coefficients. Tensor compressor 700 compresses a tensor, and multiple instances of tensor compressor 700 can be instantiated as tensor compressor 530 to compress multiple tensors. When multiple instances of tensor compressor 700 are instantiated, tensors of the same type from each instance can be packed into the same region. For example, basis vectors from each instance can be packed into a single region. These types of data are retained in the tensor and packed into three separate regions within the frame. The number of basis vectors to be used can vary during the processing of frames, and the number of coefficients used can also vary, although the number of coefficients must not exceed the number of basis vectors. In this way, the dimensions of the basis vectors and / or coefficients can be updated periodically to signal the decoder the content of the decoded, packaged frames. For example... Figure 7As shown, under the execution of processor 205, the channel averaging module 710 averages the tensors in tensor 115 across spatial dimensions (i.e., width and height) to produce a per-channel average 712 as a 1D (one-dimensional) tensor C (the channel counts of the tensors in tensor 115). The channel averaging module 710 can operate less frequently than on each received tensor 115, in which case the channel average 712 is updated only for a specific frame. The packaged frame containing the updated channel average 712 can be indicated by a non-zero channel count of tensor 712 via tensor information 1195 or via a separate tensor update flag included in SEI message 1113.

[0131] For each channel in tensor 115, a DC shift is performed by subtraction module 714, which subtracts a value obtained from the mean channel 712 that is constant for the feature map from each spatial location in the feature map. The subtracted value is the corresponding value in the reconstructed mean channel 712. As a result of subtraction module 714, a zero-centered tensor 716 is output, in which the DC components found from each feature map are removed from the corresponding feature map.

[0132] Decomposition module 718 operates under the execution of processor 205 to produce a set of basis vectors 720 for the tensors in tensor 115. Decomposition module 718 receives a zero-centered tensor 716 as input and generates the set of basis vectors 720 by performing principal component analysis (PCA) methods such as singular value decomposition (SVD). A basis vector maps all channels to a single value using a dot product operation, so in the case of 256 (256) channels in the tensor in tensor 115, a basis vector has a dimension of 256 × 1. If decomposition module 718 produces the first N basis vectors (such as 25 (25) etc.), the resulting basis vectors have a dimension of 256 × N or 256 × 25. The basis vectors are associated with the origin, so a zero-mean tensor 712 is used to ensure that an orthogonal basis can be found. Each basis vector is a vector that associates all channels with a reduced set of channels. In this way, the basis vectors collectively enable the tensor data spanning all channels to be represented as a smaller set of basis vectors. Each basis vector is derived by considering all samples in each feature map for a given channel. Vector 720 contains fewer basis vectors than the channels present in the tensor in tensor 115, which corresponds to dimensionality reduction of the tensor in tensor 115. Basis vector 720 represents the tensor in tensor 115 in a subspace that describes or "explains" the maximum variance in the tensor in tensor 115 for the number of components in basis vector 720. The basis vectors are ordered downwards from the vector with the maximum explained variance to the vector with the minimum explained variance. In other words, basis vector 720 enables the representation of the tensor in tensor 115 with minimum quality degradation for a given number of components, where the components are the top N ranked basis vectors.

[0133] like Figure 7 As shown, under the execution of processor 205, dot product module 722 performs dot product between each channel of tensor 716 and each basis vector 720 to produce coefficient tensor 724. The coefficients form tensor 724 having the same width and height as the tensor in tensor 115, but with a channel count corresponding to at most the number of components (or basis vectors) generated by decomposition module 718, which is less than the number of channels in the tensor in tensor 115. Tensor compressor 700 can change the number of channels in coefficient tensor 716 for a given frame. The number of channels present in coefficient tensor 716 for a given frame is included in tensor information 1195 and encoded in an instance of SEI message 1113 associated with the given frame. Each value in coefficient tensor 724 represents the contribution of each basis vector in reproducing each value in the feature map. Tensors 724, 720, and 712 provide compressed tensor 532.

[0134] Figure 8 This is a schematic block diagram illustrating the functional modules of a video encoder 800. Video encoder 800 provides example implementations of each of video encoders 542 and 150. (See attached diagram.) Figure 2A and Figure 2B As shown, the video encoder 800 can be implemented using a general-purpose computer system 200, wherein various functional modules can be implemented using dedicated hardware within the computer system 200 and software executable within the computer system 200 (such as one or more software code modules of a software application program 233 residing on the hard disk drive 205 and executed by the processor 205). Alternatively, the video encoder 800 can be implemented using a combination of dedicated hardware and software executable within the computer system 200. Alternatively, the video encoder 800 and the method can be implemented in dedicated hardware such as one or more integrated circuits that perform the functions or sub-functions of the method. Such dedicated hardware may include a graphics processing unit (GPU), a digital signal processor (DSP), a dedicated standard product (ASSP), an application-specific integrated circuit (ASIC), an FPGA, or one or more microprocessors and associated memory. In particular, the video encoder 800 includes modules 810 to 890, wherein each of these modules can be implemented as one or more software code modules of the software application program 233.

[0135] although Figure 8The video encoder 542 is an example of a general video coding (VVC) video encoder, but other video codecs can also be used for the processing stages described herein. For example, HEVC can be used. The described example generates a bitstream of encoded data. If other codecs are used, some implementations can package the data into different formats such as frame formats. The video encoder 800 receives frame data 802, each frame including one or more color channels. Frame data 802 can be any chroma format and bit depth supported by the profile used, such as 4:0:0 or 4:2:0 of ​​the “Master 10” profile of the VVC standard, with a sample precision of eight (8) to ten (10) bits. The frame data can be associated with the video data 113 of tensor 540. Frame data 802 is also characterized by “levels”, which specify aspects such as maximum luminance sampling rate, frame aspect ratio constraints, and slice and tile count limits. The worst-case compressed bit rate is also indicated by a combination of “level” and “tier” parameters.

[0136] like Figure 8 As shown, the block partitioner 810 first divides the frame data 802 into CTUs, typically square in shape and configured to use a specific size for the CTUs. The maximum effective size of a CTU can be, for example, 32×32, 64×64, or 128×128 luma samples, configured by the "sps_log2_ctu_size_minus5" syntax element present in the "sequence parameter set". The CTU size also provides the maximum CU size, as a CTU without further subdivision will contain one CU. The block partitioner 810 also divides each CTU into one or more CBs based on the luma coding tree and chroma coding tree. The luma channel can also be referred to as the primary color channel. Each chroma channel can also be referred to as a secondary color channel. CBs have various sizes and can include both square and non-square aspect ratios. However, in the VVC standard, CBs, CUs, PUs, and TUs always have side lengths that are powers of two. Therefore, based on the luminance coding tree and chrominance coding tree of the CTU, the current CB, represented as 812, is output from the block partitioner 810 (advancing based on iterations on one or more blocks of the CTU).

[0137] The CTUs obtained from the first segment of frame data 802 can be scanned in raster scan order, and these CTUs can be grouped into one or more "slices". A slice can be an "intra-frame" (or "I") slice. An intra-frame slice (I slice) indicates that each CU in the slice is intra-predicted. Typically, the first picture in a coding layer video sequence (CLVS) contains only I slices and is referred to as an "intra-frame picture". CLVS can contain periodic intra-frame pictures that form "random access points" (i.e., intermediate frames in the video sequence from which decoding can begin). Alternatively, slices can be single-predicted or double-predicted ("P" or "B" slices, respectively), indicating the additional availability of single and double prediction within the slice.

[0138] The video encoder 800 encodes a sequence of images according to image structures. One image structure is "low latency," in which images using inter-frame prediction may only reference previously appearing images in the sequence. Low latency allows each image to be output as soon as it is decoded, and it is also stored for possible reference by subsequent images. Another image structure is "random access," where the encoded order of images differs from the display order. Random access allows inter-frame predicted images to reference other images that have been decoded but not yet output. A certain level of image buffering is required so that reference images in the display order can exist in the decoded image buffer, resulting in multi-frame latency.

[0139] When using a chroma format other than 4:0:0, within an I-slice, the coding tree for each CTU can diverge into two separate coding trees below the 64×64 level, one for luma and the other for chroma. Using separate trees allows for different block structures for luma and chroma within the luma 64×64 region of the CTU. For example, large chroma blocks can be juxtaposed with many smaller luma blocks, and vice versa. In P- or B-slices, a single coding tree for the CTU defines a common block structure for both luma and chroma. The resulting blocks from a single tree can be intra-predicted or inter-predicted.

[0140] In addition to segmenting images into slices, images can also be segmented into "tiles." A tile is a sequence of CTUs covering a rectangular area of ​​the image. CTU scans are performed within each tile using a raster scan, progressing from one tile to the next. A slice can be an integer number of tiles, or an integer number of consecutive CTU rows within a given tile.

[0141] For each CTU, such as Figure 8The video encoder 800 operates in two phases. In the first phase (called the “search” phase), the block partitioner 810 tests various potential configurations of the coding tree. Each potential configuration of the coding tree has an associated “candidate” block divider (CB). The first phase involves testing various candidate CBs to select the CB that provides relatively high compression efficiency and relatively low distortion. The testing typically involves Lagrangian optimization, thereby evaluating the candidate CBs based on a weighted combination of rate (i.e., coding cost) and distortion (i.e., error relative to the input frame data 712). The “best” candidate CB (i.e., the CB with the lowest evaluated rate / distortion) is selected for subsequent encoding into the bitstream portion 816. Included in the evaluation of the candidate CBs are options for using the CB for a given region, or for further splitting the region according to various splitting options and encoding each smaller resulting region with a further CB, or even further splitting the region. Thus, both the coding tree and the CB are selected in the search phase.

[0142] Video encoder 800 generates prediction blocks (PBs) indicated by arrow 820 for each CB (e.g., CB 812). PB 820 is a prediction of the content of the associated CB 812. Subtractor module 822 generates a difference (or “residual”, referring to the difference in the spatial domain) between PB 820 and CB 812, indicated by 824. Difference 824 is the block size difference between the corresponding samples in PB 820 and CB 812. Difference 824 is transformed, quantized, and represented as a transform block (TB) indicated by arrow 836. PB 820 and the associated TB 836 are typically selected from one of several possible candidate CBs, for example, based on the evaluated cost or distortion.

[0143] The candidate coded block (CB) is a CB obtained from the video encoder 800 for one of the prediction modes available for the associated PB and the resulting residual. When combined with the prediction PB in the video encoder 800, the TB 836 reduces the difference between the decoded CB and the original CB 812 at the cost of additional signal notification in the bitstream.

[0144] Each candidate coded block (CB) (i.e., a prediction block (PB) combined with a transform block (TB)) has an associated coding cost (or "rate") and an associated difference (or "distortion"). The distortion of a CB is typically estimated as the difference in sample values, such as the sum of absolute differences (SAD), the sum of squared differences (SSD), or a Hadamard transform applied to the difference. The mode selector 886 uses the difference 824 to determine the estimate obtained from each candidate PB to determine the prediction mode 887. The prediction mode 887 indicates the decision to use a specific prediction mode (e.g., intra-frame prediction or inter-frame prediction) for the current CB. The coding cost associated with each candidate prediction mode and the corresponding residual coding can be estimated at a significantly lower cost compared to entropy coding of the residual. Therefore, even in a real-time video encoder, multiple candidate modes can be evaluated to determine the optimal mode in a rate-distortion sense.

[0145] Determining the preferred mode based on rate distortion is typically achieved using a variant of Lagrange optimization. Both the preferred partitioning of the CTU to CB (using block partitioner 810) and the selection of the prediction mode from multiple possibilities can be employed using Lagrange or similar optimization processes. By applying Lagrange optimization of the candidate modes in the mode selector module 886, the intra-prediction mode with the lowest cost metric is selected as the "best" mode. The lowest cost mode includes the selected quadratic transform index 888, which is also encoded into the bitstream 816 by the entropy encoder 838.

[0146] In the second phase of the operation of the video encoder 800 (referred to as the "encoding" phase), the determined coding trees (one or more) for each CTU are iterated within the video encoder 800. For CTUs using individual trees, the luma coding tree is encoded first for each 64×64 luma zone of the CTU, followed by the chroma coding tree. Within the luma coding tree, only the luma CB is encoded, and within the chroma coding tree, only the chroma CB is encoded. For CTUs using shared trees, a single tree describes the CU (i.e., luma CB and chroma CB) according to the common block structure of the shared tree.

[0147] The entropy encoder 838 supports bitwise encoding of syntactic elements using variable-length and fixed-length codewords, as well as arithmetic encoding modes for syntactic elements. Sections of the bitstream, such as "parameter sets" (e.g., Sequence Parameter Sets (SPS), Picture Parameter Sets (PPS), and Picture Headers (PH)), use a combination of fixed-length and variable-length codewords. Slices (also called contiguous sections) have a slice header using variable-length encoding, followed by slice data using arithmetic encoding. The slice header defines parameters specific to the current slice, such as slice-level quantization parameter offsets, and may include instances of PHs. Slice data includes the syntactic elements of each CTU within the slice. The use of variable-length and arithmetic encoding requires sequential parsing within the sections of the bitstream. These sections can be described using starter codes to form "Network Abstraction Layer Units" or "NAL Units." Arithmetic encoding is supported using context-adaptive binary arithmetic encoding processing.

[0148] Arithmetically encoded syntactic elements consist of a sequence of one or more "bins" (binary files). Like bits, bins have values ​​of "0" or "1". However, bins are not encoded as discrete bits in bitstream section 816. A bin has an associated prediction (or "possible" or "maximum probability") value and an associated probability (known as "context"). The "maximum probability symbol" (MPS) is encoded when the actual bin to be encoded matches the predicted value. Encoding the maximum probability symbol is relatively inexpensive in terms of bit consumption in bitstream section 816, including a cost of less than one discrete bit in total. The "minimum probability symbol" (LPS) is encoded when the actual bin to be encoded does not match the possible value. Encoding the minimum probability symbol is relatively expensive in terms of bit consumption. Bin encoding techniques enable efficient encoding of bins that skew the probabilities of "0" vs. "1". For syntactic elements with two possible values ​​(i.e., "flag"), a single bin is sufficient. For syntactic elements with many possible values, a sequence of bins is required. The convention for converting the values ​​of syntactic elements into a sequence of bins is called "binarization." When the values ​​of bins "0" and "1" may be equal (or nearly equal), the context can be omitted and an equal probability distribution is assumed. A bin with context is called a "context-encoded bin," and a bin without context is called a "bypass-encoded bin." Binarizing syntactic elements into one or more bins yields a combination of context-encoded and bypass-encoded bins. Unlike directly encoding a bit into a bitstream, bypass-encoded bins use an arithmetic coding engine, which facilitates mixing context-encoded and bypass-encoded bins into syntactic element binarizations.

[0149] For a given bin, the presence of a later bin in the sequence can be determined based on the value of an earlier bin, resulting in a variable-length bin. Furthermore, each bin can be associated with more than one context, where a context is selected for use when encoding a specific instance of the bin. The selection of a specific context can depend on earlier bins in the syntactic elements, as well as the decoded values ​​of neighboring syntactic elements (i.e., neighboring syntactic elements from neighboring blocks), etc. Each time a context-encoded bin is encoded, the context selected for that bin (if it exists) is updated to reflect the new bin value. Thus, the binary arithmetic coding scheme is called adaptive.

[0150] The absence of a bypass-encoded bin saves memory and reduces complexity, thus using a bypass bin with an unskewed distribution of values ​​for that bin. An example of a context-adaptive entropy encoder is known in the art as CABAC (Context-Adaptive Binary Arithmetic Encoder), and many variations of this encoder have been adopted in video coding.

[0151] QP controller 890 determines quantization parameter 892, which is used to establish the quantization step size for use by quantizer 834 and dequantizer 840. A larger quantization step size causes the main transform coefficient 828 to be quantized to smaller values, thereby reducing the bit rate of bitstream portion 816 at the expense of reducing the fidelity of inverse transform coefficient 846.

[0152] The entropy encoder 838 encodes the quantization parameter 892 using a combination of context-encoded and bypass-encoded bins, and encodes the LFNST index 888 if applicable to the current CB. The quantization parameter 892 is encoded at the beginning of each slice, and variations in the quantization parameter 892 within the slice are encoded using the "differential QP" syntax element. The differential QP syntax element is signaled at most once in each region known as a "quantization group". The quantization parameter 892 is applied to the residual coefficients of the luma CB. The adjusted quantization parameter is applied to the residual coefficients of the juxtaposed chroma CB. The adjusted quantization parameter may include mapping from the luma quantization parameter 892 according to a mapping table and a CU-level offset selected from an offset list. The secondary transform index 888 is signaled when the residuals associated with the transform block include valid residual coefficients only in the positions of those coefficients transformed into principal coefficients by applying a secondary transform.

[0153] Residual coefficients for each TB associated with a CB are encoded using residual syntax. Residual syntax is designed to efficiently encode coefficients with low amplitudes, primarily using arithmetic-coded bins to indicate the validity of coefficients and the amplitude of low values, while reserving bypass bins for higher-amplitude residual coefficients. Therefore, residual blocks containing very low amplitude values ​​and sparsely placed valid coefficients are efficiently compressed. Furthermore, two residual coding schemes exist. As seen when applying the transform, the conventional residual coding scheme is optimized for TBs where valid coefficients are primarily located in the upper left corner of the TB. The transform-skipped residual coding scheme can be used for TBs without the transform and efficiently encodes residual coefficients regardless of their distribution throughout the TB.

[0154] Multiplexer module 884 outputs PB 820 from intra-prediction module 864 based on the determined best intra-prediction mode selected from the test prediction modes of each candidate CB. Candidate prediction modes do not need to include every conceivable prediction mode supported by video encoder 800. Intra-prediction is divided into three types: first, “DC intra-prediction,” which involves filling PB with a single value representing the average of nearby reconstructed samples; second, “planar intra-prediction,” which involves filling PB with samples based on the plane, utilizing DC offset, and vertical and horizontal gradients derived from nearby reconstructed samples. Nearby reconstructed samples typically include a row of reconstructed samples above the current PB (extending to the right of the PB) and a column of reconstructed samples to the left of the current PB (extending downwards beyond the PB); and third, “angular intra-prediction,” which involves filling PB with reconstructed nearby samples filtered and propagated across the PB in a specific direction (or “angle”). In VVC, sixty-five (65) angles are supported, where rectangular blocks can utilize additional angles unavailable to square blocks to produce a total of eighty-seven (87) angles.

[0155] A fourth type of intra-frame prediction can be used for chroma blocks, thereby generating blocks from juxtaposed luma reconstruction samples according to a “Cross-Component Linear Model” (CCLM) mode. Three different CCLM modes are available, each using a different model derived from neighboring luma and chroma samples. The derived model is used to generate sample blocks for the chroma block from the juxtaposed luma samples. Intra-frame prediction of luma blocks can be performed using matrix multiplication of reference samples, using a matrix selected from a predefined set of matrices. This matrix intra-frame prediction (MIP) achieves gain by using matrices trained on a large set of video data, where the matrices represent the relationship between reference samples and prediction blocks that is not easily captured in angular, planar, or DC intra-frame prediction modes.

[0156] Module 864 can also generate prediction units by copying blocks from the vicinity of the current frame using an "intra-block copy" (IBC) method. The location of the reference block is constrained to an area equivalent to a CTU, which is divided into 64×64 regions called VPDUs. Regardless of the CTU size configured for the bitstream, this region covers the processed VPDU of the current CTU, as well as the VPDUs of the previous CTUs (one or more) within each row or CTU and within each slice or tile, up to a region limit corresponding to a 128×128 luma sample. This region is called the "IBC virtual buffer" and limits the IBC reference area, thereby limiting the required storage space. The IBC buffer is filled with reconstructed samples 854 (i.e., before loop filtering), thus requiring a buffer separate from the frame buffer 872. When the CTU size is 128×128, the virtual buffer includes samples only from the CTU adjacent to the left of the current CTU. When the CTU size is 32×32 or 64×64, the virtual buffer includes CTUs from up to four or sixteen CTUs to the left of the current CTU. Regardless of the CTU size, samples accessing neighboring CTUs to obtain the IBC reference block are constrained by boundaries (such as the edges of images, slices, or tiles). Particularly for feature maps with smaller FPN layers, using CTU sizes such as 32×32 or 64×64 yields a more aligned reference region to cover the set of previous feature maps. Accessing similar feature maps for IBC prediction offers the advantage of coding efficiency when feature map placement is ordered based on SAD, SSE, or other dissimilarity metrics.

[0157] The residuals of prediction blocks when encoding feature map data differ from those seen for natural video. Natural video is typically captured by image sensors or screen content, as commonly seen in operating system user interfaces, etc. Feature map residuals tend to contain a great deal of detail. The level of detail in feature map residuals is more suitable for transform skip coding compared to the significantly low-frequency coefficients of various transforms. Intra-frame predictive luminance coding blocks can be partitioned vertically or horizontally into sets of prediction blocks of equal size, each block having a minimum region of sixteen (16) luminance samples.

[0158] When neighboring samples that were previously reconstructed are not available, such as at the edge of a frame, a default halftone value of half the sample range is used. For example, for 10-bit video, a value of 512 (512) is used. Since no previous samples are available for the CB located in the upper left position of the frame, the angular and planar intra-frame prediction modes produce the same output as the DC prediction mode (i.e., a flat plane of samples with halftone values ​​as amplitude).

[0159] For inter-frame prediction, the motion compensation module 880 generates a prediction block 882 using samples from one or two frames preceding the current frame in the coded sequence of the bitstream, and the multiplexer module 884 outputs this prediction block 882 as PB 820. Furthermore, for inter-frame prediction, a single coding tree is typically used for both the luma and chroma channels. The order of the coded frames in the bitstream may differ from the order of the frames at the time of capture or display. When one frame is used for prediction, the block is called "single prediction" and has one associated motion vector. When two frames are used for prediction, the block is called "double prediction" and has two associated motion vectors. For P-slices, each CU can be intra-predicted or single-predicted. For B-slices, each CU can be intra-predicted, single-predicted, or double-predicted.

[0160] Frames are typically encoded using a "picture group" structure, achieving a temporal hierarchy. A frame can be segmented into multiple slices, each slice encoding a portion of the frame. This temporal hierarchy allows frames to reference preceding and following images in the order they are displayed. Images are encoded in the necessary order to ensure the relevance for decoding each frame is met. Affine inter-frame prediction modes are available, where instead of using one or two motion vectors to select and filter the reference sample block for a prediction unit, the prediction unit is segmented into multiple smaller blocks, generating a motion field such that each smaller block has a different motion vector. The motion field uses the motion vectors of nearby points of the prediction unit as "control points." Affine prediction allows encoding motions other than translations with less need for depth-splitting coding trees. VVC's dual prediction mode geometrically blends two reference blocks along a selected axis and signals the angle and offset relative to the block's center. This geometric partitioning mode ("GPM") allows the use of larger coding units along the boundary between two objects, with the geometry of the coding unit's boundary serving as the angle and center offset. Motion vector differences can be encoded as direction (up / down / left / right) and distance (a set of distances supporting powers of 2) instead of using Cartesian (x, y) offsets. Motion vector predictors are obtained from neighboring blocks ("merge mode") as if no offset were applied. The current block will share the same motion vectors with the selected neighboring blocks.

[0161] Samples are selected based on motion vector 878 and reference image indexes. Motion vector 878 and reference image indexes are applied to all color channels, thus describing inter-frame prediction primarily in terms of operations on the PU rather than the PB. A single coding tree describes the decomposition of each CTU into one or more inter-frame prediction blocks. Inter-frame prediction methods can vary in the number and precision of motion parameters. Motion parameters typically include reference frame indices, which indicate which reference frame(s) in the reference frame list will be used, along with spatial translations of each reference frame, but can include more frames, specific frames, or complex affine parameters (such as scaling and rotation). Additionally, predetermined motion refinement processes can be applied to generate dense motion estimates based on reference sample blocks.

[0162] PB 820 has been identified and selected, and at subtractor 822, PB 820 is subtracted from the original sample block to obtain the residual (denoted as 824) with the lowest encoding cost, which is then subjected to lossy compression. The lossy compression process includes transform, quantization, and entropy coding steps. The forward master transform module 826 applies a forward transform to the difference 824, converting it from the spatial domain to the frequency domain, and produces the master transform coefficients indicated by arrow 828. The maximum master transform size in one dimension is a 32-point DCT-2 or 64-point DCT-2 transform configured by the "sps_max_luma_transform_size_64_flag" in the sequence parameter set. If the CB being encoded is larger than the maximum supported master transform size represented as the block size (e.g., 64×64 or 32×32), the master transform 826 is applied in a tiling manner to transform all samples of the difference 824. In the case of using a non-square CB, tiling is also performed using the maximum available transform size in each dimension of the CB. For example, when using a maximum transform size of 32 (32), a 64×16 CB uses two 32×16 primary transforms arranged in a tiled manner. When the size of the CB is larger than the maximum supported transform size, the CB is filled with TBs arranged in a tiled manner. For example, a 128×128 CB with a maximum transform size of 64-pt is filled with four 64×64 TBs arranged in a 2×2 manner. A 64×128 CB with a maximum transform size of 32-pt is filled with eight 32×32 TBs arranged in a 2×4 manner.

[0163] The application of transform 826 yields multiple TBs for the CB. When each application of the transform operates on a TB with a difference 824 greater than 32×32 (e.g., 64×64), all resulting master transform coefficients 828 outside the top-left 32×32 region of the TB are set to zero (i.e., discarded). The remaining master transform coefficients 828 are passed to quantizer module 834. The master transform coefficients 828 are quantized according to quantization parameter 892 associated with the CB to produce master transform coefficients 832. In addition to quantization parameter 892, quantizer module 834 can also apply a "scaling list" to allow non-uniform quantization within the TB by further scaling the residual coefficients according to their spatial location within the TB. Quantization parameter 892 can be different for the luma CB and each chromaticity CB. Master transform coefficients 832 are passed to forward quadratic transform module 830 to produce transform coefficients indicated by arrow 836 by performing an inseparable quadratic transform (NSST) operation or by bypassing the quadratic transform. The forward master transform 826 is typically separable, thus transforming the row set and then the column set of each TB. For luminance TBs with a width and height not exceeding 16 samples, the forward master transform module 826 uses a Type II Discrete Cosine Transform (DCT-2) in both the horizontal and vertical directions, or a bypass of the transform in both the horizontal and vertical directions, or a combination of a Type VII Discrete Sine Transform (DST-7) and a Type VIII Discrete Cosine Transform (DCT-8) in either the horizontal or vertical direction. In the VVC standard, the use of a combination of DST-7 and DCT-8 is referred to as the "Multiple Transform Selection Set" (MTS).

[0164] The forward quadratic transform of module 830 is typically an inseparable transform, applied only to the residuals of the intra-frame prediction CU, and can, however, be bypassed. The forward quadratic transform operates on sixteen (16) samples (arranged in the top-left 4×4 sub-block of the main transform coefficients 828) or forty-eight (48) samples (arranged in three 4×4 sub-blocks of the top-left 8×8 coefficients of the main transform coefficients 828) to produce a set of quadratic transform coefficients. The set of quadratic transform coefficients can be smaller in number than the set of main transform coefficients from which they are derived. Because the quadratic transform is applied only to the set of coefficients that are adjacent to each other and include DC coefficients, it is called the “low-frequency inseparable quadratic transform” (LFNST). This quadratic transform can be obtained through training and, due to its inseparable nature and the trained origin, can take advantage of additional redundancy in the residual signal that cannot be captured by separable transforms (such as variations of DCT and DST). Furthermore, when LFNST is applied, all residual coefficients in the TB are zero in both the main transform domain and the quadratic transform domain.

[0165] The quantization parameter 892 is constant for a given TB, and thus a uniform scaling of the residual coefficients generated in the main transform domain for the TB is obtained. The quantization parameter 892 can be periodically varied with a signaled “differential quantization parameter”. For a CU contained within a given region (called a “quantization group”), the differential quantization parameter (differential QP) is signaled once. If the CU is larger than the quantization group size, the differential QP is signaled once using one of the TBs of the CU. That is, the differential QP is signaled once for the first quantization group of the CU by the entropy encoder 838, but not for any subsequent quantization groups of the CU. Non-uniform scaling can also be achieved by applying a “quantization matrix”, whereby the scaling factor applied for each residual coefficient is derived from a combination of the quantization parameter 892 and the corresponding entries in the scaling matrix. The scaling matrix can have a size smaller than the TB, and when applied to the TB, a nearest neighbor approach is used to provide the scaling value for each residual coefficient from a scaling matrix smaller than the TB size. The residual coefficients 836 are fed to the entropy encoder 838 for encoding in the bitstream portion 816. Typically, according to the scan pattern, the residual coefficients of each TB with at least one valid residual coefficient are scanned from the TU to produce an ordered list of values. The scan pattern typically scans the TB as a sequence of 4×4 “sub-blocks”, thus providing regular scan operations at the granularity of a 4×4 set of residual coefficients, where the arrangement of the sub-blocks depends on the size of the TB. The scan within each sub-block and the progression from one sub-block to the next typically follow a reverse diagonal scan pattern. Additionally, the quantization parameter 892 is encoded into the bitstream portion 716 using differential QP syntax elements, and the slice QP and quadratic transform index 888, given the initial values ​​in the slice or sub-picture, are encoded into the bitstream portion 816.

[0166] As described above, the video encoder 800 needs access to the frame representation corresponding to the decoded frame representation seen in the video decoder. Therefore, the residual coefficients 836 are passed through the inverse quadratic transform module 844, which operates according to the quadratic transform index 888 to produce intermediate inverse transform coefficients, indicated by arrow 842. The intermediate inverse transform coefficients 842 are inversely quantized by the dequantizer module 840 according to the quantization parameter 892 to produce the inverse transform coefficients, denoted as 846. The dequantizer module 840 can also perform inverse non-uniform scaling of the residual coefficients using a scaling list, corresponding to the forward scaling performed in the quantizer module 834. The inverse transform coefficients 846 are passed to the inverse master transform module 848 to produce residual samples of the TU (indicated by arrow 850). The inverse master transform module 848 applies the DCT-2 transform horizontally and vertically, constrained by the maximum available transform size described with reference to the forward master transform module 826. The type of inverse transformation performed by the inverse quadratic transformation module 844 corresponds to the type of forward transformation performed by the forward quadratic transformation module 830. The type of inverse transformation performed by the inverse master transformation module 848 corresponds to the type of master transformation performed by the master transformation module 826. The summation module 852 adds the residual sample 850 and PB 820 to generate the reconstructed sample of CU (indicated by arrow 854).

[0167] Reconstructed sample 854 is passed to reference sample cache 856 and in-loop filter module 868. Reference sample cache 856, typically implemented using static RAM on an ASIC to avoid expensive off-chip memory access, provides minimal sample storage to satisfy the dependencies used to generate intra-frame PBs for subsequent CUs in a frame. Minimal dependencies typically include a “line buffer” of samples below a row of CTUs for use by the next row of CTUs, as well as a column buffer (whose extent is set by the height of the CTU). Reference sample cache 856 feeds reference samples (indicated by arrow 858) to reference sample filter 860. Sample filter 860 applies a smoothing operation to produce filtered reference samples (indicated by arrow 862). Filtered reference samples 862 are used by intra-frame prediction module 864 to produce intra-frame prediction blocks of samples, indicated by arrow 866. For each candidate intra-frame prediction mode, intra-frame prediction module 864 produces sample blocks (i.e., 866). Sample blocks 866 are generated by module 864 using techniques such as DC, planar, or angular intra-frame prediction. Alternatively, a matrix multiplication approach can be used to generate sample block 866, with neighboring reference samples as input, and the video encoder 800 selects a matrix from the matrix set, using an index to signal the selected matrix in the bit stream 816 to identify which matrix in the matrix set will be used by the video decoder.

[0168] The in-loop filter module 868 applies several filtering stages to the reconstructed sample 854. The filtering stages include a "deblocking filter" (DBF), which applies smoothing aligned with the CU boundaries to reduce artifacts caused by discontinuities. Another filtering stage present in the in-loop filter module 868 is an "adaptive loop filter" (ALF), which applies a Wiener-based adaptive filter to further reduce distortion. Another available filtering stage in the in-loop filter module 868 is a "sample adaptive offset" (SAO) filter. The SAO filter operates by first classifying the reconstructed sample into one or more categories and then applying an offset at the sample level based on the assigned category.

[0169] The filtered sample, indicated by arrow 870, is output from the in-loop filter module 868. The filtered sample 870 is stored in the frame buffer 872. The frame buffer 872 typically has a capacity to store several (e.g., up to sixteen (16)) images and is therefore stored in memory 206. Due to the large memory consumption required, on-chip memory is typically not used to store the frame buffer 872. Thus, access to the frame buffer 872 is expensive in terms of memory bandwidth. The frame buffer 872 provides a reference frame (indicated by arrow 874) to the motion estimation module 876 and the motion compensation module 880. The reference frame 874 is output as the reconstructed frame 818 of the encoder module 542. Figure 8 In the example, reconstructed frame 818 is the result of lossy VVC encoding operations caused by the operations of modules 810 to 890.

[0170] Motion estimation module 876 estimates multiple "motion vectors" (denoted as 878), each a Cartesian space offset relative to the current CB position, thus referencing a block of one of the reference frames in frame buffer 872. A filtered block (denoted as 882) of reference samples is generated for each motion vector. The filtered reference samples 882 form further candidate modes for potential selection by mode selector 886. Furthermore, for a given CU, PB 820 can be formed using one reference block ("single prediction") or two reference blocks ("double prediction"). For the selected motion vector, motion compensation module 880 generates PB 820 based on filtering that supports sub-pixel accuracy in the motion vector. Thus, motion estimation module 876 (which operates on many candidate motion vectors) can perform simplified filtering compared to motion compensation module 880 (which operates only on the selected candidate), achieving reduced computational complexity. When video encoder 800 selects inter-frame prediction for a CU, motion vector 878 is encoded into bitstream portion 816.

[0171] Although the reference to Universal Video Coding (VVC) explains Figure 8The video encoder 800 can be used, but other video encoding standards or implementations can also employ the processing stages of modules 810 to 890. It can also be sourced from memory 206, hard disk drive 210, CD-ROM, Blu-ray disc. TM Read from or transfer to memory 206, hard disk drive 210, CD-ROM, Blu-ray disk, or other computer-readable storage media. TM Frame data 802 (and bit stream 816) can be written to (or other computer-readable storage media). Additionally, frame data 802 (and bit stream 816) can be received from (or transmitted to) an external source, such as a server or RF receiver connected to communication network 220. Bit stream 816 is associated with one of bit streams 121 and 151. Communication network 220 can provide limited bandwidth, thus requiring rate control in video encoder 800 to avoid network saturation when frame data 802 is difficult to compress.

[0172] Bitstream 816 may be constructed from one or more slices representing spatial portions (sets of CTUs) of frame data 802, which are generated by one or more instances of video encoder 800, each of which generates bitstream portion 816 and operates in a coordinated manner under the control of processor 205. Bitstream portion 816 may also contain a slice corresponding to a region to be output as a set of sub-pictures (each sub-picture being independently coded and independently decoded relative to other slices in the picture or any slice or sub-picture in the sub-picture) forming a picture.

[0173] Figure 9A and Figure 9B This is a schematic diagram showing how an image is divided into regions. Figure 9A Image 900 shows regions 910, 912, and 916 segmented to pack data from the inter-channel decorrelation (PCA-based) encoder network topology 700. Feature maps of each tensor are packed by packer 538. Figure 9A and Figure 9B The example shows three zones corresponding to the mean channel, basis vectors, and coefficients.

[0174] refer to Figure 9B Image 900b corresponds to image 900, region 910b corresponds to region 910, and region 912b corresponds to region 912. Region 910b retains the mean channel data, such as the mean channel 920 of each tensor in tensor 115. Figure 9BIn the example, the basis vectors include basis vector 922 and others, wherein the basis vectors are packed into the region of region 912b in a non-overlapping manner. Basis vector 922 forms a C×C' array of integer quantized values, where C is the number of channels in the tensor of tensor 115, and C' is the number of basis vectors generated by the decomposition performed by decomposition module 718. In cases where multiple tensors are decomposed into sets of basis vectors (such as when tensor 115 comprises multiple tensors), basis vector 922 comprises sets of basis vectors arranged in a non-overlapping, left-aligned, vertically concatenated manner. For example, the mean channel 920 corresponds to channel 712, and basis vector 922 corresponds to some of the basis vectors in basis vector 720.

[0175] The region in image 900 (or 900b) that is not used to store any data can be occupied by the sample value corresponding to the value "0" after applying dequantization to convert the sample value back to the floating-point domain. Similarly, the regions in areas 910 (or 910b), 912 (or 912b), and 916 that are not used to store any data can be occupied by the sample value corresponding to the value "0" after applying dequantization to convert the sample value back to the floating-point domain. Area 916 remains... Figure 7 The coefficient tensor data corresponding to 724, where the coefficients of each basis vector form a width × height feature map. The coefficient region 916 has coefficients that are, for example, packed into twenty-four feature maps (applicable to basis vectors [0…23]) such as feature map 930.

[0176] Each zone (910 / 910b, 912 / 912b, and 916 / 916b) is aligned with the CTU grid (typically 128×128). When addressing using Cartesian coordinates in units of CTU size, the location of each zone is assigned as the top-left and bottom-right Cartesian CTU addresses. For example, zone 916 occupies the CTU from (0,1) down to (6,3) (corresponding to the top-left luminance sample at position (0, 128) or the bottom-right luminance sample at (6×128 + 127, 3×128 + 127) or (895, 511)).

[0177] Figure 10A and Figure 10B This is a schematic diagram illustrating how an image is divided into sections. Figure 10A In the image 1000, the image is divided into a region 1010, which is suitable for packaging into a compressed tensor 532 that can be generated by the tensor compressor 600. Figure 10BImage 1000b, corresponding to image 1000, is shown, in which feature maps are packed into region 1010b corresponding to 1010. The feature maps of the compressed tensors (such as feature map 1030, etc.) are packed into region 1010b, such that the feature maps are packed from left to right, so that once the available space on the current line is exhausted, the next line is moved on. With the aid of tensor compressor 600, the area required to pack the compressed tensor 532 is less than the area required if the tensors in tensor 115 were packed into image 1000. Region 1010 is defined based on Cartesian CTU addresses at the top left and bottom right.

[0178] Figure 11 This is a schematic block diagram illustrating an example structure 1100 of a multi-layer bitstream 123 (including sub-bitstreams 121 or 151) that preserves encoded, packed feature maps and associated metadata. The multi-layer bitstream 123 contains groups of syntax elements, each beginning with a "Network Abstraction Layer" (NAL) unit header. For example, the NAL unit header 1108 precedes the Sequence Parameter Set (SPS) 1110a. A Scalability Dimension Information (SDI) SEI message 1112 signals the nature of the multi-layer bitstream 123 and its presence when using a VVC codec. The SEI message 1112 is included once for the first Access Unit (AU) in the encoded video sequence. The SEI message 1112 uses the sdi_max_layers_minus1 syntax element to signal the number of layers (typically two). The first (main) layer is the video bitstream 151. The second layer is an auxiliary layer, signaled by using an sdi_auxiliary_info_flag set to 1 and a corresponding sdi_aux_id[i] flag set to indicate a feature layer (i.e., for feature layer i=1) (e.g., feature layer 121). A value of sdi_aux_id[i] ranging from 3 to 127 (e.g., 3) indicates a feature layer (AUX_FEATURE) standardized in the ITU-T Recommendation H.274|ISO / IEC 23002-7. A value of sdi_aux_id[i] ranging from 128 to 159 (e.g., 128) indicates a feature layer standardized in the separate standard of ITU-T Recommendation H.274|ISO / IEC 23002-7. The auxiliary (feature) layer is associated with the primary (video) layer using the sdi_associated_primary_layer_idx syntax element.

[0179] The Video Parameter Set (VPS) 1115 signals the maximum nuh_layer_id used by sub-bitstreams existing in the multi-layer bitstream, along with flags indicating which nuh_layer_id values ​​exist in the current VPS. When using HEVC, both the syntax elements vps_base_layer_internal_flag and vps_base_layer_available_flag are set to 1, indicating that bitstream 123 includes a base layer. When using VVC, the syntax element vps_all_independent_layers_flag is set to 1, indicating that there is no inter-layer prediction between the video (base) layer and the feature (auxiliary) layer. The ptl_multilayer_enabled_flag, set to 1, signals the “profile_level_tier” syntax in VPS 1115, where separate instances of the video layer and feature layer are further indicated via the vps_num_ptls_minus1 syntax element. Additional instances of the profile_level_tier syntax exist for additional feature layers (additional second sub-bitstreams, if present). The `profile_level_tier` syntax provides a means of signaling the profile, level, and tier of the sub-bitstream. SPS 1110a includes signaling the bit depth, chroma format, and resolution of the video layer 151 (also called the "base layer"), and the associated NAL unit is encoded using a `nuh_layer_id` syntax element set to 0. SPS 1110b, depending on the compressor and decompressor selected at step 1810, includes signaling the bit depth, chroma format, and resolution of the feature images (e.g., image 900 or image 1000) of feature layer 121. The NAL unit associated with SPS 1110b (that is, with feature layer sub-bitstream 121) is signaled, where `nuh_layer_id` is not equal to zero and is typically set to 1. If multiple feature sub-bitstreams exist in multi-layer bitstream 123, each sub-bitstream will have a unique value for `nuh_layer_id` and a separate SPS. Sub-bitstreams in multi-layer bitstream 123 are extracted through sub-bitstream extraction processing. This sub-bitstream extraction processing filters bitstream 123 based on a layer identifier [nuh_layer_id] to obtain sub-bitstreams from bitstream 123. For example, video layer 151 (first sub-bitstream) is obtained through first sub-bitstream extraction processing. This first sub-bitstream extraction processing filters bitstream 123 based on identifying a first layer identifier of video layer 151 in bitstream 123 to obtain video layer 151.Furthermore, feature layer 121 (second sub-bit stream) is obtained through a second sub-bit stream extraction process. This second sub-bit stream extraction process filters bit stream 123 based on a second-layer identifier (different from the first-layer identifier) ​​that identifies feature layer 121 in bit stream 123 to obtain feature layer 121. Each sub-bit stream can be decoded independently using separate processing. For example, as described below. Figure 19 As described, different processing is used to filter video layer 151 (171) and feature layer 121 (143) from bit stream 123.

[0180] When using HEVC, a multi-layer master profile can be used to encode and decode the video parameter set. This requires using a 4:2:0 chroma format at 8-bit depth for both video layer 151 (first sub-stream) and feature layer 121 (second sub-stream). For the feature layer, 128 will be used as the neutral value for the chroma samples. `scalability_mask_flag` indicates that the scalability dimension of the feature layer is "auxiliary," where `AuxId` is used to signal the type of the auxiliary layer (i.e., the feature layer).

[0181] When using VVC, a multi-layer master 10 profile can be used to encode and decode the video parameter set, allowing 4:0:0 (monochrome) for feature layer 151 (first sub-bit stream) and a bit depth in the range of 8 to 10 bits. In some implementations, the 4:0:0 chroma format can be used for feature layer 121 (second sub-bit stream). Using sub-picture information 1111, the layout of the pictures includes the position and size of arbitrary sub-pictures (e.g., 910, 912, and 916). SPS 1110 needs to be decoded before decoding a sequence of frames that are related by the prediction structure (i.e., inter-frame prediction reference pictures) and generally have the same width, height, bit depth, and chroma format. In applications involving streaming retransmissions of SDI SEI 1112, FCVMC SEI 1113, VPS 1115, and SPS 1110a and 1110b, these streaming retransmissions can be performed so that decoding can begin at the midpoint of the bit stream. Furthermore, if the decoder network topology is modified during execution, the required frame area of ​​the compressed tensor may change, necessitating the insertion of another SPS to signal the updated frame dimension. SPS 1110a also indicates the chroma format, bit depth, and resolution of the frame data represented by bitstream 121. The encoded sub-picture 1122, which encodes sub-picture 912, includes a slice header 1130 followed by slice data 1140. Slice data 1140 includes a CTU sequence, providing an encoded representation of the frame data. Encoded sub-picture 1124 encodes sub-picture 916 corresponding to the coefficients. Multi-layer bitstream 123 includes an FCVCM decoder network instruction SEI message 1113, written by metadata encoder 544 and syntactically conforming to or equivalent to the syntax described in Appendix A. SEI message 1113 encodes the metadata required to transform the decoded frame into a set of tensors suitable for use by NN section 2 166.

[0182] Figure 12 This is a schematic block diagram showing an implementation 1200 of a tensor decoder 146. The decoder 1200 includes a tensor decompressor 1250. Figure 19 Method 1900 is illustrated, which is used to decode a multi-layer bitstream 123, reconstruct tensors using a tensor decompressor, and perform NN part 2 to complete the execution of a neural network, such that NN part 2 is selected to be compatible with the tensors decoded from the multi-layer bitstream 123. As described, method 1900 is configured to determine a set of compatible NN part 2 candidates from a library of available NN part 2 implementations in destination device 140, and select one of the compatible NN part 2 candidates for execution in destination device 140. The multi-layer bitstream 123 includes a main (video) sub-bitstream 171 and an auxiliary (feature) sub-bitstream 143, each of which is decoded to produce a video frame 172 and a decoded tensor 149.

[0183] Destination device 140 and method 1900 can be implemented as one or more software applications 233 executable within computer system 200. Destination device 140 and method 1900 can be implemented via instructions 231 in the software 233 executing within computer system 200 (see...). Figure 2B This can be achieved by... The software instructions 231 can be configured into one or more code modules, each for performing one or more specific tasks. Method 1900 begins with the video frame decoding step 1910.

[0184] At step 1910, a decoder (such as decoder 170 or decoder 146) decodes the video parameter set (VPS) 1115 used to indicate the presence of multiple sub-bitstreams in bitstream 123. The decoding of VPS 1115 is based on whether a VVC or HEVC multi-layer master profile is used.

[0185] Furthermore, at step 1910, video decoder 170, under the execution of processor 205, decodes frames from main (video) sub-bit stream 171 to produce decoded frame 172. Video decoder 160 can be implemented as described in the reference. Figure 13 The described VVC decoder, or HEVC decoder, is used. In the case of multiple sub-bit streams existing in bit stream 123, video decoder 170 operates to decode video from a first sub-bit stream (video sub-bit stream 171) obtained by performing a first sub-bit stream extraction process on bit stream 123. Control in processor 205 proceeds from step 1910 to the decoding candidate task result step 1920.

[0186] At step 1920, refer to Figure 12 The receiver / demultiplexer 142 decodes the candidate task result 11104 (FIG. 3) from the multi-layer bitstream 123. The candidate task result 11104 indicates the result of the surrogate NN portion 2 1118 performed in the source device 110, and may include, for example, one or more bounding boxes or image classifications. Control in the processor 205 proceeds from step 1920 to the task result checking step 1925.

[0187] At step 1925, the processor determines whether candidate task result 11104 indicates a frame of interest. If the destination device is configured to track people in the scene, candidate task result 11104 indicating the detection of one or more people will indicate a frame of interest, and control in processor 205 will move from step 1925 ("Yes" at step 1925) to decoding NN indication step 1930. If candidate task result 11104 does not indicate any detected people, method 1900 terminates, and processor 205 proceeds to the next frame of data in multi-layer bitstream 123.

[0188] At step 1930, receiver / demultiplexer 142, under the execution of processor 205, decodes NN topology indication 1197 and NN weight indication 1198 from multi-layer bitstream 123. NN topology indication 1197 and NN weight indication 1198 can be hashes of the overall NN topology and weights, that is, the combined network implemented in system 100, without considering splitting the network into parts to be performed in source device 110 and destination device 140. Hashes typically have fixed-length values, such as 256-bit sha1sum, sha256, sha2-384, or sha2-512 values ​​calculated from the byte-serialized network topology and network weights. Receiver / demultiplexer 142 can use the reference... Figure 13 The entropy decoder 1320 is described to parse the bitstream data. Control in processor 205 moves from step 1930 to the decoding NN section 1, indicating step 1940.

[0189] At step 1940, receiver / demultiplexer 142, under the execution of processor 205, decodes NN Part 1 topology indication 11100 and NN Part 1 weight indication 11101 (collectively referred to as 161) from multi-layer bitstream 123. Indicators 11101 and 11101 (161) are hashes used to indicate network topology and weights corresponding only to NN Part 1 as performed in NN Part 1 module 114. At step 1940, receiver / demultiplexer 142 decodes NN Part 1 output indication 11102, which encodes the number and dimensions of tensors generated by NN Part 1 (i.e., prior to further tensor compression performed by tensor encoder 116). Knowledge of the tensor dimensions output from NN Part 1 114 allows the decoder to determine the dimensions of tensors to be supplied to NN Part 2 candidates without mapping NN Part 1 topology indication 1197 to the actual network topology. In other words, destination device 140 can determine the tensor dimension without needing to determine the encoder-side network topology used to generate the multi-layer bitstream 123. Control in processor 205 proceeds from step 1940 to step 1960 of selecting NN section 2 candidates.

[0190] At step 1960, the NN Part 2 repository 160 is queried using the NN Part 1 topology indicator 11100 and the NN Part 1 weight indicator 11101. Based on the associations between the registered NN Parts 1 and 2, a set of compatible NN Part 2 candidates is determined. Each NN Part 2 candidate is able to obtain the decoded tensor output from NN Part 1 114 to produce task results. As an example, Mask R-CNN for instance segmentation is a superset of the Faster R-CNN network for object detection (both use the ResNet-101 architecture in this example). Thus, the split points can arbitrarily divide the network into parts to some extent, that is, the basis for the split can be driven by the computational offloading of the desired degree from the destination device 140 to the source device 110, rather than being restricted by topological constraints. Widely used weight files (such as those from Meta...) TM The weight files (such as those published by AI Research as part of the Detectron2 framework) are trained differently on most layers of Mask R-CNN and Faster R-CNN, so tensors encoded by one network at split points cannot be further processed by the other network. Therefore, while network topology indicators show compatibility between the two networks at, for example, split points at layer P, weight indicators show incompatibility. It is possible to use weights other than those generated by Meta... TM Weights other than those published by AI Research include weights trained to be common to layers up to the P-layer split point. When compatible (NN Part 1 indicates matching) weights are used in the source device 110, the multi-layer bitstream 123 can be used accordingly for both object detection and instance segmentation. Implementers of the destination device 140 can train the weights of NN Part 2 for tasks that are more specific than those performed by the proxy NN Part 2 118. Detection of specific objects, such as a specific car or person, is an example. A common practice when training NNs is to “freeze” the layers closer to the beginning of the network (these layers tend to learn lower-level features) and train later layers only for specific tasks, thereby reducing training time that could otherwise be quite long. Freezing earlier layers with flexible selection of later layers is similar to splitting a network topology into NN Part 1 and NN Part 2, which has a common “backbone” and the same weights for NN Part 1 and different weights for each NN Part 2 candidate. Figure 20 A method is described for generating such indications for various candidates of NN portion 2 that may be available in the memory 206 of the destination device 206. In particular, since the network topology and weights are static, the indications can be pre-calculated or obtained via external components for use at runtime. Control in processor 205 proceeds from step 1960 to step 1970 to determine the bitstream compatibility check test.

[0191] At step 1970, under the execution of processor 205, destination device 140 determines whether multilayer bitstream 123 is compatible with destination device 140 based on whether at least one candidate NN portion 2 was found at step 1960. Control in processor 205 then proceeds from step 1970 to compatibility check test step 1980.

[0192] At step 1980, if it is determined in step 1970 that no compatible NN Part 2 candidate is available in destination device 140 (“Not OK” at step 1980), then control in processor 205 proceeds to error condition step 1990. If at least one compatible NN Part 2 candidate is available in destination device 140 (“OK” at step 1980), then control in processor 205 proceeds to NN Part 2 selection step 19100. Effectively, steps 1970 and 1980 determine whether to decode the tensor from bitstream 123. Steps 1970 and 1980 determine whether to use the decoded tensor from multi-layer bitstream 123 for NN Part 2.

[0193] Steps 1970 and 1980 can be considered to have the effect of determining whether to use the second part of the network (NN part 2) to decode the tensor from the bitstream. This determination is based on the availability of, for example, multiple second parts of the network associated with the first part of the network (NN part 1) in the NN part 2 storage 160. If available, that is, if the decoding of the tensor is to be performed, one of the available networks is selected for execution.

[0194] At step 1990, under the execution of processor 205, destination device 140 signals to the user that it cannot process the feature sub-bit stream 143 within the received multi-layer bit stream 123. The decoded video 172 can still be presented to the user based on the results of the proxy NN part 2; that is, even if more specific filtering cannot be performed within destination device 140, the video can still be displayed when source device 140 detects an object of interest. Method 1900 terminates and processor 205 proceeds to the next frame.

[0195] If steps 1970 and 1980 are performed to determine the tensor to be decoded from the bitstream, then step 19100 is implemented. At step 19100, processor 205 selects an NN part 2 from the candidates determined in step 1960, wherein network weights and topology are retrieved from the NN part 2 store 160. Control in processor 205 then proceeds from step 19100 to step 19110, which instantiates the selected NN part 2.

[0196] At step 19110, the selected NN part 2 is instantiated or loaded to form NN part 2 166, where both the network topology and weights are initialized. Control in processor 205 then proceeds from step 19110 to decoding the packaged frame step 19120.

[0197] At step 19120, the image decoder 1204, under the execution of the processor 205, decodes a packed frame from the feature sub-bit stream 143 to produce a decoded frame 1210. (See reference...) Figure 13 The operation of image decoder 1204 is described below. Control in processor 205 proceeds from step 19120 to unpacking tensor step 19130.

[0198] At step 19130, the unpacker 1214 of the decoder 146, under the execution of the processor 205, according to as referenced Figure 9A and Figure 9B as well as Figure 10A and Figure 10B The described packing format is used to read feature maps from decoded frame 1210. For each tensor, multiple feature maps are decoded, the number corresponding to the number of channels used in the tensor as signaled in tensor information 1195. The channels of each tensor are unpacked into two-dimensional feature maps. The number of feature maps or channels to be decoded for a given tensor is decoded from feature sub-bit stream 143 as a "channel count". Multiple tensors, each having one channel, are arranged horizontally or vertically within slices or sub-pictures of the frame. At step 19130, unpacker 1214 unpacks the feature maps of each previously determined type using the assignment of feature maps determined for frame (e.g., 1210) in method 1800. This assignment can be decoded from the SEI message. Unpacker 1214 outputs integer tensor 1216, which has been decoded using the decoder topology of tensor decoder 146. Control in processor 205 proceeds from step 19130 to inverse tensor step 19140.

[0199] At step 19140, the inverse quantizer 1218 of the decoder 1200, under the execution of the processor 205, inverse-quantizes the integer tensor 1216 to produce the inverse-quantized tensor 1220 by applying the quantization range (such as quantization range 1196, etc.) decoded from the multi-layer bitstream 123 to a determined channel count for each tensor. For inverse quantization, the quantization range 1196 is decoded from the SEI message 1113 using the receiver / demultiplexer 142. Control in the processor 205 then proceeds from step 19140 to the buffered quantized tensor step 19150.

[0200] At step 19150, the tensor storage module 1222 of the decoder 1200 provides inter-frame storage of the inverse-quantized tensor 1220 under the execution of the processor 205. In the tensors of each region of the packet format, each tensor having at least one decoded channel or feature map is stored in the tensor storage module 1222. The tensor storage module 1222 generates an output tensor 1224, which includes the most recent tensor of each tensor for which at least one feature map has been decoded. In other words, in the case where a tensor is not decoded for the current frame (i.e., the channel count is determined to be zero), the latest value of the tensor decoded using a non-zero channel count is used. For example, when using a PCA-based method (such as encoder network topology 700, etc.), the mean channel generated by the mean channel module 710 may be updated infrequently; in this case, the tensor storage module 1222 retains the latest result from the mean channel module 710 for use by the tensor decompressor 1250. The control in processor 205 proceeds from step 19150 to step 19160 for tensor decompression.

[0201] At step 19160, tensor decompressor 1250, executed by processor 205, uses tensor 1224 as input to perform steps specified by the decoder network topology to produce decoded tensor 149. (About...) Figure 14 An example structure for a tensor decompressor is described.

[0202] In the case where multiple sub-bitstreams exist in bitstream 123, steps 19100 to 19160 are performed to decode the tensor of the decoded video from the second sub-bitstream (feature sub-bitstream 143) obtained by performing a second sub-bitstream extraction process on bitstream 123. Step 19160 decodes the tensor of the decoded video from the second sub-bitstream. The video and feature layers or bitstreams are extracted using different processes and can be decoded independently.

[0203] Control in processor 205 moves from step 19160 to step 19170, which involves performing a second part of the neural network to generate task results using a portion of the neural network.

[0204] At step 19170, under the execution of processor 205, module 166 uses tensor 149 as input and produces task result 151 as output to perform NN part 2 as selected at step 19100 and implemented as NN part 2 166. Control in processor 205 proceeds from step 19170 to the task result rendering step 19180. At step 19180, under the execution of processor 205, task result renderer 168 generates video frames that associate the task results with the video layer. For example, step 168 may present both video frame 172 and a representation of task result 151. Task result 151 may be represented as a bounding box around an object of interest, or a segmentation map with color-coded objects of interest, etc. The video frame generated from step 19180 is presented to the user of destination device 140 as the final output of system 100, for example via display 214.

[0205] If, in steps 1970 and 1980, the tensor is determined to be decoded from the bitstream, steps 19100 to 19180 are performed to perform the second part of the network (NN part 2) to produce a neural network result. Method 1900 terminates, and the processor 205 may re-invoke method 1900 when the next frame in the multi-layer bitstream 123 is received.

[0206] Figure 13 This is a schematic block diagram illustrating the functional modules of video decoder 1300. The video decoder provides example implementations of both image decoder 1204 and video decoder 170. Video decoder 1300 can be implemented as one or more software applications 233 executable within computer system 200. Video decoder 1204 can be implemented via instructions 231 (see [link to instructions]) in the software 233 executing within computer system 200. Figure 2B This can be achieved by [the following]. The software instructions 231 can be configured as one or more code modules, each for performing one or more specific tasks. The video decoder 1204 is operable to decode both the main (video) sub-stream 171 and the auxiliary (feature) sub-stream 143.

[0207] One of sub-streams 143 and 171 is input as sub-stream 1301 to entropy decoder module 1320. Entropy decoder module 1320 extracts syntactic elements from sub-stream 1301 by decoding the "bin" sequence and passes the values ​​of the syntactic elements to other modules in video decoder 1204. Entropy decoder module 1320 uses variable-length and fixed-length decoding to decode SPS, PPS, or slice headers using an arithmetic decoding engine to decode the syntactic elements of the slice data into a sequence of one or more bins. Each bin can use one or more "contexts," where the context describes the probability levels of "one" and "zero" values ​​to be used to encode the bin. When multiple contexts are available for a given bin, a "context modeling" or "context selection" step is performed to select one of the available contexts to decode the bin. When each slice can be decoded as a whole by a given instance of entropy decoder 1320, the process of decoding bins forms a sequential feedback loop.

[0208] The entropy decoder module 1320 applies an arithmetic coding algorithm, such as Context Adaptive Binary Arithmetic Coding (CABAC), to decode syntactic elements from the sub-bit stream 143 or 171 (1301). The decoded syntactic elements are used to reconstruct parameters within the video decoder 1300. Parameters include residual coefficients (indicated by arrow 1324), quantization parameters 1374, quadratic transform index 1370, and mode selection information such as intra-frame prediction modes (indicated by arrow 1358). The mode selection information also includes information such as motion vectors and partitioning each CTU into one or more CBs. Typically combined with sample data from previously decoded CBs, the parameters are used to generate PBs.

[0209] The residual coefficients 1324 are passed to the inverse quadratic transform module 1336, where a quadratic transform is applied according to the quadratic transform index, or no operation is performed (bypass). The inverse quadratic transform module 1336 produces the reconstructed transform coefficients 1332. That is, module 1336 produces the main transform domain coefficients from the quadratic transform domain coefficients. The reconstructed transform coefficients 1332 are input to the dequantizer module 1328. The dequantizer module 1328 performs inverse quantization (or "scaling") on the residual coefficients 1332 (that is, in the main transform domain) to construct the reconstructed intermediate transform coefficients indicated by arrow 1340 according to the quantization parameter 1374. The dequantizer module 1328 can also apply a scaling matrix to provide non-uniform dequantization within the TB, which corresponds to the operation of the dequantizer module 840. If a non-uniform inverse quantization matrix is ​​indicated in sub-stream 1301, the video decoder 1204 reads the quantization matrix from sub-stream 143 or 171 (1301) as a sequence of scaling factors and arranges the scaling factors into a matrix. Inverse scaling combines the quantization matrix with the quantization parameters to create reconstructed intermediate transform coefficients 1340.

[0210] The reconstructed transform coefficients 1340 are passed to the inverse master transform module 1344. Module 1344 transforms coefficients 1340 from the frequency domain back to the spatial domain. The inverse master transform module 1344 applies the inverse DCT-2 transform horizontally and vertically, constrained by the maximum available transform size as described in the reference forward master transform module 826. The result of the operation of module 1344 is the residual sample block indicated by arrow 1348. The residual sample block 1348 is equal in size to the corresponding CB. The residual sample 1348 is fed to the summing module 1350.

[0211] At summing module 1350, residual sample 1348 is added to the decoded PB (denoted as 1352) to produce a reconstructed sample block indicated by arrow 1356. Reconstructed sample 1356 is fed to reconstructed sample cache 1360 and in-loop filtering module 1388. In-loop filtering module 1388 produces a reconstructed block of frame samples, denoted as 1392. Frame sample 1392 is written to frame buffer 1396. Frame buffer 1396 outputs image or video frame 1397. If sub-bit stream 1301 is sub-bit stream 171, then frame 1397 corresponds to frame 172. If sub-bit stream 1301 is sub-bit stream 143, then frame 1397 corresponds to frame 1204.

[0212] The reconstruction sample cache 1360 operates in a manner similar to the reconstruction sample cache 856 of the video encoder 800. The reconstruction sample cache 1360 provides storage for the reconstructed samples required for intra-frame prediction of subsequent CBs in the absence of memory 206 (e.g., by using data 232, typically on-chip memory, instead). The reference sample, indicated by arrow 1364, is obtained from the reconstruction sample cache 1360 and fed to the reference sample filter 1368 to produce the filtered reference sample indicated by arrow 1372. The filtered reference sample 1372 is fed to the intra-frame prediction module 1376. Module 1376 generates blocks of intra-frame prediction samples, indicated by arrow 1380, based on the intra-frame prediction mode parameters 1358 signaled in the sub-bit stream 1301 and decoded by the entropy decoder 1320. The intra-frame prediction module 1376 supports the modes of the encoder-side module 864, including IBC and MIP. Sample block 1380 is generated using modes such as DC, planar or angular intra-frame prediction.

[0213] When the prediction mode of a color block (CB) is indicated to use intra-prediction in sub-stream 1301 (143 or 171), the intra-prediction sample 1380 is formed into a decoded block 1352 via multiplexer module 1384. Intra-prediction produces a prediction block (PB) of samples, where the PB is a block within a color component derived using "neighboring samples" from the same color component. Neighboring samples are those adjacent to the current block and have already been reconstructed due to their earlier position in the block decoding order. In the case of juxtaposition of luma and chroma blocks, the luma and chroma blocks can use different intra-prediction modes. However, two chroma CBs share the same intra-prediction mode.

[0214] When the prediction mode of the CB is indicated as inter-frame prediction in sub-stream 1301 (143 or 171), the motion compensation module 1334 generates a block of inter-frame prediction samples, represented as 1338. The inter-frame prediction sample block 1338 is generated using motion vectors decoded from sub-stream 143 or 171 by the entropy decoder 1320 and a reference frame index used to select and filter sample blocks 1398 from the frame buffer 1396. Sample block 1398 is obtained from previously decoded frames stored in the frame buffer 1396. For dual prediction, two sample blocks are generated and mixed together to produce samples for the decoded PB 1352. The frame buffer 1396 is filled with filtered block data 1392 from the in-loop filtering module 1388. Similar to the in-loop filtering module 868 of the video encoder 542, the in-loop filtering module 1388 applies any of the DBF, ALF, and SAO filtering operations. Generally, motion vectors are applied to both the luminance and chrominance channels, although the filtering processes used for subsample interpolation in the luminance and chrominance channels are different.

[0215] Figure 14 This is a schematic block diagram illustrating an implementation of a channel-based decorrelation-based tensor decoder 1400 that can be used as a tensor decompressor 1250. Multiple instances of the tensor decoder 1400 can be instantiated as tensor decompressors 1250 when multiple tensors need to be decompressed. Tensor 1224 is received from tensor storage 1222 and includes coefficient tensor 1414, basis vector tensor 1412, and channel mean tensor 1410. Dot product module 1420, executed by processor 205, generates a zero-centered tensor 1422 by performing a dot product on coefficients 1414 and basis vectors 1412. Summation module 1424 adds the zero-centered tensor 1422 to the mean channel 1410 to produce a reconstructed tensor 149 as the output from tensor decoder 1400. When it is not necessary to update an input type (such as the mean), the channel count of that type of input can be set to zero. When the channel count of the input is set to zero, the most recently received value of the input can be used instead.

[0216] Figure 15 This is a schematic block diagram illustrating a tensor decompressor 1500 (which may be used as tensor decompressor 1250 in some implementations) using a multi-scale feature reconstruction stage. The tensor decompressor 1500 includes a single-scale feature compression (SSFC) decompressor 1510. The SSFC decompressor 1510 receives a tensor 1224 with a reduced channel count (e.g., 64 channels) and passes the tensor 1224 to a convolutional layer 1512, which outputs a tensor 1513 with a restored channel count (e.g., 256 channels). The tensor 1513 is passed to a batch normalization module 1514 to produce a tensor 1515. The tensor 1515 is passed to a PreLU module 1516 to produce a tensor 1520.

[0217] The tensor decompressor 1500 includes an MSFR module 1530. The MSFR module operates to use one or more trained convolutional layers to extract data from a reference array. Figure 19The tensor 1520 generated in step 19160 produces multiple tensors. Upsampling modules 1532, 1534, and 1536 upsample tensor 1520 horizontally and vertically by factors of 2, 4, and 8, respectively, to produce tensors 1533, 1535, and 1537. Tensor 1537 forms an output (P'2, 1557) from MSFR module 1530 and is passed to downsampling module 1542. Downsampling module 1542 downsamples tensor 1537 horizontally and vertically by factors of 2 to produce tensor 1543 with the same dimensions as tensor 1535. Tensor 1543 is fed to convolutional layer 1548, which outputs tensor 1549. Summation module 1554 adds tensors 1535 and 1549 to produce tensor 1555 as the output (P'3) of MSFR module 1530.

[0218] Tensor 1535 is passed to downsampling module 1540. Downsampling module 1540 downsamples tensor 1535 horizontally and vertically by a factor of 2 to produce tensor 1541 with the same dimensions as tensor 1533. Tensor 1541 is fed to convolutional layer 1546, which outputs tensor 1547. Summation module 1552 adds tensors 1533 and 1547 to produce tensor 1553 as the output (P'4) of MSFR module 1530.

[0219] Tensor 1533 is passed to downsampling module 1538. Downsampling module 1538 downsamples tensor 1533 horizontally and vertically by a factor of 2 to produce tensor 1539 with the same dimensions as tensor 1520. Tensor 1539 is fed to convolutional layer 1544, which outputs tensor 1545. Summation module 1550 adds tensors 1520 and 1545 to produce tensor 1551 as the output (P'5) of MSFR module 1530. Tensors 1551, 1553, 1555, and 1557 provide tensor 149.

[0220] Figure 16A This is a schematic block diagram illustrating an example implementation 1600 of the NN part 2166 of a CNN for object detection (corresponding to the part of the "YOLOv3" network excluding the "DarkNet-53" backbone). For example, when as Figure 3A When implementing the CNN backbone in that way, you can use Figure 16AThe NN section 2 166. Depending on the task to be performed in the destination device 140, the NN section 2 166 can be replaced with a different network. The input tensor 149 is decomposed into tensors for each layer (i.e., tensors 1610, 1620, and 1634). Tensor 1610 is passed to the CBL module 1612 to generate tensor 1614. Tensor 1614 is passed to the detection module 1616 and the upgrade module 1622. The detection module outputs bounding boxes 1618 in the form of detection tensors. Bounding boxes 1618 are passed to the non-maximum suppression (NMS) module 1648.

[0221] To generate bounding boxes addressing coordinates in the original video data 113, scaling is performed at the upgrade module 1622 according to the original video width and height before resizing the NN part 1 of the network 114. The upgrade module 1622 receives tensors 1614 and 1620 and generates an upgrade tensor 1624, which is passed to the CBL module 1626. The CBL module 1626 generates tensor 1628 as output. Tensor 1628 is passed to the detection module 1630 and the upgrade module 1636. The detection module 1630 generates a detection tensor 1632, which is supplied to the NMS module 1648. The upgrade module 1636 is another instance of module 1622. The upgrade module 1636 receives tensors 1628 and 1634 and outputs the upgrade tensor 1638. The upgrade tensor 1638 is passed to the CBL module 1640, which outputs tensor 1642 to the detection module 1644. The detection module 1644 generates a detection tensor 1646, which is then supplied to the NMS module 1648.

[0222] CBL modules 1612, 1626, and 1640 each contain five CBL modules (e.g., Figure 3D The cascading of CBL module 360 ​​is shown. Upgrader modules 1622 and 1636 are as follows: Figure 16B Examples of upgrader module 1660 are shown. Module 1648 receives tensors 1618, 1632, and 1646 and outputs task result 151.

[0223] like Figure 16B As shown, the upgrader module 1660 accepts tensor 1662 (e.g., Figure 16A Tensor 1614 is used as input. Tensor 1662 is passed to CBL module 1666 (with the structure of module 360) to produce tensor 1668. Tensor 1668 is passed to upsampler 1670 to produce upsampled tensor 1672. Cascade module 1674 passes the upsampled tensor 1672 to a second input tensor 1664 (e.g., input to...). Figure 16AThe tensor 1620 of the upgrader 1622 is cascaded to produce tensor 1676.

[0224] Detection modules 1616, 1630, and 1644 are as follows: Figure 16C An example of detection module 1680 is shown. Detection module 1680 receives tensor 1682. Tensor 1682 is input to CBL module 1684, which has the structure of module 360. CBL module 1684 generates tensor 1686. Tensor 1686 is passed to convolution module 1688, which implements the detection kernel to generate result 1690. In some arrangements, the detection kernel applies a 1×1 kernel to produce feature map-related outputs at each of the three layers of the tensor. The detection kernel is 1×1×(B×(5+C)), where B is the number of bounding boxes that a particular unit can predict, typically three (3), and C is the number of classes, which can be eighty (80), making the kernel size two hundred and fifty-five (255) detection attributes (i.e., tensor 1690). The constant “5” represents four bounding box attributes (box center x, y and size scale x, y) and an object confidence level (“objectivity”). The detection kernel has the same spatial dimension as the input feature map, but the depth of the output corresponds to the detection attributes. The detection kernel is applied across various layers, typically three, resulting in a large number of candidate bounding boxes. The NMS module 1648 applies non-maximum suppression to the resulting bounding boxes to discard redundant boxes, such as overlapping predictions at similar scales, thus obtaining the final set of bounding boxes as the output of object detection.

[0225] Figure 17 This is a schematic block diagram showing the head 1700 of the CNN. The head 1700 can be implemented as NN part 2 166, where NN part 1 114 is implemented, for example, as the backbone 400. The head 1700 forms part of the overall network known as "Faster R-CNN" and includes a feature network (i.e., backbone 400), a region proposal network, and a detection network. The input to the head 1700 is tensor 149, which includes tensors 1710, 1712, 1714, 1716, and 1718 from layers P2 to P6. Tensors 1710, 1712, 1714, and 1716 from layers P2 to P5 are... Figure 4The outputs 477, 475, 473, and 471 from P2 to P5 correspond to these values. Tensors 1710, 1712, 1714, 1716, and 1718 from P2 to P6 are input to the Region Proposal Network (RPN) head module 1720. Tensor 1718 from P6 is generated by the max pooling module 1742, which operates on tensor 1716 from P5 to perform a 2×2 max pooling operation. The RPN head module 1720 convolves the input tensors to generate intermediate tensors. These intermediate tensors are fed into two subsequent sibling layers: (i) one for classification and (ii) one for bounding box or region of interest (ROI) regression. The resulting output is the classification and bounding box 1722. The classification and bounding box 1722 are passed to the NMS module 1724. The NMS module 1724 prunes redundant bounding boxes by removing overlapping boxes with lower scores to produce the pruned bounding box 1726. Bounding box 1726 is input to region of interest (ROI) pooler 1728. ROI pooler 1728 uses some layer tensors from tensor 149 (further described below) along with bounding box 1726 to generate fixed-size feature maps from various input-size maps using max pooling operations. In the max pooling operation, subsampling takes the maximum value from each group of input values ​​to produce a single output value in the output tensor.

[0226] The input to the ROI pooler 1728 consists of feature maps 1710, 1712, 1714, and 1716 (P2 through P5) and region of interest (ROI) proposals 1726. Each proposal (ROI) from 1726 is associated with a portion of the feature maps (1710 through 1716) to produce a fixed-size map. The size of the fixed-size map is independent of the base portion of feature maps 1710 through 1716. For example, one of the feature maps 1710 through 1716 is selected according to the following rule such that the resulting cropped map has sufficient detail: floor(4 + log2(sqrt(box_area) / 224)), where 224 is a typical box size. The ROI pooler 1728 operates to crop the input feature maps according to the proposals 1726, thereby producing tensor 1730.

[0227] Tensor 1730 is fed into the fully connected (FC) neural network head 1732. The FC head 1732 performs two fully connected layers to produce class score and bounding box prediction sub-difference tensors 1734. The class score is typically an 80-element tensor, with each element corresponding to the predicted score for the corresponding object class. The bounding box prediction sub-difference tensor is an 80 × 4 = 320-element tensor containing the bounding boxes for the corresponding object classes. Final processing is performed by the output layer module 1736, which receives tensor 1734 and performs filtering to produce a filtered tensor 1738. Low-scoring (low-classification) objects are no longer considered. The non-maximum suppression module 1740 receives the filtered tensor 1738 and removes overlapping bounding boxes by removing overlapping boxes with lower classification scores, thus obtaining the inference output tensor 1742 corresponding to tensor 151.

[0228] Figure 20 An example method 2000 is shown for determining the network topology and weights of the entire network and network indications of its parts. Method 2000 generates hashes that can be used to identify network topologies and / or network weights in an example ONNX format, and can be performed at runtime or before executing methods 1800 and 1900. If run beforehand, the results are stored in memory 206 for reference at runtime. Method 2000 is performed for each neural network supported by system 100. Furthermore, method 2000 is performed for each segmentation of the NN into NN part 1 and NN part 2. In other words, method 2000 is performed for NN part 1 and NN part 2 generated from each supporting split point of the NN. Method 2000 can use neural networks such as Faster R-CNN-X101-FPN and Mask R-CNN-X101-FPN to generate hashes, where the segmentation into NN part 1 and 2 occurs at boundaries such as P layers or Res layers. For neural networks where the dimensions of the inner layers depend on the dimensions of the input layers, proxy input resolutions such as 1920×1080 or 3840×2160 are used as sample inputs to determine the inner dimensions. The use of sample input sizes allows for a network topology byte stream independent of the actual resolution of the frame data 113. Method 2000 can also use neural networks such as YOLOv3 to generate hashes, where, for example, a segmentation at the “Darknet-53” boundary is used to divide the network into NN part 1 and NN part 2. Method 2000 begins with step 2010, which determines the NN part representation.

[0229] At step 2010, an ONNX representation of the neural network or a portion thereof is read from memory 206. The ONNX representation is a byte-serialized structural description of the NN topology and weights using a format called a "protocol buffer." The protocol buffer provides a language-independent and platform-independent scalable mechanism for serializing structured data. The protocol buffer is a series of key-value pairs with byte-level syntax, providing a more compact representation compared to alternatives such as Extensible Markup Language (XML). Each key-value pair forms a record with a "field number," "line type," and "payload," capable of signaling various data types including Boolean values, integers of varying precision, and variable-length strings. The ONNX representation operates to encode the elements (or "operators") of the network topology in a sequential execution order to avoid irrelevant ordering differences that could affect hash operations. Network node names, operator names, and variable names are specified as strings and all are replaced with simple enumerations (e.g., "var0", "var1", "operator0", "operator1", "node0", "node1", etc.) so that irrelevant naming differences do not affect hash operations. The ONNX representation may have been derived from another representation, such as Python code, using libraries such as PyTorch, Tensorflow, or Keras. This representation is refactored by replacing items such as variable names with simple enumerations constructed in execution order. The intention of the refactoring is to create a representation that depends as much as possible on the structure of the network and not on aspects (such as variable naming) determined by the programmer. The node names and dimensions of the NN's inputs and outputs are included in the resulting indications. Control in processor 205 proceeds from step 2010 to step 2020, which involves determining the NN's partial topology and weights.

[0230] At step 2020, the processor parses the byte-serialized ONNX representation of the network or a portion thereof and discards all data values ​​corresponding to trained elements (such as network weights or biases), thus preserving a topological subset of the ONNX representation. The ONNX representation is filtered to preserve graph properties (e.g., topology, convolutional dimensions, activation functions) but not trainable parameters; that is, information related to network weights outside the NN structure is removed. Additionally, the ONNX representation is filtered to produce a weight subset containing only trained elements, such as weights and biases (collectively referred to as weights). Certain additional parameters that are not trainable but data-adaptive (such as batch-normalized "running mean" and "running variance") are included as "weights" in this filtering process so that any changes in their values ​​affect the final hash calculation. In some instances, the network weight representation may be a hash of the MPEG NNC representation of the network weights. Control in processor 205 then proceeds from step 2020 to step 2030, which indicates the determination of the NN portion's topology and weights.

[0231] At step 2030, the topology subsets and weight subsets of the ONNX representation are hashed using algorithms such as md5sum, sha1sum, sha256, or crc32 to form a fixed-length and compact hash of the network topology or weights. This creates a compact and unique identifier for the original data, which typically occupies hundreds of MB. Furthermore, the aforementioned hashing algorithms have virtually no chance of two different topologies or weights producing the same hash value; that is, there is almost no chance of a "collision." Method 2000 is performed for the entire network and its parts. In other words, Method 2000 is iterated for each of (I) the entire CNN, (ii) the first network part (NN part 1 or backbone), and (iii) the second network part (NN part 2 or head). The operation of step 2030 is performed to determine a network topology indication, which is a hash of the filtered ONNX network representation that preserves the graph properties of each network. Additionally, the operation of step 2030 is performed to determine a network weight indication, which is a hash of the filtered ONNX network representation that preserves the weight properties.

[0232] When a network is divided into NN part 1 and NN part 2, the resulting indications for NN part 1 and NN part 2 include hashes of the parent (entire) network, the successor network of NN part 1 (i.e., NN part 2), and the predecessor network of NN part 2 (i.e., NN part 1). (See reference) Figure 21A and 21B To describe the additional hashes or "associations" that accompany the hashes of a given network or its parts. (For example, regarding...) Figure 21A and Figure 21BAs described, ONNX versions can also be generated and encoded in a bit stream, and the association between the first and second network portions of each ONNX version can be available. Figure 21A This is a schematic block diagram depicting the NN instruction 2100, and Figure 21B This is a schematic block diagram illustrating the association 2150 between NN indicators. The NN indicators include: an informal network name 2110 as a string; an ONNX version 2112 as an integer; a topological hash 2114 for NN part 1; a weight hash 2115 for NN part 1; a topological hash 2116 for NN; a weight hash 2117 for NN; a list of inputs 2118 to the network (that is, inputs to NN part 1), identified by enumerating the input node names in the ONNX reconstruction representation and the dimensions of each input tensor; a list of outputs 2119 from NN part 1 (corresponding to inputs to NN part 2); a list of outputs 2120 from the network (that is, outputs from NN part 2), identified by enumerating the output node names in the ONNX reconstruction representation and the dimensions of each output tensor; and an associated hash 2122. Hashes 2114 and 2115 identify NN part 1 used to generate tensors encoded in the feature sub-bit stream 143. Hashes 2116 and 2117 identify the network (NN part 1 and surrogate NN part 2) used to generate the surrogate task result 119. Hashes 2114 to 2117 are fixed-length codewords of 256 bits. The associated hash 2122 includes zero or more than zero hash pairs identifying compatible preceding NN weights and topologies. When indication 2100 is used for NN part 2 166, at least one preceding association will be included as a preceding association, that is, identifying NN part 1 114. The associated hash 2122 also includes zero or more than zero hash pairs identifying compatible successor NN weights and topologies. When indication 2100 is used for NN part 1 114, at least one successor association will be included as a successor association, typically identifying surrogate NN part 2 118. The container hash (if present) (not shown) includes topology and weight hashes used to identify the combined NN part 1 114 and surrogate NN part 2 118. The container hash identification, included together with NN Part 1 and Agent NN Part 2 118, is used to generate the agent task result 119 in the source device 110 for the entire network.

[0233] The container hashing included with NN section 2 166 identifies the entire network used to generate task result 151 in destination device 140. Thus, hashing of network topology and network weights provides a mechanism for identifying which network is used to generate task results and coupling compatible parts of the network together when executed in a distributed manner. Figure 21BThe proxy NN indication, NN section 1 indication, and proxy NN section 2 indication available in source device 110 are shown. The NN section 1 indication, which includes NN #1-4 indications and NN section 2 #1-4 indications, is signaled to destination device 140, each demonstrating compatibility with the signaled NN section 1 indication (i.e., matching network topology and network weight hash values). NN #5 and NN section 2 #5, which are incompatible with the NN section 1 indication, are also shown, and therefore their association with NN section 1 is not shown. (See reference...) Figure 19 As described, the absence of a candidate NN section 2 with a pre-hash compatible with the NN section 1 indication signaled in the multi-layer bitstream 123 results in the inability to use the tensor decoded from the feature sub-bitstream 143, even though the video sub-bitstream 171 can still be used. The existence of one or more compatible NN section 2 candidates makes it possible to use one of the candidates as NN section 2 150.

[0234] In one arrangement of System 100, network weights and network topology are hashed together in a single hash operation, thus avoiding the need for separate hashing of topology and weights. When network topology and weights are hashed together, it is no longer possible to identify cases where compatible topologies and incompatible weights are used, thereby reducing memory consumption and simplifying compatibility determination.

[0235] In the arrangement of source device 110 and destination device 140, steps 1850 and 19160 are omitted. This results in modules 116 and 146 performing video compression on feature maps from tensors obtained directly from NN section 1 114 and fed to NN section 2 166, and outputting them to feature bitstream 121 without adding network layers or using low-rank approximations. Omitting steps 1850 and 19160 results in lower complexity for source device 110 and destination device 140, but results in a higher bit rate for multi-layer bitstream 123.

[0236] In the arrangement of source device 110, feature bitstream 121 is included in multilayer bitstream 123 only when the task result 119 of the agent NN section 2 indicates some positive result (such as detecting an object in frame data 113). The inclusion of video bitstream 151 can also be conditional on the presence of task result 119. Conditional transmission of bitstreams 121 and 151 reduces the bandwidth required for communication channel 130.

[0237] In the arrangement of source device 110, a proxy task result 119 is generated and fed to video encoder 150 to influence encoding behavior (such as providing higher quality in regions of frame data 113 containing detected objects or blurring portions of frame data 113 that do not contain any objects of interest). The nn_optimized_video_bitstream_flag syntax element is encoded in the SEI message to indicate whether such a proxy task result is used as input to encode the video bitstream. Signaling the proxy task network included in the SEI message allows the decoder to determine which network to use to generate the proxy task result for use by video encoder 150.

[0238] In the arrangement of source device 110 and destination device 140, a pre-existing video bitstream conforming to standards such as HEVC master profile or VVC master 10 profile is rewritten as an HEVC master multi-layer profile or VVC master 10 multi-layer profile bitstream, wherein the original compressed video is retained as the base layer and feature layers corresponding to the output of the specified NN part 1 network are added as auxiliary layers. The bitstream rewriting may involve inserting or modifying VPS 1115, inserting SDI SEI 1112, inserting FCVCM SEI 1113, and rewriting and / or inserting SPS 1110a and 1110b. The bitstream rewriting allows NN part 1 to be executed at an earlier point in time than the execution of one of the NN part 1 candidates, thereby substantially reducing the execution burden of completing the task network when task result 167 is required.

[0239] In the arrangement of source device 110 and destination device 140, multi-layer bitstream 123 conforms to the HEVC multi-layer profile, and the base layer (i.e., the video layer) is omitted from the bitstream and any other external means for transmitting the base layer from source device 110 to destination device 140, for example, when the result 119 of the proxy NN section 2 indicates that no object was detected. VPS 1115 signals the absence of the base layer when vps_base_layer_internal_flag is set to equal to 1 and vps_base_layer_available_flag is set to equal to 0 (e.g., 1116). Even when the base layer is omitted from multi-layer bitstream 123, SPS 1110b is still included, thus signaling as if the base layer were present (e.g., 1116).

[0240] In the arrangement of source device 110 and destination device 140, NN section 1 and NN section 2 correspond to the encoder and decoder sections of the "visual transformer" network, such as those from Meta. TMThe “Segment Anything Model” (SAM) is used, where the split point (the boundary between NN parts 1 and 2) occurs at the boundary between the encoder and decoder, which is also the narrowest stage of the bottleneck architecture. When using SAM, the destination device 140 decodes the segmented map corresponding to the object selected using one or more points or bounding boxes (collectively referred to as “cues”) provided by, for example, the user of the destination device 140. Since NN part 1 (SAM encoder) only requires frame data 113 as input, NN part 1 can be run once and the result is encoded together with the video data as part of a multi-layer bitstream 123, and NN part 2 can be run each time the user of the destination device 140 provides a different cue.

[0241] In the arrangement of source device 110 and destination device 140, FCVCM SEI message 1113 includes (one or more) network topology indications and (one or more) network weight indications corresponding to tensor encoder 116 and / or tensor decoder 146, such as when tensor encoder 600 and tensor decoder 1500 are in use. The indication signaling tensor decoder 146 allows destination device 140 to select and instantiate suitable networks from a repository of candidate tensor decoders for use as tensor decoder 146.

[0242] Although the arrangement describes separate indications of network topology and network weights, the arrangement can jointly signal the network topology and network weights in a single indication. Signaling the topology and weights in a single indication avoids the need to filter information from the ONNX representation.

[0243] It should be noted that although the source device 110 and the destination device 140 are described with reference to the video source 112, which includes video and image data, other types of content, such as audio data or text data, can also be fed as input to a neural network suitable for this type of input, and the resulting intermediate feature maps can be compressed and decompressed by modules 116 and 146 using appropriate encoder and decoder network topologies.

[0244] Industrial availability

[0245] The described arrangement is applicable to the computer and data processing industry, and is particularly suitable for digital signal processing that encodes and decodes signals such as video and image signals, thereby achieving high compression efficiency.

[0246] Some embodiments described herein encode indications related to the topology and weights of the CNN into the bitstream. Encoding information related to the topology and weights provides a mechanism to identify which network head(s) or NN section(s) are compatible with the tensors provided during bitstream decoding, thereby keeping up with CNN trends and architectural changes.

[0247] Some of the embodiments described herein can include both video layers and feature layers in the bitstream. Including both video layers and feature layers in the bitstream provides a mechanism that humans can use to review or interpret CNN results.

[0248] The foregoing has only described some embodiments of the present invention, and modifications and / or changes may be made thereto without departing from the scope and spirit of the invention. These embodiments are illustrative and not restrictive.

[0249] Appendix A

[0250] The following is an example SEI message format and associated semantics used to represent metadata associated with auxiliary decoding tensor processing:

[0251]

[0252]

[0253]

[0254] Where u(n) refers to a fixed-length codeword with a length of n bits, and ue(v) refers to an unsigned exponential Columbus variable-length codeword.

[0255] FCVCM decoder information semantics:

[0256] A value of 1 for `fcvcm_network_indication_cancel_flag` indicates that this SEI message cancels the persistence of any previous FCVCM network indication information SEI messages applied to the current layer in output order. A value of 0 for `fcvcm_network_indication_cancel_flag` indicates that alpha channel information follows in the bitstream.

[0257] The information contained in the FCVCM Network Indication Information SEI message persists for the access unit starting from the current picture (i.e., the picture associated with the prefix SEI message) and continues until a new instance of the FCVCM Network Indication Information SEI message, the end of the bit stream, or the start of a new CLVS (Cyclic Layer Video Sequence).

[0258] `network_informal_name` encodes a null-terminated UTF-8 string that specifies the informal name used to identify the network from which it produces the results of included tasks. This informal name is only informational and is not used for decoding.

[0259] onnx_version is an unsigned integer in ONNX byte format that is used to preserve the serialized network topology and / or weights before hashing.

[0260] The value of nn_part1_hashes_present_flag equal to 1 indicates that the bit stream includes hashes that identify the NN part 1 of the tensors included in the feature bit stream.

[0261] nn_part1_topology_hash encodes the 256-bit sha1sum of the ONNX representation after the topology reconstruction of NN part 1.

[0262] nn_part1_weights_hash encodes the 256-bit sha1sum of the NN part 1 weights obtained from the reconstructed ONNX representation.

[0263] The value of nn_hashes_present_flag equal to 1 specifies that the bit stream includes the hash of the NN used to generate the surrogate network result (if it exists in the bit stream).

[0264] nn_topology_hash encodes the 256-bit sha1sum of the ONNX representation after NN topology reconstruction.

[0265] nn_weights_hash encodes the 256-bit sha1sum of NN weights obtained from the reconstructed ONNX representation.

[0266] The value of nn_optimized_video_bitstream_flag equal to 1 indicates that the video bitstream 151 is encoded using the result of the proxy task 119 for guidance, and when it equals 0, it indicates that the video bitstream 151 is generated independently of the result of the proxy task 119.

[0267] `nn_inputs_count` specifies the number of tensors to be provided as input to the NN. When `nn_inputs_count` is set to zero, the number and dimension of the tensors to be provided as input to the NN are unspecified. `nn_inputs_count` and related syntax elements form 2118.

[0268] nn_input_name[i] specifies the node name as the null-terminated UTF-8 string that the i-th tensor is provided to NN.

[0269] nn_input_feature_cnt[i] specifies the number of feature maps (channels) for the i-th input tensor.

[0270] nn_input_feature_width[i] specifies the width of the feature map of the i-th input tensor.

[0271] nn_input_feature_height[i] specifies the height of the feature map of the i-th input tensor.

[0272] `nn_part1_outputs_count` specifies the number of tensors provided as outputs from NN part 1. When `nn_part1_outputs_count` is set to zero, the number and dimension of tensors provided as outputs from NN part 1 are unspecified. `nn_part1_outputs_count` and related syntax elements form 2119.

[0273] nn_part1_input_name[I] specifies the node name as an empty-terminated UTF-8 string that outputs the i-th tensor from NN part 1.

[0274] nn_part1_input_feature_cnt[i] specifies the number of feature maps (channels) of the output tensor of the i-th NN part 1.

[0275] nn_part1_input_feature_width[i] specifies the width of the feature map of the output tensor of the i-th NN part 1.

[0276] nn_part1_input_feature_height[i] specifies the height of the feature map of the i-th NN part 1 output tensor.

[0277] `nn_outputs_count` specifies the number of tensors produced as outputs from the neural network. When `nn_output_count` is set to zero, the number and dimension of tensors to be provided as input to the neural network are unspecified. `nn_outputs_count` and related syntax elements form a 2120.

[0278] nn_output_name[i] specifies the node name as an empty-terminated UTF-8 string obtained from the i-th tensor of NN.

[0279] nn_output_feature_cnt[i] specifies the number of feature maps (channels) for the i-th output tensor.

[0280] nn_output_feature_width[i] specifies the width of the feature map of the i-th output tensor.

[0281] nn_output_feature_height[i] specifies the height of the feature map of the i-th output tensor.

[0282] The preceding_association_flag specifies the existence of a hash used to indicate the presence of the NN part (i.e., NN part 1 114) used to generate the tensor encoded in feature stream 121.

[0283] preceding_association_topology_hash specifies the hash of the reconstructed ONNX representation of the network topology implemented as NN part 1 114.

[0284] The preceding_association_weights_hash specifies the hash of the weights presented from the ONNX representation of NN part 1114.

[0285] The container_association_present_flag specifies the existence of the NN used to generate the surrogate task result 119 (i.e., the combined NN part 1 114 and the surrogate NN part 2 118).

[0286] container_association_topology_hash specifies the sha1sum of the reconstructed ONNX representation of the network topology, which is implemented as an unsplit network by combining NN part 1 114 with surrogate NN part 2 118.

[0287] container_association_weights_hash specifies the hash of the weights that are combined from NN part 1 114 and surrogate NN part 2 118 as the ONNX representation of the unsplit network.

[0288] If proxy_task_result_length is zero, it indicates that no proxy task result exists in the multi-level bit stream 123; if it is non-zero, it specifies the length of the number of bytes of proxy task result contained in the multi-level bit stream 123.

[0289] `proxy_task_result_byte[i]` specifies the i-th byte of the payload containing the proxy task result 119. Example proxy task results are 32-bit floating-point numbers encoded as mAP or MOTA scores, or segmented graphs compressed using, for example, run-length encoding.

Claims

1. A method for decoding a tensor from a bitstream for use by a second part of a network, the method comprising: Determine a first part of the network topology indication from the bit stream; Determine the first part of the network weight indication from the bit stream; Whether to decode the tensor from the bitstream is determined based on the network first part topology indication and the network first part weight indication; as well as If it is determined that a tensor is to be decoded from the bit stream, the tensor is decoded from the bit stream for the second part of the network.

2. A method for generating a content-oriented neural network result from a bitstream, the method comprising: Determine a first part of the network topology indication from the bit stream; Determine the first part of the network weight indication from the bit stream; Whether to proceed with the second part of the network is determined based on the network first part topology indication and the network first part weight indication; as well as If it is determined that the second part of the network is to be performed, the second part of the network is performed using tensors decoded from the bitstream to produce the neural network result.

3. The method according to claim 1, wherein, Whether to decode a tensor from the bitstream is determined based on the availability of a plurality of network second parts associated with the first part of the network, wherein one of the plurality of network second parts is selected for execution.

4. The method according to claim 2, wherein, Whether to proceed with the second part of the network is determined based on the availability of a plurality of second parts of the network associated with the first part of the network, wherein one of the plurality of second parts of the network is selected for execution.

5. The method according to claim 1 or 2, wherein, The network topology indicator is a hash of the filtered ONNX network representation, which preserves the graph properties.

6. The method according to claim 1 or 2, wherein, The network weight indicator is a hash of the filtered ONNX network representation, which preserves the weight properties.

7. The method according to claim 5, wherein, The hash is one of the sha256 hash, sha1sum hash, md5sum hash, and crc32 hash represented by the filtered ONNX network.

8. The method according to claim 1 or 2, wherein, The network weight indicator is a hash of the MPEG NNC representation of the network weight.

9. The method according to claim 1 or 2, wherein, The ONNX version is encoded in the bitstream, and the association between the first network portion and the second network portion of each ONNX version is available.

10. A decoder for decoding tensors from a bitstream for use by a second part of a network, the decoder being configured to: Determine a first part of the network topology indication from the bit stream; Determine the first part of the network weight indication from the bit stream; Whether to decode the tensor from the bitstream is determined based on the network first part topology indication and the network first part weight indication; as well as If it is determined that a tensor is to be decoded from the bit stream, the tensor is decoded from the bit stream for the second part of the network.

11. A non-transitory computer-readable storage medium storing a program for performing a method for decoding a tensor from a bitstream for use by a second part of a network, the method comprising: Determine a first part of the network topology indication from the bit stream; Determine the first part of the network weight indication from the bit stream; Whether to decode the tensor from the bitstream is determined based on the network first part topology indication and the network first part weight indication; as well as If it is determined that a tensor is to be decoded from the bit stream, the tensor is decoded from the bit stream for the second part of the network.

12. A decoder for generating a neural network result for content from a bitstream, the decoder being configured to: Determine a first part of the network topology indication from the bit stream; Determine the first part of the network weight indication from the bit stream; Whether to proceed with the second part of the network is determined based on the network first part topology indication and the network first part weight indication; as well as If it is determined that the second part of the network is to be performed, the second part of the network is performed using tensors decoded from the bitstream to produce the neural network result.

13. A non-transitory computer-readable storage medium storing a program for executing a decoding method for generating neural network results for content from a bitstream, the method comprising: Determine a first part of the network topology indication from the bit stream; Determine the first part of the network weight indication from the bit stream; Whether to proceed with the second part of the network is determined based on the network first part topology indication and the network first part weight indication; as well as If it is determined that the second part of the network is to be performed, the second part of the network is performed using tensors decoded from the bitstream to produce the neural network result.

Images