Method, apparatus and system for encoding and decoding tensor

Abstract

An apparatus and method for decoding, from a bitstream, at least a plurality of tensors forming a hierarchical representation of a feature map for a single frame. The method includes: decoding a first information unit from a bitstream; decoding a second information unit from the bitstream; and determining a first plurality of tensors from the first information unit, a feature map of at least one of the first plurality of tensors having a different spatial resolution than feature maps of other tensors of the first plurality of tensors. The method further includes determining a second plurality of tensors from the second information unit, a feature map of at least one of the second plurality of tensors having a different spatial resolution than feature maps of other tensors of the second plurality of tensors. The feature map of each tensor of the first plurality of tensors has a different spatial resolution than the feature map of each tensor of the second plurality of tensors, and the tensor of the first plurality of tensors and the second plurality of tensors corresponds to a hierarchical representation of the feature map for a single frame.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit under 35 U.S.C. §119 of the filing date of Australian patent application 2022204911, filed on July 8, 2022, which is incorporated herein by reference in its entirety as if fully set forth herein. Technical Field

[0003] The present invention generally relates to digital video signal processing, and in particular, to methods, devices and systems for encoding and decoding tensors from convolutional neural networks. The present invention also relates to a computer program product comprising a computer-readable medium having recorded thereon a computer program for encoding and decoding tensors from convolutional neural networks using video compression techniques. Background Art

[0004] Convolutional Neural Networks (CNNs) are an emerging technology for addressing use cases involving machine vision, such as object detection, instance segmentation, object tracking, and human pose estimation and action recognition, among others. Applications of CNNs may involve the use of an "edge device" with sensors and some processing power, coupled to an application server as part of the "cloud". CNNs may require relatively high computational complexity, which exceeds what can typically be provided in terms of utilizing the computational capacity or power consumption of edge devices. Executing CNNs in a distributed manner has become a solution for running cutting-edge networks using limited-capacity edge devices. In other words, distributed processing enables traditional edge devices to still provide the capabilities of cutting-edge CNNs by allocating processing between edge devices and external processing components such as cloud servers.

[0005] CNNs typically include many layers such as convolutional layers and fully connected layers, where data is passed from one layer to the next in the form of "tensors". Segmenting networks across different devices introduces the need to compress the intermediate tensor data passed from one layer to the next within a CNN, which can be referred to as "feature compression" because the intermediate tensor data is often referred to as "features" of inputs such as image frames or video frames. The International Organization for Standardization / International Electrotechnical Commission Joint Technical Committee 1 / Group Association 29 / Working Group 2-8 (ISO / IEC JTC1 / SC29 / WG2-8) (also known as the "Moving Picture Experts Group" (MPEG)) is assigned to study video-related compression techniques. WG2 "MPEG Technical Requirements" has established the "Machine Video Compression" (VCM) ad hoc group, which is authorized to study video compression for machine consumption, as well as feature compression. The feature compression mandate is in the exploratory stage where a "Call for Evidence" (CfE) is expected to be released to solicit technologies that can significantly outperform feature compression results achieved using state-of-the-art standardized techniques.

[0006] CNN requires weights for each layer to be determined in a training phase, in which a very large amount of training data is passed through the CNN and the determined results are compared with the ground truth associated with the training data. A process for updating the network weights (such as stochastic gradient descent, etc.) is applied to iteratively refine the network weights until the network performs at the desired level of accuracy. When the convolution phase has a "stride" greater than 1, the output tensor from the convolution has a lower spatial resolution than the corresponding input tensor. The pooling operation results in an output tensor with a smaller dimension than the input vector. An example of a pooling operation is "max pooling" (or "Maxpool"), which reduces the spatial size of the output tensor compared to the input tensor. Maximum pooling generates an output tensor by dividing the input tensor into groups of data samples (e.g., 2×2 groups of data samples) and selecting the maximum value from each group as the output of the corresponding value in the output tensor. The process of executing a CNN with an input and gradually transforming the input into an output is generally referred to as "inference".

[0007] Typically, a tensor has four dimensions, namely: batch, channel, height, and width. When performing inference on video data, the first dimension "batch" of size "one" indicates that one frame is passed through the CNN at a time. When training the network, the value of the batch dimension can increase so that multiple frames are passed through the network before updating the network weights according to a predetermined "batch size". Multiple frames of video can be passed through as a single tensor with the size of the batch dimension increasing according to the number of frames of a given video. However, for practical considerations related to memory consumption and access, inference on video data is typically performed on a frame basis. The "channel" dimension indicates the number of concurrent "feature maps" for a given tensor, and the height and width dimensions indicate the size of the feature map at a particular stage of the CNN. The channel count through the CNN varies depending on the network architecture. Depending on the subsampling that occurs in a particular network layer, the size of the feature map also varies.

[0008] The input to the first layer of a CNN is a batch of one or more images (e.g., a single image or video frame), which is usually resized to match the dimensions of the tensor input to the first layer. Images or video frames can also be fed in batches of size greater than one. The dimensions of the tensor depend on the CNN architecture, which usually has some dimensions related to the input width and height and an additional "channel" dimension.

[0009] Slicing a tensor based on the channel dimension (or reducing it to a collection of two-dimensional arrays based on the channel dimension) results in a collection of two-dimensional "feature maps", so called because each slice of the tensor has some relationship to the corresponding input image, capturing properties such as various edge types. At layers further from the network's input, the properties may be more abstract. The "task performance" of a CNN is measured by comparing its results for a task using a particular input to a provided ground truth, which is usually prepared by a human and is believed to indicate the "correct" result.

[0010] Once the network topology is determined, the network weights can be updated over time as more training data becomes available. The overall complexity of CNNs tends to be relatively high when performing relatively large numbers of multiplication-accumulation operations and writing and reading a large number of intermediate tensors relative to memory. In some applications, CNNs are implemented entirely in the "cloud", which results in the need for high and expensive processing power. In other applications, CNNs are implemented in edge devices (such as cameras or mobile phones), which results in lower flexibility but more distributed processing loads. Emerging architectures involve splitting the network into parts, one of which runs in the edge device and the other in the cloud. This distributed network architecture can be called "collaborative intelligence" and provides benefits such as reusing partial results from the first part of the network for several different second parts, perhaps each part is optimized for different tasks. Collaborative intelligence architectures introduce the need for efficient compression of tensor data to be sent over networks such as WANs.

[0011] As described below, video compression standards can be used for feature compression. Various methods can be used to shrink or reduce the data being presented for compression. However, some methods for shrinking or reducing the data being presented for compression may result in a decrease in accuracy that is not suitable for some tasks implemented by CNN.

[0012] Feature compression can benefit from existing video compression standards, such as the universal video coding (VVC) developed by the Joint Video Experts Group (JVET). It is expected that VVC will address the continuous demand for even higher compression performance, and address the growing market demand for service delivery over WAN (wherein the bandwidth cost is relatively high), especially with the increase in the capabilities of video formats (e.g., with higher resolution and higher frame rate). VVC can be implemented in contemporary silicon processes and provides an acceptable compromise between the achieved performance and the implementation cost. The implementation cost can be considered to be one or more than one aspect in terms of silicon area, CPU processor load, memory utilization, and bandwidth. Part of the versatility of the VVC standard is the wide selection of tools that can be used to compress video data, and the wide range of applications that VVC is suitable for. Other video compression standards (such as high-efficiency video coding (HEVC) and AV-1, etc.) can also be used for feature compression applications.

[0013] The video data comprises a sequence of frames of image data, each frame comprising one or more color channels. Typically, one primary color channel and two secondary color channels are required. The primary color channel is often referred to as the "luminance" channel, and the (one or more) secondary color channels are often referred to as the "chrominance" channels. Although video data is often displayed in an RGB (red-green-blue) color space, the RGB color space has a high degree of correlation between the three corresponding components. The video data representation seen by the encoder or decoder often uses a color space such as YCbCr. YCbCr concentrates the luminance (mapped to "luminance" according to a transfer function) in the Y (primary) channel and the chrominance in the Cb and Cr (secondary) channels. Due to the use of decorrelated YCbCr signals, the statistics of the luminance channel are significantly different from those of the chrominance channels. The main difference is that, after quantization, the chrominance channel contains relatively few significant coefficients for a given block compared to the coefficients of the corresponding luminance channel block. Additionally, the Cb and Cr channels may be spatially sampled at a lower rate (subsampled) than the luma channel, for example, half horizontally and half vertically (known as a "4:2:0 chroma format"). The 4:2:0 chroma format is commonly used in video formats such as Internet video streaming, broadcast television, and Blu-ray. TM When only luma samples are present, the resulting monochrome frame is said to use the "4:0:0 chroma format".

[0014] The VVC standard specifies a "block-based" architecture in which a frame is first partitioned into an array of square regions known as "coding tree units" (CTUs). A CTU typically occupies a relatively large area, such as 128×128 luminance samples. Other possible CTU sizes when using the VV standard are 32×32 and 64×64. However, the area of ​​the CTU at the right and lower edges of each frame may be smaller, where implicit splitting occurs to ensure that the CB remains in the frame. Associated with each CTU is a "coding tree" ("shared tree") for both the luminance channel and the chrominance channel, or a separate tree for each luminance channel and chrominance channel. The coding tree defines the decomposition of the area of ​​the CTU into a set of blocks (also called "coding blocks" (CBs)). When a shared tree is used, a single coding tree specifies blocks for both the luminance channel and the chrominance channel, in which case a set of collocated coding blocks is called a "coding unit" (CU) (i.e., each CU has a coding block for each color channel). CBs are processed in a specific order for encoding or decoding. Due to the use of 4:2:0 chroma format, a CTU with a luma coding tree for a 128×128 luma sample area has a corresponding chroma coding tree for a 64×64 chroma sample area (juxtaposed with the 128×128 luma sample area). When a single coding tree is used for the luma channel and the chroma channel, the set of juxtaposed blocks for a given area is often referred to as a "unit", such as the above-mentioned CU, as well as "prediction units" (PUs) and "transform units" (TUs). A single tree with a CU spanning the color channels of 4:2:0 chroma format video data results in chroma blocks of half the width and height of the corresponding luma blocks. When a separate coding tree is used for a given area, the above-mentioned CB is used as well as "prediction blocks" (PBs) and "transform blocks" (TBs).

[0015] Despite the above distinction between "unit" and "block", the term "block" may be used as a general term for an area or region of a frame for which operations are applied to all color channels.

[0016] For each CU, a prediction unit (PU) ("prediction unit") is generated for the content (sample values) of the corresponding region of the frame data. In addition, a representation of the difference (or "spatial domain" residual) between the prediction and the content of the region seen at the input of the encoder is formed. The differences in each color channel can be transformed and encoded into a sequence of residual coefficients, thereby forming one or more TUs for a given CU. The applied transform can be a discrete cosine transform (DCT) or other transform applied to each block of residual values. The transform is applied separately (i.e., a two-dimensional transform is performed twice). The block is first transformed by applying a one-dimensional transform to each row of samples in the block. The partial result is then transformed by applying a one-dimensional transform to each column of the partial result to produce a final block of transform coefficients that substantially decorrelate the residual samples. The VVC standard supports transforms of various sizes, including transforms of rectangular blocks whose dimensions on each side are powers of 2. The transform coefficients are quantized for entropy encoding into the bitstream.

[0017] VVC is characterized as intra prediction and inter prediction. Intra prediction involves using previously processed samples in the frame being used to generate a prediction of the current block of data samples in the frame. Inter prediction involves using a block of samples obtained from a previously decoded frame to generate a prediction of the current block of samples in the frame. The block of samples obtained from the previously decoded frame is offset relative to the spatial position of the current block according to a motion vector, which is usually filtered. The intra prediction block can be: (i) uniform sample values ​​("DC intra prediction"), (ii) a plane with an offset and horizontal and vertical gradients ("planar intra prediction"), (iii) a population of blocks with neighboring samples applied in a specific direction ("angular intra prediction"), or (iv) the result of a matrix multiplication using neighboring samples and selected matrix coefficients. Further differences between the predicted block and the corresponding input sample can be corrected to some extent by encoding a "residual" into the bitstream. The residual is usually transformed from the spatial domain to the frequency domain to form residual coefficients in the "primary transform domain". The residual coefficients may be further transformed by applying a "secondary transform" to produce residual coefficients in a "secondary transform domain". The residual coefficients are quantized according to a quantization parameter, which results in a loss of reconstruction accuracy of the samples produced at the decoder, but a reduction in the bit rate in the bitstream. A sequence of pictures may be encoded according to a specified structure of pictures using intra prediction and pictures using intra or inter prediction, and a specified dependency on previous pictures in a coding order, which may be different from the display or delivery order. The "random access" configuration results in periodic intra pictures, forming the entry point for the decoder to start decoding the bitstream. Other pictures in the random access configuration typically use inter prediction to predict content from pictures before and after the current picture in the display or delivery order according to a hierarchical structure of a specified depth. Using a picture after the current picture in the display order to predict the current picture requires a certain degree of picture buffering and delay between the decoding of a given picture and the display of the given picture (and removal from the buffer). Summary of the invention

[0018] It is an object of the present invention to substantially overcome or at least ameliorate one or more disadvantages of existing arrangements.

[0019] An aspect of the present disclosure provides a method for decoding at least multiple tensors from a bitstream, wherein the multiple tensors form a hierarchical representation of a feature map for a single frame, the method comprising: decoding a first information unit from the bitstream; decoding a second information unit from the bitstream; determining a first plurality of tensors based on the first information unit, wherein a feature map of at least one tensor in the first plurality of tensors has a different spatial resolution from a feature map of one or more other tensors in the first plurality of tensors; and determining a second plurality of tensors based on the second information unit, wherein a feature map of at least one tensor in the second plurality of tensors has a different spatial resolution from a feature map of one or more other tensors in the second plurality of tensors, wherein the feature map of each tensor in the first plurality of tensors has a different spatial resolution from the feature map of each tensor in the second plurality of tensors, and tensors in the first plurality of tensors and the second plurality of tensors correspond to the hierarchical representation of the feature map for the single frame.

[0020] Another aspect of the present disclosure provides a method for encoding at least multiple tensors into a bitstream, wherein the multiple tensors form a hierarchical representation of feature maps for a single frame, the method comprising: using a convolution operation to determine a first information unit based on a first plurality of tensors, the feature map of at least one tensor in the first plurality of tensors having a different spatial resolution from the feature map of one or more other tensors in the first plurality of tensors; using a convolution operation to determine a second information unit based on a second plurality of tensors, the feature map of at least one tensor in the second plurality of tensors having a different spatial resolution from the feature map of one or more other tensors in the second plurality of tensors, and wherein the feature map of each tensor in the first plurality of tensors has a different spatial resolution from the feature map of each tensor in the second plurality of tensors, and tensors in the first plurality of tensors and the second plurality of tensors correspond to the hierarchical representation of the feature map for the single frame; encoding the first information unit into the bitstream; and encoding the second information unit into the bitstream.

[0021] Another aspect of the present disclosure provides a decoder for decoding at least multiple tensors from a bitstream, the multiple tensors forming a hierarchical representation of feature maps for a single frame, the decoder being configured to: decode a first information unit from the bitstream; decode a second information unit from the bitstream; determine a first plurality of tensors based on the first information unit, the feature map of at least one tensor in the first plurality of tensors having a different spatial resolution from the feature map of one or more other tensors in the first plurality of tensors; and determine a second plurality of tensors based on the second information unit, the feature map of at least one tensor in the second plurality of tensors having a different spatial resolution from the feature map of one or more other tensors in the second plurality of tensors, wherein the feature map of each tensor in the first plurality of tensors has a different spatial resolution from the feature map of each tensor in the second plurality of tensors, and the tensors in the first plurality of tensors and the second plurality of tensors correspond to the hierarchical representation of the feature map for the single frame.

[0022] Another aspect of the present disclosure provides an encoder for encoding at least multiple tensors into a bitstream, the multiple tensors forming a hierarchical representation of feature maps for a single frame, the encoder being configured to: use a convolution operation to determine a first information unit based on a first plurality of tensors, the feature map of at least one tensor in the first plurality of tensors having a different spatial resolution from the feature map of one or more other tensors in the first plurality of tensors; use a convolution operation to determine a second information unit based on a second plurality of tensors, the feature map of at least one tensor in the second plurality of tensors having a different spatial resolution from the feature map of one or more other tensors in the second plurality of tensors, and wherein the feature map of each tensor in the first plurality of tensors has a different spatial resolution from the feature map of each tensor in the second plurality of tensors, and tensors in the first plurality of tensors and the second plurality of tensors correspond to the hierarchical representation of feature maps for the single frame; encode the first information unit into the bitstream; and encode the second information unit into the bitstream.

[0023] Another aspect of the present disclosure provides a non-transitory computer-readable storage medium storing a program for executing a method for decoding at least a plurality of tensors from a bitstream, the plurality of tensors forming a hierarchical representation of a feature map for a single frame, the method comprising: decoding a first information unit from the bitstream; decoding a second information unit from the bitstream; determining a first plurality of tensors based on the first information unit, the feature map of at least one of the first plurality of tensors having a different spatial resolution from the feature map of one or more other tensors in the first plurality of tensors; and determining a second plurality of tensors based on the second information unit, the feature map of at least one of the second plurality of tensors having a different spatial resolution from the feature map of one or more other tensors in the second plurality of tensors, wherein the feature map of each tensor in the first plurality of tensors has a different spatial resolution from the feature map of each tensor in the second plurality of tensors, and the tensors in the first plurality of tensors and the second plurality of tensors correspond to the hierarchical representation of the feature map for the single frame.

[0024] Another aspect of the present disclosure provides a system, comprising: a memory; and a processor, wherein the processor is configured to execute code stored on the memory for implementing a method for decoding at least multiple tensors from a bitstream, the multiple tensors forming a hierarchical representation of a feature map for a single frame, the method comprising: decoding a first information unit from the bitstream; decoding a second information unit from the bitstream; determining a first plurality of tensors based on the first information unit, wherein a feature map of at least one of the first plurality of tensors has a different spatial resolution from a feature map of one or more other tensors in the first plurality of tensors; and determining a second plurality of tensors based on the second information unit, wherein a feature map of at least one of the second plurality of tensors has a different spatial resolution from a feature map of one or more other tensors in the second plurality of tensors, wherein the feature map of each tensor in the first plurality of tensors has a different spatial resolution from the feature map of each tensor in the second plurality of tensors, and tensors in the first plurality of tensors and the second plurality of tensors correspond to the hierarchical representation of the feature map for the single frame.

[0025] Other aspects are also disclosed. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] At least one embodiment of the present invention will now be described with reference to the following drawings and appendices, in which:

[0027] Figure 1 is a schematic block diagram illustrating a distributed machine task system;

[0028] Figure 2A and Figure 2B Form a practical Figure 1 A schematic block diagram of a general computer system for a distributed machine task system;

[0029] Figure 3A is a schematic block diagram showing the functional modules of the main body of CNN;

[0030] Figure 3B It is shown Figure 3A A schematic block diagram of a residual block;

[0031] Figure 3C It is shown Figure 3A A schematic block diagram of a residual unit of ;

[0032] Figure 3D It is shown Figure 3A Schematic block diagram of the CBL module;

[0033] Figure 4 is a schematic block diagram showing the functional modules of an alternative backbone of a CNN;

[0034] Figure 5 is a schematic block diagram illustrating a cross-layer tensor bottleneck for reducing tensor dimensionality prior to compression;

[0035] Figure 6 Show a method for performing the first part of a CNN, shrinking using a bottleneck encoder, and encoding the resulting shrunken feature map;

[0036] Figure 7 is a schematic block diagram illustrating a packing arrangement for a plurality of compressed tensors;

[0037] Figure 8 is a schematic block diagram showing the functional modules of a video encoder;

[0038] Fig. 9 is a schematic block diagram showing the functional modules of a video decoder;

[0039] Fig. 10A is a schematic block diagram illustrating a cross-layer tensor inverse bottleneck system for restoring tensor dimensions after compression;

[0040] Fig. 10B is shown for use with Fig. 10A A schematic block diagram of a convolutional arrangement used with a reverse bottleneck system;

[0041] Fig.11 A method for decoding a bitstream, reconstructing a decorrelated feature map, and performing the second part of a CNN is shown;

[0042] Fig. 12A is a schematic block diagram showing the head of a CNN;

[0043] Fig. 12B It is shown Fig. 12A A schematic block diagram of an upgrader module;

[0044] Fig. 12C It is shown Fig. 12A A schematic block diagram of a detection module;

[0045] Fig.13 is a schematic block diagram showing an alternative head of a CNN; and

[0046] Fig.14 is a schematic block diagram illustrating a top-down multi-scale feature reconstruction stage for a destination device. DETAILED DESCRIPTION

[0047] Where steps and / or features having the same reference numerals are referenced in any one or more of the accompanying drawings, these steps and / or features have the same function(s) or operation(s) for the purposes of this specification, unless otherwise intended.

[0048] A distributed machine task system may include edge devices, such as a web camera or smart phone that generates intermediate compressed data. The distributed machine task system may also include end devices, such as server farm ("cloud") based applications that operate on the intermediate compressed data to generate some task result. In addition, edge device functionality may be embodied in the cloud, and the intermediate compressed data may be stored for later processing (potentially for multiple different tasks as needed). Examples of machine tasks include object detection and instance segmentation, both of which generate task results measured as "mean average precision" (mAP) for detections that exceed a threshold (such as 0.5) of intersection over union (IoU). Another example machine task is object tracking, with a mean object tracking accuracy (MOTA) score as a typical task result.

[0049] A convenient form of intermediate compressed data is a compressed video bitstream, due to the availability of high performance compression standards and their implementations. Video compression standards typically operate on integer samples of some given bit depth, such as 10 bits arranged in a planar array. Color video has three planar arrays, corresponding to the color components Y, Cb, Cr or R, G, B, depending on the application. CNNs typically operate on floating point data in the form of tensors. Tensors typically have much smaller spatial dimensions than the incoming video data on which CNNs operate, but have many more channels than the typical three channels of color video data.

[0050] Tensors typically have the following dimensions: frame, channel, height, and width. For example, a tensor of dimension [1,256,76,136] would be considered to contain two hundred and fifty-six (256) feature maps, each of size 136 × 76. For video data, inference is typically performed on one frame at a time, rather than using tensors containing multiple frames.

[0051] VVC supports partitioning a picture into multiple sub-pictures, each of which can be independently encoded and independently decoded. In one method, each sub-picture is encoded as a "slice", or a continuous sequence of encoded CTUs. A "tile" mechanism can also be used to partition a picture into multiple independently decodable areas. Sub-pictures can be specified in a flexible way, where various rectangular CTU sets are encoded as corresponding sub-pictures. The flexible definition of sub-picture size allows data types that require different areas to be efficiently maintained in one picture, avoiding large "unused" areas, that is, frame areas that are not used to reconstruct tensor data.

[0052] Figure 1 is a schematic block diagram illustrating the functional modules of a distributed machine task system 100. The concept of distributing machine tasks across multiple systems is sometimes referred to as "collaborative intelligence" (CI). The system 100 can be used to implement methods for decorrelating, packing, and quantizing feature maps into planar frames to encode feature maps and decoding feature maps from encoded data. These methods can be implemented so that the associated overhead data is not burdensome, and task performance with respect to decoding feature maps is resilient to bit rate variations of the bitstream, and quantized representations of tensors do not unnecessarily consume bits that do not provide a commensurate benefit in task performance.

[0053] The system 100 includes a source device 110 for generating encoded tensor data 115 from a CNN backbone 114 in the form of an encoded video bitstream 121. The system 100 also includes a destination device 140 for decoding the tensor data in the form of an encoded video bitstream 143. A communication channel 130 is used to communicate the encoded video bitstream 121 from the source device 110 to the destination device 140. In some arrangements, one or both of the source device 110 and the destination device 140 may include a respective mobile phone handset (e.g., a "smart phone") or a web camera and a cloud application. The communication channel 130 may be a wired connection such as Ethernet or a wireless connection such as WiFi or 5G, including a connection across a wide area network (WAN) or across an ad hoc connection. In addition, the source device 110 and the destination device 140 may include an application that captures the encoded video data on some computer-readable storage medium (such as a hard drive or memory in a file server).

[0054] like Figure 1As shown, the source device 110 includes a video source 112, a CNN backbone 114, a bottleneck encoder 116, a quantization and packing module 118, a feature map encoder 120, and a transmitter 122. The video source 112 typically includes a source of captured video frame data (represented as 113), such as a camera sensor, a previously captured video sequence stored on a non-transitory recording medium, or a video feed from a remote camera sensor. The video source 112 can also be the output of a computer graphics card, for example, a video output of a display operating system and various applications executed on a computing device (e.g., a tablet computer). Examples of source devices 110 that can include a camera sensor as a video source 112 include smart phones, video camcorders, professional cameras, and network video cameras. The system 100 uses a "bottleneck" (i.e., an additional network layer that limits the tensor dimension on the encoding side and restores the tensor dimension on the decoder side) to reduce the dimension of the tensor at the interface between the first network part and the second network part. The multi-scale representations produced by the Feature Pyramid Network (FPN) are "fused" into a single tensor using a method called "Multi-Scale Feature Compression" (MSFC). MSFC is typically used to merge all FPN layers into a single tensor. Merging all FPN layers into a single tensor is achieved at the expense of spatial detail of the less decomposed (larger) layers of the FPN. The loss of spatial detail may result in an unacceptable decrease in the accuracy of some tasks or operations implemented by the system 100.

[0055] The described arrangement separates the tensors of FPN into groups and applies the MSFC technique separately, rather than merging all FPN layers into a single tensor. Applying the MFSC technique separately permits a certain degree of cross-layer fusion without such severe degradation of spatial details. For tasks such as instance segmentation that require preservation of greater spatial details, the resulting mAP is higher using the MSFC technique separately than when all FPN layers are merged into a single tensor with low spatial resolution.

[0056] The CNN backbone 114 receives the video frame data 113 and performs a specific layer of the entire CNN (such as a layer corresponding to the "backbone" of the CNN), thereby outputting a tensor 115. The backbone layer of the CNN can produce multiple tensors as outputs corresponding to different spatial scales of the input image represented by the video frame data 113 (sometimes referred to as a "feature pyramid network" (FPN) architecture). The tensors produced by the FPN backbone form a hierarchical representation of the frame data 113 including data of the feature map. Each successive layer of the hierarchical representation has half the width and height of the previous layer. The subsequent layers generated further in the backbone network tend to contain features with more abstract representations of the frame data 113. The less decomposed layers generated earlier in the backbone network tend to contain features representing less abstract features of the frame data 113, such as various geometric attributes (such as edges at various angles, etc.). When the system 100 performs a "YOLOv3" network, the FPN can obtain three tensors corresponding to three layers output from the backbone 114 as tensors 115, and the three tensors have different spatial resolutions and channel counts. When the system 100 is performing a network such as "FasterRCNN X101-FPN" or "Mask RCNN X101-FPN", tensor 115 includes tensors P2 to P5 of four layers. A bottleneck encoder 116 receives tensor 115. Bottleneck encoder 116 acts to compress one or more internal layers of the entire CNN. The internal layers of the entire CNN provide the output of the CNN backbone 114, which is compressed or shrunk by bottleneck encoder 116 using a set of neural network layers trained to convert to a lower channel count and smaller spatial resolution than required by tensor 115. Bottleneck encoder 116 outputs bottleneck tensor 117. Bottleneck tensor 117 is passed to quantization and packing module 118. Each feature map of bottleneck tensor 117 is quantized from floating point to integer precision by module 118 and packed into monochrome frames to produce frame 119. Frame 119 is encoded by feature map encoder 120 to produce bit stream 121. The bit stream 121 is supplied to a transmitter 122 for transmission via a communication channel 130 , or the bit stream 121 is written to a storage section 132 for later use.

[0057] The source device 110 supports a particular network for the CNN backbone 114. However, the destination device 140 may use one of several networks for the CNN head 150. In this way, partially processed data in the form of packed feature maps may be stored for later use in performing various tasks without having to repeat the operations of the CNN backbone 114.

[0058] The bitstream 121 is transmitted as encoded video data (or "encoded video information") by a transmitter 122 over a communication channel 130. In some implementations, the bitstream 121 may be stored in a storage 132 until (or in lieu of) being transmitted later over the communication channel 130, where the storage 132 is a non-transitory storage device such as a "flash" memory or a hard drive. For example, the encoded video data may be provided to a client as needed over a wide area network (WAN) for a video analytics application.

[0059] The destination device 140 includes a receiver 142, a feature map decoder 144, an unpacking and inverse quantization module 146, a bottleneck decoder 148, a CNN head 150, and a CNN task result buffer 152. The receiver 142 receives the encoded video data from the communication channel 130 and passes the video bitstream 143 to the feature map decoder 144. The feature map decoder 144 operates to decode the feature map and outputs a decoded frame 145. The decoded frame 145 is passed to the unpacking and inverse quantization module 146. The module 146 unpacks and inverse quantizes the tensor of the frame 145 to generate a dequantized tensor, which is output as a decoded bottleneck tensor 147. The decoded bottleneck tensor 147 is fed to the bottleneck decoder 148. The bottleneck decoder 148 performs the inverse operation of the bottleneck encoder 116 to produce an extracted tensor 149. The extracted tensor 149 is passed to the CNN head 150. CNN head 150 performs the back layers of the task started with CNN trunk 114 to produce task results 151, which are stored in task result buffer 152. The contents of task result buffer 152 can be presented to the user, for example, via a graphical user interface, or provided to an analysis application that decides an action based on the task results, which may include a summary-level presentation of the aggregated task results to the user. The functionality of each of source device 110 and destination device 140 can also be embodied in a single device, examples of which include mobile phones and tablet computers and cloud applications.

[0060] Although example devices are described above, source device 110 and destination device 140 may each typically be configured within a general purpose computer system via a combination of hardware and software components. Figure 2ASuch a computer system 200 is illustrated, and includes: a computer module 201; input devices such as a keyboard 202, a mouse pointer device 203, a scanner 226, a camera 227 that can be configured as a video source 112, and a microphone 280; and output devices including a printer 215, a display device 214 (which can be configured as a display device for presenting task results 151), and a speaker 217. The computer module 201 can use an external modulator-demodulator (modem) transceiver device 216 to communicate with a communication network 220 via a connection 221. The communication network 220, which can represent a communication channel 130, can be a (WAN) such as the Internet, a cellular telecommunications network, or a private WAN. In the case where the connection 221 is a telephone line, the modem 216 can be a traditional "dial-up" modem. Alternatively, in the case where the connection 221 is a high-capacity (e.g., cable or optical) connection, the modem 216 can be a broadband modem. A wireless modem may also be used to make a wireless connection to the communication network 220. The transceiver device 216 may provide the functionality of the transmitter 122 and the receiver 142, and the communication channel 130 may be embodied in the connection 221.

[0061] The computer module 201 typically includes at least one processor unit 205 and a memory unit 206. For example, the memory unit 206 may have a semiconductor random access memory (RAM) and a semiconductor read-only memory (ROM). The computer module 201 also includes a plurality of input / output (I / O) interfaces, wherein the plurality of input / output (I / O) interfaces include: an audio-video interface 207 coupled to a video display 214, a speaker 217, and a microphone 280; an I / O interface 213 coupled to a keyboard 202, a mouse 203, a scanner 226, a camera 227, and an optional joystick or other human interface device (not illustrated); and an interface 208 for an external modem 216 and a printer 215. The signal from the audio-video interface 207 to the computer monitor 214 is typically the output of a computer graphics card. In some implementations, the modem 216 may be built into the computer module 201, such as built into the interface 208. The computer module 201 also has a local network interface 211, which allows the computer system 200 to be coupled to a local area communication network 222, known as a local area network (LAN), via a connection 223. Figure 2A As shown, the local area communication network 222 may also be connected to the wide area network 220 via a connection 224, wherein the local area communication network 222 will typically include a so-called "firewall" device or a device with similar functionality. The local network interface 211 may include an Ethernet (Ethernet TM)Circuit card, Bluetooth TM ) wireless arrangement or IEEE 802.11 wireless arrangement; however, for the interface 211, a variety of other types of interfaces can be implemented. The local network interface 211 can also provide the functionality of the transmitter 122 and the receiver 142, and the communication channel 130 can also be embodied in the local communication network 222.

[0062] I / O interfaces 208 and 213 may provide either or both serial connectivity and parallel connectivity, wherein the former is typically implemented in accordance with the Universal Serial Bus (USB) standard and has a corresponding USB connector (not illustrated). A storage device 209 is provided and typically includes a hard disk drive (HDD) 210. Other storage devices (not illustrated) such as floppy disk drives and tape drives may also be used. An optical disk drive 212 is typically provided to serve as a non-volatile source of data. For example, optical disks (e.g., CD-ROMs, DVDs, Blu-ray Discs, etc.) may be used. TM )), portable memory devices such as USB-RAM, portable external hard drives and floppy disks, etc., as suitable sources of data for computer system 200. Generally, any of HDD 210, optical drive 212, networks 220 and 222 may also be configured to operate as video source 112, or as a destination for decoded video data to be stored for reproduction via display 214. Source device 110 and destination device 140 of system 100 may be embodied in computer system 200.

[0063] The components 205 to 213 of the computer module 201 typically communicate via an interconnect bus 204 and in a manner that results in conventional modes of operation of the computer system 200 known to those skilled in the relevant art. For example, the processor 205 is coupled to the system bus 204 using a connection 218. Likewise, the memory 206 and the optical drive 212 are coupled to the system bus 204 via a connection 219. Examples of computers on which the described arrangement may be practiced include IBM-PC and compatibles, Sun SPARCstation, Apple Mac TM or similar computer system.

[0064] Where appropriate or desired, the source device 110 and the destination device 140 and the methods described below may be implemented using the computer system 200. In particular, the source device 110, the destination device 140 and the methods to be described may be implemented as one or more software applications 233 executable within the computer system 200. The instructions 231 (see FIG. 233 ) executed within the computer system 200 may be used to execute the source device 110, the destination device 140 and the methods to be described. Figure 2B) to implement the source device 110, the destination device 140 and the steps of the method. The software instructions 231 may be formed into one or more code modules, each for performing one or more specific tasks. The software may also be split into two separate parts, with a first part and corresponding code modules performing the method, and a second part and corresponding code modules managing a user interface between the first part and a user.

[0065] For example, the software may be stored in a computer-readable medium including a storage device described below. The software is loaded from the computer-readable medium into the computer system 200 and then executed by the computer system 200. A computer-readable medium having such software or a computer program recorded on the computer-readable medium is a computer program product. The use of the computer program product in the computer system 200 preferably implements an advantageous device for implementing the source device 110 and the destination device 140 and the method.

[0066] The software 233 is typically stored in the HDD 210 or the memory 206. The software is loaded into the computer system 200 from a computer readable medium and executed by the computer system 200. Thus, for example, the software 233 may be stored on an optically readable disk storage medium (e.g., a CD-ROM) 225 read by the optical drive 212.

[0067] In some examples, the application 233 is provided to the user in a manner encoded on one or more CD-ROMs 225 and read via corresponding drives 212, or alternatively, the application 233 can be read by the user from the network 220 or 222. Furthermore, the software can also be loaded into the computer system 200 from other computer-readable media. Computer-readable storage media refers to any non-transitory tangible storage medium that provides recorded instructions and / or data to the computer system 200 for execution and / or processing. Examples of such storage media include floppy disks, magnetic tapes, CD-ROMs, DVDs, Blu-ray discs, hard disk drives, ROMs or integrated circuits, USB memories, magneto-optical disks, or computer-readable cards such as PCMCIA cards, etc., regardless of whether these devices are internal or external to the computer module 201. Examples of transient or non-tangible computer-readable transmission media that may also participate in the provision of software, applications, instructions and / or video data or encoded video data to the computer module 201 include: radio or infrared transmission channels and network connections to another computer or networked device, and the Internet or Intranet including e-mail transmissions and information recorded on websites.

[0068] The second part of the above-mentioned application program 233 and the corresponding code module can be executed to implement one or more graphical user interfaces (GUIs) to be rendered or otherwise represented on the display 214. By typically manipulating the keyboard 202 and the mouse 203, users and applications of the computer system 200 can manipulate the interface in a functionally applicable manner to provide control commands and / or input to the applications associated with these (one or more) GUIs. Other forms of user interfaces that are functionally applicable can also be implemented, such as audio interfaces that utilize voice prompts output via the speaker 217 and user voice commands input via the microphone 280, etc.

[0069] Figure 2B is a detailed schematic block diagram of the processor 205 and the "memory" 234. The memory 234 represents Figure 2A A logical aggregation of all memory modules (including storage device 209 and semiconductor memory 206) that can be accessed by computer module 201 in.

[0070] When the computer module 201 is initially powered on, a power-on self-test (POST) program 250 is executed. The POST program 250 is usually stored in Figure 2A 249 of semiconductor memory 206. Hardware devices such as ROM 249 storing software are sometimes referred to as firmware. POST program 250 checks the hardware within computer module 201 to ensure proper operation, and typically checks processor 205, memory 234 (209, 206), and basic input-output system software (BIOS) module 251, which is also typically stored in ROM 249, for correct operation. Once POST program 250 runs successfully, BIOS 251 activates Figure 2A Activating the hard disk drive 210 causes the boot loader 252 residing on the hard disk drive 210 to be executed via the processor 205. This loads the operating system 253 into the RAM memory 206, where the operating system 253 begins to operate. The operating system 253 is a system-level application executable by the processor 205 to implement various high-level functions including processor management, memory management, device management, storage management, software application interface, and general user interface.

[0071] The operating system 253 manages the memory 234 (209, 206) to ensure that each process or application running on the computer module 201 has sufficient memory to execute without conflicting with the memory allocated to another process. Figure 2AThe different types of memory available in the computer system 200 are described so that each process can run efficiently. Therefore, the aggregate memory 234 is not intended to illustrate how specific segments of memory are allocated (unless otherwise specified), but rather to provide an overview of the memory accessible to the computer system 200 and how such memory is used.

[0072] like Figure 2B As shown, the processor 205 includes a plurality of functional modules, wherein the plurality of functional modules include a control unit 239, an arithmetic logic unit (ALU) 240, and a local or internal memory 248 sometimes referred to as a cache memory. The cache memory 248 generally includes a plurality of storage registers 244 to 246 in a register section. One or more internal buses 241 functionally connect these functional modules to each other. The processor 205 also generally has one or more interfaces 242 for communicating with external devices via the system bus 204 using a connection 218. The memory 234 is coupled to the bus 204 using a connection 219.

[0073] The application program 233 includes an instruction sequence 231 that may include conditional branch instructions and loop instructions. The program 233 may also include data 232 used when executing the program 233. The instructions 231 and data 232 are stored in memory locations 228, 229, 230 and 235, 236, 237, respectively. Depending on the relative size of the instructions 231 and the memory locations 228 to 230, a particular instruction may be stored in a single memory location, as described by the instruction shown in memory location 230. Alternatively, the instruction may be segmented into multiple portions, each stored in a separate memory location, as described by the instruction segments shown in memory locations 228 and 229.

[0074] Typically, a processor 205 is given a set of instructions, which are executed within the processor 205. The processor 205 awaits a subsequent input, which the processor 205 reacts to by executing another set of instructions. Each input may be provided from one or more of a plurality of sources, including data generated by one or more of the input devices 202, 203, data received from an external source across one of the networks 220, 202, data retrieved from one of the storage devices 206, 209, or data retrieved from a storage medium 225 inserted into a corresponding reader 212 (all of which are described in detail in the accompanying drawings). Figure 2A Execution of the instruction set may result in outputting data in some cases. Execution may also involve storing data or variables to memory 234.

[0075] The bottleneck encoder 116, the bottleneck decoder 148, and the method may use input variables 254 stored in corresponding memory locations 255, 256, 257 within the memory 234. The bottleneck encoder 116, the bottleneck decoder 148, and the method produce output variables 261 stored in corresponding memory locations 262, 263, 264 within the memory 234. The intermediate variables 258 may be stored in memory locations 259, 260, 266, and 267.

[0076] refer to Figure 2B The processor 205, registers 244, 245, 246, arithmetic logic unit (ALU) 240 and control unit 239 work together to perform the micro-operation sequence required to perform the "fetch, decode and execute" cycle for each instruction in the instruction set that constitutes the program 233. Each fetch, decode and execute cycle includes:

[0077] A fetch operation for fetching or reading an instruction 231 from a memory location 228 , 229 , 230 ;

[0078] a decode operation in which the control unit 239 determines which instruction was fetched; and

[0079] An execution operation, wherein in the execution operation, the control unit 239 and / or the ALU 240 executes the instruction.

[0080] Thereafter, a further fetch, decode and execute cycle for the next instruction may be performed. Similarly, a store cycle may be performed, whereby the control unit 239 stores or writes a value to the memory location 232 .

[0081] To be explained Figures 6 to 11 Each step or sub-process in the method is associated with one or more segments of program 233, and is typically performed by register segments 244, 245, 247, ALU 240 and control unit 239 in processor 205 working together to perform fetch, decode and execute cycles for each instruction in the instruction set of the segment of program 233.

[0082] Figure 3A is a schematic block diagram showing the functional modules of a backbone 310 of a CNN that can be used as a CNN backbone 114. Backbone 114 is sometimes referred to as "DarkNet-53" and forms the backbone of a "YOLOv3" object detection network. Different backbones are also possible, resulting in different numbers and dimensions of layers of tensor 115 for each frame.

[0083] like Figure 3AAs shown in FIG. 1 , the video data 113 is passed to a resizer module 304. The resizer module 304 resizes the frame to a resolution suitable for processing by the CNN backbone 310, thereby generating resized frame data 312. If the resolution of the frame data 113 is already suitable for the CNN backbone 310, then the operation of the resizer module 304 is not required. The resized frame data 312 is passed to a convolutional batch normalisation leaky rectified linear (CBL) module 314 to generate a tensor 316. The CBL 314 contains reference Figure 3D The modules described by the CBL module 360 ​​are shown.

[0084] refer to Figure 3D , CBL module 360 ​​takes tensor 361 as input. The tensor 361 is passed to convolution layer 362 to produce tensor 363. When convolution layer 362 has a stride of 1 and padding is set to k samples (where the size of the convolution kernel is 2k+1), tensor 363 has the same spatial dimensions as tensor 361. When convolution layer 362 has a larger stride (such as 2, etc.), tensor 363 has a smaller spatial dimension than tensor 361, for example, for a stride of 2, the size of tensor 363 is halved. Regardless of the stride, for a particular CBL block, the size of the channel dimension of tensor 363 may vary compared to the channel dimension of tensor 361. Tensor 363 is passed to batch normalization module 364, which outputs tensor 365. Batch normalization module 364 normalizes the input tensor 363, applying a scaling factor and an offset value to produce an output tensor 365. The scaling factors and offset values ​​are derived from the training process. Tensor 365 is passed to a leaky rectified linear activation ("LeakyReLU") module 366 to produce tensor 367. Module 366 provides a "leaky" activation function whereby positive values ​​in the tensor are passed through and negative values ​​are severely reduced in magnitude, e.g., to 0.1 times their previous value.

[0085] Back to Figure 3A , tensor 316 is passed from CBL block 314 to residual block 11 module 320. Module 320 comprises a sequential cascade of three residual blocks internally containing 1, 2 and 8 residual units respectively.

[0086] References Figure 3BResBlock 340 is shown describing a residual block such as that present in module 320. ResBlock 340 receives tensor 341. The tensor is zero-filled by zero-filling module 342 to produce tensor 343. Tensor 343 is passed to CBL module 344 to produce tensor 345. Tensor 345 is passed to residual unit 346, where residual block 340 contains a series of cascaded residual units. The last residual unit in residual unit 346 outputs tensor 347.

[0087] References Figure 3C 350 is used to describe a residual unit such as unit 346. ResUnit 350 takes tensor 351 as input. Tensor 351 is passed to CBL module 352 to produce tensor 353. Tensor 353 is passed to a second CBL unit 354 to produce tensor 355. Addition module 356 sums tensor 355 with tensor 351 to produce tensor 357. Addition module 356 may also be referred to as a "shortcut" because input tensor 351 substantially affects output tensor 357. For an untrained network, ResUnit 350 acts to pass through tensors. When training, CBL modules 352 and 354 act to deviate tensor 357 from tensor 351 based on training data and ground truth data.

[0088] The Res11 module 320 outputs a tensor 322. The tensor 322 is output from the backbone module 310 as one of the layers and is also provided to the Res8 module 324. The Res8 module 324 is a residual block (i.e., 340) including eight residual units (i.e., 350). The Res8 module 324 generates a tensor 326. The tensor 326 is passed to the Res4 module 328 and is output from the backbone module 310 as one of the layers. The Res4 module is a residual block (i.e., 340) including four residual units (i.e., 350). The Res4 module 328 generates a tensor 329. The tensor 329 is output from the backbone module 310 as one of the layers. In general, the layer tensors 322, 326, and 329 are output as the tensor 115. The backbone CNN 310 may take a video frame with a resolution of 1088×608 as input and produce three tensors corresponding to three layers, which have the following dimensions: [1,256,76,136], [1,512,38,68], [1,1024,19,34]. Another example of three tensors 115 corresponding to three layers may be [1,512,34,19], [1,256,68,38], [1,128,136,76], which are separated at the 75th network layer, the 90th network layer, and the 105th network layer in the CNN 310, respectively. Each tensor may have a different resolution than the next tensor. The resolution of each tensor may be doubled in height and width between the corresponding tensors. When forming the output tensor 115, the tensors 322, 326, and 329 provide a hierarchical representation of frame data including data for feature maps encoded into a bitstream. The separation point depends on CNN 310 .

[0089] Figure 4 is a schematic block diagram showing the functional modules of an alternative backbone 400 of a CNN that can be used as a CNN backbone 114. Backbone 400 implements a residual network with a feature pyramid network ("ResNet FPN") and is an alternative to CNN backbone 114. Frame data 113 is input and passed through stem network 408, res2 module 412, res3 module 416, res4 module 420, res5 module 424 and max pooling module 428 via tensors 409, 413, 417, 425, where max pooling module 428 produces P6 tensor 429 as output.

[0090] The stem network 408 includes 7×7 convolutions and max pooling operations with a stride of two (2). The res2 module 412, the res3 module 416, the res4 module 420, and the res5 module 424 perform convolution operations (LeakyReLU activation). Each module 412, 416, 420, and 424 also halves the resolution of the processed tensor via a stride setting of 2. Tensors 413, 417, 421, and 425 are passed to 1×1 lateral convolution modules 446, 444, 442, and 440, respectively. Modules 440, 442, 444, and 446 produce tensors 441, 443, 445, and 447, respectively. Tensor 441 is passed to a 3×3 output convolution module 470, which produces an output tensor P5 471. Tensor 441 is also passed to an upsampler module 450 to produce an upsampled tensor 451. Summing module 460 sums tensors 443 and 451 to produce tensor 461. Tensor 461 is passed to upsampler module 452 and 3×3 horizontal convolution module 472. Module 472 outputs P4 tensor 473. Upsampler module 452 produces upsampled tensor 453. Summing module 462 sums tensors 445 and 453 to produce tensor 463. Tensor 463 is passed to 3×3 horizontal convolution module 474 and upsampler module 454. Module 474 outputs P3 tensor 475. Upsampler module 454 outputs upsampled tensor 455. Summing module 464 sums tensors 447 and 455 to produce tensor 465, which is passed to 3×3 horizontal convolution module 476. Module 476 outputs P2 tensor 477. Upsampler modules 450, 452, and 454 use nearest neighbor interpolation to reduce computational complexity. Tensors 429, 471, 473, 475, and 477 form the output tensor 115 of CNN backbone 400. In forming output tensor 115, the FPN of tensors 429, 471, 473, 475, and 477 provides a hierarchical representation of frame data including data for feature maps encoded into a bitstream.

[0091] Figure 5 is a schematic block diagram illustrating one type of bottleneck encoder 500 that may be used as the bottleneck encoder 116 . Figure 6 A method 600 is shown for performing the first part of a CNN, performing shrinkage using a bottleneck encoder 500, and encoding the resulting shrunken feature map. Figure 7 Showing the packing arrangement from compressed tensors to feature maps in monochrome video frames.

[0092] The bottleneck encoder 500 receives the FPN tensor 501 corresponding to the tensor 115 and operates to restrict the dimensions of the received tensor to fewer layers and with a reduced spatial size. Applying the bottleneck encoder and decoder between the first part (trunk 114) and the second part (head 150) of the separated neural network enables the reduction of the spatial area in the packed tensor data frame. The reduction in spatial area is achieved by using the interface between the bottleneck encoder and the bottleneck decoder as the split point of the first part and the second part (114 and 150) of the neural network. The bottleneck encoder 116 acts as an additional layer attached to the first part of the neural network, and the bottleneck decoder 148 acts as an additional layer pre-set to the second part of the neural network.

[0093] The sensitivity of the task results to the bottleneck depends on the nature of the task. For object detection, there is less spatial sensitivity, and therefore the spatial downsampling of the larger layer of FPN is less harmful to the resulting mAP. In contrast, especially for the larger spatial tensors of FPN, the segmentation and the resulting segmentation map are more sensitive to the loss of spatial details, and therefore benefit from less severe spatial downsampling. In the described arrangement, the bottleneck encoder 500 operates at two separate scales with two different spatial resolutions, rather than applying a single scale across all FPN layers. The higher of the two resolution scales is used for the larger FPN layer, thereby retaining more details through the bottleneck encoder 500. The lower of the two resolution scales is used for the smaller FPN layer. The input FPN tensor 501 includes layers P2 502, P3 503, P4 504, and P5 505. The spatial resolution of layers P2 to P4 (502, 503, 504) is a power of 2 of the spatial resolution of P5 505. Where P5 505 has a width and height of (w,h), P2 to P4 (502,503,504) have dimensions (8w,8h), (4w,4h), (2w,2h), respectively. In other words, the individual tensors have a resolution that forms a sequence of indices that double in width and height between consecutive tensors. Layers P2 to P5 each have 256 channels.

[0094] In the above example, inputs P2 to P5 are Figure 4 The hierarchical feature pyramid network output (P2 477, P3 475, P4 473 and P5 471) generated by the CNN backbone 400 of Figure 3AThe tensors 329, 326, and 322 are implemented and output, and the tensors 329, 326, and 322 are derived into two sets of tensors: the first group has one FPN layer 329, and the second group has two FPN layers 322 and 326. The first group with one FPN layer 329 has the smallest spatial resolution and does not require the operation of the MSFF module 510 in the bottleneck encoder 116, where the tensor 329 is directly passed to the SSFC encoder 550 as tensor 529. The second group is processed by the MSFF 510 in the bottleneck encoder 116, where the tensor 326 is passed in (input) as tensor 503, and the tensor 322 is passed in as tensor 502.

[0095] The method 600 may be implemented using a device such as a configured FPGA, ASIC, or ASSP. Alternatively, as described below, the method 600 may be implemented by the source device 110 as one or more software code modules of the application 233 under execution by the processor 205. The software code modules of the application 233 implementing the method 600 may reside, for example, in the hard drive 210 and / or the memory 206. The method 600 is repeated for each frame of video data generated by the video source 112. The method 600 may be stored on a computer readable storage medium and / or in the memory 206. The method 600 begins by performing a first portion of the neural network step 610.

[0096] At step 610, the CNN backbone 114 performs a neural network layer corresponding to the first portion of the neural network under execution by the processor 205. Step 610 effectively implements the CNN backbone 114. For example, the following may be performed as described in reference Figure 4 The CNN layer is used to generate P2 to P5 tensors 501 (115). From step 610, control in processor 205 proceeds to select first set of tensors step 615.

[0097] The described arrangement effectively divides the tensor produced by the CNN backbone 114 into a first set and a second set of multiple tensors (also referred to as multiple tensors), the tensors (or at least one tensor) in each set having spatial resolution feature maps that are different from each other. At step 615, the bottleneck encoder 116, under the execution of the processor 205, selects multiple adjacent tensors in the tensor 501 as the first multiple tensors (such as two tensors P4 504 and P5 505, etc.). The tensor 501 forms a hierarchical representation of the frame data 113, which is generated by applying the FPN to the frame data 113. Using a stride equal to two convolution stages in the FPN (i.e., at modules 412, 416, 420, and 424) results in the spatial dimensions of the tensors in the tensor 501 being halved in width and height as each corresponding tensor is sorted according to the decomposition level, for example, from P2 to P5. Although the layers have different spatial resolutions, there is a certain degree of inter-layer correlation between layers P2 to P5 of the tensor 501. As long as the tensors are first spatially scaled to the same resolution (e.g., the smallest resolution in the tensors to be combined), exploiting inter-layer correlations permits a reduction in channel count relative to the concatenation of tensors across layers. Combining tensors of greatly different spatial resolutions results in an excessive loss of detail in the higher resolution tensor due to the higher rate of downsampling operations. For example, scaling P2 502 to P5 505 requires reducing the width and height to one-eighth of their previous values ​​to reduce the area to one-sixty-fourth of the area of ​​P2 502. For tasks that rely on spatial detail, such as instance segmentation, mAP degrades. For detection of small objects where the network head relies on higher resolution layers, a reduction in mAP occurs due to excessive downsampling of the larger layers. Control in processor 205 enters select second set of tensors step 620 from step 615.

[0098] At step 620, the processor 205 selects a second set of tensors that also includes tensors of adjacent spatial scales in layers P2 to P5 and includes the remaining tensors in P2 to P5 that were not selected as the second plurality of tensors at step 615. For example, tensors P2 502 and P3 503 are selected at step 620. As a result of steps 615 and 520, tensor 501 is distributed into two sets of multiple tensors. Control in processor 206 passes from step 620 to a combine first tensor step 630.

[0099] At step 630, the MSFF module 510 (see Figure 5) Under execution by processor 205, the tensors of the first set of tensors (i.e., 504, 505) are combined to produce a combined tensor 529. The downsampling module 522 operates on the tensor with the larger spatial scale (i.e., P4 504 at 2h, 2w, 256) to downsample to match the spatial scale of the smaller tensor (i.e., P5 505 at h, w, 256), thereby producing a downscaled P5 tensor 523. The cascade module 524 performs channel-by-channel cascading of tensors 505 and 523 to produce a cascaded tensor 525 of dimension h, w, 512. The cascaded tensor 525 is passed to a squeeze and excite (SE) module 526 to produce a tensor 527. The SE module 526 sequentially performs global pooling, a fully connected layer with reduced channel count, a rectified linear unit activation, a second fully connected layer with restored channel count, and a sigmoid activation function to produce a scaled tensor. Tensor 525 is scaled according to the scaling tensor to produce an output as tensor 527. SE block 526 can be trained to adaptively change the weighting of different channels in the passed tensor based on the first fully connected layer output. The first fully connected layer output reduces each feature map for each channel to a single value, which is then passed through a non-linear activation unit (ReLU) to create a conditional representation of the unit suitable for the weighting of other channels, where the recovery to the full channel count is performed by the second fully connected layer. Thus, SE block 526 is able to extract non-linear inter-channel correlation when generating tensor 527 from tensor 525 to a greater extent than is possible using purely convolutional (linear) layers. Since tensors 525 and 527 contain 512 channels (the result of the cascade of two FPN layers), the decorrelation achieved by SE block 526 spans two FPN layers P5 and P4.

[0100] Tensor 527 is passed to convolutional layer 528. Convolutional layer 528 implements one or more convolutional layers to produce a first combined tensor 529 in which the channel count is reduced to F channels (typically 256 channels). As a result of step 630, the two FPN layer tensors are reduced to a single tensor having the same channel count as the input FPN layer tensor and the spatial resolution of the smaller of the two FPN layer tensors. This dimensionality reduction is achieved using several network layers and relies on training layers (e.g., layers 526 and 528) rather than on-the-fly determination of correlations to exploit. Return to Figure 6 , control in processor 205 enters from step 630 to step 640 of combining the second tensor.

[0101] At step 640, the MSFF module 510, under the execution of the processor 205, combines the tensors in the second tensor (i.e., 502, 503) to generate the reference Figure 5The combined tensor 519 is described. The stages performed as part of step 640 correspond to the stages of step 630, except that these stages are applied to the tensor selected at step 620 instead of the tensor selected at step 615. The downsampling module 512 operates on the tensor with the larger spatial scale (i.e., P2 502 at 8h, 8w, 256) to downsample to match the spatial scale of the smaller tensor (i.e., P3 503 at 4h, 4w, 256), thereby producing a downscaled P2 tensor 513. The concatenation module 514 performs a channel-by-channel concatenation of tensors 503 and 513 to produce a concatenated tensor 515 of dimension 4h, 4w, 512. The concatenated tensor 515 is passed to a squeeze and excitation (SE) module 516 to produce a tensor 517. The SE module 516 operates in the same manner as described with reference to the SE module 526. The tensor 517 is passed to the convolutional layer 518. Convolutional layer 518 operates in a similar manner to convolutional layer 528 to produce a second combined tensor 519 in which the channel count is reduced to F channels (typically 256 channels). As a result of step 640, the two FPN layer tensors are reduced to a single tensor having the same channel count as the input FPN layer tensor and the spatial resolution of the smaller of the two FPN layer tensors. Dimensionality reduction is achieved using several network layers and relies on training layers (e.g., layers 516 and 518) rather than on-the-fly determination of correlations to exploit. As Figure 6 As shown, control in the processor 205 enters from step 640 into a single scale feature compression (SSFC) encoding first tensor step 650 .

[0102] In step 650, if Figure 5 As shown, SSFC encoder 550 is implemented under execution of processor 205. The operation of SSFC encoder 550 reduces the dimension of combined tensor 529 to produce compressed tensor 557. Combined tensor 529 is passed to convolution layer 552 to produce tensor 553. Tensor 553 has a channel count reduced from 256 to a smaller value C' (such as 64, etc.). The value 96 can also be used for C', which results in reference to Figure 7 553 is passed to a batch normalization module 554 to produce a tensor 555. The batch normalized tensor 555 has the same dimensions as the tensor 553. The tensor 555 is passed to a tanh layer 556. The tanh layer 556 implements a hyperbolic tangent (tanh) layer as per layer 536 to produce a compressed tensor 557. The compressed tensor 557 has the same dimensions as the tensor 553.

[0103] like Figure 6As shown, control in the processor 205 enters the SSFC encoding second tensor step 660 from step 650. In step 660, the SSFC encoder 530 is implemented under the execution of the processor 205 and is as described with respect to Figure 5 As described. The SSFC encoder 530 operates to further reduce the dimensionality of the combined tensor 519 to produce a compressed tensor 537. The combined tensor 519 is passed to a convolutional layer 532 to produce a tensor 533 having a channel count reduced from 256 to a smaller value C' (such as 64, etc.). Tensor 533 is passed to a batch normalization module 534 to produce a tensor 535. Batch normalized tensor 535 has the same dimension as tensor 533. Tensor 535 is passed to a tanh layer 536 to produce a compressed tensor 537. Compressed tensor 537 has the same dimension as tensor 533. The use of a hyperbolic tangent (tanh) layer compresses the dynamic range of values ​​within tensor 537 to [-1,1], thereby removing outliers.

[0104] The compressed tensors 557 and 537 each provide an information unit of a feature map of the frame data 113 obtained using the following convolution operation: (i) the MSFF 510 and SSFC encoder 550 for the tensors 505 and 504, or (ii) the MSFF 510 and SSFC encoder 530 for the tensors 503 and 502. The compressed tensors 557 and 537 provide a set of tensors 560 corresponding to the bottleneck encoded tensor 117. When step 660 is completed, as shown in FIG. Figure 6 As shown, control in processor 205 enters pack compressed tensor step 670 from step 660 .

[0105] At step 670, the quantization and packing module 118, under execution of the processor 205, quantizes the compressed tensors 537 and 557 and packs the quantized tensors into a single monochrome video frame. Figure 7 An example single monochrome video frame 700 is shown in . Frame 700 corresponds to frame data 119. Since the tanh activation function is used at 536 and 556, respectively, the range of compressed tensors 537 and 557 is [-1, 1]. The property of tanh in removing outliers results in a distribution suitable for linear quantization of the bit depth of frame 700. The channels of compressed tensor 537 are packed into feature maps of a certain size, such as feature map 710 in frame 700. The channels of compressed tensor 557 are packed into feature maps of different sizes in frame 700, such as feature map 712. One channel of compressed tensor 537 corresponds to one feature map indicated by a rectangular area (such as area 710). One channel of compressed tensor 557 corresponds to one feature map indicated by a rectangular area (such as area 712). Return to Figure 6 , control in processor 205 enters compress frame step 680 from step 670.

[0106] At step 680, the feature map encoder 120, under execution of the processor 205, encodes the frame 700 to generate a bit stream 121. Figure 8 6 to illustrate the operation of the feature map encoder 120. The method 600 terminates when executing step 680, wherein the FPN layer of the image frame 312 is dimensionally reduced and compressed into the video bitstream 121.

[0107] Figure 8 1 is a schematic block diagram showing the functional modules of the video encoder 120 (also called feature map encoder). The video encoder 120 performs packetization of the frame 119 (in Figure 7 700) to produce a bitstream 121. Typically, data is passed between functional modules within the video encoder 120 in groups of samples or coefficients (such as a partition of a block into fixed-size sub-blocks, etc.) or as an array. Figure 2A and Figure 2B As shown, the video encoder 120 can be implemented using a general-purpose computer system 200, wherein various functional modules can be implemented using dedicated hardware within the computer system 200, using software executable within the computer system 200 (such as one or more software code modules of a software application 233 residing on the hard disk drive 205 and controlled by the processor 205 for execution, etc.). Alternatively, the video encoder 120 can be implemented using a combination of dedicated hardware and software executable within the computer system 200. The video encoder 120 and the method can alternatively be implemented in dedicated hardware such as one or more integrated circuits that perform the functions or sub-functions of the method. Such dedicated hardware may include a graphics processing unit (GPU), a digital signal processor (DSP), an application-specific standard product (ASSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or one or more microprocessors and associated memory. In particular, the video encoder 120 includes modules 810 to 890, wherein each of these modules can be implemented as one or more software code modules of the software application 233.

[0108] although Figure 8 The video encoder 120 is an example of a Versatile Video Coding (VVC) video encoding pipeline, but other video codecs may be used to perform the processing stages described herein. The frame data 119 may be in any chroma format and bit depth supported by the profile being used, for example, 4:0:0, 4:2:0 of ​​the "Main 10" profile of the VVC standard, with a sample precision of eight (8) to ten (10) bits.

[0109] The block partitioner 810 first partitions the frame data 119 into CTUs, which are typically square in shape and configured so that a specific size of the CTU is used. The maximum effective size of a CTU can be, for example, 32×32, 64×64, or 128×128 luminance samples, configured by the "sps_log2_ctu_size_minus5" syntax element present in the "sequence parameter set". The CTU size also provides the maximum CU size, since a CTU that is not further split will contain one CU. The block partitioner 810 also partitions each CTU into one or more CBs based on the luminance coding tree and the chrominance coding tree. The luminance channel may also be referred to as the primary color channel. Each chrominance channel may also be referred to as a secondary color channel. CBs have various sizes and may include both square and non-square aspect ratios. However, in the VVC standard, CBs, CUs, PUs, and TUs always have side lengths that are powers of two. Thus, based on the luma coding tree and chroma coding tree of the CTU, a current CB denoted as 812 is output from the block partitioner 810 (advancing based on iteration over one or more blocks of the CTU).

[0110] The CTUs resulting from the first partitioning of the frame data 119 may be scanned in a raster scan order, and may be grouped into one or more "slices". A slice may be an "intra" (or "I") slice. An intra slice (I slice) indicates that each CU in the slice is intra predicted. Typically, the first picture in a coding layer video sequence (CLVS) contains only I slices and is referred to as an "intra picture". A CLVS may contain periodic intra pictures that form "random access points" (i.e., intermediate frames in a video sequence from which decoding may begin). Alternatively, a slice may be uni-predicted or bi-predicted ("P" or "B" slices, respectively), indicating the additional availability of uni-prediction and bi-prediction in a slice, respectively.

[0111] The video encoder 120 encodes a sequence of pictures according to a picture structure. One picture structure is "low delay", in which a picture using inter-frame prediction may only reference pictures that appeared previously in the sequence. Low delay allows each picture to be output as soon as it is decoded, and is also stored for possible reference by subsequent pictures. Another picture structure is "random access", in which the encoding order of the pictures is different from the display order. Random access allows inter-frame predicted pictures to reference other pictures that have been decoded but have not yet been output. A certain degree of picture buffering is required so that future reference pictures in display order are present in the decoded picture buffer, resulting in multiple frame delays.

[0112] When using a chroma format other than 4:0:0, in an I slice, the coding tree for each CTU can diverge below the 64×64 level into two separate coding trees, one for luma and the other for chroma. The use of separate trees allows different block structures to exist between luma and chroma within the luma 64×64 region of the CTU. For example, a large chroma CB can be co-located with many smaller luma CBs, and vice versa. In a P or B slice, a single coding tree for a CTU defines a block structure common to luma and chroma. The resulting blocks of a single tree can be intra-predicted or inter-predicted.

[0113] In addition to partitioning a picture into slices, a picture can also be partitioned into "tiles". A tile is a sequence of CTUs covering a rectangular area of ​​the picture. CTU scanning is performed within each tile in a raster scan, advancing from one tile to the next. A slice can be an integer number of tiles, or an integer number of consecutive CTU rows within a given tile.

[0114] For each CTU, the video encoder 120 operates in two stages. In the first stage (referred to as the "search" stage), the block partitioner 810 tests various potential configurations of the coding tree. Each potential configuration of the coding tree has an associated "candidate" CB. The first stage involves testing various candidate CBs to select a CB that provides relatively high compression efficiency and relatively low distortion. The test typically involves Lagrangian optimization, whereby the candidate CB is evaluated based on a weighted combination of rate (i.e., coding cost) and distortion (i.e., error relative to the input frame data 119). The "best" candidate CB (i.e., the CB with the lowest evaluated rate / distortion) is selected for subsequent encoding into the bitstream 121. Included in the evaluation of the candidate CB is the option of using a CB for a given region, or further splitting the region according to various splitting options and encoding each smaller resulting region with a further CB, or even further splitting the region. Therefore, both the coding tree and the CB itself are selected in the search stage.

[0115] The video encoder 120 generates a prediction block (PB) indicated by arrow 820 for each CB (e.g., CB 812). PB 820 is a prediction of the content of the associated CB 812. Subtractor module 822 generates a difference (or "residual", meaning the difference is in the spatial domain) between PB 820 and CB 812, represented as 824. Difference 824 is the block size difference between corresponding samples in PB 820 and CB 812. Difference 824 is transformed, quantized, and represented as a transform block (TB) indicated by arrow 836. PB 820 and associated TB 836 are typically selected from one of a plurality of possible candidate CBs, for example, based on an assessed cost or distortion.

[0116] A candidate coding block (CB) is a CB obtained from one of the prediction modes available to the video encoder 120 for the associated PB and the resulting residual. When combined with a predicted PB in the video encoder 120, the TB 836 reduces the difference between the decoded CB and the original CB 812 at the expense of additional signaling in the bitstream.

[0117] Thus, each candidate coding block (CB) (i.e., a prediction block (PB) in combination with a transform block (TB)) has an associated coding cost (or "rate") and an associated difference (or "distortion"). The distortion of the CB is typically estimated as a difference in sample values, such as a sum of absolute differences (SAD), a sum of squared differences (SSD), or a Hadamard transform applied to the difference, etc. The mode selector 886 uses the difference 824 to determine the estimates obtained from each candidate PB to determine a prediction mode 887. The prediction mode 887 indicates a decision to use a particular prediction mode (e.g., intra-frame prediction or inter-frame prediction) for the current CB. The estimation of the coding cost associated with each candidate prediction mode and the corresponding residual encoding can be performed at a significantly lower cost than entropy encoding of the residual. Therefore, even in a real-time video encoder, multiple candidate modes can be evaluated to determine the best mode in terms of rate-distortion.

[0118] Determining the best mode based on rate-distortion is usually implemented using a variation of Lagrangian optimization.

[0119] A Lagrangian or similar optimization process may be employed to select both the best partitioning of a CTU into CBs (using the block partitioner 810) and the selection of the best prediction mode from multiple possibilities. The intra prediction mode with the lowest cost metric is selected as the "best" mode by applying a Lagrangian optimization process of the candidate modes in the mode selector module 886. The lowest cost mode includes the selected secondary transform index 888, which is also encoded into the bitstream 121 by the entropy encoder 838.

[0120] In the second stage of the operation of the video encoder 120 (referred to as the "encoding" stage), the determined coding tree(s) for each CTU are iterated in the video encoder 120. For CTUs using separate trees, the luma coding tree is encoded first, followed by the chroma coding tree, for each 64x64 luma region of the CTU. Within the luma coding tree, only the luma CBs are encoded, and within the chroma coding tree, only the chroma CBs are encoded. For CTUs using shared trees, a single tree describes the CU (i.e., luma CBs and chroma CBs) according to the common block structure of the shared tree.

[0121] The entropy encoder 838 supports bitwise encoding of syntax elements using variable length and fixed length codewords, as well as arithmetic coding modes of syntax elements. Portions of the bitstream such as "parameter sets" (e.g., sequence parameter sets (SPS) and picture parameter sets (PPS)) use a combination of fixed length codewords and variable length codewords. Slices (also called continuous portions) have a slice header using variable length encoding, followed by slice data using arithmetic encoding. The slice header defines parameters specific to the current slice, such as slice-level quantization parameter offsets, etc. The slice data includes syntax elements for each CTU in the slice. The use of variable length encoding and arithmetic coding requires sequential parsing within each portion of the bitstream. These portions can be described with start codes to form "network abstraction layer units" or "NAL units". Arithmetic coding is supported using context-adaptive binary arithmetic coding processing.

[0122] The arithmetically coded syntactic elements consist of a sequence of one or more "bins" (binary files). Like bits, bin values ​​are "0" or "1". However, bins are not encoded in the bitstream 121 as discrete bits. Bins have associated predicted (or "likely" or "maximum probability") values ​​and associated probabilities (known as "context"). When the actual bin to be encoded matches the predicted value, the "maximum probability symbol" (MPS) is encoded. Encoding the maximum probability symbol is relatively cheap in terms of consumed bits in the bitstream 121, including a total cost of less than one discrete bit. When the actual bin to be encoded does not match the possible value, the "minimum probability symbol" (LPS) is encoded. Encoding the minimum probability symbol has a relatively high cost in terms of consumed bits. The bin encoding technique enables efficient encoding of bins that skew the probability of "0" vs "1". For syntactic elements with two possible values ​​(i.e., "flag"), a single bin is sufficient. For syntactic elements with many possible values, a sequence of bins is required.

[0123] The presence of a later bin in a sequence can be determined based on the value of an earlier bin in the sequence. In addition, each bin can be associated with more than one context. The selection of a particular context can depend on the bin values ​​of an earlier bin in a syntax element, an adjacent syntax element (i.e., an adjacent syntax element from an adjacent block), etc. Each time a context coding bin is encoded, the context selected for that bin (if present) is updated to reflect the new bin value. Therefore, the binary arithmetic coding scheme is called adaptive.

[0124] The entropy encoder 838 also supports bins lacking context (referred to as "bypass bins"). Bypass bins are encoded assuming an equal probability distribution between "0" and "1". Therefore, each bin has a coding cost of one bit in the bitstream 121. The absence of context saves memory and reduces complexity, so bypass bins where the distribution of values ​​for a particular bin is not skewed are used. An example of an entropy encoder that employs context and adaptation is known in the art as CABAC (Context Adaptive Binary Arithmetic Coder), and many variations of this encoder have been employed in video encoding.

[0125] The entropy encoder 838 encodes a quantization parameter 892 using a combination of context-encoded and bypass-encoded bins and, if used for the current CB, an LFNST index 888. The quantization parameter 892 is encoded using a "delta QP" generated by the QP controller module 890. The delta QP is signaled at most once in each region known as a "quantization group". The quantization parameter 892 is applied to the residual coefficients of the luma CB. The adjusted quantization parameter is applied to the residual coefficients of the collocated chroma CB. The adjusted quantization parameter may include mapping from the luma quantization parameter 892 according to a mapping table and a CU-level offset selected from an offset list. The secondary transform index 888 is signaled when the residual associated with the transform block includes valid residual coefficients only in those coefficient positions that are transformed into primary coefficients by applying a secondary transform.

[0126] The residual coefficients of each TB associated with a CB are encoded using a residual syntax. The residual syntax is designed to efficiently encode coefficients with low amplitudes, primarily using arithmetic coded bins to indicate the significance of the coefficients and the amplitude of lower values, and reserving bypass bins for residual coefficients of higher amplitudes. Therefore, residual blocks consisting of very low amplitude values ​​and sparse placement of significant coefficients are efficiently compressed. In addition, there are two residual coding schemes. As seen when the transform is applied, the conventional residual coding scheme is optimized for TBs where significant coefficients are primarily located in the upper left corner of the TB. The transform skip residual coding scheme can be used for TBs that are not transformed, and is able to efficiently encode residual coefficients regardless of their distribution throughout the TB.

[0127] The multiplexer module 884 outputs the PB 820 from the intra prediction module 864 according to the determined best intra prediction mode selected from the test prediction modes of each candidate CB. The candidate prediction modes need not include every conceivable prediction mode supported by the video encoder 120. Intra prediction is divided into three types: first, "DC intra prediction", which involves filling the PB with a single value representing the average of nearby reconstructed samples; second, "plane intra prediction", which involves filling the PB with samples according to a plane, using a DC offset and vertical and horizontal gradients derived from nearby reconstructed neighboring samples. The nearby reconstructed samples typically include a row of reconstructed samples above the current PB (extending to a certain extent to the right of the PB) and a column of reconstructed samples to the left of the current PB (extending to a certain extent downward beyond the PB); and third, "angle intra prediction", which involves filling the PB with reconstructed neighboring samples filtered and propagated across the PB in a particular direction (or "angle"). In VVC, sixty-five (65) angles are supported, with rectangular blocks being able to take advantage of additional angles not available to square blocks, yielding a total of eighty-seven (87) angles.

[0128] A fourth type of intra prediction can be used for chroma PBs, whereby the PBs are generated from collocated luma reconstructed samples according to a "cross component linear model" (CCLM) mode. Three different CCLM modes are available, each using a different model derived from adjacent luma and chroma samples. The derived model is used to generate a block of samples of the chroma PB from collocated luma samples. Matrix multiplication of reference samples can be used to intra predict luma blocks using a matrix selected from a predefined set of matrices. This matrix intra prediction (MIP) achieves gains by using matrices trained on a large set of video data, where the matrices represent the relationship between reference samples and prediction blocks that is not easily captured in angular, planar, or DC intra prediction modes.

[0129] Module 864 can also generate prediction units by copying blocks from nearby the current frame using the "intra-block copy" (IBC) method. The location of the reference block is constrained to an area equivalent to one CTU (the CTU is divided into 64×64 regions known as VPDUs), which covers the processed VPDUs of the current CTU and the VPDUs of the previous (one or more) CTUs within each row of CTUs and within each slice or tile, up to the area limit corresponding to one 128×128 luma sample, regardless of the configured CTU size of the bitstream. This area is called the "IBC virtual buffer" and limits the IBC reference area, thereby limiting the required storage. The IBC buffer is filled with reconstructed samples 854 (i.e., before loop filtering), so a buffer separate from the frame buffer 872 is required. When the CTU size is 128×128, the virtual buffer includes samples only from the CTU adjacent to the left of the current CTU. When the CTU size is 32×32 or 64×64, the virtual buffer includes CTUs from up to four or sixteen CTUs to the left of the current CTU. Regardless of the CTU size, access to adjacent CTUs to obtain samples of the IBC reference block is constrained by boundaries (such as the edges of pictures, slices, or tiles). In particular, for feature maps of FPN layers with smaller sizes, using CTU sizes such as 32×32 or 64×64 results in more aligned reference areas to cover the set of previous feature maps. Accessing similar feature maps for IBC prediction provides the advantage of efficient coding when feature map placement is sorted based on SAD, SSE, or other difference metrics.

[0130] The residuals of the prediction blocks when encoding feature map data are different from the residuals seen for natural video. Such natural video is usually captured by a camera sensor or screen content, as is commonly seen in operating system user interfaces, etc. Feature map residuals tend to contain many details, which are more suitable for transform skip coding than the significant low-frequency coefficients of various transforms. Experiments show that feature map residuals have sufficient local similarity to benefit from transform coding. However, the distribution of feature map residual coefficients does not cluster toward the DC (upper left) coefficients of the transform block. In other words, when encoding feature map data, there is enough correlation to make the transform show gain, and this also applies when using intra-block replication to generate prediction blocks for feature map data. Therefore, when encoding feature map data, when evaluating the residuals generated by candidate block vectors for intra-block replication, Hadamard cost estimates can be used instead of relying solely on SAD or SSD cost estimates. SAD or SSD cost estimates tend to select block vectors with residuals that are more suitable for transform skip coding, and may miss block vectors with residuals that will be compactly encoded using transforms. When encoding feature map data, the Multiple Transform Selection (MTS) tool of the VVC standard may be used so that, in addition to the DCT-2 transform, a combination of DCT-7 and DCT-8 transforms may also be used for residual coding horizontally and vertically.

[0131] An intra-predicted luma coding block can be partitioned vertically or horizontally into a set of equally sized prediction blocks, each with a minimum area of ​​sixteen (16) luma samples. This intra sub-partitioning (ISP) approach enables a single transform block to contribute to the prediction block generation from one sub-partition to the next in the luma coding block, thereby improving compression efficiency.

[0132] In cases where previously reconstructed neighboring samples are not available, such as at the edge of a frame, a default halftone value of half the sample range is used. For example, for 10-bit video, a value of five hundred twelve (512) is used. Since no previous samples are available for the CB located at the top left position of the frame, the angular and planar intra prediction modes produce the same output as the output of the DC prediction mode (i.e., a flat plane of samples with halftone values ​​as amplitudes).

[0133] For inter-frame prediction, a prediction block 882 is generated by a motion compensation module 880 using samples from one or two frames before the current frame in the coding order frame in the bitstream, and the prediction block 882 is output by a multiplexer module 884 as a PB820. In addition, for inter-frame prediction, a single coding tree is typically used for both the luminance channel and the chrominance channel. The order of the coded frames in the bitstream may be different from the order of the frames when captured or displayed. When one frame is used for prediction, the block is called "single prediction" and has one associated motion vector. When two frames are used for prediction, the block is called "double prediction" and has two associated motion vectors. For P slices, each CU can be intra-predicted or uni-predicted. For B slices, each CU can be intra-predicted, uni-predicted, or bi-predicted.

[0134] Frames are usually encoded using a "group of pictures" (GOP) structure to achieve the temporal hierarchy of frames. Frames can be divided into multiple slices, each of which encodes a portion of the frame. The temporal hierarchy of frames allows frames to refer to the previous and next pictures in the order in which the frames are displayed. The images are encoded in the necessary order to ensure that the correlation for decoding each frame is met. Affine inter-frame prediction mode is available, where instead of using one or two motion vectors to select and filter the reference sample blocks of the prediction unit, the prediction unit is divided into multiple smaller blocks and a motion field is generated so that each smaller block has a different motion vector. The motion field uses the motion vectors of nearby points of the prediction unit as "control points". Affine prediction allows encoding of motions other than translation, with less need for coding trees that use depth splitting. The dual prediction mode available with VVC performs geometric blending of two reference blocks along the selected axis, and signals the angle and offset relative to the center of the block. This geometric partition mode ("GPM") allows the use of larger coding units along the boundary between two objects, and the geometry of the boundary for the coding of the coding unit is used as the angle and center offset. Motion vector differences can be encoded as direction (up / down / left / right) and distance (a set of power-of-2 distances are supported) instead of using a Cartesian (x,y) offset. Motion vector predictors are obtained from neighboring blocks ("merge mode") as if no offset was applied. The current block will share the same motion vector as the selected neighboring block.

[0135] Samples are selected based on the motion vector 878 and the reference picture index. The motion vector 878 and the reference picture index are applied to all color channels, so inter-frame prediction is described mainly in terms of operations on PUs rather than PBs. A single coding tree is used to describe the decomposition of each CTU into one or more inter-frame prediction blocks. The inter-frame prediction method can vary in the number of motion parameters and their precision. The motion parameters typically include a reference frame index that indicates which reference frame(s) in the reference frame list will be used plus the spatial translation of each reference frame, but may include more frames, specific frames, or complex affine parameters (such as scaling and rotation, etc.). In addition, a predetermined motion refinement process can be applied to generate a dense motion estimate based on the reference sample block.

[0136] The PB 820 has been determined and selected, and the PB 820 is subtracted from the original sample block at the subtractor 822 to obtain a residual with the lowest coding cost (denoted as 824), and the residual is lossily compressed. The lossy compression process includes the steps of transform, quantization and entropy coding. The forward main transform module 826 applies a forward transform to the difference 824, converts the difference 824 from the spatial domain to the frequency domain, and produces the main transform coefficients represented by the arrow 828. The maximum main transform size in one dimension is a 32-point DCT-2 or 64-point DCT-2 transform configured by the "sps_max_luma_transform_size_64_flag" in the sequence parameter set. If the CB being encoded is larger than the maximum supported main transform size (e.g., 64×64 or 32×32) represented as a block size, the main transform 826 is applied in a tiled manner to transform all samples of the difference 824. In the case of using a non-square CB, the tiling is also performed using the maximum available transform size in each dimension of the CB. For example, when using a maximum transform size of thirty-two (32), a 64×16 CB uses two 32×16 primary transforms arranged in a tiled manner. When the size of the CB is larger than the maximum supported transform size, the CB is filled with TBs in a tiled manner. For example, a 128×128 CB with a 64-pt transform maximum size is filled with four 64×64 TBs in a 2×2 arrangement. A 64×128 CB with a 32-pt maximum size is filled with eight 32×32 TB transforms in a 2×4 arrangement.

[0137] The application of the transform 826 results in multiple TBs for the CBs. When each application of the transform operates on a TB of difference 824 greater than 32×32 (e.g., 64×64), all resulting main transform coefficients 828 outside the upper left 32×32 region of the TB are set to zero (i.e., discarded). The remaining main transform coefficients 828 are passed to a quantizer module 834. The main transform coefficients 828 are quantized according to a quantization parameter 892 associated with the CB to produce main transform coefficients 832. In addition to the quantization parameter 892, the quantizer module 834 may also apply a "scaling list" to allow non-uniform quantization within a TB by further scaling the residual coefficients according to their spatial position within the TB. The quantization parameter 892 may be different for the luma CB and each chroma CB. The main transform coefficients 832 are passed to a forward secondary transform module 830 to produce transform coefficients represented by arrow 836 by performing a non-separable secondary transform (NSST) operation or bypassing the secondary transform. The forward main transform is typically separable, transforming a set of rows and then a set of columns of each TB. For a luma TB with a width and height not exceeding 16 samples, the forward main transform module 826 uses a type II discrete cosine transform (DCT-2) in the horizontal and vertical directions, or a bypass of the transform in the horizontal and vertical directions, or a combination of a type VII discrete sine transform (DST-7) and a type VIII discrete cosine transform (DCT-8) in the horizontal or vertical directions. In the VVC standard, the combination of using DST-7 and DCT-8 is called a "multi-transform selection set" (MTS).

[0138] The forward secondary transform of module 830 is typically a non-separable transform that is applied only to the residual of the intra-predicted CU and can nevertheless be bypassed. The forward secondary transform operates on sixteen (16) samples (arranged as the upper left 4×4 sub-block of the main transform coefficients 828) or forty-eight (48) samples (arranged as three 4×4 sub-blocks of the upper left 8×8 coefficients of the main transform coefficients 828) to produce a set of secondary transform coefficients. The set of secondary transform coefficients can be fewer in number than the set of main transform coefficients from which they are derived. Since the secondary transform is only applied to a set of coefficients that are adjacent to each other and include the DC coefficient, the secondary transform is called a "low-frequency non-separable secondary transform" (LFNST). This secondary transform can be obtained through a training process and, due to its non-separable nature and trained origin, can exploit additional redundancy in the residual signal that cannot be captured by a separable transform (such as a variant of DCT and DST, etc.). In addition, when LFNST is applied, all remaining coefficients in the TB are zero in both the main transform domain and the secondary transform domain.

[0139] The quantization parameter 892 is constant for a given TB, and thus results in uniform scaling of the resulting residual coefficients in the main transform domain for the TB. The quantization parameter 892 may vary periodically with a signaled "delta quantization parameter". The delta quantization parameter (delta QP) is signaled once for a CU contained in a given region (referred to as a "quantization group"). If the CU is larger than the quantization group size, the delta QP is signaled once using one of the TBs of the CU. That is, the delta QP is signaled once for the first quantization group of the CU by the entropy encoder 838, and not for any subsequent quantization groups of the CU. Non-uniform scaling may also be achieved by applying a "quantization matrix", whereby the scaling factor applied for each residual coefficient is derived from a combination of the quantization parameter 892 and the corresponding entry in the scaling matrix. The scaling matrix may have a size less than the size of the TB, and when applied to a TB, a nearest neighbor method is used to provide scaling values ​​for each residual coefficient from a scaling matrix of a size less than the size of the TB. The residual coefficients 836 are supplied to the entropy encoder 838 for encoding in the bitstream 121. Typically, the residual coefficients of each TB of a TU having at least one valid residual coefficient are scanned according to a scan pattern to produce an ordered list of values. The scan pattern typically scans the TB as a sequence of 4×4 "sub-blocks", thereby providing a conventional scanning operation with a granularity of 4×4 sets of residual coefficients, where the arrangement of the sub-blocks depends on the size of the TB. The scanning within each sub-block and the progression from one sub-block to the next typically follows a reverse diagonal scan pattern. In addition, the quantization parameter 892 is encoded into the bitstream 121 using an incremental QP syntax element, and the slice QP and secondary transform index 888 of the initial value in a given slice or sub-picture are encoded into the bitstream 121.

[0140] As described above, the video encoder 120 needs to access a frame representation that corresponds to the decoded frame representation seen in the video decoder. Therefore, the residual coefficients 836 are passed through an inverse secondary transform module 844, which operates according to the secondary transform index 888 to produce intermediate inverse transform coefficients represented by arrows 842. The intermediate inverse transform coefficients 842 are inversely quantized by a dequantizer module 840 according to a quantization parameter 892 to produce inverse transform coefficients represented by arrows 846. The dequantizer module 840 can also use a scaling list to perform inverse non-uniform scaling of the residual coefficients, which corresponds to the forward scaling performed in the quantizer module 834. The inverse transform coefficients 846 are passed to an inverse main transform module 848 to produce residual samples for the TU (represented by arrows 850). The inverse main transform module 848 applies a DCT-2 transform horizontally and vertically, which is constrained by the maximum available transform size described with reference to the forward main transform module 826. The type of inverse transform performed by inverse secondary transform module 844 corresponds to the type of forward transform performed by forward secondary transform module 830. The type of inverse transform performed by inverse main transform module 848 corresponds to the type of main transform performed by main transform module 826. Summation module 852 adds residual samples 850 and PU 820 to produce reconstructed samples of the CU (indicated by arrow 854).

[0141] The reconstructed samples 854 are passed to the reference sample cache 856 and the in-loop filter module 868. The reference sample cache 856, which is typically implemented using static RAM on an ASIC to avoid expensive off-chip memory accesses, provides the minimum sample storage required to satisfy the dependencies for generating intra PBs for subsequent CUs in a frame. The minimum dependencies typically include a "line buffer" of samples along the bottom of a row of CTUs for use by the next row of CTUs and a column buffer (whose range is set by the height of the CTU). The reference sample cache 856 supplies reference samples (represented by arrow 858) to the reference sample filter 860. The sample filter 860 applies a smoothing operation to produce filtered reference samples (indicated by arrow 862). The filtered reference samples 862 are used by the intra prediction module 864 to produce an intra prediction block of samples represented by arrow 866. For each candidate intra prediction mode, the intra prediction module 864 generates a sample block (i.e., 866). The sample block 866 is generated by the module 864 using techniques such as DC, planar, or angular intra prediction. A matrix multiplication method may also be used to generate sample block 866, with adjacent reference samples as input and a matrix selected by video encoder 120 from a set of matrices, with the selected matrix signaled in bitstream 121 using an index to identify which matrix in the set of matrices will be used by video decoder 144.

[0142] The in-loop filter module 868 applies several filtering stages to the reconstructed samples 854. The filtering stages include a "deblocking filter" (DBF), which applies smoothing aligned with CU boundaries to reduce artifacts caused by discontinuities. Another filtering stage present in the in-loop filter module 868 is an "adaptive loop filter" (ALF), which applies a Wiener-based adaptive filter to further reduce distortion. Another available filtering stage in the in-loop filter module 868 is a "sample adaptive offset" (SAO) filter. The SAO filter operates by first classifying the reconstructed samples into one or more categories and applying an offset at the sample level according to the assigned category.

[0143] Filtered samples, represented by arrow 870, are output from the in-loop filter module 868. The filtered samples 870 are stored in a frame buffer 872. The frame buffer 872 typically has the capacity to store several (e.g., up to sixteen (16)) pictures and is therefore stored in the memory 206. Due to the large memory consumption required, on-chip memory is typically not used to store the frame buffer 872. As such, access to the frame buffer 872 is expensive in terms of memory bandwidth. The frame buffer 872 provides a reference frame (represented by arrow 874) to the motion estimation module 876 and the motion compensation module 880.

[0144] The motion estimation module 876 estimates a plurality of "motion vectors" (denoted as 878), each of which is a Cartesian spatial offset relative to the position of the current CB, thereby referencing a block in one of the reference frames in the frame buffer 872. A filtered block of reference samples (denoted as 882) is generated for each motion vector. The filtered reference samples 882 form a further candidate mode for potential selection by the mode selector 886. In addition, for a given CU, the PB 820 may be formed using one reference block ("uni-prediction"), or may be formed using two reference blocks ("bi-prediction"). For the selected motion vector, the motion compensation module 880 generates the PB 820 according to a filtering process that supports sub-pixel precision in the motion vector. In this way, the motion estimation module 876 (which operates on many candidate motion vectors) can perform a simplified filtering process compared to the motion compensation module 880 (which operates only on the selected candidate) to achieve reduced computational complexity. When the video encoder 120 selects inter-frame prediction for the CU, the motion vector 878 is encoded into the bitstream 121.

[0145] Although the reference to Versatile Video Coding (VVC) describes Figure 8810 to 890, but other video coding standards or implementations may also employ the processing stages of modules 810 to 890. Frame data 119 (and bitstream 121) may also be read from (or written to) memory 206, hard drive 210, CD-ROM, Blu-ray disc, or other computer-readable storage medium. In addition, frame data 119 (and bitstream 121) may be received from (or sent to) an external source (such as a server or radio frequency receiver connected to communication network 220). Communication network 220 may provide limited bandwidth, thereby requiring the use of rate control in video encoder 120 to avoid saturating the network when frame data 119 is difficult to compress.

[0146] In addition, the bitstream 121 may be constructed from one or more slices representing a spatial segment (a set of CTUs) of the frame data 119, generated by one or more instances of the video encoder 120, operating in a coordinated manner under the control of the processor 205. The bitstream 121 may also include a slice corresponding to a sub-picture, output as a set of sub-pictures forming a picture, each sub-picture being independently coded and independently decodable relative to any other slice or sub-picture in the picture.

[0147] exist Fig. 9 The video decoder 144 (also called a feature map decoder) is shown in FIG. Fig. 9 The video decoder 144 of is an example of a Versatile Video Coding (VVC) video decoding pipeline, but other video codecs may also be used to perform the processing stages described herein. Fig. 9 As shown, the bitstream 143 is input to the video decoder 144. The bitstream 143 can be read from the memory 206, the hard disk drive 210, the CD-ROM, the Blu-ray disc or other non-transitory computer-readable storage medium. Alternatively, the bitstream 143 can be received from an external source (such as a server or a radio frequency receiver connected to the communication network 220). The bitstream 143 contains the encoding syntax elements representing the captured frame data to be decoded.

[0148] The bitstream 143 is input to the entropy decoder module 920. The entropy decoder module 920 extracts syntax elements from the bitstream 143 by decoding the "bin" sequence and passes the values ​​of the syntax elements to other modules in the video decoder 144. The entropy decoder module 920 uses variable length and fixed length decoding to decode SPS, PPS or slice headers, and uses an arithmetic decoding engine to decode the syntax elements of the slice data into a sequence of one or more bins. Each bin can use one or more "contexts", where the context describes the probability level of the "one" and "zero" values ​​to be used to encode the bin. In the case where multiple contexts are available for a given bin, a "context modeling" or "context selection" step is performed to select one of the available contexts to decode the bin. The process of decoding the bin forms a sequential feedback loop, so each slice can be decoded in the entirety of the slice by a given entropy decoder 920 instance. A single (or a few) high performance entropy decoder 920 instances may decode all slices or sub-pictures of a frame or picture from the bitstream 143 , and multiple lower performance entropy decoder 920 instances may concurrently decode slices of a frame from the bitstream 143 .

[0149] The entropy decoder module 920 applies an arithmetic coding algorithm, such as "context adaptive binary arithmetic coding" (CABAC), to decode syntax elements from the bitstream 143. The decoded syntax elements are used to reconstruct parameters within the video decoder 144. The parameters include residual coefficients (represented by arrow 924), quantization parameters 974, secondary transform indexes 970, and mode selection information (represented by arrow 958) such as intra-frame prediction modes. The mode selection information also includes information such as motion vectors and partitioning of each CTU into one or more than one CB. The parameters are used to generate PBs, usually in combination with sample data from previously decoded CBs.

[0150] The residual coefficients 924 are passed to the inverse secondary transform module 936, where the secondary transform is applied or not performed (bypassed) according to the secondary transform index. The inverse secondary transform module 936 generates reconstructed transform coefficients 932, i.e., main transform domain coefficients, from the secondary transform domain coefficients. The reconstructed transform coefficients 932 are input to the dequantizer module 928. The dequantizer module 928 inverse quantizes (or "scales") the residual coefficients 932 (i.e., in the main transform coefficient domain) to create a reconstructed intermediate transform coefficient represented by arrow 940 according to the quantization parameter 974. The dequantizer module 928 can also apply a scaling matrix to provide non-uniform dequantization within the TB, which corresponds to the operation of the dequantizer module 840. If the use of a non-uniform inverse quantization matrix is ​​indicated in the bitstream 143, the video decoder 144 reads the quantization matrix from the bitstream 143 as a sequence of scaling factors and arranges the scaling factors into a matrix. Inverse scaling uses the quantization matrix in combination with the quantization parameter to create the reconstructed intermediate transform coefficients 940.

[0151] The reconstructed transform coefficients 940 are passed to an inverse main transform module 944. Module 944 transforms the coefficients 940 from the frequency domain back to the spatial domain. The inverse main transform module 944 applies an inverse DCT-2 transform horizontally and vertically, subject to the constraints of the maximum available transform size as described with reference to the forward main transform module 726. The result of the operation of module 944 is a block of residual samples represented by arrow 948. The residual sample block 948 is equal in size to the corresponding CB. The residual samples 948 are supplied to a summation module 950.

[0152] At summation module 950, residual samples 948 are added to the decoded PB, indicated as 952, to produce a block of reconstructed samples indicated by arrow 956. Reconstructed samples 956 are supplied to a reconstructed sample cache 960 and an in-loop filtering module 988. The in-loop filtering module 988 produces a reconstructed block of frame samples indicated as 992. Frame samples 992 are written to a frame buffer 996.

[0153] The reconstructed sample cache 960 operates in a manner similar to the reconstructed sample cache 856 of the video encoder 120. The reconstructed sample cache 960 provides storage for the reconstructed samples required for intra prediction of subsequent CBs without the memory 206 (e.g., by using data 232, which is typically on-chip memory, instead). Reference samples, represented by arrows 964, are obtained from the reconstructed sample cache 960 and are supplied to a reference sample filter 968 to produce filtered reference samples, represented by arrows 972. The filtered reference samples 972 are supplied to an intra prediction module 976. The module 976 produces a block of intra prediction samples, represented by arrows 980, based on the intra prediction mode parameters 958 signaled in the bitstream 143 and decoded by the entropy decoder 920. The intra prediction module 976 supports the modes of module 764, including IBC and MIP. The sample block 980 is generated using a mode such as DC, planar or angular intra prediction.

[0154] When the prediction mode of the CB is indicated in the bitstream 143 to use intra prediction, the intra prediction samples 980 form the decoded PB 952 via the multiplexer module 984. Intra prediction produces a prediction block (PB) of samples, which is a block in a color component that is derived using "neighboring samples" in the same color component. Neighboring samples are samples that are adjacent to the current block and have been reconstructed because they are at the front in the block decoding order. In the case where the luma block and the chroma block are juxtaposed, the luma block and the chroma block can use different intra prediction modes. However, the two chroma CBs share the same intra prediction mode.

[0155] When the prediction mode of the CB is indicated as inter-prediction in the bitstream 143, the motion compensation module 934 generates a block of inter-prediction samples, denoted as 938. The inter-prediction sample block 938 is generated by selecting and filtering the sample block 998 from the frame buffer 996 using the motion vector and reference frame index decoded from the bitstream 143 by the entropy decoder 920. The sample block 998 is obtained from a previously decoded frame stored in the frame buffer 996. For bi-prediction, two sample blocks are generated and mixed together to generate samples for decoding the PB 952. The frame buffer 996 is filled with filter block data 992 from the in-loop filtering module 988. Like the in-loop filtering module 868 of the video encoder 120, the in-loop filtering module 988 applies any of the DBF, ALF and SAO filtering operations. Generally, the motion vector is applied to both the luma and chroma channels, although the filtering process for sub-sample interpolation in the luma and chroma channels is different. Frames from frame buffer 996 are output as decoded frames 145 .

[0156] Figure 8 and Fig. 9 Not shown is a module for pre-processing the video prior to encoding and post-processing the video after decoding to shift sample values ​​to achieve more uniform use of the range of sample values ​​within each chroma channel. A multi-segment linear model is derived in the video encoder 120 and signaled in the bitstream for use by the video decoder 144 to undo the sample shifting. The linear model chroma scaling (LMCS) tool provides compression benefits for specific color spaces and content that have some non-uniformity in their use of sample space, especially limited range utilization, which may result in higher quality loss from the application of quantization.

[0157] Fig. 10A is a schematic block diagram illustrating a cross-layer tensor inverse bottleneck 148 for restoring tensor dimensions after compression. Fig.11 A method 1100 is shown for decoding a bitstream, reconstructing a decorrelated feature map, and performing the second part of a CNN.

[0158] The method 1100 may be implemented using a device such as a configured FPGA, ASIC, or ASSP. Alternatively, as described below, the method 1100 may be implemented by the destination device 140 as one or more software code modules of the application 233 under execution of the processor 205. The software code modules of the application 233 implementing the method 1100 may reside, for example, in the hard drive 210 and / or the memory 206. The method 1100 is repeated for each compressed data frame in the bitstream 143. The method 1100 may be stored on a computer-readable storage medium and / or in the memory 206. The method 1100 begins with a decode bitstream step 1110.

[0159] In step 1110, the video decoder 144, under the execution of the processor 205, as shown in FIG. Fig. 9 As described above, a frame 145 is decoded from the bitstream 143. The frame 145 contains the Figure 7 The control in the processor 205 enters the extract first combined tensor step 1120 from step 1110.

[0160] At step 1120, the unpacking and inverse quantization module 146 extracts a feature map for the first combined tensor (e.g., 557) from the decoded frame 145. At step 1120, the extracted feature map is inverse quantized from the integer domain to the floating point domain. The inverse quantized feature map generated by the operation of step 1120 is Fig. 10A 1011. From step 1120, control in processor 205 proceeds to extract second combined tensor step 1130.

[0161] At step 1130, the unpacking and inverse quantization module 146 extracts a feature map for the second combined tensor (e.g., 537) from the frame 145. At step 1130, the extracted feature map is inverse quantized from the integer domain to the floating point domain. The inverse quantized feature map generated by the operation of step 1130 is Fig. 10A is shown as tensor 1021. Fig. 10A As shown, the inverse quantized features 1011 and 1021 correspond to the decoding bottleneck (compressed) tensor 147. Fig.11 As shown, control in the processor 205 enters the SSFC decoding first combined tensor step 1140 from step 1130 .

[0162] Steps 1110, 1120 and 1130 operate to decode a frame of the bitstream to obtain two information units of the frame. Tensor 1011 provides a first information unit and tensor 1021 provides a second information unit. Each information unit corresponds to a feature map of a frame encoded in the bitstream.

[0163] At step 1140, the SSFC decoder 1010 is implemented under execution of the processor 205. The SSFC decoder performs a neural network layer to decompress the first decoded compressed tensor 1011 to produce a first decoded combined tensor 1017. The convolution layer 1012 receives the tensor 1011 with C'=64 channels and outputs a tensor 1013 with F=256 channels. The tensor 1013 is passed to the batch normalization layer 1014. The batch normalization layer outputs a tensor 1015. The tensor 1015 is passed to the parameterized leaky rectified linear (PReLU) layer 1016. The PReLU layer 1016 outputs a tensor 1017. Return to Fig.11 , control in the processor 205 enters the SSFC decoding second combined tensor step 1150 from step 1140.

[0164] At step 1150, the SSFC decoder 1020 is implemented under execution of the processor 205. The SSFC neural network layer decompresses the second decoded compressed tensor 1021 to produce a second decoded combined tensor 1027. The convolution layer 1022 receives the tensor 1021 with C'=64 channels and outputs a tensor 1023 with F=256 channels. The tensor 1023 is passed to the batch normalization layer 1024. The batch normalization layer 1024 outputs a tensor 1025. The tensor 1025 is passed to the PReLU layer 1026. The PReLU layer 1026 outputs a tensor 1027. Return to Fig.11 , control in processor 205 enters from step 1150 to step 1160 of reconstructing the first tensor.

[0165] At step 1160, as described with respect to Fig. 10AAs described, the multi-scale feature reconstruction (MSFR) module 1030 generates a first tensor under the execution of the processor 205. Step 1160 receives the tensor 1017 generated by the operation of step 1140. The tensor 1017 is passed to the MSFR module 1030, which generates decoded tensors 1055 and 1033. The MSFR module 1030 uses the upsampling module 1032, the downsampling module 1034, the convolution layer 1036 and the summation module 1038 when implementing step 1160. The upsampling module 1032 receives the tensor 1017 and interpolates to generate the tensor 1033. The width and height of the tensor 1033 are twice the width and height of the tensor 1017. The tensor 1033 is output as P'4 and is passed to the downsampling module 1034. The downsampling module 1034 downsamples the tensor 1033 to produce a tensor 1035 having the same dimension as the tensor 1017. The tensor 1035 is provided to the convolution layer 1036, which outputs the tensor 1037. The summation module 1038 adds the tensors 1037 and 1017 to produce a tensor 1055. The tensor 1055 is output as P'5. The tensor P'5 is the smallest tensor in the set of tensors P'4 and P'5, which is generated, for example, by adding the first information unit (1011) to another tensor 1033 produced by the convolution operation at block 1036. Return to reference Fig.11 , control in processor 205 enters from step 1160 to step 1170 of reconstructing the second tensor.

[0166] At step 1170, the MSFR module 1030 generates a second tensor from the tensor 1027 under execution of the processor 205. The MSFR module 1030 that generates the decoded tensors 1053 and 1043 uses the upsampling module 1042, the downsampling module 1044, the convolution layer 1046, and the summation module 1048 when implementing step 1170. The tensor 1027 is passed to the upsampling module 1042. The upsampling module 1042 interpolates to generate a tensor 1043 having twice the width and height of the tensor 1027. The tensor 1043 is output as P'2 and passed to the downsampling module 1044. The downsampling module 1044 downsamples the tensor 1043 to generate a tensor 1045 having the same dimensions as the tensor 1027. The tensor 1045 is provided to the convolution layer 1046, which outputs the tensor 1047. The summation module 1048 adds tensors 1047 and 1027 to produce tensor 1053, which is output as P'3. Tensor P'3 is the smallest tensor in the set of tensors P'2 and P'3, which is generated, for example, by adding the first information unit (1021) to another tensor 1053 generated by the convolution operation at box 1046. Tensors 1043, 1053, 1033, 1055 (i.e., decoded tensors P'2-P'5 corresponding to tensors P2 to P5) form tensor 149. Reference Fig.11 , control in processor 205 proceeds from step 1170 to step 1180 of performing the second part of the neural network. Fig. 10A As shown, the respective tensors in each set (1055, 1033) and (1053, 1043) of the plurality of tensors have resolutions that form an exponential sequence that doubles in width and height between consecutive tensors. For example, the width and height of 1033 are twice the width and height of 1055. The maximum tensor in each set of tensors (1033 and 1043) is determined based on the upsampling operation (1032 or 1042, respectively) applied to the feature map of the information unit.

[0167] At step 1180, CNN head 150, under execution by processor 205, takes tensor 149 as input and performs the remainder of the neural network implemented by system 100 on tensor 149. Method 1100 terminates when step 1180 is implemented, having processed tensors associated with one frame of video data. Method 1100 is re-invoked for each frame of video data encoded in bitstream 143.

[0168] Fig. 12A1 is a schematic block diagram showing a head 150 of a CNN for object detection. Depending on the task to be performed in the destination device 140, the CNN head 150 may be replaced with a different network. The input tensor 149 is separated into tensors for each layer (i.e., tensors 1210, 1220, and 1234). Tensor 1210 is passed to a CBL module 1212 to produce a tensor 1214. Tensor 1214 is passed to a detection module 1216 and an upscaling module 1222. The detection module 1216 operates to detect a bounding box 1218. The bounding box 1218 is in the form of a detection tensor. The bounding box 1218 is passed to a non-maximum suppression (NMS) module 1248. The NMS module 1248 selects one of the multiple inputs generated by the detection module to produce a detection result 151. In order to generate a bounding box addressing a coordinate in the original video data 113, scaling by the original video width and height is performed before resizing the trunk of the network 114. The upscaling module 1222 produces an upscaled tensor 1224 that is scaled to the original video width and height. The upscaled tensor 1224 is passed to the CBL module 1226. The CBL module 1226 produces a tensor 1228 as an output. The tensor 1228 is passed to the detection module 1230 and the upscaling module 1236. The detection module 1230 operates in a similar manner to the detection module 1216 and produces a detection tensor 1232. The detection tensor 1232 is fed to the NMS module 1248.

[0169] The upgrader module 1236 operates in the same manner as the module 1260 and outputs an upgraded tensor 1238. The upgraded tensor 1238 is passed to the CBL module 1240. The CBL module 1240 operates in the same manner as the modules 1212 and 1226 to output a tensor 1242 to the detection module 1244. The detection module 1244 operates in a similar manner as the detection module 1216 and produces a detection tensor 1246. The detection tensor 1246 is fed to the NMS module 1248.

[0170] CBL modules 1212, 1226, and 1240 each comprise a cascade of five CBL modules. Upgrader modules 1222 and 1236 are Fig. 12B Various instances of the upgrader module 1260 are shown.

[0171] Upscaling module 1260 accepts tensor 1262 and tensor 1264 as input. Tensor 1262 is passed to CBL module 1266 to produce tensor 1268. Using nearest neighbor interpolation or other various methods, tensor 1268 is passed to upsampler 1270 to produce upsampled tensor 1272. Concatenation module 1274 produces tensor 1276 by concatenating upsampled tensor 1272 with input tensor 1264.

[0172] Detection modules 1216, 1230, and 1244 are as follows Fig. 12C 12 is an example of a detection module 1280 shown. The detection module 1260 receives tensor 1282, which is passed to a CBL module 1284 to produce tensor 1286. Tensor 1286 is passed to a convolution module 1288, which implements a detection kernel. The detection kernel applies a 1×1 kernel to produce outputs on feature maps at three layers. The detection kernel is 1×1×(B×(5+C)), where B is the number of bounding boxes that a particular unit can predict, typically three (3), and C is the number of classes, which can be eighty (80), making the kernel size two hundred and fifty-five (255) detection attributes. Module 1288 outputs tensor 1290. The constant "5" represents four bounding box attributes (box center x, y and size scale x, y) and an object confidence level ("objectness"). The result of the detection kernel has the same spatial dimensions as the input feature map, but the depth of the output corresponds to the detection attributes. The detection kernel is applied at each layer, usually three layers, which results in a large number of candidate bounding boxes. The NMS module 1048 applies non-maximum suppression to the obtained bounding boxes to discard redundant boxes, such as overlapping predictions of similar scales, etc., thereby obtaining the final set of bounding boxes as the output of object detection.

[0173] Fig.13 1 is a schematic block diagram showing an alternative head 1300 of a CNN. The head 1300 forms part of an overall network known as “FasterRCNN” and includes a feature network (i.e., backbone portion 400), a region proposal network, and a detection network. The input to the head 1300 is a tensor 149. The tensor 149 includes P2 to P6 layer tensors 1310, 1312, 1314, 1316, and 1318. The P2 to P6 layer tensors 1310, 1312, 1314, 1316, and 1318 are input to a region proposal network (RPN) head module 1320. The RPN head module 1320 convolves the input tensor, thereby producing an intermediate tensor. The intermediate tensor is fed into two subsequent sibling layers in the module 1320, one for classification, and one for bounding box or “region of interest” (ROI) regression, as classification and bounding box 1322. The classification and bounding box 1322 are passed to the NMS module 1324. The NMS module prunes redundant bounding boxes by removing overlapping boxes with lower scores to produce pruned bounding boxes 1326. The bounding box 1326 is passed to the region of interest (ROI) pooler 1328. The ROI pooler 1328 also receives tensors P2 to P6 and uses a maximum pooling operation to produce a fixed-size feature map from various input size maps. In the operation performed by 1328, the subsampling takes the maximum value in each group of input values ​​to produce an output value in the output tensor.

[0174] In the arrangement of CNN backbone 400 and CNN head 1300, "P6" layer tensor 429 is omitted from output tensor 115, and in CNN head 1300, P6 input tensor 1318 is generated by performing a "Maxpool" operation with stride equal to 2 on P5 tensor 1316. Since the P6 layer can be reconstructed from the P5 layer, there is no need to separately encode and decode the P6 layer as an explicit FPN layer in the first set of tensors or the second set of tensors.

[0175] The input to the ROI pooler 1328 is the P2 to P5 feature maps 1310, 1312, 1314, and 1316 (corresponding to Fig. 10A 1043, 1053, 1033, and 1055) and region of interest proposals 1326. Each proposal (ROI) from 1326 is associated with a portion of the feature map (1310-1316) to produce a fixed-size map. The size of the fixed-size map is independent of the underlying portion of the feature maps 1310-1316. For example, one of the feature maps 1310-1316 is selected so that the resulting cropped map has sufficient detail according to the following rule: floor(4+log2(sqrt(box_area) / 224)), where 224 is a typical box size. The ROI pooler 1328 thus crops the incoming feature map according to the proposal 1326, producing a tensor 1330. The tensor 1330 is fed to a fully connected (FC) neural network head 1332. The FC head 1332 performs two fully connected layers to produce a class score and bounding box prediction sub-delta tensor 1334. The class score is typically a tensor of 80 elements, each element corresponding to the predicted score of the corresponding object class. The bounding box prediction delta tensor is a tensor of 80×4=320 elements, containing the bounding boxes of the corresponding object class. Final processing is performed by the output layer module 1336, which receives the tensor 1334 and performs a filtering operation to produce a filtered tensor 1338. Low scoring (low classification) objects are no longer considered further. The non-maximum suppression module 1340 receives the filtered tensor 1334 and removes overlapping bounding boxes by removing overlapping boxes with lower classification scores, thereby obtaining the inferred output tensor 151.

[0176] Fig.14 1 is a schematic block diagram illustrating a top-down multi-scale feature reconstruction (MSFR) stage 1400 for a destination device 140. The top-down MSFR stage 1400 is Fig. 10AAn alternative implementation of MSFR 1030 of . Top-down feature reconstruction involves upsampling from a lower resolution layer (at the "top" of the FPN) to generate a version of a higher resolution layer. In the case of multiple spatial representations of the layers of the FPN, a top-down operation is performed to reconstruct each tensor within the given corresponding spatial representations. The first combined tensor 1017 and the second combined tensor 1027 are received by the top-down MSFR 1400 and output directly as tensors P'5 1055a and P'3 1053a, respectively. Tensor 1017 is passed to the upsampling module 1410. Module 1410 performs 2X upsampling using an interpolation filter to produce tensor 1411. Tensor 1017 is also passed to the upsampling module 1412. Module 1412 performs nearest neighbor upsampling to produce tensor 1413 having twice the width and height of tensor 1017. Tensor 1413 is passed to convolutional layer 1414 to produce tensor 1415 having the same dimensions as tensor 1413. Summation module 1416 sums tensors 1415 and 1411 to produce tensor 1417. Tensor 1417 is passed to convolutional stage 1418 to produce tensor P'4 1033a. Tensor 1033a has the same dimensions as tensor 1417.

[0177] Tensor 1027 is passed to upsampling module 1420, which performs 2X upsampling using an interpolation filter to produce tensor 1421. Tensor 1027 is also passed to upsampling module 1422. Module 1422 performs nearest neighbor upsampling to produce tensor 1423 having twice the width and height of tensor 1027. Tensor 1423 is passed to convolutional layer 1424 to produce tensor 1425 having the same dimensions as tensor 1423. Summation module 1426 sums tensors 1425 and 1421 to produce tensor 1427. Tensor 1427 is passed to convolution stage 1428 to produce tensor P'2 1043a having the same dimensions as tensor 1427.

[0178] With the top-down MSFR module 1400 in use, the output tensors 1055a, 1033a, 1053a, and 1043a are fed to the rest of the network as tensors 1055, 1033, 1053, and 1043, respectively. Although different methods of reconstructing the FPN layers are possible, in each case, the layers of the FPN may be grouped into multiple (e.g., two) layer sets, where each layer set is fused together separately in the bottleneck encoder 116 and extracted in the bottleneck decoder 148.

[0179] The bottleneck decoder 148, whether implementing the MSFR decoder as the decoder 1030 or the decoder 1400, operates to generate a first set of tensors and a second set of tensors, respectively, using the first information unit and the second information unit decoded by the operation of steps 1110 to 11130. Each set of tensors may be determined independently so that steps 1120 and 1130 may be performed independently, concurrently, or in a different order if desired. The first set of tensors or multiple tensors is composed of tensors P'4 and P'5 ( Fig. 10A 1055 and 1033 or Fig.14 The second set of tensors or multiple tensors is provided by tensors P'2 and P'3 ( Fig. 10A 1053 and 1043 or Fig.14 1053a and 1043a in FIG. ... Figure 4 The feature pyramid network generated by the CNN backbone 400 operations or by Figure 3A A collection of tensors generated by the 300 operations of the CNN backbone.

[0180] Regardless of the implementation used by the MSFR decoder, the described arrangement allows the use of neural network layers to determine each set of the first plurality of tensors and the second plurality of tensors. These layers are typically convolutional layers, batch normalization layers, and PReLu layers. As long as decoding from a single packed frame such as frame 700 can be achieved, other layers can be used for SSFC decoders 1010 and 1020.

[0181] The first set of tensors and the second set of tensors may be provided to the CNN head 150 to implement tasks such as object detection or frame segmentation. Fig.13 As shown, P'5 to P'2 are processed.

[0182] Alternatively, Fig. 12A The CNN 150 may be equipped with tensors 1210, 1220, and 1234 from the bottleneck decoder 148. Fig. 12A In the implementation of the CNN head 150, the bottleneck decoder 148 operates to output the tensor 1017 from the SSFC decoder 1010, thereby processing the first group. The tensor 1017 is convolved with the 256-channel input and 1024-channel output to provide the tensor 1210 to the CNN head, where modules 1032, 1034, 1036 and 1038 are omitted. Fig. 10B Shown for use in Fig. 12A When implementing CNN in a similar way to the example Fig. 10AThe convolution arrangement 1090 is used in conjunction with the bottleneck decoder 148 of FIG. The arrangement 1090 includes convolutional layers 1060 , 1070 , and 1080 . Fig. 10A The tensor 1017 of is input to the convolution layer 1060, which outputs the tensor 1210 with 1024 channels. For the second group, tensors 1053 and 1043 each with 256 channels are output from the bottleneck decoder 148. The tensor 1053 is input to the convolution layer 1070, which generates the tensor 1220 with 512 channels. The tensor 1043 is input to the convolution layer 1080, which generates the tensor 1234 with 256 channels. Figure 3A , tensor 329 has 1024 channels, which are reduced to 64 channels by SSFC encoder 550. Tensors 326 and 322 have 512 channels and 256 channels, respectively, which are first reduced to 256 channels in MSFF module 510 and further reduced to 64 channels in SSFC encoder 530.

[0183] In About Figure 5 and Fig. 10A In the example implementation described, the first tensor and the second tensor (e.g., 537 and 557) have different sizes but have the same number of channels. In another arrangement of the bottleneck encoder 116 and the bottleneck decoder 148, the number of channels of the second compressed tensors 557, 1011 is different from the number of channels of the first compressed tensors 537, 1021. For example, the second compressed tensor can have a lower or smaller number of channels than the first compressed tensor. Preferably, the second compressed tensor for the higher spatial resolution FPN layer (P'2 and P'3) has a smaller number of channels than the first compressed tensor for the lower spatial resolution FPN layer (P'4 and P'5). The number of channels of the second compressed tensor 537 can be 32 or 48 channels. The second compressed tensor 537 has four times the width and height of the first compressed tensor 557, and the reduction in channel count results in a smaller area required for the frame 700. The second compressed tensor 537 represents less decomposed features of the frame 113 and shows that fewer channels are sufficient to achieve high task performance because most of the semantically meaningful information is contained in the first compressed tensor 557.

[0184] In an arrangement of a source device 110 and a destination device 140 with three layers of FPN, such as in the case of using a "YOLOv3" network, the tensors can be grouped into two sets, the first group having two FPN layers and the second group having one FPN layer. The group with one FPN layer requires bottleneck layers associated with the SSFC encoder (530 or 550) and the SSFC decoder (1010 or 1020), but does not require layers associated with the MSFF 510 or MSFR 1030. In the case where the channel counts are different between the layers of the FPN, the cascade (e.g., 514 or 522) in the MSFF module 510 cascades such tensors. Additional convolutional layers are inserted in the MSFR (e.g., 1030 or 1400) to recover the channel counts for each corresponding FPN layer.

[0185] Industrial Applicability

[0186] The described arrangement is suitable for use in the computer and data processing industry, and in particular in digital signal processing for encoding and decoding signals such as video and image signals, thereby achieving high compression efficiency.

[0187] These arrangements describe splitting the use of MFSC into two operations, thereby distinguishing how a hierarchical structure (such as a feature pyramid network, etc.) of a frame of data is compressed before encoding (and decompressed accordingly after decoding). When the MFSC technique is applied alone, a certain degree of cross-layer fusion can be achieved without the severe degradation of spatial detail caused by using a single MFSC operation. For tasks such as instance segmentation that require the preservation of greater spatial detail, the resulting mAP is higher when using a separate MSFC technique than when all FPN (hierarchical) layers are merged into a single tensor. In addition, different numbers of channels can be used in different instances of MFSC, thereby limiting the area increase when necessary. MFSC is traditionally used to "fuse" all data in FPN or shrink all data in FPN. In contrast to fusing all data in FPN or shrinking all data in FPN, using more than one instance of MFSC with a single hierarchical structure or FPN can achieve MFSC while limiting the adverse impact of mAP.

[0188] The foregoing describes only some embodiments of the present invention, and modifications and / or changes may be made thereto without departing from the scope and spirit of the present invention, the embodiments being illustrative rather than restrictive.

[0189] In the context of this specification, the word "comprising" means "mainly but not necessarily only" or "having" or "including", rather than "consisting only of..." Variations of the word "comprising" (such as "comprise" and "comprises", etc.) have correspondingly varied meanings.

Claims

1. A method for decoding at least a plurality of tensors from a bitstream, the plurality of tensors forming a hierarchical representation of a feature map for a single frame, the method comprising: decoding a first information unit from the bit stream; decoding a second information unit from the bit stream; determining a first plurality of tensors based on the first information unit, a feature map of at least one tensor in the first plurality of tensors having a different spatial resolution than a feature map of one or more other tensors in the first plurality of tensors; as well as determining a second plurality of tensors based on the second information unit, a feature map of at least one tensor in the second plurality of tensors having a different spatial resolution than a feature map of one or more other tensors in the second plurality of tensors, wherein the feature maps of each tensor in the first plurality of tensors have a different spatial resolution than the feature maps of each tensor in the second plurality of tensors, and the tensors in the first plurality of tensors and the second plurality of tensors correspond to a hierarchical representation of the feature maps for the single frame.

2. The method according to claim 1, wherein: Corresponding tensors of the first plurality of tensors and the second plurality of tensors have resolutions that form a sequence of indices that double in width and height between consecutive tensors.

3. The method according to claim 1, wherein: The first plurality of tensors and the second plurality of tensors have different numbers of channels.

4. The method according to claim 1, wherein: The plurality of tensors with higher spatial resolution in the first plurality of tensors and the second plurality of tensors have a smaller number of channels than the other plurality of tensors.

5. The method according to claim 1, wherein: A maximum tensor in each of the first plurality of tensors and the second plurality of tensors is determined based on an upsampling operation applied to a feature map of a corresponding information unit in the first information unit and the second information unit.

6. The method according to claim 1, wherein: The determination of the first plurality of tensors and the determination of the second plurality of tensors are independent of each other.

7. The method according to claim 1, wherein: The first plurality of tensors and the second plurality of tensors are determined using a neural network layer.

8. The method according to claim 1, wherein: The first information unit is used to determine a minimum tensor among the first plurality of tensors.

9. The method according to claim 1, wherein: The second information unit is used to determine a minimum tensor among the second plurality of tensors.

10. A method of encoding at least a plurality of tensors into a bitstream, the plurality of tensors forming a hierarchical representation of a feature map for a single frame, the method comprising: using a convolution operation to determine a first unit of information from a first plurality of tensors, a feature map of at least one tensor of the first plurality of tensors having a different spatial resolution than a feature map of one or more other tensors of the first plurality of tensors; using a convolution operation to determine a second unit of information from a second plurality of tensors, wherein a feature map of at least one tensor in the second plurality of tensors has a different spatial resolution than a feature map of one or more other tensors in the second plurality of tensors, and wherein the feature map of each tensor in the first plurality of tensors has a different spatial resolution than the feature map of each tensor in the second plurality of tensors, and wherein tensors in the first plurality of tensors and the second plurality of tensors correspond to a hierarchical representation of feature maps for the single frame; encoding the first information unit into the bitstream; and The second information unit is encoded into the bitstream.

11. A decoder for decoding at least a plurality of tensors from a bitstream, the plurality of tensors forming a hierarchical representation of a feature map for a single frame, the decoder being configured to: decoding a first information unit from the bit stream; decoding a second information unit from the bit stream; determining a first plurality of tensors based on the first information unit, a feature map of at least one tensor in the first plurality of tensors having a different spatial resolution than a feature map of one or more other tensors in the first plurality of tensors; as well as determining a second plurality of tensors based on the second information unit, a feature map of at least one tensor in the second plurality of tensors having a different spatial resolution than a feature map of one or more other tensors in the second plurality of tensors, wherein the feature maps of each tensor in the first plurality of tensors have a different spatial resolution than the feature maps of each tensor in the second plurality of tensors, and the tensors in the first plurality of tensors and the second plurality of tensors correspond to a hierarchical representation of the feature maps for the single frame.

12. An encoder for encoding at least a plurality of tensors into a bitstream, the plurality of tensors forming a hierarchical representation of a feature map for a single frame, the encoder being configured to: using a convolution operation to determine a first unit of information from a first plurality of tensors, a feature map of at least one tensor of the first plurality of tensors having a different spatial resolution than a feature map of one or more other tensors of the first plurality of tensors; using a convolution operation to determine a second unit of information from a second plurality of tensors, wherein a feature map of at least one tensor in the second plurality of tensors has a different spatial resolution than a feature map of one or more other tensors in the second plurality of tensors, and wherein the feature map of each tensor in the first plurality of tensors has a different spatial resolution than the feature map of each tensor in the second plurality of tensors, and wherein tensors in the first plurality of tensors and the second plurality of tensors correspond to a hierarchical representation of feature maps for the single frame; encoding the first information unit into the bitstream; and The second information unit is encoded into the bitstream.

13. A non-transitory computer-readable storage medium storing a program for performing a method of decoding at least a plurality of tensors from a bitstream, the plurality of tensors forming a hierarchical representation of a feature map for a single frame, the method comprising: decoding a first information unit from the bit stream; decoding a second information unit from the bit stream; determining a first plurality of tensors based on the first information unit, a feature map of at least one tensor in the first plurality of tensors having a different spatial resolution than a feature map of one or more other tensors in the first plurality of tensors; as well as determining a second plurality of tensors based on the second information unit, a feature map of at least one tensor in the second plurality of tensors having a different spatial resolution than a feature map of one or more other tensors in the second plurality of tensors, wherein the feature maps of each tensor in the first plurality of tensors have a different spatial resolution than the feature maps of each tensor in the second plurality of tensors, and the tensors in the first plurality of tensors and the second plurality of tensors correspond to a hierarchical representation of the feature maps for the single frame.

14. A system comprising: Memory; as well as A processor, wherein the processor is configured to execute code stored on the memory for implementing a method for decoding at least a plurality of tensors from a bitstream, the plurality of tensors forming a hierarchical representation of a feature map for a single frame, the method comprising: decoding a first information unit from the bit stream; decoding a second information unit from the bit stream; determining a first plurality of tensors based on the first information unit, a feature map of at least one tensor in the first plurality of tensors having a different spatial resolution than a feature map of one or more other tensors in the first plurality of tensors; and determining a second plurality of tensors based on the second information unit, a feature map of at least one tensor in the second plurality of tensors having a different spatial resolution than a feature map of one or more other tensors in the second plurality of tensors, wherein the feature maps of each tensor in the first plurality of tensors have a different spatial resolution than the feature maps of each tensor in the second plurality of tensors, and the tensors in the first plurality of tensors and the second plurality of tensors correspond to a hierarchical representation of the feature maps for the single frame.

Images