Method, apparatus and system for encoding and decoding tensors and video

Abstract

A system and method of decoding tensors and video from a bitstream. The network second portion is a second portion of a neural network. The decoding method comprises determining a first codec for the video based on first information decoded from the bitstream and determining a second codec for the tensors based on second information decoded from the bitstream. The video is decoded according to the first codec and the tensors are decoded according to the second codec. The first information and the second information are independent of each other and the video and the tensors are associated with each other.

Description

METHOD, APPARATUS AND SYSTEM FOR ENCODING AND DECODINGTENSORS AND VIDEOREFERENCE TO RELATED APPLICATION(S)

[0001] The application claims the benefit under 35 U.S.C. §119 of the filing date of Australian Patent Application No. 2024200305, filed 17 January 2024, hereby incorporated by reference in its entirety as if fully set forth herein.TECHNICAL FIELD

[0002] The present invention relates generally to digital video signal processing and, in particular, to a method, apparatus and system for encoding and decoding tensors and video from a bitstream. The tensors may be generated using a convolutional neural network. The present invention also relates to a computer program product including a computer readable medium having recorded thereon a computer program for encoding and decoding tensors and a video from a bitstream.BACKGROUND

[0003] Convolutional neural networks (CNNs) are an emerging technology addressing, among other things, use cases involving machine vision such as object detection, instance segmentation, object tracking, human pose estimation, and action recognition. Applications for CNNs can involve use of ‘edge devices’, with sensors and some processing capability, coupled to application servers as part of a ‘cloud’. CNNs can require relatively high computational complexity, more than can typically be afforded either in computing capacity or power consumption by an edge device. Executing a CNN in a distributed manner has emerged as one solution to running leading-edge networks using limited capability edge devices without requiring all computational complexity to be incurred within cloud servers whilst edge devices have potentially under-utilised inferencing resources. In other words, distributed processing allows legacy edge devices to still provide the capability of leading-edge CNNs by distributing processing between the edge device and other processing means, such as cloud servers. Such a distributed network architecture may be referred to as ‘collaborative intelligence’ (CI) and offers benefits such as re-using a partial result from a first portion of the network with severaldifferent second portions, perhaps with each portion being optimised for a different task. CI architectures introduce a need for efficient compression of tensor data, for transmission over a network such as a WAN.

[0004] CNNs typically include many layers, such as convolution layers and fully connected layers, with data passing from one layer to the next in the form of ‘tensors’. Splitting a network across different devices introduces a need to compress the intermediate multi-dimensional tensor data that passes from one layer to the next within a CNN in order to facilitate transmission over a network having bandwidth limitations or costs. Compression of such tensors may be referred to as ‘feature compression’ and the intermediate tensor data is often referred to as ‘features’ or ‘feature maps’. Features or feature maps are generally a collection of two-dimensional (2D) arrays of values which, when combined into a 3D (or 4D) data structure form a tensor, with each feature map corresponding to one ‘channel’ of the tensor. Intermediate tensor data represents a partially processed form of input such as an image frame or video frame, encountered within a neural network. International Organisation for Standardisation / International Electrotechnical Commission Joint Technical Committee 1 / Subcommittee 29 / Working Groups 2 (ISO / IEC JTC1 / SC29 / WG2), also known as the “Moving Picture Experts Group” (MPEG) Technical Requirements are tasked with studying requirements for compression technology in various contexts and often in relation to video. WG2 ‘MPEG Technical Requirements’ has established a ‘Feature Compression for Video Coding for Machines’ (FCVCM) ad-hoc group, mandated to study feature compression. The FCVCM AHG has issued a ‘Call for Proposals’ soliciting responses to form the basis for a standardisation project on feature compression. Previously, responses to a ‘Call for Evidence’ (CfE) demonstrated technology that can significantly outperform feature compression results achieved using state-of-the-art standardised technology directly applied to the tensors.

[0005] CNNs typically require weights for each of the layers to be predetermined in a training stage, where a very large amount of training data is passed through the CNN and a result determined by the network undergoing training being compared to ground truth associated with the training data. Discrepancy between the obtained and desired result is expressed as a Toss’ and measured with a Toss function’. Using the determined loss, a process for updating network weights, such as stochastic gradient descent (SGD), is performed. Network weight update typically involves a process of back-propagation of ‘gradients’ that begins at the output layer of the network and proceeds backward to terminate when the input layer to the network is updated, covering intermediate, or ‘hidden’, layers of the network. Gradients are indicative of deltas tobe applied to network weights and are themselves updated as part of the back propagation process. The rate of weight update is set by a ‘learning rate’ hyperparameter, typically set to facilitate the training process in finding a global minima in terms of loss (i.e., highest possible task performance for the network architecture and training data) while avoiding the training process becoming ‘stuck’ in a local minima. Becoming stuck in a local minima corresponds to obtaining sub-optimal task performance for the network architecture and being incapable of finding new weight values that could lead to higher task performance. Network weights are repeatedly updated by supplying input data and ground truth data organised into ‘batches’ to iteratively refine the network performance until further improvement in accuracy is no longer achievable. An iteration through the entire training dataset forms one ‘epoch’ of training and training typically requires performing multiple epochs to achieve a high level of performance for the task. Weights for a trained network are then available for deployment, and the network operates in a mode where weights are fixed and gradients for weight update are omitted. The process of executing a pretrained CNN with an input and progressively transforming the input into an output according to a topology of the CNN is commonly referred to as ‘inferencing’.

[0006] Generally, a tensor has four dimensions, namely: batch, channels, height and width. The first dimension, ‘batch’, is typically of size one when inferencing on video data and indicates that one frame is passed through a CNN as one batch. When training a network, the value of the batch dimension may be increased so that multiple frames are passed through the network in each batch before the network weights are updated, according to a predetermined ‘batch size’. A multi-frame video may be passed through as a single tensor with the batch dimension increased in size according to the number of frames of a given video. However, for practical considerations relating to memory consumption and access, inferencing on video data is typically performed on a frame-wise basis. The ‘channels’ dimension indicates the number of concurrent ‘feature maps’ for a given tensor and the height and width dimensions indicate the size of the feature maps at the particular stage of the CNN. Channel count varies through the layers of a CNN according to the network architecture. Feature map size also varies, depending on subsampling or upsampling occurring in specific network layers.

[0007] The overall complexity of the CNN tends to be relatively high, with relatively large numbers of multiply-accumulate (MAC) operations being performed and numerous intermediate tensors being written to and read from memory, along with reading weights for performance of each layer of the CNN. As such, dividing a neural network into portions allowsimplementation of more complex networks even in systems containing less capable edge devices, without requiring cloud servers to bear the full burden of performing the network.

[0008] Feature compression may benefit from existing video compression standards, such as Versatile Video Coding (VVC), developed by the Joint Video Experts Team (JVET). VVC is anticipated to address ongoing demand for ever-higher compression performance, especially as video formats increase in capability (for example, with higher resolution and higher frame rate) and to address increasing market demand for service delivery over WANs, where bandwidth costs are relatively high. VVC is implementable in contemporary silicon processes and offers an acceptable trade-off between achieved performance versus implementation cost. The implementation cost may be considered for example, in terms of one or more of silicon area, CPU processor load, memory utilisation and bandwidth. Other video compression standards, such as High Efficiency Video Coding (HEVC) or AV-1, may also be used for feature compression applications.

[0009] Video data includes a sequence of frames of image data, each frame including one or more colour channels. Where feature map data is to be represented in a packed frame, generally a monochrome frame having luminance only and no chroma channels is adequate. In the context of block-based coding the frame format description, a ‘sample’ is a single value, as would be obtained from one cell in an imaging sensor. When only luma samples are present, the resulting monochrome frames are said to use a “4:0:0 chroma format”.

[0010] The VVC standard specifies a ‘block based’ architecture, in which frames are firstly divided into an array of square regions known as ‘coding tree units’ (CTUs). In VVC, CTUs generally occupy 128x 128 luma samples. Other possible CTU sizes when using the VVC standard are 32x32 and 64x64. However, CTUs at the right and bottom edge of each frame may be smaller in area, with implicit splitting occurring to ensure coding blocks remain in the frame. Associated with each CTU is a ‘coding tree’ defining a decomposition of the area of the CTU into a set of blocks, also referred to as ‘coding units’ (CUs). Blocks applicable to only the luma channel or only the chroma channels are referred to as ‘coding blocks’ (CBs). A prediction of the contents of a coding block is held in a ‘prediction block’ (PB) or ‘prediction unit’ (PU) and a residual block defining an array of sample values to be additively combined with the PB or PU is referred to as a ‘transform block’ (TB) or ‘transform unit’ (TU), owing to the typical use of a transformation process in the generation of the TB or TU.

[0011] Notwithstanding the above distinction between ‘units’ and ‘blocks’, the term ‘block’ may be used as a general term to refer to areas or regions of a frame for which operations are applied to all colour channels.

[0012] For each CU, a prediction unit (PU) of the contents (sample values) of the corresponding area of frame data is generated (a ‘prediction unit’). Further, a representation of the difference (or ‘spatial domain’ residual) between the prediction and the contents of the area as seen at input to the encoder is formed. The difference in each colour channel may be transformed and coded as a sequence of residual coefficients, forming one or more TUs for a given CU. The applied transform may be a Discrete Cosine Transform (DCT) or other transform, applied to each block of residual values. The transform is applied separably, (i.e., the two-dimensional transform is performed in two passes, one horizontally and one vertically). The block is firstly transformed by applying a one-dimensional transform to each row of samples in the block. Then, the partial result is transformed by applying a one-dimensional transform to each column of the partial result to produce a final block of transform coefficients that substantially decorrelates the residual samples. Transforms of various sizes are supported by the VVC standard, including transforms of rectangular-shaped blocks, with each side dimension being a power of two. Transform coefficients are quantised for entropy encoding into a bitstream.

[0013] PBs or PUs in VVC may be generated using either an intra-frame prediction or an inter-frame prediction process. Intra-frame prediction involves the use of previously processed samples in a frame being used to generate a prediction of a current block of data samples in the frame. Inter-frame prediction involves generating a prediction of a current block of samples in a frame using a block of samples obtained from one or two previously decoded frames. The block of samples obtained from a previously decoded frame is offset from the spatial location of the current block according to a motion vector, which often has filtering applied. Intra-frame prediction blocks can be (i) a uniform sample value (“DC intra prediction”), (ii) a plane having an offset and horizontal and vertical gradient (“planar intra prediction”), (iii) a population of the block with neighbouring samples applied in a particular direction (“angular intra prediction”) or (iv) the result of a matrix multiplication using neighbouring samples and selected matrix coefficients.

[0014] VVC may be used to compress intermediate feature maps from a first portion (a ‘backbone’) of a neural network separated into two portions. In compression, the feature mapsfrom the backbone are arranged into a frame and quantised from a floating-point domain to a sample domain suitable for compression as video data. CNNs represent a quickly evolving technology, and the CNNs used for machine vision tasks can become updated with different versions or replaced with new CNN architectures as technology develops. A bitstream containing tensors from a backbone (a CNN first portion) may be decoded and supplied to one or more network heads (CNN second portions) out of many available network heads.

[0015] VVC bitstreams contain a number of Network Abstraction Layer (NAL) units. NAL units defining the properties of the video data, such as resolution and chroma format and quantization information, are stored in non-Video Coding Layer (VCL) NAL units such as a ‘Sequence Parameter Set’ (SPS) and a ‘Picture Parameter Set’ (PPS). NAL units holding coded slices, that is, runs of coding tree units, are stored in VCL-layer NAL units, such as intra slices or P slices or B slices.

[0016] Machine vision tasks performed by CNNs can provide outputs based on the particular head network used. Results can relate to matters such as bounding boxes, segmentation markers and the like. In some instances, a human may wish to review or interpret a CNN result. A need exists to assist human interpretation of CNNs in some use cases. Where a bitstream is to contain both compressed video data and compressed feature data, a need exists to define a storage format capable of holding the video and feature data. Further, flexibility is required to allow usability as encoding trends change.SUMMARY

[0017] It is an object of the present invention to substantially overcome, or at least ameliorate, one or more disadvantages of existing arrangements.

[0018] One aspect of the present disclosure provides a method of decoding tensors and video from a bitstream, the tensors being related to the video, the method comprising: determining a first codec for the video based on first information decoded from the bitstream; determining a second codec for the tensors based on second information decoded from the bitstream; decoding the video according to the first codec; decoding the tensors according to the second codec; and wherein the first information and the second information are independent of each other and the video and the tensors are associated with each other.

[0019] Another aspect of the present disclosure provides a method of encoding tensors and video into a bitstream, the tensors being related to the video, the method comprising: encoding first information into the bitstream, the first information being used to determine a first codec for the video; encoding second information into the bitstream, the second information being used to determine a second codec for the tensors; encoding the video into the bitstream according to the first codec; encoding the tensors into the bitstream according to the second codec; and wherein the first information and the second information are independent of each other and the video and the tensors are associated with each other.

[0020] Another aspect of the present disclosure provides a decoder for decoding tensors and video from a bitstream, the tensors being related to the video, the decoder configured for: determining a first codec for the video based on first information decoded from the bitstream; determining a second codec for the tensors based on second information decoded from the bitstream; decoding the video according to the first codec; decoding the tensors according to the second codec; and wherein the first information and the second information are independent of each other and the video and the tensors are associated with each other.

[0021] Another aspect of the present disclosure provides a non-transitory computer-readable storage medium which stores a program for executing a method of decoding tensors and video from a bitstream, the tensors being related to the video, the method comprising: decoding tensors and video from a bitstream, the tensors being related to the video, the method comprising: determining a first codec for the video based on first information decoded from the bitstream; determining a second codec for the tensors based on second information decoded from the bitstream; decoding the video according to the first codec; decoding the tensors according to the second codec; and wherein the first information and the second information are independent of each other and the video and the tensors are associated with each other.

[0022] Another aspect of the present disclosure provides an encoder for encoding tensors and video into a bitstream, the tensors being related to the video, the encoder configured for: encoding first information into the bitstream, the first information being used to determine a first codec for the video; encoding second information into the bitstream, the second information being used to determine a second codec for the tensors; encoding the video into the bitstream according to the first codec; encoding the tensors into the bitstream according to the second codec; and wherein the first information and the second information are independent of each other and the video and the tensors are associated with each other.

[0023] Another aspect of the present disclosure provides a computer-implemented medium non-transitory computer-readable storage medium which stores a program for executing a method of encoding tensors and video into a bitstream, the tensors being related to the video, the method comprising: encoding first information into the bitstream, the first information being used to determine a first codec for the video; encoding second information into the bitstream, the second information being used to determine a second codec for the tensors; encoding the video into the bitstream according to the first codec; encoding the tensors into the bitstream according to the second codec; and wherein the first information and the second information are independent of each other and the video and the tensors are associated with each other.

[0024] Other aspects are also disclosed.BRIEF DESCRIPTION OF THE DRAWINGS

[0025] At least one embodiment of the present invention will now be described with reference to the following drawings and an appendix, in which:

[0026] Fig. l is a schematic block diagram showing a distributed machine task system;

[0027] Figs. 2A and 2B form a schematic block diagram of a general-purpose computer system upon which the distributed machine task system of Fig. 1 may be practiced;

[0028] Fig. 3 A is a schematic block diagram showing functional modules of a backbone portion of a CNN;

[0029] Fig. 3B is a schematic block diagram showing a residual block of Fig. 3 A;

[0030] Fig. 3C is a schematic block diagram showing a residual unit of Fig. 3A;

[0031] Fig. 3D is a schematic block diagram showing a CBL module of Fig. 3 A;

[0032] Fig. 4 is a schematic block diagram showing functional modules of an alternative backbone portion of a CNN;

[0033] Fig. 5 is a schematic block diagram of a tensor encoder using a configurable tensor compressor stage;

[0034] Fig. 6 is a schematic block diagram showing a multi-scale feature fusion stage for a tensor compressor;

[0035] Fig. 7 is a schematic block diagram showing an inter-channel decorrelation-based tensor compressor;

[0036] Fig. 8 is a schematic block diagram showing functional modules of a video encoder;

[0037] Figs. 9A & 9B are schematic block diagrams showing an arrangement of regions or subpictures for holding compressed tensor data;

[0038] Figs. 10A & 10B are schematic block diagrams showing another arrangement of regions or subpictures for holding compressed tensor data;

[0039] Fig. 11 A is a schematic block diagram showing a bitstream holding encoded interchannel decorrelated feature maps, compressed video, and associated metadata;

[0040] Fig. 1 IB is a schematic block diagram showing a hierarchical arrangement of ‘boxes’ resulting in a presentation which encapsulates a video stream and a feature stream;

[0041] Fig. 12 is a schematic block diagram showing a tensor decoder with a configurable tensor decompressor;

[0042] Fig. 13 is a schematic block diagram showing functional modules of a video decoder;

[0043] Fig. 14 is a schematic block diagram showing an inter-channel decorrelation-based tensor decoder as part of a distributed machine task system;

[0044] Fig. 15 is a schematic block diagram showing an embodiment of a multi-scale feature reconstruction stage;

[0045] Fig. 16A is a schematic block diagrams showing a head portion of a CNN;

[0046] Fig. 16B is a schematic block diagram showing an upscaler module of Fig. 16A;

[0047] Fig. 16C is a schematic block diagram showing a detection module of Fig. 16A;

[0048] Fig. 17 is a schematic block diagram showing an alternative head portion of a CNN;

[0049] Fig. 18 shows a method for performing a first portion of a CNN, compressing the resulting tensors, and signalling an indication the first portion of the CNN; and

[0050] Fig. 19 shows a method for decoding a bitstream, reconstructing tensors and completing performance a task according to network topology and network weights indications.DETAILED DESCRIPTION INCLUDING BEST MODE

[0051] Where reference is made in any one or more of the accompanying drawings to steps and / or features, which have the same reference numerals, those steps and / or features have for the purposes of this description the same function(s) or operation(s), unless the contrary intention appears.

[0052] A distributed machine task system may include an edge device, such as a network camera or smartphone producing intermediate compressed data. The distributed machine task system may also include a final device, such as a server farm based (‘cloud’) application, operating on the intermediate compressed data to produce a task result. Additionally, the edge device functionality may be embodied in the cloud and the intermediate compressed data may be stored for later processing, potentially for multiple different tasks depending on need. A distributed task system may perform a task in a general form in an edge device and provide partially processed features for performance of the task in a specific form at the server side. For example, a general object detection network may be performed in the client (edge) device and the result used in the server to conditionally perform more specific detection networks using, for example, partially processed tensors produced by a first portion of the CNN that is common to the general task network and the specific task network. Such a first portion may be referred to as a ‘shared backbone’ as it is common both in network topology and network weights to multiple task networks.

[0053] A convenient form of intermediate compressed data is a compressed video bitstream, owing to the availability of high-performing compression standards and implementations thereof. Video compression standards typically operate on integer samples of some given bit depth, such as 10 bits, arranged in planar arrays. Colour video has three planar arrays, corresponding, for example, to colour components Y, Cb, Cr, or R, G, B, depending on application. CNNs typically operate on floating point data in the form of tensors. Tensors generally have a relatively smaller spatial dimensionality compared to incoming video dataupon which the CNN operates while having more channels than the three channels typical of colour video data, for example 128, 256, or 512 channels.

[0054] Tensors typically have the following dimensions: frames, channels, height, and width. For example, a tensor of dimensions [1, 256, 76, 136] would be said to contain floating-point or integer values for one frame comprising an array of two-hundred and fifty-six (256) feature maps (channels), each of size 136x76. For video data, inferencing is typically performed one frame at a time (frame value of 1), rather than using tensors containing multiple frames.

[0055] VVC supports a division of a picture into multiple subpictures, each of which may be independently encoded and independently decoded. In one approach, each subpicture is coded as one ‘slice’, or contiguous sequence of coded CTUs. A ‘tile’ mechanism is also available to divide a picture into a number of independently decodeable regions. Subpictures may be specified in a somewhat flexible manner, with various rectangular sets of CTUs coded as respective subpictures. Flexible definition of subpicture dimensions allows efficiently holding types of data requiring different areas in one picture, avoiding large ‘unused’ areas, i.e., areas of a frame that are not used for reconstruction of tensor data.

[0056] A file or stream may be encapsulated in a file format such as ISO Base Media File Format (ISOBMFF), standardised as ISO / IEC 14496-12 and ISO 14496-15 for an extension related to the carriage of NAL-unit structured video in ISOBMFF. Where a bitstream is to contain both compressed video data and compressed feature data, a need exists to define a storage format capable of holding the video and the feature data. ISOBMFF defines a hierarchical set of ‘boxes’. Each box begins with a 32-bit type identifier, typically represented as a four 8-bit (i.e., ASCII or UTF-8) code followed by a length field and then a variable-length payload, indicating the total length of the box including the associated payload. By virtue of the hierarchical nature of ISOBMFF, boxes may be declared within other boxes. Box types are defined in an object-oriented manner, enabling one box type to be derived (or to ‘extend’) another box type. Where a box type extends another box type, the child box includes all defined attributes of all ancestor box type, analoguous to class definition in an object-oriented language such as C++. A collection of boxes defining one or more motion sequences, possibly including audio, form a ‘presentation’, which may be embodied in a file or as streamed data. In the terminology of ISOBMFF, all data associated with a particular time of presentation is referred to as a ‘sample’, this includes a video frame. A presentation includes a sequence of samples, each associated with a different time, enabling delivery of media content to a useraccording to the indicated times and avoiding for example jitter in the delivery of each decoded frame (or sample) to the viewer.

[0057] Fig. l is a schematic block diagram showing functional modules of a distributed machine task system 100, capable of performing a machine task network in a distributed manner. The division of a particular neural network into two portions requires specifying a ‘split point’ in the network. Layers in the network from the input layer up to the split point are performed in a first device (or ‘source device’) and resulting intermediate tensor(s) are compressed. Layers up to the split point may be referred to as the ‘backbone’ however this term may sometimes imply a specific split point. An alternative term ‘NN part 1 ’ (neural network part one, also referred to as a first network portion) carries no implication on the location of the split point in the network. Layers from the split point to the last layer may be referred to as the ’head’ and, for the avoidance of any implied split point location, may alternatively be referred to as ‘NN part 2’ (neural network part 2, also referred to as a second network portion). A first NN part 2 (also referred to as a proxy NN part 2) performs remaining layers of the network to produce an initial task result, to be included in the bitstream for subsequent usage. Based on the result obtained from the network, layers from the split point up to the last layer in an alternative network are performed in a second device (or ‘destination device’) using decompressed tensor(s) from the first device as input to the layer(s) immediately following the split point. The alternative network has a common NN part 1 with the first network, and thus is able to produce a task result 167 with performing NN part 2 using the tensors produced by NN part 1 of the first network.

[0058] At the split point there may be one or more tensors that need to be compressed for conveyance over a communication channel with limited bandwidth compared to the bandwidth requirement for transmission of uncompressed tensors. Where a ‘feature pyramid network’ (FPN) is in use, it is common for layers in the FPN to be related in width and height such that a given layer is half the width and height of an adjacent layer among the layers. FPN architectures may also involve the width and height halving alternatively from one layer to the next layer. In some architectures, multiple tensors of the same width and height are produced within the FPN. An FPN may occur relatively early in the neural network topology, resulting in a necessity for a split point to occur within the FPN in order for a useful division of the network workload across the edge device and the cloud to be achieved. When a split occurs within the FPN of the machine task network, performance of a variety of machine task networks where layers up to the split point are common among the machine task networks (‘shared backbone’architecture) may be achieved. Where a split point occurs within the FPN, tensor compression methods may exploit redundancies across the FPN layers to improve compression performance. Compression methods applicable to the various network topologies used in contemporary CNNs are therefore beneficial for application in a wide range of scenarios.

[0059] The system 100 may be used for implementing methods for decorrelating, packing and quantising feature maps into planar frames for encoding and decoding feature maps from encoded data for various neural networks. Various neural networks may be split at different points and may result in intermediate tensors of various number and dimensionality. The multitude of possible neural networks and split points creates a need for the destination device 140 to determine compatibility with a repository of available NN part 2 implementations available in an NN part 2 repository 160. Enumerating potential neural networks is made difficult due to the need to maintain a centralised and agreed repository of network names. Furthermore, reference implementations may be available only in transient form such as public software repositories, which may change as updates are committed. An additional complication results from the machine learning community having failing to agree on the naming of newly created networks, such as seen in the ‘YOLO’ lineage of networks, where multiple different researchers may claim to have published a next YOLO version in the lineage. Open Neural Network Exchange (ONNX) is one format providing a device-independent way of specifying the weights and topology of a neural network. Stripped of weights, the ONNX topology provides a relatively lightweight complete representation of a neural network topology. Signalling an entire topology to identify a network may lack scale as network topology increases and besides, a more summarised indication of a network topology would be sufficient to uniquely identify a network without needing to reference any potentially unstable public resources. Even referencing a paper is typically insufficient as the paper will reference a public software repository (such as one hosted on github.com). Thus, to enable interoperability between different source devices and destination devices, a signalling means that enables unambiguous compatibility determination between an NN part 1 and one or more NN part 2 implementations is needed. In particular, a mechanism that does not rely upon a centralised enumeration, such as a registration authority, is preferable as establishing, maintaining, and promoting use of such a registration authority is typically burdensome and an obstacle to industry adoption. Instead, a signalling means that is more concise than sending an entire network weights and / or topology, is an alternative that can be deployed more readily whilst avoiding the need for an externally maintained and recognised authoritative registry of network weights and / or topology.

[0060] The system 100 may, in some implementations, convey both intermediate compressed features from an NN part 1 and compressed video from a video source from a source device to a destination device. Including both video and compressed features enables use-cases like overlaying task result, such as bounding boxes of specific objects, onto video for review by a human. Another use case is for the task result of the source device NN part 2 or of the destination device NN part 2 to trigger long-term storage of video that may be of future interest. In one example, video may be retained only when any person is detected, reducing storage costs compared to storage all received video. When a specific person of interest needs to be identified, an NN part 2 may be performed that is trained to detect a specific person of interest. The destination device may reprocess the NN part 1 tensors corresponding to frames containing at least one detected person using the customised NN part 2 to further filter out video data to the most relevant clips, which may then be reviewed by a human. In other implementations, video may not be retained unless a request for the video has been made.

[0061] The system 100 includes a source device 110. The source device 110 includes a video source 112 for generating unencoded frame data 113. The frame data 113 is passed to NN part 1 114 to produce tensors 115 and to a video encoder 150 to produce a video layer bitstream 151. The tensors 115 are passed to a tensor encoder 116, which produces a feature layer bitstream 121. The feature layer bitstream 121 is passed to a box encapsulator 154, which outputs a box-encapsulated bitstream 155. The bitstream 155 is passed to a transmitter 122 which outputs a transmitted bitstream 123. The box-encapsulated bitstream 155 includes a track containing video frames, obtained from the video-layer bitstream 151, and another track containing feature frames, obtained from the feature-layer bitstream 121. The packed features from the tensors 115 from the NN part 1 114 form a highly transformed version of the frame data 113. The resolution of the feature frames is dependent on the number of tensors in the tensors 115 and the channel count and resolution of each of the tensors 115, which is not related (or indirectly related) to the resolution of frames in the frame data 113. Moreover, there is no simple spatial relationship between the abstract feature map data and the samples in the frame data 113.

[0062] The system 100 also includes a destination device 140 for decoding tensor data in the form of the received bitstream 123. The destination device 140 may be used for decoding the tensor data (or tensors) for content (e.g., of audio data, video data, image data, and textual data) of the bitstream 123.

[0063] A communication channel 130 is used to communicate the bitstream 123 from the source device 110 to the destination device 140. In some arrangements, the source device 110 and destination device 140 may either or both comprise respective mobile telephone handsets (e.g., “smartphones”) or network cameras and cloud applications. The communication channel 130 may be a wired connection, such as Ethernet, or a wireless connection, such as WiFi or 5G, including connections across a Wide Area Network (WAN). The communication channel 130 may also be implemented across ad-hoc connections. Moreover, the source device 110 and the destination device 140 may comprise applications where encoded video data is captured on some computer-readable storage medium, such as a hard disk drive in a file server or memory. Although the system 100 is described as including the video source 112, which would provide the frame data 113 for a neural network targeting a computer vision application, other types of source data, such as audio or text, may be input to a suitable neural network implemented in the NN part 1 114 and an NN part 2 head 166.

[0064] As shown in Fig. 1, the source device 110 includes the video source 112, the NN part 1 114, the tensor encoder 116, the box encapsulator 154, and the transmitter 122. The video source 112 typically comprises a source of captured video frame data (shown as 113), such as an image capture sensor, a previously captured video sequence stored on a non-transitory recording medium, or a video feed from a remote image capture sensor. The video source 112 may also be an output of a computer graphics card, for example, displaying the video output of an operating system and various applications executing upon a computing device (e.g., a tablet computer). Examples of source devices 110 that may include an image capture sensor as the video source 112 include smart-phones, video camcorders, professional video cameras, and network video cameras. The video source 112 may produce independent images or may produce temporally sequential images, i.e., a video.

[0065] The system 100 implements a task network in a distributed manner, with a division into two parts, that is, NN part 1 114 and NN part 2 166. For example, a ‘ YOLOv3’ network may be used as one part of an object tracking system and a ‘FasterRCNN’ network may be used as an object detection system. The number and dimensionality of tensors 115 depends on a particular network performed in the system 100 and the split point of the particular task network.

[0066] The NN part 1 114 receives the video frame data 113 and performs specific layers of an overall CNN, such as layers corresponding to the ‘backbone’ of the CNN, outputting tensors115. The backbone (part 1) layers of the CNN may produce multiple tensors as the output 115, for example, corresponding to different spatial scales of an input image represented by the video frame data 113 when splitting the network within the FPN. An FPN may result in three tensors, corresponding to three layers, output from the backbone 114 as the tensors 115 (e.g., if a ‘Y0L0v3’ network is performed by the system 100), with varying spatial resolution and channel count. When the system 100 is performing networks such as ‘Faster RCNN X101- FPN’ or ’Mask RCNN XI 01 -FPN’ the tensors 115 may include tensors for four layers (P2-P5). Use of a FPN results in a plurality of tensors forming a hierarchical representation for a single frame to be encoded to (and decoded from) the bitstream when the split point of the network occurs within the FPN, as described hereafter. The tensor encoder 116 receives the tensors 115 and produces the feature bitstream 121, containing a compressed representation of the tensors 115.

[0067] The feature bitstream 155 is supplied to the transmitter 122 for modulation for transmission over the communications channel 130 as part of the bitstream 123. Alternatively or additionally, the bitstream 123 may be written to storage 132 for later use.

[0068] Although the tensor encoder 116 and a tensor decoder 146 are described as using a block-based encoder and decoder, specifically VVC, in conjunction with other processing, different categories of codec may be used instead. An alternative type of codec is an ‘end-to- end’ learned codec. End-to-end learned codecs rely upon learned elements to produce a sparse tensor which may be efficiently entropy coded. An analysis and synthesis stage are constructed out of learned elements such as convolutions, GDN (gradual divisive normalisation) and other stages are used to produce a sparse tensor. The sparse tensor is losslessly arithmetically encoded to produce a bitstream. Some end-to-end learned codecs make use of a hyperprior, or side-channel of information that helps capture spatial dependencies present in the main channel. Probability distributions in the entropy coding of the main channel may be adapted based on the hyperprior to achieve improved compression performance. End-to-end learned compression schemes are typically trained for a given quality level, so instead of adapting quality by varying a quantisation parameter, different network weights may be used to select a desired quality level. One example of a learned end-to-end codec is described in an IEEE paper entitled ‘Learned Image Compression with Discretized Gaussian Mixture Likelihoods and Attention Modules’ by Zhengxue Cheng et al and published at the CVPR2020 conference, informally known as the ‘Cheng2020’ codec.

[0069] The bitstream 123 is transmitted by the transmitter 122 over the communication channel 130 as encoded data. The bitstream 123 can in some implementations be stored in the storage memory 132, where the storage 132 is a non-transitory storage device such as a “Flash” memory or a hard disk drive, until later being transmitted over the communication channel 130 (or in-lieu of transmission over the communication channel 130). For example, encoded video data may be served upon demand to customers over a wide area network (WAN) for a video analytics application.

[0070] The destination device 140 includes a receiver 142, a tensor decoder 146, the box extractor 144, the NN part 2 166, a video decoder 170, and a task result Tenderer 168. The receiver 142 demodulates the bitstream 123 from the communication channel 130, sending a demodulated bitstream 143 to the box extractor 144. The box extractor 144 includes a file format parser capable of parsing high-level syntax applicable to the definition of the box structure, such as box types and box lengths in the box hierarchy, to be described with reference to Fig. 1 IB. A feature sub-bitstream 145 is passed from the box extractor 144 to the tensor decoder 146. A video sub-bitstream 171 is passed from the box extractor 144 to the video decoder 170. The tensor decoder 146 outputs decoded tensors 149, which are supplied to the NN part 2 166. The NN part 2 166 implements a network, forming the remainder of the network performed by the NN part 1 114. The NN part 2 166 receives the tensors 149 and performs the later layers of the neural network that began with the NN part 1 114 to produce a task result 167.

[0071] The task result 167 is presented to the user, such as visually by the task result Tenderer 168. The task result Tenderer 168 draws bounding boxes or segmentation maps highlighting detected objects or instances in accordance with the task result 167 and overlay ed on decoded video 172 produced by the video decoder 170. The task result Tenderer 168 is one example of presenting neural network result presentation to a user. The task result Tenderer 168 may be operative only when the task result 168 indicates a detected object.

[0072] It is also possible for the functionality of each of the source device 110 and the destination device 140 to be embodied in a single device, examples of which include mobile telephone handsets and tablet computers and cloud applications.

[0073] Notwithstanding the example devices mentioned above, each of the source device 110 and destination device 140 may be configured within a general -purpose computing system,typically through a combination of hardware and software components. Fig. 2A illustrates such a computer system 200, which includes: a computer module 201; input devices such as a keyboard 202, a mouse pointer device 203, a scanner 226, a camera 227, which may be configured as the video source 112, and a microphone 280; and output devices including a printer 215, a display device 214 and loudspeakers 217. An external Modulator-Demodulator (Modem) transceiver device 216 may be used by the computer module 201 for communicating to and from a communications network 220 via a connection 221. The communications network 220, which may represent the communication channel 130, may be a (WAN), such as the Internet, a cellular telecommunications network, or a private WAN. Where the connection 221 is a telephone line, the modem 216 may be a traditional “dial-up” modem. Alternatively, where the connection 221 is a high capacity (e.g., cable or optical) connection, the modem 216 may be a broadband modem. A wireless modem may also be used for wireless connection to the communications network 220. The transceiver device 216 may provide the functionality of the transmitter 122 and the receiver 142 and the communication channel 130 may be embodied in the connection 221.

[0074] The computer module 201 typically includes at least one processor unit 205, and a memory unit 206. For example, the memory unit 206 may have semiconductor random access memory (RAM) and semiconductor read only memory (ROM). The computer module 201 also includes a number of input / output (VO) interfaces including: an audio-video interface 207 that couples to the video display 214, loudspeakers 217 and microphone 280; an I / O interface 213 that couples to the keyboard 202, mouse 203, scanner 226, camera 227 and optionally a joystick or other human interface device (not illustrated); and an interface 208 for the external modem 216 and printer 215. The signal from the audio-video interface 207 to the computer monitor 214 is generally the output of a computer graphics card. In some implementations, the modem 216 may be incorporated within the computer module 201, for example within the interface 208. The computer module 201 also has a local network interface 211, which permits coupling of the computer system 200 via a connection 223 to a local-area communications network 222, known as a Local Area Network (LAN). As illustrated in Fig. 2A, the local communications network 222 may also couple to the wide network 220 via a connection 224, which would typically include a so-called “firewall” device or device of similar functionality. The local network interface 211 may comprise an Ethernet™ circuit card, a Bluetooth™ wireless arrangement or an IEEE 802.11 wireless arrangement; however, numerous other types of interfaces may be practiced for the interface 211. The local network interface 211 may alsoprovide the functionality of the transmitter 122 and the receiver 142 and communication channel 130 may also be embodied in the local communications network 222.

[0075] The I / O interfaces 208 and 213 may afford either or both of serial and parallel connectivity, the former typically being implemented according to the Universal Serial Bus (USB) standards and having corresponding USB connectors (not illustrated). Storage devices 209 are provided and typically include a hard disk drive (HDD) 210. Other storage devices such as a floppy disk drive and a magnetic tape drive (not illustrated) may also be used. An optical disk drive 212 is typically provided to act as a non-volatile source of data. Portable memory devices, such optical disks (e.g. CD-ROM, DVD, Blu ray Disc™), USB-RAM, portable, external hard drives, and floppy disks, for example, may be used as appropriate sources of data to the computer system 200. Typically, any of the HDD 210, optical drive 212, networks 220 and 222 may also be configured to operate as the video source 112, or as a destination for decoded video data to be stored for reproduction via the display 214. The source device 110 and the destination device 140 of the system 100 may be embodied in the computer system 200.

[0076] The components 205 to 213 of the computer module 201 typically communicate via an interconnected bus 204 and in a manner that results in a conventional mode of operation of the computer system 200 known to those in the relevant art. For example, the processor 205 is coupled to the system bus 204 using a connection 218. Likewise, the memory 206 and optical disk drive 212 are coupled to the system bus 204 by connections 219. Examples of computers on which the described arrangements can be practised include IBM-PC’s and compatibles, Sun SPARCstations, Apple Mac™ or alike computer systems.

[0077] The video encoder 150, NN part 1 114, tensor encoder 116, box encapsulator 154, modules 122 and 142, the box extractor 144, tensor decoder 146, NN part 2 166, video decoder 170 and result Tenderer 168 and methods to be described, may be implemented as one or more software application programs 233 executable within the computer system 200. In particular, the modules of the devices 110 and 140 and the steps of the described methods are effected by instructions 231 (see Fig. 2B) in the software 233 that are carried out within the computer system 200. The software instructions 231 may be formed as one or more code modules, each for performing one or more particular tasks. The software may also be divided into two separate parts, in which a first part and the corresponding code modules performs the describedmethods and a second part and the corresponding code modules manage a user interface between the first part and the user.

[0078] The software may be stored in a computer readable medium, including the storage devices described below, for example. The software is loaded into the computer system 200 from the computer readable medium, and then executed by the computer system 200. A computer readable medium having such software or computer program recorded on the computer readable medium is a computer program product. The use of the computer program product in the computer system 200 preferably effects an advantageous apparatus for implementing the source device 110 and the destination device 140 and the described methods.

[0079] The software 233 is typically stored in the HDD 210 or the memory 206. The software is loaded into the computer system 200 from a computer readable medium, and executed by the computer system 200. Thus, for example, the software 233 may be stored on an optically readable disk storage medium (e.g., CD-ROM) 225 that is read by the optical disk drive 212.

[0080] In some instances, the application programs 233 may be supplied to the user encoded on one or more CD-ROMs 225 and read via the corresponding drive 212, or alternatively may be read by the user from the networks 220 or 222. Still further, the software can also be loaded into the computer system 200 from other computer readable media. Computer readable storage media refers to any non-transitory tangible storage medium that provides recorded instructions and / or data to the computer system 200 for execution and / or processing. Examples of such storage media include floppy disks, magnetic tape, CD-ROM, DVD, Blu-ray Disc™, a hard disk drive, a ROM or integrated circuit, USB memory, a magneto-optical disk, or a computer readable card such as a PCMCIA card and the like, whether or not such devices are internal or external of the computer module 201. Examples of transitory or non-tangible computer readable transmission media that may also participate in the provision of the software, application programs, instructions and / or video data or encoded video data to the computer module 201 include radio or infra-red transmission channels, as well as a network connection to another computer or networked device, and the Internet or Intranets including e-mail transmissions and information recorded on Websites and the like.

[0081] The second part of the application program 233 and the corresponding code modules mentioned above may be executed to implement one or more graphical user interfaces (GUIs) to be rendered or otherwise represented upon the display 214. Through manipulation oftypically the keyboard 202 and the mouse 203, a user of the computer system 200 and the application may manipulate the interface in a functionally adaptable manner to provide controlling commands and / or input to the applications associated with the GUI(s). Other forms of functionally adaptable user interfaces may also be implemented, such as an audio interface utilizing speech prompts output via the loudspeakers 217 and user voice commands input via the microphone 280.

[0082] Fig. 2B is a detailed schematic block diagram of the processor 205 and a “memory” 234. The memory 234 represents a logical aggregation of all the memory modules (including the storage devices 209 and semiconductor memory 206) that can be accessed by the computer module 201 in Fig. 2A.

[0083] When the computer module 201 is initially powered up, a power-on self-test (POST) program 250 executes. The POST program 250 is typically stored in a ROM 249 of the semiconductor memory 206 of Fig. 2 A. A hardware device such as the ROM 249 storing software is sometimes referred to as firmware. The POST program 250 examines hardware within the computer module 201 to ensure proper functioning and typically checks the processor 205, the memory 234 (209, 206), and a basic input-output systems software (BIOS) module 251, also typically stored in the ROM 249, for correct operation. Once the POST program 250 has run successfully, the BIOS 251 activates the hard disk drive 210 of Fig. 2A. Activation of the hard disk drive 210 causes a bootstrap loader program 252 that is resident on the hard disk drive 210 to execute via the processor 205. This loads an operating system 253 into the RAM memory 206, upon which the operating system 253 commences operation. The operating system 253 is a system level application, executable by the processor 205, to fulfil various high level functions, including processor management, memory management, device management, storage management, software application interface, and generic user interface.

[0084] The operating system 253 manages the memory 234 (209, 206) to ensure that each process or application running on the computer module 201 has sufficient memory in which to execute without colliding with memory allocated to another process. Furthermore, the different types of memory available in the computer system 200 of Fig. 2A need to be used properly so that each process can run effectively. Accordingly, the aggregated memory 234 is not intended to illustrate how particular segments of memory are allocated (unless otherwise stated), but rather to provide a general view of the memory accessible by the computer system 200 and how such memory is used.

[0085] As shown in Fig. 2B, the processor 205 includes a number of functional modules including a control unit 239, an arithmetic logic unit (ALU) 240, and a local or internal memory 248, sometimes called a cache memory. The cache memory 248 typically includes a number of storage registers 244-246 in a register section. One or more internal busses 241 functionally interconnect these functional modules. The processor 205 typically also has one or more interfaces 242 for communicating with external devices via the system bus 204, using the connection 218. The memory 234 is coupled to the bus 204 using the connection 219.

[0086] The application program 233 includes a sequence of instructions 231 that may include conditional branch and loop instructions. The program 233 may also include data 232 which is used in execution of the program 233. The instructions 231 and the data 232 are stored in memory locations 228, 229, 230 and 235, 236, 237, respectively. Depending upon the relative size of the instructions 231 and the memory locations 228-230, a particular instruction may be stored in a single memory location as depicted by the instruction shown in the memory location 230. Alternately, an instruction may be segmented into a number of parts each of which is stored in a separate memory location, as depicted by the instruction segments shown in the memory locations 228 and 229.

[0087] In general, the processor 205 is given a set of instructions which are executed therein. The processor 205 waits for a subsequent input, to which the processor 205 reacts to by executing another set of instructions. Each input may be provided from one or more of a number of sources, including data generated by one or more of the input devices 202, 203, data received from an external source across one of the networks 220, 202, data retrieved from one of the storage devices 206, 209 or data retrieved from a storage medium 225 inserted into the corresponding reader 212, all depicted in Fig. 2 A. The execution of a set of the instructions may in some cases result in output of data. Execution may also involve storing data or variables to the memory 234.

[0088] The tensor encoder 116, the tensor decoder 146 and the described methods may use input variables 254, which are stored in the memory 234 in corresponding memory locations 255, 256, 257. The tensor encoder 116, the tensor decoder 146 and the described methods produce output variables 261, which are stored in the memory 234 in corresponding memory locations 262, 263, 264. Intermediate variables 258 may be stored in memory locations 259, 260, 266 and 267.

[0089] Referring to the processor 205 of Fig. 2B, the registers 244, 245, 246, the arithmetic logic unit (ALU) 240, and the control unit 239 work together to perform sequences of microoperations needed to perform “fetch, decode, and execute” cycles for every instruction in the instruction set making up the program 233. Each fetch, decode, and execute cycle comprises: a fetch operation, which fetches or reads an instruction 231 from a memory location 228, 229, 230; a decode operation in which the control unit 239 determines which instruction has been fetched; and an execute operation in which the control unit 239 and / or the ALU 240 execute the instruction.

[0090] Thereafter, a further fetch, decode, and execute cycle for the next instruction may be executed. Similarly, a store cycle may be performed by which the control unit 239 stores or writes a value to a memory location 232.

[0091] Each step or sub-process in the methods of Figs. 18 and 19, to be described, is associated with one or more segments of the program 233 and is typically performed by the register section 244, 245, 246, the ALU 240, and the control unit 239 in the processor 205 working together to perform the fetch, decode, and execute cycles for every instruction in the instruction set for the noted segments of the program 233.

[0092] Fig. 3A is a schematic block diagram 300 showing functional modules of a backbone portion 310 of a CNN, which may serve as an implementation of the NN part 1 114 when the system 100 is configured to perform a ‘YOLOv3’ network. The NN part 1 114 is sometimes referred to as ‘DarkNet-53’, although different backbones are also possible, resulting in a different number of and dimensionality of layers of the tensors 115 for each frame. In one implementation, the backbone portion 310 may be used as a person detector for the purpose of object tracking.

[0093] As shown in Fig. 3A, the video data 113 is passed to a resizer module 304. The resizer module 304 resizes each frame of the video data 113 to a resolution suitable for processing by the CNN backbone 310, producing resized frame data 312. If the resolution of the video data 113 is already suitable for the CNN backbone 310, operation of the resizer module 304 isnot needed. The resized frame data 312 is passed to a convolutional batch normalisation leaky rectified linear (CBL) module 314 to produce tensors 316. The CBL module 314 contains modules as described with reference to a CBL module 360 as shown in Fig 3D.

[0094] The CBL module 360 takes as input a tensor 361 of the resized frame data 312. The tensor 361 is passed to a convolutional layer 362 to produce tensor 363. If the convolutional layer 362 has a stride of one, the tensor 363 has the same spatial dimensions as the tensor 361. If the convolution layer 362 has a larger stride, such as two, the tensor 363 has smaller spatial dimensions compared to the tensor 361, for example, halved in width and height for the stride of two. Regardless of the stride, the size of channel dimension of the tensor 363 may vary compared to the channel dimension of the tensor 361 for a particular CBL block. The tensor 363 is passed to a batch normalisation module 364, which outputs a tensor 365. The batch normalisation module 364 normalises the input tensor 363 and applies a scaling factor and an offset value to produce the output tensor 365. The scaling factor and offset value are derived from a training process. The tensor 365 is passed to a leaky rectified linear activation (“LeakyReLU”) module 366 to produce a tensor 367. The module 366 provides a ‘leaky’ activation function whereby positive values in the tensor are passed through and negative values are severely reduced in magnitude, for example, to 0. IX their former value.

[0095] Returning to Fig. 3 A, the tensor 316 is passed from the CBL block 314 to a residual block module 320, such as a ‘resl+2+8’ module (also referred to as a resl 1 module) containing a concatenation of three residual blocks, each residual block containing one (1) residual unit, two (2) residual units, and eight (8) residual units, respectively. The spatial resolution of the tensors is halved horizontally and halved vertically in each of the residual blocks (see Fig. 3B) by a convolution with stride equal to two in a CBL block 344.

[0096] A residual block is described with reference to a ResBlock 340 as shown in Fig. 3B. The ResBlock 340 receives a tensor 341 (e.g., the tensor 316). The tensor 341 is zero-padded by a zero-padding module 342 to produce a tensor 343. The tensor 343 is passed to a CBL module 344 to produce a tensor 345. The CBL module 344 contains a convolution (for example similar to 362) with a stride parameter set to two, resulting in the tensor 345 having half the width and half the height of the tensor 343. The tensor 345 is passed to a residual unit 346. The residual unit 346 contains a series of concatenated residual units, based on the number of residual block (for example, eleven (11) units for the block 320). The last residual unit of the residual units 346 outputs a tensor 347.

[0097] A residual unit is described with reference to a ResUnit 350 as shown in Fig. 3C. The ResUnit 350 takes a tensor 351 (for example the tensor 345) as input. The tensor 351 is passed to a CBL module 352 to produce a tensor 353. The tensor 353 is passed to a second CBL unit 354 to produce a tensor 355. An add module 356 sums the tensor 355 with the tensor 351 to produce a tensor 357. The add module 356 may also be referred to as a ‘shortcut’ as the input tensor 351 substantially influences the output tensor 357. For an untrained network, ResUnit 350 acts to pass-through tensors. As training is performed, the CBL modules 352 and 354 act to deviate the tensor 357 away from the tensor 351 in accordance with training data and ground truth data.

[0098] Returning to Fig. 3 A, the Resl 1 module 320 outputs a tensor 322. The tensor 322 is output from the backbone module 310 as one of the layers and also provided to a Res8 module 324. The Res8 module 324 is a residual block (i.e., 340), which includes eight residual units (i.e. 350). The Res8 module 324 produces a tensor 326. The tensor 326 is passed to a Res4 module 328 and output from the backbone module 310 as one of the layers. The Res4 module is a residual block (i.e., 340), which includes four residual units (i.e., 350). The Res4 module 328 produces a tensor 329. The tensor 329 is output from the backbone module 310 as one of the layers. Collectively, the layer tensors 322, 326, and 329 are output as the tensors 115 and may be referred to as layers 0-2 or L0, LI, and L2, respectively. The backbone CNN 310 may take as input a video frame of resolution 1088x608 and produce three tensors, corresponding to three layers, with the following dimensions: [1, 256, 76, 136], [1, 512, 38, 68], [1, 1024, 19, 34], Another example of the three tensors 115 corresponding to three layers may be [1, 512, 34, 19], [1, 256, 68, 38], [1, 128, 136, 76] which are respectively separated at layer index 75, 90, and 105 when the layers are enumerated according to the YOLOv3 software implementation of the backbone 300 and a head 1200.

[0099] Each of the Resl 1 320, Res8 324 and Res4 328 operates in a similar manner to ResBlock 340. Each of the CBL 314, the CBL 344 and the CBL 354 operate in a similar manner to the CBL 360.[000100] Fig. 4 is a schematic block diagram showing functional modules of an alternative backbone portion 400 of a CNN, which may serve as an implementation of the NN part 1 114 when the system 100 is configured to perform a “FasterRCNN” or “MaskRCNN” ResNet 101 network. Frame data 113 is input and passes through a stem network 408, a res2 module 412, a res3 module 416, a res4 module 420, and a res5 module 424 via tensors 409, 413, 417, 421, 425respectively. The backbone portion 400 may be used as part of a general object detector or for instance segmentation, with various classes of object supported.[000101] The stem network 408 includes a 7x7 convolution with a stride of two (2) and a max pooling operation. The res2 module 412, the res3 module 416, the res4 module 420 and the res5 module 424 perform convolution operations, such as LeakyReLU activations. Each module 412, 416, 420 and 424 also performs one halving of the width and height of the processed tensors via a stride setting of two. Each of the tensors 413, 417, 421 and 425 are passed to one of 1x1 lateral convolution modules 446, 444, 442 and 440 respectively. The modules 446, 444, 442, and 440 produce tensors 447, 445, 443 and 441 respectively. The tensor 441 is passed to a 3x3 output convolution module 470, which produces an output tensor P5 471.[000102] The tensor 441 is also passed to upsampler module 450 to produce an upsampled tensor 451. A summation module 460 sums the tensors 443 and 451 to produce a tensor 461. The tensor 461 is passed to an upsampler module 452 and a 3x3 lateral convolution module 472. The module 472 outputs a P4 tensor 473. The upsampler module 452 produces an upsampled tensor 453. A summation module 462 sums tensors 445 and 453 to produce a tensor 463. The tensor 463 is passed to a 3x3 lateral convolution module 474 and an upsampler module 454. The module 474 outputs a P3 tensor 475. The upsampler module 454 outputs an upsampled tensor 455. A summation module 464 sums the tensors 447 and 455 to produce tensor 465, which is passed to a 3x3 lateral convolution module 476. The module 476 outputs a P2 tensor 477. The upsampler modules 450, 452, and 454 use nearest neighbour interpolation for low computational complexity. The tensors 471, 473, 475, and 477 form the output tensor 115 of the CNN backbone 400 (NN part 1 114). Although Fig. 4 shows a particular backbone portion of the Faster RCNN network architecture (a ‘P-layer split point), different divisions into backbone and head are possible. Splitting the network at tensor 409 is termed a ‘stem’ split point. Splitting the network at tensors 447, 445, 443, and 441 is termed a ‘C-layer’ split point.[000103] Fig. 5 is a schematic block diagram 500 of the tensor encoder 116 using a configurable tensor compressor stage. Fig. 5 also includes a metadata encoder 544, which provides an implementation of the box encapsulator 154. In other implementations, the box encapsulator 154 may be external to the tensor encoder 116. Fig. 18 shows a method 1800 for performing an NN part 1, encoding an indication of the codec used for the tensor compression stage, i.e., 116, and encoding resulting compressed tensors along with compressed video and anindication of the codec used for the compressed video. The tensors are related to the video as outlined in relation to operation of the video encoder 150 and the tensor encoder 116. The method 1800 may be implemented as one or more software application programs 233 executable within the computer system 200. The method 1800 may be effected by instructions 231 (see Fig. 2B) in the software 233 that are carried out within the computer system 200. The software instructions 231 may be formed as one or more code modules, each for performing one or more particular tasks. The method 1800 commences at an encode video frame step 1810.[000104] At the step 1810 the video encoder 150, under execution of the processor 205, encodes the frame data 113 to produce the video bitstream 151. Example operation of an encoder is described in relation to Fig. 8. Control in the processor 205 progresses from the step 1810 to a perform neural network first portion step 1820.[000105] At the step 1820 the NN part 1 114, under execution of the processor 205, performs the first portion of a neural network using frame data 113 from the video source 112 as input, producing the tensors 115 as output. The tensors 115 are generated by performing at least one convolution on the video data 113. Examples of operation of the NN part 1 114 are described in relation to Figs. 3 and 4. Control in the processor 205 progresses from the step 1820 to a perform tensor compression step 1830.[000106] At the step 1830, the tensor encoder 116 operates to compress the tensors 115. Fig. 5 shows an example implementation 500 of the tensor encoder 116. In the example of Fig. 5, at the step 1830, a compressor 530, under execution of the processor 205, compresses the tensors 115 to produce compressed tensors 532. The compressed tensors 532 are the same or fewer in number than the tensors 115 and reduced in dimensionality (i.e., reduced in either or both of channel count and feature map width and height). The tensor compressor may perform dimensionality reduction using a set of trained network layers, as described with reference to Fig. 6 or using methods based on low-rank approximations of the tensors 115, as described with reference to Fig. 7. Further example operation of the tensor encoder 116 is described in relation to Fig. 5. Control in the processor 205 progresses from the step 1830 to a write file header step 1840.[000107] At the step 1840, the metadata encoder 544 encodes a movie header MovieHeaderBox 11214 into a presentation 11200, as shown in Fig. 1 IB. The movie header 11214 includes creation and modification times for the presentation and the next trackID for the track to be added to the presentation, typically a video track associated with a feature track for the system 100. Control in the processor 205 progresses from the step 1840 to an encode video frame codec indication step 1850.[000108] At the step 1850, the metadata encoder 544 encodes a track TrackBox 11218 and all child boxes as described with reference to Fig. 1 IB. The TrackBox 11218 and child boxes are encoded such that the video codec used to encode the video layer bitstream 151 is indicated by virtue of the type of the child boxes within a sample table SampleTableBox 11230, i.e., boxes 11234a and 11234b which indicate use of the VVC codec in Fig. 1 IB. Other blockbased codecs, such as AVC or HEVC, may be indicated for use in storing samples of the video data. The step 1850 can be considered to encode first information into the bitstream, the first information being used to determine a first codec for the video. Control in the processor 205 progresses from the step 1850 to an encode track grouping for video track step 1855.[000109] At the step 1855 the metadata encoder 544 encodes a VideoFeatureGroupBox 11225, which is an instance of a VideoFeatureGroupBox (track group type set to ‘vftg’), which extends TrackGroupTypeBox and indicates that the containing track is one of two tracks associated with each other as a video track and a feature track. A flag, such as a ‘FeatureStreamFlag’ in the VideoFeatureGroupBox indicate that the track containing this instance of the VideoFeatureGroupBox is a video track (flag value of zero). Control in the processor 205 progresses from the step 1855 to a write encoded video frame step 1860.[000110] At the step 1860, the box encapsulator 154 inserts the video bitstream 151 into a media box MediaBox 11260 in the presentation 11200 of Fig. 1 IB. NAL units of the video bitstream 151 are separately stored such that each one is addressable by sample entries in a sample description box, such as SampleDescriptionBox 11230. Control in the processor 205 progresses from the step 1860 to an encode feature frame codec indication step 1870.[000111] At the step 1870, the metadata encoder 544 encodes a track TrackBox 11238 and all child boxes as described with reference to Fig. 1 IB such that the feature codec used for the feature bitstream 121 is indicated by virtue of the type of the child boxes within a sample table SampleTableBox 11240. Boxes 11242a and 11242b indicate a codec, such as VVC or a specific end-to-end learned codec, used for feature compression. The step 1870 can be considered to encode second information into the bitstream, the second information being used to determine a second codec for the tensors, the second codec capable of decoding tensors generated by a neural network. The first information is encoded into a first box in the bitstreamat step 1850 and the second information is encoded into a second box different from the first box in the bitstream at step 1870. The first information defines the first codec used to decode the video from the bitstream and the second information defines the second codec used to decode the tensor from the bitstream.[000112] As the feature track is associated, or references, the video track a TrackReferenceBox 11223a (type 'tref) is included in the TrackBox 11238. A TrackReferenceTypeBox indicates the type of reference via a ‘reference type’ uint32 (unsigned integer 32), which may be an auxiliary ‘auxl’ value or another custom defined reference type indicative of the nature of the track as a feature track, such as ‘axft’, shown as 11223b. Within the TrackReferenceTypeBox 11223b the track id of the video track 11218 is included. Due to the use of independent codecs for the video track and the feature track, and the nature of the feature data, there is no coding dependency of the feature track on the video track, notwithstanding the presence of the TrackReferenceBox 11223a in the feature stream TrackBox 11238.[000113] Control in the processor 205 progresses from the step 1870 to an encode track grouping for feature track step 1875.[000114] At the step 1875, the metadata encoder 544 encodes a VideoFeatureGroupBox 11241, which is an instance of a VideoFeatureGroupBox (track group type set to ‘vftg’), which extends TrackGroupTypeBox and indicates that the containing track is one of two tracks associated with each other as a video track and a feature track. A flag, such as a ‘FeatureStreamFlag’ in the VideoFeatureGroupBox indicates that this track containing that instance of the VideoFeatureGroupBox is a feature track (flag value of one). Control in the processor 205 progresses from the step 1875 to a write encoded feature frame step 1880.[000115] At the step 1880, the box encapsulator 154 inserts the feature bitstream 121 into the MediaBox 11260 in the presentation 11200. Compressed data corresponding to tensors for a given frame, which may be in the form of NAL units, are separately stored such that each one is addressable by sample entries in a SampleDescriptionBox 11240.[000116] The box encapsulator 154 uses the result of operations of steps 1840 to 1880 to form the box-encapsulated bitstream 155 including both the feature bitstream 121 and the video bitstream 151.[000117] The step 1860 operates to encode the video into the bitstream 155 according to the first codec encoded at step 1850. The step 1880 operates to encode the tensors 121 into the bitstream 155 according to the codec encoded at step 1870. The first information encoded at step 1850 and the second information encoded at 1870 are independent of each other and the video and the tensors are associated with each other at step 1855 and 1875 respectively.[000118] The method 1800 terminates and processing in the processor 205 progresses to the next instance of the source data 113 (e.g., the next frame from the video source 112).[000119] Fig. 6 is a schematic block diagram 600 showing one type of multi-scale feature fusion (MSFF) module , which may serve as the tensor compressor 530. The MSFF module 600 takes the tensors 115 and produces a compressed tensor 532, having reduced dimensionality compared to the tensors 115 and thus resulting in a reduction in bitrate when encoded as part of a packed frame. The MSFF module 600 uses trained network layers and requires a corresponding module in the tensor decoder 146 to restore tensor dimensionality so the tensors 149 may be supplied to the CNN head 150. The MSFC module 600 takes four tensors as input and requires each one the tensors to have two-hundred and fixty-six (256) channels, so that the MSFF module 600 is compatible with the P-layers of the FasterRCNN or MaskRCNN networks. However, variants of the MSFF module 600 compatible with different numbers of layers and different channel counts are possible.[000120] The MSFF module 600 produces one tensor 532 as output with sixty-four (64) channels and a feature map size corresponding to the P5 layer seen at the input, however variants with different channel count are also possible. Each variant of the MSFF module 600 requires different weights to be used for proper operation.[000121] The MSFC module 600 includes an MSFF block 610 shown in Fig. 6, which produces a single tensor 629 from the plurality of tensors 115 using one or more downsampling filters. The MSFF block 610, under execution of the processor 205, combines each tensor of a first set of tensors (i.e., 602, 603, 604, 605), to produce the combined tensor 629. The combined tensor 629 forms a representation of the FPN layer tensors. Downsample modules 622a, 622b, and 622c operate on the tensors having larger spatial scale, i.e., P4 604 at (2h, 2w, 256), and P3 603 at (4h, 4w, 256), and P2 602 at (8h, 8w, 256), respectively. Modules 622a, 622b, and 622c perform downsampling to match the spatial scale of the smallest tensor, i.e., P5 605 at (h, w, 256), producing downscaled P5 tensors 623a, 623b, 623c, respectively. A concatenation module 624 performs a channel-wise concatenation of the tensors 605, 623a, 623b, and 623c toproduce concatenated tensor 625, of dimensions (h, w, 1024). The concatenated tensor 625 is passed to a squeeze and excitation (SE) module 626 to produce a tensor 627. The SE module 626 sequentially performs a global pooling, a fully-connected layer with reduction in channel count, a rectified linear unit activation unit, a second fully-connected layer restoring the channel count, and a sigmoid activation function to produce a scaling tensor. The tensor 625 is scaled according to the scaling tensor to produce the output as the tensor 627. The SE block 626 is capable of being trained to adaptively alter the weighting of different channels in the tensor passed through, based on a first fully-connected layer output.[000122] The first fully-connected layer output reduces each feature map for each channel to a single value. Each single value is passed through a non-linear activation unit (ReLU) to create a conditional representation of the single value, suitable for weighting of other channels, with restoration to the full channel count performed by the second fully-connected layer. The SE block 626 is thus capable of extracting non-linear inter-channel correlation in producing the tensor 627 from the tensor 625, to a greater extent than is possible purely with convolutional (linear) layers. The tensor 627 is passed to a convolutional layer 628. The convolutional layer 628 implements one or more convolutional layers to produce the combined tensor 629, with channel count reduced to F channels, typically 256 channels (i.e., F = 256). Further reduction in the channel count is achieved by a single-scale feature compression (SSFC) module 650.[000123] The SSFC module 650 receives the tensor 629 and applies a convolution 652 to reduce the channel count from F (256) down to C’ (nominally set to 64 channels) to produce tensor 653. The tensor 653 is then passed to a batch normalisation module 654 to produce batch normalised tensor 655, which is passed to a hyperbolic tangent activation layer 656 to produce the compressed tensor 532. The output of the MSFC module 600 is a one tensor per frame with a fixed feature map size and fixed channel count.[000124] Fig. 7 is a schematic block diagram showing an example inter-channel decorrelationbased tensor compressor 700, which may serve as the tensor compressor 530. The tensor compressor 700 operates without need for any trained layers and relies upon extracting interchannel redundancy at runtime (i.e., using the incoming tensor data from the CNN backbone 114 rather than relying upon pretrained weights). The tensor compressor 700 outputs three types of data: mean, basis vectors, and coefficients. The tensor compressor 700 compresses one tensor and multiple instances of the tensor compressor 700 may be instantiated as the tensor compressor 530 to compress multiple tensors. When multiple instances of the tensor compressor 700 are instantiated, tensors of the same type from each instance may bepacked into the same region. For example, basis vectors from each instance may be packed into a single region. These types of data are held in tensors and packed into three separate regions within a frame. The number of basis vectors to be used may be varied during the course of processing frames and the number of coefficients used may also be varied, although the number of coefficients must not exceed the number of basis vectors. As such, dimensionality of the basis vectors and / or coefficients may be updated from time to time to signal to the decoder the contents of a decoded packed frame. As shown in Fig. 7, an extract channel mean module 710, under execution of the processor 205, performs an average operation on a tensor of the tensors 115 across the spatial dimensions (i.e., width and height) to produce a per-channel mean 712 as a ID (one-dimensional) tensor of C (channel count of a tensor of the tensors 115). The extract channel mean module 710 may operate less frequently than on each received tensors 115, in which case the channel mean 712 is updated only on specific frames. The packed frames containing an updated channel mean 712 may be indicated via a nonzero channel count for the tensor 712 via tensor information 1195 or via a separate tensor update flag included in the SEI message 1113 (see Fig. 11 A).[000125] For each channel in a tensor of the tensors 115, a DC shift is performed by a subtraction module 714 that subtracts a value, constant for the feature map and obtained from the mean channel 712, from each spatial location in the feature map. The subtracted value is a respective value in the mean reconstructed channel 712. As a result of the subtraction module 714, a zero-centred tensor 716 is output with DC component found within each feature map removed from the respective feature map.[000126] A decomposition module 718, under execution of the processor 205, operates to produce a set of basis vectors 720 for a tensor of the tensors 115. The decomposition module 718 receives the zero-centred tensor 716 as an input and generates a set of basis vectors 720 by performing a principal component analysis (PC A) method, such as singular value decomposition (SVD) or the like. One basis vector maps all channels onto a single value using a dot product operation, so with two-hundred and fifty-six (256) channels in a tensor of the tensors 115, one basis vector has dimensions 256x 1. If the decomposition module 718 produces the first N basis vectors, such as twenty-five (25), the result basis vectors have dimensions 256xN or 256x25. Basis vectors are relative to the origin point and so the zeromean tensor 712 is used to ensure an orthonormal basis can be found. Each basis vector is a vector relating all the channels to a reduced set of channels. As such, the basis vectors collectively enable representing tensor data spanning all channels into a smaller set of basis vectors. Each basis vector is derived with all samples in each feature map for a given channelbeing considered. The vectors 720 contain fewer basis vectors than there are channels in a tensor of the tensors 115, corresponding to a reduction in the dimensionality of a tensor of the tensors 115. The basis vectors 720 represent a tensor of the tensors 115 in a subspace that accounts for, or ‘explains’, the maximum amount of variance in a tensor of the tensors 115 for the number of components in the basis vectors 720. Basis vectors are ordered from the vector with the greatest explained variance down to the vector with the least explained variance. In other words, the basis vectors 720 enable representation of a tensor of the tensors 115 with minimal degradation in quality for a given number of components, with the components being the first N ranked basis vectors.[000127] As shown in Fig. 7, a dot product module 722, under execution of the processor 205, performs a dot product of each channel in the tensor 716 against each basis vector 720 to produce a coefficients tensor 724. The coefficients form a tensor 724 having the same width and height as a tensor of the tensors 115 but a channel count corresponding, at most, to the number of components (or basis vectors) produced by the decomposition module 718, which is fewer than the number of channels in a tensor of the tensors 115. The tensor compressor 700 may vary the number of channels in the coefficients tensor 716 for a given frame. The number of channels present in in the coefficients tensor 716 for a given frame is included in the tensor information 1195, coded in an instance of the SEI message 1113 associated with the given frame. Each value in the coefficients tensor 724 represents the contribution of each basis vector in reproducing each value in each feature map. The tensors 724, 720 and 712 provide the compressed tensor 532.[000128] Returning to Fig. 5, a quantiser module 534 of the encoder 116, under execution of the processor 205, quantises floating-point values in each tensor of the compressed tensors 532 to produce quantised compressed tensors 536. The quantised compressed tensors 536 have integer values and occupy a range within the sample range as defined by the operational bit depth of a video encoder 542. For example, when encoding video using 10-bit samples, integer values in the interval [0, 1023] are permitted. For each tensor, the minimum and maximum floating-point values form a quantisation range 526 provided to the metadata encoder 544.[000129] A packer module 538, under execution of the processor 205, packs feature maps of each tensor of the compressed quantised tensors 536 into the video frame 540. Feature maps of a tensor are generally packed in left-to-right then top-to-bottom manner into a region in the feature frame 540 to which the tensor is assigned. A video encoder 542, under execution of the processor 205, compresses the feature frame 540 to produce the feature bitstream 121.[000130] Fig. 8 is a schematic block diagram showing functional modules of a video encoder 800. The video encoder 800 provides an example implementation of each of the video encoders 542 and 150. The video encoder 800 may be implemented using a general-purpose computer system 200, as shown in Figs. 2A and 2B, where the various functional modules may be implemented by dedicated hardware within the computer system 200, by software executable within the computer system 200 such as one or more software code modules of the software application program 233 resident on the hard disk drive 205 and being controlled in its execution by the processor 205. Alternatively, the video encoder 800 may be implemented by a combination of dedicated hardware and software executable within the computer system 200. The video encoder 800 and the described methods may alternatively be implemented in dedicated hardware, such as one or more integrated circuits performing the functions or sub functions of the described methods. Such dedicated hardware may include graphic processing units (GPUs), digital signal processors (DSPs), application-specific standard products (ASSPs), application-specific integrated circuits (ASICs), FPGAs or one or more microprocessors and associated memories. In particular, the video encoder 800 comprises modules 810-890 which may each be implemented as one or more software code modules of the software application program 233.[000131] Although the video encoder 542 of Fig. 8 is an example of a versatile video coding (WC) video encoder, other video codecs may also be used to perform the processing stages described herein and the video frame codec indication encoded at step 1850. For example, HEVC may be used. The examples described generate a bitstream of encoded data. If other codecs were used, some implementations may pack data into a different format such as a frame format or the like. The video encoder 800 receives frame data 802, each frame including one or more colour channels. The frame data 802 may be in any chroma format and bit depth supported by the profile in use, for example 4:0:0, 4:2:0 for the “Main 10” profile of the VVC standard, at eight (8) to ten (10) bits in sample precision. The frame data my relate to the video data 113 of the tensor 540. The frame data 802 is also characterised by a ‘level’, specifying aspects such as the maximum luma sample rate, frame aspect ratio constraints, and slice and tile count limits. Worst-case compressed bitrates are also indicated by a combination of the ‘level’ and a ‘tier’ parameter.[000132] As seen in Fig. 8, a block partitioner 810 firstly divides the frame data 802 into CTUs, generally square in shape and configured such that a particular size for the CTUs is used. The maximum enabled size of the CTUs may be 32x32, 64x64, or 128x 128 luma samples forexample, configured by a ‘sps_log2_ctu_size_minus5’ syntax element present in the ‘sequence parameter set’. The CTU size also provides a maximum CU size, as a CTU with no further splitting will contain one CU. The block partitioner 810 further divides each CTU into one or more CBs according to a luma coding tree and a chroma coding tree. The luma channel may also be referred to as a primary colour channel. Each chroma channel may also be referred to as a secondary colour channel. The CBs have a variety of sizes, and may include both square and non-square aspect ratios. However, in the VVC standard, CBs, CUs, PUs, and TUs always have side lengths that are powers of two. Thus, a current CB, represented as 812, is output from the block partitioner 810, progressing in accordance with an iteration over the one or more blocks of the CTU, in accordance with the luma coding tree and the chroma coding tree of the CTU.[000133] The CTUs resulting from the first division of the frame data 802 may be scanned in raster scan order and may be grouped into one or more ‘slices’. A slice may be an ‘intra’ (or ‘I’) slice. An intra slice (I slice) indicates that every CU in the slice is intra predicted. Generally, the first picture in a coded layer video sequence (CLVS) contains only I slices, and is referred to as an ‘intra picture’. The CLVS may contain periodic intra pictures, forming ‘random access points’ (i.e., intermediate frames in a video sequence upon which decoding can commence). Alternatively, a slice may be uni- or bi-predicted (‘P’ or ‘B’ slice, respectively), indicating additional availability of uni- and bi-prediction in the slice, respectively.[000134] The video encoder 800 encodes sequences of pictures according to a picture structure. One picture structure is Tow delay’, in which case pictures using inter-prediction may only reference pictures occurring previously in the sequence. Low delay enables each picture to be output as soon as the picture is decoded, in addition to being stored for possible reference by a subsequent picture. Another picture structure is ‘random access’, whereby the coding order of pictures differs from the display order. Random access allows inter-predicted pictures to reference other pictures that, although decoded, have not yet been output. A degree of picture buffering is needed so the reference pictures in the future in terms of display order are present in the decoded picture buffer, resulting in a latency of multiple frames.[000135] When a chroma format other than 4:0:0 is in use, in an I slice, the coding tree of each CTU may diverge below the 64^64 level into two separate coding trees, one for luma and another for chroma. Use of separate trees allows different block structure to exist between luma and chroma within a luma 64x64 area of a CTU. For example, a large chroma CB may becollocated with numerous smaller luma CBs and vice versa. In a P or B slice, a single coding tree of a CTU defines a block structure common to luma and chroma. The resulting blocks of the single tree may be intra predicted or inter predicted.[000136] In addition to a division of pictures into slices, pictures may also be divided into ‘tiles’. A tile is a sequence of CTUs covering a rectangular region of a picture. CTU scanning occurs in a raster-scan manner within each tile and progresses from one tile to the next. A slice can be either an integer number of tiles, or an integer number of consecutive rows of CTUs within a given tile.[000137] For each CTU, the video encoder 800 as seen in Fig. 8 operates in two stages. In the first stage (referred to as a ‘search’ stage), the block partitioner 810 tests various potential configurations of a coding tree. Each potential configuration of a coding tree has associated ‘candidate’ CBs. The first stage involves testing various candidate CBs to select CBs providing relatively high compression efficiency with relatively low distortion. The testing generally involves a Lagrangian optimisation whereby a candidate CB is evaluated based on a weighted combination of rate (i.e., coding cost) and distortion (i.e., error with respect to the input frame data 712). ‘Best’ candidate CBs (i.e., the CBs with the lowest evaluated rate / distortion) are selected for subsequent encoding into a bitstream portion 816. Included in evaluation of candidate CBs is an option to use a CB for a given area or to further split the area according to various splitting options and code each of the smaller resulting areas with further CBs, or split the areas even further. As a consequence, both the coding tree and the CBs themselves are selected in the search stage.[000138] The video encoder 800 produces a prediction block (PB), indicated by an arrow 820, for each CB, for example, CB 812. The PB 820 is a prediction of the contents of the associated CB 812. A subtracter module 822 produces a difference, indicated as 824 (or ‘residual’, referring to the difference being in the spatial domain), between the PB 820 and the CB 812. The difference 824 is a block-size difference between corresponding samples in the PB 820 and the CB 812. The difference 824 is transformed, quantised and represented as a transform block (TB), indicated by an arrow 836. The PB 820 and associated TB 836 are typically chosen from one of many possible candidate CBs, for example, based on evaluated cost or distortion.[000139] A candidate coding block (CB) is a CB resulting from one of the prediction modes available to the video encoder 800 for the associated PB and the resulting residual. Whencombined with the predicted PB in the video encoder 800, the TB 836 reduces the difference between a decoded CB and the original CB 812 at the expense of additional signalling in a bitstream.[000140] Each candidate coding block (CB) (i.e., prediction block (PB) in combination with a transform block (TB)), has an associated coding cost (or ‘rate’) and an associated difference (or ‘distortion’). The distortion of the CB is typically estimated as a difference in sample values, such as a sum of absolute differences (SAD), a sum of squared differences (SSD) or a Hadamard transform applied to the differences. The estimate resulting from each candidate PB may be determined by a mode selector 886 using the difference 824 to determine a prediction mode 887. The prediction mode 887 indicates the decision to use a particular prediction mode for the current CB, for example, intra-frame prediction or inter-frame prediction. Estimation of the coding costs associated with each candidate prediction mode and corresponding residual coding may be performed at significantly lower cost than entropy coding of the residual.Accordingly, a number of candidate modes may be evaluated to determine an optimum mode in a rate-distortion sense even in a real-time video encoder.[000141] Determining a preferred mode in terms of rate-distortion is typically achieved using a variation of Lagrangian optimisation. Lagrangian or similar optimisation processing can be employed to both select a preferred partitioning of a CTU into CBs (by the block partitioner 810) as well as the selection of a prediction mode from a plurality of possibilities. Through application of a Lagrangian optimisation process of the candidate modes in the mode selector module 886, the intra prediction mode with the lowest cost measurement is selected as a ‘best’ mode. The lowest cost mode includes a selected secondary transform index 888, which is also encoded in the bitstream 816 by an entropy encoder 838.[000142] In the second stage of operation of the video encoder 800 (referred to as a ‘coding’ stage), an iteration over the determined coding tree(s) of each CTU is performed in the video encoder 800. For a CTU using separate trees, for each 64^64 luma region of the CTU, a luma coding tree is firstly encoded followed by a chroma coding tree. Within the luma coding tree, only luma CBs are encoded and within the chroma coding tree only chroma CBs are encoded. For a CTU using a shared tree, a single tree describes the CUs (i.e., the luma CBs and the chroma CBs) according to the common block structure of the shared tree.[000143] The entropy encoder 838 supports bitwise coding of syntax elements using variablelength and fixed-length codewords, and an arithmetic coding mode for syntax elements. Portions of the bitstream such as ‘parameter sets’, for example, the sequence parameter set (SPS), the picture parameter set (PPS), and the picture header (PH) use a combination of fixed- length codewords and variable-length codewords. Slices, also referred to as contiguous portions, have a slice header that uses variable length coding followed by slice data, which uses arithmetic coding. The slice header defines parameters specific to the current slice, such as slice-level quantisation parameter offsets, and may include an instance of the PH. The slice data includes the syntax elements of each CTU in the slice. Use of variable length coding and arithmetic coding requires sequential parsing within each portion of the bitstream. The portions may be delineated with a start code to form ‘network abstraction layer units’ or ‘NAL units’. Arithmetic coding is supported using a context-adaptive binary arithmetic coding process.[000144] Arithmetically coded syntax elements consist of sequences of one or more ‘bins’. Bins, like bits, have a value of ‘0’ or ‘ 1’ . However, bins are not encoded in the bitstream portion 816 as discrete bits. Bins have an associated predicted (or ‘likely’ or ‘most probable’) value and an associated probability, known as a ‘context’. When the actual bin to be coded matches the predicted value, a ‘most probable symbol’ (MPS) is coded. Coding a most probable symbol is relatively inexpensive in terms of consumed bits in the bitstream portion 816, including costs that amount to less than one discrete bit. When the actual bin to be coded mismatches the likely value, a ‘least probable symbol’ (LPS) is coded. Coding a least probable symbol has a relatively high cost in terms of consumed bits. The bin coding techniques enable efficient coding of bins where the probability of a ‘0’ versus a ‘ 1’ is skewed. For a syntax element with two possible values (i.e., a ‘flag’), a single bin is adequate. For syntax elements with many possible values, a sequence of bins is needed. The convention for converting values of a syntax element into a sequence of bins is termed ‘binarisation’. Where the values ‘0’ and ‘ 1’ for a bin are equally (or near equally) likely, it is possible to omit use of a context and assume an equiprobable distribution. Bins with a context are termed ‘context-coded bins’ and bins omitting a context are termed ‘bypass-coded bins’. The binarization of a syntax element into one or more bins may result in a combination of context-coded and bypass-coded bins. Unlike directly coding one bit into the bitstream, a bypass-coded bin uses the arithmetic coding engine, which facilitates mixing context-coded and bypass-coded bins into syntax element binarisations.[000145] For a given binarization, the presence of later bins in the sequence may be determined based on the value of earlier bins in the sequence, resulting in variable-length binarisations. Additionally, each bin may be associated with more than one context, with one context selected for use in coding a specific instances of the bin. The selection of a particular context may be dependent on earlier bins in the syntax element, the decoded values of neighbouring syntax elements (i.e., those from neighbouring blocks) and the like. Each time a context-coded bin is encoded, the context that was selected for that bin (if any) is updated in a manner reflective of the new bin value. As such, the binary arithmetic coding scheme is said to be adaptive.[000146] The absence of a context for bypass-coded bins saves memory and reduces complexity, and thus bypass bins are used where the distribution of values for the particular bin is not skewed. One example of an entropy coder employing context and adaption is known in the art as CABAC (context adaptive binary arithmetic coder) and many variants of this coder have been employed in video coding.[000147] A QP controller 890 determines a quantisation parameter 892, used to establish a quantisation step size for use by a quantiser 834 and a dequantiser 840. A larger quantisation step size results in primary transform coefficients 828 being quantised into smaller values, reducing bitrate of the bitstream portion 816 at the expense of a reduction in the fidelity of inverse transform coefficients 846.[000148] The entropy encoder 838 encodes the quantisation parameter 892 and, if in use for the current CB, the LFNST index 888, using a combination of context-coded and bypass-coded bins. The quantisation parameter 892 is encoded at the beginning of each slice and changes in the quantisation parameter 892 within a slice are coded using a ‘delta QP’ syntax element. The delta QP syntax element is signalled at most once in each area known as a ‘quantisation group’. The quantisation parameter 892 is applied to residual coefficients of the luma CB. An adjusted quantisation parameter is applied to the residual coefficients of collocated chroma CBs. The adjusted quantisation parameter may include mapping from the luma quantisation parameter 892 according to a mapping table and a CU-level offset, selected from a list of offsets. The secondary transform index 888 is signalled when the residual associated with the transform block includes significant residual coefficients only in those coefficient positions subject to transforming into primary coefficients by application of a secondary transform.[000149] Residual coefficients of each TB associated with a CB are coded using a residual syntax. The residual syntax is designed to efficiently encode coefficients with low magnitudes, using mainly arithmetically coded bins to indicate significance of coefficients, along with lower-valued magnitudes and reserving bypass bins for higher magnitude residual coefficients. Accordingly, residual blocks comprising very low magnitude values and sparse placement of significant coefficients are efficiently compressed. Moreover, two residual coding schemes are present. A regular residual coding scheme is optimised for TBs with significant coefficients predominantly located in the upper-left corner of the TB, as is seen when a transform is applied. A transform-skip residual coding scheme is available for TBs where a transform is not performed and is able to efficiently encode residual coefficients regardless of their distribution throughout the TB.[000150] A multiplexer module 884 outputs the PB 820 from an intra-frame prediction module 864 according to the determined best intra prediction mode, selected from the tested prediction mode of each candidate CB. The candidate prediction modes need not include every conceivable prediction mode supported by the video encoder 800. Intra prediction falls into three types, first, “DC intra prediction”, which involves populating a PB with a single value representing the average of nearby reconstructed samples; second, “planar intra prediction”, which involves populating a PB with samples according to a plane, with a DC offset and a vertical and horizontal gradient being derived from nearby reconstructed neighbouring samples. The nearby reconstructed samples typically include a row of reconstructed samples above the current PB, extending to the right of the PB to an extent and a column of reconstructed samples to the left of the current PB, extending downwards beyond the PB to an extent; and, third, “angular intra prediction”, which involves populating a PB with reconstructed neighbouring samples filtered and propagated across the PB in a particular direction (or ‘angle’). In VVC, sixty-five (65) angles are supported, with rectangular blocks able to utilise additional angles, not available to square blocks, to produce a total of eighty-seven (87) angles.[000151] A fourth type of intra prediction is available to chroma PBs, whereby the PB is generated from collocated luma reconstructed samples according to a ‘cross-component linear model’ (CCLM) mode. Three different CCLM modes are available, each mode using a different model derived from the neighbouring luma and chroma samples. The derived model is used to generate a block of samples for the chroma PB from the collocated luma samples. Luma blocks may be intra predicted using a matrix multiplication of the reference samples using one matrix selected from a predefined set of matrices. This matrix intra prediction (MIP)achieves gain by using matrices trained on a large set of video data, with the matrices representing relationships between reference samples and a predicted block that are not easily captured in angular, planar, or DC intra prediction modes.[000152] The module 864 may also produce a prediction unit by copying a block from nearby the current frame using an ‘intra block copy’ (IBC) method. The location of the reference block is constrained to an area equivalent to one CTU, divided into 64x64 regions known as VPDUs, with the area covering the processed VPDUs of the current CTU and VPDUs of the previous CTU(s) within each row or CTUs and within each slice or tile up to the area limit corresponding to one 128x 128 luma samples, regardless of the configured CTU size for the bitstream. This area is known as an ‘IBC virtual buffer’ and limits the IBC reference area, thus limiting the required storage. The IBC buffer is populated with reconstructed samples 854 (i.e., prior to loop filtering), and so a separate buffer to a frame buffer 872 is needed. When the CTU size is 128x 128 the virtual buffer includes samples only from the CTU adjacent and to the left of the current CTU. When the CTU size is 32x32 or 64x64 the virtual buffer includes CTUs from up to the four or sixteen CTUs to the left of the current CTU. Regardless of the CTU size, access to neighbouring CTUs for obtaining samples for IBC reference blocks is constrained by boundaries such as edges of pictures, slices, or tiles. Particularly for feature maps of FPN layers having smaller dimensions, use of a CTU size such as 32x32 or 64x64 results in a reference area more aligned to cover a set of previous feature maps. Where feature map placement is ordered based on SAD, SSE or other difference metric, access to similar feature maps for IBC prediction offers coding efficient advantage.[000153] The residual for a predicted block when encoding feature map data is different to the residual seen for natural video. Natural video is typically captured by an image sensor, or screen content, as generally seen in operating system user interfaces and the like. Feature map residuals tend to contain much detail. The level of detail in feature map residuals is amenable to transform skip coding more than predominantly low-frequency coefficients of various transforms. An intra-predicted luma coding block may be partitioned into a set of equal-sized prediction blocks, either vertically or horizontally, which each block having a minimum area of sixteen (16) luma samples.[000154] Where previously reconstructed neighbouring samples are unavailable, for example at the edge of the frame, a default half-tone value of one half the range of the samples is used. For example, for 10-bit video a value of five-hundred and twelve (512) is used. As no previoussamples are available for a CB located at the top-left position of a frame, angular and planar intra-prediction modes produce the same output as the DC prediction mode (i.e., a flat plane of samples having the half-tone value as magnitude).[000155] For inter-frame prediction a prediction block 882 is produced using samples from one or two frames preceding the current frame in the coding order frames in the bitstream by a motion compensation module 880 and output as the PB 820 by the multiplexer module 884. Moreover, for inter-frame prediction, a single coding tree is typically used for both the luma channel and the chroma channels. The order of coding frames in the bitstream may differ from the order of the frames when captured or displayed. When one frame is used for prediction, the block is said to be ‘uni-predicted’ and has one associated motion vector. When two frames are used for prediction, the block is said to be ‘bi-predicted’ and has two associated motion vectors. For a P slice, each CU may be intra predicted or uni -predicted. For a B slice, each CU may be intra predicted, uni -predicted, or bi-predicted.[000156] Frames are typically coded using a ‘group of pictures’ structure, enabling a temporal hierarchy of frames. Frames may be divided into multiple slices, each of which encodes a portion of the frame. A temporal hierarchy of frames allows a frame to reference a preceding and a subsequent picture in the order of displaying the frames. The images are coded in the order necessary to ensure the dependencies for decoding each frame are met. An affine inter prediction mode is available where instead of using one or two motion vectors to select and filter reference sample blocks for a prediction unit, the prediction unit is divided into multiple smaller blocks and a motion field is produced so each smaller block has a distinct motion vector. The motion field uses the motion vectors of nearby points to the prediction unit as ‘control points’. Affine prediction allows coding of motion different to translation with less need to use deeply split coding trees. A bi-prediction mode available to VVC performs a geometric blend of the two reference blocks along a selected axis, with angle and offset from the centre of the block signalled. This geometric partitioning mode (“GPM”) allows larger coding units to be used along the boundary between two objects, with the geometry of the boundary coded for the coding unit as an angle and centre offset. Motion vector differences, instead of using cartesian (x, y) offset, may be coded as a direction (up / down / left / right) and a distance, with a set of power-of-two distances supported. The motion vector predictor is obtained from a neighbouring block (‘merge mode’) as if no offset is applied. The current block will share the same motion vector as the selected neighbouring block.[000157] The samples are selected according to a motion vector 878 and reference picture index. The motion vector 878 and reference picture index applies to all colour channels and thus inter prediction is described primarily in terms of operation upon Pus rather than PBs. The decomposition of each CTU into one or more inter-predicted blocks is described with a single coding tree. Inter prediction methods may vary in the number of motion parameters and their precision. Motion parameters typically comprise a reference frame index, indicating which reference frame(s) from lists of reference frames are to be used plus a spatial translation for each of the reference frames, but may include more frames, special frames, or complex affine parameters such as scaling and rotation. In addition, a pre-determined motion refinement process may be applied to generate dense motion estimates based on referenced sample blocks.[000158] Having determined and selected the PB 820 and subtracted the PB 820 from the original sample block at the subtractor 822, a residual with lowest coding cost, represented as 824, is obtained and subjected to lossy compression. The lossy compression process comprises the steps of transformation, quantisation and entropy coding. A forward primary transform module 826 applies a forward transform to the difference 824, converting the difference 824 from the spatial domain to the frequency domain, and producing primary transform coefficients represented by an arrow 828. The largest primary transform size in one dimension is either a 32-point DCT-2 or a 64-point DCT-2 transform, configured by a ‘sps_max_luma_transform_size_64_flag’ in the sequence parameter set. If the CB being encoded is larger than the largest supported primary transform size expressed as a block size (e.g. 64x64 or 32x32), the primary transform 826 is applied in a tiled manner to transform all samples of the difference 824. Where a non-square CB is used, tiling is also performed using the largest available transform size in each dimension of the CB. For example, when a maximum transform size of thirty -two (32) is used, a 64 x 16 CB uses two 32x 16 primary transforms arranged in a tiled manner. When a CB is larger in size than the maximum supported transform size, the CB is filled with TBs in a tiled manner. For example, a 128x 128 CB with 64-pt transform maximum size is filled with four 64x64 TBs in a 2x2 arrangement. A 64x 128 CB with a 32-pt transform maximum size is filled with eight 32x32 TBs in a 2x4 arrangement.[000159] Application of the transform 826 results in multiple TBs for the CB. Where each application of the transform operates on a TB of the difference 824 larger than 32x32, e.g., 64x64, all resulting primary transform coefficients 828 outside of the upper-left 32x32 area of the TB are set to zero (i.e., discarded). The remaining primary transform coefficients 828 arepassed to the quantiser module 834. The primary transform coefficients 828 are quantised according to the quantisation parameter 892 associated with the CB to produce primary transform coefficients 832. In addition to the quantisation parameter 892, the quantiser module 834 may also apply a ‘scaling list’ to allow non-uniform quantisation within the TB by further scaling residual coefficients according to their spatial position within the TB. The quantisation parameter 892 may differ for a luma CB versus each chroma CB. The primary transform coefficients 832 are passed to a forward secondary transform module 830 to produce the transform coefficients represented by the arrow 836 by performing either a non-separable secondary transform (NSST) operation or bypassing the secondary transform. The forward primary transform 826 is typically separable, transforming a set of rows and then a set of columns of each TB. The forward primary transform module 826 uses either a type-II discrete cosine transform (DCT-2) in the horizontal and vertical directions, or bypass of the transform horizontally and vertically, or combinations of a type- VII discrete sine transform (DST-7) and a type- VIII discrete cosine transform (DCT-8) in either horizontal or vertical directions for luma TBs not exceeding 16 samples in width and height. Use of combinations of a DST-7 and DCT- 8 is referred to as ‘multi transform selection set’ (MTS) in the VVC standard.[000160] The forward secondary transform of the module 830 is generally a non-separable transform, which is only applied for the residual of intra-predicted CUs and may nonetheless also be bypassed. The forward secondary transform operates either on sixteen (16) samples (arranged as the upper-left 4x4 sub-block of the primary transform coefficients 828) or fortyeight (48) samples (arranged as three 4x4 sub-blocks in the upper-left 8x8 coefficients of the primary transform coefficients 828) to produce a set of secondary transform coefficients. The set of secondary transform coefficients may be fewer in number than the set of primary transform coefficients from which they are derived. Due to application of the secondary transform to only a set of coefficients adjacent to each other and including the DC coefficient, the secondary transform is referred to as a Tow frequency non-separable secondary transform’ (LFNST). Such secondary transforms may be obtained through a training process and due to their non-separable nature and trained origin, exploit additional redundancy in the residual signal not able to be captured by separable transforms such as variants of DCT and DST. Moreover, when the LFNST is applied, all remaining coefficients in the TB are zero, both in the primary transform domain and the secondary transform domain.[000161] The quantisation parameter 892 is constant for a given TB and thus results in a uniform scaling for producing residual coefficients in the primary transform domain for a TB.The quantisation parameter 892 may vary periodically with a signalled ‘delta quantisation parameter’. The delta quantisation parameter (delta QP) is signalled once for Cus contained within a given area, referred to as a ‘quantisation group’. If a CU is larger than the quantisation group size, delta QP is signalled once with one of the TBs of the CU. That is, the delta QP is signalled by the entropy encoder 838 once for the first quantisation group of the CU and not signalled for any subsequent quantisation groups of the CU. A non-uniform scaling is also possible by application of a ‘quantisation matrix’, whereby the scaling factor applied for each residual coefficient is derived from a combination of the quantisation parameter 892 and the corresponding entry in a scaling matrix. The scaling matrix may have a size that is smaller than the size of the TB, and when applied to the TB a nearest neighbour approach is used to provide scaling values for each residual coefficient from a scaling matrix smaller in size than the TB size. The residual coefficients 836 are supplied to the entropy encoder 838 for encoding in the bitstream portion 816. Typically, the residual coefficients of each TB with at least one significant residual coefficient of the TU are scanned to produce an ordered list of values, according to a scan pattern. The scan pattern generally scans the TB as a sequence of 4^4 ‘subblocks’, providing a regular scanning operation at the granularity of 4x4 sets of residual coefficients, with the arrangement of sub-blocks dependent on the size of the TB. The scan within each sub-block and the progression from one sub-block to the next typically follow a backward diagonal scan pattern. Additionally, the quantisation parameter 892 is encoded into the bitstream portion 816 using a delta QP syntax element, and a slice QP for the initial value in a given slice or subpicture and the secondary transform index 888 is encoded in the bitstream portion 816.[000162] As described above, the video encoder 800 needs access to a frame representation corresponding to the decoded frame representation seen in the video decoder. Thus, the residual coefficients 836 are passed through an inverse secondary transform module 844, operating in accordance with the secondary transform index 888 to produce intermediate inverse transform coefficients, represented by an arrow 842. The intermediate inverse transform coefficients 842 are inverse quantised by the dequantiser module 840 according to the quantisation parameter 892 to produce inverse transform coefficients, represented as 846. The dequantiser module 840 may also perform an inverse non-uniform scaling of residual coefficients using a scaling list, corresponding to the forward scaling performed in the quantiser module 834. The inverse transform coefficients 846 are passed to an inverse primary transform module 848 to produce residual samples, represented by an arrow 850, of the TU. The inverse primary transform module 848 applies DCT-2 transforms horizontally and vertically,constrained by the maximum available transform size as described with reference to the forward primary transform module 826. The types of inverse transform performed by the inverse secondary transform module 844 correspond with the types of forward transform performed by the forward secondary transform module 830. The types of inverse transform performed by the inverse primary transform module 848 correspond with the types of primary transform performed by the primary transform module 826. A summation module 852 adds the residual samples 850 and the PB 820 to produce reconstructed samples (indicated by an arrow 854) of the CU.[000163] The reconstructed samples 854 are passed to a reference sample cache 856 and an inloop filters module 868. The reference sample cache 856, typically implemented using static RAM on an ASIC to avoid costly off-chip memory access, provides minimal sample storage needed to satisfy the dependencies for generating intra-frame PBs for subsequent CUs in the frame. The minimal dependencies typically include a Tine buffer’ of samples along the bottom of a row of CTUs, for use by the next row of CTUs and column buffering the extent of which is set by the height of the CTU. The reference sample cache 856 supplies reference samples (represented by an arrow 858) to a reference sample filter 860. The sample filter 860 applies a smoothing operation to produce filtered reference samples (indicated by an arrow 862). The filtered reference samples 862 are used by the intra-frame prediction module 864 to produce an intra-predicted block of samples, represented by an arrow 866. For each candidate intra prediction mode the intra-frame prediction module 864 produces a block of samples, that is 866. The block of samples 866 is generated by the module 864 using techniques such as DC, planar or angular intra prediction. The block of samples 866 may also be produced using a matrix-multiplication approach with neighbouring reference sample as input and a matrix selected from a set of matrices by the video encoder 800, with the selected matrix signalled in the bitstream 816 using an index to identify which matrix of the set of matrices is to be used by the video decoder.[000164] The in-loop filters module 868 applies several filtering stages to the reconstructed samples 854. The filtering stages include a ‘deblocking filter’ (DBF) which applies smoothing aligned to the CU boundaries to reduce artefacts resulting from discontinuities. Another filtering stage present in the in-loop filters module 868 is an ‘adaptive loop filter’ (ALF), which applies a Wiener-based adaptive filter to further reduce distortion. A further available filtering stage in the in-loop filters module 868 is a ‘sample adaptive offset’ (SAO) filter. The SAOfilter operates by firstly classifying reconstructed samples into one or multiple categories and, according to the allocated category, applying an offset at the sample level.[000165] Filtered samples, represented by an arrow 870, are output from the in-loop filters module 868. The filtered samples 870 are stored in the frame buffer 872. The frame buffer 872 typically has the capacity to store several (e.g., up to sixteen (16)) pictures and thus is stored in the memory 206. The frame buffer 872 is not typically stored using on-chip memory due to the large memory consumption required. As such, access to the frame buffer 872 is costly in terms of memory bandwidth. The frame buffer 872 provides reference frames (represented by an arrow 874) to a motion estimation module 876 and the motion compensation module 880. The reference frames 874 are output as a reconstructed frame 818 of the encoder module 542. In the example of Fig. 8, the reconstructed frame 818 is a result of operation of lossy VVC encoding, that is due to operation of the modules 810 to 890.[000166] The motion estimation module 876 estimates a number of ‘motion vectors’ (indicated as 878), each being a Cartesian spatial offset from the location of the present CB, referencing a block in one of the reference frames in the frame buffer 872. A filtered block of reference samples (represented as 882) is produced for each motion vector. The filtered reference samples 882 form further candidate modes available for potential selection by the mode selector 886. Moreover, for a given CU, the PU 820 may be formed using one reference block (‘uni -predicted’) or may be formed using two reference blocks (‘bi-predicted’). For the selected motion vector, the motion compensation module 880 produces the PB 820 in accordance with a filtering process supportive of sub-pixel accuracy in the motion vectors. As such, the motion estimation module 876 (which operates on many candidate motion vectors) may perform a simplified filtering process compared to that of the motion compensation module 880 (which operates on the selected candidate only) to achieve reduced computational complexity. When the video encoder 800 selects inter prediction for a CU the motion vector 878 is encoded into the bitstream portion 816.[000167] Although the video encoder 800 of Fig. 8 is described with reference to versatile video coding (VVC), other video coding standards or implementations may also employ the processing stages of modules 810-890. The frame data 802 (and bitstream 816) may also be read from (or written to) memory 206, the hard disk drive 210, a CD-ROM, a Blu-ray disk™ or other computer readable storage medium. Additionally, the frame data 802 (and bitstream 816) may be received from (or transmitted to) an external source, such as a server connected to thecommunications network 220 or a radio-frequency receiver. The bitstream 816 relates to one of the bitstreams 121 and 151. The communications network 220 may provide limited bandwidth, necessitating the use of rate control in the video encoder 800 to avoid saturating the network at times when the frame data 802 is difficult to compress.[000168] The bitstream 816 may be constructed from one or more slices, representing spatial sections (collections of CTUs) of the frame data 802, produced by one or more instances of the video encoder 800, each producing a bitstream portion 816 and operating in a co-ordinated manner under control of the processor 205. The bitstream portion 816 may also contain one slice that corresponds to one region to be output as a collection of subpictures forming one picture, each being independently encodable and independently decodable with respect to any of the other slices or subpictures in the picture.[000169] Figs. 9A & 9B are a schematic block diagrams showing a division of a picture into regions. Fig. 9A shows a picture 900 divided into regions 910, 912, and 916, suitable for packing data from the inter-channel decorrelation (PCA-based) encoder network topology 700. Feature maps of each tensor are packed by the packer 538. The example of Figs. 9A and 9B shows three regions corresponding to mean channel, basis vectors, and coefficients.[000170] Referring to Fig. 9B, picture 900b corresponds to the picture 900, region 910b corresponds to 910 and 912b corresponds to 912. The region 910b holds mean channel data, such as mean channel 920 for each tensor of the tensors 115. In the example of Fig. 9B the basis vectors include basis vector 922 amongst others, with the basis vectors packed into the area of the region 912b in a non-overlapping manner. The basis vectors 922 form an CxC’ array of integer-quantised values, where C is the number of channels in a tensor of the tensors 115 and C’ is the number of basis vectors resulting from the decomposition performed by the decomposition module 718. Where multiple tensors are decomposed into sets of basis vectors, such as when the tensors 115 comprise a plurality of tensors, the basis vectors 922 include each set of basis vectors arranged in a non-overlapping left-aligned vertical concatenation. The mean channel 920 corresponds to the channel 712 and the basis vectors 922 correspond to some of the basis vectors 720, for example.[000171] Area in the picture 900 (or 900b) that is not used to store any data may be occupied by sample values corresponding to the value ‘0’ after application of inverse quantisation to convert from sample values back to the floating-point domain. Similarly, area in regions 910 (or 910b), 912 (or 912b), 916 that is not used to store any data, may be occupied by samplevalues corresponding to the value ‘0’ after application of inverse quantisation to convert from sample values back to the floating-point domain. The region 916 holds coefficient tensor data corresponding to 724 of Fig. 7, with coefficient for each basis vector forming a width by height feature map. The coefficient region 916 has coefficients packed as, e.g., twenty-four feature maps (applicable to basis vectors [0..23 ]), such as feature map 930.[000172] Each region (910 / 910b, 912 / 912b, and 916 / 916b) is aligned to the CTU grid, normally 128x 128. The location of each region is specified as the top-left and bottom-right cartesian CTU addresses, when addressed using cartesian co-ordinates in units of CTU size. For example, region 916 occupies CTUs from CTU at (0,1), corresponding to top-left luma sample at location (0, 128) down to the CTU at (6,3), corresponding to bottom right luma sample location at (6x 128 + 127, 3 x 128 + 127) or (895, 511).[000173] Figs. 10A & 10B are schematic block diagrams showing a division of a picture into one region. In Fig.lOA a picture 1000 is divided into one region 1010, suitable for packing the compressed tensor 532 as may be produced by the tensor compressor 600. Fig. 10B shows a picture 1000b, corresponding to the picture 1000 in which feature maps are packed into a region 1010b corresponding to 1010. Each feature map of the compressed tensor, such as feature map 1030, is packed in the region 1010b, such that feature maps are packed in a left-to-right manner, progressing to the next row once available space on the current row has been exhausted. By virtue of the tensor compressor 600, the required area to pack the compressed tensor 532 is smaller than the required area that would be required were each tensor of the tensors 115 packed into the picture 1000. The region 1010 is defined based on top-left and bottom-right cartesian CTU addresses.[000174] Fig. 11 A is a schematic block diagram showing an example structure 1100 of the bitstreams 121 and 151 holding encoded packed feature maps and associated metadata and compressed video frames. The bitstreams 121 and 151 contains groups of syntax elements each prefaced by a ‘network abstraction layer’ (NAL) unit header. For example, a NAL unit header precedes a sequence parameter set (SPS) 1110a[000175] An SPS 1110b includes signalling of the bit-depth, chroma format, and resolution of the feature picture (e.g., the picture 900 or the picture 1000) for the feature layer 121, depending on the selected compressor and decompressor at the step 1810.[000176] In applications involving streaming retransmissions of the FCVCM SEI 1113, the SPS 1110a and 1110b may be performed to enable decoding to commence at an intermediate point in the bitstream. The SPS 1110a also indicates the chroma format, the bit depth, the resolution of the frame data represented by the bitstream 121. A coded subpicture 1122 encoding subpicture 912, includes a slice header 1130 followed by slice data 1140. The slice data 1140 includes a sequence of CTUs, providing the coded representation of the frame data. Coded subpicture 1124 encodes subpicture 916, corresponding to the coefficients. The bitstream 123 includes the FCVCM decoder network indication SEI message 1113, written by the metadata encoder 544. The SEI message 1113 encodes metadata needed to convert a decoded frame into a set of tensors suitable for use by the NN part 2 166.[000177] Fig. 1 IB is a schematic block diagram showing a hierarchical arrangement of ‘boxes’ resulting in the presentation 11200, which encapsulates a video stream and a feature stream, that is, bitstreams 151 and 121, respectively to provide the box-encapsulated bitstream 155. The presentation 11200 extends the ISOBMFF specification (ISO / IEC 14496-12) of the year 2022 and the carriage of structured NAL unit data in ISOBMFF specification (ISO / IEC 14496- 15) of the year 2022.[000178] One instance of a MovieBox 11210 (type ‘moov’) or a CompressedMovieBox (type ‘moov’) is present in the presentation 11200, identified via ‘moov’. One instance of the MovieHeaderBox 11214 (type ‘mvhd’) is present in the MovieBox 11210.[000179] One or more instances of a TrackBox (type ‘trak’), such as a TrackBox 11218, are present in the MovieBox 11210. The TrackBox 11218 holds information relating to the video bitstream 151. A TrackBox 11238 holds information relating to the feature bitstream 121. Other Trackboxes may also be present, such as for audio tracks. Each TrackBox contains one TrackHeaderBox (type ‘tkhd’) , such as a TrackHeaderBox 11222. For the video track, presentation to the consumer of the video is intended or expected and so a ‘track in movie’ flag in the TrackHeaderBox 11222 is set to one.[000180] For the feature stream 121, the ‘track in movie’ flag in a corresponding TrackHeaderBox, contained within the Track 11238, is set to zero as the feature track is not directly expected to be presented to a consumer of the video. Each TrackBox includes one SampleTableBox, such as SampleTableBox 11226, containing time and data indexing for samples in the track. Each TrackBox may also include one SyncSamplesBox, such as SyncSamplesBox 11228, which provides a list of sample indices which are CRA (clean randomaccess) or IDR (instantaneous decoder refresh) points into the video sequence. In other words, each SyncSamplesBox contains a list of entry points at which decoding can commence in the corresponding track.[000181] Each SampleTableBox includes one SampleDescriptionBox (type ‘stsd’), such as the SampleDescriptionBox 11230, which defines the coding type used and where to locate the corresponding coded video sequence in one or more contained boxes of a class derived from an abstract SampleEntry class. A VvcSampleEntry (type ‘wcl’) sample box is a child of a ‘Visual Sampl eEntry’ box type, which is a child of a SampleEntry box type. Each VvcSampleEntry instance includes a VvcConfigurationBox (identified via ‘vvcC’). The ‘compressor name’ field resulting from Visual Sampl eEntry is set as ‘VVC Coding’ (with length of 10 bytes). This hierarchy permits samples to be categorised firstly as video, audio, subtitle, other, categories, and within each category for the specific codec used to be identified, such as AVC (Advanced Video Coding), HEVC (High Efficiency Video Coding), or VVC (Versatile Video Coding). The ‘SampleEntry’ class is ‘abstract’, meaning instances of SampleEntry box are not permitted, however child classes of SampleEntry may be instantiated and thus may be present in the presentation 11200. Boxes 11242a and 11242b are child types of a ‘Feature Stream Sampl eEntry’ class, which is a child of the SampleEntry class. When the features are to be compressed using VVC, a ‘ VvcFeatureSampleEntry’ (type ‘vvcf ) box type is used for the boxes 11242a and 11242b. Other child types of Feature Stream Sampl eEntry are used for other feature compression schemes. For example, if a learned end-to-end compression scheme is used for feature compression, an EndToEndFeature Sampl eEntry (type ‘e2ef ) box type may be used.[000182] A TrackGroupBox, such as TrackGroupBox 11224 or 11239 (type ‘trgr’), with one instance in each track, is used to associate multiple tracks with each other with each instance containing an instance of a child class of TrackGroupTypeBox, indicating a type of the track group via ‘track group type’. For a grouped video track and feature track the track group type may be set to ‘vfgr’ with the video track and the feature track having the same value for track group id. A separate track group id value enables multiple groups and two tracks are only associated if their respective VideoFeatureGroupBox instances have the same value for track group id.[000183] A MediaDataBox 11260 (type ‘mdat’) contained in the presentation 11200 contains the actual compressed NAL unit data for VCL-layer and non-VCL-layer NAL units of the videotrack and the feature track. NAL units contained in the MediaDataBox are addressed by the boxes within the SampleDescriptionBox of each track.[000184] Fig. 19 shows a method 1900 for decoding the bitstream 123, reconstructing tensors using a tensor decompressor, and performing an NN part 2 to complete execution of a neural network. As described, the method 1900 is configured for determining a set of compatible NN part 2 candidates from a library of available NN part 2 implementations in the destination device 140 and selecting one of the compatible NN part 2 candidates for execution in the destination device 140. The bitstream 123 includes a video track (extractable as bitstream 171) and a feature track (extractable as bitstream 145), each of which are decoded to produce video frames 172 and decoded tensors 149 respectively.[000185] The destination device 140 and the method 1900 may be implemented as one or more software application programs 233 executable within the computer system 200. The destination device 140 and the method 1900 may be effected by instructions 231 (see Fig. 2B) in the software 233 that are carried out within the computer system 200. The software instructions 231 may be formed as one or more code modules, each for performing one or more particular tasks. The bitstream 123 can comprise tensors (feature stream 121) and video (video stream 151) among other information for decoding, the tensors being related to the video, as described hereinbefore. The method 1900 begins at a decode file header step 1910.[000186] Fig. 12 is a schematic block diagram showing an implementation 1200 of the tensor decoder 146. The decoder 1200 has a metadata decoder 1230, a picture decoder 1204, an unpacker 1214, an inverse quantiser 1218, a tensor storage module 1222 and a tensor decompressor 1250. In the arrangements described, the metadata decoder 1230 provides an implementation of the box extractor 144. In other implementations, the box extractor 144 may be external to the tensor decoder 146. At the step 1910 the metadata decoder 1230 decodes the MovieHeaderBox 11214 from the presentation 11200, indicating various creation and modification time information for the presentation and a track ID for the main (video) track for presentation to an end user. Control in the processor 205 progresses from the step 1910 to a decode video track codec indication step 1920.[000187] At the step 1920 the metadata decoder 1230 decodes the TrackBox 11218 and all child boxes as described with reference to Fig. 1 IB such that the video codec used is indicated by virtue of the type of the child boxes within a sample table SampleTableBox 11230, i.e., boxes 11234a and 11234b which indicate use of a VVC codec in the example of Fig. 1 IB. Inother implementations, other types of codes may be decoded, for example, HEVC and AVC codecs. By decoding the TrackBox 11218, step 1920 operates to decode information from the bitstream 123. A type of codec for the video frames can be determined based on the decoded information. The TrackBox 11218 can be considered first information and may be in integer format or another format. Control in the processor 205 progresses from the step 1920 to a decode video track grouping indication step 1930.[000188] At the step 1930 the metadata decoder 1230 decodes theVideoFeatureGroupBox 11225, which is an instance of a VideoFeatureGroupBox(track group type set to ‘vftg’). The VideoFeatureGroupBox 11225extends TrackGroupTypeBox and indicates that the containing track is one of two tracks associated with each other as a video track and a feature track. A flag, such as a ‘FeatureStreamFlag’ in the VideoFeatureGroupBox can indicate that the track containing this instance of the VideoFeatureGroupBox is a video track (flag value of zero). Decoding the video track grouping indication by decoding the VideoFeatureGroupBox 11225 identifies the portions of the bitstream 143 which form the video bitstream 171 (output from the box extractor 144). Control in the processor 205 progresses from the step 1930 to a decode video frame step 1940.[000189] Step 1920 can be considered to define a first codec used to decode the video from the bitstream 123. At the step 1940, the video decoder 170, under execution of the processor 205, decodes a frame from the video bitstream 171 according to the codec determined based on step 1920. The video decoder 170 uses the codec type or indication as determined at the step 1920 to decode a frame from the bitstream 171 and produce the decoded frame 172. The video decoder 170 may implement a VVC decoder as described with reference to Fig. 13 or an HEVC decoder. Control in the processor 205 progresses from the step 1940 to a decode feature track codec indication step 1950.[000190] At the step 1950 the metadata decoder 1230 decodes the track TrackBox 11238 and all child boxes as described with reference to Fig. 1 IB such that the feature codec used for the feature bitstream 121 is indicated by virtue of the type of the child boxes within a sample table SampleTableBox 11240. Boxes 11242a and 11242b indicate a codec, such as VVC or a specific end-to-end learned codec, used for feature compression. By decoding theTrackBox 11238, step 1920 operates to decode information from the bitstream 123. A type of codec for decoding the tensors (i.e., feature data) can be determined based on the decoded information. The TrackBox 11238 information can be considered second information and maybe in integer format or another format. The codec determined from operation of step 1950 is capable of decoding tensors / features generated a neural network, for example the features of the feature stream 121 generated by the neural network part 1 114. The information decoded at step 1950 is included independently in the bitstream 123 to the information decoded at the step 1920 and both sets of information can be independent of each other. In other words, the codecs used for the video and feature streams can be different. The information at each of steps 1920 and 1950 is decoded from first and boxes respectively, the first and second boxes being different to one another. The metadata decoder 1230 also decodes the TrackReferenceBox 11223a and a TrackReferenceTypeBox 11223b contained in the box 11223a. The TrackReferenceTypeBox 11223a has a reference type indicative of the auxiliary nature of the feature tracl, such as ‘auxl’ and may further indicate the track as a feature track, such as using a reference type ‘axft’, although a different four character code could be used for the same purpose. The track id of the TrackBox 11218 is also decoded from the box 11223b. Control in the processor 205 progresses from the step 1950 to a decode feature track group indication step 1960.[000191] At the step 1960 the metadata decoder 1230 decodes the VideoFeatureGroupBox 11241, which is an instance of a VideoFeatureGroupBox (track group type set to ‘vftg’). The VideoFeatureGroupBox 11241extends TrackGroupTypeBox and indicates that the containing track is one of two tracks associated with each other as a video track and a feature track. A flag, such as a ‘FeatureStreamFlag’ in the VideoFeatureGroupBox indicates that this track containing that instance of the VideoFeatureGroupBox is a feature track (flag value of one). Decoding the feature track grouping indication by decoding the VideoFeatureGroupBox 11241 identifies the portions of the bitstream 143 which form the feature bitstream 145 (output from the box extractor 144). While the information decoded at step 1920 is independent of the information decoded at step 1950, the video data and the feature data in the bitstream are associated with one another. The video track grouping information decoded at step 1930 and the feature track grouping information decoded at step 1960 allow the video and feature (tensor) data to be associated with one another.[000192] Steps 1930 and 1960 operate to decode information signalling the association between the video and the tensors from the bitstream. The information signalling the association between the video and the tensors is decoded in the example described as (i) information from a first box in the bitstream for video using the first codec at step 1930 and (ii)information from a second box in the bitstream for the tensors using the second codec at step 1960.[000193] Control in the processor 205 progresses from the step 1960 to a decode feature frame step 1970.[000194] Step 1950 can be considered to define the second codec used to decode the tensors from the bitstream. At the step 1970, the picture decoder 1204, under execution of the processor 205, according to the codec determined based on step 1950, operates to decode one packed frame from the feature sub-bitstream 145 to produce a decoded frame 1210. Operation of the picture decoder 1204 is described with reference to Fig. 13. Control in the processor 205 progresses from the step 1970 to an unpack tensors step 1980.[000195] At the step 1980, the unpacker 1214 of the decoder 146, under execution of the processor 205, reads feature maps from the decoded frame 1210 in accordance with a packing format as described with reference to Figs. 9A & 9B and Figs. 10A & 10B. For each tensor, a number of feature maps are decoded, the number corresponding to the number of used channels in the tensor as signalled in the tensor information 1195. The channels for each tensor are unpacked as two-dimensional feature maps. The number of feature maps or channels to decode for a given tensor is decoded from the feature sub-bitstream 145 as a ‘channel count’. A plurality of tensors each having one channel are arranged horizontally or vertically within a slice or subpicture of the frame. The feature maps of each previously determined type are unpacked at step 1980 by the unpacker 1214 using the allocation of the feature maps determined for the frame (e.g., 1210) in the method 1800. The allocation may be decoded from an SEI message. The unpacker 1214 outputs integer tensors 1216, where the tensors 1216 have been decoded using the tensor decoder 146. Control in the processor 205 progresses from the step 1980 to an inverse quantise tensors step 1990.[000196] At the step 1990, the inverse quantiser 1218 of the decoder 1200, under execution of the processor 205, performs inverse quantisation on the integer tensors 1216 to produce inverse quantised tensors 1220 by applying quantisation ranges 1232 decoded from the bitstream 123 by the metadata decoder 1230, such as quantisation ranges 1196, to the determined channel count of each tensor. To perform inverse quantisation, the quantisation ranges 1196 are decoded from the SEI message 1113 by the receiver 142. Control in the processor 205 progresses from the step 1990 to a perform tensor decompression step 19100.[000197] At the step 19100, the tensor decompressor 1250, under execution of the processor 205, decompresses the tensors 1224 as input to produce decoded tensors 149. An example structure of a tensor decompressor is described in relation to Fig. 14. Control in the processor 205 progresses from the step 19100 to a perform neural network second portion step 19110 to generate a task result using a portion of a neural network.[000198] At the step 19110, the software 233, under execution of the processor 205, performs the NN part 2 implemented as NN part 2 166. The module 166 uses the tensors 149 as input and produces the task result 151 as output. Control in the processor 205 progresses from the step 19110 to a render task result step 19120.[000199] At the step 19120 the task result Tenderer 168, under execution of the processor 205 produces a video frame rendering the task result in association with the video layer. For example, the step 168 can present both the video frame 172 and a representation of the task result 151. The task result 151 may be represented as bounding boxes around objects of interest, segmentation map with colour tinting of objects of interest or the like. Alternatively, the video frame 172 can be rendered independently of the contents of the tensors, for example irrespective of whether a task result is present or without visual identifiers of the task result such as bounding boxes, colour mapping or the like. The video frame resulting from the step 19120 can be presented in the user of the destination device 140 as final output of the system 100, for example via the display 214. The method 1900 terminates and the processor 205 progresses to the next frame.[000200] The steps 19100 to 19120 operate to perform a network second portion (NN part 2) to produce a neural network result in a case where steps 1970 and 1980 determine to decode tensors from the bitstream. The method 1900 terminates and the processor 205 may reinvoke the method 1900 upon receiving the next frame in the bitstream 123.[000201] The box extractor 144 may parse the bitstream 123 without knowledge of the specific box structure contained therein. However a bitstream conforming to embodiments described herein contains box types can be anticipated by the method 1900. A generic parser that can parse the box structure may be used, and once parsed the anticipated elements, namely the video track codec indication, the feature track codec indication, and associated grouping information, is required to be present, enforcing the box structure as described with reference to Fig. 1 IB. Bitstreams that do not contain at least the boxes identified in Fig. 1 IB would not berequired to be decoded by the destination device 140. If the feature stream is required but not the video stream or vice versa, the decoded structure would indicate which steps of the method 1900 are to be implemented based on whether feature and / or video results are required.[000202] Fig. 13 is a schematic block diagram showing functional modules of a video decoder 1300. The video decoder provides an example implementation for each of the picture decoder 1204 and the video decoder 170. The video decoder 1300 may be implemented as one or more software application programs 233 executable within the computer system 200. The video decoder 1204 may be effected by instructions 231 (see Fig. 2B) in the software 233 that are carried out within the computer system 200. The software instructions 231 may be formed as one or more code modules, each for performing one or more particular tasks. The example video decoder 1300 is operable to decode both the video bitstream 171 and the feature bitstream 145.[000203] One of the sub-bitstreams 145 or 171 is input as a sub-bitstream 1301 to an entropy decoder module 1320. The entropy decoder module 1320 extracts syntax elements from the sub-bitstream 1301 by decoding sequences of ‘bins’ and passes the values of the syntax elements to other modules in the video decoder 1300. The entropy decoder module 1320 uses variable-length and fixed length decoding to decode SPS, PPS or slice header using an arithmetic decoding engine to decode syntax elements of the slice data as a sequence of one or more bins. Each bin may use one or more ‘contexts’, with a context describing probability levels to be used for coding a ‘one’ and a ‘zero’ value for the bin. Where multiple contexts are available for a given bin, a ‘context modelling’ or ‘context selection’ step is performed to choose one of the available contexts for decoding the bin. The process of decoding bins forms a sequential feedback loop, where each slice may be decoded in entirety by a given entropy decoder 1320 instance.[000204] The entropy decoder module 1320 applies an arithmetic coding algorithm, for example ‘context adaptive binary arithmetic coding’ (CAB AC), to decode syntax elements from the input bitstream 145 or 171 (1301). The decoded syntax elements are used to reconstruct parameters within the video decoder 1300. Parameters include residual coefficients (represented by an arrow 1324), a quantisation parameter 1374, a secondary transform index 1370, and mode selection information such as an intra prediction mode (represented by an arrow 1358). The mode selection information also includes information such as motion vectors, and the partitioning of each CTU into one or more CBs.Parameters are used to generate PBs, typically in combination with sample data from previously decoded CBs.[000205] The residual coefficients 1324 are passed to an inverse secondary transform module 1336 where either a secondary transform is applied or no operation is performed (bypass) according to a secondary transform index. The inverse secondary transform module 1336 produces reconstructed transform coefficients 1332. That is, the module 1336 produces primary transform domain coefficients from secondary transform domain coefficients. The reconstructed transform coefficients 1332 are input to a dequantiser module 1328. The dequantiser module 1328 performs inverse quantisation (or ‘scaling’) on the residual coefficients 1332, that is, in the primary transform coefficient domain, to create reconstructed intermediate transform coefficients, represented by an arrow 1340, according to the quantisation parameter 1374. The dequantiser module 1328 may also apply a scaling matrix to provide non-uniform dequantization within the TB, corresponding to operation of the dequantiser module 840. Should use of a non-uniform inverse quantisation matrix be indicated in the sub-bitstream 1301, the video decoder 1300 reads a quantisation matrix from the received bitstream 145 or l71 (1301) as a sequence of scaling factors and arranges the scaling factors into a matrix. The inverse scaling uses the quantisation matrix in combination with the quantisation parameter to create the reconstructed intermediate transform coefficients 1340.[000206] The reconstructed transform coefficients 1340 are passed to an inverse primary transform module 1344. The module 1344 transforms the coefficients 1340 from the frequency domain back to the spatial domain. The inverse primary transform module 1344 applies inverse DCT-2 transforms horizontally and vertically, constrained by the maximum available transform size as described with reference to the forward primary transform module 826. The result of operation of the module 1344 is a block of residual samples, represented by an arrow 1348. The block of residual samples 1348 is equal in size to the corresponding CB. The residual samples 1348 are supplied to a summation module 1350.[000207] At the summation module 1350, the residual samples 1348 are added to a decoded PB (represented as 1352) to produce a block of reconstructed samples, represented by an arrow 1356. The reconstructed samples 1356 are supplied to a reconstructed sample cache 1360 and an in-loop filtering module 1388. The in-loop filtering module 1388 produces reconstructed blocks of frame samples, represented as 1392. The frame samples 1392 are written to a frame buffer 1396. The frame buffer 1396 outputs image or video frames 1397.The frames 1397 correspond to the frames 172 if the sub-bitstream 1301 is the sub-bitstream 171. The frames 1397 correspond to the frames 1204 if the sub-bitstream 1301 is the subbitstream 145.[000208] The reconstructed sample cache 1360 operates similarly to the reconstructed sample cache 856 of the video encoder 800. The reconstructed sample cache 1360 provides storage for reconstructed samples needed to intra predict subsequent CBs without the memory 206 (e.g., by using the data 232 instead, which is typically on-chip memory). Reference samples, represented by an arrow 1364, are obtained from the reconstructed sample cache 1360 and supplied to a reference sample filter 1368 to produce filtered reference samples indicated by arrow 1372. The filtered reference samples 1372 are supplied to an intra-frame prediction module 1376. The module 1376 produces a block of intra-predicted samples, represented by an arrow 1380, in accordance with the intra prediction mode parameter 1358 signalled in the subbitstream 1301 and decoded by the entropy decoder 1320. The intra prediction module 1376 supports the modes of the encoder-side module 864, including IBC and MIP. The block of samples 1380 is generated using modes such as DC, planar or angular intra prediction.[000209] When the prediction mode of a CB is indicated to use intra prediction in the subbitstream 1301 (145 or 171), the intra-predicted samples 1380 form the decoded PB 1352 via a multiplexor module 1384. Intra prediction produces a prediction block (PB) of samples, which is a block in one colour component, derived using ‘neighbouring samples’ in the same colour component. The neighbouring samples are samples adjacent to the current block and by virtue of being preceding in the block decoding order have already been reconstructed. Where luma and chroma blocks are collocated, the luma and chroma blocks may use different intra prediction modes. However, the two chroma CBs share the same intra prediction mode.[000210] When the prediction mode of the CB is indicated to be inter prediction in the subbitstream 1301 (145 or 171), a motion compensation module 1334 produces a block of interpredicted samples, represented as 1338. The block of inter-predicted samples 1338 are produced using a motion vector, decoded from the sub-bitstream 145 or 171 by the entropy decoder 1320, and reference frame index to select and filter a block of samples 1398 from the frame buffer 1396. The block of samples 1398 is obtained from a previously decoded frame stored in the frame buffer 1396. For bi-prediction, two blocks of samples are produced and blended together to produce samples for the decoded PB 1352. The frame buffer 1396 is populated with filtered block data 1392 from an in-loop filtering module 1388. As with the in-loop filtering module 868 of the video encoder 542, the in-loop filtering module 1388 applies any of the DBF, the ALF and SAO filtering operations. Generally, the motion vector is applied to both the luma and chroma channels, although the filtering processes for sub-sample interpolation in the luma and chroma channel are different.[000211] Fig. 14 is a schematic block diagram showing an implementation of an inter-channel decorrelation-based tensor decoder 1400, which may serve as the tensor decompressor 1250. When multiple tensors are to be decompressed, multiple instances of the tensor decoder 1400 may be instantiated as the tensor decompressor 1250. Tensors 1224 are received from the tensor storage 1222 and include a coefficients tensor 1414, a basis vectors tensor 1412, and a channel mean tensor 1410. A zero-centred tensor 1422 is produced by a dot product module 1420, under execution of the processor 205, by performing a dot product on the coefficients tensor 1414 and the basis vectors 1412. A summation module 1424 adds the zero- centred tensor 1422 with the mean channel 1410 to produce a tensor of the reconstructed tensors 149 as output from the tensor decoder 1400. When one type of input, such as the mean, is not required to be updated, the channel count of that type of input may be set to zero. When the channel count of an input is set to zero, the most recently received value for that input may be used instead.[000212] Fig. 15 is a schematic block diagram showing a tensor decompressor 1500 using a multi-scale feature reconstruction stage, which may be used in some implementations as the tensor decompressor 1250. The tensor decompressor 1500 includes a single-scale feature compression (SSFC) decompressor 1510. The SSFC decompressor 1510 receives the tensor 1224 having a reduced channel count, such as 64 channels, and passes the tensor 1224 to a convolution layer 1512, which outputs a tensor 1513 having a restored channel count, such as 256 channels. The tensor 1513 is passed to a batch normalisation module 1514 to produce a tensor 1515. The tensor 1515 is passed to a PreLU module 1516 to produce a tensor 1520.[000213] The tensor decompressor 1500 includes a MSFR module 1530. The MSFR module operates to produce a plurality of tensors from the tensor 1520 produced by execution of step 19100, described with reference to Fig. 19, using one or more trained convolutional layers. Upsample modules 1532, 1534, and 1536 upsample the tensor 1520 horizontally and vertically by factors of two, four, and eight, respectively, to produce tensors 1533, 1535, and 1537. The tensor 1537 forms one (P’2, 1557) output from the MSFR module 1530 and is passed to a downsample module 1542. The downsample module 1542 downsamples the tensor 1537 by a factor of two horizontally and vertically to produce a tensor 1543 having the samedimensionality as the tensor 1535. The tensor 1543 is provided to a convolution layer 1548 which outputs a tensor 1549. A summation module 1554 adds the tensors 1535 and 1549 to produce tensor 1555 as an output (P’3) of the MSFR module 1530.[000214] The tensor 1535 is passed to a downsample module 1540. The downsample module 1540 downsamples the tensor 1535 by a factor of two horizontally and vertically to produce a tensor 1541 having the same dimensionality as the tensor 1533. The tensor 1541 is provided to a convolution layer 1546 which outputs a tensor 1547. A summation module 1552 adds the tensors 1533 and 1547 to produce tensor 1553 as an output (P’4) of the MSFR module 1530.[000215] The tensor 1533 is passed to a downsample module 1538. The downsample module 1538 downsamples the tensor 1533 by a factor of two horizontally and vertically to produce a tensor 1539 having the same dimensionality as the tensor 1520. The tensor 1539 is provided to a convolution layer 1544 which outputs a tensor 1545. A summation module 1550 adds the tensors 1520 and 1545 to produce tensor 1551 as an output (P’5) of the MSFR module 1530. The tensors 1551, 1553, 1555 and 1557 provide the tensors 149.[000216] Fig. 16A is a schematic block diagram showing an example implementation 1600 of the NN part 2 166 of a CNN for object detection, corresponding to a portion of a “YOLOv3” network excluding the “DarkNet-53” backbone portion. The NN part 2 1600 of Fig. 16A can be used when the CNN backbone (NN part 1 114) is implemented as in Fig. 3 A for example. Depending on the task to be performed in the destination device 140, different networks may be substituted for the NN part 2 166. Incoming tensors 149 are separated into the tensor of each layer (i.e., tensors 1610, 1620, and 1634). The tensor 1610 is passed to a CBL module 1612 to produce tensor 1614. The tensor 1614 is passed to a detection module 1616 and an upscaler module 1622. The detection module outputs bounding boxes 1618, in the form of a detection tensor. The bounding boxes 1618 are passed to a non -maximum suppression (NMS) module 1648.[000217] To produce bounding boxes addressing co-ordinates in the original video data 113, prior to resizing for the NN part 1 of the network 114, scaling by the original video width and height is performed at the upscaler module 1622. The upscaler module 1622 receives the tensor 1614 and the tensor 1620 and produces an upscaled tensor 1624, which is passed to a CBL module 1626. The CBL module 1626 produces a tensor 1628 as output. The tensor 1628 is passed to a detection module 1630 and an upscaler module 1636. The detection module 1630 produces a detection tensor 1632, which is supplied to the NMS module 1648. The upscalermodule 1636 is another instance of the module 1622. The upscaler module 1636 receives the tensor 1628 and the tensor 1634 and outputs an upscaled tensor 1638. The upscaled tensor 1638 is passed to a CBL module 1640, which outputs a tensor 1642 to a detection module 1644. The detection module 1644 produces a detection tensor 1646, which is supplied to the NMS module 1648.[000218] The CBL modules 1612, 1626, and 1640 each contain a concatenation of five CBL modules (e.g., CBL model 360 shown in Fig. 3D). The upscaler modules 1622 and 1636 are each instances of an upscaler module 1660 as shown in Fig. 16B. The module 1648 receives the tensors 1618, 1632 and 1646 and outputs the task result 151.[000219] As shown in Fig. 16B, the upscaler module 1660 accepts a tensor 1662 (for example the tensor 1614 of Fig. 16A) as an input. The tensor 1662 is passed to a CBL module 1666 (having structure of the module 360) to produce a tensor 1668. The tensor 1668 is passed to an upsampler 1670 to produce an upsampled tensor 1672. A concatenation module 1674 produces a tensor 1676 by concatenating the upsampled tensor 1672 with a second input tensor 1664 (for example the tensor 1620 input to the upscaler 1622 in Fig. 16A).[000220] The detection modules 1616, 1630, and 1644 are instances of a detection module 1680 as shown in Fig. 16C. The detection module 1680 receives a tensor 1682. The tensor 1682 is input to a CBL module 1684 having structure of the module 360. The CBL module 1684 generates a tensor 1686. The tensor 1686 is passed to a convolution module 1688, which implements a detection kernel to generate a result 1690. In some arrangements, the detection kernel applies a 1 x 1 kernel to produce the output on feature maps at each of the three layers of the tensor. The detection kernel is 1 x 1 x (B x (5 + C) ), where B is the number of bounding boxes a particular cell can predict, typically three (3), and C is the number of classes, which may be eighty (80), resulting in a kernel size of two-hundred and fifty five (255) detection attributes (i.e. tensor 1690). The constant “5” represents four boundary box attributes (box centre x, y and size scale x, y) and one object confidence level (“objectness”). The result of a detection kernel has the same spatial dimensions as the input feature map, but the depth of the output corresponds to the detection attributes. The detection kernel is applied at each layer, typically three layers, resulting in a large number of candidate bounding boxes. A process of non-maximum suppression is applied by the NMS module 1648 to the resulting bounding boxes to discard redundant boxes, such as overlapping predictions at similar scale, resulting in a final set of bounding boxes as output for object detection.[000221] Fig. 17 is a schematic block diagram showing a head portion 1700 of a CNN. The head portion 1700 can be implemented as the NN part 2 166 where the NN part 1 114 is implemented as the backbone 400 for example. The head portion 1700 forms part of an overall network known as ‘Faster RCNN’ and includes a feature network (i.e., backbone portion 400), a region proposal network, and a detection network. Input to the head portion 1700 are the tensors 149, which include the P2-P6 layer tensors 1710, 1712, 1714, 1716, and 1718. The P2- P5 layer tensors 1710, 1712, 1714, and 1716, correspond to the P2 to P5 outputs 477, 475, 473, and 471 of Fig. 4. The P2-P6 tensors 1710, 1712, 1714, 1716, and 1718 are input to a region proposal network (RPN) head module 1720. The P6 tensor 1718 is produced by max pool module 1742, operating on P5 tensor 1716 to perform a 2x2 max pooling operation.[000222] The RPN head module 1720 performs a convolution on the input tensors, generating an intermediate tensor. The intermediate tensor is fed into two subsequent sibling layers, (i) one for classifications and (ii) one for bounding box, or ‘region of interest’ (ROI), regression. A resultant output is classification and bounding boxes 1722. The classification and bounding boxes 1722 are passed to an NMS module 1724. The NMS module 1724 prunes out redundant bounding boxes by removing overlapping boxes with a lower score to produce pruned bounding boxes 1726.[000223] The bounding boxes 1726 are input to a region of interest (ROI) pooler 1728. The ROI pooler 1728 uses some of the layer tensors of the tensor 149 (described further hereafter) and the bounding boxes 1726 to produce fixed-size feature maps from various input size maps using max pooling operations. In the max pooling operation a subsampling takes the maximum value in each group of input values to produce one output value in the output tensor.[000224] Input to the ROI pooler 1728 are the P2-P5 feature maps 1710, 1712, 1714, and 1716, and region of interest proposals 1726. Each proposal (ROI) from 1726 is associated with a portion of the feature maps (1710-1716) to produce a fixed-size map. The fixed-size map is of a size independent of the underlying portion of the feature map 1710-1716. One of the feature maps 1710-1716 is selected such that the resulting cropped map has sufficient detail, for example, according to the following rule: floor(4 + log2(sqrt(box_area) / 224)), where 224 is the canonical box size. The ROI pooler 1728 operates to crop incoming feature maps according to the proposals 1726 producing a tensor 1730.[000225] The tensor 1730 is fed into a fully connected (FC) neural network head 1732. The FC head 1732 performs two fully connected layers to produce class score and bounding boxpredictor delta tensor 1734. The class score is generally an 80-element tensor, each element corresponding to a prediction score for the corresponding object category. The bounding box prediction deltas tensor is an 80x4 = 320 element tensor, containing bounding boxes for the corresponding object categories. Final processing is performed by an output layers module 1736, receiving the tensor 1734 and performing a filtering operation to produce a filtered tensor 1738. Low-scoring (low classification) objects are removed from further consideration. A non-maximum suppression module 1740 receives the filtered tensor 1738 and removes overlapping bounding boxes by removing the overlapped box with a lower classification score, resulting in an inference output tensor 1742, corresponding to the tensor 151.[000226] It should be noted that although the source device 110 and the destination device 140 are described with reference to the video source 112 comprising video and image data, other types of content such as audio data or textual data may also be supplied as input to neural networks applicable to such types of input and the resulting intermediate feature maps may be compressed and decompressed by the modules 116 and 146 with suitable encoder and decoder network topologies.[000227] The examples of methods 1800 and 1900 relate to where both a neural network result and the associated video data are required. For example, a human may wish to view the frame(s) relating to a result found by operation of the combined NN part 1 114 and NN part 2 166. For example, if presence of a person has been detected in a security application, a security guard may wish to view the relevant portion of the video. In some implementations, the video stream and associated information may only be encoded in the bitstream 155 (or decoded from the bitstream 143) if required. In other implementations, the feature stream and associated information may only be encoded in the bitstream 155 (or decoded from the bitstream 143) if required.[000228] It should be noted that although the source device 110 and the destination device 140 are described with reference to the video source 112 comprising video and image-related data, other types of content such as audio data or textual data may also be supplied as input to neural networks applicable to such types of input and the resulting intermediate feature maps may be compressed and decompressed by the modules 116 and 146 with suitable encoder and decoder network topologies. Based on the type of information being used, the encoded data may take a different form to a bitstream received from the communications channel 130, for example theencoded data could be obtained from a file server and accessed using operating system ‘read’ and ‘seek’ methods to parse the box structure contained therein.INDUSTRIAL APPLICABILITY[000229] The arrangements described are applicable to the computer and data processing industries and particularly for the digital signal processing for the encoding and decoding of signals such as video and image signals, achieving high compression efficiency.[000230] Some embodiments described herein can include both a video layer and a feature layer in a bitstream, with the layers associated with each other and with the codec selection signalled independently for the video layer and the feature layer. Including both a video layer and a feature layer in a bitstream provides a mechanism by which human may review or interpret a CNN result. Having the codec selection signalled independently for the video layer and the feature layer allows a greater flexibility and adaptability to use different codecs as encoding methods are updated, encoding requirements are altered and / or as industry trends change. In signalling codec selection independently, the arrangements described provide a mechanism by which both feature and video streams can be presented, if required, without relying on a particular encoding standard. The examples described herein relate to video coding standards such as VVC and HEVC, however other encoding methods could potentially be used, for example for the feature stream, such as coding techniques based on end-to-end learned compression techniques.[000231] Some of the techniques described herein use features of the ISO Base Media File Format (ISOBMFF). The box structure of ISOBMFF adapted to allow use of independent codecs for video and feature layers is described herein. Additionally, features such as use of track boxes and track grouping, provide further flexibility in cases where the NN result may also require viewing by a human. Moreover, the disclosed box structure for ISOBMFF may be used by other standards which convey feature and video data, such as those developed by organizations such as 3 GPP.[000232] The foregoing describes only some embodiments of the present invention, and modifications and / or changes can be made thereto without departing from the scope and spirit of the invention, the embodiments being illustrative and not restrictive.In the context of this specification, the word “comprising” means “including principally but not necessarily solely” or “having” or “including”, and not “consisting only of’. Variations of the word "comprising", such as “comprise” and “comprises” have correspondingly varied meanings.

Claims

CLAIMS1. A method of decoding tensors and video from a bitstream, the tensors being related to the video, the method comprising: determining a first codec for the video based on first information decoded from the bitstream; determining a second codec for the tensors based on second information decoded from the bitstream; decoding the video according to the first codec; decoding the tensors according to the second codec; and wherein the first information and the second information are independent of each other and the video and the tensors are associated with each other.

2. The method according to claim 1, wherein the first information is decoded from a first box in the bitstream and the second information is decoded from a second box different from the first box in the bitstream.

3. The method according to claim 1, wherein the video is rendered independently of the contents of the tensors.

4. The method according to claim 1, wherein the first codec is one of an Advanced Video Coding), a HEVC (High Efficiency Video Coding), or a VVC (Versatile Video Coding) codec.

5. The method according to claim 1, wherein the second codec is one of a HEVC (High Efficiency Video Coding), a VVC (Versatile Video Coding) codec, or an end-to-end learned codec.

6. The method according to claim 1, wherein the tensors were generated by performing at least one convolution on the video.

7. The method according to claim 1, further comprising decoding information signalling the association between the video and the tensors from the bitstream.

8. A method according to claim 7, wherein the information signalling the association between the video and the tensors is decoded as (i) information from a first box in the bitstream for video using the first codec and (ii) information from a second box in the bitstream for the tensors using the second codec.

9. A method of encoding tensors and video into a bitstream, the tensors being related to the video, the method comprising: encoding first information into the bitstream, the first information being used to determine a first codec for the video; encoding second information into the bitstream, the second information being used to determine a second codec for the tensors; encoding the video into the bitstream according to the first codec; encoding the tensors into the bitstream according to the second codec; and wherein the first information and the second information are independent of each other and the video and the tensors are associated with each other.

10. The method according to claim 9 wherein the first information is encoded into a first box in the bitstream and the second information is encoded into a second box different from the first box in the bitstream.

11. The method according to claim 9, wherein the first codec is one of an Advanced Video Coding), a HEVC (High Efficiency Video Coding), or a VVC (Versatile Video Coding) codec.

12. The method according to claim 9, wherein the second codec is one of a HEVC (High Efficiency Video Coding), a VVC (Versatile Video Coding) codec, or an end-to-end learned codec.

13. The method according to claim 9, wherein the tensors were generated by performing at least one convolution on the video.

14. A decoder for decoding tensors and video from a bitstream, the tensors being related to the video, the decoder configured for: determining a first codec for the video based on first information decoded from the bitstream;determining a second codec for the tensors based on second information decoded from the bitstream; decoding the video according to the first codec; decoding the tensors according to the second codec; and wherein the first information and the second information are independent of each other and the video and the tensors are associated with each other.

15. A non-transitory computer-readable storage medium which stores a program for executing a method of decoding tensors and video from a bitstream, the tensors being related to the video, the method comprising: decoding tensors and video from a bitstream, the tensors being related to the video, the method comprising: determining a first codec for the video based on first information decoded from the bitstream; determining a second codec for the tensors based on second information decoded from the bitstream; decoding the video according to the first codec; decoding the tensors according to the second codec; and wherein the first information and the second information are independent of each other and the video and the tensors are associated with each other.

16. An encoder for encoding tensors and video into a bitstream, the tensors being related to the video, the encoder configured for: encoding first information into the bitstream, the first information being used to determine a first codec for the video; encoding second information into the bitstream, the second information being used to determine a second codec for the tensors; encoding the video into the bitstream according to the first codec; encoding the tensors into the bitstream according to the second codec; and wherein the first information and the second information are independent of each other and the video and the tensors are associated with each other.

17. A computer-implemented medium non-transitory computer-readable storage medium which stores a program for executing a method of encoding tensors and video into a bitstream, the tensors being related to the video, the method comprising: encoding first information into the bitstream, the first information being used to determine a first codec for the video; encoding second information into the bitstream, the second information being used to determine a second codec for the tensors; encoding the video into the bitstream according to the first codec; encoding the tensors into the bitstream according to the second codec; and wherein the first information and the second information are independent of each other and the video and the tensors are associated with each other.

Images