Method, apparatus and system for encoding and decoding a bitstream

Abstract

45718737_1 Abstract METHOD, APPARATUS AND SYSTEM FOR ENCODING AND DECODING A BITSTREAM A system and method of encoding and decoding a bitstream. The method of encoding (generating) the bitstream comprises encoding, into a first bitstream portion, first information based on one or more tensors; and encoding second information into a second bitstream portion, wherein second information is associated with the first information and is related to a target object. The method generates the bitstream using the first bitstream portion and the second bitstream portion. Abstract METHOD, APPARATUS AND SYSTEM FOR ENCODING AND DECODING A BITSTREAM A system and method of encoding and decoding a bitstream. The method of encoding (generating) the bitstream comprises encoding, into a first bitstream portion, first information based on one or more tensors; and encoding second information into a second bitstream portion, wherein second information is associated with the first information and is related to a target object. The method generates the bitstream using the first bitstream portion and the second bitstream portion. 45718737_1 20 25 20 20 67 21 M ar 2 02 5 A b s t r a c t M a r 2 0 2 5 B I T S T R E A M 2 0 2 5 2 0 2 0 6 7 2 1 1 / 18 45718749_1 Fig. 1 Source device 110 Video source 112 Transmitter 122 Receiver 142 100 130 Storage 132 113 172 143 CNN backbone 114 CNN head 150 Task result buffer 152 151 Feature frame encoder 120 Feature frame decoder 144 119 145 Destination device 140 Bottleneck encoder 116 Bottleneck decoder 148 115 149 Quantise and pack 118 Unpack and inverse quantise 146 117 147 Prompt generator 160 162 Multiplexer 170 121 166 P ro m pt E nc od er 1 64 Feature encoder 168 Demultiplexer 180 182 P ro m pt d ec od er 1 86 184 188 Feature decoder 190 1 / 18 100 Source device Destination device 110 140 Prompt Task result Video source generator buffer 112 160 152 151 162 113 1 / 88 CNN head CNN backbone 150 114 115 149 Feature encoder Bottleneck Bottleneck 168 encoder 116 decoder 148 117 147 Quantise and Feature Unpack and decoder inverse pack 118 190 quantise 146 119 145 Feature Feature frame frame encoder 120 decoder 144 166 172 182 184 Multiplexer 170 Demultiplexer 180 Storage 121 132 143 Transmitter Receiver 122 142 130 Fig. 1 45718749_1 20 25 20 20 67 21 M ar 2 02 5 1 0 0 2 1 M a r 2 0 2 5 S o u r c e d e v i c e D e s t i n a t i o n d e v i c e T a s k r e s u l t V i d e o s o u r c e 1 5 1 2 0 2 5 2 0 2 0 6 7 F e a t u r e B o t t l e n e c k P r o m p t E n c o d e r 1 6 4 1 1 7 1 4 7 F e a t u r e P r o m p t d e c o d e r 1 8 6 1 1 9 1 4 5 F e a t u r e F e a t u r e f r a m e 1 6 6 1 7 2 1 8 4 1 4 3 T r a n s m i t t e r R e c e i v e r 1 3 0

Description

TECHNICAL FIELD

[0001] The present invention relates generally to digital video signal processing and, in particular, to a method, apparatus and system for encoding (generating) and decoding a bitstream. The present invention also relates to a computer program product including a computer readable medium having recorded thereon a computer program for encoding and a bitstream using video compression technology. BACKGROUND

[0002] Convolution neural networks (CNNs) are an emerging technology addressing, among other things, use cases involving machine vision such as object detection, instance segmentation, object tracking, human pose estimation and action recognition. Applications for CNNs can involve use of ‘edge devices’ with sensors and some processing capability, coupled to application servers as part of a ‘cloud’. CNNs can require relatively high computational complexity, more than can typically be afforded either in computing capacity or power consumption by an edge device. Executing a CNN in a distributed manner has emerged as one solution to running leading edge networks using limited capability edge devices. In other words, distributed processing allows legacy edge devices to still provide the capability of leading edge CNNs by distributing processing between the edge device and external processing means, such as cloud servers.

[0003] CNNs typically include many layers, such as convolution layers and fully connected layers, with data passing from one layer to the next in the form of ‘tensors’. Splitting a network across different devices introduces a need to compress the intermediate tensor data that passes from one layer to the next within a CNN, such compression may be referred to as ‘feature compression’, as the intermediate tensor data is often termed ‘features’ of input such as an image frame or video frame. International Organisation for Standardisation / International Electrotechnical Commission Joint Technical Committee 1 / Subcommittee 29 / Working Group 4 (ISO / IEC JTC 1 / SC 29 / WG 4), also known as the “Moving Picture Experts Group” (MPEG) Video is tasked developing a compression standard for features, known as “Feature Coding for Machines” (FCM). Previously, WG 2 ‘MPEG Technical Requirements’ had conducted a Call 2025202067   21 Mar 2025 for Proposals (CfP) on this topic, with several responses showing significantly improved performance than what could be achieved using state-of-the-art standardised technology directly applied to the tensors. A separate activity “Video Coding for Machines” (VCM) is also being conducted in WG 4.

[0004] CNNs require weights for each of the layers to be determined in a training stage, where a very large amount of training data is passed through the CNN and a determined result is compared to ground truth associated with the training data. A process for updating network weights, such as stochastic gradient descent, is applied to iteratively refine the network weights until the network performs at a desired level of accuracy. Where a convolution stage has a ‘stride’ greater than one, an output tensor from the convolution has a lower spatial resolution than a corresponding input tensor. Pooling operations result in an output tensor having smaller dimensions than the input tensor. One example of a pooling operation is ‘max pooling’ (or ‘Maxpool’), which reduces the spatial size of the output tensor compared to the input tensor. Max pooling produces an output tensor by dividing the input tensor into groups of data samples (e.g., a 2x2 group of data samples), and from each group selecting a maximum value as output for a corresponding value in the output tensor. The process of executing a CNN with an input and progressively transforming the input into an output is commonly referred to as ‘inferencing’.

[0005] Generally, a tensor has four dimensions, namely: batch, channels, height, and width. The first dimension, ‘batch’, of size ‘one’ when inferencing on video data indicates that one frame is passed through a CNN at a time. When training a network, the value of the batch dimension may be increased so that multiple frames are passed through the network before the network weights are updated, according to a predetermined ‘batch size’. A multi-frame video may be passed through as a single tensor with the batch dimension increased in size according to the number of frames of a given video. However, for practical considerations relating to memory consumption and access, inferencing on video data is typically performed on a framewise basis. The ‘channels’ dimension indicates the number of concurrent ‘feature maps’ for a given tensor and the height and width dimensions indicate the size of the feature maps at the particular stage of the CNN. Channel count varies through a CNN according to the network architecture. Feature map size also varies, depending on subsampling occurring in specific network layers. 2025202067   21 Mar 2025

[0006] Input to the first layer of a CNN is a batch of one or more images, for example, a single image or video frame, typically resized for compatibility with the dimensionality of the tensor input to the first layer. It is also possible to supply images or video frames in batches of size larger than one. The dimensionality of tensors is dependent on the CNN architecture, generally having some dimensions relating to input width and height and a further ‘channel’ dimension.

[0007] Slicing, or reducing a tensor to a collection of two-dimensional arrays, a tensor based on the channel dimension results in a set of two-dimensional ‘feature maps’, so-called because each slice of the tensor has some relationship to the corresponding input image, capturing properties such as various edge types. At layers further from the input to the network, the property can be more abstract. The ‘task performance’ of a CNN is measured by comparing the result of the CNN in performing a task using specific input with a provided ground truth, generally prepared by humans and deemed to indicate a ‘correct’ result.

[0008] Once a network topology is decided, the network weights may be updated over time as more training data becomes available. The overall complexity of the CNN tends to be relatively high, with relatively large numbers of multiply-accumulate operations being performed and numerous intermediate tensors being written to and read from memory. In some applications, the CNN is implemented entirely in the ‘cloud’, resulting in a need for high and costly processing power. In other applications, the CNN is implemented in an edge device, such as a camera or mobile phone, resulting in less flexibility but a more distributed processing load. An emerging architecture involves splitting a network into portions, one of the portions run in an edge device and another portion run in the cloud. Such a distributed network architecture may be referred to as ‘collaborative intelligence’ and offers benefits such as re-using a partial result from a first portion of the network with several different second portions, perhaps each portion being optimised for a different task. Collaborative intelligence architectures introduce a need for efficient compression of tensor data, for transmission over a network such as a WAN.

[0009] Some existing split network solutions have low performance in an end to end network. For example, a relatively high level of accuracy may be achieved at a cost of producing a relatively large encoded feature map. Some variants may be required to train multiple stages to achieve a sufficiently high performance. Training multiple stages can be complex and timeconsuming. 2025202067   21 Mar 2025

[00010] Existing split network solutions have generally been developed and trained for one task at one split point only. However, as one CNN backbone can be used for multiple CNN heads, solutions that are trained for one task at one split point only can be inflexible and require separate instantiations for each CNN backbone.

[00011] Video compression standards can be used for feature compression, as described below. Various methods can be used to constrict or reduce the data being presented for compression. However, some methods used to constrict or reduce the data being presented for compression can result in a decrease in accuracy unsuitable for some tasks implemented by CNNs.

[00012] Feature compression may benefit from existing video compression standards, such as Versatile Video Coding (VVC), developed by the Joint Video Experts Team (JVET). VVC is anticipated to address ongoing demand for ever-higher compression performance, especially as video formats increase in capability (e.g., with higher resolution and higher frame rate) and to address increasing market demand for service delivery over WANs, where bandwidth costs are relatively high. VVC is implementable in contemporary silicon processes and offers an acceptable trade-off between achieved performance versus implementation cost. The implementation cost may be considered for example, in terms of one or more of silicon area, CPU processor load, memory utilisation and bandwidth. Part of the versatility of the VVC standard is in the wide selection of tools available for compressing video data, as well as the wide range of applications for which VVC is suitable. Other video compression standards, such as High Efficiency Video Coding (HEVC) and AV-1, may also be used for feature compression applications.

[00013] Video data includes a sequence of frames of image data, each frame including one or more colour channels. Generally, one primary colour channel and two secondary colour channels are needed. The primary colour channel is generally referred to as the ‘luma’ channel and the secondary colour channel(s) are generally referred to as the ‘chroma’ channels. Although video data is typically displayed in an RGB (red-green-blue) colour space, this colour space has a high degree of correlation between the three respective components. The video data representation seen by an encoder or a decoder is often using a colour space such as YCbCr. YCbCr concentrates luminance, mapped to ‘luma’ according to a transfer function, in a Y (primary) channel and chroma in Cb and Cr (secondary) channels. Due to the use of a decorrelated YCbCr signal, the statistics of the luma channel differ markedly from those of the 2025202067   21 Mar 2025 chroma channels. A primary difference is that after quantisation, the chroma channels contain relatively few significant coefficients for a given block compared to the coefficients for a corresponding luma channel block. Moreover, the Cb and Cr channels may be sampled spatially at a lower rate (subsampled) compared to the luma channel, for example half horizontally and half vertically - known as a ‘4:2:0 chroma format’. The 4:2:0 chroma format is commonly used in ‘consumer’ applications, such as internet video streaming, broadcast television, and storage on Blu-RayTM disks. When only luma samples are present, the resulting monochrome frames are said to use a “4:0:0 chroma format”.

[00014] The VVC standard specifies a ‘block based’ architecture, in which frames are firstly divided into a square array of regions known as ‘coding tree units’ (CTUs). CTUs generally occupy a relatively large area, such as 128x128 luma samples. Other possible CTU sizes when using the VVC standard are 32x32 and 64x64. However, CTUs at the right and bottom edge of each frame may be smaller in area, with implicit splitting occurring the ensure the CBs remain in the frame. Associated with each CTU is a ‘coding tree’ either for both the luma channel and the chroma channels (a ‘shared tree’) or a separate tree each for the luma channel and the chroma channels. A coding tree defines a decomposition of the area of the CTU into a set of blocks, also referred to as ‘coding blocks’ (CBs). When a shared tree is in use a single coding tree specifies blocks both for the luma channel and the chroma channels, in which case the collections of collocated coding blocks are referred to as ‘coding units’ (CUs) (i.e., each CU having a coding block for each colour channel). The CBs are processed for encoding or decoding in a particular order. As a consequence of the use of the 4:2:0 chroma format, a CTU with a luma coding tree for a 128x128 luma sample area has a corresponding chroma coding tree for a 64x64 chroma sample area, collocated with the 128x128 luma sample area. When a single coding tree is in use for the luma channel and the chroma channels, the collections of collocated blocks for a given area are generally referred to as ‘units’, for example, the above-mentioned CUs, as well as ‘prediction units’ (PUs), and ‘transform units’ (TUs). A single tree with CUs spanning the colour channels of 4:2:0 chroma format video data result in chroma blocks half the width and height of the corresponding luma blocks. When separate coding trees are used for a given area, the above-mentioned CBs, as well as ‘prediction blocks’ (PBs), and ‘transform blocks’ (TBs) are used.

[00015] Notwithstanding the above distinction between ‘units’ and ‘blocks’, the term ‘block’ may be used as a general term for areas or regions of a frame for which operations are applied to all colour channels. 2025202067   21 Mar 2025

[00016] For each CU, a prediction unit (PU) of the contents (sample values) of the corresponding area of frame data is generated (a ‘prediction unit’). Further, a representation of the difference (or ‘spatial domain’ residual) between the prediction and the contents of the area as seen at input to the encoder is formed. The difference in each colour channel may be transformed and coded as a sequence of residual coefficients, forming one or more TUs for a given CU. The applied transform may be a Discrete Cosine Transform (DCT) or other transform, applied to each block of residual values. The transform is applied separably, (i.e., the two-dimensional transform is performed in two passes). The block is firstly transformed by applying a one-dimensional transform to each row of samples in the block. Then, the partial result is transformed by applying a one-dimensional transform to each column of the partial result to produce a final block of transform coefficients that substantially decorrelates the residual samples. Transforms of various sizes are supported by the VVC standard, including transforms of rectangular-shaped blocks, with each side dimension being a power of two. Transform coefficients are quantised for entropy encoding into a bitstream.

[00017] VVC features intra-frame prediction and inter-frame prediction. Intra-frame prediction involves the use of previously processed samples in a frame being used to generate a prediction of a current block of data samples in the frame. Inter-frame prediction involves generating a prediction of a current block of samples in a frame using a block of samples obtained from a previously decoded frame. The block of samples obtained from a previously decoded frame is offset from the spatial location of the current block according to a motion vector, which often has filtering applied. Intra-frame prediction blocks can be (i) a uniform sample value (“DC intra prediction”), (ii) a plane having an offset and horizontal and vertical gradient (“planar intra prediction”), (iii) a population of the block with neighbouring samples applied in a particular direction (“angular intra prediction”) or (iv) the result of a matrix multiplication using neighbouring samples and selected matrix coefficients. Further discrepancy between a predicted block and the corresponding input samples may be corrected to an extent by encoding a ‘residual’ into the bitstream. The residual is generally transformed from the spatial domain to the frequency domain to form residual coefficients in a ‘primary transform’ domain. The residual coefficients may be further transformed by application of a ‘secondary transform’ to produce residual coefficients in a ‘secondary transform domain’. Residual coefficients are quantised according to a quantisation parameter, resulting in a loss of accuracy of the reconstruction of the samples produced at the decoder but with a reduction in bitrate in the bitstream. 2025202067   21 Mar 2025

[00018] Sequences of pictures may be encoded according to a specified structure of pictures using intra-prediction and pictures using intra- or inter-prediction, and specified dependencies on preceding pictures in coding order, which may differ from display or delivery order. A ‘random access’ configuration results in periodic intra-pictures, forming entry points at which a decoder and commence decoding a bitstream. Other pictures in a random-access configuration generally use inter-prediction to predict content from pictures preceding and following a current picture in display or delivery order, according to a hierarchical structure of specified depth. The use of pictures after a current picture in display order for predicting a current picture requires a degree of picture buffering and delay between the decoding of a given picture and the display (and removal from the buffer) of the given picture.

[00019] In some applications, a need exists for identification of target objects, areas or other features of a frame of data out of all potential objects, areas or other features identifiable in the frame data by a split network system. For example, a user may wish to indicate target features or object as being important. Alternatively, a target feature, for example an object of particular interest may be required to be identified. SUMMARY

[00020] It is an object of the present invention to substantially overcome, or at least ameliorate, one or more disadvantages of existing arrangements.

[00021] One aspect of the present disclosure provides a method of generating a bitstream, the method comprising: encoding, into a first bitstream portion, first information which is based on one or more tensors; encoding second information into a second bitstream portion, wherein second information is associated with the first information and is related to a target object ; and generating the bitstream using the first bitstream portion and the second bitstream portion.

[00022] Another aspect of the present disclosure provides a method of decoding one or more tensors from a bitstream, the method comprising: obtaining a first bitstream portion and a second bitstream portion from the bitstream; producing the one or more tensors based on first information decoded from the first bitstream portion; and decoding the second bitstream portion to obtain second information, wherein the second information is associated with the first information and is related to a target object. 2025202067   21 Mar 2025

[00023] Another aspect of the present disclosure provides an encoder for generating a bitstream, the encoder configured to: encode, into a first bitstream portion, first information based on one or more tensors; encode second information into a second bitstream portion, wherein second information is associated with the first information and is related to a target object; and generate the bitstream using the first bitstream portion and the second bitstream portion.

[00024] Another aspect of the present disclosure provides a non-transitory computer- readable storage medium which stores a program for executing a method of generating a bitstream, the method comprising: encoding, into a first bitstream portion, first information based on one or more tensors; encoding second information into a second bitstream portion, wherein second information is associated with the first information and is related to a target object; and generating the bitstream using the first bitstream portion and the second bitstream portion.

[00025] Another aspect of the present disclosure provides a decoder for decoding one or more tensors from a bitstream, the decoder configured to: obtain a first bitstream portion and a second bitstream portion from the bitstream; produce the one or more tensors based on first information decoded from the first bitstream portion; and decode the second bitstream portion to obtain second information, wherein the second information is associated with the first information and is related to a target object.

[00026] Another aspect of the present disclosure provides a non-transitory computer- readable storage medium which stores a program for executing a method of decoding one or more tensors from a bitstream, the method comprising: obtaining a first bitstream portion and a second bitstream portion from the bitstream; producing the one or more tensors based on first information decoded from the first bitstream portion; and decoding the second bitstream portion to obtain second information, wherein the second information is associated with the first information and is related to a target object.

[00027] Other aspects are also disclosed. BRIEF DESCRIPTION OF THE DRAWINGS

[00028] At least one embodiment of the present invention will now be described with reference to the following drawings, in which: 2025202067   21 Mar 2025

[00029] Fig. 1 is a schematic block diagram showing a distributed machine task system;

[00030] Figs. 2A and 2B form a schematic block diagram of a general-purpose computer system upon which the distributed machine task system of Fig. 1 may be practiced;

[00031] Fig. 3A is a schematic block diagram showing functional modules of a backbone portion of a CNN;

[00032] Fig. 3B is a schematic block diagram showing a residual block of Fig. 3A;

[00033] Fig. 3C is a schematic block diagram showing a residual unit of Fig. 3A;

[00034] Fig. 3D is a schematic block diagram showing a convolutional batch normalisation leaky rectified linear (CBL)module of Fig. 3A;

[00035] Fig. 4A is a schematic block diagram showing functional modules of an alternative backbone portion of a CNN;

[00036] Fig. 4B(I) is a schematic block diagram showing functional modules of an alternative backbone portion of a CNN;

[00037] Fig. 4B(II) schematic block diagram showing architecture of a prompt encoder used in Fig. 1;

[00038] Fig. 4C is a schematic block diagram showing functional modules of an image encoder;

[00039] Fig. 5A is a schematic block diagram showing functional modules of bottleneck encoder

[00040] Fig.5B is a schematic block diagram showing an example of bottleneck encoder.

[00041] Fig. 6 shows a method for generating a bitstream;

[00042] Fig. 7 is a schematic block diagram showing a packing arrangement for a plurality of compressed tensors;

[00043] Fig. 8A is a schematic block diagram showing functional modules of a video encoder; 2025202067   21 Mar 2025

[00044] Fig. 8B is a schematic block diagram showing an FCM bitstream holding compressed feature frames;

[00045] Fig. 9 is a schematic block diagram showing functional modules of a video decoder;

[00046] Fig. 10A is a schematic block diagram showing functional modules of bottleneck decoder;

[00047] Fig.10B is a schematic block diagram showing an example of bottleneck decoder;

[00048] Fig. 11 shows a method for decoding a bitstream;

[00049] Fig. 12A is a schematic block diagrams showing a head portion of a CNN;

[00050] Fig. 12B is a schematic block diagram showing an upscaler module of Fig. 12A;

[00051] Fig. 12C is a schematic block diagram showing a detection module of Fig. 12A;

[00052] Fig. 13 is a schematic block diagram showing an alternative head portion of a CNN;

[00053] Fig. 14 is a schematic block diagram showing a cross-layer tensor inverse bottleneck for restoring tensor dimensionality after compression;

[00054] Fig.15 shows examples of different type of prompts; and

[00055] Appendix A shows an example FCM SEI message format and associated semantics for representing metadata associated with prompt information in a bitstream. DETAILED DESCRIPTION INCLUDING BEST MODE

[00056] Where reference is made in any one or more of the accompanying drawings to steps and / or features, which have the same reference numerals, those steps and / or features have for the purposes of this description the same function(s) or operation(s), unless the contrary intention appears. 2025202067   21 Mar 2025

[00057] A distributed machine task system may include an edge device, such as a network camera or smartphone producing intermediate compressed data. The distributed machine task system may also include a final device, such as a server-farm based (‘cloud’) application, operating on the intermediate compressed data to produce (generate) some task result. Additionally, the edge device functionality may be embodied in the cloud and the intermediate compressed data may be stored for later processing, potentially for multiple different tasks depending on need. Examples of machine task include object detection and instance segmentation, both of which produce a task result measured as ‘mean average precision’ (mAP) for detection over a threshold value of intersection-over-union (IoU), such as 0.5. Another example machine task is object tracking, with mean object tracking accuracy (MOTA) score as a typical task result.

[00058] A convenient form of intermediate compressed data is a compressed video bitstream, owing to the availability of high-performing compression standards and implementations thereof. Video compression standards typically operate on integer samples of some given bit depth, such as 10 bits, arranged in planar arrays. Colour video has three planar arrays, corresponding, for example, to colour components Y, Cb, Cr, or R, G, B, depending on application. CNNs typically operate on floating point data in the form of tensors. Tensors generally have a much smaller spatial dimensionality compared to incoming video data upon which the CNN operates but have many more channels than the three channels typical of colour video data.

[00059] Tensors typically have the following dimensions: Frames, channels, height, and width. For example, a tensor of dimensions [1, 256, 76, 136] is generated from one specific frame and contains two-hundred and fifty-six (256) feature maps each of which has the width of 136 and the height of 76. For video data, inferencing is typically performed one frame at a time, rather than using tensors containing multiple frames.

[00060] VVC supports a division of a picture into multiple subpictures, each of which may be independently encoded and independently decoded. In one approach, each subpicture is coded as one ‘slice’, or contiguous sequence of coded CTUs. A ‘tile’ mechanism is also available to divide a picture into a number of independently decodeable regions. Subpictures may be specified in a somewhat flexible manner, with various rectangular sets of CTUs coded as respective subpictures. Flexible definition of subpicture dimensions allows efficiently 2025202067   21 Mar 2025 holding types of data requiring different areas in one picture, avoiding large ‘unused’ areas, i.e., areas of a frame that are not used for reconstruction of tensor data.

[00061] Fig. 1 is a schematic block diagram showing functional modules of a distributed machine task system 100. The notion of distributing a machine task across multiple systems is sometimes referred to as ‘collaborative intelligence’ (CI). The system 100 may be used for implementing methods for decorrelating, packing and quantising feature maps into planar frames for encoding and decoding feature maps from encoded data. The methods may be implemented such that associated overhead data is not too burdensome and task performance on the decoded feature maps is resilient to changing bitrate of the bitstream and the quantised representation of the tensors does not needlessly consume bits where the bits do not provide a commensurate benefit in terms of task performance.

[00062] The system 100 includes a source device 110 for generating tensor data 115 from a CNN backbone 114. The source device 110 also generates prompt data 162 from a prompt generator 160. The prompt data 162 is described with reference to Fig. 15. The system 100 also includes a destination device 140 for decoding tensor data in the form of an encoded video bitstream 143. A communication channel 130 is used to communicate an encoded video bitstream 121 from the source device 110 to the destination device 140. In some arrangements, the source device 110 and destination device 140 may either or both comprise respective mobile telephone handsets (e.g., “smartphones”) or network cameras and cloud applications. The communication channel 130 may be a wired connection, such as Ethernet, or a wireless connection, such as WiFi or 5G, including connections across a Wide Area Network (WAN) or across ad-hoc connections. Moreover, the source device 110 and the destination device 140 may comprise applications where encoded video data is captured on some computer-readable storage medium, such as a hard disk drive in a file server or memory.

[00063] As shown in Fig. 1, the source device 110 includes a video source 112, the prompt generator 160, the CNN backbone 114, a feature encoder 168, and a transmitter 122. The feature encoder 168 contains a bottleneck encoder 116, a quantise and pack module 118, a feature frame encoder 120, and a prompt encoder 164.

[00064] The video source 112 typically comprises a source for capturing a video frame (shown as 113), such as an image capture sensor, a previously captured video sequence stored on a non-transitory recording medium, or a video feed from a remote image capture sensor. The 2025202067   21 Mar 2025 video source 112 may also be an output of a computer graphics card, for example, displaying the video output of an operating system and various applications executing upon a computing device (e.g., a tablet computer). Examples of source devices 110 that may include an image capture sensor as the video source 112 include smart-phones, video camcorders, professional video cameras, and network video cameras. The system 100 reduces dimensionality of tensors at the interface between the first network portion and the second network portion using a ‘bottleneck’, i.e., additional network layers that restrict tensor dimensionality on the encoding side and restore tensor dimensionality at the decoder side. The multi-scale representation produced by a feature pyramid network (FPN) may be ‘fused’ together into a single tensor using an approach named ‘multi-scale feature compression’ (MSFC). MSFC is ordinarily used to merge all FPN layers into a single tensor. Merging all FPN layers into a single tensor is implemented at the expense of spatial detail for the less decomposed (larger) layers of the FPN. The loss of spatial detail can result in an unacceptable decrease in accuracy for some tasks or operations implemented by the system 100.

[00065] The prompt generator 160 can be an interface between a user and the source device source 110, for example a graphical user interface or some other interface by which the use can enter one or more prompt indicators associated with the video frame 113. Prompt data 162 generated by the prompt generator 160 may comprise at least one of a point prompt, a bounding box prompt, a mask prompt, a text prompt, or an audio prompt. In one arrangement, a point prompt in the prompt data 162 indicates a specific position in the video frame 113. A bounding box prompt in the prompt data 162 indicates a position of a bounding box in the video frame 113. A bounding box can be represented by two points that are at the position the top-left corner of the bounding box (x min, ymin) and at the position of the bottom-right corner of the bounding box (xmax, ymax) and indicate a rectangular area wholly containing or mostly overlapping an object in an image. Bounding boxes may be represented by using width and height to specify their size. A mask prompt specifies the boundary around an object more specifically than is possible using a bounding box. A mask prompt is typically represented in one of two forms: (i) a set of points describing the boundary of an object and forming an enclosed region overlapping the object, or (ii) as a bi-level bitmap occupying either the entire frame of a region bounded by a bounding box. When using a bitmap, an encoding such as runlength encoding is typically employed to more compactly store the bitmap. An audio prompt can be presented by a waveform or frequency representation and can indicate an audio that is similar to a piece of the video source 112. A text prompt can be presented by character or string 2025202067   21 Mar 2025 that can indicate a characteristic of an object in source video. The prompt generator 160 may for example comprise a touch screen on which the captured video frame 113 is displayed, and receive user input to generate point prompt, box prompt, or mask prompt. The prompt generator 160 may also comprises a device that receives user input of text prompt and audio prompt. The prompt data 162 is associated with the video frame and is related to a target object or feature in the video frame.

[00066] In other arrangements, the prompt generator 160 may be implemented using a detection network, rather than input from a user, to produce bounding boxes around specific objects of interest. The detection network may be implemented on a subset of frames of the video data 113, the subset resulting from limited processing capacity in the source device 110. Accordingly, each instance of the prompt data 162 is associated with specific frame(s) in the video frame 113. In arrangements using a detection network rather than a user input for prompts, detected objects of interest are used to generate prompt data, for example taking the centre-point of each bounding box or directly encoding the bounding boxes as the prompt data 162.

[00067] The CNN backbone 114 receives the video frame 113 and performs specific layers of an overall CNN, such as layers corresponding to the ‘backbone’ of the CNN, outputting the tensors 115. The backbone layers of the CNN may produce (generate) multiple tensors as output, for example, corresponding to different spatial scales of an input image represented by the video frame 113, sometimes referred to as a ‘feature pyramid network’ (FPN) architecture. The tensors resulting from an FPN backbone form a hierarchical representation of the video frame 113 including data of feature maps. Each successive layer of the hierarchical representation has half the width and height of the preceding layer. Later layers that are produced further into in the backbone network tend to contain features having a more abstract representation of the video frame 113. Less decomposed layers, produced earlier in the backbone network, tend to contain features representing less abstract features of the video frame 113, such as various geometric properties such as edges of various angles. An FPN may result in three tensors, corresponding to three layers, output from the backbone 114 as the tensors 115 when a ‘YOLOv3’ network is performed by the system 100, with varying spatial resolution and channel count. When the system 100 is performing networks such as ‘Faster RCNN X101-FPN” or “Mask RCNN X101-FPN” the tensors 115 include tensors for four layers P2-P5. The system 100 may implement a network such as the Segment Anything Model (SAM) from Meta AI Research. When using the SAM, the output from the backbone 114 may contain 2025202067   21 Mar 2025 one tensor having a dimensionality of [1, 256, 64, 64]. Here, the portion of the SAM, the CNN backbone 114 operates to extract features of objects from the frame data 113 needed to perform mask determination (effectively ‘selecting’ the indicated object, with boundaries identified with high precision) when combined with the prompt data.

[00068] In the feature encoder 168, the bottleneck encoder 116 receives the tensors 115. The bottleneck encoder 116 acts to compress one or more internal layers of the overall CNN. The internal layers of the overall CNN provide the output of the CNN backbone 114, compressed or constricted by the bottleneck encoder 116 using a set of neural network layers trained to convert to a lower channel count and smaller spatial resolution than the tensors 115. The bottleneck encoder 116 outputs bottleneck tensor 117, sometimes referred to as a “fused tensor”. The bottleneck tensors 117 are passed to the quantise and pack module 118. A value of each element of each feature map of feature maps forming the bottleneck tensor 117 is quantised so that the value is converted from floating point to integer precision. Then each feature map of the quantized feature maps forming the bottleneck tensor 117 is packed into a monochrome frame by the module 118 to produce a feature frame 119. The feature frame 119 is encoded by the feature frame encoder 120 to produce a first bitstream portion 172. The prompt encoder 164 encodes the prompt data 162 to produce a second bitstream portion 166. The first bitstream portion 172 and the second bitstream portion 166 are multiplexed in a multiplexer 170 to produce the bitstream 121. The bitstream 121 is supplied to the transmitter 122 for transmission over the communications channel 130 or the bitstream 121 is written to storage 132 for later use.

[00069] The source device 110 supports a particular network for the CNN backbone 114. However, the destination device 140 may use one of several networks for a corresponding head CNN 150. In this way, partially processed data in the form of packed feature maps may be stored for later use in performing various tasks without needing to repeatedly perform the operation of the CNN backbone 114. The CNN backbone 114 may also be referred to as a neural network part 1 or “NN part 1” and the CNN head 150 may also be referred to as a neural network part 2 or “NN part 2”.

[00070] The bitstream 121 is transmitted by the transmitter 122 over the communication channel 130 as encoded video data. The bitstream 121 can in some implementations be stored in the storage 132, where the storage 132 is a non-transitory storage device such as a “Flash” memory or a hard disk drive, until later being transmitted over the communication channel 130 2025202067   21 Mar 2025 (or in-lieu of transmission over the communication channel 130). For example, encoded video data may be served upon demand to customers over a wide area network (WAN) for a video analytics application.

[00071] The destination device 140 includes a receiver 142, a feature decoder 190, the CNN head 150, and a CNN task result buffer 152. The feature decoder 190 contains a demultiplexer 180, a feature frame decoder 144, an unpack and inverse quantise module 146, a bottle neck decoder 148, and a prompt decoder 186. The receiver 142 receives encoded video data from the communication channel 130 and passes the bitstream 143 to the demultiplexer 180. The demultiplexer 180 demultiplexes the bitstream 143 into the first bitstream portion 182 (i.e., the first bitstream portion 172) and the second bitstream portion 184 (i.e., the second bitstream portion 166). The first bitstream portion 182 is input to the feature frame decoder 144. The feature frame decoder 144 operates to decode the first bitstream portion and outputs a decoded feature frame 145. The decoded feature frame 145 is passed to the unpack and inverse quantise module 146. The module 146 unpacks and inverse quantises the tensors of the feature frame 145 to generate dequantized tensors, output as decoded bottleneck tensors 147. The decoded bottleneck tensors 147 are supplied to the bottleneck decoder 148. The bottleneck decoder 148 performs the inverse operation of the bottleneck encoder 116, to produce extracted tensors 149. The prompt decoder 186 decodes the second bitstream portion 184 of the bitstream 143 to produce prompt data 188. The extracted tensors 149 and prompt data 188 are passed to the CNN head 150. The CNN head 150 performs the later layers of the task that follows the last layers performed in the CNN backbone 114, to produce (generate) a task result 151, which is stored in a task result buffer 152. The contents of the task result buffer 152 may be presented to the user, e.g., via a graphical user interface, or provided to an analytics application where some action is decided based on the task result, which may include summary level presentation of aggregated task results to a user. It is also possible for the functionality of each of the source device 110 and the destination device 140 to be embodied in a single device, examples of which include mobile telephone handsets and tablet computers and cloud applications.

[00072] Notwithstanding the example devices mentioned above, each of the source device 110 and destination device 140 may be configured within a general-purpose computing system, typically through a combination of hardware and software components. Fig. 2A illustrates such a computer system 200, which includes: a computer module 201; input devices such as a keyboard 202, a mouse pointer device 203, a scanner 226, a camera 227, which may be configured as the video source 112, and a microphone 280; and output devices including a 2025202067   21 Mar 2025 printer 215, a display device 214, which may be configured as a display device presenting the task result 151, and loudspeakers 217. An external Modulator-Demodulator (Modem) transceiver device 216 may be used by the computer module 201 for communicating to and from a communications network 220 via a connection 221. The communications network 220, which may represent the communication channel 130, may be a (WAN), such as the Internet, a cellular telecommunications network, or a private WAN. Where the connection 221 is a telephone line, the modem 216 may be a traditional “dial-up” modem. Alternatively, where the connection 221 is a high capacity (e.g., cable or optical) connection, the modem 216 may be a broadband modem. A wireless modem may also be used for wireless connection to the communications network 220. The transceiver device 216 may provide the functionality of the transmitter 122 and the receiver 142 and the communication channel 130 may be embodied in the connection 221.

[00073] The computer module 201 typically includes at least one processor unit 205, and a memory unit 206. For example, the memory unit 206 may have semiconductor random access memory (RAM) and semiconductor read only memory (ROM). The computer module 201 also includes a number of input / output (I / O) interfaces including: an audio-video interface 207 that couples to the video display 214, loudspeakers 217 and microphone 280; an I / O interface 213 that couples to the keyboard 202, mouse 203, scanner 226, camera 227 and optionally a joystick or other human interface device (not illustrated); and an interface 208 for the external modem 216 and printer 215. The signal from the audio-video interface 207 to the computer monitor 214 is generally the output of a computer graphics card. In some implementations, the modem 216 may be incorporated within the computer module 201, for example within the interface 208. The computer module 201 also has a local network interface 211, which permits coupling of the computer system 200 via a connection 223 to a local-area communications network 222, known as a Local Area Network (LAN). As illustrated in Fig. 2A, the local communications network 222 may also couple to the wide network 220 via a connection 224, which would typically include a so-called “firewall” device or device of similar functionality. The local network interface 211 may comprise an EthernetTM circuit card, a BluetoothTM wireless arrangement or an IEEE 802.11 wireless arrangement; however, numerous other types of interfaces may be practiced for the interface 211. The local network interface 211 may also provide the functionality of the transmitter 122 and the receiver 142 and communication channel 130 may also be embodied in the local communications network 222. 2025202067   21 Mar 2025

[00074] The I / O interfaces 208 and 213 may afford either or both of serial and parallel connectivity, the former typically being implemented according to the Universal Serial Bus (USB) standards and having corresponding USB connectors (not illustrated). Storage devices 209 are provided and typically include a hard disk drive (HDD) 210. Other storage devices such as a floppy disk drive and a magnetic tape drive (not illustrated) may also be used. An optical disk drive 212 is typically provided to act as a non-volatile source of data. Portable memory devices, such optical disks (e.g., CD-ROM, DVD, Blu ray DiscTM), USB-RAM, portable, external hard drives, and floppy disks, for example, may be used as appropriate sources of data to the computer system 200. Typically, any of the HDD 210, optical drive 212, networks 220 and 222 may also be configured to operate as the video source 112, or as a destination for decoded video data to be stored for reproduction via the display 214. The source device 110 and the destination device 140 of the system 100 may be embodied in the computer system 200.

[00075] The components 205 to 213 of the computer module 201 typically communicate via an interconnected bus 204 and in a manner that results in a conventional mode of operation of the computer system 200 known to those in the relevant art. For example, the processor 205 is coupled to the system bus 204 using a connection 218. Likewise, the memory 206 and optical disk drive 212 are coupled to the system bus 204 by connections 219. Examples of computers on which the described arrangements can be practiced include IBM-PC’s and compatibles, Sun SPARCstations, Apple MacTM or alike computer systems.

[00076] Where appropriate or desired, the source device 110 and the destination device 140, as well as methods described below, may be implemented using the computer system 200. In particular, the source device 110, the destination device 140 and methods to be described, may be implemented as one or more software application programs 233 executable within the computer system 200. The source device 110, the destination device 140 and the steps of the described methods are affected by instructions 231 (see Fig. 2B) in the software 233 that are carried out within the computer system 200. The software instructions 231 may be formed as one or more code modules, each for performing one or more particular tasks. The software may also be divided into two separate parts, in which a first part and the corresponding code modules performs the described methods, and a second part and the corresponding code modules manage a user interface between the first part and the user. 2025202067   21 Mar 2025

[00077] The software may be stored in a computer readable medium, including the storage devices described below, for example. The software is loaded into the computer system 200 from the computer readable medium, and then executed by the computer system 200. A computer readable medium having such software or computer program recorded on the computer readable medium is a computer program product. The use of the computer program product in the computer system 200 preferably effects an advantageous apparatus for implementing the source device 110 and the destination device 140 and the described methods.

[00078] The software 233 is typically stored in the HDD 210 or the memory 206. The software is loaded into the computer system 200 from a computer readable medium and executed by the computer system 200. Thus, for example, the software 233 may be stored on an optically readable disk storage medium (e.g., CD-ROM) 225 that is read by the optical disk drive 212.

[00079] In some instances, the application programs 233 may be supplied to the user encoded on one or more CD-ROMs 225 and read via the corresponding drive 212, or alternatively may be read by the user from the networks 220 or 222. Still further, the software can also be loaded into the computer system 200 from other computer readable media. Computer readable storage media refers to any non-transitory tangible storage medium that provides recorded instructions and / or data to the computer system 200 for execution and / or processing. Examples of such storage media include floppy disks, magnetic tape, CD-ROM, DVD, Blu-ray DiscTM, a hard disk drive, a ROM or integrated circuit, USB memory, a magneto-optical disk, or a computer readable card such as a PCMCIA card and the like, whether or not such devices are internal or external of the computer module 201. Examples of transitory or non-tangible computer readable transmission media that may also participate in the provision of the software, application programs, instructions and / or video data or encoded video data to the computer module 201 include radio or infra-red transmission channels, as well as a network connection to another computer or networked device, and the Internet or Intranets including e-mail transmissions and information recorded on Websites and the like.

[00080] The second part of the application program 233 and the corresponding code modules mentioned above may be executed to implement one or more graphical user interfaces (GUIs) to be rendered or otherwise represented upon the display 214. Through manipulation of typically the keyboard 202 and the mouse 203, a user of the computer system 200 and the application may manipulate the interface in a functionally adaptable manner to provide 2025202067   21 Mar 2025 controlling commands and / or input to the applications associated with the GUI(s). Other forms of functionally adaptable user interfaces may also be implemented, such as an audio interface utilizing speech prompt output via the loudspeakers 217 and user voice commands input via the microphone 280.

[00081] Fig. 2B is a detailed schematic block diagram of the processor 205 and a “memory” 234. The memory 234 represents a logical aggregation of all the memory modules (including the storage devices 209 and semiconductor memory 206) that can be accessed by the computer module 201 in Fig. 2A.

[00082] When the computer module 201 is initially powered up, a power-on self-test (POST) program 250 executes. The POST program 250 is typically stored in a ROM 249 of the semiconductor memory 206 of Fig. 2A. A hardware device such as the ROM 249 storing software is sometimes referred to as firmware. The POST program 250 examines hardware within the computer module 201 to ensure proper functioning and typically checks the processor 205, the memory 234 (209, 206), and a basic input-output systems software (BIOS) module 251, also typically stored in the ROM 249, for correct operation. Once the POST program 250 has run successfully, the BIOS 251 activates the hard disk drive 210 of Fig. 2A. Activation of the hard disk drive 210 causes a bootstrap loader program 252 that is resident on the hard disk drive 210 to execute via the processor 205. This loads an operating system 253 into the RAM memory 206, upon which the operating system 253 commences operation. The operating system 253 is a system level application, executable by the processor 205, to fulfil various high-level functions, including processor management, memory management, device management, storage management, software application interface, and generic user interface.

[00083] The operating system 253 manages the memory 234 (209, 206) to ensure that each process or application running on the computer module 201 has sufficient memory in which to execute without colliding with memory allocated to another process. Furthermore, the different types of memory available in the computer system 200 of Fig. 2A need to be used properly so that each process can run effectively. Accordingly, the aggregated memory 234 is not intended to illustrate how particular segments of memory are allocated (unless otherwise stated), but rather to provide a general view of the memory accessible by the computer system 200 and how such memory is used. 2025202067   21 Mar 2025

[00084] As shown in Fig. 2B, the processor 205 includes a number of functional modules including a control unit 239, an arithmetic logic unit (ALU) 240, and a local or internal memory 248, sometimes called a cache memory. The cache memory 248 typically includes a number of storage registers 244-246 in a register section. One or more internal busses 241 functionally interconnect these functional modules. The processor 205 typically also has one or more interfaces 242 for communicating with external devices via the system bus 204, using a connection 218. The memory 234 is coupled to the bus 204 using a connection 219.

[00085] The application program 233 includes a sequence of instructions 231 that may include conditional branch and loop instructions. The program 233 may also include data 232 which is used in execution of the program 233. The instructions 231 and the data 232 are stored in memory locations 228, 229, 230 and 235, 236, 237, respectively. Depending upon the relative size of the instructions 231 and the memory locations 228-230, a particular instruction may be stored in a single memory location as depicted by the instruction shown in the memory location 230. Alternately, an instruction may be segmented into a number of parts each of which is stored in a separate memory location, as depicted by the instruction segments shown in the memory locations 228 and 229.

[00086] In general, the processor 205 is given a set of instructions which are executed therein. The processor 205 waits for a subsequent input, to which the processor 205 reacts to by executing another set of instructions. Each input may be provided from one or more of a number of sources, including data generated by one or more of the input devices 202, 203, data received from an external source across one of the networks 220, 202, data retrieved from one of the storage devices 206, 209 or data retrieved from a storage medium 225 inserted into the corresponding reader 212, all depicted in Fig. 2A. The execution of a set of the instructions may in some cases result in output of data. Execution may also involve storing data or variables to the memory 234.

[00087] Each of modules of the system 100 and the described methods may use input variables 254, which are stored in the memory 234 in corresponding memory locations 255, 256, 257. The modules of the system 100 and the described methods produce output variables 261, which are stored in the memory 234 in corresponding memory locations 262, 263, 264. Intermediate variables 258 may be stored in memory locations 259, 260, 266 and 267. 2025202067   21 Mar 2025

[00088] Referring to the processor 205 of Fig. 2B, the registers 244, 245, 246, the arithmetic logic unit (ALU) 240, and the control unit 239 work together to perform sequences of micro-operations needed to perform “fetch, decode, and execute” cycles for every instruction in the instruction set making up the program 233. Each fetch, decode, and execute cycle comprises: a fetch operation, which fetches or reads an instruction 231 from a memory location 228, 229, 230; a decode operation in which the control unit 239 determines which instruction has been fetched; and an execute operation in which the control unit 239 and / or the ALU 240 execute the instruction.

[00089] Thereafter, a further fetch, decode, and execute cycle for the next instruction may be executed. Similarly, a store cycle may be performed by which the control unit 239 stores or writes a value to a memory location 232.

[00090] Each step or sub-process in the methods of Figs. 6 and 11 , to be described, is associated with one or more segments of the program 233 and is typically performed by the register section 244, 245, 247, the ALU 240, and the control unit 239 in the processor 205 working together to perform the fetch, decode, and execute cycles for every instruction in the instruction set for the noted segments of the program 233.

[00091] As for the CNN formed by the backbone 114 and the head 150, different frameworks such as Detectron2, JDE (Joint Detection and Embedding), or SAM can be used. Each framework uses a different end-to-end network architecture. When performing split inferencing, an end-to-end network architecture is split into multiple parts, such as two parts: a first portion (NN part 1) performing feature extraction and a second portion (NN part 2) producing machine task results. Fig. 3A, Fig. 4A, and Fig. 4B are arrangements of the CNN backbone 114 (NN part 1) when using YOLOv3, FPN, or SAM network architectures, respectively. Fig. 12A, 13, and 14 are arrangements of CNN head 150 (NN part 2) when using YOLOv3, FPN, or SAM network architectures, respectively.

[00092] A point in the network where an end-to-end network is separated into NN part 1 and NN part 2 is called a “split point”. In some cases, when the split points are close to the input among the end-to-end network, the NN part 2 continues extracting features, then 2025202067   21 Mar 2025 generating machine task results. At the split point, a “feature compression” network, such as the bottleneck encoder 116, acts to reduce the dimensionality of the tensors 115. The bottleneck decoder 148 acts to produced reconstructed features, i.e., the tensors 149, supplied to the NN part 2, i.e., the CNN head 150. Fig. 5A and Fig. 10A show example arrangements of the bottleneck encoder 116 and the bottleneck decoder 148, respectively.

[00093] In some end-to-end network arrangements, the NN part 2 (i.e., the CNN head 150) produces all possible results of a given machine task. For example, the NN part 2 produces all bounding boxes of all classes in an object detection task, produces all segmented areas of all classes in an object semantic segmentation task for the video frame 113, produces all human bounding boxes and their corresponding tracking IDs in a pedestrian tracking task for the video frame 113, or detect all types of sound in an audio detection task. In some other end-to-end networks arrangements, “prompt data” such as point prompt, mask prompt, box prompt, text prompt, or audio prompt is provided from the source device 110 to identify and output one or more specific targets from the results of the NN part 2 in the destination device 140. In this way, prompt data can be used to identify specific target to be outputted among the results of NN part 2.

[00094] In some arrangements where coded video data is transmitted using a video compression standard such as VCM, HEVC, or VVC, it is advantageous to encode prompt data along with the coded bitstream. Prompt data may be coded in a video bitstream using the SEI message syntax described with reference to Appendix A. In systems encoding prompt data to a video bitstream, after decoding the video bitstream, the decoded video and the prompt data can be sent to a model, such as the Segment Anything Model, to perform a task such as promptable visual segmentation. Prompt information may be coded for a given AU (access unit) using a prefix SEI (supplemental enhancement information) NAL (network abstraction layer) unit (the prefix SEI NAL unit is located immediately before the given AU in the bitstream) or a suffix SEI NAL unit (the suffix SEI NAL unit is located immediately after the given AU in the bitstream).

[00095] Fig. 3A is a schematic block diagram 300 showing functional modules of a backbone portion 310 of a CNN, which may serve as the CNN backbone 114. The CNN backbone 114 is sometimes referred to as ‘DarkNet-53’ and forms the backbone of a ‘YOLOv3’ object detection network. Different backbones are also possible, resulting in a different number of and dimensionality of layers of the tensors 115 for each video frame. 2025202067   21 Mar 2025

[00096] As shown in Fig. 3A, the video frame 113 is passed to a resizer module 304. The resizer module 304 resizes the video frame 113 to a resolution suitable for processing by the CNN backbone 310, producing resized frame data 312. If the resolution of the video frame 113 is already suitable for the CNN backbone 310, operation of the resizer module 304 is not needed. The resized frame data 312 is passed to a convolutional batch normalisation leaky rectified linear (CBL) module 314 to produce tensors 316. The CBL 314 contains modules as described with reference to a CBL module 360, as shown in Fig 3D.

[00097] Referring to Fig. 3D, the CBL module 360 takes as input a tensor 361. The tensor 361 is passed to a convolutional layer 362 to produce tensor 363. When the convolutional layer 362 has a stride of one and padding is set to k samples, with a convolutional kernel of size 2k+1, the tensor 363 has the same spatial dimensions as the tensor 361. When the convolution layer 362 has a larger stride, such as two, the tensor 363 has smaller spatial dimensions (size) compared to the tensor 361, for example, halved in size for the stride of two. Regardless of the stride, the size of channel dimension of the tensor 363 may vary compared to the channel dimension of the tensor 361 for a particular CBL block. The tensor 363 is passed to a batch normalisation module 364 which outputs a tensor 365. The batch normalisation module 364 normalises the input tensor 363, applies a scaling factor and offset value to produce the output tensor 365. The scaling factor and offset value are derived from a training process. The tensor 365 is passed to a leaky rectified linear activation (“LeakyReLU”) module 366 to produce a tensor 367. The module 366 provides a ‘leaky’ activation function whereby positive values in the tensor are passed through and negative values are severely reduced in magnitude, for example, to 0.1X their former value.

[00098] Returning to Fig. 3A, the tensor 316 is passed from the CBL block 314 to a residual block 11 module 320. The module 320 contains a sequential concatenation of three residual blocks, containing 1, 2, and 8 residual units internally, respectively.

[00099] A residual block, such as present in the module 320, is described with reference to a ResBlock 340 as shown in Fig. 3B. The ResBlock 340 receives a tensor 341. The tensor is zero-padded by a zero-padding module 342 to produce a tensor 343. The tensor 343 is passed to a CBL module 344 to produce (generate) a tensor 345. The tensor 345 is passed to a residual unit 346, of which the residual block 340 includes a series of concatenated residual units. The last residual unit of the residual units 346 outputs a tensor 347. 2025202067   21 Mar 2025 [000100]     A residual unit, such as the unit 346, is described with reference to a ResUnit 350 as shown in Fig. 3C. The ResUnit 350 takes a tensor 351 as input. The tensor 351 is passed to a CBL module 352 to produce a tensor 353. The tensor 353 is passed to a second CBL unit 354 to produce a tensor 355. An add module 356 sums the tensor 355 with the tensor 351 to produce a tensor 357. The add module 356 may also be referred to as a ‘shortcut’ as the input tensor 351 substantially influences the output tensor 357. For an untrained network, ResUnit 350 acts to pass-through tensors. As training is performed, the CBL modules 352 and 354 act to deviate the tensor 357 away from the tensor 351 in accordance with training data and ground truth data. [000101]     The Res11 module 320 outputs a tensor 322. The tensor 322 is output from the backbone module 310 as one of the layers 115 and also provided to a Res8 module 324. The Res8 module 324 is a residual block (i.e., 340), which includes eight residual units (i.e., 350). The Res8 module 324 produces a tensor 326. The tensor 326 is passed to a Res4 module 328 and output from the backbone module 310 as one of the layers 115. The Res4 module 328 contains a sequence of four residual blocks (i.e., 340), which includes residual unit (i.e., 350). The Res4 module 328 produces a tensor 329. The tensor 329 is output from the backbone module 310 as one of the layers 115. Collectively, the layer tensors 322, 326, and 329 are output as the tensors 115. The backbone CNN 310 may take as input a video frame of resolution 1088^608 and produce three tensors, corresponding to three layers, with the following dimensions: [1, 256, 76, 136], [1, 512, 38, 68], [1, 1024, 19, 34]. Another example of the three tensors corresponding to three layers may be [1, 512, 34, 19], [1, 256, 68, 38], [1, 128, 136, 76] which are respectively separated at 75th network layer, 90th network layer, and 105th network layer in the CNN 310. Each tensor can have a different resolution to the next tensor. The resolution of each tensor can double in height and width between respective tensors. In forming the output tenors 115, the tensors 322, 326, and 329 provide a hierarchical representation of the video frame data including data of feature maps for encoding to the bitstream. The separating points depend on the CNN 310. [000102]     Fig. 4A is a schematic block diagram showing functional modules of an alternative backbone portion 400 of a CNN, which may serve as the CNN backbone 114. The backbone portion 400 implements a residual network with feature pyramid network (‘ResNet FPN’) and is an alternative to the CNN backbone 114. The video frame 113 is input and passes through a stem network 408, a res2 module 412, a res3 module 416, a res4 module 420, and a res5 module 424 via tensors 409, 413, 417, 421 and 425 respectively. 2025202067   21 Mar 2025 [000103]     The stem network 408 includes a 7x7 convolution with a stride of two (2) and a max pooling operation. The res2 module 412, the res3 module 416, the res4 module 420, and the res5 module 424 perform convolution operations, LeakyReLU activations. Each module 412, 416, 420 and 424 also performs one halving of the resolution of the processed tensors via a stride setting of two. The tensors 413, 417, 421, and 425 are passed to size 1x1 pixel lateral convolution modules 446, 444, 442, and 440 respectively. The modules 440, 442, 444, and 446 produce tensors 441, 443, 445 and 447 respectively. The tensor 441 is passed to a 3x3 output convolution module 470, which produces an output tensor P5 471. The tensor 441 is also passed to upsampler module 450 to produce an upsampled tensor 451. [000104]     A summation module 460 sums the tensors 443 and 451 to produce a tensor 461. The tensor 461 is passed to an upsampler module 452 and a 3x3 lateral convolution module 472. The module 472 outputs a P4 tensor 473. The upsampler module 452 produces an upsampled tensor 453. A summation module 462 sums tensors 445 and 453 to produce a tensor 463. The tensor 463 is passed to a 3x3 lateral convolution module 474 and an upsampler module 454. The module 474 outputs a P3 tensor 475. The upsampler module 454 outputs an upsampled tensor 455. A summation module 464 sums the tensors 447 and 455 to produce tensor 465, which is passed to a 3x3 lateral convolution module 476. The module 476 outputs a P2 tensor 477. [000105]     The upsampler modules 450, 452, and 454 use nearest neighbour interpolation for low computational complexity. The tensors 471, 473, 475, and 477 form the output tensor 115 of the CNN backbone 400. In forming the output tenors 115, the FPN of tensors 471, 473, 475, and 477 provide a hierarchical representation of the video frame including data of feature maps for encoding to the bitstream. [000106]     Fig. 4B(I) shows one arrangement 400B of the CNN backbone 114 (a NN part 1) implemented as an image encoder in SAM (Segment Anything Model) network. Fig. 4B(II) shows an arrangement of the prompt encoder 164 when a similar framework to the SAM (Segment Anything Model) is used. While the prompt encoder 400B is described in relation to use with the network 400B providing the backbone 114, the prompt encoder can also be used with other types of neural network, for example with examples 300 and 400 in Figs. 3 and 4. The prompt encoder 164 can be a component of a SAM architecture implementing the neural network parts 1 and 2, or can be a separate component provided by another machine task network such as a detection network. 2025202067   21 Mar 2025 [000107]     Instance segmentation ordinarily involves detecting all objects of a predefined set of classes in a frame. SAM provides a prompt-driven (or “promptable”) segmentation task that, rather than detecting all instances of a set of predefined classes, detects object boundaries based on a provided prompt, without the constraint of being limited to operating on a set of predefined classes. A typical SAM network architecture includes three parts: an image encoder, a prompt encoder, and a mask decoder. The image encoder uses pretrained weights from a transformer model or can be trained on a relatively very large segmentation data on any network for feature extraction. The prompt encoder can be pre-trained weights for natural language processing to encode prompt data as text prompt, or can be a convolutional neural network to encode prompt data as masks. The mask decoder maps the image features from image encoder and prompt embedding from prompt decoder to generate masks. SAM models aim to always predict at least valid mask for any prompt even when the prompts are ambiguous. For example, SAM network used as the CNN head 114 and backbone 150 may comprise the encoder 400B and the prompt encoder 164 of Fig. 4B(II) in the source device 110 and a decoder arrangement including a mask decoder in the destination device 140, as described in relation to Fig. 14. [000108]     Returning Fig. 4B(I), the video frame 113 is input to a feature extraction module 401B and prompt data 162 is input to the prompt encoder 164 (Fig. 4B(II)). Output from the feature extraction module 401B is a collection of important features 402B extracted from the video frame 113. An image embedding module 403B is performed on the extracted feature 402B to reduce dimensions of the feature 402B and produce the tensor 115. The tensor 115 has a smaller size (i.e., [1, 256, 64, 64]) than tensor 402B (i.e., [1, 1280, 64, 64]). [000109]     The prompt data 162 can be provided from the prompt generator 160. The prompt data 162 may be divided into two categories. Referring to Fig. 4B(II) a first category includes prompts of a fixed size and is referred to as “sparse prompts”, such as a point prompt 408B and a box prompt 409B. Sparse or fixed size prompts may be compactly coded using a fixed sequence of syntax elements, such as Cartesian coordinates and, if needed, width and height. The second category includes variable-length data and is referred to as “dense prompts”, and includes prompts such as mask 407B, text prompt 410B, or an audio prompt 411B. Dense prompts, due to their unconstrained size, are preferably encoded into the bitstream 121 using some compression format, such as run-length encoding or an audio codec (ITU-T G.709 or ITU-T G.729.1), respectively. Notwithstanding that, the text prompt 410B is variable length, in some applications such as the original SAM (version 1), the text prompt 410B is considered to 2025202067   21 Mar 2025 be a sparse prompt. In SAM, the audio prompt is not used, but in some audio visual segmentation applications, an audio prompt 411B can be used. While Fig. 4B(II) shows the prompt 162 comprising prompts 407B, 408B, 409B, 410B and 411B, for clarity, the prompt 162 may comprise one or more of 407B, 408B, 409B, 410B and 411B in any instance. Whether any prompts are present, or the number and types of prompts present, will depend on input from the prompt generator 160, for example from a user or a detection network generating prompt indications. [000110]     In some arrangements, in cases of sparse prompts such as a box prompt or a point prompt, the sparse prompt data is encoded and decoded using a simple lossless entropy method at the encoder 164. In cases of dense prompts such as a mask prompt, prompt data can be compressed by run length encoding (RLE) method to reduce the required size to encode a mask prompt. [000111]     The prompt encoder 164 includes a point encoder 413B, a box encoder 414B, a text encoder 415B, and audio encoder 416B. Each of the encoders 413B, 414B, 415B and 416B can be a positional encoder that uses a network to extract Fourier features thus learning high frequency features from a low-dimensional domain. The encoders 413B, 414B, 415B and 416B receive the corresponding prompt type (408B, 409B, 410B an 411B respectively) and produce tensors 418B, 419B, 420B, 421B, respectively. The prompt encoders 413B, 414B, 415B and 416B may be pre-trained with pre-determined weights for natural language processing to encode prompt data relating to text or audio as prompts in tensor form. When the prompt data 162 comprises the mask prompt 407B, a mask prompt encoder 412B is used. The mask prompt encoder 412B contains convolutional layers 422B to transform the mask 407B and produce a tensor 417B. Tensor 115 is output of NN part 1 (the CNN backbone114) when SAM is used as shown in Fig. 4B(I). The prompt encoder outputs the prompt tensors 166, comprising one or more of 417B to 421B as shown in Fig. 4B(II). [000112]     Fig 4C is an arrangement 400C of the image encoder 401B and the image embedding 403B. In the example of Fig. 4C, a Vision Transformer (ViT) backbone for object detection was used. Unlike typical convolutional neural networks such as FPN having multiscale and hierarchical architecture, the original ViT is a plain, non-hierarchical architecture that maintains a single-scale feature map throughout. [000113]     The video frame 113 is input to a preprocessing block 430B to produce an image 432B. The pre-processing block 430B changes the colour mode of the video frame 113 to RGB 2025202067   21 Mar 2025 and resizes the video frame 113 to a predetermined size, typically 1024 width and 1024 height. The image 432B is input to encoder blocks 434B to produce the tensor 402B. The encoder blocks 434B contain a number of blocks B, typically 12 blocks B. Each of block B contains a normalization layer, an attention block, and a multilayer perceptron that contains two linear layers. [000114]     The two convolutional layers with kernel size equal 1 and 3, 438B and 444 respectively, are used for the image embedding module 403B. The convolutional layer 438B produces tensor 440B having a relatively high number of channels, i.e, 1280 channels. The convolutional layer 444B reduces the number of channels of tensor 440B to a smaller number, i.e. 256 channels, and produces the tensor 115 as shown in Fig. 4C. [000115]     Fig. 5A is a schematic block diagram 500 one type of bottleneck encoder, which may serve as the bottleneck encoder 116. A multi-scale feature fusion (MSFF) block 510 in the bottleneck encoder produces one tensor 530 from a plurality of tensors, i.e., the tensors 115. A single-scale feature compression (SSFC) encoder 550 performs compression of the tensor 530 to produce the tensor 117, having a lower channel count than the tensor 550, produced using learned layers such as convolutions. [000116]     Fig. 5B is one example 500B of the bottleneck encoder 116 when the CNN backbone 400 of Fig. 4A is used. The MSFF block 510, under execution of the processor 205, combines each of tensors 115 (i.e., 471, 473, 475, and 477), to produce the combined tensor 530. The combined tensor 530 forms a representation of the FPN layer tensors. Downsample modules 522a, 522b, and 522c operate on the tensors having larger spatial scale, i.e., P4 473 at (2h, 2w, 256), and P3 475 at (4h, 4w, 256), and P2 477 at (8h, 8w, 256), respectively. Modules 522a, 522b, and 522c perform downsampling to match the spatial scale of the smallest tensor, i.e., P5 471 at (h, w, 256), producing downscaled tensors 523a, 523b, 523c, respectively. A concatenation module 524 performs a channel-wise concatenation of the tensors 471, 523a, 523b, and 523c to produce concatenated tensor 525. The concatenated tensor 525 has dimensions (h, w, 1024). The concatenated tensor 525 is passed to a squeeze and excitation (SE) module 526 to produce a tensor 527. The SE module 526 sequentially performs a global pooling, a fully-connected layer with reduction in channel count, a rectified linear unit activation unit, a second fully-connected layer restoring the channel count, and a sigmoid activation function to produce a scaling tensor. The tensor 525 is scaled according to the scaling tensor to produce the output as the tensor 527. The SE block 526 is capable of being trained to 2025202067   21 Mar 2025 adaptively alter the weighting of different channels in the tensor passed through, based on the first fully-connected layer output. [000117]      The first fully-connected layer output reduces each feature map for each channel to a single value. Each single value is passed through a non-linear activation unit (ReLU) to create a conditional representation of the single value, suitable for weighting of other channels, with restoration to the full channel count performed by the second fully-connected layer. The SE block 526 is thus capable of extracting non-linear inter-channel correlation in producing the tensor 527 from the tensor 525, to a greater extent than is possible purely with convolutional (linear) layers. The tensor 527 is passed to a convolutional layer 528. The convolutional layer 528 implements one or more convolutional layers to produce the combined tensor 530, with channel count reduced to F channels, typically 256 channels (i.e., F = 256). Further reduction in the channel count is achieved by a single-scale feature compression (SSFC) module 550. [000118]      The SSFC module 550 receives the tensor 530 and applies a convolution 532 to reduce the channel count from F (256) down to C (nominally set to 64 channels) to produce tensor 553. The tensor 553 is then passed to a batch normalisation module 554 to produce batch normalised tensor 555, which is passed to a hyperbolic tangent activation layer 556 to produce the compressed tensor 117. The output of the MSFC module 500 (or 500B) is a one tensor per video frame 113 with a fixed feature map size and fixed channel count. [000119]     As described above, Fig. 5B shows an example of the bottleneck encoder 116 when an FPN network architecture described in Fig 4A is used in NN part 1. In a case of use of the architecture in Fig. 3A, a similar manner can be used in the bottleneck encoder 116. For example, in a case where the architecture in Fig. 3A is used, the MSFF block 510 combines each of tensors (i.e., 329, 326, 322) by using down-sampling, concatenation, SE process, and convolution layer, to produce the combined tensor 530. The SSFC encoder 550 performs channel-wise compression (similarly to the example of the SSFC encoder 550 in Fig. 5B) for the combined tensor 530 to produce the tensor 117. [000120]     When using the architecture in described with reference to Fig. 4B, as the CNN backbone 114, the input tensor 115 provided to the bottleneck encoder includes one tensor generated for each video frame 113. Hence, when using the architecture of Fig. 4B as the CNN backbone 114, the MSFF block 510 process may be skipped and the SSFC encoder 550 performs the channel-wise compression directly on the input tensor 115 to produce the tensor 117. 2025202067   21 Mar 2025 [000121]     Fig. 6 shows a method 600 for encoding frame data into a bitstream to generate the bistream, the method 600 includes performing a first portion of a CNN, dimensionality reduction using the bottleneck encoder 550, and encoding the resulting dimensionally-reduced feature maps to the bitstream. The method 600 also operates to encode prompt data to the bitstream. Fig. 7 shows a packing arrangement for a feature frame 700 of feature maps from compressed (dimensionally reduced) tensors into a monochrome video frame. [000122]     Referring to Fig. 6, the method 600 may be implemented using apparatus such as a configured FPGA, an ASIC, or an ASSP. Alternatively, as described below, the method 600 may be implemented by the source device 110, as one or more software code modules of the application programs 233, under execution of the processor 205. The software code modules of the application programs 233 implementing the method 600 may be resident, for example, in the hard disk drive 210 and / or the memory 206. The method 600 is repeated for each frame of video data produced by the video source 112. The method 600 may be stored on computer-readable storage medium and / or in the memory 206. The method 600 begins at a perform neural network first portion step 610. [000123]     At the step 610, the CNN backbone 114, under execution of the processor 205, performs neural network layers corresponding to the first portion (NN part 1) of the neural network. The step 610 effectively implements the CNN backbone 114. For example, the CNN layers as described with reference to Fig. 3A, Fig. 4A, or Fig. 4B are performed to produce the tensor 115. Control in the processor 205 progresses from the step 610 to a bottleneck encoder step 615. [000124]     At the step 615 bottleneck encoder 116, under execution of the processor 205 of the source device 110, compresses the tensor(s) 115 into one combined tensor having reduced dimensionality relative to the tensor 115, outputting the tensor 117. In arrangements where the tensor 115 includes tensors from a feature pyramid network, a step 617 causes the MSFF block 510 to operate to combine each tensor from the FPN (i.e., from 115) to produce the combined tensor 530. Upon completion of the step 617 control in the processor 205 of the source device 110 progresses to a step 619. At the step 619, a SSFC encoder 550 performs learned layers, such as convolutions, on the combined tensor 530 to produce the tensor 117. The tensor 117 has reduced channel count relative to the channel count of the tensor 530. In implementations using the SAM layers 400B, the step 615 implements the step 619 directly on the tensor 115 output by 400B and does not execute the step 617. Control in the processor 205 of the source device 2025202067   21 Mar 2025 110 progresses from the step 615 upon completion of sub-step (619 to a perform pre-processing step 630. [000125]     If in some arrangements (e.g. when the CNN backbone 400B is used), the step 630 is performed. At the step 630, the prompt encoder 164, under execution of the processor 205 of the source device 110, receives the prompt data 162 and performs pre-processing on the prompt data 162. The pre-processing typically operates to prepare the prompt data for encoding and / or extract data for encoding. For example, the pre-processing can be a simple process such as extracting high frequency features in an audio prompt, or more complex process such as using convolutional layers to process a mask prompt of the prompt data 162. In other arrangement (e.g. when the CNN head other than the architecture 400B is used), the step 630 can be bypassed. Or, if the CNN backbone 400B is used and prompt data is not received from the prompt generator for the frame, the step 630 can be bypassed. The method 600 continues from the step 630 to a quantize and pack step 670. [000126]     At the step 670, the quantise and pack module 118, under execution of the processor 205 of the source device 110, quantises the tensor 117 to obtain quantised tensors and packs feature maps forming the quantised tensors into a single monochrome video frame to produce the feature frame 700. In the quantising, the quantise and pack module 118 may perform conversion for value of each element of each feature map of all feature maps forming the tensor 117 so that the value is converted from a floating point value to a integer value. An example of the feature frame 700 is shown in Fig. 7. The feature frame 700 corresponds to the feature frame 119. All feature maps forming the compressed tensor 117 for one video frame 113 are packed into the feature frame 700 as shown in Fig. 7. One feature map of the compressed tensor 117 corresponds to one rectangular area, such as an area 710 or an area 720. Returning to Fig. 6, control in the processor 205 progresses from the step 670 to an encode frame step 680. [000127]     At the step 680 the feature frame encoder 120, under execution of the processor 205, encodes the feature frame 700 (119) to produce the first bitstream portion 172. Operation of the feature frame encoder 120 is described with reference to Fig. 8A. The step 680 operates to encode, into a first bitstream portion, information based on one or more tensors. The encoded information corresponds to the feature frame 700 on which feature maps of tensor(s) are packed. Control in the processor 205 progresses the method 600 from the encode frame step 680 to an encode prompt data step 685. 2025202067   21 Mar 2025 [000128]     At the step 685, the prompt encoder 164, under execution of the processor 205, encodes the pre-processed prompt data output by the step 630 into a second bitstream portion 166. The second bitstream portion 166 includes syntax structures such as FCM SPS (sequence parameter set) and FCM PH (picture header) to be described with Fig 8B. The prompt encoder 164 encodes the prompt data 162 into the FCM SEI to be described later with reference to Fig. 8B. When the NN part 1 described in Fig. 4B with the prompt encoder performed as part of the CNN backbone 114 is used, the prompt encoder 164 encodes the prompt data 162 to produce tensors (417B, 418B, 419B, 420B, 421B) corresponding to the second bitstream portion which is to be signalled into an FCM SEI. The step 685 operates to encode prompt information into a second bitstream portion, the prompt information being associated with the information encoded at step 680 and is related to a target object (as described in relation to 160 and 162). Structure of the prompt information is described in relation to Fig. 8B. Control in the processor 205 progresses from the step 685 to a multiplexing step 690. [000129]     At the step 690, the multiplexer 170 produces or generates the bitstream 121 by multiplexing the first bitstream portion 172 and the second bitstream portion 166 into a single bitstream. The first bitstream 172 contains coded feature frame and is written to an FCMFVD 80200 (described in Fig. 8B), and the second bitstream portion 172 can be delivered by FCMRSD (restoration data) 80100 will be described in Fig. 8B. [000130]     Fig. 8A is a schematic block diagram 800 showing functional modules of the feature frame encoder 120, also referred to as a feature map encoder. The feature frame encoder 120 encodes the feature frame 119, for example corresponding to the frame 700 in the example of Fig. 7, to produce the bitstream 172. Generally, data passes between functional modules within the encoder 120 in groups of samples or coefficients, such as divisions of blocks into sub-blocks of a fixed size, or as arrays. The encoder 120 may be implemented using a general-purpose computer system 200, as shown in Figs. 2A and 2B, where the various functional modules may be implemented by dedicated hardware within the computer system 200, by software executable within the computer system 200 such as one or more software code modules of the software application program 233 resident on the hard disk drive 205 and being controlled in its execution by the processor 205. Alternatively, the encoder 120 may be implemented by a combination of dedicated hardware and software executable within the computer system 200. The encoder 120 and the described methods may alternatively be implemented in dedicated hardware, such as one or more integrated circuits performing the functions or sub functions of the described methods. Such dedicated hardware 2025202067   21 Mar 2025 may include graphic processing units (GPUs), digital signal processors (DSPs), applicationspecific standard products (ASSPs), application-specific integrated circuits (ASICs), field programmable gate arrays (FPGAs) or one or more microprocessors and associated memories. In particular, the encoder 120 comprises modules 810-890 which may each be implemented as one or more software code modules of the software application program 233. [000131]     Although the encoder 120 of Fig. 8A is an example of a versatile video coding (VVC) video encoding pipeline, other video codecs may also be used to perform the processing stages described herein. The feature frame 119 may be in any chroma format and bit depth supported by the profile in use, for example 4:0:0, 4:2:0 for the “Main 10” profile of the VVC standard, at eight (8) to ten (10) bits in sample precision. [000132]     A block partitioner 810 firstly divides the feature frame 119 into CTUs, generally square in shape and configured such that a particular size for the CTUs is used. The maximum enabled size of the CTUs may be 32*32, 64*64, or 128*128 luma samples for example, configured by a ‘spslog2ctusize minus5’ syntax element presents in the ‘sequence parameter set’. The CTU size also provides a maximum CU size, as a CTU with no further splitting will contain one CU. The block partitioner 810 further divides each CTU into one or more CBs according to a luma coding tree and a chroma coding tree. The luma channel may also be referred to as a primary colour channel. Each chroma channel may also be referred to as a secondary colour channel. The CBs have a variety of sizes and may include both square and non-square aspect ratios. However, in the VVC standard, CBs, CUs, PUs, and TUs always have side lengths that are powers of two. Thus, a current CB, represented as 812, is output from the block partitioner 810, progressing in accordance with an iteration over the one or more blocks of the CTU, in accordance with the luma coding tree and the chroma coding tree of the CTU. [000133]     The CTUs resulting from the first division of the feature frame 119 may be scanned in raster scan order and may be grouped into one or more ‘slices’. A slice may be an ‘intra’ (or ‘I’) slice. An intra slice (I slice) indicates that every CU in the slice is intra predicted. Generally, the first picture in a coded layer video sequence (CLVS) contains only I slices and is referred to as an ‘intra picture’. The CLVS may contain periodic intra pictures, forming ‘random access points’ (i.e., intermediate frames in a video sequence upon which decoding can commence). Alternatively, a slice may be uni- or bi-predicted (‘P’ or ‘B’ slice, respectively), indicating additional availability of uni- and bi-prediction in the slice, respectively. 2025202067   21 Mar 2025 [000134]     The encoder 120 encodes sequences of pictures according to a picture structure. One picture structure is ‘low delay’, in which case pictures using inter-prediction may only reference pictures occurring previously in the sequence. Low delay enables each picture to be output as soon as the picture is decoded, in addition to being stored for possible reference by a subsequent picture. Another picture structure is ‘random access’, whereby the coding order of pictures differs from the display order. Random access allows inter-predicted pictures to reference other pictures that, although decoded, have not yet been output. A degree of picture buffering is needed so the reference pictures in the future in terms of display order are present in the decoded picture buffer, resulting in a latency of multiple frames. [000135]     When a chroma format other than 4:0:0 is in use, in an I slice, the coding tree of each CTU may diverge below the 64*64 level into two separate coding trees, one for luma and another for chroma. Use of separate trees allows different block structure to exist between luma and chroma within a luma 64*64 area of a CTU. For example, a large chroma CB may be collocated with numerous smaller luma CBs and vice versa. In a P or B slice, a single coding tree of a CTU defines a block structure common to luma and chroma. The resulting blocks of the single tree may be intra predicted or inter predicted. [000136]      In addition to a division of pictures into slices, pictures may also be divided into ‘tiles’. A tile is a sequence of CTUs covering a rectangular region of a picture. CTU scanning occurs in a raster-scan manner within each tile and progresses from one tile to the next. A slice can be either an integer number of tiles, or an integer number of consecutive rows of CTUs within a given tile. [000137]     For each CTU, the encoder 120 operates in two stages. In the first stage (referred to as a ‘search’ stage), the block partitioner 810 tests various potential configurations of a coding tree. Each potential configuration of a coding tree has associated ‘candidate’ CBs. The first stage involves testing various candidate CBs to select CBs providing relatively high compression efficiency with relatively low distortion. The testing generally involves a Lagrangian optimisation whereby a candidate CB is evaluated based on a weighted combination of rate (i.e., coding cost) and distortion (i.e., error with respect to the feature frame 119). ‘Best’ candidate CBs (i.e., the CBs with the lowest evaluated rate / distortion) are selected for subsequent encoding into the bitstream 172. Included in evaluation of candidate CBs is an option to use a CB for a given area or to further split the area according to various splitting options and code each of the smaller resulting areas with further CBs, or split the areas even 2025202067   21 Mar 2025 further. As a consequence, both the coding tree and the CBs themselves are selected in the search stage. [000138]     The encoder 120 produces a prediction block (PB), indicated by an arrow 820, for each CB, for example, CB 812. The PB 820 is a prediction of the contents of the associated CB 812. A subtracter module 822 produces a difference, indicated as 824 (or ‘residual’, referring to the difference being in the spatial domain), between the PB 820 and the CB 812. The difference 824 is a block-size difference between corresponding samples in the PB 820 and the CB 812. The difference 824 is transformed, quantised and represented as a transform block (TB), indicated by an arrow 836. The PB 820 and associated TB 836 are typically chosen from one of many possible candidate CBs, for example, based on evaluated cost or distortion. [000139]     A candidate coding block (CB) is a CB resulting from one of the prediction modes available to the encoder 120 for the associated PB and the resulting residual. When combined with the predicted PB in the encoder 120, the TB 836 reduces the difference between a decoded CB and the original CB 812 at the expense of additional signalling in a bitstream. [000140]     Each candidate coding block (CB), that is prediction block (PB) in combination with a transform block (TB), thus has an associated coding cost (or ‘rate’) and an associated difference (or ‘distortion’). The distortion of the CB is typically estimated as a difference in sample values, such as a sum of absolute differences (SAD), a sum of squared differences (SSD) or a Hadamard transform applied to the differences. The estimate resulting from each candidate PB may be determined by a mode selector 886 using the difference 824 to determine a prediction mode 887. The prediction mode 887 indicates the decision to use a particular prediction mode for the current CB, for example, intra-frame prediction or inter-frame prediction. Estimation of the coding costs associated with each candidate prediction mode and corresponding residual coding may be performed at significantly lower cost than entropy coding of the residual. Accordingly, a number of candidate modes may be evaluated to determine an optimum mode in a rate-distortion sense even in a real-time video encoder. [000141]     Determining an optimum mode in terms of rate-distortion is typically achieved using a variation of Lagrangian optimisation. [000142]     Lagrangian or similar optimisation processing can be employed to both select an optimal partitioning of a CTU into CBs (by the block partitioner 810) as well as the selection of 2025202067   21 Mar 2025 a best prediction mode from a plurality of possibilities. Through application of a Lagrangian optimisation process of the candidate modes in the mode selector module 886, the intra prediction mode with the lowest cost measurement is selected as the ‘best’ mode. The lowest cost mode includes a selected secondary transform index 888, which is also encoded in the first bitstream portion 172 by an entropy encoder 838. [000143]     In the second stage of operation of the encoder 120 (referred to as a ‘coding’ stage), an iteration over the determined coding tree(s) of each CTU is performed in the encoder 120. For a CTU using separate trees, for each 64x64 luma region of the CTU, a luma coding tree is firstly encoded followed by a chroma coding tree. Within the luma coding tree, only luma CBs are encoded and within the chroma coding tree only chroma CBs are encoded. For a CTU using a shared tree, a single tree describes the CUs (i.e., the luma CBs and the chroma CBs) according to the common block structure of the shared tree. [000144]     The entropy encoder 838 supports bitwise coding of syntax elements using variable-length and fixed-length codewords, and an arithmetic coding mode for syntax elements. Portions of the bitstream such as ‘parameter sets’, for example, sequence parameter set (SPS) and picture parameter set (PPS) use a combination of fixed-length codewords and variable-length codewords. Slices, also referred to as contiguous portions, have a slice header that uses variable length coding followed by slice data, which uses arithmetic coding. The slice header defines parameters specific to the current slice, such as slice-level quantisation parameter offsets. The slice data includes the syntax elements of each CTU in the slice. Use of variable length coding and arithmetic coding requires sequential parsing within each portion of the bitstream. The portions may be delineated with a start code to form ‘network abstraction layer units’ or ‘NAL units’. Arithmetic coding is supported using a context-adaptive binary arithmetic coding process. [000145]     Arithmetically coded syntax elements consist of sequences of one or more ‘bins’. Bins, like bits, have a value of ‘0’ or ‘1’. However, bins are not encoded in the bitstream 172 as discrete bits. Bins have an associated predicted (or ‘likely’ or ‘most probable’) value and an associated probability, known as a ‘context’. When the actual bin to be coded matches the predicted value, a ‘most probable symbol’ (MPS) is coded. Coding a most probable symbol is relatively inexpensive in terms of consumed bits in the bitstream 172, including costs that amount to less than one discrete bit. When the actual bin to be coded mismatches the likely value, a ‘least probable symbol’ (LPS) is coded. Coding a least probable symbol has a relatively 2025202067   21 Mar 2025 high cost in terms of consumed bits. The bin coding techniques enable efficient coding of bins where the probability of a ‘0’ versus a ‘1’ is skewed. For a syntax element with two possible values (i.e., a ‘flag’), a single bin is adequate. For syntax elements with many possible values, a sequence of bins is needed. [000146]     The presence of later bins in the sequence may be determined based on the value of earlier bins in the sequence. Additionally, each bin may be associated with more than one context. The selection of a particular context may be dependent on earlier bins in the syntax element, the bin values of neighbouring syntax elements (i.e., those from neighbouring blocks) and the like. Each time a context-coded bin is encoded, the context that was selected for that bin (if any) is updated in a manner reflective of the new bin value. As such, the binary arithmetic coding scheme is said to be adaptive. [000147]     Also supported by the entropy encoder 838 are bins that lack a context, referred to as “bypass bins”. Bypass bins are coded assuming an equiprobable distribution between a ‘0’ and a ‘1’. Thus, each bin has a coding cost of one bit in the bitstream 172. The absence of a context saves memory and reduces complexity, and thus bypass bins are used where the distribution of values for the particular bin is not skewed. One example of an entropy coder employing context and adaption is known in the art as CABAC (context adaptive binary arithmetic coder) and many variants of this coder have been employed in video coding. [000148]     The entropy encoder 838 encodes a quantisation parameter 892 and, if in use for the current CB, the LFNST index 888, using a combination of context-coded and bypass-coded bins. The quantisation parameter 892 is encoded using a ‘delta QP’ generated by a QP controller module 890. The delta QP is signalled at most once in each area known as a ‘quantisation group’. The quantisation parameter 892 is applied to residual coefficients of the luma CB. An adjusted quantisation parameter is applied to the residual coefficients of collocated chroma CBs. The adjusted quantisation parameter may include mapping from the luma quantisation parameter 892 according to a mapping table and a CU-level offset, selected from a list of offsets. The secondary transform index 888 is signalled when the residual associated with the transform block includes significant residual coefficients only in those coefficient positions subject to transforming into primary coefficients by application of a secondary transform. [000149]     Residual coefficients of each TB associated with a CB are coded using a residual syntax. The residual syntax is designed to efficiently encode coefficients with low magnitudes, 2025202067   21 Mar 2025 using mainly arithmetically coded bins to indicate significance of coefficients, along with lower-valued magnitudes and reserving bypass bins for higher magnitude residual coefficients. Accordingly, residual blocks comprising very low magnitude values and sparse placement of significant coefficients are efficiently compressed. Moreover, two residual coding schemes are present. A regular residual coding scheme is optimised for TBs with significant coefficients predominantly located in the upper-left corner of the TB, as is seen when a transform is applied. A transform-skip residual coding scheme is available for TBs where a transform is not performed and is able to efficiently encode residual coefficients regardless of their distribution throughout the TB. [000150]     A multiplexer module 884 outputs the PB 820 from an intra-frame prediction module 864 according to the determined best intra prediction mode, selected from the tested prediction mode of each candidate CB. The candidate prediction modes need not include every conceivable prediction mode supported by the encoder 120. Intra prediction falls into three types, first, “DC intra prediction”, which involves populating a PB with a single value representing the average of nearby reconstructed samples; second, “planar intra prediction”, which involves populating a PB with samples according to a plane, with a DC offset and a vertical and horizontal gradient being derived from nearby reconstructed neighbouring samples. The nearby reconstructed samples typically include a row of reconstructed samples above the current PB, extending to the right of the PB to an extent and a column of reconstructed samples to the left of the current PB, extending downwards beyond the PB to an extent The third type is “angular intra prediction”, which involves populating a PB with reconstructed neighbouring samples filtered and propagated across the PB in a particular direction (or ‘angle’). In VVC, sixty-five (65) angles are supported, with rectangular blocks able to utilise additional angles, not available to square blocks, to produce a total of eighty-seven (87) angles. [000151]     A fourth type of intra prediction is available to chroma PBs, whereby the PB is generated from collocated luma reconstructed samples according to a ‘cross-component linear model’ (CCLM) mode. Three different CCLM modes are available, each mode using a different model derived from the neighbouring luma and chroma samples. The derived model is used to generate a block of samples for the chroma PB from the collocated luma samples. Luma blocks may be intra predicted using a matrix multiplication of the reference samples using one matrix selected from a predefined set of matrices. This matrix intra prediction (MIP) achieves gain by using matrices trained on a large set of video data, with the matrices representing relationships 2025202067   21 Mar 2025 between reference samples and a predicted block that are not easily captured in angular, planar, or DC intra prediction modes. [000152]     The module 864 may also produce a prediction unit by copying a block from nearby the current frame using an ‘intra block copy’ (IBC) method. The location of the reference block is constrained to an area equivalent to one CTU, divided into 64x64 regions known as VPDUs, with the area covering the processed VPDUs of the current CTU and VPDUs of the previous CTU(s) within each row or CTUs and within each slice or tile up to the area limit corresponding to one 128x128 luma samples, regardless of the configured CTU size for the bitstream. This area is known as an ‘IBC virtual buffer’ and limits the IBC reference area, thus limiting the required storage. The IBC buffer is populated with reconstructed samples 854 (i.e., prior to loop filtering), and so a separate buffer to a frame buffer 872 is needed. When the CTU size is 128x128 the virtual buffer includes samples only from the CTU adjacent and to the left of the current CTU. When the CTU size is 32x32 or 64x64 the virtual buffer includes CTUs from up to the four or sixteen CTUs to the left of the current CTU. Regardless of the CTU size, access to neighbouring CTUs for obtaining samples for IBC reference blocks is constrained by boundaries such as edges of pictures, slices, or tiles. Especially for feature maps of FPN layers having smaller dimensions, use of a CTU size such as 32x32 or 64x64 results in a reference area more aligned to cover a set of previous feature maps. Where feature map placement is ordered based on SAD, SSE or other difference metric, access to similar feature maps for IBC prediction offers coding efficient advantage. [000153]     The residual for a predicted block when encoding feature map data is different to the residual seen for natural video. Such natural video is typically captured by an imaging sensor, or screen content, as generally seen in operating system user interfaces and the like. Feature map residuals tend to contain much detail, which is amenable to transform skip coding more than predominantly low-frequency coefficients of various transforms. Experiments show that the feature map residual has enough local similarity to benefit from transform coding. However, the distribution of feature map residual coefficients is not clustered towards the DC (top-left) coefficient of a transform block. In other words, sufficient correlation exists for a transform to show gain when encoding feature map data and this is true also for when intra block copy is used to produce prediction blocks for the feature map data. Accordingly, a Hadamard cost estimate may be used when evaluating residuals resulting from candidate block vectors for intra block copy when encoding feature map data, instead of relying solely on a SAD or SSD cost estimate. SAD or SSD cost estimates tend to select block vectors with 2025202067   21 Mar 2025 residuals more amenable to transform skip coding and may miss block vectors with residuals that would be compactly encoded using transforms. The multiple transform selection (MTS) tool of the VVC standard may be used when encoding feature map data so that, in addition to the DCT-2 transform, combinations of DST-7 and DCT-8 transforms are available horizontally and vertically for residual encoding. [000154]     An intra-predicted luma coding block may be partitioned into a set of equal sized prediction blocks, either vertically or horizontally, which each block having a minimum area of sixteen (16) luma samples. This intra sub-partition (ISP) approach enables separate transform blocks to contribute to prediction block generation from one sub-partition to the next sub-partition in the luma coding block, improving compression efficiency. [000155]     Where previously reconstructed neighbouring samples are unavailable, for example at the edge of the frame, a default half-tone value of one half the range of the samples is used. For example, for 10-bit video a value of five-hundred and twelve (512) is used. As no previous samples are available for a CB located at the top-left position of a frame, angular and planar intra-prediction modes produce the same output as the DC prediction mode (i.e. a flat plane of samples having the half-tone value as magnitude). [000156]     For inter-frame prediction a prediction block 882 is produced using samples from one or two frames preceding the current frame in the coding order frames in the bitstream by a motion compensation module 880 and output as the PB 820 by the multiplexer module 884. Moreover, for inter-frame prediction, a single coding tree is typically used for both the luma channel and the chroma channels. The order of coding frames in the bitstream may differ from the order of the frames when captured or displayed. When one frame is used for prediction, the block is said to be ‘uni-predicted’ and has one associated motion vector. When two frames are used for prediction, the block is said to be ‘bi-predicted’ and has two associated motion vectors. For a P slice, each CU may be intra predicted or uni-predicted. For a B slice, each CU may be intra predicted, uni-predicted, or bi-predicted. [000157]     Frames are typically coded using a ‘group of pictures’ (GOP) structure, enabling a temporal hierarchy of frames. Frames may be divided into multiple slices, each of which encodes a portion of the frame. A temporal hierarchy of frames allows a frame to reference a preceding and a subsequent picture in the order of displaying the frames. The images are coded in the order necessary to ensure the dependencies for decoding each frame are met. An affine 2025202067   21 Mar 2025 inter prediction mode is available where instead of using one or two motion vectors to select and filter reference sample blocks for a prediction unit, the prediction unit is divided into multiple smaller blocks and a motion field is produced so each smaller block has a distinct motion vector. The motion field uses the motion vectors of nearby points to the prediction unit as ‘control points’. Affine prediction allows coding of motion different to translation with less need to use deeply split coding trees. A bi-prediction mode available to VVC performs a geometric blend of the two reference blocks along a selected axis, with angle and offset from the centre of the block signalled. This geometric partitioning mode (“GPM”) allows larger coding units to be used along the boundary between two objects, with the geometry of the boundary coded for the coding unit as an angle and centre offset. Motion vector differences, instead of using cartesian (x, y) offset, may be coded as a direction (up / down / left / right) and a distance, with a set of power-of-two distances supported. The motion vector predictor is obtained from a neighbouring block (‘merge mode’) as if no offset is applied. The current block will share the same motion vector as the selected neighbouring block. [000158]     The samples are selected according to a motion vector 878 and reference picture index. The motion vector 878 and reference picture index applies to all colour channels and thus inter prediction is described primarily in terms of operation upon PUs rather than PBs. The decomposition of each CTU into one or more inter-predicted blocks is described with a single coding tree. Inter prediction methods may vary in the number of motion parameters and their precision. Motion parameters typically comprise a reference frame index, indicating which reference frame(s) from lists of reference frames are to be used plus a spatial translation for each of the reference frames, but may include more frames, special frames, or complex affine parameters such as scaling and rotation. In addition, a pre-determined motion refinement process may be applied to generate dense motion estimates based on referenced sample blocks. [000159]     Having determined and selected the PB 820 and subtracted the PB 820 from the original sample block at the subtractor 822, the residual with lowest coding cost, represented as 824, is obtained and subjected to lossy compression. The lossy compression process comprises the steps of transformation, quantisation and entropy coding. A forward primary transform module 826 applies a forward transform to the difference 824, converting the difference 824 from the spatial domain to the frequency domain, and producing primary transform coefficients represented by an arrow 828. The largest primary transform size in one dimension is either a 32-point DCT-2 or a 64-point DCT-2 transform, configured by a ‘spsmax luma transformsize 64flag’ in the sequence parameter set. If the CB being 2025202067   21 Mar 2025 encoded is larger than the largest supported primary transform size expressed as a block size (e.g. 64^64 or 32x32), the primary transform 826 is applied in a tiled manner to transform all samples of the difference 824. Where a non-square CB is used, tiling is also performed using the largest available transform size in each dimension of the CB. For example, when a maximum transform size of thirty-two (32) is used, a 64x16 CB uses two 32x16 primary transforms arranged in a tiled manner. When a CB is larger in size than the maximum supported transform size, the CB is filled with TBs in a tiled manner. For example, a 128x128 CB with 64-pt transform maximum size is filled with four 64x64 TBs in a 2x2 arrangement. A 64x128 CB with a 32-pt transform maximum size is filled with eight 32x32 TBs in a 2x4 arrangement. [000160]     Application of the transform 826 results in multiple TBs for the CB. Where each application of the transform operates on a TB of the difference 824 larger than 32x32, e.g. 64x64, all resulting primary transform coefficients 828 outside of the upper-left 32x32 area of the TB are set to zero (i.e., discarded). The remaining primary transform coefficients 828 are passed to a quantiser module 834. The primary transform coefficients 828 are quantised according to the quantisation parameter 892 associated with the CB to produce primary transform coefficients 832. In addition to the quantisation parameter 892, the quantiser module 834 may also apply a ‘scaling list’ to allow non-uniform quantisation within the TB by further scaling residual coefficients according to their spatial position within the TB. The quantisation parameter 892 may differ for a luma CB versus each chroma CB. The primary transform coefficients 832 are passed to a forward secondary transform module 830 to produce the transform coefficients represented by the arrow 836 by performing either a non-separable secondary transform (NSST) operation or bypassing the secondary transform. The forward primary transform is typically separable, transforming a set of rows and then a set of columns of each TB. The forward primary transform module 826 uses either a type-II discrete cosine transform (DCT-2) in the horizontal and vertical directions, or bypass of the transform horizontally and vertically, or combinations of a type-VII discrete sine transform (DST-7) and a type-VIII discrete cosine transform (DCT-8) in either horizontal or vertical directions for luma TBs not exceeding 16 samples in width and height. Use of combinations of a DST-7 and DCT-8 is referred to as ‘multi transform selection set’ (MTS) in the VVC standard. [000161]     The forward secondary transform of the module 830 is generally a non-separable transform, which is only applied for the residual of intra-predicted CUs and may nonetheless also be bypassed. The forward secondary transform operates either on sixteen (16) samples (arranged as the upper-left 4x4 sub-block of the primary transform coefficients 828) or forty- 2025202067   21 Mar 2025 eight (48) samples (arranged as three 4x4 sub-blocks in the upper-left 8x8 coefficients of the primary transform coefficients 828) to produce a set of secondary transform coefficients. The set of secondary transform coefficients may be fewer in number than the set of primary transform coefficients from which they are derived. Due to application of the secondary transform to only a set of coefficients adjacent to each other and including the DC coefficient, the secondary transform is referred to as a ‘low frequency non-separable secondary transform’ (LFNST). Such secondary transforms may be obtained through a training process and due to their non-separable nature and trained origin, exploit additional redundancy in the residual signal not able to be captured by separable transforms such as variants of DCT and DST. Moreover, when the LFNST is applied, all remaining coefficients in the TB are zero, both in the primary transform domain and the secondary transform domain. [000162]     The quantisation parameter 892 is constant for a given TB and thus results in a uniform scaling to produce residual coefficients in the primary transform domain for a TB. The quantisation parameter 892 may vary periodically with a signalled ‘delta quantisation parameter’. The delta quantisation parameter (delta QP) is signalled once for CUs contained within a given area, referred to as a ‘quantisation group’. If a CU is larger than the quantisation group size, delta QP is signalled once with one of the TBs of the CU. That is, the delta QP is signalled by the entropy encoder 838 once for the first quantisation group of the CU and not signalled for any subsequent quantisation groups of the CU. A non-uniform scaling is also possible by application of a ‘quantisation matrix’, whereby the scaling factor applied for each residual coefficient is derived from a combination of the quantisation parameter 892 and the corresponding entry in a scaling matrix. The scaling matrix may have a size that is smaller than the size of the TB, and when applied to the TB a nearest neighbour approach is used to provide scaling values for each residual coefficient from a scaling matrix smaller in size than the TB size. The residual coefficients 836 are supplied to the entropy encoder 838 for encoding in the bitstream 172. Typically, the residual coefficients of each TB with at least one significant residual coefficient of the TU are scanned to produce an ordered list of values, according to a scan pattern. The scan pattern generally scans the TB as a sequence of 4x4 ‘sub-blocks’, providing a regular scanning operation at the granularity of 4x4 sets of residual coefficients, with the arrangement of sub-blocks dependent on the size of the TB. The scan within each subblock and the progression from one sub-block to the next typically follow a backward diagonal scan pattern. Additionally, the quantisation parameter 892 is encoded into the bitstream 172 2025202067   21 Mar 2025 using a delta QP syntax element, and a slice QP for the initial value in a given slice or subpicture and the secondary transform index 888 is encoded in the bitstream 172. [000163]     As described above, the encoder 120 needs access to a frame representation corresponding to the decoded frame representation seen in the video decoder. Thus, the residual coefficients 836 are passed through an inverse secondary transform module 844, operating in accordance with the secondary transform index 888 to produce intermediate inverse transform coefficients, represented by an arrow 842. The intermediate inverse transform coefficients 842 are inverse quantised by a dequantiser module 840 according to the quantisation parameter 892 to produce inverse transform coefficients, represented by an arrow 846. The dequantiser module 840 may also perform an inverse non-uniform scaling of residual coefficients using a scaling list, corresponding to the forward scaling performed in the quantiser module 834. The inverse transform coefficients 846 are passed to an inverse primary transform module 848 to produce residual samples, represented by an arrow 850, of the TU. The inverse primary transform module 848 applies DCT-2 transforms horizontally and vertically, constrained by the maximum available transform size as described with reference to the forward primary transform module 826. The types of inverse transform performed by the inverse secondary transform module 844 correspond with the types of forward transform performed by the forward secondary transform module 830. The types of inverse transform performed by the inverse primary transform module 848 correspond with the types of primary transform performed by the primary transform module 826. A summation module 852 adds the residual samples 850 and the PU 820 to produce reconstructed samples (indicated by the arrow 854) of the CU. [000164]     The reconstructed samples 854 are passed to a reference sample cache 856 and an in-loop filters module 868. The reference sample cache 856, typically implemented using static RAM on an ASIC to avoid costly off-chip memory access, provides minimal sample storage needed to satisfy the dependencies for generating intra-frame PBs for subsequent CUs in the frame. The minimal dependencies typically include a ‘line buffer’ of samples along the bottom of a row of CTUs, for use by the next row of CTUs and column buffering the extent of which is set by the height of the CTU. The reference sample cache 856 supplies reference samples (represented by an arrow 858) to a reference sample filter 860. The sample filter 860 applies a smoothing operation to produce filtered reference samples (indicated by an arrow 862). The filtered reference samples 862 are used by the intra-frame prediction module 864 to produce an intra-predicted block of samples, represented by an arrow 866. For each candidate intra prediction mode the intra-frame prediction module 864 produces a block of 2025202067   21 Mar 2025 samples, that is 866. The block of samples 866 is generated by the module 864 using techniques such as DC, planar or angular intra prediction. The block of samples 866 may also be produced using a matrix-multiplication approach with neighbouring reference sample as input and a matrix selected from a set of matrices by the encoder 120, with the selected matrix signalled in the bitstream 172 using an index to identify which matrix of the set of matrices is to be used by the decoder 144. [000165]      The in-loop filters module 868 applies several filtering stages to the reconstructed samples 854. The filtering stages include a ‘deblocking filter’ (DBF) which applies smoothing aligned to the CU boundaries to reduce artefacts resulting from discontinuities. Another filtering stage present in the in-loop filters module 868 is an ‘adaptive loop filter’ (ALF), which applies a Wiener-based adaptive filter to further reduce distortion. A further available filtering stage in the in-loop filters module 868 is a ‘sample adaptive offset’ (SAO) filter. The SAO filter operates by firstly classifying reconstructed samples into one or multiple categories and, according to the allocated category, applying an offset at the sample level. [000166]     Filtered samples, represented by an arrow 870, are output from the in-loop filters module 868. The filtered samples 870 are stored in the frame buffer 872. The frame buffer 872 typically has the capacity to store several (e.g., up to sixteen (16)) pictures and thus is stored in the memory 206. The frame buffer 872 is not typically stored using on-chip memory due to the large memory consumption required. As such, access to the frame buffer 872 is costly in terms of memory bandwidth. The frame buffer 872 provides reference frames (represented by an arrow 874) to a motion estimation module 876 and the motion compensation module 880. [000167]     The motion estimation module 876 estimates a number of ‘motion vectors’ (indicated as 878), each being a Cartesian spatial offset from the location of the present CB, referencing a block in one of the reference frames in the frame buffer 872. A filtered block of reference samples (represented as 882) is produced for each motion vector. The filtered reference samples 882 form further candidate modes available for potential selection by the mode selector 886. Moreover, for a given CU, the PU 820 may be formed using one reference block (‘uni-predicted’) or may be formed using two reference blocks (‘bi-predicted’). For the selected motion vector, the motion compensation module 880 produces the PB 820 in accordance with a filtering process supportive of sub-pixel accuracy in the motion vectors. As such, the motion estimation module 876 (which operates on many candidate motion vectors) 2025202067   21 Mar 2025 may perform a simplified filtering process compared to that of the motion compensation module 880 (which operates on the selected candidate only) to achieve reduced computational complexity. When the encoder 120 selects inter prediction for a CU the motion vector 878 is encoded into the bitstream 172. [000168]     Although the encoder 120 of Fig. 8A is described with reference to versatile video coding (VVC), other video coding standards or implementations may also employ the processing stages of modules 810-890. The feature frame 119 (and bitstream 172) may also be read from (or written to) memory 206, the hard disk drive 210, a CD-ROM, a Blu-ray diskTM or other computer readable storage medium. Additionally, the feature frame 119 (and bitstream 172) may be received from (or transmitted to) an external source, such as a server connected to the communications network 220 or a radio-frequency receiver. The communications network 220 may provide limited bandwidth, necessitating the use of rate control in the encoder 120 to avoid saturating the network at times when the feature frame 119 is difficult to compress. Moreover, the bitstream 172 may be constructed from one or more slices, representing spatial sections (collections of CTUs) of the feature frame 119, produced by one or more instances of the encoder 120, operating in a co-ordinated manner under control of the processor 205. The bitstream 172 may also contain one slice that corresponds to one subpicture to be output as a collection of subpictures forming one picture, each being independently encodable and independently decodable with respect to any of the other slices or subpictures in the picture. [000169]     Fig. 8B shows a hierarchical syntax structure 8000 to which the FCM bitstream 121 complies. The hierarchical syntax structure 8000 defines a hierarchy of syntax structures, some of which contain other syntax structures. The source device 110 produces the bitstream 121 in accordance with the syntax structure 8000 and the destination device 140 decodes the bitstream 143 in accordance with the hierarchical syntax structure 8000. The FCM vision model parameter set (VMPS) 8020 includes information needed for operation of NN part 2, such as output frame size, and is prefixed by a header 8024 that includes a size 8022 and a type 8022b. The size 8022 indicates a total byte size of HDR 8024 and FCMVMPS 8020, and the type 8022b indicates the following syntax is to be decoded as the FCMVMPS 8020. The FCMVMPS 8020 includes a VMPS ID 8022c. The FCM restoration data (RSD) unit 80100 includes an FCMSPS (sequence parameter set) 80130, an FCMPPS (picture parameter set) 80140, an FCMPH (picture header) 80150, and an FCMSEI 80160. Parameter sets in the FCMRSD 80100 are identified by an FCM NAL unit type (not shown), prefixing each 2025202067   21 Mar 2025 respective parameter set. The FCMRSD 80100 is prefixed by header 80104. The header 80104 includes a size 80102 indicating a total byte size of HDR 80104 and FCMRSD 80100, a type 80112 indicating FCMRSD, and a VMPS ID 80114 associating the FCMRSD 80100 with the FCMVMPS 8020. The FCMSPS 80130 includes e.g., information indicating the size each of the tensors 115 and information indicating the number of the tensors 115. The FCMPPS 80140 includes parameters that may vary during operation of feature encoder 168. [000170]     For each coded feature frame, a FCM picture header (PH), such as FCMPH 80150 is included in the FCMRSD 80100 in the bitstream 121. For example, if there are two coded feature frames (in other words there are two AUs) coded in the FCM CVD 80200, there are two corresponding FCMPHs present in the FCMRSD 80100 in bitstream 121. The FCMPH 80150 includes a POC (picture order count) 80154 and a picture parameter set ID (not shown) to reference a particular FCMPPS 80140. In a case where FCMPH 80150 has the same POC as the POC of an AU 80220 (i.e., POC #0), FCMPH 80150 is to be associated with the AU 80220. In this case, parameters of the FCMPH 80150 are applied to the feature frame corresponding to AU 80220. [000171]     The first bitstream portion 172, delineated into AUs, is encoded into FCM coded video data (CVD) units, such as FCMCVD 80200. In one arrangement, an FCMCVD may also be referred to as a FCMCFD (FCM coded feature data). Each AU contains the slice(s) of one compressed feature frame, and any additional parameter sets accompanying the compressed feature frame, such as an SPS, PPS, or PH, as may be required by the VVC or HEVC video compression standards. In the example of Fig. 8B, The FCMCVD 80200 includes frame date access units AU 80220 (POC #0) and AU 80280 (POC #1), each having an associated header (80210 and 80270 respectively). [000172]     The FCM SEI 80160 may comprise prompt information 80166 as encoded to the prompt bitstream 166 at step 685. The FCMSEI 80160 includes one or more SEI messages, each SEI message identified by an SEI message type. One SEI message type is assigned to indicate the presence of the prompt information 80166. The one example of the syntax structure of the FCMSEI 80160 is described as fcmpromptinformationseimessage in the Appendix A. In the example of Fig. 8A, the second bitstream portion 166 encoded at step 685 (represented as 80160) is effectively located adjacently following the FCM PH 80150. In other examples, the FCMSEI 80160 comprising prompt information can be located adjacently preceding the FCMPH. The prompt information 80166 contains a flag 80167 (e.g. 2025202067   21 Mar 2025 promptflag in Appendix A) and prompt data 80168. The prompt flag 80167 indicates whether prompt data is used or not. If prompt data is used, the NN part 2 (described in Fig. 12, Fig. 13, and Fig. 14) receives the prompt data 80168 to generate the target results. [000173]     The prompt data 80168 comprises one or more syntax elements. The prompt data 80168 comprises the syntax element (i.e. number of promptsminus1) used to identify the total number of prompts present in the FCMSEI. [000174]     In some arrangements (e.g. when using the CNN backbone 114 in Fig. 3A or Fig. 4A), a prompt point indicating a specific position in the video frame 113 may be signaled as the part of the prompt data 80168 by the prompt encoder 164 into the FCMSEI 80160 by using the syntax elements such as promptpoint x[ i ] and promptpoint y[ i ]. Also, bounding box prompt indicating a position of a bounding box in the video frame 113 may be signaled as the part of the prompt data 80168 by the prompt encoder 164 into the FCMSEI 80160 by using the syntax elements such as promptbox xmin[ i ], promptbox ymin[ i ], promptbox xmax[ i ], promptbox ymax[ i ]. Also, a prompt mask may be signaled as the part of the prompt data 80168 by the prompt encoder 164 into the FCMSEI 80160 by using the syntax elements such as promptmask payload len and promptmask payload. Also, prompt text may be signaled as the part of the prompt data 80168 by the prompt encoder 164 in the FCMSEI 80160 by using prompttext payload. Also, prompt audio may be signaled as the part of the prompt data 80168 by the prompt encoder 164 in the FCMSEI 80160 by using the syntax elements such as promptaudio payload len and promptaudiopayload. [000175]     In other arrangements (especially when using the arrangement 400B in Fig. 4B in CNN backbone 114), the prompt tensors 166 are signaled as the part of the prompt data 80168 by the prompt encoder 164 into the FCMSEI by using the syntax elements such as promptgenericlen and promptgenericpayload byte. [000176]     The FCMSEI 80160 is to be associated with the specific AU (i.e., the specific feature frame) by using a predetermined manner. For example, the predetermined manner may require that the FCMSEI 80160 includes a value of the specific POC (poc in the Appendix A) for the feature frame for which the prompt information in the FCMSEI 80160 applies. For example, if the FCMSEI 80160 has the value of 0 as the POC, the prompt information 80166 in the FCMSEI 80160 is associated with the AU 80220 (POC #0). In this case, when the NN part 2 is performed for the reconstructed tensors derived from the feature frame of the 2025202067   21 Mar 2025 AU 80220 (POC #0), the prompt data in the prompt information 80166 is used in the NN part 2. The prompt information 80166 accordingly includes an indication to associate the prompt information with the information (e.g. the feature frame) encoded to the bitstream 172 at step 680. Examples of the prompt data are described with reference to Fig. 15. [000177]     In another arrangement, or predetermined manner, of associating the FCMSEI 80160 with the specific AU, a positional relationship between the FCMSEI 80160 and FCMPH 80150 may be used. For example, if (a) the FCMSEI 80160 is located immediately before the FCMPH 80150, that is, the FCMSEI 80160 precedes the FCMPH 80150 with no other syntax structures in between and (b) the FCMPH 80150 has the same POC value as the AU 80220, then the prompt information of the FCMSEI 80160 is associated with the AU 80220. Not limited to this arrangement, the following arrangement may also be acceptable. That is, if (a) the FCMSEI 80160 is located immediately after (i.e., subsequent to and with no intervening syntax structures) the FCMPH 80150 and (b) the FCMPH 80150 has the same POC value as the AU 80220, then the prompt information of the FCMSEI 80160 is associated with the AU 80220. [000178]     In another arrangement, or predetermined manner, of associating the FCMSEI 80160 with the specific AU, the prompt information 80166 may be coded for a given AU using a prefix SEI NAL unit (the prefix SEI NAL unit is located immediately before the given AU in the bitstream) or a suffix SEI NAL unit ( the suffix SEI NAL unit is located immediately after the given AU in the bitstream). For example, if the FCMSEI (coded in the FCMRSD 80100 with an FCM NAL unit type indicating a “prefix SEI” message, similar to PREFIX SEI_NUT of HEVC or VVC) is located immediately before the AU 80220, then the prompt information 80166 in the FCMSEI is associated with the feature frame for the AU 80220. Or if the FCMSEI is coded with a FCM NAL unit type indicating a “suffix SEI” and is located immediately after the AU 80220, the prompt information 80166 in the FCMSEI is associated with the feature frame for the AU 80220. FCM NAL unit type 33 is allocated for the FCMSEI, which may be interpreted as a ‘prefix’ SEI message in some arrangements or as a “suffix” SEI message in other arrangements. Additionally, another FCM NAL unit type, such as 34, could be used to provide an SEI such that if type 33 is prefix then type 34 is suffix and vice versa. [000179] 2025202067   21 Mar 2025 [000180]     In the above explanation, the prompt information 80166 is signaled into FCMSEI 80160 in FCMRSD 80100. In other arrangements, the prompt information 80166 may be signaled into FCMVMPS 8020. In the case where prompt information is implemented as FCMVMPS in the bitstream, the prompt information 80166 is associated with the feature frames of the AUs corresponding to FCMVMPS 8020, that is, with all AUs in the FCMRSD 80100 and FCM CVD 80200. Where prompt information 80166 is to be updated, additional signaled instances the FCMVMPS are needed. [000181]     An implementation 900 of the decoder 144, also referred to as a feature map decoder, is shown in Fig. 9. Although the decoder 144 of Fig. 9 is an example of a versatile video coding (VVC) video decoding pipeline, other video codecs may also be used to perform the processing stages described herein. The bitstream 143 is demultiplexed by the demultiplexer 180 into the first bitstream portion 182 (corresponding to the first bitstream portion 172) and the second bitstream potion 184 (corresponding to the second bitstream portion 166). The first bitstream portion 182 is input to the decoder 144, and the second bitstream portion 184 is input to the prompt decoder 186. The bitstream 182 may be read from memory 206, the hard disk drive 210, a CD-ROM, a Blu-ray diskTM or other non-transitory computer readable storage medium. Alternatively, the bitstream 182 may be received from an external source such as a server connected to the communications network 220 or a radio-frequency receiver. The bitstream 182 contains encoded syntax elements representing the captured frame data to be decoded. [000182]     The bitstream 182 is input to an entropy decoder module 920. The entropy decoder module 920 extracts syntax elements from the bitstream 182 by decoding sequences of ‘bins’ and passes the values of the syntax elements to other modules in the decoder 144. The entropy decoder module 920 uses variable-length and fixed length decoding to decode SPS, PPS or slice header an arithmetic decoding engine to decode syntax elements of the slice data as a sequence of one or more bins. Each bin may use one or more ‘contexts’, with a context describing probability levels to be used for coding a ‘one’ and a ‘zero’ value for the bin. Where multiple contexts are available for a given bin, a ‘context modelling’ or ‘context selection’ step is performed to choose one of the available contexts for decoding the bin. The process of decoding bins forms a sequential feedback loop, thus each slice may be decoded in the slice’s entirety by a given entropy decoder 920 instance. A single (or few) high-performing entropy decoder 920 instances may decode all slices or subpictures for a frame or picture from the 2025202067   21 Mar 2025 bitstream 182 multiple lower-performing entropy decoder 920 instances may concurrently decode the slices for a frame from the bitstream 182. [000183]     The entropy decoder module 920 applies an arithmetic coding algorithm, for example ‘context adaptive binary arithmetic coding’ (CABAC), to decode syntax elements from the bitstream 182. The decoded syntax elements are used to reconstruct parameters within the decoder 144. Parameters include residual coefficients (represented by an arrow 924), a quantisation parameter 974, a secondary transform index 970, and mode selection information such as an intra prediction mode (represented by an arrow 958). The mode selection information also includes information such as motion vectors, and the partitioning of each CTU into one or more CBs. Parameters are used to generate PBs, typically in combination with sample data from previously decoded CBs. [000184]     The residual coefficients 924 are passed to an inverse secondary transform module 936 where either a secondary transform is applied or no operation is performed (bypass) according to the secondary transform index 970. The inverse secondary transform module 936 produces reconstructed transform coefficients 932, that is primary transform domain coefficients, from secondary transform domain coefficients. The reconstructed transform coefficients 932 are input to a dequantiser module 928. The dequantiser module 928 performs inverse quantisation (or ‘scaling’) on the residual coefficients 932, that is, in the primary transform coefficient domain, to create reconstructed intermediate transform coefficients, represented by an arrow 940, according to the quantisation parameter 974. The dequantiser module 928 may also apply a scaling matrix to provide non-uniform dequantization within the TB, corresponding to operation of the dequantiser module 840. Should use of a nonuniform inverse quantisation matrix be indicated in the bitstream 182, the decoder 144 reads a quantisation matrix from the bitstream 182 as a sequence of scaling factors and arranges the scaling factors into a matrix. The inverse scaling uses the quantisation matrix in combination with the quantisation parameter to create the reconstructed intermediate transform coefficients 940. [000185]     The reconstructed transform coefficients 940 are passed to an inverse primary transform module 944. The module 944 transforms the coefficients 940 from the frequency domain back to the spatial domain. The inverse primary transform module 944 applies inverse DCT-2 transforms horizontally and vertically, constrained by the maximum available transform size as described with reference to the forward primary transform module 826. The result of 2025202067   21 Mar 2025 operation of the module 944 is a block of residual samples, represented by an arrow 948. The block of residual samples 948 is equal in size to the corresponding CB. The residual samples 948 are supplied to a summation module 950. [000186]     At the summation module 950 the residual samples 948 are added to a decoded PB (represented as 952) to produce a block of reconstructed samples, represented by an arrow 956. The reconstructed samples 956 are supplied to a reconstructed sample cache 960 and an in-loop filtering module 988. The in-loop filtering module 988 produces reconstructed blocks of frame samples, represented as 992. The frame samples 992 are written to a frame buffer 996. [000187]     The reconstructed sample cache 960 operates similarly to the reference sample cache 856 of the encoder 120. The reconstructed sample cache 960 provides storage for reconstructed samples needed to intra predict subsequent CBs without the memory 206 (e.g., by using the data 232 instead, which is typically on-chip memory). Reference samples, represented by an arrow 964, are obtained from the reconstructed sample cache 960 and supplied to a reference sample filter 968 to produce filtered reference samples indicated by arrow 972. The filtered reference samples 972 are supplied to an intra-frame prediction module 976. The module 976 produces a block of intra-predicted samples, represented by an arrow 980, in accordance with the intra prediction mode parameter 958 signalled in the bitstream 182 and decoded by the entropy decoder 920. The intra prediction module 976 supports the modes of the module 864, including IBC and MIP. The block of samples 980 is generated using modes such as DC, planar or angular intra prediction. [000188]     When the prediction mode of a CB is indicated to use intra prediction in the bitstream 182, the intra-predicted samples 980 form the decoded PB 952 via a multiplexor module 984. Intra prediction produces a prediction block (PB) of samples, which is a block in one colour component, derived using ‘neighbouring samples’ in the same colour component. The neighbouring samples are samples adjacent to the current block and by virtue of being preceding in the block decoding order have already been reconstructed. Where luma and chroma blocks are collocated, the luma and chroma blocks may use different intra prediction modes. However, the two chroma CBs share the same intra prediction mode. [000189]     When the prediction mode of the CB is indicated to be inter prediction in the bitstream 182, a motion compensation module 934 produces a block of inter-predicted samples, 2025202067   21 Mar 2025 represented as 938. The block of inter-predicted samples 938 are produced using a motion vector, decoded from the bitstream 182 by the entropy decoder 920, and reference frame index to select and filter a block of samples 998 from the frame buffer 996. The block of samples 998 is obtained from a previously decoded frame stored in the frame buffer 996. For bi-prediction, two blocks of samples are produced and blended together to produce samples for the decoded PB 952. The frame buffer 996 is populated with filtered block data 992 from the in-loop filtering module 988. As with the in-loop filtering module 868 of the encoder 120, the in-loop filtering module 988 applies any of the DBF, the ALF and SAO filtering operations. Generally, the motion vector is applied to both the luma and chroma channels, although the filtering processes for sub-sample interpolation in the luma and chroma channel are different. Frames from the frame buffer 996 are output as decoded feature frames 145. [000190]     Not shown in Figs. 8A and 9 is a module for pre-processing video prior to encode and post-processing video after decoding to shift sample values such that a more uniform usage of the range of sample values within each chroma channel is achieved. A multisegment linear model is derived in the encoder 120 and signalled in the bitstream for use by the decoder 144 to undo the sample shifting. This linear-model chroma scaling (LMCS) tool provides compression benefit for particular colour spaces and content that have some nonuniformity, especially utilisation of a limited range, in their utilisation of the sample space that may result in higher quality loss from application of quantisation. [000191]     Fig. 10A is a schematic block diagram 1000 showing one type of architecture for the bottleneck decoder 148. As shown in the Fig. 10A, the bottleneck decoder 148 comprises SSFC decoder 1010 and a multi-scale feature reconstruction (MSFR) module 1030. The SSFC decoder 1010 produces a combined tensor 1020 from the input tensor 147. TheMSFR module 1030 produces tensors 149 from the combined tensor 1020. [000192]     Fig. 10B is one example implementation 1000B of the bottleneck decoder 168 when the CNN backbone in Fig 4A is used. The bottleneck decoder 168 includes the singlescale feature compression (SSFC) decoder 1010. The SSFC decoder 1010 receives the tensor 147 having a reduced channel count, such as C=64 channels, and passes the tensor 147 to a convolution layer 1012, which outputs a tensor 1013 having a restored channel count, such as F=256 channels. The tensor 1013 is passed to a batch normalisation module 1014 to produce a tensor 1015. The tensor 1015 is passed to a PreLU (Parametric Rectified Linear Unit) module 1016 whish is an activation function module to produce a tensor 1020. The tensor 1020 is input 2025202067   21 Mar 2025 to MSFR module 1030. The MSFR module 1030 operates to produce a plurality of tensors from the tensor 1020, using one or more trained convolutional layers. Upsample modules 1032, 1034, and 1036 upsample the tensor 1020 horizontally and vertically by factors of two, four, and eight, respectively, to produce tensors 1033, 1035, and 1037. The tensor 1033 has the dimension (2h,2w,256), the tensor 1035 has the dimension (4h, 4w, 256), and the tensor 1037 has the dimension (8h,8w,256). The tensor 1037 is a tensor P’2. The tensor 1037 is also passed to a downsample module 1042. The downsample module 1042 downsamples the tensor 1037 by a factor of two horizontally and vertically to produce a tensor 1043 having the dimensions (4h, 4w, 256). The tensor 1043 is provided to a convolution layer 1048 which outputs a tensor 1049. A summation module 1054 adds the tensors 1035 and 1049 to produce the tensor 1055 as a tensor P’3 of the MSFR module 1030. [000193]     The tensor 1035 is also passed to a downsample module 1040. The downsample module 1040 downsamples the tensor 1035 by a factor of two horizontally and vertically to produce a tensor 1041 having the dimensions (2h, 2w, 256). The tensor 1041 is provided to a convolution layer 1046 which outputs a tensor 1047. A summation module 1052 adds the tensors 1033 and 1047 to produce the tensor 1053 as a tensor P’4 of the MSFR module 1030. [000194]     The tensor 1033 is also passed to a downsample module 1038. The downsample module 1038 downsamples the tensor 1033 by a factor of two horizontally and vertically to produce a tensor 1039 having the dimensions (h, w, 256). The tensor 1039 is provided to a convolution layer 1044 which outputs a tensor 1045. A summation module 1050 adds the tensors 1020 and 1045 to produce the tensor 1051 as a tensor P’5 of the MSFR module 1030. [000195]     As described above, Fig. 10B shows an example of the bottleneck decoder 116 when FPN network architecture is used in NN part 1. In a case of use of the architecture in Fig. 3A, a similar manner can be used in the bottleneck decoder 148. For example, in a case where the architecture in Fig. 3A is used, SSFC decoder 1010 may produce the tensor 1020 by convolution layer, batch normalization layer, and PReLU layer.The MSFR module 1030 produces the tensors 149 using operations include up-sampling, down-sampling, convolution, to produce the tensors 149. [000196]     In a case of use of the architecture in Fig. 4B, the output tensor 149 from MSFR is a single tensor. Hence in cases using the architecture of Fig. 4B, the MSFR1030 process may be skipped, and the tensor 1020 is output as the tensor 149. 2025202067   21 Mar 2025 [000197]     Fig. 11 shows a method 1100 for decoding feature frame and prompt data from a bitstream. The method 1100 includes decoding dimensionally-reduced feature maps from the bitstream, performing dimensionality expansion using the bottleneck decoder 148, and the NN part 2 150. The method 1100 also operates to decode prompt data from the bitstream. The method 1100 may be implemented using apparatus such as a configured FPGA, an ASIC, or an ASSP. Alternatively, as described below, the method 1100 may be implemented by the destination device 140, as one or more software code modules of the application programs 233, under execution of the processor 205. The software code modules of the application programs 233 implementing the method 1100 may be resident, for example, in the hard disk drive 210 and / or the memory 206. The method 1100 is repeated for each frame of compressed data in the bitstream 143. The method 1100 may be stored on computer-readable storage medium and / or in the memory 206. The method 1100 begins at a demultiplexing step 1101. [000198]     At the step 1101, the demultiplexer 180 performs demultiplexing for the bitstream 143 to obtain the first bitstream portion 182 and the second bitstream portion 1848, for example based on the structure described in relation to Fig. 8B. The method 1100 continues under control of the processor 205 from step 1101 to a decode first bitstream portion step 1102. [000199]     At the step 1102, the decoder 144, under execution of the processor 205 of the destination device 140, decodes one feature frame 145 from the first bitstream portion 182 as described with reference to Fig. 9. The feature frame 145 contains the packed feature maps corresponding to the quantised tensor obtained from the tensor 117 as described with reference to Fig. 7. The method 1100 continues under control of the processor 205 of the destination device 140 from step 1102 to a step 1103. [000200]     At the step 1103, the unpack and inverse quantise module 146 extracts feature maps in the feature frame 145. The extracted feature maps are inverse quantised from the integer domain to the floating-point domain at step 1103, to generate the tensor 147. The method 1100 continues under control of the processor 205 from step 1103 to a decode tensors step 1104. [000201]     At the step 1104, as described in relation to Fig. 10A and Fig. 10B, the bottle neck decoder 148 generates the combined tensor(s) 149. For example, if using a neural network corresponding to Fig. 3A or Fig. 4A, the SSFC decoder 1010 of the bottleneck decoder 148, under execution of the processor 205 of the destination device 140, produces the combined tensor 1020 from the input tensor 147. Further, the MSFR 1030 of the bottleneck decoder 148 2025202067   21 Mar 2025 produces the tensors 149 from the combined tensor 1020. The method 1100 continues under control of the processor 205 from step 1104 to a decode second bitstream portion step 1105. [000202]     At the step 1105, the prompt decoder 186, under execution of the processor 205 of the destination device 140, decodes the prompt data from the second bitstream portion 186. [000203]     In some arrangements (e.g. when using the CNN backbone 114 in Fig. 3A or Fig. 4A), the syntax elements such as promptpoint x[ i ] and promptpoint y[ i ] may be decoded from the FCMSEI to derive prompt point indicating a specific position in the video frame 113. Also, the syntax elements such as promptbox xmin[ i ], promptbox ymin[ i ], promptbox xmax[ i ], promptbox ymax[ i ] are decoded from the FCMSEI to derive bounding box prompt. Also, the syntax elements such as promptmask payload len and promptmask payload are decoded from the FCMSEI to derive mask prompt. Also, the syntax element such as prompttext payload is decoded from the FCMSEI to derive text prompt. Also, the syntax elements such as promptaudiopayload len and promptaudio payload are decoded from the FCMSEI to derive audio prompt. [000204]     In other arrangements (especially when using the arrangement 400B in Fig. 4B in CNN backbone 114), the syntax elements such as promptgenericlen and promptgenericpayload byte are decoded from the FCMSEI to derive the prompt tensors to be input to CNN head 150 as prompt data 188. [000205]     Also, at the step 1105, the prompt decoder 186 associates prompt information in the FCMSEI 80160 with the specific AU (i.e., the specific feature frame) by using the predetermined manner as described in Fig. 8B. Then the prompt data in the prompt information 80166 is to be used in the NN part 2 for the reconstructed tensors derived from the feature frame (i.e., the associated specific feature frame) of the associated specific AU. [000206]     Then, control in the processor 205 of the destination device 140 progresses from the step 1105 to a neural network performing step 1106. [000207]     At the step 1106, the CNN head 150 performs, under execution of the processor 205 of the destination device 140, the process of NN part 2 or CNN head 150 by using the reconstructed tensor 149 and, if present, the decoded prompt data to produce the final results of 2025202067   21 Mar 2025 a machine task. For example, the step 1106 may be performed by a CNN head as described in relation to Fig. 12A or Fig. 13. [000208]     Execution of step 1102 obtains a first bitstream portion (182) from the bitstream 143 and execution of step 1103 and 1104 produces one or more tensors based on first information decoded from the first bitstream portion. Execution of step 1105 obtains a second bitstream portion (184) from the bitstream 143. Execution of the step 1105 decodes the second bitstream portion to obtain second information, being the prompt information. The second (prompt) information is associated with the first information decoded at step 1104 and is related to a target object , thereby allowing application the second neural network portion to identify the target objection. [000209]     Fig. 12A is a schematic block diagram showing a head portion 1200 of a CNN for object detection, as implemented for the CNN head 150. The example 1200 corresponds to implementations where the backbone portion 114 was implemented according to the backbone portion 300 of Fig. 3. Depending on the task to be performed in the destination device 140, different networks may be substituted for the CNN head 150. Incoming tensors 149 are separated into the tensor of each layer (i.e., tensors 1210, 1220, and 1234). The tensor 1210 is passed to a CBL module 1212 to produce tensor 1214. The tensor 1214 is passed to a detection module 1216 and an upscaler module 1222. The detection module 1216 operates to detect bounding boxes 1218. The bounding boxes 1218 are in the form of a detection tensor. The bounding boxes 1218 are passed to a non-maximum suppression (NMS) module 1248. The NMS module 1248 selects one of multiple inputs generated by detection modules to produce the detection result 151. [000210]     To produce bounding boxes addressing co-ordinates in the original video frame 113, prior to resizing for the backbone portion of the network 114, scaling by the original video width and height is performed. The upscaler module 1222 produces an upscaled tensor 1224 scaled by original video width and height. The upscaled tensor 1224 is passed to a CBL module 1226. The CBL module 1226 produces tensor 1228 as output. The tensor 1228 is passed to a detection module 1230 and an upscaler module 1236. The detection module 1230 operates in a similar manner to the detection module 1216 and produces a detection tensor 1232. The detection tensor 1232 is supplied to the NMS module 1248. [000211]     The upscaler module 1236 operates in the same manner as the module 1222 and outputs an upscaled tensor 1238. The upscaled tensor 1238 is passed to a CBL module 1240. 2025202067   21 Mar 2025 The CBL module 1240 operates in the same manner as the modules 1212 and 1226 to output a tensor 1242 to a detection module 1244. The detection module 1244 operates in a similar manner to the detection modules 1216 and 1230 and produces a detection tensor 1246. The detection tensor 1246 is supplied to the NMS module 1248. [000212]     The CBL modules 1212, 1226, and 1240 each contain a concatenation of five CBL modules, for example as shown in Fig. 3D. The upscaler modules 1222 and 1236 are each instance of an upscaler module 1260 as shown in Fig. 12B. [000213]     The upscaler module 1260 accepts a tensor 1262 and a tensor 1264 as inputs. The tensor 1262 is passed to a CBL module 1266 to produce a tensor 1268. The tensor 1268 is passed to an upsampler 1270 to produce an upsampled tensor 1272, using upsampling methods such as nearest-neighbour interpolation or bilinear interpolation. A concatenation module 1274 produces a tensor 1276 by concatenating the upsampled tensor 1272 with the input tensor 1264. [000214]     The detection modules 1216, 1230, and 1244 are instances of a detection module 1280 as shown in Fig. 12C. The detection module 1280 receives a tensor 1282, which is passed to a CBL module 1284 to produce a tensor 1286. The tensor 1286 is passed to a convolution module 1288, which implements a detection kernel. A detection kernel a 1 x 1 kernel applied to produce the output on feature maps at the three layers. The detection kernel is 1 x 1 x (B x (5 + C)), where B is the number of bounding boxes a particular cell can predict, typically three (3), and C is the number of classes, which may be eighty (80), resulting in a kernel size of two-hundred and fifty five (255) detection attributes. The module 1288 outputs tensor 1290. The constant “5” represents four boundary box attributes (box centre x, y and size scale x, y) and one object confidence level (“objectness”). The result of a detection kernel has the same spatial dimensions as the input feature map, but the depth of the output corresponds to the detection attributes. The detection kernel is applied at each layer, typically three layers, resulting in a large number of candidate bounding boxes. [000215]     A process of non-maximum suppression is applied by the NMS module 1048 to the resulting bounding boxes to discard redundant boxes, such as overlapping predictions at similar scale, resulting in a final set of bounding boxes as output for object detection. [000216]     In Fig. 12, prompt data 188 from the second portion of the bitstream can be applied to the three detection heads 1126, 1230, and 1244 to produce results of a machine task. Results of a machine task of the NN part 2 in Fig. 12A can be one or more specific objects 2025202067   21 Mar 2025 identified based on the prompt data as described in Fig. 15 among detected objects in a video frame 113. The results 151 can be one or more specific segmented areas among segmented areas in a video frame 113. In other arrangement, the results 151 can be all objects belonging to one class specified by e.g. text prompt of the prompt data 188 among objects in a video frame 113. In other arrangement, the results can be one person among a plurality of persons presented in a video frame for pedestrian tracking. [000217]     Fig. 13 is a schematic block diagram showing an alternative head portion 1300 of a CNN, which can be implemented as the CNN head 150. The head portion 1300 forms part of an overall network known as ‘Faster RCNN’ and includes a feature network (i.e., backbone portion 400), a region proposal network, and a detection network. Input to the head portion 1300 are the tensors 149. The tensors 149 include P2-P5 tensors 1037, 1055, 1053, and 1051. The tensor P6 1318 can be derived by e.g. the CNN head 150 or bottleneck decoder 148 by performing a ‘Maxpool’ operation with stride equal to two for the tensor P5 1051. The P2-P6 tensors 1037, 1055, 1053, 1051, and 1318 are input to a region proposal network (RPN) head module 1320. The RPN head module 1320 performs a convolution on the input tensors, producing an intermediate tensor. The intermediate tensor is fed into two subsequent sibling layers in the module 1320, one for classifications and one for bounding box, or ‘region of interest’ (ROI), regression. The module 1320 generates classification and bounding boxes 1322. The classification and bounding boxes 1322 are passed to an NMS module 1324. The NMS module 1324 prunes out redundant bounding boxes by removing overlapping boxes with a lower score to produce pruned bounding boxes 1326. The bounding boxes 1326 are passed to a region of interest (ROI) pooler 1328. The ROI pooler 1328 also receives the tensors P2 1037 (corresponding to the P2 tensor 477), P3 1055 (corresponding to the P3 tensor 475), P4 1053 (corresponding to the P4 tensor 473), P5 1051 (corresponding to the tensor 471). And the ROI pooler 1328 produces fixed-size feature maps from various input size maps using max pooling operations. In the operations performed by 1328, a subsampling takes the maximum value in each group of input values to produce one output value in the output tensor. [000218]     Input to the ROI pooler 1328 are the P2-P5 tensors 1037, 1055, 1053, and 1051, and region of interest proposals 1326. Each proposal (ROI) from 1326 is associated with a portion of the feature maps (1037, 1055, 1053, 1051) to produce a fixed-size map. The fixed-size map is of a size independent of the underlying portion of the feature map of the P2-P5 tensors. One of the feature maps of the P2-P5 tensors is selected such that the resulting cropped map has sufficient detail, for example, according to the following rule: floor(4 + 2025202067   21 Mar 2025 log2(sqrt(boxarea) / 224)), where 224 is the canonical box size. The ROI pooler 1328 thus crops incoming feature maps according to the proposals 1326 producing a tensor 1330. The tensor 1330 is fed into a fully connected (FC) neural network head 1332. The FC head 1332 performs two fully connected layers to produce class score and bounding box predictor delta tensor 1334. The class score is generally an 80-element tensor, each element corresponding to a prediction score for the corresponding object category. The bounding box prediction deltas tensor is an 80*4 = 320 element tensor, containing bounding boxes for the corresponding object categories. Final processing is performed by an output layers module 1336, receiving the tensor 1334 and performing a filtering operation to produce a filtered tensor 1338. Low-scoring (low classification) objects are removed from further consideration. A non-maximum suppression module 1340 receives the filtered tensor 1338 and removes overlapping bounding boxes by removing the overlapped box with a lower classification score, producing output tensor 1342, which provides the output result 151. [000219]     In Fig. 13, the prompt data 188 is added to the final step of the head 1300 to output one or more specific bounding box for a detection task or segmented area for segmentation task to be output based on the prompt data 188. [000220]     Fig. 14 is a schematic block diagram showing an alternative head portion 1400 of a CNN, which can be implemented as the CNN head 150 when a network architecture such as SAM is used (for example as shown in Fig. 4B(I). The tensor 149 corresponds to the tensor 115 and is input to a Mask decoder 1435. The prompt data 188 may contains at least one of point data 1424, box prompt 1426, text prompt 1428, or audio prompt 1430. The prompt data 188 is processed by mask decoder 1435 together with the tensor 149 to produce segmented area corresponding to the task result 151. The segmented area is one or more specific areas identified based on the prompt data among all segmented area in the video frame. [000221]     In the mask decoder 1435, any present of the prompt data such as point prompt 1424, box prompt 1426, text prompt 1427, and audio prompt 1428 are concatenated by a concatenation layer 1440 to produce a tensor 1442. The mask prompt 1422 (if present) and the tensor 149 are added by an addition layer 1444 to produce a tensor 1446. The tensor 1142 and 1146 are processed together by inputting to a transformer block 1448. The transformer block 1148 may contain several multi-head attention blocks to combine the feature tensor 149 and prompts together. The transformer block1448 produces a tensor 1450. The tensor 1450 is input to a multilayer perceptron (MLP) block 1452 to produce a tensor 1456. The tensor 1454 is 2025202067   21 Mar 2025 input to a post processing block 1460 to produce the segmented areas 151. The post processing block 1460 removes padding of the mask if padding was added in the post-processing, or rescales the mask to the size of the original video source. A threshold can be applied in the post processing block 1460 to keep masks that satisfy the threshold condition among many masks generated from MLP 1452. The segmented areas 151 may contains one or more mask results. [000222]     Fig. 15 is a diagram 1500 showing examples of use of prompt data. Fig. 15 shows a video frame 1501 ( 113), obtained by the video source 112. In the video frame 1501, there are 6 objects in the image: a car “C”, a shoppingbag “B”, a person “Per1”, a person “Per2”, a person “Per3”, and a partyhat “H” (worn by person “Per3”). If the prompt data is not used in an object detection task in the NN part 2, six bounding boxes for the six objects are generated. Prompt data is used in the NN part 2 to identify the specific object to be outputted of the NN part 2. For example, if the point prompt indicating a position (the x-coordinate and y-coordinate) of the point “Pr1” in Fig. 15 is used as the prompt data, the object detection task of the NN part 2 may output the person “Per1” as the task result 151 of the objection detection task. If the box prompt comprising information indicating a position (e.g. the x-coordinate and y-coordinate) of the top-left corner of the box “Pr2” and comprising information indicating a size (a width and a height) of the box ”Pr2” is used as the prompt data, the object detection task of the NN part 2 may output the person “Per1” as the task result 151 of the objection detection task. . If the mask prompt indicating the mask “Pr3” is used as the prompt data, the object detection task of the NN part 2 may output the shopping bag “B” as the task result 151 of the objection detection task. If the text prompt indicating “It is a car” is used as the prompt data, the object detection task of the NN part 2 may output the car “C” as the task result 151 of the objection detection task. If an audio prompt saying “The person with a party hat” is used as the prompt data, the object detection task of the NN part 2 may output the person “Per3” as the task result of the objection detection task. In some cases, prompt data includes label of “positive” (value “1”) or “negative” (value “0”) of the object. The label “1” means the prompt is on the object, and the label “0” means the prompt is not on the object. For example a prompt is {points = [[x1, y1], [x2, y2]], labels = [1, 0]} mean this prompt contains two points of [x1, y1] and [x2, y2] and the first point is on the object, the second point is not on the object. [000223]     Available datasets with ground truth for training for processes such as detection, segmentation, tracking, and segment anything training include the OpenImage, COCO, PedTrackPP (A collection of videos from “Pexels” and “Pixabay” content websites, with additional ground-truth), PWD datasets, and SA-B1. The arrangements described can use 2025202067   21 Mar 2025 training techniques known in the machine vision industries. Implementations using a SAM model may use a model trained using feature pyramid network (FPN) or vision transformer (ViT) network for example. INDUSTRIAL APPLICABILITY [000224]     The arrangements described are applicable to the computer and data processing industries and particularly for the digital signal processing for the encoding and decoding of signals such as video and image signals, achieving high compression efficiency. [000225]     The arrangements described provide a solution by which indication of target objects or features may be embedded in the bitstream through use of prompts. Accordingly, the outputs of a task implemented by a split network arrangement can be modified to identify target objects or features of interest within the frame. [000226]      The foregoing describes only some embodiments of the present invention, and modifications and / or changes can be made thereto without departing from the scope and spirit of the invention, the embodiments being illustrative and not restrictive. [000227]     In the context of this specification, the word “comprising” means “including principally but not necessarily solely” or “having” or “including”, and not “consisting only of”. Variations of the word “comprising”, such as “comprise” and “comprises” have correspondingly varied meanings. 2025202067   21 Mar 2025 APPENDIX A FCM NAL units are encoded using the following syntax: fcm nal unit( numBytesInFCMNalUnit ) { Descriptor fcmnalunit header( ) numBytesInRbsp = 0 for( i = 2; i < numBytesInFCMNalUnit; i++ ) rbsp byte[ numBytesInRbsp++ ] } FCM NAL units are encoded using the following syntax: fcmnalunit header( ) { Descriptor fcm nalforbidden zero bit u(1) fcm nalunittype u(6) fcm nalreserved 9bits u(9) } rbsp byte[ i ] holds the ith byte of a raw bytestream payload, holding contents, such as coded syntax elements, of an FCM NAL unit. fcm nalforbidden zerobit shall be equal to zero. fcm nalreserved 9bits is reserved for future extensions of the FCL NAL unit header. When fcmnalunittype is equal to 33 (FCMNALSEI), an seirbsp( ) syntax structure is coded, as follows: sei_rbsp( ) { Descript or do sei_message( ) while( more_rbsp_data( ) ) rbsp_trailing_bits( ) } 2025202067   21 Mar 2025 sei_message( ) { Descript or payloadType = 0 do { payload_type_byte u(8) payloadType += payload_type_byte } while( payload_type_byte = = 0xFF ) payloadSize = 0 do { payloadsizebyte u(8) payloadSize += payload_size_byte } while( payload_size_byte = = 0xFF ) sei_payload( payloadType, payloadSize ) } payload typebyte encodes the n-th SEI message type as enumerated in the seipayload( ) syntax structure in the sei rbsp. For SEI message types greater than 255, one or more prefix payloadtype byte instances, each having a value of 255 are present. The SEI message type is the sum of the instances of payloadtype byte, terminated with a value not equal to 255. payload size byte encodes the size of the nth SEI message in the seirbsp. For SEI message sizes greater than 255, one or more prefix payload size byte instances, each having a value of 255 are present. The SEI message size is the sum of the instances of payloadsize byte, terminated with a value not equal to 255. sei_payload( payloadType, payloadSize ) { Descript or SeiExtensionBitsPresentFlag = 0 if( fcm_nal_unit_type = = FCMSEINUT ) / / = 33 if( payloadType = = 0 ) fcm_prompt_information_sei_message ( payloadSize ) else if( payloadType = = 1 ) other_information( payloadSize ) An example FCM SEI message format and associated semantics for representing metadata 2025202067   21 Mar 2025 associated with prompt information in a bitstream are as follows: fcm_prompt_information_sei_message( payloadSize ) { Descriptor prompt_flag   / / prompt flag 80167 u(1) prompt_persist_flag u(1) prompt_cancel_flag u(1) if( !prompt_cancel_flag || prompt_flag ) { / / the following syntax elements are prompt data 80168 poc u(32) number_of_prompts_minus1 ue(v) for ( i = 0; i <= number_of_prompts; i++ ) { prompt_type[ i ] u(1) promptpositive[ i ] u(1) prompt_has_label [ i ] u(1) if( prompt_has_label [ i ]) promptlabel [ i ] st(v) if( prompt_type[ i ] == 0 ) / / point prompt_point_x[ i ] u(16) promptpointy[ i ] u(16) else if( prompt_type[ i ] == 1 ) / / bounding box prompt_box_x_min[ i ] u(16) prompt_box_y_min[ i ] u(16) prompt_box_x_max[ i ] u(16) prompt_box_y_max[ i ] u(16) else if( prompt_type[ i ] == 2 ) / / mask prompt_mask_payload_len [ i ] ue(v) for( j = 0; j < prompt_mask_payload_len ) prompt_mask_payload[ j ] u(8) else if( prompt_type[ i ] == 3 ) / / text prompt_text_payload st(v) else if( prompt_type[ i ] == 4 ) / / audio prompt_audio_payload_len [ i ] ue(v) 2025202067   21 Mar 2025 for( j = 0; j < prompt_audio_payload_len; j++ ) prompt_audio_payload[ j ] u(8) else { prompt_generic_len [ i ] ue(v) for( j = 0; j < prompt_generic_len; j++ ) prompt_generic_payload_byte[ j ] u(8) } } } Where u(n) refers to a fixed-length codeword n bits in length, ue(v) refers to an unsigned exponential Golomb variable-length codeword, and st(v) refers to a UTF-8 encoded string. FCM prompt information SEI message semantics: promptflag is a boolean value. When promptflag is equal to 1, prompt data is parsed from the prompts information SEI message and output from the FCM decoder, i.e., output to NN part 2. When promptflag is false, prompt data is not parsed from the SEI. When promptcancel flag is present, promptflag allows the option to cancel a previous prompt and also signal a new prompt in the same prompt information SEI message. promptpersistflag when equal to 1 indicates that the signalled prompt information in the present prompt information SEI message persists for subsequent pictures, i.e., from the POC associated with the current prompt information onwards. When promptpersistflag is equal to 0 the signalled prompt information is associated only with the POC to which the prompt information SEI message applies. promptcancelflag when equal to 1 indicates that any prompt information persisting from the most recent previous instance of the prompt information SEI message (i.e., prompt information signalled in an SEI associated with a picture having the highest POC value less than the current POC) is cancelled, i.e., ceases to be associated with tensors output from the FCM decoder (i.e., 190), effective from the current POC onwards, i.e., until a new prompt information SEI message is signalled, or the current prompts SEI message is signalled with promptflag equal to one, allowing cancellation of previously signalled prompt information and replacement with newly signalled prompt information in one prompt information SEI message. 2025202067   21 Mar 2025 Note: promptflag may be omitted in the case where the FCM prompt information SEI message syntax is parsed as a result of payloadType equal to the value assigned for the fcmprompt information seimessage( ), such as 0. poc specifies the picture order count of the feature frame corresponding to the current FCM_SEI. number of prompts minus1 plus 1 specifies the total number of prompts present in the FCM_SEI. prompttype [ i ] specifies to the type of the i-th prompt, enumerated as follows: 0: Point 1: Bounding box 2: Mask 3: Text 4: Audio 5-: Reserved for future use. promptpositive [ i ] equal to 1 indicates that the i-th prompt is used to ‘select’ an object, e.g., the point or bounding box is within or encapsulates an object of interest. When promptpositive is equal to 0, the i-th prompt “deselects” or excludes the indicated object from being an object of interest. A mixture of positive and negative prompts can be used to indicate specific objects within larger group of objects, including smaller object forming part of a larger object. In arrangements using the “Segment Anything Model”, the flag “prompt positive” is only applicable when the prompt type is a point. Promptpositive may be conditionally signaled only when prompt types is equal to 0 (point). prompthaslabel [ i ] equal to 1 indicates that the i-th prompt is also provided with a class label, in order to support applications where the NN part 2 is capable of using not only the point prompt or prompt box or other form of prompt but also an associated label to assist in performance of the task, such as promptable segmentation. promptlabel [ i ], when present, contains a string representing the class of the i-th prompt, providing the capability to associate a class label with a prompt to an NN part 2 performing a 2025202067   21 Mar 2025 model that requires (or can benefit from) class labelling of a prompt. promptpoint x[ i ] specifies the x-coordinate of the point (i.e. the i-th prompt when the prompttype [ i ] = 0 ) in the video frame. promptpointy[ i ] specifies the y-coordinate of the point (i.e. the i-th prompt when the prompttype [ i ] = 0 ) of i-th prompt in the video frame. promptbox xmin[ i ] specifies the x-coordinate of the top left corner of the bounding box (i.e. the i-th prompt when the prompttype [ i ] = 1 ) in the video frame. promptboxymin[ i ] specifies the y-coordinate of the top left corner of the bounding box (i.e. the i-th prompt when the prompttype [ i ] = 1 ) in the video frame. promptboxxmax[ i ] specifies the x-coordinate of the bottom right corner of the bounding box (i.e. the i-th prompt when the prompt type [ i ] = 1 ) in the video frame. promptboxymax[ i ] specifies the x-coordinate of the bottom right corner of the bounding box (i.e. the i-th prompt when the prompt type [ i ] = 1 ) in the video frame. promptmask payload len [ i ] specifies the length in bytes of a payload of the mask (i.e. the i-th prompt when the prompt type [ i ] = 2 ). promptmask payload[ j ] specifies the j-th byte of the payload of the mask (i.e. the i-th prompt when the prompt type [ i ] = 2 ). The payload of the mask payload contains a bi-level bitmap of the image having the same resolution as the video frame 113 and the mask contained in the payload using run-length encoding. prompt text payload[ i ] specifies the payload of the prompt text (i.e. the i-th prompt when the prompttype [ i ] = 3 ) . The payload of the prompt text is a variable length strings encoded using UTF-8 or UTF-16. promptaudiopayload len [i] and prompt audio payload[ j ] specifies the length of the audio prompt (i.e. the i-th prompt when the prompttype [ i ] = 4 ) and the j-th byte of the audio 2025202067   21 Mar 2025 payload. The audio payload may be compressed using a format such as monophonic samples of 8-bits or 16-bits per sample, or using a compressed format such as the ITU-T G.711 or G.729.1, which are optimized for speech compression. Other formats such as MPEG-1 Layer III (“MP3”) may also be used. promptgenericlen [ i ] specifies the length in bytes of a predetermined prompt (i.e. the i-th prompt when the prompt type [ i ] = 5 ). The predetermined prompt is a given prompt except for the point, the bounding box, the mask, the text, the audio. promptgenericpayload byte[ j ] specifies the j-th byte of the the predetermined prompt (i.e. the i-th prompt when the prompttype [ i ] = 5 ). Arrangements supporting prefix and suffix FCM SEI messages may use the following FCM NAL unit types: fcmnalunit type = 33 / / FCMNALPREFIX SEI (this SEI precedes the FCMFPH to which this SEI message applies) fcmnalunit type = 34 / / FCMNALSUFFIX SEI (this SEI is subsequent to the FCMFPH to which this SEI message applies)

Claims

1. An method of generating a bitstream, the method comprising:encoding, into a first bitstream portion, first information based on one or more tensors;encoding second information into a second bitstream portion, wherein second information is associated with the first information and is related to a target object; and generating the bitstream using the first bitstream portion and the second bitstreamportion.

2. The method according to claim1, wherein the first information is a feature frame onwhich feature maps of a tensor are packed.

3. The method according to claim 1, wherein the second bitstream portion comprises anindication to associate the second information with the first information.

4. The method according to claim 1, wherein the second bitstream portion containing thesecond information is located adjacently preceding or following an FCMPH.

5. The method according to claim 1, wherein the first bitstream portion is an FCM CVDin the bitstream.

6. The method according to claim 1, wherein the second bitstream portion is an FCM_RSDin the bitstream.

7. The method according to claim1, wherein the second bitstream portion is an FCM_SEIin the bitstream.

8. The method according to claim1, wherein the second bitstream portion is aFCMVMPS in the bitstream.

9. The method according to claim1, wherein the second information comprises promptdata containing at least one of a point prompt, a box prompt, a mask prompt, a text prompt and an audio prompt.2025202067   21 Mar 202510. A method of decoding one or more tensors from a bitstream, the method comprising: obtaining a first bitstream portion and a second bitstream portion from the bitstream; producing the one or more tensors based on first information decoded from the first bitstream portion; anddecoding the second bitstream portion to obtain second information, wherein the second information is associated with the first information and is related to a target object.

11. The method according to claim1, wherein the first information is a feature frame on which feature maps of a tensor are packed.

12. The method according to claim 1, wherein the second bitstream portion comprises an indication to associate the second information with the first information.

13. The method according to claim1, wherein the second bitstream portion containing the second information is located adjacently preceding or following an FCMPH.

14. The method according to claim1, wherein the first bitstream portion is an FCMCFD in the bitstream.

15. The method according to claim1, wherein the second bitstream portion is an FCM_RSD in the bitstream.

16. The method according to claim1, wherein the second bitstream portion is an FCM_SEI in the bitstream.

17. The method according to claim1, wherein the second bitstream portion is an FCMVMPS in the bitstream.

18. The method according to claim1, wherein the second information comprises prompt data containing at least one of point prompt, box prompt, mask prompt, texts prompt, audio prompt.

19. An encoder for generating a bitstream, the encoder configured to:2025202067   21 Mar 2025encode, into a first bitstream portion, first information based on one or more tensors;encode second information into a second bitstream portion, wherein second information is associated with the first information and is related to a target object; andgenerate the bitstream using the first bitstream portion and the second bitstream portion.

20. A non-transitory computer-readable storage medium which stores a program for executing a method of generating a bitstream, the method comprising:encoding, into a first bitstream portion, first information based on one or more tensors; encoding second information into a second bitstream portion, wherein secondinformation is associated with the first information and is related to a target object; and generating the bitstream using the first bitstream portion and the second bitstream portion.

21. A decoder for decoding one or more tensors from a bitstream, the decoder configured to: obtain a first bitstream portion and a second bitstream portion from the bitstream; produce the one or more tensors based on first information decoded from the first bitstream portion; anddecode the second bitstream portion to obtain second information, wherein the second information is associated with the first information and is related to a target object.

22. A non-transitory computer-readable storage medium which stores a program for executing a method of decoding one or more tensors from a bitstream, the method comprising: obtaining a first bitstream portion and a second bitstream portion from the bitstream; producing the one or more tensors based on first information decoded from the first bitstream portion; anddecoding the second bitstream portion to obtain second information, wherein the second information is associated with the first information and is related to a target object.CANON KABUSHIKI KAISHAPatent Attorneys for the ApplicantSpruson & Ferguson

Images