Cross-domain neural-network filtering in video coding

Abstract

Indication(s) of a first constituent frame and a second constituent frame, to be used as input for a neural network inference, are decoded. The first and second constituent frames are rectangular regions or pictures of different scalability layers. The decoding creates a first reconstructed constituent frame from the first constituent frame and a second reconstructed constituent frame from the second constituent frame. The first reconstructed constituent frame and the second reconstructed constituent frame are provided as inputs to the neural network inference. The neural network inference is performed to enhance the first reconstructed constituent frame to an enhanced first constituent frame. Indication(s) of a first constituent frame and a second constituent frame of input video to be used as input by a decoder for a neural network inference are encoded. The encoded one or more indications are output in a bitstream.

Description

CROSS-DOMAIN NEURAL-NETWORK FILTERING IN VIDEO CODINGTECHNICAL FIELD

[0001] Examples of embodiments herein relate generally to video coding and decoding and, more specifically, relate to neural network filtering in video coding and decoding.BACKGROUND

[0002] This document uses the term domain as a video content class, wherein different domains may differ, for example, in one or more of the following: content type (e.g., regular camera-captured video, mask video, depth map), camera or view position, spatial resolution, or bit depth. In video coding and decoding, there are implementations where video from multiple different domains is used. For instance, in stereoscopic (e.g., 3D, three dimensions) video, pairs of offset images are shown separately to the left and right eye of the viewer, images for each eye possibly from a different domain.

[0003] One branch of research for obtaining compression improvement in stereoscopic video is known as asymmetric stereoscopic video coding. Asymmetric stereoscopic video coding is based on a binocular suppression theory that the Human Visual System (HVS) fuses the stereoscopic image pair such that the perceived quality is close to that of the higher quality view. Thus, compression improvement is obtained by providing a quality difference between the two coded views. An implementation of this may use different scalability layers as the domains, such as a higher definition layer with more pixels than a lower definition layer with fewer pixels. Another implementation may use information from a camera along with corresponding auxiliary video, such as object mask video, to provide the different domains.

[0004] One can perform cross-domain filtering, such as filtering a higher definition layer with more pixels into a lower definition layer with fewer pixels to provide a stereoscopic image. This cross-domain filtering can be improved.BRIEF SUMMARY

[0005] This section is intended to include examples and is not intended to be limiting.

[0006] In an exemplary embodiment, a method is disclosed that includes decoding one or more indications of a first constituent frame and a second constituent frame to be used as input for a neural network inference, wherein the first and second constituent frames are rectangular regions or pictures of different scalability layers, and the decoding creates a first reconstructed constituent frame from the first constituent frame and a second reconstructed constituent frame from the second constituent frame; providing the first reconstructed constituent frame and the second reconstructed constituent frame as inputs to the neural network inference; and performing the neural network inference to enhance the first reconstructed constituent frame to an enhanced first constituent frame.

[0007] An additional exemplary embodiment includes a computer program, comprising instructions for performing the method of the previous paragraph, when the computer program is run on an apparatus. The computer program according to this paragraph, wherein the computer program is a computer program product comprising a computer-readable medium bearing the instructions embodied therein for use with the apparatus. Another example is the computer program according to this paragraph, wherein the program is directly loadable into an internal memory of the apparatus.

[0008] An exemplary apparatus includes one or more processors and one or more memories storing instructions that, when executed by the one or more processors, cause the apparatus at least to perform: decoding one or more indications of a first constituent frame and a second constituent frame to be used as input for a neural network inference, wherein the first and second constituent frames are rectangular regions or pictures of different scalability layers, and the decoding creates a first reconstructed constituent frame from the first constituent frame and a second reconstructed constituent frame from the second constituent frame; providing the first reconstructed constituent frame and the second reconstructed constituent frame as inputs to the neural network inference; and performing the neural network inference to enhance the first reconstructed constituent frame to an enhanced first constituent frame.

[0009] An exemplary computer program product includes a computer-readable storage medium bearing instructions that, when executed by an apparatus, cause the apparatus to perform at least the following: decoding one or more indications of a first constituent frame and a second constituent frame to be used as input for a neural network inference, wherein the firstand second constituent frames are rectangular regions or pictures of different scalability layers, and the decoding creates a first reconstructed constituent frame from the first constituent frame and a second reconstructed constituent frame from the second constituent frame; providing the first reconstructed constituent frame and the second reconstructed constituent frame as inputs to the neural network inference; and performing the neural network inference to enhance the first reconstructed constituent frame to an enhanced first constituent frame.

[0010] In another exemplary embodiment, an apparatus comprises means for: decoding one or more indications of a first constituent frame and a second constituent frame to be used as input for a neural network inference, wherein the first and second constituent frames are rectangular regions or pictures of different scalability layers, and the decoding creates a first reconstructed constituent frame from the first constituent frame and a second reconstructed constituent frame from the second constituent frame; providing the first reconstructed constituent frame and the second reconstructed constituent frame as inputs to the neural network inference; and performing the neural network inference to enhance the first reconstructed constituent frame to an enhanced first constituent frame.

[0011] In an exemplary embodiment, a method is disclosed that includes encoding one or more indications of a first constituent frame and a second constituent frame of input video to be used as input by a decoder for a neural network inference, wherein the first and second constituent frames are rectangular regions or pictures of different scalability layers; and outputting the encoded one or more indications in a bitstream.

[0012] An additional exemplary embodiment includes a computer program, comprising instructions for performing the method of the previous paragraph, when the computer program is run on an apparatus. The computer program according to this paragraph, wherein the computer program is a computer program product comprising a computer-readable medium bearing the instructions embodied therein for use with the apparatus. Another example is the computer program according to this paragraph, wherein the program is directly loadable into an internal memory of the apparatus.

[0013] An exemplary apparatus includes one or more processors and one or more memories storing instructions that, when executed by the one or more processors, cause the apparatus at least to perform: encoding one or more indications of a first constituent frame and asecond constituent frame of input video to be used as input by a decoder for a neural network inference, wherein the first and second constituent frames are rectangular regions or pictures of different scalability layers; and outputting the encoded one or more indications in a bitstream.

[0014] An exemplary computer program product includes a computer-readable storage medium bearing instructions that, when executed by an apparatus, cause the apparatus to perform at least the following: encoding one or more indications of a first constituent frame and a second constituent frame of input video to be used as input by a decoder for a neural network inference, wherein the first and second constituent frames are rectangular regions or pictures of different scalability layers; and outputting the encoded one or more indications in a bitstream.

[0015] In another exemplary embodiment, an apparatus comprises means for: encoding one or more indications of a first constituent frame and a second constituent frame of input video to be used as input by a decoder for a neural network inference, wherein the first and second constituent frames are rectangular regions or pictures of different scalability layers; and outputting the encoded one or more indications in a bitstream.BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The accompanying drawings use reference numerals, where the same reference numerals may be used to refer to like parts throughout, but parts having the same reference numeral can differ in operation and components. In the attached drawings:

[0017] FIG. 1 is a block diagram illustrating a system in accordance with an example;

[0018] FIG. 2, which is split into FIGS. 2A and 2B, illustrates a WC encoder where the encoder examples can be implemented;

[0019] FIG. 3, which is split into FIGS. 3A and 3B, illustrates a video decoder where the decoder examples can be implemented;

[0020] FIG. 4 is a flow diagram of a decoding method in an example;

[0021] FIG. 5 is a flow diagram of an encoding method in an example;

[0022] FIG. 6 is a block diagram for an inter-view super-resolution neural network;

[0023] FIG. 7 is a block diagram for mask-guided or map-guided filtering; and

[0024] FIG. 8 is an example of a block diagram of an apparatus suitable for implementing any of the encoders or decoders described herein.DETAILED DESCRIPTION OF THE DRAWINGS

[0025] Abbreviations that may be found in the specification and / or the drawing figures are defined below, at the end of the detailed description section.

[0026] The word “exemplary” is used herein to mean “serving as an example, instance, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments. All of the embodiments described in this Detailed Description are exemplary embodiments provided to enable persons skilled in the art to make or use the examples.

[0027] When more than one drawing reference numeral, word, or acronym is used within this description withand in general as used within this description, the “ / ” may be interpreted as “or”, “and”, or “both”. As used herein, “at least one of the following: ” and “at least one of ” and similar wording, where the list of two or more elements are joined by “and” or “or,” mean at least any one of the elements, or at least any two or more of the elements, or at least all the elements.

[0028] As used herein, the singular forms “a”, “an” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms “comprises”, “comprising”, “has”, “having”, “includes” and / or “including”, when used herein, specify the presence of stated features, elements, and / or components etc., but do not preclude the presence or addition of one or more other features, elements, components and / or combinations thereof.

[0029] It is noted that capital and lowercase words or phrases are considered to be the same herein. For instance, the words Slice, slice, and SLICE are the same, as are the phrases Network Repository Function, network repository function, and NETWORK REPOSITORY FUNCTION.

[0030] Any flow diagram or signaling diagram herein is considered to be a logic flow diagram, and illustrates the operation of an exemplary method, results of execution of computer program instructions embodied on a computer readable memory, and / or functions performed by logic implemented in circuitry. For methods, flow diagrams, and signaling diagrams, the orders of method steps, blocks in the flow, or signaling are not critical and instead are examples.

[0031] The following uses a nested topical structure for ease of reference. This nested topical structure has the following key: I, II, III... are main categories; a, b, c... are subcategories; 1, 2, 3... are further subcategories; and i, ii, iii... are even further subcategories.

[0032] I. Introduction

[0033] Technical context is now provided for technical areas related to the understanding of the examples.

[0034] La. Video coding

[0035] A video codec includes an encoder that transforms the input video into a compressed representation suited for storage / transmission and a decoder that can decompress the compressed video representation back into a viewable form. Typically, an encoder discards some information in the original video sequence in order to represent the video in a more compact form (that is, at lower bitrate).

[0036] Typical hybrid video codecs, for example ITU-T H.263 and H.264, encode the video information in two phases. Firstly, pixel values in a certain picture area (or “block”) are predicted for example by motion compensation means (finding and indicating an area in one of the previously coded video frames that corresponds closely to the block being coded) or by spatial means (using the pixel values around the block to be coded in a specified manner). Secondly the prediction error, e.g., the difference between the predicted block of pixels and the original block of pixels, is coded. This is typically done by transforming the difference in pixel values using a specified transform (e.g., Discrete Cosine Transform (DCT) or a variant of it), quantizing the coefficients and entropy coding the quantized coefficients. By varying the fidelity of the quantization process, the encoder can control the balance between the accuracy of the pixel representation (picture quality) and size of the resulting coded video representation (file size or transmission bitrate).

[0037] Inter prediction, which may also be referred to as temporal prediction, motion compensation, or motion-compensated prediction, exploits temporal redundancy. In inter prediction the sources of prediction are previously decoded pictures (a.k.a., also known as, reference pictures).

[0038] In temporal inter prediction, the sources of prediction are previously decoded pictures in the same scalable layer. In intra block copy (IBC; a.k.a. intra-block-copy, predictionmay be applied similarly to temporal inter prediction but the reference picture is the current picture and only previously decoded samples may be referred in the prediction process. Interlayer or inter-view prediction may be applied similarly to temporal inter prediction, but the reference picture is a decoded picture from another scalable layer or from another view, respectively. In some cases, inter prediction may refer to temporal inter prediction only, while in other cases inter prediction may refer collectively to temporal inter prediction and any of intra block copy, inter-layer prediction, and inter-view prediction provided that they are performed with the same or similar process than temporal prediction. Inter prediction, temporal inter prediction, or temporal prediction may sometimes be referred to as motion compensation or motion-compensated prediction.

[0039] Intra prediction utilizes the fact that adjacent pixels within the same picture are likely to be correlated. Intra prediction can be performed in spatial or transform domain, e.g., either sample values or transform coefficients can be predicted. Intra prediction is typically exploited in intra coding, where no inter prediction is applied.

[0040] One outcome of the coding procedure is a set of coding parameters, such as motion vectors and quantized transform coefficients. Many parameters can be entropy-coded more efficiently if they are predicted first from spatially or temporally neighboring parameters. For example, a motion vector may be predicted from spatially adjacent motion vectors and only the difference relative to the motion vector predictor may be coded. Prediction of coding parameters and intra prediction may be collectively referred to as in-picture prediction.

[0041] The decoder reconstructs the output video by applying prediction means similar to the encoder to form a predicted representation of the pixel blocks (using the motion or spatial information created by the encoder and stored in the compressed representation) and prediction error decoding (inverse operation of the prediction error coding recovering the quantized prediction error signal in spatial pixel domain). After applying prediction and a prediction error decoding technique, the decoder sums up the prediction and prediction error signals (pixel values) to form the output video frame. The decoder (and encoder) can also apply additional filtering techniques to improve the quality of the output video before passing it for display and / or storing it as prediction reference for the forthcoming frames in the video sequence.

[0042] In typical video codecs, the motion information is indicated with motion vectors associated with each motion compensated image block. Each of these motion vectors represents the displacement of the image block in the picture to be coded (in the encoder side) or decoded (in the decoder side) and the prediction source block in one of the previously coded or decoded pictures. In order to represent motion vectors efficiently those are typically coded differentially with respect to block specific predicted motion vectors. In typical video codecs, the predicted motion vectors are created in a predefined way, for example calculating the median of the encoded or decoded motion vectors of the adjacent blocks. Another way to create motion vector predictions is to generate a list of candidate predictions from adjacent blocks and / or colocated blocks in temporal reference pictures and signaling the chosen candidate as the motion vector predictor. In addition to predicting the motion vector values, the reference index of previously coded / decoded picture can be predicted. The reference index is typically predicted from adjacent blocks and / or or co-located blocks in temporal reference picture. Moreover, typical high efficiency video codecs employ an additional motion information coding / decoding mechanism, often called merging / merge mode, where all the motion field information, which includes motion vector and corresponding reference picture index for each available reference picture list, is predicted and used without any modification / correction. Similarly, predicting the motion field information is carried out using the motion field information of adjacent blocks and / or co-located blocks in temporal reference pictures and the used motion field information is signaled among a list of motion field candidate list filled with motion field information of available adjacent / co-located blocks.

[0043] In typical video codecs, the prediction residual after motion compensation is first transformed with a transform kernel (like DCT) and then coded. The reason for this is that often there still exists some correlation among the residual, and transform can in many cases help reduce this correlation and provide more efficient coding.

[0044] Typical video encoders utilize Lagrangian cost functions to find optimal coding modes, e.g., the desired Macroblock mode and associated motion vectors. This kind of cost function uses a weighting factor A. to tie together the (exact or estimated) image distortion due to lossy coding methods and the (exact or estimated) amount of information that is required to represent the pixel values in an image area:C = D + XR, where C is the Lagrangian cost to be minimized, D is the image distortion (e.g., Mean Squared Error) with the mode and motion vectors considered, and R is the number of bits needed to represent the required data to reconstruct the image block in the decoder (including the amount of data to represent the candidate motion vectors).

[0045] The High Efficiency Video Coding (H.265 / HEVC a.k.a. HEVC) standard was originally developed by the Joint Collaborative Team - Video Coding (JCT-VC) of VCEG and MPEG. The standard was published by both parent standardization organizations, and it is referred to as ITU-T Recommendation H.265 and ISO / IEC International Standard 23008-2, also known as MPEG-H Part 2 High Efficiency Video Coding (HEVC). Version 2 of the H.265 / HEVC standard included scalable, multiview, fidelity range, three-dimensional, and screen content coding extensions which may be abbreviated SHVC, MV-HEVC, REXT, 3D- HEVC, and SCC, respectively.

[0046] Versatile Video Coding (VVC) (MPEG-I Part 3), a.k.a. ITU-T H.266, is a video compression standard developed by the Joint Video Experts Team (JVET) of the Moving Picture Experts Group (MPEG), (formally ISO / IEC JTC1 SC29 WG11) and Video Coding Experts Group (VCEG) of the International Telecommunication Union (ITU) to be the successor to HEVC / H.265.

[0047] A specification of the AVI bitstream format and decoding process were developed by the Alliance for Open Media (AOM). The AVI specification was published in 2018. AOM is reportedly working on the AV2 specification.

[0048] Some key definitions, bitstream and coding structures, and concepts of some video coding standards and specifications are described in this section for providing background for a video encoder, decoder, encoding method, decoding method, and a bitstream structure, wherein the embodiments may be implemented. It is to be understood that embodiments are not limited to the referenced video coding standards or specifications.

[0049] Video coding standards may specify the bitstream syntax and semantics as well as the decoding process for error-free bitstreams, whereas the encoding process might not be specified, but encoders may just be required to generate conforming bitstreams. Bitstream and decoder conformance can be verified with the Hypothetical Reference Decoder (HRD). Thestandards may contain coding tools that help in coping with transmission errors and losses, but the use of the tools in encoding may be optional and decoding process for erroneous bitstreams might not have been specified.

[0050] An elementary unit for the input to an encoder and the output of a decoder, respectively, in many cases is a picture. A picture given as an input to an encoder may also be referred to as a source picture, and a picture decoded by a decoder may be referred to as a decoded picture or a reconstructed picture.

[0051] The source and decoded pictures are each comprised of one or more sample arrays. The sample arrays of a picture may be referred to as luma (or L or Y) and chroma, where the two chroma arrays may be referred to as Cb and Cr; regardless of the actual color representation method in use. The actual color representation method in use can be indicated, e.g., in a coded bitstream e.g., using the Video Usability Information (VUI) syntax of HEVC or alike. A component may be defined as an array or single sample from one of the three sample arrays (luma and two chroma) or the array or a single sample of the array that compose a picture in monochrome format.

[0052] Samples of a sample array have a certain bit depth, such as 8 bits per sample or 10 bits per sample. A bit depth implicitly specifies a value range, which may be referred to as the full range. For example, the full range is from 0 to 255, inclusive, for 8 bits per sample, or from 0 to 1023, inclusive, for 10 bits per sample. The source video may use allocate a narrower sample value range than the full range. A specific value range, sometimes referred to as the studio range, has been specified in the ITU-T H.273 standard specifying coding-independent code points for video. A source value range may interchangeably be referred to as a source sample value range, and may be defined as the sample value range of the video that is given as input to a video encoder to be encoded.

[0053] A picture may be defined to be either a frame or a field. A frame comprises a matrix of luma samples and possibly the corresponding chroma samples. A field is a set of alternate sample rows of a frame and may be used as encoder input, when the source signal is interlaced. Chroma sample arrays may be absent (and hence monochrome sampling may be in use) or chroma sample arrays may be subsampled when compared to luma sample arrays.

[0054] A bitstream may be defined as a sequence of bits or a sequence of syntax structures. A bitstream format may constrain the order of syntax structures in the bitstream.

[0055] A syntax element may be defined as an element of data represented in a bitstream. A syntax structure may be defined as zero or more syntax elements present together in a bitstream in a specified order.

[0056] Syntax structures may be specified, for example, using arithmetic, logical, relational, bit-wise, and assignment operators similar to those available in many programming languages. For example, & may indicate a bit-wise ‘AND’ operation. Furthermore, syntax structures may be specified with reference to mathematical functions.

[0057] Syntax structures and semantics may use the values of variables derived from the values of syntax elements. Naming conventions may be defined for variables. For example, variables may be named by a mixture of lower case and upper-case letter and without any underscore characters. Variables starting with an upper-case letter may be derived for the decoding of the current syntax structure and all depending syntax structures. Variables starting with an upper-case letter may, in some cases, be used in the decoding process for later syntax structures without mentioning the originating syntax structure of the variable. Variables starting with a lower-case letter may only be used in relation to the syntax structure or function they have been defined for.

[0058] Video coding specifications may define an elementary unit that for the output an of an encoder and / or for the input to a decoder. For example, such an elementary unit may be an open bitstream unit (OBU), as specified e.g. in AVI, or a Network Abstraction Layer (NAL) unit, as specified e.g. in HEVC or VVC.

[0059] In some video codecs, an elementary unit for the output of an encoder and the input of a decoder, respectively, may be a Network Abstraction Layer (NAL) unit. For transport over packet-oriented networks or storage into structured files, NAL units may be encapsulated into packets or similar structures. A bytestream format has been specified in some video coding standards for transmission or storage environments that do not provide framing structures. The bytestream format separates NAL units from each other by attaching a start code in front of each NAL unit. To avoid false detection of NAL unit boundaries, encoders run a byte-oriented start code emulation prevention algorithm, which adds an emulation prevention byte to the NAL unitpayload if a start code would have occurred otherwise. In order to enable straightforward gateway operation between packet- and stream-oriented systems, start code emulation prevention may always be performed regardless of whether the bytestream format is in use or not. A NAL unit may be defined as a syntax structure containing an indication of the type of data to follow and bytes containing that data in the form of an RBSP interspersed as necessary with emulation prevention bytes. A raw byte sequence payload (RBSP) may be defined as a syntax structure containing an integer number of bytes that is encapsulated in a NAL unit. An RBSP is either empty or has the form of a string of data bits containing syntax elements followed by an RBSP stop bit and followed by zero or more subsequent bits equal to 0 (zero).

[0060] A bitstream may be defined to logically include a syntax structure, such as a NAL unit, when the syntax structure is transmitted along the bitstream but may be included in the bitstream according to the bitstream format. A bitstream may be defined to natively comprise a syntax structure, when the bitstream includes the syntax structure.

[0061] In some coding formats or standards, a bitstream may be in the form of a network abstraction layer (NAL) unit stream or a byte stream, that forms the representation of coded pictures and associated data forming one or more coded video sequences.

[0062] In some coding formats, such as AVI, a bitstream may comprise a sequence of open bitstream units (OBUs). An OBU comprises a header and a payload, wherein the header identifies a type of the OBU. Furthermore, the header may comprise a size of the payload in bytes.

[0063] In some coding standards, NAL units include a header and payload. In some coding standards, the NAL unit header indicates the type of the NAL unit. In some coding standards, the NAL unit header indicates a scalability layer identifier (e.g., called nuh_layer_id), which may be used, e.g., for indicating spatial or quality layers, views of a multiview video, or auxiliary layers (such as depth maps or alpha planes). In some coding standards, the NAL unit header includes a temporal sublayer identifier, which may be used for indicating temporal subsets of the bitstream, such as a 30-frames-per-second subset of a 60-frames-per-second bitstream.

[0064] Bitstreams or coded video sequences may be encoded to be temporally scalable as follows. Each picture may be assigned to a particular temporal sub-layer. A temporalsub-layer may be equivalently called a sub-layer, temporal sublayer, sublayer, or temporal level. Temporal sub-layers may be enumerated, e.g., from 0 (zero) upwards. The lowest temporal sublayer, sub-layer 0, may be decoded independently. Pictures at temporal sub-layer 1 may be predicted from reconstructed pictures at temporal sub-layers 0 and 1. Pictures at temporal sublayer 2 may be predicted from reconstructed pictures at temporal sub-layers 0, 1 , and 2, and so on. In other words, a picture at temporal sub-layer N does not use any picture at temporal sublayer greater than N as a reference for inter prediction. The bitstream created by excluding all pictures greater than or equal to a selected sub-layer value and including pictures remains conforming.

[0065] Each picture of a temporally scalable bitstream may be assigned with a temporal identifier (also known as HD, temporal layer identifier, sub-layer identifier, sublayer identifier, temporal sub-layer identifier, temporal sublayer identifier, or temporal layer ID), which may be, for example, assigned to a variable Temporalld. The temporal identifier may, for example, be indicated in a NAL unit header or in an OBU extension header. Temporalld equal to 0 corresponds to the lowest temporal level. The bitstream created by excluding all coded pictures having a Temporalld greater than or equal to a selected value and including all other coded pictures remains conforming. Consequently, a picture having Temporalld equal to tid value does not use any picture having a Temporalld greater than tid value as a prediction reference. In some video coding standards, a sub-layer or a temporal sub-layer may be defined to be a temporal scalable layer (or a temporal layer, TL) of a temporal scalable bitstream, consisting of VCL NAL units with a particular value of the Temporalld variable and the associated non-VCL NAL units.

[0066] NAL units can be categorized into Video Coding Layer (VCL) NAL units and non-VCL NAL units. VCL NAL units are typically coded slice NAL units.

[0067] A non-VCL NAL unit may be for example one of the following types: a video parameter set (VPS), a sequence parameter set (SPS), a picture parameter set (PPS), an adaptation parameter set (APS), a supplemental enhancement information (SEI) NAL unit, an access unit delimiter, an end of sequence (EOS) NAL unit, an end of bitstream (EOB) NAL unit, or a filler data NAL unit. Parameter sets may be needed for the reconstruction of decoded pictures, whereas many of the other non-VCL NAL units may not be necessary for the reconstruction of decoded sample values.

[0068] Some coding formats specify parameter sets that may carry parameter values needed for the decoding or reconstruction of decoded pictures. A parameter may be defined as a syntax element of a parameter set. A parameter set may be defined as a syntax structure that contains parameters and that can be referred to from or activated by another syntax structure, for example, using an identifier.

[0069] Instead of or in addition to parameter sets at different hierarchy levels (e.g., sequence and picture), video coding formats may include header syntax structures, such as a sequence header or a picture header.

[0070] A sequence header may precede any other data of the coded video sequence in the bitstream order. It may be allowed to repeat a sequence header in the bitstream, e.g., to provide a sequence header at a random-access point.

[0071] A picture header may precede any coded video data for the picture in the bitstream order. A picture header may be interchangeably referred to as a frame header. Some video coding specifications may enable carriage of a picture header in a dedicated picture header NAL unit or a frame header OBU or alike. Some video coding specifications may enable carriage of a picture header in a NAL unit, OBU, or alike syntax structure that also contains coded picture data.

[0072] A coded picture may be defined as a coded representation of a picture.

[0073] A random-access point may be defined as a location within a bitstream where decoding can be started.

[0074] A random-access picture or a Random-Access Point (RAP) picture may be defined as a picture that serves as a random-access point, i.e., as a picture where decoding can be started. In some contexts, the term random-access picture may be used interchangeably with the term RAP picture.

[0075] An intra random access point (IRAP) picture, when contained in a single-layer bitstream or an independent layer, may comprise only intra-coded image segments. Furthermore, an IRAP picture may constrain subsequence pictures in output order to be such that they can be correctly decoded without performing the decoding process of any pictures that precede the IRAP picture in decoding order.

[0076] Some coding standards or specifications, such as H.265 / HEVC, may use the NAL unit type of VCL NAL unit(s) of a picture to indicate a picture type. In H.266 / VVC, the NAL unit type indicates a picture type when mixed VCL NAL unit types within a coded picture are disabled (pps_mixed_nalu_types_in_pic_flag is equal to 0 in the referenced PPS, picture parameter set), while otherwise it indicates a subpicture type.

[0077] Some coding standards or specifications may indicate a picture type in a picture header or a frame header or alike.

[0078] In some coding formats, picture unit (PU) may be defined as a set of data units, such as NAL units, that are associated with each other, are consecutive in decoding order, and contain exactly one coded picture. For example, certain non- video-coding data units, such as non- VCL NAL units, may be next to coded video data units in decoding order and the respective picture unit may comprise both these non-video-coding data units and the video coding data units of a coded picture.

[0079] In some coding formats, an access unit (AU) may comprise coded pictures that have the same output time. In some coding formats, an AU may be defined as a set of NAL units that are associated with each other according to a specified classification rule, are consecutive in decoding order, and include at most one coded picture at any scalability layer (e.g., with any specific value of nuh layer id in some coding formats, such as HEVC or WC). In some coding formats, an access unit comprises one or more complete picture units. In some coding formats, in addition to including the VCL NAL units of a coded picture, an access unit may also include non-VCL NAL units associated with the coded picture. Said specified classification rule may, for example, associate pictures with the same output time or picture order count value into the same access unit.

[0080] In some coding formats, a coded video sequence (CVS) may be defined as a sequence of coded pictures in decoding order that is independently decodable and is followed by another coded video sequence or the end of the bitstream.

[0081] In some coding formats, such as AVI, a coded video sequence comprises one or more temporal units. A temporal unit consists of a series of OBUs starting from a temporal delimiter, optional sequence headers, optional metadata OBUs, a sequence of one or more frame headers, each followed by zero or more tile group OBUs as well as optional padding OBUs. Atemporal unit may be defined to comprise all the OBUs that are associated with a specific, distinct time instant. A temporal unit may comprise a temporal delimiter OBU, and all the OBUs that follow, up to but not including the next temporal delimiter. A temporal delimiter OBU may be defined as an indication that the following OBUs will have a different presentation / decoding time stamp from the one of the last frame prior to the temporal delimiter.

[0082] A coded layer video sequence (CLVS) may be defined as a sequence of pictures and associated other data within the same scalable layer (e.g., with the same value of nuh layer id) that is decodable independently of other pictures in the same layer.

[0083] Some video coding specifications enable metadata OBUs. A metadata OBU comprises a type field, which specifies the type of metadata. A metadata OBU may be understood to be similar to an SEI (supplemental enhancement information) NAU unit or an SEI message.

[0084] Video coding specifications may enable the use of supplemental enhancement information (SEI) messages or alike. Some video coding specifications include SEI NAL units, and some video coding specifications contain both prefix SEI NAL units and suffix SEI NAL units, where the former type can start a picture unit or alike and the latter type can end a picture unit or alike. An SEI NAL unit contains one or more SEI messages, which are not required for the decoding of output pictures but may assist in related processes, such as picture output timing, post-processing of decoded pictures, rendering, error detection, error concealment, and resource reservation.

[0085] ITU-T Recommendation H.274, which is equivalent to ISO / IEC 23002-7, may be called "versatile supplemental enhancement information messages for coded video bitstreams" and be referred to as "versatile supplemental enhancement information" or VSEI. The VSEI standard specifies the syntax and semantics of video usability information (VUI) parameters and supplemental enhancement information (SEI) messages. The VUI parameters and SEI messages defined in the VSEI standard are designed to be conveyed within coded video bitstreams in a manner specified in a video coding specification or to be conveyed by other means determined by the specifications for systems that make use of such coded video bitstreams. The VSEI standard is intended for use with VVC coded video bitstreams, although itis drafted in a manner intended to be sufficiently generic that it may also be used with other types of coded video bitstreams.

[0086] Several SEI messages are specified in H.264 / AVC, H.265 / HEVC,H.266 / VVC, and H.274 / VSEI standards, and the user data SEI messages enable organizations and companies to specify SEI messages for their own use. The standards may contain the syntax and semantics for the specified SEI messages but a process for handling the messages in the recipient might not be defined. Consequently, encoders may be required to follow the standard specifying a SEI message when they create SEI message(s), and decoders might not be required to process SEI messages for output order conformance. One of the reasons to include the syntax and semantics of SEI messages in standards is to allow different system specifications to interpret the supplemental information identically and hence interoperate. It is intended that system specifications can require the use of particular SEI messages both in the encoding end and in the decoding end, and additionally the process for handling particular SEI messages in the recipient can be specified.

[0087] Referring to FIG. 1, this figure is a block diagram illustrating a system 100 in accordance with an example. In the example, the encoder 130 is used to encode input video 110-I, and the encoder 130 is implemented in a transmitting apparatus 180-1. The input video 110-1 can be from a parallel camera arrangement 10 or from, see reference 11 , one camera plus auxiliary video, which could be an object mask video. In the parallel camara arrangement 10 example, there could be two outputs: Ultra-High Definition (UHD) 12-1 view; and Full High Definition (FHD) view 12-2. In the one camera plus auxiliary video 11 example, there is camera out 13-1 and aux (auxiliary) video 13-2. It is to be understood that the input video 110-1 is not limited to just these types of video content but may generally be any video including multiple domains. The encoder 130 produces a bitstream 101, using the encoding method 131 on the input video 110-1, that is received by the receiving apparatus 180-2. The receiving apparatus 180-2 implements a decoder 140, which performs a decoding method 141. The decoder 140, using the decoding method 141 on the bitstream 101, forms the output video 110-2 (as a representation of the input video 110-1). The receiving apparatus 180-2 may present the output video 110-2 to the user, e.g., via a smartphone, television, or projector among many other options and / or may provide the output video 110-2 to one or more computer vision tasks.

[0088] It is to be understood that embodiments may apply only to the receiving apparatus 180-2 or only to the transmitting apparatus 180-1.

[0089] It is to be understood that the transmitting apparatus may transmit the bitstream or may output the bitstream by other means, such as by storing the bitstream into a mass memory or by providing the bitstream through an interface, such as a shared memory bus, to another apparatus.

[0090] It is to be understood that the receiving apparatus may receive the bitstream or may input the bitstream by other means, such as by reading the bitstream from a mass memory or accessing the bitstream through an interface, such as a shared memory bus, from another apparatus.

[0091] Referring to FIG. 2, which is split into FIGS. 2A and 2B, this figure illustrates a video encoder 205 where the encoder 130 examples can be implemented. It is to be understood that the video encoder 205 is merely an example, and embodiments may likewise be implemented with any video encoder. It is noted that the terms “image” and “picture” are considered to be the same herein. FIG. 2 presents an encoder 205 for two layers, but it would be appreciated that presented encoder 205 could be similarly narrowed down to one layer or extended to encode more than two layers. It is to be understood that embodiments may be realized with any number of layers. FIG. 2 illustrates an embodiment of a video encoder 205 comprising a first encoder section 200 for a base layer and a second encoder section 200-1 for an enhancement layer (which is also referred to herein as a dependent layer). The first encoder section 200 uses a normal numbering scheme, and the second encoder section 200-1 uses a numbering scheme where each element has a “-1” added. The elements in the two encoder sections 200 and 200-1 are the same or similar and will be described similarly herein except for the differences between the two.

[0092] The input video 110-1 is split into two parts: input video 110-11; and input video 110-12. For stereoscopic video, the UHD view 12-1 can be applied to the base layer 200 as input video 110-11, and the FHD view 12-2 to the enhancement layer 200-1 as input video 110- 12. Alternatively, the FHD view 12-2 can be applied to the base layer 200 as input video 110-11, and the UHD view 12-1 to the enhancement layer 200-1 as input video 110-12. The camera- captured video 13-1 and auxiliary video 13-2 can be applied to different layers, which may beboth independent layers. For the example where the camera-captured video 13-1 and auxiliary video 13-2 are applied to different layers 200, 200-1 the link that is shown between the RFM 218 in the base layer 200 and the Pinter 206-1 in the enhancement layer 200-1 would not exist or not be used.

[0093] The encoders 200, 200-1 comprise pixel predictors 202, 202-1, prediction error encoders 203, 203-1, and prediction error decoders 204, 204-1. FIG. 2 also shows an embodiment of the pixel predictors 202, 202-1 as comprising inter-predictors (Pinter) 206, 206- 1, intra-predictors (Pintra) 208, 208-1, mode selectors 210, 210-1, filters (F) 216, 216-1, and reference frame memories (RFMs) 218, 218-1. The pixel predictors 202, 202-1 of the encoders 200, 200-1 receive base layer pictures (10, n) 201, 201-1 of input video 110-1 (e.g., a video stream) to be encoded at both the inter-predictors 206, 206-1 (which determines the difference between the picture and a motion compensated reference frame from the RFMs 218, 218-1) and the intra-predictors 208, 208-1 (which determines a prediction for an image block based only on the already processed parts of the current picture from base layer picture 201, 201-1). The output of both the inter-predictors 206, 206-1 and the intra-predictors 208, 208-1 are passed to the mode selectors 210, 210-1. The intra-predictors 208, 208-1 may have more than one intra-prediction mode. Hence, each mode may perform the intra-prediction and provide the predicted signal to the mode selectors 210, 210-1. The mode selectors 210, 210-1 also receive a copy of the base layer pictures (10, n) 201, 201-1.

[0094] Depending on which encoding mode is selected to encode the current block, the output of the inter-predictor 206, 206-1, or the output of one of the intra-predictor 208, 208- 1 modes, or the output of a surface encoder within the mode selector 210, 210-1 is passed to the output of the mode selector 210, 210-1. The output of the mode selector 210, 210-1 is passed to first summing device 221, 221-1 and comprises the prediction representation of the image block (P’n) 212, 212-1. The first summing device 221, 221-1 may subtract the prediction representation of the image block (P’n) 212, 212-1 of the pixel predictor 202, 202-1 from the base layer picture 201, 201-1 to produce a first prediction error signal (Dn) 220, 220-1, which is input to the prediction error encoder 203, 203-1.

[0095] The pixel predictor 202, 202-1 further receives from a second summing device 239, 239-1, which acts as a preliminary reconstructor, the combination of the predictionrepresentation of the image block (P’n) 212, 212-1 and the prediction error signal (D’n) 238, 238-1 of the prediction error decoder 204, 204-1. The preliminary reconstructed picture (rn) 214, 214-1 may be passed to the intra-predictor 208, 208-1 and to a filter (F) 216, 216-1. The filter 216, 216-1 receiving the preliminary representation may filter the preliminary representation and output a final reconstructed picture 240, 240-1, which may be saved in the reference frame memory (RFM) 218, 218-1. The reference frame memory 218, 218-1 may be connected to the inter-predictor 206, 206-1 to be used as the reference picture against which a future base layer picture 201, 201-1 is compared in inter-prediction operations.

[0096] Subject to the base layer (e.g., first encoder section 200) being selected and indicated to be a source for inter-layer sample prediction and / or inter-layer motion information prediction of the enhancement layer (e.g., second encoder section 200-1) according to some embodiments, the reference frame memory 218 may also be connected to the inter-predictor 206- 1 to be used as the reference image against which a future enhancement layer picture 201-1 is compared in inter-prediction operations. Moreover, the reference frame memory 218-1 may be connected to the inter-predictor 206-1 to be used as the reference image against which a future enhancement layer picture 201-1 is compared in inter-prediction operations.

[0097] Filtering parameters from the filter 216 of the first encoder section 200 may be provided to the second encoder section 200-1 subject to the base layer being selected and indicated to be source for predicting the filtering parameters of the enhancement layer according to some embodiments.

[0098] The prediction error encoder 203, 203-1 comprises a transform unit (T) 242, 242-1 and a quantizer (Q) 244, 244-1. The transform unit 242, 242-1 transforms the first prediction error signal 220, 220-1 to a transform domain. The transform is, for example, the DCT (discrete cosine transform). The quantizer 244, 244-1 quantizes the transform domain signal, e.g., the DCT coefficients, to form quantized coefficients.

[0099] The prediction error decoder 204, 204-1 receives the output from the prediction error encoder 203, and performs the opposite processes of the prediction error encoder 203, 203-1 to produce a decoded prediction error signal (D’n) 238, 238-1 which, when combined with the prediction representation of the image block 212, 212-1 at the second summing device 239, 239-1, produces the preliminary reconstructed picture 214, 214-1. The prediction errordecoder 204, 204-1 may be considered to comprise a dequantizer (Q-l) 246, 246-1, which dequantizes the quantized coefficient values, e.g., DCT coefficients, to reconstruct the transform signal and an inverse transformation unit (T-l) 248, 248-1, which performs the inverse transformation to the reconstructed transform signal wherein the output of the inverse transformation unit 248, 248-1 contains reconstructed block(s). The prediction error decoder 204, 204-1 may also comprise a block filter (not shown) which may filter the reconstructed block(s) according to further decoded information and filter parameters.

[0100] The entropy encoder 230, 230-1 receives the output of the prediction error encoder 203, 203-1 and may perform a suitable entropy encoding / variable length encoding to provide error detection and correction capability. The output of the entropy encoder 230, 230-1 can be influenced by the signaling 206, 206-1 from the mode selector 210, 210-1, e.g., which can indicate, e.g., whether inter prediction or intra prediction is being used. The output of the entropy encoder 230, 230-1 may be inserted into a bitstream 101, e.g., by a multiplexer 508. Entropy coding / decoding may be performed in many ways. For example, context-based coding / decoding may be applied, wherein both the encoder and the decoder modify the context state of a coding parameter based on previously coded / decoded coding parameters. Context based coding may for example be context adaptive binary arithmetic coding (CAB AC), or context-based variable length coding (CAVLC) or any similar entropy coding. Entropy coding / decoding may alternatively or additionally be performed using a variable length coding scheme, such as Huffman coding / decoding or Exp-Golomb coding / decoding. Decoding of coding parameters from an entropy - coded bitstream or codewords may be referred to as parsing.

[0101] Referring to FIG. 3, which is split over FIGS. 3A and 3B, this figure illustrates a video decoder 390 where the decoder 140 examples can be implemented. FIG. 3 presents a video decoder 390 for two layers, but it would be appreciated that presented decoder 390 could be similarly narrowed down to one layer or extended to decode more than two layers. It is to be understood that embodiments may be realized with any number of layers. FIG. 3 illustrates an embodiment of a video decoder 390 comprising a first decoder section 300 for a base layer and a second decoder section 300-1 for an enhancement layer. The first decoder section 300 uses a normal numbering scheme, and the second decoder section 300-1 uses a numbering scheme where each element has a “-1” added. The elements in the two decodersections 300 and 300-1 are the same or similar and will be described similarly herein except for the differences between the two.

[0102] The video decoders 300, 300-1 are coupled to the bitstream 101. There is a prediction error decoder 301, 301-1 and a pixel prediction 304, 304-1. For the prediction error decoder 301, 301-1, block 303, 303-1 illustrates inverse transforms (T4), and block 302, 302-1 illustrates inverse quantization (Q4). Block 330, 330-1 illustrates entropy decoding (E4) that has an output for the pixel prediction 304, 304-1. Reference P'nstands for a predicted representation of an image block. Reference D'nstands for a reconstructed prediction error signal. In pixel prediction 304, 304-1, block 305, 305-1 illustrates preliminary reconstructed pictures (I'n) based on the P'n from block 307, 307-1 and the output of the adder 309, 309-1, block 307, 307-1 illustrates prediction (P) (either inter-prediction or intra-prediction), which produces P'nfor the adder 309, 309-1 and as the block 305, 305-1. There is flow from the RFM 306 to the P307-1, which illustrates inter-layer prediction for at least some examples herein. Block 330, 330-1 illustrates entropy decoding (E-1). and block 306, 306-1 illustrates a reference frame memory (RFM). Adder 309, 309-1 may be used to combine decoded prediction error information (D’n) with predicted base layer / predicted layer pictures (P’n) to obtain the preliminary reconstructed pictures (I'n) 305, 305-1. Block 308, 308-1 illustrates filtering (F) and reference R'nstands for a final reconstructed picture 380, 380-1.

[0103] It is noted that cross-domain NN filtering 350 is where the examples herein take place and this block is a new part of the decoder 390 relative to decoders that existed before this document. That is, examples herein for decoding may be implemented, at least in part, as illustrated by the cross-domain NN filtering 350, which operates on the final reconstructed picture R'n380 from base layer 300 and the final reconstructed picture R'n380-1 from enhancement layer 300-1, and cross-domain NN filtering 350 replaces one of the base layer R'n380 or enhancement layer R'n380 as described below.

[0104] Lb. Reference picture resampling

[0105] In reference picture resampling (RPR), which may also be called adaptive resolution change (ARC), the decoding process of a picture may refer to one or more previous reference pictures that have a different spatial resolution for inter prediction. Consequently, a resampling of the reference pictures for operation of the inter-picture prediction process may beapplied. In general, resampling may be either picture-wise or block-wise. In picture-wise resampling, an entire reference picture is resampled, whereas in block-wise resampling, an area within a reference picture, such as a reference block for motion compensation, is resampled. The block-wise resampling may be performed as a part of the motion compensation process. Resampling may generally comprise downsampling and upsampling.

[0106] WC (versatile video coding) enables picture resolution change within a sequence at a position without encoding an IRAP (intra random access point) picture, which is always intra-coded in an independent layer. This feature is sometimes referred to as reference picture resampling (RPR), as the feature needs resampling of a reference picture used for interprediction when that reference picture has a different resolution than the current picture being decoded. In order to avoid additional processing steps, the RPR process in WC is designed to be embedded in the motion compensation process and performed at the block level.

[0107] In the motion compensation stage, the scaling ratio is used together with motion information to locate the reference samples in the reference picture to be used in the interpolation process.

[0108] In WC, the scaling ratio is restricted to be larger than or equal to 1 / 2 (2 times downsampling from the reference picture to the current picture), and less than or equal to 8 (8 times upsampling). Three sets of resampling filters with different frequency cutoffs are specified to handle various scaling ratios between a reference picture and the current picture. The three sets of resampling filters are applied respectively for the scaling ratio ranging from 1 / 2 to 1 / 1.75, from 1 / 1.75 to 1 / 1.25, and from 1 / 1.25 to 8. Each set of resampling filters has 16 phases for luma and 32 phases for chroma which is the same to the case of motion compensation interpolation filters. It is worthy to note that the filter set of normal motion compensation (MC) interpolation is used in the case of scaling ratio ranging from 1 / 1.25 to 8. Actually, the normal MC interpolation process is a special case of the resampling process with scaling ratio ranging from 1 / 1.25 to 8. The horizontal and vertical scaling ratios are derived based on picture width and height, and the left, right, top and bottom scaling offsets specified for the reference picture and the current picture. Scaling window for each picture is calculated by adding the scaling offset to the picture size. Then the scaling ratio is derived by dividing the scaling window on the reference frame to that of the current frame.

[0109] A scaling window may be signaled for and associated with a picture. Scaling windows of a picture and its reference picture may indicate the spatial correspondence between the pictures. Scaling windows may be used to achieve one or both of the following: i) a horizontal scaling ratio and a vertical scaling ratio may be derived based on the width ratio and the height ratio, respectively, of the scaling windows; ii) a left offset and a top offset for inter prediction may be derived. The horizontal and vertical scaling ratios may be used as resampling ratios for RPR. The left and top offsets may be used in deriving a spatial correspondence between a picture and its reference picture. The left and top offsets may be added in the reference picture location derivation through a motion vector scaled by a scaling ratio. For example, the top-left corner of the current block is mapped to the respective “anchor location” in the reference picture through the left and top offsets, and a motion vector scaled by a scaling ratio is then applied to obtain a reference block in relation to anchor location. In general, the scaling ratio may be 1 :1, in which case the left and top offsets may be applied without resampling. It needs to be understood that there may be other means to signal scaling ratio(s), top offset, and / or left offset than scaling windows.

[0110] In WC, a scaling window may be indicated in a PPS explicitly or implicitly. When indicated explicitly, pps_scaling_win_left_offset, pps_scaling_win_right_offset, pps scaling win top offset, and pps scaling win bottom offset specify the offsets of the scaling window boundaries relative to the left, right, top, and bottom boundaries of the conformance cropping window, respectively. When indicated implicitly, the scaling window is inferred to be identical to the conformance cropping window.

[0111] I.c. Scalable video coding

[0112] Scalable video coding may refer to coding structure where one bitstream may include multiple representations of the content, for example, at different bitrates, resolutions or frame rates. In these cases, the receiver can extract the desired representation depending on its characteristics (e.g., resolution that matches best the display device). Alternatively, a server or a network element may extract the portions of the bitstream to be transmitted to the receiver depending on, e.g., the network characteristics or processing capabilities of the receiver. A meaningful decoded representation may be produced by decoding only certain parts of a scalable bitstream. A scalable bitstream typically include of a ‘base layer’ providing the lowest qualityvideo available and one or more enhancement layers that enhance the video quality when received and decoded together with the lower layers. In order to improve coding efficiency for the enhancement layers, the coded representation of that layer typically depends on the lower layers. For example, the motion and mode information of the enhancement layer can be predicted from lower layers. Similarly, the pixel data of the lower layers can be used to create prediction for the enhancement layer.

[0113] A scalable bitstream may include a ‘base layer’ providing the lowest quality video available and one or more enhancement layers that enhance the video quality when received and decoded together with the lower layers. In order to improve coding efficiency for the enhancement layers, the coded representation of that layer may depend on the lower layers. E.g., the motion and mode information of the enhancement layer may be predicted from lower layers. Similarly, the pixel data of the lower layers can be used to create prediction for the enhancement layer. A scalable video codec for quality scalability (also known as signal-to-noise or SNR) and / or spatial scalability may be implemented as follows. For a base layer, a conventional non-scalable video encoder and decoder is used. The reconstructed / decoded pictures of the base layer are included in the reference picture buffer for an enhancement layer. In codecs using reference picture list(s) for inter prediction, the base layer decoded pictures may be inserted into a reference picture list(s) for coding / decoding of an enhancement layer picture similarly to the decoded reference pictures of the enhancement layer. Consequently, the encoder may choose a base-layer reference picture as inter prediction reference and indicate its use, e.g., with a reference picture index in the coded bitstream. The decoder decodes from the bitstream, for example from a reference picture index, that a base-layer picture is used as inter prediction reference for the enhancement layer. When a decoded base-layer picture is used as prediction reference for an enhancement layer, it is referred to as an inter-layer reference picture.

[0114] It needs to be understood that the description of scalable video coding may be generalized to any scalability hierarchy with more than two layers. In this case, a second enhancement layer may depend on a first enhancement layer in encoding and / or decoding processes, and the first enhancement layer may therefore be regarded as the base layer for the encoding and / or decoding of the second enhancement layer. Furthermore, it needs to be understood that there may be inter-layer reference pictures from more than one layer in areference picture buffer or reference picture lists of an enhancement layer, and each of these inter-layer reference pictures may be considered to reside in a base layer or a reference layer for the enhancement layer being encoded and / or decoded. Furthermore, it needs to be understood that other types of inter-layer processing than reference-layer picture upsampling may take place instead or additionally. For example, the bit-depth of the samples of the reference-layer picture may be converted to the bit depth of the enhancement layer and / or the sample values may undergo a mapping from the color space of the reference layer to the color space of the enhancement layer.

[0115] A scalable video coding and / or decoding scheme may use multi-loop coding and / or decoding, which may be characterized as follows. In the encoding / decoding, a base layer picture may be reconstructed / decoded to be used as a motion-compensation reference picture for subsequent pictures, in coding / decoding order, within the same layer or as a reference for interlayer (or inter-view or inter-component) prediction. The reconstructed / decoded base layer picture may be stored in the decoded picture buffer (DPB). An enhancement layer picture may likewise be reconstructed / decoded to be used as a motion-compensation reference picture for subsequent pictures, in coding / decoding order, within the same layer or as reference for inter-layer (or interview or inter-component) prediction for higher enhancement layers, when any. In addition to reconstructed / decoded sample values, syntax element values of the base / reference layer or variables derived from the syntax element values of the base / reference layer may be used in the inter-lay er / inter-component / inter-view prediction.

[0116] Inter-layer prediction may be defined as prediction in a manner that is dependent on data elements (e.g., sample values or motion vectors) of reference pictures from a different layer than the layer of the current picture (being encoded or decoded). Many types of inter-layer prediction exist and may be applied in a scalable video encoder / decoder.

[0117] The types of inter-layer prediction may comprise, but are not limited to, one or more of the following: inter-layer sample prediction, inter-layer motion prediction, inter-layer residual prediction. In inter-layer sample prediction, at least a subset of the reconstructed sample values of a source picture for inter-layer prediction are used as a reference for predicting sample values of the current picture. In inter-layer motion prediction, at least a subset of the motion vectors of a source picture for inter-layer prediction are used as a reference for predicting motionvectors of the current picture. Typically, predicting information on which reference pictures are associated with the motion vectors is also included in inter-layer motion prediction. For example, the reference indices of reference pictures for the motion vectors may be inter-layer predicted and / or the picture order count or any other identification of a reference picture may be interlayer predicted. In some cases, inter-layer motion prediction may also comprise prediction of block coding mode, header information, block partitioning, and / or other similar parameters. In some cases, coding parameter prediction, such as inter-layer prediction of block partitioning, may be regarded as another type of inter-layer prediction. In inter-layer residual prediction, the prediction error or residual of selected blocks of a source picture for inter-layer prediction is used for predicting the current picture.

[0118] A direct reference layer may be defined as a layer that may be used for interlayer prediction of another layer for which the layer is the direct reference layer. A direct predicted layer may be defined as a layer for which another layer is a direct reference layer. An indirect reference layer may be defined as a layer that is not a direct reference layer of a second layer but is a direct reference layer of a third layer that is a direct reference layer or indirect reference layer of a direct reference layer of the second layer for which the layer is the indirect reference layer. An indirect predicted layer may be defined as a layer for which another layer is an indirect reference layer. A dependent layer may be defined as a layer that uses inter-layer prediction from one or more other layers. A dependent layer may be a direct predicted layer or an indirect predicted layer. An independent layer may be defined as a layer that does not have direct reference layers. In other words, an independent layer is not predicted using inter-layer prediction. A non-base layer may be defined as any other layer than the base layer, and the base layer may be defined as the lowest layer in the bitstream. An independent non-base layer may be defined as a layer that is both an independent layer and a non-base layer.

[0119] A multi-layer bitstream is a bitstream comprising multiple layers, which may be, but are not limited to, base and enhancement layers as discussed above for scalable video coding. A multilayer bitstream may additionally or alternatively comprise independent layers that do not have inter-layer prediction relationship between each other and may even represent different types of content. Any multi-layer bitstream may be regarded as a scalable video bitstream.

[0120] An output layer set (OLS) may be defined as a set of layers for which one or more layers are specified as the output layers. Output layer sets may be indicated in a VPS.

[0121] An output layer may be defined as a layer of an output layer set that is output.

[0122] An output layer set (OLS) layer index may be defined as an index, of a layer in an OLS, to the list of layers in the OLS, which may be indicated in a VPS.

[0123] Scalability types, scalability modes, scalability domains, or scalability dimensions may include but are not limited to the following:

[0124] 1) Quality scalability: Base layer pictures are coded at a lower quality than enhancement layer pictures, which may be achieved for example using a greater quantization parameter value (i.e., a greater quantization step size for transform coefficient quantization) in the base layer than in the enhancement layer.

[0125] 2) Spatial scalability: Base layer pictures are coded at a lower resolution (i.e. have fewer samples) than enhancement layer pictures. Spatial scalability and quality scalability may sometimes be considered the same type of scalability.

[0126] 3) Bit-depth scalability: Base layer pictures are coded at lower bit-depth (e.g.8 bits) than enhancement layer pictures (e.g. 10 or 12 bits).

[0127] 4) Dynamic range scalability: Scalable layers represent a different dynamic range and / or images obtained using a different tone mapping function and / or a different optical transfer function.

[0128] 5) Chroma format scalability: Base layer pictures provide lower spatial resolution in chroma sample arrays (e.g. coded in 4:2:0 chroma format) than enhancement layer pictures (e.g. 4:4:4 format).

[0129] 6) Color gamut scalability: enhancement layer pictures have a richer / broader color representation range than that of the base layer pictures - for example the enhancement layer may have UHDTV (ITU-R BT.2020) color gamut and the base layer may have the ITU-R BT.709 color gamut.

[0130] 7) Region-of-interest (ROI) scalability: An enhancement layer represents of spatial subset of the base layer. ROI scalability may be used together with other types of scalability, e.g. quality or spatial scalability so that the enhancement layer provides higher subjective quality for the spatial subset.

[0131] 8) View scalability, which may also be referred to as multiview coding. The base layer represents a first view, whereas an enhancement layer represents a second view. A view may be defined as a sequence of pictures representing one camera or viewpoint. It may be considered that in stereoscopic or two-view video, one video sequence or view is presented for the left eye while a parallel view is presented for the right eye.

[0132] 9) Auxiliary video scalability. A scalable layer represents auxiliary video, which may not be intended to be displayed as such. A scalable layer may represent for example as alpha mask, depth map, or an object mask video.

[0133] When one or more auxiliary layers represent depth, the type of scalability may be referred to as depth scalability, which may also be referred to as depth-enhanced coding. A layer or some layers of a bitstream may represent texture view(s), while other layer or layers may represent depth view(s).

[0134] It should be understood that many of the scalability types may be combined and applied together. For example, color gamut scalability and bit-depth scalability may be combined.

[0135] I .d. Neural networks

[0136] A neural network (NN) may be described as a computation graph consisting of several layers of computation. Each layer may include one or more units, where each unit performs an elementary computation. A unit is connected to one or more other units, and the connection may be associated with a weight. The weight may be used for scaling the signal passing through the associated connection. Weights are learnable parameters, i.e., values which can be learned from training data. There may be other learnable parameters, such as those of batch-normalization layers.

[0137] In some neural networks, such as convolutional neural networks for image classification, initial layers (those close to the input data) extract semantically low-level features such as edges and textures in images, whereas intermediate layers extract more high-level features. After the feature extraction layers there may be one or more layers performing a certain task, such as classification, semantic segmentation, object detection, denoising, style transfer, super-resolution, and the like.

[0138] Neural networks are being utilized in an ever-increasing number of applications for many different types of devices, such as mobile phones. Examples include imageand video analysis and processing, social media data analysis, device usage data analysis, and the like.

[0139] One property of neural nets (and other machine learning tools) is that they are able to learn properties from input data, e.g., in supervised way or in unsupervised way. Such learning is a result of a training algorithm, or of a meta-level neural network providing the training signal.

[0140] In general, the training algorithm includes changing some properties of the neural network so that its output is as close as possible to a desired output. For example, in the case of classification of objects in images, the output of the neural network can be used to derive a class or category index which indicates the class or category that the object in the input image belongs to. Training usually happens by minimizing or decreasing the output’s error, also referred to as the loss or loss function. Examples of losses are mean squared error, cross-entropy, etc. In recent deep learning techniques, training is an iterative process, where at each iteration the algorithm modifies the weights of the neural net to make a gradual improvement of the network’s output, i.e., to gradually decrease the loss, by means of gradient descent technique. In one example, at each training iteration, gradients of the loss function with respect to one or more weights or parameters of the NN are computed, for example by backpropagation technique; the computed gradients are then used by an optimization routine, such as Adam or Stochastic Gradient Descent (SGD) to obtain an update to the one or more weights or parameters.

[0141] The terms “model”, “neural network”, “neural net” and “network” are used interchangeably herein, and also the weights of neural networks are sometimes referred to as learnable parameters or simply as parameters.

[0142] Training a neural network is an optimization process, but the final goal may be different from the typical goal of optimization. In optimization, the only goal is to minimize a function. In machine learning, the goal of the optimization or training process is to make the model learn the properties of the data distribution from a limited training dataset. In other words, the goal is to learn to use a limited training dataset in order to learn to generalize to previously unseen data, i.e., data which was not used for training the model. This is usually referred to as generalization. In practice, data is usually split into at least two sets, the training set and the validation set. The training set is used for training the network, i.e., to modify its learnableparameters in order to minimize the loss. The validation set is used for checking the performance of the network on data which was not used to minimize the loss, as an indication of the final performance of the model. In particular, the errors on the training set and on the validation set are monitored during the training process to understand the following things:

[0143] 1) If the network is learning at all - in this case, the training set error should decrease, otherwise the model is in the regime of underfitting.

[0144] 2) If the network is learning to generalize - in this case, also the validation set error needs to decrease and to be not too much higher than the training set error.

[0145] 3) If the training set error is low, but the validation set error is much higher than the training set error, or it does not decrease, or it even increases, the model may be in the regime of overfitting. This means that the model has just memorized the training set’s properties and performs well only on that set, but performs poorly on a set not used for tuning its parameters.

[0146] Neural network inference may refer to applying the trained neural network with input data to obtain output data. Neural network inference may additionally comprise formatting the input data to be suitable to be provided to the neural network, which may comprise, for example, data format conversions (e.g. from integer values to floating point values), value range conversions (e.g. from a given integer range to a floating point range from 0 to 1 or from -1 to 1), and / or adding random noise to the input data. Neural network inference may comprise multiple iterations, i.e. applying the neural network multiple times, initially to the given input data and each subsequent time to the output of the previous iteration.

[0147] I.e. Use of neural networks in video coding

[0148] I.e.l. Neural-network post-filtering

[0149] Version 3 of the VSEI (versatile supplemental enhancement information) standard includes the specification of the neural -network post-filter characteristics (NNPFC) and neural-network post-filter activation (NNPFA) supplemental enhancement information (SEI) messages. Extensions to NNPFC SEI message are being specified for version 4 of the VSEI standard.

[0150] The syntax structure specifying the NNPFC SEI message may be called nn_post_filter_characteristics. The syntax structure specifying the NNPFA SEI message may becalled nn_post_filter_activation. The NNPFC SEI message comprises the nnpfc id syntax element, which includes an identifying number that may be used to identify a post-processing filter.

[0151] The NNPFC SEI message syntax comprises nnpfc_base_flag. nnpfc_base_flag equal to 1 specifies that the SEI message specifies the base NNPF. nnpfc base flag equal to 0 specifies that the SEI message specifies an update relative to the base NNPF. The following constraints apply to the value of nnpfc base flag:

[0152] i) When an NNPFC SEI message is the first NNPFC SEI message, in decoding order, that has a particular nnpfc id value within the current CL VS, the value of nnpfc base flag is required to be equal to 1.

[0153] ii) All NNPFC SEI messages in a CLVS that have a particular nnpfc id value and nnpfc base flag equal to 1 are required to have identical SEI payload content.

[0154] When nnpfc base flag is equal to 0, the following applies:

[0155] i) This SEI message defines an update relative to the preceding base NNPF in decoding order with the same nnpfc id value. Updates are not cumulative but rather each update is applied on the base NNPF, which is the NNPF specified by the first NNPFC SEI message, in decoding order, that has a particular nnpfc id value within the current CLVS. The NNPF defined by this SEI message is obtained by applying the update defined by this SEI message relative to the base NNPF with the same nnpfc_id value.

[0156] ii) This SEI message pertains to the current decoded picture and all subsequent decoded pictures of the current layer, in output order, until the end of the current CLVS or up to but excluding the decoded picture that follows the current decoded picture in output order within the current CLVS and is associated with a subsequent NNPFC SEI message, in decoding order, having nnpfc base flag equal to 0 and that particular nnpfc id value within the current CLVS, whichever is earlier.

[0157] The NNPFC SEI message comprises the nnpfc mode idc syntax element, the semantics of which may be defined as follows:

[0158] nnpfc mode idc equal to 1 specifies that the base post-processing filter or the update relative to the base post-processing filter associated with the nnpfc id value is a neuralnetwork identified by the Uniform Resource Identifier (URI) nnpfc uri with the format identified by the tag URI nnpfc tag uri.

[0159] nnpfc mode idc equal to 0 indicates that this SEI message contains an ISO / IEC 15938-17 bitstream that specifies the base post-processing filter or updates relative to the base postprocessing filter with the same nnpfc id value.

[0160] When nnpfc mode idc is equal to 0 and nnpfc base flag is equal to 0, the update may be obtained by decoding the coded neural network bitstream included in the NNPFC SEI message.

[0161] The NNPFC SEI message may also comprise:

[0162] i) Purpose of the post-processing filter, which may comprise, but may not be limited to, one or more of the following:

[0163] - Visual quality improvement;

[0164] - Chroma upsampling from the 4:2:0 chroma format to the 4:2:2 or 4:4:4 chroma format, or from the 4:2:2 chroma format to the 4:4:4 chroma format;

[0165] - Increasing the width or height of the input picture;

[0166] - Frame rate upsampling;

[0167] - Bit depth upsampling; or

[0168] - Colorization.

[0169] - Temporal extrapolation (i.e., generating one or more future pictures)

[0170] - Spatial extrapolation (i.e., extending one or more pictures, possibly additionally including cropping).

[0171] ii) Formatting of the input tensors that are given as input to the neural network inference.

[0172] iii) Formatting of the output tensors that are resulting from the neural network inference; and

[0173] iv) Characterization of the complexity of the neural network.

[0174] The NNPFC SEI message syntax includes the nnpfc_num_input_pics_minusl syntax element. nnpfc_num_input_pics_minusl plus 1 specifies the number of pictures used as input for the NNPF. The variable numlnputPics may be set equal to nnpfc_num_input_pics_minusl + 1.

[0175] When nnpfc_num_input_pics_minusl is greater than 0, the NNPFC SEI message syntax includes nnpfc_input_pic_filtering _flag[ i ] for each value of i in the range of 0 to nnpfc_num_input_pics_minusl, inclusive. nnpfc_input_pic_filtering_flag[ i ] equal to 1 indicates that for the i-th input picture the NNPF generates a corresponding output picture. nnpfc_input_pic_filtering_flag[ i ] equal to 0 indicates that for the i-th input picture the NNPF does not generate a corresponding output picture. Each NNPF-generated picture is stored in the output tensor of the NNPF. When nnpfc_num_input_pics_minusl is equal to 0, nnpfc_input_pic_filtering_flag

[0000] is inferred to be equal to 1.

[0176] A frame rate upsampling filter may interchangeably be called a picture rate upsampling filter. Such a filter generates or interpolates one or more pictures between a pair of pictures given as A frame rate upsampling filter may interchangeably be called a picture rate upsampling filter. Such a filter generates or interpolates one or more pictures between a pair of pictures given as input to the filter. It is also possible to have a frame rate upsampling filter where the number of input pictures may be greater than 2. Such a frame rate upsampling filter may generate pictures between more than one pair of input pictures. A frame rate upsampling filter may comprise a neural network, in which case the generation of the interpolated pictures between a pair of input pictures is performed by the inference of the neural network. It is possible to have a frame rate upsampling filter that extrapolates a picture before input picture(s) or after input picture(s), instead of or in addition to between input pictures.

[0177] When the filtering purpose comprises frame rate upsampling, the NNPFC SEI message includes nnpfc_interpolated_pics[ i ] syntax elements for the values of i in the range of 0, inclusive, to nnpfc_num_input_pics_minusl, exclusive. nnpfc_interpolated_pics[ i ] specifies the number of interpolated pictures generated by the NNPF between the i-th and the ( i + 1 )-th picture used as input for the NNPF.

[0178] When the filtering purpose comprises temporal extrapolation, the NNPFC SEI message includes the nnpfc_extrapolated_pics_minusl syntax element. nnpfc_extrapolated_pics_minusl plus 1 specifies the number of extrapolated pictures generated by the NNPF subsequent to all input pictures for the NNPF in output order.

[0179] When the filtering purpose comprises spatial extrapolation, a text prompt may be included in the NNPFC SEI message. When the input tensor for the NNPF is specified asinputTensor[ batchldx ][ inpPicIdx ][ compldx ][ y ][ x ] (when nnpfc component last flag is equal to 0) or inputTensor[ batchldx ][ inpPicIdx ] [ y ][ x ][ compldx ] (when nnpfc component last flag is equal to 1) and compldx is the component index (e.g., 0 for luma, 1 for Cb, and 2 for Cr), the text prompt may be provided to the NNPF inference by extending the range of compldx in the input tensor (e.g., 3 for text prompt). The text prompt may be included in the input tensor by including numerical value representations of the characters in the text prompt in a pre-defined scan order. The text prompt may comprise instructions or guidance which type of picture content is suggested to be included within the extrapolated area.

[0180] The NNPFC SEI message syntax may comprise an indication, which may be called nnpfc_absent_input_pic_zero_flag, that indicates how pictures that would not originate from the current bitstream are expected to be replaced in the input tensor. nnpfc_absent_input_pic_zero_flag equal to 1 (one) indicates that the NNPF expects an input picture that is not present in the current bitstream to be represented sample arrays with sample values equal to 0. nnpfc_absent_input_pic_flag equal to 0 (zero) indicates that the NNPF expects an input picture that is not present in the current bitstream to be represented by the closest input picture in output order within the current bitstream.

[0181] The NNPFC SEI message includes the nnpfc inp order idc syntax element, which indicates the method of ordering the sample arrays of an input picture to form an input tensor to the NNPF. The currently specified values of nnpfc inp order idc may be described as follows, while other values may be specified in the future. When nnpfc inp order idc is equal to 0, one luma matrix is present in the input tensor for each input picture. When nnpfc inp order idc is equal to 1, two chroma matrices are present in the input tensor. When nnpfc inp order idc is equal to 2, one luma and two chroma matrices are present in the input tensor. When nnpfc inp order idc is equal to 3, four luma matrices and two chroma matrices are present in the input tensor and the luma channels are derived in an interleaved manner, nnpfc inp order idc equal to 3 can only be used when the input chroma format is 4:2:0.

[0182] The NNPFC SEI message provides an extensible auxiliary input mechanism, which enables to input data other than the input pictures to the neural network inference. The NNPFC SEI message syntax may comprise an indication, which may be called nnpfc auxiliary inp idc, that indicates if auxiliary input data in addition to sample array(s) ofinput picture(s) is present in the input tensor of the NNPF. nnpfc auxiliary inp idc greater than 0 indicates that auxiliary input data is present in the input tensor of the NNPF. Specific semantics may be specified for specific non-zero values of nnpfc auxiliary inp idc. nnpfc auxiliary inp idc equal to 0 indicates that auxiliary input data is not present in the input tensor.

[0183] The value range of nnpfc auxiliary inp idc is specified to be up to 255, inclusive. In VSEI version 3, only one auxiliary input type, namely the QP-based StrengthControlVal[ i ], was defined, which is indicated by nnpfc auxiliary inp idc equal to 1.

[0184] The NNPFA SEI message specifies the neural-network post-processing filter (NNPF) that may be used for post-processing filtering for the current picture, or for postprocessing filtering for the current picture and one or more other pictures. The NNPFA SEI message comprises the nnpfa target id syntax element, which indicates that the neural-network post-processing filter with nnpfc id equal to nnpfa target id may be used for post-processing filtering for the indicated persistence. The indicated persistence may be the current picture only (indicated by nnpfa_persistence_flag equal to 0). Alternatively, the NNPF activation may be indicated to be persistent by nnpfa_persistence_flag equal to 1, in which case the persistence of the NNPF activation may last until the end of the current CLVS or the next picture, in output order, in the current layer associated with a NNPFA SEI message with the same nnpfa target id as the current SEI message.

[0185] The NNPFA SEI message syntax may comprise a syntax element indicative if the base post-processing filter or the latest post-processing filter is activated, where the latest post-processing filter is defined by the base post-processing filter relative to which the latest filter update, if any, has been applied. The syntax element may be called nnpfa target base flag. nnpfa target base flag equal to 1 specifies that the target NNPF is the base NNPF with nnpfc_id equal to nnpfa target id. nnpfa target base flag equal to 0 specifies that the target NNPF is the NNPF specified by the last NNPFC SEI message with nnpfc_id equal to nnpfa_target_id that precedes the first VCL NAL unit of the current picture in decoding order and is not a repetition of the NNPFC SEI message that contains the base NNPF.

[0186] The NNPFA SEI message syntax may comprise indications which ones of the filtered pictures corresponding to the input pictures are output by the NNPF process. For the i-thinput picture that is filtered by the NNPF, the NNPFA SEI message syntax may comprise nnpfa_output_flag[ i ] syntax element, which when equal to 0, specifies that the filtered picture is not output by the NNPF process, and when equal to 1 , specifies that the filtered picture is output by the NNPF process.

[0187] In relation to an NNPFA SEI message, two sets of pictures may be defined, namely nnpfcTargetPictures and nnpfaTargetPictures. nnpfcTargetPictures may be defined to be the set of pictures to which the last NNPFC SEI message with nnpfc id equal to nnpfa target id that precedes the current NNPFA SEI message in decoding order pertains. nnpfaTargetPictures may be defined to be the set of pictures for which the target NNPF is activated by the current NNPFA SEI message. It may be required for a conforming bitstream that any picture included in nnpfaTargetPictures shall also be included in nnpfcTargetPictures.

[0188] An NNPF process comprises performing the NNPF inference for given input pictures. The NNPF inference may be performed in a patch-wise manner so that the entire picture area gets filtered. The NNPF inference may be followed by outputting NNPF -generated pictures in their increasing index order, where all NNPF-generated pictures that were interpolated by the NNPF are output and those NNPF-generated pictures that correspond to any input pictures to the NNPF are output as specified in the semantics of the NNPFA SEI message.

[0189] A general post-processing filtering process using NNPFs may be described as follows. Input to this process is a bitstream BitstreamToFilter. Output of this process is a list of NNPF output pictures ListNnpfOutputPics. First, BitstreamToFilter is decoded, and the list CroppedDecodedPictures is set to be the list of the cropped decoded pictures in output order resulted from decoding BitstreamToFilter. Second, the filtering process for one picture, as described below, is repeatedly invoked, in output order, for each cropped decoded picture that is in CroppedDecodedPictures and for which one or more NNPFs are activated. The order of the pictures in ListNnpfOutputPics is in output order. It may be required that within ListNnpfOutputPics there shall be no more than one picture pertaining to any particular output time instance. When for any particular picture in CroppedDecodedPictures there are multiple NNPFs activated and only one the NNPFs is allowed to be chosen to be applied although any of the NNPFs may be chosen, the above constraint shall apply regardless of which NNPF is chosen to be applied to the particular picture.

[0190] A filtering process for one picture using an NNPF may be described as follows. The filtering process for one picture using an NNPF may be applied to each cropped decoded picture, referred to as the current picture, that is in CroppedDecodedPictures and for which one or more NNPFs are activated. When applying an NNPF to the current picture, the filtered and / or interpolated pictures are generated by the NNPF by applying the NNPF process to the current picture. When applying an NNPF to the current picture, the order of the pictures generated by the NNPF by applying the NNPF process being stored into the output tensor of the NNPF is in output order. When the applied NNPF is the last NNPF that is applied to the current picture, the pictures generated by the NNPF and output by the NNPF process are included into ListNnpfOutputPics, in the same order as when the pictures are stored into the output tensor of the NNPF.

[0191] The use of NNPFC and NNPFA SEI messages for WC has been described in version 3 of the versatile video coding (WC) standard. It is to be understood that NNPFC and NNPFA SEI message may be similarly used for any other video coding specification.

[0192] When NNPFC and NNPFA SEI messages are used for WC, a decoder selects input pictures for the NNPF. The input pictures may be selected in reverse output order starting from a picture for which the NNPF is activated through an NNPFA SEI message. The input pictures may be indexed, starting from index 0 that is assigned for the picture for which the NNPF is activated through an NNPFA SEI message. In an example, the decoder selects the input picture with index i, where i is greater than 0, to be the latest cropped decoded output picture, in output order, that precedes the input picture with index i-1 in output order. When there is no cropped decoded output picture, in output order, that precedes the input picture with index i-1 in output order as a result of decoding the bitstream, it may be considered that the input picture with index i is not present in the current bitstream (e.g., missing) and the subsequent input pictures, when any, with index i+1 to numlnputPics-l, inclusive, are likewise missing. A missing input picture may be treated like described above in relation to nnpfc_absent_input_pic_zero_flag syntax element.

[0193] When NNPFC and NNPFA SEI messages are used for WC and a picture rate upsampling NNPF that interpolates pictures between a single pair of input pictures is activated persistently until the end of the bitstream, the NNPF is applied repeatedly at the end of thebitstream for different sets of input pictures up to but excluding a set of input pictures that would cause creation of any interpolated picture after the last picture of the bitstream in output order. In these sets of input pictures, some of the pictures may be missing and may be, for example, replaced by the last picture within the bitstream in output order.

[0194] I.e.2. Neural-network in-loop filtering

[0195] More than one NN (neural network) may be available for performing decoding operation. When more than one NNs are available, an NN may need to be identified or addressed separately or explicitly (e.g., through IDs, identifications) so that a correct NN is used in a decoding operation.

[0196] The validity of a NN NAL (network abstraction layer) unit may be defined similarly to parameter sets. A NN NAL unit should be available to the decoder before being referenced.

[0197] The NN may be signaled as part of a parameter set. For example, a new APS (adaptation parameter set) type may be defined for a video coding standard such as WC. In other words, an NN NAL unit may be an adaptation parameter set NAL unit with an APS type indicating NN filtering. In an example embodiment, the following syntax may be used for an APS:

[0198] NN_APS may be a pre-defined constant value, e.g. equal to 3.

[0199] nn_data( ) may comprise an NNR bitstream.

[0200] nn_data( ) may comprise syntax and semantics like those of the NNPFC SEI message, although it should be noticed that the syntax structures within nn_data( ) do not necessarily comply with the SEI message syntax.

[0201] nn_data( ) may comprise syntax and semantics comprising a combination of two NNPFC SEI messages, one for the base filter and another for a filter update, although it may be noted that the syntax structures within nn_data( ) do not necessarily comply with the SEI message syntax. The nn_data( ) may comprise one or both of a reference or an identifier, such as a URI, that identifies a (de-facto) standard neural network or a separately provided neural network, and an update of the identified neural network.

[0202] I.f Spatial resolution and resampling

[0203] Spatial resolution of an image or a picture may be defined as the number of pixels or samples representing the image / picture in horizontal and vertical direction. In this document, expressions such as “images at different resolution” may be interpreted as two images have different number of pixels either in horizontal direction, or in vertical direction, or in both directions.

[0204] In signal processing, resampling of images is usually understood as changing the sampling rate of the current image in horizontal or / and vertical directions. Resampling results in a new image which is represented with different number of pixels in horizontal or / and vertical direction. In some applications, the process of image resampling is equal to image resizing. In general, resampling is classified in two processes: downsampling and upsampling.

[0205] Downsampling or subsampling process may be defined as reducing the sampling rate of a signal, and it typically results in reducing of the image sizes in horizontal and / or vertical directions. In image downsampling, the spatial resolution of the output image, i.e. the number of pixels in the output image, is reduced compared to the spatial resolution of the input image. Downsampling ratio may be defined as the horizontal or vertical resolution of the downsampled image divided by the respective resolution of the input image for downsampling. Downsampling ratio may alternatively be defined as the number of samples in the downsampled image divided by the number of samples in the input image for downsampling. As the two definitions differ, the term downsampling ratio may be further characterized by indicating whether it is indicated along one coordinate axis or both coordinate axes (and hence as a ratio of number of pixels in the images). Image downsampling may be performed for example by decimation, i.e. by selecting a specific number of pixels, based on the downsampling ratio, out of the total number of pixels in the original image. In some embodiments, downsampling may include low-pass filtering or other filtering operations, which may be performed before or after image decimation. Any low-pass filtering method may be used, including but not limited to linear averaging.

[0206] An upsampling process may be defined as increasing the sampling rate of the signal, and it typically results in increasing of the image sizes in horizontal and / or vertical directions. In image upsampling, the spatial resolution of the output image, i.e. the number of pixels in the output image, is increased compared to the spatial resolution of the input image. Upsampling ratio may be defined as the horizontal or vertical resolution of the upsampled image divided by the respective resolution of the input image. Upsampling ratio may alternatively be defined as the number of samples in the upsampled image divided by the number of samples in the input image. As the two definitions differ, the term upsampling ratio may be further characterized by indicating whether it is indicated along one coordinate axis or both coordinate axes (and hence as a ratio of number of pixels in the images). Image upsampling may be performed for example by copying or interpolating pixel values such that the total number of pixels is increased. In some cases, upsampling may include filtering operations, such as edge enhancement filtering.

[0207] I.g. Stereoscopic 3D video

[0208] Stereoscopic video content includes pairs of offset images that are shown separately to the left and right eye of the viewer. These offset images may be captured with a specific stereoscopic camera setup. A stereo baseline distance between cameras may be selected in a manner that it approximately matches the inter-pupillary distance.

[0209] Binocular rivalry affords a unique opportunity to discover aspects of perceptual processing that transpire outside of visual awareness. In stereoscopic presentation, the brain registers slight perspective differences between left and right views to create a stable, three-dimensional representation incorporating both views. In other words, the visual cortex receives information from each eye and combines this information to form a single stereoscopic image. Left- and right-eye image differences along any one of a wide range of stimulus dimensions are sufficient to instigate binocular rivalry. These include differences in color, luminance, contrast polarity, form, spatial resolution, or velocity. Rivalry can be triggered by very simple stimulus differences or by differences between complex images. Stronger, high- contrast stimuli lead to stronger perceptual competition. Rivalry can even occur under dim viewing conditions, when light levels are so low they can only be detected by the retina's rod photoreceptors. Under some conditions, rivalry can be triggered by physically identical stimuli that differ in appearance owing to simultaneous luminance or color contrast. If both views are provided with equal quality, the perceived quality of stereoscopic image is proportional to the quality of both views. On the other hand, if the quality or other factors of left and right view differs, the view with the better visual quality may dominate in the quality perceived by the human visual system (HVS). In other words, the perceived quality may be close to the visual quality of the view with the better visual quality.

[0210] One branch of research for obtaining compression improvement in stereoscopic video is known as asymmetric stereoscopic video coding. Asymmetric stereoscopic video coding is based on the above-described assumption of the binocular suppression theory that the Human Visual System (HVS) fuses the stereoscopic image pair such that the perceived quality is close to that of the higher quality view. Thus, compression improvement is obtained by providing a quality difference between the two coded views.

[0211] Asymmetry between the two views can be achieved, for example, by one or more of the following methods:

[0212] a) Mixed-resolution (MR) stereoscopic video coding, also referred to as resolution-asymmetric stereoscopic video coding. One of the views is low-pass filtered and hence has a smaller number of spatial details or a lower spatial resolution as compared to without low-pass filtering. Furthermore, the low-pass filtered view is usually sampled with a coarser sampling grid, i.e., represented by fewer pixels.

[0213] b) Mixed-resolution chroma sampling. The chroma pictures of one view are represented by fewer samples than the respective chroma pictures of the other view.

[0214] c) Asymmetric sample-domain quantization. The sample values of the two views are quantized with a different step size. For example, the luma samples of one view may be represented with the range of 0 to 255 (i.e., 8 bits per sample) while the range may be scaled to the range of 0 to 159 for the second view. Thanks to fewer quantization steps, the second view can be compressed with a higher ratio compared to the first view. Different quantization step sizes may be used for luma and chroma samples. As a special case of asymmetric sample-domain quantization, one can refer to bit-depth-asymmetric stereoscopic video when the number of quantization steps in each view matches a power of two.

[0215] d) Asymmetric transform-domain quantization. The transform coefficients of the two views are quantized with a different step size. As a result, one of the views has a lower fidelity and may be subject to a greater amount of visible coding artifacts, such as blocking and ringing.

[0216] e) Mixed temporal resolution (i.e., different picture rate) between views.

[0217] f) A combination of different encoding techniques above.

[0218] In Mixed Resolution (MR) stereoscopic images, one of the views, i.e., either the left view or the right view, is provided with lower resolution, typically through downsampling, and consequently with a different quality. Obviously, for the final presentation of the stereoscopic image lower resolution view should be upsampled to the display resolution e.g. if display has resolution of W for width and H for height (W*H) an arbitrary view can have resolutions aW*bH where a and b are coefficients smaller or equal to one, defining the resolution of downsampled view compared to the display resolution.

[0219] The use of mixed-resolution stereoscopic video has been proposed in ATSC A / 345:2024-03 Candidate Standard for WC video. For instance a HMD (head-mounted display) using multilayer WC may be used, see Figure B.5 of the ATSC A / 345. Such an HMD may include an ultra-high definition (UHD, e.g. 3840x2160) left view and full high definition (FHD, e.g. 1920x1080) right view, which is upsampled to UHD resolution subsequent to decoding. The UHD right video is down-sampled into FHD, and is applied to a multilayer WC along with the UHD left video. The FHD is output as the FHD right view, considered to be a base layer, and upsampling is performed on the FHD and output as part of the stereoscopic UHD 3D (three- dimensional) output, which also contains the UHD left view (e.g., which has an enhancement layer plus the base layer).

[0220] I.h. Types of mask / map images / video

[0221] I .h.l Overview

[0222] The terms mask image, mask information image, and map image are used interchangeably herein. Uikewise, terms mask video, mask information video, and map video are used interchangeably herein. These terms may be prefixed by the type of the mask or map.

[0223] Types of mask information video may include but are not limited to:

[0224] i) Object mask.

[0225] ii) Region of Interest (ROI) mask.

[0226] iii) Saliency map.

[0227] iv) Alpha mask (also known as alpha plane).

[0228] v) Occupancy mask.

[0229] vi) Filtering mask.

[0230] vii) Depth map.

[0231] viii) Gain map.

[0232] ix) Attenuation map.

[0233] I. h.2 Obj ect and ROI masks

[0234] An object mask may identify one or more objects of a picture in a pixel- accurate manner. In other words, an object mask may map pixel locations to identified objects. An object mask may result from an instance segmentation or object detection algorithm.

[0235] Sending object masks as video data allows results of video analysis performed by the encoder to be sent to the decoder. This could reduce the workload and power consumption of the decoder.

[0236] In some cases, an object mask may be a binary matrix or image, wherein a value, such as “0”, may represent background and another value, such as “1”, may represent the foreground.

[0237] In some cases, an object mask may be a monochrome (grayscale) image. An object mask may be represented with a luma sample array, having bit-depth equal to BitDepthY. Thus, there are l«BitDepthY different values for each luma sample. To distinguish different masks within one picture, a selected sample value or a selected sample value range may represent a single object of the object mask.

[0238] As masks can be overlapped, multiple object mask pictures can used for one primary picture to handle overlapping case. In that case, the samples with same position but in the different mask picture could belong to different masks overlapped with each other.

[0239] Other terms that may be used similarly to object include annotated region and region of interest (ROI). An annotated region or an ROI does not necessarily represent a single object but may be more generally defined to identify region(s) having some annotation and / or of different interest or importance. A region-of-interest mask may indicate one or more regions of interest in an associated image. A region of interest may be perceptually more important to human viewers or may be more important to improve a computer vision task accuracy when compared to areas outside of the regions of interest. Annotated region and / or ROI masks may be represented similarly to object masks.

[0240] I.h.3 Saliency map

[0241] A saliency map may be defined as an image that highlights the region(s) on which viewers are likely to focus. Alternatively, a saliency may reflect the likely most relevant regions for computer vision tasks. Sample values of a saliency map may correspond to a degree of importance of the samples to the human visual system or to computer vision task(s). A saliency map may be a continuous tone image.

[0242] I.h.4. Alpha mask

[0243] An alpha mask may be used to provide transparency information for an associated image. A first value of an alpha mask may represent a fully opaque pixel, and a second value may represent a fully transparent pixel. Values between the first and second values may represent different levels of transparency between fully opaque and fully transparent.

[0244] I. h.5. Occupancy mask

[0245] An occupancy mask may be used in patch-based volumetric video coding to indicate which sample locations of an associated texture image (a.k.a. texture atlas) are occupied by pixel values to be used in volumetric video reconstruction and which sample locations are unoccupied, i.e., should not be used in volumetric video reconstruction. A first value of an occupancy mask may indicate an occupied sample location, and a second value may indicate an unoccupied sample location.

[0246] I.h.6. Filtering mask

[0247] A filtering mask may be used to indicate the areas that are subject to filtering and / or are not subject to filtering, and / or a filtering strength. The type of filtering may be, but may not be limited to, one or more of the following: film grain noise synthesis, adaptive loop filtering (ALF), post-filtering. A first value of a filtering mask may indicate that no filtering is applied for the collocated pixel in an associated image, and a second value may indicate that filtering is applied for the collocated pixel in an associated image. Values between the first and second values may represent different levels of filtering strength between no filtering and filtering at full strength.

[0248] I.h.7. Depth map

[0249] A texture view refers to a view that represents ordinary video content, for example has been captured using an ordinary camera, and is usually suitable for rendering on a display. A texture view typically comprises pictures having three components, one luma component and two chroma components.

[0250] A depth view refers to a view that represents distance information of a texture sample from the camera sensor, disparity or parallax information between a texture sample and a respective texture sample in another view, or similar information. A depth view may comprise depth pictures (a.k.a. depth maps) having one component or sample array. A depth map is animage with per-pixel depth information or similar. For example, each sample in a depth map represents the distance of the respective texture sample or samples from the plane on which the camera resides. In other words, if the z axis is along the shooting axis of the cameras (and hence orthogonal to the plane on which the cameras he), a sample in a depth map represents the value on the z axis. The semantics of depth map values may for example include the following:

[0251] i) Each luma sample value in a depth map represents an inverse of real- world distance (Z) value, i.e., 1 / Z, normalized in the dynamic range of the luma samples, such to the range of 0 to 255, inclusive, for 8-bit luma representation. The normalization may be done in a manner where the quantization 1 / Z is uniform in terms of disparity.

[0252] ii) Each luma sample value in a depth map represents an inverse of real-world distance (Z) value, i.e., 1 / Z, which is mapped to the dynamic range of the luma samples, such to the range of 0 to 255, inclusive, for 8-bit luma representation, using a mapping function f(l / Z) or table, such as a piece-wise linear mapping. In other words, depth map values result in applying the function f(l / Z).

[0253] iii) Each luma sample value in a depth map represents a real-world distance (Z) value normalized in the dynamic range of the luma samples, such to the range of 0 to 255, inclusive, for 8-bit luma representation.

[0254] iv) Each luma sample value in a depth map represents a disparity or parallax value from the present depth view to another indicated or derived depth view or view position.

[0255] Phrases such as depth view, depth view component, depth picture and depth map may be used interchangeably herein. It is to be understood that any semantics of depth map values may be used in various embodiments including but not limited to the ones described above. For example, embodiments herein may be applied for depth pictures where sample values indicate disparity values.

[0256] Depth-enhanced video refers to texture video having one or more views associated with depth video having one or more depth views. A number of approaches may be used for representing depth-enhanced video, including the use of video plus depth (V+D), multiview video plus depth (MVD), and layered depth video (LDV). In the video plus depth (V+D) representation, a single view of texture and the respective view of depth are represented as sequences of texture picture and depth pictures, respectively. The MVD representationcontains a number of texture views and respective depth views. In the LDV representation, the texture and depth of the central view are represented conventionally, while the texture and depth of the other views are partially represented and cover only the dis-occluded areas required for correct view synthesis of intermediate views.

[0257] I.h.8 Gain map

[0258] A gain map may be used for dynamic range conversion, for example from a standard dynamic range (SDR) image to a high dynamic range (HDR) image, or vice versa. A gain map may assist in displaying an HDR image / video on an SDR display, or an SDR image / video on an HDR display. A gain map includes per-pixel information for the dynamic range conversion from an image of a "base" dynamic range to an image of a "destination" dynamic range.

[0259] Terms tone mapping and dynamic range conversion may be used interchangeably.

[0260] I.h.9. Attenuation map

[0261] An attenuation map may be used for per-pixel brightness scaling to achieve energy saving on a display device. Sample values of an attenuation map may represent the amount or magnitude of brightness scaling. An attenuation map may be applied as postprocessing after decoding to scale the brightness of an image on a pixel-adaptive manner. The perceived quality of an attenuated image may be indistinguishable from or close to the perceived quality of the respective decoded image.

[0262] Li. Some SEI message information

[0263] Li l Scalable nesting SEI message

[0264] A scalable nesting SEI message contains one or more SEI messages. The SEI messages included in the scalable nesting SEI message are also referred to as the scalable-nested SEI messages. A scalable nesting SEI message comprises information indicative of which subset of the bitstream the scalable-nested SEI messages apply. For example, WC includes a scalable nesting (SN) SEI message, which provides information to associate scalable-nested SEI messages with specific output layer sets (OLSs), specific layers, or specific sets of subpictures. Thus, the scalable nesting SEI message may provide the capability to specify that one or moreSEI messages applies to one or more output layer sets, one or more layers, and / or one or more subpictures.

[0265] The variable NestedLayers may be derived from the scalable nesting SEI message to indicate the layers that the scalable nesting SEI message applies to. The variable NestingNumLayers may be derived to be equal to the count of layers to which the scalable nesting SEI message applies to. The array variable NestingLayerld may be derived to include the layer identifier values of the layers that the scalable nesting SEI message applies to.

[0266] Some of the syntax elements of the scalable nesting SEI message of WC are described below.

[0267] sn ols flag equal to 1 specifies that the scalable-nested SEI messages apply to specific OLSs. sn ols flag equal to 0 specifies that the scalable-nested SEI messages apply to specific layers.

[0268] sn subpic flag equal to 1 specifies that the scalable-nested SEI messages that apply to specified OLSs or layers apply only to specific subpictures of the specified OLSs or layers, sn subpic flag equal to 0 specifies that the scalable-nested SEI messages that apply to specific OLSs or layers apply to all subpictures of the specified OLSs or layers.

[0269] sn num olss minusl plus 1 specifies the number of OLSs to which the scalable-nested SEI messages apply. The value of sn num olss minusl shall be in the range of 0 to TotalNumOlss - 1, inclusive, where TotalNumOlss specifies the total number of OLSs derived from the VPS if a VPS is present and is equal to 1 otherwise.

[0270] 1.1.2 Constituent rectangle and constituent rectangle nesting SEI messages

[0271] The constituent rectangle (CR) SEI message has been proposed, e.g., in JVET-AH0162. It enables composition of multiple constituent rectangles within a coded picture and provides information about the rectangles, including ID, type, text description, location, and size.

[0272] The constituent rectangle nesting SEI message was proposed, e.g., in JVET- AI0177. The CR nesting SEI message includes one or more SEI messages, which may be referred to as CR-nested SEI messages. The CR nesting SEI message specifies that the CR- nested SEI messages apply to one or more constituent rectangles.

[0273] The proposed functionality would be useful, for example, if a coded picture contained multiple alpha-channel constituent rectangles along with texture constituent rectangles, by enabling an alpha channel info SEI message to be applied to the multiple alpha channel constituent rectangles.

[0274] The CR nesting SEI message included crn_rect_id[ i ], which indicates the identifier of the i-th constituent rectangle. The CR nesting SEI message may be included in a scalable nesting SEI message, in which case the crn_rect_id[ i ] values may indicate the constituent rectangles in each picture in the NestedLayers to which the CR-nested SEI messages apply.

[0275] 1.1.3 SEI processing order and processing order nesting SEI messages

[0276] Standardization is ongoing for specifying the SEI processing order (SPO) SEI message and the processing order nesting (PON) SEI message.

[0277] The SEI processing order (SPO) SEI message carries information indicating the preferred processing order, as determined by the encoder (i.e., the content producer), for a group of types of SEI messages that may be present in a CVS (coded video sequence).

[0278] A processing chain comprises a list of types of SEI messages identified by an SPO SEI message in the preferred processing order indicated in the SPO SEI message.

[0279] The processing order nesting (PON) SEI message includes one or more SEI messages that should be applied only as parts of the processing chain identified by an associated SEI processing order SEI message and should not be applied in a manner that would contradict with the processing chain identified by the associated SEI processing order SEI message.

[0280] I.j Reconstruction guided in-loop filters for video coding

[0281] One method for reconstruction guided in-loop filters for video coding comprises obtaining a video signal comprising blocks of video frames, there the blocks comprises a plurality of pixels; obtaining information indicative about reconstruction or visibility of said pixels in said blocks upon decoding; and disabling, based on said information indicative about reconstruction or visibility of said pixels in said blocks, in-loop filtering for such blocks having only a subset of the pixels to be reconstructed or visible upon decoding.

[0282] Thus, the method enables to remove at least a part of distortion and increase the coding efficiency through reduced bitrate by activating and deactivating some or all videocodec in-loop filters, based on a secondary input indicating information about reconstruction of said pixels in said blocks upon decoding, e.g., an occupancy signal. Based on this information, the encoder may disable in-loop filtering for such blocks having only a subset of the pixels to be reconstructed or visible upon decoding.

[0283] Another aspect relates to the operation of a decoder (or a Tenderer / receiver / player / client). The method comprises receiving a bitstream comprising an encoded video signal comprising blocks of video frames, said blocks comprising a plurality of pixels; receiving, in or along said bitstream, information indicative about reconstruction or visibility of said pixels in said blocks; decoding the an encoded video signal according to at least said information indicative about reconstruction or visibility of said pixels in said blocks by disabling in-loop filtering for such blocks having only a subset of the pixels to be reconstructed or visible.

[0284] Hence, the decoding apparatus receives a bitstream, containing reconstruction guided in-loop filter signaling as disclosed above, and comprising encoded video data, and in or along the bitstream the reconstruction information. The reconstruction information may be provided in the form of a bitstream comprising an encoded occupancy or alpha map video, either as individual video bitstreams or as layers in a multi-layer encoding, or a reconstruction signal, multiplexed in the same or yet another video, e.g., as one or more sample values within a specific value range of a depth map. If necessary, the decoder first decodes the reconstruction information, then the decoder decodes the video data, activating / disabling in-loop filters according to said information indicative about reconstruction or visibility of said pixels in said blocks, i.e., reconstruction information state (full, none, partial) and the received signaling.

[0285] For a neural-network post-processing filter, an input tensor may be formed. In an embodiment, it may be indicated by an encoder, e.g., in a neural-network post-filter characteristics (NNPFC) SEI message, that the post-filter expects auxiliary input in the input tensor for the reconstruction information and / or variables controlling the reconstruction information guided filtering, such as one or more threshold values defining the sample value range in the reconstruction information that indicates a non-visible pixel for filtering. In an embodiment, it may be decoded by a decoder, e.g., from a NNPFC SEI message, that the postfilter expects auxiliary input in the input tensor for the reconstruction information and / or variables controlling the reconstruction information guided filtering. Accordingly, the decoderforms an input tensor with reconstruction information and / or variables controlling the reconstruction information guided filtering.

[0286] II. Examples of problems

[0287] Now that technical context has been provided, problems in these technical areas are described. The neural-network post-filter (NNPF) design in VSEI has not considered filtering where input pictures originate from multiple scalability layers. Consequently, the filtering purposes are limited, and filtering is not guided by information available from other scalability layers. See below (Section IV.1) for application use cases enabled by input pictures from multiple scalability layers.

[0288] The NNPF design in VSEI has not considered filtering where input pictures are extracted from rectangular regions, such as constituent rectangles. Constituent rectangles may be used similarly to scalability layers for packing pictures of different modalities or different views onto the same coded picture.

[0289] Auxiliary input in the input tensor for the reconstruction information and / or variables controlling the reconstruction information guided filtering may be used for NNPF but there are no known methods to identify auxiliary inputs and to insert them in the input tensor.

[0290] One may use occupancy or alpha for guided filtering, such as disabling filtering on pixel basis. This, however, does not address many of the use cases in Section IV.1 below.

[0291] III. Overview of Examples

[0292] These problems are addressed herein. An overview is provided, then additional information is provided.

[0293] Referring to FIG. 4, this figure is a flow diagram of a decoding method 141 in an example. The decoding method 141 is assumed to be performed by the decoder 140 in the receiving apparatus 180-2. Decoding, by the decoder 140, according to an example herein comprises the following:

[0294] 1) Decoding one or more indications of a first constituent frame and a second constituent frame to be used as input for a neural network inference, wherein the first and second constituent frames are rectangular regions or pictures of different scalability layers. Thedecoding creates a first reconstructed constituent frame from the first constituent frame and a second reconstructed constituent frame from the second constituent frame. See block 410.

[0295] 2) Providing the first reconstructed constituent frame and the second reconstructed constituent frame as inputs to the neural network inference. See block 420.

[0296] 3) Performing the neural network inference to enhance the first reconstructed constituent frame to an enhanced first constituent frame. See block 430.

[0297] Turning to FIG. 5, this figure is a flow diagram of an encoding method 131 in an example. The encoding method 131 is assumed to be performed by the encoder 130 in the transmitting apparatus 180-1. Encoding, by the encoder 130, according to an example herein comprises:

[0298] 1) Encoding one or more indications of a first constituent frame and a second constituent frame to be used as input for a neural network inference, wherein the first and second constituent frames are rectangular regions or pictures of different scalability layers. See block 510.

[0299] 2) outputting the encoded one or more indications. See block 520. The output would be, e.g., to the bitstream 101 and toward the decoder 140.

[0300] One typical implementation is to put the FHD and UHD pictures in different layers. They could, however, be frame-packed as rectangular regions to the same frame. The rectangular regions are considered in the neural network inference as if the rectangular regions were standalone frames.

[0301] IV. Further details

[0302] IV. a Applications

[0303] Some non-limiting embodiments of applications or usages are presented in this subsection. The applications presented here provide corresponding technical effects and advantages.

[0304] IV.a.l Inter-view super-resolution

[0305] In an embodiment, examples are applied to asymmetric stereoscopic video, such as mixed-resolution stereoscopic video. A neural network inputs pictures from both views and enhances the lower-quality view. For example, an upsampling or super-resolution neural network may increase the resolution of a low-resolution (LR) view of a mixed-resolutionstereoscopic video as illustrated in FIG. 6, which is a block diagram for an inter-view superresolution neural network. This is an example of cross-domain NN filtering 350 and is indicated as cross-domain NN filtering 350-1. This example uses two views, view 0 610-0, and view 1 610-1. View 1 610-1 is a LR view, and view 0 610-0 is a full resolution (FR) view. Reconstruction / decoding 620 has been performed to form the views, view 0 610-0 and view 1 610-1. The reconstruct! on / decoding 620 is meant to be the entire decoding process, although multiple options for the same are described below. The views 610-0 and 610-1 can be, in one example, considered to be outputs of the filtering performed by block 308 of FIG. 3 and part of the R’n that form the output video (e.g., stream) 110-2. Other examples are described below. Processing is performed on the LR view, view 1 610-1, and the view 0 610-0 by the inter-view super-resolution NN (neural network) 630 to form view 1 640 that has full resolution (FR), which may be equal to the resolution of view 0 610-0. This is an example where a first constituent frame, e.g., from view 0 610-0, and a second constituent frame, from view 1 610-1, are to be used as input for a neural network inference (e.g., reference 630), and the first and second constituent frames can include pictures of different scalability layers, which are the FR (e.g., UHD) and LR (e.g., FHD) pictures based on different resolutions.

[0306] As described above, the use of mixed-resolution stereoscopic video has been proposed in ATSC A / 345:2024-03 Candidate Standard for WC video. View 0 610-1 could be (see block 650) an ultra high definition (UHD, e.g., 3840x2160) left view (as video / picture) and the view 1 610-1 could be (see block 660) a full high definition (FHD, e.g., 1920x1080) right view (as video / picture), which is upsampled to UHD resolution as an enhanced FHD right view (as video / picture), see block 670. It is noted that the terms “left” and “right” may be switched, e.g., the UHD could be “right”, and the FHD could be “left”.

[0307] These are additional examples of the view 610-0 and 610-1. In an embodiment, view 0 610-1 is encoded as the base layer 200 and / or decoded from the base layer 300, and view 1 610-1 is encoded as a dependent layer 200-1 and / or decoded from a dependent layer 300-1, wherein the dependent layer 300-1 is predicted from the base layer 300.

[0308] In an embodiment, view 1 610-1 is encoded as the base layer 200 and / or decoded from the base layer 300-1, and view 0 610-0 is encoded as a dependent layer 200-1and / or decoded from a dependent layer 300-1, wherein the dependent layer 300-1 is predicted from the base layer 300.

[0309] In an embodiment, view 0 610-0 is encoded and / or decoded without dependencies from view 1 610-1 and view 1 610-1 is encoded and / or decoded without dependencies from view 0 610-0. For example, constituent frames of a frame-packed picture may comprise coinciding pictures of view 0 and view 1.

[0310] FIG. 6 shows two views. Examples herein may not be limited to two views but instead may apply to any number of views.

[0311] IV. a.2 Mask- or map-guided filtering

[0312] In an embodiment, an example is applied to a scheme where a texture image / video is accompanied by at least one mask or map image / video. As described previously, mask video may also refer to map video. Refer to FIG. 7, which is a block diagram for mask- or map-guided filtering. This is an example of cross-domain NN filtering 350 and is indicated as cross-domain NN filtering 350-2. This example involves reconstruction / decoding 720 that forms a mask / map picture 710 and a reconstructed picture 711. The mask / map picture 710 and reconstructed picture 711 are processed by NN filtering 730 to form an enhanced reconstructed picture 740. The neural network filtering 730 inputs pictures from both the texture video (e.g., the reconstructed picture 711) and the mask or map video (e.g., mask / map picture 710) and enhances the texture video. The mask video (also referred to as map video) may be of any type, including but not limited to those presented earlier, which included the following: i) Object mask; ii) Region of Interest (ROI) mask; iii) Saliency map; iv) Alpha mask (also known as alpha plane); v) Occupancy mask; vi) Filtering mask; vii) Depth map; viii) Gain map; ix) Attenuation map. In an example, the neural network filtering 730 may have been trained in a manner that filtering takes place only within object boundaries. In another example, the neural network filtering 730 may have been trained in a manner that filtering takes place only outside object boundaries for the background. In yet another example, the filtering strength may be controlled by a map image. This is an example where a first constituent frame, e.g., from reconstructed picture 711, and a second constituent frame, from mask / map picture 710, are to be used as input for a neural network inference (e.g., reference 730), and the first and second constituent framescan include pictures of different scalability layers, which are the pictures 710 and 711 pictures based on different resolutions.

[0313] Other examples are as follows. In an embodiment, a first scalability layer of a bitstream 101 comprises a first coded picture that may be decoded to the reconstructed picture 711, and a second scalability layer of the bitstream 101 comprises a second coded picture that may be decoded to the mask / map picture 710.

[0314] In an embodiment, a decoded picture comprises both the mask / map picture 710 and the reconstructed picture 711. An encoding system packs a texture image and a mask / map image spatially into a source picture and encodes the source picture into a coded picture. A decoding system decodes the coded picture into a decoded picture and extracts the mask / map picture 710 and the reconstructed (texture) picture 711 from the decoded picture.

[0315] IV. a.3 Depth from stereoscopic video

[0316] In an embodiment, an example is applied to stereoscopic video. A neural network inputs pictures from both views and produces one or two depth views, which may represent depth or disparity. In the example of FIG. 6, The inputs to the NN 630 would be as in FIG. 6, but can be of the same or different spatial resolution. The output of the NN 630 would be depth view 0 and / or depth view 1.

[0317] IV.a.4 View synthesis

[0318] In an embodiment, an example is applied to synthesize a view. A neural network inputs one or more texture views and one or more depth views and generates a texture view of another viewpoint. The neural network may also generate a depth view of that viewpoint.

[0319] IV. a.5 Bokeh effect filtering

[0320] In an embodiment, an example is applied to introduce bokeh effect with a neural network that takes a texture picture and a respective depth picture as inputs and generates a texture picture with bokeh effect.

[0321] IV. a.6 Mask-driven inpainting

[0322] Inpainting may be defined as a process to fill in indicted or missing part(s) of an image in a manner that looks indistinguishable from other parts of the image. Inpainting may be used for object removal or object replacement. For example, an a person may be removedfrom an image and replaced by background that could appear as if the person were not present in the image.

[0323] In an embodiment, an example is applied to a scheme where a texture image / video is accompanied by at least one mask image / video. This embodiment is similar to the mask- or map-guided filtering described in IV. a.2. The block diagram of FIG. 7 applies to this embodiment too. This is an example of cross-domain NN filtering 350 and is indicated as crossdomain NN filtering 350-2. This example involves reconstruct! on / decoding 720 that forms a mask picture 710 and a reconstructed picture 711. The mask picture 710 and reconstructed picture 711 are processed by NN filtering 730, wherein the NN filtering 730 applies the mask to select areas or objects of the reconstructed picture 711 that are removed and inpainted by the NN filtering 730 to form an enhanced reconstructed picture 740. The neural network filtering 730 inputs pictures from both the texture video (e.g., the reconstructed picture 711) and the mask video (e.g., mask picture 710) and inpaints areas or objects of the texture video as indicated by the mask video. The mask video may be of any type, including but not limited to those presented earlier, such as: i) Object mask; ii) Filtering mask. This is an example where a first constituent frame, e.g., from reconstructed picture 711, and a second constituent frame, from mask picture 710, are to be used as input for a neural network inference (e.g., reference 730), and the first and second constituent frames can include pictures of different scalability layers, which are the pictures 710 and 711 pictures, which may have the same or different resolutions.

[0324] In an embodiment, a first scalability layer of a bitstream comprises a first coded picture that may be decoded to the reconstructed picture 711, and a second scalability layer of the bitstream comprises a second coded picture that may be decoded to the mask picture 710.

[0325] In an embodiment, a decoded picture comprises both the mask picture 710 and the reconstructed picture 711. An encoding system packs a texture image and a mask image spatially into a source picture and encodes the source picture into a coded picture. A decoding system decodes the coded picture into a decoded picture and extracts the mask picture 710 and the reconstructed (texture) picture 711 from the decoded picture.

[0326] In an embodiment, an indication of an inpainting purpose may be defined for a description of the NN filter. For example, a bit of the nnpfc_purpose value may be defined toindicate inpainting. In embodiments described under IV.a.6., an encoder may indicate the inpainting purpose, e.g. in the value of nnpfc_purpose, for the NN filtering 730 to be performed by the decoder. In embodiment described under IV.a.6, a decoder may decode an indication of the inpainting purpose, e.g. from the value of nnpfc_purpose.

[0327] In an embodiment, an inpainting purpose may be accompanied by a text prompt similarly to the spatial extrapolation purpose, e.g. as described in relation to the NNPFC SEI message. The text prompt may affect the inpainting performed by the NN filtering 730. For example, a text prompt "inpaint with background" would replace the removed areas with content similar to the background present in the reconstructed picture 711. In another example, a text prompot "anonymize people within the mask" would replace the people indicated by the mask picture 710 with anonymized people.

[0328] IV. b Packing and indication of constituent frames

[0329] A constituent frame may be defined as the following

[0330] 1. a rectangular region within a frame that contains one or more other constituent frames; or

[0331] 2. a frame within two or more adjacent frames, in output order, that correspond to the same display timestamp; or

[0332] 3. a first frame in a first scalability layer.

[0333] Following the same numbering as above, constituent frames may be associated with each other in one or more of the following ways:

[0334] 1. all constituent frames within the same frame may be associated with each other; or

[0335] 2. all adjacent frames, in output order, that correspond to the same display timestamp may be associated with each other; or

[0336] 3. all frames of the same access at different scalability layers may be associated with each other.

[0337] Constituent frames that are associated with each other may be used as inputs to the neural network inference according to embodiments of this invention.

[0338] A constituent frame that is a rectangular region within a frame may be identified by a constituent rectangles SEI message or alike. Characteristics of a constituentrectangle may be indicated by one or more SEI messages included in a constituent rectangle nesting SEI message or alike.

[0339] In an embodiment, an encoder encodes, into or along a bitstream, one or more indications of constituent frames used as input for a neural network inference.

[0340] In an embodiment, a decoder decodes, from or along a bitstream, one or more indications of constituent frames used as input for a neural network inference. The decoder performs a neural network inference with the inputs decoded from the one or more indications.

[0341] In an embodiment, an encoder encodes a nesting SEI message that includes an NNPFC SEI message and / or an NNPFA SEI message, and / or a decoder decodes a nesting SEI message that includes an NNPFC SEI message and / or an NNPFA SEI message. The nesting SEI message indicates the scalability layers and / or regions that comprise the constituent frames used as input to the neural network inference. The messages include the SEI, such as including NNPFC SEI and / or NNPFA SEI . The nesting SEI message may be, but may not be limited to, any of the following:

[0342] 1. The scalable nesting SEI message, e.g., as specified in VVC. The scalable nesting SEI message may indicate the scalability layers and / or subpictures used as input to the neural network inference. When subpictures are indicated, the subpicture indicated by the scalable nesting SEI message may be extracted from the cropped decoded picture and treated as input picture for the NNPF inference.

[0343] 2. The constituent rectangle nesting SEI message as described earlier. The constituent rectangle nesting SEI message indicates the constituent rectangles that comprise the constituent frames used as input to the neural network inference.

[0344] In an embodiment, an encoder indicates in a first nesting SEI message (e.g., a scalable nesting SEI message) the scalability layers comprising rectangular region(s) used as constituent frames, and in a second nesting SEI message (e.g., a constituent rectangle nesting SEI message) included in the first nesting SEI message the rectangular region(s) used as constituent frame(s), and includes an NNPFC SEI message and / or an NNPFA SEI message in the second nesting SEI message. In an embodiment, a decoder decodes from a first nesting SEI message (e.g., a scalable nesting SEI message) the scalability layers comprising rectangular region(s) used as constituent frames, and from a second nesting SEI message (e.g., a constituent rectanglenesting SEI message) included in the first nesting SEI message the rectangular region(s) used as constituent frame(s), and decodes an NNPFC SEI message and / or an NNPFA SEI message from the second nesting SEI message.

[0345] IV. c Methods for inputting constituent frames to the neural network inference

[0346] Temporal input pictures to a neural network inference may be defined as pictures residing in the same scalability layer.

[0347] Methods for inputting several constituent frames to the neural network inference may include, but may not be limited to, the following:

[0348] 1. In an embodiment, the input tensor readily has a dimension that is used for a multitude of temporal input pictures, and the sample array(s) of constituent frame(s) are treated like any temporal input pictures in input tensor. In this option the constituent frame(s) may be required to have the same spatial resolution.

[0349] 2. In an embodiment, a new array dimension is added to indicate a multitude of constituent frames, in addition to a readily existing a dimension that is used for a multitude of temporal input pictures. In this option the constituent frame(s) may be required to have the same spatial resolution.

[0350] 3. Since constituent frames used as input to the neural network inference may have different spatial resolutions or other properties, such as bit depth, it is asserted that it may not be possible to include sample arrays of constituent frames into a single input tensor since the dimensions do not necessarily match. Therefore, in an embodiment, sample array(s) of each constituent frame is included in a different tensor provided as input to the neural network inference.

[0351] To avoid the requirement of all the constituent frames having the same spatial resolution in options 1 and 2, a pre-defined arrangement may be used to arrange a constituent frame in a tensor having greater dimensions. For example, if a second constituent frame has width and height less than those of a first constituent frame, the tensor dimensions may be allocated according to the width and height of the first constituent frame, and the second constituent frame may be considered to be located on the top-left corner and be padded to the right and below with a pre-defined value, such as 0, to occupy the tensor dimensions. In anotherexample, the second constituent frame is upsampled with a pre-defined method to the dimensions of the first constituent frame.

[0352] These embodiments are described in further detail in the following subsections.

[0353] Many of the embodiments in the following sub-sections allow inputting both temporal input pictures and constituent frames to the neural network inference, in which case it may be needed to indicate that both are expected as input. In an embodiment, an encoder indicates and / or a decoder decodes the use of multiple constituent frames as input through any of the following methods in an NNPFC SEI message:

[0354] 1. One or more (e.g., new) nnpfc_purpose values that indicate the use of multiple constituent frames.

[0355] 2. One or more (e.g., new) nnpfc mode idc values that indicate the use of multiple constituent frames.

[0356] 3. One or more (e.g., new) nnpfc auxiliary inp idc values that indicate the use of multiple constituent frames.

[0357] IV.c.l Reusing the tensor dimension for multiple input pictures

[0358] IV.c.l.i General

[0359] In part, this section describes the handling of temporal input pictures, where a first constituent frame and a second constituent frame are described at least in part using an input tensor, and the input tensor has a dimension that is used for a multitude of temporal input pictures. The one or more sample arrays of the first constituent frame and second constituent frame may be treated like any temporal input pictures in the input tensor. For instance, the first and second constituent frames could take different inpPicIdx values in the input tensor formatted as described in this paragraph. It is remarked that the NNPF design already defines an input tensor with a dimension that represents an input picture index. The input tensor for the NNPF is specified as inputTensor[ batchldx ][ inpPicIdx ][ compldx ][ y ][ x ] (when nnpfc component last flag is equal to 0) or inputTensor[ batchldx ][ inpPicIdx ] [ y ][ x ][ compldx ] (when nnpfc component last flag is equal to 1), where

[0360] - batchldx is the batch index and may be set equal to 0,

[0361] - inpPicIdx is the input picture index in the range of 0 to nnpfc_num_input_pics_minus 1 , inclusive,

[0362] - y and x are the vertical and horizontal coordinates of a sample,

[0363] - compldx is the component index, e.g., 0 (zero) for luma, 1 for Cb, and 2 forCr when the input pictures have three components.

[0364] In certain techniques, the input pictures are selected within the current layer in inverse output order starting from the current picture.

[0365] In an embodiment, an encoder encodes and / or a decoder decodes an indication of the number of input pictures to a neural network inference to be equal to the number of constituent frames.

[0366] In an embodiment, an encoder sets nnpfc_num_input_pics_minusl + 1 being equal to the number of constituent frames.

[0367] In an embodiment, the constituent frames are assigned to input picture indices according to the following:

[0368] - a pre-defined method, and / or

[0369] - an ordering provided or inferred by the nesting SEI message, and / or

[0370] - an ordering controlled by one or more syntax elements in the NNPFC SEI message.

[0371] In an embodiment, constituent frames are assigned to the interface variables luma sample array CroppedYPic[ idx ] and chroma sample arrays CroppedCbPic[ idx ] and CroppedCrPic[ idx ], when present, of the input pictures with index idx in the range of 0 (zero) to nnpfc_num_input_pics_minusl, inclusive. The idx order of assigning constituent frames may be pre-defined and / or may depend on the nesting SEI message.

[0372] In an embodiment, NestingLayerId[i] may be derived from a scalable nesting SEI message and the picture that has NestingLayerId[i] in an access unit may be assigned as the i-th input picture in the input tensor.

[0373] In an embodiment, a scalable nesting SEI message includes an NNPFC SEI message and / or an NNPFA SEI message to indicate that the NNPF applies for the layers specified by the scalable nesting SEI message.

[0374] In an additional embodiment, the scalable nesting SEI message includes the layer identifiers of the layers to which the NNPF applies. In the scalable nesting SEI message specified in WC, sn ols flag may be equal to 0 to indicate that layer identifiers are included in the scalable nesting SEI message.

[0375] In an alternative additional embodiment, the scalable nesting identifies an OLS, and the NNPF applies to the output layers of the OLS. In this case, in the scalable nesting SEI message specified in VVC, sn ols flag may be equal to 1.

[0376] In an embodiment, an encoder encodes and / or a decoder decodes an indication of the number of input pictures to a neural network inference to be greater than the number of constituent frames, which may indicate that the input pictures comprise both constituent frames and temporal input pictures.

[0377] In an embodiment, the constituent frames and temporal input pictures are assigned to input picture indices according to a pre-defined method and / or according to an ordering provided or inferred by the nesting SEI message. For example, the constituent frames of the current access unit may be assigned first, followed by the constituent frame(s) of the temporally previous access unit, in output order.

[0378] Vl.c.l.ii Example embodiment

[0379] In an example embodiment, when an NNPFA SEI message is included in a scalable nesting SEI message with sn ols flag equal to 0 (zero) and sn subpic flag equal to 0 (zero), the following process is used for deriving interface variables for interpreting NNPFC and NNPFA SEI messages.

[0380] Let currAu be an access unit for which the scalable-nested NNPFA SEI message is active.

[0381] If currAu does not include each picture that has nuh layer id equal to NestingLayerId[ i ] and ph_pic_output_flag equal to 1 for each value of i in the range of 0 (zero) to NestingNumLayers - 1, inclusive, the NNPF is not applied to currAu. Otherwise, the NNPF may be applied to currAu as follows.

[0382] Let currPic be the picture that is present in currAu and has nuh layer id equal to NestingLayerId

[0000] .

[0383] An NNPFC SEI message that has nnpfc id equal to nnpfa target id is an associated NNPFC SEI message.

[0384] It is not required to include an associated NNPFC SEI message in a scalable nesting SEI message.

[0385] It may be required that the indicated purpose of the NNPF does not include picture rate upsampling or temporal extrapolation, since their interpretation may be ambiguous when all input pictures are not temporal input pictures. With reference to the semantics of the NNPFC SEI message specified in VSEI, it may be required that both PictureRateUpsamplingFlag and TemporalExtrapolationFlag shall be equal to 0 for the associated NNPFC SEI message.

[0386] It may be required that nnpfc_num_input_pics_minusl + 1 shall be equal to N estingN umLay ers .

[0387] It may be required that the width, height, and bit depth shall be the same in each picture that is present in currAu and has nuh layer id equal to NestingLayerId[ i ] for i in the range of 0 to NestingNumLayers - 1, inclusive.

[0388] For purposes of interpretation of the scalable-nested NNPFA SEI message with the associated NNPFC SEI messages, the following variables are specified:

[0389] - CroppedWidth is set equal to the value of pps_pic_width_in_luma_samples- SubWidthC * ( pps conf win left offset + pps conf win right offset ) for currPic.

[0390] - CroppedHeight is set equal to the value of pps_pic_height_in_luma_samples- SubHeightC * ( pps conf win top offset + pps conf win bottom offset ) for currPic.

[0391] - When nnpfc inp order idc is equal to 0, 2 or 3, the luma sample array of the picture that is present in currAu and has nuh layer id equal to NestingLayerId[ i ] is assigned to the luma sample array CroppedYPic[ i ] for all values of i in the range of 0 to NestingNumLayers - 1, inclusive.

[0392] - When nnpfc inp order idc is equal to 1, 2, or 3, the following applies for each value of i in the range of 0 to NestingNumLayers - 1, inclusive:

[0393] - If there is a picture that is present in currAu, has nuh layer id equal toNestingLayerId[ i ], and has chroma sample arrays, these chroma sample arrays are assigned to CroppedCbPic[ i ] and CroppedCrPic[ i ].

[0394] - Otherwise, all the sample values of CroppedCbPic[ i ] and andCroppedCrPic[ i ] are set equal to ( 1 « ( BitDepth - 1 ) ).

[0395] - BitDepthY and BitDepthC are both set equal to BitDepth of currPic.

[0396] - ChromaFormatldc is set equal to sps chroma format idc applying to currPic.

[0397] - When nnpfc auxiliary inp idc is equal to 1, the following applies for each value of i in the range of 0 to NestingNumLayers - 1, inclusive:

[0398] - StrengthControlVal[ i ] is set equal to the value of ( firstSliceQpY +QpBdOffset ) -^ ( 63 + QpBdOffset ), where firstSliceQpY is equal to SliceQpY of the first slice of the picture that is present in currAu and has nuh layer id equal to NestingLayerId[ i ].

[0399] VI.c.2 Adding a tensor dimension for constituent frames

[0400] In an embodiment, a new dimension is added in the input tensor for constituent tensors. For example, the input tensor for the NNPF may be specified as inputTensor[ batchldx ][ inpPicIdx ][ constFrldx ][ compldx ][ y ][ x ] (when nnpfc component last flag is equal to 0) or inputTensor[ batchldx ][ inpPicIdx ][ constFrldx ][ y ][ x ][ compldx ] (when nnpfc component last flag is equal to 1), where inpPicIdx is the temporal input picture index and constFrldx specifies the index of the constituent frame and other index variables are as described before. constFrldx may alternatively be referred to as the cross-domain input picture index.

[0401] In an embodiment, an encoder encodes and / or a decoder decodes an indication of the number of constituent frames to a neural network inference. Without loss of generality, the indication may be called nnpfc_num_cd_input_pics_minusl, which may indicate that constFrldx is in the range of 0 to nnpfc_num_cd_input_pics_minusl, inclusive, and nesting SEI message enclosing an NNPFA SEI message for this NNPF identify nnpfc_num_cd_input_pics_minusl + 1 constituent frames.

[0402] In an embodiment, the constituent frames are assigned to cross-domain input picture indices according to a pre-defined method and / or according to an ordering provided or inferred by the nesting SEI message.

[0403] VI. c.3 Increasing the count of input tensors

[0404] Without loss of generality, embodiments are described with reference to the term auxiliary input tensor. It is to be understood that "auxiliary" is merely a qualifier and could be removed or replaced by any other qualifier.

[0405] Without loss of generality, the first reconstructed constituent frame may be referred to as the primary input picture and second reconstructed constituent frame may be referred to as an auxiliary input picture. It is to be understood that "auxiliary" is merely a qualifier and does not necessarily refer to an auxiliary picture or an auxiliary picture layer as defined in some video coding formats.

[0406] A neural network inference takes multiple tensors as input, where one is the input tensor of the primary input picture that may be intended to be filtered and the others are called auxiliary input tensors comprising the sample arrays of the auxiliary input pictures that may assist in the filtering. It is to be understood that embodiments are not limited to filtering only the input tensor and auxiliary input pictures may be filtered by the neural network inference.

[0407] In an embodiment, the auxiliary input tensors are derived the same way as the input tensor. Consequently, the syntax elements controlling the derivation of the input tensor are replicated for each auxiliary input tensor. The same process is used to form both the input tensor and the auxiliary input tensors, controlled by their respective syntax elements.

[0408] In an embodiment, nnpfc auxiliary inp idc may be specified as a bit field to allow mixed use of auxiliary input types. A previously unused bit position of nnpfc auxiliary inp idc, may be used for the proposed auxiliary input type. For example, when nnpfc auxiliary inp idc & 4 is greater than 0 (zero), input derived from indicated constituent frames is in use for the NNPF inference.

[0409] In an example embodiment, the NNPFC SEI message syntax is appended as follows. It is to be understood that the syntax is merely an example and embodiments may be realized with any syntax.

[0410] The semantics may be described as follows:

[0411] nnpfc_num_aux_pic_tensors_minusl plus 1 specifies the number of picturebased auxiliary input tensors that are derived from sample arrays. If nnpfc_num_aux_pic_tensors_minusl is present, the value of numAuxInputTensors is set equal to nnpfc_num_aux_pic_tensors_minusl + 1. Otherwise (nnpfc_num_aux_pic_tensors_minusl is not present), the value of num numAuxInputTensors is set equal to 0.

[0412] nnpfc_aux_pic_idx[ i ] specifies the index of the picture used to derive the i-th picture-based auxiliary input tensor among the candidate auxiliary input picture list CandAuxList.

[0413] nnpfc_aux_inp_format_idc[ i ] indicates the method of converting a sample value of the picture CandAuxList[ nnpfc_aux_pic_idx[ i ] ] to a value of the i-th picture-based auxiliary input tensor. The semantics of nnpfc_aux_inp_format_idc[ i ] are the same as those for nnpfc inp format idc with the following replacements and the functions InpY and InpC, when applied to a picture in CandAuxList, apply with the following replacements:

[0414] - BitDepthY is replaced by the luma bit depth of CandAuxList[ nnpfc_aux_pic_idx[ i ] ].

[0415] - BitDepthC is replaced by the chroma bit depth of CandAuxList[ nnpfc_aux_pic_idx[ i ] ], when applicable.

[0416] - inpTensorBitDepthY is replaced by auxInpTensorBitDepthY[ i ] derived below.

[0417] inpTensorBitDepthC is replaced by aux!npTensorBitDepthC[ i ] derived below.

[0418] nnpfc_aux_inp_order_idc[ i ] indicates the method of ordering the sample arrays of the picture CandAuxList[ nnpfc_aux_pic_idx[ i ] ] to form the i-th picture-based auxiliary input tensor and otherwise has the same semantics as nnpfc inp order idc with ChromaFormatldc replaced by the chroma format indicator of CandAuxList[ nnpfc_aux_pic_idx[ i ] ] and ChromaUpsamplingFlag set equal to 0 for the semantics.

[0419] nnpfc_aux_inp_tensor_luma_bitdepth_minus8[ i ] plus 8 specifies the bit depth of luma sample values in the i-th picture-based auxiliary input tensor. The value of aux!npTensorBitDepthY[ i ] is derived as follows:

[0420] aux!npTensorBitDepthY[ i ] = nnpfc_aux_inp_tensor_luma_bitdepth_minus8[ i ] + 8.

[0421] nnpfc_aux_inp_tensor_chroma_bitdepth_minus8[ i ] plus 8 specifies the bit depth of chroma sample values in the i-th picture-based auxiliary input tensor. The value of aux!npTensorBitDepthC[ i ] is derived as follows:

[0422] aux!npTensorBitDepthC[ i ] = nnpfc_aux_inp_tensor_chroma_bitdepth_minus8[ i ] + 8.

[0423] In an embodiment, the semantics of an NNPFC SEI message use an additional interface variable: candidate auxiliary input picture list CandAuxList, which includes pictures from which auxiliary input tensors may be derived. CandAuxList may be considered as a multidimensional array, with array index idx in CandAuxList[ idx ] indicating an entry the idx-th position within the list, and Y, Cb, and Cr sample arrays assigned to CandAuxList[ idx ]

[0000] , CandAuxList[ idx ]

[0001] , and CandAuxList[ idx ]

[0002] , respectively.

[0424] In an embodiment, when an NNPFC SEI message is included in a scalable nesting SEI message, the candidate auxiliary input picture list CandAuxList[ idx ] is set to consist of the cropped decoded pictures in increasing order of layer identifier values (e.g., nuh layer id values in WC) for which both of the following conditions are true:

[0425] - The cropped decoded pictures are decoded from the access unit that includes the current picture currPic for which the NNPF is activated.

[0426] - The cropped decoded pictures have a layer identifier identified in the scalable nesting SEI message. In WC, the cropped decoded pictures have nuh layer id that is equal to NestingLayerId[ i ] for each value of i in the range of 0 to nestingNumLayers - 1, inclusive, in increasing order of i, excluding the nuh layer id value of currPic.

[0427] In an embodiment, when an NNPFC SEI message is included in a constituent rectangle nesting SEI message, the candidate auxiliary input picture list CandAuxList[ idx ] is set to consist of the constituent rectangles identified by crn_rect_id[ i ] in a pre-defined order the values of i, such as increasing order of i.

[0428] IV. d Other embodiments and generalizations

[0429] Some embodiments may be limited to having the same properties in all constituent frames. In an embodiment, a processing chain is inferred or explicitly indicated by an encoder or decoded by a decoder, e.g., using the SEI processing order SEI message. The processing chain may include one or more conversion operations to align the properties of constituent frames. The properties may include, but may not be limited to:

[0430] 1. spatial resolution (width and height);

[0431] 2. bit depth; and / or

[0432] 3. chroma format.

[0433] The conversion operation may, for example, be specified through a neural- network filter.

[0434] For example, an SEI processing order SEI message may define a processing chain including spatial resolution upsampling NNPF followed by a quality-enhancement NNPF. The SEI processing order SEI message may comprise NNPFC SEI message types with respective SEI prefixes (e.g. including nnpfc_purpose and nnpfc id syntax elements) and / or NNPFA SEI message types with respective SEI prefixes (e.g. including nnpfa target id syntax element). The NNPFA SEI message(s) for the spatial resolution upsampling may be present in SEI NAL unit(s) that has nuh layer id of the layer that contains mask or map video (see use case IV.a.2 as an example). Consequently, the NNPF inference causes spatial resolution upsampling of the reconstructed mask or map video. The NNPFA SEI message(s) of the qualityenhancement NNPF may be present in a scalable nesting SEI message that identifies layers ofthe mask / map video and texture video and causes the NNPF to enhance the texture video as guided by the spatially upsampled mask / map video.

[0435] It is noted that "first" and "second" are merely qualifiers in the terms the first constituent frame and second constituent frame and could be replaced by some other qualifiers. They do not necessarily correspond to the order that constituent frames are assigned to input picture indices.

[0436] It is to be understood that even though embodiments have been described with reference to enhancing the first reconstructed constituent frame to an enhanced first constituent frame, the embodiments similarly apply to enhancing any of the constituent frames. In an embodiment, an encoder uses nnpfc_input_pic_filtering_flag[ i ] to indicate which constituent frames given as input are enhanced or filtered. Furthermore, it is to be understood that filtering may generate one or more output picture(s) in addition to or instead of enhancing constituent frame(s) given as input.

[0437] When filtering generates one or more output picture(s) in addition to or instead of enhancing constituent frame(s) given as input, embodiments for an encoding system may comprise indicating a filtering purpose (e.g., within nnpfc_purpose) and embodiments for a decoding system may comprise decoding a filtering purpose (e.g., from nnpfc_purpose), wherein the filtering purpose indicates the type of the one or more output picture(s) and may be one or more of the following, but need not be limited to them:

[0438] 1. Depth map generation purpose (from stereoscopic video), wherein the one or more output picture(s) are depth maps;

[0439] 2. View synthesis purpose, wherein the one or more output picture(s) represent views not among those represented by the input picture(s).

[0440] In an embodiment, when the depth map filtering purpose is indicated, one or more syntax elements are additionally present to characterize the generated depth map(s), which may be indicative of one or more of the following: the view(s) that depth map(s) represent (in relation to the views of the input pictures); a depth representation type, which may be for example an inverse of a distance that is uniformly mapped to an output value range, a disparity that is uniformly mapped to an output value range, a distance that is uniformly mapped to an output value range; a distance that corresponds to the minimum value of an output value range; a distance that corresponds to the maximum value of an output value range; a disparity that corresponds to the minimum value of an output value range; a disparity that corresponds to the maximum value of an output value range.

[0441] In an embodiment, when the view synthesis filtering purpose is indicated, one or more syntax elements are additionally present to characterize the synthesized view(s), which may be indicative of one or more of the following: the view position(s) in relation to the views of the input pictures;

[0442] IV. e. In-loop embodiments

[0443] It is to be understood that embodiments that have been described in relation to post-filtering may likewise be used for in-loop filtering. Any of the embodiments relating to post-processing stages may be respectively realized for in-loop processing

[0444] In an embodiment, an encoder and / or a decoder further comprises: using the enhanced first constituent frame as a reference for prediction, wherein the prediction may, for example, be inter prediction or inter-layer prediction.

[0445] It is to be understood that embodiments described with reference to SEI message(s) may likewise use syntax structures that are decoded normatively. An encoder may encode indication(s) in a syntax structure that is normatively decoded by a decoder, as opposed to encoding indication(s) as supplemental enhancement information or alike.

[0446] In an embodiment, an encoder encodes in a syntax structure, such as a video parameter set, indication(s) of the scalability layer(s) that comprise the constituent frames used as input to the neural network inference. In an embodiment, a decoder decodes from a syntaxstructure, such as a video parameter set, indication(s) of the scalability layer(s) that comprise the constituent frames used as input to the neural network inference.

[0447] In an embodiment, an encoder encodes in a syntax structure, such as a sequence parameter set or a picture parameter set, indication(s) of rectangular region(s) that comprise the constituent frames used as input to the neural network inference. In an embodiment, a decoder decodes from a syntax structure, such as a sequence parameter set or a picture parameter set, indication(s) of rectangular region(s) that comprise the constituent frames used as input to the neural network inference.

[0448] In an embodiment, an encoder encodes in-loop neural-network filter characteristics in a syntax structure, such as an adaptation parameter set. In an embodiment, a decoder decodes in-loop neural-network filter characteristics from a syntax structure, such as an adaptation parameter set.

[0449] In an embodiment, an encoder encodes in-loop neural-network filter activation in a picture-level syntax structure, such as a picture header or a picture trailer. In an embodiment, a decoder decodes in-loop neural-network filter activation from a picture-level syntax structure, such as a picture header or a picture trailer. A picture trailer data unit (which may alternatively have other names, such as a coded picture completion data unit) may indicate the end of coded data of a picture or indicate the end of a picture unit.

[0450] V. Example apparatus and other information

[0451] Turning to FIG. 8, this figure is an example of a block diagram of an apparatus 180 suitable for implementing any of the encoders or decoders described herein. The apparatus 180 includes circuitry comprising one or more processors 820, one or more memories 825, one or more transceivers 830, one or more network (N / W) interface(s) (I / F(s)) 855 and user interface (UI) circuitry and elements 857, interconnected through one or more buses 827. Depending on implementation, some apparatus may not have all of the circuitry. For example, an apparatus 180 might not have UI circuitry and elements 857. An apparatus may have additional circuitry, not described here. FIG. 8 is presented merely as an example.

[0452] Each of the one or more transceivers 830 includes a receiver, Rx, 832 and a transmitter, Tx, 833. The one or more buses 827 may be address, data, and / or control buses, and may include any interconnection mechanism, such as a series of lines on a motherboard orintegrated circuit, fiber optics or other optical communication equipment, and the like. The one or more transceivers 830 are connected to one or more antennas 805, and may communicate using wireless link 811, which could implement any number of wireless communication interfaces such as Wi-Fi, cellular, or satellite.

[0453] The one or more memories 825 include computer program code 823. The apparatus 180 includes a program 840, comprising one of or both parts 840-1 and / or 840-2. The program 840 may implement an encoder 130, a decoder 140, or a codec (130 + 140), which implements both encoding and decoding. The program itself may be implemented in a number of ways. The program 840 may be implemented in circuitry as program 840-1, such as being implemented as part of the one or more processors 820, and contains instructions implemented in circuitry. The program 840-1 may be implemented also as an integrated circuit or through other circuitry such as a programmable gate array. In another example, the program 840 may be implemented as program 840-2, which is implemented as computer program code (having corresponding instructions) 823 and is executed by the one or more processors 820. For instance, the one or more memories 825 store instructions that, when executed by the one or more processors 820, cause the apparatus 180 to perform one or more of the operations as described herein.

[0454] The network interface(s) (N / W I / F(s)) 855 are wired interfaces communicating using link(s) 856, which could be fiber optic or other wired interfaces. The apparatus 180 could include only wireless transceiver(s) 830, only N / W I / Fs 855, or both wireless transceiver(s) 830 and N / W I / Fs 855.

[0455] The apparatus 180 may or may not include UI circuitry and elements 857. These could include a display such as a touchscreen, speakers, or interface elements such as for headsets. For instance, an apparatus 180 of a smartphone would typically include at least a touchscreen and speakers. The UI circuitry and elements 857 may also include circuity to communicate with external UI elements (not shown) such as displays, keyboards, mice, headsets, and the like.

[0456] The computer readable memories 825 may be of any type suitable to the local technical environment and may be implemented using any suitable data storage technology, such as semiconductor-based memory devices, flash memory, firmware, magnetic memory devicesand systems, optical memory devices and systems, fixed memory and removable memory. The processor(s) 820 may be of any type suitable to the local technical environment, and may include one or more of general-purpose computers, special purpose computers, microprocessors, digital signal processors (DSPs) and processors based on a multi-core processor architecture, as nonlimiting examples. The processor(s) 820 control the apparatus 180 to perform the operations as described herein. The processor(s) 820 may execute instructions, including microcode, but are not implemented solely in software.

[0457] Some embodiments have been described with reference to constituent frames being either rectangular regions of a picture or pictures from different scalability layers. It is to be understood that embodiments may be similarly realized with reference to constituent frames being rectangular regions of pictures from one or more scalability layers.

[0458] Some embodiments have been described with the help of syntax of the bitstream. It needs to be understood, however, that the corresponding structure and / or computer program may reside at the encoder for generating the bitstream and / or at the decoder for decoding the bitstream.

[0459] Where example embodiments have been described with reference to an encoder, it needs to be understood that the resulting bitstream and the decoder have corresponding elements in them. Likewise, where example embodiments have been described with reference to a decoder, it needs to be understood that the encoder has structure and / or computer program for generating the bitstream to be decoded by the decoder.

[0460] Some example embodiments have been described with the help of certain syntax elements the bitstream. It needs to be understood, however, that embodiments may be similarly realized with different sets of syntax elements that partly or fully cover the semantics of one or more syntax elements described in example embodiments.

[0461] Some example embodiments have been described with reference to SEI message(s). It needs to be understood, however, that embodiments may similarly be realized with any similar structures or data units, such as a metadata open bitstream unit (OBU), as specified in AVI or AV2, for example. Specific SEI message syntax structures have been presented in some example embodiments, but it needs to be understood that embodiments generally apply to anySEI messages or any syntax structures that may have at least partly a similar intent as the intent of those specific SEI messages.

[0462] The following are additional examples.

[0463] Example 1. A method, comprising: decoding one or more indications of a first constituent frame and a second constituent frame to be used as input for a neural network inference, wherein the first and second constituent frames are rectangular regions or pictures of different scalability layers, and the decoding creates a first reconstructed constituent frame from the first constituent frame and a second reconstructed constituent frame from the second constituent frame; providing the first reconstructed constituent frame and the second reconstructed constituent frame as inputs to the neural network inference; and performing the neural network inference to enhance the first reconstructed constituent frame to an enhanced first constituent frame.

[0464] Example 2. The method according to example 1, wherein: the first constituent frame and the second constituent frame are frames of stereoscopic video, wherein the first constituent frame is at a first resolution and the second constituent frame is at a second resolution, and wherein the first resolution is lower than the second resolution; and the performing the neural network inference enhances the first reconstructed constituent frame to an enhanced first reconstructed constituent frame by enhancing the first reconstructed constituent frame from the first resolution to the second resolution.

[0465] Example 3. The method according to example 1, wherein: the first constituent frame comprises a reconstructed picture and the second constituent frame comprises a mask corresponding to the reconstructed picture; and the performing the neural network inference enhances the first reconstructed constituent frame using the corresponding mask.

[0466] Example 4. The method according to example 3, wherein the mask comprises at least one of the following: an object mask; a Region of Interest (ROI) mask; a saliency map; an alpha mask; an occupancy mask; a filtering mask; a depth map; a gain map; or an attenuation map.

[0467] Example 5. The method according to example 3 or 4, wherein the neural network inference has been trained in a manner for one of the following: filtering takes place only within object boundaries; or the filtering takes place only outside the object boundaries.

[0468] Example 6. The method according to example 1, wherein the first constituent frame and the second constituent frame are frames of stereoscopic video, wherein the first constituent frame is at a first resolution and the second constituent frame is at a second resolution, and wherein the first resolution is lower than the second resolution; and the performing the neural network inference enhances the first reconstructed constituent frame to one or more depth views, as the enhanced first constituent frame, representing one of depth or disparity.

[0469] Example 7. The method according to example 1, wherein: the first constituent frame comprises a texture view from a first viewpoint and the second constituent frame comprises a depth view from the first viewpoint; and the performing the neural network inference enhances the first reconstructed constituent frame by generating a texture view of another viewpoint.

[0470] Example 8. The method according to example 1, wherein: the first constituent frame comprises a texture picture and the second constituent frame comprises a depth picture; and the performing the neural network inference enhances the first reconstructed constituent frame by generating a picture with bokeh effect.

[0471] Example 9. The method according to example 1, wherein constituent frames for the first and second constituent frames comprise one of the following; a rectangular region within a frame that contains one or more other constituent frames; or a frame within two or more adjacent frames, in output order, that correspond to a same display timestamp; or a first frame in a first scalability layer.

[0472] Example 10. The method according to example 9, wherein constituent frames are associated with each other in one or more of the following ways: all constituent frames within a same frame are associated with each other; all adjacent frames, in output order, that correspond to a same display timestamp are associated with each other; or all frames of a same access at different scalability layers are associated with each other.

[0473] Example 11. The method according to any of examples 1 to 10, wherein the decoding comprises decoding a nesting message comprising supplemental enhancement information (SEI) that includes a message having one or both of a neural-network post-filter characteristic SEI or neural-network post-filter activation SEI, wherein the nesting messageindicates one or both of scalability layers or regions that comprise the first and second constituent frames used as input to the neural network inference.

[0474] Example 12. The method according to example 11, wherein the nesting message indicates the one or both of the or subpictures used as input via the first and second constituent frames to the neural network inference.

[0475] Example 13. The method according to example 12, wherein subpictures are indicated, and the subpictures are extracted from a cropped decoded picture and treated as an input picture for inference using a neural network post filter of the neural network inference.

[0476] Example 14. The method according to any of examples 1 to 13, wherein first constituent frame and the second constituent frame are described at least in part using an input tensor, the input tensor has a dimension that is used for a multitude of temporal input pictures, and one or more sample arrays of the first constituent frame and second constituent frame are treated as are any temporal input pictures in the input tensor.

[0477] Example 15. A method, comprising: encoding one or more indications of a first constituent frame and a second constituent frame of input video to be used as input by a decoder for a neural network inference, wherein the first and second constituent frames are rectangular regions or pictures of different scalability layers; and outputting the encoded one or more indications in a bitstream.

[0478] Example 16. The method according to example 15, wherein: the first constituent frame and the second constituent frame are frames of stereoscopic video, wherein the first constituent frame is at a first resolution and the second constituent frame is at a second resolution, and wherein the first resolution is lower than the second resolution; and performing the neural network inference enhances the first constituent frame to an enhanced first constituent frame by enhancing the first constituent frame from the first resolution to the second resolution.

[0479] Example 17. The method according to example 15, wherein: the first constituent frame comprises a reconstructed picture and the second constituent frame comprises a mask corresponding to the reconstructed picture; and performing the neural network inference enhances the first constituent frame using the corresponding mask.

[0480] Example 18. The method according to example 17, wherein the mask comprises at least one of the following: an object mask; a Region of Interest (ROI) mask; asaliency map; an alpha mask; an occupancy mask; a filtering mask; a depth map; a gain map; or an attenuation map.

[0481] Example 19. The method according to example 17 or 18, wherein the neural network inference has been trained in a manner for one of the following: filtering takes place only within object boundaries; or the filtering takes place only outside the object boundaries.

[0482] Example 20. The method according to example 15, wherein the first constituent frame and the second constituent frame are frames of stereoscopic video, wherein the first constituent frame is at a first resolution and the second constituent frame is at a second resolution, and wherein the first resolution is lower than the second resolution; and performing the neural network inference enhances the first constituent frame to one or more depth views, as the enhanced first constituent frame, representing one of depth or disparity.

[0483] Example 21. The method according to example 15, wherein: the first constituent frame comprises a texture view from a first viewpoint and the second constituent frame comprises a depth view from the first viewpoint; and performing the neural network inference enhances the first constituent frame by generating a texture view of another viewpoint.

[0484] Example 22. The method according to example 15, wherein: the first constituent frame comprises a texture picture and the second constituent frame comprises a depth picture; and performing the neural network inference enhances the first constituent frame by generating a picture with bokeh effect.

[0485] Example 23. The method according to example 15, wherein a constituent frame for one or both of the first constituent frame or the second constituent frame is defined as the following a rectangular region within a frame that contains one or more other constituent frames; a frame within two or more adjacent frames, in output order, that correspond to a same display timestamp; or a first frame in a first scalability layer.

[0486] Example 24. The method according to example 23, wherein constituent frames are associated with each other in one or more of the following ways: all constituent frames within a same frame are associated with each other; all adjacent frames, in output order, that correspond to a same display timestamp are associated with each other; or all frames of a same access at different scalability layers are associated with each other.

[0487] Example 25. The method according to any of examples 15 to 24, wherein the encoding comprises encoding a nesting message comprising supplemental enhancement information (SEI) that includes a message having one or both of a neural-network post-filter characteristic SEI or neural-network post-filter activation SEI, wherein the nesting message indicates one or both of scalability layers or regions that comprise the first and second constituent frames used as input to the neural network inference.

[0488] Example 26. The method according to example 25, wherein the nesting message indicates one or both of the scalability layers or subpictures used as input via the first and second constituent frames to the neural network inference.

[0489] Example 27. An apparatus, comprising means for: decoding one or more indications of a first constituent frame and a second constituent frame to be used as input for a neural network inference, wherein the first and second constituent frames are rectangular regions or pictures of different scalability layers, and the decoding creates a first reconstructed constituent frame from the first constituent frame and a second reconstructed constituent frame from the second constituent frame; providing the first reconstructed constituent frame and the second reconstructed constituent frame as inputs to the neural network inference; and performing the neural network inference to enhance the first reconstructed constituent frame to an enhanced first constituent frame.

[0490] Example 28. The apparatus according to example 27, wherein: the first constituent frame and the second constituent frame are frames of stereoscopic video, wherein the first constituent frame is at a first resolution and the second constituent frame is at a second resolution, and wherein the first resolution is lower than the second resolution; and the performing the neural network inference enhances the first reconstructed constituent frame to an enhanced first reconstructed constituent frame by enhancing the first reconstructed constituent frame from the first resolution to the second resolution.

[0491] Example 29. The apparatus according to example 27, wherein: the first constituent frame comprises a reconstructed picture and the second constituent frame comprises a mask corresponding to the reconstructed picture; and the performing the neural network inference enhances the first reconstructed constituent frame using the corresponding mask.

[0492] Example 30. The apparatus according to example 29, wherein the mask comprises at least one of the following: an object mask; a Region of Interest (ROI) mask; a saliency map; an alpha mask; an occupancy mask; a filtering mask; a depth map; a gain map; or an attenuation map.

[0493] Example 31. The apparatus according to example 29 or 30, wherein the neural network inference has been trained in a manner for one of the following: filtering takes place only within object boundaries; or the filtering takes place only outside the object boundaries.

[0494] Example 32. The apparatus according to example 27, wherein the first constituent frame and the second constituent frame are frames of stereoscopic video, wherein the first constituent frame is at a first resolution and the second constituent frame is at a second resolution, and wherein the first resolution is lower than the second resolution; and the performing the neural network inference enhances the first reconstructed constituent frame to one or more depth views, as the enhanced first constituent frame, representing one of depth or disparity.

[0495] Example 33. The apparatus according to example 27, wherein: the first constituent frame comprises a texture view from a first viewpoint and the second constituent frame comprises a depth view from the first viewpoint; and the performing the neural network inference enhances the first reconstructed constituent frame by generating a texture view of another viewpoint.

[0496] Example 34. The apparatus according to example 27, wherein: the first constituent frame comprises a texture picture and the second constituent frame comprises a depth picture; and the performing the neural network inference enhances the first reconstructed constituent frame by generating a picture with bokeh effect.

[0497] Example 35. The apparatus according to example 27, wherein constituent frames for the first and second constituent frames comprise one of the following; a rectangular region within a frame that contains one or more other constituent frames; or a frame within two or more adjacent frames, in output order, that correspond to a same display timestamp; or a first frame in a first scalability layer.

[0498] Example 36. The apparatus according to example 35, wherein constituent frames are associated with each other in one or more of the following ways: all constituent frames within a same frame are associated with each other; all adjacent frames, in output order, that correspond to a same display timestamp are associated with each other; or all frames of a same access at different scalability layers are associated with each other.

[0499] Example 37. The apparatus according to any of examples 27 to 36, wherein the decoding comprises decoding a nesting message comprising supplemental enhancement information (SEI) that includes a message having one or both of a neural-network post-filter characteristic SEI or neural-network post-filter activation SEI, wherein the nesting message indicates one or both of scalability layers or regions that comprise the first and second constituent frames used as input to the neural network inference.

[0500] Example 38. The apparatus according to example 37, wherein the nesting message indicates the one or both of the or subpictures used as input via the first and second constituent frames to the neural network inference.

[0501] Example 39. The apparatus according to example 38, wherein subpictures are indicated, and the subpictures are extracted from a cropped decoded picture and treated as an input picture for inference using a neural network post filter of the neural network inference.

[0502] Example 40. The apparatus according to any of examples 27 to 39, wherein first constituent frame and the second constituent frame are described at least in part using an input tensor, the input tensor has a dimension that is used for a multitude of temporal input pictures, and one or more sample arrays of the first constituent frame and second constituent frame are treated as are any temporal input pictures in the input tensor.

[0503] Example 41. An apparatus, comprising means for: encoding one or more indications of a first constituent frame and a second constituent frame of input video to be used as input by a decoder for a neural network inference, wherein the first and second constituent frames are rectangular regions or pictures of different scalability layers; and outputting the encoded one or more indications in a bitstream.

[0504] Example 42. The apparatus according to example 41, wherein: the first constituent frame and the second constituent frame are frames of stereoscopic video, wherein the first constituent frame is at a first resolution and the second constituent frame is at a secondresolution, and wherein the first resolution is lower than the second resolution; and performing the neural network inference enhances the first constituent frame to an enhanced first constituent frame by enhancing the first constituent frame from the first resolution to the second resolution.

[0505] Example 43. The apparatus according to example 41, wherein: the first constituent frame comprises a reconstructed picture and the second constituent frame comprises a mask corresponding to the reconstructed picture; and performing the neural network inference enhances the first constituent frame using the corresponding mask.

[0506] Example 44. The apparatus according to example 43, wherein the mask comprises at least one of the following: an object mask; a Region of Interest (ROI) mask; a saliency map; an alpha mask; an occupancy mask; a filtering mask; a depth map; a gain map; or an attenuation map.

[0507] Example 45. The apparatus according to example 43 or 44, wherein the neural network inference has been trained in a manner for one of the following: filtering takes place only within object boundaries; or the filtering takes place only outside the object boundaries.

[0508] Example 46. The apparatus according to example 41, wherein the first constituent frame and the second constituent frame are frames of stereoscopic video, wherein the first constituent frame is at a first resolution and the second constituent frame is at a second resolution, and wherein the first resolution is lower than the second resolution; and performing the neural network inference enhances the first constituent frame to one or more depth views, as the enhanced first constituent frame, representing one of depth or disparity.

[0509] Example 47. The apparatus according to example 41, wherein: the first constituent frame comprises a texture view from a first viewpoint and the second constituent frame comprises a depth view from the first viewpoint; and performing the neural network inference enhances the first constituent frame by generating a texture view of another viewpoint.

[0510] Example 48. The apparatus according to example 41, wherein: the first constituent frame comprises a texture picture and the second constituent frame comprises a depth picture; and performing the neural network inference enhances the first constituent frame by generating a picture with bokeh effect.

[0511] Example 49. The apparatus according to example 41, wherein a constituent frame for one or both of the first constituent frame or the second constituent frame is defined as the following a rectangular region within a frame that contains one or more other constituent frames; a frame within two or more adjacent frames, in output order, that correspond to a same display timestamp; or a first frame in a first scalability layer.

[0512] Example 50. The apparatus according to example 49, wherein constituent frames are associated with each other in one or more of the following ways: all constituent frames within a same frame are associated with each other; all adjacent frames, in output order, that correspond to a same display timestamp are associated with each other; or all frames of a same access at different scalability layers are associated with each other.

[0513] Example 51. The apparatus according to any of examples 41 to 50, wherein the encoding comprises encoding a nesting message comprising supplemental enhancement information (SEI) that includes a message having one or both of a neural-network post-filter characteristic SEI or neural-network post-filter activation SEI, wherein the nesting message indicates one or both of scalability layers or regions that comprise the first and second constituent frames used as input to the neural network inference.

[0514] Example 52. The apparatus according to example 51, wherein the nesting message indicates one or both of the scalability layers or subpictures used as input via the first and second constituent frames to the neural network inference.

[0515] Example 53. An apparatus, comprising: one or more processors; and one or more memories storing instructions that, when executed by the one or more processors, cause the apparatus at least to perform: decoding one or more indications of a first constituent frame and a second constituent frame to be used as input for a neural network inference, wherein the first and second constituent frames are rectangular regions or pictures of different scalability layers, and the decoding creates a first reconstructed constituent frame from the first constituent frame and a second reconstructed constituent frame from the second constituent frame; providing the first reconstructed constituent frame and the second reconstructed constituent frame as inputs to the neural network inference; and performing the neural network inference to enhance the first reconstructed constituent frame to an enhanced first constituent frame.

[0516] Example 54. The apparatus according to example 53, wherein: the first constituent frame and the second constituent frame are frames of stereoscopic video, wherein the first constituent frame is at a first resolution and the second constituent frame is at a second resolution, and wherein the first resolution is lower than the second resolution; and the performing the neural network inference enhances the first reconstructed constituent frame to an enhanced first reconstructed constituent frame by enhancing the first reconstructed constituent frame from the first resolution to the second resolution.

[0517] Example 55. The apparatus according to example 53, wherein: the first constituent frame comprises a reconstructed picture and the second constituent frame comprises a mask corresponding to the reconstructed picture; and the performing the neural network inference enhances the first reconstructed constituent frame using the corresponding mask.

[0518] Example 56. The apparatus according to example 55, wherein the mask comprises at least one of the following: an object mask; a Region of Interest (ROI) mask; a saliency map; an alpha mask; an occupancy mask; a filtering mask; a depth map; a gain map; or an attenuation map.

[0519] Example 57. The apparatus according to example 55 or 56, wherein the neural network inference has been trained in a manner for one of the following: filtering takes place only within object boundaries; or the filtering takes place only outside the object boundaries.

[0520] Example 58. The apparatus according to example 53, wherein the first constituent frame and the second constituent frame are frames of stereoscopic video, wherein the first constituent frame is at a first resolution and the second constituent frame is at a second resolution, and wherein the first resolution is lower than the second resolution; and the performing the neural network inference enhances the first reconstructed constituent frame to one or more depth views, as the enhanced first constituent frame, representing one of depth or disparity.

[0521] Example 59. The apparatus according to example 53, wherein: the first constituent frame comprises a texture view from a first viewpoint and the second constituent frame comprises a depth view from the first viewpoint; and the performing the neural networkinference enhances the first reconstructed constituent frame by generating a texture view of another viewpoint.

[0522] Example 60. The apparatus according to example 53, wherein: the first constituent frame comprises a texture picture and the second constituent frame comprises a depth picture; and the performing the neural network inference enhances the first reconstructed constituent frame by generating a picture with bokeh effect.

[0523] Example 61. The apparatus according to example 53, wherein constituent frames for the first and second constituent frames comprise one of the following; a rectangular region within a frame that contains one or more other constituent frames; or a frame within two or more adjacent frames, in output order, that correspond to a same display timestamp; or a first frame in a first scalability layer.

[0524] Example 62. The apparatus according to example 61, wherein constituent frames are associated with each other in one or more of the following ways: all constituent frames within a same frame are associated with each other; all adjacent frames, in output order, that correspond to a same display timestamp are associated with each other; or all frames of a same access at different scalability layers are associated with each other.

[0525] Example 63. The apparatus according to any of examples 53 to 62, wherein the decoding comprises decoding a nesting message comprising supplemental enhancement information (SEI) that includes a message having one or both of a neural-network post-filter characteristic SEI or neural-network post-filter activation SEI, wherein the nesting message indicates one or both of scalability layers or regions that comprise the first and second constituent frames used as input to the neural network inference.

[0526] Example 64. The apparatus according to example 63, wherein the nesting message indicates the one or both of the or subpictures used as input via the first and second constituent frames to the neural network inference.

[0527] Example 65. The apparatus according to example 64, wherein subpictures are indicated, and the subpictures are extracted from a cropped decoded picture and treated as an input picture for inference using a neural network post filter of the neural network inference.

[0528] Example 66. The apparatus according to any of examples 53 to 65, wherein first constituent frame and the second constituent frame are described at least in part using aninput tensor, the input tensor has a dimension that is used for a multitude of temporal input pictures, and one or more sample arrays of the first constituent frame and second constituent frame are treated as are any temporal input pictures in the input tensor.

[0529] Example 67. An apparatus, comprising: one or more processors; and one or more memories storing instructions that, when executed by the one or more processors, cause the apparatus at least to perform: encoding one or more indications of a first constituent frame and a second constituent frame of input video to be used as input by a decoder for a neural network inference, wherein the first and second constituent frames are rectangular regions or pictures of different scalability layers; and outputting the encoded one or more indications in a bitstream.

[0530] Example 68. The apparatus according to example 67, wherein: the first constituent frame and the second constituent frame are frames of stereoscopic video, wherein the first constituent frame is at a first resolution and the second constituent frame is at a second resolution, and wherein the first resolution is lower than the second resolution; and performing the neural network inference enhances the first constituent frame to an enhanced first constituent frame by enhancing the first constituent frame from the first resolution to the second resolution.

[0531] Example 69. The apparatus according to example 67, wherein: the first constituent frame comprises a reconstructed picture and the second constituent frame comprises a mask corresponding to the reconstructed picture; and performing the neural network inference enhances the first constituent frame using the corresponding mask.

[0532] Example 70. The apparatus according to example 69, wherein the mask comprises at least one of the following: an object mask; a Region of Interest (ROI) mask; a saliency map; an alpha mask; an occupancy mask; a filtering mask; a depth map; a gain map; or an attenuation map.

[0533] Example 71. The apparatus according to example 69 or 70, wherein the neural network inference has been trained in a manner for one of the following: filtering takes place only within object boundaries; or the filtering takes place only outside the object boundaries.

[0534] Example 72. The apparatus according to example 67, wherein the first constituent frame and the second constituent frame are frames of stereoscopic video, wherein the first constituent frame is at a first resolution and the second constituent frame is at a secondresolution, and wherein the first resolution is lower than the second resolution; and performing the neural network inference enhances the first constituent frame to one or more depth views, as the enhanced first constituent frame, representing one of depth or disparity.

[0535] Example 73. The apparatus according to example 67, wherein: the first constituent frame comprises a texture view from a first viewpoint and the second constituent frame comprises a depth view from the first viewpoint; and performing the neural network inference enhances the first constituent frame by generating a texture view of another viewpoint.

[0536] Example 74. The apparatus according to example 67, wherein: the first constituent frame comprises a texture picture and the second constituent frame comprises a depth picture; and performing the neural network inference enhances the first constituent frame by generating a picture with bokeh effect.

[0537] Example 75. The apparatus according to example 67, wherein a constituent frame for one or both of the first constituent frame or the second constituent frame is defined as the following a rectangular region within a frame that contains one or more other constituent frames; a frame within two or more adjacent frames, in output order, that correspond to a same display timestamp; or a first frame in a first scalability layer.

[0538] Example 76. The apparatus according to example 75, wherein constituent frames are associated with each other in one or more of the following ways: all constituent frames within a same frame are associated with each other; all adjacent frames, in output order, that correspond to a same display timestamp are associated with each other; or all frames of a same access at different scalability layers are associated with each other.

[0539] Example 77. The apparatus according to any of examples 67 to 76, wherein the encoding comprises encoding a nesting message comprising supplemental enhancement information (SEI) that includes a message having one or both of a neural-network post-filter characteristic SEI or neural-network post-filter activation SEI, wherein the nesting message indicates one or both of scalability layers or regions that comprise the first and second constituent frames used as input to the neural network inference.

[0540] Example 78. The apparatus according to example 77, wherein the nesting message indicates one or both of the scalability layers or subpictures used as input via the first and second constituent frames to the neural network inference.

[0541] Example 79. A computer program, comprising instructions which, when the program is executed by an apparatus, cause the apparatus to carry out the methods of any of examples 1 to 26.

[0542] Example 80. The computer program according to example 79, wherein the computer program is a computer program product comprising a computer-readable medium bearing the instructions embodied therein for use with the apparatus.

[0543] Example 81. The computer program according to example 79, wherein the computer program is directly loadable into an internal memory of the apparatus.

[0544] As used in this application, the term “circuitry” may refer to one or more or all of the following:

[0545] (a) hardware-only circuit implementations (such as implementations in only analog and / or digital circuitry) and

[0546] (b) combinations of hardware circuits and software, such as (as applicable): (i) a combination of analog and / or digital hardware circuit(s) with software / firmware and (ii) any portions of hardware processor(s) (including digital signal processor(s)) with software, and memory(ies) that work together to cause an apparatus, such as a mobile phone or server, to perform various functions) and

[0547] (c) hardware circuit(s) and or processor(s), such as a microprocessor(s) or a portion of a microprocessor(s), that requires software (e.g., firmware) for operation, but the software may not be present when it is not needed for operation.

[0548] This definition of circuitry applies to all uses of this term in this application, including in any claims. As a further example, as used in this application, the term circuitry also covers an implementation of merely a hardware circuit or processor (or multiple processors) or portion of a hardware circuit or processor and its (or their) accompanying software and / or firmware. The term circuitry also covers, for example and if applicable to the particular claim element, a baseband integrated circuit or processor integrated circuit for a mobile device or a similar integrated circuit in server, a cellular network device, or other computing or network device.

[0549] Embodiments herein may be implemented in software (executed by one or more processors), hardware (e.g., an application specific integrated circuit), or a combination ofsoftware and hardware. In an example embodiment, the software (e.g., application logic, an instruction set) is maintained on any one of various conventional computer-readable media. In the context of this document, a “computer-readable medium” may be any media or means that can contain, store, communicate, propagate or transport the instructions for use by or in connection with an instruction execution system, apparatus, or device, such as a computer, with one example of a computer described and depicted, e.g., in FIG. 8. A computer-readable medium may comprise a computer-readable storage medium (e.g., memories 15, 75, and 95 or other device) that may be any media or means that can contain, store, and / or transport the instructions for use by or in connection with an instruction execution system, apparatus, or device, such as a computer. A computer-readable storage medium does not comprise propagating signals, and therefore may be considered to be non-transitory. The term “non-transitory”, as used herein, is a limitation of the medium itself (i.e., tangible, not a signal) as opposed to a limitation on data storage persistency (e.g., RAM, random access memory, versus ROM, read-only memory).

[0550] If desired, the different functions discussed herein may be performed in a different order and / or concurrently with each other. Furthermore, if desired, one or more of the above-described functions may be optional or may be combined.

[0551] Although various aspects of an example are set out in the independent claims, other aspects of an example comprise other combinations of features from the described embodiments and / or the dependent claims with the features of the independent claims, and not solely the combinations explicitly set out in the claims.

[0552] It is also noted herein that while the above describes example embodiments of an example, these descriptions should not be viewed in a limiting sense. Rather, there are several variations and modifications which may be made without departing from the scope of the present invention as defined in the appended claims.

[0553] The following abbreviations that may be found in the specification and / or the drawing figures are defined as follows:

[0554] 3D three-dimensional

[0555] a.k.a also known as

[0556] ALF adaptive loop filtering

[0557] APS adaptation parameter set

[0558] ARC adaptive resolution change

[0559] CABAC context adaptive binary arithmetic coding

[0560] CAVLC context-based variable length coding

[0561] CLVS Coded Layer Video Sequence

[0562] CR constituent rectangle

[0563] CVS coded video sequence

[0564] DCT discrete cosine transform

[0565] DPB decoded picture buffer

[0566] FHD full high definition

[0567] FR full resolution

[0568] HDR high dynamic range

[0569] HEVC High Efficiency Video Coding

[0570] HMD head-mounted display

[0571] HVS human visual system

[0572] ID identification, identifier

[0573] LDV layered depth video

[0574] LR low-resolution

[0575] MC motion compensation

[0576] MR mixed resolution

[0577] MVD multiview video plus depth

[0578] NAL network abstraction layer

[0579] NN neural network

[0580] NNPF neural-network post-filter

[0581] NNPF A neural-network post-filter activation

[0582] NNPFC neural-network post-filter characteristic(s)

[0583] OLS output layer set

[0584] PPS Picture parameter set

[0585] PON processing order nesting

[0586] RFM reference frame memory

[0587] ROI region of interest

[0588] RPR reference picture resampling

[0589] SDR standard dynamic range

[0590] SEI supplemental enhancement information

[0591] SPO SEI processing order

[0592] SN scalable nesting

[0593] UHD ultra high definition

[0594] URI Uniform Resource Identifier

[0595] VSEI versatile supplemental enhancement information

[0596] WC versatile video coding

Claims

What is claimed is:

1. A method, comprising: decoding one or more indications of a first constituent frame and a second constituent frame to be used as input for a neural network inference, wherein the first and second constituent frames are rectangular regions or pictures of different scalability layers, and the decoding creates a first reconstructed constituent frame from the first constituent frame and a second reconstructed constituent frame from the second constituent frame; providing the first reconstructed constituent frame and the second reconstructed constituent frame as inputs to the neural network inference; and performing the neural network inference to enhance the first reconstructed constituent frame to an enhanced first constituent frame.

2. The method according to claim 1, wherein: the first constituent frame and the second constituent frame are frames of stereoscopic video, wherein the first constituent frame is at a first resolution and the second constituent frame is at a second resolution, and wherein the first resolution is lower than the second resolution; and the performing the neural network inference enhances the first reconstructed constituent frame to an enhanced first reconstructed constituent frame by enhancing the first reconstructed constituent frame from the first resolution to the second resolution.

3. The method according to claim 1, wherein: the first constituent frame comprises a reconstructed picture and the second constituent frame comprises a mask corresponding to the reconstructed picture; and the performing the neural network inference enhances the first reconstructed constituent frame using the corresponding mask.

924. The method according to claim 3, wherein the mask comprises at least one of the following: an object mask; a Region of Interest (ROI) mask; a saliency map; an alpha mask; an occupancy mask; a filtering mask; a depth map; a gain map; or an attenuation map.

5. The method according to claim 3 or 4, wherein the neural network inference has been trained in a manner for one of the following: filtering takes place only within object boundaries; or the filtering takes place only outside the object boundaries.

6. The method according to claim 1 , wherein the first constituent frame and the second constituent frame are frames of stereoscopic video, wherein the first constituent frame is at a first resolution and the second constituent frame is at a second resolution, and wherein the first resolution is lower than the second resolution; and the performing the neural network inference enhances the first reconstructed constituent frame to one or more depth views, as the enhanced first constituent frame, representing one of depth or disparity.

7. The method according to claim 1, wherein: the first constituent frame comprises a texture view from a first viewpoint and the second constituent frame comprises a depth view from the first viewpoint; and93the performing the neural network inference enhances the first reconstructed constituent frame by generating a texture view of another viewpoint.

8. The method according to claim 1, wherein: the first constituent frame comprises a texture picture and the second constituent frame comprises a depth picture; and the performing the neural network inference enhances the first reconstructed constituent frame by generating a picture with bokeh effect.

9. The method according to claim 1, wherein constituent frames for the first and second constituent frames comprise one of the following; a rectangular region within a frame that contains one or more other constituent frames; or a frame within two or more adjacent frames, in output order, that correspond to a same display timestamp; or a first frame in a first scalability layer.

10. The method according to claim 9, wherein constituent frames are associated with each other in one or more of the following ways: all constituent frames within a same frame are associated with each other; all adjacent frames, in output order, that correspond to a same display timestamp are associated with each other; or all frames of a same access at different scalability layers are associated with each other.

11. The method according to any of claims 1 to 10, wherein the decoding comprises decoding a nesting message comprising supplemental enhancement information (SEI) that includes a message having one or both of a neural-network post-filter characteristic SEI or neural-network post-filter activation SEI, wherein the nesting message indicates one or both of scalability layers or regions that comprise the first and second constituent frames used as input to the neural network inference.9412. The method according to claim 11, wherein the nesting message indicates the one or both of the or subpictures used as input via the first and second constituent frames to the neural network inference.

13. The method according to claim 12, wherein subpictures are indicated, and the subpictures are extracted from a cropped decoded picture and treated as an input picture for inference using a neural network post filter of the neural network inference.

14. The method according to any of claims 1 to 13, wherein first constituent frame and the second constituent frame are described at least in part using an input tensor, the input tensor has a dimension that is used for a multitude of temporal input pictures, and one or more sample arrays of the first constituent frame and second constituent frame are treated as are any temporal input pictures in the input tensor.

15. A method, comprising: encoding one or more indications of a first constituent frame and a second constituent frame of input video to be used as input by a decoder for a neural network inference, wherein the first and second constituent frames are rectangular regions or pictures of different scalability layers; and outputting the encoded one or more indications in a bitstream.

16. The method according to claim 15, wherein: the first constituent frame and the second constituent frame are frames of stereoscopic video, wherein the first constituent frame is at a first resolution and the second constituent frame is at a second resolution, and wherein the first resolution is lower than the second resolution; and performing the neural network inference enhances the first constituent frame to an enhanced first constituent frame by enhancing the first constituent frame from the first resolution to the second resolution.9517. The method according to claim 15, wherein: the first constituent frame comprises a reconstructed picture and the second constituent frame comprises a mask corresponding to the reconstructed picture; and performing the neural network inference enhances the first constituent frame using the corresponding mask.

18. The method according to claim 17, wherein the mask comprises at least one of the following: an object mask; a Region of Interest (ROI) mask; a saliency map; an alpha mask; an occupancy mask; a filtering mask; a depth map; a gain map; or an attenuation map.

19. The method according to claim 17 or 18, wherein the neural network inference has been trained in a manner for one of the following: filtering takes place only within object boundaries; or the filtering takes place only outside the object boundaries.

20. The method according to claim 15, wherein the first constituent frame and the second constituent frame are frames of stereoscopic video, wherein the first constituent frame is at a first resolution and the second constituent frame is at a second resolution, and wherein the first resolution is lower than the second resolution; and96performing the neural network inference enhances the first constituent frame to one or more depth views, as the enhanced first constituent frame, representing one of depth or disparity.

21. The method according to claim 15, wherein: the first constituent frame comprises a texture view from a first viewpoint and the second constituent frame comprises a depth view from the first viewpoint; and performing the neural network inference enhances the first constituent frame by generating a texture view of another viewpoint.

22. The method according to claim 15, wherein: the first constituent frame comprises a texture picture and the second constituent frame comprises a depth picture; and performing the neural network inference enhances the first constituent frame by generating a picture with bokeh effect.

23. The method according to claim 15, wherein a constituent frame for one or both of the first constituent frame or the second constituent frame is defined as the following a rectangular region within a frame that contains one or more other constituent frames; a frame within two or more adjacent frames, in output order, that correspond to a same display timestamp; or a first frame in a first scalability layer.

24. The method according to claim 23, wherein constituent frames are associated with each other in one or more of the following ways: all constituent frames within a same frame are associated with each other; all adjacent frames, in output order, that correspond to a same display timestamp are associated with each other; or all frames of a same access at different scalability layers are associated with each other.9725. The method according to any of claims 15 to 24, wherein the encoding comprises encoding a nesting message comprising supplemental enhancement information (SEI) that includes a message having one or both of a neural-network post-filter characteristic SEI or neural-network post-filter activation SEI, wherein the nesting message indicates one or both of scalability layers or regions that comprise the first and second constituent frames used as input to the neural network inference.

26. The method according to claim 25, wherein the nesting message indicates one or both of the scalability layers or subpictures used as input via the first and second constituent frames to the neural network inference.

27. An apparatus, comprising means for: decoding one or more indications of a first constituent frame and a second constituent frame to be used as input for a neural network inference, wherein the first and second constituent frames are rectangular regions or pictures of different scalability layers, and the decoding creates a first reconstructed constituent frame from the first constituent frame and a second reconstructed constituent frame from the second constituent frame; providing the first reconstructed constituent frame and the second reconstructed constituent frame as inputs to the neural network inference; and performing the neural network inference to enhance the first reconstructed constituent frame to an enhanced first constituent frame.

28. The apparatus according to claim 27, wherein: the first constituent frame and the second constituent frame are frames of stereoscopic video, wherein the first constituent frame is at a first resolution and the second constituent frame is at a second resolution, and wherein the first resolution is lower than the second resolution; and98the performing the neural network inference enhances the first reconstructed constituent frame to an enhanced first reconstructed constituent frame by enhancing the first reconstructed constituent frame from the first resolution to the second resolution.

29. The apparatus according to claim 27, wherein: the first constituent frame comprises a reconstructed picture and the second constituent frame comprises a mask corresponding to the reconstructed picture; and the performing the neural network inference enhances the first reconstructed constituent frame using the corresponding mask.

30. The apparatus according to claim 29, wherein the mask comprises at least one of the following: an object mask; a Region of Interest (ROI) mask; a saliency map; an alpha mask; an occupancy mask; a filtering mask; a depth map; a gain map; or an attenuation map.

31. The apparatus according to claim 29 or 30, wherein the neural network inference has been trained in a manner for one of the following: filtering takes place only within object boundaries; or the filtering takes place only outside the object boundaries.

32. The apparatus according to claim 27, wherein the first constituent frame and the second constituent frame are frames of stereoscopic video, wherein the first constituent frame is at a first resolution and the second 99constituent frame is at a second resolution, and wherein the first resolution is lower than the second resolution; and the performing the neural network inference enhances the first reconstructed constituent frame to one or more depth views, as the enhanced first constituent frame, representing one of depth or disparity.

33. The apparatus according to claim 27, wherein: the first constituent frame comprises a texture view from a first viewpoint and the second constituent frame comprises a depth view from the first viewpoint; and the performing the neural network inference enhances the first reconstructed constituent frame by generating a texture view of another viewpoint.

34. The apparatus according to claim 27, wherein: the first constituent frame comprises a texture picture and the second constituent frame comprises a depth picture; and the performing the neural network inference enhances the first reconstructed constituent frame by generating a picture with bokeh effect.

35. The apparatus according to claim 27, wherein constituent frames for the first and second constituent frames comprise one of the following; a rectangular region within a frame that contains one or more other constituent frames; or a frame within two or more adjacent frames, in output order, that correspond to a same display timestamp; or a first frame in a first scalability layer.

36. The apparatus according to claim 35, wherein constituent frames are associated with each other in one or more of the following ways: all constituent frames within a same frame are associated with each other; all adjacent frames, in output order, that correspond to a same display timestamp are associated with each other; or100all frames of a same access at different scalability layers are associated with each other.

37. The apparatus according to any of claims 27 to 36, wherein the decoding comprises decoding a nesting message comprising supplemental enhancement information (SEI) that includes a message having one or both of a neural-network post-filter characteristic SEI or neural-network post-filter activation SEI, wherein the nesting message indicates one or both of scalability layers or regions that comprise the first and second constituent frames used as input to the neural network inference.

38. The apparatus according to claim 37, wherein the nesting message indicates the one or both of the or subpictures used as input via the first and second constituent frames to the neural network inference.

39. The apparatus according to claim 38, wherein subpictures are indicated, and the subpictures are extracted from a cropped decoded picture and treated as an input picture for inference using a neural network post filter of the neural network inference.

40. The apparatus according to any of claims 27 to 39, wherein first constituent frame and the second constituent frame are described at least in part using an input tensor, the input tensor has a dimension that is used for a multitude of temporal input pictures, and one or more sample arrays of the first constituent frame and second constituent frame are treated as are any temporal input pictures in the input tensor.

41. An apparatus, comprising means for: encoding one or more indications of a first constituent frame and a second constituent frame of input video to be used as input by a decoder for a neural network inference, wherein the first and second constituent frames are rectangular regions or pictures of different scalability layers; and outputting the encoded one or more indications in a bitstream.

42. The apparatus according to claim 41, wherein: the first constituent frame and the second constituent frame are frames of stereoscopic video, wherein the first constituent frame is at a first resolution and the second constituent frame is at a second resolution, and wherein the first resolution is lower than the second resolution; and performing the neural network inference enhances the first constituent frame to an enhanced first constituent frame by enhancing the first constituent frame from the first resolution to the second resolution.

43. The apparatus according to claim 41, wherein: the first constituent frame comprises a reconstructed picture and the second constituent frame comprises a mask corresponding to the reconstructed picture; and performing the neural network inference enhances the first constituent frame using the corresponding mask.

44. The apparatus according to claim 43, wherein the mask comprises at least one of the following: an object mask; a Region of Interest (ROI) mask; a saliency map; an alpha mask; an occupancy mask; a filtering mask; a depth map; a gain map; or an attenuation map.

45. The apparatus according to claim 43 or 44, wherein the neural network inference has been trained in a manner for one of the following: filtering takes place only within object boundaries; orthe filtering takes place only outside the object boundaries.

46. The apparatus according to claim 41, wherein the first constituent frame and the second constituent frame are frames of stereoscopic video, wherein the first constituent frame is at a first resolution and the second constituent frame is at a second resolution, and wherein the first resolution is lower than the second resolution; and performing the neural network inference enhances the first constituent frame to one or more depth views, as the enhanced first constituent frame, representing one of depth or disparity.

47. The apparatus according to claim 41, wherein: the first constituent frame comprises a texture view from a first viewpoint and the second constituent frame comprises a depth view from the first viewpoint; and performing the neural network inference enhances the first constituent frame by generating a texture view of another viewpoint.

48. The apparatus according to claim 41, wherein: the first constituent frame comprises a texture picture and the second constituent frame comprises a depth picture; and performing the neural network inference enhances the first constituent frame by generating a picture with bokeh effect.

49. The apparatus according to claim 41, wherein a constituent frame for one or both of the first constituent frame or the second constituent frame is defined as the following a rectangular region within a frame that contains one or more other constituent frames; a frame within two or more adjacent frames, in output order, that correspond to a same display timestamp; or a first frame in a first scalability layer.10350. The apparatus according to claim 49, wherein constituent frames are associated with each other in one or more of the following ways: all constituent frames within a same frame are associated with each other; all adjacent frames, in output order, that correspond to a same display timestamp are associated with each other; or all frames of a same access at different scalability layers are associated with each other.

51. The apparatus according to any of claims 41 to 50, wherein the encoding comprises encoding a nesting message comprising supplemental enhancement information (SEI) that includes a message having one or both of a neural-network post-filter characteristic SEI or neural-network post-filter activation SEI, wherein the nesting message indicates one or both of scalability layers or regions that comprise the first and second constituent frames used as input to the neural network inference.

52. The apparatus according to claim 51, wherein the nesting message indicates one or both of the scalability layers or subpictures used as input via the first and second constituent frames to the neural network inference.

53. An apparatus, comprising: one or more processors; and one or more memories storing instructions that, when executed by the one or more processors, cause the apparatus at least to perform: decoding one or more indications of a first constituent frame and a second constituent frame to be used as input for a neural network inference, wherein the first and second constituent frames are rectangular regions or pictures of different scalability layers, and the decoding creates a first reconstructed constituent frame from the first constituent frame and a second reconstructed constituent frame from the second constituent frame; providing the first reconstructed constituent frame and the second reconstructed constituent frame as inputs to the neural network inference; and104performing the neural network inference to enhance the first reconstructed constituent frame to an enhanced first constituent frame.

54. The apparatus according to claim 53, wherein: the first constituent frame and the second constituent frame are frames of stereoscopic video, wherein the first constituent frame is at a first resolution and the second constituent frame is at a second resolution, and wherein the first resolution is lower than the second resolution; and the performing the neural network inference enhances the first reconstructed constituent frame to an enhanced first reconstructed constituent frame by enhancing the first reconstructed constituent frame from the first resolution to the second resolution.

55. The apparatus according to claim 53, wherein: the first constituent frame comprises a reconstructed picture and the second constituent frame comprises a mask corresponding to the reconstructed picture; and the performing the neural network inference enhances the first reconstructed constituent frame using the corresponding mask.

56. The apparatus according to claim 55, wherein the mask comprises at least one of the following: an object mask; a Region of Interest (ROI) mask; a saliency map; an alpha mask; an occupancy mask; a filtering mask; a depth map; a gain map; or an attenuation map.10557. The apparatus according to claim 55 or 56, wherein the neural network inference has been trained in a manner for one of the following: filtering takes place only within object boundaries; or the filtering takes place only outside the object boundaries.

58. The apparatus according to claim 53, wherein the first constituent frame and the second constituent frame are frames of stereoscopic video, wherein the first constituent frame is at a first resolution and the second constituent frame is at a second resolution, and wherein the first resolution is lower than the second resolution; and the performing the neural network inference enhances the first reconstructed constituent frame to one or more depth views, as the enhanced first constituent frame, representing one of depth or disparity.

59. The apparatus according to claim 53, wherein: the first constituent frame comprises a texture view from a first viewpoint and the second constituent frame comprises a depth view from the first viewpoint; and the performing the neural network inference enhances the first reconstructed constituent frame by generating a texture view of another viewpoint.

60. The apparatus according to claim 53, wherein: the first constituent frame comprises a texture picture and the second constituent frame comprises a depth picture; and the performing the neural network inference enhances the first reconstructed constituent frame by generating a picture with bokeh effect.

61. The apparatus according to claim 53, wherein constituent frames for the first and second constituent frames comprise one of the following; a rectangular region within a frame that contains one or more other constituent frames; or106a frame within two or more adjacent frames, in output order, that correspond to a same display timestamp; or a first frame in a first scalability layer.

62. The apparatus according to claim 61, wherein constituent frames are associated with each other in one or more of the following ways: all constituent frames within a same frame are associated with each other; all adjacent frames, in output order, that correspond to a same display timestamp are associated with each other; or all frames of a same access at different scalability layers are associated with each other.

63. The apparatus according to any of claims 53 to 62, wherein the decoding comprises decoding a nesting message comprising supplemental enhancement information (SEI) that includes a message having one or both of a neural-network post-filter characteristic SEI or neural-network post-filter activation SEI, wherein the nesting message indicates one or both of scalability layers or regions that comprise the first and second constituent frames used as input to the neural network inference.

64. The apparatus according to claim 63, wherein the nesting message indicates the one or both of the or subpictures used as input via the first and second constituent frames to the neural network inference.

65. The apparatus according to claim 64, wherein subpictures are indicated, and the subpictures are extracted from a cropped decoded picture and treated as an input picture for inference using a neural network post filter of the neural network inference.

66. The apparatus according to any of claims 53 to 65, wherein first constituent frame and the second constituent frame are described at least in part using an input tensor, the input tensor has a dimension that is used for a multitude of temporal input pictures, and one or107more sample arrays of the first constituent frame and second constituent frame are treated as are any temporal input pictures in the input tensor.

67. An apparatus, comprising: one or more processors; and one or more memories storing instructions that, when executed by the one or more processors, cause the apparatus at least to perform: encoding one or more indications of a first constituent frame and a second constituent frame of input video to be used as input by a decoder for a neural network inference, wherein the first and second constituent frames are rectangular regions or pictures of different scalability layers; and outputting the encoded one or more indications in a bitstream.

68. The apparatus according to claim 67, wherein: the first constituent frame and the second constituent frame are frames of stereoscopic video, wherein the first constituent frame is at a first resolution and the second constituent frame is at a second resolution, and wherein the first resolution is lower than the second resolution; and performing the neural network inference enhances the first constituent frame to an enhanced first constituent frame by enhancing the first constituent frame from the first resolution to the second resolution.

69. The apparatus according to claim 67, wherein: the first constituent frame comprises a reconstructed picture and the second constituent frame comprises a mask corresponding to the reconstructed picture; and performing the neural network inference enhances the first constituent frame using the corresponding mask.10870. The apparatus according to claim 69, wherein the mask comprises at least one of the following: an object mask; a Region of Interest (ROI) mask; a saliency map; an alpha mask; an occupancy mask; a filtering mask; a depth map; a gain map; or an attenuation map.

71. The apparatus according to claim 69 or 70, wherein the neural network inference has been trained in a manner for one of the following: filtering takes place only within object boundaries; or the filtering takes place only outside the object boundaries.

72. The apparatus according to claim 67, wherein the first constituent frame and the second constituent frame are frames of stereoscopic video, wherein the first constituent frame is at a first resolution and the second constituent frame is at a second resolution, and wherein the first resolution is lower than the second resolution; and performing the neural network inference enhances the first constituent frame to one or more depth views, as the enhanced first constituent frame, representing one of depth or disparity.

73. The apparatus according to claim 67, wherein: the first constituent frame comprises a texture view from a first viewpoint and the second constituent frame comprises a depth view from the first viewpoint; and109performing the neural network inference enhances the first constituent frame by generating a texture view of another viewpoint.

74. The apparatus according to claim 67, wherein: the first constituent frame comprises a texture picture and the second constituent frame comprises a depth picture; and performing the neural network inference enhances the first constituent frame by generating a picture with bokeh effect.

75. The apparatus according to claim 67, wherein a constituent frame for one or both of the first constituent frame or the second constituent frame is defined as the following a rectangular region within a frame that contains one or more other constituent frames; a frame within two or more adjacent frames, in output order, that correspond to a same display timestamp; or a first frame in a first scalability layer.

76. The apparatus according to claim 75, wherein constituent frames are associated with each other in one or more of the following ways: all constituent frames within a same frame are associated with each other; all adjacent frames, in output order, that correspond to a same display timestamp are associated with each other; or all frames of a same access at different scalability layers are associated with each other.

77. The apparatus according to any of claims 67 to 76, wherein the encoding comprises encoding a nesting message comprising supplemental enhancement information (SEI) that includes a message having one or both of a neural-network post-filter characteristic SEI or neural-network post-filter activation SEI, wherein the nesting message indicates one or both of scalability layers or regions that comprise the first and second constituent frames used as input to the neural network inference.11078. The apparatus according to claim 77, wherein the nesting message indicates one or both of the scalability layers or subpictures used as input via the first and second constituent frames to the neural network inference.

79. A computer program, comprising instructions which, when the program is executed by an apparatus, cause the apparatus to carry out the methods of any of claims 1 to 26.

80. The computer program according to claim 79, wherein the computer program is a computer program product comprising a computer-readable medium bearing the instructions embodied therein for use with the apparatus.

81. The computer program according to claim 79, wherein the computer program is directly loadable into an internal memory of the apparatus.I l l

Images