QP and picture position input for nn in-loop filters

Abstract

A method of decoding a video having a sequence of pictures from a bitstream using neural network-based video coding is provided. The method includes deriving a first quantization parameter (QP) delta value from a first syntax element from the bitstream. The method includes determining a first QP value based on the first QP delta value. The method includes decoding a second QP delta value from a second syntax element from the bitstream. The method includes determining a second QP value based on the second QP delta value. The method includes passing the second QP value to a neural network as an input. The method includes using an output of the neural network to further decode the bitstream.

Description

QP AND PICTURE POSITION INPUT FOR NN IN-LOOP FILTERSTECHNICAL FIELD

[0001] Disclosed are embodiments related to video coding.BACKGROUND

[0002] VVC and HEVC

[0003] Versatile Video Coding (VVC) and its predecessor High Efficiency Video Coding (HEVC) are block-based video codecs standardized and developed jointly by ITU- T and MPEG. The codecs utilize both temporal and spatial prediction. VVC and HEVC are similar in many aspects. Spatial prediction is achieved using intra (I) prediction from within the current picture. Temporal prediction is achieved using uni-directional (P) or bidirectional inter (B) prediction on the block level from previously decoded reference pictures. The VVC version 1 specification was published as Rec. ITU-T H.266 | ISO / IEC 23090-3, “Versatile Video Coding”, in 2020.

[0004] Blocks and Units

[0005] In many video coding standards, such as HEVC and VVC, each component is split into blocks and the coded video bitstream consists of a series of coded blocks. A block is a two-dimensional array of samples. It is common in video coding that the picture is split into units that cover a specific area of the picture. Each unit consists of all blocks from all components that make up that specific area and each block belongs fully to one unit. The macroblock in H.264 and the Coding unit (CU) in HEVC and VVC are examples of units.

[0006] A block can alternatively be defined as a two-dimensional array that a transform used in coding is applied to. These blocks are known under the name “transform blocks”. Alternatively, a block can be defined as a two-dimensional array that a single prediction mode is applied to. These blocks can be called “prediction blocks”. In this application, the word block is not tied to one of these definitions but the descriptions herein can apply to either definition.

[0007] NAL units

[0008] Both HEVC and VVC define a Network Abstraction Layer (NAL). All the data in HEVC and VVC is encapsulated in NAL units. A so-called Video Coding Layer (VCL) NAL unit contains data that represents picture sample values. A non-VCL NAL unit contains additional associated data such as parameter sets and supplemental enhancement information (SEI) messages. The NAL unit in HEVC begins with a 2-byte header that identifies what type of data is carried in the NAL unit, as well as what layer ID and what temporal ID the NAL unit belongs to. The NAL unit type is transmitted in the nal unit type codeword in the NAL unit header and the type indicates and defines how the NAL unit should be parsed and decoded. The bytes after the 2-byte NAL unit header make up the payload of the type indicated by the NAL unit type. A bitstream consists of a series of concatenated NAL units.

[0009] Slices and tiles

[0010] The concept of slices in HEVC divides the picture into independently coded slices, where the decoding of one slice in a picture is independent of other slices of the same picture. Different coding types could be used for slices of the same picture, i.e., a slice could either be an I-slice, P-slice or B-slice. One purpose of slices is to enable resynchronization in case of data loss. In HEVC, a slice is a set of CTUs.

[0011] The VVC and HEVC video coding standards includes a tool called tiles that divides a picture into rectangular spatially independent regions. Tiles in VVC are similar to the tiles used in HEVC. Using tiles, a picture in VVC can be partitioned into rows and columns of CTUs where a tile is an intersection of a row and a column.

[0012] In VVC, a slice is defined as an integer number of complete tiles or an integer number of consecutive complete CTU rows within a tile of a picture that are exclusively contained in a single NAL unit. In VVC, a picture may be partitioned into either raster scan slices or rectangular slices. A raster scan slice consists of a number of complete tiles in raster scan order. A rectangular slice consists of a group of tiles that together occupy a rectangular region in the picture or a consecutive number of CTU rows inside one tile. Each slice has a slice header comprising syntax elements. Decoded slice header values from these syntax elements are used when decoding the slice. Each slice is carried in one VCL NAL unit. In an early draft of the VVC specification, slices were referred to as tile groups.

[0013] Decoding order, output order and temporal sublayers

[0014] Two important concepts in video coding are the concepts of decoding order and output order. Decoding order is the order in which the pictures are decoded. This is typically the same order as the pictures are encoded in as well as the order in which the coded pictures are transmitted. Output order is the order in which pictures are output from the decoder, and is the order in which they typically would be viewed by an end user who is watching the video.

[0015] In HEVC and VVC, the NAL unit has a nuh_temporal_id_plus 1 syntax element and the Temporal ID of the NAL unit is set to the value of nuh_temporal_id_plusl minus 1. All VCL NAL units for one picture must have the same Temporal ID value which then specifies what temporal sublayer the picture belongs to. A sublayer with Temporal ID equal to x is said to be the x-th sublayer or sublayer x. The encoder is required to set Temporal ID values such that pictures belonging to a lower temporal sublayer is perfectly decodable if higher temporal sublayers are discarded. This is ensured by restrictions in the HEVC and VVC specifications that the encoder must comply with. An example of temporal sublayers is illustrated in Table 1.

[0016] Table 1 - Temporal sublayer example (sub-GOP size = 4)

[0017] Table 1 illustrates 8 pictures, each indicated by an X, and where each picture is associated with an output order value, a decoding order value and a Temporal ID value. The 8 pictures are output or displayed in the order from left to right, in increasing value of output order that is. The decoding order values shows the order in which the pictures are decoded. This is also the order of the coded pictures in the bitstream. As an example, the entry ‘2’ in the first column of the ‘Decoding order’ row signifies that there are two pictures in front of this picture in decoding order in the sub-GOP, namely the pictures in thesecond and fourth column. The entry ‘0’ in the fourth column indicates that this is the first picture in decoding order in the sub-GOP. There are three temporal sublayers in the example, sublayer 0, 1 and 2. The sublayers are shown by the Temporal ID values and are illustrated by the vertical position of each picture X in the figure. As said earlier, in HEVC and VVC, there is a rule that no picture of a lower Temporal ID may use any picture of a higher Temporal ID for prediction. That is an important rule since it enables removal of higher temporal sublayers without affecting the decodability of the remaining lower temporal sublayers. For instance, if temporal sublayer 2 were to be removed in the example above, temporal sublayers 0 and 1 would be decodable since no sublayer 2 picture is allowed to be referenced by any sublayer 0 or 1 picture. Note that an HEVC or VVC encoder may assign all pictures to sublayer 0. Then the rule cannot be violated, but no coded picture can be safely removed based on Temporal IDs.

[0018] In this disclosure we use the term “temporal sublayer” or “sublayer” when we refer to temporal sublayers.

[0019] Hierarchical structure of pictures

[0020] Table 1 is an example of a so-called hierarchical structure of pictures. The sub-GOP size in this example is 4 since the distance in terms of pictures between the pictures in the lowest sub-layer is equal to 4. It is common in video coding to use hierarchical structures of pictures since they have been proven to provide good compression efficiency. Commonly a sub-GOP size is a power of two, such that the sub-GOP size is equal to one of 2, 4, 8, 16, 32, etc. Each such structure of pictures of size N can be constructed from the structure of size N / 2 by adding a new highest sublayer where new pictures are added in-between the pictures of the size N / 2 structure. Table 2 and Table 3 below show sub-GOP sizes 8 and 16.

[0021] Table 2 - Temporal sublayer example (sub-GOP size = 8)

[0022] Table 3 - Temporal sublayer example (sub-GOP size = 16)

[0023] Parameter Sets

[0024] HEVC and VVC specifies three types of parameter sets, the picture parameter set (PPS), the sequence parameter set (SPS) and the video parameter set (VPS). The PPS contains data that is common for a whole picture, the SPS contains data that is common for a coded video sequence (CVS) and the VPS contains data that is common for multiple CVSs, e.g., data for multiple scalability layers in the bitstream. The PPS is sent as its own NAL. The same goes for SPS and VPS.

[0025] VVC also specifies one additional parameter set, the adaptation parameter set (APS). The APS carries parameters needed for the adaptive loop filter (ALF) tool, the luma mapping and chroma scaling (LMCS) tool and the scaling list tool.

[0026] Both HEVC and VVC allow certain information (e.g. parameter sets) to be provided by external means. “By external means” should be interpreted as the information is not provided in the coded video bitstream but by some other means not specified in thevideo codec specification, e.g. via metadata possibly provided in a different data channel, as a constant in the decoder, or provided through an API to the decoder.

[0027] Picture header

[0028] VVC includes a picture header syntax structure. This syntax structure can either be conveyed in its own NAL unit or be included in a slice header, given that there is only one slice for the picture. When conveyed in a NAL unit, the NAL unit type is equal to a value that indicates that the NAL unit contains a picture header. The values of the syntax elements in the picture header are used to decode all slices of one picture.

[0029] FIG. 15 describes some of the relationships between some of the entities so far described, in particular, between the PPS, SPS, VPS, Slices, and picture headers. Option (a) shown in the figure is only allowed if there is only one slice per picture. Option (b) is allowed if there is one or more slices per picture. That is, if there is only one slice per picture, both options (a) and (b) are allowed.

[0030] POC

[0031] Pictures in HEVC are identified by their picture order count (POC) values, also known as full POC values. The POC value also indicates the output order of the pictures, such as a picture with a lower POC value is output before a picture with a higher POC value. Each slice contains a code word, pic order cnt lsb, that shall be the same for all slices in a picture. pic_order_cnt_lsb is also known as the least significant bits (Isb) of the full POC since it is a fixed-length code word and only the least significant bits of the full POC is signaled. Both encoder and decoder keep track of POC and assign POC values to each picture that is encoded / decoded. The pic order cnt lsb can be signaled by 4-16 bits. There is a variable MaxPicOrderCntLsb used in HEVC and VVC which is set to the maximum pic order cnt lsb value plus 1. This means that if 8 bits are used to signal pic_order_cnt_lsb, the maximum value is 255 and MaxPicOrderCntLsb is set to 2A8 = 256. The picture order count value of a picture is called PicOrderCntVal in HEVC and VVC. Usually, PicOrderCntVal for the current picture is simply called PicOrderCntVal.

[0032] Residuals, transforms and quantization

[0033] A residual block consists of samples that represent sample value differences between sample values of the original source block and the prediction block. A non-zero residual block is processed using a spatial transform. In the encoder, the transform coefficients are quantized according to a quantization parameter (QP) value which controlsthe precision of the quantized coefficients. The quantized coefficients can be referred to as residual coefficients. A high QP value will result in low precision of the coefficients and thus low fidelity of the residual block. A decoder receives the residual coefficients, applies inverse quantization and inverse transform to derive the decoded residual block.

[0034] In VVC, a QP value is typically assigned to each block during decoding. Each block belongs to a slice and a slice QP value is derived for each slice from parameter set and slice header syntax elements. In VVC, the derivation of the slice QP value includes decoding a delta QP value, qp delta, that is either decoded from the picture header syntax structure or from the slice header as follows:SliceQpY = 26 + pps_init_qp_minus26 + qp_deltaSliceQpY is the slice QP value for the slice, pps_init_qp_minus26 is the decoded value of a syntax element in the PPS that the slice refers to and qp delta is the decoded value of a delta QP syntax element in the picture header syntax structure or slice header. Whether the qp delta is located in the picture header syntax structure or slice header is specified by the pps qp delta info in ph flag which is located in the PPS syntax structure. In the VVC specification, the name of the qp delta syntax element is ph qp delta if it is located in the picture header and sh qp delta if it is located in the slice header.

[0035] In this description, the PPS QP refers to the QP value signalled in the PPS. For VVC, the PPS QP value is equal to 26 + pps_init_qp_minus26.

[0036] The blocks in a slice are decoded in a deterministic scan order. During decoding of the blocks, a block QP value is derived for each block . The block QP value for the first block in a slice is initialized to be equal to the slice QP value. For any block that follows in scan order, a delta QP syntax element for the block might be decoded and then the value of the syntax element is added to a predicted QP value to form the block QP value . The predicted QP value is derived from block QP values of blocks earlier in scan order. In this manner, the VVC codec supports flexible assignment of QP values for the blocks. In VVC, the granularity of the block QP is flexible, this means that the size of the spatial area to which one block QP value applies is configurable and thereby flexible. The VVC specification uses the term “quantization group” instead of “block” as used above, and it should be understood that the embodiments in this disclosure applies also to quantization groups.

[0037] The QP for each of the Cb and Cr chroma channels may be signaled as an offset to the luma QP. This may be done in the PPS using the pps cb qp offset and pps_cr_qp_offset syntax elements, in the slice header using the sh_cb_qp_offset and sh_cr_qp_offset syntax elements or by deriving the CuQpOffsetCb and CuQpOffsetCr at CU level by specifying a cu chroma qp offset idx in a list of offsets signalled in PPS, pps_cb_qp_offset_list[] and pps_cr_qp_offset_list[], respectively.

[0038] SEI messages

[0039] Supplementary Enhancement Information (SEI) messages are codepoints in the coded bitstream that do not influence the decoding process of coded pictures from VCL NAL units. SEI messages usually address issues of representation / rendering of the decoded bitstream. The overall concept of SEI messages and many of the messages themselves have been inherited from the H.264 and HEVC specifications into the VVC specification. In VVC, an SEI RBSP contains one or more SEI messages.

[0040] SEI messages assist in processes related to decoding, display or other purposes. However, SEI messages are not required for constructing the luma or chroma samples by the decoding process. Some SEI messages are required for checking bitstream conformance and for output timing decoder conformance. Other SEI messages are not required for checking bitstream conformance. A decoder is not required to support all SEI messages. Usually, if a decoder encounters an unsupported SEI message, it is discarded.

[0041] ITU-T H.274 | ISO / IEC 23002-7, also referred to as VSEI, specifies the syntax and semantics of Video Usability Information (VUI) parameters and SEI messages and is particularly intended for use with VVC, although it is written in a manner intended to be sufficiently generic that it may also be used with other types of coded video bitstreams. The first version of ITU-T H.274 | ISO / IEC 23002-7 was finalized in July 2020. At the time of writing, versions 2 and 3 of VSEI have also been published.

[0042] The NN post filter SEI messages

[0043] Several additional SEI messages were added to version 3 of VSEI, including the NN postfilter characteristics (NNPFC) SEI message and the NN post filter activation (NNPFA) SEI message, which are used for applying NN postfiltering to pictures of the decoded video. The NNPFC SEI message may specify a neural network that may be used as a post-processing filter for the decoded pictures, whereas the NNPFA SEI message indicates to which pictures an NN post-filter specified by the NNPFC SEI message shouldbe applied. More precisely, it activates or de-activates the possible use of the target neural- network post-processing filter for post-processing filtering of a set of pictures. The neural network parameters, i.e., the weights of the neural network, may be carried with the NNPFC SEI message. Alternatively, the NNPFC SEI message includes an URI for identifying the neural network and its parameters.

[0044] The NNPFC SEI message also includes parameters for describing the characteristics of the neural network, including information about the input to the NN, output from the NN and the purpose of the NN post filter. The following purposes with the NN post filter SEI message are defined:• General visual quality improvement• Chroma upsampling• Resolution resampling (increasing or decreasing the width or height)• Picture rate upsampling• Bit depth upsampling (increasing the luma bit depth or the chroma bit depth)• Colourization

[0045] The input tensor to the NN postfilter may, as specified with the nnpfc inp order idc syntax element, include either a single luma channel, two chroma channels, one luma channel and two chroma channels, or four luma channels and two chroma channels where each of the four luma channels comprises every fourth sample of the input luma samples. In addition, the input tensor may also include an auxiliary input matrix in a separate channel. Similarly, the output tensors from the NN post-filter may, as specified with the nnpfc out order idc syntax element, include either a single luma channel, two chroma channels, one luma channel and two chroma channels, or four luma channels and two chroma channels.

[0046] The bit depth of the luma and chroma samples for the input tensors and output tensors may be set individually for each of luma and chroma.

[0047] Exploration Experiment on Neural Network based Video Coding (NNVC)

[0048] At the 20th JVET meeting it was decided to set up an exploration experiment(EE) on neural network-based (NN-based) video coding [1], The exploration experiment continued at the subsequent JVET meetings 21 through 31 with many tests: NN-based in-loop filtering, NN-based post filtering, NN-based super resolution and NN-based intra prediction. The first two are of special interest to this invention and will be described further.

[0049] VVC contains three in-loop filters that are not based on neural networks: A deblocking filter, a sample adaptive offset (SAO) filter, and an adaptive loop filter (ALF). The deblocking filter is used to remove block artifacts by smoothening discontinuities in horizontal and vertical directions across block boundaries. The deblocking filter uses a block boundary strength (BS) parameter to determine the filtering strength. The BS can have values 0, 1, and 2, where a larger value indicates a stronger filtering. The output of the deblocking filter is further processed by SAO, and the output of SAO is then processed by ALF. The output of the ALF filter can then be put into the decoded picture buffer (DPB), which contains decoded pictures that may be used for prediction of subsequently encoded (or decoded) pictures. Since the deblocking filter, SAO filter and ALF filter in this way influence the pictures in the DPB used for prediction, they are classified as in-loop filters, also known as loop filters. This means that changes done by the loop filter may influence not only the current picture but future pictures. It is possible for a decoder to further filter the picture in the DPB, but not store the filtered output in the DPB. In contrast to loop filters, such a filter is not influencing future predictions and is therefore classified as a postprocessing filter, also known as a postfilter. Postfiltering is generally optional for decoders and thereby not required to be performed for decoder implementations to conform to a standard specification.

[0050] The contributions JVET-X0066 [2] and JVET-Y0143 [3] are two successive contributions that describe NN-based in-loop filtering. Both contributions use the same NN models for filtering. The NN-based in-loop filter is placed before SAO and ALF and the samples before the deblocking filter are used as input to the filter. The output of the NN- based in-loop filter is mixed with the output of the deblocking filter and forwarded as the input to SAO. The purpose of using the NN-based filter is to improve the quality of the reconstructed samples. Here it is helpful that the NN model is non-linear. While deblocking, SAO and ALF all contain non-linear elements such as conditions, and are thus not strictly linear, all three of them are based on linear filters. In contrast, a sufficiently large NN model can in principle leam any non-linear mapping and is therefore capable of representing a wider class of functions compared to deblocking, SAO and ALF. In JVET- X0066 and JVET-Y0143, there are four NN models, i.e., four NN-based in-loop filters. In arefined version of that work presented in the contribution JVET-AB0052, only two models are used: One for luma samples and another model for chroma samples.

[0051] JVET-AD0380 [4] proposed a new unified design for the NN-based in-loop filtering, which captures the benefits of previous NN structures. The unified filter has only one NN model, to filter luma and chroma samples and intra and inter pictures. FIG. 16 shows the structure of the unified filter. There are six inputs, the reconstructed samples of luma and chroma before deblocking (‘rec’), the prediction samples of luma and chroma (‘pred’), the BS information of luma and chroma (‘bs’), the quantization parameter for a sequence (‘QPbase’), the quantization parameter for each slice ('QPslice’), as well as information on whether a particular sample was intra-predicted, uni-predicted or bipredicted (‘IPB’). These inputs first go through a convolutional layer (3x3 or 1x1) and a parametric rectified linear unit (PReLU) layer separately, then they are concatenated and fused together with a 1x1 convolutional layer.

[0052] In FIG. 16, a convolutional layer with a kernel size of 3x3 is denoted “conv 3x3” and a convolutional layer with a kernel size of 1x1 is denoted “conv 1x1”. The values dl, d2, ... are the numbers of the output channels. “ I 2” denotes downsampling with a factor of 2. There are N backbone blocks. Inside a backbone block, a convolutional layer “conv 3x1 CxC21” means the convolution kernel is of size 3x1, the number of input channels is C and the number of output channels is C21.

[0053] The NN-based in-loop filters presented in JVET-X0066, JVET-AB0053, JVET-AB0052 and JVET-AD0380 increase the compression efficiency of the codec substantially, i.e., they lower the bit rate substantially without lowering the objective quality as measured by MSE-based PSNR. Increases in compression efficiency, often referred to simply as “gain”, are often measured as the Bjontegaard-delta rate (BDR) against an anchor. As an example, a BDR of -1% means that the same PSNR distortion can be reached with 1% bitrate saving on average. As reported in JVET-AF0041, for the random access (RA) configuration, the BDR for the luma component (Y) is -10.27%, and for the all-intra (Al) configuration, the BDR for the luma component is -7.86%. The complexity of NN models used for compression are often measured by MAC / pixel (multiply-accumulate operations per pixel). The high bitrate savings of an NN model is typically directly related to the high complexity of the NN model. The model described in JVET-AF0041 has a complexity of 477 kMAC / pixel, i.e., 477,000 multiply-accumulateoperations per pixel. There are also other measures of complexity, such as total model size in terms of stored parameters.

[0054] In JVET standardization, development and study of codecs is done using a common test conditions (CTC). The CTC specifies how a codec under test should be configured and what test sequences that should be used. Keeping the test conditions statis enables apples-to-apples evaluations, but with the drawback that configurations outside the CTC are not tested much and the codec performance may become too optimized towards the CTC.

[0055] The terms “neural network”, “neural network model”, and “NN model” are used interchangeably throughout this disclosure.

[0056] Details regarding QPbase and QPslice

[0057] In NNVC, QPbase and QPslice are just single numbers. They are expanded into the same size as other inputs such as a 144x144 sample array, before being used as input to the neural network. It should be noted that in many applications it may be the case that different samples inside the CTU will be associated with different QP values, due to the possibility of using delta-QP. In this case the QP could effectively be of the same resolution as the other inputs.

[0058] In NNVC version 6.0, the NN filtering operates on an NN block. For luma, the size of the NN block is either 128x128 or 256x256. The larger 256x256 NN block size is used for Intra pictures, and if both the picture width is greater than 832 and the slice QP is greater than 29. Otherwise, 128x128 is used.

[0059] The use of NN filtering can be controlled (turned on or off) at the sequence level, the slice level, or the NN block level. The NNVC version 6.0 decoder controls the filtering for the blocks where it is turned on by the QPbase and QPslice NN input parameters. These parameters are determined for each luma NN block. Source code related to QPbase and QPslice in NNVC 6.0 looks as follows (the code has been edited for clarity):Line code0 if(prmID != -1) {1 sign = (temporalld >= 4) ? 1 : -1;2 intrafactor = (inter ? 5 : 2);3 qpOffset = intrafactor * sign;4 QPpps = pic!nitQPMinus26 + 26;5 QPslice = QPpps + sliceQpDelta;6 QPbase = QPpps + (prmld ? qpOffset : 0);7 }

[0060] Line 0 may cause the NN filtering to be turned off for a current luma NN block. The value of prmld is decoded from the bitstream and can be decoded into one of the values {-1, 0, 1}. If prmld is equal to -1, the filtering is turned off, otherwise the filter is turned on for the NN block.

[0061] Line 1 determines the sign of the variable qpOffset on line 3. The variable termporalld corresponds to the temporal sublayer of a picture, also known as the Temporal ID. Looking at Table 3, the sign value for the pictures belonging to the highest temporal sublayer is set equal to 1 while the sign value for the other pictures is set equal to -1. This means that only the pictures that were coded using the lowest quality would have a positive sign.

[0062] Line 2 determines a scale factor that depends on whether the slice that comprises the current NN block is coded in Inter mode or not. The scale factor becomes equal to 5 for Inter coded slices and 2 for Intra coded slices.

[0063] Line 3 determines the value of the qpOffset variable.

[0064] Line 4 determines the value of the QPpps variable, where the value of picInitQPMinus26 is decoded from the picture parameter set. For standardization work, the QPpps value can be seen as the QP value for one entire sequence since the same picInitQPMinus26 is used for all PPSs in a bitstream. In the standardization work, each sequence is encoded using a QPpps value equal to one of { 22, 27, 32, 37, 42 }.

[0065] Line 5 sets the value of QPslice, which is input to the NN. sliceQpDelta is a value that is set equal to the value of the sh qp delta syntax element in the slice header. In the standardization work, the sh qp delta syntax element is always set so that the QPslice value is equal to the single QP value that is used for all CTUs of one picture. The single QP value may vary for different pictures. For Inter pictures, the QPslice value used for a picture in the standardization work is the same for all pictures of the same temporal sublayer.

[0066] Line 6 determines the value of QPbase, which also is input to the NN. The value of prmld, decoded from the bitstream, determines whether qpOffset shall be added or not, where qpOffset may take one of the values { -5, -2, 2, 5}. The value 2 never occurs in standardization work since no picture of temporal ID larger than 3 contains any Intra slices.

[0067] In the standardization work, the QPslice values for two random-access examples using an Intra refresh interval of 32 (which is used for 30 fps input sequences) and QPpps values equal to 22 and 32 can be described as follows. Let POC of a picture denote the position in which the picture is output, with the position of the first output picture being 0, the position of the second output picture being 1, and so on. Then source code variable values for all pictures in the bitstream with particular POC values modulo 32 are as shown in Table 4 and Table 5 below.

[0068] Table 4 - Example with QPpps equal to 22

[0069] Table 5 - Example with QPpps equal to 32

[0070] In the standardization work, the training of the neural network is done by training data. The training data consists of sets of training elements, where each training element consists of one instance of the six inputs plus the source sample data. The NN that is being trained uses one instance as input, and weights of the NN is adjusted based on the output and the source sample data. Each instance of six inputs has been generated by encoding a training video sequence and storing the instance from the encoding. The instances used for random-access training were encoded using QPpps values equal to one of { 22, 27, 32, 37, 42 }, which is the same values that are used in the standardization work. The QPbase and QPslice values for an instance are derived using the source code above with prmID set equal to 0. This means that QPbase is set equal to QPpps during training, while QPslice is set according to e.g. Table 4 or Table 5.

[0071] In the standardization work, when NN filtering is turned on, the encoder has two options for each NN block, to signal prmld as equal to 0 or 1. For intra slices, QPbase can be set to either QPpps or QPpps-2. For inter slices, when the temporal layer is smaller than 4, QPbase can be set to either QPpps or QPpps-5. When the temporal layer is larger than or equal to 4 for an inter slice, QPbase can be set to either QPpps or QPpps+5. For each NN block during encoding, both available options are tried, and the encoder selects to signal the prmld value that gives the best rate-distortion performance.

[0072] NNVC 6.0 syntax and semantics

[0073] The focus of the description in this section is on the NN architecture in NNVC 6.0 when NN_HOP_UNIFIED=1 in the code. Note that there is no specification text containing the syntax and semantics shown in this section, this was all written by the authors of this disclosure.

[0074] Sequence parameter set

[0075] sps_nnlf_enabled_flag equal to 1 specifies that NN in-loop filtering is enabled for the CLVS. sps_nnlf_enabled_flag equal to 0 specifies that NN in-loop filtering is disabled for the CLVS.

[0076] sps_nnlf_set equal to 0 specifies that NN in-loop filtering set 0 may be used. sps_nnlf_set equal to 1 specifies that NN in-loop filtering set 1 may be used. sps_nnlf_set equal to 2 specifies that the LC NN in-loop filtering set may be used. sps_nnlf_set equal to 3 specifies that the unified NN in-loop filtering set may be used.

[0077] Slice header

[0078] slice nnlf hop mode equal to 0 specifies that no filtering is done for the slice, slice nnlf hop mode equal to 1 specifies that filtering is done with prmld equal to 0 for all NN blocks in the slice. slice_nnlf_hop_mode equal to 2 specifies that filtering is done with prmld equal to 1 for all NN blocks in the slice. slice_nnlf_hop_mode equal to 3 specifies that prmld is decoded for each NN block in the slice.

[0079] nn scaling mode specifies the mode for which reconstructed sample values after deblocking shall be combined with NN filtered output. The value of nn_scaling_mode shall be in the range of 0 to 8, inclusive.

[0080] Slice data (coding tree unit)

[0081] use_nnlf_hop equal to 0 specifies that no filtering is done for the NN block

[0082] use_first_param specifies that the CTUs in the NN block is filtered using prmld equal to (1 - use_first_param).

[0083] Details on nn scaling mode

[0084] nn scaling mode specifies the mode for which reconstructed sample values after deblocking shall be combined with NN filtered output. The value of nn_scaling_mode shall be in the range of 0 to 8, and the meaning of nn_scaling_mode for NNVC-6.0 is as follows:

[0085] The bold source code additions between “# position 1” and “# position 2” below was used to test whether some modes are redundant. The code swaps the scaleFlag value with a value that is believed to be equivalent (and the equivalency was confirmed by the test). • scaleFlag { -1, 1, 5 } are equivalent• scaleFlag { 2, 6 } are equivalent• scaleFlag { 3, 7 } are equivalent

[0086] The source code follows:#if NN HOP UNIFIED void HLSyntaxReader: :readHopScalingMode(NNFilterHOP: : SliceParameters& prms) const int inv_map_list[9] = { 1, 5, 0, 2, 6, 3, 7, 4, 8 };uint32_t code;READ_UVLC(code, "nn scaling mode");CHECK(code < 0 || code > 8, " Invalid scaling mode "); prms.scaleFlag = inv_map_list[code] - 1;# position 1 if (prms.scaleFlag >= 1 && prms.scaleFlag <= 3) prms.scaleFlag += 4; printf(" ## ++++4 ##"); else if (prms.scaleFlag >= 5 && prms.scaleFlag <= 7) prms.scaleFlag -= 4; printf(" ## -—4 ##"); else if (prms.scaleFlag == -1) prms.scaleFlag = 1; printf(" ## scaleFlag changed from -1 to +1 ##");# position 2#endif

[0087] The following table shows the decoded and used scaleFlag values for 4 Class D sequences coded at QP = {22, 27, 32, 37, 42}. The 20 bitstreams contained 9500 decoded pictures in total.

[0088] The next table shows the decoded and used scaleFlag values for 23 CTC sequences coded at QP = {22, 27, 32, 37, 42}. The 115 bitstreams contained 51,470 decoded pictures in total.

[0089] The bitstreams with prms.scaleFlag=O was BQTerrace QP 27 (3 codewords) and SlideShow QP 22 (3 codewords). In all 6 cases, y nnScale, cb nnScale, and cr_nnScale were all decoded to the value -240.

[0090] The values of slice nnlf hop mode was decoded as follows:SUMMARY

[0091] Certain challenges presently exist. A first problem with existing methods is that the sign of qpOffset depends on the comparison “(temporallD >= 4)”. This seems hardcoded for GOP sizes 32 and 64 with temporal sublayers being turned on. It would be possible for an encoder to put pictures into temporal layers such that the desired sign will be derived, but temporal layers restrict picture referencing such that pictures of a lower sublayer are not allowed to reference a higher sublayer picture. The dependency between the sign of the qpOffset and temporal layers therefore seems problematic.

[0092] A second problem is that in NNVC, implementors have to be aware that the value of sliceQpDelta controls the NNVC filtering and it may therefore not be optimal to set it to fit the QP to use for the first block of the slice.

[0093] A third problem is that the average QP value within one picture typically varies throughout a bitstream with higher average QP for difficult content such as sport and lower average QP for easy content such as news. Deriving QPbase from QPpps seems problematic in light of this variation, since either a single QPpps value must be suitable for many pictures, or multiple PPSs must be used, each with a different QPpps value. Using multiple PPSs may need a significant number of bits which is inefficient.

[0094] A fourth problem with the existing solution is that the method is tailored to a specific configuration, for instance the method depends on use of temporal layers which is not always the case. This makes the existing solution for other GoP structures and number of temporal layers suboptimal. This makes the current solution not desirable for other setups for instance configurations and use cases outside the CTC.

[0095] A fifth problem is that in the current solution QPbase and QPslice are single numbers which are expanded in their dimensionality to the same size as other inputs to the NN filter. On the other hand, the QP value for each block typically varies within one picture depending on the complexity of the block. This information is usually available to the decoder through the encoded bitstream. In the current solution, the information about this QP variation is not exploited. Hence the current solution using a single QP value for all the sample values to the NN loop filter is suboptimal.

[0096] A sixth problem with the current solution is that it cannot convey separate QP values to the NN for the different luma and chroma channels. Using separate QP values for the chroma channels is something that is typically supported by codecs, e.g. HEVC and VVC, and something that would be useful to have as input for a neural network.

[0097] Embodiments herein provide improvements on the existing methods, e.g., by conveying QP value and / or picture position information to a neural network.

[0098] Accordingly, in one aspect there is provided a method of decoding a video comprising a sequence of pictures using neural network-based video coding. The method includes deriving a first quantization parameter (QP) delta value from a first syntax element from a bitstream. The method includes determining a first QP value based on the first QP delta value. The method includes decoding a second QP delta value from a second syntax element from the bitstream. The method includes determining a second QP value based on the second QP delta value. The method includes passing the second QP value to a neural network as an input. The method includes using an output of the neural network to further decode the bitstream.

[0099] In another aspect there is provided a method of encoding a video comprising a sequence of pictures using a neural network-based video coding. The method includes deriving a first quantization parameter (QP) delta value from a first QP value. The method includes encoding the first QP delta value into a first syntax element in a bitstream. The method includes deriving a second QP delta value based on a second QP value. The methodincludes encoding the second QP delta value into a second syntax element in the bitstream. The method includes passing the second QP value to a neural network as an input. The method includes using an output of the neural network to further encode the bitstream.

[0100] In another aspect there is provided a method of decoding a video comprising a sequence of pictures using neural network-based video coding. The method includes deriving a first quantization parameter (QP) delta value from a first syntax element from a bitstream. The method includes determining a first QP value based on the QP delta value. The method includes decoding a second QP delta value from a second syntax element from the bitstream. The method includes determining whether to use a second QP value or a third QP value for a neural network. The method includes deriving the second QP value and / or the third QP value based on the second QP delta value, wherein the second QP value is equal in magnitude to the third QP value and of an opposite sign. The method includes passing the second QP value and / or the third QP value to the neural network as an input. The method includes using an output of the neural network to further decode the bitstream.

[0101] In another aspect there is provided a method of decoding a video comprising a sequence of pictures using neural network-based video coding. The method includes deriving a first quantization parameter (QP) delta value from a first syntax element from a bitstream. The method includes determining a first QP value based on the first QP delta value. The method includes decoding a second QP value from a second syntax element in the bitstream. The method includes decoding a third QP value from a third syntax element in the bitstream. The method includes selecting whether to use the second QP value or the third QP value for a neural network. The method includes passing the selected second or third QP value to the neural network as an input. The method includes using an output of the neural network to further decode the bitstream.

[0102] In another aspect there is provided a method of encoding a video comprising a sequence of pictures using a neural network-based video coding. The method includes obtaining a first quantization parameter (QP) value. The method includes encoding the first QP value into a first syntax element in a bitstream. The method includes obtaining a second QP value and a third QP value. The method includes encoding the second QP value into a second syntax element in the bitstream. The method includes encoding the third QP value into a third syntax element in the bitstream. The method includes determining whether to use the second QP value or the third QP value for the neural network The method includespassing the selected QP value to the neural network as an input. The method includes using an output of the neural network to further encode the bitstream.

[0103] In another aspect there is provided a method of decoding a video comprising a sequence of pictures using neural network-based video coding. The method includes decoding an offset value V from a syntax element in a bitstream. The method includes deriving a first QP value QP1 for a first block from the bitstream. The method includes deriving a second QP value QP2 for a second block from the bitstream. The method includes assigning the value QP1+V to at least one entry of a matrix M. The method includes assigning the value QP2+V to at least one entry of the matrix M. The method includes passing the matrix M to a neural network as an input. The method includes using an output of the neural network to further decode the bitstream.

[0104] In another aspect there is provided a method of decoding a video comprising a sequence of pictures using neural network-based video coding. The method includes determining a first quantization parameter (QP) value based on decoding one or more syntax elements from a bitstream, wherein the first QP value is for the first chroma channel. The method includes determining a second QP value based on decoding one or more syntax elements from the bitstream, wherein the second QP value is for a luma channel. The method includes passing the first and second QP values to a neural network as an input. The method includes using an output of the neural network to further decode the bitstream.

[0105] In another aspect there is provided a method of encoding a video comprising a sequence of pictures using a neural network-based video coding. The method includes deriving a first quantization parameter (QP) delta value from a first QP value, wherein the first QP value is for a first chroma channel. The method includes encoding the first QP delta value into a first syntax element in a bitstream. The method includes deriving a second QP delta value based on a second QP value, wherein the second QP value is for a luma channel. The method includes encoding the second QP delta value into a second syntax element in the bitstream. The method includes passing the first and second QP values to a neural network as an input. The method includes using an output of the neural network to further encode the bitstream.

[0106] In another aspect there is provided a method of encoding a video comprising a sequence of pictures using a neural network-based video coding. The method includes deriving a first quantization parameter (QP) delta value from a first QP value. The methodincludes encoding the first QP delta value into a first syntax element in a bitstream. The method includes deriving a first position value. The method includes encoding the first position value into a second syntax element in the bitstream. The method includes passing the first QP value and the first position value to a neural network as an input. The method includes using an output of the neural network to further encode the bitstream.

[0107] In another aspect there is provided a computer program comprising instructions which when executed by processing circuitry of an apparatus causes the apparatus to perform any of the methods disclosed herein. In one embodiment, there is provided a carrier containing the computer program wherein the carrier is one of an electronic signal, an optical signal, a radio signal, and a computer readable storage medium.

[0108] Advantages of embodiments disclosed herein include improved compression efficiency, reliability, and flexibility. Embodiments improve on at least one or more of the above-identified problems. For example, the first, second, third, and fourth problem described above are improved on by replacing the current derivation of QPbase and QPslice used for input to a neural network by novel derivation methods. As another example, by providing different QP values for the chroma channels to the NN, the NN can be better finetuned for different combinations of QP values. In this way, a decoder need not rely on the same offset always being used between luma QP and chroma QP in a codec. As another example, by using other means to describe the hierarchy of a picture in the bitstream to the NN other than using the temporal ID as used in existing methods, embodiments provide better reliability and flexibility. Since the temporal ID need not be followed by the encoder, having it as input to the NN can be misleading in many cases. Using a specific parameter for describing the pictures position in the GOP hierarchy in embodiments provides better stability and may also be set independently to better tune the NN. As another example, deriving QPbase from the QP of a reference picture instead of from the PPS QP, may provide better results as it gives a better level of detail of the input used for producing the current picture.BRIEF DESCRIPTION OF THE DRAWINGS

[0109] The accompanying drawings, which are incorporated herein and form part of the specification, illustrate various embodiments.

[0110] FIG. 1 illustrates a system according to an embodiment.

[0111] FIG. 2 is a block diagram of an encoder according to an embodiment.

[0112] FIG. 3 is a block diagram of a decoder according to an embodiment.

[0113] FIG. 4 is a flowchart illustrating a process according to an embodiment.

[0114] FIG. 5 is a flowchart illustrating a process according to an embodiment.

[0115] FIG. 6 is a flowchart illustrating a process according to an embodiment.

[0116] FIG. 7 is a flowchart illustrating a process according to an embodiment.

[0117] FIG. 8 is a flowchart illustrating a process according to an embodiment.

[0118] FIG. 9 is a flowchart illustrating a process according to an embodiment.

[0119] FIG. 10 is a flowchart illustrating a process according to an embodiment.

[0120] FIG. 11 is a flowchart illustrating a process according to an embodiment.

[0121] FIG. 12 is a flowchart illustrating a process according to an embodiment.

[0122] FIG. 13 is a flowchart illustrating a process according to an embodiment.

[0123] FIG. 14 is a block diagram of an apparatus according to an embodiment.

[0124] FIG. 15 is a chart illustrating relationships between the PPS, SPS, VPS,Slices, and picture headers.

[0125] FIG. 16 illustrates the structure of the unified filter.

[0126] FIG. 17A illustrates a 128x128 matrix and four block QP values (64x64 entries assigned to each block QP value) according to an embodiment.

[0127] FIG. 17B illustrates a 128x128 matrix to input to the NN according to an embodiment.DETAILED DESCRIPTION

[0128] A first aspect of provided embodiments is to send the value of qpOffset in the bitstream instead of deriving it from the variables intra and temporallD.

[0129] A second aspect of provided embodiments is to send what offset values to use. One example is to be able to let prmld select between qpOffset =-l and qpOffset =+l and not always have one qpOffset option being the qpOffset=0 option.

[0130] In existing methods there are two options, qpOffset equal to 0 or equal to the derived offset. A third aspect of provided embodiments is to have more than two options. One example on the encoder side is to search broadly in the first picture to decide whatoption values to use, and then keep using the same option values without complex searches in future pictures.

[0131] In one embodiment, qpOffset is associated with the weighting between deblocking filter and NN filter or the nn_scaling_mode. For example, in NNVC-6.0, when NN is enabled, the weighting of NN filter can be derived or chosen from 100%, 75%, 50%, and nn scaling mode is used to represent the modes of weighting. Then, qpOffset can take different values depending on the weighting or the values of nn_scaling_mode.

[0132] A fourth aspect of the provided embodiments is to allow some offset also to QPslice. In one embodiment, there is a another offset “qpOffset2” applied to QPslice. This allows the QPslice to be adjusted. The value of qpOffset2 may be obtained following the methods described in the disclosed embodiments. The value of qpOffset2 may be derived or signalled. In one embodiment, the derivation of qpOffset and qpOffset2 are independent. The derivation of qpOffset2 is not related to the existence or value of qpOffset. In another embodiment, the existence of qpOffset2 depends on the value of qpOffset. For example, only when qpOffset is non-zero, qpOffset2 exists and the value of qpOffset2 is derived.

[0133] A fifth aspect of provided embodiments is to derive QP addition steps based on QP. In this aspect, QP variation is set as a function of (e.g. proportional to) one or a combination of one or more parameters, either already in the bitstream or newly signalled. Here are two examples in this context:• Set the steps of the QP additions proportional to the QP.• Set the steps of the QP additions equal to f 1 (QP) if QP is smaller than a threshold and equal to f2(QP) if QP is larger than a threshold.

[0134] A sixth aspect of provided embodiments is to derive QP addition steps from the distance to an intra coded frame, for instance by setting the QP additions proportional to the distance of the picture from the closest I frame.

[0135] A seventh aspect of provided embodiments is to increase the QP input resolution. It seems like using a scalar input as QP and then expanding it to the size of a block is a waste of resources. A higher resolution QP input seems more informative to the NN. Two embodiments in the context of increasing the QP input resolution are:Increase the QP resolution either by a LUT / function / mapping or a separate NN that could take inputs such as the scalar QP, BS, etc. and output per pixel QP.Increase the QP resolution by inputing the current QP of the pixels to the NN (signal the slice QP for the first block and the delta QP for every following block in the CTU). Potentially no extra signalling.

[0136] A eighth aspect of provided embodiments is to use individual QP values for the chroma channels. In one embodiment, individual QP values are input to the NN for the chroma channels. The chroma QP values may be the same as derived from the signaled chroma QP offsets in the bitstreams, or it may additionally be derived from a chroma QP offset specific to be used for the NN.

[0137] A ninth aspect of provided embodiments is to use the current picture position in the GOP hierarchy for deriving the input parameters to the NN. In one embodiment, a specific parameter for describing the current picture's position in the GOP hierarchy is used for deriving the input parameters instead of using the temporal ID. In another version, QPbase is derived from the QP of a reference picture. The reference picture may be a reference picture directly referenced from the current CU. The reference picture may also be the lowest picture in the hierarchy, e.g. the first picture in the GOP. In yet another version, a new input QPblock is added to the NN, in addition to the inputs QPbase and QPslice. QPblock may vary from block to block due to the use of delta QP, whereas QPbase and QPslice may be constant over the block.

[0138] FIG. 1 illustrates a system 100 according to an embodiment. System 100 includes an encoder 102 and a decoder 104. In some embodiments, encoder 102 is in communication with decoder 104 via a network 110 (e.g., the Internet or other network). Encoder 102 encodes a source video 101, produced by a camera 199, into a bitstream comprising an encoded video and may transmit the bitstream to decoder 104 via network 110. In some embodiments, encoder 102 is not in communication with decoder 104, and, in such an embodiment, rather than transmiting bitstream to decoder 104, the bitstream is stored in a data storage unit 190 and decoder 104 retrieves the bitstream from data storage unit 190.

[0139] Decoder 104 decodes the pictures included in the encoded video to produce video data for display and / or further image processing (e.g. a machine vision task, such as generating a 3D of an object-of-interest (OOI) in the pictures of the video). Accordingly, decoder 104 may be part of a device 103 having an image processor 105 and / or a display 106. The image processor 105 may perform machine vision tasks on the decoded pictures. One such machine vision task may be locating an object in the picture and creating a 3D model ofthe object. The device 103 may be a mobile device, a set-top device, a head-mounted display, or any other device.

[0140] FIG. 2 illustrates functional components of encoder 102 according to some embodiments. It should be noted that encoders may be implemented differently so implementation other than this specific example can be used.

[0141] As shown in FIG. 2, encoder 102 employs a subtractor 241 to produce a residual block which is the difference in sample values between an input block provided by the source video and a prediction block (i.e., the output of a selector 251, which is either an inter prediction block output by an inter predictor 250 (a.k.a., motion compensator) or an intra prediction block output by an intra predictor 249). Then a forward transform 242 is performed on the residual block to produce a transformed block comprising transform coefficients. A quantization unit 243 quantizes the transform coefficients based on a quantization parameter (QP) value (e.g., a QP value obtained based on a picture QP value for the picture in which the input block is a part and a block specific QP offset value for the input block), thereby producing quantized transform coefficients which are then encoded into the bitstream by encoder 244 (e.g., an entropy encoder) and the bitstream with the encoded transform coefficients is output from encoder 102. Next, encoder 102 uses the quantized transform coefficients to produce a reconstructed block. This is done by first applying inverse quantization 245 and inverse transform 246 to the transform coefficients to produce a reconstructed residual block and using an adder 247 to add the prediction block to the reconstructed residual block, thereby producing the reconstructed block, which is stored in the reconstruction picture buffer (RPB) 266. Loop filtering by a loop filter (LF) stage 267 is applied and the final decoded picture is stored in a decoded picture buffer (DPB) 268, where it can then be used by the inter predictor 250 to produce an inter prediction block for the next picture to be processed. LF stage 267 may include three sub-stages: i) a deblocking filter, ii) a sample adaptive offset (SAG) filter, and iii) an Adaptive Loop Filter (ALF).

[0142] FIG. 3 illustrates functional components of decoder 104 according to some embodiments. It should be noted that decoder 104 may be implemented differently so implementations other than this specific example can be used. Decoder 104 includes a decoder module 361 (e.g., an entropy decoder) that decodes from the bitstream quantized transform coefficient values of a block. Decoder 104 also includes a reconstruction stage 398 in which the quantized transform coefficient values are subject to an inverse quantization process 362 and inverse transform process 363 to produce a residual block.This residual block is input to adder 364 that adds the residual block and a prediction block output from selector 390 to form a reconstructed block. Selector 390 either selects to output an inter prediction block or an intra prediction block. The reconstructed block is stored in a RPB 365. The inter prediction block is generated by the inter prediction module 350 and the intra prediction block is generated by the intra prediction module 369. Following the reconstruction stage 398, a loop filter stage 367 applies loop filtering to the reconstructed picture or block and the final decoded, reconstructed picture may be stored in a decoded picture buffer (DPB) 368 and output to image processor (IP) 105 (also may be output to display 106). Pictures are stored in the DPB for two primary reasons: 1) to wait for picture output and 2) to be used for reference when decoding future pictures.

[0143] Additional Examples

[0144] Example 1 - A novel syntax element to signal QP values to an NN.

[0145] In this example, the value of qpOffset is conveyed by a syntax element in the bitstream instead of it being derived from the slice type and temporal ID as in the prior art.

[0146] In a first variant, the derivation of sliceQpDelta is replaced by a novel delta QP syntax element sh qp offset nn as shown in the syntax table below.

[0147] In this syntax table, the sh qp delta syntax element may be used to determine the value of SliceQpY as previously described for the existing methods:SliceQpY = 26 + pps_init_qp_minus26 + sh_qp_delta

[0148] The novel syntax element, sh qp offset nn, is used to derive a QP value to use as input to the neural network. Two variants are shown below. In the first, sh qp offset nn is an offset to QPpps and in the second, sh qp offset is an offset toQPpps+sliceQpDelta. (As in the existing methods, sliceQpDelta is in both variants equal to the value of sh qp delta or ph qp delta.)

[0149] Variant 1

[0150] Variant 2

[0151] A decoder may perform one or more of the following steps in this example to derive a value to input to a neural network:1) Derive a first QP delta value from a first syntax element (sh qp delta or ph qp delta) from the bitstream.2) Determine a first QP value (QPslice or similar) from the QP delta value. 3) Decode a set of quantized coefficient values from the bitstream.4) Derive a set of dequantized coefficient values from the set of quantized coefficient values and the first QP value (QPslice).5) Decode a second QP delta value from a second syntax element (sh qp offet nn) from the bitstream.6) Determine a second QP value V (QPbase or similar) from the second QP delta value.7) Input the second QP value V to the neural network.

[0152] The determining of the second QP value in Step 6 may alternatively be done from both the first and second QP delta values, for example, as shown in the second variant above.

[0153] An encoder may perform the following steps in this example to encode an input value to a neural network into a bitstream:1) Derive a first QP delta value from a first QP value (QPslice or similar).2) Encode the first QP delta value into a first syntax element (sh qp delta or ph qp delta) in the bitstream.3) Decode a set of quantized coefficient values from the bitstream.4) Derive a set of dequantized coefficient values from the set of quantized coefficient values and the first QP value (QPslice).5) Derive a second QP delta value from a second QP value V (QPbase or similar).6) Encode the second QP delta value into a second syntax element (sh qp offet nn) in the bitstream.7) Input the second QP value V to the neural network.

[0154] The derivation of the second QP delta value in Step 5 may alternatively be done from both the first and second QP values.

[0155] The second syntax element may be located in a slice data, a slice header, a picture header or a parameter set such as an SPS, a PPS, or an APS.

[0156] In one variant, there is one second syntax element for each NN block. This may result in multiple second syntax elements for a slice or picture. The multiple syntax elements may be located in a slice data, a slice header, a picture header or a parameter set such as an SPS, a PPS, or an APS. In some variants, the number of NN blocks may first beobtained or derived, and then this number of second syntax elements are decoded by a decoder, or encoded into a bitstream by an encoder.

[0157] Here, a slice contains a slice header part (in VVC this corresponds to the slice_header() syntax structure) and a slice data part (in VVC this corresponds to the slice_data() syntax structure).

[0158] Example 2 - Use of both a positive and negative QP value.

[0159] In an alternative of the previous example, the second syntax element provides two options. One value V is signalled, and both +V and -V are used. An example is shown in the syntax table below:

[0160] For example, if sh abs qp offset nn minusl is equal to 2, then values +3 and -3 are both provided.

[0161] A decoder may perform the following steps in this embodiment to derive a value to input to a neural network:1) Derive a first QP delta value from a first syntax element (sh qp delta or ph qp delta) from the bitstream.2) Determine a first QP value (QPslice or similar) from the QP delta value.3) Decode a set of quantized coefficient values from the bitstream.4) Derive a set of dequantized coefficient values from the set of quantized coefficient values and the first QP value (QPslice).5) Decode a second QP delta value from a second syntax element (sh abs nn minusl) from the bitstream.6) Determine whether to use a second QP value (V2) or a third QP value (V3) for a neural network.7) Derive V2 and / or V3 from the second QP delta value wherein V2 is equal to -V3.8) Input the value of V2 or V3 to the neural network.

[0162] Example 3 - Signaling multiple QP values to use for an NN.

[0163] In this example, two or more QP values are signalled using two or more corresponding novel syntax elements. Each novel syntax element provides at least one QP value as an option to use for the NN. The novel syntax elements may be located in a slice header, a picture header, or a parameter set such as an SPS, a PPS, or an APS.

[0164] The syntax table below shows a PPS example with two novel syntax elements, but more than two could also be used.

[0165] In one version of this example, the number of novel syntax elements are not fixed. Instead, there may be a preceding syntax element that specifies the number of novel syntax elements as shown below.

[0166] A decoder may perform the following steps in this embodiment to derive a value to input to a neural network.1) Decode a second QP value from a second syntax element (qp offet nnfO]) in the bitstream.2) Decode a third QP value from a third syntax element (qp_offet_nn[l]) in the bitstream.3) Select whether to use the second QP value or the third QP value for a neural network4) Input the selected QP value V to the neural network.

[0167] The selecting step 3 may be done by decoding a value from a fourth syntax element where the value specifies whether to use the second or third QP value. As an example, the second and third syntax elements may be located in a parameter set such as an SPS, a PPS, an APS, etc., and the fourth syntax element may be located in a slice (slice header or slice data).

[0168] An encoder may perform the following steps in this embodiment to encode an input value to a neural network into a bitstream.1) Obtain a second QP value and a third QP value.2) Encode the second QP value into a second syntax element in the bitstream.3) Encode the third QP value into a third syntax element in the bitstream.4) Determine whether to use the second QP value or the third QP value for the neural network.5) Input the selected QP value V to the neural network.

[0169] This encoder method may additionally comprise encoding a fourth syntax element into the bitstream where the value of the fourth syntax element specifies whether to use the second or third QP value for the NN.

[0170] The following table shows the QPbase options of existing methods and some of the disclosed embodiments:

[0171] Example 4 - Input a matrix of QP values to a neural network.

[0172] In this example, which may be combined with other examples disclosed herein, the input to the neural network that represents the QP value or values is a matrix. Thematrix is derived by adding the input value V derived as described by previous embodiments to each entry of the matrix.

[0173] In one variant of this example, the values in the matrix are derived in three steps. In the first step, the input value V is derived as shown in other embodiments. In the second step, the entries are assigned to the QP value that was used for encoding or decoding the corresponding sample values or corresponding block of sample values. In the third step, the input value V is added to each element of the matrix.

[0174] As an example, let’s assume that there is one slice in a current picture and that the novel syntax element located in the slice header as described in example 1. Assume further that the input to the NN comprises an NN block of 128x128 luma samples and that each of the 64x64 luma blocks of the NN block is assigned an individual block QP value.

[0175] In the second step, the slice header is decoded and the input value V is derived from the decoded value of the novel syntax element.

[0176] In the second step, the current picture is decoded without applying NN filtering. This involves decoding each of the four block QP values for the four 64x64 blocks, let’s call them QP0, QP1, QP2, and QP3, as illustrated below. The values QPO, QP1, QP2, QP3 are then assigned to entries of a matrix. The matrix may have the size 128x128 to match the size of the NN block, and the values are assigned such that the entry values matches to corresponding block QP value. This means that the entries in the matrix has the values illustrated in FIG. 17A. FIG. 17A illustrates a 128x128 matrix and four block QP values (64x64 entries assigned to each block QP value).

[0177] In the third step, the input value V is added to all 128*128 entries in the matrix as illustrated in FIG. 17B. FIG. 17B illustrates a 128x128 matrix to input to the NN.

[0178] The matrix is then input to the NN.

[0179] A decoder may perform the following steps in this embodiment to derive a matrix to input to a neural network.1) Decode an offset value V from a syntax element in the bitstream.2) Derive a first QP value QP1 for a first block from the bitstream.3) Derive a second QP value QP2 for a second block from the bitstream.4) Assign the value QP1+V to at least one entry of a matrix M.5) Assign the value QP2+V to at least one entry of the matrix M.6) Input the matrix M to a neural network.

[0180] Optionally, the method may additionally include using the first QP value QP1 in an inverse quantization process for the first block and using the second QP value QP2 in an inverse quantization process for the second block.

[0181] Example 5 - Chroma QP.

[0182] In this example, a QP value is also provided to the NN for the chroma channels, where the QP value is different from the luma QP value. The chroma QP value may be the same for all chroma channels or there may be individual chroma QP values for each chroma channel, e.g., one chroma QP value for Cb and one chroma QP value for Cr.

[0183] In one version, the chroma QP value is derived from a QP offset value where the chroma QP is offset from a luma QP value. In another version, the chroma QP value for a second chroma channel, e.g., Cr, is derived from a chroma QP value of a first chroma channel, e.g., Cb. The chroma QP offset may for instance be signaled in a parameter set such as SPS or PPS or in a header such as a slice header or a picture header.

[0184] In another version, an offset for at least one of the chroma channels specific for the NN is signaled in the bitstream. This is illustrated in the syntax table below for signalling this in the slice header. For simplicity, the example only shows the values for Cb, but it is to be understood that the corresponding values for Cr could be signaled and / or derived in a similar way.

[0185] sh cb qp offset is an offset used for deriving the chroma QP value for the Cb chroma channel, QP'Cb.

[0186] sh cb qp nn offset is an offset to a QP value to be used to derive a chroma QP value for Cb, QPnnCb, that is to be provided to the NN. In one version, QPnnCb is derived from the derived chroma QP value QP'Cb and the sh cb qp nn offset, e.g., QPnnCb = QP'Cb + sh cb qp offset.

[0187] In one version, the value of sh cb nn qp offset is different from the value of sh cb qp offset, wherein both sh cb nn qp offset and sh cb qp offset are non-zero values.

[0188] In another version, QPnnCb is derived from the derived luma QP value and the sh cb qp nn offset, e.g., QPnnCb = QP'luma + sh cb qp nn offset.

[0189] Example 6 - Value for indicating the current picture’s position in the GOP hierarchy.

[0190] As described above, temporallD may not be the best parameter to use to derive the input to the NN. In this example, a new value, posInHierarchy, is signaled in the bitstream, e.g., in the picture header or slice header, to be used to indicate to the NN the current picture’s position in the GOP hierarchy.

[0191] In one version, the posInHierarchy describes a value in a range. E.g., if the range is [0, 10], the posInHierarchy may be 3. In one version, the posInHierarchy is directly input to the NN. In one version, the posInHierarchy is used to derive a weight or an offset to an input to the NN. In one version a lower value of posInHierarchy results in a larger weight or offset compared to a higher value of posInHierarchy. In another version a lower value of posInHierarchy results in a smaller weight or offset compared to a higher value of posInHierarchy. In one version, at least a part of the range (e.g. max value) is signaled in the bitstream.

[0192] In one version, the posInHierarchy may be signalled in the bitstream with different precision, e.g. a syntax element may first be signalled specifying the number of bits to decode for the posInHierachy value.

[0193] Compared to using the temporallD for the derivation of the input parameters, a designer of the NN may alter this parameter to fine-tune the results of the NN without having to change the temporallD value, which is primarily used for describing the temporal sublayers and indicate what pictures that may be removed from the bitstream.

[0194] Example 7 - Derive QPbase from the QP of a reference picture.

[0195] In existing methods, QPbase is derived from the PPS QP value. In this example, Qpbase is instead (or in addition) derived from one or more QP values of one or more reference pictures referenced by the current picture. Alternatively, another parameter than QPbase is derived from one or more QP values of one or more reference pictures referenced by the current picture and input to the NN. In one version the one or more reference pictures are one or both of the reference pictures used by the current CU.

[0196] In one version the reference picture for which the QP is derived from is the lowest picture in the GOP hierarchy, e.g. the first picture in the GOP. The lowest picture in the GOP hierarchy may also refer to the reference picture with the lowest QP, the reference picture with the lowest POC value or the reference picture that is first in decoding order.

[0197] The QP value of a reference picture may for instance be the derived slice QP of the picture, or the average used QP of the picture. In one version, the QP value of the reference picture is here the CU QP value of CU in the referenced picture that is collocated with the current CU of the current picture.

[0198] Example 8 - QP addition steps as a function of QP.

[0199] This example defines QP addition steps as a function of (e.g. proportional to) one or a combination of one or more parameters. A QP addition step is a value added to a given QP to prepare the input QP to the NN model. In a first version of this example, the QP addition steps are derived as a function of the QP. In one example, QP is the QPpps and the QP addition steps is the term based on the QPpps which is added to QPpps to obtain QPbase which will be input to the NN:QPbase = QPpps + f(QPpps)

[0200] In this example, the derivation of the QP addition steps may also be a function of one or more other parameters in the bitstream, shown by “param” here:QPbase = QPpps + f(QPpps, param)

[0201] In an embodiment, the QP addition steps is a first order polynomial function of the QP:QPbase = QPpps + a * QPpps + b where a and b are provided to the decoder, for instance, by signalling, by a configuration, or by other means.

[0202] In a second embodiment, QP addition steps only scale the QP:QPbase = QPpps + a * QPpps

[0203] In a variant, the function used for deriving the QP addition steps is a piecewise function. In one example, the piecewise function has different subfunctions each applied to a specific QP interval. The QP interval may be specified with threshold value(s) T1 for instance.

[0204] The table below summarises how the QPbase is derived in this embodiment:

[0205] Table 6

[0206] Example 9 - QP based on the distance to the closest I frame.

[0207] In this example, the QP input is a function of the distance of the current picture to the closest intra coded frame. In one variant, the distance to the closest intra coded frame is calculated as a POC difference. In another variant, the distance to the closest intracoded frame is calculated as minimum prediction steps the current picture is away from the closest intra coded frame. The two last rows in the table below give an example of distance values for the first and second variants for the GOP structure in Table 3.

[0208] Example 10 - Increasing the QP input resolution.

[0209] In this example, the QP input to the NN filter has originally a higher resolution than one contrary to the existing methods where one scalar value is scaled up to the size of other inputs to the neural network.

[0210] In one version of this example, the input QP to the neural network is the current QP of the pixels to the neural network. In one embodiment, one CTU is the input block to the neural network and the input QP to the neural network for the first block consistsof the slice QP of the slice the CTU belongs to and for the following blocks in the CTU the QP is updated by the delta QP for every following block in the CTU.

[0211] In a second version, the single QP input to the network is upsampled using a look-up-table or a function that takes the single QP and at least one other parameters such as the boundary strength as input.

[0212] Example 11 - Add offset to QPslice.

[0213] In one embodiment, there is another offset “qpOffset2” applied to QPslice.This allows the QPslice to be adjusted. Given a QPbase, if qpOffset2 is larger than 0, QPslice increases and the gap between QPbase and QPslice is larger, which could mean that the network is treated with a current picture / slice that has a larger temporal distance to the intra slice and the quality needs to be reduced. On the other hand, if qpOffset2 is smaller than 0, QPslice decreases and the gap between QPbase and QPslice is smaller, which could mean that the network is treated with a current picture / slice that has a smaller temporal distance to the intra slice and the quality of the current picture / slice needs to be improved. The value of qpOffset2 may be obtained following the methods described in the disclosed embodiments. The value of qpOffset2 may be derived or signalled.

[0214] In one version, the derivation of qpOffset and qpOffset2 are independent. The derivation of qpOffset2 is not related to the existence or value of qpOffset.

[0215] In another version, the existence of qpOffset2 depends on the value of qpOffset. For example, only when qpOffset is non-zero, qpOffset2 exists and the value of qpOffset2 is derived.

[0216] Example 12

[0217] In one embodiment, qpOffset is associated with the weighting between deblocking filter and NN filter or the nn_scaling_mode.

[0218] For example, in NNVC-6.0, when NN is enabled, the weighting of NN filter can be derived or chosen from 100%, 75%, 50%, and nn_scaling_mode is used to represent the modes of weighting. nn_scaling_mode have a range between 0 to 8.

[0219] When the weighting is derived, it is not limited to a value from 100%, 75%, 50% and can have more values than the range of nn_scaling_mode.

[0220] Then, qpOffset can take different values depending on the weighting value or the values of nn_scaling_mode.

[0221] Example 13

[0222] In one embodiment, a novel input QPblock is added to the inputs of the neural network. This can be in addition to QPbase and QPslice. QPblock may have different values over the inputs; if the input is of size 144x144, not all values need to have the same value. This can be advantageous in the case where deltaQP is used in the input. The resulting QP actually used in each block is available in the decoder and can be used as the QPblock input to the neural network.

[0223] Summary of Various EmbodimentsAl. A method 400 (see FIG. 4) of decoding a video comprising a sequence of pictures from a bitstream using neural network-based video coding, the method comprising: deriving (s402) a first quantization parameter (QP) delta value from a first syntax element from the bitstream; determining (s404) a first QP value based on the first QP delta value; decoding (s406) a second QP delta value from a second syntax element from the bitstream; determining (s408) a second QP value based on the second QP delta value; passing (s410) the second QP value to a neural network as an input; and using (s412) an output of the neural network to further decode the bitstream.A2. The method of embodiment Al, further comprising: decoding a set of quantized coefficient values from the bitstream; andderiving a set of dequantized coefficient values based on the set of quantized coefficient values and the first QP value.A3. The method of any one of embodiments A1-A2, wherein determining a second QP value based on the second QP delta value is further based on the first QP delta value.A4. The method of any one of embodiments Al -A3, wherein the second syntax element is located in one of: a slice data, a slice header, a picture header, and a parameter set.A5. The method of any one of embodiments A1-A4, wherein the second syntax element is variable for each block.A6. The method of any one of embodiments A1-A5, wherein first and second QP values are for a chroma channel, and additional QP values for a luma channel are decoded from the bitstream and passed to the neural network.A7. The method of any one of embodiments A1-A6, wherein the second QP value is further based on a reference QP value of a reference picture.A8. The method of any one of embodiments A1-A7, wherein the second QP value is further based on a function of a picture parameter set (PPS) QP value.A9. The method of embodiment A8, wherein the function of a PPS QP value is a polynomial function.A10. The method of any one of embodiments A1-A9, wherein the second QP value is further based on a distance to the closest intra-coded frame.All. The method of any one of embodiments A1-A10, wherein the second QP value is further based on a value decoded from the bitstream that specifies how reconstructed sample values after deblocking shall be combined with NN filtered output (e.g., nn_scaling_mode).A12. The method of any one of embodiments Al-All, wherein passing the second QP value to a neural network as an input further comprises passing a QPblock as input to the neural network.Bl. A method 500 (see FIG. 5) of encoding a video comprising a sequence of pictures to a bitstream using a neural network-based video coding, the method comprising: deriving (s502) a first quantization parameter (QP) delta value from a first QP value; encoding (s504) the first QP delta value into a first syntax element in the bitstream; deriving (s506) a second QP delta value based on a second QP value; encoding (s508) the second QP delta value into a second syntax element in the bitstream; passing (s510) the second QP value to a neural network as an input; and using (s512) an output of the neural network to further encode the bitstream.B2. The method of embodiment Bl, further comprising: encoding a set of quantized coefficient values from the bitstream. deriving a set of dequantized coefficient values from the set of quantized coefficient values and the first QP value (QPslice).B3. The method of any one of embodiments B1-B2, wherein deriving a second QP delta value based on the second QP value is further based on the first QP value.B4. The method of any one of embodiments B1-B3, wherein the second syntax element is located in one of: a slice data, a slice header, a picture header, and a parameter set.B5. The method of any one of embodiments B1-B4, wherein the second syntax element is variable for each block.B6. The method of any one of embodiments B1-B5, wherein first and second QP values are for a chroma channel, and additional QP values for a luma channel are encoded into the bitstream and passed to the neural network.B7. The method of any one of embodiments B1-B6, wherein the second QP value is further based on a reference QP value of a reference picture.B8. The method of any one of embodiments B1-B7, wherein the second QP value is further based on a function of a picture parameter set (PPS) QP value.B9. The method of embodiment B8, wherein the function of a PPS QP value is a polynomial function.BIO. The method of any one of embodiments B1-B9, wherein the second QP value is further based on a distance to the closest intra-coded frame.Bl 1. The method of any one of embodiments Bl -BIO, wherein the second QP value is further based on a value decoded from the bitstream that specifies how reconstructed sample values after deblocking shall be combined with NN filtered output (e.g., nn_scaling_mode).Bl 2. The method of any one of embodiments Bl-Bl 1, wherein passing the second QP value to a neural network as an input further comprises passing a QPblock as input to the neural network.Cl. A method 600 (see FIG. 6) of decoding a video comprising a sequence of pictures from a bitstream using neural network-based video coding, the method comprising: deriving (s602) a first quantization parameter (QP) delta value from a first syntax element from the bitstream; determining (s604) a first QP value based on the QP delta value; decoding (s606) a second QP delta value from a second syntax element from the bitstream; determining (s608) whether to use a second QP value or a third QP value for a neural network; deriving (s610) the second QP value and / or the third QP value based on the second QP delta value, wherein the second QP value is equal in magnitude to the third QP value and of an opposite sign; passing (s612) the second QP value and / or the third QP value to the neural network as an input; and using (s614) an output of the neural network to further decode the bitstream.C2. The method of embodiment Cl, further comprising: decoding a set of quantized coefficient values from the bitstream; deriving a set of dequantized coefficient values from the set of quantized coefficient values and the first QP value.C3. The method of any one of embodiments C1-C2, wherein first, second, and third QP values are for a chroma channel, and additional QP values are provided for a luma channel.C4. The method of any one of embodiments C1-C3, wherein the second and / or third QP value is further based on a reference QP value of a reference picture.C5. The method of any one of embodiments C1-C4, wherein the second QP value is further based on a function of a picture parameter set (PPS) QP value.C6. The method of embodiment C5, wherein the function of a PPS QP value is a polynomial function.C7. The method of any one of embodiments C1-C6, wherein the second QP value is further based on a distance to the closest intra-coded frame.C8. The method of any one of embodiments C1-C7, wherein the second QP value is further based on a value decoded from the bitstream that specifies how reconstructed sample values after deblocking shall be combined with NN filtered output (e.g., nn_scaling_mode).C9. The method of any one of embodiments C1-C8, wherein passing the second QP value to a neural network as an input further comprises passing a QPblock as input to the neural network.DI. A method 700 (see FIG. 7) of decoding a video comprising a sequence of pictures from a bitstream using neural network-based video coding, the method comprising: deriving (s702) a first quantization parameter (QP) delta value from a first syntax element from the bitstream;determining (s704) a first QP value based on the first QP delta value; decoding (s706) a second QP value from a second syntax element in the bitstream. decoding (s708) a third QP value from a third syntax element in the bitstream. selecting (s710) whether to use the second QP value or the third QP value for a neural network; passing (s712) the selected second or third QP value to the neural network as an input; and using (s714) an output of the neural network to further decode the bitstream.D2. The method of embodiment DI, wherein selecting whether to use the second QP value or the third QP value for a neural network is based on decoding a value from a fourth syntax element where the value specifies whether to use the second or third QP value.D3. The method of any one of embodiments D1-D2, wherein the second and third syntax elements are located in a parameter set, and the fourth syntax element is located in a slice.D4. The method of any one of embodiments D1-D3, wherein first, second, and third QP values are for a chroma channel, and additional QP values for a luma channel are decoded from the bitstream and passed to the neural network.El. A method 800 (see FIG. 8) of encoding a video comprising a sequence of pictures to a bitstream using a neural network-based video coding, the method comprising: obtaining (s802) a first quantization parameter (QP) value; encoding (s804) the first QP value into a first syntax element in the bitstream; obtaining (s806) a second QP value and a third QP value; encoding (s808) the second QP value into a second syntax element in the bitstream; encoding (s810) the third QP value into a third syntax element in the bitstream; determining (s812) whether to use the second QP value or the third QP value for the neural network; passing (s814) the selected QP value to the neural network as an input; and using (s816) an output of the neural network to further encode the bitstream.E2. The method of embodiment El, further comprising encoding a fourth syntax element into the bitstream where a value of the fourth syntax element specifies whether to use the second or third QP value as input for the neural network.E3. The method of any one of embodiments E1-E2, wherein first, second, and third QP values are for a chroma channel, and additional QP values are provided for a luma channel.Fl. A method 900 (see FIG. 9) of decoding a video comprising a sequence of pictures from a bitstream using neural network-based video coding, the method comprising: decoding (s902) an offset value V from a syntax element in the bitstream; deriving (s904) a first QP value QP1 for a first block from the bitstream; deriving (s906) a second QP value QP2 for a second block from the bitstream; assigning (s908) the value QP1+V to at least one entry of a matrix M; assigning (s910) the value QP2+V to at least one entry of the matrix M; passing (s912) the matrix M to a neural network as an input; and using (s914) an output of the neural network to further decode the bitstream.F2. The method of embodiment Fl, further comprising using the first QP value QP1 in an inverse quantization process for the first block and using the second QP value QP2 in an inverse quantization process for the second block.F3. The method of any one of embodiments F1-F2, wherein first and second QP values are for a chroma channel, and additional QP values for a luma channel are decoded from the bitstream and passed to the neural network.F4. The method of any one of embodiments F1-F3, wherein the syntax element is located in one of: a slice data, a slice header, a picture header, and a parameter set.F5. The method of any one of embodiments A1-A6, wherein the offset value V is further based on a reference QP value of a reference picture.F6. The method of any one of embodiments F1-F5, wherein the offset value V is further based on a function of a picture parameter set (PPS) QP value.F7. The method of embodiment F6, wherein the function of a PPS QP value is a polynomial function.F8. The method of any one of embodiments F1-F7, wherein the offset value V is further based on a value decoded from the bitstream that specifies how reconstructed sample values after deblocking shall be combined with NN filtered output (nn_scaling_mode).Gl. A method 1000 (see FIG. 10) of decoding a video comprising a sequence of pictures from a bitstream using neural network-based video coding, the method comprising: determining (si 002) a first quantization parameter (QP) value based on decoding one or more syntax elements from the bitstream, wherein the first QP value is for the first chroma channel; determining (si 004) a second QP value based on decoding one or more syntax elements from the bitstream, wherein the second QP value is for a luma channel; passing (si 006) the first and second QP values to a neural network as an input; and using (si 008) an output of the neural network to further decode the bitstream.G2. The method of embodiment Gl, further comprising determining a third QP value based on decoding one or more syntax elements from the bitstream, wherein the third QP value is for a second chroma channel, and wherein passing the first and second QP values to a neural network as an input further comprises passing the third QP value to the neural network as an input.G3. The method of embodiment G2, wherein the third QP value is based on the first QP value.G4. The method of embodiment G2, wherein the third QP value is based on the second QP value.Hl. A method 1100 (see FIG. 11) of encoding a video comprising a sequence of pictures to a bitstream using a neural network-based video coding, the method comprising: deriving (si 102) a first quantization parameter (QP) delta value from a first QP value, wherein the first QP value is for a first chroma channel;encoding (si 104) the first QP delta value into a first syntax element in the bitstream; deriving (si 106) a second QP delta value based on a second QP value, wherein the second QP value is for a luma channel; encoding (si 108) the second QP delta value into a second syntax element in the bitstream; passing (si 110) the first and second QP values to a neural network as an input; and using (si 112) an output of the neural network to further encode the bitstream.H2. The method of embodiment Hl, further comprising determining a third QP value, wherein the third QP value is for a second chroma channel and wherein passing the first and second QP values to a neural network as an input further comprises passing the third QP value to the neural network as an input.H3. The method of embodiment H2, wherein the third QP value is based on the first QP value.H4. The method of embodiment H2, wherein the third QP value is based on the second QP value.11. A method 1200 (see FIG. 12) of decoding a video comprising a sequence of pictures from a bitstream using neural network-based video coding, the method comprising: deriving (si 202) a first quantization parameter (QP) delta value from a first syntax element from the bitstream; determining (s!204) a first QP value based on the first QP delta value; decoding (si 206) a first position value from a second syntax element from the bitstream, wherein the position value indicates a position in a group of pictures (GOP) hierarchy; passing (s!208) the first QP value and a second position value based on the first position value to a neural network as an input; and using (si 210) an output of the neural network to further decode the bitstream.12. The method of embodiment II, wherein the second position value equals the first position value.13. The method of embodiment II, wherein the second position value is a weight for another input of the neural network.14. The method of embodiment II, wherein the second position value is an offset for another input of the neural network.JI. A method 1300 (see FIG. 13) of encoding a video comprising a sequence of pictures to a bitstream using a neural network-based video coding, the method comprising: deriving (si 302) a first quantization parameter (QP) delta value from a first QP value; encoding (si 304) the first QP delta value into a first syntax element in the bitstream; deriving (si 306) a first position value; encoding (si 308) the first position value into a second syntax element in the bitstream; passing (si 310) the first QP value and the first position value to a neural network as an input; and using (si 312) an output of the neural network to further encode the bitstream.J2. The method of embodiment JI, wherein the second position value equals the first position value.J3. The method of embodiment JI, wherein the second position value is a weight for another input of the neural network.J4. The method of embodiment JI, wherein the second position value is an offset for another input of the neural network.KI. A computer program (1443) comprising instructions (1444) which when executed by processing circuitry (1402) of an apparatus causes the apparatus to perform the method of any one of embodiments A1-A12, B1-B12, C1-C9, D1-D4, E1-E3, F1-F3, G1-G4, H1-H4, 11-14, and Jl-J4.K2. A carrier containing the computer program of embodiment KI, wherein the carrier is one of an electronic signal, an optical signal, a radio signal, and a computer readable storage medium (1442).LI. An apparatus 1400 (see FIG. 14) for decoding a video comprising a sequence of pictures from a bitstream using a neural network-based video coding, wherein the apparatus is configured to perform the method of any one of embodiments Al -Al 2, C1-C9, D1-D4, Fl- F3, G1-G4, and 11-14.Ml. An apparatus 1400 (see FIG. 14) for encoding a video comprising a sequence of pictures to a bitstream using a neural network-based video coding, wherein the apparatus is configured to perform the method of any one of embodiments B1-B12, E1-E3, H1-H4, and J1-J4.

[0224] FIG. 14 is a block diagram of an apparatus 1400 for implementing encoder 102 or decoder 104 according to some embodiments. When apparatus 1400 implements encoder 102, apparatus 1400 may be referred to as an encoder apparatus, when apparatus 1400 implements decoder 104, apparatus 1400 may be referred to as a decoder apparatus. As shown in FIG. 14, apparatus 1400 may comprise: processing circuitry (PC) 1402, which may include one or more processors (P) 1455 (e.g., one or more general purpose microprocessors and / or one or more other processors, such as an application specific integrated circuit (ASIC), field-programmable gate arrays (FPGAs), and the like), which processors may be co-located in a single housing or in a single data center or may be geographically distributed (i.e., encoder apparatus 1400 may be a distributed computing apparatus); at least one network interface 1448 (e.g., a physical interface or air interface) comprising a transmitter (Tx) 1445 and a receiver (Rx) 1447 for enabling apparatus 1400 to transmit data to and receive data from other nodes connected to a network 110 (e.g., an Internet Protocol (IP) network) to which network interface 1448 is connected (physically or wirelessly) (e.g., network interface 1448 may be coupled to an antenna arrangement comprising one or more antennas for enabling encoder apparatus 1400 to wirelessly transmit / receive data); and a storage unit (a.k.a., “data storage system”) 1408, which may include one or more non-volatile storage devices and / or one or more volatile storage devices. In embodiments where PC 1402 includes a programmable processor, a computer readable storage medium (CRSM) 1442 may be provided. CRSM 1442 may store a computer program (CP) 1443 comprising computer readable instructions (CRI) 1444. CRSM 1442 may be a non-transitory computer readable medium, such as, magnetic media (e.g., a hard disk), opticalmedia, memory devices (e.g., random access memory, flash memory), and the like. In some embodiments, the CRI 1444 of computer program 1443 is configured such that when executed by PC 1402, the CRI causes encoder apparatus 1400 to perform steps described herein (e.g., steps described herein with reference to the flow charts). In other embodiments, encoder apparatus 1400 may be configured to perform steps described herein without the need for code. That is, for example, PC 1402 may consist merely of one or more ASICs. Hence, the features of the embodiments described herein may be implemented in hardware and / or software.

[0225] While various embodiments are described herein, it should be understood that they have been presented by way of example only, and not limitation. Thus, the breadth and scope of this disclosure should not be limited by any of the above-described exemplary embodiments. Moreover, any combination of the above-described elements in all possible variations thereof is encompassed by the disclosure unless otherwise indicated herein or otherwise clearly contradicted by context.

[0226] As used herein transmitting a message “to” or “toward” an intended recipient encompasses transmitting the message directly to the intended recipient or transmitting the message indirectly to the intended recipient (i. e. , one or more other nodes are used to relay the message from the source node to the intended recipient). Likewise, as used herein receiving a message “from” a sender encompasses receiving the message directly from the sender or indirectly from the sender (i.e., one or more nodes are used to relay the message from the sender to the receiving node). Further, as used herein “a” means “at least one” or “one or more.”

[0227] Additionally, while the processes described above and illustrated in the drawings are shown as a sequence of steps, this was done solely for the sake of illustration. Accordingly, it is contemplated that some steps may be added, some steps may be omitted, the order of the steps may be re-arranged, and some steps may be performed in parallel.

Claims

CLAIMS1. A method 400 (see FIG. 4) of decoding a video comprising a sequence of pictures from a bitstream using neural network-based video coding, the method comprising: deriving (s402) a first quantization parameter (QP) delta value from a first syntax element from the bitstream; determining (s404) a first QP value based on the first QP delta value; decoding (s406) a second QP delta value from a second syntax element from the bitstream; determining (s408) a second QP value based on the second QP delta value; passing (s410) the second QP value to a neural network as an input; and using (s412) an output of the neural network to further decode the bitstream.

2. The method of claim 1, wherein determining a second QP value based on the second QP delta value is further based on the first QP delta value and the second QP value is further based on a reference QP value of a reference picture.

3. The method of any one of claims 1-2, wherein the second syntax element is located in one of: a slice data, a slice header, a picture header, and a parameter set; and wherein the second syntax element is variable for each block.

4. The method of any one of claims 1-3, wherein (i) the second QP value is further based on a function of a picture parameter set (PPS) QP value and the function of a PPS QP value is a polynomial function and / or (ii) the second QP value is further based on a distance to a closest intra-coded frame.

5. The method of any one of claims 1-4, wherein the second QP value is further based on a value decoded from the bitstream that specifies how reconstructed sample values after deblocking shall be combined with NN filtered output (e.g., nn_scaling_mode).

6. A method 600 (see FIG. 6) of decoding a video comprising a sequence of pictures from a bitstream using neural network-based video coding, the method comprising:deriving (s602) a first quantization parameter (QP) delta value from a first syntax element from the bitstream; determining (s604) a first QP value based on the first QP delta value; decoding (s606) a second QP delta value from a second syntax element from the bitstream; determining (s608) whether to use a second QP value or a third QP value for a neural network; deriving (s610) the second QP value and / or the third QP value based on the second QP delta value, wherein the second QP value is equal in magnitude to the third QP value and of an opposite sign; passing (s612) the second QP value and / or the third QP value to the neural network as an input; and using (s614) an output of the neural network to further decode the bitstream.

7. The method of claim 6, wherein the second and / or third QP value is further based on a reference QP value of a reference picture.

8. The method of any one of claims 6-7, wherein the second QP value is further based on a function of a picture parameter set (PPS) QP value.

9. The method of claim 8, wherein the function of a PPS QP value is a polynomial function.

10. The method of any one of claims 6-9, wherein the second QP value is further based on a distance to a closest intra-coded frame.

11. The method of any one of claims 6-10, wherein the second QP value is further based on a value decoded from the bitstream that specifies how reconstructed sample values after deblocking shall be combined with NN filtered output (e.g., nn_scaling_mode).

12. The method of any one of claims 1-11, wherein passing the second QP value to a neural network as an input further comprises passing a QPblock as input to the neural network.

13. A method 900 (see FIG. 9) of decoding a video comprising a sequence of pictures from a bitstream using neural network-based video coding, the method comprising: decoding (s902) an offset value V from a first syntax element in the bitstream; deriving (s904) a first QP value QP1 for a first block from the bitstream; deriving (s906) a second QP value QP2 for a second block from the bitstream; assigning (s908) the value QP1+V to at least a first entry of a matrix M; assigning (s910) the value QP2+V to at least a second entry of the matrix M; passing (s912) the matrix M to a neural network as an input; and using (s914) an output of the neural network to further decode the bitstream.

14. The method of claim 13, further comprising using the first QP value QP1 in an inverse quantization process for the first block and using the second QP value QP2 in an inverse quantization process for the second block.

15. The method of any one of claims 13-14, wherein first and second QP values are for a chroma channel, and additional QP values for a luma channel are decoded from the bitstream and passed to the neural network.

16. The method of any one of claims 13-15, wherein the first syntax element is located in one of: a slice data, a slice header, a picture header, and a parameter set.

17. The method of any one of claims 13-16, wherein the offset value V is further based on a reference QP value of a reference picture.

18. The method of any one of claims 13-17, wherein the offset value V is further based on a function of a picture parameter set (PPS) QP value.

19. The method of claim 18, wherein the function of a PPS QP value is a polynomial function.

20. The method of any one of claims 13-19, wherein the offset value V is further based on a value decoded from the bitstream that specifies how reconstructed sample values after deblocking shall be combined with NN filtered output (nn_scaling_mode).

21. A method 1000 (see FIG. 10) of decoding a video comprising a sequence of pictures from a bitstream using neural network-based video coding, the method comprising: determining (si 002) a first quantization parameter (QP) value based on decoding a first syntax element from the bitstream, wherein the first QP value is for the first chroma channel; determining (si 004) a second QP value based on decoding a second syntax element from the bitstream, wherein the second QP value is for a luma channel; passing (si 006) the first and second QP values to a neural network as an input; and using (si 008) an output of the neural network to further decode the bitstream.

22. The method of claim 21, further comprising determining a third QP value based on decoding a third syntax element from the bitstream, wherein the third QP value is for a second chroma channel, and wherein passing the first and second QP values to a neural network as an input further comprises passing the third QP value to the neural network as an input.

23. The method of claim 22, wherein the third QP value is based on the first QP value.

24. The method of claim 22, wherein the third QP value is based on the second QP value.

25. A method 1200 (see FIG. 12) of decoding a video comprising a sequence of pictures from a bitstream using neural network-based video coding, the method comprising: deriving (si 202) a first quantization parameter (QP) delta value from a first syntax element from the bitstream; determining (sl204) a first QP value based on the first QP delta value; decoding (si 206) a first position value from a second syntax element from the bitstream, wherein the position value indicates a position in a group of pictures (GOP) hierarchy; passing (si 208) the first QP value and a second position value based on the first position value to a neural network as an input; and using (si 210) an output of the neural network to further decode the bitstream.

26. The method of claim 25, wherein the second position value equals the first position value.

27. The method of claim 25, wherein the second position value is a weight for another input of the neural network.

28. The method of claim 25, wherein the second position value is an offset for another input of the neural network.

29. An apparatus 1400 (see FIG. 14) for decoding a video comprising a sequence of pictures from a bitstream using a neural network-based video coding, wherein the apparatus is configured to perform the method of any one of claims 1-28.

30. A method 500 (see FIG. 5) of encoding a video comprising a sequence of pictures to a bitstream using a neural network-based video coding, the method comprising: deriving (s502) a first quantization parameter (QP) delta value from a first QP value; encoding (s504) the first QP delta value into a first syntax element in the bitstream; deriving (s506) a second QP delta value based on a second QP value; encoding (s508) the second QP delta value into a second syntax element in the bitstream; passing (s510) the second QP value to a neural network as an input; and using (s512) an output of the neural network to further encode the bitstream.

31. The method of claim 30, wherein deriving a second QP delta value based on the second QP value is further based on the first QP value and wherein the second QP value is further based on a reference QP value of a reference picture..

32. The method of any one of claims 30-31, wherein first and second QP values are for a chroma channel, and additional QP values for a luma channel are encoded into the bitstream and passed to the neural network.

33. A method 800 (see FIG. 8) of encoding a video comprising a sequence of pictures to a bitstream using a neural network-based video coding, the method comprising: obtaining (s802) a first quantization parameter (QP) value;encoding (s804) the first QP value into a first syntax element in the bitstream; obtaining (s806) a second QP value and a third QP value; encoding (s808) the second QP value into a second syntax element in the bitstream; encoding (s810) the third QP value into a third syntax element in the bitstream; determining (s812) whether to use the second QP value or the third QP value for the neural network; passing (s814) the selected QP value to the neural network as an input; and using (s816) an output of the neural network to further encode the bitstream.

34. The method of claim 33, further comprising encoding a fourth syntax element into the bitstream where a value of the fourth syntax element specifies whether to use the second or third QP value as input for the neural network; and wherein first, second, and third QP values are for a chroma channel, and additional QP values are provided for a luma channel.

35. A method 1100 (see FIG. 11) of encoding a video comprising a sequence of pictures to a bitstream using a neural network-based video coding, the method comprising: deriving (si 102) a first quantization parameter (QP) delta value from a first QP value, wherein the first QP value is for a first chroma channel; encoding (si 104) the first QP delta value into a first syntax element in the bitstream; deriving (si 106) a second QP delta value based on a second QP value, wherein the second QP value is for a luma channel; encoding (si 108) the second QP delta value into a second syntax element in the bitstream; passing (si 110) the first and second QP values to a neural network as an input; and using (si 112) an output of the neural network to further encode the bitstream.

36. The method of claim 35, further comprising determining a third QP value, wherein the third QP value is for a second chroma channel and wherein passing the first and second QP values to a neural network as an input further comprises passing the third QP value to the neural network as an input; and wherein the third QP value is based on the first QP value or the second QP value.

37. A method 1300 (see FIG. 13) of encoding a video comprising a sequence of pictures to a bitstream using a neural network-based video coding, the method comprising: deriving (si 302) a first quantization parameter (QP) delta value from a first QP value; encoding (si 304) the first QP delta value into a first syntax element in the bitstream; deriving (si 306) a first position value; encoding (si 308) the first position value into a second syntax element in the bitstream; passing (si 310) the first QP value and the first position value to a neural network as an input; and using (si 312) an output of the neural network to further encode the bitstream.

38. The method of claim 37, wherein the second position value equals the first position value.

39. The method of claim 38, wherein the second position value is (i) a weight for another input of the neural network or (ii) an offset for another input of the neural network.

40. An apparatus 1400 (see FIG. 14) for encoding a video comprising a sequence of pictures from a bitstream using a neural network-based video coding, wherein the apparatus is configured to perform the method of any one of claims 30-39.

Images