Method for sharing neural network inference information in video compression

Abstract

A method, the method comprising: obtaining a video stream; obtaining metadata associated with the video stream, the metadata representing an allowable margin around a patch for an inference process of a neural network-based image processing tool; and applying an image processing tool based on the neural network to decode the video stream.

Description

Technical Field

[0001] At least one of the embodiments of the present invention generally relates to a method and apparatus for encoding and decoding video data using a neural network, and in particular, to a method that allows sharing of neural network information, thereby allowing a flexible reasoning process on the decoder side. Background Art

[0002] In order to achieve high compression efficiency, video coding schemes generally use prediction and transformation to exploit spatial and temporal redundancy in video content. During encoding, a picture of the video content is divided into sample blocks (i.e., pixels), and then these blocks are divided into one or more sub-blocks (hereinafter referred to as original sub-blocks). Then, intra-frame or inter-frame prediction is applied to each sub-block to exploit the correlation within or between images. Regardless of which prediction method (intra-frame or inter-frame) is used, a predicted sub-block is determined for each original sub-block. Then, the sub-block representing the difference between the original sub-block and the predicted sub-block (usually represented as a prediction error sub-block, a prediction residual sub-block, or simply a residual sub-block) is transformed, quantized, and entropy encoded to generate an encoded video stream. In order to reconstruct the video, the compressed data is decoded by the inverse process corresponding to the transformation, quantization, and entropy encoding.

[0003] In recently explored video coding solutions, neural network based processing has been proposed, for example in the post-filtering stage or for block prediction. Before practical use, the neural network needs to be trained in order to be able to provide accurate results. The training of a neural network is a computationally intensive process that typically requires comparing the output data provided by the neural network with the expected values ​​of these output data for a large amount of input data. Once trained, the neural network can apply what it has learned to input data, even if these input data have never been considered during the training process. The process of applying a trained neural network to input data to obtain output data is called inference.

[0004] It is well known in the field of video compression that the process applied on the encoder side should be completely replicable on the decoder side to ensure that there is no drift between the encoder and the decoder. The same applies to the neural network (NN) based process applied in the prediction loop of the encoder and decoder. This requirement for replicability means that the output data inferred on the decoder side should be the same as the output data inferred on the encoder side. In addition, it is usually expected that two decoders (possibly with different implementations) systematically provide the same results. However, encoders and decoders are usually designed to have different software or hardware constraints. For example, an encoder may be able to store more data in memory than a decoder. A decoder, where processing speed is usually a key issue, may be able to process more data in parallel than an encoder without the same processing speed issue. Decoders implemented in smartphones generally do not have the same hardware constraints as decoders implemented on PCs. In this case, the developers of encoders and decoders should leave as much flexibility as possible when designing the reasoning process of the NN-based process, while first ensuring the replicability of the output provided by the NN-based in-loop process on the encoder side on the decoder side, and secondly ensuring that two decoders with different implementations provide the same results.

[0005] It is desirable to propose a solution that can overcome the above-mentioned problems. In particular, it is desirable to propose a solution that allows flexibility in the reasoning process of the NN reasoning process to be ensured. Summary of the invention

[0006] In a first aspect, one or more embodiments of the present invention provide a method, the method comprising:

[0007] Get the video stream;

[0008] obtaining metadata associated with the video stream, the metadata representing an allowable margin around a patch for use in an inference process of a neural network-based image processing tool; and

[0009] The neural network-based image processing tool is applied to decode the video stream.

[0010] In an embodiment, the allowable margin depends on a receptive field, which depends on a neural network used in the neural network based image processing tool.

[0011] In an embodiment, said metadata comprises representation of at least one syntactic element, said at least one syntactic element being dependent on said receptive field of said neural network used in said neural network based image processing tool.

[0012] In an embodiment, the at least one syntax element comprises: a first syntax element vertically defining the receptive field and a second syntax element horizontally defining the receptive field.

[0013] In an embodiment, the ability of the inference process to process patches of a larger size than those considered during the definition of the neural network used in the neural network-based image processing tool is determined by comparing at least one value representing the receptive field dependent on the neural network with a margin around the patch considered during the definition of the neural network used in the neural network-based image processing tool.

[0014] In an embodiment, the ability of the inference process to handle patches of larger sizes than considered during definition of the neural network used in the neural network based image processing tool is specified in the metadata by a syntax element.

[0015] In an embodiment, the metadata comprises at least one syntax element representing at least one offset added to represent a value of the receptive field depending on the neural network or a margin around a patch considered during definition of the neural network used in the neural network-based image processing tool, the offset being used in response to processing the current patch by the inference process based on the position of the current patch, the size of the current patch being smaller than the size of the patch considered during definition of the neural network used in the neural network-based image processing tool.

[0016] In an embodiment, the metadata comprises at least one syntax element, the at least one syntax element representing a position of an output patch of the inference process of the neural network-based image processing tool in an output tensor generated by the inference process.

[0017] In a second aspect, one or more of the embodiments of the present invention provide a method, the method comprising:

[0018] Obtaining a video stream; and

[0019] Information representing an allowable margin around a patch for use in an inference process of a neural network based image processing tool is signaled in the form of metadata associated with the video stream.

[0020] In an embodiment, the allowable margin depends on a receptive field, which depends on a neural network used in the neural network based image processing tool.

[0021] In an embodiment, said metadata comprises at least one syntax element representing said receptive field depending on said neural network used on said neural network based image processing tool.

[0022] In an embodiment, the at least one syntax element comprises: a first syntax element vertically defining the receptive field and a second syntax element horizontally defining the receptive field.

[0023] In an embodiment, the ability of the inference process to process patches of a larger size than those considered during the definition of the neural network used in the neural network-based image processing tool is determined by comparing at least one value representing the receptive field dependent on the neural network with a margin around the patch considered during the definition of the neural network used in the neural network-based image processing tool.

[0024] In an embodiment, the ability of the inference process to handle patches of larger sizes than considered during definition of the neural network used in the neural network based image processing tool is specified in the metadata by a syntax element.

[0025] In an embodiment, the metadata comprises at least one syntax element representing at least one offset added to represent a value of the receptive field depending on the neural network or a margin around a patch considered during definition of the neural network used in the neural network-based image processing tool, the offset being used in response to processing the current patch by the inference process based on the position of the current patch, the size of the current patch being smaller than the size of the patch considered during definition of the neural network used in the neural network-based image processing tool.

[0026] In an embodiment, the metadata comprises at least one syntax element, the at least one syntax element representing a position of an output patch of the inference process of the neural network-based image processing tool in an output tensor generated by the inference process.

[0027] In an embodiment, the video stream is obtained by applying a video compression process to an original video, wherein the video compression process includes the neural network-based image processing tool in a prediction loop of the video compression process, or as a post-processing tool.

[0028] In a third aspect, one or more of the embodiments of the present invention provide a signal comprising metadata associated with a video stream, the metadata representing an allowable margin around a patch for an inference process of a neural network-based image processing tool.

[0029] In a fourth aspect, one or more of the embodiments of the present invention provide a computer program, the computer program comprising program code instructions for implementing the method of the first aspect or the second aspect.

[0030] In a fifth aspect, one or more of the embodiments of the present invention provide a non-transitory information storage medium, wherein the non-transitory information storage medium includes program code instructions for implementing the method of the first aspect or the second aspect.

[0031] In a sixth aspect, one or more of the embodiments of the present invention provide a device, the device comprising an electronic circuit, the electronic circuit being configured to:

[0032] Get the video stream;

[0033] obtaining metadata associated with the video stream, the metadata representing an allowable margin around a patch for use in an inference process of a neural network based image processing tool; and

[0034] The neural network-based image processing tool is applied to decode the video stream.

[0035] In an embodiment, the allowable margin depends on a receptive field, which depends on a neural network used in the neural network based image processing tool.

[0036] In an embodiment, said metadata comprises at least one syntax element representing said receptive field depending on said neural network used in said neural network based image processing tool.

[0037] In an embodiment, the at least one syntax element comprises: a first syntax element vertically defining the receptive field and a second syntax element horizontally defining the receptive field.

[0038] In an embodiment, the ability of the inference process to process patches of a larger size than those considered during the definition of the neural network used in the neural network-based image processing tool is determined by comparing at least one value representing the receptive field dependent on the neural network with a margin around the patch considered during the definition of the neural network used in the neural network-based image processing tool.

[0039] In an embodiment, the ability of the inference process to handle patches of larger sizes than considered during definition of the neural network used in the neural network based image processing tool is specified in the metadata by a syntax element.

[0040] In an embodiment, the metadata comprises at least one syntax element representing at least one offset added to represent a value of the receptive field depending on the neural network or a margin around a patch considered during definition of the neural network used in the neural network-based image processing tool, the offset being used in response to processing the current patch by the inference process based on the position of the current patch, the size of the current patch being smaller than the size of the patch considered during definition of the neural network used in the neural network-based image processing tool.

[0041] In an embodiment, the metadata comprises at least one syntax element, the at least one syntax element representing a position of an output patch of the inference process of the neural network-based image processing tool in an output tensor generated by the inference process.

[0042] In a seventh aspect, one or more of the embodiments of the present invention provide a device, the device comprising an electronic circuit, the electronic circuit being configured to:

[0043] Obtaining a video stream; and

[0044] Information representing an allowable margin around a patch for use in an inference process of a neural network based image processing tool is signaled in the form of metadata associated with the video stream.

[0045] In an embodiment, the allowable margin depends on a receptive field, which depends on a neural network used in the neural network based image processing tool.

[0046] In an embodiment, said metadata comprises at least one syntax element representing said receptive field depending on said neural network used on said neural network based image processing tool.

[0047] In an embodiment, the at least one syntax element comprises: a first syntax element vertically defining the receptive field and a second syntax element horizontally defining the receptive field.

[0048] In an embodiment, the ability of the inference process to process patches of a larger size than those considered during the definition of the neural network used in the neural network-based image processing tool is determined by comparing at least one value representing the receptive field dependent on the neural network with a margin around the patch considered during the definition of the neural network used in the neural network-based image processing tool.

[0049] In an embodiment, the ability of the inference process to handle patches of larger sizes than considered during definition of the neural network used in the neural network based image processing tool is specified in the metadata by a syntax element.

[0050] In an embodiment, the metadata comprises at least one syntax element representing at least one offset added to represent a value of the receptive field depending on the neural network or a margin around a patch considered during definition of the neural network used in the neural network-based image processing tool, the offset being used in response to processing the current patch by the inference process based on the position of the current patch, the size of the current patch being smaller than the size of the patch considered during definition of the neural network used in the neural network-based image processing tool.

[0051] In an embodiment, the metadata comprises at least one syntax element, the at least one syntax element representing a position of an output patch of the inference process of the neural network-based image processing tool in an output tensor generated by the inference process.

[0052] In an embodiment, the video stream is obtained by applying a video compression process to an original video, wherein the video compression process includes the neural network-based image processing tool in a prediction loop of the video compression process, or as a post-processing tool. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 Schematically illustrates the background for implementing the embodiments;

[0054] Figure 2 schematically shows an example of the segmentation that a pixel picture of an original video undergoes;

[0055] Figure 3 A method for encoding a video stream is schematically depicted;

[0056] Figure 4 schematically depicts a method for decoding an encoded video stream;

[0057] Figure 5A An example of a hardware architecture of a processing module capable of implementing an encoding module or a decoding module for implementing various aspects and embodiments is schematically shown;

[0058] Figure 5B A block diagram illustrating an example of a first system implementing various aspects and embodiments;

[0059] Figure 5CA block diagram showing an example of a second system implementing various aspects and embodiments;

[0060] Figure 6 An example of a NN-based process is schematically shown;

[0061] Fig. 7A Schematically represent the block-based reasoning process and the sub-block-based reasoning process;

[0062] Figure 7B Schematically represent the block-based reasoning process and the superblock-based reasoning process;

[0063] Fig. 8A Schematically represents an example of region expansion of the sub-block based reasoning process;

[0064] Figure 8B schematically represents an example of region expansion of the sub-block based inference process when the receptive field of the NN is characterized by different horizontal and vertical extents;

[0065] Fig. 9 Schematic representation of the receptive field and boundaries of a block;

[0066] Fig.10 Schematically representing an example of a NN with varying tensor sizes;

[0067] Fig.11 An example of a layout schematically showing the sub-block reasoning process for the lower right sub-block;

[0068] Fig.12 Schematically showing an embodiment in which the output is offset;

[0069] Fig.13 Schematic showing the impact of the bias introduced during NN inference with some tensor size changes.

[0070] Fig.14A schematically illustrating application of an embodiment during an encoding process; and

[0071] Fig. 14B The application of the embodiments during the decoding process is schematically represented. DETAILED DESCRIPTION

[0072] The following examples of embodiments are described in the context of a video format similar to VVC, which is being developed by a joint collaborative team of ITU-T and ISO / IEC experts, known as the Joint Video Experts Team (JVET). However, the embodiments are not limited to video encoding / decoding methods corresponding to VVC. The embodiments are particularly suitable for various video formats, including, for example, HEVC (ISO / IEC 23008-2 - MPEG-H Part 2, High Efficiency Video Coding / ITU-T H.265), AVC (ISO / CEI 14496-10), EVC (Basic Video Coding / MPEG-5), AV1, AV2, and VP9.

[0073] Figure 1 The background for implementing the embodiments is schematically shown.

[0074] exist Figure 1 In the embodiment, system 11 (which may be a camera, a storage device, a computer, a server, or any device capable of transmitting a video stream) transmits a video stream to system 13 using a communication channel 12. The video stream is encoded and transmitted by system 11, or is received and / or stored by system 11 and then transmitted. Communication channel 12 is a wired network link (e.g., the Internet or Ethernet) or a wireless network link (e.g., WiFi, 3G, 4G, or 5G).

[0075] The system 13, which may be, for example, a set-top box, receives the video stream and decodes it to generate a sequence of decoded pictures.

[0076] The obtained decoded picture sequence is then transmitted to a display system 15 using a communication channel 14, which may be a wired or wireless network. The display system 15 then displays the pictures.

[0077] In an embodiment, the system 13 is included in the display system 15. In this case, the system 13 and the display 15 are included in a TV, a computer, a tablet, a smartphone, a head-mounted display, or the like.

[0078] Figure 2 , Figure 3 and Figure 4 An example that introduces video formats.

[0079] Figure 2 An example of segmentation of a pixel picture 21 of an original video sequence 20 is shown. Here, a pixel is considered to consist of three components: a luminance component and two chrominance components. However, other types of pixels including fewer or more components (such as only a luminance component or an additional depth component or transparency component) are also possible.

[0080] The picture is divided into multiple encoding entities. First, Figure 2 As indicated by reference numeral 23 in the figure, the picture is divided into a grid of blocks, which are called coding tree units (CTUs). A CTU consists of an N×N block of luminance samples and two corresponding blocks of chrominance samples. For example, N is usually a power of two, with a maximum value of "128". Secondly, the picture is divided into one or more groups of CTUs. For example, it can be divided into one or more tile rows and tile columns, and a tile is a sequence of CTUs covering a rectangular area of ​​the picture. In some cases, a tile can be divided into one or more bricks, each brick consisting of at least one row of CTUs within the tile. On top of the concept of tiles and tiles, there is another coding entity (called a slice), which can contain at least one tile of a picture or at least one tile of a tile.

[0081] exist Figure 2 In the example in FIG. 2 , as indicated by reference numeral 22 , the picture 21 is divided into three slices S1 , S2 and S3 in a raster scan slice mode, each slice includes a plurality of tiles (not shown), and each tile includes only one tile.

[0082] like Figure 2 As indicated by reference numeral 24 in FIG. 1 , a CTU may be partitioned into a hierarchical tree of one or more sub-blocks, which are referred to as coding units (CUs). A CTU is the root (i.e., parent node) of the hierarchical tree and may be partitioned into multiple CUs (i.e., child nodes). Each CU will be a leaf of the hierarchical tree if not further partitioned into smaller CUs, or will be a parent node of a smaller CU (i.e., child node) if further partitioned.

[0083] exist Figure 2 In the example of , CTU 24 is first partitioned into "4" square CUs using quadtree partitioning. The upper left CU is a leaf of the hierarchy tree because it is not further partitioned, i.e., it is not the parent of any other CU. The upper right CU is further partitioned into "4" smaller square CUs using quadtree partitioning again. The lower right CU is vertically partitioned into "2" rectangular CUs using binary tree partitioning. The lower left CU is vertically partitioned into "3" rectangular CUs using ternary tree partitioning.

[0084] During encoding of a picture, the partitioning is adaptive and each CTU is partitioned to optimize the compression efficiency of the CTU scale.

[0085] In HEVC, the concepts of prediction unit (PU) and transform unit (TU) appear. In fact, in HEVC, the coding entity for prediction (i.e., PU) and transform (i.e., TU) can be a subdivision of CU. For example, Figure 2As shown in FIG. 2 , a CU of size 2N×2N may be divided into PUs 2411 of size N×2N or PUs 2412 of size 2N×N. In addition, the CU may be divided into “4” TUs 2412 of size N×N or TUs of size 2N×N. The “16” TUs.

[0086] It can be noted that in VVC, except in some special cases, the boundaries of TU and PU are aligned on the boundaries of CU. Therefore, a CU usually includes one TU and one PU.

[0087] In this application, the term "block" or "picture block" may be used to refer to any of CTU, CU, PU, ​​and TU. In addition, the term "block" or "picture block" may be used to refer to macroblocks, partitions, and sub-blocks, as specified in H.264 / AVC or other video coding standards, and more generally to sample arrays of various sizes.

[0088] In this application, the terms "reconstruction" and "decoding" can be used interchangeably, the terms "pixel" and "sample" can be used interchangeably, and the terms "image", "picture", "sub-picture", "slice" and "frame" can be used interchangeably. Usually, but not necessarily, the term "reconstruction" is used on the encoder side, while "decoding" is used on the decoder side.

[0089] Figure 3 Schematically depicts a method for encoding a video stream performed by an encoding module. Variations of this encoding method are envisioned, but for clarity, the following describes Figure 3 encoding methods without describing all anticipated variations.

[0090] Before encoding, the current original picture of the original video sequence may be preprocessed. For example, in step 301, a color transform is applied to the current original picture (e.g., from RGB 4:4:4 to YCbCr4:2:0), or a remapping is applied to the current original picture components to obtain a signal distribution that is more resilient to compression (e.g., using histogram equalization of one of the color components). The picture obtained by preprocessing is referred to as a preprocessed picture below.

[0091] The encoding of the pre-processed picture begins with segmenting the pre-processed picture during step 302, as described with respect to Figure 2 Therefore, the pre-processed picture is divided into CTU, CU, PU, ​​TU, etc. For each block, the encoding module determines the encoding mode between intra prediction and inter prediction.

[0092] Intra-prediction consists in predicting the pixels of the current block from a prediction block derived from pixels of a reconstructed block located in the causal vicinity of the current block to be coded according to an intra-prediction method during step 303. The result of the intra-prediction is a prediction direction indicating which pixels of the nearby blocks are to be used, and a residual block obtained by calculating the difference between the current block and the predicted block.

[0093] Inter prediction consists in predicting the pixels of the current block from a block of pixels of a picture preceding or following the current picture, called a reference block, which picture is called a reference picture. During the encoding of the current block according to the inter prediction method, the block of the reference picture which is closest to the current block according to a similarity criterion is determined by a motion estimation step 304. During step 304, a motion vector indicating the position of the reference block in the reference picture is determined. Said motion vector is used during a motion compensation step 305 during which a residual block is calculated in the form of a difference between the current block and the reference block. In the first video compression standards, the above-mentioned unidirectional inter prediction mode was the only inter mode available. With the development of video compression standards, the family of inter modes has grown significantly and now includes many different inter modes.

[0094] During the selection step 306 , the prediction mode that optimizes the compression performance according to the rate / distortion optimization criterion (ie, the RDO criterion) is selected by the encoding module from among the tested prediction modes (intra-prediction mode, inter-prediction mode).

[0095] When a prediction mode is selected, the residual block is transformed during step 307. The transformed block is then quantized during step 309.

[0096] It should be noted that the encoding module can skip the transformation and directly apply quantization to the untransformed residual signal. When the current block is encoded according to an intra-frame prediction mode, during step 310, the prediction direction and the transformed and quantized residual block are encoded by the entropy encoder. When the current block is encoded according to inter-frame prediction, when appropriate, the motion vector of the block is predicted according to a prediction vector selected from a set of motion vector predictors, which are derived from reconstructed blocks located in the spatial and temporal vicinity of the block to be encoded. Next, during step 310, the motion information is encoded by the entropy encoder in the form of a motion residual and an index for identifying the prediction vector. During step 310, the transformed and quantized residual block is encoded by the entropy encoder.

[0097] It should be noted that the encoding module may bypass both transform and quantization, ie, apply entropy encoding to the residual without applying a transform or quantization process. The result of the entropy encoding is inserted into the encoded video stream 311 .

[0098] Metadata such as SEI (Supplemental Enhancement Information) messages may be appended to the encoded video stream 311. A SEI message as defined in standards such as AVC, HEVC or VVC is a data container associated with a video stream and includes metadata providing information about the video stream.

[0099] After the quantization step 309, the current block is reconstructed so that the pixels corresponding to the block can be used for future predictions. This reconstruction phase is also called a prediction loop. Therefore, inverse quantization is applied to the transformed and quantized residual block during step 312, and an inverse transform is applied during step 313. According to the prediction mode for the block obtained during step 314, the prediction block of the block is reconstructed. If the current block is encoded according to the inter-frame prediction mode, during step 316, the encoding module uses the motion vector of the current block to apply motion compensation when appropriate in order to identify the reference block of the current block. If the current block is encoded according to the intra-frame prediction mode, during step 315, the prediction direction corresponding to the current block is used to reconstruct the prediction block of the current block. The prediction block and the reconstructed residual block are added to obtain the reconstructed current block.

[0100] After reconstruction, in-loop filtering aimed at reducing coding artifacts is applied to the reconstructed blocks during step 317. This filtering is called in-loop filtering because it occurs in the prediction loop to obtain the same reference picture at the decoder as at the encoder and thus avoid drift between the encoding and decoding processes. In-loop filtering tools include deblocking filtering, SAO (Sample Adaptive Offset) and ALF (Adaptive Loop Filtering).

[0101] When a block is reconstructed, it is inserted into a reconstructed picture stored in a memory of reconstructed pictures 319, usually called a decoded picture buffer (DPB), during a step 318. The reconstructed picture stored in this way can then serve as a reference picture for other pictures to be encoded.

[0102] Figure 4 Schematically depicts a method performed by a decoding module for decoding an encoded video stream 311, the video stream 311 according to Figure 3 Variations of this decoding method are envisioned, but for clarity the following describes Figure 4 without describing all anticipated variations.

[0103] The decoding is done block by block. For the current block, it starts with entropy decoding of the current block during a step 410. Entropy decoding allows obtaining at least the prediction mode of the block.

[0104] If the block has been coded according to inter prediction mode, entropy decoding allows obtaining the prediction vector index, the motion residual and the residual block, if appropriate.During a step 408, the motion vector is reconstructed for the current block using the prediction vector index and the motion residual.

[0105] If the block has been coded according to an intra prediction mode, entropy decoding allows obtaining the prediction direction and the residual block. Steps 412, 413, 414, 415, 416 and 417 implemented by the decoding module are identical in all respects to steps 312, 313, 314, 315, 316 and 317 respectively implemented by the encoding module.

[0106] In step 418, the decoded blocks are saved in a decoded picture and the decoded picture is stored in the DPB 419. When the decoding module decodes a given picture, the picture stored in the DPB 419 is the same as the picture stored in the DPB 319 by the encoding module during encoding of the given picture. The decoded picture may also be output by the decoding module, for example, for display.

[0107] The post-processing step 421 may include an inverse color transform (e.g., conversion from YCbCr 4:2:0 to RGB 4:4:4), an inverse mapping of the remapping process performed in the pre-processing of step 301, and post-filtering to improve the reconstructed picture based on filter parameters provided in, for example, an SEI message.

[0108] Among the recently explored video coding solutions, neural network based processing has been proposed, for example for post filtering or block prediction (intra and inter). In addition, solutions have been proposed that allow information representing the NN to be exchanged between the encoder and the decoder. For example, such information is transmitted as side information in the form of SEI messages.

[0109] Figure 6 An example of a NN-based process is schematically shown.

[0110] exist Figure 6 In , a block 51 of the original picture 50 is processed. Before processing, the block 51 is enlarged by some margin. The enlarged block is then input into the NN 52. The output of the NN is a block 53 having the same size as the block 51. The block 53 is then inserted into the picture 54 so that it can be used for the encoding or decoding process, or displayed without being used for the encoding or decoding process. In some variants, the block 53 has the same size as the enlarged block. In this case, the margin is used for blending at the block boundaries to reduce blocking artifacts.

[0111] Table TAB1 provides an example of an SEI message nnr_post_filter that allows the transmission of information representing the NN post-filtering process.

[0112]

[0113]

[0114]

[0115] Table TAB1

[0116] The semantics of the syntactic elements in Table TAB1 are briefly described as follows (the reader is referred to the following document for detailed semantics: JVET_Z0052: Joint Video Experts Team (JVET) of ITU-T SG 16WP 3 and ISO / IEC JTC 1 / SC 29, 26th Meeting, Teleconference, April 20-29, 2022, Hannuksela et al.):

[0117] nnrpf_constant_patch_size_flag equal to "0" specifies that the post-processing filter accepts as input any patch size that is a positive integer multiple of the patch size indicated by nnrpf_patch_size_minus1. nnrpf_constant_patch_size_flag equal to "1" specifies that the post-processing filter accepts as input exactly the patch size indicated by nnrpf_patch_size_minus1.

[0118] nnrpf_patch_size_minus1+1 specifies the horizontal and vertical sample counts of the patch size of the post-processing filter. The value of nnrpf_patch_size_minus1 should be in the range of "0" to "32766", inclusive.

[0119] nnrpf_overlap specifies the overlapping horizontal and vertical sample counts of adjacent input tensors for the post-processing filter. The value of nnrpf_overlap should be in the range of "0" to "16383", inclusive.

[0120] nnrpf_component_last_flag indicates whether channels are indicated as the second or last dimension in tensors at the input of the NN and in tensors at the output of the NN.

[0121] nnrpf_pic_width_in_luma_samples and nnrpf_pic_height_in_luma_samples specify the width and height, respectively, of the luma sample array of the picture obtained by applying the post-processing filter identified by nnrpf_id to the cropped decoded output picture.

[0122] nnrpf_id contains an identification number that can be used to identify the post-processing filter.

[0123] In this semantics, a patch is a tensor that includes picture data related to, for example, a block or a whole picture, which may be picture samples (e.g., block samples + samples near the block), quantization parameters, motion information, etc. For example, a simple patch is a block of samples.

[0124] Export the variables inpPatchWidth, inpPatchHeight, outPatchWidth, outPatchHeight, horCScaling, verCScaling, outPatchCWidth, outPatchCHeight, and overlapSize as follows:

[0125] ·inpPatchWidth=nnrpf_patch_size_minus1+1;

[0126] ·inpPatchHeight=nnrpf_patch_size_minus1+1;

[0127] ·outPatchWidth=(nnrpf_pic_width_in_luma_samples*inpPatchWidth) /

[0128] InpPicWidthInLumaSamples;

[0129] ·outPatchHeight=(nnrpf_pic_height_in_luma_samples*inpPatchHeight) /

[0130] InpPicHeightInLumaSamples;

[0131] ·horCScaling=InpSubWidthC / outSubWidthC;

[0132] ·verCScaling=InpSubHeightC / outSubHeightC;

[0133] ·outPatchCWidth=outPatchWidth*horCScaling;

[0134] ·outPatchCHeight=outPatchHeight*verCScaling;

[0135] ·overlapSize=nnrpf_overlap.

[0136] InpPicHeightInLumaSamples represents the input picture height in luminance samples.

[0137] InpPicWidthtInLumaSamples represents the input picture width in luminance samples.

[0138] Using the above variables, the following pseudocode gives an example of the derivation process of the input tensor inputTensor given the patch size (inpPatchWidth, inpPatchHeight) and the overlap size overlapSize:

[0139]

[0140] Clip3(a,b,c) represents a function that keeps the value of variable b between a and c.

[0141] (cTop, cLeft) represents the coordinates of the pixel at the top left corner of the current block in the luma channel of the current image. It should be noted that the convention in video compression is used. This means that the column coordinate cTop is placed first in the coordinate tuple.

[0142] inputTensor represents the tensor fed into the NN. It should be noted that for indexing inputTensor, the convention in machine learning is used. This means that row coordinates come before column coordinates. If nnrpf_component_last_flag is equal to "0", that is, the channel is indicated as the second dimension, then the rows and columns are indicated as the third and fourth dimensions. If nnrpf_component_last_flag is equal to "1", that is, the channel is indicated as the fourth dimension in inputTensor, then the rows and columns are indicated as the second and third dimensions in inputTensor. inpY indicates the luminance channel of the current picture. CroppedYPic[y][x] indicates the cropped decoded output picture.

[0143] The following pseudocode describes an example of the process of deriving new filtered samples given the output tensor outputTensor.

[0144]

[0145] In this pseudocode, FilteredYPic indicates the array of luminance filtered samples.

[0146] OutY and OutC indicate functions for converting luma sample values ​​and chroma sample values ​​output by post-processing into integer values ​​at bit depths BitDepthY and BitDepthC, respectively. OutY and OutC are specified as follows:

[0147] OutY(x)=Clip3(0,(1< <BitDepthY)-1,Round(x*((1<<BitDepthY)-1)))

[0148] OutC(x)=Clip3(0,(1< <BitDepthC)-1,Round(x*((1<<BitDepthC)-1)))

[0149] Where Round(a) rounds the value a to the nearest integer value.

[0150] Typically, the SEI message nnr_post_filter describes:

[0151] The patch size corresponding to the output size;

[0152] The input size is equal to the patch size plus some overlap size overlapSize which is the same on all boundaries of the patch.

[0153] Some information is needed to describe the input and output of such a NN in order for the decoder to perform correct and reproducible inference.

[0154] Fig. 7A The block-based reasoning process and the sub-block-based reasoning process are schematically shown. Fig. 7A In Fig. 1, the left half shows a B with d samples, which is used by default in the inference process. For example, inpPatchWidth and inpPatchHeight are related to the patch width (w) and height (h), and overlapSize is the additional margin d.

[0155] exist Fig. 7A On the right side of , an example is shown where the same block B is divided into “4” sub-blocks for the inference process: for example, when the original NN operates on the entire block B, the memory constraints on the decoder side do not allow the entire block B to be processed, but rather “4” separate inference processes are needed to process smaller blocks, such as sub-blocks B1 to B4.

[0156] Figure 7B The block-based reasoning process and the super-block-based reasoning process are schematically represented.

[0157] Figure 7B Shows Fig. 7A The inverse problem of : While the original NN acts on block B, the decoder is able to process more data in parallel and process larger blocks. Figure 7B In , the processing area of ​​the inference process consists of 4 blocks B1 to B4, thus forming a super block that is processed simultaneously.

[0158] As can be seen, the inference process requires flexibility. However, in the current signaling proposed in nnr_post_filter, some information is missing to achieve this flexibility, in particular to allow handling of sub-blocks and super-blocks during the inference process, while using blocks to define the original NN. The various embodiments described below propose solutions to this problem.

[0159] In particular, various embodiments introduce information that allows signaling to a decoder of possible block size variations on which an inference process may operate.

[0160] Figure 5A , Figure 5B and Figure 5C Examples of apparatus, devices and / or systems are described that enable implementation of various embodiments.

[0161] Figure 5A Schematically shows an example of a hardware architecture of a processing module 500 capable of implementing an encoding module or a decoding module, wherein the encoding module or the decoding module can respectively implement the modified according to different aspects and embodiments. Figure 3 The encoding method and Figure 4 For example, when system 11 is responsible for encoding a video stream, the encoding module is included in the system. The decoding module is included in system 13, for example.

[0162] The processing module 500 includes the following items connected by a communication bus 5005: a processor or CPU (central processing unit) 5000, which includes, as non-limiting examples, one or more microprocessors, general-purpose computers, special-purpose computers, and processors based on multi-core architectures; a random access memory (RAM) 5001; a read-only memory (ROM) 5002; a storage unit 5003, which may include non-volatile memory and / or volatile memory, including but not limited to electrically erasable programmable read-only memory (EEPROM), read-only memory (ROM), programmable read-only memory (PROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, magnetic disk drive and / or optical disk drive, or storage medium reader, such as SD (Secure Digital) card reader and / or hard disk drive (HDD) and / or network accessible storage device; at least one communication interface 5004 for exchanging data with other modules, devices or systems. The communication interface 5004 may include but is not limited to a transceiver configured to send and receive data through a communication channel. The communication interface 5004 may include, but is not limited to, a modem or a network card.

[0163] If the processing module 500 implements a decoding module, the communication interface 5004 enables, for example, the processing module 500 to receive an encoded video stream and provide a decoded picture sequence. If the processing module 500 implements an encoding module, the communication interface 5004 enables, for example, the processing module 500 to receive an original picture data sequence for encoding and provide an encoded video stream.

[0164] The processor 5000 can execute instructions loaded into the RAM 5001 from the following: ROM 5002, external memory (not shown), storage medium, or communication network. When the processing module 500 is powered on, the processor 5000 can read instructions from the RAM 5001 and execute them. These instructions form a computer program, so that, for example, the processor 5000 implements the following Figure 4 Decoding methods described and / or Figure 3 The encoding method described and the Fig.14A or Fig. 14B Described methods include various aspects and embodiments of this document described below.

[0165] about Figure 3 , Figure 4 , Fig.14A and Fig. 14BAll or part of the algorithms and steps of the described methods may be implemented in software by a programmable machine such as a DSP (digital signal processor) or a microcontroller executing a set of instructions, or in hardware by a machine or dedicated components such as an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit).

[0166] As can be seen, microprocessors, general purpose computers, special purpose computers, processors based on or not based on multi-core architectures, DSPs, microcontrollers, FPGAs, and ASICs are suitable for at least partially implementing Figure 3 , Figure 4 , Fig.14A and Fig. 14B Method of electronic circuit.

[0167] Figure 5C A block diagram of an example of a system 13 that implements various aspects and embodiments is shown. System 13 may be embodied as a device including various components described below, and is configured to perform one or more aspects and embodiments described in this document. Examples of such devices include, but are not limited to, various electronic devices, such as personal computers, laptop computers, smart phones, tablet computers, digital multimedia set-top boxes, digital television receivers, personal video recording systems, connected household appliances, and head-mounted displays. The elements of system 13 may be embodied in a single integrated circuit (IC), multiple ICs, and / or discrete components, either individually or in combination. For example, in at least one embodiment, system 13 includes one or more processing modules 500 that implement a decoding module. In various embodiments, system 13 is communicatively coupled to one or more other systems or other electronic devices via, for example, a communication bus or through dedicated input and / or output ports. In various embodiments, system 13 is configured to implement one or more aspects described in this document.

[0168] As indicated in block 531, inputs may be provided to the processing module 500 through various input modules. Such input modules include, but are not limited to, (i) a radio frequency (RF) module that receives an RF signal transmitted over the air, for example, by a broadcaster, (ii) a component (COMP) input module (or a group of COMP input modules), (iii) a universal serial bus (USB) input module, and / or (iv) a high-definition multimedia interface (HDMI) input module. Figure 5C Other examples not shown include composite video.

[0169] In various embodiments, the input module of block 531 has associated corresponding input processing elements known in the art. For example, the RF module may be associated with elements suitable for the following operations: (i) selecting a desired frequency (also referred to as selecting a signal, or band limiting a signal to a frequency band), (ii) down-converting the selected signal, (iii) again band-limiting to a narrower frequency band to select a signal frequency band that may be referred to as a channel in some embodiments, (iv) demodulating the down-converted and band-limited signal, (v) performing error correction, and (vi) demultiplexing to select a desired packet stream. The RF module of various embodiments includes one or more elements that perform these functions, such as a frequency selector, a signal selector, a frequency band limiter, a channel selector, a filter, a down-converter, a demodulator, an error corrector, and a demultiplexer. The RF portion may include a tuner that performs various of these functions, including, for example, down-converting a received signal to a lower frequency (e.g., an intermediate frequency or a near-baseband frequency) or baseband. In a set-top box embodiment, RF module and its associated input processing element receive the RF signal transmitted by wired (for example, cable) medium, and filter to the desired frequency band again by filtering, down-conversion and perform frequency selection.Various embodiments rearrange the order of above-mentioned (and other) elements, remove some in these elements and / or add other elements that perform similar or different functions.Adding element can include inserting element between existing element, for example, such as inserting amplifier and analog-to-digital converter.In various embodiments, RF module comprises antenna.

[0170] Additionally, the USB and / or HDMI modules may include respective interface processors for connecting the system 13 to other electronic devices via USB and / or HDMI connections. It should be appreciated that aspects of input processing (e.g., Reed-Solomon error correction) may be implemented, for example, within a separate input processing IC or within the processing module 500 as desired. Similarly, aspects of USB or HDMI interface processing may be implemented within a separate interface IC or within the processing module 500 as desired. The demodulated, error corrected, and demultiplexed stream is provided to the processing module 500.

[0171] The various elements of the system 13 may be disposed within an integrated housing. Within the integrated housing, the various elements may be interconnected and data may be transmitted between them using a suitable connection arrangement, such as an internal bus known in the art, including an inter-IC (I2C) bus, wiring, and a printed circuit board. For example, in the system 13, the processing module 500 is interconnected with the other elements of the system 13 via a bus 5005.

[0172] The communication interface 5004 of the processing module 500 allows the system 13 to communicate on the communication channel 12. As already mentioned above, the communication channel 12 may be implemented, for example, within a wired medium and / or a wireless medium.

[0173] In various embodiments, data is streamed or otherwise provided to the system 13 using a wireless network, such as a Wi-Fi network, e.g., IEEE 802.11 (IEEE refers to the Institute of Electrical and Electronics Engineers). The Wi-Fi signal of these embodiments is received via a communication channel 12 suitable for Wi-Fi communication and a communication interface 5004. The communication channel 12 of these embodiments is typically connected to an access point or router that provides access to external networks, including the Internet, to allow streaming applications and other top communications. Other embodiments use an RF connection of an input block 531 to provide streaming data to the system 13. As indicated above, various embodiments provide data in a non-streaming manner. Additionally, various embodiments use a wireless network other than Wi-Fi, such as a cellular network or a Bluetooth network.

[0174] System 13 can provide output signals to various output devices, including display system 15, speakers 535, and other peripheral devices 536. The display system 15 of various embodiments includes, for example, one or more of a touch screen display, an organic light emitting diode (OLED) display, a curved display, and / or a foldable display. Display 15 can be used for a television, a tablet computer, a laptop computer, a mobile phone (mobile phone), a head-mounted display, or other devices. Display system 15 can also be integrated with other components (for example, as in a smartphone), or separate (for example, an external monitor of a laptop computer). In various examples of embodiments, other peripheral devices 536 include one or more of a stand-alone digital video disc (or digital versatile disc) (DVR, for both terms), a disc player, a stereo system, and / or a lighting system. Various embodiments use one or more peripheral devices 536 that provide functions based on the output of system 13. For example, a disc player performs the function of playing the output of system 13.

[0175] In various embodiments, control signals are communicated between system 13 and display system 15, speakers 535, or other peripheral devices 536 using signaling such as AV.Link, Consumer Electronics Control (CEC), or other communication protocols that enable device-to-device control with or without user intervention. Output devices may be communicatively coupled to system 13 via dedicated connections through respective interfaces 532, 533, and 534. Alternatively, output devices may be connected to system 13 using communication channel 12 via communication interface 5004, or connected to a peripheral device 536 via communication interface 5004. Figure 5CThe display interface 532 is a dedicated communication channel corresponding to the communication channel 12 in the display system 15. The display system 15 and the speaker 535 can be integrated into a single unit with other components of the system 13 in an electronic device (e.g., such as a television). In various embodiments, the display interface 532 includes a display driver, such as a timing controller (T Con) chip, for example.

[0176] The display system 15 and the speaker 535 may optionally be separate from one or more of the other components. In various embodiments where the display system 15 and the speaker 535 are external components, output signals may be provided via dedicated output connections including, for example, an HDMI port, a USB port, or a COMP output.

[0177] Figure 5B A block diagram of an example of a system 11 that implements various aspects and embodiments is shown. System 11 is very similar to system 13. System 11 can be embodied as a device including various components described below, and is configured to perform one or more aspects and embodiments described in this document. Examples of such devices include, but are not limited to, various electronic devices, such as personal computers, laptop computers, smart phones, tablet computers, cameras, and servers. The elements of system 11 can be embodied in a single integrated circuit (IC), multiple ICs, and / or discrete components, either alone or in combination. For example, in at least one embodiment, system 11 includes one or more processing modules 500 that implement a coding module. In various embodiments, system 11 is communicatively coupled to one or more other systems or other electronic devices via, for example, a communication bus or through dedicated input and / or output ports. In various embodiments, system 11 is configured to implement one or more aspects described in this document.

[0178] As indicated in block 531 already described with respect to FIG. 5D , input may be provided to the processing module 500 through various input modules.

[0179] The various elements of the system 11 may be disposed within an integrated housing. Within the integrated housing, the various elements may be interconnected and data may be transmitted therebetween using a suitable connection arrangement, such as an internal bus known in the art, including an inter-IC (I2C) bus, wiring, and a printed circuit board. For example, in the system 11, the processing module 500 is interconnected with the other elements of the system 11 via a bus 5005.

[0180] The communication interface 5004 of the processing module 500 allows the system 11 to communicate over the communication channel 12 .

[0181] In various embodiments, a wireless network, such as a Wi-Fi network, e.g., IEEE 802.11 (IEEE refers to the Institute of Electrical and Electronics Engineers), is used to stream or otherwise provide data to the system 11. The Wi-Fi signals of these embodiments are received through a communication channel 12 and a communication interface 5004 suitable for Wi-Fi communications. The communication channel 12 of these embodiments is typically connected to an access point or router that provides access to external networks, including the Internet, to allow streaming applications and other top communications. Other embodiments use the RF connection of the input block 531 to provide streaming data to the system 11.

[0182] As indicated above, various embodiments provide data in a non-streaming manner.Additionally, various embodiments use wireless networks other than Wi-Fi, such as cellular networks or Bluetooth networks.

[0183] The data provided to the system 11 may be provided in different formats. In various embodiments, these data are encoded and conform to known video compression formats, such as AV1, VP9, ​​VVC, HEVC, AVC, etc. In various embodiments, these data are raw data provided, for example, by a picture and / or audio acquisition module connected to the system 11 or included in the system 11. In this case, the processing module 500 is responsible for the encoding of these data.

[0184] System 11 may provide output signals to various output devices, such as system 13, which may be capable of storing and / or decoding the output signals.

[0185] Various implementations involve decoding. "Decoding" as used in this application can cover, for example, all or part of a process performed on a received encoded video stream to produce a final output suitable for display. In various embodiments, such processes include one or more of the processes typically performed by a decoder, such as entropy decoding, inverse quantization, inverse transform, and prediction. In various embodiments, such processes also include or alternatively include processes performed by a decoder of the various implementations described in this application, for example, to apply an NN-based coding mode, such as NN-based post-processing or an in-loop filter.

[0186] Whether the phrase "decoding process" is intended to refer specifically to a subset of operations or generally to a broader decoding process will be clear based on the context of the specific description and is believed to be well understood by those skilled in the art.

[0187] Various implementations involve encoding. In a manner similar to the above discussion about "decoding", "encoding" as used in this application can cover, for example, all or part of a process performed on an input video sequence to produce an encoded video stream. In various embodiments, such processes include one or more of the processes typically performed by an encoder, such as segmentation, prediction, transformation, quantization, and entropy coding. In various embodiments, such processes also include or alternatively include processes performed by encoders of the various implementations described in this application, for example, to signal to a decoder possible block size changes that an inference process based on an NN post-processing or in-loop filter can operate.

[0188] Whether the phrase "encoding process" is intended to refer specifically to a subset of operations or generally to a broader encoding process will be clear based on the context of the specific description and is believed to be well understood by those skilled in the art.

[0189] It should be noted that the syntactic element names used herein are descriptive terms. Therefore, they do not exclude the use of other syntactic element names. For example, in the following, the following syntactic element names nn_subblock_dx, nn_subblock_dy, nn_subblock_dxb, nn_subblock_dyb are used. These names can be replaced by nn_subblock_dL, nn_subblock_dT, nn_subblock_dxR, nn_subblock_dyB respectively.

[0190] When a figure is presented as a flow chart, it should be understood that it also provides a block diagram of the corresponding device. Similarly, when a figure is presented as a block diagram, it should be understood that it also provides a flow chart of the corresponding method / device.

[0191] Various embodiments relate to rate-distortion optimization. In particular, during the encoding process, a balance or trade-off between rate and distortion is generally considered. Rate-distortion optimization is generally formulated as minimizing a rate-distortion function, which is a weighted sum of rate and distortion. There are different approaches to solving the rate-distortion optimization problem. For example, these approaches can be based on extensive testing of all coding options, including all considered modes or coding parameter values, wherein a comprehensive evaluation of the coding cost and the associated distortion of the reconstructed signal is performed after encoding and decoding. A faster approach can also be used to save coding complexity, particularly to calculate approximate distortion based on a prediction or prediction residual signal rather than a reconstructed signal. A mixture of these two approaches can also be used, such as by using approximate distortion only for some possible coding options and using full distortion for other coding options. Other approaches only evaluate a subset of possible coding options. More generally, many approaches employ any of a variety of techniques to perform optimization, but optimization is not necessarily a comprehensive evaluation of coding costs and associated distortions.

[0192] The implementations and aspects described herein may be implemented in, for example, a method or process, a device, a software program, a data stream, or a signal. Even if discussed only in the context of a single implementation (e.g., discussed only as a method), the implementation of the features discussed may also be implemented in other forms (e.g., a device or program). The device may be implemented, for example, with appropriate hardware, software, and firmware. The method may be implemented, for example, in a processor, which generally refers to a processing device, including, for example, a computer, a microprocessor, an integrated circuit, or a programmable logic device. The processor also includes a communication device, for example, such as a computer, a cellular phone, a portable / personal digital assistant ("PDA"), and other devices that facilitate information communication between end users.

[0193] Reference to "one embodiment" or "an embodiment" or "an implementation" or "implementation" and other variations thereof means that a particular feature, structure, characteristic, etc. described in connection with the embodiment is included in at least one embodiment. Thus, the appearances of the phrases "in one embodiment" or "in an embodiment" or "in one implementation" or "in an implementation" and any other variations appearing in various places throughout this application are not necessarily all referring to the same embodiment.

[0194] Additionally, the present application may refer to "determining" various information. Determining information may include, for example, estimating information, calculating information, predicting information, retrieving information from a memory, or obtaining information, for example, from another device, module, or user.

[0195] Furthermore, the present application may refer to "accessing" various information. Accessing information may include one or more of: for example, receiving information, retrieving information (e.g., from a memory), storing information, moving information, copying information, calculating information, determining information, predicting information, or estimating information.

[0196] Additionally, the present application may involve "receiving" various information. Like "accessing," receiving is a broad term. Receiving information may include one or more of: for example, accessing information or retrieving information (e.g., retrieving from a memory). Moreover, "receiving" is often involved in one way or another during operation, such as, for example, storing information, processing information, transmitting information, moving information, copying information, erasing information, calculating information, determining information, predicting information, or estimating information.

[0197] It should be understood that, for example, in the case of "A / B", "A and / or B", and "at least one of A and B", "one or more of A and B", the use of any of the following " / ", "and / or", and "at least one", "one or more" is intended to cover selecting only the first listed option (A), or selecting only the second listed option (B), or selecting both options (A and B). As another example, in the case of "A, B, and / or C" and "at least one of A, B, and C", "one or more of A, B, and C", such wording is intended to cover selecting only the first listed option (A), or selecting only the second listed option (B), or selecting only the third listed option (C), or selecting only the first and second listed options (A and B), or selecting only the first and third listed options (A and C), or selecting only the second and third listed options (B and C), or selecting all three options (A and B and C). For ordinary technicians in this and related fields, this can be extended to as many listed items as possible.

[0198] In addition, as used herein, the word "signaling" especially refers to indicating something to the corresponding decoder. For example, in some embodiments, the encoder signals the use of some coding tools. In this way, in an embodiment, the same parameters can be used on both the encoder side and the decoder side. Therefore, for example, the encoder can send (explicit signaling) specific parameters to the decoder so that the decoder can use the same specific parameters. On the contrary, if the decoder already has specific parameters and other parameters, signaling can be used without sending (implicit signaling) to simply allow the decoder to know and select specific parameters. By avoiding the transmission of any actual function, bit saving is achieved in various embodiments. It should be understood that signaling can be implemented in a variety of ways. For example, in various embodiments, one or more syntactic elements, signs, etc. are used to send information to the corresponding decoder. Although the verb form of the word "signaling (signal)" is related to the above, the word "signal (signal)" can also be used as a noun in this article.

[0199] It will be apparent to one of ordinary skill in the art that implementations may generate a variety of signals formatted to carry information that may be stored or transmitted, for example. The information may include, for example, instructions for executing a method, or data generated by one of the described implementations. For example, a signal may be formatted to carry an encoded video stream and SEI message of the described embodiment. Such a signal may be formatted as, for example, an electromagnetic wave (e.g., using a radio frequency portion of a spectrum) or a baseband signal. Formatting may include, for example, encoding an encoded video stream and modulating a carrier with the encoded video stream. The information carried by the signal may be, for example, analog or digital information. As is well known, a signal may be transmitted over a variety of different wired or wireless links. The signal may be stored on a processor-readable medium.

[0200] In the following, various embodiments propose to signal to the decoder various patch dimensions on which the inference process of the NN-based process may operate.

[0201] Fig.14A The application of an embodiment during an encoding process is schematically represented.

[0202] Fig.14A The embodiment of is executed by, for example, the processing module 500 of the system 11.

[0203] In step 1401 , the processing module 500 obtains original video data to be encoded.

[0204] In step 1402, the processing module 500 uses, for example, Figure 3 The encoding process encodes the raw video data into an encoded video stream.

[0205] In step 1403, the processing module 500 signals information referred to as NN information in the form of metadata associated with the encoded video stream, which represents the allowable margin around the patch for the inference process of the NN-based image processing tool.

[0206] In step 1404, the processing module 500 provides the encoded video stream with the signaled information in the form of metadata to the receiving device.

[0207] In an embodiment, the encoding process of step 1402 includes at least one NN-based image processing tool, such as, for example, at least one NN-based in-loop filtering process.

[0208] In an embodiment, the signaled NN information of step 1403 provides information about each NN-based image processing tool performed during the encoding process of step 1402, or about at least one NN-based post-processing process implemented by the receiving device on the decoded video data.

[0209] In an embodiment, the NN information is signaled in a SEI message, wherein the NN information is represented in the form of syntax elements described with respect to the following Tables TAB2, TAB3 and TAB4.

[0210] Fig. 14B The application of the embodiments during the decoding process is schematically represented.

[0211] Fig. 14B The embodiment of is executed by, for example, the processing module 500 of the system 13.

[0212] In step 1411 , the processing module 500 obtains a video stream.

[0213] In step 1412, the processing module 500 obtains signaled NN information in the form of metadata associated with the video stream, the information representing an allowable margin around a patch for at least one inference process of a NN-based image processing tool.

[0214] In step 1413, the processing module 500 applies, for example, Figure 4 Each NN-based image processing tool is applied by the processing module 500 during the decoding process (e.g., during an in-loop filtering process) or during post-processing of the decoded video data using information representing the allowable margin around the patch of the inference process of the NN-based image processing tool.

[0215] In step 1414 , the processing module 500 provides the decoded video data to a display module, such as the display system 15 .

[0216] In the following, several types of additional NN information are proposed, including receptive field information and offsets. The syntax of SEI messages suitable for transmitting these types of information is also proposed. However, as described below, the transmission of NN information is not limited to SEI messages.

[0217] Receptive field information :

[0218] As mentioned above, the NN used in the NN-based image processing tool is usually defined (trained) using a given block (or patch) size of the input and output of the inference process. When the input or output block (or patch) size of the inference process is changed relative to the original input or output block (or patch) size considered when defining the NN, it is necessary to know which area is to be considered for the input of the inference process in order to calculate the output whose sample values ​​are the same as the corresponding sample values ​​of the output provided by the inference process when applied to a block (or patch) of a given size. In the following, the term receptive field is used to represent the additional margin value used to define the margin around the block (or patch) so that all values ​​within the block (or patch) at the output depend only on the input block (or patch) plus the margin. In general, the receptive field value depends on the NN network. In this case, we call it the NN receptive field. However, in the following, a broader definition is used, in which the receptive field is large enough to define the value of the margin around each sample of the block (or patch) that is equal to or includes the NN receptive field. In the following, the words block or patch are used indifferently, and the block is a special case of the patch.

[0219] Depending on whether a subblock or a superblock is considered, several situations arise.

[0220] In with Fig. 7A The illustrated case corresponds to the first embodiment in which the original inference process of the NN-based image processing tool considers a block B with a given overlapSize of wxh samples and d samples, but the inference process is applied to a sub-block of the block B, thereby applying a separate inference process. w and h can be different, so that the block and the sub-block are rectangular, or can be equal, so that the block and the sub-block are square.

[0221] In a first embodiment, when the margin overlapSize is smaller than the receptive field of the NN (ie, the NN receptive field), an asymmetric margin around the sub-block is used.

[0222] Fig. 8A Schematically illustrating an example of region expansion of a sub-block based reasoning process.

[0223] exist Fig. 8A In the example, the original block B is divided into 4 square sub-blocks B1 to B4 of equal size. Fig. 8A Represents two top sub-blocks B1 and B2. For sub-block B1:

[0224] The top margin and left margin are d, the same as the margin of the original block B;

[0225] The bottom margin and right margin are defined with a single value r, where r corresponds to the receptive field of the NN.

[0226] For sub-block B2, the left margin and bottom margin use the receptive field size r, and the right margin and top margin use the margin value d.

[0227] It may be noted that block B may correspond to the entire picture.

[0228] In terms of syntax, in a first embodiment, it is proposed to add a syntax element nn_receptive_field to the existing syntax of the SEI message described in table TAB1 for encoding a value representing the receptive field value r. In this case, the margin around a patch (such as block B1, B2, B3 or B4) is defined by the margin value d and the receptive field value r provided by the syntax element nn_receptive_field.

[0229] In a variation of the syntax suitable for the first embodiment, the syntax element nn_receptive_field is replaced by the syntax element nn_receptive_field_plus1. The value of the receptive field is then nn_receptive_field_plus1 - 1. The value of the receptive field is "-1" (ie, nn_receptive_field_plus1 = 0) indicating that the NN has no receptive field.

[0230] In a variation of the first embodiment, the receptive field may have different horizontal and vertical ranges depending on the architecture of the NN. For example, if a neural network contains a convolutional layer whose horizontal convolution stride is twice as large as the vertical convolution stride, the convolutional layer may cause the receptive field of the NN to be different horizontally and vertically. Figure 8B Consider this situation.

[0231] Figure 8B Schematically represents an example of region expansion of the sub-block based inference process when the receptive field of the NN is characterized by different horizontal and vertical extents.

[0232] For sub-block B1:

[0233] The top margin and left margin are d, the same as the margin of the original block B;

[0234] The bottom margin is defined as r h And the right margin is defined as r w , r h and r w Define the receptive field vertically and horizontally separately.

[0235] For sub-block B2, the right margin is equal to r w , the bottom margin is equal to r h , and the right margin and top margin use margin d.

[0236] Regarding the syntax related to the variant of the first embodiment, it is proposed to add the syntax element nn_receptive_field_vertically for encoding the value r h and the syntax element nn_receptive_field_horizontally for encoding the value r w to the existing syntax of table TAB1.

[0237] In the second embodiment corresponding to the case shown Figure 7B the original inference process of the NN-based image processing tool considers a block B with wxh samples and a margin overlapSize of d samples, but this inference process is applied to the superblock of block B. The superblock B can have any size, including the size corresponding to the entire frame constituting block B. In the case of the second embodiment, in order to recover the same result as the processing based on block B, the margin needs to be greater than or equal to the receptive field of the NN.

[0238] Fig. 9 Schematically represents the receptive field and boundary of block B.

[0239] In Fig. 9 a block with a margin of d samples and a receptive field of r samples is shown. When d is less than r, it is assumed that the samples in the margin between d and r are equal to a specific value. Usually, zero-padding is used, that is, it is assumed that the samples outside the block are zero. Other padding (mirroring, etc.) can be used. In this case (when d is less than r), without changing the sample values, blocks larger than the size defined in the original NN cannot be processed.

[0240] In the first variant of the second embodiment, the ability to process blocks larger than the block size considered during the definition of the NN network used in the NN-based image processing tool is determined by comparing the receptive_field value r with the overlap overlapSize value d. If d < r, blocks of larger size cannot be processed. Otherwise, if d ≥ r, blocks of larger size can be processed.

[0241] In the variant of the second embodiment, the ability to process blocks larger than the block size considered during the definition of the NN network used in the NN-based image processing tool is explicitly signaled by adding the flag nn_possible_superblock_processing_flag to the syntax of table TAB1.

[0242] Offset information :

[0243] Problems arise when NN-based image processing tools use a NN that contains some tensor size changes during the NN inference process. Fig.10 Example schematically representing a NN with varying tensor sizes:

[0244] Assume that a 3D (three-dimensional) image corresponding to a square block B 100 of size w×w and depth D

[0245] The input tensor in the form of tensor is used as input for the NN inference process;

[0246] First, in step 101, a 2D convolution using a kernel of size 3×3 and a stride of “1” is applied to the input tensor. The stride is a parameter of the NN that modifies the amount by which the convolution kernel moves over the image or video.

[0247] Then in step 102, max pooling with a stride of "2" is applied to the output tensor of step 101. Max pooling is a pooling operation that computes the maximum value of a sub-part of a tensor and uses that maximum value to create a downsampled (pooled) tensor. This max pooling is usually applied after a convolutional layer. It adds a small amount of translation invariance, i.e., translating the image by a small amount does not significantly affect the value of most pooled outputs. After this max pooling layer, the subsampled tensor is w / 2×w / 2×D, where w / 2 is an integer division and D is the depth of the subsampled tensor. It can be noted that max pooling can be replaced by any reduction layer (convolution with stride, other pooling with stride, etc.).

[0248] In step 103, a 2D convolution using a kernel of size 3×3 and stride “1” is applied to the downsampled tensor;

[0249] In step 104, upscaling is applied, for example, a pixel shuffle method (exchanging width / height with depth of the tensor), a transposed convolution with stride "2", or interpolation, etc. After upscaling, the size of the upscaled tensor is w×w×D', where D' is the depth of the tensor.

[0250] In step 105, a 2D convolution using a kernel of size 3×3 and stride “1” is applied to the upscaled tensor to obtain an output tensor. The resulting block B′ is then extracted from the output tensor.

[0251] When the NN inference process includes some internal tensor size changes, some output tensor properties may differ from the input tensor properties. This is usually Fig.10 The situation of NN inference process.

[0252] These characteristics include the output tensor size which may be different from the input tensor size, and / or the sample positions in the output tensor may differ from the corresponding sample positions in the input tensor by an offset value. These differences may be caused by the parameters of the NN, such as the stride strategy during NN inference, the eventual use of padding, etc.

[0253] For example, let us consider Fig.10 The situation shown:

[0254] The original block size of block B is 16×16, where the margin is 2, i.e., the input size is 20×20;

[0255] After the max pooling step 102, the subsampled tensor size is 10×10×D;

[0256] After the upscaling step 104, the upscaled tensor size returns to 20×20;

[0257] For this particular NN, the receptive field is "5".

[0258] When considering the case of the first embodiment with sub-blocks B1 to B4, the input size will be:

[0259] Sub-block size = 16 / 2 × 16 / 2 = 8 × 8;

[0260] For the upper left sub-block B1:

[0261] The margin is "2";

[0262] The margin at the bottom and right is "5" (equal to the receptive field);

[0263] This gives an input tensor size of (2+8+5)×(2+8+5)=15×15.

[0264] In the case of sub-block B1, by Fig.10 The output sub-block B1' obtained by applying the process of Fig.10 The process is applied to the entire block B and the sub-block B1' extracted from the resulting block B' is similar.

[0265] However, for the top right sub-block B2:

[0266] The upper right boundary, i.e., the margin is 2;

[0267] The margin on the bottom and left is 5 (equal to the receptive field);

[0268] This gives a full input tensor size of (5+8+2)×(2+8+5)=15×15

[0269] However, due to Fig.10The offset introduced during the process is Fig.10 The output sub-block B2″ obtained by applying the process of to the sub-block B2 is different from the sub-block B2′ extracted from the block B′.

[0270] Fig.13 Schematic illustration of the impact of the bias introduced during NN inference with some tensor size variations.

[0271] In this case, the size of block B is 8×8, the value of margin d is “2”, and the value of receptive field r is “5”. Fig. 7A In , block B is divided into four sub-blocks B1 to B4 of size 4×4. Therefore, the margin around block B is two samples. The left margin and top margin of sub-block B1 are two samples. The right margin and bottom margin of sub-block B1 are five samples. The right margin and top margin of sub-block B2 are two samples. The left margin and bottom margin of sub-block B2 are five samples.

[0272] Block B and sub-blocks B1 and B2 are input to Fig.10 NN inference process, assuming that the process has been designed to process blocks of size 8×8. In the case of block B, the input tensor size is (2+2+8)×(2+2+8)=12×12. In the case of blocks B1 and B2, the input tensor size is (2+4+5)×(2+4+5)=11×11. Fig.10 As can be seen from the description of , the NN inference process includes a reduction step 102. Assuming that Fig.13 The reduction process used in the case of takes one sample from both input tensors starting with the top left sample. Fig.13 In FIG. 1 , samples marked with “B” represent samples retained by the downsampling process of step 102 when applied to block B. Samples marked with “B1” represent samples retained by the downsampling process of step 102 when applied to sub-block B1. Samples marked with “B2” represent samples retained by the downsampling process of step 102 when applied to sub-block B2. As can be seen, when Fig.10 When the NN inference process of is applied to block B and sub-block B1, the same samples are retained by the downsampling process. Fig.10 When the NN inference process is applied to sub-block B2, the samples retained by the downsampling process are the same as those in Fig.10 The samples retained by the downsampling process are different when the NN inference process of is applied to sub-block B. This explains why when applied to sub-block B2 Fig.10 The application of the NN inference process does not provide the same Fig.10 The NN inference process has the same result.

[0273] In this example, in order to apply at the full block level or sub-block level Fig.10To systematically restore the same sub-blocks during the inference process, the following parameters are used:

[0274] For the top right border, the margin should be 2;

[0275] The bottom margin should be "5";

[0276] The left margin should be 5+1;

[0277] This gives a full input tensor size of (6+8+2)×(2+8+5)=16×15

[0278] As can be seen in this example, more parameters need to be signaled to better account for these offsets.In a third embodiment, additional parameters specify the input size of the sub-blocks and the offsets in the output tensor.

[0279] about Fig.11 Examples of added parameters according to the first variation of the third embodiment are shown and provided in the following table TAB2.

[0280] Fig.11 Schematically showing an example layout of the sub-block inference process for the lower right sub-block.

[0281] exist Fig.11 In FIG. 4 , an example of using “4” sub-blocks to process an original block B with an overlap size (overlapSize) of d is shown. Fig.11 In FIG. 5 , the parameters of the lower right sub-block B4 are shown. dT, dB, dL, dR are the top margin, bottom margin, left margin, and right margin, respectively. These margins are usually obtained by adding / subtracting the offset values ​​ox and oy to the horizontal and vertical margins / receptive field values ​​of the original block.

[0282] Using an offset allows encoding the residuals with respect to default parameters (depending on the original residuals and the receptive field). By denote the receptive field of the NN by r and the original residual by d, the residuals of the "4" sub-blocks B1 to B4 of block B are calculated using the following equation (as Fig. 7A shown):

[0283] Block B1:

[0284] o dT = d + oy1;

[0285] o dL=d+ox1;

[0286] o dB = r;

[0287] o dR = r;

[0288] Block B2:

[0289] o dT = d + oy2;

[0290] o dL = r + ox2;

[0291] o dB = r;

[0292] o dR = d;

[0293] Block B3:

[0294] o dT=r+oy3;

[0295] o dL=d+ox3;

[0296] o dB = d;

[0297] o dR = r;

[0298] Block B4:

[0299] o dT = r + oy4;

[0300] o dL=r+ox4;

[0301] o dB = d;

[0302] o dR = d;

[0303] Typically, the parameters are:

[0304] ox1=oy1=0;

[0305] ox2=1,oy2=0;

[0306] ox3=0,oy3=1;

[0307] ox4=oy4=1;

[0308] In a second variation of the second embodiment, the left offset dx1 (respectively the top offset dy1) and the right offset dx1b (respectively the bottom offset dy1b) are different:

[0309] Block B1:

[0310] o dT = d + oy1;

[0311] o dL=d+ox1;

[0312] o dB = r + oy1b;

[0313] o dR=r+ox1b;

[0314] Block B2:

[0315] o dT = d + oy2;

[0316] o dL = r + ox2;

[0317] o dB = r + oy2b;

[0318] o dR=d+ox2b;

[0319] Block B3:

[0320] o dT=r+oy3;

[0321] o dL=d+ox3;

[0322] o dB = d + oy3b;

[0323] o dR=r+ox3b;

[0324] Block B4:

[0325] o dT = r + oy4;

[0326] o dL=r+ox4;

[0327] o dB = d + oy4b;

[0328] o dR=d+ox4b;

[0329] A second variant of the third embodiment is associated with an additional syntax provided in table TAB3.

[0330] In a variant, the margins are defined absolutely rather than relative to the margin, ie for each sub-block Bi the margins dT, dB, dL and dR are signalled.

[0331] It can be noted that the input size of the sub-block is derived from the size of the original block.

[0332] In a third variation of the third embodiment, additional parameters are signaled to gain more flexibility. Fig.12 An embodiment in which the output is offset wx and wy is schematically represented. wx and wy represent the vertical and horizontal offsets of the blocks within the output tensor, respectively, to extract the output block B'. ​​In other words, wx and wy represent the position of the output block B' in the output tensor of the NN inference process. The above offsets are typically applied when the output size of a layer in the NN (e.g., a convolutional layer) is reduced compared to the input tensor, because only operations that use samples within the input tensor are used.

[0333]

[0334] Table TAB2

[0335] Table TAB2 represents a first embodiment of an additional syntax to be added to the syntax of table TAB1. The following semantics are used:

[0336] nn_receptive_field: Specifies half the size of the network's receptive field. It represents the number of samples around the input such that translating the input produces the same useful output. It should be noted that the border or overlap parameter is usually less than or equal to the receptive field. nn_receptive_field values ​​range from "0" to nn_patch_size / 2. Assuming the input patch size is nn_patch_size and the border around the input patch is border_size, the output patch size is assumed to be equal to the nn_patch_size obtained by cropping the center portion of the output tensor.

[0337] nn_possible_superblock_processing: When set to "1", it allows the decoder to perform processing on larger block sizes using multiples of nn_patch_size plus border_size. For example, the input tensor will have width and height:

[0338] o W=nn_patch_size*2+border_size;

[0339] o H=nn_patch_size*2+border_size.

[0340] nn_possible_halfblock_processing: When set to "1", it allows the decoder to perform processing on smaller (half) block sizes. For example, the input tensor will have width and height (without margin):

[0341] o W = nn_patch_size / 2;

[0342] o H = nn_patch_size / 2;

[0343] Assume that the sub-block Bi (where Bi is from "1" to "4") is as follows Fig. 7A As described.

[0344] Using nn_half_block_dx[i] and nn_half_block_dy[i] in the first variation of the third embodiment

[0345] The boundaries of each sub-block Bi are derived by setting oxi = nn_half_block_dx[i] and oyi = nn_half_block_dy[i]. Set r to nn_receptive_field and d to border_size:

[0346] The output samples are obtained by clipping the output with the margin defined as the input. By default, the output samples are located at the position of the input samples, thus removing the added margin.

[0347] Table TAB3 represents a second embodiment of an additional syntax to be added to the syntax of table TAB1.

[0348]

[0349]

[0350] Table TAB3

[0351] The following semantics are used:

[0352] In the second variation of the third embodiment, nn_half_block_dx[i], nn_half_block_dy[i], nn_half_block_dxb[i], and nn_half_block_dyb[i] are used by setting oxi=nn_half_block_dx[i], oyi=nn_half_block_dy[i], oxib=

[0353] nn_half_block_dxb[i] and oyib=nn_half_block_dyb[i] to derive the boundary of each sub-block Bi. Set r to nn_receptive_field and d to border_size.

[0354] Table TAB4 represents a third embodiment of additional syntax to be added to the syntax of Table TAB1. The additional syntax of Table TAB4 shows that extension to quarter or even smaller sub-block sizes is straightforward: the margin around each sub-block is signaled using the same syntax as shown in the table below for quarter size.

[0355]

[0356]

[0357] Table TAB4

[0358] The additional syntax of Tables TAB2, TAB3, and TAB4 can be applied to a post-processing NN inference process, or to any NN inference process in the prediction loop of an encoder or decoder (such as an in-loop filter).

[0359] It may be noted that so far in this disclosure, block B has been divided into four or "16" sub-blocks. Table TAB5 below provides syntax to be added to table TAB1 to allow a flexible syntax for dividing a block into 4, 16, 64, 256, etc. sub-blocks.

[0360]

[0361] Table TAB5

[0362] In table TAB5, nn_nb subdivision_minus1+1 is the number of possible recursive subdivisions of the patch. Each offset of each level is read in the corresponding variable nn_sublock dzz[level][i], where dzz is dx, dy, dxb or dyb. The value of nn_nb subdivision_minus1+1 is in the range of "1" to floor(log2(nn_patch_size)). When not present, the value is inferred to be "0".

[0363] When there are possible sub-blocks, for a particular level L, where L is in the range of "1" to nn_nb_subdivision_minus1+1, each sub-block of size (nn_patch_size >> L) is used as input to the NN following the already described procedure with the following margins:

[0364] ·dT[i]=defaultMarginT+nn_subblock_dy[L][i];

[0365] ·dL[i]=defaultMarginL+nn_subblock_dx[L][i];

[0366] ·dB[i]=defaultMarginB+nn_subblock_dyb[L][i];

[0367] ·dR[i]=defaultMarginR+nn_subblock_dxb[L][i].

[0368] As a reminder, dT, dL, dB, and dR are top margin, left margin, bottom margin, and right margin, respectively, and:

[0369] defaultMarginT is equal to d(original block border_size) for the sub-block at the top of the original block, and equal to r(receptive field) otherwise;

[0370] For the sub-blocks on the left side of the original block, defaultMarginL is equal to d(original block border_size), otherwise it is equal to r(receptive field);

[0371] For the sub-blocks on the right side of the original block, defaultMarginR is equal to d(original block border_size), otherwise it is equal to r(receptive field);

[0372] For the sub-block at the bottom of the original block, defaultMarginB is equal to d(original block border_size),

[0373] Otherwise it is equal to r (receptive field).

[0374] Table TAB6 provides the same flexible syntax when the syntax element nn_receptive_field is replaced by the syntax element nn_receptive_field_plus1.

[0375]

[0376]

[0377] Table TAB6

[0378] In addition, so far, the shape of the sub-block depends on the shape of block B. If block B is a square, then the sub-block is a square. If block B is a rectangle, then the sub-block is a rectangle. The syntax proposed in this disclosure is also suitable for the case where the shape of the sub-block is independent of the shape of block B. For example:

[0379] • A square block B can be divided into rectangular sub-blocks.

[0380] • A rectangular block B can be divided into square sub-blocks.

[0381] • Block B can also be divided into any combination of square and rectangular sub-blocks of various sizes.

[0382] This disclosure describes various NN information that can be transmitted or stored, such as syntax, for example. The NN information can be packaged or arranged in a variety of ways, including, for example, ways commonly used in video standards, such as placing the NN information in an SEI message (such as the SEI message described in this disclosure). Other ways are also available, including, for example, ways common to system-level or application-level standards, such as placing the NN information in the following items:

[0383] SDP (Session Description Protocol), i.e., a format for describing multimedia communication sessions for session announcement and session invitation purposes, e.g. as described in RFCs and in conjunction with RTP (Real-time Transport Protocol)

[0384] Used for transmission.

[0385] • DASH MPD (Media Presentation Description) descriptor, e.g. as used in DASH and transmitted over HTTP. A descriptor is associated with a representation or a set of representations to provide additional characteristics to the content representation.

[0386] • RTP header extensions, e.g. as used during RTP streaming.

[0387] - ISO Base Media File Format, e.g. as used in OMAF and which uses boxes of object-oriented building blocks defined by a unique type identifier and length, also referred to as atoms in some specifications.

[0388] HLS (HTTP Live Streaming) manifests delivered over HTTP. A manifest is associated with a version or set of versions of the content to provide characteristics of the version or set of versions.

[0389] Many embodiments are described above. The features of these embodiments may be provided individually or in any combination. In addition, embodiments may include one or more of the following features, devices, or aspects, individually or in any combination, across various claim categories and types:

[0390] • A bitstream or signal comprising one or more of the described syntax elements or variations thereof.

[0391] • Creating and / or transmitting and / or receiving and / or decoding a bitstream or signal comprising one or more of the described syntax elements or variations thereof.

[0392] • TV, set-top box, mobile phone, tablet or other electronic device that implements at least one of the described embodiments.

[0393] A TV, set-top box, mobile phone, tablet, or other electronic device that executes at least one of the described embodiments and displays (e.g., using a monitor, screen, or other type of display)

[0394] The resulting picture.

[0395] • A TV, set-top box, mobile phone, tablet or other electronic device that tunes (eg, using a tuner) a channel to receive a signal including an encoded video stream and performs at least one of the described embodiments.

[0396] TV, set-top box, mobile phone, tablet or other electronic device that receives the signal over the air (e.g.

[0397] Using an antenna) a signal including an encoded video stream and performing at least one of the described embodiments.

[0398] A server, camera, phone, tablet, or other electronic device that transmits over the air (eg, using an antenna) a signal comprising an encoded video stream and performs at least one of the described embodiments.

[0399] • A server, camera, cell phone, tablet or other electronic device that tunes (eg, using a tuner) a channel transmitting a signal including an encoded video stream and performs at least one of the described embodiments.

Claims

1. A method, include: Get (1411) video stream; obtaining (1412) metadata associated with the video stream, the metadata representing a first margin around a patch for use in an inference process of a neural network based image processing tool; and The video stream is decoded using the metadata by applying the neural network-based image processing tool (1413).

2. The method according to claim 1, in, The first margin depends on a receptive field, which depends on a neural network used in the neural network based image processing tool.

3. The method according to claim 2, in, The metadata comprises at least one syntax element representing the receptive field depending on the neural network used in the neural network based image processing tool.

4. The method according to claim 3, in, The at least one syntax element includes: a first syntax element vertically defining the receptive field and a second syntax element horizontally defining the receptive field.

5. A method according to any of the preceding claims 2 to 4, in, The ability of the inference process to process patches of a larger size than those considered during the definition of the neural network used in the neural network-based image processing tool is determined by comparing at least one value representing the receptive field dependent on the neural network to a margin around the patch considered during the definition of the neural network used in the neural network-based image processing tool.

6. A method according to any preceding claim from claim 2 to claim 4, in, The ability of the inference process to handle patches of larger sizes than considered during definition of the neural network used in the neural network-based image processing tool is specified in the metadata by syntax elements.

7. A method according to any preceding claim from claim 2 to claim 6, in, The metadata includes at least one syntax element representing at least one offset added to a value representing the receptive field depending on the neural network or to a second margin around a patch considered during definition of the neural network used in the neural network-based image processing tool, the offset being used in response to processing a current patch by the inference process based on a position of the current patch, the current patch having a size smaller than a patch size considered during definition of the neural network used in the neural network-based image processing tool.

8. A method according to any preceding claim, in, The metadata includes at least one syntax element, and the at least one syntax element represents a position of an output patch of the inference process of the neural network-based image processing tool in an output tensor generated by the inference process.

9. A method, include: Get (1402) the video stream; as well as Information representing a first margin around the patch for use in an inference process of a neural network based image processing tool is signaled (1403) in the form of metadata associated with the video stream.

10. The method according to claim 9, in, The first margin depends on a receptive field, which depends on a neural network used in the neural network based image processing tool.

11. The method according to claim 10, in, The metadata comprises at least one syntax element representing the receptive field depending on the neural network used on the neural network based image processing tool.

12. The method according to claim 11, in, The at least one syntax element includes: a first syntax element vertically defining the receptive field and a second syntax element horizontally defining the receptive field.

13. A method according to any preceding claim from claim 10 to 12, in, Determining the ability of the inference process to process patches of a larger size than those considered during definition of the neural network used in the neural network-based image processing tool by comparing at least one value representing the receptive field dependent on the neural network to a second margin around the patch considered during definition of the neural network used in the neural network-based image processing tool.

14. A method according to any preceding claim from claim 10 to 12, in, The ability of the inference process to handle patches of larger sizes than considered during definition of the neural network used in the neural network-based image processing tool is specified in the metadata by syntax elements.

15. A method according to any preceding claim from claim 9 to claim 14, in, The metadata includes at least one syntax element representing at least one offset added to a value representing the receptive field depending on the neural network or added to a margin around a patch considered during definition of the neural network used in the neural network-based image processing tool, the offset being used in response to processing a current patch by the inference process based on a position of the current patch, the size of the current patch being smaller than a size of a patch considered during definition of the neural network used in the neural network-based image processing tool.

16. A method according to any preceding claim from claim 9 to claim 15, in, The metadata includes at least one syntax element, and the at least one syntax element represents a position of an output patch of the inference process of the neural network-based image processing tool in an output tensor generated by the inference process.

17. A method according to any preceding claim from claim 9 to claim 16, in, The video stream is obtained by applying a video compression process to an original video, wherein the video compression process includes the neural network based image processing tool in a prediction loop of the video compression process or as a post-processing tool.

18. A signal comprising metadata associated with a video stream, the metadata representing margins around patches for use in an inference process of a neural network based image processing tool.

19. A computer program comprising program code instructions for implementing the method according to any preceding claim of claims 1 to 17.

20. A non-transitory information storage medium storing program code instructions for implementing the method according to any preceding claim of claims 1 to 17.

21. An apparatus comprising an electronic circuit, the electronic circuit being configured to: Get (1411) video stream; obtaining (1412) metadata associated with the video stream, the metadata representing a first margin around a patch for use in an inference process of a neural network based image processing tool; and The video stream is decoded using the metadata by applying the neural network-based image processing tool (1413).

22. The device according to claim 21, in, The first margin depends on a receptive field, which depends on a neural network used in the neural network based image processing tool.

23. The device according to claim 22, in, The metadata comprises at least one syntax element representing the receptive field depending on the neural network used in the neural network based image processing tool.

24. The device according to claim 23, in, The at least one syntax element includes: a first syntax element vertically defining the receptive field and a second syntax element horizontally defining the receptive field.

25. Apparatus according to any preceding claim from claim 22 to 24, in, Determining the ability of the inference process to process patches of a larger size than those considered during definition of the neural network used in the neural network-based image processing tool by comparing at least one value representing the receptive field dependent on the neural network to a second margin around the patch considered during definition of the neural network used in the neural network-based image processing tool.

26. The device according to any one of the preceding claims 22 to 24, in, The ability of the inference process to handle patches of larger sizes than considered during definition of the neural network used in the neural network-based image processing tool is specified in the metadata by syntax elements.

27. A device according to any preceding claim 21 to 26, in, The metadata includes at least one syntax element representing at least one offset added to a value representing the receptive field depending on the neural network or to a second margin around a patch considered during definition of the neural network used in the neural network-based image processing tool, the offset being used in response to processing a current patch by the inference process based on a position of the current patch, the current patch having a size smaller than a patch size considered during definition of the neural network used in the neural network-based image processing tool.

28. Apparatus according to any preceding claim from claim 21 to claim 27, in, The metadata includes at least one syntax element, and the at least one syntax element represents a position of an output patch of the inference process of the neural network-based image processing tool in an output tensor generated by the inference process.

29. An apparatus comprising an electronic circuit, the electronic circuit being configured to: obtaining (1402) a video stream; and Information representing a first margin around the patch for use in an inference process of a neural network based image processing tool is signaled (1403) in the form of metadata associated with the video stream.

30. The device according to claim 29, in, The first margin depends on a receptive field, which depends on a neural network used in the neural network based image processing tool.

31. The device according to claim 30, in, The metadata comprises at least one syntax element representing the receptive field depending on the neural network used on the neural network based image processing tool.

32. The device according to claim 31, in, The at least one syntax element includes: a first syntax element vertically defining the receptive field and a second syntax element horizontally defining the receptive field.

33. Apparatus according to any preceding claim 30 to 32, in, Determining the ability of the inference process to process patches of a larger size than those considered during definition of the neural network used in the neural network-based image processing tool by comparing at least one value representing the receptive field dependent on the neural network to a second margin around the patch considered during definition of the neural network used in the neural network-based image processing tool.

34. Apparatus according to any preceding claim 30 to 32, in, The ability of the inference process to handle patches of larger sizes than considered during definition of the neural network used in the neural network-based image processing tool is specified in the metadata by syntax elements.

35. Apparatus according to any preceding claim from claim 29 to claim 34, in, The metadata includes at least one syntax element representing at least one offset added to a value representing the receptive field depending on the neural network or to a second margin around a patch considered during definition of the neural network used in the neural network-based image processing tool, the offset being used in response to processing a current patch by the inference process based on a position of the current patch, the current patch having a size smaller than a patch size considered during definition of the neural network used in the neural network-based image processing tool.

36. Apparatus according to any preceding claim from claim 29 to claim 35, in, The metadata includes at least one syntax element, and the at least one syntax element represents a position of an output patch of the inference process of the neural network-based image processing tool in an output tensor generated by the inference process.

37. Apparatus according to any preceding claim from claim 29 to claim 36, in, The video stream is obtained by applying a video compression process to an original video, wherein the video compression process includes the neural network-based image processing tool in a prediction loop of the video compression process or as a post-processing tool.

Images