Image encoding / decoding method, method of transmitting bitstream, and recording medium storing bitstream

Abstract

Provided are an image encoding / decoding method, a bitstream transmission method, and a computer-readable recording medium for storing a bitstream. An image decoding method according to the disclosure is an image decoding method performed by an image decoding apparatus, and can be an image decoding method including the steps of obtaining size information from a neural network post-filter characteristic (NNPFC) supplemental enhancement information (SEI) message, and determining a size of an output picture based on the size information, wherein the size information includes information on a numerator value of a ratio between the size of the output picture and a size of an input picture and information on a denominator value of the ratio.

Description

Technical Field

[0001] This disclosure relates to an image encoding / decoding method, a method for transmitting a bit stream, and a recording medium for storing the bit stream, and more specifically, to a method for processing information about image size. Background Technology

[0002] Recently, there has been a growing demand for high-resolution and high-quality images, such as high-definition (HD) and ultra-high-definition (UHD) images, across various fields. As the resolution and quality of image data increase, the amount of information or bits transmitted increases relative to existing image data. This increase in the amount of information or bits transmitted leads to increased transmission and storage costs.

[0003] Therefore, efficient image compression techniques are needed to efficiently send, store, and reproduce information about high-resolution and high-quality images. Summary of the Invention

[0004] Technical issues

[0005] One object of this disclosure is to provide an image encoding / decoding method and apparatus with improved encoding / decoding efficiency.

[0006] Furthermore, one object of this disclosure is to provide a method for signaling size information.

[0007] Furthermore, one object of this disclosure is to provide a method for determining image size based on size information from signal notifications.

[0008] Another object of this disclosure is to provide a non-transitory computer-readable recording medium that stores a bitstream generated by an image encoding method according to this disclosure.

[0009] Another object of this disclosure is to provide a non-transitory computer-readable recording medium that stores a bitstream received, decoded and used to reconstruct an image by an image decoding device according to this disclosure.

[0010] Another object of this disclosure is to provide a method for transmitting a bitstream generated by an image encoding method according to this disclosure.

[0011] The technical problems solved by this disclosure are not limited to those described above, and those skilled in the art will clearly understand from the following description other technical problems not described herein.

[0012] Technical solution

[0013] An image decoding method according to one aspect of this disclosure is performed by an image decoding device. The image decoding method may include obtaining size information from Neural Network Post-Filter Feature (NNPFC) Supplemental Enhancement Information (SEI) messages, and determining the size of an output image based on the size information. The size information may include information about the numerator of the ratio between the size of the output image and the size of the input image, and information about the denominator of the ratio.

[0014] The image encoding method according to another aspect of this disclosure is performed by an image encoding device. The image encoding method may include determining a ratio between the size of an output image and the size of an input image, and encoding size information of this ratio in a Neural Network Post-Filter Feature (NNPFC) Supplemental Enhancement Information (SEI) message. The size information may include information about the numerator value of the ratio and information about the denominator value of the ratio.

[0015] According to another aspect of this disclosure, a computer-readable recording medium can store a bitstream generated by the image encoding method or apparatus of this disclosure.

[0016] According to another aspect of this disclosure, a bit stream generated by the image encoding method or device of this disclosure can be transmitted.

[0017] The features briefly outlined above are merely exemplary aspects of the present disclosure as described in the following detailed description and do not limit the scope of the disclosure.

[0018] Beneficial effects

[0019] According to this disclosure, an image encoding / decoding method and apparatus with improved encoding / decoding efficiency can be provided.

[0020] Furthermore, according to this disclosure, size information can be efficiently signaled.

[0021] Furthermore, according to this disclosure, the size of the output image can be adaptively determined based on changes in the size of the input image.

[0022] Additionally, according to this disclosure, a non-transitory computer-readable recording medium may be provided that stores a bitstream generated by an image encoding method according to this disclosure.

[0023] Additionally, according to this disclosure, a non-transitory computer-readable recording medium may be provided, which stores a bitstream that is received, decoded, and used to reconstruct an image by an image decoding device according to this disclosure.

[0024] Additionally, according to this disclosure, a method for transmitting a bitstream generated by an image encoding method according to this disclosure can be provided.

[0025] Those skilled in the art will understand that the effects achievable through this disclosure are not limited to those specifically described herein, and that other advantages of this disclosure will become clearer from the detailed description. Attached Figure Description

[0026] Figure 1 This is a view schematically illustrating a video encoding system to which embodiments of the present disclosure can be applied.

[0027] Figure 2 This is a schematic view illustrating an image encoding device to which embodiments of the present disclosure can be applied.

[0028] Figure 3 This is a schematic view of an image decoding device to which embodiments of the present disclosure can be applied.

[0029] Figure 4 This is a view illustrating the interlacing method used to derive the luminance channel.

[0030] Figure 5 This is a flowchart illustrating an image encoding method according to an embodiment of the present disclosure.

[0031] Figure 6 This is a flowchart illustrating an image decoding method according to an embodiment of the present disclosure.

[0032] Figure 7 This is a flowchart illustrating an image encoding method according to another embodiment of the present disclosure.

[0033] Figure 8 This is a flowchart illustrating an image decoding method according to another embodiment of the present disclosure.

[0034] Figure 9 This is a flowchart illustrating an image encoding method according to another embodiment of the present disclosure.

[0035] Figure 10 This is a flowchart illustrating an image decoding method according to another embodiment of the present disclosure.

[0036] Figure 11 This is a view illustrating a content streaming system to which embodiments of the present disclosure can be applied. Detailed Implementation

[0037] The embodiments of this disclosure will now be described in detail with reference to the accompanying drawings to enable those skilled in the art to readily implement them. However, this disclosure can be implemented in various different forms and is not limited to the embodiments described herein.

[0038] In describing this disclosure, detailed descriptions of relevant known functions or constructions will be omitted if it is determined that such detailed descriptions would unnecessarily obscure the scope of this disclosure. In the accompanying drawings, portions irrelevant to the description of this disclosure have been omitted, and similar portions have been labeled with similar reference numerals.

[0039] In this disclosure, when a component is “connected,” “linked,” or “coupled” to another component, it may include not only a direct connection but also an indirect connection with intermediate components. Furthermore, when a component “includes” or “has” other components, unless otherwise stated, it means that other components may be further included, rather than excluded.

[0040] In this disclosure, the terms first, second, etc., are used only to distinguish one component from other components and do not limit the order or importance of the components unless otherwise stated. Therefore, within the scope of this disclosure, a first component in one embodiment may be referred to as a second component in another embodiment, and similarly, a second component in one embodiment may be referred to as a first component in another embodiment.

[0041] In this disclosure, the distinction between individual components is for the purpose of clearly describing each feature and does not imply that these components must be separate. That is, multiple components may be integrated and implemented in a single hardware or software unit, or a single component may be distributed and implemented in multiple hardware or software units. Therefore, such implementations of component integration or component distribution are included within the scope of this disclosure, even if not otherwise stated.

[0042] In this disclosure, the components described in the various embodiments are not necessarily essential components, and some components may be optional. Therefore, embodiments comprising a subset of the components described in one embodiment are also included within the scope of this disclosure. Furthermore, embodiments that include other components besides those described in the various embodiments are also included within the scope of this disclosure.

[0043] This disclosure relates to the encoding and decoding of images. Unless defined new in this disclosure, the terms used herein may have the general meaning commonly used in the art to which this disclosure pertains.

[0044] In this disclosure, "picture" generally refers to the basis representing an image within a specific time period, and a slice / tile is the coding basis that constitutes a part of the picture. A picture may consist of one or more slices / tiles. Furthermore, a slice / tile may include one or more coding tree units (CTUs).

[0045] In this disclosure, "pixel" or "cell" can refer to the smallest unit that constitutes a picture (or image). Furthermore, "sample" can be used as the term corresponding to a pixel. A sample can generally represent a pixel or a pixel value, and can represent only the pixel / pixel value of the luminance component or only the pixel / pixel value of the chrominance component.

[0046] In this disclosure, "unit" can refer to a basic unit of image processing. A unit may include a specific region of an image and at least one of the following information associated with that region. A unit may include a luminance block and two chrominance (e.g., Cb, Cr) blocks. In some cases, the term "unit" may be used interchangeably with terms such as "sample array," "block," or "region." Generally, an M×N block may include a set (or array) of samples (or transform coefficients) with M columns and N rows.

[0047] In this disclosure, "current block" can refer to one of "current coding block," "current coding unit," "coding target block," "decoding target block," or "processing target block." When performing prediction, "current block" can refer to "current prediction block" or "prediction target block." When performing transform (inverse transform) / quantization (dequantization), "current block" can refer to "current transform block" or "transform target block." When performing filtering, "current block" can refer to "filter target block."

[0048] Furthermore, in this disclosure, unless explicitly stated as a chroma block, "current block" may refer to a block that includes both luma component blocks and chroma component blocks, or "the luma block of the current block." The luma component block of the current block may be explicitly represented by terms such as "luma block" or "current luma block," which clearly indicates that it is a luma component block. Additionally, the chroma component block of the current block may be explicitly represented by terms such as "chroma block" or "current chroma block," which clearly indicates that it is a chroma component block.

[0049] In this disclosure, the terms “ / ” and “,” should be interpreted as meaning “and / or”. For example, the expressions “A / B” and “A, B” can mean “A and / or B”. Furthermore, “A / B / C” and “A, B, C” can mean “at least one of the following: A, B and / or C”.

[0050] In this disclosure, the term "or" should be interpreted as meaning "and / or". For example, the expression "A or B" can include 1) only "A", 2) only "B" and / or 3) "both A and B". In other words, in this disclosure, the term "or" should be interpreted as meaning "additionally or alternatively".

[0051] Overview of Video Encoding Systems

[0052] Figure 1This is a view illustrating a video encoding system to which embodiments of the present disclosure can be applied.

[0053] The video encoding system according to the embodiment may include an encoding device 10 and a decoding device 20. The encoding device 10 may deliver encoded video and / or image information or data to the decoding device 20 in the form of a file or stream via a digital storage medium or network.

[0054] The encoding device 10 according to an embodiment may include a video source generator 11, an encoding unit (encoder) 12, and a transmitter 13. The decoding device 20 according to an embodiment may include a receiver 21, a decoding unit (decoder) 22, and a renderer 23. The encoding unit 12 may be referred to as a video / image encoding device, and the decoding unit 22 may be referred to as a video / image decoding device. The transmitter 13 may be included in the encoding unit 12. The receiver 21 may be included in the decoding unit 22. The renderer 23 may include a display, and the display may be configured as a separate device or an external component.

[0055] The video source generator 11 can obtain video / images through processes of capturing, compositing, or generating video / images. The video source generator 11 may include a video / image capture device and / or a video / image generation device. The video / image capture device may include, for example, one or more cameras, a video / image archive containing previously captured video / images, etc. The video / image generation device may include, for example, a computer, tablet computer, and smartphone, and can generate video / images (electronically). For example, virtual video / images can be generated by a computer, etc. In this case, the video / image capture process can be replaced by a process that generates related data.

[0056] The encoding unit 12 can encode the input video / image. The encoding unit 12 can perform a series of processes such as prediction, transformation, and quantization to achieve compression and encoding efficiency. The encoding unit 12 can output encoded data (encoded video / image information) in the form of a bitstream.

[0057] The transmitter 13 can acquire encoded video / image information or data output in bitstream form and forward it to the receiver 21 of the decoding device 20 or another external object via a digital storage medium or network in the form of a file or stream. The digital storage medium can include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, SSD, etc. The transmitter 13 can include elements for generating media files in a predetermined file format and can include elements for transmission via a broadcast / communication network. The transmitter 13 can be provided as a transmission device separate from the encoding device 12. In this case, the transmission device can include at least one processor and a transmission unit that acquires encoded video / image information or data output in bitstream form and that transmits the information or data in the form of a file or stream. The receiver 21 can extract / receive the bitstream from the storage medium or network and send the bitstream to the decoding unit 22.

[0058] The decoding unit 22 can decode the video / image by performing a series of processes (such as dequantization, inverse transform, and prediction) corresponding to the operations of the encoding unit 12.

[0059] Renderer 23 can render the decoded video / images. The rendered video / images can be displayed on a monitor.

[0060] Overview of Image Encoding Devices

[0061] Figure 2 This is a schematic view illustrating an image encoding device to which embodiments of the present disclosure can be applied.

[0062] like Figure 2 As shown, the image encoding device 100 may include an image segmenter 110, a subtractor 115, a transformer 120, a quantizer 130, a dequantizer 140, an inverse transformer 150, an adder 155, a filter 160, a memory 170, an inter-frame prediction unit 180, an intra-frame prediction unit 185, and an entropy encoder 190. The inter-frame prediction unit 180 and the intra-frame prediction unit 185 may be collectively referred to as "prediction units". The transformer 120, quantizer 130, dequantizer 140, and inverse transformer 150 may be included in a residual processor. The residual processor may further include a subtractor 115.

[0063] In some implementations, all or at least some of the components constituting the image encoding device 100 may be configured by a single hardware component (e.g., an encoder or a processor). Furthermore, the memory 170 may include a decoded image buffer (DPB) and may be configured by a digital storage medium.

[0064] Image segmenter 110 can segment an input image (or picture or frame) input to image encoding device 100 into one or more processing units. For example, a processing unit may be referred to as a coding unit (CU). A coding unit can be obtained by recursively segmenting a coding tree unit (CTU) or a maximum coding unit (LCU) according to a quadtree-binary-truncate (QT / BT / TT) structure. For example, a coding unit can be segmented into multiple deeper coding units based on a quadtree structure, a binary tree structure, and / or a truncate structure. For the segmentation of coding units, a quadtree structure can be applied first, followed by a binary tree structure and / or a truncate structure. The encoding process of this disclosure can be performed based on a final coding unit that is no longer segmented. A maximum coding unit can be used as the final coding unit, or a deeper coding unit obtained by segmenting a maximum coding unit can be used as the final coding unit. Here, the encoding process may include prediction, transformation, and reconstruction processes, which will be described later. As another example, the processing unit of the encoding process may be a prediction unit (PU) or a transformation unit (TU). Prediction units and transform units can be derived from or segmented from the final coding unit. Prediction units can be units for predicting samples, and transform units can be units for deriving transform coefficients and / or units for deriving residual signals from transform coefficients.

[0065] The prediction unit (inter-frame prediction unit 180 or intra-frame prediction unit 185) can perform prediction on the block to be processed (the current block) and generate a prediction block containing prediction samples for the current block. The prediction unit can determine whether to apply intra-frame prediction or inter-frame prediction based on the current block or coding unit. The prediction unit can generate various information related to the prediction of the current block and send the generated information to the entropy encoder 190. The information about the prediction can be encoded in the entropy encoder 190 and output as a bitstream.

[0066] Intra-prediction unit (intra-predictor) 185 can predict the current block by referencing samples in the current image. The referenced samples can be located in adjacent regions of the current block, or at a more distant location depending on the intra-prediction mode and / or intra-prediction technique. Intra-prediction modes can include multiple non-directional modes and multiple directional modes. Non-directional modes can include, for example, DC mode and planar mode. Directional modes can include, for example, 33 or 65 directional prediction modes, depending on the level of detail in the prediction direction. However, this is merely an example; more or fewer directional prediction modes can be used depending on the configuration. Intra-prediction unit 185 can determine the prediction mode to be applied to the current block by using prediction modes applied to adjacent blocks.

[0067] The inter-frame prediction unit (inter-frame predictor) 180 can deduce the prediction block of the current block based on a reference block (reference sample array) specified by motion vectors on a reference image. In this case, to reduce the amount of motion information transmitted in inter-frame prediction mode, motion information can be predicted on a block, sub-block, or sample basis based on the correlation of motion information between neighboring blocks and the current block. Motion information may include motion vectors and reference image indices. Motion information may also include inter-frame prediction direction (L0 prediction, L1 prediction, Bi prediction, etc.) information. In the case of inter-frame prediction, neighboring blocks may include spatially neighboring blocks existing in the current image and temporally neighboring blocks existing in the reference image. The reference image containing the reference block and the reference image containing the temporally neighboring block may be the same or different. The temporally neighboring block may be referred to as a co-location reference block, a co-location coding unit (colCU), etc. The reference image containing the temporally neighboring block may be referred to as a co-location image (colPic). For example, the inter-frame prediction unit 180 can construct a motion information candidate list based on neighboring blocks and generate information indicating which candidate to use to deduce the motion vector and / or reference image index of the current block. Inter-frame prediction can be performed based on various prediction modes. For example, in skip mode and merge mode, the inter-frame prediction unit 180 can use motion information from neighboring blocks as motion information for the current block. In skip mode, unlike merge mode, residual signals may not be sent. In motion vector prediction (MVP) mode, motion vectors from neighboring blocks can be used as motion vector predictors, and the motion vector of the current block can be signaled by encoding the motion vector difference and an indicator for the motion vector predictor. The motion vector difference can refer to the difference between the motion vector of the current block and the motion vector predictor.

[0068] The prediction unit can generate a prediction signal based on various prediction methods and techniques described below. For example, the prediction unit can apply intra-frame prediction or inter-frame prediction not only for the prediction of the current block, but also both intra-frame prediction and inter-frame prediction simultaneously. A prediction method that applies both intra-frame prediction and inter-frame prediction simultaneously for the prediction of the current block can be called Inter-Frame Intra-Frame Combined Prediction (CIIP). Furthermore, the prediction unit can perform Intra-Block Copying (IBC) for the prediction of the current block. Intra-Block Copying can be used for content image / video coding, such as in games, i.e., Screen Content Coding (SCC). IBC is a method for predicting the current image using a previously reconstructed reference block located at a predetermined distance from the current block. When IBC is applied, the position of the reference block in the current image can be encoded as a vector (block vector) corresponding to the predetermined distance. IBC essentially performs prediction within the current image, but because the reference block is derived within the current image, its execution can be similar to inter-frame prediction. That is, IBC can use at least one of the inter-frame prediction techniques described in this disclosure.

[0069] The prediction signal generated by the prediction unit can be used to generate a reconstructed signal or a residual signal. The subtractor 115 can generate a residual signal (residual block or residual sample array) by subtracting the prediction signal (prediction block or prediction sample array) output from the prediction unit from the input image signal (original block or original sample array). The generated residual signal can be sent to the converter 120.

[0070] Transformer 120 can generate transform coefficients by applying transform techniques to the residual signal. For example, the transform techniques may include at least one of Discrete Cosine Transform (DCT), Discrete Sine Transform (DST), Karhunen-Loeve Transform (KLT), Graph-Based Transform (GBT), or Conditional Nonlinear Transform (CNT). Here, GBT refers to the transform obtained from a graph when the relationship information between pixels is represented by a graph. CNT refers to the transform obtained based on the prediction signal generated using all previously reconstructed pixels. Furthermore, the transform process can be applied to square pixel blocks of the same size, or it can be applied to non-square, variable-sized blocks.

[0071] Quantizer 130 quantizes the transform coefficients and sends them to entropy encoder 190. Entropy encoder 190 encodes the quantized signal (information about the quantized transform coefficients) and outputs a bitstream. This information about the quantized transform coefficients can be referred to as residual information. Quantizer 130 can rearrange the block-type quantized transform coefficients into a one-dimensional vector form based on the coefficient scan order and generate information about the quantized transform coefficients based on this one-dimensional vector form.

[0072] The entropy encoder 190 can perform various encoding methods, such as exponential Golomb coding, context-adaptive variable-length coding (CAVLC), and context-adaptive binary arithmetic coding (CABAC). In addition to quantization transform coefficients, the entropy encoder 190 can also encode information required for video / image reconstruction (e.g., values ​​of syntax elements, etc.) together or separately. The encoded information (e.g., encoded video / image information) can be transmitted or stored as a bitstream in a Network Abstraction Layer (NAL) unit. The video / image information can also include information about various parameter sets, such as adaptive parameter sets (APS), picture parameter sets (PPS), sequence parameter sets (SPS), or video parameter sets (VPS). Furthermore, the video / image information can also include general constraint information. The signaling notification information, transmitted information, and / or syntax elements described in this disclosure can be encoded through the above encoding process and included in the bitstream.

[0073] The bitstream can be transmitted over a network or stored in a digital storage medium. The network may include broadcast networks and / or communication networks, and the digital storage medium may include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, SSD, etc. A transmitter (not shown) for transmitting the signal output from the entropy encoder 190 and / or a storage unit (not shown) for storing the signal may be included as internal / external components of the image encoding device 100. Alternatively, the transmitter may be provided as a component of the entropy encoder 190.

[0074] The quantization transform coefficients output from quantizer 130 can be used to generate residual signals. For example, the residual signal (residual block or residual sample) can be reconstructed by applying dequantization and inverse transform to the quantization transform coefficients through dequantizer 140 and inverse transformer 150.

[0075] Adder 155 adds the reconstructed residual signal to the prediction signal output from inter-frame prediction unit 180 or intra-frame prediction unit 185 to generate a reconstructed signal (reconstructed image, reconstructed block, reconstructed sample array). If there is no residual for the block to be processed (such as in the case of applying a skip mode), the predicted block can be used as a reconstructed block. Adder 155 may be referred to as a reconstructor or reconstructed block generator. The generated reconstructed signal can be used for intra-frame prediction of the next block to be processed in the current image, and can be used for inter-frame prediction of the next image by filtering as described below.

[0076] Filter 160 can improve subjective / objective image quality by applying filtering to the reconstructed signal. For example, filter 160 can generate a modified reconstructed image by applying various filtering methods to the reconstructed image and store the modified reconstructed image in memory 170, specifically in the DPB of memory 170. Various filtering methods can include, for example, deblocking filtering, sample adaptive offsetting, adaptive loop filtering, bilateral filtering, etc. As described later in the description of each filtering method, filter 160 can generate various filtering-related information and send the generated information to entropy encoder 190. The filtering-related information can be encoded by entropy encoder 190 and output as a bitstream.

[0077] The modified reconstructed image sent to memory 170 can be used as a reference image in inter-frame prediction unit 180. When inter-frame prediction is applied by image coding device 100, prediction mismatch between image coding device 100 and image decoding device can be avoided, and coding efficiency can be improved.

[0078] The DPB of memory 170 can store modified reconstructed images for use as reference images in inter-frame prediction unit 180. Memory 170 can store motion information of blocks in the current frame whose motion information has been derived (or encoded) and / or motion information of blocks in the reconstructed image. The stored motion information can be sent to inter-frame prediction unit 180 and used as motion information for spatially adjacent blocks or temporally adjacent blocks. Memory 170 can store reconstructed samples of reconstructed blocks in the current image and can transmit the reconstructed samples to intra-frame prediction unit 185.

[0079] Overview of image decoding devices

[0080] Figure 3 This is a schematic view of an image decoding device to which embodiments of the present disclosure can be applied.

[0081] like Figure 3 As shown, the image decoding device 200 may include an entropy decoder 210, a dequantizer 220, an inverse transformer 230, an adder 235, a filter 240, a memory 250, an inter-frame predictor 260, and an intra-frame predictor 265. The inter-frame predictor 260 and the intra-frame predictor 265 may be collectively referred to as "prediction units (predictors)". The dequantizer 220 and the inverse transformer 230 may be included in a residual processor.

[0082] According to an embodiment, all or at least some of the components constituting the image decoding device 200 may be configured by hardware components (e.g., a decoder or a processor). Furthermore, the memory 170 may include a decoded image buffer (DPB) or may be configured by a digital storage medium.

[0083] The image decoding device 200, having received a bitstream containing video / image information, can perform operations with... Figure 2 The image is reconstructed using the processing performed by the image encoding device 100, corresponding to the processing used in the image encoding device. For example, the image decoding device 200 can perform decoding using a processing unit applied in the image encoding device. Therefore, the processing unit used for decoding can be, for example, an encoding unit. The encoding unit can be obtained by segmenting a coding tree unit or a maximum coding unit. The reconstructed image signal decoded and output by the image decoding device 200 can be reproduced by a reproduction device (not shown).

[0084] Image decoding device 200 can receive from Figure 2The image encoding device outputs a signal in the form of a bitstream. The received signal can be decoded by an entropy decoder 210. For example, the entropy decoder 210 can parse the bitstream to derive information (e.g., video / image information) required for image reconstruction (or image reconstruction). The video / image information may also include information about various parameter sets such as adaptation parameter sets (APS), image parameter sets (PPS), sequence parameter sets (SPS), or video parameter sets (VPS). In addition, the video / image information may also include general constraint information. The image decoding device can further decode the image based on the information about the parameter sets and / or the general constraint information. The signaling information / received information and / or syntax elements described in this disclosure can be decoded and obtained from the bitstream through a decoding process. For example, the entropy decoder 210 decodes the information in the bitstream based on encoding methods such as exponential Golomb coding, CAVLC, or CABAC, and outputs the values ​​of the syntax elements required for image reconstruction and the quantized values ​​of the transform coefficients for the residuals. More specifically, the CABAC entropy decoding method can receive bins corresponding to each syntax element in the bitstream, use information about the target syntax element, decoding information of adjacent blocks, and information about the target block or symbols / bins decoded in the previous stage to determine a context model, predict the occurrence probability of bins based on the determined context model, perform arithmetic decoding on the bins, and generate symbols corresponding to the value of each syntax element. In this case, after determining the context model, the CABAC entropy decoding method can use the information of the decoded symbols / bins to update the context model of the next symbol / bin. Prediction-related information from the information decoded by the entropy decoder 210 can be provided to the prediction units (inter-frame predictor 260 and intra-frame prediction unit 265), and the residuals of the entropy decoding performed in the entropy decoder 210 (i.e., quantization transform coefficients and related parameter information) can be input to the dequantizer 220. Furthermore, filtering-related information from the information decoded by the entropy decoder 210 can be provided to the filter 240. Meanwhile, the receiver (not shown) for receiving signals output from the image encoding device can be further configured as an internal / external element of the image decoding device 200, or the receiver can be a component of the entropy decoder 210.

[0085] Furthermore, the image decoding device according to this disclosure can be referred to as a video / image / image decoding device. The image decoding device can be divided into an information decoder (video / image / image information decoder) and a sample decoder (video / image / image sample decoder). The information decoder may include an entropy decoder 210. The sample decoder may include at least one of a dequantizer 220, an inverse transformer 230, an adder 235, a filter 240, a memory 250, an inter-frame prediction unit 160, or an intra-frame prediction unit 265.

[0086] Dequantizer 220 can dequantize the quantized transform coefficients and output transform coefficients. Dequantizer 220 can rearrange the quantized transform coefficients into two-dimensional blocks. In this case, the rearrangement can be performed based on the coefficient scan order performed in the image encoding device. Dequantizer 220 can obtain the transform coefficients by performing dequantization on the quantized transform coefficients using quantization parameters (e.g., quantization step size information).

[0087] The inverse transformer 230 can perform an inverse transformation on the transform coefficients to obtain the residual signal (residual block, residual sample array).

[0088] The prediction unit can perform prediction on the current block and generate a prediction block containing prediction samples for the current block. The prediction unit can determine whether to apply intra-frame prediction or inter-frame prediction to the current block based on the prediction information output from the entropy decoder 210, and can determine a specific intra-frame / inter-frame prediction mode (prediction technique).

[0089] The prediction unit can generate prediction signals based on various prediction methods (techniques) described below, which are the same as those described in the prediction unit of the image coding device 100.

[0090] Intra-predictor 265 can predict the current block by referring to samples in the current image. The description of intra-predictor unit 185 also applies to intra-predictor unit 265.

[0091] Inter-frame predictor 260 can deduce the predicted block of the current block based on reference blocks (reference sample arrays) specified by motion vectors on a reference image. In this case, to reduce the amount of motion information transmitted in inter-frame prediction mode, motion information can be predicted on a block, sub-block, or sample basis based on the correlation of motion information between neighboring blocks and the current block. Motion information may include motion vectors and reference image indices. Motion information may also include inter-frame prediction direction (L0 prediction, L1 prediction, Bi prediction, etc.) information. In the case of inter-frame prediction, neighboring blocks may include spatially adjacent blocks existing in the current image and temporally adjacent blocks existing in the reference image. For example, inter-frame predictor 260 can configure a motion information candidate list based on neighboring blocks and deduce the motion vector and / or reference image index of the current block based on the received candidate selection information. Inter-frame prediction can be performed based on various prediction modes, and the information about the prediction may include information indicating the inter-frame prediction mode of the current block.

[0092] Adder 235 generates a reconstruction signal (reconstructed image, reconstruction block, reconstruction sample array) by adding the obtained residual signal to the prediction signal (prediction block, prediction sample array) output from the prediction unit (including inter-frame predictor 260 and / or intra-frame prediction unit 265). If there is no residual for the block to be processed (such as in the case of applying a skip mode), the prediction block can be used as a reconstruction block. The description of adder 155 also applies to adder 235. Adder 235 may be referred to as a reconstructor or reconstruction block generator. The generated reconstruction signal can be used for intra-frame prediction of the next block to be processed in the current image, and can be used for inter-frame prediction of the next image by filtering as described below.

[0093] Filter 240 can improve subjective / objective image quality by applying filtering to the reconstructed signal. For example, filter 240 can generate a modified reconstructed image by applying various filtering methods to the reconstructed image and store the modified reconstructed image in memory 250, specifically in the DPB of memory 250. Various filtering methods may include, for example, deblocking filtering, adaptive sample shifting, adaptive loop filtering, bilateral filtering, etc.

[0094] The (modified) reconstructed image stored in the DPB of memory 250 can be used as a reference image in inter-frame predictor 260. Memory 250 can store motion information of blocks in the current image whose motion information has been derived (or decoded) and / or motion information of blocks in the reconstructed image. The stored motion information can be sent to inter-frame predictor 260 to be used as motion information of spatially adjacent blocks or temporally adjacent blocks. Memory 250 can store reconstructed samples of reconstructed blocks in the current image and transmit the reconstructed samples to intra-frame prediction unit 265.

[0095] The embodiments described in this disclosure in the filter 160, inter-frame prediction unit 180 and intra-frame prediction unit 185 of the image encoding device 100 can be applied equally or correspondingly to the filter 240, inter-frame predictor 260 and intra-frame predictor 265 of the image decoding device 200.

[0096] Post-filter characteristics of neural networks (NNPFC)

[0097] Tables 1 to 3 represent the NNPFC syntax structure.

[0098] [Table 1]

[0099]

[0100] [Table 2]

[0101]

[0102] [Table 3]

[0103]

[0104] The NNPFC syntax structures in Tables 1 to 3 can be signaled in the form of Supplemental Enhancement Information (SEI) messages. These SEI messages signaling the NNPFC syntax structures in Tables 1 to 3 can be referred to as NNPFC SEI messages.

[0105] The Neural Network Post-Filter Property (NNPFC) SEI message specifies the neural network that can be used as a post-processing filter. The Neural Network Post-Filter Activation (NNPFA) SEI message indicates that a specified Neural Network Post-Processing Filter (NNPF) should be used on a particular image. Here, "post-processing filter" and "post-filter" can have the same meaning.

[0106] To use this SEI message, the following variables need to be defined:

[0107] - Input the image width and height (in brightness samples), which are denoted as CroppedWidth and CroppedHeight in this paper, respectively.

[0108] - The luminance sample array CroppedYPic[idx] and chrominance sample array CroppedCbPic[idx] and CroppedCrPic[idx] (if present) of the input image are used as inputs to NNPF, with the index idx ranging from 0 to numInputPics-1 (inclusive).

[0109] -BitDepthY can represent the bit depth of the brightness sample array of the input image.

[0110] -BitDepthC can represent the bit depth of the chroma sample array of the input image (if any).

[0111] -ChromaFormatIdc can represent a chroma format identifier.

[0112] - When the value of nnpfc_auxiliary_inp_idc is equal to 1, the filter strength control value StrengthControlVal should be a real number in the range of 0 to 1 (inclusive).

[0113] The input image at index 0 can correspond to the image whose NNPF is activated by the NNPFA SEI message defined by this NNPFC SEI message. Input images with indices i in the range of 1 to numInputPics-1 (inclusive) can be output before the input image at index i-1.

[0114] If nnpfc_purpose&0x08 is not equal to 0, and the input image at index 0 is associated with a frame packing arrangement SEI message with fp_arrangement_type equal to 5, then all input images can be associated with a frame packing arrangement SEI message with fp_arrangement_type equal to 5 and have the same value as fp_current_frame_is_frame0_flag.

[0115] More than one NNPFC SEI message can exist for the same image. If more than one NNPFC SEI message with different nnpfc_id values ​​exists or is active for the same image, then these more than one NNPFC SEI message can have the same or different nnpfc_purpose and nnpfc_mode_idc values.

[0116] `nnpfc_purpose` indicates the purpose of NNPF, as specified in Table 4. In the bitstream, the value of `nnpfc_purpose` should be in the range of 0 to 63 (inclusive). Values ​​of `nnpfc_purpose` from 64 to 65535 (inclusive) are reserved for future use. The decoder should ignore NNPFCSEI messages with `nnpfc_purpose` in the range of 64 to 65535 (inclusive). If the value of `nnpfc_purpose` is reserved for future use, the syntax elements of the SEI message can be expanded to the existing syntax elements, provided that `nnpfc_purpose` equals the corresponding value. If `ChromaFormatIdc` equals 3, then `nnpfc_purpose&0x02` should equal 0. If `ChromaFormatIdc` or `nnpfc_purpose&0x02` is not equal to 0, then `nnpfc_purpose&0x20` should equal 0.

[0117] [Table 4]

[0118]

[0119] `nnpfc_id` can contain an identifier that can be used to identify NNPF. In the bitstream, the value of `nnpfc_id` should be between 0 and 2. 32-2 (including 0 and 2) 32 The range is -2). nnpfc_id from 256 to 511 (inclusive) and 2 31 Up to 2 32 -2 (including 2) 31 and 2 32 The value of -2) can be reserved for future use. The decoder encounters nnpfc_id in the range of 256 to 511 (inclusive) or in the range of 2. 31 Up to 2 32 -2 (including 2) 31 and 2 32 When an NNPFC SEI message falls within the range of -2), the SEI message should be ignored.

[0120] When the -NNPFC SEI message is the first NNPFC SEI message in the current CLVS with a specific nnpfc_id value in decoding order, the following rules apply:

[0121] - This SEI message specifies the baseline NNPF.

[0122] - This SEI message pertains to the current decoded image of the current layer and all subsequent decoded images in output order, up to the end of the current CLVS.

[0123] The NNPFC SEI message may be a repetition of previous NNPFC SEI messages in the current CLVS in decoding order, and subsequent semantics may apply under the assumption that the SEI message is the only NNPFC SEI message with the same content in the current CLVS.

[0124] A value of 0 for nnpfc_mode_idc can indicate that the SEI message can contain a bitstream representing the baseline NNPF, or it can represent an update relative to the baseline NNPF with the same nnpfc_id value.

[0125] When the NNPFC SEI message is the first NNPFC SEI message in the current CLVS with a specific nnpfc_id value in decoding order, nnpfc_mode_idc equal to 1 can specify that the baseline NNPF associated with the nnpfc_id value is a neural network, and the neural network can be a neural network identified by a URI represented by nnpfc_uri in the format identified by nnpfc_tag_uri.

[0126] If the NNPFC SEI message is neither the first NNPFC SEI message in the current CLVS with a specific nnpfc_id value in decoding order, nor a duplicate of the first NNPFC SEI message, then nnpfc_mode_idc equal to 1 can specify that the update relative to the baseline NNPF with the same nnpfc_id value uses the URI definition represented by nnpfc_uri in the format identified by nnpfc_tag_uri.

[0127] In the bitstream, the value of nnpfc_mode_idc should be in the range of 0 to 1 (inclusive). Values ​​of nnpfc_mode_idc from 2 to 255 (inclusive) can be reserved for future use and may not exist in the bitstream. The decoder should ignore NNPFC SEI messages with nnpfc_mode_idc in the range of 2 to 255 (inclusive). nnpfc_mode_idc values ​​greater than 255 should not exist in the bitstream and should not be reserved for future use.

[0128] If the SEI message is the first NNPFC SEI message in the current CLVS with a specific nnpfc_id value in decoding order, then the NNPF postprocessing filter (PostProcessingFilter()) can be assigned the same as the baseline NNPF.

[0129] If the SEI message is not the first NNPFC SEI message with a specific nnpfc_id value in the current CLVS in decoding order, and is not a duplicate of the first NNPFC SEI message, then the NNPF postprocessing filter (PostProcessingFilter()) can be obtained by applying the update defined by the SEI message to the baseline NNPF.

[0130] Updates are not cumulative; instead, each update is applied to a baseline NNPF, which is the NNPF specified by the first NNPFC SEI message with a specific nnpfc_id value in the current CLVS in decoding order.

[0131] In the bitstream, nnpfc_reserved_zero_bit_a should be equal to 0. The decoder should ignore NNPFC SEI messages where nnpfc_reserved_zero_bit_a is not equal to 0.

[0132] The `nnpfc_tag_uri` can contain a tag URI with syntax and semantics as specified in IETF RFC 4151, which identifies a neural network used as a baseline NNPF or an update relative to a baseline NNPF with the same `nnpfc_id` value specified by the `nnpfc_uri`. The `nnpfc_tag_uri` uniquely identifies the format of the neural network data specified by the `nnpfc_uri` without requiring a central registry. An `nnpfc_tag_uri` equal to "tag:iso.org,2023:15938-17" indicates that the neural network data identified by the `nnpfc_uri` conforms to ISO / IEC 15938-17.

[0133] nnpfc_uri can contain a URI with syntax and semantics as specified in IETF Internet Standard 66, which identifies a neural network used as a baseline NNPF or an update relative to a baseline NNPF with the same nnpfc_id value.

[0134] A value of 1 for `nnpfc_property_present_flag` indicates the presence of syntax elements related to filter purpose, input format, output format, and complexity. A value of 0 for `nnpfc_property_present_flag` indicates the absence of syntax elements related to filter purpose, input format, output format, and complexity. If the SEI message is the first NNPFC SEI message in the current CLVS with a specific `nnpfc_id` value in decoding order, then the value of `nnpfc_property_present_flag` should be 1. If the value of `nnpfc_property_present_flag` is 0, then all syntax elements that exist only when the value of `nnpfc_property_present_flag` is 1 and for which no inferred value is specified can be inferred to be equal to the corresponding syntax element in the NNPFC SEI message containing the updated baseline NNPF provided by this SEI message.

[0135] A value of 1 for nnpfc_base_flag indicates that the SEI message represents a baseline NNPF. A value of 0 for nnpfc_base_flag indicates that the SEI message represents an update associated with the baseline NNPF. If nnpfc_base_flag does not exist, its value can be inferred to be 0.

[0136] The value of nnpfc_base_flag is subject to the following constraints:

[0137] - When the NNPFC SEI message is the first NNPFC SEI message in the current CLVS with a specific nnpfc_id value in decoding order, the value of nnpfc_base_flag should be equal to 1.

[0138] - When NNPFC SEI message nnpfcB is not the first NNPFC SEI message with a specific nnpfc_id value in the current CLVS in decoding order, and the value of nnpfc_base_flag is equal to 1, the NNPFC SEI message should be a duplicate of the first NNPFC SEI message nnpfcA with the same nnpfc_id value in decoding order, that is, the payload content of nnpfcB should be the same as the payload content of nnpfcA.

[0139] If the NNPFC SEI message is not the first NNPFC SEI message with a specific nnpfc_id value in the current CLVS in decoding order, and is not a duplicate of the first NNPFC SEI message with that specific nnpfc_id value, then the following rules apply:

[0140] -SEI messages can define updates to baseline NNPFs relative to those that precede them in decoding order and have the same nnpfc_id value.

[0141] - SEI messages are associated only with the current reconstructed image of the current layer and all subsequent reconstructed images in output order (until the end of the current CLVS or the next reconstructed image within the current CLVS), and with subsequent NNPFC SEI messages within the current CLVS that have the earlier of a specific nnpfc_id value in decoding order.

[0142] When the NNPFC SEI message nnpfcCurr is not the first NNPFC SEI message with a specific nnpfc_id value in the current CLVS in decoding order, is not a duplicate of the first NNPFC SEI message with that specific nnpfc_id value (i.e., the value of nnpfc_base_flag is equal to 0), and the value of nnpfc_property_present_flag is equal to 1, the following constraints apply:

[0143] - The nnpfc_purpose value in this NNPFC SEI message should be the same as the nnpfc_purpose value in the first NNPFC SEI message in the current CLVS that has that specific nnpfc_id value in the decoding order.

[0144] - The value of the syntax element in the NNPFC SEI message that is located after nnpfc_property_present_flag and before nnpfc_complexity_info_present_flag in the decoding order should be the same as the value of the corresponding syntax element in the first NNPFC SEI message in the current CLVS that has that specific nnpfc_id value in the decoding order.

[0145] -nnpfc_complexity_info_present_flag equals 0; or nnpfc_complexity_info_present_flag equals 1 in the first NNPFC SEI message with that particular nnpfc_id value in the current CLVS in decoding order, and all of the following apply:

[0146] (1) The nnpfc_parameter_type_idc in nnpfcCurr should be equal to the nnpfc_parameter_type_idc in nnpfcBase.

[0147] (2) When nnpfc_log2_parameter_bit_length_minus3 exists in nnpfcCurr, its value should be less than or equal to nnpfc_log2_parameter_bit_length_minus3 in nnpfcBase.

[0148] (3) If nnpfc_num_parameters_idc in nnpfcBase is equal to 0, then nnpfc_num_parameters_idc in nnpfcCurr should be equal to 0.

[0149] (4) Otherwise (nnpfc_num_parameters_idc in nnpfcBase is greater than 0), nnpfc_num_parameters_idc in nnpfcCurr should be greater than 0 and less than or equal to nnpfc_num_parameters_idc in nnpfcBase.

[0150] (5) If nnpfc_num_kmac_operations_idc in nnpfcBase is equal to 0, then nnpfc_num_kmac_operations_idc in nnpfcCurr should be equal to 0.

[0151] (6) Otherwise (nnpfc_num_kmac_operations_idc in nnpfcBase is greater than 0), nnpfc_num_kmac_operations_idc in nnpfcCurr should be greater than 0 and less than or equal to nnpfc_num_kmac_operations_idc in nnpfcBase.

[0152] (7) If nnpfc_total_kilobyte_size in nnpfcBase is equal to 0, then nnpfc_total_kilobyte_size in nnpfcCurr should be equal to 0.

[0153] (8) Otherwise (nnpfc_total_kilobyte_size in nnpfcBase is greater than 0), nnpfc_total_kilobyte_size in nnpfcCurr should be greater than 0 and less than or equal to nnpfc_total_kilobyte_size in nnpfcBase.

[0154] When `nnpfc_purpose&0x02` is not equal to 0, `nnpfc_out_sub_c_flag` can specify the values ​​of variables `outSubWidthC` and `outSubHeightC`. `nnpfc_out_sub_c_flag` equal to 1 specifies that both `outSubWidthC` and `outSubHeightC` are equal to 1. `nnpfc_out_sub_c_flag` equal to 0 specifies that both `outSubWidthC` and `outSubHeightC` are equal to 1. When `ChromaFormatIdc` equals 2 and `nnpfc_out_sub_c_flag` exists, its value should be equal to 1.

[0155] When `nnpfc_purpose&0x20` is not equal to 0, `nnpfc_out_colour_format_idc` can specify the color format of the NNPFC output, and thus specify the values ​​of the variables `outSubWidthC` and `outSubHeightC`. `nnpfc_out_colour_format_idc` equal to 1 specifies the NNPFC output color format as 4:2:0, and both `outSubWidthC` and `outSubHeightC` are equal to 2. `nnpfc_out_colour_format_idc` equal to 2 specifies the NNPFC output color format as 4:2:2, and `outSubWidthC` equals 2 and `outSubHeightC` equals 1. `nnpfc_out_colour_format_idc` equal to 3 specifies the NNPFC output color format as 4:2:4, and both `outSubWidthC` and `outSubHeightC` are equal to 1. The value of `nnpfc_out_colour_format_idc` should not be equal to 0.

[0156] When both nnpfc_purpose&0x02 and nnpfc_purpose&0x20 are equal to 0, outSubWidthC and outSubHeightC can be inferred to be equal to SubWidthC and SubHeightC, respectively.

[0157] `nnpfc_pic_width_in_luma_samples` and `nnpfc_pic_height_in_luma_samples` can represent the width and height of the luminance sample array of the image obtained after applying the NNPF identified by `nnpfc_id` to the cropped decoded output image, respectively. If `nnpfc_pic_width_in_luma_samples` and `nnpfc_pic_height_in_luma_samples` do not exist, they can be inferred to be equal to `CroppedWidth` and `CroppedHeight`, respectively. The value of `nnpfc_pic_width_in_luma_samples` should be in the range of `CroppedWidth` to `CroppedWidth×16-1` (inclusive). The value of nnpfc_pic_height_in_luma_samples should be in the range of CroppedHeight to CroppedHeight×16-1 (inclusive).

[0158] `nnpfc_num_input_pics_minus1+1` represents the number of decoded output images used as input to NNPF. The value of `nnpfc_num_input_pics_minus1` should be in the range of 0 to 63 (inclusive).

[0159] `nnpfc_interpolated_pics[i]` specifies the number of interpolated pictures that NNPF generates between the i-th picture and the (i+1)-th picture used as input to NNPF. The value of `nnpfc_interpolated_pics[i]` should be in the range of 0 to 63 (inclusive). For at least one value of `i` in the range of 0 to `nnpfc_num_input_pics_minus1-1` (inclusive), the value of `nnpfc_interpolated_pics[i]` should be greater than 0.

[0160] If nnpfc_input_pic_output_flag[i] equals 1, it indicates that NNPF will generate the corresponding output image for the i-th input image. If nnpfc_input_pic_output_flag[i] equals 0, it indicates that NNPF will not generate the corresponding output image for the i-th input image.

[0161] The variables numInputPics, which indicate the number of images used as input to NNPF, and numOutputPics, which indicate the total number of images generated as a result of NNPF, can be derived as shown in Table 5.

[0162] [Table 5]

[0163]

[0164] Setting `nnpfc_component_last_flag` to 1 indicates that the last dimension of both the NNPF input tensor (`inputTensor`) and the NNPF output tensor (`outputTensor`) is used for the current channel. Setting `nnpfc_component_last_flag` to 0 indicates that the third dimension of both the NNPF input tensor (`inputTensor`) and the NNPF output tensor (`outputTensor`) is used for the current channel.

[0165] The first dimension of both the input and output tensors can be used for batch indexing, a common practice in some neural network frameworks. Although the formula in this SEI message semantics uses the batch size corresponding to a batch index of 0, the batch size used by the neural network to infer the input is determined by the post-processing implementation.

[0166] For example, when nnpfc_inp_order_idc equals 3 and nnpfc_auxiliary_inp_idc equals 1, the input tensor has 7 channels, including 4 luminance matrices, 2 chrominance matrices, and 1 auxiliary input matrix. In this case, the DeriveInputTensors() process derives each of these 7 channels of the input tensor one by one, and when processing a specific channel, that channel can be referred to as the current channel during processing.

[0167] `nnpfc_inp_format_idc` indicates the method for converting sample values ​​of the cropped decoded output image into NNPF input values. When `nnpfc_inp_format_idc` equals 0, the NNPF input values ​​are real numbers, and the functions `InpY()` and `InpC()` can be specified as shown in Equation 1.

[0168] [Formula 1]

[0169] InpY(x)=x÷((1<<BitDepth Y )-1)

[0170] InpC(x)=x÷((1<<BitDepth C )-1)

[0171] When nnpfc_inp_format_idc equals 1, the input value of NNPF is an unsigned integer, and the functions InpY() and InpC() are specified as shown in Table 6.

[0172] [Table 6]

[0173]

[0174] The variable `inpTensorBitDepthY` can be derived from the syntax element `nnpfc_inp_tensor_luma_bitdepth_minus8`, as specified below. The variable `inpTensorBitDepthC` can be derived from the syntax element `nnpfc_inp_tensor_chroma_bitdepth_minus8`, as specified below.

[0175] Values ​​greater than 1 in nnpfc_inp_format_idc are reserved for future use and may not exist in the bitstream. The decoder must ignore NNPFC SEI messages containing reserved values ​​of nnpfc_inp_format_idc.

[0176] The value of nnpfc_inp_tensor_luma_bitdepth_minus8+8 can specify the bit depth of the luminance sample values ​​in the input integer tensor. The value of inpTensorBitDepthY is derived as shown in Equation 2.

[0177] [Equation 2]

[0178] inpTensorBitDepth Y =nnpfc_inp_tensor_luma_bitdepth_minus8+8

[0179] Bitstream consistency requires the value of nnpfc_inp_tensor_luma_bitdepth_minus8 to be in the range of 0 to 24 (inclusive).

[0180] `nnpfc_inp_tensor_chroma_bitdepth_minus8+8` specifies the bit depth of the chroma sample values ​​in the input integer tensor. The value of `inpTensorBitDepthC` is derived as shown in Equation 3.

[0181] [Formula 3]

[0182] inpTensorBitDepth C=nnpfc_inp_tensor_chroma_bitdepth_minus8+8

[0183] Bitstream consistency requires the value of nnpfc_inp_tensor_chroma_bitdepth_minus8 to be in the range of 0 to 24 (inclusive).

[0184] nnpfc_inp_order_idc can indicate the method for sorting the sample array of the cropped decoded output image into one of the NNPF input images.

[0185] In the bitstream, the value of nnpfc_inp_order_idc should be in the range of 0 to 3 (inclusive). Values ​​of nnpfc_inp_order_idc from 4 to 255 (inclusive) may not exist in the bitstream. The decoder should ignore NNPFC SEI messages with nnpfc_inp_order_idc in the range of 4 to 255 (inclusive). nnpfc_inp_order_idc values ​​greater than 255 may not exist in the bitstream and should not be reserved for future use.

[0186] When ChromaFormatIdc is not equal to 1, the value of nnpfc_inp_order_idc should not be 3.

[0187] Table 7 provides a reference description of the nnpfc_inp_order_idc values.

[0188] [Table 7]

[0189]

[0190] A patch is a rectangular array of samples extracted from the components of an image (such as luminance or chrominance components).

[0191] A value greater than 0 for nnpfc_auxiliary_inp_idc indicates the presence of auxiliary input data in the input tensor of NNPF. A value equal to 0 for nnpfc_auxiliary_inp_idc indicates the absence of auxiliary input data in the input tensor. A value equal to 1 for nnpfc_auxiliary_inp_idc specifies that the auxiliary input data is derived using the methods shown in Tables 8 to 10.

[0192] In the bitstream, the value of nnpfc_auxiliary_inp_idc should be in the range of 0 to 1 (inclusive). Values ​​of nnpfc_inp_order_idc in the range of 2 to 255 (inclusive) should not exist in the bitstream. The decoder should ignore NNPFC SEI messages with nnpfc_inp_order_idc in the range of 2 to 255 (inclusive). nnpfc_inp_order_idc values ​​greater than 255 should not exist in the bitstream and should not be reserved for future use.

[0193] When the value of nnpfc_auxiliary_inp_idc is equal to 1, the variable strengthControlScaledVal can be derived as shown in Equation 4.

[0194] [Formula 4]

[0195]

[0196] The process of deriving the input tensor inputTensors() for the given vertical sample coordinates cTop and horizontal sample coordinates cLeft of the top-left sample position of the sample patch included in the input tensor can be represented as a combination of Tables 8 to 10.

[0197] [Table 8]

[0198]

[0199] [Table 9]

[0200]

[0201] [Table 10]

[0202]

[0203] A value of 1 for `nnpfc_separate_colour_description_present_flag` indicates that the SEI message syntax structure specifies a unique combination of primary colors, transformation properties, and matrix coefficients of the image resulting from NNPF. A value of 0 for `nnpfc_separate_colour_description_present_flag` indicates that the combination of primary colors, transformation properties, and matrix coefficients of the image resulting from NNPF is the same as indicated in the CLVS VUI parameters.

[0204] Except for the following, the semantics of nnpfc_colour_primaries are the same as those defined for the vui_colour_primaries syntax element:

[0205] -nnpfc_colour_primaries can indicate the primary colors of the image obtained after applying the NNPF specified in the SEI message, instead of the primary colors used in CLVS.

[0206] - If nnpfc_colour_primaries does not exist in the NNPFC SEI message, its value can be inferred to be the same as the value of vui_colour_primaries.

[0207] Except for the following, the semantics of nnpfc_transfer_characteristics are the same as those defined for the vui_transfer_characteristics syntax element:

[0208] -nnpfc_transfer_characteristics can indicate the transformation characteristics of the image obtained after applying the NNPF specified in the SEI message, rather than the transformation characteristics used in CLVS.

[0209] - If nnpfc_transfer_characteristics is not present in the NNPFC SEI message, its value can be inferred to be the same as the value of vui_transfer_characteristics.

[0210] Except for the following, the semantics of nnpfc_matrix_coeffs are the same as those specified by the vui_matrix_coeffs syntax element:

[0211] -nnpfc_matrix_coeffs can indicate the matrix coefficients of the image obtained after applying the NNPF specified in the SEI message, instead of the matrix coefficients used by CLVS.

[0212] - If nnpfc_matrix_coeffs does not exist in the NNPFC SEI message, then the value of nnpfc_matrix_coeffs can be inferred to be the same as the value of vui_matrix_coeffs.

[0213] The allowed values ​​for -nnpfc_matrix_coeffs are not limited by the decoded video image chroma format indicated by the ChromaFormatIdc value in the VUI parameter semantics.

[0214] - If the value of nnpfc_matrix_coeffs is equal to 0, the value of nnpfc_out_order_idc shall not be equal to 1 or 3.

[0215] nnpfc_out_format_idc equal to 0 may indicate that the sample values output by the NNPF are real numbers with a value range from 0 to 1, which are linearly mapped to unsigned integers in the value range 0 to (1<<bitDepth)-1 for the bit depth bitDepth required by subsequent post-processing or display. nnpfc_out_format_idc equal to 1 may indicate that the luma sample values output by the NNPF are unsigned integers in the range from 0 to (1<<(nnpfc_out_tensor_luma_bitlength_minus8+8))-1, and the chroma sample values output by the NNPF are unsigned integers in the range from 0 to (1<<(nnpfc_out_tensor_chroma_bitlength_minus8+8))-1.

[0216] Values of nnpfc_out_format_idc greater than 1 are reserved for future use and may not be present in the bitstream. The decoder shall ignore the NNPFC SEI message that contains a reserved value of nnpfc_out_format_idc.

[0217] nnpfc_out_tensor_luma_bitdepth_minus8+8 may indicate the bit depth of luma sample values in the output integer tensor. The value of nnpfc_out_tensor_luma_bitdepth_minus8 shall be in the range of 0 to 24 (inclusive of 0 and 24).

[0218] nnpfc_out_tensor_chroma_bitdepth_minus8+8 may indicate the bit depth of chroma sample values in the output integer tensor. The value of nnpfc_out_tensor_chroma_bitdepth_minus8 shall be in the range of 0 to 24 (inclusive of 0 and 24).

[0219] If nnpfc_purpose&0x10 is not equal to 0, the value of nnpfc_out_format_idc shall be equal to 1, and at least one of the following constraints shall be satisfied:

[0220] - nnpfc_out_tensor_luma_bitdepth_minus8+8 is greater than BitDepth Y

[0221] -nnpfc_out_tensor_chroma_bitdepth_minus8+8 is greater than BitDepth C

[0222] `nnpfc_out_order_idc` indicates the output order of NNPF output samples. In the bitstream, the value of `nnpfc_out_order_idc` should be in the range of 0 to 3 (inclusive). `nnpfc_out_order_idc` values ​​in the range of 4 to 255 may not exist in the bitstream. The decoder should ignore NNPFCSEI messages with `nnpfc_out_order_idc` in the range of 4 to 255 (inclusive). `nnpfc_out_order_idc` values ​​greater than 255 may not exist in the bitstream and are not reserved for future use. If the value of `nnpfc_purpose&0x02` is 0, then the value of `nnpfc_out_order_idc` should not be equal to 3.

[0223] Table 11 describes the values ​​of nnpfc_out_order_idc.

[0224] [Table 11]

[0225]

[0226] Given the vertical sample coordinates cTop and horizontal sample coordinates cLeft of the top-left sample position of the patch of samples included in the input tensor, the processing of sample values ​​in the filtered output sample arrays FilteredYPic, FilteredCbPic, and FilteredCrPic from the output tensor outputTensor, StoreOutputTensors(), can be represented as a combination of Tables 12 and 13.

[0227] [Table 12]

[0228]

[0229] [Table 13]

[0230]

[0231] nnpfc_overlap indicates the horizontal and vertical sample overlap counts of adjacent input tensors in NNPF. The value of nnpfc_overlap should be in the range of 0 to 16383 (inclusive).

[0232] Setting nnpfc_constant_patch_size_flag to 1 indicates that NNPF only accepts the patch size indicated by nnpfc_patch_width_minus1 and nnpfc_patch_height_minus1 as input. Setting `nnpfc_constant_patch_size_flag` to 0 instructs NNPF to accept any patch size (width of `inpPatchWidth` and height of `inpPatchHeight`) as input, such that the width of the extended patch (i.e., the patch plus the overlapping area) (equal to `inpPatchWidth + 2 × nnpfc_overlap`) is a positive integer multiple of `nnpfc_extended_patch_width_cd_delta_minus1 + 1 + 2 × nnpfc_overlap`, and the height of the extended patch (equal to `inpPatchHeight + 2 × nnpfc_overlap`) is a positive integer multiple of `nnpfc_extended_patch_height_cd_delta_minus1 + 1 + 2 × nnpfc_overlap`.

[0233] When nnpfc_constant_patch_size_flag equals 1, nnpfc_patch_width_minus1+1 indicates the horizontal sample count for the required patch size at the NNPF input. The value of nnpfc_patch_width_minus1 should be in the range of 0 to Min(32766,CroppedWidth-1) (inclusive).

[0234] When nnpfc_constant_patch_size_flag equals 1, nnpfc_patch_height_minus1+1 indicates the vertical sample count for the required patch size in the NNPF input. The value of nnpfc_patch_height_minus1 should be in the range of 0 to Min(32766,CroppedHeight-1) (inclusive).

[0235] When nnpfc_constant_patch_size_flag equals 0, nnpfc_extended_patch_width_cd_delta_minus1+1+2×nnpfc_overlap indicates the common divisor of all allowed extended patch width values ​​required by the NNPF input. The value of nnpfc_extended_patch_width_cd_delta_minus1 should be in the range of 0 to Min(32766,CroppedWidth-1) (inclusive).

[0236] When nnpfc_constant_patch_size_flag equals 0, nnpfc_extended_patch_height_cd_delta_minus1+1+2×nnpfc_overlap indicates the common divisor of all allowed extended patch height values ​​required by the NNPF input. The value of nnpfc_extended_patch_height_cd_delta_minus1 should be in the range of 0 to Min(32766,CroppedHeight-1) (inclusive).

[0237] The variables inpPatchWidth and inpPatchHeight can be set to the width and height of the patch size, respectively.

[0238] If nnpfc_constant_patch_size_flag is equal to 0, then the following rules apply:

[0239] The values ​​of –inpPatchWidth and inpPatchHeight can be provided externally or set by the post-processor itself.

[0240] The value of `-inpPatchWidth+2×nnpfc_overlap` should be a positive integer multiple of `nnpfc_extended_patch_width_cd_delta_minus1+1+2×nnpfc_overlap`, and `inpPatchWidth` should be less than or equal to `CroppedWidth`. Similarly, the value of `inpPatchHeight+2×nnpfc_overlap` should be a positive integer multiple of `nnpfc_extended_patch_height_cd_delta_minus1+1+2×nnpfc_overlap`, and `inpPatchHeight` should be less than or equal to `CroppedHeight`.

[0241] Otherwise (nnpfc_constant_patch_size_flag equals 1), the value of inpPatchWidth can be set to nnpfc_patch_width_minus1+1, and the value of inpPatchHeight can be set to nnpfc_patch_height_minus1+1.

[0242] The variables outPatchWidth, outPatchHeight, horCScaling, verCScaling, outPatchCWidth, and outPatchCHeight can be derived as shown in Table 14.

[0243] [Table 14]

[0244]

[0245] Bitstream consistency requires that outPatchWidth × CroppedWidth equal nnpfc_pic_width_in_luma_samples × inpPatchWidth, and outPatchHeight × CroppedHeight equal nnpfc_pic_height_in_luma_samples × inpPatchHeight.

[0246] The `nnpfc_padding_type` parameter indicates the padding used when referencing sample positions outside the boundaries of the cropped decoded output image, as detailed in Table 15. The value of `nnpfc_padding_type` should be in the range of 0 to 15 (inclusive).

[0247] [Table 15]

[0248] 0 Zero fill 1 Copy fill 2 Reflection fill 3 Surround fill 4 Fixed fill 5…15 reserve

[0249] nnpfc_luma_padding_val indicates the luminance value used for padding when the value of nnpfc_padding_type is 4.

[0250] nnpfc_cb_padding_val indicates the Cb value used for padding when the value of nnpfc_padding_type is 4.

[0251] nnpfc_cr_padding_val indicates the Cr value used for padding when the value of nnpfc_padding_type is 4.

[0252] The InpSampleVal(y,x,picHeight,picWidth,CroppedPic) function takes the vertical sample position y, the horizontal sample position x, the image height picHeight, the image width picWidth, and the sample array CroppedPic as input. This function returns the SampleVal value derived as shown in Table 16.

[0253] For the input to the InpSampleVal() function, to be compatible with the input tensor rules of some inference engines, the vertical position can be listed before the horizontal position.

[0254] [Table 16]

[0255]

[0256] The NNPF PostProcessingFilter() function, used in conjunction with the procedures shown in Table 17, can be used to generate filtered and / or interpolated images in a patchwise manner. As indicated by nnpfc_out_order_idc, these images can contain sample arrays of Y, Cb, and Cr, respectively: FilteredYPic, FilteredCbPic, and FilteredCrPic.

[0257] [Table 17]

[0258]

[0259] The order in which the images are stored in the output tensor can be the output order, and the output order generated by applying NNPF to that output order can be interpreted as an output order that does not conflict with the output order of the input images.

[0260] A value of 1 for nnpfc_complexity_info_present_flag can specify the presence of one or more syntax elements indicating the complexity of the NNPF associated with nnpfc_id. A value of 0 for nnpfc_complexity_info_present_flag can specify the absence of syntax elements indicating the complexity of the NNPF associated with nnpfc_id.

[0261] A `nnpfc_parameter_type_idc` value of 0 indicates that the neural network should use only integer parameters. A `nnpfc_parameter_type_flag` value of 1 indicates that the neural network should use either floating-point or integer parameters. A `nnpfc_parameter_type_idc` value of 2 indicates that the neural network should use only binary parameters. A `nnpfc_parameter_type_idc` value of 3 can be reserved for future use and may not exist in the bitstream. The decoder should ignore NNPFC SEI messages with `nnpfc_parameter_type_idc` equal to 3.

[0262] The values ​​of 0, 1, 2, and 3 for nnpfc_log2_parameter_bit_length_minus3 can respectively instruct the neural network not to use parameters with bit lengths greater than 8, 16, 32, and 64. When nnpfc_parameter_type_idc exists and nnpfc_log2_parameter_bit_length_minus3 does not exist, the neural network can choose not to use parameters with a bit length greater than 1.

[0263] `nnpfc_num_parameters_idc` indicates the maximum number of neural network parameters in NNPF, expressed as powers of 2048. A value of 0 for `nnpfc_num_parameters_idc` indicates that the maximum number of neural network parameters is unknown. The value of `nnpfc_num_parameters_idc` should be in the range of 0 to 52 (inclusive). Values ​​greater than 52 are reserved for future use and may not exist in the bitstream. The decoder should ignore NNPFC SEI messages with `nnpfc_num_parameters_idc` greater than 52.

[0264] If the value of nnpfc_num_parameters_idc is greater than 0, then the variable maxNumParameters is derived as shown in Equation 5.

[0265] [Formula 5]

[0266] maxNumParameters=(2048<<nnpfc_num_parameters_idc)-1

[0267] The requirement for bitstream consistency is that the number of neural network parameters in NNPF should be less than or equal to maxNumParameters.

[0268] A value greater than 0 for `nnpfc_num_kmac_operations_idc` indicates that the maximum number of multiply-accumulate operations per sample in NNPF is less than or equal to `nnpfc_num_kmac_operations_idc × 1000`. A value equal to 0 for `nnpfc_num_kmac_operations_idc` indicates that the maximum number of multiply-accumulate operations for the network is unknown. The value of `nnpfc_num_kmac_operations_idc` should be between 0 and 2. 32 -2 (including 0 and 2) 32 Within the range of -2).

[0269] A `nnpfc_total_kilobyte_size` greater than 0 indicates the total size, in kilobytes, required to store the uncompressed parameters of the neural network. The total size, in bits, is equal to or greater than the sum of the bits used to store each parameter. `nnpfc_total_kilobyte_size` is the total size in bits divided by 8000 and rounded up. A `nnpfc_total_kilobyte_size` equal to 0 indicates that the total size required to store the parameters of the neural network is unknown. The value of `nnpfc_total_kilobyte_size` should be between 0 and 2. 32 -2 (including 0 and 2) 32 Within the range of -2).

[0270] In the bitstream, nnpfc_reserved_zero_bit_b should be equal to 0. The decoder should ignore NNPFC SEI messages where nnpfc_reserved_zero_bit_b is not equal to 0.

[0271] nnpfc_payload_byte[i] may contain the i-th byte of the bitstream. The sequence of bytes for all existing values ​​of i, nnpfc_payload_byte[i], should be a complete bitstream conforming to ISO / IEC 15938-17.

[0272] Post-filter activation (NNFPA) of neural networks

[0273] The syntax structure of NNFPA is shown in Table 18.

[0274] [Table 18]

[0275]

[0276] The NNPFA syntax structure in Table 18 can be signaled in the form of an SEI message. The SEI message that signals the NNPFA syntax structure in Table 18 can be called an NNPFA SEI message.

[0277] The Neural Network Post-Filter Activation (NNPFA) SEI message can activate or deactivate the target Neural Network Post-Processing Filter (NNPF) identified by nnpfa_target_id for possible post-processing filtering of a set of images. For a specific image where the NNPF is activated, the target NNPF can be the NNPF specified by the last NNPFC SEI message with an nnpfc_id equal to nnpfa_target_id, where the last NNPFC SEI message may precede the first VCL NAL unit of the current image in the decoding order, and may not correspond to a repetition of the NNPFC SEI message containing the underlying NNPF.

[0278] Several NNPFA SEI messages can exist for the same image, for example, when NNPF is used for different purposes or to filter different color components.

[0279] nnpfa_target_id can indicate the target NNPF, which is specified by one or more NNPFC SEI messages that are associated with the current image and have an nnpfc_id equal to nnpfa_target_id.

[0280] The value of nnpfa_target_id should be between 0 and 2. 32 -2 (including 0 and 2) 32 The range is -2). The range is from 256 to 511 (inclusive) and 2. 31 Up to 2 32 -2 (including 2) 31 and 2 32 -2) nnpfa_target_id values ​​within the range can be reserved for future use. The decoder should ignore nnpfa_target_id values ​​in the range of 256 to 511 (inclusive) or 2. 31 Up to 2 32 -2 (including 2) 31 and 2 32 -2) NNPFA SEI messages within the range.

[0281] NNPFASEI messages with a specific nnpfa_target_id value should not exist in the current PU unless one or both of the following conditions are true:

[0282] - Within the current CLVS, there exists an NNPFCSEI message with an nnpfc_id that is equal to the specific nnpfa_target_id value, which exists in a PU that is decoded before the current PU.

[0283] - An NNPFC SEI message exists in the current PU with an nnpfc_id that is equal to the specific nnpfa_target_id value.

[0284] When a PU contains both an NNPFC SEI message with a specific nnpfc_id value and an NNPFA SEI message with an nnpfa_target_id that is equal to that specific nnpfc_id value, the NNPFC SEI message should be decoded before the NNPFA SEI message.

[0285] A `nnpfa_cancel_flag` equal to 1 indicates that the persistence of a target NNPF established by any previous NNPFA SEI message with the same `nnpfa_target_id` as the current SEI message is cancelled; that is, the target NNPF is no longer used unless it is activated by another NNPFA SEI message with the same `nnpfa_target_id` as the current SEI message and `nnpfa_cancel_flag` equal to 0. A `nnpfa_cancel_flag` equal to 0 indicates adherence to the `nnpfa_persistence_flag`.

[0286] `nnpfa_persistence_flag` specifies the persistence of the target NNPF in the current layer. A `nnpfa_persistence_flag` of 0 specifies that the target NNPF can be used only for post-processing filtering of the current image. A `nnpfa_persistence_flag` of 1 specifies that the target NNPF can be used for post-processing filtering of the current image and all subsequent images in the current layer in output order, until one or more of the following conditions are true:

[0287] - A new CLVS begins for the current layer.

[0288] -End of bitstream.

[0289] - In terms of output order, output the image in the current layer that is associated with the NNPFA SEI message that has the same nnpfa_target_id as the current SEI message and whose nnpfa_cancel_flag is equal to 1, after the current image.

[0290] For the subsequent image in the current layer that is associated with an NNPFA SEI message that has the same nnpfa_target_id as the current SEI message and whose nnpfa_cancel_flag is equal to 1, no target NNPF is applied.

[0291] `nnpfcTargetPictures` can be the set of pictures involved in the last NNPFA SEI message that precedes the current NNPFA SEI message in the decoding order and has an `nnpfc_id` equal to `nnpfa_target_id`. `nnpfaTargetPictures` can also be the set of pictures that activate the target NNPF in the current NNPFA SEI message. A requirement for bitstream consistency is that any pictures included in `nnpfaTargetPictures` should also be included in `nnpfcTargetPictures`.

[0292] Post-filter hints

[0293] The syntax structure of post-filter hints is shown in Table 19.

[0294] [Table 19]

[0295]

[0296] The post-filter hint syntax structure in Table 19 can be signaled as an SEI message. The SEI message signaling the post-filter hint syntax structure in Table 19 can be referred to as a post-filter hint SEI message.

[0297] The SEI message provides information about the coefficients of the post-filter or the relevance of the post-filter design, which can be potentially used in post-processing after a set of images has been decoded and output, thereby achieving improved display quality.

[0298] A filter_hint_cancel_flag value of 1 indicates that the SEI message cancels the persistence of any preceding or following filter hints SEI messages to be applied to the current layer in the output order. A filter_hint_cancel_flag value of 0 indicates that the following filter hints should be followed.

[0299] `filter_hint_persistence_flag` specifies the persistence of the post-filter hint SEI message for the current layer. A `filter_hint_persistence_flag` of 0 specifies that the post-filter hint applies only to the current decoded picture. A `filter_hint_persistence_flag` of 1 specifies that the post-filter hint SEI message applies to the current decoded picture and remains valid for all subsequent pictures in the current layer in output order, until one or more of the following conditions are true:

[0300] - A new CLVS begins for the current layer.

[0301] -End of bitstream.

[0302] - In terms of output order, output the image in the AU of the current layer that is associated with the post-filter hint SEI message after the current image.

[0303] The filter_hint_size_y parameter specifies the vertical size of the filter coefficients or the correlation array. The value of filter_hint_size_y should be in the range of 1 to 15 (inclusive).

[0304] The filter_hint_size_x parameter specifies the horizontal size of the filter coefficients or the correlation array. The value of filter_hint_size_x should be in the range of 1 to 15 (inclusive).

[0305] `filter_hint_type` identifies the type of filter hint sent, as specified in Table 20. The value of `filter_hint_type` should be in the range of 0 to 2 (inclusive). A `filter_hint_type` value equal to 3 may not exist in the bitstream. The decoder should ignore post-filter hint SEI messages with `filter_hint_type` equal to 3.

[0306] [Table 20]

[0307] 0 Coefficients of 2D-FIR filter 1 Coefficients of a 1D-FIR filter 2 Cross-correlation matrix

[0308] A filter_hint_chroma_coeff_present_flag value of 1 can specify filter coefficients that have chroma. A filter_hint_chroma_coeff_present_flag value of 0 can specify filter coefficients that do not have chroma.

[0309] The `filter_hint_value[cIdx][cy][cx]` method specifies the filter coefficients or elements of the cross-correlation matrix between the original and decoded signals with 16-bit precision. The value of `filter_hint_value[cIdx][cy][cx]` should be in the range of -2. 31 +1 to 2 31 -1 (including -2) 31 +1 and 2 31 Within the range of -1). cIdx can specify the associated color components, cy represents the counter in the vertical direction, and cx represents the counter in the horizontal direction. Depending on the value of filter_hint_type, the following items can be applied:

[0310] - If filter_hint_type equals 0, the coefficients of a two-dimensional finite impulse response (FIR) filter with size filter_hint_size_y × filter_hint_size_x can be sent.

[0311] Otherwise, if `filter_hint_type` equals 1, the filter coefficients of two one-dimensional FIR filters can be sent. In this case, `filter_hint_size_y` should be equal to 2. A `cy` index equal to 0 specifies the filter coefficients of the horizontal filter, and a `cy` index equal to 1 specifies the filter coefficients of the vertical filter. In the filtering process, the horizontal filter is applied first, and the result is then filtered by the vertical filter.

[0312] - Otherwise (filter_hint_type equals 2), the hint sent can specify the cross-correlation matrix between the original signal s and the decoded signal s'.

[0313] The normalized cross-correlation matrix of the correlated color components identified by cIdx and having a size of filter_hint_size_y × filter_hint_size_x can be defined as shown in Equation 6.

[0314] [Formula 6]

[0315]

[0316] In Equation 6, s denotes the sample array of the color component cIdx of the original picture, s' denotes the corresponding array of the decoded picture, h denotes the vertical height of the relevant color component, w denotes the horizontal width of the relevant color component, bitDepth denotes the bit depth of this color component, OffsetY is equal to (filter_hint_size_y>>1), OffsetX is equal to (filter_hint_size_x>>1), cy ranges from 0<=cy<filter_hint_size_y, and cx ranges from 0<=cx<filter_hint_size_x.

[0317] The decoder may derive the Wiener post-filter from the cross-correlation matrix of the original signal and the decoded signal and the auto-correlation matrix of the decoded signal.

[0318] Related technical issues

[0319] To control the size of the output picture, the following design is currently applied:

[0320] - If the purpose of NNPF does not include spatial upsampling, the size of the output picture is inferred to be the same as the size of the input picture.

[0321] - Otherwise, the size of the output picture is signaled.

[0322] Considering that an NNPF carrying an NNPFC SEI message may be activated by multiple NNPFA SEI messages in different pictures, the size of the input picture may vary according to the activation time, but currently, if the NNPF includes the purpose of spatial upsampling, the size of the output picture is fixed to be the same.

[0323] Implementation

[0324] The present disclosure proposes multiple embodiments to solve the above problems. The embodiments proposed in the present disclosure may be executed individually or in combination with each other.

[0325] Through the present disclosure, the following possible and suitable embodiments are proposed.

[0326] 1. The bitstream may include various picture sizes, and the NNPF may be used to upsample the resolution of all pictures to the same size.

[0327] 2. The bitstream may include various picture sizes, and the NNPF may be used to upsample the resolution of all pictures while maintaining the ratio of the picture resolutions of the input bitstream.

[0328] The embodiments proposed by the present disclosure are summarized as follows.

[0329] 1. When signaling the output image size, a flag (i.e., `nnpfc_output_pic_size_flag`) can be added to indicate whether the output image size is a fixed size (i.e., a given width and height) or a ratio relative to the input image size. A `nnpfc_output_pic_size_flag` of 0 indicates that there are widths and heights of luminance samples for the output image size. A `nnpfc_output_pic_size_flag` of 1 indicates that there is a ratio between the output and input image sizes.

[0330] 2. When the value of nnpfc_output_pic_size_flag is 1, it can send signals to indicate the numerator and denominator of the ratio of the output image width to the input image width, nnpfc_pic_width_ratio_num and nnpfc_pic_width_ratio_denum, and it can also send signals to indicate the numerator and denominator of the ratio of the output image height to the input image height, nnpfc_pic_height_ratio_num and nnpfc_pic_height_ratio_denum.

[0331] 3. When signaling the output image size, the scaling ratio values ​​for width and height can be signaled. This ratio can be signaled as both the numerator and denominator. The ratio (or ratio value) can be signaled using the "minus1" terminology to save bits and avoid signaling zero values. Additionally, the ratio (or ratio value) can be encoded and signaled via zero-order exponent Golomb (ue(v)) encoding.

[0332] 4. `nnpfc_pic_width_num_minus1` indicates the numerator of the ratio between the input image width and the output image width, and `nnpfc_pic_width_num_minus1` indicates the denominator of the ratio between the input image width and the output image width. Similarly, `nnpfc_pic_height_num_minus1` indicates the numerator of the ratio between the input image height and the output image height, and `nnpfc_pic_height_num_minus1` indicates the denominator of the ratio between the input image height and the output image height.

[0333] 5. The output image width can be derived by multiplying a non-integer ratio by the input image width and then rounding up. Similarly, the output image height can be derived by multiplying a non-integer ratio by the input image height and then rounding up.

[0334] 6. If nnpfc_pic_width_num_minus1, nnpfc_pic_width_denum_minus1, nnpfc_pic_height_num_minus1, and nnpfc_pic_height_denum_minus1 exist, then the value of (nnpfc_pic_width_num_minus1+1)÷(nnpfc_pic_width_denum_minus1+1) should be non-1, or the value of (nnpfc_pic_height_num_minus1+1)÷(nnpfc_pic_height_denum_minus1+1) should be non-1.

[0335] 7. Alternatively, if nnpfc_pic_width_num_minus1, nnpfc_pic_width_denum_minus1, nnpfc_pic_height_num_minus1, and nnpfc_pic_height_denum_minus1 exist, then the value of (nnpfc_pic_width_num_minus1+1)÷(nnpfc_pic_width_denum_minus1+1) should be greater than 1, or the value of (nnpfc_pic_height_num_minus1+1)÷(nnpfc_pic_height_denum_minus1+1) should be greater than 1.

[0336] The embodiments of this disclosure will now be described in detail.

[0337] Implementation Method 1

[0338] Implementation method 1 relates to 1 and 2 in the above summary.

[0339] According to implementation method 1, the NNPFC SEI message can be configured as shown in Table 21.

[0340] [Table 21]

[0341]

[0342] A value of 0 for nnpfc_output_pic_size_flag indicates the existence of nnpfc_pic_width_in_luma_samples and nnpfc_pic_width_in_luma_samples. A value of 1 for nnpfc_output_pic_size_flag indicates the existence of nnpfc_pic_width_ratio_num, nnpfc_pic_width_ratio_denum, nnpfc_pic_height_ratio_num, and nnpfc_pic_height_ratio_denum.

[0343] `nnpfc_pic_width_in_luma_samples` and `nnpfc_pic_height_in_luma_samples` respectively indicate the width and height of the luminance sample array of the image obtained by applying NNPF (identified by `nnpfc_id`) to the cropped decoded output image. The value of `nnpfc_pic_width_in_luma_samples` can be in the range of `CroppedWidth` to `CroppedWidth×16-1` (inclusive), and the value of `nnpfc_pic_height_in_luma_samples` can be in the range of `CroppedHeight` to `CroppedHeight×16-1` (inclusive). `CroppedWidth` indicates the width of the input image, and `CroppedHeight` indicates the height of the input image.

[0344] `nnpfc_pic_width_ratio_num` and `nnpfc_pic_width_ratio_denum` can respectively indicate the numerator and denominator of the ratio of the output image width to the cropped width. The values ​​of `nnpfc_pic_width_ratio_num` and `nnpfc_pic_width_ratio_denum` can be in the range of 1 to 16-1 (inclusive).

[0345] The variable PicWidthInLumaSamples, which indicates the width of the output image, can be derived as follows.

[0346] - If nnpfc_output_pic_size_flag does not exist, the value of PicWidthInLumaSamples can be set to equal CroppedWidth.

[0347] Otherwise, if nnpfc_output_pic_size_flag is 0, the value of PicWidthInLumaSamples can be set to equal nnpfc_pic_width_in_luma_samples.

[0348] Otherwise (i.e., nnpfc_output_pic_size_flag is 1), the value of PicWidthInLumaSamples can be set to equal to (CroppedWidth × nnpfc_pic_width_ratio_num) / nnpfc_pic_width_ratio_denum.

[0349] `nnpfc_pic_height_ratio_num` and `nnpfc_pic_height_ratio_denum` can respectively indicate the numerator and denominator of the ratio of the output image height to the cropped height. The values ​​of `nnpfc_pic_height_ratio_num` and `nnpfc_pic_height_ratio_denum` can be in the range of 1 to 16-1 (inclusive).

[0350] The variable PicHeightInLumaSamples, which indicates the height of the output image, can be derived as follows:

[0351] - If nnpfc_output_pic_size_flag does not exist, the value of PicHeightInLumaSamples can be set to equal CroppedHeight.

[0352] Otherwise, if nnpfc_output_pic_size_flag is 0, the value of PicHeightInLumaSamples can be set to equal nnpfc_pic_height_in_luma_samples.

[0353] - Otherwise (i.e., nnpfc_output_pic_size_flag is 1), the value of PicHeightInLumaSamples can be set to equal to (CroppedHeight × nnpfc_pic_height_ratio_num) / nnpfc_pic_height_ratio_denum.

[0354] The procedure StoreOutputTensors(), which derives the sample values ​​in the filtered output sample arrays FilteredYPic, FilteredCbPic, and FilteredCrPic from the output tensor for a given vertical sample coordinate cTop and horizontal sample coordinate cLeft indicating the top-left sample position of a sample patch contained in the input tensor, can be modified to a combination of Tables 22 and 23.

[0355] [Table 22]

[0356]

[0357] [Table 23]

[0358]

[0359] The derivation process of variables outPatchWidth, outPatchHeight, horCScaling, verCScaling, outPatchCWidth, and outPatchCHeight can also be modified as shown in Table 24.

[0360] [Table 24]

[0361]

[0362] The requirements for bitstream consistency are: outPatchWidth × CroppedWidth should be equal to PicWidthInLumaSamples × inpPatchWidth, and outPatchHeight × CroppedHeight should be equal to PicHeightInLumaSamples × inpPatchHeight.

[0363] The image encoding method and image decoding method according to Embodiment 1 will be described below. The image encoding method can be executed by the image encoding device 100, and the image decoding method can be executed by the image decoding device 200.

[0364] Figure 5 An example of an image encoding method is shown, and Figure 6 An example of an image decoding method is shown.

[0365] Reference Figure 5 The size of the output image can be determined (S510). Furthermore, the type of size information indicating the size of the output image can be determined (S520). For example, the size information can consist of `nnpfc_pic_width_in_luma_samples` indicating the width of the luminance sample array of the output image and `nnpfc_pic_height_in_luma_samples` indicating the height of the luminance sample array of the output image. As another example, the size information can consist of `nnpfc_pic_width_ratio_num` and `nnpfc_pic_width_ratio_denum` indicating the ratio of the output image width to the input image width, and `nnpfc_pic_height_ratio_num` and `nnpfc_pic_height_ratio_denum` indicating the ratio of the output image height to the input image height. The size information composed of `nnpfc_pic_width_in_luma_samples` and `nnpfc_pic_height_in_luma_samples` can be referred to as "non-ratio information". The dimensional information composed of nnpfc_pic_width_ratio_num, nnpfc_pic_width_ratio_denum, nnpfc_pic_height_ratio_num, and nnpfc_pic_height_ratio_denum can be referred to as "ratio information" or "dimensional information". Here, non-ratio information and ratio information can be associated with the type of dimensional information.

[0366] It can be determined whether the type of the size information is ratio (S530). If the type of the size information is determined to be ratio (i.e., ratio information), then nnpfc_output_pic_size_flag can be encoded with a first value (e.g., 1), and the ratio information can be encoded (S540). On the other hand, if the type of the size information is determined not to be ratio (i.e., non-ratio information), then nnpfc_output_pic_size_flag can be encoded with a second value (e.g., 0), and the ratio information can be encoded in the bitstream (NNPFC SEI message) (S540). The size information can be encoded based on zero-order exponential Golomb coding. nnpfc_output_pic_size_flag can be referred to as "first information".

[0367] Reference Figure 6The `nnpfc_output_pic_size_flag` is obtained from the bitstream (NNPFC SEI message) (S610), and the value of `nnpfc_output_pic_size_flag` can be determined (S620). If the value of `nnpfc_output_pic_size_flag` is a first value (e.g., 1) indicating the presence of ratio information (or size information), the ratio information can be obtained from the bitstream (S630). Conversely, if the value of `nnpfc_output_pic_size_flag` is a second value (e.g., 0) indicating the presence of non-ratio information, the non-ratio information can be obtained from the bitstream (S650).

[0368] The size of the output image can be determined based on the obtained ratio information or non-ratio information (S640). For example, when ratio information is obtained, the width of the output image (PicWidthInLumaSamples) can be determined by applying the width ratio (nnpfc_pic_width_ratio_num / nnpfc_pic_width_ratio_denum) indicated by the ratio information to the width of the input image, and the height of the output image (PicHeightInLumaSamples) can be determined by applying the height ratio (nnpfc_pic_height_ratio_num / nnpfc_pic_height_ratio_denum) indicated by the ratio information to the height of the input image. As another example, when non-ratio information is obtained, the width (PicWidthInLumaSamples) of the output image can be determined to be equal to the value of the width (nnpfc_pic_width_in_luma_samples) indicated by the non-ratio information, and the height (PicHeightInLumaSamples) of the output image can be determined to be equal to the value of the height (nnpfc_pic_height_in_luma_samples) indicated by the non-ratio information.

[0369] Implementation Method 2

[0370] Implementation method 2 relates to points 3 to 7 in the above summary.

[0371] According to implementation method 2, the NNPFC SEI message can be configured as shown in Table 25.

[0372] [Table 25]

[0373]

[0374] nnpfc_pic_width_num_minus1+1 and nnpfc_pic_width_denum_minus1+1 can indicate the numerator and denominator of the ratio of the output image width to the CroppedWidth. The value of (nnpfc_pic_width_num_minus1+1)÷(nnpfc_pic_width_denum_minus1+1) can be in the range of 1 to 16-1 (inclusive).

[0375] The value of the variable nnpfcOutputPicWidth indicates the width of the luminance sample array of the image obtained by applying NNPF identified by nnpfc_id to the cropped decoded output image, and can be derived as shown in Equation 7.

[0376] [Formula 7]

[0377] nnpfcOutputPicWidth =

[0378] Ceil((CroppedWidth*((nnpfc_pic_width_num_minus1+1))÷(nnpfc_pic_width_denum_minus1+1))

[0379] nnpfc_pic_height_num_minus1+1 and nnpfc_pic_height_denum_minus1+1 can indicate the numerator and denominator of the ratio of the output image height to the CroppedHeight. The value of (nnpfc_pic_height_num_minus1+1)÷(nnpfc_pic_height_denum_minus1+1) can be in the range of 1 to 16-1 (inclusive).

[0380] The value of the variable nnpfcOutputPicHeight indicates the height of the luminance sample array of the image obtained by applying NNPF identified by nnpfc_id to the cropped decoded output image, and can be derived as shown in Equation 8.

[0381] [Formula 8]

[0382] nnpfcOutputPicHeight=

[0383] Ceil((CroppedHeight*(nnpfc_pic_height_num_minus1+1))÷(nnpfc_pic_height_denum_minus1+1))

[0384] If nnpfc_pic_width_num_minus1, nnpfc_pic_width_denum_minus1, nnpfc_pic_height_num_minus1, and nnpfc_pic_height_denum_minus1 exist, then the value of (nnpfc_pic_width_num_minus1+1)÷(nnpfc_pic_width_denum_minus1+1) can be greater than 1, or the value of (nnpfc_pic_height_num_minus1+1)÷(nnpfc_pic_height_denum_minus1+1) can be greater than 1. Alternatively, if nnpfc_pic_width_num_minus1, nnpfc_pic_width_denum_minus1, nnpfc_pic_height_num_minus1, and nnpfc_pic_height_denum_minus1 exist, then the value of (nnpfc_pic_width_num_minus1+1)÷(nnpfc_pic_width_denum_minus1+1) can be a non-1 value, or the value of (nnpfc_pic_height_num_minus1+1)÷(nnpfc_pic_height_denum_minus1+1) can be a non-1 value.

[0385] The procedure StoreOutputTensors(), which derives the sample values ​​in the filtered output sample arrays FilteredYPic, FilteredCbPic, and FilteredCrPic from the output tensor outputTensor for a given vertical sample coordinate cTop and horizontal sample coordinate cLeft indicating the top-left sample position of the sample patch contained in the input tensor, can be modified to a combination of Tables 26 and 27.

[0386] [Table 26]

[0387]

[0388] [Table 27]

[0389]

[0390] The derivation process of variables outPatchWidth, outPatchHeight, horCScaling, verCScaling, outPatchCWidth, and outPatchCHeight can also be changed as shown in Table 28.

[0391] [Table 28]

[0392]

[0393] The requirements for bitstream consistency are: outPatchWidth × CroppedWidth should be equal to nnpfcOutputPicWidth × inpPatchWidth, and outPatchHeight × CroppedHeight should be equal to nnpfcOutputPicHeight × inpPatchHeight.

[0394] The image encoding method and image decoding method according to Embodiment 2 will be described below. The image encoding method can be executed by the image encoding device 100, and the image decoding method can be executed by the image decoding device 200.

[0395] Figure 7 An example of an image encoding method is shown, and Figure 8 An example illustrating an image decoding method.

[0396] Reference Figure 7 The ratio between the size of the output image and the size of the input image can be determined (S710). Furthermore, the determined ratio and size information can be encoded in a bitstream (NNPFC SEI message) (S720). The size information can be encoded based on zero-order exponential Golomb coding.

[0397] The size information can include information about the numerator and denominator values ​​of the determined ratios. Information about the numerator values ​​can include the numerator value of the ratio between the width of the output image and the width of the input image (width ratio) (nnpfc_pic_width_num_minus1) and the numerator value of the ratio between the height of the output image and the height of the input image (height ratio) (nnpfc_pic_height_num_minus1). Similarly, information about the denominator values ​​can include the denominator value of the width ratio (nnpfc_pic_width_denum_minus1) and the denominator value of the height ratio (nnpfc_pic_height_denum_minus1).

[0398] The information regarding the numerator values ​​(nnpfc_pic_width_num_minus1, nnpfc_pic_height_num_minus1) can indicate the value obtained by subtracting 1 from the numerator value of the width ratio, and can also indicate the value obtained by subtracting 1 from the numerator value of the height ratio. Therefore, the numerator value of the width ratio can be determined by adding 1 to the information regarding the numerator value, and the numerator value of the height ratio can be determined by adding 1 to the information regarding the numerator value. Similarly, the information regarding the denominator values ​​(nnpfc_pic_width_denum_minus1, nnpfc_pic_height_denum_minus1) can indicate the value obtained by subtracting 1 from the denominator value of the width ratio, and can also indicate the value obtained by subtracting 1 from the denominator value of the height ratio. Therefore, the denominator value of the width ratio can be determined by adding 1 to the information regarding the denominator value, and the denominator value of the height ratio can be determined by adding 1 to the information regarding the denominator value.

[0399] Reference Figure 8 When size information is obtained from the bitstream (NNPFC SEI message) (S810), the size of the output image can be determined based on the obtained size information (S820). Here, the size determination of the output image can be performed by resolution upsampling.

[0400] For example, the width of the output image (nnpfcOutputPicWidth) can be determined by applying a width ratio ((nnpfc_pic_width_num_minus1+1)÷(nnpfc_pic_width_denum_minus1+1)) to the width value of the input image (CroppedWidth) and rounding up the input image width to the nearest integer. As another example, the height of the output image (nnpfcOutputPicHeight) can be determined by applying a height ratio ((nnpfc_pic_height_num_minus1+1)÷(nnpfc_pic_height_denum_minus1+1)) to the height value of the input image (CroppedHeight) and rounding up the input image height to the nearest integer.

[0401] Figure 9 An image coding method with limited width and height ratios is shown, and Figure 10 An image decoding method with limited width and height ratios is shown.

[0402] Reference Figure 9It can be determined whether the size of the output image is equal to the size of the input image (S910). For example, it can be determined whether the width ratio is equal to 1 and the height ratio is equal to 1. As another example, it can be determined whether the width ratio is greater than 1 and the height ratio is greater than 1.

[0403] If the output image size is not equal to the input image size, the size information can be encoded in the bitstream (NNPFC SEI message) (S920). Conversely, if the output image size is equal to the input image size, the size information does not need to be encoded. For example, if the width ratio is not equal to 1 and the height ratio is not equal to 1, the size information is encoded; if the width ratio is equal to 1 and the height ratio is equal to 1, the size information does not need to be encoded. As another example, if the width ratio is greater than 1 and the height ratio is greater than 1, the size information is encoded; if the width ratio is less than or equal to 1 and the height ratio is less than or equal to 1, the size information does not need to be encoded.

[0404] Reference Figure 10 It can be determined whether size information exists (S1010). That is, it can be determined whether size information was obtained from the bitstream (NNPFC SEI message). Referring to Tables 4 and 25, the existence of size information can have the same meaning as whether resolution upsampling is performed. That is, the existence of size information can mean that resolution upsampling is performed ((nnpfc_purpose&0x04) != 0), and the absence of size information can mean that resolution upsampling is not performed ((nnpfc_purpose&0x04) == 0).

[0405] When size information is available, the dimensions of the output image can be constrained to have values ​​different from those of the input image. For example, the width ratio can be constrained to have a value not equal to 1, and the height ratio can be constrained to have a value not equal to 1. As another example, the width ratio can be constrained to have a value greater than 1, and the height ratio can be constrained to have a value greater than 1.

[0406] Figure 11 This is a diagram illustrating a content streaming system to which embodiments of the present disclosure can be applied.

[0407] like Figure 11 As shown, a content streaming system to which embodiments of the present disclosure can be applied may generally include an encoding server, a streaming server, a web server, a media storage device, a user device, and a multimedia input device.

[0408] An encoding server compresses content input from multimedia input devices such as smartphones, cameras, and camcorders into digital data to generate a bitstream, and then sends the bitstream to a streaming server. As another example, when multimedia input devices such as smartphones, cameras, and camcorders directly generate bitstreams, the encoding server can be omitted.

[0409] The bitstream can be generated by an image encoding method or image encoding device applicable to embodiments of the present disclosure, and the streaming server can temporarily store the bitstream during the sending or receiving of the bitstream.

[0410] A streaming server sends multimedia data to a user's device based on a user's request via a web server, and the web server acts as a medium for notifying the user of services. When a user requests a desired service from the web server, the web server can deliver the request to the streaming server, and the streaming server can send multimedia data to the user. In this scenario, the content streaming system may include a separate control server. In this case, the control server is used to control the commands / responses between the devices in the content streaming system.

[0411] A streaming server can receive content from media storage and / or encoding servers. For example, when receiving content from an encoding server, the content can be received in real time. In this case, to provide a smooth streaming service, the streaming server can store the bitstream for a predetermined period of time.

[0412] Examples of user equipment may include mobile phones, smartphones, laptops, digital broadcast terminals, personal digital assistants (PDAs), portable multimedia players (PMPs), navigators, tablet PCs, ultrabooks, wearable devices (e.g., smartwatches, smart glasses, head-mounted displays), digital TVs, desktop computers, digital signage, etc.

[0413] In a content streaming system, each server can operate as a distributed server, in which case data received from each server can be processed in a distributed manner.

[0414] The scope of this disclosure includes software or machine-executable commands (e.g., operating systems, applications, firmware, programs, etc.) for enabling operation of methods according to various embodiments to be executed on a device or computer, and non-transitory computer-readable media on which such software or commands are stored and executable on a device or computer.

[0415] Industrial applicability

[0416] The embodiments disclosed herein can be used to encode or decode images.

Claims

1. A decoding device for image decoding, the decoding device comprising: Memory; as well as At least one processor, connected to the memory, wherein the at least one processor is configured to: After obtaining the neural network filter characteristics, NNPFC supplements the enhancement information SEI message, and... The input image is post-filtered based on the NNPFC SEI message to obtain the output image. The NNPFC SEI message includes ratio information for determining the size of the output image. The ratio information includes numerator and denominator information, which specify the ratio of the size of the output image to the size of the input image. The first flag information is obtained based on the NNPFC usage information in the NNPFC SEI message, and Specifically, if the first flag information is equal to 1, the ratio information is included in the NNPFC SEI message, and if the first flag information is equal to 0, the ratio information is not included in the NNPFC SEI message.

2. The decoding device according to claim 1, wherein, The molecular information includes first molecular information relating to a first ratio of the width of the output image to the width of the input image, and second molecular information relating to a second ratio of the height of the output image to the height of the input image.

3. The decoding device according to claim 2, wherein, The denominator information includes first denominator information for the first ratio and second denominator information for the second ratio.

4. The decoding device according to claim 2, wherein, The molecular value for the first ratio is obtained by adding 1 to the first molecular information, and The molecule value for the second ratio is obtained by adding 1 to the second molecule information.

5. The decoding device according to claim 3, wherein, The denominator value for the first ratio is obtained by adding 1 to the first denominator information, and The denominator value for the second ratio is obtained by adding 1 to the second denominator information.

6. The decoding device according to claim 1, wherein, The ratio information is encoded based on the zero-order exponential Columbus code.

7. The decoding device according to claim 2, wherein, The width of the output image is determined by rounding up the product of the width of the input image and the first ratio.

8. The decoding device according to claim 2, wherein, The height of the output image is determined by rounding up the product of the height of the input image and the second ratio.

9. The decoding device according to claim 1, wherein, The ratio of the size of the output image to the size of the input image, as specified by the ratio information, is not equal to 1.

10. The decoding device according to claim 1, wherein, Based on the fact that the first flag information is equal to 0, the specific size information of the output image is included in the NNPFC SEI message.

11. An encoding device for image encoding, the encoding device comprising: Memory; as well as At least one processor, connected to the memory, wherein the at least one processor is configured to: After generating the neural network, the filter characteristics NNPFC are supplemented with enhancement information SEI messages, and... The image information, including the NNPFC SEI message, is encoded. The NNPFC SEI message includes ratio information for determining the size of the output image, which is obtained by post-filtering the input image. The ratio information includes numerator and denominator information, which specify the ratio of the size of the output image to the size of the input image. The first flag information is obtained based on the NNPFC usage information in the NNPFC SEI message, and Specifically, if the first flag information is equal to 1, the ratio information is included in the NNPFC SEI message, and if the first flag information is equal to 0, the ratio information is not included in the NNPFC SEI message.

12. An apparatus for transmitting a bit stream, the apparatus comprising: Transmitter; as well as At least one processor, The device is configured to perform a method including the following steps: Generate a bitstream including neural network post-filter characteristics, NNPFC supplemental enhancement information, SEI messages; and Send the bit stream, The NNPFC SEI message includes ratio information for determining the size of the output image, which is obtained by post-filtering the input image. The ratio information includes numerator and denominator information, which specify the ratio of the size of the output image to the size of the input image. The first flag information is obtained based on the NNPFC usage information in the NNPFC SEI message, and Specifically, if the first flag information is equal to 1, the ratio information is included in the NNPFC SEI message, and if the first flag information is equal to 0, the ratio information is not included in the NNPFC SEI message.

Images