Image encoding / decoding method and device, and recording medium for storing bitstream

Abstract

An image decoding method and device according to the present disclosure can receive a bitstream including an encoded video picture and reconstruct the encoded video picture included in the bitstream. The bitstream can include information related to a neural-network post-filter (NNPF). The NNPF-related information can include at least one from among an NNPFA SEI message, an NNPFC SEI message, and an NNPFU SEI message.

Description

Video encoding / decoding method and device, and recording medium storing bitstream

[0001] The present invention relates to a video encoding / decoding method and device, and a recording medium storing a bitstream.

[0002] Recently, the demand for high-resolution, high-quality images, such as HD (High Definition) images and UHD (Ultra High Definition) images, is increasing in various application fields, and accordingly, high-efficiency image compression technologies are being discussed.

[0003] There are various technologies for image compression, such as inter prediction technology that predicts pixel values ​​included in the current picture from pictures before or after the current picture, intra prediction technology that predicts pixel values ​​included in the current picture using pixel information within the current picture, and entropy encoding technology that assigns short codes to values ​​with high frequency of appearance and long codes to values ​​with low frequency of appearance, and these technologies can be used to effectively compress and transmit or store image data.

[0004] The present disclosure provides a method and device for configuring NNPF related information.

[0005] The present disclosure provides a method and device for signaling NNPF related information.

[0006] A video decoding method and device according to the present disclosure can receive a bitstream including an encoded video picture and restore the encoded video picture included in the bitstream. The bitstream can be configured to include information related to a neural network post-processing filter (NNPF).

[0007] In the image decoding method and device according to the present disclosure, the NNPF-related information may include at least one of a neural-network post-filter activation (NNPFA) SEI (supplemental enhancement information) message, a neural-network post-filter characteristics (NNPFC) SEI message, or a neural-network post-filter update (NNPFU) SEI message.

[0008] In the video decoding method and device according to the present disclosure, the NNPF-related information can be obtained from a NAL (network abstraction layer) unit of the bitstream.

[0009] In the image decoding method and device according to the present disclosure, the NNPFA SEI message may include prompt information. The prompt information may specify a text string prompt for generating the contents of a spatial extrapolation image area.

[0010] In the video decoding method and device according to the present disclosure, the NNPFA SEI message may further include a prompt presence flag. The prompt presence flag having a value of 1 may indicate that the prompt information exists and that zero bit information may exist. The prompt presence flag having a value of 0 may indicate that the prompt information and the zero bit information do not exist.

[0011] In the video decoding method and device according to the present disclosure, the prompt information can be adaptively signaled based on the prompt presence flag.

[0012] In the video decoding method and device according to the present disclosure, the NNPFA SEI message may include a prompt update flag. The prompt update flag may be related to whether the prompt information is updated.

[0013] In the video decoding method and device according to the present disclosure, the prompt information can be adaptively signaled based on the prompt update flag.

[0014] In the image decoding method and device according to the present disclosure, the NNPFC SEI message may include prompt information. The prompt information may specify a text string prompt for generating the contents of a spatial extrapolation image area.

[0015] In the video decoding method and device according to the present disclosure, the NNPFC SEI message may further include a prompt presence flag. The prompt presence flag having a value of 1 may indicate that the prompt information exists and that zero bit information may exist. The prompt presence flag having a value of 0 may indicate that the prompt information and the zero bit information do not exist.

[0016] In the video decoding method and device according to the present disclosure, the prompt information can be adaptively signaled based on the prompt presence flag.

[0017] In the image decoding method and device according to the present disclosure, the NNPFU SEI message may include prompt information. The prompt information may specify a text string prompt for generating the contents of a spatial extrapolation image area.

[0018] In the video decoding method and device according to the present disclosure, the NNPFU SEI message may further include a prompt presence flag. The prompt presence flag having a value of 1 may indicate that the prompt information exists and that zero bit information may exist. The prompt presence flag having a value of 0 may indicate that the prompt information and the zero bit information do not exist.

[0019] In the video decoding method and device according to the present disclosure, the prompt information can be adaptively signaled based on the prompt presence flag.

[0020] A video encoding method and device according to the present disclosure may receive a video picture to be encoded, encode the received video picture to generate video information about the video picture, generate neural-network post-filter (NNPF) related information, and generate a bitstream including the video information and the NNPF related information. The NNPF related information may include at least one of a neural-network post-filter activation (NNPFA) SEI (supplemental enhancement information) message, a neural-network post-filter characteristics (NNPFC) SEI message, or a neural-network post-filter update (NNPFU) SEI message.

[0021] A computer-readable digital storage medium is provided, which stores encoded video / image information that causes a decoding device according to the present disclosure to perform a video decoding method.

[0022] A computer-readable digital storage medium storing video / image information generated by a video encoding method according to the present disclosure is provided.

[0023] A method and device for transmitting video / image information generated by a video encoding method according to the present disclosure are provided.

[0024] By configuring prompt-related information in a specific location, such as NNPFA information, as described herein, the problem of having to repeatedly signal prompt-related information whenever the prompt information is updated can be resolved.

[0025] FIG. 1 illustrates a video / image coding system according to the present disclosure.

[0026] FIG. 2 is a schematic block diagram of an encoding device to which an embodiment of the present disclosure can be applied and in which encoding of a video / image signal is performed.

[0027] FIG. 3 is a schematic block diagram of a decoding device to which an embodiment of the present disclosure can be applied and in which decoding of a video / image signal is performed.

[0028] FIG. 4 illustrates a method for restoring a video picture performed in a decoding device (300) according to the present disclosure.

[0029] FIG. 5 illustrates a method for deriving a luma channel as an embodiment according to the present disclosure.

[0030] FIG. 6 illustrates a schematic configuration of a decoding device (300) that performs a method for restoring a video picture according to the present disclosure.

[0031] FIG. 7 illustrates a method for generating a bitstream performed in an encoding device (200) according to the present disclosure.

[0032] FIG. 8 illustrates a schematic configuration of an encoding device (200) that performs a method for generating a bitstream according to the present disclosure.

[0033] FIG. 9 illustrates an example of a content streaming system to which embodiments of the present disclosure can be applied.

[0034] The present disclosure may be modified in various ways and encompasses numerous embodiments. Specific embodiments are illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the present disclosure to specific embodiments, but rather to encompass all modifications, equivalents, and alternatives falling within the spirit and technical scope of the present disclosure. Throughout the description of each drawing, similar reference numerals have been used to designate similar components.

[0035] While terms such as "first" and "second" may be used to describe various components, these components should not be limited by these terms. These terms are used solely to distinguish one component from another. For example, without departing from the scope of the present disclosure, a first component could be referred to as a "second component," and similarly, a second component could also be referred to as a "first component." The term "and / or" includes a combination of multiple related items described herein or any of multiple related items described herein.

[0036] When a component is referred to as being "connected" or "connected" to another component, it should be understood that it may be directly connected or connected to that other component, but that there may be other components intervening. Conversely, when a component is referred to as being "directly connected" or "connected" to another component, it should be understood that there are no other components intervening.

[0037] The terminology used in this application is only used to describe specific embodiments and is not intended to limit the present disclosure. The singular expression includes the plural expression unless the context clearly indicates otherwise. In this application, it should be understood that the terms "comprise" or "have" indicate the presence of a feature, number, step, operation, component, part, or combination thereof described in the specification, but do not preclude the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0038] The present disclosure relates to video / image coding. For example, the methods / embodiments disclosed in this specification can be applied to methods disclosed in the versatile video coding (VVC) standard. In addition, the methods / embodiments disclosed in this specification can be applied to methods disclosed in the essential video coding (EVC) standard, the AOMedia Video 1 (AV1) standard, the second generation of audio video coding standard (AVS2), or the next generation of video / image coding standards (e.g., H.267 or H.268).

[0039] This specification presents various embodiments of video / image coding, and unless otherwise stated, the embodiments may be performed in combination with each other.

[0040] In this specification, a video may refer to a set of images over time. A picture generally refers to a unit representing one image at a specific time point, and a slice / tile is a unit that constitutes part of a picture in coding. A slice / tile may include one or more coding tree units (CTUs). A picture may be composed of one or more slices / tiles. A tile is a rectangular area consisting of multiple CTUs within a specific tile column and a specific tile row of a picture. A tile column is a rectangular area of ​​CTUs that has a height equal to the height of the picture and a width specified by the syntax requirements of the picture parameter set. A tile row is a rectangular area of ​​CTUs that has a height specified by the picture parameter set and a width equal to the width of the picture. CTUs within a tile are arranged consecutively according to the CTU raster scan, while tiles within a picture may be arranged consecutively according to the tile raster scan. A slice may contain an integer number of complete tiles or an integer number of contiguous complete CTU rows within a picture, which may be exclusively contained within a single NAL unit. Meanwhile, a picture may be divided into two or more subpictures. A subpicture may be a rectangular region of one or more slices within a picture.

[0041] A pixel, or pel, can refer to the smallest unit that constitutes a picture (or image). Additionally, the term "sample" can be used as a counterpart to a pixel. A sample can generally represent a pixel or a pixel value, and can represent only the pixel / pixel value of the luminance component, or only the pixel / pixel value of the chrominance component.

[0042] A unit may represent a basic unit of image processing. A unit may include at least one of a specific region of a picture and information related to the region. One unit may include one luma block and two chroma (e.g., cb, cr) blocks. In some cases, the term "unit" may be used interchangeably with terms such as "block" or "area." In general, an MxN block may include a set (or array) of samples (or sample array) or transform coefficients consisting of M columns and N rows.

[0043] As used herein, "A or B" can mean "only A," "only B," or "both A and B." In other words, as used herein, "A or B" can be interpreted as "A and / or B." For example, as used herein, "A, B or C" can mean "only A," "only B," "only C," or "any combination of A, B and C."

[0044] As used herein, a slash ( / ) or a comma can mean "and / or." For example, "A / B" can mean "A and / or B." Accordingly, "A / B" can mean "only A," "only B," or "both A and B." For example, "A, B, C" can mean "A, B, or C."

[0045] In this specification, "at least one of A and B" may mean "only A", "only B" or "both A and B". Additionally, in this specification, the expressions "at least one of A or B" or "at least one of A and / or B" may be interpreted identically to "at least one of A and B".

[0046] Additionally, in this specification, “at least one of A, B and C” can mean “only A,” “only B,” “only C,” or “any combination of A, B and C.” Additionally, “at least one of A, B or C” or “at least one of A, B and / or C” can mean “at least one of A, B and C.”

[0047] Additionally, parentheses used herein may mean "for example." Specifically, when "prediction (intra-prediction)" is indicated, "intra-prediction" may be suggested as an example of "prediction." In other words, "prediction" in this specification is not limited to "intra-prediction," and "intra-prediction" may be suggested as an example of "prediction." Furthermore, even when "prediction (i.e., intra-prediction)" is indicated, "intra-prediction" may be suggested as an example of "prediction."

[0048] Technical features individually described in a single drawing in this specification may be implemented individually or simultaneously.

[0049] FIG. 1 illustrates a video / image coding system according to the present disclosure.

[0050] Referring to FIG. 1, a video / image coding system may include a first device (source device) and a second device (receiving device).

[0051] A source device can transmit encoded video / image information or data to a receiving device via a digital storage medium or a network in the form of a file or streaming. The source device may include a video source, an encoding device, and a transmitting device. The receiving device may include a receiving device, a decoding device, and a renderer. The encoding device may be referred to as a video / image encoding device, and the decoding device may be referred to as a video / image decoding device. The transmitter may be included in the encoding device. The receiver may be included in the decoding device. The renderer may include a display unit, and the display unit may be configured as a separate device or an external component.

[0052] A video source may obtain video / images through a process of capturing, synthesizing, or generating video / images. The video source may include a video / image capture device and / or a video / image generation device. The video / image capture device may include one or more cameras, a video / image archive containing previously captured video / images, etc. The video / image generation device may include a computer, a tablet, a smartphone, etc., and may (electronically) generate video / images. For example, a virtual video / image may be generated through a computer, etc., in which case the video / image capture process may be replaced by a process of generating related data.

[0053] An encoding device can encode input video / images. The encoding device can perform a series of procedures, such as prediction, transformation, and quantization, to improve compression and coding efficiency. The encoded data (encoded video / image information) can be output in the form of a bitstream.

[0054] The transmission unit can transmit encoded video / image information or data output in the form of a bitstream to the receiving unit of a receiving device via a digital storage medium or network in the form of a file or streaming. The digital storage medium can include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, SSD, etc. The transmission unit can include an element for generating a media file via a predetermined file format and an element for transmission via a broadcasting / communication network. The receiving unit can receive / extract the bitstream and transmit it to a decoding device.

[0055] The decoding device can decode the video / image by performing a series of procedures such as inverse quantization, inverse transformation, and prediction corresponding to the operation of the encoding device.

[0056] The renderer can render decoded video / images. The rendered video / images can be displayed through the display unit.

[0057] FIG. 2 is a schematic block diagram of an encoding device to which an embodiment of the present disclosure can be applied and in which encoding of a video / image signal is performed.

[0058] Referring to FIG. 2, the encoding device (200) may be configured to include an image partitioner (210), a prediction unit (predictor) 220, a residual processor (residual processor) 230, an entropy encoder (entropy encoder) 240, an adder (adder) 250, a filter (filter) 260, and a memory (memory) 270. The prediction unit (220) may include an inter prediction unit (221) and an intra prediction unit (222). The residual processor (230) may include a transformer (transformer) 232, a quantizer (quantizer) 233, a dequantizer (dequantizer) 234, and an inverse transformer (inverse transformer) 235. The residual processing unit (230) may further include a subtractor (231). The addition unit (250) may be called a reconstructor or a recontructed block generator. The image segmentation unit (210), the prediction unit (220), the residual processing unit (230), the entropy encoding unit (240), the addition unit (250), and the filtering unit (260) described above may be configured by one or more hardware components (e.g., an encoding device chipset or processor) according to an embodiment. In addition, the memory (270) may include a decoded picture buffer (DPB) and may be configured by a digital storage medium. The hardware component may further include the memory (270) as an internal / external component.

[0059] The image segmentation unit (210) can segment an input image (or picture, frame) input to the encoding device (200) into one or more processing units. For example, the processing unit may be called a coding unit (CU). In this case, the coding unit may be recursively segmented from a coding tree unit (CTU) or a largest coding unit (LCU) according to a QTBTTT (Quad-tree binary-tree ternary-tree) structure.

[0060] For example, a single coding unit may be split into multiple coding units with deeper depths based on a quad-tree structure, a binary tree structure, and / or a ternary structure. In this case, for example, the quad-tree structure may be applied first, and the binary tree structure and / or the ternary structure may be applied later. Alternatively, the binary tree structure may be applied before the quad-tree structure. The coding procedure according to the present specification may be performed based on the final coding unit that is no longer split. In this case, based on coding efficiency according to image characteristics, etc., the largest coding unit may be used directly as the final coding unit, or, if necessary, the coding unit may be recursively split into coding units of lower depths, and the coding unit with the optimal size may be used as the final coding unit. Here, the coding procedure may include procedures such as prediction, transformation, and restoration, which will be described later.

[0061] As another example, the processing unit may further include a prediction unit (PU) or a transform unit (TU). In this case, the prediction unit and the transform unit may each be split or partitioned from the final coding unit described above. The prediction unit may be a unit of sample prediction, and the transform unit may be a unit for deriving a transform coefficient and / or a unit for deriving a residual signal from a transform coefficient.

[0062] The term "unit" may be used interchangeably with terms such as "block" or "area" depending on the case. In general, an MxN block can represent a set of samples or transform coefficients consisting of M columns and N rows. A sample can generally represent a pixel or a pixel value, and can represent only the pixel / pixel value of the luminance component, or only the pixel / pixel value of the chrominance component. A sample can be used as a term corresponding to a pixel or pel in a picture (or image).

[0063] The encoding device (200) can generate a residual signal (residual block, residual sample array) by subtracting a prediction signal (prediction block, prediction sample array) output from an inter prediction unit (221) or an intra prediction unit (222) from an input video signal (original block, original sample array), and the generated residual signal is transmitted to a conversion unit (232). In this case, a unit that subtracts a prediction signal (prediction block, prediction sample array) from an input video signal (original block, original sample array) within the encoding device (200) may be called a subtraction unit (231).

[0064] The prediction unit (220) can perform a prediction on a block to be processed (hereinafter, referred to as a current block) and generate a predicted block including prediction samples for the current block. The prediction unit (220) can determine whether intra prediction or inter prediction is applied on a current block or CU basis. The prediction unit (220) can generate various information related to prediction, such as prediction mode information, as described later in the description of each prediction mode, and transmit the information to the entropy encoding unit (240). The information related to prediction can be encoded by the entropy encoding unit (240) and output in the form of a bitstream.

[0065] The intra prediction unit (222) can predict the current block by referring to samples in the current picture. The referenced samples may be located in the neighborhood of the current block, or may be located a certain distance away from the current block, depending on the prediction mode. In intra prediction, the prediction modes may include one or more non-directional modes and multiple directional modes. The non-directional mode may include at least one of a DC mode or a planar mode. The directional mode may include 33 directional modes or 65 directional modes depending on the degree of detail in the prediction direction. However, this is only an example, and a greater or lesser number of directional modes may be used depending on the settings. The intra prediction unit (222) may also determine the prediction mode applied to the current block by using the prediction mode applied to the neighboring blocks.

[0066] The inter prediction unit (221) can derive a prediction block for the current block based on a reference block (reference sample array) specified by a motion vector on a reference picture. At this time, in order to reduce the amount of motion information transmitted in the inter prediction mode, the motion information can be predicted in units of blocks, subblocks, or samples based on the correlation of motion information between neighboring blocks and the current block. The motion information can include a motion vector and a reference picture index. The motion information can further include inter prediction direction information (L0 prediction, L1 prediction, Bi prediction, etc.). In the case of inter prediction, the neighboring block can include a spatial neighboring block existing in the current picture and a temporal neighboring block existing in the reference picture. The reference picture including the reference block and the reference picture including the temporal neighboring block may be the same or different. The above temporal neighboring blocks may be called collocated reference blocks, collocated CUs (colCUs), etc., and the reference pictures including the temporal neighboring blocks may be called collocated pictures (colPic). For example, the inter prediction unit (221) may construct a motion information candidate list based on the neighboring blocks, and generate information indicating which candidate is used to derive the motion vector and / or reference picture index of the current block. Inter prediction may be performed based on various prediction modes, and for example, in the case of skip mode and merge mode, the inter prediction unit (221) may use the motion information of the neighboring blocks as the motion information of the current block. In the case of skip mode, unlike the merge mode, a residual signal may not be transmitted.In the motion vector prediction (MVP) mode, the motion vector of the surrounding blocks is used as a motion vector predictor, and the motion vector of the current block can be indicated by signaling the motion vector difference.

[0067] The prediction unit (220) can generate a prediction signal based on various prediction methods described below. For example, the prediction unit can apply intra prediction or inter prediction for prediction of a single block, and can also apply intra prediction and inter prediction simultaneously. This can be called combined inter and intra prediction (CIIP) mode. In addition, the prediction unit can be based on an intra block copy (IBC) prediction mode or a palette mode for prediction of a block. The IBC prediction mode or palette mode can be used for content image / video coding such as games, such as screen content coding (SCC). IBC basically performs prediction within the current picture, but can be performed similarly to inter prediction in that it derives a reference block within the current picture. That is, IBC can utilize at least one of the inter prediction techniques described herein. Palette mode can be viewed as an example of intra coding or intra prediction. When the palette mode is applied, sample values ​​within a picture can be signaled based on information about the palette table and palette index. The prediction signal generated through the prediction unit (220) can be used to generate a restoration signal or a residual signal.

[0068] The transform unit (232) can apply a transform technique to the residual signal to generate transform coefficients. For example, the transform technique can include at least one of a Discrete Cosine Transform (DCT), a Discrete Sine Transform (DST), a Karhunen-Loeve Transform (KLT), a Graph-Based Transform (GBT), or a Conditionally Non-linear Transform (CNT). Here, GBT refers to a transform obtained from a graph when the relationship information between pixels is expressed as a graph. CNT refers to a transform obtained based on generating a prediction signal using all previously restored pixels. In addition, the transform process can be applied to a pixel block having a square size and the same size, or can be applied to a block of a non-square variable size.

[0069] The quantization unit (233) quantizes the transform coefficients and transmits them to the entropy encoding unit (240), and the entropy encoding unit (240) can encode the quantized signal (information about the quantized transform coefficients) and output it as a bitstream. The information about the quantized transform coefficients can be called residual information. The quantization unit (233) can rearrange the quantized transform coefficients in a block form into a one-dimensional vector form based on the coefficient scan order, and can also generate information about the quantized transform coefficients based on the quantized transform coefficients in the one-dimensional vector form.

[0070] The entropy encoding unit (240) can perform various encoding methods such as exponential Golomb, context-adaptive variable length coding (CAVLC), context-adaptive binary arithmetic coding (CABAC), etc. The entropy encoding unit (240) can also encode information necessary for video / image restoration (e.g., values ​​of syntax elements, etc.) together or separately from quantized transform coefficients.

[0071] Encoded information (e.g., encoded video / image information) can be transmitted or stored in the form of a bitstream in units of NAL (network abstraction layer) units. The video / image information may further include information on various parameter sets, such as an adaptation parameter set (APS), a picture parameter set (PPS), a sequence parameter set (SPS), or a video parameter set (VPS). In addition, the video / image information may further include general constraint information. In the present specification, information and / or syntax elements transmitted / signaled from an encoding device to a decoding device may be included in the video / image information. The video / image information may be encoded through the above-described encoding procedure and included in the bitstream. The bitstream may be transmitted via a network or stored in a digital storage medium. Here, the network may include a broadcasting network and / or a communication network, and the digital storage medium may include various storage media, such as a USB, SD, CD, DVD, Blu-ray, HDD, or SSD. The signal output from the entropy encoding unit (240) may be configured as an internal / external element of the encoding device (200) by a transmitting unit (not shown) and / or a storing unit (not shown), or the transmitting unit may be included in the entropy encoding unit (240).

[0072] The quantized transform coefficients output from the quantization unit (233) can be used to generate a prediction signal. For example, by applying inverse quantization and inverse transformation to the quantized transform coefficients through the inverse quantization unit (234) and the inverse transform unit (235), a residual signal (residual block or residual samples) can be reconstructed. The addition unit (250) can generate a reconstructed signal (reconstructed picture, reconstructed block, reconstructed sample array) by adding the reconstructed residual signal to the prediction signal output from the inter prediction unit (221) or the intra prediction unit (222). When there is no residual for the block to be processed, such as when skip mode is applied, the predicted block can be used as a reconstructed block. The addition unit (250) may be called a reconstructor or a reconstructed block generation unit. The generated restoration signal can be used for intra prediction of the next processing target block within the current picture, and can also be used for inter prediction of the next picture after filtering as described below. Meanwhile, LMCS (luma mapping with chroma scaling) may be applied during the picture encoding and / or restoration process.

[0073] The filtering unit (260) can improve subjective / objective picture quality by applying filtering to the restoration signal. For example, the filtering unit (260) can apply various filtering methods to the restoration picture to generate a modified restoration picture, and store the modified restoration picture in the memory (270), specifically, in the DPB of the memory (270). The various filtering methods can include deblocking filtering, sample adaptive offset, adaptive loop filter, bilateral filter, etc. The filtering unit (260) can generate various information regarding filtering and transmit it to the entropy encoding unit (240). The information regarding filtering can be encoded by the entropy encoding unit (240) and output in the form of a bitstream.

[0074] The modified restored picture transmitted to the memory (270) can be used as a reference picture in the inter prediction unit (221). Through this, when inter prediction is applied, the encoding device can avoid prediction mismatch between the encoding device (200) and the decoding device, and can also improve encoding efficiency.

[0075] The DPB of the memory (270) can store the modified restored picture to be used as a reference picture in the inter prediction unit (221). The memory (270) can store motion information of a block from which motion information is derived (or encoded) within the current picture and / or motion information of blocks within a picture that has already been restored. The stored motion information can be transferred to the inter prediction unit (221) to be used as motion information of a spatial neighboring block or motion information of a temporal neighboring block. The memory (270) can store restored samples of restored blocks within the current picture and transfer them to the intra prediction unit (222).

[0076] FIG. 3 is a schematic block diagram of a decoding device to which an embodiment of the present disclosure can be applied and in which decoding of a video / image signal is performed.

[0077] Referring to FIG. 3, the decoding device (300) may be configured to include an entropy decoder (310), a residual processor (320), a predictor (330), an adder (340), a filter (350), and a memory (360). The predictor (330) may include an inter-prediction unit (332) and an intra-prediction unit (331). The residual processor (320) may include a dequantizer (321) and an inverse transformer (321).

[0078] The entropy decoding unit (310), residual processing unit (320), prediction unit (330), addition unit (340), and filtering unit (350) described above may be configured by a single hardware component (e.g., a decoding device chipset or processor) depending on the embodiment. In addition, the memory (360) may include a decoded picture buffer (DPB) and may be configured by a digital storage medium. The hardware component may further include the memory (360) as an internal / external component.

[0079] When a bitstream including video / image information is input, the decoding device (300) can restore the image corresponding to the process in which the video / image information is processed in the encoding device of FIG. 2. For example, the decoding device (300) can derive units / blocks based on block division-related information obtained from the bitstream. The decoding device (300) can perform decoding using a processing unit applied in the encoding device. Accordingly, the processing unit for decoding may be a coding unit, and the coding unit may be divided from a coding tree unit or a maximum coding unit according to a quad tree structure, a binary tree structure, and / or a ternary tree structure. One or more transform units may be derived from the coding unit. Then, the restored image signal decoded and output by the decoding device (300) can be reproduced through a reproduction device.

[0080] The decoding device (300) can receive a signal output from the encoding device of FIG. 2 in the form of a bitstream, and the received signal can be decoded through the entropy decoding unit (310). For example, the entropy decoding unit (310) can parse the bitstream to derive information (e.g., video / image information) necessary for image restoration (or picture restoration). The video / image information may further include information on various parameter sets, such as an adaptation parameter set (APS), a picture parameter set (PPS), a sequence parameter set (SPS), or a video parameter set (VPS). In addition, the video / image information may further include general constraint information. The decoding device can decode the picture further based on the information on the parameter set and / or the general constraint information. The signaling / received information and / or syntax elements described later in this specification can be decoded through the decoding procedure and obtained from the bitstream. For example, the entropy decoding unit (310) can decode information in a bitstream based on a coding method such as exponential Golomb coding, CAVLC, or CABAC, and output the values ​​of syntax elements required for image restoration and the quantized values ​​of transform coefficients for residuals. More specifically, the CABAC entropy decoding method receives a bin corresponding to each syntax element in the bitstream, determines a context model using information of the syntax element to be decoded and decoding information of the surrounding and decoding target blocks or information of symbols / bins decoded in the previous step, and predicts the occurrence probability of the bin according to the determined context model to perform arithmetic decoding of the bin to generate a symbol corresponding to the value of each syntax element.At this time, the CABAC entropy decoding method can update the context model using the information of the decoded symbol / bin for the context model of the next symbol / bin after determining the context model. Information regarding prediction among the information decoded by the entropy decoding unit (310) is provided to the prediction unit (inter prediction unit (332) and intra prediction unit (331)), and residual values ​​on which entropy decoding is performed by the entropy decoding unit (310), i.e., quantized transform coefficients and related parameter information, can be input to the residual processing unit (320). The residual processing unit (320) can derive a residual signal (residual block, residual samples, residual sample array). In addition, information regarding filtering among the information decoded by the entropy decoding unit (310) can be provided to the filtering unit (350). Meanwhile, a receiving unit (not shown) that receives a signal output from an encoding device may be further configured as an internal / external element of a decoding device (300), or the receiving unit may be a component of an entropy decoding unit (310).

[0081] Meanwhile, a decoding device according to the present specification may be called a video / video / picture decoding device, and the decoding device may be divided into an information decoding device (video / video / picture information decoding device) and a sample decoding device (video / video / picture sample decoding device). The information decoding device may include the entropy decoding unit (310), and the sample decoding device may include at least one of the inverse quantization unit (321), the inverse transformation unit (322), the adding unit (340), the filtering unit (350), the memory (360), the inter prediction unit (332), and the intra prediction unit (331).

[0082] The inverse quantization unit (321) can inverse quantize the quantized transform coefficients and output the transform coefficients. The inverse quantization unit (321) can rearrange the quantized transform coefficients into a two-dimensional block form. In this case, the rearrangement can be performed based on the coefficient scanning order performed in the encoding device. The inverse quantization unit (321) can perform inverse quantization on the quantized transform coefficients using quantization parameters (e.g., quantization step size information) and obtain transform coefficients.

[0083] In the inverse transform unit (322), the transform coefficients are inversely transformed to obtain a residual signal (residual block, residual sample array).

[0084] The prediction unit (320) can perform a prediction on the current block and generate a predicted block including prediction samples for the current block. The prediction unit (320) can determine whether intra-prediction or inter-prediction is applied to the current block based on the information regarding the prediction output from the entropy decoding unit (310), and can determine a specific intra / inter-prediction mode.

[0085] The prediction unit (320) can generate a prediction signal based on various prediction methods described below. For example, the prediction unit (320) can apply intra prediction or inter prediction for prediction of a single block, and can also apply intra prediction and inter prediction simultaneously. This can be called combined inter and intra prediction (CIIP) mode. In addition, the prediction unit can be based on an intra block copy (IBC) prediction mode or a palette mode for prediction of a block. The IBC prediction mode or palette mode can be used for content image / video coding such as games, such as screen content coding (SCC). IBC basically performs prediction within the current picture, but can be performed similarly to inter prediction in that it derives a reference block within the current picture. That is, IBC can utilize at least one of the inter prediction techniques described herein. Palette mode can be viewed as an example of intra coding or intra prediction. When palette mode is applied, information about the palette table and palette index may be included and signaled in the video / image information.

[0086] The intra prediction unit (331) can predict the current block by referring to samples within the current picture. The referenced samples may be located in the neighborhood of the current block, or may be located a certain distance away from the current block, depending on the prediction mode. In intra prediction, the prediction modes may include one or more non-directional modes and multiple directional modes. The intra prediction unit (331) may also determine the prediction mode applied to the current block by using the prediction mode applied to the neighboring blocks.

[0087] The inter prediction unit (332) can derive a prediction block for the current block based on a reference block (reference sample array) specified by a motion vector on a reference picture. At this time, in order to reduce the amount of motion information transmitted in the inter prediction mode, the motion information can be predicted in units of blocks, subblocks, or samples based on the correlation of the motion information between the neighboring blocks and the current block. The motion information can include a motion vector and a reference picture index. The motion information can further include inter prediction direction information (L0 prediction, L1 prediction, Bi prediction, etc.). In the case of inter prediction, the neighboring blocks can include spatial neighboring blocks existing in the current picture and temporal neighboring blocks existing in the reference picture. For example, the inter prediction unit (332) can construct a motion information candidate list based on the neighboring blocks, and derive the motion vector and / or reference picture index of the current block based on the received candidate selection information. Inter prediction can be performed based on various prediction modes, and information about the prediction can include information indicating an inter prediction mode for the current block.

[0088] The addition unit (340) can generate a restoration signal (restored picture, restoration block, restoration sample array) by adding the acquired residual signal to the prediction signal (prediction block, prediction sample array) output from the prediction unit (including the inter-prediction unit (332) and / or intra-prediction unit (331)). When there is no residual for the block to be processed, such as when skip mode is applied, the prediction block can be used as the restoration block.

[0089] The addition unit (340) may be referred to as a restoration unit or restoration block generation unit. The generated restoration signal may be used for intra prediction of the next processing target block within the current picture, may be output after filtering as described below, or may be used for inter prediction of the next picture. Meanwhile, LMCS (luma mapping with chroma scaling) may be applied during the picture decoding process.

[0090] The filtering unit (350) can improve subjective / objective image quality by applying filtering to the restored signal. For example, the filtering unit (350) can apply various filtering methods to the restored picture to generate a modified restored picture, and transmit the modified restored picture to the memory (360), specifically, to the DPB of the memory (360). The various filtering methods can include deblocking filtering, sample adaptive offset, adaptive loop filter, bilateral filter, etc.

[0091] The (corrected) reconstructed picture stored in the DPB of the memory (360) can be used as a reference picture in the inter prediction unit (332). The memory (360) can store motion information of a block from which motion information is derived (or decoded) in the current picture and / or motion information of blocks in a picture that has already been reconstructed. The stored motion information can be transferred to the inter prediction unit (332) to be used as motion information of a spatial neighboring block or motion information of a temporal neighboring block. The memory (360) can store reconstructed samples of reconstructed blocks in the current picture and transfer them to the intra prediction unit (331).

[0092] In this specification, the embodiments described in the filtering unit (260), the inter prediction unit (221), and the intra prediction unit (222) of the encoding device (200) can be applied to the filtering unit (350), the inter prediction unit (332), and the intra prediction unit (331) of the decoding device (300) in the same or corresponding manner, respectively.

[0093] FIG. 4 illustrates a method for restoring a video picture performed in a decoding device (300) according to the present disclosure.

[0094] A bitstream including an encoded video picture can be received (S400).

[0095] The encoded video picture of the bitstream can be restored (S410).

[0096] Video information about an encoded video picture can be extracted from a bitstream. The encoded video picture can be restored based on the extracted video information.

[0097] The bitstream may include information related to a neural network post-filter (NNPF). The NNPF-related information may include at least one of neural network post-filter activation (NNPF) information, neural network post-filter characteristics (NNPFC) information, or neural network post-filter update (NNPFU) information.

[0098] NNPFA information, NNPFC information, and NNPFU information can be configured in SEI (supplemental enhancement information) messages, respectively. In this case, NNPFA information, NNPFC information, and NNPFU information can be referred to as NNPFA SEI messages, NNPFC SEI messages, and NNPFU SEI messages, respectively. For convenience of explanation, it is assumed below that NNPF-related information is configured in SEI messages.

[0099] The NNPFA SEI message can enable or disable the possible use of a target NNPF for post-processing filtering of pictures. Here, the target NNPF can be identified based on at least one of a target identifier (nnpfa_target_id) or a target base flag (nnpfa_target_base_flag) described below.

[0100] For a given picture with NNPF enabled, the target NNPF can be: If nnpfa_target_base_flag is 1, the target NNPF can be the base NNPF with nnpfc_id equal to nnpfa_target_id. Otherwise (i.e., if nnpfa_target_base_flag is 0), the target NNPF can be the NNPF specified by the last NNPFC SEI message, which can be the one with nnpfc_id equal to nnpfa_target_id that precedes the first VCL NAL unit of the current picture in decoding order. If nnpfa_target_base_flag is 0, the target NNPF can be a non-repeating NNPFC SEI message containing the base NNPF.

[0101] There may be multiple NNPFA SEI messages for the same picture. For example, if NNPF is used for different purposes or to filter different color components, there may be multiple NNPFA SEI messages for the same picture.

[0102] For example, the NNPFA SEI message according to the present disclosure may be structured as shown in Table 1 below.

[0103] nn_post_filter_activation( payloadSize ) {Descriptornnpfa_target_idue(v)nnpfa_cancel_flagu(1)if( !nnpfa_cancel_flag ) {nnpfa_persistence_flagu(1)nnpfa_target_base_flagu(1)if( nnpfa_persistence_flag )nnpfa_no_foll_clvs_flagu(1)nnpfa_num_output_entriesue(v)for( i = 0; i < nnpfa_num_output_entries; i++ )nnpfa_output_flag[ i ]u(1)if( more_data_in_payload( ) ) {nnpfa_spatial_extrapolation_prompt_present_flagu(1)if (nnpfa_spatial_extrapolation_prompt_present_flag){while( !byte_aligned( ) )nnpfa_alignment_zero_bit_cu(1)nnpfa_promptst(v)}}}

[0104] According to Table 1, the NNPFA SEI message may include a target identifier (nnpfa_target_id). The nnpfa_target_id may indicate an identifier (nnpfc_id) of a target NNPF. The target NNPF may be specified based on one or more NNPFC SEI messages related to the current picture. The target NNPF may have an nnpfc_id that is the same as the nnpfa_target_id. The value of the nnpfa_target_id may be from 0 to (2 32 -2) may be limited to the range of up to 10 ...

[0105] If either or both of the following conditions are not met, an NNPFA SEI message with a specific value of nnpfa_target_id may be restricted from being present in the current picture unit (PU).

[0106] (Condition 1) There is an NNPFC SEI message with an nnpfc_id that is identical to a specific value of nnpfa_target_id that exists in a PU preceding the current PU in decoding order within the current CLVS (coded layer video sequence).

[0107] (Condition 2) There is an NNPFC SEI message in the current PU with an nnpfc_id equal to a specific value of nnpfa_target_id.

[0108] If a PU contains both an NNPFC SEI message having an nnpfc_id of a specific value and an NNPFA SEI message having an nnpfa_target_id equal to the nnpfc_id of the specific value, the NNPFC SEI message must precede the NNPFA SEI message in decoding order.

[0109] The NNPFA SEI message may include a cancel flag (nnpfa_cancel_flag). If nnpfa_cancel_flag is 1, this may indicate that the persistence of the target NNPF established by a previous NNPFA SEI message with the same nnpfa_target_id as the current SEI message is canceled. That is, the target NNPF may no longer be used unless activated by another NNPFA SEI message with the same nnpfa_target_id as the current SEI message and a cancel flag with a value of 0. If nnpfa_cancel_flag is 0, this may indicate that it is followed by nnpfa_persistence_flag, nnpfa_target_base_flag, nnpfa_no_prev_clvs_flag, nnpfa_no_foll_clvs_flag, and nnpfa_num_output_entries, which are described below.

[0110] The NNPFA SEI message may include a persistence flag (nnpfa_persistence_flag). nnpfa_persistence_flag may specify the persistence of the target NNPF for the current layer. For example, if the value of nnpfa_persistence_flag is 0, it may specify that the target NNPF can be used for post-processing filtering only for the current picture. If the value of nnpfa_persistence_flag is 1, it may specify that the target NNPF can be used for post-processing filtering for the current picture and all subsequent pictures of the current layer in output order.

[0111] The set of pictures for which the target NNPF is activated by the current NNPFA SEI message is called nnpfaTargetPictures, and the set of pictures for which the NNPFC SEI message corresponding to the target NNPF is applied is called nnpfcTargetPictures. At this time, the pictures included in nnpfaTargetPictures can be restricted to also be included in nnpfcTargetPictures.

[0112] The NNPFA SEI message may include a target base flag (nnpfa_target_base_flag). If the value of nnpfa_target_base_flag is 1, this may indicate that the target NNPF is a base NNPF. Here, the base NNPF may be one having the same nnpfc_id as the nnpfa_target_id described above. If the value of nnpfa_target_base_flag is 0, this may indicate that the target NNPF is an NNPF specified by the last NNPFC SEI message. Here, the last NNPFC SEI message may be one having the same nnpfc_id as the nnpfa_target_id that precedes the first VCL NAL unit of the current picture in decoding order. In addition, if nnpfa_target_base_flag is 0, the target NNPF may not be a repetition of an NNPFC SEI message including the base NNPF.

[0113] The NNPFA SEI message can activate a primary NNPF with a specific value of nnpfc_id when the primary NNPFDML update is enabled and switches the target NNPF from the updated NNPF to the primary NNPF.

[0114] If the nnpfa_target_base_flag of the NNPFA SEI message is 0, it may be restricted that there is at least one NNPFC SEI message that precedes the NNPFA SEI message in decoding order, has an nnpfc_id equal to the nnpfa_target_id, and has an nnpfc_base_flag value of 0.

[0115] The NNPFA SEI message may include a previous CLVS flag (nnpfa_no_prev_clvs_flag). If the value of nnpfa_no_prev_clvs_flag is 1, this may indicate that the input pictures for NNPF are not generated from the previous CLVS. If the value of nnpfa_no_prev_clvs_flag is 0, this may indicate that the input pictures for NNPF may or may not be generated from the previous CLVS.

[0116] If the current CLVS is spliced ​​from another bitstream next to a previous CLVS and one or more input pictures are selected from one or more previous CLVSs due to the NNPFA SEI message, which may negatively affect the output of the target NNPF, the value of nnpfa_no_prev_clvs_flag may be changed from 0 to 1.

[0117] The NNPFA SEI message may include a fall CLVS flag (nnpfa_no_foll_clvs_flag). If nnpfa_no_foll_clvs_flag is 1, this may indicate that when the NNPFA SEI message persists for the last PU of the CLVS in output order, it is treated as if it persisted for the last PU in output order of the current layer in the bitstream. If the NNPFA SEI message does not persist for the last PU of the CLVS in output order or the value of nnpfa_no_foll_clvs_flag is 0, the value of nnpfa_no_foll_clvs_flag may have no effect.

[0118] nnpfa_no_foll_clvs_flag can be adaptively signaled based on nnpfa_persistence_flag. For example, nnpfa_no_foll_clvs_flag can be signaled based on nnpfa_persistence_flag having a value of 1. nnpfa_no_foll_clvs_flag can be unsignaled based on nnpfa_persistence_flag having a value of 0.

[0119] When the next CLVS is spliced ​​from another bitstream next to the current CLVS, the value of nnpfa_no_foll_clvs_flag can be changed from 0 to 1 for the picture-rate-upsampling NNPF. As a result, the NNPF process can interpolate pictures up to the end of the current CLVS using only the input pictures generated from the current CLVS.

[0120] The NNPFA SEI message may include output entry information (nnpfa_num_output_entries). nnpfa_num_output_entries may specify the number of output flags (nnpfa_output_flag) present in the NNPFA SEI message. The value of nnpfa_num_output_entries may be restricted to fall in the range of 0 to NumInpPicsInOutputTensor. When the value of PictureRateUpsamplingFlag is 0 and nnpfa_num_output_entries is equal to NumInpPicsInOutputTensor, at least one of the output flags present in the NNPFA SEI message may be restricted to have a value of 1.

[0121] The NNPFA SEI message may include an output flag (nnpfa_output_flag). If the value of nnpfa_output_flag is 1, this may indicate that the NNPF-generated picture corresponding to the input picture with the index (InpIdx) is output by the NNPF process activated by the NNPFA SEI message. If the value of nnpfa_output_flag is 0, this may indicate that the NNPF-generated picture corresponding to the input picture with the index (InpIdx) is not output by the NNPF process activated by the NNPFA SEI message.

[0122] If nnpfa_num_output_entries is less than NumInpPicsInOutputTensor, nnpfa_output_flag can be inferred to be 1 for each value in the range from nnpfa_num_output_entries to (NumInpPicsInOutputTensor-1).

[0123] The NNPFA SEI message may include a prompt presence flag (nnpfa_spatial_extrapolation_prompt_present_flag). nnpfa_spatial_extrapolation_prompt_present_flag may indicate whether prompt information (nnpfa_prompt) exists.

[0124] For example, if the value of nnpfa_spatial_extrapolation_prompt_present_flag is 1, this may indicate that nnpfa_prompt exists. In addition, if the value of nnpfa_spatial_extrapolation_prompt_present_flag is 1, this may indicate that zero bit information (nnpfa_alignment_zero_bit_c) may exist. If the value of nnpfa_spatial_extrapolation_prompt_present_flag is 0, this may indicate that nnpfa_prompt does not exist. In addition, if the value of nnpfa_spatial_extrapolation_prompt_present_flag is 0, this may indicate that nnpfa_alignment_zero_bit_c does not exist.

[0125] nnpfa_spatial_extrapolation_prompt_present_flag may indicate whether the NNPFC identified by nnpfa_target_id includes a spatial extrapolation prompt. The value of nnpfa_spatial_extrapolation_prompt_present_flag can be 1 only if a spatial extrapolation prompt is present.

[0126] The NNPFA SEI message may contain zero bit information (nnpfa_alignment_zero_bit_c). nnpfa_alignment_zero_bit_c may be constrained to have a value of 0.

[0127] The NNPFA SEI message may include prompt information (nnpfa_prompt). nnpfa_prompt may specify a text string prompt for generating the content of the spatial extrapolation image region. If the value of nnpfa_spatial_extrapolation_prompt_present_flag is 1, nnpfa_prompt may be restricted to not be a null string.

[0128] At least one of nnpfa_alignment_zero_bit_c or nnpfa_prompt can be adaptively signaled based on nnpfa_spatial_extrapolation_prompt_present_flag.

[0129] For example, nnpfa_alignment_zero_bit_c may be signaled based on nnpfa_spatial_extrapolation_prompt_present_flag having a value of 1. nnpfa_alignment_zero_bit_c may not be signaled based on nnpfa_spatial_extrapolation_prompt_present_flag having a value of 0. nnpfa_prompt may be signaled based on nnpfa_spatial_extrapolation_prompt_present_flag having a value of 1. nnpfa_prompt may not be signaled based on nnpfa_spatial_extrapolation_prompt_present_flag having a value of 0.

[0130] For example, the NNPFA SEI message according to the present disclosure may be structured as shown in Table 2 below.

[0131] nn_post_filter_activation( payloadSize ) {Descriptornnpfa_target_idue(v)nnpfa_cancel_flagu(1)if( !nnpfa_cancel_flag ) {nnpfa_persistence_flagu(1)nnpfa_target_base_flagu(1)if( nnpfa_persistence_flag )nnpfa_no_foll_clvs_flagu(1)nnpfa_num_output_entriesue(v)for( i = 0; i < nnpfa_num_output_entries; i++ )nnpfa_output_flag[ i ]u(1)if( more_data_in_payload( ) ) {nnpfa_spatial_extrapolation_prompt_update_flagu(1)if (nnpfa_spatial_extrapolation_prompt_update_flag) {nnpfa_spatial_extrapolation_prompt_presnt_flagu(1)if(nnpfa_spatial_extrapolation_prompt_presnt_flag){while( !byte_aligned( ) )nnpfa_alignment_zero_bit_cu(1)nnpfa_promptst(v)}}}}}

[0132] According to Table 2, the NNPFA SEI message may further include a prompt update flag (nnpfa_spatial_extrapolation_prompt_update_flag). nnpfa_spatial_extrapolation_prompt_update_flag may be related to updating nnpfa_prompt.

[0133] For example, if the value of nnpfa_spatial_extrapolation_prompt_update_flag is 1, this may indicate that nnpfa_spatial_extrapolation_prompt_presnt_flag and nnpfa_prompt will be updated. On the other hand, if the value of nnpfa_spatial_extrapolation_prompt_update_flag is 0, this may indicate that nnpfa_spatial_extrapolation_prompt_presnt_flag and nnpfa_prompt will remain the same as before.

[0134] If the value of SpatialExtrapolationFlag of the target NNPF is 0, nnpfa_spatial_extrapolation_prompt_update_flag may be restricted to have a value of 0. Here, SpatialExtrapolationFlag may be information derived based on the purpose information (nnpfc_purpose) of the NNPFC SEI message described later.

[0135] In Table 2, syntax elements that overlap with those in Table 1 are the same as those in Table 1, and a detailed explanation is omitted here.

[0136] The NNPFC SEI message can specify a neural network that can be used as a post-processing filter. The use of a specific neural network post-processing filter (NNPF) for a specific picture can be indicated based on the NNPFA SEI message.

[0137] The following variables can be defined for use with the NNPFC SEI message:

[0138] - Input picture width (CroppedWidth) and height (CroppedHeight) in luma sample units

[0139] - Luma sample array (CroppedYPic[idx]) and chroma sample arrays (CroppedCbPic[idx] and CroppedCrPic[idx]) of input pictures with indices (idx) ranging from 0 to (numInputPics-1), which are used as input to NNPF.

[0140] - Bit depth (BitDepthY) for the luma sample array of input pictures

[0141] - Bit depth (BitDepthC) for the chroma sample arrays of input pictures

[0142] - Chroma format identifier (ChromaFormatIdc)

[0143] - If nnpfc_auxiliary_inp_idc is 1, an array of filtering strength control values ​​(StrengthControlVal[idx]) containing real numbers in the range 0 to 1 for input pictures with indices (idx) in the range 0 to (numInputPics-1).

[0144] An input picture with index 0 may correspond to a picture for which the NNPF defined in the NNPFC SEI message is activated by the NNPFA SEI message. An input picture with index i in the range of 1 to (numInputPics-1) may precede an input picture with index (i-1) in the output order.

[0145] The variables SubWidthC and SubHeightC can be derived based on ChromaFormatIdc.

[0146] For example, the NNPFC SEI message according to the present disclosure may be structured as shown in Table 3 below.

[0147] nn_post_filter_characteristics( payloadSize ) {Descriptornnpfc_purposeu(16)nnpfc_idue(v)nnpfc_base_flagu(1)nnpfc_mode_idcue(v)if( nnpfc_mode_idc = = 1 ) {while( !byte_aligned( ) )nnpfc_alignment_zero_bit_au(1)nnpfc_tag_urist(v)nnpfc_urist(v)}nnpfc_property_present_flagu(1)if( nnpfc_property_present_flag ) { / * input and output formatting * / nnpfc_num_input_pics_minus1ue(v)if( nnpfc_num_input_pics_minus1 > 0 ) {for( i = 0; i <= nnpfc_num_input_pics_minus1; i++ )nnpfc_input_pic_filtering_flag[ i ]u(1)nnpfc_absent_input_pic_zero_flagu(1)}if( ChromaUpsamplingFlag )nnpfc_out_sub_c_flagu(1)if( ColourizationFlag )nnpfc_out_colour_format_idcu(2)if( ResolutionResamplingFlag ) {nnpfc_pic_width_num_minus1ue(v)nnpfc_pic_width_denom_minus1ue(v)nnpfc_pic_height_num_minus1ue(v)nnpfc_pic_height_denom_minus1ue(v)}if( PictureRateUpsamplingFlag )for( i = 0; i < nnpfc_num_input_pics_minus1;i++ )nnpfc_interpolated_pics[ i ]ue(v)if( TemporalExtrapolationFlag )nnpfc_extrapolated_pics_minus1ue(v)if( SpatialExtrapolationFlag ) {nnpfc_spatial_extrapolation_left_offsetue(v)nnpfc_spatial_extrapolation_right_offsetue(v)nnpfc_spatial_extrapolation_top_offsetue(v)nnpfc_spatial_extrapolation_bottom_offsetue(v)nnpfc_spatial_extrapolation_prompt_present_flagu(1)if (nnpfc_spatial_extrapolation_prompt_present_flag) {while( !byte_aligned( ) )nnpfc_alignment_zero_bit_cu(1)nnpfc_promptst(v)}}nnpfc_component_last_flagu(1)nnpfc_inp_format_idcue(v)nnpfc_auxiliary_inp_idcue(v)nnpfc_inp_order_idcue(v)... / * ISO / IEC 15938-17 bitstream * / if( nnpfc_mode_idc = = 0 ) {while( !byte_aligned( ) )nnpfc_alignment_zero_bit_bu(1)for( i = 0; more_data_in_payload( ); i++ )nnpfc_payload_byte[ i ]b(8)}};

[0148] According to Table 3, the NNPFC SEI message may include purpose information (nnpfc_purpose). nnpfc_purpose may indicate the purpose of the NNPF defined in Table 4 below. Here, if (nnpfc_purpose & bitMask) is not 0, it may indicate that the NNPF has a purpose associated with the bitMask value in Table 1. If nnpfc_purpose is greater than 0 and (nnpfc_purpose & bitMask) is 0, the purpose associated with the bitMask value is not applicable to the NNPF. If the value of nnpfc_purpose is 0, the NNPF may be used as determined by the application and specified by nnpfc_application_purpose_tag_uri.

[0149] bitMaskInterpretation0x01General visual quality improvement0x02Chroma upsampling (from the 4:2:0 chroma format to the 4:2:2 or 4:4:4 chroma format, or from the 4:2:2 chroma format to the 4:4:4 chroma format)0x04Resolution resampling (increasing or decreasing the width or height)0x08Picture rate upsampling0x10Bit depth upsampling (increasing the luma bit depth or the chroma bit depth)0x20Colourization0x40Temporal extrapolation (ie, generating one or more future pictures)0x80Spatial extrapolation (ie, generating content outside of the spatial area of ​​the input pictures)

[0150] Within a CLVS, all NNPFC SEI messages with a specific nnpfc_id value can be constrained to have the same nnpfc_purpose value. The value of nnpfc_purpose can be constrained to fall in the range 0 to 255.

[0151] The NNPFC SEI message may contain an identifier (nnpfc_id). nnpfc_id may contain an identification number that can be used to identify the NNPF. The value of nnpfc_id may be from 0 to (2 32 -2) may be limited to the range of up to 10 ...

[0152] If the NNPFC SEI message is the first NNPFC SEI message with an nnpfc_id of a particular value within the current CLVS in decoding order, the following may apply:

[0153] - This SEI message specifies the default NNPF.

[0154] - This SEI message relates to the currently decoded picture and all subsequent decoded pictures of the current layer in output order, up to the end of the current CLVS.

[0155] The NNPFC SEI message may include a base flag (nnpfc_base_flag). If the value of nnpfc_base_flag is 1, this may indicate that the NNPFC SEI message specifies the base NNPF. If the value of nnpf_base_flag is 0, this may indicate that the NNPFC SEI message specifies an update to the base NNPF.

[0156] The value of nnpfc_base_flag may be subject to the constraints in Table 5 below.

[0157] - If the NNPFC SEI message is the first NNPFC SEI message in decoding order with an nnpfc_id of a specific value within the current CLVS, the value of nnpfc_base_flag shall be equal to 1. - If the NNPFC SEI message nnpfcB is not the first NNPFC SEI message in decoding order with an nnpfc_id of a specific value within the current CLVS and the value of nnpfc_base_flag is 1, the NNPFC SEI message shall be a repetition of the first NNPFC SEI message nnpfcA with the same nnpfc_id of a specific value within the current CLVS. That is, the payload content of nnpfcB shall be identical to the payload content of nnpfcA.

[0158] If the value of nnpfc_base_flag is 0, the following Table 6 can be applied.

[0159] - This SEI message defines an update to the previous base NNPF in decoding order with the same nnpfc_id. Updates are not cumulative, and each update is applied to the base NNPF. The base NNPF is the NNPF specified by the first NNPFC SEI message in decoding order and has the specific nnpfc_id within the current CLVS. The NNPF defined in this SEI message is obtained by applying the update defined in this SEI message to the base NNPF with the same nnpfc_id. - This SEI message is associated with the current decoded picture and all subsequent decoded pictures of the current layer in output order. However, this SEI message may be associated until the end of the current CLVC or until a given decoded picture is excluded, whichever comes first. Here, the given decoded picture may be the decoded picture that follows the current decoded picture in output order within the current CLVS. A given decoded picture may be associated with a subsequent NNPFC SEI message in decoding order within the current CLVS with an nnpfc_base_flag of 0 and an nnpfc_id of a particular value.

[0160] An NNPFC SEI message may include a mode indicator (nnpfc_mode_idc). A value of nnpfc_mode_idc of 0 indicates that the NNPFC SEI message contains an ISO / IEC 15938-17 bitstream specifying a base NNPF (if nnpfc_base_flag is 1), or that the NNPFC SEI message is an update to a base NNPF with the same nnpfc_id (if nnpfc_base_flag is 0).

[0161] When the value of nnpfc_base_flag is 1, nnpfc_mode_idc with a value of 1 may indicate that the base NNPF associated with the value of nnpfc_id is a neural network identified by a URI pointed to by nnpfc_uri, having a format identified by a tag URI nnpfc_tag_uri.

[0162] When the value of nnpfc_base_flag is 0, nnpfc_mode_idc with a value of 1 can indicate that updates to the base NNPF with the same value of nnpfc_id are defined by the URI pointed to by nnpfc_uri, in a format identified by the nnpfc_tag_uri tag URI.

[0163] The value of nnpfc_mode_idc can be constrained to fall in the range 0 to 1.

[0164] The NNPFC SEI message may contain zero bit information (nnpfc_reserved_zero_bit_a). nnpfc_reserved_zero_bit_a may be restricted to have a value of 0.

[0165] The NNPFC SEI message may include tag URI information (nnpfc_tag_uri). nnpfc_tag_uri may include a tag URI with the syntax and semantics defined in IETF RFC 4151. Here, the tag URI may identify the format and related information for a neural network used as an update to a base NNPF or a base NNPF with an nnpfc_id of the same value specified by nnpfc_uri. If nnpfc_tag_uri is equal to "tag:iso.org,2023:15938-17", this may indicate that the neural network data identified by nnpfc_uri is compliant with ISO / IEC 15938-17.

[0166] The NNPFC SEI message may include URI information (nnpfc_uri). nnpfc_uri may include a URI with the syntax and semantics specified in IETF Internet Standard 66. Here, the URI may identify a neural network used as a base NNPF or as an update to a base NNPF with an nnpfc_id of the same value.

[0167] The NNPFC SEI message may include a property presence flag (nnpfc_property_present_flag). If the value of nnpfc_property_present_flag is 1, this may indicate the presence of syntax elements related to filter purpose, input format, output format, and complexity. On the other hand, if the value of nnpfc_property_present_flag is 0, this may indicate the absence of syntax elements related to filter purpose, input format, output format, and complexity.

[0168] If the value of nnpfc_base_flag is 1, nnpfc_property_present_flag can be restricted to have a value of 1.

[0169] When the value of nnpfc_property_present_flag is 0, the values ​​of all syntax elements that can only be present when the value of nnpfc_property_present_flag is 1 can be inferred from the values ​​of the respective corresponding syntax elements in the NNPFC SEI message that contains the base NNPF for which this SEI message provides updates.

[0170] If the NNPFC SEI message nnpfcCurr is not the first NNPFC SEI message in decoding order with a particular value of nnpfc_id within the current CLVS, is not a repeat of the first NNPFC SEI message with that nnpfc_id (i.e., has the value of nnpfc_base_flag equal to 0), and has the value of nnpfc_property_present_flag equal to 1, the constraints in Table 7 below may apply.

[0171] - The value of nnpfc_purpose in an NNPFC SEI message must be the same as the value of nnpfc_purpose in the first NNPFC SEI message in decoding order that has the corresponding nnpfc_id value in the current CLVS. - In an NNPFC SEI message, the values ​​of the syntax elements from nnpfc_property_present_flag to nnpfc_complexity_info_present_flag in decoding order must be the same as the values ​​of the corresponding syntax elements in the first NNPFC SEI message in decoding order that has the corresponding nnpfc_id value in the current CLVS. - The value of nnpfc_complexity_info_present_flag must be 0. Alternatively, the value of nnpfc_complexity_info_present_flag shall be 1 in the first NNPFC SEI message in decoding order (hereinafter referred to as nnpfcBase) with the corresponding nnpfc_id value in the current CLVS. - nnpfc_parameter_type_idc in nnpfcCurr shall be equal to nnpfc_parameter_type_idc in nnpfcBase. - nnpfc_log2_parameter_bit_length_minus3 in nnpfcCurr shall be less than or equal to nnpfc_log2_parameter_bit_length_minus3 in nnpfcBase. - If the value of nnpfc_num_parameters_idc in nnpfcBase is 0, nnpfc_num_parameters_idc in nnpfcCurr shall be 0. - If the value of nnpfc_num_parameters_idc is greater than 0, nnpfc_num_parameters_idc in nnpfcCurr must be greater than 0 and less than or equal to nnpfc_num_parameters_idc in nnpfcBase.- If nnpfc_num_kmac_operations_idc in nnpfcBase is 0, nnpfc_num_kmac_operations_idc in nnpfcCurr must be 0. - If nnpfc_num_kmac_operations_idc is greater than 0, nnpfc_num_kmac_operations_idc in nnpfcCurr must be greater than 0 and less than or equal to nnpfc_num_kmac_operations_idc in nnpfcBase. - If nnpfc_total_kilobyte_size in nnpfcBase is 0, nnpfc_total_kilobyte_size in nnpfcCurr must be 0. If nnpfc_total_kilobyte_size is greater than 0, nnpfc_num_kmac_operations_idc in nnpfcCurr must be nnpfc_total_kilobyte_size must be greater than 0 and less than or equal to nnpfc_total_kilobyte_size in nnpfcBase.

[0172] The NNPFC SEI message may include input number information (nnpfc_num_input_pics_minus1). The value of nnpfc_num_input_pics_minus1 plus 1 may indicate the number of input pictures used in NNPF. The value of nnpfc_num_input_pics_minus1 may be constrained to fall in the range of 0 to 63. If the value of PictureRateUpsamplingFlag is 1, nnpfc_num_input_pics_minus1 may be constrained to have a value greater than 0.

[0173] The variable (numInputPics) that specifies the number of pictures used as input to NNPF can be derived as in the following mathematical expression 1.

[0174] [Mathematical Formula 1]

[0175] numInputPics = nnpfc_num_input_pics_minus1 + 1

[0176] The NNPFC SEI message may include a picture output flag (nnpfc_input_pic_output_flag). If the value of nnpfc_input_pic_output_flag[i] is 1, this may indicate that NNPF generates a corresponding output picture for the i-th input picture. If the value of nnpfc_input_pic_output_flag[i] is 0, this may indicate that NNPF does not generate a corresponding output picture for the i-th input image.

[0177] If the value of nnpfc_num_input_pics_minus1 is 0, the value of nnpfc_input_pic_output_flag[0] can be inferred to be 1. If the value of PictureRateUpsamplingFlag is 0 and the value of nnpfc_num_input_pics_minus1 is greater than 0, nnpfc_input_pic_output_flag[i] can be constrained to have a value of 1 for at least one i that is greater than or equal to 0 and less than or equal to nnpfc_num_input_pics_minus1.

[0178] The NNPFC SEI message may include a picture zero flag (nnpfc_absent_input_pic_zero_flag). If the value of nnpfc_absent_input_pic_zero_flag is 1, this may indicate that NNPF expects that an input picture that is not present in the bitstream is represented by a sample array with sample values ​​of 0. If the value of nnpfc_absent_input_pic_zero_flag is 0, this may indicate that NNPF expects that an input picture that is not present in the bitstream is represented by the closest input picture in output order in the bitstream.

[0179] The NNPFC SEI message can include a sub-chroma flag (nnpfc_out_sub_c_flag). nnpfc_out_sub_c_flag can specify the values ​​of the variables outSubWidthC and outSubHeightC when chromaUpsamplingFlag is 1.

[0180] When the value of nnpfc_out_sub_c_flag is 1, outSubWidthC and outSubHeightC can each be 1. When the value of nnpfc_out_sub_c_flag is 0, outSubWidthC can be 2 and outSubHeightC can be 1. When the value of ChromaFormatIdc is 2 and nnpfc_out_sub_c_flag is present, nnpfc_out_sub_c_flag can be constrained to have a value of 1.

[0181] nnpfc_out_sub_c_flag can be adaptively signaled based on the chroma upsampling flag (ChromaUpsamplingFlag). ChromaUpsamplingFlag can indicate whether the purpose information indicates that the purpose of NNPF includes chroma upsampling. ChromaUpsamplingFlag can be derived based on the purpose information. For example, ChromaUpsamplingFlag can be derived as shown in the following mathematical expression (2).

[0182] [Equation 2]

[0183] ChromaUpsamplingFlag = ( ( nnpfc_purpose & 0x02 ) > 0 ) ? 1:0

[0184] The NNPFC SEI message can include a color format indicator (nnpfc_out_colour_format_idc). When the colorizationFlag value is 1, nnpfc_out_colour_format_idc can specify the color format of the NNPF output and the corresponding values ​​of outSubWidthC and outSubHeightC.

[0185] If the value of nnpfc_out_colour_format_idc is 1, this can indicate that the color format of the NNPF output is 4:2:0, and outSubWidthC and outSubHeightC are both 2. If the value of nnpfc_out_colour_format_idc is 2, this can indicate that the color format of the NNPF output is 4:2:2, outSubWidthC is 2, and outSubHeightC is 1. If the value of nnpfc_out_colour_format_idc is 3, this can indicate that the color format of the NNPF output is 4:4:4, and outSubWidthC and outSubHeightC are both 1. nnpfc_out_colour_format_idc can be constrained to not have a value of 0.

[0186] If both chromaUpsamplingFlag and colorizationFlag are 0, outSubWidthC and outSubHeightC can be inferred to be the same values ​​as SubWidthC and SubHeightC, respectively.

[0187] nnpfc_out_colour_format_idc can be adaptively signaled based on the colourization flag (ColourizationFlag). ColourizationFlag can indicate whether the purpose information indicates that the NNPF purpose includes colourization. ColourizationFlag can be derived based on the purpose information. For example, ColourizationFlag can be derived as shown in the following mathematical expression (3).

[0188] [Equation 3]

[0189] ColourizationFlag = ( ( nnpfc_purpose & 0x20 ) > 0 ) ? 1:0

[0190] The NNPFC SEI message can include width ratio information (nnpfc_pic_width_num_minus1, nnpfc_pic_width_denom_minus1). The values ​​of nnpfc_pic_width_num_minus1 plus 1 and nnpfc_pic_width_denom_minus1 plus 1 can specify the numerator and denominator, respectively, of the resampling ratio of the NNPF output picture width for CroppedWidth.

[0191] The value of (nnpfc_pic_width_num_minus1 + 1) / (nnpfc_pic_width_denom_minus1 + 1) may be constrained to be in the range of 1 / 16 to 16. If nnpfc_pic_width_num_minus1 and nnpfc_pic_width_denom_minus1 do not exist, their values ​​may both be inferred to be 0.

[0192] The variable nnpfcOutputPicWidth can represent the width of the luma sample array of the picture(s) generated as a result of applying the NNPF identified by nnpfc_id to the input picture(s). nnpfcOutputPicWidth can be derived as shown in the following mathematical expression 4.

[0193] [Equation 4]

[0194] nnpfcOutputPicWidth = Ceil( CroppedWidth * ( nnpfc_pic_width_num_minus1 + 1 ) ÷ ( nnpfc_pic_width_denom_minus1 + 1 ) )

[0195] Here, (nnpfcOutputPicWidth % outSubWidthC) can be constrained to have a value of 0.

[0196] The NNPFC SEI message can include height ratio information (nnpfc_pic_height_num_minus1, nnpfc_pic_height_denom_minus1). The values ​​of nnpfc_pic_height_num_minus1 plus 1 and nnpfc_pic_height_denom_minus1 plus 1 can specify the numerator and denominator, respectively, of the resampling ratio of the NNPF output picture height for CroppedHeight.

[0197] The value of (nnpfc_pic_height_num_minus1 + 1) / (nnpfc_pic_height_denom_minus1 + 1) may be constrained to be in the range of 1 / 16 to 16. If nnpfc_pic_height_num_minus1 and nnpfc_pic_height_denom_minus1 do not exist, their values ​​may both be inferred to be 0.

[0198] The variable nnpfcOutputPicHeight may represent the height of the luma sample array of the picture(s) generated as a result of applying the NNPF identified by nnpfc_id to the input picture(s). nnpfcOutputPicHeight may be derived as in the following mathematical expression 5.

[0199] [Equation 5]

[0200] nnpfcOutputPicHeight = Ceil( CroppedHeight * ( nnpfc_pic_height_num_minus1 + 1 ) ÷ ( nnpfc_pic_height_denom_minus1 + 1 ) )

[0201] Here, (nnpfcOutputPicHeight % outSubHeightC) can be constrained to have a value of 0.

[0202] If nnpfc_pic_width_num_minus1, nnpfc_pic_width_denom_minus1, nnpfc_pic_height_num_minus1, and nnpfc_pic_height_denom_minus1 exist, at least one of the following conditions must be true:

[0203] - The value of nnpfcOutputPicWidth is not equal to CroppedWidth.

[0204] - The value of nnpfcOutputPicHeight is not equal to CroppedHeight.

[0205] The above width ratio information and height ratio information can be adaptively signaled based on the resolution resampling flag (ResolutionResamplingFlag). ResolutionResamplingFlag may indicate whether the purpose information indicates that the purpose of NNPF includes resolution resampling. ResolutionResamplingFlag can be derived based on the purpose information. For example, ResolutionResamplingFlag can be derived as shown in the following mathematical expression (6).

[0206] [Equation 6]

[0207] ResolutionResamplingFlag = ( ( nnpfc_purpose & 0x04 ) > 0 ) ? 1:0

[0208] The NNPFC SEI message may include information on the number of interpolated pictures (nnpfc_interpolated_pics). nnpfc_interpolated_pics[i] may specify the number of interpolated pictures generated by the NNPF between the i-th and (i+1)-th pictures used as inputs of the NNPF. The value of nnpfc_interpolated_pics[i] may be constrained to fall in the range of 0 to 63. The value of nnpfc_interpolated_pics[i] may be constrained to have a value greater than 0 for at least one of the values ​​of i in the range of 0 to (nnpfc_num_input_pics_minus1-1).

[0209] nnpfc_interpolated_pics can be adaptively signaled based on the PictureRateUpsamplingFlag. PictureRateUpsamplingFlag can be about whether the purpose information indicates that the purpose of NNPF includes picture rate upsampling. PictureRateUpsamplingFlag can be derived based on the purpose information. For example, PictureRateUpsamplingFlag can be derived as shown in the following mathematical expression (7).

[0210] [Equation 7]

[0211] PictureRateUpsamplingFlag = ((nnpfc_purpose & 0x08) > 0)? 1:0

[0212] The NNPFC SEI message may include information on the number of extrapolated pictures (nnpfc_extrapolated_pics_minus1). The value of nnpfc_extrapolated_pics_minus1 plus 1 may specify the number of extrapolated pictures generated by NNPF after all input pictures in output order. The value of nnpfc_extrapolated_pics_minus1 may be constrained to fall in the range of 0 to 62.

[0213] A variable (NumInpPicsInOutputTensor) that specifies the number of pictures that exist in the output tensor of NNPF and have a corresponding input picture, a variable (InpIdx[idx]) that specifies the input picture index of the idx-th picture that exists in the output tensor of NNPF and has a corresponding input picture, and a variable (numOutputPics) that specifies the total number of pictures that exist in the output tensor of NNPF can be derived as shown in Table 8 below.

[0214] for( i = 0, numPicsInOutputTensor = 0; i < numInputPics; i++ )if( nnpfc_input_pic_filtering_flag[ i ] ) {InpIdx[ numPicsInOutputTensor ] = inumPicsInOutputTensor++}NumInpPicsInOutputTensor = numPicsInOutputTensorif( PictureRateUpsamplingFlag )for( i = 0; i <= numInputPics - 2; i++ )numPicsInOutputTensor += nnpfc_interpolated_pics[ i ]if( TemporalExtrapolationFlag )numPicsInOutputTensor += nnpfc_extrapolated_pics + 1

[0215] nnpfc_extrapolated_pics_minus1 can be adaptively signaled based on the temporal extrapolation flag (TemporalExtrapolationFlag). TemporalExtrapolationFlag can indicate whether the purpose information indicates that the NNPF's purpose includes temporal extrapolation. TemporalExtrapolationFlag can be derived based on the purpose information. For example, TemporalExtrapolationFlag can be derived as shown in the following mathematical expression (8).

[0216] [Equation 8]

[0217] TemporalExtrapolationFlag = ( ( nnpfc_purpose & 0x40 ) > 0 ) ? 1:0

[0218] The NNPFC SEI message may include extrapolation area information (nnpfc_spatial_extrapolation_left_offset, nnpfc_spatial_extrapolation_right_offset, nnpfc_spatial_extrapolation_top_offset, and nnpfc_spatial_extrapolation_bottom_offset).

[0219] The above extrapolation region information can specify a spatial extrapolation region. A luma sample whose horizontal picture coordinate is (outSubWidthC * nnpfc_spatial_extrapolation_left_offset) to (nnpfcOutputPicWidth - ( outSubWidthC * nnpfc_spatial_extrapolation_right_offset )) and whose vertical picture coordinate is (outSubWidthC * nnpfc_spatial_extrapolation_top_offset) to (nnpfcOutputPicWidth - ( outSubWidthC * nnpfc_spatial_extrapolation_bottom_offset )) can correspond to a spatial region of an input picture. The values ​​of nnpfc_spatial_extrapolation_left_offset, nnpfc_spatial_extrapolation_right_offset, nnpfc_spatial_extrapolation_top_offset, and nnpfc_spatial_extrapolation_bottom_offset may be constrained to be in the range of 0 to 65,536, inclusive. At least one of nnpfc_spatial_extrapolation_left_offset, nnpfc_spatial_extrapolation_right_offset, nnpfc_spatial_extrapolation_top_offset, and nnpfc_spatial_extrapolation_bottom_offset may be constrained to have a value greater than 0.

[0220] The NNPFC SEI message may include a prompt presence flag (nnpfc_spatial_extrapolation_prompt_present_flag). nnpfc_spatial_extrapolation_prompt_present_flag may indicate whether prompt information (nnpfa_prompt) exists.

[0221] For example, if the value of nnpfc_spatial_extrapolation_prompt_present_flag is 1, this may indicate that nnpfa_prompt exists. In addition, if the value of nnpfc_spatial_extrapolation_prompt_present_flag is 1, this may indicate that zero bit information (nnpfc_alignment_zero_bit_c) may exist. If the value of nnpfc_spatial_extrapolation_prompt_present_flag is 0, this may indicate that nnpfc_prompt does not exist. In addition, if the value of nnpfc_spatial_extrapolation_prompt_present_flag is 0, this may indicate that nnpfc_alignment_zero_bit_c does not exist.

[0222] The NNPFA SEI message may contain zero bit information (nnpfc_alignment_zero_bit_c). nnpfc_alignment_zero_bit_c may be constrained to have a value of 0.

[0223] The NNPFA SEI message may include prompt information (nnpfc_prompt). nnpfc_prompt may specify a text string prompt for generating the content of the spatial extrapolation image region. If the value of nnpfc_spatial_extrapolation_prompt_present_flag is 1, nnpfc_prompt may be restricted to not be a null string.

[0224] When NNPFC is activated by NNPFA, based on nnpfa_spatial_extrapolation_prompt_update_flag defined in NNPFA being 1 (true), nnpfc_spatial_extrapolation_prompt_present_flag and nnpfc_prompt defined in NNPFC can be replaced with nnpfa_spatial_extrapolation_prompt_present_flag and nnpfa_prompt defined in NNPFA, respectively.

[0225] At least one of nnpfc_alignment_zero_bit_c or nnpfc_prompt can be adaptively signaled based on nnpfc_spatial_extrapolation_prompt_present_flag.

[0226] For example, nnpfc_alignment_zero_bit_c may be signaled based on nnpfc_spatial_extrapolation_prompt_present_flag having a value of 1. nnpfc_alignment_zero_bit_c may not be signaled based on nnpfc_spatial_extrapolation_prompt_present_flag having a value of 0. nnpfc_prompt may be signaled based on nnpfc_spatial_extrapolation_prompt_present_flag having a value of 1. nnpfc_prompt may not be signaled based on nnpfc_spatial_extrapolation_prompt_present_flag having a value of 0.

[0227] At least one of nnpfc_alignment_zero_bit_c or nnpfc_prompt can be adaptively signaled based on the spatial extrapolation flag (SpatialExtrapolationFlag).

[0228] For example, nnpfc_alignment_zero_bit_c may be signaled based on the value of SpatialExtrapolationFlag being 1. nnpfc_alignment_zero_bit_c may not be signaled based on the value of SpatialExtrapolationFlag being 0. nnpfc_prompt may be signaled based on the value of SpatialExtrapolationFlag being 1. nnpfc_prompt may not be signaled based on the value of SpatialExtrapolationFlag being 0.

[0229] At least one of nnpfc_alignment_zero_bit_c or nnpfc_prompt may not be signaled based on the value of nnpfc_spatial_extrapolation_prompt_present_flag being 0, even if the value of SpatialExtrapolationFlag is 1, and may be signaled based on the value of nnpfc_spatial_extrapolation_prompt_present_flag being 1.

[0230] The above SpatialExtrapolationFlag may indicate whether the purpose information indicates that the NNPF's purpose includes spatial extrapolation. SpatialExtrapolationFlag may be derived based on the purpose information. For example, SpatialExtrapolationFlag may be derived as shown in the following mathematical expression (9).

[0231] [Equation 9]

[0232] SpatialExtrapolationFlag = ( ( nnpfc_purpose & 0x80 ) > 0 ) ? 1:0

[0233] The NNPFC SEI message may include a last flag (nnpfc_component_last_flag). If the value of nnpfc_component_last_flag is 1, it may indicate that the last dimension of the input tensor for NNPF (inputTensor) and the output tensor generated by NNPF (outputTensor) is used for the current channel. If the value of nnpfc_component_last_flag is 0, it may indicate that the third dimension of the input tensor for NNPF (inputTensor) and the output tensor generated by NNPF (outputTensor) is used for the current channel.

[0234] The NNPFC SEI message may include an input format indicator (nnpfc_inp_format_idc). nnpfc_inp_format_idc may indicate how to convert the sample values ​​of the input picture into the input values ​​of the NNPF.

[0235] When nnpfc_inp_format_idc is 0, the input value of NNPF is a real number, and the functions InpY() and InpC() can be specified as shown in Table 9 below.

[0236] InpY(x) = x ÷ ( ( 1 << BitDepthY ) - 1 )InpC(x) = x ÷ ( ( 1 << BitDepthC ) - 1 )

[0237] When nnpfc_inp_format_idc is 1, the input value to NNPF is an unsigned integer, and the functions InpY( ) and InpC( ) can be specified as shown in Table 10 below.

[0238] shiftY = BitDepthY - inpTensorBitDepthYif( inpTensorBitDepthY >= BitDepthY)InpY( x ) = x << ( inpTensorBitDepthY - BitDepthY )elseInpY( x ) = Clip3(0, ( 1 << inpTensorBitDepthY ) - 1, ( ) >> shiftY )shiftC = BitDepthC - inpTensorBitDepthCif( inpTensorBitDepthC >= BitDepthC )InpC( x ) = x << ( inpTensorBitDepthC - BitDepthC )elseInpC( x ) = Clip3(0, ( 1 << inpTensorBitDepthC ) - 1, ( x + ( 1 << ( shiftC - 1 ) ) ) >> shiftC )

[0239] The variable inpTensorBitDepthY can be derived from nnpfc_inp_tensor_luma_bitdepth_minus8, which will be described later. The variable inpTensorBitDepthC can be derived from nnpfc_inp_tensor_chroma_bitdepth_minus8, which will be described later.

[0240] The NNPFC SEI message may include an auxiliary input indicator (nnpfc_auxiliary_inp_idc). If the value of nnpfc_auxiliary_inp_idc is greater than 0, it may indicate that auxiliary input data exists in the input tensor of the NNPF. If the value of nnpfc_auxiliary_inp_idc is 0, it may indicate that auxiliary input data does not exist in the input tensor. If the value of nnpfc_auxiliary_inp_idc is 1, it may indicate that the auxiliary input data is derived based on a predefined formula. The value of nnpfc_auxiliary_inp_idc may be constrained to fall in the range of 0 to 1.

[0241] The NNPFC SEI message may include an alignment method indicator (nnpfc_inp_order_idc). nnpfc_inp_order_idc may indicate how to align the sample array of the input picture to form the input tensor for NNPF. The value of nnpfc_inp_order_idc may be constrained to fall in the range of 0 to 3.

[0242] If ChromaFormatIdc is not 1, nnpfc_inp_order_idc may be constrained to not have a value of 3. If ChromaFormatIdc is 0, nnpfc_inp_order_idc may be constrained to not have a value of 0. If chromaUpsamplingFlag is 1, nnpfc_inp_order_idc may be constrained to not have a value of 0.

[0243] The following Table 11 describes the values ​​of nnpfc_inp_order_idc. Fig. 5 illustrates an example of deriving four luma channels (right) from the luma component (left) when the value of nnpfc_inp_order_idc is 3.

[0244] nnpfc_inp_order_idcDescription0 If nnpfc_auxiliary_inp_idc is 0, there is one luma matrix in the input tensor for each input picture, and the number of channels is 1. Otherwise, if nnpfc_auxiliary_inp_idc is 1, there is one luma matrix and one auxiliary input matrix, and the number of channels is 2.1 If nnpfc_auxiliary_inp_idc is 0, there are two chroma matrices in the input tensor, and the number of channels is 2. Otherwise, if nnpfc_auxiliary_inp_idc is 1, there are two chroma matrices and one auxiliary input matrix, and the number of channels is 3.2 If nnpfc_auxiliary_inp_idc is 0, there is one luma matrix and two chroma matrices in the input tensor, and the number of channels is 3. Otherwise, if nnpfc_auxiliary_inp_idc is 1, there are one luma matrix, two chroma matrices, and one auxiliary input matrix, and the number of channels is 4.3 If nnpfc_auxiliary_inp_idc is 0, there are four luma matrices and two chroma matrices in the input tensor, and the number of channels is 6. Otherwise, if nnpfc_auxiliary_inp_idc is 1, there are four luma matrices, two chroma matrices, and one auxiliary input matrix in the input tensor, and the number of channels is 7. The luma channels are derived in an interleaved manner as shown in Fig. 5. nnpfc_inp_order_idc can only be used when the input chroma format is 4:2:0. 4..255 Reserved

[0245] The NNPFU SEI message can update the properties of a target NNPF. Here, the target NNPF can be identified based on a target identifier (nnpfu_target_id) for post-processing filtering of a picture.

[0246] For example, the NNPFU SEI message according to the present disclosure may be configured as shown in Table 12 below.

[0247] nn_post_filter_update( payloadSize ) {Descriptornnpfu_target_idue(v)nnpfu_spatial_extrapolation_prompt_presnt_flagu(1)if(nnpfu_spatial_extrapolation_prompt_presnt_flag){while( !byte_aligned( ) )nnpfu_alignment_zero_bit_cu(1)nnpfu_promptst(v)}}

[0248] The NNPFU SEI message may include a target identifier (nnpfu_target_id). The nnpfu_target_id may indicate the nnpfc_id of a target NNPF. Here, the target NNPF may be specified based on one or more NNPFC SEI messages related to the current picture. The target NNPF may have the same nnpfc_id as the nnpfu_target_id. The value of the nnpfu_target_id may be from 0 to (2 32 -2) may be limited to the range of up to 10 ...

[0249] The NNPFU SEI message may include a prompt presence flag (nnpfu_spatial_extrapolation_prompt_present_flag). nnpfu_spatial_extrapolation_prompt_present_flag may indicate whether prompt information (nnpfu_prompt) exists.

[0250] For example, if the value of nnpfu_spatial_extrapolation_prompt_present_flag is 1, this may indicate that nnpfu_prompt exists. In addition, if the value of nnpfu_spatial_extrapolation_prompt_present_flag is 1, this may indicate that zero bit information (nnpfu_alignment_zero_bit_c) may exist. If the value of nnpfu_spatial_extrapolation_prompt_present_flag is 0, this may indicate that nnpfu_prompt does not exist. In addition, if the value of nnpfu_spatial_extrapolation_prompt_present_flag is 0, this may indicate that nnpfu_alignment_zero_bit_c does not exist.

[0251] nnpfu_spatial_extrapolation_prompt_present_flag may indicate whether the NNPFC identified by nnpfa_target_id includes a spatial extrapolation prompt. The value of nnpfu_spatial_extrapolation_prompt_present_flag can be 1 only if a spatial extrapolation prompt is present.

[0252] The NNPFU SEI message may contain zero bit information (nnpfu_alignment_zero_bit_c). nnpfu_alignment_zero_bit_c may be constrained to have a value of 0.

[0253] The NNPFU SEI message may include prompt information (nnpfu_prompt). nnpfu_prompt may specify a text string prompt for generating the content of the spatial extrapolation image region. If the value of nnpfu_spatial_extrapolation_prompt_present_flag is 1, nnpfu_prompt may be restricted to not be a null string.

[0254] At least one of nnpfu_alignment_zero_bit_c or nnpfu_prompt can be adaptively signaled based on nnpfu_spatial_extrapolation_prompt_present_flag.

[0255] For example, nnpfu_alignment_zero_bit_c may be signaled based on nnpfu_spatial_extrapolation_prompt_present_flag having a value of 1. nnpfu_alignment_zero_bit_c may not be signaled based on nnpfu_spatial_extrapolation_prompt_present_flag having a value of 0. nnpfu_prompt may be signaled based on nnpfu_spatial_extrapolation_prompt_present_flag having a value of 1. nnpfu_prompt may not be signaled based on nnpfu_spatial_extrapolation_prompt_present_flag having a value of 0.

[0256] NNPF related information according to the present disclosure may be configured in an SEI message of a bitstream. The SEI message may be included in a network abstraction layer (NAL) unit of the bitstream. However, the present disclosure is not limited thereto. For example, the NNPF related information according to the present disclosure may be configured in a high level syntax of the bitstream. Here, the high level syntax may be at least one of a sequence parameter set (SPS), a picture parameter set (PPS), a picture header (PH), or a slice header (SH). Alternatively, the NNPF related information according to the present disclosure may be defined as a separate NAL unit type within the bitstream.

[0257] FIG. 6 illustrates a schematic configuration of a decoding device (300) that performs a method for restoring a video picture according to the present disclosure.

[0258] Referring to FIG. 6, the decoding device (300) may include a receiving unit (600), a video information extraction unit (610), and a video restoration unit (620).

[0259] The receiving unit (600) can receive a bitstream including an encoded video picture.

[0260] The video information extraction unit (610) can extract video information about an encoded video picture from a bitstream. In addition, the video information extraction unit (610) can extract NNPF-related information from the bitstream, as described with reference to FIG. 4.

[0261] The video restoration unit (620) can restore an encoded video picture based on the extracted video information.

[0262] FIG. 7 illustrates a method for generating a bitstream performed in an encoding device (200) according to the present disclosure.

[0263] A video picture to be encoded can be received (S700).

[0264] The received video picture can be encoded to generate video information about the video picture (S710).

[0265] A bitstream including video information about a video picture can be generated (S720).

[0266] Additionally, NNPF-related information applied to the bitstream can be generated, as described with reference to Fig. 4. The generated NNPF-related information can be included in the bitstream.

[0267] FIG. 8 illustrates a schematic configuration of an encoding device (200) that performs a method for generating a bitstream according to the present disclosure.

[0268] Referring to FIG. 8, the encoding device (200) may include a receiving unit (800), a video compression unit (810), and a bitstream generation unit (820).

[0269] The receiving unit (800) can receive one or more video pictures to be encoded.

[0270] The video compression unit (810) can encode one or more received video pictures to generate video information about the video pictures. The video compression unit (810) can generate NNPF-related information applied to the bitstream.

[0271] The bitstream generation unit (820) can generate a bitstream including the video information. The bitstream generation unit (820) can generate a bitstream further including the generated NNPF-related information.

[0272] In the embodiments described above, the methods are described based on a flowchart as a series of steps or blocks. However, the embodiments are not limited to the order of the steps, and some steps may occur in a different order or simultaneously with other steps described above. Furthermore, those skilled in the art will understand that the steps depicted in the flowchart are not exclusive, and other steps may be included, or one or more steps in the flowchart may be deleted without affecting the scope of the embodiments of this document.

[0273] The method according to the embodiments of the present document described above can be implemented in the form of software, and the encoding device and / or decoding device according to the present document can be included in a device that performs image processing, such as a TV, a computer, a smartphone, a set-top box, a display device, etc.

[0274] When the embodiments in this document are implemented as software, the above-described method can be implemented as a module (process, function, etc.) that performs the above-described function. The module can be stored in memory and executed by a processor. The memory can be internal or external to the processor and can be connected to the processor by various well-known means. The processor can include an application-specific integrated circuit (ASIC), another chipset, logic circuit, and / or data processing device. The memory can include a read-only memory (ROM), a random access memory (RAM), flash memory, a memory card, a storage medium, and / or other storage devices. That is, the embodiments described in this document can be implemented and performed on a processor, a microprocessor, a controller, or a chip. For example, the functional units illustrated in each drawing can be implemented and performed on a computer, a processor, a microprocessor, a controller, or a chip. In this case, information for implementation (e.g., information on instructions) or an algorithm can be stored on a digital storage medium.

[0275] In addition, the decoding device and encoding device to which the embodiment(s) of the present specification are applied may be included in a multimedia broadcasting transmitting and receiving device, a mobile communication terminal, a home cinema video device, a digital cinema video device, a surveillance camera, a video conversation device, a real-time communication device such as a video communication, a mobile streaming device, a storage medium, a camcorder, a video-on-demand (VoD) service providing device, an OTT (Over the top video) device, an Internet streaming service providing device, a three-dimensional (3D) video device, a VR (virtual reality) device, an AR (argumente reality) device, a video phone video device, a transportation terminal (ex. a vehicle (including an autonomous vehicle) terminal, an airplane terminal, a ship terminal, etc.), and a medical video device, and may be used to process a video signal or a data signal. For example, the OTT (Over the top video) device may include a game console, a Blu-ray player, an Internet-connected TV, a home theater system, a smartphone, a tablet PC, a DVR (Digital Video Recorder), etc.

[0276] In addition, the processing method to which the embodiment(s) of the present specification are applied can be produced in the form of a computer-executable program and can be stored in a computer-readable recording medium. Multimedia data having a data structure according to the embodiment(s) of the present specification can also be stored in a computer-readable recording medium. The computer-readable recording medium includes all types of storage devices and distributed storage devices in which computer-readable data is stored. The computer-readable recording medium can include, for example, a Blu-ray disc (BD), a universal serial bus (USB), a ROM, a PROM, an EPROM, an EEPROM, a RAM, a CD-ROM, a magnetic tape, a floppy disk, and an optical data storage device. In addition, the computer-readable recording medium includes a medium implemented in the form of a carrier wave (e.g., transmission via the Internet). In addition, a bitstream generated by an encoding method can be stored in a computer-readable recording medium or transmitted via a wired or wireless communication network.

[0277] Additionally, the embodiments of the present disclosure may be implemented as a computer program product by program code, and the program code may be executed on a computer by the embodiments of the present disclosure. The program code may be stored on a computer-readable carrier.

[0278] FIG. 9 illustrates an example of a content streaming system to which embodiments of the present disclosure can be applied.

[0279] Referring to FIG. 9, a content streaming system to which the embodiment(s) of the present specification are applied may largely include an encoding server, a streaming server, a web server, a media storage, a user device, and a multimedia input device.

[0280] The encoding server compresses content input from multimedia input devices such as smartphones, cameras, and camcorders into digital data, generates a bitstream, and transmits it to the streaming server. Alternatively, if multimedia input devices such as smartphones, cameras, and camcorders directly generate bitstreams, the encoding server may be omitted.

[0281] The above bitstream can be generated by an encoding method or a bitstream generation method to which the embodiment(s) of the present specification are applied, and the streaming server can temporarily store the bitstream during the process of transmitting or receiving the bitstream.

[0282] The streaming server transmits multimedia data to a user device based on a user request via a web server, and the web server acts as an intermediary to inform the user of available services. When a user requests a desired service from the web server, the web server transmits the request to the streaming server, and the streaming server transmits the multimedia data to the user. At this time, the content streaming system may include a separate control server, in which case the control server controls commands / responses between each device within the content streaming system.

[0283] The streaming server can receive content from a media repository and / or an encoding server. For example, when receiving content from the encoding server, the content can be received in real time. In this case, to provide a smooth streaming service, the streaming server can store the bitstream for a certain period of time.

[0284] Examples of the user devices may include mobile phones, smart phones, laptop computers, digital broadcasting terminals, personal digital assistants (PDAs), portable multimedia players (PMPs), navigation devices, slate PCs, tablet PCs, ultrabooks, wearable devices (e.g., smartwatches, smart glasses, HMDs), digital TVs, desktop computers, digital signage, etc.

[0285] Each server within the above content streaming system can be operated as a distributed server, in which case data received from each server can be processed in a distributed manner.

[0286] The claims set forth in this specification may be combined in various ways. For example, the technical features of the method claims of this specification may be combined and implemented as a device, and the technical features of the device claims of this specification may be combined and implemented as a method. Furthermore, the technical features of the method claims and the technical features of the device claims of this specification may be combined and implemented as a device, and the technical features of the method claims and the technical features of the device claims of this specification may be combined and implemented as a method.

Claims

1. A step of receiving a bitstream including an encoded video picture; and A step of restoring an encoded video picture included in the above bitstream, The above bitstream is configured to include information related to a neural network post-processing filter (NNPF), The above NNPF-related information includes at least one of a neural-network post-filter activation (NNPFA) SEI (supplemental enhancement information) message, a neural-network post-filter characteristics (NNPFC) SEI message, or a neural-network post-filter update (NNPFU) SEI message, A method in which the above NNPF related information is obtained from a NAL (network abstraction layer) unit of the bitstream.

2. In paragraph 1, The above NNPFA SEI message contains prompt information, The above prompt information is a method for specifying a text string prompt for generating the contents of a spatial extrapolation image area.

3. In paragraph 2, The above NNPFA SEI message further includes a prompt presence flag, The above prompt presence flag with a value of 1 indicates that the above prompt information exists and that zero bit information may exist, A method wherein the above prompt presence flag having a value of 0 indicates that the above prompt information and the above zero bit information do not exist.

4. In paragraph 3, A method wherein the above prompt information is adaptively signaled based on the prompt presence flag.

5. In paragraph 2, The above NNPFA SEI message includes a prompt update flag, The above prompt update flag is related to whether the above prompt information is updated.

6. In paragraph 5, A method wherein the above prompt information is adaptively signaled based on the above prompt update flag.

7. In paragraph 1, The above NNPFC SEI message contains prompt information, The above prompt information is a method for specifying a text string prompt for generating the contents of a spatial extrapolation image area.

8. In paragraph 7, The above NNPFC SEI message further includes a prompt presence flag, The above prompt presence flag with a value of 1 indicates that the above prompt information exists and that zero bit information may exist, A method wherein the above prompt presence flag having a value of 0 indicates that the above prompt information and the above zero bit information do not exist.

9. In paragraph 8, A method wherein the above prompt information is adaptively signaled based on the prompt presence flag.

10. In paragraph 1, The above NNPFU SEI message contains prompt information, The above prompt information is a method for specifying a text string prompt for generating the contents of a spatial extrapolation image area.

11. In paragraph 10, The above NNPFU SEI message further includes a prompt presence flag, The above prompt presence flag with a value of 1 indicates that the above prompt information exists and that zero bit information may exist, A method wherein the above prompt presence flag having a value of 0 indicates that the above prompt information and the above zero bit information do not exist.

12. In paragraph 11, A method wherein the above prompt information is adaptively signaled based on the prompt presence flag.

13. A step of receiving a video picture to be encoded; A step of encoding the received video picture to generate video information about the video picture; A step of generating information related to a neural network post-filter (NNPF); and Including a step of generating a bitstream including the above video information and the above NNPF related information, The above NNPF-related information includes at least one of a neural-network post-filter activation (NNPFA) SEI (supplemental enhancement information) message, a neural-network post-filter characteristics (NNPFC) SEI message, or a neural-network post-filter update (NNPFU) SEI message, A method in which the above NNPF related information is encoded in a NAL (network abstraction layer) unit of the bitstream.

14. A computer-readable storage medium storing a bitstream generated by the method according to Article 13.

15. A step of generating a bitstream; wherein the bitstream is generated based on a step of receiving a video picture to be encoded, a step of encoding the received video picture to generate video information about the video picture, and a step of generating information related to a neural network post-filter (NNPF), and Including a step of transmitting data including the above bitstream, The above NNPF-related information includes at least one of a neural-network post-filter activation (NNPFA) SEI (supplemental enhancement information) message, a neural-network post-filter characteristics (NNPFC) SEI message, or a neural-network post-filter update (NNPFU) SEI message, A method in which the above NNPF related information is encoded in a NAL (network abstraction layer) unit of the bitstream.

Images