Image encoding / decoding methods and devices, and recording media for storing bitstreams.

Abstract

The image decoding method and apparatus according to this disclosure are capable of receiving a bitstream including encoded video images and reconstructing the encoded video images included in the bitstream. The bitstream can include information related to the Neural Network Post-Filter (NNPF). The NNPF-related information can include at least one of NNPFA SEI messages, NNPFC SEI messages, and NNPFU SEI messages.

Description

Technical Field

[0001] This disclosure relates to image encoding / decoding methods and apparatus, as well as recording media for storing bit streams. Background Technology

[0002] Recently, the demand for high-resolution and high-quality images, such as HD (high-definition) and UHD (ultra-high-definition) images, has been increasing in various application fields, and therefore, efficient image compression technology is being discussed.

[0003] There are various techniques, such as inter-frame prediction, which uses video compression technology to predict the pixel values ​​included in the current image from images before or after the current image; intra-frame prediction, which uses pixel information in the current image to predict the pixel values ​​included in the current image; and entropy coding technology, which assigns short symbols to values ​​that occur frequently and long symbols to values ​​that occur infrequently. These image compression techniques can be used to effectively compress image data and transmit or store it. Summary of the Invention

[0004] Technical issues

[0005] This disclosure provides a method and apparatus for configuring NNPF-related information.

[0006] This disclosure provides a method and apparatus for transmitting NNPF-related information using signals.

[0007] Technical solution

[0008] The image decoding method and apparatus according to this disclosure can receive a bitstream including encoded video images and reconstruct the encoded video images included in the bitstream. The bitstream can be configured to include information related to a neural network post-filter (NNPF).

[0009] In the image decoding method and apparatus according to the present disclosure, NNPF-related information may include at least one of the following: Neural Network Post-Filter Activation (NNPFA) Supplemental Enhancement Information (SEI) message, Neural Network Post-Filter Characteristics (NNPFC) SEI message, or Neural Network Post-Filter Update (NNPFU) SEI message.

[0010] In the image decoding method and apparatus according to the present disclosure, NNPF-related information can be obtained from the Network Abstraction Layer (NAL) unit of the bitstream.

[0011] In the image decoding method and apparatus according to this disclosure, the NNPFA SEI message may include prompt information. This prompt information may specify a text string prompting the content used to generate the spatially extrapolated image region.

[0012] In the image decoding method and apparatus according to this disclosure, the NNPFA SEI message may further include a presence flag. A presence flag with a value of 1 may indicate the presence of a presence message and may contain zero-bit information. A presence flag with a value of 0 may indicate a presence message and the absence of zero-bit information.

[0013] In the image decoding method and apparatus according to the present disclosure, a prompting message can be adaptively sent by signal based on a prompting presence flag.

[0014] In the image decoding method and apparatus according to this disclosure, the NNPFA SEI message may include a prompt update flag. The prompt update flag may be related to whether the prompt information has been updated.

[0015] In the image decoding method and apparatus according to the present disclosure, prompt information can be adaptively sent with signals based on the prompt update flag.

[0016] In the image decoding method and apparatus according to this disclosure, the NNPFC SEI message may include prompt information. This prompt information may specify a text string prompt for generating the content of a spatially extrapolated image region.

[0017] In the image decoding method and apparatus according to this disclosure, the NNPFC SEI message may further include a presence flag. A presence flag with a value of 1 may indicate the presence of a presence message and may also indicate the presence of zero-bit information. A presence flag with a value of 0 may indicate the absence of both a presence message and zero-bit information.

[0018] In the image decoding method and apparatus according to the present disclosure, a prompting message can be adaptively sent by signal based on a prompting presence flag.

[0019] In the image decoding method and apparatus according to this disclosure, the NNPFU SEI message may include prompt information. This prompt information may specify a text string prompt for generating the content of a spatially extrapolated image region.

[0020] In the image decoding method and apparatus according to this disclosure, the NNFPFU SEI message may further include a presence flag. A presence flag with a value of 1 may indicate the presence of a presence message and may also indicate the presence of zero-bit information. A presence flag with a value of 0 may indicate the absence of both a presence message and zero-bit information.

[0021] In the image decoding method and apparatus according to the present disclosure, a prompting message can be adaptively sent by signal based on a prompting presence flag.

[0022] The image encoding method and apparatus according to this disclosure can receive a video image to be encoded, encode the received video image to generate video information related to the video image, generate neural network post-filter (NNPF) related information, and generate a bitstream including the video information and the NNPF related information. The NNPF related information may include at least one of a neural network post-filter activation (NNPFA) supplementary enhancement information (SEI) message, a neural network post-filter feature (NNPFC) SEI message, or a neural network post-filter update (NNPFU) SEI message.

[0023] A computer-readable digital storage medium is provided that stores encoded video / image information, thereby causing an image decoding method to be performed by a decoding apparatus according to the present disclosure.

[0024] A computer-readable digital storage medium is provided according to the present disclosure for storing video / image information generated according to an image encoding method.

[0025] A method and apparatus for transmitting video / image information generated according to an image encoding method are provided according to the present disclosure.

[0026] Beneficial effects

[0027] By configuring prompt-related information, such as NNPFA information, at specific locations as shown in this disclosure, the problem of having to repeatedly send prompt-related information via signaling whenever the prompt information is updated can be solved. Attached Figure Description

[0028] Figure 1 A video / image encoding system according to this disclosure is shown.

[0029] Figure 2 A schematic block diagram illustrating an encoding apparatus to which embodiments of the present disclosure are applicable and to perform encoding of video / image signals is shown.

[0030] Figure 3 A schematic block diagram of a decoding apparatus to which embodiments of the present disclosure are applicable and to perform decoding of video / image signals is shown.

[0031] Figure 4 The illustration shows a method for reconstructing video images performed by a decoding device 300 according to the present disclosure.

[0032] Figure 5 The illustration illustrates a method for deriving a luminance channel according to an embodiment of the present disclosure.

[0033] Figure 6 The illustration shows a schematic configuration of a decoding apparatus 300 that performs a method for reconstructing video images according to the present disclosure.

[0034] Figure 7 The figure illustrates a method for generating a bitstream performed by an encoding device 200 according to the present disclosure.

[0035] Figure 8 The figure shows a schematic configuration of an encoding apparatus 200 for performing a method for generating a bitstream according to the present disclosure.

[0036] Figure 9 Examples of content streaming systems to which embodiments of this disclosure can be applied are shown. Detailed Implementation

[0037] Because this disclosure can be modified in various ways and has several embodiments, specific embodiments will be illustrated in the accompanying drawings and described in detail in the detailed description. However, this disclosure is not intended to be limited to the specific embodiments and should be understood to include all variations, equivalents, and substitutions included within the spirit and scope of this disclosure. Similar reference numerals are used for similar components in the description of each drawing.

[0038] Terms such as "first," "second," etc., may be used to describe various components, but components should not be limited by these terms. These terms are used only to distinguish one component from other components. For example, without departing from the scope of this disclosure, a first component may be referred to as a second component, and similarly, a second component may be referred to as a first component. Terms and / or combinations of any one or more related statement items are included.

[0039] When a component is described as "connected" or "linked" to another component, it should be understood that it can be directly connected or linked to another component, but there may also be another component in between. On the other hand, when a component is described as "directly connected" or "directly linked" to another component, it should be understood that there is no other component in between.

[0040] The terminology used in this application is for describing particular embodiments only and is not intended to limit this disclosure. Singular expressions include plural expressions unless the context clearly indicates otherwise. In this application, it should be understood that terms such as “comprising” or “having” are intended to designate the presence of features, numbers, steps, operations, components, portions, or combinations thereof described in the specification, but do not preclude the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, portions, or combinations thereof.

[0041] This disclosure relates to video / image coding. For example, the methods / exercises disclosed herein can be applied to methods disclosed in the Essential Video Coding (VVC) standard. Additionally, the methods / exercises disclosed herein can be applied to methods disclosed in the Basic Video Coding (EVC) standard, the AOMedia Video 1 (AV1) standard, the Audio Video Coding Standard 2 (AVS2), or next-generation video / image coding standards (e.g., H.267 or H.268).

[0042] This specification presents various embodiments of video / image encoding, and unless otherwise stated, these embodiments may be combined with each other to perform the task.

[0043] Here, "video" can refer to a collection of images over time. "Image" generally refers to a unit representing an image within a specific time period, and a tile is a unit that forms part of an image during encoding. A tile can include at least one coding tree unit (CTU). An image can consist of at least one tile. A tile is a rectangular area consisting of multiple CTUs within a specific tile column and a specific tile row of an image. A tile column is a rectangular area of ​​CTUs with the same height as the image and a width assigned by the syntax requirements of the image parameter set. A tile row is a rectangular area of ​​CTUs with the same height assigned by the image parameter set and a width equal to the width of the image. CTUs within a tile can be arranged consecutively according to a CTU raster scan, and tiles within an image can be arranged consecutively according to a tile raster scan. A tile can include an integer number of complete tiles or an integer number of consecutive complete CTU rows that can be exclusively included within a single NAL unit of the image. Simultaneously, an image can be divided into at least two sub-images. A sub-image can be a rectangular area of ​​at least one tile within an image.

[0044] A pixel, cell, or pixel unit can refer to the smallest unit that makes up a picture (or image). Additionally, "sample" can be used as the term corresponding to a pixel. A sample can typically represent a pixel or pixel value, and can represent only the pixel / pixel value of the luminance component, or only the pixel / pixel value of the chrominance component.

[0045] A unit can represent a basic unit of image processing. A unit may include a specific region of an image and at least one of the information associated with that region. A unit may include a luminance block and two chrominance (e.g., cb, cr) blocks. In some cases, units may be used interchangeably with terms such as block or region. In general, an MxN block may include a set (or array) of transform coefficients or samples (or sample arrays) consisting of M columns and N rows.

[0046] Here, "A or B" can refer to "A only", "B only", or "both A and B". In other words, "A or B" can be interpreted as "A and / or B". For example, "A, B or C" can refer to "A only", "B only", "C only", or "any combination of A, B and C".

[0047] The forward slash ( / ) or comma used in this article can refer to "and / or". For example, "A / B" can refer to "A and / or B". Therefore, "A / B" can refer to "A only", "B only", or "both A and B". For example, "A, B, C" can refer to "A, B, or C".

[0048] Here, "at least one of A and B" can refer to "only A", "only B" or "both A and B". Furthermore, expressions such as "at least one of A or B" or "at least one of A and / or B" can be interpreted in the same way as "at least one of A and B".

[0049] Additionally, here, "at least one of A, B, and C" can refer to "A only", "B only", "C only" or "any combination of A, B, and C". Furthermore, "at least one of A, B, or C" or "at least one of A, B, and / or C" can refer to "at least one of A, B, and C".

[0050] Additionally, the parentheses used in this document can refer to "for example". Specifically, when the indication is "prediction (intra-frame prediction)", "intra-frame prediction" can be cited as an example of "prediction". In other words, "prediction" here is not limited to "intra-frame prediction", and "intra-frame prediction" can be cited as an example of "prediction". Furthermore, even when the indication is "prediction (i.e., intra-frame prediction)", "intra-frame prediction" can be cited as an example of "prediction".

[0051] Here, a technical feature described individually in a single figure can be implemented individually or simultaneously.

[0052] Figure 1 A video / image encoding system according to this disclosure is shown.

[0053] refer to Figure 1A video / image encoding system may include a first device (source device) and a second device (receiving device).

[0054] A source device can transmit encoded video / image information or data to a receiving device in the form of a file or stream via digital storage media or a network. The source device may include a video source, an encoding apparatus, and a transmitting unit. The receiving device may include a receiving unit, a decoding apparatus, and a renderer. The encoding apparatus may be referred to as a video / image encoding apparatus, and the decoding apparatus may be referred to as a video / image decoding apparatus. A transmitter may be included in the encoding apparatus. A receiver may be included in the decoding apparatus. The renderer may include a display unit, and the display unit may consist of a separate device or an external component.

[0055] A video source can acquire video / images through the process of capturing, compositing, or generating video / images. A video source can include devices for capturing video / images and devices for generating video / images. Devices for capturing video / images can include at least one camera, video / image archives containing previously captured video / images, etc. Devices for generating video / images can include computers, tablets, smartphones, etc., and can generate video / images (electronically). For example, virtual video / images can be generated by computers, etc., and in this case, the process of capturing video / images can be replaced by the process of generating related data.

[0056] An encoding device can encode input video / images. The encoding device can perform a series of processes such as prediction, transformation, and quantization for compression and coding efficiency. The encoded data (encoded video / image information) can be output as a bitstream.

[0057] The transmitting unit can send encoded video / image information or data, output in bitstream form, to the receiving unit of the receiving device via digital storage media or a network, either as a file or through streaming. Digital storage media can include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, SSD, etc. The transmitting unit can include elements for generating media files according to a predetermined file format and may include elements for transmission over broadcast / communication networks. The receiving unit can receive / extract the bitstream and send it to a decoding device.

[0058] Decoding devices can decode video / images by performing a series of processes such as inverse quantization, inverse transform, and prediction, which correspond to the operations of encoding devices.

[0059] The renderer can render decoded video / images. The rendered video / images can be displayed through a display unit.

[0060] Figure 2 A rough block diagram of an encoding apparatus that can be applied to embodiments of the present disclosure and perform encoding of video / image signals is shown.

[0061] refer to Figure 2 The encoding device 200 may consist of an image segmenter 210, a predictor 220, a residual processor 230, an entropy encoder 240, an adder 250, a filter 260, and a memory 270. The predictor 220 may include an inter-frame predictor 221 and an intra-frame predictor 222. The residual processor 230 may include a transform 232, a quantizer 233, an inverse quantizer 234, and an inverse transform 235. The residual processor 230 may further include a subtractor 231. The adder 250 may be referred to as a reconstructor or a reconstruction block generator. According to an embodiment, the image segmenter 210, predictor 220, residual processor 230, entropy encoder 240, adder 250, and filter 260 may be configured by at least one hardware component (e.g., an encoder chipset or processor). Additionally, the memory 270 may include a decoded picture buffer (DPB) and may be configured by a digital storage medium. The hardware component may further include the memory 270 as an internal / external component.

[0062] Image segmenter 210 can partition an input image (or picture, frame) input to encoding device 200 into at least one processing unit. As an example, a processing unit can be referred to as a coding unit (CU). In this case, the coding unit can be recursively partitioned from the coding tree unit (CTU) or the largest coding unit (LCU) according to a quadtree-binary-trinary-tree (QTBTTT) structure.

[0063] For example, a coding unit can be segmented into multiple coding units of greater depth based on a quadtree structure, a binary tree structure, and / or a ternary structure. In this case, for example, a quadtree structure can be applied first, and a binary tree structure and / or a ternary structure can be applied later. Alternatively, a binary tree structure can be applied before the quadtree structure. The coding process according to this specification can be performed based on the final coding unit that is no longer segmented. In this case, based on coding efficiency according to image characteristics, the largest coding unit can be directly used as the final coding unit, or if necessary, the coding unit can be recursively segmented into deeper coding units, and the coding unit with the optimal size can be used as the final coding unit. Here, the coding process can include processes such as prediction, transformation, and reconstruction, as described later.

[0064] As another example, the processing unit may further include a prediction unit (PU) or a transform unit (TU). In this case, the prediction unit and the transform unit may be divided or segmented from the aforementioned final encoding unit, respectively. The prediction unit may be a unit for predicting samples, and the transform unit may be a unit for deriving transform coefficients and / or a unit for deriving residual signals from transform coefficients.

[0065] In some cases, a unit can be used interchangeably with terms such as block or region. Generally, an MxN block can represent a set of transform coefficients or samples consisting of M columns and N rows. Samples can typically represent pixels or pixel values, and can represent only the pixel / pixel value of the luminance component, or only the pixel / pixel value of the chrominance component. Samples can be used as a term to correspond a picture (or image) to pixels or cells.

[0066] The encoding device 200 can subtract the prediction signal (prediction block, prediction sample array) output from the inter-frame predictor 221 or the intra-frame predictor 222 from the input image signal (original block, original sample array) to generate a residual signal (residual signal, residual sample array), and the generated residual signal is sent to the converter 232. In this case, the unit in the encoding device 200 that subtracts the prediction signal (prediction block, prediction sample array) from the input image signal (original block, original sample array) can be called the subtractor 231.

[0067] Predictor 220 can perform prediction on the block to be processed (hereinafter referred to as the current block) and generate a block of predictions including prediction samples for the current block. Predictor 220 can determine whether to apply intra-frame prediction or inter-frame prediction on a block or CU basis. Predictor 220 can generate various information about the prediction, such as prediction mode information, and send it to entropy encoder 240, as described later in the description of each prediction mode. The information about the prediction can be encoded in entropy encoder 240 and output as a bitstream.

[0068] Intra-predictor 222 can predict the current block by referencing samples within the current image. Depending on the prediction mode, the referenced samples can be located near the current block or at a distance from it. In intra-prediction, the prediction mode can include at least one non-directional mode and multiple directional modes. The non-directional mode can include at least one of a DC mode or a planar mode. Depending on the level of detail of the prediction direction, the directional modes can include 33 or 65 directional modes. However, this is just an example, and more or fewer directional modes can be used depending on the configuration. Intra-predictor 222 can determine the prediction mode applied to the current block by using prediction modes applied to neighboring blocks.

[0069] Inter-frame predictor 221 can derive a prediction block for the current block based on a reference block (reference sample array) specified by motion vectors on a reference image. In this case, to reduce the amount of motion information transmitted in the inter-frame prediction mode, motion information can be predicted on a block, sub-block, or sample basis based on the correlation between motion information between neighboring blocks and the current block. Motion information may include motion vectors and reference image indices. Motion information may further include inter-frame prediction direction information (L0 prediction, L1 prediction, Bi prediction, etc.). For inter-frame prediction, neighboring blocks may include spatially neighboring blocks existing in the current image and temporally neighboring blocks existing in the reference image. The reference image including the reference block and the reference image including the temporally neighboring block may be the same or different. The temporally neighboring block may be referred to as a juxtaposed reference block, a juxtaposed CU (colCU), etc., and the reference image including the temporally neighboring block may be referred to as a juxtaposed image (colPic). For example, inter-frame predictor 221 can configure a motion information candidate list based on neighboring blocks and generate information indicating which candidate is used to derive the motion vector and / or reference image index of the current block. Inter-frame prediction can be performed based on various prediction modes, and for example, for skip mode and merge mode, the inter-frame predictor 221 can use the motion information of neighboring blocks as the motion information of the current block. For skip mode, unlike merge mode, residual signals may not be sent. For motion vector prediction (MVP) mode, the motion vectors of neighboring blocks are used as motion vector predictors, and the motion vector difference is signaled to indicate the motion vector of the current block.

[0070] Predictor 220 can generate a prediction signal based on various prediction methods described later. For example, the predictor can not only apply intra-frame prediction or inter-frame prediction to predict a block, but also apply both intra-frame prediction and inter-frame prediction simultaneously. This can be referred to as a combined intra-frame and inter-frame prediction (CIIP) mode. Alternatively, the predictor can be based on an intra-block copy (IBC) prediction mode or a palette mode for prediction against a block. The IBC prediction mode or palette mode can be used for content image / video coding such as screen content coding (SCC) in games, etc. IBC essentially performs prediction within the current image, but it can be performed similarly to inter-frame prediction because it derives a reference block within the current image. In other words, IBC can use at least one of the inter-frame prediction techniques described herein. A palette mode can be considered an example of intra-frame coding or intra-frame prediction. When a palette mode is applied, sample values ​​within the image can be signaled based on information about the palette table and palette index. The prediction signal generated by predictor 220 can be used to generate a reconstructed signal or a residual signal.

[0071] Transformer 232 can generate transform coefficients by applying a transform technique to the residual signal. For example, the transform technique may include at least one of Discrete Cosine Transform (DCT), Discrete Sine Transform (DST), Karhunen-Loève Transform (KLT), Graphical Based Transform (GBT), or Conditional Nonlinear Transform (CNT). Here, GBT refers to the transform obtained from a graphic when the relationship information between pixels is expressed as a graphic. CNT refers to the transform obtained based on generating a prediction signal using all previously reconstructed pixels. Furthermore, the transform process can be applied to square pixel blocks of the same size or to non-square blocks of variable size.

[0072] Quantizer 233 can quantize the transform coefficients and send them to entropy encoder 240, which can encode the quantized signal (information about the quantized transform coefficients) and output it as a bitstream. The information about the quantized transform coefficients can be referred to as residual information. Quantizer 233 can rearrange the quantized transform coefficients in block form into a 1D vector form based on the coefficient scan order, and can generate information about the quantized transform coefficients based on the 1D vector form of the quantized transform coefficients.

[0073] The entropy encoder 240 can perform various encoding methods, such as exponential Columbus coding, context-adaptive variable-length coding (CAVLC), and context-adaptive binary arithmetic coding (CABAC). The entropy encoder 240 can encode information necessary for video / video image reconstruction (e.g., values ​​of syntax elements, etc.) in addition to the transform coefficients quantized together or individually.

[0074] Encoded information (e.g., encoded video / image information) can be transmitted or stored in bitstream form at the network abstraction layer (NAL) unit level. The video / image information may further include information about various parameter sets such as adaptive parameter sets (APS), picture parameter sets (PPS), sequence parameter sets (SPS), or video parameter sets (VPS). Additionally, the video / image information may further include general constraint information. Here, information transmitted from the encoding device / signaled to the decoding device and / or syntax elements can be included in the video / image information. The video / image information can be encoded by the above-described encoding process and included in the bitstream. The bitstream can be transmitted over a network or stored in a digital storage medium. Here, the network may include broadcast networks and / or communication networks, and the digital storage medium may include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, SSD, etc. Transmission units (not shown) for transmission and / or storage units (not shown) for storing signals output from the entropy encoder 240 can be configured as internal / external elements of the encoding device 200, or the transmission unit may also be included in the entropy encoder 240.

[0075] The quantized transform coefficients output from quantizer 233 can be used to generate a prediction signal. For example, the residual signal (residual block or residual sample) can be reconstructed by applying dequantization and inverse transform to the quantized transform coefficients using dequantizer 234 and inverse transformer 235. Adder 250 can add the reconstructed residual signal to the prediction signal output from inter-frame predictor 221 or intra-frame predictor 222 to generate a reconstructed signal (reconstructed image, reconstructed block, reconstructed sample array). When there is no residual for the block to be processed, such as when a skip mode is applied, the prediction block can be used as a reconstructed block. Adder 250 can be referred to as a reconstructor or reconstructed block generator. The generated reconstructed signal can be used for intra-frame prediction of the next block to be processed within the current image, and can also be used for inter-frame prediction of the next image by filtering, which will be described later. Meanwhile, a luminance mapping with chroma scaling (LMCS) can be applied during image encoding and / or reconstruction.

[0076] Filter 260 can improve subjective / objective image quality by applying filtering to the reconstructed signal. For example, filter 260 can generate a modified reconstructed image by applying various filtering methods to the reconstructed image, and the modified reconstructed image can be stored in memory 270, specifically in the DPB of memory 270. Various filtering methods can include deblocking filtering, sample adaptive shifting, adaptive loop filtering, bilateral filtering, etc. Filter 260 can generate various information about the filtering and send it to entropy encoder 240. The information about the filtering can be encoded in entropy encoder 240 and output as a bitstream.

[0077] The modified reconstructed image sent to memory 270 can be used as a reference image in inter-frame predictor 221. When inter-frame prediction is applied through it, the encoding device can avoid prediction mismatch in encoding device 200 and decoding device, and can also improve encoding efficiency.

[0078] The DPB of memory 270 can store modified reconstructed images for use as reference images in inter-frame predictor 221. Memory 270 can store motion information of blocks from which motion information in the current image is derived (or encoded) and / or motion information of blocks in the pre-reconstructed image. The stored motion information can be sent to inter-frame predictor 221 to be used as motion information for spatially or temporally neighboring blocks. Memory 270 can store reconstructed samples of reconstructed blocks in the current image and send them to intra-frame predictor 222.

[0079] Figure 3 A rough block diagram of a decoding apparatus that can be applied to embodiments of the present disclosure and perform decoding of video / image signals is shown.

[0080] refer to Figure 3 The decoding device 300 can be configured to include an entropy decoder 310, a residual processor 320, a predictor 330, an adder 340, a filter 350, and a memory 360. The predictor 330 may include an inter-frame predictor 332 and an intra-frame predictor 331. The residual processor 320 may include an inverse quantizer 321 and an inverse transformer 321.

[0081] According to an embodiment, the entropy decoder 310, residual processor 320, predictor 330, adder 340, and filter 350 described above can be configured by a single hardware component (e.g., a decoder chipset or processor). Additionally, the memory 360 may include a decoded image buffer (DPB) and can be configured by a digital storage medium. The hardware component may further include the memory 360 as an internal / external component.

[0082] When the input includes a bitstream containing video / image information, the decoding device 300 can respond to... Figure 2 The decoding device 300 reconstructs an image by processing video / image information in its encoding apparatus. For example, the decoding device 300 can derive units / blocks based on information related to block segmentation obtained from the bitstream. The decoding device 300 can perform decoding by using processing units applied in the encoding apparatus. Therefore, the processing unit for decoding can be an encoding unit, and the encoding unit can be segmented from encoding tree units or maximally encoded units according to a quadtree structure, binary tree structure, and / or ternary tree structure. At least one transform unit can be derived from the encoding unit. Furthermore, the reconstructed image signal decoded and output by the decoding device 300 can be played back by a playback device.

[0083] Decoding device 300 can receive data in bitstream form from... Figure 2 The signal output by the encoding device and the received signal can be decoded by the entropy decoder 310. For example, the entropy decoder 310 can parse the bitstream to derive information (e.g., video / image information) necessary for image reconstruction (or picture reconstruction). The video / image information may further include information about various parameter sets such as adaptive parameter sets (APS), picture parameter sets (PPS), sequence parameter sets (SPS), or video parameter sets (VPS). In addition, the video / image information may further include general constraint information. The decoding device can further decode the picture based on the information about the parameter sets and / or the general constraint information. The information sent / received by the signal and / or the syntax elements described later herein can be decoded and obtained from the bitstream through the decoding process. For example, the entropy decoder 310 can decode the information in the bitstream based on encoding methods such as exponential Golomb coding, CAVLC, CABAC, etc., and output the values ​​of the syntax elements necessary for image reconstruction and the quantized values ​​of the transform coefficients of the residuals. More specifically, the CABAC entropy decoding method can receive bins corresponding to each syntax element from the bitstream, determine a context model using information about the syntax element to be decoded, decoding information of neighboring blocks and the block to be decoded, or information about symbols / bins decoded in the previous step, perform arithmetic decoding on the bins by predicting the occurrence probability of the bins based on the determined context model, and generate symbols corresponding to the value of each syntax element. In this case, after determining the context model, the CABAC entropy decoding method can update the context model by using information about the decoded symbols / bins for the context model used for the next symbol / bin. Among the information decoded in the entropy decoder 310, information about prediction is provided to the predictors (inter-frame predictor 332 and intra-frame predictor 331), and the residual values ​​of entropy decoding performed on them in the entropy decoder 310, i.e., the quantized transform coefficients and related parameter information, can be input to the residual processor 320. The residual processor 320 can derive residual signals (residual blocks, residual samples, residual sample arrays). In addition, information about filtering among the information decoded in the entropy decoder 310 can be provided to the filter 350. Meanwhile, the receiving unit (not shown) that receives the signal output from the encoding device can be further configured as an internal / external element of the decoding device 300 or the receiving unit can be a component of the entropy decoder 310.

[0084] Furthermore, the decoding device according to this specification can be referred to as a video / image / picture decoding device, and the decoding device can be divided into an information decoder (video / image / picture information decoder) and a sample decoder (video / image / picture sample decoder). The information decoder may include an entropy decoder 310, and the sample decoder may include at least one of an inverse quantizer 321, an inverse transformer 322, an adder 340, a filter 350, a memory 360, an inter-frame predictor 332, and an intra-frame predictor 331.

[0085] The dequantizer 321 can dequantize the quantized transform coefficients and output the transform coefficients. The dequantizer 321 can rearrange the quantized transform coefficients into a two-dimensional block form. In this case, the rearrangement can be performed based on the coefficient scan order executed in the encoding device. The dequantizer 321 can obtain the transform coefficients by performing dequantization on the quantized transform coefficients using quantization parameters (e.g., quantization step size information).

[0086] The inverse transformer 322 performs an inverse transformation on the transformation coefficients to obtain the residual signal (residual block, residual sample array).

[0087] Predictor 320 can perform prediction on the current block and generate a prediction block including prediction samples for the current block. Predictor 320 can determine whether to apply intra-frame prediction or inter-frame prediction to the current block based on the prediction information output from entropy decoder 310, and determine a specific intra-frame / inter-frame prediction mode.

[0088] Predictor 320 can generate prediction signals based on various prediction methods described later. For example, predictor 320 can not only apply intra-frame prediction or inter-frame prediction to predict a block, but also apply intra-frame prediction and inter-frame prediction simultaneously. This can be referred to as a combined intra-frame and inter-frame prediction (CIIP) mode. Alternatively, the predictor can be based on an intra-block copy (IBC) prediction mode or a palette mode for block prediction. The IBC prediction mode or palette mode can be used for content image / video coding such as screen content coding (SCC) in games, etc. IBC essentially performs prediction within the current frame, but it can be performed similarly to inter-frame prediction because it derives a reference block within the current frame. In other words, IBC can use at least one of the inter-frame prediction techniques described herein. Palette mode can be considered an example of intra-frame coding or intra-frame prediction. When a palette mode is applied, information about the palette table and palette index can be included in the video / image information and transmitted as a signal.

[0089] The intra-predictor 331 can predict the current block by referencing samples within the current image. Depending on the prediction mode, the referenced samples can be located near the current block or at a certain distance away from it. In intra-prediction, the prediction mode can include at least one non-directional mode and multiple directional modes. The intra-predictor 331 can determine the prediction mode applied to the current block by using prediction modes applied to neighboring blocks.

[0090] Inter-frame predictor 332 can derive a prediction block for the current block based on a reference block (reference sample array) specified by motion vectors on a reference image. In this case, to reduce the amount of motion information transmitted in the inter-frame prediction mode, motion information can be predicted on a block, sub-block, or sample basis based on the correlation between motion information of neighboring blocks and the current block. Motion information may include motion vectors and a reference image index. Motion information may further include inter-frame prediction direction information (L0 prediction, L1 prediction, Bi prediction, etc.). For inter-frame prediction, neighboring blocks may include spatially neighboring blocks existing in the current image and temporally neighboring blocks existing in the reference image. For example, inter-frame predictor 332 can configure a motion information candidate list based on neighboring blocks and derive the motion vector and / or reference image index of the current block based on the received candidate selection information. Inter-frame prediction can be performed based on various prediction modes, and information about the prediction may include information indicating the inter-frame prediction mode used for the current block.

[0091] Adder 340 can add the obtained residual signal to the prediction signal (prediction block, prediction sample array) output from the predictor (including inter-frame predictor 332 and / or intra-frame predictor 331) to generate a reconstruction signal (reconstructed image, reconstruction block, reconstruction sample array). When there is no residual for the block to be processed, such as when a skip mode is applied, the prediction block can be used as the reconstruction block.

[0092] Adder 340 can be referred to as a reconstructor or reconstructed block generator. The generated reconstructed signal can be used for intra-frame prediction of the next block to be processed in the current image, output by filtering as described later, or it can be used for inter-frame prediction of the next image. Meanwhile, a luminance map with chroma scaling (LMCS) can be applied during image decoding.

[0093] Filter 350 can improve subjective / objective image quality by applying filtering to the reconstructed signal. For example, filter 350 can generate a modified reconstructed image by applying various filtering methods to the reconstructed image, and send the modified reconstructed image to memory 360, specifically the DPB of memory 360. Various filtering methods can include deblocking filtering, adaptive sampling offset, adaptive loop filtering, bilateral filtering, etc.

[0094] The (modified) reconstructed image stored in the DPB of memory 360 can be used as a reference image in inter-frame predictor 332. Memory 360 can derive (or decode) motion information of blocks from its current image and / or motion information of blocks in the pre-reconstructed image. The stored motion information can be sent to inter-frame predictor 332 as motion information for spatially or temporally neighboring blocks. Memory 360 can store reconstructed samples of reconstructed blocks in the current image and send them to intra-frame predictor 331.

[0095] Here, the embodiments described in the encoding device 200’s filter 260, inter-frame predictor 221 and intra-frame predictor 222 can also be applied equally or correspondingly to the decoding device 300’s filter 350, inter-frame predictor 332 and intra-frame predictor 331, respectively.

[0096] Figure 4 The illustration shows a method for reconstructing video images performed by a decoding device 300 according to the present disclosure.

[0097] The S400 can receive bitstreams including encoded video images.

[0098] The encoded video image S410 can be reconstructed from the bitstream.

[0099] Video information related to encoded video images can be extracted from the bitstream. The extracted video information can then be used to reconstruct the encoded video images.

[0100] The bitstream may include information related to the Neural Network Post-Filter (NNPF). NNPF-related information may include at least one of Neural Network Post-Filter Activation (NNPFA) information, Neural Network Post-Filter Characteristics (NNPFC) information, or Neural Network Post-Filter Update (NNPFU) information.

[0101] NNPFA, NNPFC, and NNPFU information can be configured in supplementary enhancement information (SEI) messages, and in this case, the NNPFA, NNPFC, and NNPFU information can be referred to as NNPFA SEI messages, NNPFC SEI messages, and NNPFU SEI messages, respectively. For ease of description, it is assumed below that NNPF-related information is configured in SEI messages.

[0102] The NNPFA SEI message can activate or disable the possible use of a target NNPF for post-processing filtering of an image. Here, the target NNPF can be identified based on at least one of the target identifier (nnpfa_target_id) or target base flag (nnpfa_target_base_flag) described below.

[0103] For a specific image in which the NNPF is activated, the target NNPF can be as follows: When nnpfa_target_base_flag is 1, the target NNPF can be the base NNPF with the same nnpfc_id as nnpfa_target_id. Otherwise (i.e., when nnpfa_target_base_flag is 0), the target NNPF can be the NNPF specified by the last NNPFC SEI message. Here, the last NNPFC SEI message can have the same nnpfc_id as the nnpfa_target_id of the first VCL NAL unit that precedes the current image in decoding order. When nnpfa_target_base_flag is 0, the target NNPF does not need to be a duplicate of the NNPFC SEI message that includes the base NNPF.

[0104] For the same image, there may be multiple NNPFA SEI messages. For example, when NNPF is used for different purposes or for filtering different color components, there may be multiple NNPFA SEI messages for the same image.

[0105] As an example, the NNPFA SEI message according to this disclosure can be configured as shown in Table 1 below.

[0106] [Table 1]

[0107] According to Table 1, the NNPFA SEI message may include a target identifier (nnpfa_target_id). nnpfa_target_id can represent the identifier (nnpfc_id) of the target NNPF. The target NNPF can be specified based on one or more NNPFA SEI messages associated with the current image. The target NNPF may have the same nnpfc_id as nnpfa_target_id. The value of nnpfa_target_id may be restricted to falling within the range of 0 to (2^3). 32 Within the range of -2).

[0108] The presence of NNPFA SEI messages with a specific value of nnpfa_target_id in the current picture unit (PU) can be restricted if one or both of the following conditions are not met.

[0109] (Condition 1) Within the current coding layer video sequence (CLVS), there exists an NNPFC SEI message with the same nnpfc_id as the nnpfa_target_id that exists in a PU that precedes the current PU in the decoding order.

[0110] (Condition 2) There exists an NNPFCSEI message with the same nnpfc_id as the specific value of nnpfa_target_id in the current PU.

[0111] When both an NNPFC SEI message with a specific value of nnpfc_id and an NNPFA SEI message with the same specific value of nnpfc_id as nnpfa_target_id are included in the PU, the NNPFC SEI message will precede the NNPFA SEI message in the decoding order.

[0112] NNPFA SEI messages can include a cancellation flag (nnpfa_cancel_flag). When nnpfa_cancel_flag is 1, it indicates that the persistence of a target NNPF established by a previous NNPFA SEI message with the same nnpfa_target_id as the current SEI message is cancelled. In other words, the target NNPF may no longer be used unless it is activated by another NNPFA SEI message with the same nnpfa_target_id as the current SEI message and a cancellation flag with a value of 0. When nnpfa_cancel_flag is 0, it can indicate that nnpfa_persistence_flag, nnpfa_target_base_flag, nnpfa_no_prev_clvs_flag, nnpfa_no_foll_clvs_flag, and nnpfa_num_output_entry, which will be described later, follow.

[0113] The NNPFA SEI message may include a persistence flag (nnpfa_persistence_flag). nnpfa_persistence_flag specifies the persistence of the target NNPF used for the current layer. For example, when nnpfa_persistence_flag is 0, it specifies that the target NNPF can be used only for post-processing filtering of the current image. When nnpfa_persistence_flag is 1, it specifies that the target NNPF can be used for post-processing filtering of the current image and all subsequent images in the current layer, in output order.

[0114] Suppose that the set of images for which the target NNPF is activated by the current NNPFA SEI message is nnpfTargetPictures, and the set of images for which the NNPFC SEI message corresponding to the target NNPF is applied is nnpfcTargetPictures. In this case, images included in nnpfTargetPictures may be restricted to being included in nnpfcTargetPictures.

[0115] The NNPFA SEI message may include a target base flag (nnpfa_target_base_flag). When nnpfa_target_base_flag is 1, it indicates that the target NNPF is the base NNPF. Here, the base NNPF may have the same nnpfc_id as the nnpfa_target_id mentioned above. When nnpfa_target_base_flag is 0, it indicates that the target NNPF is the NNPF assigned by the last NNPFC SEI message. Here, the last NNPFC SEI message may have the same nnpfc_id as the nnpfa_target_id of the first VCL NAL unit that precedes the current image in decoding order. Additionally, when nnpfa_target_base_flag is 0, the target NNPF may not be a duplicate of the NNPFC SEI message that includes the base NNPF.

[0116] When the underlying NNPFDML update is activated to switch the target NNPF from the updated NNPF to the underlying NNPF, the NNPFASEI message can activate the underlying NNPF with a specific value of nnpfc_id.

[0117] When the nnpfa_target_base_flag of the NNPFA SEI message is 0, it may restrict the existence of at least one NNPFC SEI message that has an nnpfc_id equal to nnpfa_target_id and an nnpfc_base_flag value equal to 0, and is decoded before the NNPFASEI message.

[0118] The NNPFA SEI message may include the previous CLVS flag (nnpfa_no_prev_clvs_flag). When nnpfa_no_prev_clvs_flag is 1, it indicates that the input image used for NNPF was not generated from the previous CLVS. When nnpfa_no_prev_clvs_flag is 1, it indicates that the input image used for NNPF may or may not be generated from the previous CLVS.

[0119] When the current CLVS is spliced ​​from another bitstream immediately following the previous CLVS and one or more input images may have a negative impact on the output of the target NNPF, the value of nnpfa_no_prev_clvs_flag can change from 0 to 1 because they are selected from one or more previous CLVSs due to the NNPFA SEI message.

[0120] The NNPFA SEI message may include a follow CLVS flag (nnpfa_no_foll_clvs_flag). When nnpfa_no_foll_clvs_flag is 1, it indicates that when the NNPFA SEI message is continuously used for the last PU of the CLVS in output order, it is processed as if it were continuously used for the last PU of the current layer in the bitstream in output order. The value of nnpfa_no_foll_clvs_flag may have no effect when the NNPFA SEI message is not continuously used for the last PU of the CLVS in output order, or when the value of nnpfa_no_foll_clvs_flag is 0.

[0121] The `nnpfa_no_foll_clvs_flag` can be adaptively signaled based on `nnpfa_persistence_flag`. For example, `nnpfa_no_foll_clvs_flag` can be signaled based on a value of 1. `nnpfa_persistence_flag` can be substituted without signaling if its value is 0.

[0122] When the next CLVS is concatenated from another bitstream immediately following the current CLVS, the value of nnpfa_no_foll_clvs_flag can change from 0 to 1 for image rate upsampling NNPF. As a result, the NNPF process can interpolate images to the last current CLVS using only the input images generated in the current CLVS.

[0123] The NNPFA SEI message may include output entry information (nnpfa_num_output_entries). nnpfa_num_output_entries specifies the number of output flags present in the NNPFA SEI message. The value of nnpfa_num_output_entries can be restricted to fall within the range of 0 to NumInpPicsInOutputTensor. When the value of PictureRateUpsampingFlag is 0 and nnpfa_num_output_entries is the same as NumInpPicsInOutputTensor, at least one of the output flags present in the NNPFA SEI message can be restricted to have a value of 1.

[0124] The NNPFA SEI message may include an output flag (nnpfa_output_flag). When nnpfa_output_flag is 1, it indicates that the NNPF process activated by the NNPFA SEI message outputs an NNPF-generated image corresponding to the input image with index (InpIdx). When nnpfa_output_flag is 0, it indicates that the NNPF process activated by the NNPFA SEI message does not output an NNPF-generated image corresponding to the input image with index (InpIdx).

[0125] When nnpfa_num_output_entries is less than NumInpPicsInOutputTensor, for each value falling within the range of nnpfa_num_output_entries to (NumInpPicsInOutputTensor-1), nnpfa_output_flag can be inferred to be a value of 1.

[0126] The NNPFA SEI message may include a presence flag (nnpfa_spatial_extrapolation_prompt_flag). nnpfa_spatial_extrapolation_prompt_flag may be related to the presence of a prompt message (nnpfa_prompt).

[0127] For example, a value of 1 for `nnpfa_spatial_extrapolation_prompt_present_flag` indicates the presence of `nnpfa_prompt`. Similarly, a value of 1 for `nnpfa_spatial_extrapolation_prompt_present_flag` indicates the presence of zero-bit information (`nnpfa_alignment_zero_bit_c`). A value of 0 for `nnpfa_spatial_extrapolation_prompt_present_flag` indicates the absence of `nnpfa_prompt`. Furthermore, a value of 0 for `nnpfa_spatial_extrapolation_prompt_present_flag` indicates the absence of `nnpfa_alignment_zero_bit_c`.

[0128] The `nnpfa_spatial_extrapolation_prompt_present_flag` can be related to whether the spatial extrapolation hint is included in the NNPFC identified by `nnpfa_target_id`. The value of `nnpfa_spatial_extrapolation_prompt_present_flag` can be 1 only if a spatial extrapolation hint is present.

[0129] The NNPFA SEI message may include zero-bit information (nnpfa_alignment_zero_bit_c). nnpfa_alignment_zero_bit_c may be restricted to a value of 0.

[0130] The NNPFA SEI message can include a prompt message (nnpfa_prompt). nnpfa_prompt can specify a text string prompt for the content used to generate the spatially extrapolated image region. When the value of nnpfa_spatial_extrapolation_prompt_present_flag is 1, nnpfa_prompt can be restricted to an empty string.

[0131] At least one of nnpfa_alignment_zero_bit_c or nnpfa_prompt can be adaptively sent with a signal based on nnpfa_spatial_extrapolation_prompt_present_flag.

[0132] For example, `nnpfa_alignment_zero_bit_c` can be signaled based on a value of 1 for `nnpfa_spatial_extrapolation_prompt_present_flag`. `nnpfa_alignment_zero_bit_c` can be signaled without a value of 0 for a value of 0 for a value of 1 for a value of 1 for a value of 0 for a value of 0 for a value of 0 for a value of 0 for a value of 0 for a value of 1 ...

[0133] As an example, the NNPFA SEI message according to this disclosure can be configured as shown in Table 2 below.

[0134] [Table 2]

[0135] According to Table 2, the NNPFA SEI message may further include a prompt update flag (nnpfa_spatial_extrapolation_prompt_update_flag). The nnpfa_spatial_extrapolation_prompt_update_flag may be related to an update of the nnpfa_prompt.

[0136] For example, when the value of nnpfa_spatial_extrapolation_prompt_update_flag is 1, it indicates that nnpfa_spatial_extrapolation_prompt_presnt_flag and nnpfa_prompt will be updated. On the other hand, when the value of nnpfa_spatial_extrapolation_prompt_update_flag is 0, it indicates that nnpfa_spatial_extrapolation_prompt_presnt_flag and nnpfa_prompt will remain the same as before.

[0137] When the SpatialExtrapolationFlag value of the target NNPF is 0, the nnpfa_spatial_extrapolation_prompt_update_flag can be restricted to a value of 0. Here, SpatialExtrapolationFlag can be information derived based on the purpose information (nnpfc_purpose) of the NNPFC SEI message, which will be described later.

[0138] In Table 2, the grammatical elements that overlap with those in Table 1 are as described in Table 1, and detailed descriptions are omitted here.

[0139] The NNPFC SEI message can specify the neural network that can be used as a post-filter. The NNPFC SEI message can be used to indicate the use of a specific neural network post-filter (NNPF) for a specific image.

[0140] The following variables can be defined for using NNPFC SEI messages.

[0141] - The width and height of the input image in the brightness sample unit.

[0142] - The luminance sample array (CroppedYPic[idx]) and chrominance sample array (CroppedCbPic[idx] and CroppedCrPic[idx]) of the input image with indices (idx) falling in the range of 0 to (numInputPics-1) are used as input to NNPF.

[0143] - Bit depth (BitDepthY) of the luminance sample array used for the input image.

[0144] - Bit depth (BitDepthC) of the chroma sample array used for the input image.

[0145] - ChromaFormatIdc

[0146] - When npfc_auxiliary_inp_idc is 1, for input images with indices (idx) ranging from 0 to (numInputPics-1), include the array of filter strength control values ​​(StrengthControlVal[idx]) that fall within the range of 0 to 1.

[0147] An input image with index 0 can correspond to an NNPF image defined in the NNPFC SEI message, which is activated by the NNPFA SEI message. An input image with index i falling in the range of 1 to (numInputPics-1) can be output before the input image with index (i-1).

[0148] The variables SubWidthC and SubHeightC can be exported based on ChromaFormatIdc.

[0149] As an example, the NNPFC SEI message according to this disclosure can be configured as shown in Table 3 below.

[0150] [Table 3]

[0151] According to Table 3, the NNPFC SEI message may include purpose information (nnpfc_purpose). nnpfc_purpose can represent the purpose of the NNPF as defined in Table 4 below. Here, when (nnpfc_purpose & bitMask) is not 0, it indicates that the NNPF has a purpose associated with the bitmask value in Table 1. When nnpfc_purpose is greater than 0 and (nnpfc_purpose & bitMask) is 0, the purpose associated with the bitmask value may not be applied to the NNPF. When the value of nnpfc_purpose is 0, the NNPF determined by the application and assigned by nnpfc_application_purpose_tag_uri can be used.

[0152] [Table 4]

[0153] Within CLVS, all NNPFC SEI messages with a specific nnpfc_id value can be restricted to having the same nnpfc_purpose value. The value of nnpfc_purpose can be restricted to falling within the range of 0 to 255.

[0154] The NNPFC SEI message may include an identifier (nnpfc_id). nnpfc_id may include an identification number that can be used to identify the NNFP. The value of nnpfc_id may be restricted to falling within the range of 0 to (2^3). 32 Within the range of -2).

[0155] The following can be applied when the NNPFC SEI message is the first NNPFC SEI message in decoding order that has a specific value of nnpfc_id within the current CLVS.

[0156] - This SEI message specifies the underlying NNPF.

[0157] - This SEI message, ending with the current CLVS, involves the currently decoded image of the current layer and all subsequently decoded images in decoding order.

[0158] The NNPFC SEI message may include a base flag (nnpfc_base_flag). When the value of nnpfc_base_flag is 1, it indicates that the NNPFC SEI message specifies the base NNPF. When the value of nnpf_base_flag is 0, it indicates that the NNPFC SEI message specifies an update to the base NNPF.

[0159] The constraints in Table 5 below can be applied to the value of nnpfc_base_flag.

[0160] [Table 5]

[0161] When the value of nnpfc_base_flag is 0, the following table 6 can be applied.

[0162] [Table 6]

[0163] The NNPFC SEI message may include a mode indicator (nnpfc_mode_idc). When the value of nnpfc_mode_idc is 0, it can indicate that the NNPFC SEI message includes an ISO / IEC 15938-17 bitstream specifying the underlying NNPF (when nnpfc_base_flag is 1), or it can indicate that the NNPFC SEI message is an update to the underlying NNPF with the same value of nnpfc_id (when nnpfc_base_flag is 0).

[0164] When the value of nnpfc_base_flag is 1, an nnpfc_mode_idc with a value of 1 can represent the underlying NNPF associated with the value of nnpfc_id as a neural network identified by the URI indicated by nnpfc_uri in the format identified by the tag URI nnpfc_tag_uri.

[0165] When the value of nnpfc_base_flag is 0, an nnpfc_mode_idc with a value of 1 can represent an update of the underlying NNPF with the same value of nnpfc_id. The update is defined by the URI indicated by nnpfc_uri in the format identified by the tag URI nnpfc_tag_uri.

[0166] The value of nnpfc_mode_idc can be restricted to fall within the range of 0 to 1.

[0167] The NNPFC SEI message may include zero-bit information (nnpfc_reserved_zero_bit_a). nnpfc_reserved_zero_bit_a can be restricted to a value of 0.

[0168] The NNPFC SEI message may include tag URI information (nnpfc_tag_uri). The nnpfc_tag_uri may include a tag URI with the semantics and syntax defined in IETF RFC 4151. Here, the tag URI can identify the format and related information of a neural network used as an update to a base NNPF or a base NNPF having the same value as the nnpfc_id specified by the nnpfc_uri. When the nnpfc_tag_uri is the same as “tag:iso.org,2023:15938-17”, it can indicate that the neural network data identified by the nnpfc_uri conforms to ISO / IEC 15938-17.

[0169] The NNPFC SEI message may include URI information (nnpfc_uri). nnpfc_uri may include a URI with the semantics and syntax specified in IETF Internet Standard 66. Here, the URI may identify the neural network used as an update to the underlying NNPF or the underlying NNPF with the same value nnpfc_id.

[0170] The NNPFC SEI message may include an attribute presence flag (nnpfc_property_present_flag). When nnpfc_property_present_flag is 1, it indicates the presence of syntax elements related to the filter's purpose, input format, output format, and complexity. Conversely, when nnpfc_property_present_flag is 0, it indicates the absence of syntax elements related to the filter's purpose, input format, output format, and complexity.

[0171] When the value of nnpfc_base_flag is 1, nnpfc_property_present_flag can be restricted to a value of 1.

[0172] When the value of nnpfc_property_present_flag is 0, the values ​​of all syntax elements that can exist only when the value of nnpfc_property_present_flag is 1 can be inferred as the values ​​of the corresponding syntax elements in the NNPFC SEI message of the underlying NNPF that includes this SEI message for which updates.

[0173] The constraints in Table 7 below can be applied when the NNPFC SEI message nnpfcCurr is not the first NNPFC SEI message in the current CLVS with a specific value of nnpfc_id in the decoding order, nor is it a duplicate of the first NNPFC SEI message with the corresponding nnpfc_id (i.e., the value of nnpfc_base_flag is 0) and the value of nnpfc_property_present_flag is 1.

[0174] [Table 7]

[0175] The NNPFC SEI message can include input quantity information (nnpfc_num_input_pics_minus1). The value obtained by incrementing nnpfc_num_input_pics_minus1 by 1 represents the number of input images used in NNPF. The value of nnpfc_num_input_pics_minus1 can be restricted to fall within the range of 0 to 63. When the value of PictureRateUpsampingFlag is 1, nnpfc_num_input_pics_minus1 can be restricted to have a value greater than 0.

[0176] You can export a variable (numInputPics) that specifies the number of images used as input to NNPF, as shown in Equation 1 below.

[0177] [Equation 1]

[0178] The NNPFC SEI message can include an image output flag (nnpfc_input_pic_output_flag). When nnpfc_input_pic_output_flag[i] is 1, it indicates that NNPF generates the corresponding output image for the i-th input image. When nnpfc_input_pic_output_flag[i] is 0, it indicates that NNPF does not generate the corresponding output image for the i-th input image.

[0179] When the value of nnpfc_num_input_pics_minus1 is 0, the value of nnpfc_input_pic_output_flag[0] can be inferred to be 1. When the value of PictureRateUpsampingFlag is 0 and the value of nnpfc_num_input_pics_minus1 is greater than 0, for at least one i falling within the range of 0 to nnpfc_num_input_pics_minus1, nnpfc_input_pic_output_flag[i] can be restricted to have a value of 1.

[0180] The NNPFC SEI message may include an image zero flag (nnpfc_absent_input_pic_zero_flag). When nnpfc_absent_input_pic_zero_flag is 1, it indicates that an input image that NNPF expects to be absent from the bitstream is represented as a sample array with a sample value of 0. When nnpfc_absent_input_pic_zero_flag is 0, it indicates that an input image that NNPF expects to be absent from the bitstream is represented as the most recent input image in the bitstream in output order.

[0181] The NNPFC SEI message may include a sub-chroma flag (nnpfc_out_sub_c_flag). When chromaUpsamplingFlag is 1, nnpfc_out_sub_c_flag can specify the values ​​of the variables outSubWidthC and outSubHeightC.

[0182] When nnpfc_out_sub_c_flag is 1, outSubWidthC and outSubHeightC can both be 1. When nnpfc_out_sub_c_flag is 0, outSubWidthC can be 2 and outSubHeightC can be 1. When ChromaFormatIdc is 1 and nnpfc_out_sub_c_flag exists, nnpfc_out_sub_c_flag can be restricted to a value of 1.

[0183] The `nnpfc_out_sub_c_flag` can be adaptively sent using a signal based on the `CromaUpsamplingFlag`. The `CromaUpsamplingFlag` can be related to whether the destination information indicates that the purpose of NNPF will include chroma upsampling. The `CromaUpsamplingFlag` can be derived based on the destination information. For example, the `CromaUpsamplingFlag` can be derived as shown in Equation 2 below.

[0184] [Equation 2]

[0185] The NNPFC SEI message can include a color format indicator (nnpfc_out_colour_format_idc). When the colorizationFlag value is 1, nnpfc_out_colour_format_idc can specify the color format of the NNPF output and the corresponding values ​​for outSubWidthC and outSubHeightC.

[0186] When nnpfc_out_colour_format_idc is 1, it indicates that the NNPF output color format is 4:2:0, and both outSubWidthC and outSubHeightC are 2. When nnpfc_out_colour_format_idc is 2, it indicates that the NNPF output color format is 4:2:2, outSubWidthC is 2, and outSubHeightC is 1. When nnpfc_out_colour_format_idc is 3, it indicates that the NNPF output color format is 4:4:4, and both outSubWidthC and outSubHeightC are 1. nnpfc_out_colour_format_idc can be restricted to a value of 0.

[0187] When both chromaUpsampingFlag and colourizationFlag are 0, outSubWidthC and outSubHeightC can be inferred to have the same values ​​as SubWidthC and SubHeightC, respectively.

[0188] The `nnpfc_out_colour_format_idc` can be adaptively signaled based on the colorization flag (ColourizationFlag). The ColourizationFlag can be related to whether the purpose information indicates that the NNPF purpose will include colorization. The ColourizationFlag can be derived based on the purpose information. For example, the ColourizationFlag can be derived as shown in Equation 3 below.

[0189] [Equation 3]

[0190] The NNPFC SEI message can include width ratio information (nnpfc_pic_width_num_minus1, nnpfc_pic_width_deom_minus1). The values ​​obtained by incrementing nnpfc_pic_width_num_minus1 and nnpfc_pic_width_deom_minus1 can respectively specify the numerator and denominator of the resampling ratio used for the width of the NNPF output image in CroppedWidth.

[0191] The value of (nnpfc_pic_width_num_minus1+1) / (nnpfc_pic_width_deom_minus1+1) can be restricted to falling within the range of 1 / 16 or greater and 16 or less. When nnpfc_pic_width_num_minus1 and nnpfc_pic_width_deom_minus1 do not exist, their values ​​can be inferred to be 0.

[0192] The variable nnpfcOutputPicWidth can represent the width of the luminance sample array of the image generated by applying the NNPF identified by nnpfc_id to the input image. nnpfcOutputPicWidth can be derived as shown in Equation 4 below.

[0193] [Equation 4]

[0194] Here, (nnpfcOutputPicWidth%outSubWidthC) can be restricted to having a value of 0.

[0195] The NNPFC SEI message can include height ratio information (nnpfc_pic_height_num_minus1, nnpfc_pic_height_deom_minus1). The values ​​obtained by incrementing nnpfc_pic_height_num_minus1 and nnpfc_pic_height_deom_minus1 by 1 can respectively specify the numerator and denominator of the resampling ratio used for the height of the NNPF output image for CroppedHeight.

[0196] The value of (nnpfc_pic_height_num_minus1+1) / (nnpfc_pic_height_deom_minus1+1) can be restricted to falling within the range of 1 / 16 or greater and 16 or less. When nnpfc_pic_height_num_minus1 and nnpfc_pic_height_deom_minus1 do not exist, their values ​​can be inferred to be 0.

[0197] The variable nnpfcOutputPicHeight can represent the height of the luminance sample array of the image generated by applying the NNPF identified by nnpfc_id to the input image. nnpfcOutputPicHeight can be derived as shown in Equation 5 below.

[0198] [Equation 5]

[0199] Here, (nnpfcOutputPicHeight%outSubHeightC) can be restricted to having a value of 0.

[0200] When nnpfc_pic_width_num_minus1, nnpfc_pic_width_deom_minus1, nnpfc_pic_height_num_minus1, and nnpfc_pic_height_deom_minus1 exist, at least one of the following conditions should be true.

[0201] The value of nnpfcOutputPicWidth is different from that of CroppedWidth.

[0202] The value of nnpfcOutputPicHeight is different from CroppedHeight.

[0203] Width ratio and height ratio information can be adaptively transmitted using signals based on the ResolutionResamingFlag. The ResolutionResamingFlag can be related to whether the purpose information indicates that NNPF will include resolution resampling. The ResolutionResamplingFlag can be derived based on the purpose information. For example, the ResolutionResamplingFlag can be derived as shown in Equation 6 below.

[0204] [Equation 6]

[0205] The NNPFC SEI message may include interpolated image count information (nnpfc_interpolated_pics). nnpfc_interpolated_pics specifies the number of interpolated images generated by NNPF between the i-th image and the (i+1)-th image used as input to NNPF. The value of nnpfc_interpolated_pics[i] can be restricted to fall within the range of 0 to 63. The value of nnpfc_interpolated_pics[i] can also be restricted to have a value greater than 0 for at least one of the values ​​of i in the range of 0 to (nnpfc_num_input_pics_minus1-1).

[0206] The `nnpfc_interpolated_pics` can be adaptively signaled based on the `PictureRateUpsamplingFlag`. The `PictureRateUpsamplingFlag` can be related to whether the purpose information indicates that NNPF's purpose will include picture rate upsampling. The `PictureRateUpsamplingFlag` can be derived based on the purpose information. For example, the `PictureRateUpsamplingFlag` can be derived as shown in Equation 7 below.

[0207] [Equation 7]

[0208] The NNPFC SEI message can include information about the number of extrapolated pictures (nnpfc_extrapolated_pics_minus1). The value obtained by incrementing nnpfc_extrapolated_pics_minus1 by 1 specifies the number of extrapolated pictures generated by NNPF after all input pictures in output order. The value of nnpfc_extrapolated_pics_minus1 can be restricted to fall within the range of 0 to 62.

[0209] You can export a variable (NumInpPicsInOutputTensor) that specifies the number of images in the output tensor of NNPF and has a corresponding input image, a variable (InpIdx[idx]) that specifies the index of the input image of the idx-th image in the output tensor of NNPF and has a corresponding input image, and a variable (numOutputPics) that specifies the total number of images in the output tensor of NNPF, as shown in Table 8 below.

[0210] [Table 8]

[0211] The `nnpfc_extrapolated_pics_minus1` signal can be adaptively sent based on the `TemporalExtrapolationFlag`. The `TemporalExtrapolationFlag` can be related to whether the destination information indicates that the NNPF destination will include time extrapolation. The `TemporalExtrapolationFlag` can be derived based on the destination information. For example, the `TemporalExtrapolationFlag` can be derived as shown in Equation 8 below.

[0212] [Equation 8]

[0213] NNPFC SEI messages may include extrapolation region information (nnpfc_spatial_extrapolation_left_offset, nnpfc_spatial_extrapolation_right_offset, nnpfc_spatial_extrapolation_top_offset, and nnpfc_spatial_extrapolation_bottom_offset).

[0214] Extrapolation region information can specify the spatial extrapolation region. The horizontal image coordinates are (outSubWidthC...). nnpfc_spatial_extrapolation_left_offset) to (nnpfcOutputPicWidth-(outSubWidthC nnpfc_spatial_extrapolation_right_offset) and the vertical image coordinates are (outSubWidthC) nnpfc_spatial_extrapolation_top_offset) to (nnpfcOutputPicWidth - (outSubWidthC The brightness samples of `nnpfc_spatial_extrapolation_bottom_offset` can correspond to spatial regions of the input image. The values ​​of `nnpfc_spatial_extrapolation_left_offset`, `nnpfc_spatial_extrapolation_right_offset`, `nnpfc_spatial_extrapolation_top_offset`, and `nnpfc_spatial_extrapolation_bottom_offset` can be restricted to fall within the range of 0 to 65,536. At least one of `nnpfc_spatial_extrapolation_left_offset`, `nnpfc_spatial_extrapolation_right_offset`, `nnpfc_spatial_extrapolation_top_offset`, and `nnpfc_spatial_extrapolation_bottom_offset` can be restricted to have a value greater than 0.

[0215] The NNPFC SEI message may include a presence flag (nnpfc_spatial_extrapolation_prompt_present_flag). nnpfc_spatial_extrapolation_prompt_present_flag may be associated with a presence prompt message (nnpfa_prompt).

[0216] For example, a value of 1 for `nnpfc_spatial_extrapolation_prompt_present_flag` indicates the presence of `nnpfc_prompt`. Similarly, a value of 1 for `nnpfc_spatial_extrapolation_prompt_present_flag` indicates the presence of zero-bit information (`nnpfc_alignment_zero_bit_c`). A value of 0 for `nnpfc_spatial_extrapolation_prompt_present_flagdml` indicates the absence of `nnpfc_prompt`. Furthermore, a value of 0 for `nnpfc_spatial_extrapolation_prompt_present_flag` indicates the absence of `nnpfc_alignment_zero_bit_c`.

[0217] The NNPFA SEI message may include zero-bit information (nnpfc_alignment_zero_bit_c). nnpfc_alignment_zero_bit_c can be restricted to a value of 0.

[0218] The NNPFA SEI message can include a prompt message (nnpfc_prompt). nnpfc_prompt can specify a text string prompt for the content used to generate the spatially extrapolated image region. When the value of nnpfc_spatial_extrapolation_prompt_present_flag is 1, nnpfc_prompt can be restricted to an empty string.

[0219] When NNPFC is activated by NNPFA, based on nnpfa_spatial_extrapolation_prompt_update_flag defined in NNPFA being 1 (true), nnpfc_spatial_extrapolation_prompt_present_flag and nnpfc_prompt defined in NNPFC can be replaced by nnpfa_spatial_extrapolation_prompt_present_flag and nnpfa_prompt defined in NNPFA, respectively.

[0220] At least one of nnpfc_alignment_zero_bit_c or nnpfc_prompt can be adaptively sent with a signal based on nnpfc_spatial_extrapolation_prompt_present_flag.

[0221] For example, `nnpfc_alignment_zero_bit_c` can be signaled with a value of 1 for `nnpfc_spatial_extrapolation_prompt_present_flag`. `nnpfc_alignment_zero_bit_c` can be signaled without a value of 0 for a value of 0 for a value of 1 for a value of 1 for a value of 0 for a value of 0 for a value of 0 for a value of 0 for a value of 0 for a value of 0 for a value of 1 for a value of 1 for a value of 0 for a value of 0 for a value of 1 ...

[0222] It is possible to adaptively send at least one of nnpfc_alignment_zero_bit_c or nnpfc_prompt based on the SpatialExtrapolationFlag.

[0223] For example, `nnpfc_alignment_zero_bit_c` can be signaled with a value of 1 for `SpatialExtrapolationFlag`. `nnpfc_alignment_zero_bit_c` can be signaled without a value of 0 for a value of 0 for a value of 1 for a value of 1 for a value of 0 ...

[0224] Even when the value of SpatialExtrapolationFlag is 1, either nnpfc_alignment_zero_bit_c or nnpfc_prompt can be used without being signaled based on the value of nnpfc_spatial_extrapolation_prompt_present_flag being 0, and can be used with a signal based on the value of nnpfc_spatial_extrapolation_prompt_present_flag being 1.

[0225] SpatialExtrapolationFlag can be related to whether the purpose information indicates that the purpose of NNPF will include spatial extrapolation. SpatialExtrapolationFlag can be derived based on the purpose information. For example, SpatialExtrapolationFlag can be derived as shown in Equation 9 below.

[0226] [Equation 9]

[0227] The NNPFC SEI message may include a last flag (nnpfc_component_last_flag). When nnpfc_component_last_flag is 1, it indicates that the last dimension of the input tensor (inputTensor) used for NNPF and the output tensor (outputTensor) generated from NNPF are used for the current channel. When nnpfc_component_last_flag is 0, it indicates that the third dimension of the input tensor (inputTensor) used for NNPF and the output tensor (outputTensor) generated from NNPF are used for the current channel.

[0228] The NNPFC SEI message may include an input format indicator (nnpfc_inp_format_idc). nnpfc_inp_format_idc can indicate the method used to convert sample values ​​of the input image into NNPF input values.

[0229] When nnpfc_inp_format_idc is 0, the input values ​​of NNPF are real numbers, and the functions InpY() and InpC() can be specified as shown in Table 9 below.

[0230] [Table 9]

[0231] When nnpfc_inp_format_idc is 1, the input value of NNPF is an unsigned integer, and the functions InpY() and InpC() can be specified as shown in Table 10 below.

[0232] [Table 10]

[0233] The variable inpTensorBitDepthY can be derived from nnpfc_inp_tensor_luma_bitdepth_minus8, which will be described later. The variable inpTensorBitDepthC can be derived from nnpfc_inp_tenson_chroma_bitdepth_minus8, which will be described later.

[0234] The NNPFC SEI message may include an auxiliary input indicator (nnpfc_auxiliary_inp_idc). When the value of nnpfc_auxiliary_inp_idc is greater than 0, it indicates the presence of auxiliary input data in the NNPF input tensor. When the value of nnpfc_auxiliary_inp_idc is 0, it indicates the absence of auxiliary input data in the input tensor. When the value of nnpfc_auxiliary_inp_idc is 1, it indicates that auxiliary input data is derived based on a predefined equation. The value of nnpfc_auxiliary_inp_idc can be restricted to fall within the range of 0 to 1.

[0235] The NNPFC SEI message may include a sorting method indicator (nnpfc_inp_order_idc). nnpfc_inp_order_idc can represent the method used to sort the sample array of the input image to form the input tensor for NNPF. The value of nnpfc_inp_order_idc can be restricted to fall within the range of 0 to 3.

[0236] When ChromaFormatIdc is not 1, it can restrict nnpfc_inp_order_idc to a value of 3. When ChromaFormatIdc is 0, it can restrict nnpfc_inp_order_idc to a value of 0. When chromaUpsampingFlag is 1, it can restrict nnpfc_inp_order_idc to a value of 0.

[0237] Table 11 below describes the values ​​of nnpfc_inp_order_idc. Figure 5The illustration shows an example of deriving four luminance channels (right) from the luminance component (left) when the value of nnpfc_inp_order_idc is 3.

[0238] [Table 11]

[0239] The NNPFU SEI message can update the attributes of the target NNPF. Here, the target NNPF can be identified based on the target identifier (nnpfu_target_id) used for post-processing filtering of the image.

[0240] As an example, the NNPFU SEI message according to this disclosure can be configured as shown in Table 12 below.

[0241] [Table 12]

[0242] The NNPFU SEI message may include a target identifier (nnpfu_target_id). nnpfu_target_id can represent the nnpfc_id of the target NNPF. Here, the target NNPF can be specified based on one or more NNPFC SEI messages associated with the current image. The target NNPF can have the same nnpfc_id as nnpfu_target_id. The value of nnpfu_target_id can be restricted to falling within the range of 0 to (2^3). 32 Within the range of -2).

[0243] The NNPFU SEI message may include a presence flag (nnpfu_spatial_extrapolation_present_flag). The npfu_spatial_extrapolation_present_flag may be associated with a presence prompt (nnpfu_prompt).

[0244] For example, a value of 1 for `nnpfu_spatial_extrapolation_prompt_present_flag` indicates the presence of `nnpfu_prompt`. Similarly, a value of 1 for `nnpfu_spatial_extrapolation_prompt_present_flag` indicates the presence of zero-bit information (`nnpfu_alignment_zero_bit_c`). A value of 0 for `nnpfu_spatial_extrapolation_prompt_present_flag` indicates the absence of `nnpfu_prompt`. Furthermore, a value of 0 for `nnpfu_spatial_extrapolation_prompt_present_flag` indicates the absence of `nnpfu_alignment_zero_bit_c`.

[0245] The `nnpfu_spatial_extrapolation_prompt_present_flag` can be related to whether the spatial extrapolation hint is included in the NNPFC identified by `nnpfa_target_id`. The value of `nnpfu_spatial_extrapolation_prompt_present_flag` can be 1 only if a spatial extrapolation hint is present.

[0246] The NNPFU SEI message may include zero-bit information (nnpfu_alignment_zero_bit_c). nnpfu_alignment_zero_bit_c can be restricted to a value of 0.

[0247] The NNPFU SEI message can include a prompt message (nnpfu_prompt). nnpfu_prompt can specify a text string prompt for the content used to generate the spatially extrapolated image region. When the value of nnpfu_spatial_extrapolation_prompt_present_flag is 1, nnpfu_prompt can be restricted to an empty string.

[0248] At least one of nnpfu_alignment_zero_bit_c or nnpfu_prompt can be adaptively sent with a signal based on nnpfu_spatial_extrapolation_prompt_present_flag.

[0249] For example, `nnpfu_alignment_zero_bit_c` can be signaled based on a value of 1 for `nnpfu_spatial_extrapolation_prompt_present_flag`. `nnpfu_alignment_zero_bit_c` can be signaled without a value of 0 for a value of 0 for a value of 1 for a value of 1 for a value of 0 for a value of 0 for a value of 0 for a value of 0 for a value of 0 for a value of 1 for a value of 1 for a value of 0 for a value of 0 for a value of 1 ...

[0250] The NNPF-related information according to this disclosure can be configured in the SEI message of the bitstream. The SEI message can be included in the Network Abstraction Layer (NAL) unit of the bitstream. However, it is not limited thereto. As an example, the NNPF-related information according to this disclosure can be configured in the high-level syntax of the bitstream. Here, the high-level syntax can be at least one of Sequence Parameter Set (SPS), Picture Parameter Set (PPS), Picture Header (PH), or Slice Header (SH). Alternatively, the NNPF-related information according to this disclosure can also be defined as a separate NAL unit type within the bitstream.

[0251] Figure 6 The illustration shows a schematic configuration of a decoding apparatus 300 that performs a method for reconstructing video images according to the present disclosure.

[0252] refer to Figure 6 The decoding device 300 may include a receiver 600, a video information extractor 610, and a video reconstructor 620.

[0253] Receiver 600 can receive bitstreams including encoded video images.

[0254] The video information extractor 610 can extract video information related to the encoded video images from the bitstream. Additionally, the video information extractor 610 can extract NNPF-related information from the bitstream, which, along with information obtained through reference... Figure 4 The description is the same.

[0255] The video reconstructor 620 can reconstruct encoded video images based on extracted video information.

[0256] Figure 7The figure illustrates a method for generating a bitstream performed by an encoding device 200 according to the present disclosure.

[0257] The S700 can receive video images to be encoded.

[0258] The received video images can be encoded to generate video information S710 associated with the video images.

[0259] The S720 can generate a bitstream that includes video information related to video images.

[0260] Additionally, NNPF-related information applied to the bitstream can be generated, which is consistent with information obtained by referencing... Figure 4 The description is the same. The generated NNPF-related information can be included in the bitstream.

[0261] Figure 8 The figure shows a schematic configuration of an encoding apparatus 200 for performing a method for generating a bitstream according to the present disclosure.

[0262] refer to Figure 8 The encoding device 200 may include a receiver 800, a video compressor 810, and a bitstream generator 820.

[0263] The receiver 800 can receive one or more video images to be encoded.

[0264] The video compressor 810 can encode one or more received video images to generate video information associated with the video images. The video compressor 810 can generate NNPF-related information applied to the bitstream.

[0265] Bitstream generator 820 can generate a bitstream that includes video information. Bitstream generator 820 can further generate a bitstream that includes information related to the generated NNPF.

[0266] In the above embodiments, the method is described as a series of steps or blocks based on the flowchart. However, the corresponding embodiments are not limited to the order of the steps, and some steps may occur simultaneously or in a different order than the other steps described above. Furthermore, those skilled in the art will understand that the steps shown in the flowchart are not exclusive, and other steps may be included or one or more steps in the flowchart may be deleted without affecting the scope of the embodiments of this disclosure.

[0267] The methods described above according to embodiments of the present disclosure can be implemented in software, and the encoding and / or decoding apparatus according to the present disclosure can be included in devices performing image processing, such as TVs, computers, smartphones, set-top boxes, display devices, etc.

[0268] In this disclosure, when the embodiments are implemented as software, the above methods can be implemented as modules (processes, functions, etc.) performing the above functions. Modules can be stored in memory and can be executed by a processor. Memory can be located inside or outside the processor and can be connected to the processor by various well-known means. The processor may include application-specific integrated circuits (ASICs), another chipset, logic circuits, and / or data processing devices. Memory may include read-only memory (ROM), random access memory (RAM), flash memory, memory cards, storage media, and / or other storage devices. In other words, the embodiments described herein can be implemented on a processor, microprocessor, controller, or chip. For example, the functional units shown in each figure can be implemented on a computer, processor, microprocessor, controller, or chip. In this case, information for implementation (e.g., information about instructions) or algorithms can be stored in a digital storage medium.

[0269] Furthermore, the decoding and encoding devices using embodiments of this disclosure can be included in multimedia broadcasting transmitting and receiving devices, mobile communication terminals, home theater video devices, digital cinema video devices, surveillance cameras, video conferencing devices, real-time communication devices such as video communication, mobile streaming devices, storage media, cameras, devices for providing video-on-demand (VoD) services, over-the-top (OTT) devices, devices for providing internet streaming services, three-dimensional (3D) video devices, virtual reality (VR) devices, augmented reality (AR) devices, videophone video devices, transportation terminal devices (e.g., vehicle (including autonomous vehicles) terminals, aircraft terminals, ship terminals, etc.), and medical video devices, and can be used to process video signals or data signals. For example, over-the-top (OTT) devices can include game consoles, Blu-ray players, networked TVs, home theater systems, smartphones, tablets, digital video recorders (DVRs), etc.

[0270] Furthermore, the processing methods applying embodiments of this disclosure can be generated in the form of a computer-executable program and can be stored in a computer-readable recording medium. Multimedia data having data structures according to embodiments of this disclosure can also be stored in a computer-readable recording medium. Computer-readable recording media include all types of storage devices and distributed storage devices that store computer-readable data. Computer-readable recording media can include, for example, Blu-ray discs (BD), Universal Serial Bus (USB), ROM, PROM, EPROM, EEPROM, RAM, CD-ROM, magnetic tape, floppy disks, and optical media storage devices. Additionally, computer-readable recording media include media implemented in carrier wave form (e.g., transmitted via the Internet). Furthermore, bitstreams generated by encoding methods can be stored in a computer-readable recording medium or transmitted via wired or wireless communication networks.

[0271] Furthermore, the embodiments of this disclosure can be implemented by a computer program product using program code, and this program code can be executed on a computer by the embodiments of this disclosure. The program code can be stored on a computer-readable medium.

[0272] Figure 9 Examples of content streaming systems to which embodiments of the present disclosure may be applied are shown.

[0273] refer to Figure 9 The content streaming system using embodiments of this disclosure may mainly include an encoding server, a streaming server, a web server, media storage, user equipment, and multimedia input devices.

[0274] An encoding server generates a bitstream by compressing content input from multimedia input devices such as smartphones, cameras, and camcorders into digital data, and then sends it to a streaming server. As another example, when multimedia input devices such as smartphones, cameras, and camcorders generate bitstreams directly, the encoding server can be omitted.

[0275] A bitstream can be generated by applying the encoding method or bitstream generation method of the embodiments of this disclosure, and the streaming server can temporarily store the bitstream during the sending or receiving of the bitstream.

[0276] A streaming server sends multimedia data to a user's device via a web server based on the user's request, and the web server acts as a medium to notify the user what services are available. When a user requests a service from the web server, the web server delivers it to the streaming server, and the streaming server sends the multimedia data to the user. In this scenario, the content streaming system may include a separate control server, which in this case controls the commands / responses between each device in the content streaming system.

[0277] A streaming server can receive content from media storage and / or encoding servers. For example, when receiving content from an encoding server, the content can be received in real time. In this case, to provide a smooth streaming service, the streaming server can store a bitstream for a certain period of time.

[0278] Examples of user equipment may include mobile phones, smartphones, laptops, digital broadcast terminals, personal digital assistants (PDAs), portable multimedia players (PMPs), navigation devices, tablet PCs, tablet computers, ultrabooks, wearable devices (e.g., smartwatches, smart glasses, head-mounted displays (HMDs), digital TVs, desktop computers, digital signage, etc.).

[0279] In a content streaming system, each server can be operated as a distributed server, and in this case, data received from each server can be distributed and processed.

[0280] The claims set forth herein can be combined in various ways. For example, the technical features of the method claims of this disclosure can be combined and implemented as a device, and the technical features of the device claims of this disclosure can be combined and implemented as a method. Furthermore, the technical features of the method claims and the technical features of the device claims of this disclosure can be combined and implemented as a device, and the technical features of the method claims and the technical features of the device claims of this disclosure can be combined and implemented as a method.

Claims

1. A method comprising: Receive bitstreams including encoded video images; as well as Reconstruct the encoded video images included in the bitstream. The bitstream is configured to include information related to the Neural Network Post-Filter (NNPF). The NNPF-related information includes at least one of the following: Neural Network Post-Filter Activation (NNPFA) Supplemental Enhancement Information (SEI) message, Neural Network Post-Filter Feature (NNPFC) SEI message, or Neural Network Post-Filter Update (NNPFU) SEI message. Specifically, the NNPF-related information is obtained from the Network Abstraction Layer (NAL) unit of the bitstream.

2. The method according to claim 1, wherein, The NNPFA SEI message includes a prompt message, and The prompt information specifies a text string prompt for the content of the spatial extrapolation image region.

3. The method according to claim 2, wherein, The NNPFA SEI message further includes a presence flag. The presence flag, with a value of 1, indicates the presence of the prompt information and may include zero-bit information. The presence flag with a value of 0 indicates that the prompt information and the zero-bit information do not exist.

4. The method according to claim 3, wherein, Based on the presence of the aforementioned prompt flag, the prompt information is adaptively sent using a signal.

5. The method according to claim 2, wherein, The NNPFA SEI message includes an update prompt flag, and The notification update flag is related to whether the notification information has been updated.

6. The method according to claim 5, wherein, Based on the aforementioned prompt update flag, the prompt information is adaptively sent using signals.

7. The method according to claim 1, wherein, The NNPFC SEI message includes a prompt message, and The prompt information specifies a text string prompt for the content of the spatial extrapolation image region.

8. The method according to claim 7, wherein, The NNPFC SEI message further includes a presence flag. The presence flag, with a value of 1, indicates the presence of the prompt information and may include zero-bit information. The presence flag with a value of 0 indicates that the prompt information and the zero-bit information do not exist.

9. The method according to claim 8, wherein, Based on the presence of the aforementioned prompt flag, the prompt information is adaptively sent using a signal.

10. The method according to claim 1, wherein, The NNPFU SEI message includes a prompt message, and The prompt information specifies a text string prompt for the content of the spatial extrapolation image region.

11. The method according to claim 10, wherein, The NNPFU SEI message further includes a presence flag. The presence flag, with a value of 1, indicates the presence of the prompt information and may include zero-bit information. The presence flag with a value of 0 indicates that the prompt information and the zero-bit information do not exist.

12. The method according to claim 11, wherein, Based on the presence of the aforementioned prompt flag, the prompt information is adaptively sent using a signal.

13. A method comprising: Receive the video and images to be encoded; The received video images are encoded to generate video information related to the video images; Information related to the neural network filter (NNPF) after generation; as well as Generate a bitstream that includes the video information and the NNPF-related information. The NNPF-related information includes at least one of the following: Neural Network Post-Filter Activation (NNPFA) Supplemental Enhancement Information (SEI) message, Neural Network Post-Filter Feature (NNPFC) SEI message, or Neural Network Post-Filter Update (NNPFU) SEI message. The NNPF-related information is encoded into the Network Abstraction Layer (NAL) unit of the bitstream.

14. A computer-readable storage medium for storing a bit stream generated by the method of claim 13.

15. A method comprising: Generate a bitstream, wherein the bitstream is generated based on: receiving a video image to be encoded, encoding the received video image to generate video information associated with the video image, and generating neural network post-filter (NNPF) related information; and Send data including the bit stream. The NNPF-related information includes at least one of the following: Neural Network Post-Filter Activation (NNPFA) Supplemental Enhancement Information (SEI) message, Neural Network Post-Filter Feature (NNPFC) SEI message, or Neural Network Post-Filter Update (NNPFU) SEI message. The NNPF-related information is encoded into the Network Abstraction Layer (NAL) unit of the bitstream.

Images