Image encoding / decoding method and apparatus and recording medium for storing bitstream

Abstract

The image encoding method and apparatus according to the present disclosure can receive a video picture to be encoded, encode the received video picture to generate a compressed video picture, generate encoder optimization information, and generate a bitstream including the compressed video picture and the encoder optimization information. The encoder optimization information can include size information of an original picture, and the size information of the original picture can be generated based on a flag indicating whether there is information related to the original picture.

Description

Technical Field

[0001] This disclosure relates to image encoding / decoding methods and apparatus, as well as recording media for storing bit streams. Background Technology

[0002] Recently, the demand for high-resolution and high-quality images, such as HD (high-definition) and UHD (ultra-high-definition) images, has been increasing in various application fields, and therefore, efficient image compression technology is being discussed.

[0003] There are various techniques, such as inter-frame prediction, which uses video compression technology to predict the pixel values ​​included in the current image from images before or after the current image; intra-frame prediction, which uses pixel information in the current image to predict the pixel values ​​included in the current image; and entropy coding, which assigns short codes to values ​​that occur frequently and long codes to values ​​that occur infrequently. These image compression techniques can be used to effectively compress image data and transmit or store it. Summary of the Invention

[0004] Technical issues

[0005] This disclosure provides a method and apparatus for configuring encoder optimization information.

[0006] This disclosure provides a method and apparatus for transmitting encoder optimization information using signals.

[0007] Technical solution

[0008] The image encoding method and apparatus of this disclosure can receive a video image to be encoded, encode the received video image to generate a compressed video image, generate encoder optimization information, and generate a bitstream including the compressed video image and the encoder optimization information.

[0009] In the image encoding method and apparatus according to this disclosure, the encoder optimization information may include the size information of the original image.

[0010] In the image encoding method and apparatus according to the present disclosure, the size information of the original image can be generated based on a flag indicating whether information related to the original image exists.

[0011] In the image encoding method and apparatus according to the present disclosure, the size information of the original image may include information representing the width of the original image and information representing the height of the original image.

[0012] In the image encoding method and apparatus according to the present disclosure, when the value of the flag is 1, information related to the original image may exist, and when the value of the flag is 0, information related to the original image may not exist.

[0013] In the image encoding method and apparatus according to the present disclosure, information representing the width of the original image and information representing the height of the original image can be generated based on the value of the flag being 1.

[0014] In the image encoding method and apparatus according to the present disclosure, based on the value of the flag being 0, information representing the width and height of the original image may not be generated.

[0015] In the image encoding method and apparatus according to the present disclosure, a spatial resampling type identifier can be generated based on the value of a flag being 0, which identifies the type of spatial resampling applied.

[0016] In the image coding method and apparatus according to the present disclosure, a flag can be generated based on a spatial resampling flag related to whether spatial resampling optimization is applied.

[0017] In the image coding method and apparatus according to this disclosure, spatial resampling flags can be derived based on EOI type information related to the attributes of the optimization method.

[0018] The image decoding method and apparatus according to this disclosure can receive a bitstream including encoded video images and reconstruct the encoded video images included in the bitstream. Here, the bitstream may include encoder optimization information, and the encoder optimization information may include the size information of the original image. The size information of the original image may be included based on a flag indicating whether there is information related to the original image.

[0019] A computer-readable digital storage medium is provided that stores encoded video / image information that enables the decoding apparatus according to the present disclosure to perform an image decoding method.

[0020] A computer-readable digital storage medium is provided according to the present disclosure for storing video / image information generated according to an image encoding method.

[0021] A method and apparatus for transmitting video / image information generated according to an image encoding method are provided according to the present disclosure.

[0022] Technical effect

[0023] According to this disclosure, the attributes of the applied spatial resampling can be identified by configuring the size information of the original image as encoder optimization information.

[0024] According to this disclosure, detailed information related to spatial resampling optimization can be effectively transmitted using signals. Attached Figure Description

[0025] Figure 1 A video / image encoding system according to this disclosure is shown.

[0026] Figure 2 A schematic block diagram illustrating an encoding apparatus to which embodiments of the present disclosure are applicable and to perform encoding of video / image signals is shown.

[0027] Figure 3 A schematic block diagram of a decoding apparatus to which embodiments of the present disclosure are applicable and to perform decoding of video / image signals is shown.

[0028] Figure 4 The figure illustrates a method for generating a bitstream performed by an encoding device 200 according to the present disclosure.

[0029] Figure 5 The figure shows a schematic configuration of an encoding apparatus 200 for performing a method for generating a bitstream according to the present disclosure.

[0030] Figure 6 The illustration shows a method for reconstructing a video image performed by a decoding device 300 according to the present disclosure.

[0031] Figure 7 The illustration shows a schematic configuration of a decoding apparatus 300 that performs a method for reconstructing video images according to the present disclosure.

[0032] Figure 8 Examples of content streaming systems to which embodiments of this disclosure can be applied are shown. Detailed Implementation

[0033] Because this disclosure can be modified in various ways and has several embodiments, specific embodiments will be illustrated in the accompanying drawings and described in detail in the detailed description. However, this disclosure is not intended to be limited to the specific embodiments and should be understood to include all variations, equivalents, and substitutions included within the spirit and scope of this disclosure. In describing each drawing, similar reference numerals are used for similar components.

[0034] Terms such as "first," "second," etc., may be used to describe various components, but components should not be limited by these terms. These terms are used only to distinguish one component from other components. For example, without departing from the scope of this disclosure, a first component may be referred to as a second component, and similarly, a second component may be referred to as a first component. Terms and / or combinations of any one or more related statement items are included.

[0035] When a component is described as "connected" or "linked" to another component, it should be understood that it can be directly connected or linked to another component, but there may also be another component in between. On the other hand, when a component is described as "directly connected" or "directly linked" to another component, it should be understood that there is no other component in between.

[0036] The terminology used in this application is for describing particular embodiments only and is not intended to limit this disclosure. Singular expressions include plural expressions unless the context clearly indicates otherwise. In this application, it should be understood that terms such as “comprising” or “having” are intended to designate the presence of features, numbers, steps, operations, components, portions, or combinations thereof described in the specification, but do not preclude the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, portions, or combinations thereof.

[0037] This disclosure relates to video / image coding. For example, the methods / exercises disclosed herein can be applied to methods disclosed in the Universal Video Coding (VVC) standard. Additionally, the methods / exercises disclosed herein can be applied to methods disclosed in the Basic Video Coding (EVC) standard, the AOMedia Video 1 (AV1) standard, the second-generation Audio Video Coding (AVS2) standard, or next-generation video / image coding standards (e.g., H.267 or H.268).

[0038] This specification presents various embodiments of video / image encoding, and unless otherwise stated, these embodiments may be combined with each other to perform the task.

[0039] Here, video can refer to a collection of images over time. An image generally refers to a unit representing an image within a specific time period, and a tile is a unit that forms part of an image during encoding. A tile can include at least one Code Tree Unit (CTU). An image can consist of at least one tile. A tile is a rectangular area consisting of multiple CTUs within a specific tile column and a specific tile row of an image. A tile column is a rectangular area of ​​CTUs with the same height as the image and a width assigned by the syntax requirements of the image parameter set. A tile row is a rectangular area of ​​CTUs with the same height assigned by the image parameter set and a width equal to the width of the image. CTUs within a tile can be arranged consecutively according to a CTU raster scan, and tiles within an image can be arranged consecutively according to a tile raster scan. A tile can include an integer number of complete tiles or an integer number of consecutive complete CTU rows that can be exclusively included within a single NAL unit of an image. Simultaneously, an image can be divided into at least two sub-images. A sub-image can be a rectangular area of ​​at least one tile within an image.

[0040] A pixel, cell, or pixel unit can refer to the smallest unit that makes up a picture (or image). Additionally, "sample" can be used as the term corresponding to a pixel. A sample can typically represent a pixel or pixel value, and can represent only the pixel / pixel value of the luminance component, or only the pixel / pixel value of the chrominance component.

[0041] A unit can represent a basic unit of image processing. A unit may include a specific region of an image and at least one of the information associated with that region. A unit may include a luminance block and two chrominance (e.g., Cb, cr) blocks. In some cases, the term "unit" may be used interchangeably with terms such as "block" or "region". In general, an MxN block may include a set (or array) of transform coefficients or samples (or sample arrays) consisting of M columns and N rows.

[0042] Here, "A or B" can refer to "A only", "B only", or "both A and B". In other words, "A or B" can be interpreted as "A and / or B". For example, "A, B or C" can refer to "A only", "B only", "C only", or "any combination of A, B and C".

[0043] The forward slash ( / ) or comma used in this article can refer to "and / or". For example, "A / B" can refer to "A and / or B". Therefore, "A / B" can refer to "A only", "B only", or "both A and B". For example, "A, B, C" can refer to "A, B, or C".

[0044] Here, "at least one of A and B" can refer to "only A", "only B" or "both A and B". Furthermore, expressions such as "at least one of A or B" or "at least one of A and / or B" can be interpreted in the same way as "at least one of A and B".

[0045] Additionally, here, "at least one of A, B, and C" can refer to "A only", "B only", "C only" or "any combination of A, B, and C". Furthermore, "at least one of A, B, or C" or "at least one of A, B, and / or C" can refer to "at least one of A, B, and C".

[0046] Additionally, the parentheses used in this document can refer to "for example". Specifically, when the indication is "prediction (intra-frame prediction)", "intra-frame prediction" can be cited as an example of "prediction". In other words, "prediction" here is not limited to "intra-frame prediction", and "intra-frame prediction" can be cited as an example of "prediction". Furthermore, even when the indication is "prediction (i.e., intra-frame prediction)", "intra-frame prediction" can be cited as an example of "prediction".

[0047] Here, a technical feature described individually in a single figure can be implemented individually or simultaneously.

[0048] Figure 1 A video / image encoding system according to this disclosure is shown.

[0049] refer to Figure 1 A video / image encoding system may include a first device (source device) and a second device (receiving device).

[0050] A source device can transmit encoded video / image information or data to a receiving device in the form of a file or stream via digital storage media or a network. The source device may include a video source, an encoding device, and a transmitting unit. The receiving device may include a receiving unit, a decoding device, and a renderer. The encoding device may be referred to as a video / image encoding device, and the decoding device may be referred to as a video / image decoding device. The transmitter may be included in the encoding device. The receiver may be included in the decoding device. The renderer may include a display unit, and the display unit may consist of a separate device or an external component.

[0051] A video source can acquire video / images through the process of capturing, compositing, or generating video / images. A video source can include devices for capturing video / images and devices for generating video / images. Devices for capturing video / images can include at least one camera, video / image archives containing previously captured video / images, etc. Devices for generating video / images can include computers, tablets, smartphones, etc., and can generate video / images (electronically). For example, virtual video / images can be generated by computers, etc., and in this case, the process of capturing video / images can be replaced by the process of generating related data.

[0052] An encoding device can encode input video / images. The encoding device can perform a series of processes such as prediction, transformation, and quantization for compression and encoding efficiency. The encoded data (encoded video / image information) can be output as a bitstream.

[0053] The transmitting unit can send encoded video / image information or data, output in bitstream form, to the receiving unit of the receiving device via digital storage media or a network, either as a file or a stream. Digital storage media can include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, SSD, etc. The transmitting unit can include elements for generating media files according to a predetermined file format and may include elements for transmission over a broadcast / communication network. The receiving unit can receive / extract the bitstream and send it to a decoding device.

[0054] Decoding devices can decode video / images by performing a series of processes, such as dequantization, inverse transform, and prediction, that correspond to the operations of encoding devices.

[0055] The renderer can render decoded video / images. The rendered video / images can be displayed through a display unit.

[0056] Figure 2 A schematic block diagram of an encoding apparatus that can be applied to embodiments of the present disclosure and perform encoding of video / image signals is shown.

[0057] refer to Figure 2 The encoding device 200 may consist of an image segmenter 210, a predictor 220, a residual processor 230, an entropy encoder 240, an adder 250, a filter 260, and a memory 270. The predictor 220 may include an inter-frame predictor 221 and an intra-frame predictor 222. The residual processor 230 may include a transformer 232, a quantizer 233, a dequantizer 234, and an inverse transformer 235. The residual processor 230 may further include a subtractor 231. The adder 250 may be referred to as a reconstructor or a reconstruction block generator. According to an embodiment, the image segmenter 210, predictor 220, residual processor 230, entropy encoder 240, adder 250, and filter 260 may be configured by at least one hardware component (e.g., an encoder chipset or processor). Additionally, the memory 270 may include a decoded picture buffer (DPB) and may be configured by a digital storage medium. The hardware component may further include the memory 270 as an internal / external component.

[0058] Image segmenter 210 can partition an input image (or picture, frame) input to encoding device 200 into at least one processing unit. As an example, a processing unit can be referred to as a coding unit (CU). In this case, the coding unit can be recursively partitioned from coding tree unit (CTU) or maximum coding unit (LCU) according to a quadtree-binary-tritree (QTBTTT) structure.

[0059] For example, a coding unit can be segmented into multiple coding units of greater depth based on a quadtree structure, a binary tree structure, and / or a ternary structure. In this case, for example, a quadtree structure can be applied first, and a binary tree structure and / or a ternary structure can be applied later. Alternatively, a binary tree structure can be applied before the quadtree structure. The coding process according to this specification can be performed based on the final coding unit that is no longer segmented. In this case, based on coding efficiency according to image characteristics, the largest coding unit can be directly used as the final coding unit, or if necessary, the coding unit can be recursively segmented into deeper coding units, and the coding unit with the optimal size can be used as the final coding unit. Here, the coding process can include processes such as prediction, transformation, and reconstruction, as described later.

[0060] As another example, the processing unit may further include a prediction unit (PU) or a transform unit (TU). In this case, the prediction unit and the transform unit may be divided or segmented from the aforementioned final encoding unit, respectively. The prediction unit may be a unit for predicting samples, and the transform unit may be a unit for deriving transform coefficients and / or a unit for deriving residual signals from transform coefficients.

[0061] In some cases, a unit can be used interchangeably with terms such as block or region. Generally, an MxN block can represent a set of transform coefficients or samples consisting of M columns and N rows. Samples can typically represent pixels or pixel values, and can represent only the pixel / pixel value of the luminance component, or only the pixel / pixel value of the chrominance component. Samples can be used as a term to correspond a picture (or image) to pixels or cells.

[0062] The encoding device 200 can subtract the prediction signal (prediction block, prediction sample array) output from the inter-frame predictor 221 or the intra-frame predictor 222 from the input image signal (original block, original sample array) to generate a residual signal (residual signal, residual sample array), and the generated residual signal is sent to the converter 232. In this case, the unit in the encoding device 200 that subtracts the prediction signal (prediction block, prediction sample array) from the input image signal (original block, original sample array) can be called the subtractor 231.

[0063] Predictor 220 can perform prediction on the block to be processed (hereinafter referred to as the current block) and generate a block of predictions including prediction samples for the current block. Predictor 220 can determine whether to apply intra-frame prediction or inter-frame prediction on a block or CU basis. Predictor 220 can generate various information about the prediction, such as prediction mode information, and send it to entropy encoder 240, as described later in the description of each prediction mode. The information about the prediction can be encoded in entropy encoder 240 and output as a bitstream.

[0064] Intra-predictor 222 can predict the current block by referencing samples within the current image. Depending on the prediction mode, the referenced samples can be located near the current block or at a specific distance away from the current block. In intra-prediction, the prediction mode can include at least one non-directional mode and multiple directional modes. The non-directional mode can include at least one of a DC mode or a planar mode. Depending on the level of detail in the prediction direction, the directional modes can include 33 or 65 directional modes. However, this is just an example, and more or fewer directional modes can be used depending on the configuration. Intra-predictor 222 can determine the prediction mode applied to the current block by using prediction modes applied to neighboring blocks.

[0065] Inter-frame predictor 221 can derive a prediction block for the current block based on a reference block (reference sample array) specified by motion vectors on a reference image. In this case, to reduce the amount of motion information transmitted in the inter-frame prediction mode, motion information can be predicted on a block, sub-block, or sample basis based on the correlation between motion information between neighboring blocks and the current block. Motion information may include motion vectors and reference image indices. Motion information may further include inter-frame prediction direction information (L0 prediction, L1 prediction, Bi prediction, etc.). For inter-frame prediction, neighboring blocks may include spatially neighboring blocks existing in the current image and temporally neighboring blocks existing in the reference image. The reference image including the reference block and the reference image including the temporally neighboring block may be the same or different. The temporally neighboring block may be referred to as a juxtaposed reference block, juxtaposed CU (colCU), etc., and the reference image including the temporally neighboring block may be referred to as a juxtaposed image (colPic). For example, inter-frame predictor 221 can configure a motion information candidate list based on neighboring blocks and generate information indicating which candidate is used to derive the motion vector and / or reference image index of the current block. Inter-frame prediction can be performed based on various prediction modes, and for example, for skip mode and merge mode, the inter-frame predictor 221 can use the motion information of neighboring blocks as the motion information of the current block. For skip mode, unlike merge mode, residual signals may not be sent. For motion vector prediction (MVP) mode, the motion vectors of neighboring blocks are used as motion vector predictors, and the motion vector difference is signaled to indicate the motion vector of the current block.

[0066] Predictor 220 can generate a prediction signal based on various prediction methods described later. For example, the predictor can not only apply intra-frame prediction or inter-frame prediction to predict a block, but also apply both intra-frame prediction and inter-frame prediction simultaneously. This can be referred to as a combined intra-frame and inter-frame prediction (CIIP) mode. Alternatively, the predictor can be based on an intra-block copy (IBC) prediction mode or a palette mode for prediction against a block. The IBC prediction mode or palette mode can be used for content image / video coding such as screen content coding (SCC) in games, etc. IBC essentially performs prediction within the current image, but it can be performed similarly to inter-frame prediction because it derives a reference block within the current image. In other words, IBC can use at least one of the inter-frame prediction techniques described herein. A palette mode can be considered an example of intra-frame coding or intra-frame prediction. When a palette mode is applied, sample values ​​within the image can be signaled based on information about the palette table and palette index. The prediction signal generated by predictor 220 can be used to generate a reconstructed signal or a residual signal.

[0067] Transformer 232 can generate transform coefficients by applying a transform technique to the residual signal. For example, the transform technique may include at least one of Discrete Cosine Transform (DCT), Discrete Sine Transform (DST), Karhunen-Loève Transform (KLT), Graphical Based Transform (GBT), or Conditional Nonlinear Transform (CNT). Here, GBT refers to the transform obtained from a graphic when the relationship information between pixels is expressed as a graphic. CNT refers to the transform obtained based on generating a prediction signal using all previously reconstructed pixels. Furthermore, the transform process can be applied to square pixel blocks of the same size or to non-square blocks of variable size.

[0068] Quantizer 233 can quantize the transform coefficients and send them to entropy encoder 240, which can encode the quantized signal (information about the quantized transform coefficients) and output it as a bitstream. The information about the quantized transform coefficients can be referred to as residual information. Quantizer 233 can rearrange the quantized transform coefficients in block form into a one-dimensional vector form based on the coefficient scan order, and can generate information about the quantized transform coefficients based on the one-dimensional vector form of the quantized transform coefficients.

[0069] The entropy encoder 240 can perform various encoding methods, such as exponential Golomb, context-adaptive variable-length coding (CAVLC), and context-adaptive binary arithmetic coding (CABAC). The entropy encoder 240 can encode information necessary for video / video image reconstruction (e.g., values ​​of syntax elements) in addition to the transform coefficients quantized together or individually.

[0070] Encoded information (e.g., encoded video / image information) can be transmitted or stored in bitstream form at the network abstraction layer (NAL) unit level. The video / image information may further include information about various parameter sets such as adaptive parameter sets (APS), picture parameter sets (PPS), sequence parameter sets (SPS), or video parameter sets (VPS). Additionally, the video / image information may further include general constraint information. Here, information transmitted from the encoding device / signaled to the decoding device and / or syntax elements can be included in the video / image information. The video / image information can be encoded by the above-described encoding process and included in the bitstream. The bitstream can be transmitted over a network or stored in a digital storage medium. Here, the network may include broadcast networks and / or communication networks, and the digital storage medium may include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, SSD, etc. Transmission units (not shown) for transmission and / or storage units (not shown) for storing signals output from the entropy encoder 240 can be configured as internal / external elements of the encoding device 200, or the transmission unit may also be included in the entropy encoder 240.

[0071] The quantized transform coefficients output from quantizer 233 can be used to generate a prediction signal. For example, the residual signal (residual block or residual sample) can be reconstructed by applying dequantization and inverse transform to the quantized transform coefficients using dequantizer 234 and inverse transformer 235. Adder 250 can add the reconstructed residual signal to the prediction signal output from inter-frame predictor 221 or intra-frame predictor 222 to generate a reconstructed signal (reconstructed image, reconstructed block, reconstructed sample array). When there is no residual for the block to be processed, such as when a skip mode is applied, the prediction block can be used as a reconstructed block. Adder 250 can be referred to as a reconstructor or reconstructed block generator. The generated reconstructed signal can be used for intra-frame prediction of the next block to be processed within the current image, and can also be used for inter-frame prediction of the next image by filtering, which will be described later. Meanwhile, a luminance mapping with chroma scaling (LMCS) can be applied during image encoding and / or reconstruction.

[0072] Filter 260 can improve subjective / objective image quality by applying filtering to the reconstructed signal. For example, filter 260 can generate a modified reconstructed image by applying various filtering methods to the reconstructed image, and the modified reconstructed image can be stored in memory 270, specifically in the DPB of memory 270. Various filtering methods can include deblocking filtering, sample adaptive offset, adaptive loop filtering, bilateral filtering, etc. Filter 260 can generate various information about the filtering and send it to entropy encoder 240. The information about the filtering can be encoded in entropy encoder 240 and output as a bitstream.

[0073] The modified reconstructed image sent to memory 270 can be used as a reference image in inter-frame predictor 221. When inter-frame prediction is applied through it, the encoding device can avoid prediction mismatch in encoding device 200 and decoding device, and can also improve encoding efficiency.

[0074] The DPB of memory 270 can store modified reconstructed images for use as reference images in inter-frame predictor 221. Memory 270 can store motion information of blocks from which motion information in the current image is derived (or encoded) and / or motion information of blocks in the pre-reconstructed image. The stored motion information can be sent to inter-frame predictor 221 to be used as motion information for spatially or temporally neighboring blocks. Memory 270 can store reconstructed samples of reconstructed blocks in the current image and send them to intra-frame predictor 222.

[0075] Figure 3 A schematic block diagram of a decoding apparatus that can be applied to embodiments of the present disclosure and perform decoding of video / image signals is shown.

[0076] refer to Figure 3 The decoding device 300 can be configured to include an entropy decoder 310, a residual processor 320, a predictor 330, an adder 340, a filter 350, and a memory 360. The predictor 330 may include an inter-frame predictor 332 and an intra-frame predictor 331. The residual processor 320 may include a dequantizer 321 and an inverse transformer 321.

[0077] According to an embodiment, the entropy decoder 310, residual processor 320, predictor 330, adder 340, and filter 350 described above can be configured by a single hardware component (e.g., a decoder chipset or processor). Additionally, the memory 360 may include a decoded image buffer (DPB) and can be configured by a digital storage medium. The hardware component may further include the memory 360 as an internal / external component.

[0078] When the input includes a bitstream containing video / image information, the decoding device 300 can respond to... Figure 2 The decoding device 300 reconstructs an image by processing video / image information in its encoding apparatus. For example, the decoding device 300 can derive units / blocks based on information related to block segmentation obtained from the bitstream. The decoding device 300 can perform decoding by using processing units applied in the encoding apparatus. Therefore, the processing unit for decoding can be an encoding unit, and the encoding unit can be segmented from encoding tree units or maximally encoded units according to a quadtree structure, binary tree structure, and / or ternary tree structure. At least one transform unit can be derived from the encoding unit. Furthermore, the reconstructed image signal decoded and output by the decoding device 300 can be played back by a playback device.

[0079] Decoding device 300 can receive data in bitstream form from... Figure 2 The signal output by the encoding device and the received signal can be decoded by the entropy decoder 310. For example, the entropy decoder 310 can parse the bitstream to derive information (e.g., video / image information) necessary for image reconstruction (or picture reconstruction). The video / image information may further include information about various parameter sets such as adaptive parameter sets (APS), picture parameter sets (PPS), sequence parameter sets (SPS), or video parameter sets (VPS). In addition, the video / image information may further include general constraint information. The decoding device can further decode the picture based on the information about the parameter sets and / or the general constraint information. The information sent / received by the signal and / or the syntax elements described later herein can be decoded and obtained from the bitstream through the decoding process. For example, the entropy decoder 310 can decode the information in the bitstream based on encoding methods such as exponential Golomb coding, CAVLC, CABAC, etc., and output the values ​​of the syntax elements necessary for image reconstruction and the quantized values ​​of the transform coefficients of the residuals. More specifically, the CABAC entropy decoding method can receive bins corresponding to each syntax element from the bitstream, determine a context model using information about the syntax element to be decoded, decoding information of neighboring blocks and the block to be decoded, or information about symbols / bins decoded in the previous step, perform arithmetic decoding on the bins by predicting the occurrence probability of the bins based on the determined context model, and generate symbols corresponding to the value of each syntax element. In this case, after determining the context model, the CABAC entropy decoding method can update the context model by using information about the decoded symbols / bins for the context model used for the next symbol / bin. Among the information decoded in the entropy decoder 310, information about prediction is provided to the predictors (inter-frame predictor 332 and intra-frame predictor 331), and the residual values ​​of entropy decoding performed on them in the entropy decoder 310, i.e., the quantized transform coefficients and related parameter information, can be input to the residual processor 320. The residual processor 320 can derive residual signals (residual blocks, residual samples, residual sample arrays). In addition, information about filtering among the information decoded in the entropy decoder 310 can be provided to the filter 350. Meanwhile, the receiving unit (not shown) that receives the signal output from the encoding device can be further configured as an internal / external element of the decoding device 300 or the receiving unit can be a component of the entropy decoder 310.

[0080] Furthermore, the decoding device according to this specification can be referred to as a video / image / picture decoding device, and the decoding device can be divided into an information decoder (video / image / picture information decoder) and a sample decoder (video / image / picture sample decoder). The information decoder may include an entropy decoder 310, and the sample decoder may include at least one of a dequantizer 321, an inverse transformer 322, an adder 340, a filter 350, a memory 360, an inter-frame predictor 332, and an intra-frame predictor 331.

[0081] Dequantizer 321 can dequantize the quantized transform coefficients and output the transform coefficients. Dequantizer 321 can rearrange the quantized transform coefficients into a two-dimensional block form. In this case, the rearrangement can be performed based on the coefficient scan order performed in the encoding device. Dequantizer 321 can obtain the transform coefficients by performing dequantization on the quantized transform coefficients using quantization parameters (e.g., quantization step size information).

[0082] The inverse transformer 322 performs an inverse transformation on the transformation coefficients to obtain the residual signal (residual block, residual sample array).

[0083] Predictor 320 can perform prediction on the current block and generate a prediction block including prediction samples for the current block. Predictor 320 can determine whether to apply intra-frame prediction or inter-frame prediction to the current block based on the prediction information output from entropy decoder 310, and determine a specific intra-frame / inter-frame prediction mode.

[0084] Predictor 320 can generate prediction signals based on various prediction methods described later. For example, predictor 320 can not only apply intra-frame prediction or inter-frame prediction to predict a block, but also apply intra-frame prediction and inter-frame prediction simultaneously. This can be referred to as a combined intra-frame and inter-frame prediction (CIIP) mode. Alternatively, the predictor can be based on an intra-block copy (IBC) prediction mode or a palette mode for block prediction. The IBC prediction mode or palette mode can be used for content image / video coding such as screen content coding (SCC) in games, etc. IBC essentially performs prediction within the current frame, but it can be performed similarly to inter-frame prediction because it derives a reference block within the current frame. In other words, IBC can use at least one of the inter-frame prediction techniques described herein. Palette mode can be considered an example of intra-frame coding or intra-frame prediction. When a palette mode is applied, information about the palette table and palette index can be included in the video / image information and transmitted as a signal.

[0085] Intra-predictor 331 can predict the current block by referencing samples within the current image. Depending on the prediction mode, the referenced samples can be located near the current block or at a specific distance away from the current block. In intra-prediction, the prediction mode can include at least one non-directional mode and multiple directional modes. Intra-predictor 331 can determine the prediction mode applied to the current block by using prediction modes applied to neighboring blocks.

[0086] Inter-frame predictor 332 can derive a prediction block for the current block based on a reference block (reference sample array) specified by motion vectors on a reference image. In this case, to reduce the amount of motion information transmitted in the inter-frame prediction mode, motion information can be predicted on a block, sub-block, or sample basis based on the correlation between motion information of neighboring blocks and the current block. Motion information may include motion vectors and reference image indices. Motion information may further include inter-frame prediction direction information (L0 prediction, L1 prediction, Bi prediction, etc.). For inter-frame prediction, neighboring blocks may include spatially neighboring blocks existing in the current image and temporally neighboring blocks existing in the reference image. For example, inter-frame predictor 332 can configure a motion information candidate list based on neighboring blocks and derive the motion vector and / or reference image index of the current block based on received candidate selection information. Inter-frame prediction can be performed based on various prediction modes, and information about the prediction may include information indicating the inter-frame prediction mode used for the current block.

[0087] Adder 340 can add the obtained residual signal to the prediction signal (prediction block, prediction sample array) output from the predictor (including inter-frame predictor 332 and / or intra-frame predictor 331) to generate a reconstruction signal (reconstructed image, reconstruction block, reconstruction sample array). When there is no residual for the block to be processed, such as when a skip mode is applied, the prediction block can be used as the reconstruction block.

[0088] Adder 340 can be referred to as a reconstructor or reconstructed block generator. The generated reconstructed signal can be used for intra-frame prediction of the next block to be processed in the current image, output by filtering as described later, or it can be used for inter-frame prediction of the next image. Meanwhile, a luminance map with chroma scaling (LMCS) can be applied during image decoding.

[0089] Filter 350 can improve subjective / objective image quality by applying filtering to the reconstructed signal. For example, filter 350 can generate a modified reconstructed image by applying various filtering methods to the reconstructed image and send the modified reconstructed image to memory 360, specifically the DPB of memory 360. Various filtering methods can include deblocking filtering, adaptive sampling offset, adaptive loop filtering, bilateral filtering, etc.

[0090] The (modified) reconstructed image stored in the DPB of memory 360 can be used as a reference image in inter-frame predictor 332. Memory 360 can derive (or decode) motion information of blocks from its current image and / or motion information of blocks in the pre-reconstructed image. The stored motion information can be sent to inter-frame predictor 332 as motion information for spatially or temporally neighboring blocks. Memory 360 can store reconstructed samples of reconstructed blocks in the current image and send them to intra-frame predictor 331.

[0091] Here, the embodiments described in the encoding device 200’s filter 260, inter-frame predictor 221 and intra-frame predictor 222 can also be applied equally or correspondingly to the decoding device 300’s filter 350, inter-frame predictor 332 and intra-frame predictor 331, respectively.

[0092] Figure 4 The figure illustrates a method for generating a bitstream performed by an encoding device 200 according to the present disclosure.

[0093] It can receive video images to be encoded using the S400.

[0094] The received video images can be encoded to generate compressed video images S410.

[0095] Encoder optimization information (EOI) can be generated in S420.

[0096] The encoder optimization information disclosed herein can be related to received or compressed video images. The encoder optimization information can define the purpose, attributes, and scope (or target) of encoder optimization.

[0097] As an example, encoder optimization information may include an EOI cancellation flag (eoi_cancel_flag). eoi_cancel_flag can be associated with the persistence of encoder optimization information. When eoi_cancel_flag is 1, it indicates that the persistence of previously applied encoder optimization information is cancelled. For example, when eoi_cancel_flag is 1, it indicates that the persistence of encoder optimization information included in previous prediction units (PUs) in output order is cancelled. When eoi_cancel_flag is 0, it indicates that optimization-related information applied during preprocessing or encoding follows.

[0098] Optimization-related information may include at least one of the following: EOI persistence flag (eoi_persistence_flag), EOI identifier for human viewing (eoi_for_human_viewing_idc), EOI identifier for machine analysis (eoi_for_machine_analysis_idc), or EOI type information (eoi_type). In other words, the above optimization-related information can be encoded based on the value of eoi_cancel_flag being 0.

[0099] `eoi_persistence_flag` relates to the persistence of optimization information. `eoi_persistence_flag` represents the persistence of optimization information as indicated by `eoi_type`. When `eoi_persistence_flag` is 0, it indicates that the optimization information (based on `eoi_type`) is applied only to the current image. When `eoi_persistence_flag` is 1, it indicates that the optimization information (based on `eoi_type`) is applied to the current image and all subsequent images. Here, "subsequent images" can refer to all subsequent images of the current layer in output order.

[0100] `eoi_for_human_viewing_idc` represents information used to identify the purpose and objective of the optimization. A value of 3 for `eoi_for_human_viewing_idc` likely means the optimization aims at human viewing. A value of 2 likely means the video is suitable for human viewing but has not been specifically optimized for it. A value of 1 likely means the video is not suitable for human viewing. A value of 0 likely means it is unknown whether the video is suitable for human viewing.

[0101] `eoi_for_machine_analysis_idc` represents information used to identify the purpose and objective of optimization. A value of 3 for `eoi_for_machine_analysis_idc` likely indicates that the optimization objective includes machine analysis. A value of 2 likely indicates that the video is suitable for machine analysis but has not been specifically optimized for it. A value of 1 likely indicates that the video is not suitable for machine analysis. A value of 0 likely indicates that it is unknown whether the video is suitable for machine analysis.

[0102] `eoi_type` can represent an attribute of an optimization method. For example, `eoi_type` can be defined as shown in Table 1 below. `eoi_type` can identify the optimization attribute (or optimization type) defined in Table 1. However, the optimization attributes defined in Table 1 are only examples, and new optimization attributes can also be defined under the corresponding structure. Alternatively, `eoi_type` can identify only a portion of the optimization attributes defined in Table 1. Other optimization attributes not defined in Table 1 can also be defined in addition to those in Table 1.

[0103] [Table 1]

[0104] When (eoi_type & bitMask) is not 0, it indicates that an optimization attribute with the bitmask value in Table 1 is applied. When eoi_type is greater than 0 and (eoi_type & bitMask) is 0, it indicates that an optimization attribute with the corresponding bitmask value is not applied. When eoi_type is 0, it indicates that an optimization determined by the application is used. For example, optimization attributes based on eoi_type can also be defined in Table 2 below.

[0105] [Table 2]

[0106] The encoder optimization information related to the objectives, attributes, and scope of application of the optimizations described above can also be applied in the same way to the embodiments described below. The encoder optimization information can be defined in the SEI message, as shown in Table 3 below.

[0107] [Table 3]

[0108] The applied optimization attributes can be derived based on the information in Table 1 or Table 2. For example, the spatial resampling flag (EoiSpatialResamplingFlag) can indicate whether spatial resampling optimization is applied. A value of 1 for EoiSpatialResamplingFlag likely means spatial resampling is applied. A value of 0 for EoiSpatialResamplingFlag likely means spatial resampling is not applied. In this way, based on eoi_type, the applied optimization attributes can be identified, and further details related to the applied optimization attributes can be defined in the encoder optimization information. The value of EoiSpatialResamplingFlag can be derived as shown in Equation 1 or 2 below.

[0109] [Formula 1]

[0110] EoiSpatialResamplingFlag = ((eoi_type & 0x04 ) > 0 ) ? 1 : 0

[0111] [Equation 2]

[0112] if ((eoi_type & 0x04 ) != 0)

[0113] EoiSpatialResamplingFlag= 1; else EoiSpatialResamplingFlag=0; This disclosure relates to methods for defining detailed information related to spatial resampling optimization.

[0114] A spatial resampling type identifier (eoi_spatial_resampling_type_idc) can be generated based on the value of EoiSpatialResamplingFlag being 1. eoi_spatial_resampling_type_idc can contain information that identifies the type of spatial resampling applied.

[0115] As an example, eoi_spatial_resampling_type_idc can be defined in Table 4 below.

[0116] [Table 4]

[0117] In Table 4, a value of 0 for `eoi_spatial_resampling_type_idc` indicates that the spatial resampling type is upsampling. A value of 1 indicates that the spatial resampling type is downsampling. A value of 2 indicates that the spatial resampling type is upsampling based on neural networks (e.g., AI, SR). A value of 3 indicates that the spatial resampling type is downsampling based on neural networks. Values ​​of 0 and 1 for `eoi_spatial_resampling_type_idc` may mean that it is not a neural network-based resampling method. Here, upsampling and downsampling are examples of spatial resampling, and other resampling methods can also be identified by `eoi_spatial_resampling_type_idc`.

[0118] eoi_spatial_resampling_type_idc can be encoded as a descriptor of u(3). In other words, eoi_spatial_resampling_type_idc can be encoded as an unsigned integer with 3 bits. According to Table 4, eoi_spatial_resampling_type_idc can be expressed as 3 bits, as shown in Table 5 below.

[0119] [Table 5]

[0120] As an example, eoi_spatial_resampling_type_idc can also be defined as shown in Table 6 below.

[0121] [Table 6]

[0122] In Table 6, a value of 0 for `eoi_spatial_resampling_type_idc` indicates that the spatial resampling type is upsampling. A value of 1 for `eoi_spatial_resampling_type_idc` indicates that the spatial resampling type is downsampling. In this case, as shown in Table 5, based on a value of 1 for `EoiSpatialResamplingFlag`, an additional neural network resampling flag (`eoi_nn_based_resampling_flag`) can be generated.

[0123] When the value of eoi_nn_based_resampling_flag is 1, it indicates that spatial resampling is based on a neural network. When the value of eoi_nn_based_resampling_flag is 0, it indicates that spatial resampling is not based on a neural network.

[0124] The eoi_nn_based_resampling_flag can be encoded after eoi_spatial_resampling_type_idc is encoded.

[0125] The `eoi_nn_based_resampling_flag` can be encoded as a descriptor of `u(1)`. In other words, the `eoi_nn_based_resampling_flag` can be encoded as an unsigned integer with 1 bit. The `eoi_spatial_resampling_type_idc` can be encoded as a descriptor of `u(3)`. In other words, the `eoi_spatial_resampling_type_idc` can be encoded as an unsigned integer with 3 bits. According to Table 6, the `eoi_spatial_resampling_type_idc` can be expressed as 3 bits, as shown in Table 7 below.

[0126] [Table 7]

[0127] As described in Tables 5 or 7, when the type of spatial resampling according to eoi_spatial_resampling_type_idc is unknown, it indicates that the determination of the type of spatial resampling is unclear. For example, when upsampling the width of an image and downsampling the height, the overall size of the image may not change. In this case, the spatial resampling attribute can be known from the size information of the original image. In order to reconstruct the decoded image at the original source resolution, it may be necessary to provide the size of the original image to a receiver such as a decoding device.

[0128] According to this disclosure, the size information of the original image can be defined as eoi_original_width and eoi_original_height. Here, eoi_original_width can be information representing the width of the input image (i.e., the original image) before spatial resampling is applied. eoi_original_height can be information representing the height of the input image (i.e., the original image) before spatial resampling is applied.

[0129] As an example, the size information of the original image can be defined in Table 8 below.

[0130] [Table 8]

[0131] According to Table 8, based on the value of EoiSpatialResamplingFlag being 1, eoi_spatial_resampling_type_idc and original_pic_info_present_flag can be generated respectively.

[0132] The eoi_spatial_resampling_type_idc is the same as above, and overlapping descriptions will be omitted here.

[0133] `original_pic_info_present_flag` can be a flag indicating whether information related to the input image (i.e., the original image) exists before spatial resampling is applied. A value of 1 for `original_pic_info_present_flag` likely means that information related to the original image exists. A value of 0 for `original_pic_info_present_flag` likely means that information related to the original image does not exist. Here, information related to the original image may include the size of the original image.

[0134] When the value of `original_pic_info_present_flag` is 1, `eoi_original_width` and `eoi_original_height` can be generated as the size information of the original image. On the other hand, when the value of `original_pic_info_present_flag` is 0, `eoi_original_width` and `eoi_original_height` are not generated.

[0135] As an example, the size information of the original image can also be defined in Table 9 below.

[0136] [Table 9]

[0137] The size information of the encoded video images can be included in the bitstream. For example, information representing the width of the encoded video images (sps_pic_width_max_in_luma_samples) and information representing the height of the encoded video images (sps_pic_height_max_in_luma_samples) can be included in the sequence parameter set (SPS).

[0138] The type of spatial resampling can also be identified by comparing the size information of the encoded video image with the size information of the original image. For example, when the size indicated by `eoi_original_width` is greater than the size indicated by `sps_pic_width_max_in_luma_samples`, the applied spatial resampling can be inferred to be upsampling. Conversely, when the size indicated by `eoi_original_width` is less than the size indicated by `sps_pic_width_max_in_luma_samples`, the applied spatial resampling can be inferred to be downsampling. Similarly, the type of spatial resampling can be identified by comparing the sizes of `eoi_original_height` and `sps_pic_height_max_in_luma_samples`.

[0139] Therefore, as defined in Table 9, based on a value of 1 for `original_pic_info_present_flag`, the original image size information (`eoi_original_width`, `eoi_original_height`) is generated, but `eoi_spatial_resampling_type_idc` may not be generated. In other words, based on a value of 1 for `original_pic_info_present_flag`, the original image size information can be encoded in the bitstream. Based on a value of 1 for `original_pic_info_present_flag`, `eoi_spatial_resampling_type_idc` may not be encoded in the bitstream.

[0140] On the other hand, based on a value of 0 for `original_pic_info_present_flag`, `eoi_spatial_resampling_type_idc` can be generated. In other words, based on a value of 0 for `original_pic_info_present_flag`, `eoi_spatial_resampling_type_idc` can be encoded in the bitstream. Therefore, when the size information of the original image is unavailable, `eoi_spatial_resampling_type_idc` can be encoded in the bitstream because it is unnecessary when the size information of the original image is available.

[0141] eoi_spatial_resampling_type_idc can be encoded as a descriptor of u(3). In other words, eoi_spatial_resampling_type_idc can be encoded as an unsigned integer with 3 bits. Here, original_pic_info_present_flag can be generated based on the value of EoiSpatialResamplingFlag being 1.

[0142] As an example, eoi_spatial_resampling_type_idc can also be defined in Table 10 below.

[0143] [Table 10]

[0144] In Table 10, a value of 0 for `eoi_spatial_resampling_type_idc` indicates that the spatial resampling type is upsampling. A value of 1 indicates that the spatial resampling type is downsampling. A value of 2 indicates that the spatial resampling type is upsampling based on neural networks (e.g., AI, SR). A value of 3 indicates that the spatial resampling type is downsampling based on neural networks. Values ​​of 0 and 1 for `eoi_spatial_resampling_type_idc` may mean that it is not a neural network-based resampling method. Here, upsampling and downsampling are examples of spatial resampling, and other resampling methods can also be identified by `eoi_spatial_resampling_type_idc`.

[0145] eoi_spatial_resampling_type_idc can be encoded as a descriptor of u(2). In other words, eoi_spatial_resampling_type_idc can be encoded as an unsigned integer with 2 bits. According to Table 10, eoi_spatial_resampling_type_idc can be expressed as 2 bits, as shown in Table 11 below.

[0146] [Table 11]

[0147] As an example, eoi_spatial_resampling_type_idc can also be defined in Table 12 below.

[0148] [Table 12]

[0149] In Table 12, a value of 0 for `eoi_spatial_resampling_type_idc` indicates that the spatial resampling type is upsampling. A value of 1 for `eoi_spatial_resampling_type_idc` indicates that the spatial resampling type is downsampling. In this case, based on a value of 1 for `EoiSpatialResamplingFlag` in Table 12, an additional neural network resampling flag (`eoi_nn_based_resampling_flag`) can be generated.

[0150] When the value of eoi_nn_based_resampling_flag is 1, it indicates that spatial resampling is based on a neural network. When the value of eoi_nn_based_resampling_flag is 0, it indicates that spatial resampling is not based on a neural network.

[0151] eoi_nn_based_resampling_flag can be encoded before eoi_spatial_resampling_type_idc is encoded.

[0152] The `eoi_nn_based_resampling_flag` can be encoded as a descriptor of `u(1)`. In other words, the `eoi_nn_based_resampling_flag` can be encoded as an unsigned integer with 1 bit. The `eoi_spatial_resampling_type_idc` can be encoded as a descriptor of `u(2)`. In other words, the `eoi_spatial_resampling_type_idc` can be encoded as an unsigned integer with 2 bits. According to Table 12, the `eoi_spatial_resampling_type_idc` can be expressed as 2 bits, as shown in Table 13 below.

[0153] [Table 13]

[0154] As described in Table 13, when the type of spatial resampling according to eoi_spatial_resampling_type_idc is unknown, it indicates that the determination of the type of spatial resampling is unclear. For example, when upsampling the width of an image and downsampling the height, the overall size of the image may not change. In this case, the spatial resampling attribute can be known from the size information of the original image.

[0155] According to this disclosure, the size information of the original image can be defined as eoi_original_width and eoi_original_height. Here, eoi_original_width can be information representing the width of the input image (i.e., the original image) before spatial resampling is applied. eoi_original_height can be information representing the height of the input image (i.e., the original image) before spatial resampling is applied.

[0156] As an example, the size information of the original image can be defined in Table 14 below.

[0157] [Table 14]

[0158] According to Table 14, based on the value of EoiSpatialResamplingFlag being 1, eoi_spatial_resampling_type_idc and original_pic_info_present_flag can be generated respectively.

[0159] eoi_spatial_resampling_type_idc is the same as above, and overlapping descriptions will be omitted here.

[0160] `original_pic_info_present_flag` can be a flag indicating whether information related to the input image (i.e., the original image) exists before spatial resampling is applied. A value of 1 for `original_pic_info_present_flag` likely means that information related to the original image exists. A value of 0 for `original_pic_info_present_flag` likely means that information related to the original image does not exist. Here, information related to the original image may include the size of the original image.

[0161] With `original_pic_info_present_flag` set to 1, `eoi_original_width` and `eoi_original_height` can be generated as size information for the original image. Conversely, with `original_pic_info_present_flag` set to 0, `eoi_original_width` and `eoi_original_height` are not generated as size information for the original image.

[0162] As an example, the size information of the original image can also be defined in Table 15 below.

[0163] [Table 15]

[0164] The size information of the encoded video images can be included in the bitstream. For example, information representing the width of the encoded video images (sps_pic_width_max_in_luma_samples) and information representing the height of the encoded video images (sps_pic_height_max_in_luma_samples) can be included in the sequence parameter set (SPS).

[0165] The type of spatial resampling can also be identified by comparing the size information of the encoded video image with the size information of the original image. For example, when the size indicated by `eoi_original_width` is greater than the size indicated by `sps_pic_width_max_in_luma_samples`, the applied spatial resampling can be inferred to be upsampling. Conversely, when the size indicated by `eoi_original_width` is less than the size indicated by `sps_pic_width_max_in_luma_samples`, the applied spatial resampling can be inferred to be downsampling. Similarly, the type of spatial resampling can be identified by comparing the sizes of `eoi_original_height` and `sps_pic_height_max_in_luma_samples`.

[0166] Therefore, as defined in Table 15, based on a value of 1 for `original_pic_info_present_flag`, the original image size information (`eoi_original_width`, `eoi_original_height`) is generated, but `eoi_spatial_resampling_type_idc` may not be generated. In other words, based on a value of 1 for `original_pic_info_present_flag`, the original image size information can be encoded in the bitstream. Based on a value of 1 for `original_pic_info_present_flag`, `eoi_spatial_resampling_type_idc` may not be encoded in the bitstream.

[0167] On the other hand, based on the value of `original_pic_info_present_flag` being 0, `eoi_spatial_resampling_type_idc` can be generated. In other words, based on the value of `original_pic_info_present_flag` being 0, `eoi_spatial_resampling_type_idc` can be encoded in the bitstream. Here, `eoi_spatial_resampling_type_idc` can be encoded as a descriptor of `u(2)`. In other words, `eoi_spatial_resampling_type_idc` can be encoded as an unsigned integer with 2 bits. Here, `original_pic_info_present_flag` can be generated based on the value of `EoiSpatialResamplingFlag` being 1.

[0168] This disclosure relates to methods for defining detailed information related to spatial resampling optimization.

[0169] Based on the value of 1 for the above-mentioned EoiSpatialResamplingFlag, detailed information related to spatial resampling optimization can be generated. Here, the detailed information related to spatial resampling optimization may include at least one of the following: upsampling flag (eoi_upsampling_flag), downsampling flag (eoi_downsampling_flag), neural network-based upsampling flag (eoi_nn_based_upsampling_flag), neural network-based downsampling flag (eoi_nn_based_downsampling_flag), or neural network-based resampling flag (eoi_nn_based_resampling_flag).

[0170] When the value of eoi_upsampling_flag is 1, it indicates that the applied spatial resampling is upsampling. When the value of eoi_upsampling_flag is 0, it indicates that the applied spatial resampling is not upsampling.

[0171] When the value of eoi_downsampling_flag is 1, it indicates that the applied spatial resampling is downsampling. When the value of eoi_downsampling_flag is 0, it indicates that the applied spatial resampling is not downsampling.

[0172] When the value of eoi_nn_based_upsampling_flag is 1, it indicates that the applied spatial resampling is based on upsampling from a neural network. When the value of eoi_nn_based_upsampling_flag is 0, it indicates that the applied spatial resampling is not based on upsampling from a neural network.

[0173] When the value of eoi_nn_based_downsampling_flag is 1, it indicates that the applied spatial resampling is based on neural network downsampling. When the value of eoi_nn_based_downsampling_flag is 0, it indicates that the applied spatial resampling is not based on neural network downsampling.

[0174] When the value of eoi_nn_based_resampling_flag is 1, it indicates that the applied spatial resampling is based on a neural network. When the value of eoi_nn_based_resampling_flag is 0, it indicates that the applied spatial resampling is not based on a neural network.

[0175] As an example, details related to spatial resampling optimization may include eoi_upsampling_flag, eoi_downsampling_flag, and eoi_nn_based_resampling_flag, and can be defined in the encoder optimization information in Table 16 below.

[0176] [Table 16]

[0177] In Table 16, when both `eoi_upsampling_flag` and `eoi_nn_based_resampling_flag` are 1, it likely means that neural network-based upsampling is applied. When both `eoi_upsampling_flag` and `eoi_nn_based_resampling_flag` are 0, it likely means that upsampling is applied but neural network-based upsampling is not applied.

[0178] `eoi_upsampling_flag` and `eoi_downsampling_flag` can be restricted to not having the same value. In other words, `eoi_upsampling_flag` and `eoi_downsampling_flag` cannot both be generated as 1 or 0 at the same time. For example, when the value of `eoi_upsampling_flag` is 1, `eoi_downsampling_flag` can be not 1. Conversely, when the value of `eoi_upsampling_flag` is 0, `eoi_downsampling_flag` can be not 0.

[0179] Alternatively, details related to spatial resampling optimization may include eoi_upsampling_flag, eoi_downsampling_flag, eoi_nn_based_upsampling_flag, and eoi_nn_based_downsampling_flag, and may also be defined in the encoder optimization information in Table 17 below.

[0180] [Table 17]

[0181] The S430 can generate a bitstream that includes compressed video images and encoder optimization information. The encoder optimization information can be configured in the SEI message of the bitstream. The SEI message can be included in the Network Abstraction Layer (NAL) unit of the bitstream.

[0182] Figure 5 The figure shows a schematic configuration of an encoding apparatus 200 for performing a method for generating a bitstream according to the present disclosure.

[0183] refer to Figure 5 The encoding device 200 may include a receiver 500, a video compressor 510, an EOI generator 520, and a bitstream generator 530.

[0184] Receiver 500 can receive one or more encoded video images.

[0185] The video compressor 510 can encode one or more received video images to generate compressed video images.

[0186] The EOI generator 520 can generate encoder optimization information.

[0187] Bitstream generator 530 can generate bitstreams that include compressed video images and encoder optimization information.

[0188] Figure 6 The illustration shows a method for reconstructing a video image performed by a decoding device 300 according to the present disclosure.

[0189] The S600 can receive bitstreams including encoded video images.

[0190] The S610 is a video image encoding that can reconstruct the bitstream.

[0191] Video information related to encoded video images can be extracted from the bitstream. The extracted video information can then be used to reconstruct the encoded video images.

[0192] Additionally, the SEI message can be included in the NAL unit of the bitstream. Encoder optimization information (EOI) can be defined in the SEI message S420. The encoder optimization information can be related to the encoded video image. The encoder optimization information can be included in the SEI message and transmitted via signaling through the bitstream.

[0193] Encoder optimization information may include the EOI cancellation flag (eoi_cancel_flag). Based on eoi_cancel_flag being 0, optimization-related information can be included in the encoder optimization information, which is consistent with the information obtained by referencing... Figure 4 The description is the same.

[0194] In addition, based on the spatial resampling flag (EoiSpatialResamplingFlag) indicating whether spatial resampling optimization is applied, detailed information related to spatial resampling optimization can be included in the encoder optimization information.

[0195] The method for deriving the value of EoiSpatialResamplingFlag and the reference Figure 4 The description is the same. Based on a value of 1 for EoiSpatialResamplingFlag, detailed information related to spatial resampling optimization can be included in the encoder optimization information. This is consistent with the information obtained through reference... Figure 4 The description is the same, and a detailed description will be omitted here.

[0196] As an example, based on a value of 1 for `EoiSpatialResamplingFlag`, the spatial resampling type identifier (`eoi_spatial_resampling_type_idc`) can be included in the encoder optimization information. Here, `eoi_spatial_resampling_type_idc` can be encoded as a 3-bit unsigned integer. Alternatively, `eoi_spatial_resampling_type_idc` can be encoded as a 2-bit unsigned integer.

[0197] As an example, based on the value of EoiSpatialResamplingFlag being 1, the spatial resampling type identifier (eoi_spatial_resampling_type_idc) and the neural network resampling flag (eoi_nn_based_resampling_flag) can be included in the encoder optimization information.

[0198] Here, `eoi_nn_based_resampling_flag` can be encoded after `eoi_spatial_resampling_type_idc` is encoded. Alternatively, `eoi_nn_based_resampling_flag` can be encoded before `eoi_spatial_resampling_type_idc` is encoded. `eoi_nn_based_resampling_flag` can be encoded as a 1-bit unsigned integer. `eoi_spatial_resampling_type_idc` can be encoded as a 3-bit unsigned integer. Alternatively, `eoi_spatial_resampling_type_idc` can be encoded as a 2-bit unsigned integer.

[0199] As an example, based on a value of 1 for `EoiSpatialResamplingFlag`, `eoi_spatial_resampling_type_idc` and `original_pic_info_present_flag` can be included in the encoder optimization information. Based on a value of 1 for `original_pic_info_present_flag`, `eoi_original_width` and `eoi_original_height`, which represent the size information of the original image, can be included in the encoder optimization information. Conversely, based on a value of 0 for `original_pic_info_present_flag`, `eoi_original_width` and `eoi_original_height` may not be included in the encoder optimization information.

[0200] As an example, with `original_pic_info_present_flag` set to 1, the original image size information (eoi_original_width, eoi_original_height) can be included in the encoder optimization information, but `eoi_spatial_resampling_type_idc` may not be included. With `original_pic_info_present_flag` set to 1, the original image size information can be extracted from the bitstream.

[0201] On the other hand, based on the value of `original_pic_info_present_flag` being 0, `eoi_spatial_resampling_type_idc` can be included in the encoder optimization information. Based on the value of `original_pic_info_present_flag` being 0, `eoi_spatial_resampling_type_idc` can be extracted from the bitstream. Here, `original_pic_info_present_flag` can be encoded based on the value of `EoiSpatialResamplingFlag` being 1. `original_pic_info_present_flag` can be extracted from the bitstream based on the value of `EoiSpatialResamplingFlag` being 1.

[0202] As an example, based on a value of 1 for EoiSpatialResamplingFlag, detailed information related to spatial resampling optimization can be included in the encoder optimization information, which is consistent with information obtained by referencing... Figure 4 The description is the same.

[0203] Figure 7 The illustration shows a schematic configuration of a decoding apparatus 300 that performs a method for reconstructing video images according to the present disclosure.

[0204] refer to Figure 7 The decoding device 300 may include a receiver 700, a video information extractor 710, and a video reconstructor 720.

[0205] Receiver 700 can receive bitstreams including encoded video images.

[0206] The video information extractor 710 can extract video information related to encoded video images from the bitstream. Additionally, the video information extractor 710 can extract SEI messages from the NAL units of the bitstream. The extracted SEI messages may include encoder optimization information.

[0207] The video reconstructor 720 can reconstruct encoded video images based on the extracted video information.

[0208] In the above embodiments, the method is described as a series of steps or blocks based on the flowchart. However, the corresponding embodiments are not limited to the order of the steps, and some steps may occur simultaneously or in a different order than the other steps described above. Furthermore, those skilled in the art will understand that the steps shown in the flowchart are not exclusive, and other steps may be included or one or more steps in the flowchart may be deleted without affecting the scope of the embodiments of this disclosure.

[0209] The methods described above according to embodiments of the present disclosure can be implemented in software, and the encoding and / or decoding apparatus according to the present disclosure can be included in devices performing image processing, such as TVs, computers, smartphones, set-top boxes, display devices, etc.

[0210] In this disclosure, when the embodiments are implemented as software, the above methods can be implemented as modules (processes, functions, etc.) performing the above functions. Modules can be stored in memory and can be executed by a processor. Memory can be located inside or outside the processor and can be connected to the processor by various well-known means. The processor may include application-specific integrated circuits (ASICs), another chipset, logic circuits, and / or data processing devices. Memory may include read-only memory (ROM), random access memory (RAM), flash memory, memory cards, storage media, and / or other storage devices. In other words, the embodiments described herein can be implemented on a processor, microprocessor, controller, or chip. For example, the functional units shown in each figure can be implemented on a computer, processor, microprocessor, controller, or chip. In this case, information for implementation (e.g., information about instructions) or algorithms can be stored in a digital storage medium.

[0211] Furthermore, the decoding and encoding devices using embodiments of this disclosure can be included in multimedia broadcasting transmitting and receiving devices, mobile communication terminals, home theater video devices, digital cinema video devices, surveillance cameras, video conferencing devices, real-time communication devices such as video communication, mobile streaming devices, storage media, cameras, devices for providing video-on-demand (VoD) services, over-the-top (OTT) devices, devices for providing internet streaming services, three-dimensional (3D) video devices, virtual reality (VR) devices, augmented reality (AR) devices, videophone video devices, transportation terminal devices (e.g., vehicle (including autonomous vehicles) terminals, aircraft terminals, ship terminals, etc.), and medical video devices, and can be used to process video signals or data signals. For example, over-the-top (OTT) devices can include game consoles, Blu-ray players, networked TVs, home theater systems, smartphones, tablets, digital video recorders (DVRs), etc.

[0212] Furthermore, the processing methods applying embodiments of this disclosure can be generated in the form of a computer-executable program and can be stored in a computer-readable recording medium. Multimedia data having data structures according to embodiments of this disclosure can also be stored in a computer-readable recording medium. Computer-readable recording media include all types of storage devices and distributed storage devices that store computer-readable data. Computer-readable recording media can include, for example, Blu-ray discs (BD), Universal Serial Bus (USB), ROM, PROM, EPROM, EEPROM, RAM, CD-ROM, magnetic tape, floppy disks, and optical media storage devices. Additionally, computer-readable recording media include media implemented in carrier wave form (e.g., transmitted via the Internet). Furthermore, bitstreams generated by encoding methods can be stored in a computer-readable recording medium or transmitted via wired or wireless communication networks.

[0213] Furthermore, the embodiments of this disclosure can be implemented by a computer program product using program code, and this program code can be executed on a computer by the embodiments of this disclosure. The program code can be stored on a computer-readable medium.

[0214] Figure 8 Examples of content streaming systems to which embodiments of the present disclosure may be applied are shown.

[0215] refer to Figure 8 The content streaming system using embodiments of this disclosure may mainly include an encoding server, a streaming server, a web server, media storage, user equipment, and multimedia input devices.

[0216] An encoding server generates a bitstream by compressing content input from multimedia input devices such as smartphones, cameras, and camcorders into digital data, and then sends it to a streaming server. As another example, when multimedia input devices such as smartphones, cameras, and camcorders generate bitstreams directly, the encoding server can be omitted.

[0217] A bitstream can be generated by applying the encoding method or bitstream generation method of the embodiments of this disclosure, and the streaming server can temporarily store the bitstream during the sending or receiving of the bitstream.

[0218] A streaming server sends multimedia data to a user's device via a web server based on the user's request, and the web server acts as a medium to notify the user what services are available. When a user requests a service from the web server, the web server delivers it to the streaming server, and the streaming server sends the multimedia data to the user. In this scenario, the content streaming system may include a separate control server, which in this case controls the commands / responses between each device in the content streaming system.

[0219] A streaming server can receive content from media storage and / or encoding servers. For example, when receiving content from an encoding server, the content can be received in real time. In this case, to provide a smooth streaming service, the streaming server can store bitstreams for specific time periods.

[0220] Examples of user equipment may include mobile phones, smartphones, laptops, digital broadcast terminals, personal digital assistants (PDAs), portable multimedia players (PMPs), navigation devices, tablet PCs, tablet computers, ultrabooks, wearable devices (e.g., smartwatches, smart glasses, head-mounted displays (HMDs), digital TVs, desktop computers, digital signage, etc.).

[0221] In a content streaming system, each server can be operated as a distributed server, and in this case, data received from each server can be distributed and processed.

[0222] The claims set forth herein can be combined in various ways. For example, the technical features of the method claims of this disclosure can be combined and implemented as a device, and the technical features of the device claims of this disclosure can be combined and implemented as a method. Furthermore, the technical features of the method claims and the technical features of the device claims of this disclosure can be combined and implemented as a device, and the technical features of the method claims and the technical features of the device claims of this disclosure can be combined and implemented as a method.

Claims

1. A method comprising: Receive the video and images to be encoded; The received video images are encoded to generate compressed video images; Generate encoder optimization information; as well as Generate a bitstream that includes the compressed video images and the encoder optimization information. The encoder optimization information includes the size information of the original image, and The size information of the original image is generated based on a flag indicating whether there is information related to the original image.

2. The method according to claim 1, wherein, The size information of the original image includes information representing the width of the original image and information representing the height of the original image.

3. The method according to claim 2, wherein, When the value of the flag is 1, information related to the original image exists, and Specifically, when the value of the flag is 0, there is no information related to the original image.

4. The method according to claim 3, wherein, Based on the value of the flag being 1, information representing the width of the original image and information representing the height of the original image are generated respectively.

5. The method according to claim 3, wherein, If the value of the flag is 0, no information representing the width or height of the original image will be generated.

6. The method according to claim 5, wherein, Based on the value of the flag being 0, a spatial resampling type identifier is generated to identify the type of spatial resampling applied.

7. The method according to claim 1, wherein, The flag is generated based on a spatial resampling flag related to whether or not spatial resampling optimization is applied.

8. The method according to claim 7, wherein, The spatial resampling flag is derived based on EOI type information related to the attributes of the optimization method.

9. A method comprising: Receive bitstreams including encoded video images; as well as Reconstructing the encoded video images included in the bitstream The bitstream includes encoder optimization information. The encoder optimization information includes the size information of the original image, and The size information of the original image is included based on a flag indicating whether information related to the original image exists.

10. A computer-readable storage medium for storing a bit stream generated by the method of claim 1.

11. A method comprising: A bitstream is generated, wherein the bitstream is generated based on: receiving a video image to be encoded, encoding the received video image to generate a compressed video image, generating encoder optimization information, and generating a bitstream including the compressed video image and the encoder optimization information; and Send data including the bit stream. The encoder optimization information includes the size information of the original image, and The size information of the original image is generated based on a flag indicating whether there is information related to the original image.

Images