Apparatus for encoding / decoding / transmitting image information

Abstract

The present disclosure relates to an apparatus for encoding / decoding / transmitting image information. An image decoding method performed by a decoding device according to the present document is characterized by comprising the steps of: obtaining a flag indicating whether or not there is a picture header (PH) network abstraction layer (NAL) unit including a PH of a current picture; obtaining a video coding layer (VCL) NAL unit including a slice header and slice data of a current slice of the current picture; obtaining the PH based on the flag; and decoding the current picture based on the PH, the slice header, and the slice data.

Description

[0001] This application is a divisional application of the original invention patent application No. 202080097938.3 (International Application No.: PCT / KR2020 / 019322, Application Date: December 29, 2020, Invention Title: PH NAL Unit Encoding Related Image Decoding Method and Apparatus). Technical Field

[0002] This disclosure relates to image coding technology, and more specifically, to a video decoding method and apparatus for adaptively encoding PH NAL units in an image coding system. Background Technology

[0003] Recently, there has been a growing demand for high-resolution, high-quality images, such as HD (high-definition) and UHD (ultra-high-definition) images, across various fields. Because image data is high-resolution and high-quality, the amount of information or bits to be transmitted increases compared to traditional image data. Therefore, transmission and storage costs increase when using media such as traditional wired / wireless broadband lines to transmit image data or when storing image data using existing storage media.

[0004] Therefore, there is a need for efficient image compression technology to effectively transmit, store, and reproduce information from high-resolution, high-quality images. Summary of the Invention

[0005] Technical issues

[0006] The technical objective of this disclosure is to provide a method and apparatus for improving image coding efficiency.

[0007] Another technical objective of this disclosure is to provide a method and apparatus for encoding a flag indicating the presence of a PH NAL unit.

[0008] Technical solution

[0009] According to embodiments of this disclosure, an image decoding method performed by a decoding device is provided. The method includes the following steps: obtaining a flag indicating the presence of a PH Network Abstraction Layer (NAL) unit that includes a PH (picture header) of the current frame; obtaining a Video Coding Layer (VCL) NAL unit that includes a slice header and slice data of the current slice in the current frame; obtaining the PH based on the flag; and decoding the current frame based on the PH, the slice header, and the slice data.

[0010] According to another embodiment of this disclosure, a decoding apparatus for performing image decoding is provided. The decoding apparatus includes: an entropy decoder configured to obtain a flag indicating the presence of a PH Network Abstraction Layer (NAL) unit that includes a frame header (PH) of the current frame, obtain a Video Coding Layer (VCL) NAL unit that includes a slice header and slice data of a current slice in the current frame, and obtain the PH based on the flag; and a predictor configured to decode the current frame based on the PH, the slice header, and the slice data.

[0011] According to another embodiment of this disclosure, a video encoding method performed by an encoding device is provided. The method includes the following steps: generating a Video Coding Layer (VCL) Network Abstraction Layer (NAL) unit that includes a slice header and slice data of a slice in the current frame; determining whether a PH NAL unit that includes a PH of the current frame exists; generating a PH and a flag indicating the presence or absence of a PH NAL unit based on the determined result; and encoding image information including the VCL NAL unit, the PH, and the flag.

[0012] According to another embodiment of this disclosure, a video encoding apparatus is provided. The encoding apparatus includes: an entropy encoder configured to generate a Video Coding Layer (VCL) Network Abstraction Layer (NAL) unit that includes a slice header and slice data of a slice in the current frame; determine whether a PH NAL unit including a PH of the current frame exists; generate a PH and a flag indicating the presence or absence of a PH NAL unit based on the determination result; and encode image information including the VCL NAL unit, the PH, and the flag.

[0013] According to another embodiment of this disclosure, a computer-readable digital storage medium is provided that stores a bitstream including image information and enables the execution of an image decoding method. In this computer-readable digital storage medium, the image decoding method includes the following steps: obtaining a flag indicating the presence of a PH Network Abstraction Layer (NAL) unit that includes a Frame Header (PH) of the current frame; obtaining a Video Coding Layer (VCL) NAL unit that includes a Slice Header and Slice Data of the current slice in the current frame; obtaining the PH based on the flag; and decoding the current frame based on the PH, the Slice Header, and the Slice Data.

[0014] Beneficial effects

[0015] According to this disclosure, a flag indicating the presence of a PH NAL unit can be used to adaptively adjust the bit rate of the bitstream based on the flag, thereby improving overall coding efficiency.

[0016] According to this disclosure, constraints on the number of slices in the current frame and the presence of PH NAL units can be set for the relevant frame based on a flag indicating the presence of PH NAL units, so as to adaptively control the NAL units according to the bit rate of the bitstream, thereby improving the overall coding efficiency. Attached Figure Description

[0017] Figure 1 Examples of video / image encoding apparatuses to which embodiments of the present disclosure may be applied are briefly illustrated.

[0018] Figure 2 This is a schematic diagram illustrating the configuration of a video / image encoding device to which embodiments of the present disclosure can be applied.

[0019] Figure 3 This is a schematic diagram illustrating the configuration of a video / image decoding device to which embodiments of the present disclosure can be applied.

[0020] Figure 4 An example of a video / image coding method based on inter-frame prediction is shown.

[0021] Figure 5 An example of a video / image decoding method based on inter-frame prediction is shown.

[0022] Figure 6 The inter-frame prediction process is illustrated schematically.

[0023] Figure 7 The hierarchical structure for encoding image information is illustrated schematically.

[0024] Figure 8 The coding process according to an embodiment of the present disclosure is illustrated schematically.

[0025] Figure 9 The decoding process according to an embodiment of the present disclosure is illustrated schematically.

[0026] Figure 10 The image header configuration in the NAL unit is schematically shown based on the presence or absence of the PH NAL unit.

[0027] Figure 11 An image encoding method according to the encoding device of this document is illustrated schematically.

[0028] Figure 12 An encoding device for performing an image encoding method according to this document is illustrated schematically.

[0029] Figure 13 An image decoding method according to the decoding device of this document is illustrated schematically.

[0030] Figure 14A decoding device for performing an image decoding method according to this document is illustrated schematically.

[0031] Figure 15 A structural diagram of a content streaming system applying this disclosure is shown. Detailed Implementation

[0032] This disclosure can be modified in various forms, and specific embodiments thereof will be described and illustrated in the accompanying drawings. However, the embodiments are not intended to limit this disclosure. The terminology used in the following description is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. Singular expressions include plural expressions, provided that they are clearly understood in different ways. Terms such as “comprising” and “having” are intended to indicate the presence of the features, quantities, steps, operations, elements, components, or combinations thereof used in the following description, and therefore it should be understood that the possibility of having or adding one or more different features, quantities, steps, operations, elements, components, or combinations thereof is not excluded.

[0033] Furthermore, the elements in the accompanying drawings described in this disclosure are drawn independently for ease of explanation of different specific functions and do not imply that these elements are specifically implemented by independent hardware or independent software. For example, two or more elements may be combined to form a single element, or a single element may be divided into multiple elements. Embodiments of combining and / or dividing elements are part of this disclosure and do not depart from the concept of this disclosure.

[0034] In the following, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. Furthermore, throughout the drawings, similar reference numerals are used to indicate similar elements, and identical descriptions of similar elements will be omitted.

[0035] Figure 1 Examples of video / image encoding apparatuses to which embodiments of the present disclosure may be applied are briefly illustrated.

[0036] Reference Figure 1 A video / image encoding system may include a first device (source device) and a second device (receiving device). The source device may transmit encoded video / image information or data to the receiving device in the form of a file or stream via a digital storage medium or network.

[0037] The source device may include a video source, an encoding device, and a transmitter. The receiving device may include a receiver, a decoding device, and a renderer. The encoding device may be referred to as a video / image encoding device, and the decoding device may be referred to as a video / image decoding device. The transmitter may be included in the encoding device. The receiver may be included in the decoding device. The renderer may include a display, and the display may be configured as a separate device or an external component.

[0038] Video sources can acquire video / images through processes that capture, synthesize, or generate video / images. Video sources may include video / image capture devices and / or video / image generation devices. Video / image capture devices may include, for example, one or more cameras, video / image archives including previously captured video / images, etc. Video / image generation devices may include, for example, computers, tablets, and smartphones, and can generate video / images (electronically). For example, virtual video / images can be generated by computers, etc. In this case, the video / image capture process can be replaced by a process that generates related data.

[0039] Encoding devices can encode input video / images. They can perform a series of processes such as prediction, transformation, and quantization to achieve compression and encoding efficiency. The encoded data (encoded video / image information) can be output as a bitstream.

[0040] A transmitter can send encoded video / image information or data, output as a bitstream, to a receiver in a receiving device in the form of a file or stream via a digital storage medium or network. The digital storage medium can include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, SSD, etc. The transmitter can include elements for generating media files according to a predetermined file format and may include elements for transmission via a broadcast / communication network. The receiver can receive / extract the bitstream and send the received bitstream to a decoding device.

[0041] Decoding devices can decode video / images by performing a series of processes, such as dequantization, inverse transform, and prediction, that correspond to the operations of encoding devices.

[0042] The renderer can render decoded video / images. Rendered video / images can then be displayed on a monitor.

[0043] This disclosure relates to video / image coding. For example, the methods / implementations disclosed in this disclosure can be applied to methods disclosed in Multifunctional Video Coding (VVC), EVC (Essential Video Coding) standard, AOMedia Video 1 (AV1) standard, AVS2 (Audio Video Coding 2) standard, or next-generation video / image coding standards (e.g., H.267, or H.268, etc.).

[0044] This disclosure presents various implementations of video / image encoding, and unless otherwise stated, the implementations may be combined with each other.

[0045] In this disclosure, video can refer to a series of images over time. Generally, a frame refers to a unit representing an image within a specific time region, and sub-frames / slices / tiles are units that constitute part of a frame in encoding. Sub-frames / slices / tiles may include one or more coding tree units (CTUs). A frame may consist of one or more sub-frames / slices / tiles. A frame may consist of one or more tile groups. A tile group may include one or more tiles. A brick can represent a rectangular area of ​​a CTU row within a tile in a frame. A tile can be partitioned into multiple tiles, each tile consisting of one or more CTU rows within the tile. A tile that is not partitioned into multiple tiles may also be referred to as a tile. Tile scanning is a specific sequential ordering of the CTUs that partition the frame, wherein CTUs are sequentially ordered by CTU raster scan within a tile, tiles within a tile are sequentially ordered by raster scan of the tiles of a tile, and tiles in a frame are sequentially ordered by raster scan of the tiles of a frame. Additionally, a sub-picture can represent a rectangular area of ​​one or more slices within a picture. That is, a sub-picture contains one or more slices that collectively cover a rectangular area of ​​the picture. A tile is a rectangular area of ​​a CTU within a specific tile column and a specific tile row in the picture. A tile column is a rectangular area of ​​a CTU whose height is equal to the height of the picture and whose width is specified by a syntax element in the picture parameter set. A tile row is a rectangular area of ​​a CTU whose height is specified by a syntax element in the picture parameter set and whose width is equal to the width of the picture. A tile scan is a specific sequential ordering of CTUs that partition the picture, wherein CTUs can be ordered consecutively by CTU raster scan within a tile, and tiles in the picture can be ordered consecutively by raster scan of the picture's tiles. A slice comprises an integer number of tiles in the picture that can be exclusively contained in a single NAL unit. A slice can consist of multiple complete tiles or only a consecutive sequence of complete tiles of a single tile. In this disclosure, tile groups and slices can be used interchangeably. For example, in this disclosure, a tile group / tile group header may be referred to as a slice / slice header.

[0046] A pixel, or pel, can represent the smallest unit that makes up a picture (or image). Additionally, "sample" can be used as the term corresponding to a pixel. A sample can typically represent a pixel or a pixel value, and can represent only the pixel / pixel value of the luminance component or only the pixel / pixel value of the chrominance component.

[0047] A unit can represent the basic unit of image processing. A unit may include a specific region of the image and at least one of the information associated with that region. A unit may include a luminance block and two chrominance (e.g., cb, cr) blocks. In some cases, the term "unit" may be used interchangeably with terms such as "block" or "region". In general, an M×N block may include a set (or array) of samples (or transform coefficients) with M columns and N rows.

[0048] In this specification, “A or B” can mean “A only”, “B only”, or “A and B”. In other words, in this specification, “A or B” can be interpreted as “A and / or B”. For example, “A, B or C” in this document means “A only”, “B only”, “C only”, or “any one and any combination of A, B and C”.

[0049] The forward slash ( / ) or comma used in this specification can mean "and / or". For example, "A / B" can mean "A and / or B". Therefore, "A / B" can mean "A only", "B only", or "A and B". For example, "A, B, C" can mean "A, B, or C".

[0050] In this specification, "at least one of A and B" can mean "A only", "B only" or "both A and B". Furthermore, in this specification, the expression "at least one of A or B" or "at least one of A and / or B" can be interpreted as the same as "at least one of A and B".

[0051] Additionally, in this specification, "at least one of A, B, and C" means "A only", "B only", "C only" or "any combination of A, B, and C". Furthermore, "at least one of A, B, or C" or "at least one of A, B, and / or C" can mean "at least one of A, B, and C".

[0052] Furthermore, the parentheses used in this specification may mean "for example". Specifically, when "prediction (intra-frame prediction)" is indicated, "intra-frame prediction" may be given as an example of "prediction". In other words, "prediction" in this specification is not limited to "intra-frame prediction", and "intra-frame prediction" may be given as an example of "prediction". Moreover, even when "prediction (i.e., intra-frame prediction)" is indicated, "intra-frame prediction" may be given as an example of "prediction".

[0053] In this specification, the technical features described individually in a single figure may be implemented individually or simultaneously.

[0054] The following figures were created to illustrate specific examples of this specification. Since the names of specific devices or signals / messages / fields described in the figures are presented by way of example, the technical features of this specification are not limited to the specific names used in the following figures.

[0055] Figure 2 This is a schematic diagram illustrating the configuration of a video / image encoding apparatus to which embodiments of the present disclosure may be applied. In the following, the video encoding apparatus may include an image encoding apparatus.

[0056] Reference Figure 2 The encoding device 200 includes an image segmenter 210, a predictor 220, a residual processor 230, an entropy encoder 240, an adder 250, a filter 260, and a memory 270. The predictor 220 may include an inter-frame predictor 221 and an intra-frame predictor 222. The residual processor 230 may include a transformer 232, a quantizer 233, a dequantizer 234, and an inverse transformer 235. The residual processor 230 may also include a subtractor 231. The adder 250 may be referred to as a reconstructor or a reconstruction block generator. According to embodiments, the image segmenter 210, predictor 220, residual processor 230, entropy encoder 240, adder 250, and filter 260 may be constituted by at least one hardware component (e.g., an encoder chipset or a processor). Additionally, the memory 270 may include a decoded frame buffer (DPB) or may be constituted by a digital storage medium. The hardware component may also include the memory 270 as an internal / external component.

[0057] Image segmenter 210 can segment an input image (or picture or frame) input to encoding device 200 into one or more processors. For example, a processor may be referred to as a coding unit (CU). In this case, coding units can be recursively segmented from coding tree units (CTUs) or maximum coding units (LCUs) according to a quadtree-binary-trinary (QTBTTT) structure. For example, a coding unit can be segmented into multiple deeper coding units based on a quadtree structure, a binary tree structure, and / or a ternary structure. In this case, for example, a quadtree structure can be applied first, followed by a binary tree structure and / or a ternary structure. Alternatively, a binary tree structure can be applied first. The encoding process according to this disclosure can be performed based on the final coding unit that is no longer segmented. In this case, the maximum coding unit can be used as the final coding unit based on encoding efficiency according to image characteristics, or, if necessary, the coding unit can be recursively segmented into deeper coding units, and the coding unit with the optimal size can be used as the final coding unit. Here, the encoding process may include prediction, transformation, and reconstruction processes, which will be described later. As another example, the processor may also include a prediction unit (PU) or a transform unit (TU). In this case, the prediction unit and the transform unit can be separated or divided from the final encoding unit described above. The prediction unit may be a unit for predicting samples, and the transform unit may be a unit for deriving transform coefficients and / or a unit for deriving residual signals from transform coefficients.

[0058] In some cases, a unit can be used interchangeably with terms such as a block or region. Generally, an M×N block can represent a set of samples or transform coefficients consisting of M columns and N rows. A sample can typically represent a pixel or pixel value, and may represent only the pixel / pixel value of the luminance component or only the pixel / pixel value of the chrominance component. A sample can be used as a term corresponding to a frame (or image) of pixels or cells.

[0059] In encoding device 200, a prediction signal (prediction block, prediction sample array) output from inter-frame predictor 221 or intra-frame predictor 222 is subtracted from the input image signal (original block, original sample array) to generate a residual signal (residual block, residual sample array), and the generated residual signal is sent to converter 232. In this case, as shown, the unit in encoding device 200 used to subtract the prediction signal (prediction block, prediction sample array) from the input image signal (original block, original sample array) can be called subtractor 231. The predictor can perform prediction on the block to be processed (hereinafter referred to as the current block) and generate a prediction block including the prediction samples of the current block. The predictor can determine whether to apply intra-frame prediction or inter-frame prediction on a unit of the current block or CU. As described later in the description of each prediction mode, the predictor can generate various information related to the prediction (such as prediction mode information) and send the generated information to entropy encoder 240. The information about the prediction can be encoded in entropy encoder 240 and output as a bitstream.

[0060] Intra-predictor 222 can predict the current block by referencing samples in the current frame. Depending on the prediction mode, the referenced samples may be located near or far from the current block. In intra-prediction, the prediction mode can include multiple non-directional modes and multiple directional modes. Non-directional modes can include, for example, DC mode and planar mode. Depending on the level of detail in the prediction direction, the directional modes can include, for example, 33 or 65 directional prediction modes. However, this is just an example, and more or fewer directional prediction modes may be used depending on the settings. Intra-predictor 222 can determine the prediction mode to be applied to the current block by using the prediction modes applied to neighboring blocks.

[0061] Inter-frame predictor 221 can deduce the prediction block of the current block based on a reference block (reference sample array) specified by motion vectors on a reference frame. Here, to reduce the amount of motion information transmitted in inter-frame prediction mode, motion information can be predicted on a block, sub-block, or sample basis based on the correlation between motion information between neighboring blocks and the current block. Motion information may include motion vectors and reference frame indices. Motion information may also include inter-frame prediction direction (L0 prediction, L1 prediction, Bi prediction, etc.) information. In the case of inter-frame prediction, neighboring blocks may include spatially adjacent blocks existing in the current frame and temporally adjacent blocks existing in the reference frame. The reference frame including the reference block and the reference frame including the temporally adjacent block may be the same or different. The temporally adjacent block may be called a co-located reference block, a co-located CU (colCU), etc., and the reference frame including the temporally adjacent block may be called a co-located frame (colPic). For example, inter-frame predictor 221 can configure a motion information candidate list based on neighboring blocks and generate information indicating which candidate to use to deduce the motion vector and / or reference frame index of the current block. Inter-frame prediction can be performed based on various prediction modes. For example, in skip mode and merge mode, the inter-frame predictor 221 can use the motion information of neighboring blocks as the motion information of the current block. In skip mode, unlike merge mode, residual signals may not be able to be sent. In motion vector prediction (MVP) mode, the motion vectors of neighboring blocks can be used as motion vector predictors, and the motion vector of the current block can be indicated by signaling the motion vector difference.

[0062] Predictor 220 can generate a prediction signal based on various prediction methods described below. For example, the predictor can not only apply intra-frame prediction or inter-frame prediction to predict a block, but can also apply both intra-frame prediction and inter-frame prediction simultaneously. This can be referred to as Inter-intra-frame Combined Prediction (CIIP). Alternatively, the predictor can predict blocks based on an Intra-Block Copy (IBC) prediction mode or a palette mode. IBC prediction mode or palette mode can be used for content image / video coding such as games, for example, Screen Content Coding (SCC). IBC essentially performs prediction within the current frame, but can be performed similarly to inter-frame prediction because the reference block is derived within the current frame. That is, IBC can use at least one of the inter-frame prediction techniques described in this disclosure. Palette mode can be considered as an example of intra-frame coding or intra-frame prediction. When applying palette mode, sample values ​​within the frame can be signaled based on information about the palette table and palette index.

[0063] The predicted signal generated by the predictor (including inter-frame predictor 221 and / or intra-frame predictor 222) can be used to generate a reconstructed signal or a residual signal. Transformer 232 can generate transform coefficients by applying transform techniques to the residual signal. For example, the transform technique can include at least one of Discrete Cosine Transform (DCT), Discrete Sine Transform (DST), Karhunen-Loève Transform (KLT), Graph-Based Transform (GBT), or Conditional Nonlinear Transform (CNT). Here, GBT refers to a transform obtained from a graph when the relationship information between pixels is represented by a graph. CNT refers to a transform generated based on the predicted signal generated using all previously reconstructed pixels. Furthermore, the transform processing can be applied to square pixel blocks of the same size, or it can be applied to blocks of variable size that are not square.

[0064] Quantizer 233 quantizes the transform coefficients and sends them to entropy encoder 240, which encodes the quantized signal (information about the quantized transform coefficients) and outputs a bitstream. This information about the quantized transform coefficients can be called residual information. Quantizer 233 can rearrange the block-type quantized transform coefficients into a one-dimensional vector form based on the coefficient scan order and generate information about the quantized transform coefficients based on this one-dimensional vector form. Entropy encoder 240 can perform various encoding methods, such as Golomb, Context Adaptive Variable Length Coding (CAVLC), and Context Adaptive Binary Arithmetic Coding (CABAC). Entropy encoder 240 can encode information required for video / image reconstruction other than the quantized transform coefficients (e.g., values ​​of syntax elements) together or separately. Encoded information (e.g., encoded video / image information) can be sent or stored in units of NAL (Network Abstraction Layer) in the form of a bitstream. The video / image information may also include information about various parameter sets such as Adaptive Parameter Set (APS), Picture Parameter Set (PPS), Sequence Parameter Set (SPS), or Video Parameter Set (VPS). Additionally, the video / image information may include general constraint information. In this disclosure, information and / or syntax elements that transmit / signal notifications from the encoding device to the decoding device may be included in the video / picture information. The video / image information may be encoded by the encoding process described above and included in a bitstream. The bitstream may be transmitted over a network or stored in a digital storage medium. The network may include broadcast networks and / or communication networks, and the digital storage medium may include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, SSD, etc. A transmitter (not shown) that transmits a signal output from the entropy encoder 240 and / or a storage unit (not shown) that stores the signal may be included as internal / external components of the encoding device 200; alternatively, the transmitter may be included in the entropy encoder 240.

[0065] The quantized transform coefficients output from quantizer 233 can be used to generate a prediction signal. For example, the residual signal (residual block or residual sample) can be reconstructed by applying dequantization and inverse transform to the quantized transform coefficients using dequantizer 234 and inverse transformer 235. Adder 250 adds the reconstructed residual signal to the prediction signal output from inter-frame predictor 221 or intra-frame predictor 222 to generate a reconstructed signal (reconstructed frame, reconstructed block, reconstructed sample array). If the block to be processed has no residual (such as when a skip mode is applied), the prediction block can be used as a reconstructed block. Adder 250 can be called a reconstructor or reconstructed block generator. The generated reconstructed signal can be used for intra-frame prediction of the next block to be processed in the current frame, and can be used for inter-frame prediction of the next frame by filtering as described below.

[0066] In addition, Luminance Mapping and Chroma Scaling (LMCS) can be applied during image encoding and / or reconstruction.

[0067] Filter 260 can improve subjective / objective image quality by applying filtering to the reconstructed signal. For example, filter 260 can generate a modified reconstructed image by applying various filtering methods to the reconstructed image and store the modified reconstructed image in memory 270 (specifically, the DPB of memory 270). Various filtering methods may include, for example, deblocking filtering, sample adaptive offset, adaptive loop filtering, bilateral filtering, etc. Filter 260 can generate various filtering-related information and send the generated information to entropy encoder 240, as described later in the description of various filtering methods. The filtering-related information can be encoded by entropy encoder 240 and output as a bitstream.

[0068] The modified reconstructed frame sent to memory 270 can be used as a reference frame in inter-frame predictor 221. When inter-frame prediction is applied via the encoding device, prediction mismatch between the encoding device 200 and the decoding device can be avoided, and encoding efficiency can be improved.

[0069] The DPB of memory 270 can store a modified reconstructed frame that serves as a reference frame in inter-frame predictor 221. Memory 270 can store motion information of blocks from which motion information in the current frame is derived (or encoded) and / or motion information of reconstructed blocks in the frame. The stored motion information can be sent to inter-frame predictor 221 and used as motion information for spatially adjacent blocks or temporally adjacent blocks. Memory 270 can store reconstructed samples of reconstructed blocks in the current frame and can transmit these reconstructed samples to intra-frame predictor 222.

[0070] Figure 3 This is a schematic diagram illustrating the configuration of a video / image decoding device to which embodiments of the present disclosure can be applied.

[0071] Reference Figure 3 The decoding device 300 may include an entropy decoder 310, a residual processor 320, a predictor 330, an adder 340, a filter 350, and a memory 360. The predictor 330 may include an inter-frame predictor 332 and an intra-frame predictor 331. The residual processor 320 may include a dequantizer 321 and an inverse transformer 322. According to embodiments, the entropy decoder 310, residual processor 320, predictor 330, adder 340, and filter 350 may be constructed from hardware components (e.g., a decoder chipset or processor). Additionally, the memory 360 may include a decoded picture buffer (DPB) or may be constructed from a digital storage medium. The hardware components may also include the memory 360 as an internal / external component.

[0072] When the input includes a bitstream containing video / image information, the decoding device 300 can interact with... Figure 2 The processing of video / image information in the encoding device correspondingly reconstructs the image. For example, the decoding device 300 can deduce units / blocks based on block segmentation information obtained from the bitstream. The decoding device 300 can use a processor applied in the encoding device to perform decoding. Therefore, the decoding processor can be, for example, an encoding unit, and the encoding unit can be segmented from the encoding tree unit or the maximum encoding unit according to a quadtree structure, binary tree structure, and / or ternary tree structure. One or more transform units can be derived from the encoding unit. The reconstructed image signal decoded and output by the decoding device 300 can be reproduced by a reproduction device.

[0073] Decoding device 300 can receive data in bitstream form from... Figure 2The signal output by the encoding device can be decoded by the entropy decoder 310. For example, the entropy decoder 310 can parse the bitstream to derive information (e.g., video / image information) required for image reconstruction (or picture reconstruction). The video / image information may also include information about various parameter sets such as Adaptive Parameter Set (APS), Picture Parameter Set (PPS), Sequence Parameter Set (SPS), or Video Parameter Set (VPS). In addition, the video / image information may also include general constraint information. The decoding device can also decode the picture based on the information about the parameter sets and / or general constraint information. The signaling / receiving information and / or syntax elements described later in this disclosure can be decoded and obtained from the bitstream through the decoding process. For example, the entropy decoder 310 decodes the information in the bitstream based on encoding methods such as exponential Golomb coding, CAVLC, or CABAC, and outputs the quantized values ​​of the syntax elements and transform coefficients of the residuals required for image reconstruction. More specifically, the CABAC entropy decoding method can receive a bin corresponding to each syntax element in the bitstream, determine the context model using information about the target syntax element, decoding information about the target block, or information about symbols / bins decoded in previous stages, and perform arithmetic decoding on the bin by predicting the occurrence probability of the bin based on the determined context model, generating a symbol corresponding to the value of each syntax element. In this case, after determining the context model, the CABAC entropy decoding method can update the context model by using the information of the decoded symbol / bin for the context model of the next symbol / bin. The prediction-related information in the information decoded by the entropy decoder 310 can be provided to the predictors (inter-frame predictor 332 and intra-frame predictor 331), and the residual values ​​(i.e., quantization transform coefficients and related parameter information) from which entropy decoding has been performed in the entropy decoder 310 can be input to the residual processor 320. The residual processor 320 can derive the residual signal (residual block, residual sample, residual sample array). Additionally, the filtering information in the information decoded by the entropy decoder 310 can be provided to the filter 350. Furthermore, the receiver (not shown) for receiving the signal output from the encoding device can be further configured as an internal / external element of the decoding device 300, or the receiver can be a component of the entropy decoder 310. Additionally, the decoding device according to this disclosure can be referred to as a video / image / picture decoding device, and the decoding device can be classified as an information decoder (video / image / picture information decoder) and a sample decoder (video / image / picture sample decoder). The information decoder may include the entropy decoder 310, and the sample decoder may include at least one of a dequantizer 321, an inverse transformer 322, an adder 340, a filter 350, a memory 360, an inter-frame predictor 332, and an intra-frame predictor 331.

[0074] Dequantizer 321 can dequantize the quantized transform coefficients and output the transform coefficients. Dequantizer 321 can rearrange the quantized transform coefficients in the form of two-dimensional blocks. In this case, the rearrangement can be performed based on the coefficient scan order performed in the encoding device. Dequantizer 321 can perform dequantization on the quantized transform coefficients using quantization parameters (e.g., quantization step size information) and obtain the transform coefficients.

[0075] The inverse transformer 322 performs an inverse transformation on the transformation coefficients to obtain the residual signal (residual block, residual sample array).

[0076] The predictor can perform prediction on the current block and generate a prediction block that includes the prediction samples of the current block. The predictor can determine whether to apply intra-frame prediction or inter-frame prediction to the current block based on the prediction information output from the entropy decoder 310, and can determine the specific intra-frame / inter-frame prediction mode.

[0077] The predictor can generate a prediction signal based on various prediction methods described below. For example, the predictor can not only apply intra-frame prediction or inter-frame prediction to predict a block, but can also apply intra-frame prediction and inter-frame prediction simultaneously. This can be called Inter-Frame Intra-Frame Combined Prediction (CIIP). Alternatively, the predictor can predict blocks based on an Intra-Frame Block Copy (IBC) prediction mode or a palette mode. IBC prediction modes or palette modes can be used for content image / video coding such as in games, for example, Screen Content Coding (SCC). IBC essentially performs prediction within the current frame, but can be performed similarly to inter-frame prediction because a reference block is derived within the current frame. That is, IBC can use at least one of the inter-frame prediction techniques described in this disclosure. A palette mode can be considered an example of intra-frame coding or intra-frame prediction. When a palette mode is applied, sample values ​​within the frame can be signaled based on information about the palette table and palette index.

[0078] Intra-predictor 331 can predict the current block by referencing samples in the current frame. Depending on the prediction mode, the referenced samples may be located near or far from the current block. In intra-prediction, the prediction mode can include multiple non-directional modes and multiple directional modes. Intra-predictor 331 can determine the prediction mode applied to the current block by using prediction modes applied to neighboring blocks.

[0079] Inter-frame predictor 332 can deduce the prediction block of the current block based on reference blocks (reference sample arrays) specified by motion vectors on a reference frame. In this case, to reduce the amount of motion information transmitted in the inter-frame prediction mode, motion information can be predicted on a block, sub-block, or sample basis based on the correlation of motion information between neighboring blocks and the current block. Motion information may include motion vectors and reference frame indices. Motion information may also include inter-frame prediction direction (L0 prediction, L1 prediction, Bi prediction, etc.) information. In the case of inter-frame prediction, neighboring blocks may include spatially adjacent blocks existing in the current frame and temporally adjacent blocks existing in the reference frame. For example, inter-frame predictor 332 can configure a motion information candidate list based on neighboring blocks and deduce the motion vector and / or reference frame index of the current block based on received candidate selection information. Inter-frame prediction can be performed based on various prediction modes, and the information about the prediction may include information indicating the mode of inter-frame prediction for the current block.

[0080] Adder 340 can generate a reconstruction signal (reconstructed frame, reconstruction block, reconstruction sample array) by adding the obtained residual signal to the prediction signal (prediction block, prediction sample array) output from the predictor (including inter-frame predictor 332 and / or intra-frame predictor 331). If the block to be processed has no residual (e.g., when a skip mode is applied), the prediction block can be used as the reconstruction block.

[0081] Adder 340 can be called a reconstructor or reconstruction block generator. The generated reconstructed signal can be used for intra-frame prediction of the next block to be processed in the current frame, and can be filtered and output as described below, or it can be used for inter-frame prediction of the next frame.

[0082] In addition, Luminance Mapping and Chromaticity Scaling (LMCS) can be applied during the image decoding process.

[0083] Filter 350 can improve subjective / objective image quality by applying filtering to the reconstructed signal. For example, filter 350 can generate a modified reconstructed image by applying various filtering methods to the reconstructed image and store the modified reconstructed image in memory 360 (specifically, the DPB of memory 360). Various filtering methods may include, for example, deblocking filtering, sample adaptive shifting, adaptive loop filtering, bilateral filtering, etc.

[0084] The (modified) reconstructed frame stored in the DPB of memory 360 can be used as a reference frame in inter-frame predictor 332. Memory 360 can store motion information of blocks from which motion information in the current frame is derived (or decoded) and / or motion information of reconstructed blocks in the frame. The stored motion information can be sent to inter-frame predictor 221 for use as motion information of spatially adjacent blocks or temporally adjacent blocks. Memory 360 can store reconstructed samples of reconstructed blocks in the current frame and can transmit the reconstructed samples to intra-frame predictor 331.

[0085] In this disclosure, the embodiments described in the filter 260, inter-frame predictor 221, and intra-frame predictor 222 of the encoding device 200 can be the same as, or applied to, the filter 350, inter-frame predictor 332, and intra-frame predictor 331 of the decoding device 300, respectively. The same content can also be applied to the inter-frame predictor 332 and intra-frame predictor 331.

[0086] In this disclosure, at least one of quantization / inverse quantization and / or transformation / inverse transformation may be omitted. When quantization / inverse quantization is omitted, the transformation coefficients of the quantization may be referred to as transformation coefficients. When transformation / inverse transformation is omitted, the transformation coefficients may be referred to as coefficients or residual coefficients, or for the sake of consistency, they may still be referred to as transformation coefficients.

[0087] In this disclosure, quantization transform coefficients and transform coefficients can be referred to as transform coefficients and scaling transform coefficients, respectively. In this case, residual information can include information about the transform coefficients, and this information can be signaled via residual coding syntax. Transform coefficients can be derived based on residual information (or information about the transform coefficients), and scaling transform coefficients can be derived by inverse transforming (scaling) the transform coefficients. Residual samples can be derived based on the inverse transform (scaling) of the scaling transform coefficients. This can also be applied / expressed in other parts of this disclosure.

[0088] Furthermore, as mentioned above, prediction is performed during video encoding to improve compression efficiency. This allows the generation of a prediction block, including prediction samples of the current block, as the block to be encoded (i.e., the target encoding block). Here, the prediction block includes prediction samples in the spatial domain (or pixel domain). The prediction block is derived in the same manner in both the encoding and decoding devices, and the encoding device can signal the decoding device with information about the residual between the original block and the prediction block (residual information), rather than the original sample values ​​of the original block, thereby improving image encoding efficiency. The decoding device can derive a residual block including residual samples based on the residual information, add the residual block and the prediction block to generate a reconstructed block including reconstructed samples, and generate a reconstructed image including the reconstructed block.

[0089] Residual information can be generated through transformation and quantization processes. For example, the encoding device can derive a residual block between the original block and the prediction block, perform a transformation process on the residual samples (residual sample array) included in the residual block to derive transform coefficients, perform a quantization process on the transform coefficients to derive quantized transform coefficients, and signal the relevant residual information (via bitstream) to the decoding device. Here, the residual information may include the values ​​of the quantized transform coefficients, their positions, the transform technique, the transform core, and the values ​​of quantization parameters. The decoding device can perform dequantization / inverse transform processes based on the residual information and derive residual samples (or residual blocks). The decoding device can generate a reconstructed frame based on the prediction block and the residual block. Furthermore, as a reference for inter-frame prediction of subsequent reference frames, the encoding device can dequantize / inverse transform the quantized transform coefficients to derive residual blocks and generate a reconstructed frame based on them.

[0090] Intra-frame prediction can refer to the generation of prediction samples for the current block based on reference samples in the frame to which the current block belongs (hereinafter referred to as the current frame). When applying intra-frame prediction to the current block, neighboring reference samples to be used for intra-frame prediction of the current block can be derived. The neighboring reference samples of the current block may include samples adjacent to the left boundary of the current block of size nW×nH and a total of 2×nH samples adjacent to the lower left of the current block, samples adjacent to the top boundary of the current block and a total of 2×nW samples adjacent to the upper right of the current block, and samples adjacent to the upper left of the current block. Alternatively, the neighboring reference samples of the current block may include multiple columns of upper adjacent samples and multiple rows of left adjacent samples. In addition, the neighboring reference samples of the current block may include a total of nH samples adjacent to the right boundary of the current block of size nW×nH, a total of nW samples adjacent to the lower boundary of the current block, and samples adjacent to the lower right of the current block.

[0091] However, some neighboring reference samples of the current block may not have been decoded or may be unavailable. In this case, the decoder can construct neighboring reference samples to be used for prediction by replacing unavailable samples with available samples. Alternatively, the neighboring reference samples to be used for prediction can be configured by interpolation of available samples.

[0092] When deriving neighboring reference samples, (i) the predicted sample can be derived based on the average or interpolation of the neighboring reference samples of the current block, or (ii) the predicted sample can be derived based on a reference sample existing in a specific (predictive) direction relative to the predicted sample in the neighboring reference samples of the current block. Case (i) can be referred to as non-directional mode or non-angular mode, and case (ii) can be referred to as directional mode or angular mode.

[0093] Alternatively, prediction samples can be generated by interpolating the first neighboring sample in the prediction direction of the intra-prediction mode of the current block and the second neighboring sample in the opposite direction from the prediction sample in the adjacent reference samples. This can be called linear interpolation intra-prediction (LIP). Furthermore, a linear model (LM) can be used to generate chroma prediction samples based on luminance samples. This can be called LM mode or chroma component LM (CCLM) mode.

[0094] Alternatively, provisional prediction samples for the current block can be derived based on filtered neighboring reference samples, or by weighted summing of the provisional prediction samples with at least one reference sample derived from the existing neighboring reference samples (i.e., unfiltered neighboring reference samples) according to the intra-prediction mode. This process can be referred to as position-dependent intra-prediction (PDPC).

[0095] Alternatively, the reference sample line with the highest prediction accuracy among the adjacent reference sample lines of the current block is selected, and the predicted sample is derived using the reference sample in the prediction direction of the selected line. In this case, intra-frame prediction coding can be performed by instructing (signaling) the reference sample line used to the decoding device. The above situation can be referred to as multi-reference line intra-frame prediction or MRL-based intra-frame prediction.

[0096] Furthermore, the current block can be divided into vertical or horizontal sub-partitions and intra-prediction can be performed based on the same intra-prediction mode. However, neighboring reference samples can be derived and used on a sub-partition basis. In other words, the intra-prediction mode for the current block applies to sub-partitions in this case, but in some situations, intra-prediction performance can be improved by deriving and using neighboring reference samples on a sub-partition basis. This prediction method can be called intra-prediction based on intra-partition sub-partitions (ISP).

[0097] The intra-prediction methods described above can be referred to as intra-prediction types to distinguish them from intra-prediction modes. Intra-prediction types can be referred to using various terms, such as intra-prediction techniques or additional intra-prediction modes. For example, an intra-prediction type (or additional intra-prediction mode, etc.) can include at least one of LIP, PDPC, MRL, and ISP. General intra-prediction methods that exclude specific intra-prediction types such as LIP, PDPC, MRL, and ISP can be referred to as normal intra-prediction types. When no specific intra-prediction type is applied, normal intra-prediction types can generally be applied, and prediction can be performed based on the intra-prediction modes described above. Furthermore, post-processing filtering can be performed on the derived prediction samples if necessary.

[0098] Specifically, the intra-frame prediction process may include an intra-frame prediction mode / type determination step, a neighboring reference sample derivation step, and a prediction sample derivation step based on the intra-frame prediction mode / type. Furthermore, if necessary, a post-filtering step may be performed on the derived prediction samples.

[0099] When applying intra-prediction, the intra-prediction modes of neighboring blocks can be used to determine the intra-prediction mode to be applied to the current block. For example, the decoding device can select one of the most probable mode (MPM) candidates in the MPM list derived based on additional candidate modes and the intra-prediction modes of the current block's neighboring blocks (e.g., left and / or top neighboring blocks), or select one of the remaining intra-prediction modes not included in the MPM candidates (and planar modes) based on remaining intra-prediction mode information. The MPM list can be configured to include or exclude planar modes as candidates. For example, the MPM list can have 6 candidates when it includes planar modes as candidates, and 5 candidates when it does not. When the MPM list does not include planar modes as candidates, a non-planar flag (e.g., intra_luma_not_planar_flag) indicating whether the intra-prediction mode of the current block is not a planar mode can be signaled. For example, the MPM flag can be signaled first, and when the MPM flag is 1, the MPM index and non-planar flag can be signaled. Furthermore, when the non-flat flag is set to 1, the MPM index can be signaled. Here, the fact that the MPM list does not include flat patterns as candidates is that flat patterns are always considered MPMs, not that they are not MPMs. Therefore, the flag (non-flat flag) is signaled first to check if it is a flat pattern.

[0100] For example, MPM flags (e.g., `intra_luma_mpm_flag`) can indicate whether the intra-prediction mode applied to the current block is among the MPM candidates (and planar modes) or among the remaining modes. An MPM flag of 1 indicates that the intra-prediction mode of the current block is among the MPM candidates (and planar modes), while an MPM flag of 0 indicates that the intra-prediction mode of the current block is not among the MPM candidates (and planar modes). Non-planar flags of 0 (e.g., `intra_luma_not_planar_flag`) indicate that the intra-prediction mode of the current block is planar, while a non-planar flag of 1 indicates that the intra-prediction mode of the current block is not planar. The MPM index can be signaled in the form of `mpm_idx` or `intra_luma_mpm_idx` syntax elements, and the remaining intra-prediction mode information can be signaled in the form of `rem_intra_luma_pred_mode` or `intra_luma_mpm_remainder` syntax elements. For example, the remaining intra-prediction mode information can be indexed in the order of prediction mode numbers to indicate one of the remaining intra-prediction modes not included in the MPM candidates (and planar modes) among all intra-prediction modes. The intra-prediction mode can be an intra-prediction mode for the luma component (sample). In the following, the intra-prediction mode information may include at least one of the following: an MPM flag (e.g., intra_luma_mpm_flag), a non-planar flag (e.g., intra_luma_not_planar_flag), an MPM index (e.g., mpm_idx or intra_luma_mpm_idx), or remaining intra-prediction mode information (rem_intra_luma_luma_mpm_mode or intra_luma_mpminder). In this disclosure, the MPM list can be referred to using various terms such as the MPM candidate list and candModeList. When applying MIP to the current block, the individual MPM flags used for MIP (e.g., intra_mip_mpm_flag), MPM index (e.g., intra_mip_mpm_idx), and remaining intra-prediction mode information (e.g., intra_mip_mpm_remainder) can be signaled, but the non-plane flags can be left unsigned.

[0101] In other words, generally speaking, when performing block segmentation of an image, the current block to be encoded and its neighboring blocks have similar image characteristics. Therefore, there is a high probability that the current block and its neighboring blocks have the same or similar intra-prediction modes. Thus, the encoder can use the intra-prediction modes of neighboring blocks to encode the intra-prediction mode of the current block.

[0102] For example, the decoding / encoding unit can construct a most probable mode (MPM) list for the current block. This MPM list can be referred to as an MPM candidate list. Here, MPM can refer to a mode used to improve coding efficiency during intra-frame prediction mode coding by considering the similarity between the current block and neighboring blocks. As mentioned above, the MPM list can be constructed to include planar modes or to exclude planar modes. For example, when the MPM list includes planar modes, the number of candidates in the MPM list can be 6. When the MPM list does not include planar modes, the number of candidates in the MPM list can be 5.

[0103] The encoder / decoder can build a list of MPMs that includes five or six MPMs.

[0104] To construct an MPM list, three types of modes can be considered, such as default intra-frame mode, neighbor intra-frame mode, and deduced intra-frame mode.

[0105] For the neighbor intra-frame mode, two neighbor blocks can be considered: the left neighbor block and the upper neighbor block.

[0106] As mentioned above, if the MPM list is constructed to exclude flat patterns, flat patterns can be excluded from the list, and the number of candidates in the MPM list can be set to five.

[0107] In addition, non-directional (or non-corner) modes in intra-frame prediction modes can include DC modes based on the average of neighboring reference samples of the current block or planar modes based on interpolation.

[0108] Furthermore, when applying inter-frame prediction, the predictor of the encoding / decoding device can derive prediction samples by performing inter-frame prediction on a block-by-block basis. Inter-frame prediction can be applied when performing prediction on the current block. That is, the predictor of the encoding / decoding device (more specifically, the inter-frame predictor) can derive prediction samples by performing inter-frame prediction on a block-by-block basis. Inter-frame prediction can represent a prediction derived by a method that depends on data elements (e.g., sample values ​​or motion information) of (one or more) frames other than the current frame. When applying inter-frame prediction to the current block, the prediction block (prediction sample array) for the current block can be derived based on a reference block (reference sample array) on a reference frame indicated by a reference frame index, specified by a motion vector. In this case, to reduce the amount of motion information transmitted in the inter-frame prediction mode, the motion information of the current block can be predicted on a block, sub-block, or sample-by-sample basis based on the correlation of motion information between adjacent blocks and the current block. Motion information can include motion vectors and reference frame indices. Motion information can further include inter-frame prediction type (L0 prediction, L1 prediction, Bi prediction, etc.) information. In the application of inter-frame prediction, neighboring blocks can include spatially adjacent blocks existing in the current frame and temporally adjacent blocks existing in a reference frame. The reference frame including the reference block and the reference frame including the temporally adjacent blocks can be the same or different from each other. Temporally adjacent blocks can be referred to by names such as juxtaposed reference blocks, juxtaposed CUs (ColCU), etc., and the reference frame including the temporally adjacent blocks can be referred to as a juxtaposed frame (ColPic). For example, a motion information candidate list can be configured based on the neighboring blocks of the current block, and a signal can be sent indicating which candidate's flag or index information to select (use) for deriving the motion vector of the current block and / or the reference frame index. Inter-frame prediction can be performed based on various prediction modes, and for example, in skip mode and merge mode, the motion information of the current block can be the same as the motion information of the selected neighboring block. In skip mode, unlike merge mode, residual signals may not be sent. In motion vector prediction (MVP) mode, the motion vector of the selected neighboring block can be used as a motion vector predictor, and the motion vector difference can be signaled. In this case, the motion vector of the current block can be derived by using the sum of the motion vector predictor and the motion vector difference.

[0109] Depending on the inter-frame prediction type (L0 prediction, L1 prediction, Bi prediction, etc.), motion information may further include L0 motion information and / or L1 motion information. The L0 direction motion vector may be referred to as the L0 motion vector or MVL0, and the L1 direction motion vector may be referred to as the L1 motion vector or MVL1. Prediction based on the L0 motion vector may be called L0 prediction, prediction based on the L1 motion vector may be called L1 prediction, and prediction based on both L0 and L1 motion vectors may be called bi-prediction. Here, the L0 motion vector may indicate the motion vector associated with the reference frame list L0, and the L1 motion vector may indicate the motion vector associated with the reference frame list L1. The reference frame list L0 may include frames that precede the current frame in output order, and the reference frame list L1 may include frames that follow the current frame in output order, serving as reference frames. Previous frames may be referred to as forward (reference) frames, and subsequent frames may be referred to as reverse (reference) frames. The reference frame list L0 may further include frames that follow the current frame in output order, serving as reference frames. In this scenario, previous frames can be indexed first in the reference frame list L0, and then subsequent frames can be indexed. The reference frame list L1 can further include frames that precede the current frame in the output order as reference frames. In this case, subsequent frames can be indexed first in the reference frame list L1, and then previous frames can be indexed. Here, the output order can correspond to the frame sequence count (POC) order.

[0110] The video / image coding process based on inter-frame prediction can schematically include, for example, the following.

[0111] Figure 4 An example of a video / image coding method based on inter-frame prediction is shown.

[0112] The encoding device performs inter-frame prediction on the current block (S400). The encoding device can deduce the inter-frame prediction mode and motion information of the current block, and generate prediction samples for the current block. Here, the inter-frame prediction mode determination process, the motion information derivation process, and the prediction sample generation process can be executed simultaneously, and any one of these processes can be executed earlier than the others. For example, the inter-frame prediction unit of the encoding device may include a prediction mode determination unit, a motion information derivation unit, and a prediction sample derivation unit. The prediction mode determination unit can determine the prediction mode of the current block, the motion information derivation unit can deduce the motion information of the current block, and the prediction sample derivation unit can deduce the prediction samples for the current block. For example, the inter-frame prediction unit of the encoding device can search for blocks similar to the current block in a predetermined area (search area) of a reference frame through motion estimation, and deduce a reference block whose difference from the current block is the smallest or equal to or less than a predetermined standard. Based on this derivation, a reference frame index indicating the reference frame where the reference block is located can be derived, and a motion vector can be derived based on the positional difference between the reference block and the current block. The encoding device can determine the mode applicable to the current block among various prediction modes. The encoding device can compare the RD costs of various prediction modes and determine the optimal prediction mode for the current block.

[0113] For example, when a skip mode or merge mode is applied to the current block, the encoding device can configure a merge candidate list (described below) and deduce a reference block among the reference blocks indicated by the merge candidates included in the merge candidate list that has a difference from the current block that is smallest, equal to, or less than a predetermined standard. In this case, a merge candidate associated with the deduced reference block can be selected, and merge index information indicating the selected merge candidate can be generated and signaled to the decoding device. The motion information of the current block can be deduced using the motion information of the selected merge candidate.

[0114] As another example, when the (A)MVP mode is applied to the current block, the encoding device can configure an (A)MVP candidate list, as described below, and use the motion vector of the selected MVP candidate from the motion vector prediction sub-candidates included in the (A)MVP candidate list as the MVP of the current block. In this case, for example, the motion vector of a reference block derived through motion estimation can be used as the motion vector of the current block, and the MVP candidate with the motion vector having the smallest difference from the motion vector of the current block can be selected as the MVP candidate. The motion vector difference (MVD) can be derived, which is the difference obtained by subtracting the MVP from the motion vector of the current block. In this case, information about the MVD can be signaled to the decoding device. Furthermore, when the (A)MVP mode is applied, the value of the reference screen index can be configured as reference screen index information and signaled to the decoding device separately.

[0115] The encoding device can derive residual samples based on the predicted samples (S410). The encoding device can derive residual samples by comparing the original samples and the predicted samples of the current block.

[0116] The encoding device encodes image information, including prediction information and residual information (S420). The encoding device can output the encoded image information in the form of a bitstream. The prediction information may include information about prediction mode information (e.g., skip flag, merge flag, or mode index, etc.) and information about motion information, as information related to the prediction process. The information about motion information may include candidate selection information (e.g., merge index, MVP flag, or MVP index), which is information used to derive motion vectors. In addition, the information about motion information may include information about MVD and / or reference frame index information. Furthermore, the information about motion information may include information indicating whether L0 prediction, L1 prediction, or dual prediction is applied. The residual information is information about residual samples. The residual information may include information about the quantization transform coefficients used for the residual samples.

[0117] The output bitstream can be stored in a (digital) storage medium and transmitted to a decoding device, or transmitted to a decoding device via a network.

[0118] Furthermore, as mentioned above, the encoding device can generate a reconstructed picture (including reconstructed samples and reconstructed blocks) based on reference samples and residual samples. This is to derive the same prediction result as that performed by the decoding device, which can increase encoding efficiency. Therefore, the encoding device can store the reconstructed picture (or reconstructed samples or reconstructed blocks) in memory and use the reconstructed picture as a reference picture. As mentioned above, the in-loop filtering process can be further applied to the reconstructed picture.

[0119] The video / image decoding process based on inter-frame prediction can schematically include, for example, the following.

[0120] Figure 5 An example of a video / image decoding method based on inter-frame prediction is shown.

[0121] Reference Figure 5 The decoding device can perform operations corresponding to those performed by the encoding device. The decoding device can perform predictions on the current block and derive prediction samples based on the received prediction information.

[0122] Specifically, the decoding device can determine the prediction mode of the current block based on the received prediction information (S500). The decoding device can determine which inter-frame prediction mode to apply to the current block based on the prediction mode information in the prediction information.

[0123] For example, a merge flag can be used to determine whether to apply a merge mode or (A)MVP mode to the current block. Alternatively, a variety of inter-frame prediction mode candidates can be selected based on a mode index. Inter-frame prediction mode candidates may include skip mode, merge mode, and / or (A)MVP mode, or may include various inter-frame prediction modes described below.

[0124] The decoding device derives motion information for the current block based on the determined inter-frame prediction mode (S510). For example, when a skip mode or merge mode is applied to the current block, the decoding device can be configured to select a merge candidate from the merge candidate list described below. Here, the selection can be performed based on selection information (merge index). The motion information for the current block can be derived using the motion information of the selected merge candidate. The motion information of the selected merge candidate can be used as the motion information for the current block.

[0125] As another example, when applying the (A)MVP mode to the current block, the decoding device can configure an (A)MVP candidate list, as described below, and use the motion vector of the selected MVP candidate from the motion vector prediction sub-candidates included in the (A)MVP candidate list as the MVP of the current block. Here, selection can be performed based on selection information (MVP flag or MVP index). In this case, the MVD of the current block can be derived based on information about the MVD, and the motion vector of the current block can be derived based on the MVP and MVD of the current block. Furthermore, the reference frame index of the current block can be derived based on reference frame index information. The frame indicated by the reference frame index in the reference frame list of the current block can be derived as the reference frame referenced by the inter-frame prediction of the current block.

[0126] Furthermore, as described below, the motion information of the current block can be derived without a candidate list configuration, and in this case, the motion information of the current block can be derived based on the process disclosed in the prediction mode. In this case, the candidate list configuration can be omitted.

[0127] The decoding device can generate prediction samples for the current block based on the motion information of the current block (S520). In this case, a reference frame can be derived based on the reference frame index of the current block, and prediction samples for the current block can be derived by using samples of the reference block indicated by the motion vector of the current block on the reference frame. In some cases, a prediction sample filtering process for all or some of the prediction samples for the current block can be further performed.

[0128] For example, the inter-frame prediction unit of the decoding device may include a prediction mode determination unit, a motion information derivation unit, and a prediction sample derivation unit. The prediction mode determination unit can determine the prediction mode of the current block based on the received prediction mode information, the motion information derivation unit can derive the motion information (motion vector and / or reference frame index) of the current block based on information about the received motion information, and the prediction sample derivation unit can derive the prediction samples of the current block.

[0129] The decoding device generates a residual sample for the current block based on the received residual information (S530). The decoding device can generate a reconstruction sample for the current block based on the prediction sample and the residual sample, and generate a reconstructed image based on the generated reconstruction sample (S540). Thereafter, as described above, the in-loop filtering process can be further applied to the reconstructed image.

[0130] Figure 6 The inter-frame prediction process is illustrated schematically.

[0131] Reference Figure 6 As described above, the inter-frame prediction process may include an inter-frame prediction mode determination step, a motion information derivation step based on the determined prediction mode, and a prediction processing step (prediction sample generation) based on the derivation of motion information. The inter-frame prediction process may be performed by the encoding and decoding devices described above. In this document, the encoding device may include an encoding device and / or a decoding device.

[0132] Reference Figure 6 The encoding device determines the inter-frame prediction mode for the current block (S600). Various inter-frame prediction modes can be used for the prediction of the current block in the frame. For example, various modes can be used, such as merge mode, skip mode, motion vector prediction (MVP) mode, affine mode, sub-block merge mode, merge with MVD (MMVD) mode, and history motion vector prediction (HMVP) mode. Decoder-side motion vector refinement (DMVR) mode, adaptive motion vector resolution (AMVR) mode, dual prediction with CU-level weights (BCW), and bidirectional optical flow (BDOF) can be further used as additional modes. Affine mode can also be called affine motion prediction mode. MVP mode can also be called advanced motion vector prediction (AMVP) mode. In this paper, some modes and / or motion information candidates derived from some modes can also be included in one of the motion information-related candidates in other modes. For example, HMVP candidates can be added to the merge candidates of merge / skip mode, or to the MVP candidates of MVP mode. If an HMVP candidate is used as a motion information candidate for merge mode or skip mode, then the HMVP candidate can be called an HMVP merge candidate.

[0133] Prediction mode information, indicating the inter-frame prediction mode of the current block, can be signaled from the encoding device to the decoding device. In this case, the prediction mode information can be included in the bitstream and received by the decoding device. The prediction mode information may include index information indicating one of a plurality of candidate modes. Alternatively, the inter-frame prediction mode can be indicated by hierarchical signaling of flag information. In this case, the prediction mode information may include one or more flags. For example, a skip flag can be signaled to indicate whether a skip mode is applied; a merge flag can be signaled to indicate whether a merge mode is applied when a skip mode is not applied; and an MVP mode can be signaled when a merge mode is not applied, or further flags for additional differentiation can be signaled. Affine modes can be signaled as independent modes or as subordinate modes to merge or MVP modes. For example, affine modes may include affine merge mode and affine MVP mode.

[0134] The encoding device derives the motion information of the current block (S610). Motion information can be derived based on the inter-frame prediction mode.

[0135] The encoding device can use motion information of the current block to perform inter-frame prediction. The encoding device can derive the optimal motion information for the current block through a motion estimation process. For example, the encoding device can use the original block in the original frame of the current block to search for highly correlated similar reference blocks within a predetermined search range in the reference frame, on a fractional pixel basis, and derive motion information using the searched reference blocks. Block similarity can be derived based on the difference in phase-based sample values. For example, block similarity can be calculated based on the sum of absolute differences (SAD) between the current block (or its template) and a reference block (or its template). In this case, motion information can be derived based on the reference block with the minimum SAD in the search area. The derived motion information can be signaled to the decoding device based on the inter-frame prediction mode and various methods.

[0136] The encoding device performs inter-frame prediction based on the motion information of the current block (S620). The encoding device can derive (one or more) prediction samples for the current block based on the motion information. The current block, which includes the prediction samples, can be called a prediction block.

[0137] Figure 7 The hierarchical structure of encoded image information is illustrated schematically.

[0138] Figure 7 The video / image encoded according to the coding layer and structure of this disclosure can be illustrated schematically. (See reference...) Figure 7Encoded video / images can be divided into a Video Coding Layer (VCL) that processes video / images and performs video / image decoding, a subsystem that transmits and stores encoded information, and a Network Abstraction Layer (NAL) that exists between the VCL and the subsystems and is responsible for the functions.

[0139] For example, in VCL, VCL data that includes compressed image data (slice data) can be generated, or a set of parameters that includes a Picture Parameter Set (PPS), a Sequence Parameter Set (SPS), a Video Parameter Set (VPS), or a set of parameters that includes additional supplemental enhancement information (SEI) messages required for image decoding processing can be generated.

[0140] For example, in NAL, a NAL cell can be generated by adding header information (NAL cell header) to the raw byte sequence payload (RBSP) generated in VCL. In this case, RBSP can refer to slice data, parameter sets, SEI messages, etc. generated in VCL. The NAL cell header can include NAL cell type information specified based on the RBSP data included in the corresponding NAL cell.

[0141] For example, such as Figure 7 As shown, NAL units can be classified into VCL NAL units and non-VCL NAL units based on the RBSP generated in the VCL. VCL NAL units can refer to NAL units that include information about the image (slice data), while non-VCL NAL units can refer to NAL units that include information required for image decoding (parameter set or SEI message).

[0142] Header information can be attached to the aforementioned VCL NAL units and non-VCL NAL units according to the subsystem's data standard, and both VCL NAL units and non-VCL NAL units including header information can be transmitted over a network. For example, NAL units can be converted to predetermined standard data formats such as H.266 / VVC file format, Real-time Transport Protocol (RTP), Transport Streaming (TS), etc., and transmitted via various networks.

[0143] In addition, as mentioned above, the type of NAL cell can be specified according to the RBSP data structure included in the NAL cell, and information about the NAL cell type can be stored in the NAL cell header and notified by a signal.

[0144] For example, NAL units can be classified into VCL NAL unit types and non-VCL NAL unit types based on whether they include information about the image (slice data). Additionally, VCL NAL unit types can be classified based on the characteristics and type of the image included within the VCL NAL unit, while non-VCL NAL unit types can be classified based on the type of parameter set.

[0145] The following can be an example of a NAL cell type specified based on the type of the parameter set included in a non-VCL NAL cell type.

[0146] - Adaptive Parameter Set (APS) NAL Units: Includes NAL unit types from the APS.

[0147] - Decoding Parameter Set (DPS) NAL Unit: Includes NAL unit types of DPS.

[0148] - Video Parameter Set (VPS) NAL Unit: Includes the NAL unit type of the VPS.

[0149] - Sequence Parameter Set (SPS) NAL Units: Includes NAL unit types of SPS.

[0150] - Picture Parameter Set (PPS) NAL Unit: Includes NAL unit types for PPS.

[0151] - Header (PH) NAL Unit: Includes the NAL unit type of PH.

[0152] The aforementioned NAL unit type can have syntax information about the NAL unit type, which can be stored in the NAL unit header and signaled. For example, this syntax information can be nal_unit_type, and the NAL unit type can be specified as the nal_unit_type value.

[0153] Furthermore, as described above, a frame can include multiple slices, and each slice can include a slice header and slice data. In this case, a frame header can be added (embedded) to multiple slices (a set of slice headers and slice data). The frame header (frame header syntax) can include information / parameters that can be applied collectively to the frame. The slice header (slice header syntax) can include information / parameters that can be applied collectively to the slice. APS (APS syntax) or PPS (PPS syntax) can include information / parameters that can be applied collectively to one or more slices or frames. SPS (SPS syntax) can include information / parameters that can be applied collectively to one or more sequences. VPS (VPS syntax) can include information / parameters that can be applied collectively to multiple layers. DPS (DPS syntax) can include information / parameters that can be applied collectively to the entire image. DPS can include information / parameters related to the concatenation of coded video sequences (CVS). In this disclosure, the High-Level Syntax (HLS) can include at least one of APS syntax, PPS syntax, SPS syntax, VPS syntax, DPS syntax, frame header syntax, and slice header syntax.

[0154] Furthermore, as mentioned above, typically a NAL unit type can be set for a frame, and as also mentioned above, the NAL unit type can be notified by a signal via the `nal_unit_type` in the NAL unit header, which includes the NAL unit of the slice. The following shows an example of the NAL unit type code and NAL unit type class.

[0155] [Table 1]

[0156] Furthermore, as mentioned above, a frame can consist of one or more slices. Additionally, parameters describing the frame can be signaled via the frame header (PH), and parameters describing a slice can be signaled via the slice header (SH). The PH can be transmitted within its own NAL cell type. Furthermore, the SH can exist at the beginning of the NAL cell containing the slice's payload (i.e., slice data).

[0157] For example, the syntax elements of a PH that use signals for notification can be as follows.

[0158] [Table 2]

[0159] Furthermore, using PH can mean that each encoded frame must have at least two NAL units. For example, one of the two units could be a NAL unit for the PH, and the other could be a NAL unit for the encoded slice that includes the slice header (SH) and slice data. This can be problematic for bitstreams with low bitrates, as the additional NAL units per frame can significantly impact the bitrate. Therefore, it might be preferable for PH to have a mode that does not consume new NAL units.

[0160] Therefore, this disclosure provides embodiments for solving the above-mentioned problems. The proposed embodiments can be applied individually or in combination.

[0161] As an example, a method is proposed to signal a flag indicating the presence of PH NAL units in a coded video layer sequence (CLVS). That is, the flag can indicate the presence of a picture header in a NAL unit (i.e., a PH NAL unit) or a slice header. Here, for example, CLVS can refer to a sequence of picture units (PUs) with the same nuh_layer_id value. A picture unit can be a set of NAL units of a coded picture. Furthermore, the high-level parameter set can be, for example, a sequence parameter set (SPS), a picture parameter set (PPS), or a slice header. The flag can be called ph_nal_present_flag. Alternatively, the flag can be called a PH NAL presence flag.

[0162] Furthermore, regarding the PH NAL presence flag, this disclosure proposes an implementation where the value of ph_nal_present_flag is constrained to be the same for all SPS referenced for frames of the same CVS. This constraint can mean that the value of ph_nal_present_flag must be the same for a coded video sequence in a multi-layer bitstream.

[0163] Furthermore, as an example, when the value of ph_nal_present_flag is equal to 1, there exists a PH NAL unit, and this PH NAL unit is associated with the video coding layer (VCL) NAL unit of the image.

[0164] Furthermore, as an example, when the value of ph_nal_present_flag is equal to 0 (i.e., when there is no PH NAL unit for each frame), a method is proposed to apply the following constraints.

[0165] For example, the above constraints can be as follows.

[0166] First, all frames in CLVS can consist of only one slice.

[0167] Secondly, the PH NAL unit may not exist. The PH syntax table can exist in the slice layer RBSP along with the slice header (SH) and slice data. That is, the PH syntax table can exist in the slice header.

[0168] Third, the PH and SH syntax tables can begin at byte alignment positions. To achieve this, byte alignment bits can be added between the PH and SH.

[0169] Fourth, the value of picture_header_extension_present_flag in all PPS instances of SPS can be 0.

[0170] Fifth, all syntax elements that can exist in PH or SH can exist in PH but not SH.

[0171] Furthermore, as an example, a method for updating access unit detection can be proposed. That is, instead of checking the PH, each new VCL NAL unit can signify a new access unit (AU). Specifically, when the value of ph_nal_present_flag indicates that a PH NAL unit does not exist, the VCL NAL unit including ph_nal_present_flag is not the VCL NAL unit of the previous AU (i.e., the frame of the previous AU), and can signify the VCL NAL unit resolving the new AU (i.e., the frame of the new AU). Therefore, when the value of ph_nal_present_flag indicates that a PH NAL unit does not exist, the VCL NAL unit including ph_nal_present_flag can be the first VCL NAL unit of the frame of the new AU (e.g., the current frame to be decoded). Here, AU can signify a set of frame units (PUs) belonging to different layers and including encoded frames that are related to the output of the decoded frame buffer (DPB) at the same time. In addition, PU can signify a set of NAL units that are associated and have a sequential decoding order and include a single encoded frame. That is, PU can refer to a set of NAL units associated with each other and having a sequential decoding order for an encoded frame. Furthermore, when the bitstream is a single-layer bitstream rather than a multi-layer bitstream, AU can be the same as PU.

[0172] The embodiments proposed in this disclosure can be implemented as follows.

[0173] For example, the SPS syntax for signaling the ph_nal_present_flag proposed in the embodiments of this disclosure can be as follows.

[0174] [Table 3]

[0175] Referring to Table 3, SPS may include ph_nal_present_flag.

[0176] For example, the semantics of the syntax element ph_nal_present_flag can be shown in the table below.

[0177] [Table 4]

[0178] For example, referring to Table 4, the syntax element `ph_nal_present_flag` can indicate whether each encoded sheet of the CLVS for the reference SPS has a NAL unit with the same `nal_unit_type` as `PH_NUT`. For example, `ph_nal_present_flag` equal to 1 can indicate that each encoded sheet of the CLVS for the reference SPS has a NAL unit with the same `nal_unit_type` as `PH_NUT`. Furthermore, for example, `ph_nal_present_flag` equal to 0 can indicate that each encoded sheet of the CLVS for the reference SPS does not have a NAL unit with the same `nal_unit_type` as `PH_NUH`.

[0179] In addition, for example, the following can be applied when ph_nal_present_flag is 1.

[0180] -nal_unit_type for PH_NUT NAL units (i.e., PH NAL units) may not exist in the CLVS of the reference SPS.

[0181] - Referring to SPS's CLVS, each frame may include a slice.

[0182] -PH can be present in sliced ​​RBSP layers.

[0183] Furthermore, although the method of signaling ph_nal_present_flag via SPS is proposed in Tables 3 and 4, the method shown in Tables 3 and 4 is an implementation proposed in this disclosure. Implementations of signaling ph_nal_present_flag via PPS or slice header instead of SPS are also proposed.

[0184] Furthermore, for example, according to the embodiments proposed in this disclosure, the header syntax table and the header RBSP can be separately signaled as shown in the table below.

[0185] [Table 5]

[0186] In addition, the syntax for the signal notification header can be as follows.

[0187] [Table 6]

[0188] Furthermore, according to the embodiments proposed in this disclosure, for example, the slice layer RBSP can be signaled as follows.

[0189] [Table 7]

[0190] Furthermore, for example, according to the embodiments proposed in this disclosure, one or more constraints shown in the table below may be applied.

[0191] [Table 8]

[0192] For example, referring to Table 8, if ph_nal_present_flag is 0, then the value of picture_header_extension_present_flag can be 0.

[0193] Additionally, for example, bitstream suitability may require the value of pic_rpl_present_flag to be equal to 1 if the following two conditions are true.

[0194] -ph_nal_present_flag is 0, and the screen associated with PH is not an IDR screen.

[0195] -ph_nal_present_flag is 0, the screen associated with PH is the IDR screen, and sps_id_rpl_present_flag is equal to 1.

[0196] Here, rpl can refer to a list of reference screens.

[0197] Additionally, for example, when the value of ph_nal_present_flag is 0, bitstream suitability can require the value of pic_sao_enabled_present_flag to be equal to 1.

[0198] Furthermore, for example, when the value of ph_nal_present_flag is 0, bitstream suitability can require the value of pic_alf_enabled_present_flag to be equal to 1.

[0199] Furthermore, for example, when the value of ph_nal_present_flag is 0, bitstream suitability can require that the value of pic_deblocking_filter_override_present_flag must be equal to 1.

[0200] In addition, for example, implementation methods can be applied according to the following process.

[0201] Figure 8 The coding process according to an embodiment of the present disclosure is illustrated schematically.

[0202] Reference Figure 8 The encoding device can generate a NAL unit that includes information about the frame (S800). The encoding device can determine whether a NAL unit for the frame header exists (S810) and can decide whether a NAL unit for the frame header exists (S820).

[0203] For example, when a NAL unit for the picture header is present, the encoding device can generate a bitstream that includes a VCLNAL unit containing a slice header and a PH NAL unit containing a picture header (S830).

[0204] Furthermore, for example, when no NAL unit is available for the picture header, the encoding device can generate a bitstream that includes VCL NAL units containing both the slice header and the picture header (S840). That is, the picture header syntax structure can exist in the slice header.

[0205] Figure 9 The decoding process according to an embodiment of the present disclosure is illustrated schematically.

[0206] Reference Figure 9 The decoding device can receive a bitstream including NAL units (S900). Afterward, the decoding device can determine whether a NAL unit for the frame header exists (S910).

[0207] For example, when a NAL unit for the picture head is present, the decoding device can decode / reconstruct the picture / slice / block / sample based on the slice head in the VCL NAL unit and the picture head in the PH NAL unit (S920).

[0208] Furthermore, for example, when there is no NAL unit for the picture head, the decoding device can decode / reconstruct the picture / slice / block / sample based on the slice head and picture head in the VCL NAL unit (S930).

[0209] Here, the (encoded) bitstream may include one or more NAL units for decoding the picture. Additionally, the NAL unit may be a VCL NAL unit or a non-VCL NAL unit. For example, a VCL NAL unit may include information about the encoded slice, and a VCL NAL unit may have a NAL unit type of the "VCL" class shown in Table 1 above.

[0210] Furthermore, according to the embodiments proposed in this disclosure, the bitstream may include PH NAL units (NAL units for the frame header), or the bitstream may not include PH NAL units for the current frame. Information indicating the presence of PH NAL units (e.g., ph_nal_present_flag) can be signaled via an HLS (e.g., VPS, DPS, SPS, slice header, etc.).

[0211] Figure 10 This schematically illustrates the header configuration in a NAL unit based on the presence or absence of a PH NAL unit. For example, Figure 10 (a) shows the case where a PH NAL unit is present for the current frame. Figure 10 (b) shows the case where there is no PH NAL unit for the current frame, but the frame header is included in the VCL NAL unit.

[0212] For example, when a PH NAL unit is present, the frame header can be included within the PH NAL unit. Conversely, when a PH NAL unit is absent, the frame header can still be configured, but it can be included within another type of NAL unit. For example, the frame header can be included within a VCL NAL unit. A VCL NAL unit can include information about encoded slices. A VCL unit can include a slice header for encoding slices. For example, when a particular slice header includes information indicating that the encoded / associated slice is the first slice in a frame or sub-frame, the frame header can be included within a specific VAL NAL unit that includes that particular slice header. Alternatively, for example, when a PH NAL unit is absent, the frame header can be included within a non-VCL NAL unit such as a PPS NAL unit, an APS NAL unit, etc.

[0213] Figure 11 An image encoding method of the encoding device according to this document is illustrated schematically. Figure 11 The method disclosed in the article can be derived from Figure 2 The encoding device shown performs this. Specifically, for example, Figure 11 S1100 to S1130 can be performed by the entropy encoder of the encoding device. Furthermore, although not shown, the decoding process for the current frame can be performed by the predictor and residual processor of the encoding device.

[0214] The encoding device generates a Video Coding Layer (VCL) Network Abstraction Layer (NAL) unit (S1100) that includes a slice header and slice data of a slice in the current frame. The encoding device can generate a slice header and slice data of a slice in the current frame. The slice header and slice data can be included in the VCL NAL unit.

[0215] For example, the slice header may include syntax elements representing the parameters of the slice. Furthermore, the slice data may include, for example, prediction information and residual information for the blocks within the slice.

[0216] Furthermore, for example, the encoding device can generate and encode prediction information for blocks in a slice of the current frame. In this case, various prediction methods disclosed in this disclosure (e.g., inter-frame prediction or intra-frame prediction) can be applied. For example, the encoding device can determine whether to perform inter-frame prediction or intra-frame prediction on the block, and can determine a specific inter-frame prediction mode or a specific intra-frame prediction mode based on RD cost. Based on the determined mode, the encoding device can derive prediction samples for the block. The prediction information may include prediction mode information for the block.

[0217] In addition, for example, the encoding device can encode the residual information of the blocks in the current frame.

[0218] For example, an encoding device can derive residual samples by subtracting predicted samples from the original samples of a block.

[0219] Then, for example, the encoding device can quantize the residual samples to derive quantized residual samples, derive transform coefficients based on the quantized residual samples, and generate and encode residual information based on the transform coefficients. Alternatively, for example, the encoding device can quantize the residual samples to derive quantized residual samples, transform the quantized residual samples to derive transform coefficients, and generate and encode residual information based on the transform coefficients.

[0220] Furthermore, for example, the encoding device can derive prediction samples and residual samples of blocks in a slice of the current frame based on prediction information and residual information, and generate reconstructed samples / reconstructed frames of the current frame based on the prediction samples and residual samples.

[0221] The encoding device determines whether a PH NAL unit, which includes the picture header (PH) of the current frame, exists (S1110). For example, the encoding device may determine whether a PH NAL unit exists. For example, when a PH NAL unit exists, the encoding device may generate a PH NAL unit that includes the PH related to the current frame and / or a Video Coding Layer (VCL) NAL unit that includes information about a slice of the current frame (e.g., slice header and slice data). Alternatively, for example, when a PH NAL unit does not exist, the encoding device may generate a Video Coding Layer (VCL) NAL unit that includes the PH related to the current frame and information about a slice of the current frame (e.g., slice header and slice data). Furthermore, for example, when a flag indicates that a PH NAL unit does not exist, the current frame may include only one slice. Here, for example, the PH may include syntax elements representing parameters of the current frame.

[0222] The encoding device generates a PH and a flag indicating the presence or absence of the PH NAL cell based on the determined result (S1120). For example, the encoding device may generate a PH and a flag indicating the presence or absence of the PH NAL cell based on the determined result. For example, the flag may indicate the presence or absence of the PH NAL cell. For example, when the value of the flag is 1, the flag may indicate the presence of the PH NAL cell, and when the value of the flag is 0, the flag may indicate the absence of the PH NAL cell. Alternatively, for example, when the value of the flag is 0, the flag may indicate the presence of the PH NAL cell, and when the value of the flag is 1, the flag may indicate the absence of the PH NAL cell. The syntax element of this flag may be the aforementioned ph_nal_present_flag.

[0223] Furthermore, for example, when it is determined that a PH NAL unit exists, the PH of the current frame can be included in the PH NAL unit; when it is determined that a PH NAL unit does not exist, the PH of the current frame can be included in the Video Coding Layer (VCL) NAL unit containing the slice header and slice data of the slice of the current frame.

[0224] The encoding device encodes image information including the VCL NAL unit, PH, and the flag (S1130). The encoding device can encode image information including the VCL NAL unit, PH, and the flag. The image information may include the VCL NAL unit, PH, and the flag. Furthermore, for example, the image information may include a high-level syntax, and the flag may be included in the high-level syntax. That is, for example, the flag may be indicated by a signal in the high-level syntax. For example, the high-level syntax may be a sequence parameter set (SPS). Or, for example, the high-level syntax may be a slice header (SH). That is, for example, the flag may be included in the slice header.

[0225] Furthermore, for example, when the flag indicates the presence of a PH NAL unit, PH can be included in the PH NAL unit; when the flag indicates the absence of a PH NAL unit, PH can be included in the slice header of the current frame slice. For example, when the flag indicates the presence of a PH NAL unit, PH can be included in the PH NAL unit; when the flag indicates the absence of a PH NAL unit, PH can be included in the VCL NAL unit, which includes the slice header and slice data. That is, for example, when the flag indicates the presence of a PH NAL unit, PH can be included in the PH NAL unit; when the flag indicates the absence of a PH NAL unit, PH can be included in the slice header. For example, when the flag indicates the presence of a PH NAL unit, the image information can include a PH NAL unit containing PH and at least one VCL NAL unit containing the slice header and slice data of the current frame slice; when the flag indicates the absence of a PH NAL unit, the image information can include a VCL NAL unit containing PH, the slice header, and the slice data. Furthermore, for example, when the flag indicates the absence of a PH NAL unit, the image information may not include a PH NAL unit.

[0226] Furthermore, for example, when the flag indicates that a PH NAL unit is absent, the PH NAL unit may be absent for all frames in the coded video layer sequence (CLVS) including the current frame. That is, for example, the flag indicating whether a PH NAL unit exists for all frames in the coded video layer sequence (CLVS) may have the same value. Furthermore, for example, when the flag indicates that a PH NAL unit is absent, the frame headers of all frames in the CLVS may be included in the VCL NAL unit containing the slice headers of all frames.

[0227] Furthermore, for example, AU detection can be modified from existing methods. For instance, a new VCL NAL unit can refer to a new AU. That is, for example, when the flag indicates the absence of a PH NAL unit, the VCL NAL unit including the slice header can be the first VCL NAL unit of the current frame (for a new AU (i.e., the AU for the current frame)). For example, the flag can be included in the slice header of the VCL NAL unit. Or, for example, when the flag indicates the presence of a PH NAL unit, it can be the first VCL NAL unit of the current frame following the PH NAL unit (i.e., signaled after the PH NAL unit).

[0228] Furthermore, the encoding device can decode the current frame. For example, the encoding device can decode the current frame based on the syntax elements of the PH (Programmable Context), the syntax elements of the slice header, and the slice data. For example, the syntax elements of the PH can be the syntax elements shown in Table 6. The PH can include syntax elements representing parameters of the current frame, the slice header can include syntax elements representing parameters of slices in the current frame, and the slice data can include prediction information and residual information of blocks in the slice.

[0229] Furthermore, the bitstream including image information can be transmitted to the decoding device via a network or (digital) storage medium. Here, the network can include broadcast networks and / or communication networks, and the digital storage medium can include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, and SSD.

[0230] Figure 12 An encoding device for performing the image encoding method according to this document is shown schematically. Figure 11 The method shown can be derived from Figure 12 The encoding device shown performs this. Specifically, for example, Figure 12 The entropy encoder of the encoding device can execute S1100 to S1130. Although not shown, the decoding process for the current frame can be performed by the predictor and residual processor of the encoding device.

[0231] Figure 13 An image decoding method according to the decoding device of this document is illustrated schematically. Figure 13 The method shown can be derived from Figure 3 The decoding device shown performs this operation. Specifically, for example, Figure 13 S1300 to S1310 can be executed by the entropy decoder of the decoding device. Figure 13 The S1320 can be executed by the predictor and residual processor of the decoding device.

[0232] The decoding device obtains a flag indicating whether a PH Network Abstraction Layer (NAL) unit containing a frame header (PH) of the current frame exists (S1300). The decoding device can obtain image information via a bitstream. For example, the decoding device can obtain a flag indicating whether a PH Network Abstraction Layer (NAL) unit containing a frame header (PH) of the current frame exists via a bitstream. For example, the decoding device can obtain image information via a bitstream, and the image information may include this flag. Alternatively, for example, the image information may include high-level syntax, and the flag may be included in the high-level syntax. That is, for example, the flag may be obtained via high-level syntax. For example, the high-level syntax may be a sequence parameter set (SPS). Or, for example, the high-level syntax may be a slice header (SH). That is, for example, the flag may be included in the slice header.

[0233] For example, this flag can indicate the presence or absence of a PH NAL cell. For instance, a value of 1 indicates the presence of a PH NAL cell, and a value of 0 indicates its absence. Alternatively, for example, a value of 0 indicates the presence of a PH NAL cell, and a value of 1 indicates its absence. The syntax element for this flag can be the aforementioned `ph_nal_present_flag`.

[0234] The decoding device obtains a Video Coding Layer (VCL) NAL unit (S1310) that includes the slice header and slice data of the current slice in the current frame. The decoding device can obtain image information including information about the current frame via a bitstream. For example, the decoding device can obtain image information including a VCL NAL unit that includes the slice header and slice data of the current slice in the current frame via a bitstream. For example, the slice header and slice data may include syntax elements representing parameters of the current slice.

[0235] Additionally, for example, the slice data may include prediction information and residual information for the blocks in the current slice. For example, the prediction information may include the prediction pattern information of the blocks. Additionally, for example, the residual information is information about the residual samples of the blocks. The residual information may include information about the quantization transformation coefficients of the residual samples.

[0236] The decoding device obtains PH based on this flag (S1320). The decoding device can obtain PH from the PH NAL unit or the VCL NAL unit, which includes the slice header and slice data, based on this flag. That is, for example, the decoding device can obtain PH from the PHNAL unit or the slice header based on this flag.

[0237] For example, when the flag indicates the presence of a PH NAL unit, PH can be obtained from the PH NAL unit; when the flag indicates the absence of a PH NAL unit, PH can be obtained from the VCL NAL unit, which includes the slice header and slice data. That is, for example, when the flag indicates the presence of a PH NAL unit, PH can be obtained from the PH NAL unit; when the flag indicates the absence of a PH NAL unit, PH can be obtained from the slice header. For example, when the flag indicates the presence of a PH NAL unit, PH can be included in the PH NAL unit; when the flag indicates the absence of a PH NAL unit, PH can be included in the VCL NAL unit, which includes the slice header and slice data. That is, for example, when the flag indicates the presence of a PH NAL unit, PH can be included in the PH NAL unit; when the flag indicates the absence of a PH NAL unit, PH can be included in the slice header. For example, when the flag indicates the presence of a PH NAL unit, the image information can include a PH NAL unit containing PH and a VCL NAL unit containing the slice header; when the flag indicates the absence of a PH NAL unit, the image information can include a VCL NAL unit containing PH, the slice header, and slice data. Furthermore, for example, when the flag indicates that the PHNAL unit is not present, the image information may not include the PHNAL unit.

[0238] Furthermore, for example, when the flag indicates that no PH NAL unit exists, the current frame of PH may consist of only one slice. That is, for example, when the flag indicates that no PH NAL unit exists, the image information may include a VCL NAL unit containing a slice header and slice data of one slice in the current frame (the current slice).

[0239] Furthermore, for example, when the flag indicates the presence of a PH NAL unit, the PH NAL unit for the current frame and at least one VCL NAL unit including the slice header and slice data of the current frame can be obtained through the bitstream. That is, for example, when the flag indicates the presence of a PH NAL unit, the image information may include a VCL NAL unit containing the slice header and slice data other than the current slice in the current frame, as well as the PH NAL unit of the current frame and the VCL NAL unit of the current slice.

[0240] Furthermore, for example, when the flag indicates that a PH NAL unit is absent, the PH NAL unit may be absent for all frames in the coded video layer sequence (CLVS) including the current frame. That is, for example, the flag indicating whether a PH NAL unit exists for all frames in the coded video layer sequence (CLVS) may have the same value. Furthermore, for example, when the flag indicates that a PH NAL unit is absent, the frame headers of all frames in the CLVS may be included in the slice headers of all frames. That is, for example, when the flag indicates that a PH NAL unit is absent, the frame headers of all frames in the CLVS may be included in the VCL NAL unit that includes the slice headers of all frames.

[0241] Furthermore, for example, AU detection can be modified from existing methods. For instance, a new VCL NAL unit can refer to a new AU. That is, for example, when the flag indicates the absence of a PH NAL unit, the VCL NAL unit including the slice header can be the first VCL NAL unit of the current frame (for a new AU (i.e., the AU for the current frame)). For example, the flag can be included in the slice header of the VCL NAL unit. Or, for example, when the flag indicates the presence of a PH NAL unit, it can be the first VCL NAL unit of the current frame following the PH NAL unit (i.e., signaled after the PH NAL unit).

[0242] The decoding device decodes the current frame based on the PH, slice header, and slice data (S1330). The decoding device can decode the current frame based on the PH, slice header, and slice data. For example, the decoding device can decode the current frame based on the syntax elements of the PH, the syntax elements of the slice header, and the slice data. For example, the syntax elements of the PH can be the syntax elements shown in Table 6. The PH may include syntax elements representing parameters of the current frame, and the slice header may include syntax elements representing parameters of the current slice. Additionally, for example, the slice data may include prediction information and residual information of blocks in the current slice. For example, the decoding device can derive prediction samples and residual samples based on the prediction information and residual information of blocks in the current slice of the current frame, and generate a reconstructed sample / reconstructed frame of the current frame based on the prediction samples and residual samples.

[0243] As mentioned above, in-loop filtering processes such as deblocking filtering, SAO, and / or ALF can be applied to the reconstructed samples as needed to improve subjective / objective image quality.

[0244] Figure 14 A decoding device for performing the image decoding method according to this document is shown schematically. Figure 13 The method shown can be derived from Figure 14 The decoding device shown performs this operation. Specifically, for example, Figure 14 The entropy decoder of the decoding device can perform Figure 13 S1300 to S1320, Figure 14 The predictor and residual processor of the decoding device can perform Figure 13 The S1330.

[0245] According to the above disclosure, a flag indicating the presence of a PH NAL unit can be used to adaptively adjust the bit rate of the bit stream based on the flag, thereby improving the overall coding efficiency.

[0246] Furthermore, according to this disclosure, constraints regarding the number of slices in the current frame and the presence of PH NAL units can be set for the relevant frame based on a flag indicating the presence or absence of PH NAL units, and the bit rate control NAL units can be adapted to improve overall coding efficiency.

[0247] In the above embodiments, the method is described based on a flowchart having a series of steps or blocks. This disclosure is not limited to the order of the above steps or blocks. Some steps or blocks may be performed in a different order than the other steps or blocks described above, or may be performed simultaneously. Furthermore, those skilled in the art will understand that the steps shown in the flowchart are not exclusive, and may include other steps, or one or more steps in the flowchart may be deleted without affecting the scope of this disclosure.

[0248] The embodiments described in this specification can be executed by being implemented on a processor, microprocessor, controller, or chip. For example, the functional units shown in each figure can be executed by being implemented on a computer, processor, microprocessor, controller, or chip. In this case, the information for implementation (e.g., information about instructions) or algorithms can be stored in a digital storage medium.

[0249] Furthermore, the decoding and encoding devices using this disclosure can be included in the following devices: multimedia broadcasting transmitting / receiving devices, mobile communication terminals, home theater video devices, digital cinema video devices, surveillance cameras, video chat devices, real-time communication devices such as video communication, mobile streaming devices, storage media, portable video cameras, VoD service providing devices, over-the-top (OTT) video devices, internet streaming service providing devices, three-dimensional (3D) video devices, teleconferencing video devices, transportation user equipment (e.g., vehicle user equipment, aircraft user equipment, and ship user equipment), and medical video devices; and the decoding and encoding devices using this disclosure can be used to process video signals or data signals. For example, over-the-top (OTT) video devices can include game consoles, Blu-ray players, internet-access televisions, home theater systems, smartphones, tablet computers, digital video recorders (DVRs), etc.

[0250] Furthermore, the processing methods of this disclosure can be generated in the form of a computer-executable program and can be stored in a computer-readable recording medium. Multimedia data with data structures according to this disclosure can also be stored in a computer-readable recording medium. A computer-readable recording medium includes all types of storage devices in which computer-readable data is stored. Computer-readable recording media can include, for example, BD, Universal Serial Bus (USB), ROM, PROM, EPROM, EEPROM, RAM, CD-ROM, magnetic tape, floppy disk, and optical data storage devices. Additionally, computer-readable recording media include media implemented in the form of a carrier wave (e.g., transmission via the Internet). Furthermore, bitstreams generated by encoding methods can be stored in a computer-readable recording medium or transmitted via wired / wireless communication networks.

[0251] Furthermore, embodiments of this disclosure can be implemented using computer program products based on program code, and the program code can be executed on a computer using embodiments of this disclosure. The program code can be stored on a computer-readable medium.

[0252] Figure 15 A structural diagram of a content streaming system applying this disclosure is shown.

[0253] The content streaming system using the embodiments of this disclosure may mainly include an encoding server, a streaming server, a network server, a media storage device, a user device, and a multimedia input device.

[0254] An encoding server compresses content input from multimedia input devices such as smartphones, cameras, or camcorders into digital data to generate a bitstream, which is then sent to a streaming server. As another example, when multimedia input devices such as smartphones, cameras, or camcorders generate bitstreams directly, the encoding server can be omitted.

[0255] A bitstream can be generated by an encoding method or bitstream generation method that applies the embodiments of this disclosure, and the stream server can temporarily store the bitstream during the sending or receiving of the bitstream.

[0256] The streaming server sends multimedia data to the user device via a web server based on user requests, and the web server acts as a medium for notifying the user of services. When a user requests a desired service from the web server, the web server transmits the request to the streaming server, and the streaming server sends the multimedia data to the user. In this scenario, the content streaming system may include a separate control server. In this case, the control server is used to control the commands / responses between devices within the content streaming system.

[0257] A streaming server can receive content from media storage and / or encoding servers. For example, when receiving content from an encoding server, the content can be received in real time. In this case, to provide a smooth streaming service, the streaming server can store the bitstream for a predetermined period of time.

[0258] Examples of user devices may include mobile phones, smartphones, laptops, digital broadcast terminals, personal digital assistants (PDAs), portable multimedia players (PMPs), navigators, touchscreen PCs, tablet PCs, ultrabooks, wearable devices (e.g., smartwatches, smart glasses, and head-mounted displays), digital TVs, desktop computers, and digital signage. Each server within the content streaming system can operate as a distributed server, in which case data received from each server can be distributed.

[0259] The claims described in this disclosure can be combined in various ways. For example, the technical features of the method claims of this disclosure can be combined to implement a device, and the technical features of the device claims of this disclosure can be combined to implement a method. Furthermore, the technical features of the method claims and the device claims of this disclosure can be combined to implement a device, and the technical features of the method claims and the device claims of this disclosure can be combined to implement a method.

Claims

1. An apparatus for decoding image information, the apparatus comprising: Memory; and at least one processor, the at least one processor being coupled to the memory, the at least one processor being configured to: Obtain a flag indicating whether a PH network abstraction layer NAL unit exists, which includes a pictorial header for the current frame, and the pictorial header includes multiple parameters; Obtain a Video Coding Layer (VCL) NAL unit that includes the slice header and slice data for the current slice in the current frame; The image header is obtained based on the aforementioned logo; as well as The current frame is decoded based on the frame header, the slice header, and the slice data. Wherein, based on the flag indicating the absence of the PH NAL unit, the current frame includes only a single current slice. Specifically, the presence of the PH NAL unit is indicated by the flag, and the PH NAL unit includes all of the plurality of parameters included in the image header. Specifically, based on the flag indicating the absence of the PH NAL unit, the VCL NAL unit includes the slice header, the slice data, and all the plurality of parameters included in the frame header. The slice data includes residual information for the current block in the current slice, and The residual sample of the current block is derived based on the residual information.

2. An apparatus for encoding image information, the apparatus comprising: Memory; and at least one processor, the at least one processor being coupled to the memory, the at least one processor being configured to: Generate Video Coding Layer (VCL) Network Abstraction Layer (NAL) units that include slice headers and slice data for slices in the current frame; Determine whether there exists a frame header PH NAL unit that includes a frame header for the current frame, the frame header including multiple parameters; Based on the determined results, the image header and a flag indicating the presence or absence of the PH NAL unit are generated; as well as Image information, including the VCL NAL unit, the image header, and the logo, is encoded. Wherein, based on the flag indicating the absence of the PH NAL unit, the current frame includes only a single current slice. Specifically, based on the existence of the PH NAL unit, the PH NAL unit includes all the plurality of parameters included in the image header, and Wherein, based on the absence of the PH NAL unit, the VCL NAL unit includes the slice header, the slice data, and all the plurality of parameters included in the frame header. This involves generating residual information for the current block within the current slice, and The slice data includes the residual information for the current block in the current slice.

3. A device for transmitting image information, the device comprising: At least one processor is configured to perform the following steps to generate a bitstream of image information: generating a Video Coding Layer (VCL) Network Abstraction Layer (NAL) unit including a slice header and slice data for a slice in the current frame; determining whether a PH NAL unit including a frame header for the current frame, the frame header including multiple parameters; generating the frame header and a flag indicating the presence or absence of the PH NAL unit based on the determined result; and encoding the image information including the VCL NAL unit, the frame header, and the flag. as well as A transmitter configured to transmit the bit stream. Wherein, based on the flag indicating the absence of the PH NAL unit, the current frame includes only a single current slice. Specifically, based on the existence of the PH NAL unit, the PH NAL unit includes all the plurality of parameters included in the image header, and Wherein, based on the absence of the PH NAL unit, the VCL NAL unit includes the slice header, the slice data, and all the plurality of parameters included in the frame header. This involves generating residual information for the current block within the current slice, and The slice data includes the residual information for the current block in the current slice.

Images