MIP-based image encoding / decoding method, method for transmitting bit stream, and recording medium having bit stream stored therein

Abstract

An image encoding / decoding method and apparatus are provided. An image decoding method according to the present disclosure comprises the steps of: determining an intra prediction mode of a current block; and generating a prediction block of the current block by performing intra prediction on the basis of the determined intra prediction mode, the intra prediction being restricted to be performed by using only reference samples within a predetermined range among the available reference samples from the current block.

Description

Technical Field

[0001] This disclosure relates to a MIP-based image encoding / decoding method, a method for transmitting a bitstream, and a recording medium for storing the bitstream, and more specifically, to a matrix-based intra-frame prediction (MIP) image encoding / decoding method, a method for transmitting a bitstream, and a recording medium for storing the bitstream, using only a portion of a reference sample. Background Technology

[0002] Recently, the demand for high-resolution and high-quality images, such as high-definition (HD) and ultra-high-definition (UHD) images, has increased across various fields. As image data becomes higher resolution and higher quality, the amount of information transmitted, or the bit rate, increases relative to traditional image data. This increase in the amount of information transmitted or the bit rate leads to increased transmission and storage costs.

[0003] Therefore, an efficient image compression technology is needed to effectively send, store, and reproduce information from high-resolution and high-quality images. Summary of the Invention

[0004] Technical issues

[0005] This disclosure is intended to provide an image encoding / decoding method and apparatus with improved encoding / decoding efficiency.

[0006] In addition, this disclosure is intended to provide an image encoding / decoding method and apparatus based on intra-frame prediction and some selected reference samples.

[0007] In addition, this disclosure is to provide an image coding / decoding method and apparatus that improves the intra-frame prediction efficiency of Enhanced Compression Model (ECM).

[0008] In addition, this disclosure is intended to provide a non-transitory computer-readable recording medium for storing bit streams generated by an image encoding method or apparatus according to this disclosure.

[0009] In addition, this disclosure is intended to provide a non-transitory computer-readable recording medium for storing bitstreams received and decoded by an image decoding apparatus according to this disclosure and used for image reconstruction.

[0010] In addition, this disclosure is intended to provide a method for transmitting a bit stream generated by an image encoding method or apparatus according to this disclosure.

[0011] The technical problems to be solved in this disclosure are not limited to those described above, and other technical problems not described will be clearly understood by those skilled in the art from the following description.

[0012] Technical solution

[0013] An image decoding method according to one aspect of this disclosure includes: determining whether a matrix-based intra-frame prediction (MIP) mode is applied to the current block; and generating a prediction block for the current block based on the application of the MIP mode to the current block, wherein, based on the MIP mode, the prediction block can be generated by using only a portion of the left reference sample or the upper reference sample of the current block.

[0014] Meanwhile, as an example, a prediction block can be generated based on the MIP pattern by using only the upper reference sample of the current block, but it can also be generated based on at least one upper reference sample line.

[0015] Meanwhile, as an example, a prediction block can be generated based on the MIP pattern by using only the upper reference sample of the current block, but the upper reference sample can further include the upper right reference sample of the current block.

[0016] Meanwhile, as an example, a prediction block can be generated based on the MIP pattern by using only the left reference sample of the current block, but it can also be generated based on at least one left reference sample line.

[0017] Meanwhile, as an example, a prediction block can be generated based on the MIP pattern by using only the left reference sample of the current block, but the left reference sample may include the lower left reference sample of the current block.

[0018] Meanwhile, as an example, a prediction block can be generated using a portion of the left reference sample and the upper reference sample of the current block, but at least one of the left reference sample or the upper reference sample can be partially adjacent to the current block.

[0019] Meanwhile, as an example, a prediction block can be generated using a portion of the left reference sample and the top reference sample of the current block, but the left reference sample can further include the bottom-left reference sample of the current block, and the top reference sample can further include the top-right reference sample of the current block.

[0020] Meanwhile, as an example, based on the application of MIP mode, it can be determined whether to execute MIP mode based on the left reference sample or the top reference sample.

[0021] Meanwhile, as an example, it is possible to determine whether to execute MIP mode based on the information obtained from the bitstream or based on the left reference sample or the top reference sample.

[0022] An image coding method according to one aspect of this disclosure includes: determining whether a matrix-based intra-frame prediction (MIP) mode will be applied to the current block; and generating a prediction block of the current block based on the application of the MIP mode to the current block, wherein the prediction block can be generated based on the MIP mode by using only a portion of the left reference sample or the upper reference sample of the current block.

[0023] According to another aspect of this disclosure, a computer-readable recording medium may store a bitstream generated by the image encoding method or image encoding apparatus of this disclosure.

[0024] According to another aspect of the transmission method of this disclosure, a bit stream generated by the image encoding apparatus or image encoding method of this disclosure can be transmitted.

[0025] The features of this disclosure that have been briefly summarized above are merely exemplary embodiments described in detail below and are not intended to limit the scope of this disclosure.

[0026] Beneficial effects

[0027] According to this disclosure, an image encoding / decoding method and apparatus with improved encoding / decoding efficiency can be provided.

[0028] In addition, according to this disclosure, an image encoding / decoding method and apparatus based on intra-frame prediction and some selected reference samples can be provided.

[0029] In addition, according to this disclosure, an image encoding / decoding method and apparatus that can generate adaptive intra-prediction blocks in a reference sample region during intra-frame prediction can be provided.

[0030] In addition, according to this disclosure, an image encoding / decoding method and apparatus with improved ECM intra-frame prediction efficiency can be provided.

[0031] Additionally, according to this disclosure, a non-transitory computer-readable recording medium may be provided for storing bit streams generated by an image encoding method or apparatus according to this disclosure.

[0032] Additionally, according to this disclosure, a non-transitory computer-readable recording medium may be provided for storing bitstreams received and decoded by an image decoding apparatus according to this disclosure and used for image reconstruction.

[0033] Additionally, according to this disclosure, a method for transmitting a bit stream generated by an image encoding method or apparatus according to this disclosure can be provided.

[0034] The effects achievable by this disclosure are not limited to those described above, and other effects not described will be readily apparent to those skilled in the art from the following description. Attached Figure Description

[0035] Figure 1 A schematic diagram is shown of a video encoding system that can be applied according to embodiments of the present disclosure.

[0036] Figure 2 A schematic diagram of an image encoding apparatus that can be applied according to embodiments of the present disclosure is shown.

[0037] Figure 3 A schematic diagram of an image decoding apparatus that can be applied according to embodiments of the present disclosure is shown.

[0038] Figure 4 A flowchart illustrating an example of an intra-frame prediction mode signaling method in an encoding apparatus.

[0039] Figure 5 A flowchart illustrating an example of an intra-frame prediction mode determination method in a decoding apparatus.

[0040] Figure 6 A diagram showing an example of neighboring blocks used to derive the MPM list.

[0041] Figures 7 to 11 This is a diagram illustrating examples of the overall processes that can be applied to averaging, matrix-vector multiplication, and linear interpolation in this disclosure.

[0042] Figure 12 This is a diagram illustrating an example of the boundary averaging process that can be applied to this disclosure.

[0043] Figure 13 This is a diagram illustrating an example of a linear interpolation process that can be applied to this disclosure.

[0044] Figure 14 This is a diagram used to describe the MIP prediction method that can be applied to this disclosure.

[0045] Figure 15 This is a diagram illustrating a method for performing MIP prediction according to embodiments of the present disclosure.

[0046] Figure 16 This is a diagram used to describe a MIP-based image encoding or decoding method according to embodiments of the present disclosure.

[0047] Figure 17 This is a diagram used to describe the MIP prediction method that can be applied to this disclosure.

[0048] Figure 18 This is a diagram illustrating a method for performing MIP prediction according to embodiments of the present disclosure.

[0049] Figure 19 This is a diagram used to describe a MIP-based image encoding or decoding method according to embodiments of the present disclosure.

[0050] Figure 20 This is a diagram used to describe the MIP prediction method that can be applied to this disclosure.

[0051] Figure 21 This is a diagram illustrating a method for performing MIP prediction according to embodiments of the present disclosure.

[0052] Figure 22 This is a diagram used to describe a MIP-based image encoding or decoding method according to embodiments of the present disclosure.

[0053] Figure 23 This is a diagram illustrating an image decoding method according to an embodiment of the present disclosure.

[0054] Figure 24 This is a diagram illustrating an image encoding method according to an embodiment of the present disclosure.

[0055] Figure 25 An exemplary diagram is shown of a content streaming system to which embodiments of the present disclosure may be applied. Detailed Implementation

[0056] In the following description, embodiments of the present disclosure will be detailed with reference to the accompanying drawings to facilitate implementation by those skilled in the art. However, the present disclosure may be implemented in various different forms and is not limited to the embodiments described herein.

[0057] In describing embodiments of this disclosure, detailed explanations of well-known configurations or functions that are considered to obscure the essential points of this disclosure have been omitted. Additionally, portions irrelevant to the description of this disclosure have been omitted from the accompanying drawings, and similar reference numerals have been assigned to similar portions.

[0058] In this disclosure, when certain components are described as “connected,” “coupled,” or “linked” to another component, this can include not only direct connections but also indirect connections in which another component may be present in between. Additionally, when a component is described as “including” or “having” another component, this means that, unless otherwise expressly stated, it does not exclude other components but may further include additional components.

[0059] In this disclosure, unless otherwise expressly stated, the terms first, second, etc., are used only to distinguish one component from another and do not limit the order or importance of the components. Therefore, within the scope of this disclosure, a first component in one embodiment may be referred to as a second component in another embodiment, and similarly, a second component in one embodiment may be referred to as a first component in another embodiment.

[0060] In this disclosure, distinguishable components are described to clearly explain their respective characteristics, and this does not necessarily mean that the components are separate. In other words, multiple components may be integrated into a single hardware or software unit, or a single component may be distributed across multiple hardware or software units. Therefore, such integrated or distributed embodiments are also included within the scope of this disclosure unless they are explicitly described.

[0061] In this disclosure, the components described in the various embodiments are not necessarily essential components, and some may be optional components. Therefore, embodiments comprising a subset of the components described in one embodiment are also included within the scope of this disclosure. Furthermore, embodiments that include additional components in addition to those described in the various embodiments are also included within the scope of this disclosure.

[0062] This disclosure relates to the encoding and decoding of images, and the terms used herein may have their common meanings as commonly used in the art to which this disclosure pertains, unless these terms are redefined herein.

[0063] In this disclosure, "picture" generally refers to a unit representing a single image at a specific point in time. A slice / tile is a coding unit that constitutes part of a picture, and a picture may consist of one or more slices / tiles. In addition, a slice / tile may include one or more coding tree units (CTUs).

[0064] In this disclosure, "pixel" or "cell" can refer to the smallest unit that constitutes a picture (or image). Additionally, the term "sample" can be used as the corresponding term for a pixel. A sample can generally represent a pixel or a pixel value, and can indicate only the pixel / pixel value of the luminance component or only the pixel / pixel value of the chrominance component.

[0065] In this disclosure, "unit" can refer to a basic unit of image processing. A unit may include at least one of a specific region of an image or information associated with that region. Depending on the context, the term "unit" may be used interchangeably with "sample array," "block," "region," etc. Typically, an M×N block may include a set (or array) of samples (or sample arrays) or a set (or array) of transform coefficients, consisting of M columns and N rows.

[0066] In this disclosure, the term "current block" may refer to one of "current coding block," "current coding unit," "coding target block," "decoding target block," or "processing target block." When performing prediction, "current block" may refer to "current prediction block" or "prediction target block." When performing transform (inverse transform) / quantization (dequantization), "current block" may refer to "current transform block" or "transform target block." When performing filtering, "current block" may refer to "filter target block."

[0067] In this disclosure, unless explicitly stated as a chroma block, the term "current block" may refer to a block that includes both luma component blocks and chroma component blocks, or it may refer to "the luma block of the current block." The luma component block of the current block may be explicitly expressed using terms such as "luma block" or "current luma block," clearly indicating that it is a luma component block. Similarly, the chroma component block of the current block may be explicitly expressed using terms such as "chroma block" or "current chroma block," clearly indicating that it is a chroma component block.

[0068] In this disclosure, " / " and "," can mean "and / or". For example, "A / B" and "A, B" can mean "A and / or B". In addition, "A / B / C" and "A, B, C" can mean "at least one of A, B and / or C".

[0069] In this disclosure, "or" can mean "and / or". For example, "A or B" can mean 1) only "A", 2) only "B", or 3) "A and B". Alternatively, in this disclosure, "or" can also mean "in addition or alternatively".

[0070] Overview of Video Encoding Systems

[0071] Figure 1 A schematic diagram is shown of a video encoding system that can be applied according to embodiments of the present disclosure.

[0072] The video encoding system according to the embodiment may include an encoder device 10 and a decoder device 20. The encoder device 10 may transmit encoded video and / or image information or data to the decoder device 20 in the form of a file or stream via a digital storage medium or network.

[0073] The encoder device 10 according to an embodiment may include a video source generator 11, an encoder 12, and a transmitter 13. The decoder device 20 according to an embodiment may include a receiver 21, a decoder 22, and a renderer 23. The encoder 12 may be referred to as a video / image encoder, and the decoder 22 may be referred to as a video / image decoder. The transmitter 13 may be included in the encoder 12. The receiver 21 may be included in the decoder 22. The renderer 23 may include a display, and the display may be configured as a separate device or an external component.

[0074] The video source generator 11 can obtain video / images through a process of capturing, compositing, or generating video / images. The video source generator 11 may include a video / image capture device and / or a video / image generation device. The video / image capture device may include, for example, one or more cameras, a video / image archive containing previously captured video / images, etc. The video / image generation device includes, for example, a computer, tablet, or smartphone, and can generate video / images (electronically). For example, virtual video / images can be generated by a computer, etc., and in this case, the video / image capture process can be replaced by a process of generating related data.

[0075] Encoder 12 can encode input video / images. Encoder 12 can perform a series of processes such as prediction, transformation, and quantization for compression and encoding efficiency. Encoder 12 can output encoded data (encoded video / image information) in the form of a bitstream.

[0076] Transmitter 13 can acquire encoded video / image information or data output in bitstream form and transmit it to receiver 21 of decoder device 20 or another external object via digital storage medium or network in the form of file or stream. Digital storage medium can include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, SSD, etc. Transmitter 13 can include elements for generating media files in a predetermined file format and elements for transmission via broadcast / communication network. Transmitter 13 can be provided as a transmission device separate from encoder 12, in which case the transmission device can include at least one processor for acquiring encoded video / image information or data in bitstream form and a transmitter for delivering it in the form of file or stream. Receiver 21 can extract / receive bitstreams from storage medium or network and transmit them to decoder 22.

[0077] Decoder 22 can decode video / images by performing a series of processes such as dequantization, inverse transform, prediction, etc., corresponding to the operations of encoder 12.

[0078] Renderer 23 can render decoded video / images. The rendered video / images can be displayed through the display unit.

[0079] Overview of image encoding devices

[0080] Figure 2 A schematic diagram of an image encoding apparatus that can be applied according to embodiments of the present disclosure is shown.

[0081] like Figure 2As described above, the image encoding apparatus 100 may include an image partitioner 110, a subtractor 115, a transformer 120, a quantizer 130, a dequantizer 140, an inverse transformer 150, an adder 155, a filter 160, a memory 170, an inter-frame predictor 180, an intra-frame predictor 185, and an entropy encoder 190. The inter-frame predictor 180 and the intra-frame predictor 185 may be collectively referred to as "predictors". The transformer 120, quantizer 130, dequantizer 140, and inverse transformer 150 may be included in a residual processor. The residual processor may further include a subtractor 115.

[0082] Depending on the embodiment, all or at least some of the multiple components constituting the image encoding apparatus 100 may be implemented as a single hardware component (i.e., an encoder or a processor). Additionally, the memory 170 may include a decoded image buffer (DPB) and may be implemented using a digital storage medium.

[0083] Image partitioner 110 can partition an input image (or picture, frame) input to image encoding device 100 into at least one processing unit. As an example, a processing unit may be referred to as an encoding unit (CU). Encoding units can be obtained by recursively partitioning encoding tree units (CTUs) or maximum encoding units (LCUs) according to a quadtree, binary tree, or ternary tree (QT / BT / TT) structure. For example, encoding units can be partitioned into deeper encoding units based on a quadtree, binary tree, and / or ternary tree structure. For partitioning encoding units, a quadtree structure can be applied first, followed by a binary tree structure and / or a ternary tree structure. The encoding process according to this disclosure can be performed based on a final encoding unit without further partitioning that final encoding unit. A maximum encoding unit can be directly used as the final encoding unit, or a deeper encoding unit obtained by partitioning the maximum encoding unit can be used as the final encoding unit. Here, the encoding process may include processes such as prediction, transformation, and / or reconstruction, which will be described later. As another example, the processing unit used for the encoding process may be a prediction unit (PU) or a transformation unit (TU). The prediction unit and the transform unit can each be derived from or partitioned from the final coding unit. The prediction unit can be a unit used for sample prediction, and the transform unit can be a unit used for deriving transform coefficients and / or deriving residual signals from transform coefficients.

[0084] The predictor (inter-frame predictor 180 or intra-frame predictor 185) can perform prediction on the target block (current block) and generate a prediction block including prediction samples for the current block. The predictor can determine whether to apply intra-frame or inter-frame prediction to the current block or coding unit (CU). The predictor can generate various information related to the prediction of the current block and send it to the entropy encoder 190. The prediction-related information can be encoded by the entropy encoder 190 and can be output as a bitstream.

[0085] Intra-predictor 185 can predict the current block by referencing samples within the current image. The referenced samples can be located in the vicinity of the current block or at a more distant location, depending on the intra-prediction mode and / or intra-prediction method. Intra-prediction modes can include multiple non-directional modes and multiple directional modes. Non-directional modes can include, for example, DC modes and planar modes. Directional modes can include, for example, 33 or 65 directional prediction modes, depending on the granularity of the prediction orientation. However, this is just an example, and more or fewer directional prediction modes may be used depending on the configuration. Intra-predictor 185 can also determine the prediction mode to be applied to the current block by using prediction modes applied to neighboring blocks.

[0086] Inter-frame predictor 180 can derive the predicted block of the current block based on a reference block (reference sample array) specified by motion vectors on a reference image. To reduce the amount of motion information transmitted in inter-frame prediction mode, motion information can be predicted at the block, sub-block, or sample level based on the correlation of motion information between neighboring blocks and the current block. Motion information may include motion vectors and reference image indices. Motion information may further include information about inter-frame prediction orientation (i.e., L0 prediction, L1 prediction, Bi prediction, etc.). In inter-frame prediction, neighboring blocks may include spatially neighboring blocks existing in the current image and temporally neighboring blocks existing in the reference image. The reference image containing the reference block and the reference image containing the temporally neighboring block may be the same or different. The temporally neighboring block may be referred to as a co-located reference block or a co-located coding unit (colCU). The reference image containing the temporally neighboring block may be referred to as a co-located image (colPic). For example, inter-frame predictor 180 can construct a motion information candidate list based on neighboring blocks and generate information indicating which candidate is used to derive the motion vector and / or reference image index of the current block. Inter-frame prediction can be performed based on various prediction modes, and for example, in skip mode and merge mode, the inter-frame predictor 180 can use motion information from neighboring blocks as motion information for the current block. In skip mode, unlike merge mode, the residual signal may not be sent. In motion vector prediction (MVP) mode, the motion vectors of neighboring blocks can be used as the motion vector predictor, and the motion vector of the current block can be signaled by encoding the motion vector difference and an indicator for the motion vector predictor. The motion vector difference can refer to the difference between the motion vector of the current block and the motion vector predictor.

[0087] The predictor can generate a prediction signal based on various prediction methods and / or techniques described later. For example, the predictor can apply intra-frame prediction or inter-frame prediction to the prediction of the current block, or it can apply both intra-frame prediction and inter-frame prediction simultaneously. A prediction method that applies both intra-frame and inter-frame prediction to the prediction of the current block can be called Combined Inter-Frame and Intra-Frame Prediction (CIIP). Additionally, the predictor can perform Intra-Frame Block Copying (IBC) on the prediction of the current block. Intra-Frame Block Copying can be used, for example, in applications such as Screen Content Coding (SCC) in game content image / video encoding. IBC is a method of predicting the current block by using a pre-reconstructed reference block within the current image, located at a predetermined distance from the current block. When IBC is applied, the position of the reference block within the current image can be encoded as a vector (block vector) corresponding to the predetermined distance. IBC essentially performs prediction within the current image, but because it derives a reference block within the current image, it can operate similarly to inter-frame prediction. In other words, IBC can use at least one of the inter-frame prediction methods described in this disclosure.

[0088] The predicted signal generated by the predictor can be used to generate a reconstructed signal or a residual signal. Subtractor 115 generates a residual signal (residual block, residual sample array) by subtracting the predicted signal (predicted block, predicted sample array) output from the predictor from the input image signal (original block, original sample array). The generated residual signal can be sent to converter 120.

[0089] Transformer 120 can generate transform coefficients by applying a transform method to the residual signal. For example, the transform method may include at least one of Discrete Cosine Transform (DCT), Discrete Sine Transform (DST), Karhunen-Loeve Transform (KLT), Graphical Based Transform (GBT), or Conditional Nonlinear Transform (CNT). Here, GBT refers to a transform obtained from a graph when the relationship information between pixels is represented as a graph. CNT refers to a transform based on a predicted signal generated using all previously reconstructed pixels. The transform process can be applied to pixel blocks of the same square size or to non-square blocks of variable size.

[0090] Quantizer 130 can quantize the transform coefficients and send them to entropy encoder 190. Entropy encoder 190 can encode the quantized signal (information about the quantized transform coefficients) and output it as a bitstream. The information about the quantized transform coefficients can be referred to as residual information. Quantizer 130 can rearrange the block-shaped quantized transform coefficients into a one-dimensional vector based on the coefficient scan order, and can generate information about the quantized transform coefficients based on the one-dimensional vector of quantized transform coefficients.

[0091] The entropy encoder 190 can perform various encoding methods, such as exponential Columbus, context-adaptive variable-length coding (CAVLC), or context-adaptive binary arithmetic coding (CABAC). The entropy encoder 190 can encode not only quantized transform coefficients but also information necessary for video / image reconstruction (i.e., values ​​of syntax elements) either together with or separately from the quantized transform coefficients. The encoded information (i.e., encoded video / image information) can be transmitted or stored in a network abstraction layer (NAL) unit in the form of a bitstream. The video / image information may further include information about various parameter sets, such as adaptive parameter sets (APS), picture parameter sets (PPS), sequence parameter sets (SPS), or video parameter sets (VPS). Additionally, the video / image information may further include general constraint information. The signaling information, transmitted information, and / or syntax elements described in this disclosure can be included in the bitstream by being encoded through the above-described encoding process.

[0092] The bitstream can be transmitted over a network or stored in a digital storage medium. Here, the network may include broadcast networks and / or communication networks, etc., and the digital storage medium may include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, SSD, etc. A transmitter (not shown) for transmitting the signal output from the entropy encoder 190 and / or a storage unit (not shown) for storing the signal may be provided as internal / external components of the image encoding apparatus 100, or the transmitter may be configured as a component of the entropy encoder 190.

[0093] The quantized transform coefficients output from quantizer 130 can be used to generate residual signals. For example, the residual signal (residual block or residual sample) can be reconstructed by applying dequantization and inverse transform to the quantized transform coefficients through dequantizer 140 and inverse transform 150.

[0094] Adder 155 generates a reconstruction signal (reconstructed image, reconstruction block, or reconstruction sample array) by adding the reconstruction residual signal to the prediction signal output from inter-frame predictor 180 or intra-frame predictor 185. When there is no residual for the target block, such as when a skip mode is applied, the prediction block can be used as a reconstruction block. Adder 155 can be referred to as a reconstructor or reconstruction block generator. The generated reconstruction signal can be used for intra-frame prediction of the next target block within the current image, and, as described later, can also be used for inter-frame prediction of the next image after filtering.

[0095] Filter 160 can apply filtering to the reconstructed signal to enhance subjective / objective quality. For example, filter 160 can apply various filtering methods to the reconstructed image to generate a modified reconstructed image, and the modified reconstructed image can be stored in memory 170, specifically in the DPB of memory 170. Various filtering methods can include, for example, deblocking filtering, sample adaptive offset, adaptive loop filtering, bilateral filtering, etc. Filter 160 can generate various filtering-related information, as described later in the explanation of each filtering method, and can send it to entropy encoder 190. The filtering-related information can be encoded by entropy encoder 190 and output as a bitstream.

[0096] The modified reconstructed image sent to memory 170 can be used as a reference image in inter-frame predictor 180. When inter-frame prediction is applied in this case, image coding device 100 can avoid prediction mismatch between image coding device 100 and image decoding device, and can improve coding efficiency.

[0097] The DPB in memory 170 can store modified reconstructed images for use as reference images in inter-frame predictor 180. Memory 170 can store motion information of blocks in the current image with derived (or encoded) motion information and / or motion information of blocks in reconstructed images. The stored motion information can be sent to inter-frame predictor 180 for use as motion information for spatially or temporally neighboring blocks. Memory 170 can store reconstructed samples of reconstructed blocks in the current image and send them to intra-frame predictor 185.

[0098] Overview of image decoding devices

[0099] Figure 3 A schematic diagram of an image decoding apparatus to which embodiments of the present disclosure may be applied is shown.

[0100] like Figure 3 As shown, the image decoding device 200 may include an entropy decoder 210, a dequantizer 220, an inverse transformer 230, an adder 235, a filter 240, a memory 250, an inter-frame predictor 260, and an intra-frame predictor 265. The inter-frame predictor 260 and the intra-frame predictor 265 may be collectively referred to as "predictors". The dequantizer 220 and the inverse transformer 230 may be included in a residual processor.

[0101] All or at least some of the multiple components constituting the image decoding device 200 may be implemented as a single hardware component (i.e., a decoder or processor), depending on the embodiment. Additionally, the memory 170 may include a digital storage block (DPB) and may be implemented using a digital storage medium.

[0102] The image decoding device 200, which receives a bitstream containing video / image information, can perform operations related to... Figure 2 The image encoding device 100 performs a process corresponding to the process of reconstructing the image. For example, the image decoding device 200 can perform decoding using a processing unit applied in the image encoding device. Therefore, the processing unit used for decoding can be, for example, an encoding unit. The encoding unit can be a coding tree unit, or it can be obtained by segmenting the maximum coding unit. In addition, the reconstructed image signal decoded and output by the image decoding device 200 can be played by a playback device (not shown).

[0103] Image decoding device 200 can receive data from [source] in bitstream form. Figure 2The received signal is output by the image encoding device. The entropy decoder 210 can decode the bitstream. For example, the entropy decoder 210 can parse the bitstream to extract information necessary for image reconstruction (or picture reconstruction) (i.e., video / image information). The video / image information may further include information about various parameter sets, such as adaptive parameter sets (APS), picture parameter sets (PPS), sequence parameter sets (SPS), or video parameter sets (VPS). Additionally, the video / image information may further include general constraint information. The image decoding device can further use the information about the parameter sets and / or general constraint information to decode the image. The signaling information, received information, and / or syntax elements described in this disclosure can be obtained from the bitstream through the decoding process. For example, the entropy decoder 210 can decode the information in the bitstream based on encoding methods such as exponential Golomb coding, CAVLC, or CABAC, and can output the syntax element values ​​necessary for image reconstruction and the quantized values ​​of the transform coefficients related to the residuals. More specifically, the CABAC entropy decoding method can receive bins corresponding to syntax elements in the bitstream, determine the context model using information about the target syntax element, decoding information of neighboring blocks and the target block, or information about previously decoded symbols / bins, predict the probability of bin occurrence based on the determined context model, and perform arithmetic decoding on the bins to generate symbols corresponding to each syntax element. In this case, the CABAC entropy decoding method can update the context model for the next symbol / bin using the decoded symbol / bin information after determining the context model. In the decoding information from the entropy decoder 210, prediction-related information can be provided to the predictors (inter-frame predictor 260 and intra-frame predictor 265), and the residual values ​​entropied by the entropy decoder 210, in other words, the quantized transform coefficients and related parameter information, can be input to the dequantizer 220. Additionally, in the decoding information from the entropy decoder 210, filtering-related information can be provided to the filter 240. Meanwhile, the receiver (not shown) that receives the signal output from the image encoding device can be additionally configured as an internal / external element of the image decoding device 200, or the receiver can be configured as a component of the entropy decoder 210.

[0104] Furthermore, the image decoding apparatus according to this disclosure can also be referred to as a video / image / picture decoding apparatus. The image decoding apparatus may include an information decoder (video / image / picture information decoder) and / or a sample decoder (video / image / picture sample decoder). The information decoder may include an entropy decoder 210, and the sample decoder may include at least one of a dequantizer 220, an inverse transformer 230, an adder 235, a filter 240, a memory 250, an inter-frame predictor 260, or an intra-frame predictor 265.

[0105] The dequantizer 220 can dequantize the quantized transform coefficients and output the transform coefficients. The dequantizer 220 can rearrange the quantized transform coefficients into two-dimensional blocks. In this case, the rearrangement can be performed based on the coefficient scanning order applied in the image encoding device. The dequantizer 220 can perform dequantization on the quantized transform coefficients using quantization parameters (i.e., quantization step size information) and obtain the transform coefficients.

[0106] The inverse transformer 230 can perform an inverse transformation on the transform coefficients to obtain the residual signal (residual block or residual sample array).

[0107] The predictor can perform predictions on the current block and generate a prediction block that includes prediction samples for the current block. The predictor can determine whether to apply intra-frame prediction or inter-frame prediction to the current block based on prediction-related information output from the entropy decoder 210, and can determine a specific intra-frame / inter-frame prediction mode (prediction method).

[0108] The predictor can generate a prediction signal based on various prediction methods (techniques) that will be described later, which is the same as described in the explanation of the predictor in the image coding device 100.

[0109] Intra-predictor 265 can predict the current block by referring to samples within the current image. The interpretation of intra-predictor 185 can also be applied to intra-predictor 265 in the same way.

[0110] Inter-frame predictor 260 can derive a prediction block for the current block based on a reference block (reference sample array) specified by motion vectors on a reference image. In this case, to reduce the amount of motion information transmitted in the inter-frame prediction mode, motion information can be predicted at the block, sub-block, or sample level based on the correlation of motion information between neighboring blocks and the current block. Motion information may include motion vectors and reference image indices. Motion information may further include information about inter-frame prediction orientation (i.e., L0 prediction, L1 prediction, Bi prediction, etc.). In inter-frame prediction, neighboring blocks may include spatially neighboring blocks within the current image and temporally neighboring blocks in the reference image. For example, inter-frame predictor 260 can construct a motion information candidate list based on neighboring blocks and derive the motion vector and / or reference image index of the current block based on the received candidate selection information. Inter-frame prediction can be performed based on various prediction modes (methods), and prediction-related information may include information indicating the inter-frame prediction mode (method) applied to the current block.

[0111] Adder 235 generates a reconstructed signal (reconstructed image, reconstructed block, reconstructed sample array) by adding the obtained residual signal to the prediction signal (predicted block, predicted sample array) output from the predictors (including inter-frame predictor 260 and / or intra-frame predictor 265). When there is no residual for the target block, such as when a skip mode is applied, the predicted block can be used as a reconstructed block. The interpretation of adder 155 can also be applied to adder 235 in the same way. Adder 235 can be referred to as a reconstructor or reconstructed block generator. The generated reconstructed signal can be used for intra-frame prediction of the next target block in the current image, and, as described later, can also be used for inter-frame prediction of the next image after filtering.

[0112] Filter 240 can apply filtering to the reconstructed signal to enhance subjective / objective quality. For example, filter 240 can apply various filtering methods to the reconstructed image to generate a modified reconstructed image, and the modified reconstructed image can be stored in memory 250, specifically in the DPB of memory 250. Various filtering methods may include, for example, deblocking filtering, adaptive sample shifting, adaptive loop filtering, bilateral filtering, etc.

[0113] The (modified) reconstructed image stored in the DPB of memory 250 can be used as a reference image in inter-frame predictor 260. Memory 250 can store motion information of blocks in the current image whose motion information has been derived (or decoded) and / or motion information of blocks in the reconstructed image. The stored motion information can be sent to inter-frame predictor 260 to be used as motion information for spatially or temporally neighboring blocks. Memory 250 can store reconstructed samples of reconstructed blocks in the current image and send them to intra-frame predictor 265.

[0114] The embodiments described in this specification for the filter 160, inter-frame predictor 180, and intra-frame predictor 185 of the image encoding apparatus 100 can be applied in the same or corresponding manner to the filter 240, inter-frame predictor 260, and intra-frame predictor 265 of the image decoding apparatus 200.

[0115] Intra-prediction mode / type determination

[0116] When intra-prediction is applied, the intra-prediction mode to be applied to the current block can be determined by using the intra-prediction modes of neighboring blocks. For example, the decoding device can select one of the most probable mode (MPM) candidates within an MPM list derived based on the intra-prediction modes of the current block's neighboring blocks (e.g., the left neighboring block and / or the upper neighboring block) and additional candidate modes based on the received MPM index. Alternatively, one of the remaining intra-prediction modes (and planar modes) not included in the MPM candidates can be selected based on remaining intra-prediction mode information. The MPM list can be configured to include or exclude planar modes as candidates. For example, when the MPM list includes planar modes as candidates, it can have six candidates, and when it excludes planar modes as candidates, it can have three candidates. When the MPM list excludes planar modes as candidates, a non-planar flag (i.e., intra_luma_not_planar_flag) indicating whether the intra-prediction mode of the current block is not a planar mode can be used to indicate this. For example, the MPM flag can be signaled first, and when the MPM flag is 1, the MPM index and non-planar flag can be signaled. Additionally, when the non-planar flag is 1, the MPM index can be signaled. Here, the configuration of excluding planar patterns as candidates from the MPM list does not mean that planar patterns are not considered MPMs, but rather that since planar patterns are always considered MPMs, the non-planar flag is signaled first to determine whether a pattern is a planar pattern.

[0117] For example, the MPM flag (e.g., `intra_luma_mpm_flag`) can be used to indicate whether the intra prediction mode applied to the current block is among the MPM candidates (and planar modes) or among the remaining modes. A value of 1 for the MPM flag indicates that the intra prediction mode of the current block is among the MPM candidates (and planar modes), and a value of 0 for the MPM flag indicates that the intra prediction mode of the current block is not among the MPM candidates (and planar modes). A value of 0 for the non-planar flag (i.e., `intra_luma_not_planar_flag`) indicates that the intra prediction mode of the current block is planar, and a value of 1 for the non-planar flag indicates that the intra prediction mode of the current block is not planar. The MPM index can be signaled in the form of syntax elements such as `mpm_idx` or `intra_luma_mpm_idx`, and the remaining intra prediction mode information can be signaled in the form of syntax elements such as `rem_intra_luma_pred_mode` or `intra_luma_mpm_remainder`. For example, the remaining intra-prediction mode information may indicate one of the remaining intra-prediction modes that is not included in the MPM candidates (and planar modes) and is indexed in the order of prediction mode numbers among all intra-prediction modes. The intra-prediction mode may be an intra-prediction mode for the luma component (samples). In the following, the intra-prediction mode information may include at least one of the following: an MPM flag (i.e., intra_luma_mpm_flag), a non-planar flag (i.e., intra_luma_not_planar_flag), an MPM index (i.e., mpm_idx or intra_luma_mpm_idx), and remaining intra-prediction mode information (i.e., rem_intra_luma_pred_mode or intra_luma_mpm_remainder). In this disclosure, the MPM list may also be referred to by various terms, such as the MPM candidate list or candModeList. When MIP is applied to the current block, the separate MPM flag (i.e., intra_mip_mpm_flag), MPM index (i.e., intra_mip_mpm_idx), and remaining intra-prediction mode information (i.e., intra_mip_mpm_remainder) used for MIP can be signaled, and the non-plane flag can be signaled without signaling.

[0118] The following examples demonstrate the intra-prediction mode signaling process in the encoding device and the intra-prediction mode determination process in the decoding device.

[0119] Figure 4 A flowchart illustrating an example of a signaling method for intra-frame prediction modes in an encoding apparatus is shown.

[0120] refer to Figure 4 The encoding device constructs an MPM list for the current block S400. The MPM list may include candidate intra-prediction modes (MPM candidates) with high probability of being applied to the current block. The MPM list may include intra-prediction modes of neighboring blocks, and may additionally include specific intra-prediction modes according to a predetermined method. Specific methods for configuring the MPM list will be described later.

[0121] The encoding device determines the intra-prediction mode of the current block S410. The encoding device can perform prediction based on various intra-prediction modes and can determine the optimal intra-prediction mode based on rate distortion optimization (RDO). In this case, the encoding device can determine the optimal intra-prediction mode by using only the MPM candidates and planar modes included in the MPM list, or by using the remaining intra-prediction modes in addition to the MPM candidates and planar modes included in the MPM list. Specifically, for example, when the intra-prediction type of the current block is a specific type (i.e., LIP, MRL, or ISP) instead of the normal intra-prediction type, the encoding device can determine the optimal intra-prediction mode by considering only the MPM candidates and planar modes as intra-prediction mode candidates for the current block. In other words, in this case, the intra-prediction mode of the current block can be determined only from the MPM candidates and planar modes, and in this case, the MPM flag can be notified by signaling without encoding. In this case, decoding can estimate the MPM flag as 1 without separately receiving the MPM flag.

[0122] Simultaneously, typically, when the intra-prediction mode of the current block is not a planar mode and is one of the MPM candidates in the MPM list, the coding device generates an MPM index (mpm idx) indicating one of the MPM candidates. When the intra-prediction mode of the current block is not included in the MPM list, the coding device generates remaining intra-prediction mode information, which indicates modes among the remaining intra-prediction modes (and planar modes) not included in the MPM list, and is the same as the intra-prediction mode of the current block.

[0123] The encoding device can encode intra-prediction mode information and output it as a bitstream. Intra-prediction mode information may include the aforementioned MPM flag, non-planar flag, MPM index, and / or remaining intra-prediction mode information. Typically, the MPM index and remaining intra-prediction mode information are interchangeable and may not be simultaneously signaled to indicate the intra-prediction mode of the block. In other words, an MPM flag value of 1 may be signaled along with either the non-planar flag or the MPM index, and an MPM flag value of 0 may be signaled along with the remaining intra-prediction mode information. However, as mentioned above, when a specific intra-prediction type is applied to the current block, the MPM flag may not be signaled, and only the non-planar flag and / or the MPM index may be signaled. In other words, in this case, the intra-prediction mode information may only include the non-planar flag and / or the MPM index.

[0124] The decoding device can determine the intra-prediction mode corresponding to the intra-prediction mode information determined by the encoding device and signaled by the signal.

[0125] Figure 5 A flowchart illustrating an example of an intra-frame prediction mode determination method in a decoding apparatus is shown.

[0126] refer to Figure 5 The decoding device obtains intra-prediction mode information from the bitstream S500. As described above, the intra-prediction mode information may include at least one of the following: MPM flag, non-planar flag, MPM index, and remaining intra-prediction modes.

[0127] The decoding device constructs the MPM list S510. The MPM list is configured in the same manner as the MPM list configured by the encoding device. In other words, the MPM list may include intra-prediction modes of neighboring blocks, and may further include specific intra-prediction modes according to a predetermined method. The specific method for configuring the MPM list will be described later.

[0128] Although S510 is illustrated as executing after S500, this is merely an example, and S510 can be executed before or simultaneously with S500.

[0129] The decoding device determines the intra-prediction mode of the current block S520 based on the MPM list and intra-prediction mode information. For example, when the MPM flag is 1, the decoding device can derive a planar mode as the intra-prediction mode of the current block (based on the non-planar flag), or it can derive a candidate indicated by the MPM index in the MPM candidates in the MPM list as the intra-prediction mode of the current block. In another example, when the MPM flag is 0, the decoding device can derive the intra-prediction mode indicated by the remaining intra-prediction mode information as the intra-prediction mode of the current block from the remaining intra-prediction modes and planar modes not included in the MPM list. Meanwhile, in another example, when the intra-prediction type of the current block is a specific type (i.e., LIP, MRL, IST, etc.), the decoding device can derive a planar mode from the MPM list or a candidate indicated by the MPM index as the intra-prediction mode of the current block without checking the MPM flag.

[0130] For example, when the MRL is not applied to the current block (i.e., when intra_luma_ref_idx == 0), the non-plane flag can be signaled, and when the MRL is applied to the current block (i.e., when intra_luma_ref_idx != 0), the non-plane flag can be omitted. When the non-plane flag is omitted, its value can be estimated as 1 by the decoding device.

[0131] Simultaneously, intra-prediction modes can include two directional intra-prediction modes and 65 directional prediction modes. Non-directional intra-prediction modes can include planar intra-prediction modes and DC intra-prediction modes, and directional intra-prediction modes can include intra-prediction modes numbered from 2 to 66. Extended directional intra-prediction modes can be applied to blocks of all sizes and can be applied to both the luma and chroma components.

[0132] In addition to the intra-prediction modes mentioned above, intra-prediction modes can further include cross-component linear model (CCLM) modes for chroma samples. Depending on whether left samples, top samples, or both are considered for deriving LM parameters, CCLM modes can be classified as LT_CCLM, L_CCLM, and T_CCLM, and can be applied only to the chroma component.

[0133] For example, intra-frame prediction modes can be indexed as shown in Table 1 below.

[0134] [Table 1]

[0135]

[0136] As mentioned above, the intra-prediction type (or additional intra-prediction mode, etc.) can include at least one of LIP, PDPC, MRL, ISP, and MIP. The intra-prediction type can be indicated based on intra-prediction type information, and this information can be implemented in various forms. As an example, the intra-prediction type information can include intra-prediction type index information indicating one of the intra-prediction types. In another example, the intra-prediction type information can include at least one of the following: reference sample line information indicating whether MRL is applied to the current block and which reference sample line is used when applying it (i.e., intra_luma_ref_idx); an ISP flag indicating whether ISP is applied to the current block (i.e., intra_subpartitions_mode_flag); ISP type information indicating the segmentation type of the subpartition when applying ISP (i.e., intra_subpartitions_split_flag); flag information indicating whether PDPC is applied; or flag information indicating whether LIP is applied. Additionally, the intra-prediction type information can include a MIP flag indicating whether MIP is applied to the current block (which can be referred to as intra_mip_flag).

[0137] Meanwhile, as mentioned above, when MIP is applied to the current block (i.e., when the value of intra_mip_flag is 1), the MPM list for MIP can be configured separately, and the intra-prediction mode information for MIP can include an MPM flag called intra_mip_mam_flag, an MPM index called intra_mip_mpm_idx, and the remaining intra-prediction mode information called intra_mip_mpm_remainder.

[0138] Furthermore, various prediction modes can be used for MIP, and the matrix and offset for MIP can be derived based on the intra-prediction modes used for MIP. As mentioned above, the matrix can be called the (MIP) weight matrix, and the offset can be called the (MIP) offset vector or (MIP) bias vector. The number of intra-prediction modes used for MIP can be set differently based on the size of the current block. For example, i) when the height and width of the current block (i.e., CB or TB) are both 4, 35 intra-prediction modes (i.e., intra-prediction modes 0 to 34) can be available; ii) when the height and width of the current block are both equal to or less than 8, 19 intra-prediction modes (i.e., intra-prediction modes 0 to 18) can be available; iii) in other cases, 11 intra-prediction modes (i.e., intra-prediction modes 0 to 10) can be available. For example, when the height and width of the current block are both 4, it can be referred to as block size type 0; when the height and width of the current block are both equal to or less than 8, it can be referred to as block size type 1; otherwise, it can be referred to as block size type 2. The number of intra-prediction modes used for MIP can be organized in the table below. However, this is merely an example, and the block size type and the number of available intra-prediction modes can be changed. In this disclosure, the intra-prediction mode used for MIP can be referred to as MIP intra-prediction mode, MIP prediction mode, or MIP mode.

[0139] [Table 2]

[0140]

[0141] Meanwhile, an auxiliary MPM list has been introduced in the Enhanced Compression Model (ECM). The traditional Primary MPM (PMPM) list includes six entries, and the Auxiliary MPM (SMPM) list includes sixteen entries. First, a general MPM list with twenty-two entries is configured, with the first six entries included in the PMPM list, and the remaining entries included in the SMPM list. The first entry in the general MPM list is the planar mode, and the remaining entries consist of the intra-mode of neighboring blocks on the left (L), top (A), bottom left (BL), top right (AR), and top left (AL), the directional mode with offset added from the first two available directional modes of neighboring blocks, and the default mode, such as... Figure 6 As described in [the text].

[0142] When the CU block is vertically oriented, the order of neighboring blocks can be top (A), left (L), bottom left (BL), top right (AR), and top left (AL). Otherwise, the order can be left (L), top (A), bottom left (BL), top right (AR), and top left (AL).

[0143] The PMPM flag is parsed, and when its value is 1, the PMPM index can be parsed to determine which entry in the PMPM list to select. Otherwise, the SPMPM flag can be parsed to determine whether to parse the SPMPM index or the remaining patterns.

[0144] Nearest reference sample export

[0145] When intra-frame prediction is applied to the current block, neighbor reference samples to be used for intra-frame prediction of the current block can be derived. The neighbor reference samples of the current block may include a total of 2×nH samples adjacent to the left boundary of the current block (nW×nH size) and its lower left corner, a total of 2×nW samples adjacent to the top boundary of the current block and its upper right corner, and a sample adjacent to the upper left corner of the current block. Alternatively, the neighbor reference samples of the current block may include multiple columns of upper neighbor samples and multiple rows of left neighbor samples. Additionally, the neighbor reference samples of the current block may include a total of nH samples adjacent to the right boundary of the current block (nW×nH size), a total of nW samples adjacent to the lower boundary of the current block, and a sample adjacent to the lower right corner of the current block.

[0146] Furthermore, when applying the MRL described below, the reference sample may be located in rows 1 through 3, instead of row 0 which is adjacent to the current block on the left / top, and in this case, the number of neighboring reference samples may further increase. The specific regions and numbers of neighboring reference samples are described below.

[0147] At the same time, when applying ISP, neighboring reference samples can be exported on a sub-partition basis.

[0148] Some neighboring reference samples of the current block may not have been decoded yet, or may be unavailable. In this case, the decoder can configure the neighboring reference samples to be used for prediction by interpolation of available samples.

[0149] Some neighboring reference samples of the current block may not have been decoded yet, or may be unavailable. In this case, the decoder can configure the neighboring reference samples to be used for prediction by extrapolating the available samples. Reference samples from the bottom left corner to the top right corner can be configured by replacing or filling pixels that have not been decoded or are unavailable with the last available sample while updating the reference samples to the latest sample (the last available sample).

[0150] Prediction samples derived based on intra-frame prediction mode / type

[0151] The predictor of the encoding / decoding device can derive reference samples from the intra-prediction mode of the current block in the neighboring reference samples of the current block, and can generate prediction samples for the current block based on the reference samples.

[0152] For example, (i) a predicted sample can be derived based on the average or interpolation of the neighboring reference samples of the current block, and (ii) a predicted sample can be derived based on a reference sample located on a specific (predictive) orientation in the neighboring reference samples of the current block. Case (i) can be referred to as a non-directional mode or a non-angular mode, and case (ii) can be referred to as a directional mode or an angular mode. Alternatively, a predicted sample can be generated by interpolation between a first neighboring sample and a second neighboring sample, the second neighboring sample being located on an orientation opposite to the predictive orientation of the intra-prediction mode of the current block relative to the predicted sample of the current block in the neighboring reference samples. The above cases can be referred to as Linear Interpolation Intra-Prediction (LIP). Additionally, a provisional predicted sample of the current block can be derived based on filtered neighboring reference samples, and a predicted sample of the current block can be derived by performing a weighted sum of the provisional predicted sample and at least one reference sample derived from the intra-prediction mode in conventional neighboring reference samples (in other words, unfiltered neighboring reference samples). The above cases can be referred to as Position-Related Intra-Prediction (PDPC). Additionally, among multiple neighboring reference samples in the current block, the reference sample row with the highest prediction accuracy can be selected, and the prediction sample is derived by using the reference sample located on the prediction orientation on the selected line. In this case, intra-frame prediction coding can be performed by signaling the used reference samples to the decoding device. This can be referred to as Multi-Reference Line Intra-Frame Prediction (MRL) or MRL-based intra-frame prediction. Furthermore, the current block can be divided into vertical or horizontal sub-partitions, and intra-frame prediction can be performed based on the same intra-frame prediction mode, while neighboring reference samples can be derived and used in the sub-partition units. In this case, the intra-frame prediction mode of the current block is applied to the sub-partitions in the same way, and intra-frame prediction performance can be improved by deriving and using neighboring reference samples in the sub-partition units. This prediction method can be referred to as Intra-Frame Sub-Partition (ISP) or ISP-based intra-frame prediction. This will be described in detail later. Additionally, when the prediction orientation based on the prediction sample points to a position between neighboring reference samples—in other words, when the prediction orientation points to a fractional sample position—the value of the prediction sample can be derived by interpolation of multiple reference samples located around the corresponding prediction orientation (i.e., those adjacent to the corresponding fractional sample position).

[0153] The aforementioned intra-prediction methods can be referred to as intra-prediction types, which are distinct from intra-prediction modes. Intra-prediction types can also be referred to by various terms, such as intra-prediction techniques or additional intra-prediction modes. For example, an intra-prediction type (or additional intra-prediction mode, etc.) may include at least one of LIP, PDPC, MRL, or ISP. Information about the intra-prediction type can be encoded by the encoding device and included in the bitstream to be signaled to the decoding device. Information about the intra-prediction type can be implemented in various forms, such as flag information indicating whether each intra-prediction type is applied, or index information indicating one of multiple intra-prediction types.

[0154] The MPM list for exporting the above intra-prediction modes can be configured differently depending on the intra-prediction type. Alternatively, the MPM list can be configured uniformly regardless of the intra-prediction type.

[0155] Nearest reference sample export

[0156] When intra-frame prediction is applied to the current block, neighbor reference samples to be used for intra-frame prediction of the current block can be derived. The neighbor reference samples of the current block may include a total of 2×nH samples adjacent to the left boundary of the current block (nW×nH size) and its lower left corner, a total of 2×nW samples adjacent to the top boundary of the current block and its upper right corner, and a sample adjacent to the upper left corner of the current block. Alternatively, the neighbor reference samples of the current block may include multiple columns of upper neighbor samples and multiple rows of left neighbor samples. Additionally, the neighbor reference samples of the current block may include a total of nH samples adjacent to the right boundary of the current block (nW×nH size), a total of nW samples adjacent to the lower boundary of the current block, and a sample adjacent to the lower right corner of the current block.

[0157] Furthermore, when multiple reference lines (MRLs) are applied, reference samples may be located in rows 1 through 3, rather than in row 0 which is adjacent to the current block on the left / top. In this case, the number of neighboring reference samples may further increase. The specific regions and numbers of neighboring reference samples are described below.

[0158] At the same time, when applying intra-fractional sub-partitions (ISP), neighboring reference samples can be exported on a per-sub-partition basis.

[0159] DIMD (Decoder-side Intra-Frame Mode Export)

[0160] In DIMD, intra-frame prediction can be derived as a planar orientation and a weighted average of the two derived orientations. To achieve this, two angular modes can be selected from the histogram of gradients (HoG) computed from the neighboring pixels of the current block. When two modes are selected, their predictors (predicted blocks) and planar predictors can be computed normally, and the corresponding weighted average can be used as the final predictor (final predicted block) for the current block. In this case, to determine the weights, the corresponding magnitudes in the HoG can be used for each of the two modes.

[0161] Since the exported intra-frame modes are included in the main list of intra-frame MPMs, the DIMD process can be performed before configuring the MPM list. The main exported intra-frame modes of the DIMD block can be stored with the block and can be used to configure the MPM list of neighboring blocks.

[0162] In DIMD chroma mode, such as Figure 12 As shown, the DIMD derivation method can be used to derive the chroma intra-prediction mode of the current block based on pre-reconstructed neighboring Y, Cb, and Cr samples in the second neighboring row and column. Specifically, to construct the HoG, the horizontal and vertical gradients can be computed not only for the pre-reconstructed Cb and Cr samples but also for each juxtaposed pre-reconstructed luminance sample of the current chroma block. Then, the intra-prediction mode with the highest histogram magnitude can be used to perform chroma intra-prediction of the current chroma block.

[0163] When the intra-prediction mode derived from the DIMD chroma mode is the same as the intra-prediction mode derived from the DM mode, the intra-prediction mode with the second highest histogram amplitude can be used as the DIMD chroma mode. To indicate whether the above-mentioned DIMD chroma mode has been applied, a predetermined CU-level flag can be signaled.

[0164] (6) Fusion for Template-Based Intra-Frame Mode Derivation (TIMD)

[0165] For each intra-prediction mode in the MPM, the SATD between the predicted and reconstructed samples of the template can be calculated. Then, the two intra-prediction modes with the minimum SATD can be selected as TIMD modes. These two TIMD modes can be fused according to weights, and this weighted intra-prediction can be used to encode the current CU. The derivation of TIMD modes can include the aforementioned Position-Related Intra-Prediction Combination (PDPC).

[0166] The costs of the two selected modes can be compared with a predetermined threshold, and a cost factor of 2 can be applied, as shown in Equation 1 below.

[0167] [Formula 1]

[0168] costMode2 < 2 * costModel

[0169] When the condition in Equation 1 is true, the above fusion can be applied. Conversely, when the condition in Equation 1 is false, only Mode 1 can be used.

[0170] Meanwhile, the weights of the patterns can be calculated based on their corresponding SATD costs, as shown in Equation 2 below.

[0171] [Equation 2]

[0172] weight1=costMode2 / (costMode1+costMode2)

[0173] weight2 = 1 - weight1

[0174] MIP (Matrix-based Intra-frame Prediction)

[0175] Figures 7 to 11 It is a diagram describing the matrix-based intra-frame prediction (MIP) process that can be applied to this disclosure.

[0176] Matrix-based intra-prediction (MIP) can be called affine line-weighted intra-prediction (ALWIP) or matrix-weighted intra-prediction (MIP or MWIP). To predict samples of a rectangular block with width W and height H, MIP uses a line consisting of H reconstructed samples from the left boundary of the adjacent block and W reconstructed samples from the upper boundary as input. When reconstructed samples are unavailable, they can be generated in the same way as those generated through existing intra-prediction.

[0177] The prediction signal can be generated essentially by following these steps:

[0178] 1. The average can be extracted by averaging the four samples when W=H=4 and averaging the eight samples for other cases among the boundary samples (averaging process).

[0179] 2. Matrix-vector multiplication can be performed after offset addition by using the averaged sample as input. As a result of this process, a simplified prediction signal for the subsampled sample set of the original block can be obtained (matrix-vector multiplication process).

[0180] 3. The predicted signal at the remaining location can be generated by linear interpolation of the predicted signal from the subsample set, which is a single-step linear interpolation ((linear) interpolation process) in each direction.

[0181] The matrices and offset vectors required to generate the predicted signal (predicted block or predicted sample) can be obtained from three sets of matrices S0, S1, and S2. Set S0 consists of 18 matrices. The matrix is ​​composed of 16 rows, 4 columns, and 18 offset vectors. Each of these has a size of 16. The matrices and offset vectors of the corresponding sets can be used to form 4x4 blocks. Meanwhile, set S1 consists of 10 matrices. Each matrix may have 16 rows, 8 columns, and 10 offset vectors. Each of these has a size of 16. The matrices and offset vectors of the corresponding sets can be used for blocks of 4x8, 8x4, and 8x8 sizes. Finally, set S2 consists of 6 matrices. The matrix is ​​composed of 64 rows, 8 columns, and 6 offset vectors. Each of these has a size of 64. The corresponding set of matrices and offset vectors can be used for blocks of all other sizes except those mentioned above.

[0182] The number of multiplications required for a matrix-vector product operation can always be less than or equal to 4·W·H. In other words, in MIP mode, up to four multiplication operations may be required for each sample.

[0183] Overview of the entire MIP process

[0184] By reference Figures 7 to 11 Describe the entire process of averaging, matrix-vector multiplication, and linear interpolation. Meanwhile, Figures 7 to 11 The block shape not shown in the figure can also be like... Figures 7 to 11 It was handled in the manner disclosed in the document.

[0185] 1. In a 4x4 block, MIP can take two averages based on the axes of each boundary. The resulting four input samples can be fed into matrix-vector multiplication. The matrix can be obtained from set S0. After offset addition, 16 final prediction samples can be obtained. Linear interpolation may not be necessary for generating the prediction signal. In other words, (4·16) / (4·4) = 4 multiplications can be performed for each sample.

[0186] 2. In a 4x4 block, MIP can take four averages based on the axes of each boundary. The resulting eight input samples can be fed into matrix-vector multiplication. The matrix can be obtained from set S1. Therefore, 16 samples can be obtained at odd positions in the prediction block. In other words, (8·16) / (8·8) = 2 multiplications can be performed for each sample. After offset summation, these samples can be vertically interpolated using the reduced upper boundary. Horizontal interpolation can be performed using the original left boundary. In this case, multiplications may not be required during interpolation. In other words, it is possible that only a total of 2 multiplications are needed per sample to derive the MIP prediction.

[0187] 3. In an 8x4 block, MIP can take the four original boundary values ​​of the left boundary and the four averages based on the horizontal axis of the boundary. The resulting eight input samples can be fed into a matrix-vector multiplication. The matrix can be obtained from set S1. Therefore, 16 samples can be obtained at each vertical position and odd horizontal position of the prediction block. In other words, (8·16) / (8·4) = 4 multiplications can be performed for each sample. After offset addition, these samples can be horizontally interpolated using the original left boundary. In this case, no multiplication process needs to be added during interpolation. In other words, only a total of 4 multiplications are needed for each sample to derive the MIP prediction. The transpose case can also be handled in this way.

[0188] 4. In a 16x16 block, MIP can take 4 averages based on each axis of the boundary. The resulting 8 input samples can be fed into matrix-vector multiplication. The matrix can be obtained from set S2. Therefore, 64 samples can be obtained at odd positions in the prediction block. In other words, (8·64) / (16·16) = 2 multiplications can be performed for each sample. After offset summation, these samples can be vertically interpolated using the 8 averages at the upper boundary. Horizontal interpolation can be performed using the original left boundary. In this case, no multiplication process needs to be added during interpolation. In other words, it is possible that only a total of 2 multiplications are needed per sample to derive the MIP prediction.

[0189] The corresponding process can also be applied in essentially the same way to larger blocks. In this case, it can be shown that the number of multiplications per sample is less than 4.

[0190] For a Wx8 block with W > 8, horizontal interpolation alone may be necessary because samples are provided at odd horizontal positions and for each vertical position. In this case, (8·64) / (W·8) = 64 / W multiplications can be performed per sample to compute the reduced prediction. For W = 16, linear interpolation may not require additional multiplications, but for W > 16, the additional number of multiplications per sample required for linear interpolation can be less than 2. Therefore, the total number of multiplications per sample can be less than or equal to 4.

[0191] Finally, for a W×4 block with W>8, assume A kThis is a matrix generated by excluding all rows corresponding to odd-numbered terms based on the horizontal axis of the downsampled block. In this case, the output size is 32, and only horizontal interpolation can be performed again. The reduced prediction can be computed using (8·32) / (W·4) = 64 / W multiplications per sample. For W = 16, no additional multiplications are needed, but for W > 16, the number of multiplications per sample required for linear interpolation can be less than 2. Therefore, the total number of multiplications can be less than or equal to 4. The transpose case can also be handled in this way.

[0192] Boundary average

[0193] Figure 12 This is a diagram illustrating an example of a boundary averaging process that can be applied to this disclosure. As an example, boundary averaging can be included in the MIP process.

[0194] According to the averaging process, the average can be applied to each boundary (left or top boundary). Here, a boundary can refer to a neighboring reference sample adjacent to the boundary of the current block, such as a reference... Figures 7 to 11 As described above. For example, the left boundary (bdry) left ) can represent the left neighbor reference sample adjacent to the left boundary of the current block, while the upper boundary (bdry) top This can represent the upper nearest neighbor reference sample adjacent to the upper boundary of the current block. When the current block size is 4x4, the size of each boundary can be reduced to 2 samples through averaging. When the current block is not 4x4 in size, the size of each boundary can be reduced to 4 samples through averaging.

[0195] In the first step, input boundary bdry top and bdry left It can be reduced to a smaller boundary. and Here, for a 4x4 block, and Both can consist of two samples, and for all other cases, and Both can consist of 4 samples.

[0196] For a 4x4 block, in the case of 0 ≤ i < 2, it can be defined as follows:

[0197] [Formula 3]

[0198]

[0199] and, It can also be defined similarly to Equation 3.

[0200] For other cases, when the block width W is W = 4.2k When i ≤ 4, it can be defined as follows:

[0201] [Formula 4]

[0202]

[0203] and, It can also be defined similarly to Equation 4.

[0204] Because of these two shrinkage boundaries and Connected to the reduced boundary vector bdry red Therefore, a 4x4 block can have a size of 4, while other shapes can have a size of 8.

[0205] Additionally, when the variable `mode` represents the MIP mode, the following cascading can be defined:

[0206] [Formula 5]

[0207]

[0208] Finally, in the interpolation of the predicted signal from the subsampled data, for larger blocks, a further modified version of the average boundary may be needed, i.e., a second version. In other words, for min(W, H) > 8 and W ≥ H, W = 8 * 2 l And in cases where 0 ≤ i < 8, it can be defined as follows:

[0209] [Formula 6]

[0210]

[0211] When min(W, H) > 8 and H > W, it can also be defined similarly.

[0212] Reduced prediction signals are generated through matrix-vector multiplication.

[0213] A reduced input vector bdry red A reduced prediction signal can be generated. The reduced prediction signal can be used for signals with a width of W. red And the height is H red The signal of the downsampled block. Here, W red and H red It can be defined as follows:

[0214] [Formula 7]

[0215]

[0216] Reduced prediction signal pred redIt can be derived through matrix-vector product calculation and offset addition.

[0217] [Formula 8]

[0218] pred red =A·bdry red +b

[0219] Here, A can be any value for W = H = 4 with W red ·H red A matrix with 1 row and 4 columns, or, in other cases, an 8-column matrix. Where b can be W red ·H red A vector of size.

[0220] Matrix A and vector b can be obtained from one of sets S0, S1, and S2 as follows. The index idx = idx(W, H) can be defined as follows:

[0221] [Formula 9]

[0222]

[0223] Additionally, m can be derived as follows:

[0224]

[0225] Then, in the case where idx ≤ 1 or idx = 2 and min(W, H) > 4, it can be and When idx = 2 and min(W, H) = 4, for W = 4, A can correspond to the odd x-coordinate of the downsampled block, or for H = 4, it can correspond to the odd y-coordinate of the downsampled block.

[0226] Finally, the transposed signal can be used instead of the reduced prediction signal in the following cases:

[0227] 1. W = H = 4 and mode ≥ 18

[0228] 2. max(W, H) = 8 and mode ≥ 10

[0229] 3. max(W, H) > 8 and mode ≥ 6

[0230] For W = H = 4, calculate pred red The total number of multiplications required is likely 4, and in this case, it's because A has 16 rows and 4 columns. In other cases, A has 8 columns and W... red ·H red Yes, and in this case, it may be indicated that 8W is required. red ·H red≤ 4·W·H multiplications. In other words, in this case, each sample may require up to 4 multiplications to compute pred red .

[0231] linear interpolation

[0232] Figure 13 is a diagram showing an example of a linear interpolation process that can be applied to the present disclosure. As an example, the linear interpolation process may be included in the MIP process. As an example, the interpolation process may be a linear interpolation or bilinear interpolation process. The interpolation process may include two steps consisting of 1) vertical interpolation and 2) horizontal interpolation as illustrated in Figure 13 . When W >= H, vertical linear interpolation may be applied first, and then horizontal linear interpolation may be applied. When W < H, horizontal linear interpolation may be applied first, and then vertical linear interpolation may be applied. In a 4x4 block, the interpolation process may be omitted.

[0233] For a W×H block with max(W, H) ≥ 8, a prediction signal may be generated by linear interpolation based on a downsampled prediction signal. Based on the block shape, linear interpolation may be applied vertically, horizontally, or in both directions. When linear interpolation is applied in both directions, it may be applied first in the horizontal direction; and for W < H, it may be applied first in the vertical direction.

[0234] Assume there is a WxH block with max(W, H) ≥ 8 and W ≥ H. Here, one-dimensional linear interpolation may be performed as follows. For generality, vertical linear interpolation may be performed. First, the downsampled prediction signal may be extended to the top by a boundary signal. The vertical upsampling factor may be defined as U ver = H / H red , and may be In this case, the extended downsampled prediction signal may be defined as follows:

[0235] [Equation 10]

[0236]

[0237] In this case, a vertically linearly interpolated prediction signal may be generated from the extended downsampled prediction signal as follows.

[0238] [Equation 11]

[0239]

[0240] For 0 ≤ x < W red , 0 ≤ y < H red and 0 ≤ k < U ver .

[0241] Example

[0242] This disclosure relates to intra-frame prediction, and more specifically, to a technique for performing matrix-based intra-frame prediction (MIP) using only a portion of a reference sample. Figure 14 This is a diagram used to describe the MIP prediction method that can be applied to this disclosure. According to Figure 14 For example, for the prediction of the current block, neighboring reference samples can be used as input to obtain the predicted value of the current block as output through a neural network.

[0243] Meanwhile, the current MIP prediction can generate a predicted block by using both the left reference sample and the top reference sample around the block. However, the samples of the current block may have partial correlation with some reference samples such as the top reference sample or the left reference sample, and therefore, higher prediction accuracy can be obtained when performing the prediction of the current block by using some reference samples.

[0244] Therefore, this disclosure proposes a method for adaptively performing MIP prediction in a reference sample region. In other words, this disclosure proposes a method for performing MIP prediction by using only a portion of the neighboring reference samples of the current prediction block.

[0245] In the following, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings.

[0246] Figure 15 This is a diagram illustrating a method for performing MIP prediction according to embodiments of the present disclosure, and Figure 16 This is a diagram used to describe a MIP-based image encoding or decoding method according to embodiments of the present disclosure. In this embodiment, reference will also be made to... Figure 14 , 15 Sections 1 and 16 describe a method for performing MIP predictions using only some of the reference samples from the current block.

[0247] at the same time, Figure 16 The image encoding method or image decoding method can be executed in the image encoder or image decoder, respectively, and the image encoder and image decoder can include the means described with reference to the other figures above.

[0248] First, the MIP-based image encoding / decoding method S1610 according to embodiments of this disclosure can be executed. As an example, the method in S1610 can be executed based on some selected reference samples.

[0249] As an example, to perform MIP-based image encoding or decoding, some of the left and top reference samples can be selected. For example, the top reference sample S1620 of the current block can be selected for use in performing MIP. As mentioned above, the pixels of the current block may have partial correlations with some reference samples such as the top or left reference samples, and therefore, higher prediction accuracy can be obtained when performing prediction of the current block using partial reference samples. Here, according to... Figure 15 In embodiments of this disclosure, only the upper reference sample can be selected for MIP prediction and used as the MIP prediction input. In this case, only the upper reference sample can be selected as the neural network input for MIP prediction. Meanwhile, as... Figure 15 As illustrated in (a), the upper reference sample can correspond to the sample immediately adjacent to the current block. In other words, according to... Figure 15 In the embodiment of (a), a reference sample can be used that is immediately adjacent to the top of the current block. However, as another example, as illustrated in (b), adjacent samples that are adjacent to the top of the current block but whose number is greater than the width of the current block (e.g., extending further to the right) can be included in the upper reference sample, or reference samples from multiple reference sample lines can be included in the upper reference sample. Figure 15 In the embodiment of (b), multiple lines from the upper reference sample and the upper right reference sample of the current block can be used. As another example, as illustrated in (c), adjacent samples that are adjacent to the top of the current block but whose number is greater than the width of the current block (e.g., extending further to the left and / or right) can be included in the upper reference sample, or reference samples from multiple reference sample lines can be included. Figure 15 In embodiment (c), the top, top-right, or top-left reference sample of the current block can be used. Meanwhile, the number of reference samples or reference sample lines further extending to the left or right is not limited to... Figure 15 The number disclosed in the document. Furthermore, for the method proposed in this embodiment, some samples within one or more reference sample lines can be used for MIP prediction, and only some regions of all upper reference samples of the current block can be used for the MIP prediction proposed in this embodiment.

[0250] Then, the MIP mode S1630 can be selected based on the MIP mode information. The MIP mode can be applied to the current block, or it can be explicitly sent as a signal via a bitstream.

[0251] Then, a neural network-based MIP (S1640) can be applied based on the selected MIP pattern. In other words, MIP-based prediction can be performed. Meanwhile, as an example, based on the selected reference sample input, prediction can be made through methods such as... Figure 14 and / or Figure 15The neural network shown obtains predicted block sample values ​​as output. Here, the neural network can be adaptively selected based on the reference sample input. In other words, S1640 may include a process in which MIP-based prediction is applied based on the selected input sample values ​​and the selected neural network. As an example, a neural network consisting of a matrix can be used to perform MIP. Meanwhile, the predicted block sample values ​​as output can be derived as follows:

[0252] [Equation 12]

[0253] pred = Clip(A·r)

[0254] In the formula, *r* can refer to the reference sample input selected by the method proposed in this embodiment, and *pred* can refer to the predicted block sample value as the output. In other words, the predicted block sample value can be derived based on the selected reference sample value. As an example, *A* can refer to the trained neural network matrix, and the *Clip* function is a clipping function, which can be a function used to limit the base and / or apex values ​​as a function to clip the output predicted sample (pred) value within a certain range of sample values. In this case, the clipping range can be predefined. In other words, the output *pred* can be obtained by operating on a matrix *A* over the input *r*. The size of matrix *A* can be adaptively changed according to the size of the input and output.

[0255] Furthermore, the configuration of the neural network presented in this embodiment for obtaining the output pred for the input r is not limited to this example. In other words, in addition to using a neural network with a simple matrix, multiple matrix operations can be used by configuring multiple hidden layers, and bias terms and activation functions can be used for each neural network layer.

[0256] In addition, when outputting predicted samples, values ​​can be output in the frequency domain in addition to the spatial region. In this case, an inverse transformation can be performed on the predicted (pred) value to obtain the final predicted sample value.

[0257] Furthermore, as in the example above, the size of the neural network matrix can be adaptively selected based on the input and output. In this embodiment, the selected input (i.e., the selected reference sample) can be sampled using appropriate methods such as downsampling, and then used as the input for the neural network. In other words, reference samples obtained by downsampling the selected reference sample at a ratio of 1:2, 1:4, or 1:8 can be used as the input for the neural network. In this case, the size of the input can be reduced, thus reducing the size of the neural network matrix and significantly reducing the total number of MIP operations.

[0258] Furthermore, the sampling method and scale of the reference samples selected in this example are not limited to this example, and in addition to downsampling, upsampling can also be used to increase the input. The sampling method and scale of the reference samples can also be adaptively selected according to the shape and size of the block and the number of selected reference samples. In addition, the selected reference samples can be preprocessed by various filters such as low-pass filtering and high-pass filtering, and then used as input to the neural network. Alternatively, the selected reference samples can be transformed into the frequency domain by applying a transformation and then input into the neural network. After finding the average value of the selected reference samples, the residual sample value obtained by subtracting the corresponding average value from all reference samples can also be used as input to the neural network.

[0259] Furthermore, the aforementioned input filtering and sampling methods can be similarly applied to the output. In other words, when the number of outputs of the neural network (i.e., the number of output samples) is less than or greater than the number of prediction samples required for the current block size, the output values ​​can be upsampled or downsampled to match the number of prediction samples required for the current block size. Additionally, when a transformation is applied to the input and frequency domain inputs are used, or residual samples excluding the average are used as inputs, modifications can be made to the output.

[0260] As in the method in the example above, the number of input and output samples for neural network matrix operations proposed in this disclosure is not limited to a specific number.

[0261] Furthermore, post-processing filtering can also be applied to prediction blocks predicted using the methods proposed in this disclosure. The Position-Related Intra-Prediction Combination (PDPC) method can also be applied to prediction blocks predicted using the methods proposed in this disclosure. As an example, a smoothing filter can also be applied to prediction blocks predicted using the methods proposed in this invention.

[0262] Furthermore, as an example, when configuring a neural network for the method proposed in this embodiment, the coefficients of the matrices used in the operations within the neural network can have a wide variety of bit precision. For example, when performing 10-bit precision matrix operations, the matrix coefficients have a specific range of values ​​(e.g., 0–1023 or -512–511), and the inputs and outputs can be adaptively adjusted accordingly. These bit precision values ​​are not limited to the examples above and can be adaptively changed based on factors such as the number of layers in the neural network.

[0263] Furthermore, in the method proposed in this embodiment, the number of neural network matrix patterns (the number of patterns in the proposed MIP method) can be configured in various ways based on patterns (e.g., intra-frame patterns or inter-frame patterns), the width of the input block, the height of the input block, the number of pixels in the input block, the position of sub-blocks within the block, the syntax for explicit signal transmission, the statistical characteristics of neighboring pixels, and whether a quadratic transformation is used. For example, the number of neural network matrix patterns can be adaptively selected based on the size of the current block. Alternatively, the number of neural network matrix patterns can be selected with the same size for all blocks.

[0264] As an example, the coefficients of the neural network matrix can be adaptively defined for all block shapes. For example, neural network coefficients can be defined corresponding to each block shape from 4x4 to 256x256. The number of inputs and outputs of the neural network matrix can be predetermined based on the block shape. For example, as in... Figure 15 In the example of (a), MIP prediction is performed using only one upper reference sample line and 49 types of neural network matrices corresponding to 4, 8, 16, 32, 64, 128, and 256 as inputs and 4, 8, 16, 32, 64, 128, and 256 as outputs. Furthermore, each of the 49 neural network matrices can have matrix coefficients according to the number of MIP patterns proposed in this embodiment. As an example, when the number of MIP patterns with a neural network matrix having 8 inputs and 16 outputs (which can correspond to an 8x16 block shape) is 35, the corresponding neural network matrix can have a total of 8x16x35 matrix coefficients. Alternatively, as another example, only neural network matrices suitable for a specific block shape can be defined, and they can have restricted inputs and outputs. In this case, the specific block shape neural network matrix can be utilized by appropriately preprocessing or post-processing the inputs and outputs. As an example, when only a neural network matrix applicable to square block shapes is defined, it can correspond to the input of the corresponding square block neural network matrix by preprocessing (e.g., downsampling or upsampling) reference samples of non-square blocks. The output can also correspond to the predicted values ​​of non-square blocks by postprocessing (e.g., downsampling or upsampling).

[0265] Meanwhile, according to the method proposed in this embodiment, the neural network matrix mode (the mode in the proposed MIP method) can be selected based on at least one of the following: mode (e.g., intra-frame mode or inter-frame mode), input block width, input block height, number of pixels in the input block, position of sub-blocks within the block, syntax of explicit signal transmission, statistical characteristics of neighboring pixels, and whether a quadratic transform is used. In this case, in order to send the selected mode information to the decoder, the encoder can send the mode information by selecting and applying an appropriate binary method. As an example, the encoder can configure a specific mode of transmission, such as intra-frame luma mode, as an MPM mode, and then perform binarization after configuring the remaining modes separately. In this case, binarization bits can be saved by performing appropriate context modeling in each binarization. Alternatively, as another example, considering the total number of MIP modes, the mode information can also be sent to the decoder by using binarization such as truncated binary / truncated unary / fixed length.

[0266] Furthermore, to select the primary and secondary transform types in the prediction block predicted by the method proposed in this embodiment, the transform mode can be selected based on at least one of the following: neural network matrix pattern, input block width, input block height, number of pixels in the input block, position of sub-blocks within the block, explicit signal transmission syntax, and statistical characteristics of neighboring pixels. As an example, when selecting the primary transform type of the block to which the prediction method proposed in this disclosure is applied, the Multiple Transform Set (MTS) transform method can be applied. As an example, when selecting the secondary transform type of the block to which the prediction method proposed in this disclosure is applied, a secondary transform kernel suitable for intra-planar frame modes can be selected.

[0267] Simultaneously, candidate MIP prediction modes for the current block, such as DIMD or TIMD, can be determined to select the MIP mode proposed in this embodiment, or the MIP prediction mode for the current block can be determined. In other words, candidate MIP prediction modes or the MIP prediction mode for the corresponding block can be inferred by utilizing neighboring reference samples such as DIMD or TIMD.

[0268] As another example, in the method proposed in this embodiment, the mode candidate inference method used in DIMD can be applied to MIP mode prediction. In other words, pixel gradients can be calculated using only the region of the upper reference sample, and the first or second MIP prediction mode obtained by the corresponding gradient can be configured as the MIP prediction mode candidate for the current block.

[0269] As another example, in the method proposed in this embodiment, the pattern inference method used in TIMD can be applied to MIP pattern prediction. In other words, TIMD template matching can be performed using only the region of the upper reference sample. MIP prediction pattern candidates for which TIMD template matching will be applied can be selected, similar to MPM, or multiple MIP prediction patterns can be selected.

[0270] Furthermore, whether the method proposed in this embodiment is applied can be signaled as information in the High-Level Syntax (HLS), such as VPS, SPS, PPS, image header, slice header, DCI, etc. As an example, to determine whether the method proposed in this embodiment is applied on a PPS basis, information regarding whether the method proposed in this embodiment is applied can be included within the PPS and signaled.

[0271] Additional information regarding whether to apply the method proposed in this embodiment can be explicitly sent via signaling. However, as another example, the method proposed in this embodiment can be inferred or adaptively selected without additional information signaling. For example, whether the method proposed in this embodiment is applied at the CTU or CU level can be signaled via a 1-bit flag.

[0272] As an example, the method proposed in this embodiment can only be used if the MIP pattern is defined to be used in the High-Level Syntax (HLS). Alternatively, the application of the method proposed in this embodiment can be determined by signaling additional information within the MIP pattern. For example, when the value of information regarding whether MIP is applied (e.g., a MIP flag) is a specific value (e.g., 1, true), information regarding whether the method proposed in this embodiment is applied can be signaled using a 1-bit flag.

[0273] Alternatively, when the method proposed in this embodiment can be applied based on the size or shape of a specific block or the presence of specific conditions, only in this case can a 1-bit flag be used to signal whether the method proposed in this embodiment is applied. As an example, when the block height is at least four times the block width, the method proposed in this embodiment may not be applied, and therefore, the signaling of additional information regarding whether it is applied accordingly can be omitted.

[0274] Furthermore, as another example, under certain conditions, it can be implicitly inferred whether the method proposed in this embodiment is applied. For example, when the left reference sample of the current block does not exist (such as a block existing at the left boundary of an image) or when the left reference sample cannot be used, signaling information regarding whether the method proposed in this embodiment is applied can be omitted. Additionally, signaling information regarding whether the MIP mode is applied can also be omitted. In this case, it can be inferred that the MIP method proposed in this embodiment is always applied. When the left reference sample of the current block is a CTU boundary / tile boundary / slice boundary / sub-image boundary, signaling information regarding whether the method proposed in this embodiment is applied can be omitted. Additionally, signaling information regarding whether the MIP mode is applied can also be omitted. In this case, it can be inferred that the MIP method proposed in this embodiment is always applied.

[0275] Furthermore, as an example, when determining whether to apply the method proposed in this embodiment at a higher level as defined in the High-Level Syntax (HLS), it is possible to adaptively determine whether to send information about the application of the method proposed in this embodiment in the coding unit. For example, when the value of the information indicating whether the method proposed in this embodiment is applied, transmitted by signal in the SPS, is a specific value (e.g., 0, false) (i.e., when it indicates that the method proposed in this embodiment is not used at the SPS level), the method proposed in this embodiment is also not applied at the coding unit level, and signaling regarding whether it is used accordingly may not be performed.

[0276] Figure 17 and Figure 18 This is a diagram illustrating a method for performing MIP prediction according to embodiments of the present disclosure, and Figure 19 This is a diagram used to describe a MIP-based image encoding or decoding method according to embodiments of the present disclosure. In this embodiment, reference will also be made to... Figures 17 to 19 This section describes a method for performing MIP predictions using only some reference samples from the current block.

[0277] at the same time, Figure 19 The image encoding method or image decoding method can be executed in the image encoder or image decoder, respectively, and the image encoder and image decoder can include the means described with reference to the other figures above.

[0278] First, the MIP-based image encoding / decoding method S1910 according to an embodiment of this disclosure can be executed. As an example, the method in S1910 can be executed based on some selected reference samples.

[0279] As an example, to perform MIP-based image encoding or decoding, some of the left and top reference samples can be selected. For example, the left reference sample S1920 of the current block can be selected for MIP execution. As mentioned above, the pixels of the current block may have partial correlations with some reference samples such as the top or left reference samples, and therefore, higher prediction accuracy can be obtained when performing prediction of the current block using partial reference samples. Here, according to... Figure 18 In embodiments of this disclosure, only the left reference sample can be selected for MIP prediction and used as the MIP prediction input. In this case, only the left reference sample can be selected as the neural network input for MIP prediction. Meanwhile, as... Figure 18 As illustrated in (a), the left reference sample can correspond to the sample immediately adjacent to the current block. In other words, according to... Figure 18 In the embodiment of (a), only the reference sample immediately to the left of the current block may be used. However, as another example, such as Figure 18 As illustrated in (b), adjacent samples that are to the left of the current block but whose number is greater than the height of the current block (e.g., further extended to the bottom) can be included in the left reference sample, or, reference samples from multiple reference sample lines can be included in the left reference sample. Figure 18 In the embodiment of (b), multiple lines of the left and lower-left reference samples of the current block can be used. As another example, such as Figure 18 As illustrated in (c), adjacent samples that are to the left of the current block but whose number is greater than the height of the current block (e.g., extending further to the top and / or bottom) can be included in the left reference sample, or reference samples from multiple reference sample lines can be included. Figure 18 In embodiment (c), a left, top-left, or bottom-left reference sample of the current block can be used. Meanwhile, the number of reference samples or reference sample lines further extending to the top or bottom are not limited to... Figure 18 The number disclosed in the document. Additionally, for the method proposed in the embodiments, some samples within one or more reference sample lines can be used for MIP prediction, and only some regions of all left reference samples of the current block can be used for the MIP prediction proposed in this embodiment.

[0280] Then, the MIP mode S1930 can be selected based on the MIP mode information. The MIP mode can be applied to the current block, or it can be explicitly sent as a signal via a bit stream.

[0281] Then, a neural network-based MIP S1940 can be applied based on the selected MIP pattern. In other words, MIP-based prediction can be performed. Meanwhile, as an example, based on the selected reference sample input, prediction can be made through methods such as... Figure 14 and / or Figure 17 The neural network shown obtains predicted block sample values ​​as output. Here, the neural network can be adaptively selected based on the reference sample input. In other words, S1940 may include a process of applying MIP-based predictions based on the selected input sample values ​​and the selected neural network.

[0282] As an example, because the configuration of the neural network, the neural network matrix, the processing of the neural network's inputs and outputs, the signaling of information related to the embodiment, etc., are similar to those described above... Figure 14 , 15 The description is the same as in 16, so overlapping descriptions are omitted.

[0283] Figure 20 and Figure 21 This is a diagram illustrating a method for performing MIP prediction according to embodiments of the present disclosure, and Figure 22 This is a diagram used to describe a MIP-based image encoding or decoding method according to embodiments of the present disclosure. In this embodiment, reference will also be made to... Figure 20 , 21 The entirety of 22 describes a method for performing MIP predictions using only some of the reference samples from the current block.

[0284] at the same time, Figure 22 The image encoding method or image decoding method can be executed in the image encoder or image decoder, respectively, and the image encoder and image decoder can include the means described with reference to the other figures above.

[0285] First, the MIP-based image encoding / decoding method S2210 according to an embodiment of this disclosure can be executed. As an example, the method in S2210 can be executed based on some selected reference samples.

[0286] As an example, to perform MIP-based image encoding or decoding, some of the left and top reference samples can be selected. For example, some of the left and top reference samples of the current block (S2220) can be selected for use in performing MIP. In other words, in Figure 20 and Figure 21 In the embodiments described, a partial or extended left reference sample and a partial or extended upper reference sample can be selected and used. As mentioned above, the pixels of the current block may have a partial correlation with some reference samples such as the upper or left reference sample, and therefore, higher prediction accuracy can be obtained when performing prediction of the current block using partial or extended reference samples. Here, according to... Figure 21In embodiments of this disclosure, a portion or extended reference sample existing in adjacent reference samples on the left or top can be selected for MIP prediction and used as the MIP prediction input. In this case, a portion of the left reference sample and a portion of the top reference sample can be selected as the neural network input for MIP prediction. Meanwhile, as... Figure 21 As illustrated in (a), the left reference sample and the top reference sample can correspond to the samples immediately adjacent to the current block. In other words, according to... Figure 21 In the embodiment of (a), only the reference sample immediately to the left of the current block or only the reference sample immediately to the top of the current block can be used. In other words, only the reference samples adjacent to the current block can be used. However, as Figure 21 As shown in (a), the reference sample immediately adjacent to the current block is used, but blocks located on top of the left reference sample corresponding to the height of the current block are excluded, and left reference samples extending further to the bottom can be included in the reference sample. Additionally, as... Figure 21 As shown in (a), the reference sample immediately adjacent to the current block is used, but blocks to the left of the upper reference sample corresponding to the width of the current block are excluded, and upper reference samples extending further to the right can be included in the reference sample. However, as another example, as Figure 21 As shown in (b), adjacent reference samples that are to the left of the current block but whose number is less than the number of samples corresponding to the height of the current block can be included in the left reference sample, or reference samples from multiple reference sample lines can be included in the left reference sample. Additionally, as... Figure 21 As shown in (b), adjacent reference samples that are adjacent to the top of the current block but whose number is less than the number of samples corresponding to the width of the current block can be included in the upper reference sample, or reference samples from multiple reference sample lines can be included in the upper reference sample. Additionally, samples located at the top left corner of the current block can be further included in the reference sample. In other words, reference samples extending further to the left from the upper reference sample or extending further upward from the left reference sample can be further selected. Figure 21 In the embodiment of (b), multiple lines of the left, top, and / or top-left reference samples of the current block can be used. Meanwhile, as another example, as shown in (c), the left reference sample of the current block may include reference samples corresponding to the height of the current block, while the top reference sample may only include fewer adjacent samples than the number of reference samples corresponding to the width of the current block. In other words, only some of the top adjacent reference samples can be included in the top reference sample.

[0287] Meanwhile, the number of reference samples or reference sample lines that are further extended to the top or bottom are not limited to... Figure 21The number disclosed in the document. Furthermore, for the method proposed in this embodiment, some samples within one or more reference sample lines can be used for MIP prediction, and only some regions of all left or upper reference samples of the current block can be used for the MIP prediction proposed in this embodiment.

[0288] Then, the MIP mode S2230 can be selected based on the MIP mode information. The MIP mode can be applied to the current block, or it can be explicitly sent as a signal via a bitstream.

[0289] Then, a neural network-based MIP S2240 can be applied based on the selected MIP pattern. In other words, MIP-based prediction can be performed. Meanwhile, as an example, based on the selected reference sample input, prediction can be made through methods such as... Figure 14 and / or Figure 15 The neural network shown obtains predicted block sample values ​​as output. Here, the neural network can be adaptively selected based on the reference sample input. In other words, S2240 may include a process in which MIP-based prediction is applied based on the selected input sample values ​​and the selected neural network. As an example, because the configuration of the neural network, the neural network matrix, the processing of the neural network's input and output, the signaling of information associated with the embodiment, etc., are similar to those described above... Figure 14 , 15 The descriptions in 16, 17, 18, 19, 20 and 21 are the same, so overlapping descriptions are omitted.

[0290] Simultaneously, information related to the MIP according to embodiments of this disclosure can be transmitted by signals. Information that can be transmitted by signals by referring to exemplary tables is described below, but the names and order of the described information can be changed, and it can be transmitted by signals at various levels (e.g., Sequence Parameter Set (SPS), Picture Parameter Set (PPS), Slice Header, Picture Header, Encoding Unit (CU), Prediction Unit (PU), etc.).

[0291] Table 3 below shows examples of how information related to MIP is transmitted using signals according to embodiments of this disclosure.

[0292] [Table 3]

[0293]

[0294] As an example, the first syntax `intra_mip_flag` can be information used to determine whether MIP is applied. For example, when the value of the first syntax is a first value (e.g., 1), it can be determined that MIP is applied, and when the value of the first syntax is a second value (e.g., 0), it can be determined that MIP is not applied. The first syntax can be obtained from the bitstream based on another syntax (e.g., `sps_mip_enabled_flag`). For example, the other syntax `sps_mip_enabled_flag` can be information indicating whether MIP is enabled. For example, when the first syntax is not present in the bitstream, its value can be derived to a specific value (e.g., 0). As an example, the second syntax `intra_mip_above_flag` can be information used to determine whether MIP using only the above reference sample is applied. It can be obtained by referring to the above... Figure 14 , Figure 15 and 16 The information described is used to determine whether to apply a MIP. As an example, when the value of the second syntax is a first value (e.g., 1), it can be determined that a MIP using only the upper reference sample is applied, and when the value of the second syntax is a second value (e.g., 0), it can be determined that a MIP using only the upper reference sample is not applied. As an example, the second syntax can be obtained from the bitstream based on another syntax (e.g., the first syntax).

[0295] As an example, the third syntax `intra_mip_above_mode` could be information used to determine the mode of a MIP that only uses the above reference sample. This could be achieved by referencing the above... Figure 14 , 15 The information described in section 16 for determining the mode of the MIP. As an example, a third syntax can be obtained from the bitstream based on another syntax (e.g., a second syntax). For example, the third syntax can only be obtained from the bitstream if the value of the second syntax is a specific value (e.g., 1).

[0296] Furthermore, when the value of the second syntax is a specific value (e.g., 0), other syntaxes besides the third syntax can be obtained. In this case, it can be determined that the application uses a MIP with a generic left reference sample and a top reference sample, and information related to the generic MIP (e.g., intra_mip_transposed_flag and / or intra_mip_mode) can be obtained.

[0297] Meanwhile, Table 4 below is another example in which information related to MIP according to embodiments of this disclosure is transmitted by signal.

[0298] [Table 4]

[0299]

[0300] As an example, the first syntax, `intra_mip_flag`, can be information used to determine whether a MIP is applied. Since it is the same as described by referring to Table 3, overlapping descriptions are omitted. As an example, the fourth syntax, `intra_mip_left_flag`, can be information used to determine whether a MIP using only the left reference sample is applied. It can be described by referring to the above... Figures 17 to 19 The information described is used to determine whether to apply the MIP. As an example, when the value of the fourth syntax is a first value (e.g., 1), it can be determined that the MIP using only the left reference sample is applied, and when the value of the fourth syntax is a second value (e.g., 0), it can be determined that the MIP using only the left reference sample is not applied. As an example, the fourth syntax can be obtained from the bitstream based on another syntax (e.g., the first syntax).

[0301] As an example, the fifth syntax, `intra_mip_left_mode`, can be information used to determine the mode of a MIP that uses only the left reference sample. It can be used to determine the mode by referencing the above... Figures 17 to 19 Information about the pattern of the described MIP. As an example, a fifth syntax can be obtained from the bitstream based on another syntax (e.g., a fourth syntax). For instance, the fifth syntax can only be obtained from the bitstream if the value of the fourth syntax is a specific value (e.g., 1).

[0302] Furthermore, when the value of the fourth syntax is a specific value (e.g., 0), other syntaxes besides the fifth syntax can be obtained. In this case, it can be determined that the application uses a MIP with a generic left reference sample and a top reference sample, and generic MIP-related information (e.g., intra_mip_transposed_flag and / or intra_mip_mode) can be obtained.

[0303] Meanwhile, Table 5 below is another example of sending MIP-related information according to embodiments of this disclosure using signals.

[0304] [Table 5]

[0305]

[0306] As an example, the first syntax, `intra_mip_flag`, can be information used to determine whether a MIP is applied. Since it is the same as described in Table 3, overlapping descriptions are omitted. As an example, the sixth syntax, `intra_mip_extended_flag`, can be information used to determine whether a MIP using only the upper reference sample or the left reference sample is applied. It can be obtained by referring to the above... Figures 14 to 22The information described is used to determine whether to apply a MIP. As an example, when the value of the sixth syntax is a first value (e.g., 1), it can be determined that a MIP using only the top or left reference sample is applied, and when the value of the sixth syntax is a second value (e.g., 0), it can be determined that a MIP using only the top or left reference sample is not applied. As an example, the sixth syntax can be obtained from the bitstream based on another syntax (e.g., the first syntax).

[0307] As an example, the second syntax `intra_mip_above_flag` can be obtained based on the value of the sixth syntax. As an example, the second syntax can be obtained when the value of the sixth syntax is a specific value (e.g., 1). The second syntax can be information used, as described above, to determine whether to apply a MIP that uses only the upper reference sample. However, as an example, when the value of the second syntax in Table 5 is a first value (e.g., 1), it can be determined that a MIP using only the upper reference sample is applied, while when it is a second value (e.g., 0), it can be determined that a MIP using only the left reference sample is applied.

[0308] Then, based on the value of the second grammar, either the third or fifth grammar can be obtained. As an example, when the value of the second grammar is the first value (e.g., 1), the application only uses the MIP of the upper reference sample, thus obtaining the third grammar `intra_mip_above_mode`. The description of the third grammar is the same as that described by referring to Table 3 above, therefore overlapping descriptions are omitted. As another example, when the value of the second grammar is the second value (e.g., 2), the application only uses the MIP of the left reference sample, thus obtaining the fifth grammar `intra_mip_left_mode`. The description of the fifth grammar is the same as that described by referring to Table 4 above, therefore overlapping descriptions are omitted.

[0309] Furthermore, when the value of the sixth syntax is a specific value (e.g., 0), other syntaxes besides the second, third, or fifth syntax can be obtained. In this case, it can be determined that the application uses a MIP with a generic left reference sample and a top reference sample, and generic MIP-related information (e.g., intra_mip_transposed_flag and / or intra_mip_mode) can be obtained.

[0310] Meanwhile, in Tables 3 to 5 above, information indicating whether to apply a MIP using only the upper reference sample and information indicating whether to apply a MIP using only the left reference sample are respectively transmitted by signals, but this is only for clarity of description. Therefore, according to another embodiment of this disclosure, a message may indicate whether to apply a MIP using only the upper reference sample, whether to apply a MIP using only the left reference sample, or whether to apply a MIP using only a portion of the left and upper reference samples, and obviously, other embodiments are also included in this disclosure.

[0311] Figure 23 This is a diagram illustrating an image decoding method performed by an image decoding apparatus (decoder) according to an embodiment of the present disclosure.

[0312] For example, the image decoding device can determine whether a matrix-based intra-prediction (MIP) mode is applied to the current block S2310. As an example, determining the applied MIP mode in the corresponding step may include referencing... Figures 14 to 22 The MIP described. As an example, it can be determined whether to apply the MIP mode based on the MIP application information described above, which can be obtained from the bitstream. Meanwhile, step S2310 can further include determining whether to apply the MIP mode by using only the left reference sample or the top reference sample, or by applying the MIP mode by using only a portion of the left reference sample and the top reference sample.

[0313] As an example, based on the MIP pattern being determined to be applied to the current block, a prediction block S2320 for the current block can be generated. Based on the MIP pattern, the prediction block can be generated by using only a portion of the left or upper reference sample of the current block. As an example, when generating a prediction block based on the MIP pattern by using only the upper reference sample of the current block, it can be generated based on at least one upper reference sample line. Additionally, as an example, when generating a prediction block based on the MIP pattern by using only the upper reference sample of the current block, the upper reference sample can further include the upper-right reference sample of the current block. Also, as another example, when generating a prediction block based on the MIP pattern by using only the left reference sample of the current block, it can be generated based on at least one left reference sample line. Additionally, as another example, when generating a prediction block based on the MIP pattern by using only the left reference sample of the current block, the left reference sample can include the lower-left reference sample of the current block. Also, as another example, when generating a prediction block using a portion of both the left and upper reference samples of the current block, at least one of the left or upper reference samples can be partially adjacent to the current block. Furthermore, as another example, a predicted block is generated using a portion of the left and upper reference samples of the current block. However, the left reference sample may further include the lower-left reference sample of the current block, and the upper reference sample may further include the upper-right reference sample of the current block. Also, as another example, based on the application of the MIP mode, it can be determined whether to execute the MIP mode based on the left or upper reference sample. In other words, when it is determined that the MIP mode should be applied, it can be determined how to execute the MIP mode subsequently. For example, it can be determined whether to apply the MIP mode using information obtained from the bitstream, and it can be determined whether to execute the MIP mode based on the left or upper reference sample based on the information obtained from the bitstream.

[0314] At the same time, because Figure 23 The embodiments correspond to those of this disclosure, but some steps may be changed, or the order of some steps may be changed, or some steps may be deleted or performed simultaneously, and obviously, these are also included in the embodiments of this disclosure.

[0315] Figure 24 This diagram illustrates an image encoding method that can be performed by an image encoding apparatus according to an embodiment of the present disclosure.

[0316] As an example, the image coding device can determine whether to apply a matrix-based intra-frame prediction (MIP) mode to the current block S2410. For example, determining the MIP mode to be applied in the corresponding step may include referencing... Figures 14 to 22The MIP described. As an example, whether a MIP mode is applied can be encoded in the bitstream as the MIP application information described above. Meanwhile, step S2410 may further include determining whether to apply the MIP mode by using only the left reference sample or the top reference sample, or by applying the MIP mode by using only a portion of the left reference sample and the top reference sample.

[0317] As an example, based on the MIP pattern being determined to be applied to the current block, a prediction block S2420 for the current block can be generated. The prediction block can be generated based on the MIP pattern by using only a portion of the left or upper reference sample of the current block. As an example, when generating a prediction block based on the MIP pattern by using only the upper reference sample of the current block, it can be generated based on at least one upper reference sample line. Additionally, as an example, when generating a prediction block based on the MIP pattern by using only the upper reference sample of the current block, the upper reference sample can further include the upper-right reference sample of the current block. Meanwhile, as another example, when generating a prediction block based on the MIP pattern by using only the left reference sample of the current block, it can be generated based on at least one left reference sample line. Additionally, as an example, when generating a prediction block based on the MIP pattern by using only the left reference sample of the current block, the left reference sample can include the lower-left reference sample of the current block. Meanwhile, as another example, when generating a prediction block using a portion of both the left and upper reference samples of the current block, at least one of the left or upper reference samples can be partially adjacent to the current block. Furthermore, as an example, a predicted block is generated using a portion of the left and upper reference samples of the current block. However, the left reference sample may further include the lower-left reference sample of the current block, and the upper reference sample may further include the upper-right reference sample of the current block. Also, as an example, based on the application of the MIP mode, it can be determined whether to execute the MIP mode based on the left or upper reference sample. In other words, when it is determined that the MIP mode is to be applied, it can be determined how the MIP mode will be executed subsequently. As an example, whether the MIP mode is applied can be encoded as a single piece of information in the bitstream, and whether the MIP mode is executed based on the left or upper reference sample can also be encoded as a single piece of information in the bitstream.

[0318] Meanwhile, as an example, the bitstream generated by the image encoding method can be stored in a non-transitory computer-readable medium.

[0319] Additionally, as an example, a method for sending a bitstream generated by an image encoding method can be proposed.

[0320] At the same time, because Figure 24Corresponding to the embodiments of this disclosure, some steps may be changed, or the order of some steps may be changed, or some steps may be deleted or performed simultaneously, and obviously, these are also included in the embodiments of this disclosure.

[0321] For clarity, the exemplary method of this disclosure is described as a series of operations; however, this is not intended to limit the order in which the steps are performed, and each step may be performed simultaneously or in a different order when necessary. In order to implement the method according to this disclosure, additional steps may be included in addition to the illustrated steps, and some steps may be omitted while including the remaining steps, or some steps may be omitted while including the additional steps.

[0322] In this disclosure, the image encoding device or image decoding device that performs a predetermined operation (step) can perform the operation (step) to check the execution conditions or circumstances of the corresponding operation (step). For example, when describing the execution of a predetermined operation when predetermined conditions are met, the image encoding device or image decoding device can perform an operation to check whether the predetermined conditions are met before performing the predetermined operation.

[0323] The various embodiments disclosed herein are not a list of all possible combinations, but are provided to illustrate representative aspects of the disclosure, and the elements described in the various embodiments may be applied independently or in combination with two or more elements.

[0324] Furthermore, the various embodiments of this disclosure can be implemented using hardware, firmware, software, or a combination thereof. When implemented in hardware, the various embodiments can be implemented using one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers, or microprocessors.

[0325] Furthermore, the image decoding apparatus and image encoding apparatus applying the embodiments of this disclosure can be included in various devices, such as multimedia broadcasting transmitting / receiving devices, mobile communication terminals, home theater video devices, digital cinema video devices, surveillance cameras, video conferencing equipment, real-time communication devices such as video communication devices, mobile streaming media devices, storage media, cameras, video-on-demand (VoD) service providers, over-the-top (OTT) devices, internet streaming service providers, three-dimensional (3D) video devices, video telephony devices, and medical video devices, and can be used to process video signals or data signals. For example, over-the-top (OTT) devices can include game consoles, Blu-ray players, internet-connected televisions, home theater systems, smartphones, tablets, and digital video recorders (DVRs), etc.

[0326] Figure 21 An exemplary diagram of a content streaming system to which embodiments of the present disclosure may be applied is shown.

[0327] like Figure 21 As shown, the content streaming system applying the embodiments of this disclosure can broadly include encoding servers, streaming servers, web servers, media storage, user equipment, and multimedia input devices.

[0328] An encoding server compresses content input from multimedia input devices such as smartphones, cameras, or camcorders into digital data, generates a bitstream, and sends it to a streaming server. As another example, when multimedia input devices such as smartphones, cameras, or camcorders directly generate bitstreams, an encoding server can be omitted.

[0329] The bitstream can be generated by applying the video encoding method and / or image encoding apparatus of the embodiments of this disclosure, and the streaming server can temporarily store the bitstream during the process of sending or receiving the bitstream.

[0330] A streaming server can send multimedia data to a user's device based on a user request via a web server, and the web server can act as a medium to notify the user of available services. When a user requests a service from the web server, the web server can send the request to the streaming server, and the streaming server can transmit the multimedia data to the user. In this scenario, the content streaming system may include a separate control server, which in turn controls the command / response exchanges between devices within the content streaming system.

[0331] A streaming server can receive content from media storage and / or encoding servers. For example, when receiving content from an encoding server, the content can be received in real time. In this case, to provide a seamless streaming service, the streaming server can store a bitstream for a certain period of time.

[0332] Examples of user devices may include mobile phones, smartphones, laptops, digital broadcasting terminals, personal digital assistants (PDAs), portable multimedia players (PMPs), navigation devices, tablet PCs, tablet computers, ultrabooks, wearable devices (i.e., smartwatches, smart glass, head-mounted displays (HMDs)), digital televisions, desktop computers, and digital signage.

[0333] In a content streaming system, each server can operate as a distributed server, in which case the data received by each server can be processed in a distributed manner.

[0334] The scope of this disclosure includes software or machine-executable instructions (i.e., operating systems, applications, firmware, programs, etc.) that enable the operation of methods according to various embodiments to be performed on a device or computer, and non-transitory computer-readable media in which such software or instructions are stored and can be executed on a device or computer.

[0335] [Industrial Applicability]

[0336] The embodiments disclosed herein can be used to encode / decode images.

Claims

1. An image decoding method performed by an image decoding device, comprising: Determine the intra-prediction mode for the current block; as well as Based on a defined intra-prediction mode, a prediction block for the current block is generated by performing intra-prediction. Specifically, the intra-frame prediction mode is determined based on range constraint information indicating whether the intra-frame prediction mode is restricted to a specific range, and Specifically, based on the range limitation information indicating that the intra-prediction mode is limited to the specific range, and based on the intra-prediction mode determined to be one of the intra-prediction modes within the specific range, the intra-prediction is performed by primarily using reference samples within a predetermined range of the available reference samples of the current block.

2. The method according to claim 1, wherein, Based on the fact that the reference samples within the predetermined range are primarily used, the intra-frame prediction mode is determined to be a directional mode within the specific range.

3. The method according to claim 2, wherein, The specific range is determined based on whether the intra-frame prediction is based on wide-angle intra-frame prediction.

4. The method according to claim 2, wherein, The specific range is any one of the orientation pattern ranges of 50 to 66, 35 to 66, or 19 to 34.

5. The method according to claim 2, wherein, The specific range is any one of the orientation pattern ranges 2 to 18, 2 to 33, or 18 to 49.

6. The method according to claim 1, wherein, The prediction block is generated by a weighted sum of a first prediction block obtained primarily using reference samples within a first range and a second prediction block obtained primarily using reference samples within a second range.

7. The method according to claim 6, wherein, The first range and the second range do not overlap.

8. The method according to claim 6, wherein, The first weight to be applied to the first prediction block is determined differently from the weight to be applied to the second prediction block.

9. The method according to claim 8, wherein, The first weight is determined based on the size of the current block.

10. The method according to claim 8, wherein, The first weight and the second weight are predefined values.

11. An image encoding method performed by an image encoding device, comprising: Determine the intra-prediction mode for the current block; as well as Based on the determined intra-frame prediction mode, the prediction mode information of the current block is encoded; The intra-frame prediction mode is determined based on whether the intra-frame prediction mode is restricted to a specific range, and Wherein, based on the intra prediction mode being restricted to the specific range, the intra prediction of the current block is performed based on the intra prediction mode determined to be one of the intra prediction modes within the specific range, by primarily using reference samples within a predetermined range of the available reference samples of the current block.

12. A computer-readable medium for storing a bitstream generated by the image encoding method of claim 11.

13. A method for transmitting a bitstream generated by an image encoding method, the image encoding method comprising: Determine the intra-prediction mode for the current block; as well as Based on the determined intra-frame prediction mode, the prediction mode information of the current block is encoded. The intra-frame prediction mode is determined based on whether the intra-frame prediction mode is restricted to a specific range, and Wherein, based on the intra prediction mode being restricted to the specific range, the intra prediction of the current block is performed based on the intra prediction mode determined to be one of the intra prediction modes within the specific range, by primarily using reference samples within a predetermined range of the available reference samples of the current block.

Images