Method and apparatus for matrix-based intra prediction based on various reference pixels

Abstract

Disclosed are methods and apparatuses for matrix-based intra prediction based on various reference pixels. According to the disclosed methods, coding efficiency and prediction accuracy can be improved by utilizing various reference pixels for MIP prediction.

Description

Technical Field

[0001] This invention relates to a video encoding / decoding method, apparatus, and recording medium for storing bitstreams, and more specifically, to a matrix-based intra-frame prediction method and apparatus based on various reference pixels. Background Technology

[0002] The statements in this section are provided only as background information in relation to the present invention and do not necessarily constitute prior art.

[0003] Compared to audio or still image data, video data is much larger in volume, and unprocessed or uncompressed video data requires significant hardware resources (including memory) to store or transmit it in its raw form.

[0004] Accordingly, when storing or transmitting video data, an encoder is typically used to compress the video data for storage or transmission, and a decoder receives the compressed video data and performs decompression and reconstruction. Such video compression technologies include H.264 / AVC, High Efficiency Video Coding (HEVC), and Versatile Video Coding (VVC), which offers approximately 30% or more improvement in coding efficiency compared to HEVC.

[0005] However, as video size, resolution, and frame rate continue to increase, and the amount of data that needs to be encoded and decoded also increases accordingly, a new compression technology is needed that is more efficient in terms of encoding and decoding and more effective in improving video quality than existing compression technologies.

[0006] In particular, the demand for high-resolution, high-quality video (e.g., High Definition (HD) and Ultra High Definition (UHD) video) has been increasing in various fields recently. As the resolution and quality of image data become higher, the amount of information or bits transmitted also increases compared to existing image data. This increase in the amount of information or bits transmitted leads to increased transmission and storage costs. Furthermore, there is a growing interest in and demand for immersive media such as Virtual Reality (VR), Artificial Reality (AR) content, and holograms, and the broadcasting of videos / images with characteristics different from real-world images (e.g., game footage) is increasing. Accordingly, there is a need for an efficient video compression technology to efficiently compress, transmit, store, and play information from high-resolution, high-quality video with all the aforementioned characteristics. Summary of the Invention

[0007] Technical issues This invention provides a method and apparatus for video encoding or decoding, for effectively encoding or decoding video and improving the objective and subjective quality of reconstructed video, as well as a recording medium for storing the bitstream generated by the video encoding method / apparatus.

[0008] Technical solution At least one aspect of the present invention provides a video decoding method, comprising: determining a reference template including pixels to be decoded, and determining a weight matrix and an offset for matrix-based intra-frame prediction of the current block. The video decoding method further comprises: generating a prediction block for the current block based on the weight matrix, the offset, and the reference template of the current block.

[0009] Another aspect of the present invention provides a video coding method, comprising: determining a reference template including encoded pixels, and determining a weight matrix and an offset for matrix-based intra-frame prediction of the current block. The video coding method further comprises: generating a prediction block for the current block based on the weight matrix, the offset, and the reference template of the current block.

[0010] Another aspect of the present invention provides a non-volatile computer-readable recording medium storing a bitstream generated by a video coding apparatus. The bitstream is generated by a video coding method. The video coding method includes: determining a reference template comprising encoded pixels, and determining a weight matrix and an offset for matrix-based intra-frame prediction of the current block. The video coding method further includes: generating a prediction block for the current block based on the weight matrix, the offset, and the reference template of the current block.

[0011] Another aspect of the present invention provides a method executed by at least one processor. The method includes: generating a bitstream of video, and transmitting data including the bitstream. Generating the bitstream includes: determining a reference template including encoded pixels, and determining a weight matrix and an offset for matrix-based intra-frame prediction of the current block. Generating the bitstream further includes: generating a prediction block for the current block based on the weight matrix, offset, and reference template of the current block.

[0012] Beneficial effects According to embodiments of the present invention, the video encoding / decoding apparatus can efficiently encode / decode video by diversifying the reference pixels used for MIP prediction, and can improve the objective / subjective quality of the reconstructed video. Attached Figure Description

[0013] Figure 1 This is a block diagram of a video encoding device that can implement the technology of the present invention.

[0014] Figure 2This paper demonstrates a method for segmenting blocks using a quadtree plus binary tree ternary tree (QTBTTT) structure.

[0015] Figure 3a and Figure 3b Several intra-prediction modes, including a wide-angle intra-prediction mode, are shown.

[0016] Figure 4 The adjacent blocks of the current block are shown.

[0017] Figure 5 This is a block diagram of a video decoding device that can implement the technology of the present invention.

[0018] Figure 6 This is a flowchart of a matrix-based intra-frame prediction method according to an embodiment of the present invention.

[0019] Figure 7a , Figure 7b and Figure 7c Each shows a reference template adjacent to the current block according to an embodiment of the invention.

[0020] Figure 8 A reference template that is not adjacent to the current block is shown according to an embodiment of the present invention.

[0021] Figure 9 A reference template indicating the motion information of adjacent blocks according to an embodiment of the present invention is shown.

[0022] Figure 10 This is a flowchart of a video encoding method according to an embodiment of the present invention.

[0023] Figure 11 This is a flowchart of a video decoding method according to an embodiment of the present invention. Detailed Implementation

[0024] In the following description, some embodiments of the invention are described in detail with reference to the accompanying illustrative drawings. In the following description, the same reference numerals denote the same elements, although these elements are shown in different drawings. Furthermore, in the following description of some embodiments, detailed descriptions of relevant known components and functions may be omitted for clarity and brevity when it is considered that such detailed descriptions obscure the subject matter of the invention.

[0025] Figure 1 This is a block diagram of a video encoding apparatus that can implement the technology of the present invention. In the following, reference is made to... Figure 1 The diagram illustrates a video encoding device and its components.

[0026] The encoding device may include: an image segmenter 110, a predictor 120, a subtractor 130, a transformer 140, a quantizer 145, a reordering unit 150, an entropy encoder 155, an inverse quantizer 160, an inverse transformer 165, an adder 170, a loop filter unit 180, and a memory 190.

[0027] Each component of the encoding device can be implemented as hardware or software, or a combination of hardware and software. Furthermore, the function of each component can be implemented as software, and the microprocessor can be implemented to execute the software function corresponding to each component.

[0028] A video consists of one or more sequences of images. Each image is segmented into multiple regions, and encoding is performed on each region. For example, an image is segmented into one or more tiles and / or slices. Here, one or more tiles can be defined as a tile group. Each tile and / or slice is segmented into one or more coding tree units (CTUs). Additionally, each CTU is segmented into one or more coding units (CUs) in a tree structure. Information applied to each coding unit (CU) is encoded as the syntax of the CU, and information commonly applied to CUs included in a CTU is encoded as the syntax of the CTU. Furthermore, information commonly applied to all blocks in a slice is encoded as the syntax of the slice header, and information applied to all blocks constituting one or more images is encoded as a picture parameter set (PPS) or picture header. Additionally, information commonly referenced by multiple images is encoded as a sequence parameter set (SPS). Furthermore, information commonly referenced by one or more SPSs is encoded as a video parameter set (VPS). Furthermore, information commonly applied to a tile or tile group can also be encoded as the syntax of the tile or tile group header. The syntax included in SPS, PPS, slice headers, and tile or tile group headers can be called high-level syntax.

[0029] Image splitter 110 determines the size of the coding tree unit (CTU). Information about the size of the CTU (CTU size) is encoded into SPS or PPS syntax and transmitted to the video decoding device.

[0030] Image segmenter 110 divides each image constituting the video into multiple coding tree units (CTUs) of a predetermined size, and then recursively segments the CTUs using a tree structure. The leaf nodes in the tree structure are called coding units (CUs), which are the basic units of encoding.

[0031] A tree structure can be a quadtree (QT) that divides a higher node (or parent node) into four lower nodes (or child nodes) of the same size. A tree structure can also be a binary tree (BT) that divides a higher node into two lower nodes. A tree structure can also be a ternary tree (TT) that divides a higher node into three lower nodes in a 1:2:1 ratio. A tree structure can also be a combination of two or more of the QT, BT, and TT structures. For example, a quadtree plus binary tree (QTBT) structure can be used, or a quadtree plus binary tree ternary tree (QTBTTT) structure can be used. Here, a binary tree ternary tree (BTTT) is added to the tree structure to form a multiple-type tree (MTT).

[0032] Figure 2 This is a schematic diagram illustrating a method for segmenting blocks using a QTBTT structure.

[0033] like Figure 2 As shown, the CTU can first be segmented into a QT structure. Quadtree segmentation can be recursive until the size of the segmented blocks reaches the minimum block size (MinQTSize) allowed for leaf nodes in the QT. The entropy encoder 155 encodes a first flag (QT_split_flag) indicating whether each node of the QT structure is segmented into the four nodes below it and signals this flag to the video decoding device. When the leaf node of the QT is not larger than the maximum block size (MaxBTSize) allowed for the root node in the BT, the leaf node can be further segmented using at least one of the BT or TT structures. Multiple segmentation directions can exist in the BT and / or TT structures. For example, two directions can exist: a horizontal direction for segmenting the blocks of the corresponding node and a vertical direction for segmenting the blocks of the corresponding node. Figure 2 As shown, when MTT splitting begins, the entropy encoder 155 encodes a second flag (mtt_split_flag) indicating whether a node has been split, as well as a flag indicating the splitting direction (vertical or horizontal) and / or the splitting type (binary or ternary) if the node has been split, and signals this to the video decoding device.

[0034] Alternatively, before encoding the first flag (QT_split_flag) indicating whether each node has been split into the four lower-level nodes, the CU splitting flag (split_cu_flag) indicating whether a node has been split can also be encoded. When the value of the CU splitting flag (split_cu_flag) indicates that each node has not been split, the block of the corresponding node becomes a leaf node in the split tree structure and becomes a CU, which is the basic unit of encoding. When the value of the CU splitting flag (split_cu_flag) indicates that each node has been split, the video encoding device begins encoding the first flag using the above scheme.

[0035] When QTBT is used as another example of a tree structure, two types can exist: a type where the corresponding node's block is horizontally divided into two blocks of the same size (i.e., symmetrical horizontal division) and a type where the corresponding node's block is vertically divided into two blocks of the same size (i.e., symmetrical vertical division). The entropy encoder 155 encodes a split flag (split_flag) indicating whether each node of the BT structure is divided into blocks of the lower layer and split type information indicating the split type, and transmits this information to the video decoding device. Alternatively, a type can exist where the corresponding node's block is divided into two blocks that are asymmetrically divided. Asymmetrical forms can include dividing the corresponding node's block into two rectangular blocks with a size ratio of 1:3, or it can also include dividing the corresponding node's block along a diagonal direction.

[0036] The CU can have various sizes depending on the QTBT or QTBTTT segmentation from the CTU. In the following text, the block corresponding to the CU to be encoded or decoded (i.e., the leaf node of the QTBTTT) is called the "current block". When using QTBTTT segmentation, the current block can also be rectangular in shape in addition to a square shape.

[0037] Predictor 120 predicts the current block to generate a prediction block. Predictor 120 includes an intra-frame predictor 122 and an inter-frame predictor 124.

[0038] Typically, each block of the current image can be predictively encoded. This prediction can usually be performed using intra-frame prediction (which utilizes data from the image containing the current block) or inter-frame prediction (which utilizes data from an image encoded before the image containing the current block). Inter-frame prediction includes both one-way and two-way prediction.

[0039] Intra-predictor 122 predicts pixels in the current block by utilizing pixels (reference pixels) that are adjacent to the current block in the current image, including the current block. Depending on the prediction direction, there are various intra-prediction modes. For example, such as... Figure 3aAs shown, various intra-frame prediction modes can include two non-directional modes: Planar mode and DC mode, and can include 65 directional modes. The neighboring pixels and algorithm equations to be used are defined differently for each prediction mode.

[0040] To perform efficient orientation prediction for the current block with a rectangular shape, additional methods such as... can be used. Figure 3b The dashed arrows indicate the directional modes (intra-prediction modes #67 to #80, #-1 to #-14). These directional modes can be referred to as "wide-angle intra-prediction modes." Figure 3b In the diagram, the arrows indicate the corresponding reference samples used for prediction, not the prediction direction. The prediction direction is opposite to the direction indicated by the arrow. When the current block has a rectangular shape, the wide-angle intra-frame prediction mode is a mode that performs prediction in the opposite direction to the specific direction mode without additional bit transmission. In this case, in the wide-angle intra-frame prediction mode, some wide-angle intra-frame prediction modes available for the current block can be determined by the ratio of the width to the height of the current block with a rectangular shape. For example, when the current block has a rectangular shape with a height less than its width, wide-angle intra-frame prediction modes with angles less than 45 degrees (intra-frame prediction modes #67 to #80) are available. When the current block has a rectangular shape with a width greater than its height, wide-angle intra-frame prediction modes with angles greater than -135 degrees (intra-frame prediction modes #-1 to #-14) are available.

[0041] Intra-predictor 122 can determine the intra-prediction to be used for encoding the current block. In some examples, intra-predictor 122 can encode the current block by utilizing multiple intra-prediction modes, and can also select the appropriate intra-prediction mode to use from test modes. For example, intra-predictor 122 can calculate rate-distortion values ​​by utilizing rate-distortion analysis of intra-prediction modes across multiple tests, and can also select the intra-prediction mode with the best rate-distortion characteristics from the test modes.

[0042] Intra-predictor 122 selects one intra-prediction mode from a variety of intra-prediction modes and predicts the current block by using neighboring pixels (reference pixels) determined according to the selected intra-prediction mode and an algorithm equation. Entropy encoder 155 encodes the information about the selected intra-prediction mode and transmits it to the video decoding device.

[0043] Inter-frame predictor 124 generates a predicted block for the current block by utilizing motion compensation processing. Inter-frame predictor 124 searches for the most similar block to the current block in a reference image that was encoded and decoded earlier than the current image, and generates the predicted block for the current block using the searched block. Additionally, a motion vector (MV) is generated, corresponding to the displacement between the current block in the current image and the predicted block in the reference image. Typically, motion estimation is performed on the luma component, and the motion vector calculated based on the luma component is used for both the luma and chroma components. The motion information, including information about the reference image and information about the motion vector used to predict the current block, is encoded by entropy encoder 155 and transmitted to the video decoding device.

[0044] The inter-frame predictor 124 can also perform interpolation on a reference image or reference block to increase prediction accuracy. In other words, subsamples are interpolated between two consecutive integer samples by applying filter coefficients to multiple consecutive integer samples, including two integer samples. When performing a process to search for the most similar block to the current block on the interpolated reference image, the motion vector can be represented with fractional-unit precision instead of integer-sample-unit precision. For each target region to be encoded, such as a cell like a slice, tile, CTU, or CU, the precision or resolution of the motion vector can be set differently. When this Adaptive Motion Vector Resolution (AMVR) is applied, information about the motion vector resolution to be applied to each target region should be signaled for each target region. For example, when the target region is a CU, information about the motion vector resolution applied to each CU is signaled. The information about the motion vector resolution can be information representing the precision of the motion vector difference to be described below.

[0045] On the other hand, the inter-frame predictor 124 can perform inter-frame prediction using bidirectional prediction. In the case of bidirectional prediction, two reference images and two motion vectors representing the location of the block most similar to the current block in each reference image are used. The inter-frame predictor 124 selects a first reference image and a second reference image from reference image list 0 (RefPicList0) and reference image list 1 (RefPicList1), respectively. The inter-frame predictor 124 also searches for the block most similar to the current block in each reference image to generate a first reference block and a second reference block. In addition, the predicted block of the current block is generated by averaging or weighted averaging the first reference block and the second reference block. In addition, motion information including information about the two reference images used to predict the current block and information about the two motion vectors is transmitted to the entropy encoder 155. Here, reference image list 0 may consist of images in pre-reconstructed images that are displayed before the current image in display order, and reference image list 1 may consist of images in pre-reconstructed images that are displayed after the current image in display order. However, although not particularly limited to this, pre-reconstructed images that are displayed after the current image in display order may be additionally included in reference image list 0. Conversely, pre-reconstructed images preceding the current image can also be additionally included in the reference image list 1.

[0046] Various methods can be used to minimize the number of bits consumed in encoding motion information.

[0047] For example, when the reference image and motion vector of the current block are the same as those of the adjacent blocks, the information that can identify the adjacent blocks is encoded to transmit the motion information of the current block to the video decoding device. This method is called merge mode.

[0048] In merge mode, the inter-frame predictor 124 selects a predetermined number of merge candidate blocks (hereinafter referred to as "merge candidates") from the neighboring blocks of the current block.

[0049] As adjacent blocks used to derive merge candidates, all or some of the following can be used: left block A0, lower left block A1, upper block B0, upper right block B1, and upper left block B2, which are adjacent to the current block in the current image. Figure 4 As shown. In addition to the current image containing the current block, blocks located within a reference image (which may be the same as or different from the reference image used to predict the current block) can also be used as merge candidates. For example, a co-located block of the current block within the reference image, or a block adjacent to that co-located block, can be additionally used as a merge candidate. If the number of merge candidates selected using the above method is less than a preset number, a zero vector is added to the merge candidates.

[0050] The inter-frame predictor 124 configures a merge list including a predetermined number of merge candidates by utilizing neighboring blocks. It selects a merge candidate from the merge candidates included in the merge list to be used as motion information for the current block, and generates merge index information for identifying the selected candidate. The generated merge index information is encoded by the entropy encoder 155 and transmitted to the video decoding device.

[0051] The merge skip mode is a special case of the merge mode. After quantization, when all transform coefficients used for entropy coding are close to zero, only the adjacent block selection information is transmitted, and the residual signal is not transmitted. By utilizing the merge skip mode, relatively high coding efficiency can be achieved for images with slight motion, still images, and screen content images.

[0052] From then on, merge mode and merge skip mode were collectively referred to as merge / skip mode.

[0053] Another approach for encoding motion information is the Advanced Motion Vector Prediction (AMVP) model.

[0054] In AMVP mode, the inter-frame predictor 124 derives motion vector prediction candidates for the current block by utilizing neighboring blocks of the current block. As neighboring blocks used to derive motion vector prediction candidates, they can use... Figure 4 The current block in the current image shown includes all or some of the adjacent blocks: left block A0, lower left block A1, upper block B0, upper right block B1, and upper left block B2. Furthermore, blocks located in a reference image (which may be the same as or different from the reference image used to predict the current block) can also be used as neighboring blocks for deriving motion vector prediction candidates, in addition to the current image containing the current block. For example, a co-located block of the current block in the reference image or a block adjacent to that co-located block can be used. If the number of motion vector candidates selected by the above method is less than a preset number, a zero vector is added to the motion vector candidates.

[0055] The inter-frame predictor 124 derives motion vector prediction candidates by utilizing motion vectors from neighboring blocks, and determines the motion vector prediction for the current block by utilizing these candidates. Furthermore, the motion vector difference is calculated by subtracting the motion vector prediction from the motion vector prediction for the current block.

[0056] Motion vector predictions can be obtained by applying predefined functions (e.g., median and average calculations) to motion vector prediction candidates. In this case, the video decoding device also knows the predefined functions. Furthermore, since the neighboring blocks used to derive the motion vector prediction candidates are blocks that have already been encoded and decoded, the video decoding device may also already know the motion vectors of the neighboring blocks. Therefore, the video encoding device does not need to encode the information used to identify the motion vector prediction candidates. Accordingly, in this case, information about the motion vector difference and information about the reference image used to predict the current block are encoded.

[0057] Alternatively, motion vector prediction can be determined by selecting any of the motion vector prediction candidates. In this case, additional encoding is performed on the information used to identify the selected motion vector prediction candidate, as well as information about the motion vector difference and information about the reference image used to predict the current block.

[0058] Subtractor 130 generates a residual block by subtracting the predicted block generated by intra-predictor 122 or inter-predictor 124 from the current block.

[0059] Transformer 140 transforms the residual signal in the residual block, which has pixel values ​​in the spatial domain, into transform coefficients in the frequency domain. Transformer 140 can transform the residual signal in the residual block by using the entire size of the residual block as a transform unit, or it can divide the residual block into multiple sub-blocks and perform the transformation by using the sub-blocks as transform units. Alternatively, the residual block is divided into two sub-blocks (i.e., a transformed region and a non-transformed region) to transform the residual signal by using only the transformed region sub-block as a transform unit. Here, the transformed region sub-block can be one of two rectangular blocks with a size ratio of 1:1 based on a horizontal axis (or a vertical axis). In this case, the entropy encoder 155 encodes a flag (cu_sbt_flag) indicating that only the transformed sub-block is used, as well as orientation (vertical / horizontal) information (cu_sbt_horizontal_flag) and / or position information (cu_sbt_pos_flag), and signals this information to the video decoding device. Furthermore, the size of the transformed region sub-blocks can be based on a 1:3 size ratio along the horizontal (or vertical) axis. In this case, the entropy encoder 155 additionally encodes the flag (cu_sbt_quad_flag) for dividing the corresponding segment and signals it to the video decoding device.

[0060] On the other hand, the transformer 140 can perform transformations on the residual blocks individually in the horizontal and vertical directions. Various types of transformation functions or transformation matrices can be used for this transformation. For example, a pair of transformation functions for horizontal and vertical transformations can be defined as a Multiple Transform Set (MTS). The transformer 140 can select the transformation function pair with the highest transformation efficiency in the MTS and can transform the residual block in each of the horizontal and vertical directions. The entropy encoder 155 encodes the information (mts_idx) about the transformation function pair in the MTS and signals it to the video decoding device.

[0061] Quantizer 145 quantizes the transform coefficients output from transformer 140 using quantization parameters and outputs the quantized transform coefficients to entropy encoder 155. Quantizer 145 can also quantize the relevant residual blocks immediately without transforming any block or frame. Quantizer 145 can also apply different quantization coefficients (scaling values) based on the position of the transform coefficients in the transform block. The quantization matrix applied to the quantized transform coefficients arranged in two dimensions can be encoded and signaled to the video decoding device.

[0062] The reordering unit 150 can perform a rearrangement of the coefficient values ​​on the quantized residual values.

[0063] The reordering unit 150 can transform a 2D coefficient array into a 1D coefficient sequence using coefficient scanning. For example, the reordering unit 150 can output a 1D coefficient sequence by scanning the DC coefficients to high-frequency region coefficients using a zig-zag scan or a diagonal scan. Depending on the size of the transform unit and the intra-frame prediction mode, a vertical scan scanning the 2D coefficient array in the column direction and a horizontal scan scanning the 2D block-type coefficients in the row direction can also be used instead of a zig-zag scan. In other words, the scanning method to be used can be determined from zig-zag scan, diagonal scan, vertical scan, and horizontal scan, depending on the size of the transform unit and the intra-frame prediction mode.

[0064] The entropy encoder 155 encodes the 1D quantized transform coefficient sequence output from the reordering unit 150 by utilizing various encoding schemes, including Context-based Adaptive Binary Arithmetic Code (CABAC) and Exponential Golomb, to generate a bitstream.

[0065] Furthermore, the entropy encoder 155 encodes block-segmentation related information (e.g., CTU size, CTU segmentation flag, QT segmentation flag, MTT segmentation type, MTT segmentation direction, etc.) so that the video decoding device can segment blocks equivalently to the video encoding device. Additionally, the entropy encoder 155 encodes information indicating whether the current block is encoded using intra-frame prediction or inter-frame prediction. The entropy encoder 155 encodes intra-frame prediction information (i.e., information about the intra-frame prediction mode) or inter-frame prediction information (the merge index in the case of merge mode, and information about the reference image index and motion vector difference in the case of AMVP mode) according to the prediction type. Furthermore, the entropy encoder 155 encodes quantization-related information (i.e., information about quantization parameters and information about the quantization matrix).

[0066] Inverse quantizer 160 inverse quantizes the quantized transform coefficients output from quantizer 145 to generate transform coefficients. Inverse transformer 165 transforms the transform coefficients output from inverse quantizer 160 from the frequency domain to the spatial domain to reconstruct the residual block.

[0067] Adder 170 adds the reconstructed residual block to the predicted block generated by predictor 120 to reconstruct the current block. When performing intra-frame prediction for the next sequential block, the pixels in the reconstructed current block can be used as reference pixels.

[0068] The loop filtering unit 180 performs filtering on the reconstructed pixels to reduce blocking artifacts, ringing artifacts, blurring artifacts, etc., that occur due to block-based prediction and transform / quantization. The loop filtering unit 180, as an in-loop filter, may include all or some of the following: a deblocking filter 182, a sample adaptive offset (SAO) filter 184, and an adaptive loop filter (ALF) 186.

[0069] Deblocking filter 182 filters the boundaries between reconstructed blocks to remove blocking artifacts caused by block unit encoding / decoding, and SAO filter 184 and ALF 186 perform additional filtering on the deblocked video. SAO filter 184 and ALF 186 are filters used to compensate for the difference between reconstructed pixels and original pixels caused by lossy coding. SAO filter 184 applies an offset in CTU units to enhance subjective image quality and coding efficiency. On the other hand, ALF 186 performs block unit filtering and applies different filters to compensate for distortion by dividing the boundaries of the corresponding blocks and the degree of variation. Information about the filter coefficients to be used for ALF can be encoded and signaled to the video decoding device.

[0070] The reconstructed blocks filtered by deblocking filter 182, SAO filter 184, and ALF 186 are stored in memory 190. When all blocks in an image are reconstructed, the reconstructed image can be used as a reference image for inter-frame prediction of blocks in subsequent images to be encoded.

[0071] Video encoding devices can store the bitstream of encoded video data in a non-volatile storage medium or send the bitstream to a video decoding device via a communication network.

[0072] Figure 5 This is a functional block diagram of a video decoding device that can implement the technology of the present invention. In the following text, reference is made to... Figure 5 This describes a video decoding device and its components.

[0073] The video decoding device may include: an entropy decoder 510, a reordering unit 515, an inverse quantizer 520, an inverse transformer 530, a predictor 540, an adder 550, a loop filter unit 560, and a memory 570.

[0074] Similar to Figure 1 The video encoding device and the video decoding device each component can be implemented as hardware or software, or a combination of hardware and software. Furthermore, the function of each component can be implemented as software, and the microprocessor can also be implemented to execute the software functions corresponding to each component.

[0075] The entropy decoder 510 extracts information related to block segmentation by decoding the bitstream generated by the video encoding device to determine the current block to be decoded, and extracts the prediction information and information about the residual signal required to reconstruct the current block.

[0076] The entropy decoder 510 determines the size of the CTU by extracting information about the CTU size from the sequence parameter set (SPS) or the picture parameter set (PPS), and segments the image into CTUs of a defined size. Furthermore, the CTU is identified as the highest level (i.e., the root node) of the tree structure, and segmentation information for the CTU can be extracted to segment the CTU by utilizing the tree structure.

[0077] For example, when segmenting a CTU using a QTBTT structure, the first flag (QT_split_flag) related to the QT segmentation is first extracted to split each node into four lower-level nodes. Additionally, a second flag (mtt_split_flag), segmentation direction (vertical / horizontal), and / or segmentation type (binary / ternary) related to the MTT segmentation are extracted relative to the nodes corresponding to the leaf nodes of the QT to segment the corresponding leaf nodes using the MTT structure. Thus, each node below a leaf node of the QT is recursively segmented using either a BT or TT structure.

[0078] As another example, when splitting the CTU using the QTBTTT structure, a CU split flag (split_cu_flag) indicating whether the CU has been split is extracted. A first flag (QT_split_flag) can also be extracted when the corresponding block is split. During the splitting process, for each node, zero or more recursive MTT splits may occur after zero or more recursive QT splits. For example, with respect to the CTU, MTT splits may occur immediately, or conversely, only multiple QT splits may occur.

[0079] As another example, when segmenting a CTU using a QTBT structure, the first flag (QT_split_flag) associated with the segmentation of the QT is extracted to split each node into four nodes at the lower level. Additionally, a split flag (split_flag) indicating whether the node corresponding to a leaf node of the QT will be further segmented using BT, as well as segmentation direction information, is extracted.

[0080] On the other hand, when the entropy decoder 510 determines the current block to be decoded by utilizing the segmentation of the tree structure, the entropy decoder 510 extracts information about the prediction type indicating whether the current block is predicted intra-frame or inter-frame. When the prediction type information indicates intra-frame prediction, the entropy decoder 510 extracts the syntax elements for the intra-frame prediction information (intra-frame prediction mode) for the current block. When the prediction type information indicates inter-frame prediction, the entropy decoder 510 extracts information representing the syntax elements for the inter-frame prediction information, namely, the motion vector and the reference picture for the motion vector reference.

[0081] In addition, the entropy decoder 510 extracts quantization-related information and extracts information about the quantization transform coefficients of the current block as information about the residual signal.

[0082] The reordering unit 515 can again change the 1D quantization transform coefficient sequence entropy decoded by the entropy decoder 510 into a 2D coefficient array (i.e., a block) in the reverse order of the coefficient scan order performed by the video encoding device.

[0083] The inverse quantizer 520 performs inverse quantization on the quantized transform coefficients, and performs inverse quantization on the quantized transform coefficients using quantization parameters. The inverse quantizer 520 can also apply different quantization coefficients (scaling values) to the quantized transform coefficients arranged in 2D. The inverse quantizer 520 can perform inverse quantization by applying a matrix of quantization coefficients (scaling values) from the video encoding device to a 2D array of quantized transform coefficients.

[0084] The inverse transformer 530 reconstructs the residual signal by inversely transforming the inverse-quantized transform coefficients from the frequency domain to the spatial domain, thereby generating the residual block of the current block.

[0085] Furthermore, when the inverse transformer 530 performs an inverse transform on a portion (sub-block) of the transform block, it extracts a flag (cu_sbt_flag) indicating that only the sub-blocks of the transform block are transformed, the orientation (vertical / horizontal) information (cu_sbt_horizontal_flag) of the sub-blocks, and / or the position information (cu_sbt_pos_flag) of the sub-blocks. The inverse transformer 530 also inverse transforms the transform coefficients of the corresponding sub-blocks from the frequency domain to the spatial domain to reconstruct the residual signal, and fills the regions not inversely transformed with values ​​of "0" as the residual signal to generate the final residual block of the current block.

[0086] Furthermore, when applying MTS, the inverse transformer 530 determines the transform function or transform matrix to be applied in both the horizontal and vertical directions by utilizing the MTS information (mts_idx) signaled from the video encoding device. The inverse transformer 530 also performs inverse transform on the transform coefficients in the transform block in both the horizontal and vertical directions using the determined transform function.

[0087] Predictor 540 may include intra-predictor 542 and inter-predictor 544. Intra-predictor 542 is activated when the prediction type of the current block is intra-prediction, and inter-predictor 544 is activated when the prediction type of the current block is inter-prediction.

[0088] Intra-predictor 542 determines the intra-prediction mode for the current block among multiple intra-prediction modes based on the syntax elements for the intra-prediction mode extracted from entropy decoder 510. Intra-predictor 542 also predicts the current block by utilizing neighboring reference pixels of the current block according to the intra-prediction mode.

[0089] The inter-frame predictor 544 determines the motion vector and the reference picture of the motion vector reference for the current block by using the syntax elements for the inter-frame prediction mode extracted from the entropy decoder 510, and predicts the current block by using the motion vector and the reference picture.

[0090] Adder 550 reconstructs the current block by adding the residual block output from inverse transform 530 to the predicted block output from inter-frame predictor 544 or intra-frame predictor 542. Pixels within the reconstructed current block are used as reference pixels when performing intra-frame prediction on a subsequent block to be decoded.

[0091] The loop filtering unit 560, acting as an in-loop filter, may include a deblocking filter 562, a SAO filter 564, and an ALF 566. The deblocking filter 562 performs deblocking filtering on the boundaries between reconstructed blocks to remove block artifacts caused by block unit decoding. The SAO filter 564 and ALF 566 perform additional filtering on the reconstructed blocks after deblocking to compensate for differences between reconstructed pixels and original pixels caused by lossy encoding. The filter coefficients of the ALF are determined by utilizing information about the filter coefficients decoded from the bitstream.

[0092] The reconstructed blocks filtered by deblocking filter 562, SAO filter 564, and ALF 566 are stored in memory 570. When all blocks in an image are reconstructed, the reconstructed image can be used as a reference image for inter-frame prediction of blocks in subsequent images to be encoded.

[0093] The following describes an improved encoding / decoding tool executed by the aforementioned video encoding or video decoding apparatus. The following embodiments can be executed by the intra-frame predictor 122 in the video encoding apparatus. Additionally, the following embodiments can be executed by the intra-frame predictor 542 in the video decoding apparatus. In this invention, the image processing apparatus refers to at least one of the video encoding or video decoding apparatuses.

[0094] This invention relates to a matrix-based intra prediction (MIP) mode. The MIP mode derives the predicted sample for the current block by multiplying the reference pixels of the neighboring reference templates of the current block by a weight matrix.

[0095] The process of deriving the predicted sample for the current block in MIP mode can be represented by Equation 1.

[0096] [Equation 1] In Equation 1, P(x, y) represents the predicted sample of the current block, x and y represent the positions of the predicted sample in the current block, r(k) represents the k-th reference pixel of the current block, and F(x, y, k) represents the matrix weights corresponding to the predicted sample (x, y). k represents the index of the reference pixel in the reference template of the current block. The value of the predicted sample of the current block is determined by multiplying the weight matrix of the current block with the reference pixel. The size of the predicted block is the same as the size of the current block.

[0097] MIP mode can be activated or deactivated based on the current block size. MIP mode can be activated for blocks whose width and height are each less than or equal to a predefined value. For example, the predefined value could be 32. MIP mode can be used for blocks with both width and height up to 32. The predefined value can be any value other than 32. On the other hand, MIP mode may not be activated for blocks of certain sizes. For example, MIP mode may not be available for blocks with width × height of 4 × 32, 32 × 4, 8 × 32, or 32 × 8.

[0098] MIP mode can be activated or deactivated based on the intra-prediction mode of the current block. For example, MIP mode can be activated when the intra-prediction mode number of the current block is 0, 1, or (2+2×k). Here, k is an integer including 0. As another example, MIP mode can be activated when the intra-prediction mode number of the current block is 0, 1, or (2+4×k).

[0099] MIP mode can be activated or deactivated based on at least one of the current block size or intra-prediction mode. MIP mode can be activated for a preset size or smaller block and a preset intra-prediction mode. For example, MIP mode can be activated when the width and height of the current block are 16 or less, and the intra-prediction mode number of the current block is 0, 1, or (2+2×k). Here, k is an integer including 0. As another example, MIP mode can be activated when the width and height of the current block are 32 or less, and the intra-prediction mode number of the current block is 0, 1, or (2+4×k). Here, k is an integer including 0.

[0100] When the width or height of the current block is greater than 16, the MIP mode is used to generate a 16×16 prediction block, and the prediction block of the current block can be generated by performing interpolation or bilinear interpolation on the 16×16 prediction block.

[0101] When MIP mode is activated, the intra-prediction mode of the current block is replaced by MIP mode. Specifically, the existing or derived intra-prediction mode of the current block can be replaced by MIP mode. Some directional intra-prediction modes can also be replaced by MIP mode. Once it is determined whether the intra-prediction mode and MIP mode of the current block are activated, the current block is predicted using MIP mode without additional signaling.

[0102] Specifically, the present invention provides a video encoding / decoding method and apparatus for performing MIP prediction using various reference pixels. According to the present invention, various reference pixels are used for MIP prediction of the current block so as not to limit the shape, position, and characteristics of the reference pixels. According to the present invention, the accuracy of intra-frame prediction modes can be improved, and encoding and decoding efficiency can be enhanced. Furthermore, offsets can be used in the MIP prediction according to the present invention.

[0103] Figure 6 This is a flowchart of a matrix-based intra-frame prediction method according to an embodiment of the present invention.

[0104] In step S610, the image processing device determines the reference template for the current block.

[0105] A reference template refers to a set of pixels that were encoded or decoded before the current block.

[0106] The reference template can include pixels adjacent to the current block. For example, the reference template can include at least one reference line adjacent to the current block.

[0107] The reference template can include pixels that are not adjacent to the current block. For example, the reference template can include a reference block that is not adjacent to the current block.

[0108] The reference template may include a reference region indicated by motion information from the current block's neighboring blocks. This motion information may be referred to as a motion vector or a block vector.

[0109] Reference templates can be pre-specified between video encoding devices and video decoding devices.

[0110] Alternatively, the reference template can be determined based on signaling information or the intra-prediction mode of the current block. The signaling information may include information about the shape, position, size, etc., of the reference template. For example, the signaling information may include information indicating a reference template that includes pixels adjacent to the current block, a reference template that includes non-adjacent pixels, or a reference template indicated using motion information from neighboring blocks.

[0111] The size of the reference template can be predefined. Alternatively, the size of the reference template can be determined based on at least one of the size of the current block or the size of the weight matrix. The size of the reference template can also be determined more based on the intra-frame prediction mode.

[0112] At least one pixel in the reference template may be a pixel generated from other pixels using at least one of padding or interpolation.

[0113] The image processing device can adjust the reference template by performing a weighted summation of pixels in the reference template in units of a predetermined shape.

[0114] Information regarding whether to perform fill, interpolation, adjustment, etc., on the reference template can be specified between the video encoding device and the video decoding device or transmitted by signal.

[0115] In step S620, the image processing device determines a weight matrix for matrix-based intra-prediction of the current block. Furthermore, the image processing device may determine the offset for matrix-based intra-prediction.

[0116] The weight matrix can be derived based on at least one of the intra-prediction mode or the size of the current block. As an example, the weight matrix for the current block can be selected from a predefined weight matrix based on at least one of the current block size or the intra-prediction mode. As another example, each element of the weight matrix for the current block can be derived based on at least one of the current block size or the intra-prediction mode.

[0117] Alternatively, the weight matrix can be determined based on signaling information between the video encoding and decoding devices. Information indicating the weight matrix of the current block can be signaled from a predefined weight matrix.

[0118] Offset is a deviation used to adjust the result of matrix multiplication. Offset can be 0 or a real value. The offset can have the same value regardless of the location of the predicted sample in the current block. Alternatively, the offset can have a different value for each location of the predicted sample in the current block.

[0119] The offset can be a value selected from predefined values ​​based on signaling information between the video encoding and decoding devices. Alternatively, the offset can be derived based on information about the current block or adjacent blocks.

[0120] In step S630, the image processing device generates a prediction block for the current block based on the weight matrix and the reference template of the current block. The offset can be further used to generate the prediction block for the current block.

[0121] The process of generating the prediction block for the current block can be represented by Equation 2.

[0122] [Equation 2] In Equation 2, b(k) represents the offset corresponding to the k-th reference pixel. P(x, y) represents the predicted sample of the current block, where x and y represent the positions of the predicted sample within the current block, r(k) represents the k-th reference pixel of the current block, and F(x, y,k) represents the matrix weights corresponding to the predicted sample (x, y). k represents the index of the reference pixel in the reference template of the current block. The value of the predicted sample of the current block is determined by multiplying the weight matrix of the current block with the reference pixel. The size of the predicted block is the same as the size of the current block.

[0123] According to another embodiment of the invention, the prediction block of the current block is generated by reducing the reference template, matrix multiplication and upsampling.

[0124] The reference pixel is derived and can be used for MIP prediction. The derivation of the reference pixel may include reduction of the reference template, matrix calculation, or upsampling.

[0125] First, the reference template is reduced by averaging or sampling the pixels within it. The reduced reference template can be in vector or matrix form. Here, the number of pixels to be averaged, the sampling rate, etc., can be specified between the video encoding and decoding devices.

[0126] The boundary vector is then derived by multiplying the reduced reference template by one of the predefined matrices. The matrix used for calculation can be selected from the predefined matrices based on at least one of the current block size or intra-prediction modes.

[0127] A prediction block for the current block is generated by upsampling the boundary vector. For example, a prediction block for the current block can be derived by interpolating the elements of the boundary vector.

[0128] The generation of the prediction block for the current block based on the reduction and upsampling of the reference template can be performed based on the fact that at least one of the current block size or the intra-prediction mode is a predefined value. In other words, when at least one of the intra-prediction mode or the current block size is within a predetermined range, the prediction block for the current block can be generated from the reduction, matrix multiplication and upsampling of the reference template.

[0129] Alternatively, signaling information can be used to determine whether to generate a prediction block for the current block based on a reduction and upsampling of a reference template.

[0130] Similarly, image processing devices can improve encoding / decoding efficiency and prediction accuracy by utilizing various reference templates for MIP prediction.

[0131] Figure 7a , Figure 7b and Figure 7c Each shows a reference template adjacent to the current block according to an embodiment of the present invention.

[0132] refer to Figure 7a , Figure 7b and Figure 7c The reference template can be one of the following: an L-shaped reference template, a left-side reference template, or a top-side reference template.

[0133] In another implementation, reference templates of various shapes can be used.

[0134] exist Figure 7a The image shows an L-shaped reference template.

[0135] The L-shaped reference template includes at least one reference line adjacent to the current block. Specifically, the L-shaped reference template includes at least one left reference line, at least one top reference line, and at least one top-left reference pixel.

[0136] In other words, the L-shaped reference template can include the adjacent pixels to the left and top of the current block. Specifically, the L-shaped reference template can include the lower left adjacent pixels, the left adjacent pixels, the upper left adjacent pixels, the upper adjacent pixels, and the upper right adjacent pixels of the current block.

[0137] The size or area of ​​the L-shaped reference template can be represented as (C) W ×W×L1)+(C H ×H×L2)+(L1×L2). Here, W is the width of the current block, H is the height of the current block, and C W and C H These are the weights for the width and height of the current block, respectively. L1 is the number of top reference lines, and L2 is the number of left reference lines. C W C H L1 and L2 are positive integers.

[0138] For example, the reference template may include samples of reconstruction line 4 adjacent to the left of the current block (L2=4) and samples of reconstruction line 4 adjacent to the top of the current block (L1=4). The width and height of the reference template may have values ​​of 2W and 2H, respectively.

[0139] exist Figure 7b The left-hand reference template is shown in the image.

[0140] The left reference template includes at least one left reference line adjacent to the current block. In other words, the left reference template includes the left-side adjacent pixels of the current block. Specifically, the left reference template includes the lower-left adjacent pixels and the left-side adjacent pixels of the current block. Furthermore, the left reference template may include the upper-left adjacent pixels of the current block.

[0141] The size or area of ​​the reference template on the left can be represented as (C H×H×L3). L3 represents the number of reference lines on the left.

[0142] exist Figure 7c The upper reference template is shown in the image.

[0143] The top reference template includes at least one top reference line adjacent to the current block. In other words, the top reference template includes the top-adjacent pixels of the current block. Specifically, the top reference template includes the top-upper-right adjacent pixels of the current block. Furthermore, the top reference template may include the top-left pixel of the current block.

[0144] The size or area of ​​the upper reference template can be expressed as (C W ×W×L4). L4 represents the number of upper reference lines.

[0145] exist Figure 7a , Figure 7b and Figure 7c In the middle, the size of the reference template (that is, C) W C H L1, L2, L3 and L4 can be determined through various schemes.

[0146] C W C H L1, L2, L3, and L4 can be pre-specified between the video encoding device and the video decoding device.

[0147] As an example, C W and C H It can be 2, and L1, L2, L3 and L4 can be 4.

[0148] As another example, C W and C H It can be 2, and L1, L2, L3 and L4 can all be 2.

[0149] As another example, C W and C H It can be 2, and L1, L2, L3 and L4 can be 1.

[0150] As another example, C W and C H It can be 1, and L1, L2, L3 and L4 can all be 1.

[0151] Alternatively, C can be determined based on the size of the current block. W C H L1, L2, L3, and L4. L1, L2, L3, and L4 can be selected from predefined values ​​based on the size of the current block. Here, the predefined values ​​can be 1, 2, or 4.

[0152] As an example, when the size of the current block is less than or equal to 16×16, C W and C H It can be 2, and L1, L2, L3 and L4 can all be 2.

[0153] As another example, when the width and height of the current block are greater than 16×16, and the width and height of the current block are less than or equal to 32×32, C W and C H It can be 2, and L1, L2, L3 and L4 can be 1.

[0154] Alternatively, C can be determined based on at least one of the size of the current block or the size of the weight matrix. W C H L1, L2, L3, and L4. In the left reference template, when the size of the weight matrix is ​​M×N, the height of the left reference template is N, and C H It can be a value obtained by dividing N by H.

[0155] As an example, L3 can be 2 when the size of the current block is less than or equal to 16×16.

[0156] Alternatively, C can be determined based on at least one of the intra-frame prediction mode or the size of the current block. W C H L1, L2, L3 and L4.

[0157] As an example, when the size of the current block is less than or equal to 16×16, and the intra-prediction mode of the current block includes mode numbers 0, 1, or (2+2×k), C W and C H The value is 2, and L1, L2, L3, and L4 can all be 2. Here, k is 0 or a positive integer.

[0158] As another example, when the width and height of the current block are greater than 16×16, the width and height of the current block are less than or equal to 32×32, and the intra-prediction mode of the current block includes mode number 0, 1, or (2+4×k), C W and C H The value can be 2, and L1, L2, L3, and L4 can be 1. Here, k is 0 or a positive integer.

[0159] Alternatively, C can be determined based on the intra-frame prediction mode. W and C H For example, for intra-frame prediction modes 18 to 50, C W and C H It can be set to 1. For the remaining intra-frame prediction mode, C W and C HIt can be set to 2.

[0160] On the other hand, the reference template adjacent to the current block can be determined by implicit or explicit methods.

[0161] As an implicit method, a reference template can be determined based on the intra-prediction mode replaced by the MIP mode. Various reference templates can be predefined or mapped to intra-prediction modes. If the intra-prediction mode is a predefined value or included within a predefined range, a specific reference template can be used for MIP prediction. The relationship between the reference template and the intra-prediction mode can be specified between the video encoding and decoding devices.

[0162] As an example, when the intra-prediction mode number of the current block is 50, it indicates the intra-prediction mode in the vertical direction, and the upper reference template can be used for MIP prediction.

[0163] As another example, when the intra-prediction mode number of the current block is 18, indicating the intra-prediction mode in the horizontal direction, the left reference template can be used for MIP prediction.

[0164] As another example, when the intra-prediction mode number of the current block is 34, indicating the intra-prediction mode in the diagonal direction, the L-shaped reference template can be used for MIP prediction.

[0165] As another example, when the intra-prediction mode number of the current block is 45, indicating the intra-prediction mode in the diagonal direction, the entire upper reference template and the upper part of the left reference template can be used.

[0166] As another implicit method, the reference template can be determined based on information other than the intra-frame prediction mode.

[0167] As an explicit method, the reference template can be determined based on the signaling information between the video encoding device and the video decoding device.

[0168] As an example, signaling information may include an index or flag used to identify the reference template for the current block from a predefined reference template. The reference template for the current block may be selected from a predefined reference template based on the signaling information.

[0169] Figure 8 A reference template that is not adjacent to the current block is shown according to an embodiment of the present invention.

[0170] You can specify a non-adjacent neighboring block between the video encoding device and the video decoding device for the current block.

[0171] exist Figure 8The diagram shows an example of a non-neighboring block of the current block. At least one of blocks 14 through 31 can be used as a reference template to predict the MIP of the current block.

[0172] The number of each pixel indicates the non-nearest neighboring blocks of the corresponding pixel. For example, the non-nearest neighboring block including pixel number 14 can be called block number 14. The pixel included in block number 14 is called the reference pixel of the current block.

[0173] In another implementation, various non-adjacent neighboring blocks can be used. Depending on the location, size, number, or shape of the non-adjacent neighboring blocks, various neighboring blocks can be used as reference templates.

[0174] Non-nearest neighboring blocks for MIP prediction of the current block can be predefined or derived.

[0175] As an example, the location, size, number, or shape of non-adjacent adjacent blocks can be specified in advance.

[0176] As another example, the position, size, number, or shape of non-nearest neighboring blocks can be inferred based on the size of the current block, its prediction mode, etc. The video decoding device can select non-nearest neighboring blocks for MIP prediction from a predefined pool of non-nearest neighboring blocks based on the size of the current block, its prediction mode, etc.

[0177] As another example, the location, size, number, or shape of non-adjacent neighboring blocks can be inferred based on signaling information. Signaling information may include flags or indices used to specify non-adjacent neighboring blocks.

[0178] Figure 9 A reference template indicating the motion information of adjacent blocks according to an embodiment of the present invention is shown.

[0179] exist Figure 9 In the image, the neighboring block to the left of the current block is a block predicted using the reconstructed block indicated by the motion information of the neighboring blocks in the current image.

[0180] When predicting neighboring blocks, template matching can be used to determine the reconstructed blocks. Specifically, template matching means finding a template in the search region that minimizes the difference between the template of the target block and the template of any block in the search region. The difference between the template of the target block and the template of any block in the search region is defined as the cost function. The template with the minimum cost is searched, and the block containing the searched template is used as the predicted block for the target block. As cost functions, sum of absolute differences (SAD), sum of absolute transformed differences (SATD), mean-removed sum of absolute differences (MR-SAD), mean squared error (MSE), sum of squared error (SSE), etc., can be used. For example, in... Figure 9 In this process, the difference between the template of an adjacent block and the template of a reconstructed block is minimized, and the predicted block of an adjacent block can be the same as the predicted block of a reconstructed block. The reconstructed block is located at a distance equal to the motion information of the adjacent blocks and can have the same size as the adjacent blocks.

[0181] Motion information of adjacent blocks can indicate the relative positional relationship between the reconstructed block of the current block and its adjacent blocks. The motion information of adjacent blocks can be called block vectors.

[0182] When neighboring blocks have motion information, the pixel at the position indicated by the motion information of the neighboring block, or the neighboring pixel at the indicated position, can be used as a reference template for the current block.

[0183] In the example, a reconstructed block in the current image, based on the motion information of neighboring blocks, can be used as a reference template for the current block.

[0184] In another example, as a reference template for the current block, a template of a reconstructed block in the current image, based on the motion information of neighboring blocks, can be used.

[0185] In another example, a block in the current image, indicated by motion information from neighboring blocks based on the current block, can be used as a reference template for the current block. Here, the size and orientation of the motion information of neighboring blocks are preserved, and the block indicated by the motion information of neighboring blocks can be the right-hand block of the reconstructed block.

[0186] In another example, as a reference template for the current block in the current image, a template of a block based on the motion information of neighboring blocks can be used.

[0187] In another implementation, the neighboring blocks of the current block can be predicted using blocks in a reconstructed block in a reference image different from the current image, indicated by motion information of neighboring blocks.

[0188] Motion information of adjacent blocks indicates the positional relationship between corresponding blocks in a reference image and reconstructed blocks in the reference image. Motion information of adjacent blocks also indicates the relative positional relationship between adjacent blocks in the current image and reconstructed blocks in the reference image. This motion information of adjacent blocks can be referred to as motion vectors.

[0189] The corresponding block of an adjacent block refers to a block in the reference image that has the same relative position as the adjacent block in the current image. In other words, the position of an adjacent block in the current image is the same as the position of its corresponding block in the reference image. Furthermore, the corresponding block of the current block refers to a block in the reference image that has the same relative position as the current block in the current image.

[0190] In the example, a block in the reference image, indicated by motion information of adjacent blocks relative to their corresponding blocks, can be used as a reference template for the current block. The indicated block is located at an equidistant point from the motion information of the corresponding blocks of adjacent blocks.

[0191] In another example, a template of a block in a reference image, indicated by motion information of the adjacent blocks relative to their corresponding blocks, can be used as a reference template for the current block.

[0192] In another example, as a reference template for the current block, a block in the reference image indicated by the motion information of adjacent blocks based on the current block can be used. Here, the size and orientation of the motion information of adjacent blocks are preserved. The block indicated by the motion information of adjacent blocks can be the right-hand block of the corresponding block in the reconstructed block.

[0193] In another example, as a reference template for the current block, a template of a block in the reference image, indicated by the motion information of adjacent blocks based on the corresponding block of the current block, can be used.

[0194] According to an embodiment of the present invention, if the reference template of the current block is not available for MIP prediction, at least one pixel in the reference template can be generated from other pixels using at least one of padding or interpolation.

[0195] Specifically, if matrix computation is impossible due to a mismatch between the size of the reference template and the size of the weight matrix, or if pixels in the reference template are unavailable, or if the reference template crosses a predetermined boundary, pixels in the reference template can be generated or replaced. For example, the reference template can be expanded when its height is less than the width of the weight matrix.

[0196] To generate the pixels of the reference template, you can use padding, interpolation, vertical interpolation, horizontal interpolation, or both vertical and horizontal interpolation.

[0197] As an example, a reference template can be expanded by filling it with any pixels within the reference template. Alternatively, the reference template can be filled with predefined values ​​or values ​​based on the bit depth.

[0198] As another example, a reference template can be extended by interpolating one or more pixels of the reference template. Alternatively, a reference template can be extended by interpolating adjacent pixels of the reference template. Alternatively, a reference template can be extended by performing interpolation using pixels within the reference template and adjacent pixels of the reference template.

[0199] As another example, unavailable pixels in a reference template can be replaced by pixels generated by interpolating adjacent pixels.

[0200] According to an embodiment of the present invention, the reference template of the current block can be adjusted by performing a weighted summation of pixels within the reference template in units of a predetermined shape. The adjusted reference template is used for MIP prediction of the current block.

[0201] The adjustment of the reference template can be represented by Equation 3.

[0202] [Equation 3] In equation 3, ref adj This is the adjusted reference template, w is the weight, and ref is the unit of weighted summation performed within the reference template. n The shape can be a line or a block. The sum of the weights is 1.

[0203] As an example, the predefined unit can be a line unit. When the reference template of the current block is an upper reference template that includes multiple upper reference lines, a weighted summation of the multiple upper reference lines in the vertical direction can be performed. Here, the number of rows in the reference lines after performing the weighted summation is less than the number of rows in the multiple upper reference lines.

[0204] As another example, the predetermined unit can be an L-shaped unit. When the reference template of the current block is an L-shaped template, a weighted summation of the L-shaped reference lines in the reference template can be performed in L-shaped units. A weighted summation of the top reference line and the left reference line can be performed, as can a weighted summation of the top-left pixel of the current block. In Equation 3, ref0 is the L-shaped reference line adjacent to the current block, and ref n It can be the L-shaped reference line furthest from the current block.

[0205] As another example, the predetermined unit can be a block unit. When the reference template of the current block includes non-nearest neighboring blocks of the current block, a weighted summation of the non-nearest neighboring blocks can be performed. The number of non-nearest neighboring blocks after performing the weighted summation is less than the number of non-nearest neighboring blocks without performing the weighted summation.

[0206] The position, order, size, or orientation of the predetermined units used to adjust the reference template can be varied. For example, a weighted summation of the upper reference lines in the horizontal direction can be performed.

[0207] A portion of a reference template can be adjusted. In other words, the adjusted reference template can be the result of a weighted sum of a portion of a block or a portion of a line from an existing reference template.

[0208] Information used to adjust the reference template can be specified between the video encoding and decoding devices or transmitted via signals. This information may include whether the reference template is to be adjusted, the unit of the weighted summation, and the position, size, number, or direction of the weighted summation of reference lines or reference blocks within the reference template.

[0209] Figure 10 This is a flowchart of a video encoding method according to an embodiment of the present invention.

[0210] exist Figure 10 In step S1010, the video encoding device determines the intra-frame prediction mode for the current block.

[0211] It can be determined that MIP mode is the best intra-frame prediction mode.

[0212] In step S1020, the video encoding device performs intra-frame prediction for the current block based on the intra-frame prediction mode.

[0213] If the intra-frame prediction mode is MIP mode, the video coding device performs MIP prediction for the current block. This can be utilized... Figures 6 to 9 The method described herein is used to perform MIP prediction.

[0214] In step S1030, the video encoding device encodes information about the intra-frame prediction mode.

[0215] Information about intra-prediction modes can include information about the use of MIP, as well as information about the reference template, matrix, offset, etc., used for MIP prediction.

[0216] Information regarding the use of MIP can include the MIP constraint flag (MIP_constraint_flag) and the MIP flag. The MIP flag can be either the MIP enabled flag (MIP_enabled_flag) or the MIP disabled flag (MIP_disabled_flag). Setting the MIP constraint flag to 0 indicates that there are no constraints on MIP prediction. The MIP enabled or disabled flag can be set to either 0 or 1. Conversely, if the MIP constraint flag is set to 1, the MIP enabled flag is set to 0, and the MIP disabled flag is set to 1.

[0217] Information about the use of MIP can be signaled independently at higher or lower levels, or it can be signaled in conjunction with other syntax elements.

[0218] As an example, information about the use of MIP can be signaled at a higher level, such as in the video parameter set (VPS), sequence parameter set (SPS), picture parameter set (PPS), picture header (PH), or slice header (SH).

[0219] As another example, information about the use of MIPs can be signaled at lower levels, such as in coding tree units (CTUs), coding units (CUs), prediction units (PUs), or transformation units (TUs). The use of MIPs is controlled on a block-by-block basis.

[0220] As another example, information regarding the use of MIP is not signaled by explicit flags, but rather can be signaled dependently on flags such as the current block size or other prediction modes. The video decoding device can determine the MIP mode based on flags such as the current block size or other prediction modes.

[0221] Figure 11 This is a flowchart of a video decoding method according to an embodiment of the present invention.

[0222] exist Figure 11 In step S1110, the video decoding device decodes information about the intra-prediction mode of the current block.

[0223] Information about intra-prediction modes can include information about the use of MIP, as well as information about the reference template, matrix, offset, etc., used for MIP prediction.

[0224] In step S1120, the video decoding device determines the intra-prediction mode of the current block based on information about the intra-prediction mode. Here, the intra-prediction mode can be the MIP mode.

[0225] In step S1130, the video decoding device performs MIP prediction on the current block. This can be achieved using... Figures 6 to 9 The method described herein is used to perform MIP prediction.

[0226] Although the steps in the various flowcharts are described in sequence, these steps only illustrate the technical ideas of some embodiments of the invention. Therefore, those skilled in the art to which this invention pertains can perform these steps by changing the order described in the various figures or by performing two or more steps in parallel. Thus, the steps in the various flowcharts are not limited to the chronological order shown.

[0227] It should be understood that the above description presents illustrative embodiments that can be implemented in various other ways. The functionality described in some embodiments can be implemented by hardware, software, firmware, and / or combinations thereof. It should also be understood that functional components described in this invention are designated as "...units" to emphasize their potential for independent implementation.

[0228] On the other hand, the various methods or functions described in some embodiments can be implemented as instructions stored in a non-volatile recording medium, which can be read and executed by one or more processors. The non-volatile recording medium can include various types of recording devices, such as those storing data in a computer system-readable form. For example, the non-volatile recording medium can include storage media such as erasable programmable read-only memory (EPROM), flash memory drives, optical disk drives, magnetic hard disk drives, and solid-state drives (SSDs).

[0229] Although embodiments of the invention have been described for illustrative purposes, those skilled in the art will understand that various modifications, additions, and substitutions can be made without departing from the spirit and scope of the invention. Therefore, embodiments of the invention have been described for the sake of brevity and clarity. The scope of the technical concept of the embodiments of the invention is not limited by the examples. Accordingly, those skilled in the art will understand that the scope of the invention should not be limited by the embodiments explicitly described above, but rather by the claims and their equivalents.

[0230] Cross-references to related applications This application claims priority and benefit to Korean Patent Application No. 10-2024-0041177, filed March 26, 2024, and Korean Patent Application No. 10-2025-0037374, filed March 24, 2025, the entire contents of which are incorporated herein by reference.

Claims

1. A video decoding method, comprising: Determine a reference template that includes the pixels to be decoded; Determine the weight matrix for matrix-based intra-frame prediction of the current block; as well as The prediction block for the current block is generated based on the weight matrix and the reference template of the current block.

2. The video decoding method of claim 1, wherein, The reference template includes pixels that are adjacent to the current block among the decoded pixels.

3. The video decoding method according to claim 2, wherein, The reference template is determined based on signaling information or the intra-frame prediction mode of the current block.

4. The video decoding method according to claim 1, wherein, The reference template includes pixels that are not adjacent to the current block in the decoded pixels.

5. The video decoding method according to claim 1, wherein, The reference template includes a reference region indicated by motion information of adjacent blocks of the current block.

6. The video decoding method according to claim 1, wherein, At least one pixel in the reference template is generated from other pixels using at least one of padding or interpolation.

7. The video decoding method according to claim 1, further comprising: The reference template is adjusted by performing a weighted summation of pixels within the reference template in units of a predetermined shape.

8. The video decoding method according to claim 1, wherein, The predicted blocks that generated the current block include: The reference template is reduced based on the fact that at least one of the intra-prediction mode or the size of the current block is a predefined value. Boundary vectors are generated by multiplying a predefined matrix by a reduced reference template; and The prediction block for the current block is generated by upsampling the boundary vector.

9. The video decoding method according to claim 1, wherein, The size of the reference template is determined based on at least one of the size of the current block or the size of the weight matrix.

10. The video decoding method according to claim 1, wherein, The predicted block for the current block is further generated based on the offset of the matrix-based intra-frame prediction used for the current block.

11. A video encoding method, comprising: Determine a reference template that includes the encoded pixels; Determine the weight matrix for matrix-based intra-frame prediction of the current block; as well as The prediction block for the current block is generated based on the weight matrix and the reference template of the current block.

12. A method for transmitting data including a bitstream of video, the method comprising: Generate the bitstream of the video; as well as Transmission includes data in bit stream format. The generation of the bitstream includes: Determine a reference template that includes the encoded pixels; Determine the weight matrix for matrix-based intra-frame prediction of the current block; and The prediction block for the current block is generated based on the weight matrix and the reference template of the current block.

Images