Video encoding and decoding method and apparatus utilizing temporal MPM candidates
By obtaining intra-frame patterns from images other than the current image during intra-frame prediction and adding them to the MPM list, the problem of insufficient video encoding and decoding efficiency in existing technologies is solved, achieving more efficient video encoding and decoding and quality improvement.
Patent Information
- Application Number
- CN202580028244.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2025-04-07
- Filing Date
- 2025-04-09
- Publication Date
- 2026-11-17
AI Technical Summary
Existing video encoding and decoding technologies are inefficient when processing large-size, high-definition, and high-frame-rate video data, and cannot effectively improve video quality.
When configuring the most probable mode (MPM) list in intra-frame prediction, intra-frame modes are obtained from images other than the current image and added to the candidate list, and then processed by the video encoding and decoding devices.
It improves video encoding and decoding efficiency and enhances video quality.
Smart Images

Figure CN123125855A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to a video coding and decoding method and apparatus that utilizes temporal MPM candidates in intra-frame prediction. Background Technology
[0002] The statements in this section are provided only as background information in relation to the present invention and do not necessarily constitute prior art.
[0003] Compared to audio or still image data, video data is much larger in volume, and unprocessed or uncompressed video data requires significant hardware resources (including memory) to store or transmit it in its raw form.
[0004] Accordingly, when storing or transmitting video data, an encoder is typically used to compress the video data for storage or transmission, and a decoder receives the compressed video data and performs decompression and reconstruction. Such video compression technologies include H.264 / AVC, High Efficiency Video Coding (HEVC), and Versatile Video Coding (VVC), which offers approximately 30% or more improvement in coding efficiency compared to HEVC.
[0005] However, as video size, resolution, and frame rate continue to increase, and the amount of data that needs to be encoded and decoded also increases accordingly, a new compression technology is needed that is more efficient in terms of encoding and decoding and more effective in improving video quality than existing compression technologies. Summary of the Invention
[0006] Technical issues The present invention aims to provide a video encoding and decoding method and apparatus, which, when configuring a Most Probable Mode (MPM) list in intra-frame prediction, adds intra-frame modes obtained from images other than the current image to the MPM list.
[0007] Technical solution At least one aspect of the present invention provides a method for reconstructing a current block, performed by a video decoding apparatus. The method includes: configuring a candidate list for the current block. Here, the current block is included in an inter-frame slice and intra-frame prediction is performed; the candidate list includes candidate intra-frame modes. The method further includes: selecting at least one reference image associated with the current block. The method further includes: obtaining at least one intra-frame mode from the at least one reference image. The method further includes: updating the candidate list by adding at least one intra-frame mode to the candidate list.
[0008] Another aspect of the present invention provides a method for encoding a current block, performed by a video encoding apparatus. The method includes: configuring a candidate list for the current block. Here, the current block is included in an inter-frame slice and intra-frame prediction is performed; the candidate list includes candidate intra-frame modes. The method further includes: selecting at least one reference image associated with the current block. The method further includes: obtaining at least one intra-frame mode from the at least one reference image. The method further includes: updating the candidate list by adding at least one intra-frame mode to the candidate list.
[0009] Another aspect of the present invention provides a method for providing video data to a video decoding apparatus. The method includes: encoding video data into a bitstream, and transmitting the bitstream to the video decoding apparatus. Encoding the video data includes: configuring a candidate list for a current block. Here, the current block is included in an inter-frame slice and intra-frame prediction is performed; the candidate list includes candidate intra-frame modes. Encoding the video data further includes: selecting at least one reference image associated with the current block. Encoding the video data further includes: obtaining at least one intra-frame mode from the at least one reference image. Encoding the video data further includes: updating the candidate list by adding at least one intra-frame mode to the candidate list.
[0010] Beneficial effects As described above, the present invention provides a video encoding / decoding method and apparatus that, when configuring an MPM list in intra-frame prediction, adds intra-frame modes obtained from images other than the current image to the MPM list. Therefore, the video encoding / decoding method and apparatus improve video encoding / decoding efficiency and enhance video quality. Attached Figure Description
[0011] Figure 1 This is a block diagram of a video encoding device that can implement the technology of the present invention.
[0012] Figure 2 This paper demonstrates a method for partitioning blocks using a quadtree plus binary tree ternary tree (QTBTTT) structure.
[0013] Figure 3a and Figure 3b Several intra-prediction modes, including a wide-angle intra-prediction mode, are shown.
[0014] Figure 4 The adjacent blocks of the current block are shown.
[0015] Figure 5 This is a block diagram of a video decoding device that can implement the technology of the present invention.
[0016] Figure 6This is an example diagram showing the pixels used for the Most Probable Mode (MPM) configuration.
[0017] Figure 7 This is an example diagram illustrating the time hierarchy in a hierarchical prediction structure according to an embodiment of the present invention.
[0018] Figure 8 This is an example diagram illustrating the occurrence rate predicted according to a hierarchical method based on an embodiment of the present invention.
[0019] Figure 9 This is an example diagram illustrating the time hierarchy in a hierarchical prediction structure according to another embodiment of the present invention.
[0020] Figure 10 This is an example diagram illustrating the occurrence rate predicted according to a hierarchy based on another embodiment of the present invention.
[0021] Figure 11 This is an example diagram illustrating the problem configured according to the MPM list.
[0022] Figure 12 This is an example diagram illustrating a solution to a problem based on an MPM list configuration according to an embodiment of the present invention.
[0023] Figures 13a to 13c This is an example diagram showing a selection of reference images according to an embodiment of the present invention.
[0024] Figure 14 This is an example diagram showing a predetermined position in a reference picture according to an embodiment of the present invention.
[0025] Figure 15a and Figure 15b This is an example diagram illustrating the acquisition of intra-frame mode according to an embodiment of the present invention.
[0026] Figures 16a to 16c This is an example diagram illustrating the acquisition of intra-frame mode according to another embodiment of the present invention.
[0027] Figure 17 This is a flowchart illustrating a method for encoding the current block performed by a video encoding apparatus according to an embodiment of the present invention.
[0028] Figure 18 This is a flowchart illustrating a method for reconstructing the current block performed by a video decoding device according to an embodiment of the present invention. Detailed Implementation
[0029] In the following description, some embodiments of the invention are described in detail with reference to the accompanying illustrative drawings. In the following description, the same reference numerals denote the same elements, although these elements are shown in different drawings. Furthermore, in the following description of some embodiments, detailed descriptions of relevant known components and functions may be omitted for clarity and brevity when it is considered that such detailed descriptions obscure the subject matter of the invention.
[0030] Figure 1 This is a block diagram of a video encoding apparatus that can implement the technology of the present invention. In the following, reference is made to... Figure 1 The illustration depicts a video encoding apparatus and its components.
[0031] The encoding device may include: an image segmenter 110, a predictor 120, a subtractor 130, a transformer 140, a quantizer 145, a reordering unit 150, an entropy encoder 155, an inverse quantizer 160, an inverse transformer 165, an adder 170, a loop filter unit 180, and a memory 190.
[0032] Each component of the encoding device can be implemented as hardware or software, or a combination of hardware and software. Furthermore, the function of each component can be implemented as software, and the microprocessor can be implemented to execute the software function corresponding to each component.
[0033] A video consists of one or more sequences comprising multiple images. Each image is segmented into multiple regions, and encoding is performed on each region. For example, an image is segmented into one or more tiles and / or slices. Here, one or more tiles can be defined as a tile group. Each tile and / or slice is segmented into one or more Coding Tree Units (CTUs). Additionally, each CTU is segmented into one or more Coding Units (CUs) through a tree structure. Information applied to each CU is encoded as the CU's syntax, and information commonly applied to CUs included in a CTU is encoded as the CTU's syntax. Furthermore, information commonly applied to all blocks within a slice is encoded as the slice header's syntax, and information applied to all blocks constituting one or more images is encoded as a Picture Parameter Set (PPS) or picture header. Additionally, information commonly referenced by multiple images is encoded as a Sequence Parameter Set (SPS). Furthermore, information commonly referenced by one or more SPSs is encoded as a Video Parameter Set (VPS). Furthermore, information commonly applied to a tile or tile group can also be encoded as the syntax of the tile or tile group header. The syntax included in SPS, PPS, slice headers, and tile or tile group headers can be called high-level syntax.
[0034] Image splitter 110 determines the size of the coding tree unit (CTU). Information about the size of the CTU (CTU size) is encoded into SPS or PPS syntax and transmitted to the video decoding device.
[0035] Image segmenter 110 divides each image constituting the video into multiple coding tree units (CTUs) of a predetermined size, and then recursively segments the CTUs using a tree structure. The leaf nodes in the tree structure are called coding units (CUs), which are the basic units of encoding.
[0036] A tree structure can be a quadtree (QT) that divides a higher node (or parent node) into four lower nodes (or child nodes) of the same size. A tree structure can also be a binary tree (BT) that divides a higher node into two lower nodes. A tree structure can also be a ternary tree (TT) that divides a higher node into three lower nodes in a 1:2:1 ratio. A tree structure can also be a combination of two or more of the QT, BT, and TT structures. For example, a quadtree plus binary tree (QTBT) structure can be used, or a quadtree plus binary tree ternary tree (QTBTTT) structure can be used. Here, a binary tree ternary tree (BTTT) is added to the tree structure to form a multiple-type tree (MTT).
[0037] Figure 2 This is a schematic diagram illustrating a method for segmenting blocks using a QTBTT structure.
[0038] like Figure 2 As shown, the CTU can first be segmented into a QT structure. Quadtree segmentation can be recursive until the size of the segmented blocks reaches the minimum block size (MinQTSize) allowed for leaf nodes in the QT. The entropy encoder 155 encodes a first flag (QT_split_flag) indicating whether each node of the QT structure is segmented into the four nodes below it and signals this flag to the video decoding device. When the leaf node of the QT is not larger than the maximum block size (MaxBTSize) allowed for the root node in the BT, the leaf node can be further segmented using at least one of the BT or TT structures. Multiple segmentation directions can exist in the BT and / or TT structures. For example, two directions can exist: a horizontal direction for segmenting the blocks of the corresponding node and a vertical direction for segmenting the blocks of the corresponding node. Figure 2 As shown, when MTT splitting begins, the entropy encoder 155 encodes a second flag (mtt_split_flag) indicating whether a node has been split, as well as a flag indicating the splitting direction (vertical or horizontal) and / or the splitting type (binary or ternary) if the node has been split, and signals this to the video decoding device.
[0039] Alternatively, before encoding the first flag (QT_split_flag) indicating whether each node has been split into the four lower-level nodes, the CU splitting flag (split_cu_flag) indicating whether a node has been split can also be encoded. When the value of the CU splitting flag (split_cu_flag) indicates that each node has not been split, the block of the corresponding node becomes a leaf node in the split tree structure and becomes a CU, which is the basic unit of encoding. When the value of the CU splitting flag (split_cu_flag) indicates that each node has been split, the video encoding device begins encoding the first flag using the above scheme.
[0040] When QTBT is used as another example of a tree structure, two types can exist: a type where the corresponding node's block is horizontally divided into two blocks of the same size (i.e., symmetrical horizontal division) and a type where the corresponding node's block is vertically divided into two blocks of the same size (i.e., symmetrical vertical division). The entropy encoder 155 encodes a split flag (split_flag) indicating whether each node of the BT structure is divided into blocks of the lower layer and split type information indicating the split type, and transmits this information to the video decoding device. Alternatively, a type can exist where the corresponding node's block is divided into two blocks that are asymmetrically divided. Asymmetrical forms can include dividing the corresponding node's block into two rectangular blocks with a size ratio of 1:3, or it can also include dividing the corresponding node's block along a diagonal direction.
[0041] The CU can have various sizes depending on the QTBT or QTBTTT segmentation from the CTU. In the following text, the block corresponding to the CU to be encoded or decoded (i.e., the leaf node of the QTBTTT) is called the "current block". When using QTBTTT segmentation, the current block can also be rectangular in shape in addition to a square shape.
[0042] Predictor 120 predicts the current block to generate a prediction block. Predictor 120 includes an intra-frame predictor 122 and an inter-frame predictor 124.
[0043] Typically, each block of the current image can be predictively encoded. This prediction can usually be performed using intra-frame prediction (which utilizes data from the image containing the current block) or inter-frame prediction (which utilizes data from an image encoded before the image containing the current block). Inter-frame prediction includes both one-way and two-way prediction.
[0044] Intra-predictor 122 predicts pixels in the current block by utilizing pixels (reference pixels) that are adjacent to the current block in the current image, including the current block. Depending on the prediction direction, there are various intra-prediction modes. For example, such as... Figure 3aAs shown, various intra-frame prediction modes can include two non-directional modes: Planar mode and DC mode, and can include 65 directional modes. The neighboring pixels and algorithm equations to be used are defined differently for each prediction mode.
[0045] To perform efficient orientation prediction for the current block with a rectangular shape, additional methods such as... can be used. Figure 3b The dashed arrows indicate the directional modes (intra-prediction modes #67 to #80, #-1 to #-14). These directional modes can be referred to as "wide-angle intra-prediction modes." Figure 3b In the diagram, the arrows indicate the corresponding reference samples used for prediction, not the prediction direction. The prediction direction is opposite to the direction indicated by the arrow. When the current block has a rectangular shape, the wide-angle intra-frame prediction mode is a mode that performs prediction in the opposite direction to the specific direction mode without additional bit transmission. In this case, in the wide-angle intra-frame prediction mode, some wide-angle intra-frame prediction modes available for the current block can be determined by the ratio of the width to the height of the current block with a rectangular shape. For example, when the current block has a rectangular shape with a height less than its width, wide-angle intra-frame prediction modes with angles less than 45 degrees (intra-frame prediction modes #67 to #80) are available. When the current block has a rectangular shape with a width greater than its height, wide-angle intra-frame prediction modes with angles greater than -135 degrees (intra-frame prediction modes #-1 to #-14) are available.
[0046] Intra-predictor 122 can determine the intra-prediction to be used for encoding the current block. In some examples, intra-predictor 122 can encode the current block by utilizing multiple intra-prediction modes, and can also select the appropriate intra-prediction mode to use from test modes. For example, intra-predictor 122 can calculate rate-distortion values by utilizing rate-distortion analysis of intra-prediction modes across multiple tests, and can also select the intra-prediction mode with the best rate-distortion characteristics from the test modes.
[0047] Intra-predictor 122 selects one intra-prediction mode from a variety of intra-prediction modes and predicts the current block by using neighboring pixels (reference pixels) determined according to the selected intra-prediction mode and an algorithm equation. Entropy encoder 155 encodes the information about the selected intra-prediction mode and transmits it to the video decoding device.
[0048] Inter-frame predictor 124 generates a predicted block for the current block by utilizing motion compensation processing. Inter-frame predictor 124 searches for the most similar block to the current block in a reference image that was encoded and decoded earlier than the current image, and generates the predicted block for the current block using the searched block. Additionally, a motion vector (MV) is generated, corresponding to the displacement between the current block in the current image and the predicted block in the reference image. Typically, motion estimation is performed on the luma component, and the motion vector calculated based on the luma component is used for both the luma and chroma components. The motion information, including information about the reference image and information about the motion vector used to predict the current block, is encoded by entropy encoder 155 and transmitted to the video decoding device.
[0049] The inter-frame predictor 124 can also perform interpolation on a reference image or reference block to increase prediction accuracy. In other words, subsamples are interpolated between two consecutive integer samples by applying filter coefficients to multiple consecutive integer samples, including two integer samples. When performing a process to search for the most similar block to the current block on the interpolated reference image, motion vectors can be represented with fractional-unit precision instead of integer-sample-unit precision. For each target region to be encoded, such as a cell like a slice, tile, CTU, or CU, the precision or resolution of the motion vectors can be set differently. When applying this Adaptive Motion Vector Resolution (AMVR), information about the motion vector resolution to be applied to each target region should be signaled for each target region. For example, when the target region is a CU, information about the motion vector resolution applied to each CU is signaled. The information about the motion vector resolution can be information representing the precision of the motion vector difference to be described below.
[0050] On the other hand, the inter-frame predictor 124 can perform inter-frame prediction using bidirectional prediction. In the case of bidirectional prediction, two reference images and two motion vectors representing the position of the block most similar to the current block in each reference image are used. The inter-frame predictor 124 selects a first reference image and a second reference image from reference image list 0 (RefPicList0) and reference image list 1 (RefPicList1), respectively. The inter-frame predictor 124 also searches for the block most similar to the current block in the corresponding reference image to generate a first reference block and a second reference block. In addition, the predicted block of the current block is generated by averaging or weighted averaging the first reference block and the second reference block. In addition, motion information including information about the two reference images used to predict the current block and information about the two motion vectors is transmitted to the entropy encoder 155. Here, reference image list 0 may consist of images in the pre-reconstructed images that are displayed before the current image in order of appearance, and reference image list 1 may consist of images in the pre-reconstructed images that are displayed after the current image in order of appearance. However, although not particularly limited to this, pre-reconstructed images that are displayed after the current image in order of appearance may be additionally included in reference image list 0. Conversely, pre-reconstructed images preceding the current image can also be additionally included in the reference image list 1.
[0051] Various methods can be used to minimize the number of bits consumed in encoding motion information.
[0052] For example, when the reference image and motion vector of the current block are the same as those of the adjacent blocks, the information that can identify the adjacent blocks is encoded to transmit the motion information of the current block to the video decoding device. This method is called merge mode.
[0053] In merge mode, the inter-frame predictor 124 selects a predetermined number of merge candidate blocks (hereinafter referred to as "merge candidates") from the neighboring blocks of the current block.
[0054] As adjacent blocks used to derive merge candidates, all or some of the following can be used: left block A0, lower left block A1, upper block B0, upper right block B1, and upper left block B2, which are adjacent to the current block in the current image. Figure 4 As shown. In addition to the current image containing the current block, blocks located within a reference image (which may be the same as or different from the reference image used to predict the current block) can also be used as merge candidates. For example, a co-located block of the current block within the reference image, or a block adjacent to that co-located block, can be additionally used as a merge candidate. If the number of merge candidates selected using the above method is less than a preset number, a zero vector is added to the merge candidates.
[0055] The inter-frame predictor 124 configures a merge list including a predetermined number of merge candidates by utilizing neighboring blocks. It selects a merge candidate from the merge candidates included in the merge list to be used as motion information for the current block, and generates merge index information for identifying the selected candidate. The generated merge index information is encoded by the entropy encoder 155 and transmitted to the video decoding device.
[0056] The merge skip mode is a special case of the merge mode. After quantization, when all transform coefficients used for entropy coding are close to zero, only the adjacent block selection information is transmitted, and the residual signal is not transmitted. By utilizing the merge skip mode, relatively high coding efficiency can be achieved for images with slight motion, still images, and screen content images.
[0057] From then on, merge mode and merge skip mode were collectively referred to as merge / skip mode.
[0058] Another approach for encoding motion information is the Advanced Motion Vector Prediction (AMVP) model.
[0059] In AMVP mode, the inter-frame predictor 124 derives motion vector prediction candidates for the current block by utilizing neighboring blocks of the current block. As neighboring blocks used to derive motion vector prediction candidates, they can use... Figure 4 The current block in the current image shown includes all or some of the adjacent blocks: left block A0, lower left block A1, upper block B0, upper right block B1, and upper left block B2. Furthermore, blocks located in a reference image (which may be the same as or different from the reference image used to predict the current block) can also be used as neighboring blocks for deriving motion vector prediction candidates, in addition to the current image containing the current block. For example, a co-located block of the current block in the reference image or a block adjacent to that co-located block can be used. If the number of motion vector candidates selected by the above method is less than a preset number, a zero vector is added to the motion vector candidates.
[0060] The inter-frame predictor 124 derives motion vector prediction candidates by utilizing motion vectors from neighboring blocks, and determines the motion vector prediction for the current block by utilizing these candidates. Furthermore, the motion vector difference is calculated by subtracting the motion vector prediction from the motion vector prediction for the current block.
[0061] Motion vector predictions can be obtained by applying predefined functions (e.g., median and average calculations) to motion vector prediction candidates. In this case, the video decoding device also knows the predefined functions. Furthermore, since the neighboring blocks used to derive the motion vector prediction candidates are blocks that have already been encoded and decoded, the video decoding device may also already know the motion vectors of the neighboring blocks. Therefore, the video encoding device does not need to encode the information used to identify the motion vector prediction candidates. Accordingly, in this case, information about the motion vector difference and information about the reference image used to predict the current block are encoded.
[0062] Alternatively, motion vector prediction can be determined by selecting any of the motion vector prediction candidates. In this case, additional encoding is performed on the information used to identify the selected motion vector prediction candidate, as well as information about the motion vector difference and information about the reference image used to predict the current block.
[0063] Subtractor 130 generates a residual block by subtracting the predicted block generated by intra-predictor 122 or inter-predictor 124 from the current block.
[0064] Transformer 140 transforms the residual signal in the residual block, which has pixel values in the spatial domain, into transform coefficients in the frequency domain. Transformer 140 can transform the residual signal in the residual block by using the entire size of the residual block as a transform unit, or it can divide the residual block into multiple sub-blocks and perform the transformation by using the sub-blocks as transform units. Alternatively, the residual block is divided into two sub-blocks (i.e., a transformed region and a non-transformed region) to transform the residual signal by using only the transformed region sub-block as a transform unit. Here, the transformed region sub-block can be one of two rectangular blocks with a size ratio of 1:1 based on a horizontal axis (or a vertical axis). In this case, the entropy encoder 155 encodes a flag (cu_sbt_flag) indicating that only the transformed sub-block is used, as well as orientation (vertical / horizontal) information (cu_sbt_horizontal_flag) and / or position information (cu_sbt_pos_flag), and signals this information to the video decoding device. Furthermore, the size of the transformed region sub-blocks can be based on a 1:3 size ratio along the horizontal (or vertical) axis. In this case, the entropy encoder 155 additionally encodes the flag (cu_sbt_quad_flag) for dividing the corresponding segment and signals it to the video decoding device.
[0065] On the other hand, the transformer 140 can perform transformations on the residual blocks individually in the horizontal and vertical directions. Various types of transformation functions or transformation matrices can be used for this transformation. For example, a pair of transformation functions for horizontal and vertical transformations can be defined as a Multiple Transform Set (MTS). The transformer 140 can select the transformation function pair with the highest transformation efficiency in the MTS and can transform the residual block in each of the horizontal and vertical directions. The entropy encoder 155 encodes the information (mts_idx) about the transformation function pair in the MTS and signals it to the video decoding device.
[0066] Quantizer 145 quantizes the transform coefficients output from transformer 140 using quantization parameters and outputs the quantized transform coefficients to entropy encoder 155. Quantizer 145 can also quantize the relevant residual blocks immediately without transforming any block or frame. Quantizer 145 can also apply different quantization coefficients (scaling values) based on the position of the transform coefficients in the transform block. The quantization matrix applied to the quantized transform coefficients arranged in two dimensions can be encoded and signaled to the video decoding device.
[0067] The reordering unit 150 can reorder the coefficient values of the quantized residual values.
[0068] The reordering unit 150 can transform a 2D coefficient array into a 1D coefficient sequence using coefficient scanning. For example, the reordering unit 150 can output a 1D coefficient sequence by scanning the DC coefficients to high-frequency region coefficients using a zig-zag scan or a diagonal scan. Depending on the size of the transform unit and the intra-frame prediction mode, a vertical scan scanning the 2D coefficient array in the column direction and a horizontal scan scanning the 2D block-type coefficients in the row direction can also be used instead of a zig-zag scan. In other words, the scanning method to be used can be determined from zig-zag scan, diagonal scan, vertical scan, and horizontal scan, depending on the size of the transform unit and the intra-frame prediction mode.
[0069] The entropy encoder 155 encodes the 1D quantized transform coefficient sequence output from the reordering unit 150 by utilizing various encoding schemes, including Context-based Adaptive Binary Arithmetic Code (CABAC) and Exponential Golomb, to generate a bitstream.
[0070] Furthermore, the entropy encoder 155 encodes block-segmentation related information (e.g., CTU size, CTU segmentation flag, QT segmentation flag, MTT segmentation type, MTT segmentation direction, etc.) so that the video decoding device can segment blocks equivalently to the video encoding device. Additionally, the entropy encoder 155 encodes information indicating whether the current block is encoded using intra-frame prediction or inter-frame prediction. The entropy encoder 155 encodes intra-frame prediction information (i.e., information about the intra-frame prediction mode) or inter-frame prediction information (the merge index in the case of merge mode, and information about the reference image index and motion vector difference in the case of AMVP mode) according to the prediction type. Furthermore, the entropy encoder 155 encodes quantization-related information (i.e., information about quantization parameters and information about the quantization matrix).
[0071] Inverse quantizer 160 inverse quantizes the quantized transform coefficients output from quantizer 145 to generate transform coefficients. Inverse transformer 165 transforms the transform coefficients output from inverse quantizer 160 from the frequency domain to the spatial domain to reconstruct the residual block.
[0072] Adder 170 adds the reconstructed residual block to the prediction block generated by predictor 120 to reconstruct the current block. When performing intra-frame prediction for the next block, the pixels in the reconstructed current block can be used as reference pixels.
[0073] The loop filtering unit 180 performs filtering on the reconstructed pixels to reduce blocking artifacts, ringing artifacts, blurring artifacts, etc., that occur due to block-based prediction and transform / quantization. The loop filtering unit 180, as an in-loop filter, may include all or some of the following: a deblocking filter 182, a sample adaptive offset (SAO) filter 184, and an adaptive loop filter (ALF) 186.
[0074] Deblocking filter 182 filters the boundaries between reconstructed blocks to remove blocking artifacts caused by block unit encoding / decoding, and SAO filter 184 and ALF 186 perform additional filtering on the deblocked video. SAO filter 184 and ALF 186 are filters used to compensate for the difference between reconstructed pixels and original pixels caused by lossy coding. SAO filter 184 applies an offset in CTU units to enhance subjective image quality and coding efficiency. On the other hand, ALF 186 performs block unit filtering and applies different filters to compensate for distortion by dividing the boundaries of the corresponding blocks and the degree of variation. Information about the filter coefficients to be used for ALF can be encoded and signaled to the video decoding device.
[0075] The reconstructed blocks filtered by deblocking filter 182, SAO filter 184, and ALF 186 are stored in memory 190. When all blocks in an image are reconstructed, the reconstructed image can be used as a reference image for inter-frame prediction of blocks in subsequent images to be encoded.
[0076] Video encoding devices can store the bitstream of encoded video data in a non-volatile storage medium or send the bitstream to a video decoding device via a communication network.
[0077] Figure 5 This is a functional block diagram of a video decoding device that can implement the technology of the present invention. In the following text, reference is made to... Figure 5 This describes a video decoding device and its components.
[0078] The video decoding device may include: an entropy decoder 510, a reordering unit 515, an inverse quantizer 520, an inverse transformer 530, a predictor 540, an adder 550, a loop filter unit 560, and a memory 570.
[0079] Similar to Figure 1 The video encoding device and the video decoding device each component can be implemented as hardware or software, or a combination of hardware and software. Furthermore, the function of each component can be implemented as software, and the microprocessor can also be implemented to execute the software functions corresponding to each component.
[0080] The entropy decoder 510 extracts information related to block segmentation by decoding the bitstream generated by the video encoding device to determine the current block to be decoded, and extracts the prediction information and information about the residual signal required to reconstruct the current block.
[0081] The entropy decoder 510 determines the size of the CTU by extracting information about the CTU size from the sequence parameter set (SPS) or the picture parameter set (PPS), and segments the image into CTUs of a defined size. Furthermore, the CTU is identified as the highest level (i.e., the root node) of the tree structure, and segmentation information for the CTU can be extracted to utilize the tree structure for CTU segmentation.
[0082] For example, when segmenting a CTU using a QTBTT structure, the first flag (QT_split_flag) related to the QT segmentation is first extracted to split each node into four lower-level nodes. Additionally, a second flag (mtt_split_flag), segmentation direction (vertical / horizontal), and / or segmentation type (binary / ternary) related to the MTT segmentation are extracted relative to the nodes corresponding to the leaf nodes of the QT to segment the corresponding leaf nodes using the MTT structure. Thus, each node below a leaf node of the QT is recursively segmented using either a BT or TT structure.
[0083] As another example, when splitting the CTU using the QTBTTT structure, a CU split flag (split_cu_flag) indicating whether the CU has been split is extracted. A first flag (QT_split_flag) can also be extracted when the corresponding block is split. During the splitting process, for each node, zero or more recursive MTT splits may occur after zero or more recursive QT splits. For example, with respect to the CTU, MTT splits may occur immediately, or conversely, only multiple QT splits may occur.
[0084] As another example, when segmenting a CTU using a QTBT structure, the first flag (QT_split_flag) associated with the QT segmentation is extracted to split each node into four nodes at the lower level. Additionally, a split flag (split_flag) indicating whether the node corresponding to a leaf node of the QT is further segmented using BT, as well as segmentation direction information, is extracted.
[0085] On the other hand, when the entropy decoder 510 determines the current block to be decoded by utilizing the segmentation of the tree structure, the entropy decoder 510 extracts information about the prediction type indicating whether the current block is predicted intra-frame or inter-frame. When the prediction type information indicates intra-frame prediction, the entropy decoder 510 extracts the syntax elements for the intra-frame prediction information (intra-frame prediction mode) for the current block. When the prediction type information indicates inter-frame prediction, the entropy decoder 510 extracts information representing the syntax elements for the inter-frame prediction information, namely, the motion vector and the reference picture for the motion vector reference.
[0086] In addition, the entropy decoder 510 extracts quantization-related information and extracts information about the quantization transform coefficients of the current block as information about the residual signal.
[0087] The reordering unit 515 can again change the 1D quantization transform coefficient sequence entropy decoded by the entropy decoder 510 into a 2D coefficient array (i.e., a block) in the reverse order of the coefficient scan order performed by the video encoding device.
[0088] The inverse quantizer 520 performs inverse quantization on the quantized transform coefficients, and performs inverse quantization on the quantized transform coefficients using quantization parameters. The inverse quantizer 520 can also apply different quantization coefficients (scaling values) to the quantized transform coefficients arranged in 2D. The inverse quantizer 520 can perform inverse quantization by applying a matrix of quantization coefficients (scaling values) from the video encoding device to a 2D array of quantized transform coefficients.
[0089] The inverse transformer 530 reconstructs the residual signal by inversely transforming the inverse-quantized transform coefficients from the frequency domain to the spatial domain, thereby generating the residual block of the current block.
[0090] Furthermore, when the inverse transformer 530 performs an inverse transform on a portion (sub-block) of the transform block, it extracts a flag (cu_sbt_flag) indicating that only the sub-blocks of the transform block are transformed, the orientation (vertical / horizontal) information (cu_sbt_horizontal_flag) of the sub-blocks, and / or the position information (cu_sbt_pos_flag) of the sub-blocks. The inverse transformer 530 also inverse transforms the transform coefficients of the corresponding sub-blocks from the frequency domain to the spatial domain to reconstruct the residual signal, and fills the regions not inversely transformed with values of "0" as the residual signal to generate the final residual block of the current block.
[0091] Furthermore, when applying MTS, the inverse transformer 530 determines the transform function or transform matrix to be applied in both the horizontal and vertical directions by utilizing MTS information (mts_idx) signaled from the video encoding device. The inverse transformer 530 also performs inverse transform on the transform coefficients in the transform block in both the horizontal and vertical directions using the determined transform function.
[0092] Predictor 540 may include intra-predictor 542 and inter-predictor 544. Intra-predictor 542 is activated when the prediction type of the current block is intra-prediction, and inter-predictor 544 is activated when the prediction type of the current block is inter-prediction.
[0093] Intra-predictor 542 determines the intra-prediction mode for the current block among multiple intra-prediction modes based on the syntax elements for the intra-prediction mode extracted from entropy decoder 510. Intra-predictor 542 also predicts the current block by utilizing neighboring reference pixels of the current block according to the intra-prediction mode.
[0094] The inter-frame predictor 544 determines the motion vector and the reference picture of the motion vector reference for the current block by using the syntax elements for the inter-frame prediction mode extracted from the entropy decoder 510, and predicts the current block by using the motion vector and the reference picture.
[0095] Adder 550 reconstructs the current block by adding the residual block output from inverse transform 530 to the predicted block output from inter-frame predictor 544 or intra-frame predictor 542. Pixels within the reconstructed current block are used as reference pixels when performing intra-frame prediction on a subsequent block to be decoded.
[0096] The loop filtering unit 560, acting as an in-loop filter, may include a deblocking filter 562, a SAO filter 564, and an ALF 566. The deblocking filter 562 performs deblocking filtering on the boundaries between reconstructed blocks to remove block artifacts caused by block unit decoding. The SAO filter 564 and ALF 566 perform additional filtering on the reconstructed blocks after deblocking to compensate for differences between reconstructed pixels and original pixels caused by lossy encoding. The filter coefficients of the ALF are determined by utilizing information about the filter coefficients decoded from the bitstream.
[0097] The reconstructed blocks filtered by deblocking filter 562, SAO filter 564, and ALF 566 are stored in memory 570. When all blocks in an image are reconstructed, the reconstructed image can be used as a reference image for inter-frame prediction of blocks in subsequent images to be encoded.
[0098] In some embodiments, the present invention relates to encoding and decoding video images as described above. More specifically, the present invention provides a video encoding / decoding method and apparatus that, when configuring a Most Probable Mode (MPM) list in intra-frame prediction, adds intra-frame modes obtained from images other than the current image to the MPM list.
[0099] The following implementation scheme can be performed by the predictor 120 in the video encoding device. The following implementation scheme can also be performed by the predictor 540 in the video decoding device.
[0100] When encoding the current block, the video encoding apparatus can generate signaling information associated with this embodiment in terms of optimizing rate distortion. The video encoding apparatus can encode the signaling information using entropy encoder 155 and can send the encoded signaling information to the video decoding apparatus. The video decoding apparatus can decode the signaling information associated with the decoding of the current block from the bitstream using entropy decoder 510.
[0101] In the following description, the term "target block" may be used interchangeably with the current block or coding unit (CU), or may refer to certain regions of a coding unit.
[0102] Furthermore, a flag value of true indicates that the flag is set to 1. Conversely, a flag value of false indicates that the flag is set to 0.
[0103] Decoder side includes Figure 1 The video encoding apparatus shown includes all or part of the inverse quantizer 160, inverse transformer 165, predictor 120, adder 170, loop filter unit 180, and memory 190. Alternatively, the decoder side includes... Figure 5 The video decoding apparatus shown includes all or part of the inverse quantizer 520, inverse converter 530, predictor 540, adder 550, loop filter unit 560, and memory 570. Regarding a series of decoding processes, the decoder side of the video encoding apparatus and the decoder side of the video decoding apparatus perform the same operations. The video encoding apparatus determines information related to the operation of the decoder side and notifies the video decoding apparatus of the determined information via a signal. The video decoding apparatus can decode the information notified via the signal and can operate the decoder side based on the decoded information.
[0104] I-1. Intra-frame prediction Intra-frame prediction uses pixel information within the same image to predict the pixel values of the current block to be encoded. In intra-frame prediction, the most suitable mode can be selected from multiple intra-frame prediction modes based on the characteristics of the image, and then the selected mode can be used for prediction of the current block. The video encoding device selects a mode from multiple intra-frame prediction modes and then uses the selected mode to encode the current block. Afterwards, the video encoding device can transmit information about the corresponding mode to the video decoding device.
[0105] HEVC technology uses a total of 35 intra-prediction modes, including 33 directional modes (angle modes) and 2 non-directional modes (non-angle modes) for intra-prediction. However, as image spatial resolution increases from 720×480 to 2048×1024 or 8192×4096, the size of prediction block units also gradually increases, correspondingly increasing the need to add more diverse intra-prediction modes. For example... Figure 3a As shown, VVC technology uses 65 more refined prediction modes for intra-frame prediction, thus enabling more diverse utilization of prediction orientation compared to previous techniques. With the introduction of Wide Angular Intra Prediction (WAIP), for rectangular blocks, intra-frame modes -14 to -1 and intra-frame modes 67 to 80 (which are directional modes with larger angles) can be used depending on the block's aspect ratio.
[0106] According to VVC technology, intra-prediction modes can represent 67 modes. These modes can be signaled in intra-prediction modes ranging from -14 to 80 based on the block aspect ratio, and include planar modes and DC modes for non-directional prediction. In addition to intra-prediction techniques utilizing intra-prediction modes, intra-block copy (IBC), palette, matrix-based intra-prediction (MIP), block-based delta pulse code modulation (BDPCM), and inter-frame prediction techniques can also be used to predict the current block.
[0107] I-2. Most Probable Pattern (MPM) Technique As described above, in intra-frame prediction, the prediction of the luminance channel can be generated based on 67 Intra Prediction Modes (IPMs). The 67 IPMs refer to 67 intra-frame prediction modes, which can be signaled in prediction modes ranging from -14 to 80 depending on the block aspect ratio, and include planar modes and DC modes of non-directional prediction modes. When generating a prediction using one of the 67 prediction modes, the video coding device signals the prediction mode using the Most Probable Mode (MPM) to efficiently transmit the prediction mode information.
[0108] Most Probable Mode (MPM) technology leverages the property that when a block is encoded using an intra-prediction mode, the prediction modes of adjacent blocks are likely to be similar. MPM selects six MPM candidates based on the intra-prediction modes used by neighboring blocks, the nearest neighboring modes, and statistically frequently used modes. The set of these six selected MPM candidates is called the MPM list. When the intra-prediction mode of the current block is included in the MPM list, the MPM index corresponding to the intra-prediction mode of the current block among the candidates included in the list is encoded. Conversely, when the intra-prediction mode of the current block is not included in the six MPM candidates, six MPM candidates are excluded from 67 intra-prediction modes to form the MPM remainder, which is the set of intra-prediction modes. The intra-prediction mode of the current block can then be encoded using the MPM remainder.
[0109] like Figure 6 For example, for a block including pixel A, which is to the left of the bottom-left pixel of the current block, and pixel B, which is above the top-right pixel of the current block, the prediction modes of each block are defined as modeA and modeB. An MPM list can be generated based on modeA and modeB by selecting six MPM candidates as follows. When the current block is located on the boundary of a CTU, tile, slice, sub-image, picture, etc., and pixel A or pixel B is unavailable, the prediction mode of the block including the corresponding pixel is considered planar. Subsequently, an MPM list can be generated according to the following MPM list generation algorithm (hereinafter referred to as the "generation algorithm").
[0110] First, when modeA and modeB are the same, and modeA is greater than INTRA_DC, select {plane, modeA, 2+((modeA+61)%64), 2+((modeA-1)%64), 2+((modeA+60)%64), 2+(modeA%64)} as MPM candidates.
[0111] Next, when modeA and modeB are not the same, and modeA or modeB is greater than INTRA_DC, the MPM candidate configuration is as follows. In this case, minAB = Min(modeA, modeB) and maxAB = Max(modeA, modeB) are defined.
[0112] When both modeA and modeB are greater than INTRA_DC, and maxAB-minAB=1, select {plane, modeA, modeB, 2+((minAB+61)%64), 2+((maxAB-1)%64), 2+((minAB+60)%64)} as MPM candidate.
[0113] When both modeA and modeB are greater than INTRA_DC, and maxAB-minAB≥62, select {Planar, modeA, modeB, 2+((minAB-1)%64), 2+((maxAB+61)%64), 2+(minAB%64)} as MPM candidates.
[0114] When both modeA and modeB are greater than INTRA_DC, and maxAB-minAB=2, select {Planar, modeA, modeB, 2+((minAB-1)%64), 2+((minAB+61)%64), 2+((maxAB-1)%64)} as MPM candidates.
[0115] When both modeA and modeB are greater than INTRA_DC, and 2<maxAB-minAB<62, select {Planar, modeA, modeB, 2+((minAB+61)%64), 2+((minAB-1)%64), 2+((maxAB+61)%64)} as MPM candidates.
[0116] When modeA and modeB are different, and one of modeA and modeB is greater than INTRA_DC, select {Planar, maxAB, 2+((maxAB+61)%64), 2+((maxAB-1)%64), 2+((maxAB+60)%64), 2+(maxAB%64)} as MPM candidates.
[0117] In addition, when both modeA and modeB are equal to or less than INTRA_DC, select {Planar, INTRA_DC, INTRA_ANGULAR50, INTRA_ANGULAR18, INTRA_ANGULAR46, INTRA_ANGULAR54} as MPM candidates.
[0118] On the other hand, when using MPM, the video decoding apparatus parses the intra prediction mode of the current block, as shown in Table 1.
[0119] [Table 1] First, when `intra_luma_ref_idx` (which is the reference line index indicating one of multiple reference lines) is 0, the video encoding device can signal the video decoding device with `intra_luma_mpm_flag` (which is a flag indicating whether MPM is used). When `intra_luma_mpm_flag` is true and `intra_luma_ref_idx` is 0, the video encoding device can signal the video decoding device with `intra_luma_not_planar_flag` (which is a flag indicating whether planar mode is used). When `intra_luma_not_planar_flag` is false, the intra-frame prediction mode is set to planar mode, and when `intra_luma_not_planar_flag` is true, `intra_luma_mpm_idx` can be additionally signaled. If `intra_luma_not_planar_flag` does not exist, it can be inferred to be 1.
[0120] On the other hand, when intra_luma_ref_idx is not 0, planar mode is not used. Therefore, intra_luma_not_planar_flag is not transmitted and is considered true. Additionally, since intra_luma_not_planar_flag is true, intra_luma_mpm_idx can be signaled separately.
[0121] Next, when intra_luma_mpm_flag is false, the remaining MPM is emitted as an intra-prediction mode signal.
[0122] II. Embodiments according to the present invention The encoding of the intra-prediction mode according to the present invention can be efficient, especially for the encoding and decoding of intra-prediction blocks present in inter-frame slices. The occurrence rate of blocks in inter-frame slices encoded by intra-prediction, inter-prediction, and IBC can be represented as shown in Table 2. Table 2 shows the test results based on Random Access (RA) configuration and Quantization Parameter (QP) values 22, 27, 32, and 37 in the software of VVC test model version 22.0.
[0123] [Table 2] The frame rate and intra-frame period (using the period of intra-frame slices) of the videos are shown in Table 2, and the Group of Pictures (GOP) value for all videos is 32. According to Table 2, the proportion of blocks encoded based on intra-frame prediction in inter-frame slices is approximately 1% to 50%, and in some images, it may exceed 50%. Conventional methods related to intra-frame pattern coding mainly use intra-frame prediction blocks present within intra-frame slices, thus adding the intra-frame patterns of neighboring blocks in the current image to the MPM list when configuring the MPM list for the current block. However, as the experimental results in Table 2 show, many blocks within inter-frame slices are also encoded based on intra-frame prediction. In the case of intra-frame prediction blocks within inter-frame slices, there may not be neighboring blocks encoded based on intra-frame prediction. When configuring the MPM list using neighboring blocks in the current image encoded based on intra-frame prediction, the encoding / decoding efficiency related to the intra-frame prediction pattern of the current block may be significantly reduced. Therefore, by utilizing the technique according to the present invention, intra-frame prediction in inter-frame slices can be encoded more efficiently.
[0124] Figure 7 This is an example diagram illustrating the time hierarchy in a hierarchical prediction structure according to an embodiment of the present invention.
[0125] exist Figure 7 In the example, the GOP is 32, and the intra-frame period is 64. L0 represents an intra-frame slice, and all blocks within the intra-frame slice are encoded and decoded according to intra-frame prediction. L1 to L6 represent inter-frame slices, and blocks within the inter-frame slices are encoded and decoded according to inter-frame prediction or intra-frame prediction. Inter-frame slices can use previously decoded images as reference images. Images at the L1 level can use images at the L0 level as reference images, and images at the L2 level can use images at both the L0 and L1 levels as reference images. When encoding according to inter-frame prediction in an inter-frame slice, images existing at levels lower than or equal to the current image can be used as reference images.
[0126] Regarding the image with an intra-frame periodicity value of 64 in Table 2, the occurrence rate predicted according to the hierarchy is as follows: Figure 8 As shown, the lower the level, the higher the utilization rate of intra-frame prediction; in the L1 level of a specific image, 50% or more of the blocks are encoded based on intra-frame prediction. Figure 8 The example illustrates the importance of improving the encoding and decoding efficiency of intra-frame prediction in inter-frame slices.
[0127] Figure 9 This is an example diagram illustrating the time hierarchy in a hierarchical prediction structure according to another embodiment of the present invention.
[0128] exist Figure 9In the example, the GOP is 32, and the intra-frame period is 32. L0 represents an intra-frame slice, and L1 to L5 represent inter-frame slices.
[0129] Regarding the image with an intra-frame periodicity value of 32 in Table 2, the occurrence rate predicted according to the level is as follows: Figure 10 As shown, the lower the level, the higher the utilization rate of intra-frame prediction; in the L1 level of a specific image, 50% or more of the blocks are encoded based on intra-frame prediction. Figure 10 The example illustrates the importance of improving the encoding and decoding efficiency of intra-frame prediction in inter-frame slices.
[0130] In the process of generating an MPM list based on conventional techniques, such as Figure 6 As shown, the anchor pixels used to determine the anchor block are fixed at positions L and A, making intra-frame modes of blocks other than those containing pixels L and A unusable in the MPM list generation process. When configuring the MPM list in an inter-frame slice, there may be many blocks encoded using inter-frame or IBC encoding near the current block. Intra-frame prediction may not be performed correctly when configuring the MPM list according to a conventional scheme. Therefore, obtaining intra-frame modes from images other than the current image and adding the obtained intra-frame modes to the MPM list may be preferable.
[0131] like Figure 11 As shown, when each block including pixels L and A is determined as an anchor block to generate the MPM list, the intra-prediction mode of the anchor block is completely replaced with a plane. Since the two modes of the anchor block are the same non-directional mode, the MPM list is generated as {0 (plane), 1 (DC), 50, 18, 46, 54} according to the above generation algorithm. When the intra-prediction mode of the current block is mode 19, since mode 19, which does not exist in the MPM list, is selected (based on the MPM remainder), the video encoding device must consume many bits to transmit the corresponding mode.
[0132] According to the present invention, such as Figure 12 As shown, when configuring the MPM list, if the intra-frame mode of the reference image is added to the MPM list, mode 19 can be used to generate the MPM list. Since mode 19 selected by the current block exists in the MPM list, the number of bits used to transmit mode 19 is reduced, and encoding / decoding efficiency can be improved.
[0133] This invention addresses the problem present in the example described above: when generating the MPM list for the current block, the encoding / decoding efficiency of intra-frame prediction is reduced due to the presence of many blocks near the current block encoded via inter-frame prediction or IBC. Since the aforementioned problem occurs more frequently, particularly in inter-frame slices, the effects according to this invention may be more pronounced in inter-frame slices.
[0134] The following embodiments are described with a focus on video decoding apparatuses, but can be implemented in the same or similar manner in video encoding apparatuses. Alternatively, the following embodiments are described with a focus on the decoder side of the video decoding apparatus, but can be implemented in the same or similar manner on the decoder side of the video encoding apparatus.
[0135] In this invention, when a reference image is available during the generation of the MPM list, the video decoding device can select the reference image, obtain the intra-frame mode from the selected image, and add the intra-frame mode to the MPM list. Hereinafter, the intra-frame modes present in the reference image are referred to as temporal MPM candidates.
[0136] As an example, an MPM list containing intra-frame modes obtained from a reference image can be used. In this embodiment, the video decoding device can configure the MPM list by adding intra-frame modes obtained from the reference image to the MPM list, and can use the configured MPM list to decode the intra-frame prediction mode of the current block. This embodiment can be implemented in three steps. In the first step, the video decoding device selects a reference image. In the second step, the video decoding device obtains the intra-frame mode from the selected reference image and adds the obtained intra-frame mode to the MPM list. In the third step, the video decoding device decodes the intra-frame prediction mode of the current block based on the MPM list.
[0137] In the first step, the video decoding device selects a reference image from the list of reference images. The video decoding device can select one or more reference images.
[0138] As an example, the video decoding device can select a predetermined reference picture. The video decoding device can select a predetermined reference picture, or it can use a reference picture indicated at a higher level (e.g., a picture header or slice header). When a predetermined reference picture is selected, a co-position picture, a specific picture in a reference picture list, etc., can be used as the predetermined reference picture. The co-position picture can be a reference picture set at a higher level (e.g., a picture header or slice header) for motion vector prediction generation. A specific picture in the reference picture list can be a specific picture in reference picture list 0 (L0) (e.g., the first picture), or a specific picture in reference picture list 1 (L1) (e.g., the second picture). Which reference picture list to use and which picture within the corresponding reference picture list to use can be predefined according to the protocol between the video encoding device and the video decoding device.
[0139] Figures 13a to 13c This is an example diagram showing a selection of reference images according to an embodiment of the present invention.
[0140] As another example, the video decoding device can select a reference image of an inter-frame prediction block near the current block. A neighboring block of the current block can refer to a block existing within a predetermined distance of the current block in the same image that includes the current block. In this case, the neighboring block can include an anchor block within the same image.
[0141] When the reference image is a single image, such as Figure 13a In the example, the video decoding device can select a corresponding reference picture. When the reference picture for a block near the current block that is inter-frame predictive coded is a single reference picture, the video decoding device can select the corresponding reference picture. Figure 13a In the reference image index RefIdx, the reference image is indicated within the reference image list (L0 or L1).
[0142] When there are multiple reference images, such as Figure 13b In the example, the video decoding device can select a single reference image that satisfies predetermined conditions. For example, the video decoding device can select a single reference image that satisfies the condition of having the highest weight. When multiple reference images exist, the weight represents the value of the weighted sum used for the reference block during bidirectional prediction. Figure 13b In this context, w0=1 and w1=7, therefore the video decoding device selects reference image 1, which includes the reference block with the highest weight. Figure 13b For example, a video decoding device may use images including the reference block with the highest weight, the image with the smallest difference in Picture Order Count (POC) from the current image, the image existing in L0, the image existing in L1, etc., as predetermined conditions.
[0143] When there are multiple reference images, such as Figure 13c In the example, the video decoding device can select multiple reference images that meet predetermined conditions. For instance, if a block coded according to inter-frame prediction exists near the current block, and two reference images related to the corresponding block exist, the video decoding device can select two reference images. Figure 13c For example, a video decoding device may use all reference images for bidirectional prediction, including images of reference blocks with weights (e.g., greater than a predetermined threshold), images with small POC differences (e.g., less than a predetermined threshold) from the current image, etc., as predetermined conditions.
[0144] In the second step, the video decoding device can obtain intra-frame modes from the reference images and add the obtained intra-frame modes to the MPM list. The video decoding device can obtain one or more intra-frame modes from each reference image and add the obtained intra-frame modes to the MPM list.
[0145] As an example, a video decoding device can set a pixel at a predetermined position in a selected reference image as an anchor pixel, obtain the intra-frame mode from the set anchor pixel, and add the obtained intra-frame mode to the MPM list. Figure 14 As shown, the predetermined position can be within a block that is at the same position as the current block, a neighboring position, a non-neighboring position, etc. The video decoding device can set the pixels at each position as anchor pixels, obtain the intra-frame mode based on the pixels at the predetermined positions in the selected reference picture, and add the obtained intra-frame mode to the MPM list.
[0146] As another example, the video decoding device can set anchor pixels in a selected reference image by utilizing motion information near the current block, and can add the intra-frame mode of the set anchor pixels to the MPM list. The video decoding device can use motion information near the current block to set pixels that are separated by block vectors (BV) or motion vectors (MV) from blocks in the reference image that are at the same position as the current block as anchor pixels, obtain the intra-frame mode from the set anchor pixels, and add the obtained intra-frame mode to the MPM list.
[0147] Figure 15a and Figure 15b This is an example diagram illustrating the acquisition of intra-frame mode according to an embodiment of the present invention.
[0148] If the block decoded according to IBC or inter-frame prediction exists near the current block, the video decoding device can obtain the intra-frame mode from the reference picture by utilizing the block vector or motion vector of the corresponding block, and can add the obtained intra-frame mode to the MPM list.
[0149] exist Figure 15a In this process, the video decoding device utilizes the block vectors of blocks near the current block. When decoding anchor blocks near the current block according to the IBC, the video decoding device can obtain the intra-frame mode of blocks separated from the co-located blocks of the current block in the reference image by the block vectors of the anchor blocks, and can add the obtained intra-frame mode to the MPM list. Figure 15a In this process, intra-frame mode 52, which can be used as a temporal MPM candidate, can be added to the MPM list.
[0150] exist Figure 15b In this process, the video decoding device utilizes motion vectors near the current block. When decoding anchor blocks near the current block based on inter-frame prediction, the video decoding device can obtain the intra-frame mode of blocks separated by the motion vectors of the anchor blocks and blocks in the reference image that are at the same position as the current block, and can add the obtained intra-frame mode to the MPM list. Figure 15b In this process, intra-frame mode 52, which can be used as a temporal MPM candidate, can be added to the MPM list.
[0151] Figures 16a to 16c This is an example diagram illustrating the acquisition of intra-frame mode according to another embodiment of the present invention.
[0152] A video decoding device can obtain an intra-frame mode from a reference image by recursively generating block vectors or motion vectors from the block vectors or motion vectors used by blocks near the current block, and can add the obtained intra-frame mode to the MPM list. The above process can be performed as follows.
[0153] The video decoding device searches for block vectors and motion vectors near the current block and obtains a reference block indicated by the searched block vectors or motion vectors (S10).
[0154] The video decoding device searches for block vectors and motion vectors within the obtained reference block and obtains a reference block indicated by the searched block vectors or motion vectors (S20 to S26).
[0155] The video decoding device finds an intra-prediction block by repeating recursive steps N times (where N is a natural number of 2 or greater). The video decoding device treats the found intra-prediction block as the final reference block and generates a new block vector or motion vector indicating the relationship between the current block and the final reference block (S30). The video decoding device may repeat the recursive steps N times until an intra-prediction block is found. A new block vector or motion vector may be generated based on the N motion information obtained through the recursive search. If no intra-prediction block is found even after repeating the recursive steps N times, the video decoding device may omit adding the intra-prediction mode to the MPM list.
[0156] The video decoding device obtains the intra-frame mode of the block separated from the block in the reference picture that is in the same position as the current block by the new block vector or motion vector, and adds the obtained intra-frame mode to the MPM list (S40).
[0157] exist Figure 16a In this process, the video decoding device utilizes block vectors recursively generated based on block vectors. The video decoding device can recursively generate block vectors from blocks near the current block that are decoded using Intra-Block Copy (IBC), obtain intra-frame modes by utilizing the recursively generated block vectors in a selected reference picture, and add the obtained intra-frame modes to the Most Probable Mode (MPM) list. Figure 16a In this process, intra-frame mode 19, as a candidate for temporal MPM, can be added to the MPM list. As described above, a reference image can be selected based on the first step.
[0158] exist Figure 16bIn this process, the video decoding device utilizes motion vectors recursively generated based on motion vectors. The video decoding device can recursively generate motion vectors from blocks near the current block that are decoded according to inter-frame prediction. Intra-frame modes are obtained by using the recursively generated motion vectors in selected reference images, and the obtained intra-frame modes can be added to the MPM list. Figure 16b In this process, intra-frame mode 54, which can be used as a temporal MPM candidate, can be added to the MPM list. As mentioned above, a reference image can be selected based on the first step.
[0159] exist Figure 16c In this process, the video decoding apparatus utilizes motion vectors recursively generated based on block vectors and motion vectors. The video decoding apparatus obtains motion vectors from blocks near the current block that are decoded according to inter-frame prediction, and obtains block vectors existing within a reference block indicated by the obtained motion vectors or within the reference block indicated by the motion vectors. The video decoding apparatus may repeat this process (e.g., four times) to obtain motion vectors, obtain intra-frame modes by utilizing motion vectors obtained from selected reference images, and add the obtained intra-frame modes to the MPM list. Figure 16c In this process, intra-frame mode 54, which can be used as a temporal MPM candidate, can be added to the MPM list. As mentioned above, a reference image can be selected based on the first step.
[0160] In the third step, the intra-frame prediction mode can be encoded or decoded using an MPM list configured by adding temporal MPM candidates.
[0161] As an example, a video coding apparatus can encode the intra-prediction mode of the current block using an MPM list configured by adding temporal MPM candidates. When the intra-prediction mode is included in the MPM list, the video coding apparatus can encode the index indicating the intra-prediction mode in the MPM list. On the other hand, when the intra-prediction mode is not included in the MPM list, the video coding apparatus can encode the intra-prediction mode based on the remaining MPMs.
[0162] As another example, the video decoding device can decode the intra-prediction mode of the current block using an MPM list configured by adding temporal MPM candidates. When resolving the index indicating the intra-prediction mode in the MPM list, the video decoding device can decode the intra-prediction mode based on the resolved index and the MPM list. If no index is resolved, the video decoding device can decode the intra-prediction mode based on the remaining MPMs.
[0163] On the other hand, although an implementation of adding temporal MPM candidates in inter-frame slices has been described, the method for adding MPM candidates according to the present invention can also be used in intra-frame slices. For example, in the first step, the current image is used instead of the reference image. In the second step, instead of selecting an intra-frame mode from the reference image, an intra-frame mode can be obtained from a reference block within the current image, and the obtained intra-frame mode can be added to the MPM list. In the third step, the intra-frame prediction mode of the current block can be encoded and decoded based on the configured MPM list.
[0164] In the following text, using Figure 17 and Figure 18 The example illustrates how to configure an MPM list by adding temporal MPM candidates to inter-frame slices.
[0165] Figure 17 This is a flowchart illustrating a method for encoding the current block performed by a video encoding apparatus according to an embodiment of the present invention.
[0166] The video encoding device configures the candidate list for the current block (S1700).
[0167] The current block is included in the inter-frame slice and intra-frame prediction is performed. The candidate list is a list of MPMs and may include candidate intra-frame modes selected according to existing algorithms used to generate the MPM list.
[0168] The video encoding device selects at least one reference picture associated with the current block (S1702).
[0169] As an example, a video encoding device may obtain a predetermined reference image or a reference image indicated at a higher level as at least one reference image.
[0170] As another example, the video encoding apparatus can search for inter-frame prediction blocks in blocks that exist within a predetermined distance from the current block, and can obtain at least one reference image of the searched block as at least one reference image.
[0171] As another example, when a searched block has multiple reference images, the video encoding device can obtain the reference image with the highest weight among the multiple reference images.
[0172] The video encoding device obtains at least one intra-frame mode from at least one reference picture (S1704).
[0173] As an example, a video encoding device can set an anchor pixel in a selected reference image to a pixel at a predetermined position. The video encoding device can then obtain at least one intra-frame mode based on the set anchor pixel.
[0174] As another example, the video encoding apparatus can obtain motion information of blocks existing within a predetermined distance from the current block. This motion information can be provided using motion vectors or block vectors. The video encoding apparatus can set anchor pixels within a selected reference frame by applying the motion information to co-located blocks of the current block. Based on the set anchor pixels, the video encoding apparatus can obtain at least one intra-frame mode.
[0175] As another example, the video encoding apparatus can obtain first motion information of blocks existing within a predetermined distance from the current block. Until the Nth motion information (where N is a natural number of 2 or greater) indicates an intra-prediction block, the video encoding apparatus can recursively generate second to Nth motion information based on the first motion information. For each piece of motion information, a motion vector or a block vector can be used. When the Nth motion information indicates an intra-prediction block, the video encoding apparatus can generate new motion information based on the first to Nth motion information. The video encoding apparatus can set anchor pixels within a selected reference picture by applying the new motion information to the co-located blocks of the current block. The video encoding apparatus can obtain at least one intra-frame mode based on the set anchor pixels. On the other hand, when the Nth motion information does not indicate an intra-prediction block, the video encoding apparatus can omit the process of obtaining at least one intra-frame mode.
[0176] The video encoding device updates the candidate list by adding at least one intra-frame mode to the candidate list (S1706).
[0177] The addition of at least one intra-frame mode represents a temporal candidate.
[0178] The video coding apparatus determines the intra-prediction mode for the current block (S1708). For example, in terms of rate-distortion optimization, the video coding apparatus can determine the intra-prediction mode for the current block.
[0179] The video coding unit determines a candidate index corresponding to the intra-prediction mode of the current block based on the updated candidate list (S1710). The candidate index may represent the MPM index.
[0180] The video encoding device encodes the candidate index (S1712).
[0181] Subsequently, the video coding apparatus can generate a prediction block for the current block based on the intra-frame prediction mode, and can generate a residual block by subtracting the prediction block from the current block. The video coding apparatus can generate quantized transform coefficients by applying transform / quantization to the residual block, and can generate a bitstream by encoding the quantized transform coefficients.
[0182] Figure 18 This is a flowchart illustrating a method for reconstructing the current block performed by a video decoding device according to an embodiment of the present invention.
[0183] The video decoding device configures the candidate list for the current block (S1800).
[0184] The current block is included in the inter-frame slice and intra-frame prediction is performed. The candidate list is a list of MPMs and may include candidate intra-frame modes selected according to existing algorithms used to generate the MPM list.
[0185] The video decoding device selects at least one reference image associated with the current block (S1802).
[0186] As an example, a video decoding device may obtain a predetermined reference image or a reference image indicated at a higher level as at least one reference image.
[0187] As another example, the video decoding device can search for inter-frame prediction blocks in blocks that exist within a predetermined distance from the current block, and can obtain at least one reference image of the searched block as at least one reference image.
[0188] As another example, when a searched block has multiple reference images, the video decoding device can obtain the reference image with the highest weight among the multiple reference images.
[0189] The video decoding device obtains at least one intra-frame mode from at least one reference picture (S1804).
[0190] As an example, a video decoding device can set an anchor pixel in a selected reference image to a pixel at a predetermined position. The video decoding device can then obtain at least one intra-frame mode based on the set anchor pixel.
[0191] As another example, the video decoding device can obtain motion information of blocks existing within a predetermined distance from the current block. Motion information can be provided using motion vectors or block vectors. The video decoding device can set anchor pixels within a selected reference frame by applying the motion information to co-located blocks of the current block. Based on the set anchor pixels, the video decoding device can obtain at least one intra-frame mode.
[0192] As another example, the video decoding device can obtain first motion information of blocks existing within a predetermined distance from the current block. Until the Nth motion information (where N is a natural number of 2 or greater) indicates an intra-prediction block, the video decoding device can recursively generate second to Nth motion information based on the first motion information. For each piece of motion information, a motion vector or a block vector can be used. When the Nth motion information indicates an intra-prediction block, the video decoding device can generate new motion information based on the first to Nth motion information. The video decoding device can set anchor pixels within a selected reference picture by applying the new motion information to the co-located blocks of the current block. The video decoding device can obtain at least one intra-frame mode based on the set anchor pixels. On the other hand, when the Nth motion information does not indicate an intra-prediction block, the video decoding device can omit the process of obtaining at least one intra-frame mode.
[0193] The video decoding device updates the candidate list by adding at least one intra-frame mode to the candidate list (S1806).
[0194] The addition of at least one intra-frame mode represents a temporal candidate.
[0195] The video decoding device decodes candidate indices from the bitstream (S1808). Candidate indices can represent MPM indices.
[0196] The video decoding device extracts the intra-prediction mode of the current block from the updated candidate list based on the candidate index (S1810).
[0197] Subsequently, the video decoding device generates a prediction block for the current block based on the intra-frame prediction mode. The video decoding device can decode the quantized transform coefficients from the bitstream and can generate a residual block by applying inverse quantization / inverse transform to the quantized transform coefficients. The video decoding device can generate a reconstructed block for the current block by adding the residual block and the prediction block.
[0198] Although the steps in the various flowcharts are described as being executed sequentially, these steps merely exemplify the technical ideas of some embodiments of the invention. Therefore, those skilled in the art to which this invention pertains can perform these steps by changing the order described in the various figures or by performing two or more steps in parallel. Thus, the steps in the various flowcharts are not limited to the chronological order shown.
[0199] It should be understood that the above description presents illustrative embodiments that can be implemented in various other ways. The functionality described in some embodiments can be implemented by hardware, software, firmware, and / or combinations thereof. It should also be understood that functional components described in this invention are designated as "...units" to emphasize their potential for independent implementation.
[0200] On the other hand, the various methods or functions described in some embodiments can be implemented as instructions stored in a non-volatile recording medium, which can be read and executed by one or more processors. The non-volatile recording medium can include various types of recording devices, such as those storing data in a computer system-readable form. For example, the non-volatile recording medium can include storage media such as erasable programmable read-only memory (EPROM), flash memory drives, optical disk drives, magnetic hard disk drives, and solid-state drives (SSDs).
[0201] Although embodiments of the invention have been described for illustrative purposes, those skilled in the art will understand that various modifications, additions, and substitutions can be made without departing from the spirit and scope of the invention. Therefore, embodiments of the invention have been described for the sake of brevity and clarity. The scope of the technical concept of the embodiments of the invention is not limited by the examples. Accordingly, those skilled in the art will understand that the scope of the invention should not be limited by the embodiments explicitly described above, but rather by the claims and their equivalents.
[0202] Cross-references to related applications This application claims priority and benefit to Korean Patent Application No. 10-2024-0050288, filed on April 15, 2024, and Korean Patent Application No. 10-2025-0044808, filed on April 7, 2025, the entire contents of each of which are incorporated herein by reference.
Claims
1. A method for reconstructing a current block, performed by a video decoding device, the method comprising: Configure a candidate list for the current block, wherein the current block is included in inter-frame slices and intra-frame prediction is performed, and the candidate list includes candidate intra-frame modes; Select at least one reference image related to the current block; Obtain at least one intra-frame mode from at least one reference image; and The candidate list is updated by adding at least one intra-frame mode to the candidate list.
2. The method according to claim 1, further comprising: Decode candidate indices from the bitstream; The intra-block prediction mode is extracted from the updated candidate list based on the candidate index. as well as The prediction block for the current block is generated based on the intra-frame prediction mode.
3. The method according to claim 1, wherein, Select at least one reference image, including: Obtain a predetermined reference image or a reference image indicated by a high-level authority as at least one reference image.
4. The method according to claim 1, wherein, Select at least one reference image, including: Search for inter-frame prediction blocks among blocks that exist within a predetermined distance from the current block; and Obtain at least one reference image of the searched block as at least one reference image.
5. The method according to claim 4, wherein, Select at least one reference image, including: When a searched block has multiple reference images, the reference image with the highest weight among the multiple reference images is obtained.
6. The method according to claim 1, wherein, Obtaining at least one intra-frame mode includes: In the selected reference image, set the anchor pixel to the pixel at the predetermined position; and At least one intra-frame mode is obtained based on the set anchor pixel.
7. The method according to claim 1, wherein, Obtaining at least one intra-frame mode includes: Obtain motion information of blocks that exist within a predetermined distance from the current block; Within the selected reference image, anchor pixels are set by applying motion information to the co-located blocks of the current block; and At least one intra-frame mode is obtained based on the set anchor pixel.
8. The method according to claim 7, wherein, Obtaining exercise information includes: Obtain motion vectors or block vectors as motion information.
9. The method according to claim 1, wherein, Obtaining at least one intra-frame mode includes: Obtain first motion information of blocks existing within a predetermined distance from the current block; and The motion information continues until the Nth (where N is a natural number of 2 or greater) indicates the intra-frame prediction block, and then the second motion information is recursively generated up to the Nth motion information based on the first motion information.
10. The method according to claim 9, wherein, When the Nth motion information indicates an intra-prediction block, obtaining at least one intra-mode further includes: Within the selected reference image, anchor pixels are set by applying the first to Nth motion information to the co-located blocks of the current block; and At least one intra-frame mode is obtained based on the set anchor pixel.
11. A method for encoding a current block, performed by a video encoding apparatus, the method comprising: Configure a candidate list for the current block, wherein the current block is included in inter-frame slices and intra-frame prediction is performed, and the candidate list includes candidate intra-frame modes; Select at least one reference image related to the current block; Obtain at least one intra-frame mode from at least one reference image; and The candidate list is updated by adding at least one intra-frame mode to the candidate list.
12. The method of claim 11, further comprising: Determine the intra-prediction mode for the current block; The candidate index corresponding to the intra-prediction mode of the current block is determined based on the updated candidate list. The prediction block for the current block is generated based on the intra-frame prediction mode; as well as Encode the candidate indexes.
13. The method according to claim 11, wherein, Obtaining at least one intra-frame mode includes: Obtain motion information of blocks that exist within a predetermined distance from the current block; Within the selected reference image, anchor pixels are set by applying motion information to the co-located blocks of the current block; and At least one intra-frame mode is obtained based on the set anchor pixel.
14. The method according to claim 13, wherein, Obtaining exercise information includes: Obtain motion vectors or block vectors as motion information.
15. A method for providing video data to a video decoding device, the method comprising: Encode video data into a bitstream; as well as Transmit the bitstream to the video decoding device. Encoding video data includes: Configure a candidate list for the current block, wherein the current block is included in inter-frame slices and intra-frame prediction is performed, and the candidate list includes candidate intra-frame modes; Select at least one reference image related to the current block; Obtain at least one intra-frame mode from at least one reference image; and The candidate list is updated by adding at least one intra-frame mode to the candidate list.
Citation Information
Patent Citations
Method for AI-based manufacturing process standard integrated management system and monitoring system
KR1020240050288A
Speech enhancement using clustering of cues
KR1020250044808A
