Method and apparatus for video encoding using proximity information based pallete mode
By using a neighborhood-based palette pattern in video encoding to generate palette tables and index maps, the problem of insufficient encoding efficiency and quality for high-resolution and high-frame-rate videos is solved, achieving more efficient video encoding and image enhancement.
Patent Information
- Application Number
- CN202380029217.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2023-03-02
- Filing Date
- 2023-03-03
- Publication Date
- 2024-11-12
AI Technical Summary
Existing video coding technologies are inadequate in terms of coding efficiency and image enhancement when processing high-resolution and high-frame-rate videos, especially when encoding screen content such as animation and computer graphics.
A neighborhood-based palette model is adopted, which reconstructs the current block by generating a palette table and index map, and uses the neighborhood information of the current block for encoding and decoding, selecting the best method to improve encoding efficiency and video quality.
It improves the efficiency and quality of video encoding, especially when processing screen content such as animation and computer graphics, providing higher encoding efficiency and image enhancement effects.
Smart Images

Figure CN118947109A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a method and device for video encoding using a palette mode based on neighboring information. Background Art
[0002] The statements in this section merely provide background information related to the present disclosure and may not constitute prior art.
[0003] Since video data has a large amount of data compared to audio or still image data, without a process for compression, the video data requires a large amount of hardware resources (including memory) to store or transmit the video data.
[0004] Thus, encoders are generally used to compress and store or transmit video data. A decoder receives compressed video data, decompresses the received compressed video data, and plays the decompressed video data. Video compression technologies include H.264 / Advanced Video Coding (AVC), High Efficiency Video Coding (HEVC), and Versatile Video Coding (VVC), which have improved coding efficiency of about 30% or more compared to HEVC.
[0005] However, as image size, resolution and frame rate gradually increase, the amount of data to be encoded also increases. Thus, new compression techniques are needed that provide higher coding efficiency and improved image enhancement effects than existing compression techniques. Specifically, there is a need for more efficient encoding and decoding techniques for video of screen content such as animation and computer graphics. Summary of the invention
[0006]
Technical issues
[0007] The present invention attempts to provide a video encoding method and apparatus for increasing video encoding efficiency and enhancing video quality by utilizing a palette mode based on neighboring information when predicting a current block.
[0008]
Technical solution
[0009] At least one aspect of the present invention provides a method for reconstructing a current block performed by a video decoding device. The method includes generating a palette table for the current block. The method also includes deriving an index map by using neighboring information of the current block. Here, the neighboring information includes a neighboring block vector of the current block or a template within a previously reconstructed neighboring area of the current block. The index map includes a per-sample index of the current block, and the index indicates an entry in the palette table having a color value corresponding to a sample of the current block. The method also includes reconstructing the samples of the current block based on the index map and the palette table.
[0010] Another aspect of the present invention provides a method for encoding a current block performed by a video encoding device. The method includes determining a palette table and deriving an index map according to a first method using neighboring information of the current block. Here, the neighboring information includes a neighboring block vector of the current block or a template within a previously reconstructed neighboring area of the current block. The method also includes determining the palette table and deriving the index map according to a second method using samples within the current block. The method also includes selecting an optimal method between the first method and the second method. The method also includes encoding the palette table according to the optimal method.
[0011] Another aspect of the present invention provides a computer-readable recording medium storing a bitstream generated by a video encoding method. The video encoding method includes determining a palette table and deriving an index map according to a first method using neighboring information of a current block. Here, the neighboring information includes a neighboring block vector of the current block or a template within a previously reconstructed neighboring area of the current block. The video encoding method also includes determining the palette table and deriving the index map according to a second method using samples within the current block. The video encoding method also includes selecting an optimal method between the first method and the second method. The video encoding method also includes encoding the palette table according to the optimal method.
[0012]
Beneficial Effects
[0013] As described above, the present invention provides a video encoding method and device that utilizes a palette mode based on neighboring information when predicting a current block, thereby improving video encoding efficiency and enhancing video quality. BRIEF DESCRIPTION OF THE DRAWINGS
[0014] Figure 1 is a block diagram of a video encoding device that may implement the techniques of this disclosure.
[0015] Figure 2 A method for partitioning blocks using a quadtree plus binary tree ternary tree (QTBTTT) structure is described.
[0016] Figure 3a and 3b Multiple intra prediction modes are shown including a wide-angle intra prediction mode.
[0017] Figure 4 Neighboring blocks of the current block are shown.
[0018] Figure 5 is a block diagram of a video decoding device that may implement the techniques of this disclosure.
[0019] Figure 6 A diagram illustrating a color palette table.
[0020] Figure 7 A diagram illustrating initialization of a palette prediction list when 1-CTU delayed wavefront parallel processing (WPP) is enabled according to at least one embodiment of the present disclosure.
[0021] Figure 8 A diagram illustrating a configuration of a palette table according to at least one embodiment of the present disclosure.
[0022] Fig. 9 A diagram illustrating a palette table including escape symbols.
[0023] Fig.10 FIG. 5 is a diagram illustrating scanning of each coefficient group based on multiple lines according to at least one embodiment of the present disclosure.
[0024] Fig.11 A diagram illustrating index run length encoding for coefficient groups according to at least one embodiment of the present disclosure.
[0025] Fig.12 is a diagram illustrating derivation of an index map based on proximity information according to at least one embodiment of the present disclosure.
[0026] Fig.13 is a diagram illustrating the location of a block vector according to at least one embodiment of the present disclosure.
[0027] Fig.14 is a diagram illustrating the signaling of difference values mapped to corresponding indexes in a palette according to at least one embodiment of the present invention.
[0028] Fig.15 is a flowchart of a method for encoding a current block by using a palette mode, performed by a video encoding apparatus according to at least one embodiment of the present invention.
[0029] Fig.16 is a flowchart of a method for decoding a current block using a palette mode, performed by a video decoding device according to at least one embodiment of the present invention. DETAILED DESCRIPTION
[0030] Hereinafter, some embodiments of the present disclosure are described in detail with reference to the accompanying drawings. In the following description, the same reference numerals represent the same elements, although the elements are shown in different drawings. In addition, in the following description of some embodiments, for the purpose of clarity and brevity, detailed descriptions of related known components and functions may be omitted when it is considered to obscure the subject matter of the present disclosure.
[0031] Figure 1 is a block diagram of a video encoding device that can implement the technology of the present disclosure. Figure 1A diagram of a video encoding device and components of the device is described.
[0032] The encoding apparatus may include a picture separator 110 , a predictor 120 , a subtractor 130 , a transformer 140 , a quantizer 145 , a rearrangement unit 150 , an entropy encoder 155 , an inverse quantizer 160 , an inverse transformer 165 , an adder 170 , a loop filter unit 180 , and a memory 190 .
[0033] Each component of the encoding device can be implemented as hardware or software or as a combination of hardware and software. In addition, the function of each component can be implemented as software, and a microprocessor can also be implemented to execute the function of the software corresponding to each component.
[0034] A video is composed of one or more sequences including multiple pictures. Each picture is divided into multiple regions, and encoding is performed on each region. For example, a picture is divided into one or more tiles or / and slices. Here, one or more tiles can be defined as tile groups. Each tile or / and slice is divided into one or more coding tree units (CTUs). In addition, each CTU is divided into one or more coding units (CUs) through a tree structure. The information applied to each coding unit (CU) is encoded as the syntax of the CU, and the information commonly applied to the CU contained in a CTU is encoded as the syntax of the CTU. In addition, the information commonly applied to all blocks in a slice is encoded as the syntax of the slice header, and the information applied to all blocks constituting one or more pictures is encoded as a picture parameter set (PPS) or a picture header. In addition, the information commonly referenced by multiple pictures is encoded into a sequence parameter set (SPS). In addition, the information commonly referenced by one or more SPSs is encoded into a video parameter set (VPS). Further, the information commonly applied to a tile or tile group can also be encoded as the syntax of the tile or tile group header. The syntax included in the SPS, PPS, slice header, tile, or tile group header may be referred to as a high-level syntax.
[0035] The picture partitioner 110 determines the size of a coding tree unit (CTU). Information on the size of the CTU (CTU size) is encoded as a syntax of an SPS or a PPS and delivered to a video decoding apparatus.
[0036] The picture divider 110 divides each picture constituting a video into a plurality of coding tree units (CTUs) having a predetermined size, and then recursively divides the CTUs by using a tree structure. A leaf node in the tree structure becomes a coding unit (CU), which is a basic unit of encoding.
[0037] The tree structure may be a quadtree (QT), in which a higher node (or parent node) is divided into four lower nodes (or child nodes) of the same size. The tree structure may also be a binary tree (BT), in which a higher node is divided into two lower nodes. The tree structure may also be a ternary tree (TT), in which a higher node is divided into three lower nodes at a ratio of 1:2:1. The tree structure may also be a structure in which two or more of the QT structure, the BT structure, and the TT structure are mixed. For example, a quadtree plus a binary tree (QTBT) structure may be used, or a quadtree plus a binary tree ternary tree (QTBTTT) structure may be used. Here, a binary tree ternary tree (BTTT) is added to the tree structure to be referred to as a multi-type tree (MTT).
[0038] Figure 2 is a diagram for describing a method of partitioning a block by using a QTBTTT structure.
[0039] like Figure 2 As shown, the CTU can be first split into a QT structure. The quadtree partitioning can be recursive until the size of the partitioned block reaches the minimum block size (MinQTSize) of the leaf node allowed in QT. The first flag (QT_split_flag) indicating whether each node of the QT structure is divided into four nodes of the lower layer is encoded by the entropy encoder 155 and transmitted to the video decoding device. When the leaf node of QT is not larger than the maximum block size (MaxBTSize) of the root node allowed in BT, the leaf node can be further divided into at least one of the BT structure or the TT structure. There may be multiple partitioning directions in the BT structure and / or the TT structure. For example, there may be two directions, namely, the direction in which the blocks of the corresponding node are divided horizontally and the direction in which the blocks of the corresponding node are divided vertically. As Figure 2 As shown, when MTT splitting starts, a second flag (mtt_split_flag) indicating whether a node is split and a flag further indicating the split direction (vertical or horizontal) and / or a flag indicating the split type (binary or ternary) if the node is split are encoded by the entropy encoder 155 and notified to the video decoding device with a signal.
[0040] Alternatively, before encoding the first flag (QT_split_flag) indicating whether each node is divided into four nodes of the lower layer, the CU split flag (split_cu_flag) indicating whether the node is divided may also be encoded. When the value of the CU split flag (split_cu_flag) indicates that each node is not split, the block of the corresponding node becomes a leaf node in the split tree structure and becomes a CU as a basic unit of encoding. When the value of the CU split flag (split_cu_flag) indicates that each node is split, the video encoding device first starts encoding the first flag through the above scheme.
[0041] When QTBT is used as another example of a tree structure, there may be two types, namely, a type in which the block of the corresponding node is horizontally divided into two blocks of the same size (i.e., symmetrical horizontal division) and a type in which the block of the corresponding node is vertically divided into two blocks of the same size (i.e., symmetrical vertical division). A split flag (split_flag) indicating whether each node of the BT structure is split into blocks of the lower layer and split type information indicating the split type are encoded by the entropy encoder 155 and delivered to the video decoding device. At the same time, a type in which the block of the corresponding node is split into two blocks that are asymmetric to each other may be presented in addition. The asymmetric form may include a form in which the block of the corresponding node is divided into two rectangular blocks having a size ratio of 1:3, or may also include a form in which the block of the corresponding node is divided in a diagonal direction.
[0042] A CU may have different sizes depending on the QTBT or QTBTTT partition from a CTU. Hereinafter, a block corresponding to a CU to be encoded or decoded (i.e., a leaf node of a QTBTTT) is referred to as a "current block". Due to the QTBTTT partition, the shape of the current block may be a rectangular shape in addition to a square shape.
[0043] The predictor 120 predicts the current block to generate a predicted block. The predictor 120 includes an intra predictor 122 and an inter predictor 124.
[0044] In general, each of the current blocks in a picture may be predictively encoded. Typically, the prediction of the current block may be performed using intra-prediction techniques (using data from the picture including the current block) or inter-prediction techniques (using data from a picture coded before the picture including the current block). Inter-prediction includes both unidirectional prediction and bidirectional prediction.
[0045] The intra-frame predictor 122 predicts the pixels in the current block by using the pixels (reference pixels) located on the neighbors of the current block in the current picture including the current block. According to the prediction direction, there are multiple intra-frame prediction modes. For example, as shown in FIG. 3A, multiple intra-frame prediction modes may include 2 non-directional modes including a plane mode and a DC mode, and may include 65 directional modes. Neighboring pixels and the arithmetic equations to be used are defined differently according to each prediction mode.
[0046] In order to perform effective directional prediction on the current block having a rectangular shape, a directional mode (#67 to #80, intra prediction mode #-1 to #-14) as illustrated by the dotted arrows in FIG. 3B may be additionally used. The directional mode may be referred to as a "wide-angle intra prediction mode". In FIG. 3B, the arrow indicates the corresponding reference sample used for prediction and does not indicate the prediction direction. The prediction direction is opposite to the direction indicated by the arrow. When the current block has a rectangular shape, the wide-angle intra prediction mode is a mode in which prediction is performed in a direction opposite to a specific directional mode without additional bit transmission. In this case, in the wide-angle intra prediction mode, some wide-angle intra prediction modes that can be used for the current block may be determined by the ratio of the width and height of the current block having a rectangular shape. For example, when the current block has a rectangular shape with a height less than the width, a wide-angle intra prediction mode (intra prediction mode #67 to #80) with an angle less than 45 degrees is available. When the current block has a rectangular shape with a width greater than the height, a wide-angle intra prediction mode with an angle greater than -135 degrees may be used.
[0047] The intra-frame predictor 122 may determine an intra-frame prediction to be used for encoding the current block. In some embodiments, the intra-frame predictor 122 may encode the current block by using a plurality of intra-frame prediction modes, and may further select an appropriate intra-frame prediction mode to be used from a test mode. For example, the intra-frame predictor 122 may calculate a rate-distortion value by using a rate-distortion analysis for a plurality of tested intra-frame prediction modes, and may further select an intra-frame prediction mode having the best rate-distortion characteristics in the test mode.
[0048] The intra-frame predictor 122 selects an intra-frame prediction mode from among a plurality of intra-frame prediction modes and predicts the current block by using adjacent pixels (reference pixels) and an arithmetic equation determined according to the selected intra-frame prediction mode. Information about the selected intra-frame prediction mode is encoded by the entropy encoder 155 and delivered to the video decoding device.
[0049] The inter-frame predictor 124 generates a prediction block for the current block by using a motion compensation process. The inter-frame predictor 124 searches for a block most similar to the current block in a reference picture that is encoded and decoded earlier than the current picture, and generates a prediction block for the current block by using the searched block. In addition, a motion vector (MV) is generated, which corresponds to the displacement between the current block in the current picture and the prediction block in the reference picture. Typically, motion estimation is performed on the luminance component, and the motion vector calculated based on the luminance component is used for both the luminance component and the chrominance component. Motion information including information about the reference picture and information about the motion vector for predicting the current block is encoded by the entropy encoder 155 and delivered to the video decoding device.
[0050] The inter-frame predictor 124 may also perform interpolation on a reference picture or a reference block in order to increase the accuracy of the prediction. In other words, the subsamples between two consecutive integer samples are interpolated by applying the filter coefficients to a plurality of consecutive integer samples including two integer samples. When the process of searching for the block most similar to the current block is performed for the interpolated reference picture, the decimal unit precision may be used instead of the integer sample unit precision for the motion vector representation. The precision or resolution of the motion vector may be set differently for each target region to be encoded (e.g., units such as slices, tiles, CTUs, CUs, etc.). When this adaptive motion vector resolution (AMVR) is applied, information about the motion vector resolution to be applied to each target region should be signaled for each target region. For example, when the target region is a CU, information about the motion vector resolution applied to each CU is signaled. The information about the motion vector resolution may be information representing the precision of the motion vector difference to be described below.
[0051] Meanwhile, the inter-frame predictor 124 may perform inter-frame prediction by using bidirectional prediction. In the case of bidirectional prediction, two reference pictures and two motion vectors representing the block position most similar to the current block in each reference picture are used. The inter-frame predictor 124 selects a first reference picture and a second reference picture from a reference picture list 0 (RefPicList0) and a reference picture list 1 (RefPicList1), respectively. The inter-frame predictor 124 also searches for a block most similar to the current block in the corresponding reference picture to generate a first reference block and a second reference block. In addition, a prediction block of the current block is generated by averaging or weighted averaging the first reference block and the second reference block. In addition, motion information including information about two reference pictures for predicting the current block and including information about two motion vectors is delivered to the entropy encoder 155. Here, the reference picture list 0 may be composed of pictures before the current picture in the display order among the pre-reconstructed pictures, and the reference picture list 1 may be composed of pictures after the current picture in the display order among the pre-reconstructed pictures. However, although not particularly limited thereto, a pre-reconstructed picture after the current picture in the display order may be additionally included in the reference picture list 0. Conversely, a pre-reconstructed picture before the current picture may also be additionally included in the reference picture list 1.
[0052] In order to minimize the number of bits consumed for encoding motion information, various methods may be used.
[0053] For example, when the reference picture and motion vector of the current block are the same as those of the adjacent block, information capable of identifying the adjacent block is encoded to deliver the motion information of the current block to the video decoding device. This method is called merge mode.
[0054] In the merge mode, the inter predictor 124 selects a predetermined number of merge candidate blocks (hereinafter, referred to as “merge candidates”) from the neighboring blocks of the current block.
[0055] As a neighboring block for deriving merge candidates, such as Figure 4 As illustrated in FIG, all or some of the left block A0, the lower left block A1, the upper block B0, the upper right block B1, and the upper left block B2 adjacent to the current block in the current picture may be used. Further, blocks other than the current picture where the current block is located, which are located within a reference picture (which may be the same as or different from the reference picture used to predict the current block), may also be used as merge candidates. For example, a block located at the same location as the current block or a block adjacent to the block located at the same location in the reference picture may be additionally used as a merge candidate. If the number of merge candidates selected by the method described above is less than the preset number, a zero vector is added to the merge candidate.
[0056] The inter-frame predictor 124 configures a merge list including a predetermined number of merge candidates by using adjacent blocks. A merge candidate to be used as motion information of the current block is selected from the merge candidates included in the merge list, and merge index information for identifying the selected candidate is generated. The generated merge index information is encoded by the entropy encoder 155 and delivered to the video decoding device.
[0057] Merge skip mode is a special case of merge mode. After quantization, when all transform coefficients for entropy coding are close to zero, only the neighbor block selection information is transmitted without the residual signal. By using merge skip mode, relatively high coding efficiency can be achieved for pictures with slight motion, still pictures, screen content pictures, etc.
[0058] Hereinafter, the merge mode and the merge skip mode are collectively referred to as the merge / skip mode.
[0059] Another method for encoding motion information is the Advanced Motion Vector Prediction (AMVP) mode.
[0060] In the AMVP mode, the inter-frame predictor 124 derives a motion vector predictor candidate for a motion vector of a current block by using neighboring blocks of the current block. As neighboring blocks for deriving motion vector predictor candidates, the same as in Figure 4All or some of the left block A0, lower left block A1, upper block B0, upper right block B1 and upper left block B2 adjacent to the current block in the current picture illustrated in the figure. In addition, blocks other than the current picture where the current block is located, which are located in a reference picture (which may be the same as or different from the reference picture used to predict the current block), may also be used as adjacent blocks for deriving motion vector prediction sub-candidates. For example, blocks co-located with the current block in the reference picture or blocks adjacent to the co-located blocks may be used. If the number of motion vector candidates selected by the above method is less than the preset number, a zero vector is added to the motion vector candidates.
[0061] The inter predictor 124 derives a motion vector predictor candidate by using the motion vectors of the neighboring blocks, and determines a motion vector predictor for the motion vector of the current block by using the motion vector predictor candidate. In addition, a motion vector difference is calculated by subtracting the motion vector predictor from the motion vector of the current block.
[0062] The motion vector predictor can be obtained by applying a predefined function (e.g., center value and average value calculation, etc.) to the motion vector predictor candidate. In this case, the video decoding device also knows the predefined function. In addition, since the neighboring blocks used to derive the motion vector predictor candidate are blocks for which encoding and decoding have been completed, the video decoding device may also already know the motion vectors of the neighboring blocks. Therefore, the video encoding device does not need to encode the information for identifying the motion vector predictor candidate. Thus, in this case, information about the motion vector difference and information about the reference picture used to predict the current block are encoded.
[0063] Meanwhile, the motion vector predictor may also be determined by selecting any one of the motion vector predictor candidates. In this case, information for identifying the selected motion vector predictor candidate is additionally encoded jointly with information about the motion vector difference and information about the reference picture used to predict the current block.
[0064] The subtractor 130 generates a residual block by subtracting a prediction block generated by the intra predictor 122 or the inter predictor 124 from the current block.
[0065] The transformer 140 converts the residual signal in the residual block having the pixel value in the spatial domain into a transform coefficient in the frequency domain. The transformer 140 may transform the residual signal in the residual block by using the total size of the residual block as a transform unit, or may further divide the residual block into a plurality of sub-blocks, and may perform the transform by using the sub-block as a transform unit. Optionally, the residual block is divided into two sub-blocks, namely a transform region and a non-transform region, so as to transform the residual signal using only the transform region sub-block as a transform unit. Here, the transform region sub-block may be one of two rectangular blocks having a size ratio of 1:1 based on the horizontal axis (or vertical axis). In this case, the flag (cu_sbt_flag) indicates that only the sub-block is transformed, and the directional (vertical / horizontal) information (cu_sbt_horizontal_flag) and / or the position information (cu_sbt_pos_flag) are encoded by the entropy encoder 155 and signaled to the video decoding device. In addition, the size of the transform region sub-block may have a size ratio of 1:3 based on the horizontal axis (or vertical axis). In this case, a flag (cu_sbt_quad_flag) of dividing the corresponding division is additionally encoded by the entropy encoder 155 and notified to the video decoding device by signal.
[0066] Meanwhile, the transformer 140 may perform transformation on the residual block separately in the horizontal direction and the vertical direction. For the transformation, different types of transformation functions or transformation matrices may be used. For example, a pair of transformation functions for horizontal transformation and vertical transformation may be defined as a multiple transformation set (MTS). The transformer 140 may select a transformation function pair with the highest transformation efficiency in the MTS, and may transform the residual block in each of the horizontal and vertical directions. Information (mts_idx) of the transformation function pair in the MTS is encoded by the entropy encoder 155 and notified to the video decoding device with a signal.
[0067] The quantizer 145 quantizes the transform coefficients output from the transformer 140 using the quantization parameter and outputs the quantized transform coefficients to the entropy encoder 155. The quantizer 145 can also immediately quantize the relevant residual block without the need for transforming any block or frame. The quantizer 145 can also apply different quantization coefficients (scaling values) according to the position of the transform coefficient in the transform block. The quantization matrix applied to the quantized transform coefficients arranged in 2 dimensions can be encoded and notified to the video decoding device with a signal.
[0068] The rearrangement unit 150 may perform rearrangement of coefficient values for the quantized residual value.
[0069] The rearrangement unit 150 may change the 2D coefficient array into a 1D coefficient sequence by using coefficient scanning. For example, the rearrangement unit 150 may output an ID coefficient sequence by scanning the DC coefficient into a high frequency domain coefficient using a zigzag scan or a diagonal scan. Depending on the size of the transform unit and the intra prediction mode, a vertical scan that scans the 2D coefficient array in the column direction and a horizontal scan that scans the 2D block type coefficients in the row direction may also be used instead of the zigzag scan. In other words, depending on the size of the transform unit and the intra prediction mode, the scanning method to be used may be determined among zigzag scanning, diagonal scanning, vertical scanning, and horizontal scanning.
[0070] The entropy encoder 155 generates a bitstream by encoding the sequence of ID-quantized transform coefficients output from the rearrangement unit 150 using various encoding schemes including context-based adaptive binary arithmetic coding (CABAC), exponential Golomb, and the like.
[0071] In addition, the entropy encoder 155 encodes information related to block partitioning (such as CTU size, CTU split flag, QT split flag, MTT partition type, MTT partition direction, etc.) to allow the video decoding device to equally partition the blocks to the video encoding device. In addition, the entropy encoder 155 encodes information about the prediction type indicating whether the current block is encoded by intra-frame prediction or inter-frame prediction. The entropy encoder 155 encodes intra-frame prediction information (i.e., information about intra-frame prediction mode) or inter-frame prediction information (in the case of merge mode, merge index, and in the case of AMVP mode, information about reference picture index and motion vector difference) according to the prediction type. In addition, the entropy encoder 155 encodes information related to quantization (i.e., information about quantization parameters and information about quantization matrices).
[0072] The inverse quantizer 160 dequantizes the quantized transform coefficient output from the quantizer 145 to generate a transform coefficient. The inverse transformer 165 transforms the transform coefficient output from the inverse quantizer 160 from the frequency domain to the spatial domain to reconstruct a residual block.
[0073] The adder 170 reconstructs the current block by adding the reconstructed residual block and the prediction block generated by the predictor 120. When intra prediction is performed on a next sequential block, pixels in the reconstructed current block may be used as reference pixels.
[0074] The loop filter unit 180 performs filtering on the reconstructed pixels in order to reduce block artifacts, ringing artifacts, blur artifacts, etc. that occur due to block-based prediction and transform / quantization. The loop filter unit 180 as a loop filter may include all or some of the deblocking filter 182, the sample adaptive offset (SAO) filter 184, and the adaptive loop filter (ALF) 186.
[0075] The deblocking filter 182 filters the boundaries between the reconstructed blocks to remove the block effect caused by block unit encoding / decoding, and the SAO filter 184 and the ALF 186 perform additional filtering for the deblocked filtered video. The SAO filter 184 and the ALF 186 are filters for compensating the difference between the reconstructed pixels and the original pixels, which is caused by lossy coding. The SAO filter 184 applies an offset as a CTU unit to enhance subjective picture quality and coding efficiency. On the other hand, the ALF 186 performs block unit filtering and applies different filters to compensate for distortion by dividing the boundaries of the corresponding blocks and the degree of variation. Information about the filter coefficients to be used for the ALF can be encoded and notified to the video decoding device with a signal.
[0076] The reconstructed blocks filtered by the deblocking filter 182, the SAO filter 184, and the ALF 186 are stored in the memory 190. When all blocks in one picture are reconstructed, the reconstructed picture can be used as a reference picture for inter-frame prediction of blocks in a picture to be encoded later.
[0077] Figure 5 is a functional block diagram of a video decoding device that can implement the techniques of the present invention. Figure 5 , describes a video decoding device and components of the device.
[0078] The video decoding apparatus may include an entropy decoder 510 , a rearrangement unit 515 , an inverse quantizer 520 , an inverse transformer 530 , a predictor 540 , an adder 550 , a loop filter unit 560 , and a memory 570 .
[0079] and Figure 1 Similar to the video encoding device of the present invention, each component of the video decoding device can be implemented as hardware or software or as a combination of hardware and software. In addition, the function of each component can be implemented as software, and a microprocessor can also be implemented to execute the function of the software corresponding to each component.
[0080] The entropy decoder 510 extracts information related to block partitioning by decoding a bit stream generated by a video encoding apparatus to determine a current block to be decoded, and extracts prediction information required to reconstruct the current block and information about a residual signal.
[0081] The entropy decoder 510 determines the size of the CTU by extracting information about the CTU size from a sequence parameter set (SPS) or a picture parameter set (PPS), and divides the picture into CTUs of a determined size. In addition, the CTU is determined to be the highest level of the tree structure, i.e., the root node, and the partition information of the CTU can be extracted to partition the CTU using the tree structure.
[0082] For example, when the CTU is divided using the QTBTTT structure, the first flag (QT_split_flag) related to the division of the QT is first extracted to divide each node into four nodes of the lower layer. In addition, for the node corresponding to the leaf node of the QT, the second flag (mtt_split_flag) related to the division of the MTT, the division direction (vertical / horizontal) and / or the division type (binary / ternary) are extracted to divide the corresponding leaf node into the MTT structure. As a result, each node below the leaf node of the QT is recursively split into a BT or TT structure.
[0083] As another embodiment, when a CTU is split by using a QTBTTT structure, a CU split flag (split_cu_flag) indicating whether the CU is split is extracted. When the corresponding block is split, a first flag (QT_split_flag) may also be extracted. During the splitting process, for each node, 0 or more recursive MTT splits may occur after 0 or more recursive QT splits. For example, for a CTU, MTT splitting may occur immediately, or conversely, only multiple QT splits may occur.
[0084] For another example, when the CTU is split using the QTBT structure, the first flag (QT_split_flag) related to the splitting of the QT is extracted to split each node into four nodes of the lower layer. In addition, the split flag (split_flag) indicating whether the node corresponding to the leaf node of the QT is further split into BTs and the split direction information are extracted.
[0085] Meanwhile, when the entropy decoder 510 determines the current block to be decoded by using the partitioning of the tree structure, the entropy decoder 510 extracts information about the prediction type indicating whether the current block is intra-predicted or inter-predicted. When the prediction type information indicates intra-prediction, the entropy decoder 510 extracts syntax elements of intra-prediction information (intra-prediction mode) for the current block. When the prediction type information indicates inter-prediction, the entropy decoder 510 extracts information representing syntax elements for inter-prediction information (i.e., motion vectors and reference pictures referenced by motion vectors).
[0086] Furthermore, the entropy decoder 510 extracts quantization-related information and extracts information on a quantized transform coefficient of the current block as information on a residual signal.
[0087] The rearrangement unit 515 may change the sequence of 1D quantized transform coefficients entropy-decoded by the entropy decoder 510 into a 2D coefficient array (ie, block) again in the reverse order of the coefficient scanning order performed by the video encoding apparatus.
[0088] The inverse quantizer 520 dequantizes the quantized transform coefficients and dequantizes the quantized transform coefficients by using quantization parameters. The inverse quantizer 520 may also apply different quantization coefficients (scaling values) to the quantized transform coefficients arranged in 2D. The inverse quantizer 520 may perform inverse quantization by applying a matrix (scaling value) of quantization coefficients from a video encoding device to a 2D array of quantized transform coefficients.
[0089] The inverse transformer 530 generates a residual block for the current block by reconstructing a residual signal by inversely transforming the dequantized transform coefficients from the frequency domain into the spatial domain.
[0090] In addition, when the inverse transformer 530 inversely transforms a partial region (sub-block) of the transform block, the inverse transformer 530 extracts a flag (cu_sbt_flag) that only the sub-block of the transform block is transformed, direction (vertical / horizontal) information (cu_sbt_horizontal_flag) of the sub-block, and / or position information (cu_sbt_pos_flag) of the sub-block. The inverse transformer 530 also inversely transforms the transform coefficient of the corresponding sub-block from the frequency domain to the spatial domain to reconstruct the residual signal and fills the region that is not inversely transformed with a value of "0" as the residual signal to generate a final residual block for the current block.
[0091] In addition, when MTS is applied, the inverse transformer 530 determines a transform index or a transform matrix applied in each of the horizontal and vertical directions by using MTS information (mts_idx) signaled from the video encoding device. The inverse transformer 530 also performs inverse transform on the transform coefficients in the transform block in the horizontal and vertical directions by using the determined transform function.
[0092] The predictor 540 may include an intra predictor 542 and an inter predictor 544. The intra predictor 542 is activated when the prediction type of the current block is intra prediction, and the inter predictor 544 is activated when the prediction type of the current block is inter prediction.
[0093] The intra predictor 542 determines an intra prediction mode of a current block among a plurality of intra prediction modes according to a syntax element of the intra prediction mode extracted from the entropy decoder 510. The intra predictor 542 also predicts the current block by using neighboring reference pixels of the current block according to the intra prediction mode.
[0094] The inter predictor 544 determines a motion vector of a current block and a reference picture to which the motion vector refers by using a syntax element for the inter prediction mode extracted from the entropy decoder 510 .
[0095] The adder 550 reconstructs the current block by adding the residual block output from the inverse transformer 530 and the prediction block output from the inter predictor 544 or the intra predictor 542. When intra prediction is performed on a block to be decoded later, pixels within the reconstructed current block are used as reference pixels.
[0096] The loop filter unit 560 as a loop filter may include a deblocking filter 562, an SAO filter 564, and an ALF 566. The deblocking filter 562 performs deblocking filtering on the boundary between the reconstructed blocks in order to remove the block effect that occurs due to block unit decoding. The SAO filter 564 and the ALF 566 perform additional filtering on the reconstructed blocks after deblocking filtering in order to compensate for the difference between the reconstructed pixels and the original pixels that occurs due to lossy encoding. The filter coefficients of the ALF are determined by using information about the filter coefficients decoded from the bitstream.
[0097] The reconstructed blocks filtered by the deblocking filter 562, the SAO filter 564, and the ALF 566 are stored in the memory 570. When all blocks in one picture are reconstructed, the reconstructed picture can be used as a reference picture for inter-frame prediction of blocks in a picture to be encoded later.
[0098] In some embodiments, the present invention relates to encoding and decoding a video image as described above. More specifically, the present invention provides a video encoding method and apparatus that utilizes a palette mode based on neighbor information when predicting a current block.
[0099] The following embodiments may be performed by the predictor 120 in the video encoding device. The following embodiments may also be performed by the predictor 540 in the video decoding device.
[0100] The video encoding device in the encoding of the current block may generate signaling information associated with the present embodiment in terms of rate-distortion optimization. The video encoding device may encode the signaling information using the entropy encoder 155 and transmit the encoded signaling information to the video decoding device. The video decoding device may decode the signaling information associated with predicting the current block from the bitstream using the entropy decoder 510.
[0101] In the following description, the term "target block" may be used interchangeably with a current block or a coding unit (CU). The term "target block" may refer to some areas of a coding unit.
[0102] In addition, a flag value of true indicates when the flag is set to 1. In addition, a flag value of false indicates when the flag is set to 0.
[0103] I. Palette Mode
[0104] When certain colors frequently appear in a video such as a screen content video, a palette mode may be applied. In the palette mode, frequently appearing colors are stored in a table format. The video encoding device may transmit an index of a corresponding palette table to a video decoding device, and the video decoding device may then use the parsed index to predict the current block.
[0105] In one example, in the palette mode, the video encoding device will be signaled by using index table information for the value of each pixel that appears K times (an integer greater than 1, where K≤M×N) or more in the current block of size M×N and an index map for mapping to the palette table at each pixel position in the current block. The video decoding device may also parse the table information and the index map, and may perform a prediction process to reconstruct the current block. In this case, K may be defined as a fixed value set according to an agreement between the video encoding device and the video decoding device, or may be adaptively predefined according to the size of the current block.
[0106] The palette mode can be applied to 4:4:4, 4:2:0, 4:2:2, and monochrome formats. When the palette mode is enabled, a flag indicating the activation of the palette mode can be sent at the CU level. The palette mode is applied to blocks 64×64 or smaller, but not to blocks containing 16 or fewer samples. The palette mode is treated as a prediction mode different from intra prediction, inter prediction, and intra block copy (IBC) mode. For example, if the prediction mode of the current block is the palette mode, a reconstructed signal of the current block can be generated based on a prediction process that omits the transform process.
[0107] On the other hand, in the case of a slice using a dual tree with different CU partitions between luma and chroma components, a palette may be used for each color component (e.g., Y palette, Cb palette, Cr palette), or two palettes may be used (e.g., Y palette, Cb / Cr palette). For a single tree, one palette containing all color component (Y, Cb, Cr) values may be used. For monochrome, a single palette may be used.
[0108] For a slice using a single tree, the maximum size of the palette prediction list is 63, and the maximum size of the palette table for the current block is 31. For a dual tree, the maximum size of the palette prediction list and the maximum size of the palette table are halved. That is, for each luma palette and chroma palette, the maximum size of the palette prediction list is 31, and the maximum size of the palette table for the current block is 15. In addition, depending on the color format, if the palette size of the luma component is P, the palette size of Cb and the palette size of Cr may each be P / 2.
[0109] In one example, the size of the palette table may be predetermined based on the size of the current block according to an agreement between the video encoding device and the video decoding device. Alternatively, a fixed size of the palette may be predetermined regardless of the size of the current block.
[0110] When palette mode is used, samples in the current block may be represented by representative color values. The index of the palette may be signaled for locations with sample values close to the palette color. Figure 6 The configuration shown in the example of includes a palette table of indices and corresponding colors. The video encoding device may determine the palette table of the current block, for example, by applying clustering to samples in the current block.
[0111] like Figure 6 As shown in , an entry represents a pair of index and color.
[0112] To populate the palette table, a palette predictor or palette prediction list is maintained. The palette prediction list may have a maximum size that can be transmitted on the SPS, and as described above, the palette prediction list may typically be twice as large as the palette table.
[0113] Initializing a palette prediction list refers to the process of generating a palette prediction list for a first block of a group of video blocks (e.g., a picture, a sub-picture, a slice, a tile, etc.). Since no previous palette prediction list is available for the first block, the palette prediction list for the first block may be initialized to zero. Thus, the entry in the palette table for the first block may be a new entry signaled by the video encoding device.
[0114] Additionally, when wavefront parallel processing (WPP) is enabled, the palette prediction list may need to be initialized in the first CTU (or virtual pipeline data unit (VPDU)) of each CTU row to allow parallel processing of CTU rows. In this case, instead of initializing the palette prediction list to zero, the palette prediction list for the first CTU (or VPDU) of the current CTU row can be initialized by using the palette data from the decoded CTU or VPDU at the top of the current CTU row. In other words, the palette prediction list of the decoded CTU of the top CTU row can be used as the palette prediction list for the first CTU of the current CTU row. For example, if Figure 7 As shown in the example, when 1-CTU delayed WPP (i.e., 4-VPDU delayed WPP) is used, the palette prediction list of the palette table for the first CTU constituting the current CTU row can be initialized by using the palette prediction list of the last VPDU (i.e., the top CTU of the current CTU) that has been decoded in the previous CTU row.
[0115] In order to reuse an entry included in the palette prediction list, the video encoding device may signal a flag indicating whether the entry is to be reused. If the corresponding flag is 1, the corresponding entry is reused in a palette table (hereinafter used interchangeably with "current palette table" or "current palette") of the current block, and if the corresponding flag is 0, the corresponding entry is not reused. A sequence of reuse flags (0 or 1 binary) of an entry in the palette prediction list may be encoded by using run length encoding.
[0116] The video decoding device parses the series of reuse flags and stores those entries in the palette prediction list specified by the reuse flags in the current palette table. If the corresponding flag is 1, the video decoding device includes the corresponding entry in the current palette for reuse, and if the corresponding flag is 0, the corresponding entry is not reused.
[0117] Additionally, the remainder of the current palette table may be implemented using one or more new entries that are explicitly transmitted from the video encoding device or implicitly determined by the video decoding device. In this case, the one or more new entries may be added to the current palette table after the reused entries. If explicitly transmitted, the video encoding device may signal the number of new entries and their corresponding color values to the video decoding device. On the other hand, if all entries in the current palette are implemented by a series of reuse flags, encoding of the new entries may be omitted.
[0118] Figure 8 A diagram illustrating a configuration of a palette table according to at least one embodiment of the present disclosure.
[0119] exist Figure 8 In the example of , the palette prediction list has eight entries and the current palette has four entries. A series of reuse flags indicates that the first and fifth entries of the palette prediction list (i.e., the entries with indices 0 and 4) are included as the first and second entries of the current palette, and the remaining entries of the palette prediction list (i.e., the entries corresponding to indices 1 to 3, 5 to 7) are not included in the current palette. It also indicates that new entries explicitly transmitted from the video encoding device or implicitly determined by the video decoding device are included in the current palette as the third and fourth entries.
[0120] After encoding the current block by using the palette table, the video encoding apparatus may update the palette prediction list. Figure 8As described in , the video encoding device updates the palette prediction list by using the current palette table. The video encoding device may then include any non-reused entries of the previous palette prediction list in the new palette prediction table until the maximum allowed size is reached. After decoding the current block by using the palette table, the video decoding device may update the palette prediction list in the same manner as described above.
[0121] Any sample in the current block may not be the same as or similar to any color contained in the palette. Such samples may not be suitable for encoding based on the palette. Therefore, an escape symbol can be signaled to identify samples outside the color range of the palette. First, the video encoding device can signal a flag indicating whether any sample in the current block is to be encoded to the video decoding device based on the escape symbol. The flag is zero, indicating that no samples in the current block are encoded by using the escape symbol. For example, all samples in the current block can be determined based on the entries contained in the palette table. On the other hand, the flag is 1, indicating that some samples in the current block are encoded by using the escape symbol.
[0122] For samples in the current block that are encoded using an escape symbol, the corresponding sample values may be quantized and sent directly to the video decoding device. If the escape symbol is present in the current block, the size of the palette table may be increased by one, and the last index in the table may be assigned to the escape symbol. Thus, the video encoding device may assign the last index of the palette table, where the index is increased by one to indicate that a particular sample of the current block is encoded with an escape symbol. If the index of a particular sample in the current block is the same as the index assigned to the escape symbol, the video decoding device may reconstruct the escape symbol by decoding the corresponding sample from the bitstream and then dequantizing the decoded corresponding sample.
[0123] Fig. 9 A diagram illustrating a palette table including escape symbols.
[0124] When the flag indicating whether a particular sample in the current block is encoded based on the escape symbol is 1, the last index of the corresponding palette table indicates the escape symbol, such as Fig. 9 As shown in the example of . On the other hand, if the corresponding flag is 1, the video decoding device decodes the escape symbol of the sample indicated by the last index of the palette table, and then dequantizes the decoded escape symbol. This is equivalent to storing the dequantized escape sample at the end of the current palette table if the corresponding flag is 1.
[0125] After generating the palette table, the video encoding device may generate an index map by determining the index of each sample of the current block being encoded according to the palette mode. For example, the video encoding device may derive the index map by using the palette table of the current block in terms of optimizing rate distortion. The video encoding device may then encode the index map by using index run coding. In index run coding, the current block is divided into multiple row-based coefficient groups, and an index run is generated for each coefficient group. The index run is then signaled / parsed. Here, for example, the coefficient group may include 16 samples (m=16). In addition to the index run, the index of the palette table and the quantized escape symbol are signaled / parsed in groups.
[0126] Fig.10 FIG. 5 is a diagram illustrating scanning of each coefficient group based on multiple lines according to at least one embodiment of the present disclosure.
[0127] like Fig.10 As shown in the example of , horizontal or vertical traversal scanning can be used to scan the samples of the current block. Fig.10 In the example of , m=8. The video encoding device may signal a flag indicating whether to use horizontal traversal scanning or vertical traversal scanning. If this flag is 0, the video decoding device may apply horizontal traversal scanning to the current block. On the other hand, if this flag is 1, the video decoding device may apply vertical traversal scanning to the current block.
[0128] For each coefficient group, index run encoding may be performed as follows. For each sample position, a run_copy_flag may be signaled, which is a flag indicating whether the run type at the current position is the same as the run type of the previously scanned sample. If this flag is 1, it indicates that the index run type at the current position is the same as the run type of the previously scanned sample. That is, the index run type at the current position and the run type of the previously scanned sample may both be COPY_INDEX or COPY_ABOVE (or COPY_LEFT for vertical traversal scanning).
[0129] On the other hand, this flag is zero, indicating that the index run type at the current position is different from the run type of the previously scanned sample. In this case, copy_above_palette_index_flag can be signaled, which is a flag indicating the run type at the current position. This flag is 1 for COPY_ABOVE (or COPY_LEFT for a vertical traversal scan), indicating that the palette index at the current position is set to be the same as the palette index at the same position in the top row (or the same position in the left row for a vertical traversal scan). On the other hand, this flag is 0 for COPY_INDEX. That is, the palette index at the current position can be signaled or derived.
[0130] In the first row (for horizontal traversal scan) or first column (for vertical traversal scan) of the current block, the run type of the corresponding sample is the default COPY_INDEX and is therefore not signaled. In addition, if the previously parsed run type is COPY_ABOVE, the run type of the current position is not signaled.
[0131] The above-mentioned index run may include a flag indicating whether the run type at the current position is the same as the run type in the previous scan sample, and a flag of copy_above_palette_index_flag indicating the run type at the current position.
[0132] Fig.11 A diagram illustrating index run length encoding for coefficient groups according to at least one embodiment of the present disclosure.
[0133] exist Fig.11 In the example of , for one coefficient group (m=16), the index map can be encoded based on index run length coding. Fig.11 In the example of , three run types based on horizontal traversal scanning are used, and the length of each run type is 4, 2, and 10 samples.
[0134] Based on the decoded index map and the current palette, the video decoding device can predict the value of each sample in the current block in the palette mode. That is, for each sample, an index is derived by using the index map, and the value of each sample can be predicted by using the color value indicated by the index in the current palette table.
[0135] II. Palette Mode Based on Neighborhood Information
[0136] In contrast to signaling / parsing an index run encoding index map, in one example, a palette table and index map for a current block may be derived based on neighbor information of the current block. Here, the neighbor information may be a block vector of the M×N current block, or information about a previously reconstructed region in the current frame based on template matching using a template region around the current block.
[0137] In one example, a flag indicating whether to derive the index map of the current block based on the neighboring information may be signaled. If this flag is 1, the video decoding device may derive the index map of the current block based on the neighboring information of the current block. If the flag is 0, the video decoding device may decode the index run coded index map, such as Fig.11 as shown in the example.
[0138] Fig.12 is a diagram illustrating derivation of an index map based on proximity information according to at least one embodiment of the present disclosure.
[0139] In one example, when an index map is derived based on template matching, such as Fig.12 As shown, template matching is a process of finding the best cost value between the template in the previously reconstructed adjacent area of the current block and the adjacent template of the random reference block in the previously reconstructed area in the current frame. As a cost function, SAD (Sum of Absolute Difference), SATD (Sum of Absolute Transform Difference), SSE (Sum of Square Difference), etc. can be used. The block with the best cost in the random reference block is set as a candidate block for deriving the index map.
[0140] When using block vectors, the block vectors existing at the left, top, upper left, upper right, and lower left positions of the current block can be used, such as Fig.13 As shown in the example of . At this time, the video encoding device and the video decoding device can configure the same block vector candidate list, and then the index of the selected block vector can be notified / parsed by signaling according to block matching. Block matching is the process of finding the best cost value between the current block and the reference block indicated by the block vector. In this case, SAD, SATD, SSE, etc. can be used as the cost function. Similarly, the block with the best cost in the random reference block is set as a candidate block to derive the index map.
[0141] As another example, without signaling and parsing, the video encoding apparatus and the video decoding apparatus may equally form a block vector candidate list, and then may perform template matching on each block vector candidate to derive a candidate block having an optimal cost value.
[0142] Alternatively, without additional signaling and parsing information, and when the block vector of the current block is not available, template matching can be performed on previously reconstructed neighboring regions of the current block to derive a candidate block with the best cost value, as in Fig.12 as shown in the example.
[0143] As an example, in order to derive a palette table and an index map from a previously reconstructed region (ie, a candidate block), the video encoding apparatus may use methods such as quantization, clustering, segmentation, and the like.
[0144] For example, when determining the palette table by using quantization, the video encoding apparatus may implicitly determine the quantization step size (Q_step) based on the quantization parameter of the current block, or may implicitly determine the quantization step size based on the reconstructed pixel value of the previously reconstructed region used for derivation. In this case, the number of quantization steps may be the size of the palette, which is the number of indexes in the palette. The video encoding apparatus may determine the palette table of the candidate block by quantizing the samples of the candidate block based on the quantization step size and deriving an entry in the palette corresponding to each quantized sample.
[0145] As another example, when clustering is used to determine the palette table, the size of the palette (which is the number of indices in the palette) may be implicitly determined by the number of clusters (which is the set to be clustered). The palette table for the candidate block may then be determined by performing clustering of the candidate block based on the number of clusters and deriving a palette entry corresponding to each cluster.
[0146] As another example, when segmentation is used to determine the palette table, the size of the palette (which is the number of indices in the palette) can be implicitly determined as the set to be segmented (i.e., the number of segments). Then, by performing segmentation of the candidate block based on the number of segments and deriving the entries of the palette corresponding to each segment, the palette table of the candidate block can be determined.
[0147] In determining the palette table for the candidate block, the video encoding device may use an escape symbol to identify samples outside the color range of the palette. If the escape symbol is present in the candidate block, the video encoding device may increase the size of the palette table by one and may assign the last index in the table to the escape symbol.
[0148] The video encoding device may then use the palette table of the candidate block to derive an index map in terms of cost optimization. Here, the cost represents the cost value between the candidate block and the reconstructed block generated by the index map. In this case, SAD, SATD, SSE, etc. may be used as a cost function.
[0149] Fig.14 is a diagram illustrating the signaling of difference values mapped to corresponding indexes in a palette according to at least one embodiment of the present invention.
[0150] In one example, after selecting a previously reconstructed region in the current frame and deriving an index map from information about the region, when the magnitude of each index value in the derived index map is proportional to the magnitude of the pixel value in the selected previously reconstructed region and proportional to the magnitude of the value mapped to each index in the palette, the difference in the value mapped to each index in the palette may be signaled for the new entry being signaled, such as Fig.14 The video decoding apparatus may reconstruct the palette by reconstructing the value of each index mapped to the new entry and then adding the reconstructed difference value of each index to the value mapped to the previous index based on the first new entry. If the entry is a reuse entry based on a reuse flag, the value mapped to the corresponding index may be derived from a palette prediction list and used for the current palette.
[0151] Reference now Fig.15 and Fig.16 , describes a method for encoding / decoding a current block by using a palette mode based on neighboring information. Fig.15 and16 In , the palette mode is applied to the current block determined according to the prediction unit (PU).
[0152] Fig.15 is a flowchart of a method for encoding a current block by using a palette mode, performed by a video encoding apparatus according to at least one embodiment of the present invention.
[0153] The video encoding apparatus determines a palette table and derives an index map according to a first method (S1500). Here, the first method uses neighboring information of the current block, and the neighboring information includes a block vector of a neighbor of the current block or a template within a previously reconstructed adjacent region of the current block.
[0154] In the first method, the video encoding device generates a candidate block by using neighboring information of the current block, and then determines a palette table of the candidate block. The video encoding device then uses the palette table of the candidate block to derive an index map. Thus, the index map includes a per-sample index of the candidate block, and the index indicates an entry in the palette table of the candidate block having a color value corresponding to a sample of the candidate block.
[0155] As an example of generating a candidate block, the video encoding device synthesizes a block vector candidate list by using block vectors existing at left, top, upper left, upper right, and lower left positions of the current block, such as Fig.13 As shown in the example of . The video encoding device may select the best block vector from the block vector candidate list by using block matching, and may generate a candidate block based on the selected block vector. The video encoding device may then encode the index of the selected block vector.
[0156] As another example, without further encoding, the video encoding apparatus may apply template matching to the current block and a block indicated by each candidate block vector in the block vector candidate list to set a block having the best cost as a candidate block.
[0157] As another example, if there is no additional parsing and the block vector cannot be utilized, the video encoding apparatus may apply template matching to a previously reconstructed neighboring area of the current block to set a block corresponding to a template having the best cost as a candidate block.
[0158] Meanwhile, the video encoding apparatus may apply a palette table derivation method to the candidate block to determine the palette table. Here, the palette table derivation method may utilize quantization steps, clustering, or segmentation.
[0159] In one example, the video encoding device implicitly derives an optimal quantization step size and then quantizes samples of the candidate block based on the derived quantization step size. The video encoding device may determine the palette table by deriving an entry in the palette table corresponding to each quantized sample.
[0160] As another example, the video encoding apparatus determines the number of clusters for clustering, and clusters the candidate blocks based on the number of clusters.The video encoding apparatus may then determine the palette table by deriving an entry corresponding to each cluster in the palette table.
[0161] As yet another example, the video encoding apparatus determines the number of segments used for segmentation, and segments the candidate blocks based on the number of segments.The video encoding apparatus may then determine the palette table by deriving an entry in the palette table corresponding to each segment.
[0162] In terms of cost optimization, the video encoding device can use the palette table of the candidate block to derive the index map. Here, the cost indicates the cost value between the reconstructed block generated by the index map and the candidate block. In this case, SAD, SATD, SSE, etc. can be used as the cost function.
[0163] When determining the palette table for the candidate block, the video encoding device may use an escape symbol to identify samples outside the color range of the palette. If the escape symbol is present in the candidate block, the video encoding device may increase the size of the palette table by one and may assign the last index in the table to the escape symbol.
[0164] The video encoding apparatus determines a palette table and derives an index map according to the second method ( S1502 ).
[0165] Here, the second method uses samples in the current block. That is, the video encoding apparatus determines a palette table of the current block by using samples of the current block, and derives an index map by using the palette table of the current block. Thus, the index map contains a per-sample index of the current block, and the index indicates an entry in the palette table of a candidate block having a color value corresponding to the sample of the current block.
[0166] The video encoding apparatus may determine the palette table of the current block, for example, by applying clustering to samples of the current block.
[0167] When generating the palette table for the current block, the video encoding device may use an escape symbol to identify samples outside the color range of the palette. If the escape symbol exists in the current block, the video encoding device may increase the size of the palette table by one and may assign the last index in the table to the escape symbol.
[0168] The video encoding apparatus may derive an index map using a palette table of a current block in terms of optimizing rate-distortion.
[0169] The video encoding apparatus may then encode the index map based on index run coding. Here, the index run coding partitions the current block into a plurality of row-based coefficient groups and determines an index run and a corresponding index of each coefficient group.
[0170] The video encoding apparatus selects an optimal method between the first method and the second method (S1504).
[0171] The video encoding apparatus determines an index map derivation flag according to the optimal method (S1506). Here, the index map derivation flag indicates whether to derive the index map based on the first method using neighbor information of the current block.
[0172] The video encoding apparatus encodes the index map derivation flag (S1508).
[0173] The video encoding apparatus encodes the palette table according to the optimal method (S1510).
[0174] The video encoding device may set a series of reuse flags based on the reused entries in the palette table from the palette prediction list. Further, the video encoding device may set the remaining entries in the palette table except the reused entries as new entries.
[0175] The video encoding device encodes the series of reuse flags and encodes the new entry.
[0176] After encoding the palette table, the video encoding device may update the palette prediction list. The video encoding device may include entries from the palette table in the palette prediction list. The video encoding device may also add non-reused entries from a previous palette prediction list to the palette prediction list until a maximum allowed size is reached.
[0177] Fig.16 is a flowchart of a method for decoding a current block by using a palette mode, performed by a video decoding apparatus according to at least one embodiment of the present invention.
[0178] The video decoding apparatus generates a palette table for a current block (S1600).
[0179] To generate the palette table, the video decoding device first decodes a series of reuse flags from the bitstream. Here, the series of reuse flags indicates whether an entry in a palette prediction list is reused. The video decoding device includes the reused entry from the palette prediction list in the palette table based on the value of the series of reuse flags. In addition, the video decoding device may first decode a new entry from the bitstream, or may implicitly derive the new entry, and then may add the new entry to the palette table.
[0180] After generating the palette table, the video decoding device may update the palette prediction list. The video decoding device includes entries from the palette table surface in the palette prediction list. The video decoding device may also add non-reused entries from a previous palette prediction list to the palette prediction list until a maximum allowed size is reached.
[0181] The video decoding apparatus decodes an index map derivation flag from a bitstream (S1602). Here, the index map derivation flag indicates whether to derive an index map based on neighboring information of a current block.
[0182] The video decoding device checks the index map derivation flag (S1604).
[0183] If the index map derivation flag is true (yes in S1604), the video decoding apparatus derives an index map by using neighboring information of the current block (S1606). Here, the index map includes a per-sample index of the current block, and the index indicates an entry in a palette table having a color value corresponding to a sample of the current block.
[0184] As the neighboring information, the video decoding apparatus may use a block vector of a neighbor of the current block or a template within a previously reconstructed neighboring region of the current block. The video decoding apparatus generates a candidate block for deriving the index map using the neighboring information.
[0185] In one example, the video decoding device generates a block vector candidate list by using block vectors existing at the left, top, upper left, upper right, and lower left of the current block, such as Fig.13 Then, the video decoding device may decode the candidate index and may generate a candidate block based on a block vector derived from the block vector candidate list by using the candidate index.
[0186] As another example, without further parsing, the video decoding apparatus may apply template matching to the current block and a block indicated by each candidate block vector in the block vector candidate list to set a block having the best cost as the candidate block.
[0187] As yet another example, if there is no additional parsing and the block vector cannot be utilized, the video decoding apparatus may apply template matching to a previously reconstructed neighboring area of the current block to set a block corresponding to a template having the best cost as a candidate block.
[0188] The video decoding device can use the palette table to derive the index map of the candidate block in terms of cost optimization. Here, the cost indicates the cost value between the reconstructed block generated by the index map and the candidate block. In this case, SAD, SATD, SSE, etc. can be used as the cost function.
[0189] On the other hand, if the index map derivation flag is false (No in S1604), the video decoding apparatus decodes the index map based on index run decoding (S1608).
[0190] Here, index run decoding partitions the current block into multiple row-based coefficient groups, and then decodes the index run and corresponding index from the bitstream for each coefficient group.
[0191] The video decoding apparatus reconstructs samples of a current block based on the index map and the palette table (S1610).
[0192] If the index indicates the last entry in the palette table that is extended to support the escape symbol, the video decoding device decodes the escape symbol from the bitstream.Then, the video decoding device dequantizes the decoded escape symbol.
[0193] Although the steps in each flowchart are described as being performed sequentially, these steps are merely illustrative of the technical concepts of some embodiments of the present disclosure. Therefore, a person of ordinary skill in the art to which the present disclosure belongs can perform the steps by changing the order described in each figure or by performing two or more steps in parallel. Therefore, the steps in each flowchart are not limited to the time series order shown.
[0194] It should be understood that the above description presents illustrative embodiments that can be implemented in different other ways. The functions described in some embodiments can be implemented by hardware, software, firmware and / or a combination thereof. It should also be understood that the functional components described in this disclosure are marked with "... units" to strongly emphasize the possibility of their independent implementation.
[0195] At the same time, the various methods or functions described in some embodiments may be implemented as instructions stored in a non-transitory recording medium that can be read and executed by one or more processors. For example, the non-transitory recording medium may include various types of recording devices, in which data is stored in a form readable by a computer system. For example, the non-transitory recording medium may include storage media such as erasable programmable read-only memory (EPROM), flash drives, optical drives, magnetic hard drives, and solid-state drives (SSDs).
[0196] Although the embodiments of the present disclosure have been described for illustrative purposes, however, those of ordinary skill in the art to which the present disclosure belongs should recognize that various modifications, additions, and substitutions are possible without departing from the concept and scope of the present disclosure. Therefore, for the sake of brevity and clarity, the embodiments of the present disclosure have been described. The scope of the technical concept of the embodiments of the present disclosure is not limited by the diagrams. Therefore, those of ordinary skill in the art to which the present disclosure belongs should understand that the scope of the present disclosure should not be limited by the embodiments explicitly described above, but by the claims and their equivalents.
[0197] (reference number)
[0198] 120: Predictor
[0199] 155: Entropy Encoder
[0200] 510: Entropy Decoder
[0201] 540: Predictor
[0202] CROSS-REFERENCE TO RELATED APPLICATIONS
[0203] This application claims priority to and the benefit of Korean Patent Application No. 10-2022-0035562, filed on Mar. 22, 2022, and Korean Patent Application No. 10-2023-0027858, filed on Mar. 2, 2023, the entire contents of each of which are incorporated herein by reference.
Claims
1. A method for reconstructing a current block performed by a video decoding device, the method include: generating a palette table for the current block; deriving an index map by using neighboring information of the current block, wherein the neighboring information comprises neighboring block vectors of the current block or a template within a previously reconstructed neighboring region of the current block, wherein the index map comprises indices according to samples of the current block, and wherein the indices indicate entries in the palette table having color values corresponding to the samples of the current block; and The samples of the current block are reconstructed based on the index map and the palette table.
2. The method according to claim 1, further comprising: include: decoding an index map derivation flag from a bitstream, the index map derivation flag indicating whether the index map is derived based on the neighboring information; as well as Check the index map derivation flag, Wherein, when the index map derivation flag is true, the method performs derivation of the index map by using the proximity information.
3. The method according to claim 2, further comprising: include: When the index map derivation flag is false, the index map is decoded from the bitstream.
4. The method according to claim 1, in, Generating the palette table includes: decoding a series of reuse flags from the bitstream, the reuse flags indicating whether entries contained in the palette prediction list are to be reused; and A reuse entry from the palette prediction list is included in the palette table based on the series of reuse flag values.
5. The method according to claim 4, in, Generating the palette table includes: decoding a new entry from the bitstream or implicitly deriving the new entry; adding the new entry to the palette table; and The palette prediction list is updated.
6. The method according to claim 1, in, Deriving the index map includes: generating candidate blocks for deriving the index map; and An index map of the candidate blocks is derived by using the palette table.
7. The method according to claim 6, in, Generating the candidate block comprises: composing a block vector candidate list by using block vectors existing at left, top, upper left, upper right, and lower left positions of the current block; decoding the candidate index; and The candidate block is generated based on the block vector derived from the block vector candidate list by using the candidate index.
8. The method according to claim 6, in, Generating the candidate block comprises: composing a block vector candidate list by using block vectors existing at left, top, upper left, upper right, and lower left positions of the current block; and By applying template matching to the current block and the block indicated by each candidate block vector in the block vector candidate list, a block having the best cost is set as the candidate block.
9. The method according to claim 6, in, Generating the candidate block comprises: By applying template matching to a previously reconstructed neighboring region of the current block, a block corresponding to a template having a best cost is set as the candidate block.
10. The method according to claim 6, in, Deriving the index map includes: deriving an index map of the candidate blocks in terms of cost optimization; and The index map of the candidate block is set as the index map of the current block.
11. A method for encoding a current block performed by a video encoding device, the method include: Determine a palette table and derive an index map according to a first method, wherein the first method uses neighboring information of the current block, wherein the neighboring information includes a neighboring block vector of the current block or a template within a previously reconstructed neighboring region of the current block; Determine the palette table and derive the index map according to a second method, wherein the second method uses samples within the current block; selecting an optimal method between the first method and the second method; and The palette table is encoded according to the optimal method.
12. The method according to claim 11, in, The first method comprises: generating a candidate block by using neighboring information of the current block; Determining a palette table for the candidate block; and In terms of cost optimization, the index map is derived by using the palette table of the candidate blocks, The index map includes a per-sample index of the candidate block; and The index indicates an entry in the palette table having a color value corresponding to a sample of the candidate block.
13. The method according to claim 11, in, The second method comprises: determining a palette table for the current block by using the samples of the current block; deriving the index map by using the palette table of the current block in terms of rate-distortion optimization; and The index map is encoded.
14. The method according to claim 11, further comprising: include: determining an index map derivation flag based on the optimal method, the index map derivation flag indicating whether to derive the index map based on the first method; as well as The index map derivation flag is encoded.
15. The method according to claim 11, in, Encoding the palette table includes: setting a series of reuse flags based on reuse entries in the palette table from a palette prediction list; setting the remaining entries in the palette table except the reused entry as new entries; and The palette prediction list is updated.
16. The method according to claim 15, in, Encoding the palette table further comprises: encoding the series of reuse flags; and The new entry is encoded.
17. The method according to claim 12, in, Determining the palette table includes: The palette table is determined by applying a method of derivation of the palette table to the candidate region, wherein the method of derivation utilizes quantization step size, clustering or segmentation.
18. A computer-readable recording medium storing a bit stream generated by a video encoding method, the video encoding method include: Determine a palette table and derive an index map according to a first method, wherein the first method uses neighboring information of a current block, wherein the neighboring information includes a neighboring block vector of the current block or a template within a previously reconstructed neighboring region of the current block; Determine the palette table and derive the index map according to a second method, wherein the second method uses samples within the current block; selecting an optimal method between the first method and the second method; and The palette table is encoded according to the optimal method.
Citation Information
Cited By
A palette candidate generation method and device for VVC SCC
CN122718468A
