Method and apparatus for DIMD position related mixing and encoder / decoder including the same
Through position-related mixing technology, the mixing weight is adjusted according to the sample to the template block distance of the codec unit CU, which solves the problem of insufficient global weighting of the intra prediction mode and large calculation overhead in the prior art, and improves the intra prediction accuracy and efficiency.
Patent Information
- Application Number
- CN202380030568.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2022-04-07
- Filing Date
- 2023-03-03
- Publication Date
- 2024-11-12
AI Technical Summary
The global weighting processing of intra prediction mode in the prior art results in the predictor being unreliable and has a high computational overhead, especially when only one IPM is selected, the single prediction mode generated is inefficient.
Using position-dependent mixing technology, the mixing weight is determined based on the distance from each sample of the codec unit CU to the template block, balance the contribution of the IPM and Planar/DC modes, and generate a more reliable intra-predictor.
It improves intra prediction accuracy, reduces computing overhead, and takes advantage of the advantages of DIMD mode in local scenarios to provide more accurate and efficient intra prediction processing.
Smart Images

Figure CN118947118A_ABST
Abstract
Description
[0001] CROSS-REFERENCE TO RELATED APPLICATIONS
[0002] This application is based on and claims priority from European patent application No. “22167223.1” filed on April 7, 2022, the entire contents of which are incorporated herein by reference. Technical Field
[0003] The present invention generally relates to the field of encoding / decoding pictures, images or videos, and embodiments of the present invention relate to improvements regarding intra prediction, and more specifically, to improvements regarding decoder-side intra mode derivation (DIMD) processing. More specific embodiments of the present invention relate to DIMD position-dependent mixing. Background Art
[0004] Encoding and decoding of pictures, images or videos are performed according to certain standards, for example, according to the Advanced Video Codec (AVC) standard (see reference [1]), the High Efficiency Video Codec (HEVC) standard (see reference [2]) or the Versatile Video Codec (VVC) standard (see reference [3]).
[0005] Figure 1 A block diagram of a standard video compression system 100 operating according to the VVC standard is illustrated in FIG. The standard video codec 100 compresses and encodes pictures 102 of a video sequence. The pictures 102 to be encoded are partitioned into blocks 104, also referred to as codec units (CUs). The encoder 100 includes a pre-coding filter 106 and a prediction residual signal former 108, which generates a prediction residual signal 110 to measure the deviation between the prediction signal 112 and a signal 114 output by the filter 106. The encoder also includes a transformer 116, a quantizer 118, and provides an output bitstream or data stream 120 using an entropy codec 122. In addition, the encoder 100 includes a prediction stage 124 for determining the prediction signal 112, and the prediction stage 124 includes an inverse quantizer or inverse quantizer 126, an inverse transformer 128, a combiner 130, an in-loop filter 134, a motion estimator 138, an intra-frame / inter-frame mode selector 140, an inter-frame predictor 142 and an intra-frame predictor 144.
[0006] As reference Figure 1The described video codec 100 compresses and encodes a picture 102 of a video sequence as follows. The picture 102 to be encoded is partitioned into blocks or CUs 104. Each codec unit 104 is encoded using an intra or inter codec mode. When a CU is encoded in intra mode, intra prediction is performed by an intra predictor 144. Intra prediction includes predicting the current CU 114 being encoded with the help of coded, decoded and reconstructed picture samples located around the current CU (e.g., the top and left of the current CU). Intra prediction is performed in the spatial domain. In inter mode, motion compensation and motion estimation are performed by a motion estimator 138, which searches for a CU that is a good predictor of the current CU in one or more reference pictures provided by a picture buffer 136 and used to predictively encode and decode the current picture. For example, a good predictor of the current CU is a predictor similar to the current CU, that is, the distortion between the two CUs is low or below a certain threshold. Motion estimation can also consider signaling to the predictor to optimize the rate cost of the rate-distortion trade-off. The output of the motion estimation step is one or more motion vectors and reference indices associated with the current CU. Motion compensation is then used to predict the current CU with the aid of one or more motion vectors and reference picture indices determined by the motion estimator 138. Basically, the block or CU contained in the selected reference picture and pointed to by the determined motion vector is used as the prediction block of the current CU. The encoder 100 selects one of the intra-frame coding mode or inter-frame coding mode for encoding the CU with the aid of the selector 140 and indicates the intra / inter decision, for example, with the aid of a prediction mode flag. The prediction residual 110 is then transformed and quantized by blocks 116 and 118, and the quantized transform coefficients as well as the motion vectors and other syntax elements are entropy encoded and written to the output bitstream 120. The encoder 100 can skip the transform stage 116 and directly apply quantization to the untransformed residual signal 110 in a so-called transform-skip coding mode. After encoding a block or CU, the encoder decodes the CU and reconstructs it to obtain a reconstructed signal 132, which can be used as reference data for predicting future CUs or blocks to be encoded. The quantized transform coefficients 110' are dequantized and inverse transformed, resulting in a decoded predicted CU or block 110", and the decoded prediction residual and the predicted block are then combined (usually added) at 130 to provide a reconstructed block or CU 132. An in-loop filter 134 is applied to the reconstructed picture to reduce compensation artifacts. For example, a deblocking filter, a sample adaptive offset (SAO) filter, and an adaptive loop filter (ALF) can be applied to reduce coding artifacts. The filtered picture is stored in a buffer 136 (also called a decoded picture buffer (DPB)) so that it can be used as a reference picture for encoding and decoding subsequent pictures.
[0007] Figure 21 is a block diagram of a video decoder 150 for predictively decoding a picture or video provided at an output 154 of the decoder 150 from a data stream 152. The decoder 150 includes an entropy decoder 156, a partition block 158, an inverse quantizer 160, an inverse transformer 162, a combiner 164, an in-loop filter 166, an optional post-decoding processor 168, and a prediction module 170. The prediction module 170 includes a decoded picture buffer 180, a motion compensator 182, and an intra-frame predictor 184.
[0008] The encoded pictures of the video sequence are decompressed and decoded by the decoder 150 as follows. The input bitstream 152 is entropy decoded by the decoder 156, which provides, for example, block partition information, the codec mode for each codec unit, the transform coefficients contained in each transform block, prediction information (such as intra-frame prediction mode), motion vectors, reference picture indexes, and other codec information. The block partition information indicates how the picture is partitioned, and the decoder 150 can divide the input picture into codec tree units (CTUs) (usually 64×64 or 128×128 pixels in size) and divide each CTU into rectangular or square codec units (CUs) according to the decoded partition information. The entropy decoded quantized coefficients 172 are dequantized 160 and inverse transformed 162 to obtain a decoded residual picture or CU 174. The decoded prediction parameters are used to predict the current block or CU, that is, whether the predicted block is obtained by its intra-frame prediction or by its motion compensated temporal prediction. The prediction process performed on the decoder side is the same as the prediction process performed on the encoder side. The decoded residual block 174 is added to the predicted block 176, thereby producing a reconstructed current image block 164. The in-loop filter 166 is applied to the reconstructed picture or image, which is also stored in the decoded picture buffer 180 for use with reference pictures for future picture decoding. As mentioned above, the decoded picture may also be subjected to post-decoding processing, such as for performing an inverse color transform, such as a conversion from YCbCr4:2:0 to RGB 4:4:4.
[0009] As mentioned above, the intra-frame prediction mode to be used for decoding can be carried in the bitstream provided by the encoder and received by the decoder, but, according to other methods, instead of introducing the actual intra-frame prediction mode into the bitstream, the intra-frame prediction mode can be derived by gradient analysis using reconstructed pixels adjacent to the currently processed CU. In other words, the intra-frame prediction mode is not explicitly indicated in the bitstream, but is implicit. This method is called decoder-side intra-frame mode derivation (DIMD), which can be signaled using a simple flag and then the actual intra-frame mode is derived during the reconstruction process, such as the reconstruction process performed by the prediction block 124 / 170 of the encoder 100 or decoder 150. The encoder 100 can encode information whether DIMD is used for the current CU into the bitstream 120, and in the case where DIMD is not used, the actual intra-frame mode is signaled by the encoder and parsed from the bitstream by the decoder.
[0010] Taken from reference [5] Figure 3 An example of an intra-block general reconstruction process is illustrated, which may be applied at the encoder 100 or at the decoder 150. During the parsing process 200, intra-block decoding is determined, as indicated at block 202, and pixel residual parsing 204 is performed, as well as DIMD flag parsing 206. If the DMID flag 206 is zero, then intra-prediction mode parsing 208 is performed to obtain an intra-mode 210 to be used when predicting and reconstructing pixels during a reconstruction process 212, as indicated at 216. If the DIMD flag 206 is set to one, then intra-prediction derivation 218 is performed during the reconstruction process 212, which produces a derived intra-mode 220, which is used in the reconstruction process together with the pixel residual 214 at 216.
[0011] Further details of the DIMD method are described in references [5] to [8]. Generally speaking, DIMD processing is based on a reconstructed template area adjacent to the currently processed CU, the width and height of which are three samples wide. The template area includes a left area, an upper area, and an upper left area. These areas are also referred to as the left template area, the upper template area, and the upper left template area in the following text. In the template area, a corresponding edge detection filter (such as a 3×3 horizontal and vertical Sobel filter) is applied to determine the amplitude and angle of the brightness direction or orientation of each midline sample of the template area. A histogram of gradients (HoG) is calculated, and each entry corresponds to a conventional intra-angle pattern and the accumulated intensity is stored:
[0012] angle=arctan(G hor / G ver )
[0013] amplitude=|Ghor |+|G ver |
[0014] Among them G hor and G ver is the pure horizontal and vertical intensity calculated by the Sobel filter. FIG. 4 taken from reference [8] illustrates a DIMD template block or region used to calculate HoG. FIG. 4( a) illustrates a partially reconstructed picture 250, as indicated by a reconstructed block 252 and an unavailable block 254. In other words, the reconstructed block 252 includes the already reconstructed codec units or blocks of the picture to be reconstructed, while the corresponding codec units or blocks in the unavailable block 254 are still to be reconstructed. The currently processed CU or block 256 and the template block 258 mentioned above are illustrated. FIG4( b) illustrates that the template block 258 is 3 samples or pixels wide in width and height, and as described above, the HoG 260 is calculated using 3×3 horizontal and vertical Sobel filters 259 to obtain an angular intra-prediction mode (also called intra-prediction mode, IPM) based on the amplitude and angle of the luma direction of each intermediate line sample 262 of the template block 258, and obtain the associated cumulative intensity or amplitude for each IPM in the template block. The two most representative IPMs (indicated as M1 and M2 in FIG4( b)) are selected from the HoG 260 and combined with the Planar mode using weights for M1 and M2 derived from the ratio of their cumulative amplitudes and a fixed weight for the Planar mode (e.g., 1 / 3, i.e., 21 / 64 with 6-bit integer precision).
[0015] Taken from reference [8] Figure 5 The diagram illustrates prediction fusion by weighted averaging of M1, M2 and Planar modes. Assuming a square CU 256, corresponding pixels of CU 256 are predicted by the two most representative IPMs M1, M2 taken from HoG 260, thereby obtaining predictor blocks Pred1, Pred2 and Pred3. The corresponding predictor blocks are weighted by weights w1, w2, w3 and fused or combined into a final predictor block including DMID predictors for corresponding samples or pixels in CU 256.
[0016] Figure 6An example of a conventional DIMD predictor derivation process is illustrated. At S100, using the input CU 256, the neighborhood of the input CU 256 is analyzed to determine the number of available upper and left inner samples. At S102, the HoG 260 is constructed for the template block 258, that is, based on the available samples, the angle intra prediction mode IPM is determined by its cumulative amplitude in the HoG 260 (step S104). At S106, it is determined how many IPMs are selected, that is, whether no IPM is selected, whether one IPM is selected, or whether two IPMs are selected. If no IPM is selected, then only the Planar mode (which can be the default mode) is applied, as indicated at S108, to obtain the DIMD predictor. If only one IPM is selected, then the selected IPM is used to obtain the DIMD predictor, as indicated at S110. If two IPMs are selected, then at S112, the mixing weight of each selected mode is determined according to its cumulative amplitude ratio, and the weight for the Planar mode is set to 1 / 3. Predictor blocks Pred1 to Pred3 for each selected IPM and Planar mode are calculated at S114 , and a DIMD predictor is obtained by mixing the two selected IPM and Planar modes using the determined mixing weight at S116 .
[0017] Figure 7 An example of a currently processed CU 256 and an associated template block 258 of width / height T=3 samples or pixels is illustrated. For each intermediate line sample 262, the corresponding orientation is illustrated by the direction of the corresponding arrow, and the corresponding strength of the gradient is indicated by the length of the arrow. The HoG is generated by accumulating the strength of each orientation / IPM entry, and the two most significant modes in the HoG (if they exist) are selected as the modes to be mixed with the Planar mode or the planar intra prediction mode. Figure 7 In the example of , IPMs 5 and 28 have the first and second highest cumulative strengths / amplitudes. As described above, these modes are weighted, and the blending weights are conventionally fixed and / or determined according to the ratio of the cumulative strengths / amplitudes of the selected IPMs. Although different blending modes are proposed in the prior art, all blending modes are global with respect to the currently processed block or CU. More specifically, when considering the conventional approach and when two IPMs (such as M1 and M2) are selected, the blending weights can be calculated as follows
[0018]
[0019] Where w1 is associated with IPM M1, w2 is associated with IPM M2, and w3 is associated with Planar mode. Different modifications of the blending process are described in the prior art, for example, reference
[10] proposes selecting multiple blending modes based on rate-distortion optimization, reference
[11] proposes using HoG to implicitly derive two of the three blending modes, reference
[13] proposes using template matching to implicitly derive the blending mode, and reference
[20] proposes selecting different blending modes based on multiple DIMD modes determined. When DIMD is applied, one or two IPMs are derived from reconstructed neighboring samples (i.e., from template blocks), and if two intra modes are derived, they are combined (i.e., blended) with the Planar mode predictor. Conventional solutions always implement a global or CU-level weighted blending of the two IPMs with the Planar mode.
[0020] Conventionally, in the reconstructed portion 252 of the picture 250, three columns and rows of reconstructed samples or pixels are used for DIMD processing, i.e., the HoG 260 is calculated using a filter having a fixed filter window of 3×3 samples and located at the same position in the template block 258. In reference [6], when calculating the gradient on the pixel located in the left column, top row or upper left sample and directly adjacent to the current CU, the 3×3 Sobel filter 259 can be replaced by a 3×2 filter. In reference [8], instead of using the 3×3 Sobel filter 259 on all pixels or samples in the middle line of the template block 258, the 3×3 Sobel filter 259 can be applied more sparsely, for example, only at one middle line sample in the left template block and one middle line sample in the upper template block. Further aspects of the DIMD processing are described, for example, in the following references:
[0021] - References
[10] ,
[11] ,
[13] and
[20] discuss the mixing of modes,
[0022] - References
[16] and
[18] discuss behavior correction (error repair) relative to the original intent,
[0023] - References [9],
[12] and
[14] discuss combinations with other modes,
[0024] - References
[15] ,
[17] ,
[19] or
[21] discuss syntax or simplifications for DIMD processing.
[0025] However, the above reference Figure 6 The basic DIMD processing remains the same as described in reference [8].
[0026] The conventional hybrid process for obtaining the final intra-prediction block is disadvantageous because it uses weights that are applied globally to the entire block or CU currently being processed, so that the weights do not take into account the local characteristics of the IPM, such as the orientation and strength of the IPM in the reconstructed template block, which can lead to a less reliable intra-predictor. Another disadvantage is that if only one IPM is selected, the predictor is not combined with the Planar mode or DC mode predictor, that is, it is the same as the conventional intra-prediction mode. However, this mode is encoded as a DIMD mode, so that when compared to the conventional intra-prediction mode, additional computational overhead is incurred due to the need to calculate the HoG, which may ultimately result in only a single IPM to be used.
[0027] Therefore, there is a need to provide further improvements in DIMD processing. Summary of the invention
[0028] The present invention provides a method for deriving a decoder-side intra-mode derivation DIMD predictor for corresponding samples of a coding unit CU of a picture, the method comprising:
[0029] Select one or more intra prediction modes IPM in the template block adjacent to the CU.
[0030] For each sample of the CU, determining a mixing weight for mixing at least one or more selected IPMs according to a distance between the sample of the CU and the template block, and
[0031] The DIMD predictor is generated by mixing one or more selected IPMs and the Planar mode or the DC mode using the determined mixing weights.
[0032] Optionally, the mixing weights are determined such that samples of a CU closer to the template block, such as prediction samples, are weighted with a higher IPM contribution, while samples of a CU farther from the template block are weighted with a higher Planar or DC contribution.
[0033] Optionally, the mixing weight of the selected IPM for samples of the CU is determined by weighting a first value obtained from one or more predefined characteristics associated with the IPM according to the distance.
[0034] Optionally, only the distance is used to determine the blending weights for Planar mode or DC mode.
[0035] Optionally, only the position of the sample in the CU and the size of the CU are used to determine the distance for the sample.
[0036] Optionally, one or more IPMs are selected using IPM statistics determined globally by:
[0037] - an entire template block, which includes multiple template block regions, where the selection is performed globally for the entire CU, or
[0038] - The selection includes a global selection of the IPM set for the entire CU, followed by further region-by-region selection from the IPM set for each CU region.
[0039] Optionally, one or more IPMs are selected using IPM statistics, such as a histogram of gradients HoG, determined separately on each of multiple template block areas of the template block, the multiple template block areas including a left template block area and an upper template block area, wherein the selection includes performing a region-by-region selection from the IPMs determined separately for each CU area.
[0040] Alternatively, if the selected IPM exists only in one of the template block regions, the mixing weight of the selected IPM is determined according to the distances of samples in the CU to the adjacent template block region in which the selected IPM exists.
[0041] Optionally, only one of the position coordinates of the sample in the CU and only one dimension of the CU are used to determine the distance for the sample.
[0042] Optionally, the method further comprises:
[0043] Split a CU into two or more CU regions,
[0044] wherein for at least one of the CU regions adjacent to the template block, for each sample of the CU region, a mixing weight for mixing at least one or more selected IPMs is determined according to a distance from the sample of the CU region to the template block, and
[0045] The DIMD predictor is generated using the hybrid weights determined for each CU region.
[0046] Optionally, for only one or some but not all CU regions of the CU, the mixing weight is determined according to the distance between the sample of the corresponding CU region and the template block, and for each remaining CU region, the mixing weight for mixing at least one or more selected IPMs is
[0047] - a global blending weight determined for the entire CU, or
[0048] - Local blending weights determined based on the proximity of the CU region to the template block.
[0049] Optionally, for a CU region that is not located adjacent to the template block,
[0050] ○ Select only Planar mode as DIMD predictor, or
[0051] ○ Select only DC mode as DIMD predictor, or
[0052] o using one or more selected IPMs in one or more or all template block regions to determine blending weights, or
[0053] ○ The blending weight is determined by weighting the blending weights of the CU regions adjacent to the template block region.
[0054] The present invention provides a non-transitory computer-readable storage medium including instructions, which, when executed by a computer, cause the computer to perform the method of the present invention.
[0055] The present invention provides a device for deriving a decoder-side intra-mode derivation DIMD predictor for corresponding samples of a coding unit CU of a picture, the device comprising:
[0056] An intra-frame prediction mode IPM selection module, wherein the intra-frame prediction mode IPM selection module is configured to select one or more IPMs in a template block adjacent to the CU,
[0057] a mixing weight determination module configured to determine, for each sample of the CU, a mixing weight for mixing at least one or more selected IPMs according to a distance between the sample of the CU and the template block, and
[0058] A DIMD predictor generation module is configured to generate a DIMD predictor by mixing one or more selected IPMs and a Planar mode or a DC mode using the determined mixing weights provided by the mixing weight determination module.
[0059] The present invention provides a device for decoding a coded data stream, wherein the coded data stream includes data representing a picture, the device comprising:
[0060] a decoder module configured to receive an encoded data stream and decode a picture from the encoded data stream, and
[0061] A prediction module comprises a device of the present invention for deriving a decoder-side intra mode derivation DIMD predictor for corresponding samples of a coding unit CU of a picture.
[0062] The present invention provides an apparatus for encoding a picture into a coded data stream, wherein the coded data stream includes data representing the picture, the apparatus comprising:
[0063] an encoder module configured to receive an original picture and encode the picture into an encoded data stream, and
[0064] A prediction module comprises a device of the present invention for deriving a decoder-side intra mode derivation DIMD predictor for corresponding samples of a coding unit CU of a picture.
[0065] The technical solution provided according to the embodiment of the present invention has the following beneficial effects.
[0066] In the prior art methods, the hybrid weights are calculated for the entire CU currently being processed or predicted, i.e., conventional methods focus on a global or CU-level approach when determining the hybrid weights for the selected IPM. However, the signal characteristics of the IPM may differ across template blocks, such that the selected weights may be more suitable for obtaining a DIMD predictor for samples in one CU region than for obtaining a DIMD predictor for samples in another CU region. In other words, the weights may not take into account the local characteristics of the IPM, such as the orientation and strength of the IPM in the reconstructed template block, which results in a less reliable intra-frame predictor.
[0067] The present invention solves the shortcomings found in the prior art methods by modifying the conventional DIMD hybrid weighting process by considering the distance of each sample within the currently processed codec unit CU to the template block. In other words, DMID position-dependent hybrid is proposed, which is also referred to as sample-by-sample adaptive hybrid, sample-by-sample hybrid or sample-by-sample weighting hereinafter. For example, (bi)linear sample-by-sample weighting is proposed, in which the contributions of the two IPMs and the contribution of the Planar / DC mode are balanced, so that the predicted samples closer to the template block relative to the current CU are weighted with a higher IPM contribution, while the samples closer to the bottom right of the current CU are weighted with a higher Planar / DC contribution. Therefore, during the hybrid process, an intra-frame predictor is created that more reliably fits the local orientation of the CU to be predicted, thereby improving the prediction accuracy.
[0068] Another disadvantage of the conventional approach is that if only one IPM is selected, the predictor is not combined with the Planar mode or DC mode predictor, i.e. it is the same as the conventional intra prediction mode. However, the mode is encoded as a DIMD mode, so that when compared to the conventional intra prediction mode, additional computational overhead is incurred due to the need to calculate the HoG, which may ultimately result in only a single IPM to be used. The present invention is advantageous because, according to an embodiment, when only one IPM is derived by combining only one IPM and the Planar mode or DC mode using the sample-by-sample mixing weights determined for one selected IPM, the sample-by-sample mixing weights are also used, thereby also utilizing the DIMD mode for such scenarios.
[0069] It should be understood that the content described in this section is not intended to identify key or important features of the embodiments of the present invention, nor is it intended to limit the scope of the present invention. Other features of the present invention will become apparent from the following description. BRIEF DESCRIPTION OF THE DRAWINGS
[0070] The drawings are illustrative and serve to explain the invention and are not to be construed as limiting the invention to the illustrated embodiments.
[0071] Figure 1 is a block diagram of a conventional video encoder;
[0072] Figure 2 is a block diagram of a conventional video decoder;
[0073] Figure 3 The diagram shows the application Figure 1 The encoder or Figure 2 An example of a generic intra-block reconstruction process at a decoder of;
[0074] FIG4 illustrates a DIMD template block or region used to calculate a histogram of gradients (HoG) in a DIMD process;
[0075] Figure 5 The diagram shows the prediction fusion by weighted averaging of the selected IPM and Planar modes;
[0076] Figure 6 An example of a conventional DIMD predictor derivation process is illustrated;
[0077] Figure 7 illustrates an example of a currently processed and associated template block with a width / height of samples or pixels;
[0078] Figure 8 A flowchart of a method for deriving a DIMD predictor for corresponding samples of a CU of a picture according to an embodiment of the present invention is illustrated;
[0079] Fig. 9 The diagram illustrates sample-by-sample mixing weighting according to the distance from the current CU sample to the template block according to an embodiment of the present invention;
[0080] Fig.10 illustrates a flow chart illustrating a DIMD predictor derivation process according to an embodiment of the first aspect of the present invention;
[0081] Fig.11 illustrates the locations of relevant templates / CU blocks and samples according to an embodiment of the present invention;
[0082] FIG12 illustrates the distance between a sample in the current CU and one or more samples in a template block for a square CU;
[0083] FIG13 illustrates the distance between a sample in the current CU and one or more samples in a template block for a rectangular CU;
[0084] Fig.14 The IPM selection for each template block area is illustrated;
[0085] FIG15 illustrates an embodiment of the second aspect of the invention, wherein the distance used in the weighting function depends on the presence of an IPM in each template block region;
[0086] Fig.16 An embodiment is illustrated, according to which a CU is partitioned into four CU regions;
[0087] Fig.17 A block diagram of an apparatus for deriving a DIMD predictor according to an embodiment of the present invention is illustrated;
[0088] Fig.18 A block diagram of a decoder according to an embodiment of the present invention is illustrated;
[0089] Fig.19 A block diagram of an encoder according to an embodiment of the present invention is illustrated; and
[0090] Fig. 20 A block diagram of an electronic device according to an embodiment of the present invention is illustrated. DETAILED DESCRIPTION
[0091] Illustrative embodiments of the present invention will be described below with reference to the accompanying drawings, in which various details of the embodiments of the present invention are included to facilitate understanding and should be regarded as illustrative only. Thus, those of ordinary skill in the art recognize that various changes and modifications may be made to the embodiments described herein without departing from the scope of the present invention. Moreover, for the sake of clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.
[0092] In the present invention, the term “and / or” is intended to cover all possible combinations and sub-combinations of the listed elements, including any one, any sub-combination or all of the elements listed individually, and does not necessarily exclude additional elements.
[0093] In the present invention, the phrase "at least one of... or..." is intended to cover any one or more of the listed elements, including any one, any subcombination or all of the elements listed individually, without necessarily excluding any additional elements, and without necessarily requiring all elements.
[0094] In the present invention, the term "codec" refers to "encoding" or "decoding", as is apparent from the context of the described embodiments. Likewise, the term "codec" refers to "encoder" or "decoder".
[0095] Figure 8 A flowchart of a method for deriving a decoder-side intra mode derivation predictor for corresponding samples of a coding unit of a picture according to an embodiment of the present invention is illustrated, the method comprising the following steps:
[0096] S200: Select one or more intra prediction modes IPM in a template block adjacent to the CU.
[0097] S202: For each sample of the CU, determine a mixing weight for mixing at least one or more selected IPMs according to a distance between the sample of the CU and a template block.
[0098] S204: Generate a DIMD predictor by mixing one or more selected IPMs and the Planar mode or the DC mode using the determined mixing weights.
[0099] In other words, an embodiment provides a method for deriving a DIMD predictor for samples of a CU based on a blending weight for a hybrid IPM / Planar mode or a DC mode, wherein the blending weight is determined for each sample based on the distance between the sample and a template block, instead of using the same blending weight for each sample in the CU.
[0100] According to an embodiment, linear or bilinear sample-by-sample weighting is used, according to which the contribution of the first intra-frame prediction mode M1 and the contribution of the second intra-frame prediction mode M2 and the contribution of the Planar or DC intra-frame prediction mode M3 are balanced, so that with respect to the currently processed CU, prediction samples closer to the template block are weighted with higher M1 and M2 contributions than samples closer to the lower right corner of the currently processed CU, and samples closer to the lower right corner of the currently processed CU are weighted with higher Planar / DC contributions. Fig. 9 The diagram illustrates sample-by-sample mixing weighting according to the distance of the current CU sample to the template block according to an embodiment of the present invention. Fig. 9 The CU 256 and the template block 258 are shown. It is assumed that the first IPM M1 and the second IPM M2 are selected from the HoG generated for the entire template block 258, for example, the IPM whose cumulative amplitude in the HoG exceeds a predetermined value (such as 50), or the IPM whose number of sample positions for determining the corresponding IPM exceeds a certain threshold. Fig. 9As schematically shown in , prediction samples of CU 256 closer to template 258 experience more M1 and M2 contributions and less M3 contribution associated with Planar / DC mode, while prediction samples closer to the bottom right of CU 256 experience less M1 and M2 contributions, but more M3 contribution.
[0101] According to an embodiment of the present invention, the conventional DIMD predictor derivation process described above is modified for DIMD predictor calculation by blending selected IPM and Planar / DC modes (see Figure 6 ).
[0102] It should be noted that the present invention is not limited to determining IPM statistics based on HoG, based on which one or more IPMs are selected. According to other embodiments, other methods can be applied to determine IPM statistics, for example, a Hough transform or a gradient-based Hough transform can be used on the filter window size to determine the direction for each center pixel (of the filter window), and then the histogram of the found directions can allow the selection of the peak direction (and the matching IPM).
[0103] First aspect: sample-by-sample mixing weights
[0104] Fig.10A flow chart of a DIMD predictor derivation process according to an embodiment of the first aspect of the invention is illustrated. At S220, the neighborhood of the currently processed CU 256 is analyzed to determine the number of available samples above and to the left. At S222, a histogram of gradients HoG is constructed for the entire template block 258, i.e., by filtering the samples in the template block in the manner described above with reference to FIG. 4. At S224, one or more intra-angle modes IPM are determined. In the depicted embodiment, up to two most representative intra-angle modes are selected from the HoG. As mentioned above, the HoG indicates for each IPM the amplitude accumulated over all occurrences of the corresponding IPM in the template block, and according to an embodiment, the IPM with the highest cumulative amplitude and the IPM with the second highest cumulative amplitude can be selected at S224. At S226, it is determined how many IPMs are actually selected, and if no or zero IPMs are selected, the Planar mode or DC mode, which can be the default mode, is used as the DIMD predictor, as shown in S228. If only one IPM is selected, the selected IPM is used as a DIMD predictor, as shown in S230. If two IPMs are selected, a mixing weight of the two selected modes is determined according to the cumulative amplitude ratio at S232, and a prediction for each selected IPM is determined or calculated at S234. At S236, the DIMD predictor is obtained by linearly mixing the two selected modes with the Planar mode or the DC mode according to the distance between each sample in the currently processed CU 256 and the template block 258 using the determined weights.
[0105] Therefore, in the embodiment described above, the selection of the IPM is not modified, i.e., it can be as suggested in the conventional prior art method, but the mixing weight of the IPM selected for the samples of the CU is determined by weighting a first value (e.g., weight) obtained from one or more predefined characteristics associated with the IPM (e.g., cumulative amplitude from HoG) according to the distance. The mixing weight of the Planar mode or DC mode can be determined using only the distance.
[0106] Further embodiments for determining or setting the mixing weights are now described in more detail without limiting the invention to these embodiments. According to an embodiment, when considering selecting at most two IPMs, initially the conventional processing (as described above with reference to Figure 6The mixed weights w1, w2, w3 are calculated by using the process described in the previous section. In order to obtain the DIMD predictor, the mixed weights of the first selected IPM and the second selected IPM (i.e., w1 for IPM1, w2 for IPM2, and w3 for Planar / DC mode) are modified so that the corresponding weights for weighting the cumulative amplitude ratio of the IPM are further weighted using the distance between the samples in the CU and the template block.
[0107] In other words, conventionally, for each sample at position (x, y) in the current CU 256, weighting is performed as follows:
[0108]
[0109] The resulting prediction can generally be written as:
[0110]
[0111] Where i is the index of the weight, P(x,y) is the predicted sample obtained, and M i (x, y) is the i-th IPM sample, S(x, y) is the sample of the current CU, and w i is the weight associated with the ith selected IPM. Note that the sum of the weights equals 1:
[0112]
[0113] and:
[0114]
[0115] where i and j are between 1 and N (N is the number of IPMs selected in the HoG).
[0116] Assume that d is the sample at position (x, y) in the current CU and the sample T (or a set of samples T) located in the template block i ), where d=0 for samples adjacent to the template block and d=1 for samples at the lower right corner of the current CU.
[0117] d = distance (S; T)
[0118] Therefore, samples of the current CU located on the top and left edges of the current CU can be mixed with the following weights (d=1):
[0119]
[0120] w 3,边缘 =0
[0121] And the samples of the current CU located at the lower right corner of the current CU are mixed with the following weights (d=1):
[0122] w 1,BR =0
[0123] w 2,BR =0
[0124] w 3,BR =1
[0125] Fig.11 The diagram illustrates the positions of relevant blocks and samples according to an embodiment of the present invention. The diagram illustrates the currently processed CU 256 and the adjacent template block 258, with the coordinate x extending in the horizontal direction and the coordinate y extending in the vertical direction. The position (0,0) is located at the upper left corner of the CU 256, and the position (1,1) is located at the lower right corner of the CU 256. The edge sample 256a located on the top and left edges of the CU 256 is weighted as described above, and similarly, the lower right sample 256b is weighted as described above. All samples within the remaining block 256c of the CU 256 are mixed using weights modified according to the distance of the corresponding samples in the area 258c to the template block 258. In the embodiment described above, it is assumed that the normalized distance at the upper left sample of the CU 256 is d=0, and the normalized distance at the lower right sample is d=1. According to other embodiments, for the bottom right sample, the normalized distance may be selected as 0, and for the edge sample 256a, the normalized distance may be selected as d=1.
[0126] It can be based on the sample T or T i The mathematical formulas for several distances are selected based on the positioning of the samples. In the following, two embodiments for determining the distances of corresponding samples are provided, however, the present invention is not limited to such embodiments, and any other suitable mathematical method for determining the distance between two samples may be applied. Preferably, as mentioned above, the distances are normalized for direct use in the mixing weight equations, and minimal editing of the existing formulas is performed.
[0127] According to an embodiment, only the position of the sample in the CU and the size of the CU are used to determine the distance for the sample.
[0128] Further embodiments for determining or setting the blending weight according to the distance are now described in more detail, without limiting the invention to these embodiments. According to an embodiment, the distance from the template block 158 to the lower right corner sample is considered to increase, and the distance can be determined as follows:
[0129]
[0130] Where (x, y) is the position of sample S in the current CU, where (0, 0) is the top left sample position, W is the width of the current CU 256, and H is the height of the current CU 256, such as Fig.11 shown. Fig.11 Further included is an arrow d1 indicating that the distance increases from the upper left sample to the lower right sample.
[0131] According to another embodiment, the distance may be considered from the top left sample position, the (0,0) position in the currently processed CU 256, and may be expressed as follows:
[0132]
[0133] It should be noted that the distance d depends only on the position of the considered sample in the current CU 256 and the size of the current CU 256 .
[0134] 12 illustrates distances d1 and d2 between samples in the current CU 256 and one or more samples in the template block 258 for a square CU 256. FIG12(a) shows the normalized distance d according to the first embodiment described above, and FIG12(b) shows the normalized distance d according to the second embodiment described above.
[0135] Figure 13 illustrates the corresponding distances in a similar manner to Figure 12, but for the rectangular CU 256. Likewise, Figure 13(a) illustrates the normalized distances according to the first embodiment described above, and Figure 13(b) illustrates the normalized distances according to the second embodiment described above.
[0136] According to an embodiment of the present invention, the modified weight calculation may be as follows (with floating point precision):
[0137] The proposed modified weights are calculated as follows (with floating point precision):
[0138]
[0139] w3(x,y)=d(x,y)
[0140] Where w1(x,y), w2(x,y), and w3(x,y) represent the weights associated with the sample S(x,y) at position (x,y) at distance d in the current CU. w2(x,y) and w3(x,y) can also be written as follows:
[0141] w2(x,y)=(1-d(x,y))-w1(x,y)
[0142] and
[0143] w3(x,y)=1-(w1(x,y)+w2(x,y))
[0144] The above formula can be summarized as follows, as a function of the number N of IPMs selected in the template block area, and further considering the N+1th mode as a Planar mode or a DC mode:
[0145]
[0146] w N+1 (x,y)=d(x,y)
[0147] It should be noted that:
[0148]
[0149] In the embodiments described above, the weighting coefficients may be represented using floating point arithmetic, but, according to other embodiments, the weighting coefficients may also be represented according to any other precision, and in particular, may be represented according to integer precision, such as 6-bit precision, which may be advantageous in the case where the IPM amplitude ratio is also implemented as integer arithmetic, as described, for example, in reference
[17] . According to a further embodiment, the weights are represented as powers of 2, so that the mixing process average is calculated as a right shift, rather than using division.
[0150] The embodiments described above apply the method of the present invention when at least two IPMs are used or at least two IPMs are selected from the HoG process. However, the present invention is not limited to such embodiments. Instead, the present invention can also be applied to the case where no IPM is selected from the HoG process, one or more IPMs are selected from the HoG process. If no IPM is selected, it means that the Planar mode or the DC mode is selected. If only one IPM is selected in the template block 258, the IPM can be mixed with the Planar mode or the DC mode to take into account the distance of the prediction sample in the CU 256 to the template block 258, for example as follows:
[0151] w1(x,y)=1-d(x,y)
[0152] w3(x,y)=d(x,y)
[0153] As described above, according to other embodiments, the distance d may be set to 0 at the lower right corner sample 256b, and may be set to 1 for the sample 256a adjacent to the template block 258. According to such an embodiment, when it is assumed that two IPMs are selected from the HoG and mixed with the Planar / DC mode, the weighting formula may be expressed as follows:
[0154]
[0155] w3(x,y)=1-d(x,y)
[0156] The remaining processing may not be modified, and the position-calculated weights are used during the blending process. For each sample at position (x, y) in the current CU, the prediction may be determined as follows:
[0157]
[0158] Where i is the index of the weight, P(x,y) is the predicted sample obtained, and M i (x,y) is the i-th IPM sample, S(x,y) is the sample of the current CU, and w i (x,y) is the weight associated with the ith selected IPM (there may be 2 selected angular IPMs, and the 3rd is Planar mode or DC mode).
[0159] According to an embodiment, implementing or achieving the blending process may depend on the size of the CU 256. The blending process may be restricted to not operate below a certain CU size (e.g., for CUs having a size of 4×4 or less and / or CUs exceeding a certain size, such as CUs having a size of 16×16 samples or more).
[0160] Second aspect: sample-by-sample mixing weights and local IPM
[0161] According to a second aspect of the invention, the method of the invention described above with reference to the first aspect is combined with a method according to which one or more IPMs for hybrid processing are selected for each template block area, i.e., one or more IPMs are not determined for the entire template block 258, but are determined separately for different template block areas. This method (also referred to as DIMD IPM selection per template block area) is described in a co-pending patent application by the applicant entitled "METHOD AND APPARATUS FOR DIMD INTRA PREDICTION MODE SELECTION IN ATEMPLATE AREA, AND ENCODER / DECODER INCLUDING THE SAME" filed on the same day as the present application, the contents of which are incorporated herein by reference.
[0162] Fig.14The IPM selection per template block region is illustrated. The template block 258 adjacent to the CU 256 is divided into a left template block region 258a, an upper template block region 258b, and an upper left template block region 258c. For each of the template block regions 258a to 258c, the HoG is calculated, and one or more IPMs mentioned above are selected from the peaks in the HoG of the corresponding template block region, that is, each template block region is associated with multiple IPMs. Fig.14 The embodiment is illustrated, according to which, in each template block area, an IPM has been selected from the corresponding template block HoG, i.e., a first IPM M1 has been selected for the left template block area 258a and the upper left template block area 258c, while a second IPM M2 has been selected for the upper template block area 258b. Similar to the first aspect, each weight w i With IPM M i However, according to the second aspect, the distance function d i Depends on the samples in CU 256 to the presence of IPM M i According to an embodiment, only the left template block area 258a and the upper template block area 258b for which the IPM is selected are considered, that is, the IPM selected from the upper left template area 258c can be ignored. The weighting formula described above has not changed, but the distance calculation has changed and depends on whether each selected IPM exists in the template block area.
[0163] According to an embodiment, the distance of the sample is determined using only one of the position coordinates of the sample in the CU and only one dimension of the CU.
[0164] Further embodiments of determining or setting the mixing weights according to a distance determined using only one of the position coordinates of the sample in the CU and only one dimension of the CU are now described in more detail, without limiting the invention to these embodiments. FIG. 15 illustrates an embodiment of the second aspect of the invention, according to which the distance used in the weighting function depends on the IP M in each template block area. i In Figures 15(a) and 15(b), it is assumed that the first IPM M1 exists only in the upper template block area 258b, and the second IPM M2 exists in both the left template block area 258a and the upper template block area 258b, while in Figure 15(c), the first IPM M1 exists only in the left template block area 258a, and the second IPM M2 exists only in the upper template block area 258b.
[0165] In the examples of FIG. 15( a) and FIG. 15( b), since IPM M2 exists in two template block regions, the distance calculation for w2 is the same as described above with reference to the first aspect, but the distance used to calculate the weight w2 associated with IPM M2 may be d1 or d2 as described above with reference to the first aspect, and FIG. 15( b) illustrates the normalized distance d1 described above also with reference to FIG. 13( a). On the other hand, since IPM M1 exists only on the upper template block region 258 v, the distance d1 used to calculate w1 is modified as follows:
[0166]
[0167] On the other hand, if IPM M1 only exists in the left template block area, the distance d1 used to calculate w1 is modified as follows:
[0168]
[0169] More generally, for an IPM M that exists only in one of the upper and left template blocks, i , weight w i It can be written as follows:
[0170]
[0171] Aspect 3: Combining sample-by-sample and region-by-region hybrid weighted operations
[0172] According to the third aspect, when the blending weight is calculated region by region based on the proximity of the CU region to the corresponding template block, the method of the present invention can be applied.
[0173] Fig.16An embodiment is illustrated, according to which the CU 256 is partitioned into four CU regions, namely, an upper left region TL, an upper right region TR, a lower left region BL, and a lower right region BR. The template block 258 includes a left template block region 258a, an upper template block region 258b, and an upper left template block region 258c. According to an embodiment of the third aspect, one or more IPMs are obtained from the HoG generated for the entire template block 258, and for each of the CU regions TL, TR, BL, and BR, corresponding blending weights w1, w2, w3 are calculated based on the adjacency of the corresponding region to one of the template block regions 258 to 258c. The DIMD region-wise adaptive planning described in more detail in the applicant's co-pending patent application entitled "METHOD AND APPARATUS FOR DIMD REGION-WISE ADAPTIVE BLENDING, AND ENCODER / DECODER INCLUDING THE SAME" and its contents are incorporated herein by reference. Fig.16 In the described example, the upper left CU region TL is adjacent to all template block regions, and when it is assumed that two types of IPM M1 and IPM M2 are selected, weights w1 and w2 are used for the upper left CU region TL, while the upper right CU region TR and the lower left CU region BL, which are adjacent only to the left template block region 258a and the upper template block region 258b, respectively, select IPM M1 and IPM M2 as DIMD predictors for samples in the corresponding regions. For the lower right region BR, which is not adjacent to any template block region, the Planar mode or the DC mode is provided as the DIMD predictor for samples in the region.
[0174] According to an embodiment of the third aspect of the present invention, in each CU region, the corresponding weights are adjusted and calculated as described above according to the first and second aspects of the present invention, so that, for example, in the upper left CU region TL, the weights w1, w2 can be modified according to the distance between the samples in the corresponding CU region and the template block region adjacent to the CU region. For example, for the upper left region TL, the distance can be calculated according to the embodiment of the first aspect (see, for example, FIG. 12 or FIG. 13), while for the upper right region TR and the lower left region BL, the distance can be calculated according to the embodiment of the second aspect (see, for example, FIG. 15).
[0175] According to other embodiments, only the upper left CU region TL of CU 256 applies sample-by-sample weighting as described in the previous embodiments, while other CU regions only apply region-by-region weighting, and according to such embodiments, the upper left CU region may implement the weight w3 associated with the Planar mode or the DC mode.
[0176] According to a further embodiment, for the continuity principle, the final weight (independent of the presence of the selected IPM in the template region) is related to the neighboring IPMs associated with the template block region. Fig.16 In FIG. 5 , the current BR region may have a weight w3 assigned to M2 (the upper right region of the current CU) or, in another embodiment, to M1 (the lower left region of the current CU).
[0177] Further embodiments
[0178] So far, the inventive concept has been described with reference to aspects and embodiments relating to a method for determining a DIMD predictor. According to further embodiments, the invention also provides an apparatus for deriving a DIMD predictor and an encoder / decoder comprising such an apparatus.
[0179] Fig.17 The block diagram of an apparatus for deriving a decoder-side intra-mode derivation DIMD predictor for corresponding samples of a coding unit CU of a picture is illustrated. The apparatus 300 comprises the following modules:
[0180] 302: An intra-frame prediction mode IPM selection module, wherein the intra-frame prediction mode IPM selection module is configured to select one or more IPMs in a template block adjacent to the CU.
[0181] 304: A mixing weight determination module, wherein the mixing weight determination module is configured to determine, for each sample of the CU, a mixing weight for mixing at least one or more selected IPMs according to a distance between the sample of the CU and the template block.
[0182] 306: A DIMD predictor generation module configured to generate a DIMD predictor by mixing one or more selected IPMs and a Planar mode or a DC mode using the determined mixing weights provided by the mixing weight determination module.
[0183] Fig.18 A block diagram of a decoder for decoding a coded data stream including data representing a picture according to an embodiment of the present invention is illustrated. The decoder 400 includes the following modules:
[0184] 402: A decoder module, the decoder module being configured to decode a picture from an encoded data stream.
[0185] 404: A prediction module, the prediction module comprising a device for deriving a decoder-side intra mode derivation 300 DIMD predictor for corresponding samples of a coding unit CU of a picture according to an embodiment of the present invention.
[0186] Fig.19 The block diagram of an encoder for encoding a picture into a coded data stream according to an embodiment of the present invention is illustrated, the coded data stream including data representing the picture. The encoder 500 includes the following modules:
[0187] 502: An encoder module, the encoder module is configured to receive an original picture and encode the picture into an encoded data stream.
[0188] 504: A prediction module, the prediction module comprising means for deriving a decoder-side intra mode derivation 300 DIMD predictor for corresponding samples of a coding unit CU according to an embodiment of the present invention.
[0189] Although some aspects of the disclosed concepts have been described in the context of an apparatus, it is apparent that these aspects also represent a description of a corresponding method, where a block or device corresponds to a method step or a feature of a method step. Similarly, aspects described in the context of a method step also represent a description of a corresponding block or item or feature of a corresponding apparatus.
[0190] Fig. 20 900 is a block diagram illustrating an electronic device 900 according to an embodiment of the present invention. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smart phones, wearable devices and other similar computing devices. The components shown herein, their connections and relationships, and their functions are described only as examples and are not intended to limit the embodiments of the present invention described and / or claimed herein. The device 900 includes a computing unit 901 to perform various appropriate actions and processes according to computer program instructions stored in a read-only memory (ROM) 902 or loaded from a storage unit 908 into a random access memory (RAM) 903. In the RAM 903, various programs and data for the operation of the storage device 900 can also be stored. The computing unit 901, ROM 902, and RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.
[0191] The components in the device 900 are connected to the I / O interface 905, including: an input unit 906, such as a keyboard, a mouse; an output unit 907, such as various types of displays, speakers; a storage unit 908, such as a disk, an optical disk; and a communication unit 909, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 909 allows the device 900 to exchange information / data with other devices through a computer network (such as the Internet and / or various telecommunications networks). The computing unit 901 can be composed of various general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any suitable processors, controllers, microcontrollers, etc. The computing unit 901 performs the various methods and processes described above, such as image processing methods. For example, in some embodiments, the image processing method can be implemented as a computer software program tangibly implemented on a machine-readable medium (such as a storage unit 908). In some embodiments, part or all of the computer program may be loaded and / or installed on the device 900 via the ROM 902 and / or the communication unit 909. When the computer program is loaded into the RAM 903 and executed by the computing unit 901, one or more steps of the above-described image processing method may be performed. In some embodiments, the computing unit 901 may be configured to perform the image processing method in any other suitable manner (e.g., by means of firmware).
[0192] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementation in one or more computer programs that can be executed and / or interpreted on a programmable system that includes at least one programmable processor, and the programmable processor may be a special purpose or general purpose programmable processor, and may receive data and instructions from a storage system, at least one input device, and at least one output device, and may transmit data and instructions to a storage system, at least one input device, and at least one output device.
[0193] The program code for implementing the method of the present invention can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer or other programmable data processing device, so that when the processor or controller executes the program code, the functions and / or operations specified in the flow chart and / or block diagram are executed. The program code can be executed completely on the machine, partially on the machine, partially on the machine as an independent software package and partially on a remote machine or completely on a remote machine or server.
[0194] In the context of the present invention, a machine-readable medium may be a tangible medium that may contain or store a program that is used by or in connection with an instruction execution system, device, or apparatus. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include an electrical connection based on one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
[0195] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having a display device (e.g., a cathode ray tube (CRT) or a liquid crystal display (LCD)) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user, for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and any form (including acoustic input, voice input, or tactile input) can be used to receive input from the user.
[0196] The systems and techniques described herein may be implemented on a computing system that includes a back-end component (e.g., as a data server), or on a computing system that includes a middleware component (e.g., an application server), or on a computing system that includes a front-end component (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described herein), or on a computer system that includes such back-end components, middleware components, front-end components, or any combination thereof. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), the Internet, and a blockchain network.
[0197] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the respective computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or a cloud host, which is a host product in a cloud computing service system and solves the defects of difficult management and weak business scalability in traditional physical hosts and virtual private servers (referred to as "VPS"). The server may also be a server of a distributed system, or a server combined with a blockchain.
[0198] It should be understood that steps can be reordered, added or deleted by using the various forms of processes shown above. For example, the steps described in the present invention can be performed in parallel, sequentially or in different orders, as long as the expected results of the technical solutions in the present invention can be achieved, and are not limited herein.
[0199] The specific embodiments mentioned above do not limit the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and replacements can be made according to design requirements and other factors. Any modifications, equivalent replacements and improvements made within the principles of the present invention should be included in the protection scope of the present invention.
[0200] References
[0201] [1]H.264:Advanced video coding for generic audiovisual services,https: / / www.itu.int / rec / T-REC-H.264-202108-P / en
[0202] [2]H.265: High efficiency video coding, https: / / www.itu.int / rec / T-REC-H.265-202108-P / en
[0203] [3]H.266:Versatile video coding,https: / / www.itu.int / rec / T-REC-H.266-202008-I / en
[0204] [4]AV1 Bitstream&Decoding Process Specification, http: / / aomedia.org / av1 / specification /
[0205] [5] E.Mora et al., “CE3-related: Decoder-side Intra Mode Derivation”, JVET-L0164, Macao, October 2018
[0206] [6]CE3:Decoder-side Intra Mode Derivation(tests 3.1.1,3.1.2,3.1.3and3.1.4),JVET-M0094,Marrakech,January 2019
[0207] [7] M. Abdoli et al., E. Mora, T. Guionnet, M. Raulet, Non-CE3: Decoder-side IntraMode Derivation with Prediction Fusion, JVET-N0342, Geneva, March 2019
[0208] [8] M. Abdoli et al., “Non-CE3: Decoder-side Intra Mode Derivation with Prediction Fusion Using Planar”, JVET-O0449, Gothenburg, July 2019
[0209] [9] J. Zhao et al., “EE2-Related: Improvements of Decoder-Side Intra Mode Derivation”, JVET-V0087, online, April 2021.
[0210]
[10] J. Zhao et al., “EE2 Test 2.2: DIMD with multiple blending modes”, JVET-W0054, online, July 2021.
[0211]
[11] X. Li et al., “EE2-related: Implicit derivation of DIMD blend modes”, JVET-W0067, online, July 2021.
[0212]
[12] X. Li et al., “EE2-related: A combination of CIIP and DIMD / TIMD”, JVET-W0068, online, July 2021.
[0213]
[13] J. Zhao et al., “EE2 Related–DIMD with implicitly derived multiple blending modes”, JVET-W0126, online, July 2021.
[0214]
[14] X. Li et al., “EE2-3.1: Combination of CIIP and DIMD / TIMD”, JVET-X0098, online, October 2021.
[0215]
[15] C. Zhou et al., “EE2-related: Optimization on the second mode derivation of DIMD blending mode”, JVET-X0115, online, October 2021.
[0216]
[16] T. Dumas et al., “AHG12: removing a discontinuity in the discrete angle comparison in DIMD,” JVET-X0139, online, October 2021.
[0217]
[17] X. Xiu et al., “AHG12: Removal of floating operations in DIMD and TIMD,” JVET-X0149, online, October 2021.
[0218]
[18] K. Cao et al., “Non-EE2: Fix for histogram of gradients derivation in DIMD mode”, JVET-X0156, online, October 2021.
[0219]
[19] K. Naser et al., “AhG12: Removed DIMD from MPM list of TIMD”, JVET-Y0097, online, January 2022.
[0220]
[20] S. Yoo et al., “EE2-related: Clean-up on DIMD,” JVET-Y0131, online, January 2022.
[0221]
[21] J. Zhao et al., “Non-EE2: DIMD Flag Signalling Clean-up”, JVET-Y0144, online, January 2022.
[0222]
[22] J. Pfaff et al., “Intra Prediction and Mode Coding in VVC”, IEEE Transactions on Circuits and Systems for Video Technology, Vol. 31, No. 10, October 2021.
Claims
1. A method for deriving a decoder-side intra mode derivation DIMD predictor for corresponding samples of a coding unit CU (256) of a picture (102), the method comprising: include: Selecting (S200) one or more intra prediction modes IPM in a template block (258) adjacent to the CU (256), for each sample of the CU (256), determining (S202) a blending weight for blending at least one or more selected IPMs according to a distance from the sample of the CU (256) to the template block (258), and The DIMD predictor is generated (S204) by mixing the one or more selected IPMs and the Planar mode or the DC mode using the determined mixing weight.
2. The method of claim 1 , wherein the blending weights are determined such that the samples of the CU ( 256 ) closer to the template block ( 258 ), such as prediction samples, are weighted with a higher IPM contribution, while the samples of the CU ( 256 ) further away from the template block ( 258 ) are weighted with a higher Planar or DC contribution.
3. The method of claim 1 or 2, wherein the blending weight of the selected IPM for samples of the CU (256) is determined by weighting a first value obtained from one or more predefined characteristics associated with the IPM according to the distance. The method of claim 3 , wherein only the distance is used to determine the blending weight for Planar mode or DC mode.
5. The method of any of the preceding claims, wherein the distance for a sample is determined using only the location of the sample in the CU (256) and the size of the CU (256).
6. A method according to any one of the preceding claims, wherein the one or more IPMs are selected using IPM statistics determined globally by: - an entire template block (258), the template block (258) comprising a plurality of template block regions, wherein the selection is performed globally for the entire CU (256), or - the selection comprises a global selection of an IPM set for the entire CU (256), followed by a further region-by-region selection from the IPM set for each CU region.
7. The method according to any one of claims 1 to 5, wherein the one or more IPMs are selected using IPM statistics such as a histogram of gradients HoG determined separately on each of a plurality of template block regions of the template block (258), the plurality of template block regions comprising a left template block region (258a) and an upper template block region (258b), wherein the selecting comprises a region-by-region selection from the IPMs determined separately for each CU region.
8. The method of claim 6 or 7, wherein if the selected IPM exists only in one of the template block regions, the blending weight of the selected IPM is determined according to the distance of samples in the CU (256) to the adjacent template block region in which the selected IPM exists.
9. The method of claim 8, wherein the distance for a sample is determined using only one of the position coordinates of the sample in the CU (256) and only one dimension of the CU (256).
10. The method according to any one of claims 6 to 9, comprising Splitting the CU (256) into two or more CU regions, wherein for at least one of the CU regions adjacent to the template block (258), for each sample of the CU region, a mixing weight for mixing at least the one or more selected IPMs is determined according to a distance from the sample of the CU region to the template block (258), and The DIMD predictor is generated using the hybrid weight determined for each CU region.
11. The method according to claim 9, wherein For only one or some but not all of the CU regions of the CU (256), the blending weight is determined according to the distance of the sample of the corresponding CU region to the template block (258), and for each remaining CU region, the blending weight for blending at least the one or more selected IPMs is - a global blending weight determined for the entire CU (256), or - a local blending weight determined according to the proximity of the CU region to the template block (258), and Optionally, for a CU region that is not located adjacent to the template block (258), o Select only the Planar mode as the DIMD predictor, or ○ Only the DC mode is selected as the DIMD predictor, or o using the one or more selected IPMs in one or more or all template block regions to determine the blending weights, or ○ The mixing weight is determined by weighting the mixing weights of the CU regions adjacent to the template block region.
12. A non-transitory computer-readable storage medium comprising instructions, which, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 11.
13. A device (300) for deriving a decoder-side intra-mode derivation DIMD predictor for corresponding samples of a codec unit CU (256) of a picture (102), the device (300) include: an intra-frame prediction mode IPM selection module (302), the intra-frame prediction mode IPM selection module (302) being configured to select one or more IPMs in a template block (258) adjacent to the CU (256), a blending weight determination module (304), the blending weight determination module (304) being configured to determine, for each sample of the CU (256), a blending weight for blending at least one or more selected IPMs according to a distance between the sample of the CU (256) and the template block (258); and A DIMD predictor generation module (306) is configured to generate the DIMD predictor by mixing the one or more selected IPMs and the Planar mode or the DC mode using the determined mixing weights provided by the mixing weight determination module (304).
14. A device (400) for decoding an encoded data stream (152), the encoded data stream (152) comprising data representing a picture (102), the device (400) include: a decoder module (402) configured to receive the encoded data stream (152) and decode a picture (102) from the encoded data stream (152), and A prediction module (404) comprising means (300) for deriving a decoder-side intra-mode derivation DIMD predictor for corresponding samples of a codec unit CU (256) of the picture (102) according to claim 13.
15. A device (500) for encoding a picture (102) into a coded data stream (120), the coded data stream (120) comprising data representing the picture (102), the device (500) include: an encoder module (502) configured to receive an original picture (102) and encode the picture (102) into the encoded data stream (120), and A prediction module (504) comprising means (300) for deriving a decoder-side intra-mode derivation DIMD predictor for corresponding samples of a codec unit CU (256) of the picture (102) according to claim 13.
