Coordination method and device for intra transform decoding and wide-angle intra prediction

Abstract

The invention discloses a method and a device for coordinating intra-frame transformation decoding and wide-angle intra-frame prediction. Methods and apparatus for wide angle intra prediction using a combination of location-dependent intra prediction. Wide-angle intra-frame prediction enables an intra-frame prediction direction angle to be higher than traditional 45 degrees. Furthermore, a position dependent intra prediction combination (PDPC) is employed in the specification of next generation video coding H.266 / VVC, and the PDPC enables more reference pixels along the edge of the block. In one embodiment, when a video block to be coded or decoded is non-square, an additional intra prediction direction is enabled in the direction of a longer block edge. The index is used to indicate prediction directions and may be adjusted according to additional intra prediction in longer directions with correspondingly fewer prediction directions along shorter block edges. This preserves the number of prediction modes that need to be indexed, but allows their angles to correspond to the shape of the block.

Description

[0001] This application is a divisional application of the Chinese invention patent application with the application date of September 19, 2019, application number 201980061647.6, and invention name “Coordination of intra-frame transform decoding and wide-angle intra-frame prediction”. Technical Field

[0002] At least one embodiment of the invention is generally directed to a method or apparatus for video encoding or decoding, compression or decompression. Background Art

[0003] To achieve high compression efficiency, image and video coding schemes typically employ prediction (including motion vector prediction) as well as transforms to exploit spatial and temporal redundancy in video content. Typically, intra-frame or inter-frame prediction is used to exploit intra-frame or inter-frame correlations, and then the difference between the original image and the predicted image, which is typically represented as a prediction error or prediction residual, is transformed, quantized, and entropy decoded. To reconstruct the video, the compressed data is decoded by the inverse processes corresponding to entropy decoding, quantization, transform, and prediction.

[0004] In the development of the Versatile Video Coding (VVC) standard, the block shape can be rectangular. Rectangular blocks lead to wide-angle intra prediction mode. Summary of the invention

[0005] At least one embodiment of the present invention relates to a method and apparatus for video encoding or decoding, and more particularly to a method and apparatus for interaction between a maximum transform size and a transform decoding tool in a video encoder or a video decoder.

[0006] According to a first aspect, a method is provided. The method comprises the steps of predicting samples of a rectangular video block using at least one of N reference samples from a row above the rectangular video block or at least one of M reference samples from a column to the left of the rectangular video block, wherein the number of wide angles increases in proportion to the aspect ratio of the rectangular block, wherein if a prediction mode for the rectangular video block is set to exceed a maximum prediction angle, a prediction mode corresponding to the maximum prediction angle is used; and encoding the rectangular video block using the prediction in an intra coding mode.

[0007] According to a second aspect, a method is provided. The method comprises the steps of predicting samples of a rectangular video block using at least one of N reference samples from a row above the rectangular video block or at least one of M reference samples from a column to the left of the rectangular video block, wherein the number of wide angles increases in proportion to the aspect ratio of the rectangular block, wherein if a prediction mode for the rectangular video block is set to exceed a maximum prediction angle, a prediction mode corresponding to the maximum prediction angle is used; and decoding the rectangular video block using the prediction in an intra coding mode.

[0008] According to another aspect, a device is provided. The device includes a processor. The processor can be configured to encode a block of a video or decode a bitstream by performing any one of the above methods.

[0009] According to another general aspect of at least one embodiment, there is provided an apparatus comprising an apparatus according to any of the decoding embodiments; and at least one of: (i) an antenna configured to receive a signal comprising a video block, (ii) a band limiter configured to limit the received signal to a frequency band comprising the video block, or (iii) a display configured to display an output representing the video block.

[0010] According to another general aspect of at least one embodiment, a non-transitory computer-readable medium is provided that contains data content generated according to any of the described encoding embodiments or variations.

[0011] According to another general aspect of at least one embodiment, there is provided a signal comprising video data generated according to any of the described encoding embodiments or variations.

[0012] According to another general aspect of at least one embodiment, a bitstream is formatted to include data content generated according to any of the described encoding embodiments or variations.

[0013] According to another general aspect of at least one embodiment, there is provided a computer program product comprising instructions which, when a computer executes the program, cause the computer to perform any of the described decoding embodiments or variations.

[0014] These and other aspects, features and advantages of the general aspects will become apparent from the following detailed description of illustrative embodiments which is to be read in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 An example of replacing the interior direction in the case of a flat rectangle with greater width than height is shown, where 2 patterns (#2 and #3) are replaced by wide angle patterns (35 and 36).

[0016] Figure 2 A standard, common video compression scheme is shown.

[0017] Figure 3 A standard, common video decompression scheme is shown.

[0018] Figure 4 An exemplary processor-based subsystem is shown for implementing the generally described aspects.

[0019] Figure 5 An embodiment of a method according to the described aspects is shown.

[0020] Figure 6 A further embodiment of a method according to the described aspects is shown.

[0021] Figure 7 Example apparatus according to the described aspects is shown. DETAILED DESCRIPTION

[0022] At least one embodiment of the present invention generally relates to a method or apparatus for video encoding or decoding and video compression, and more specifically, to a portion related to transform decoding of intra-frame prediction residuals, wherein enhanced multiple transforms and / or auxiliary transforms are used in conjunction with wide-angle intra-frame prediction.

[0023] To achieve high compression efficiency, image and video coding schemes typically employ prediction (including motion vector prediction) and transforms to exploit spatial and temporal redundancy in video content. Typically, intra-frame or inter-frame prediction is used to exploit intra-frame or inter-frame correlations, and then the difference between the original image and the predicted image, which is typically represented as a prediction error or prediction residual, is transformed, quantized, and entropy encoded. To reconstruct the video, the compressed data is decoded by the inverse processes corresponding to entropy encoding, quantization, transform, and prediction.

[0024] The embodiments described herein belong to the field of video compression and relate to video compression as well as video encoding and decoding.

[0025] In the HEVC (High Efficiency Video Coding, ISO / IEC 23008-2, ITU-T H.265) video compression standard, motion compensated temporal prediction is used to exploit the redundancy existing between consecutive pictures in a video.

[0026] To do this, a motion vector is associated with each prediction unit (PU). Each coding tree unit (CTU) is represented by a coding tree in the compressed domain. This is a quadtree partitioning of the CTU, where each leaf is called a coding unit (CU).

[0027] Each CU is then given some intra or inter prediction parameters (prediction information). To do this, it is spatially split into one or more prediction units (PUs), each of which is assigned some prediction information. Intra or inter coding modes are assigned on a CU level.

[0028] In the JVET (Joint Video Exploration Team) proposal for a new video compression standard, called the Joint Exploration Model (JEM), it has been proposed to accept a quadtree-binary tree (QTBT) block partitioning structure due to high compression performance. A block in a binary tree (BT) can be split into two sub-blocks of equal size by splitting it horizontally or vertically in the middle. Therefore, a BT block can have a rectangular shape with unequal width and height, unlike a block in QT, which always has a square shape with equal height and width. In HEVC, the angular intra prediction directions are defined from 45° to -135° over a 180° angle, and they have been maintained in JEM, which has made the definition of the angular direction independent of the target block shape.

[0029] To encode these blocks, intra prediction is used to provide an estimated version of the block using previously reconstructed neighboring samples. The difference between the source block and the prediction is then encoded. In the classical codecs described above, a single row of reference samples is used to the left and top of the current block.

[0030] In recent work, wide-angle intra prediction is proposed, which enables intra prediction direction angles higher than the traditional 45 degrees. In addition, position-dependent intra prediction combining (PDPC) is adopted in the current specification for the next generation video coding H.266 / VVC.

[0031] In the JVET (Joint Video Exploration Team) proposal for a new video compression standard, called the Joint Exploration Model (JEM), it has been proposed to accept a quadtree-binary tree (QTBT) block partitioning structure due to high compression performance. A block in a binary tree (BT) can be split into two sub-blocks of equal size by splitting it horizontally or vertically in the middle. Therefore, a BT block can have a rectangular shape with unequal width and height, unlike a block in a quadtree (QT), where a block always has a square shape with equal height and width. In HEVC, the angular intra prediction directions are defined from 45° to -135° over a 180° angle, and they have been maintained in JEM, which has made the definition of the angular direction independent of the target block shape. However, since the idea of ​​dividing a coding tree unit (CTU) into CUs is to capture an object or a part of an object, and the shape of a block is associated with the directionality of the object, it makes sense to adapt the defined prediction direction according to the block shape for higher compression efficiency. In this context, the described general aspects propose to redefine the intra prediction direction for rectangular target blocks.

[0032] In HEVC (High Efficiency Video Coding, H.265), the encoding of frames of a video sequence is based on a quadtree (QT) block partitioning structure. The frame is divided into square coding tree units (CTUs), all of which are subjected to quadtree-based partitioning into multiple coding units (CUs) based on the rate-distortion (RD) criterion. Each CU is intra-predicted, that is, it is spatially predicted from causally adjacent CUs, or inter-predicted, that is, it is temporally predicted from a decoded reference frame. In I slices, all CUs are intra-predicted, while in P and B slices, CUs can be intra- or inter-predicted. For intra prediction, HEVC defines 35 prediction modes, which include a plane mode (indexed as mode 0), a DC mode (indexed as mode 1), and 33 angle modes (indexed as modes 2 to 34). The angle mode is associated with a prediction direction ranging from 45 degrees to -135 degrees in the clockwise direction. Since HEVC supports a quadtree (QT) block partitioning structure, all prediction units (PUs) have a square shape. Therefore, from the perspective of PU (prediction unit) shape, the definition of prediction angles from 45 degrees to -135 degrees is reasonable. For a target prediction unit of size N×N pixels, the top reference array and the left reference array each have a size of 2N+1 samples, which is required to cover the above angle range of all target pixels. Considering that the height and width of the PU have equal lengths, it also makes sense that the lengths of the two reference arrays are equal.

[0033] For the next video coding standard, JVET's attempt as a joint exploration model (JEM) proposes to use 65 angular intra prediction modes in addition to planar mode and DC mode. However, the prediction direction is defined within the same angular range, i.e., from 45 degrees to -135 degrees in a clockwise direction. For a target block of size W×H pixels, the top reference array and the left reference array each have a pixel size of (W+H+1), which is required to be the aforementioned angular range covering all target pixels. This definition of the angle in JEM is for simplicity rather than for any other specific reason. However, doing so introduces some inefficiencies.

[0034] Figure 1 An example of how to replace the angular intra mode with the wide mode for non-square blocks in the case of 35 intra directional modes is shown. In this example, modes 2 and 3 are replaced by wide modes 35 and 36, where the direction of mode 35 points in the opposite direction of mode 3, and the direction of mode 36 points in the opposite direction of mode 4.

[0035] Figure 1 Shown is replacing the interior direction in the case of a flat rectangle (with >height). In this example, 2 modes (#2 and #3) are replaced by wide angle modes (35 and 36).

[0036] For the case of 65 intra directional modes, wide angle intra prediction can shift up to 10 modes. For example, if a block has a width greater than a height, then according to the general embodiment described herein, modes #2 to #11 are removed and modes #67 to #76 are added.

[0037] PDPC, as currently adopted in the draft of the future standard H.266 / VVC, applies to several intra modes: planar, DC, horizontal, vertical, diagonal mode and the so-called adjacent diagonal mode (i.e. a direction close to a diagonal). Figure 1 In the example of , the diagonal patterns correspond to patterns 2 and 34. If two adjacent patterns are added per diagonal direction, the adjacent patterns may include, for example, patterns 3, 4, 32, 33. In the current design of the PDPC employed, 8 patterns are considered per diagonal, i.e., a total of 16 adjacent diagonal patterns. The PDPC for diagonal and adjacent diagonal patterns is described in detail below.

[0038] Wide-angle intra prediction (WAIP) has recently been adopted in current test models for general video coding VVC (H.266), which is expected to be the successor to H.265 / HEVC. WAIP basically adapts the range of intra-directional modes to better fit the shape of rectangular target blocks. For example, when WAIP is used for flat blocks (i.e. blocks with a width greater than their height), some horizontal modes are replaced by additional vertical modes in the opposite direction beyond anti-tilt mode #34 (-135 degrees). Similarly, for tall blocks (i.e. blocks with a height greater than their width) some vertical modes are replaced by additional horizontal modes in the opposite direction beyond mode #2 (45 degrees). Figure 1 An exemplary case is shown where modes #2 and #3 are replaced by #35 and #36, which are not considered in classical intra prediction. To support the additional prediction modes, the reference array on the longer side of the block is extended to twice the length of the side. On the other hand, since some modes originating from the shorter side are removed, the reference array on the shorter side is shortened to twice the length of the side.

[0039] The newly introduced mode is called wide-angle mode. Modes beyond mode number #34 (-135 degrees) are numbered in sequence as #35, #36, and so on. Similarly, newly introduced modes beyond mode #2 (45 degrees) are numbered in sequence as #1, #2, and so on. Modes #0 and #1 correspond to planes and DC, respectively, as in HEVC. It should be noted that in the current VVC, the number of intra-frame prediction modes has been extended to 67, where modes #0 and #1 correspond to planes (PLANAR) and DC modes, and the remaining 65 modes correspond to directional modes. For WAIP, the number of directions has been extended to 85, with 10 additional directions each added beyond mode #66 (-135 degrees) and mode #2 (45 degrees). In this case, the modes added beyond mode #66 (-135 degrees) are numbered in the order of #67, #68…#76. Similarly, the modes added in addition to mode #2 (45 degrees) are sequentially numbered as modes #-1, #-2, ... #-10. Of the 85 directional modes, only 65 modes are considered for any given block. When the target block is a square, the directional mode remains unchanged. That is, the range of modes is from #2 to #66. When the target block is flat and its width is equal to twice the height, the range of directional modes is from #8 to #72. For all other flat blocks, i.e., blocks with an aspect ratio greater than or equal to 4, the range of directional modes is from #12 to #76. Similarly, when the target block is tall and its height is equal to twice the width, the range of directional modes is from #-6 to #60. For all other tall blocks, i.e., blocks with an aspect ratio greater than or equal to 4, the range of directional modes is from #-10 to #56. Since the total number of directional modes is still 65, the encoding of the mode index remains unchanged. That is, for encoding purposes, the wide-angle mode is indexed in the opposite direction with the same index as the corresponding original mode (the index is removed). In other words, the wide mode is mapped to the original mode index. For a given target block, this mapping is one-to-one, so there is no difference between the encodings followed by the encoder and decoder.

[0040] When WAIP is used, the actual coded intra prediction direction then corresponds to the direction opposite to the coded intra prediction mode index, i.e. the coding mode index remains unchanged and the decoder derives the actual mode knowing the dimensions of the block. This has an impact on other coding tools that rely on prediction modes. In the general aspects described herein, we consider the impact of the selection and coding of the indexes of both sets of enhanced multiple transforms (EMT) and non-separable sub-transforms (NSST).

[0041] Both EMT and NSST rely on intra-frame prediction modes. For example, for EMT, there is currently a table lookup that maps intra-frame modes to appropriate transform sequences. The table is the size of the number of intra-frame modes, i.e., 67 in the current VVC. In each set of EMT, 4 pairs of horizontal and vertical transforms are predefined. For each prediction mode, the NSST set contains 3 offline learned transforms in addition to the identity transform (i.e., NSST is not applied). When WAIP is considered, the actual prediction mode may exceed the original maximum prediction mode index (#66) and may also have negative values. As mentioned earlier, in the current design, up to 85 intra-frame directions are considered. Therefore, in the case of wide-angle prediction mode, the mapping table that associates prediction modes with transform sets cannot be used as is. The general aspects described in this article propose three ways to solve this problem:

[0042] 1) Constant value expansion. Whenever the prediction mode exceeds the maximum value (#66), the transform set corresponding to the maximum prediction mode value (#66) is used. Similarly, when the prediction mode is negative, the transform set of the lowest angle prediction mode value (#2) is used.

[0043] 2) Mirror extension: For prediction modes exceeding the maximum value or negative values, the transform set corresponding to the opposite direction is used, and the horizontal and vertical pairs are interchanged.

[0044] 3) Extending with offline training values: The dependencies between EMT and predicted patterns are learned through offline data. A similar process can be followed in order to learn the optimal set of new patterns resulting from the use of WAIP. Additionally, the NSST transformation matrices of these patterns can be learned,

[0045] and add it to the existing collection.

[0046] Recently, it has been noted that the encoding of the EMT index can be optimized by taking into account the prediction mode index. For example, different CABAC contexts can be used for each prediction mode or even for modes above and below the diagonal mode. In addition, different strategies can be used to encode horizontal, vertical and diagonal modes. When using WAIP, the same problem as in the previous section occurs. This is because the actual prediction mode is different from the encoded prediction mode.

[0047] The general aspects described here address this problem in a similar manner as in the previous section. That is, there are two solutions:

[0048] 1-Constant value extension: Whenever the prediction mode exceeds the maximum value (#66), the decoding of the transform set index considers the maximum prediction mode value (#66), and when the prediction mode is negative, the decoding of the transform set index considers the lowest angular prediction mode value (#2) to be used.

[0049] 2- Extension with new values: Whenever the prediction mode exceeds the maximum value (#66) or becomes negative, the decoding of the transform set index adopts these new values ​​for CABAC context. Moreover, these new values ​​can be used to distinguish between horizontal, vertical and diagonal modes.

[0050] In the JEM software, the mapping between intra prediction modes and transform sets is described as follows:

[0051] For each prediction mode (from 0 to 66), the horizontal (g_aucTrSetHorz) and vertical (g_aucTrSetVert) mapping tables are defined as:

[0052] g_aucTrSetVert

[67] =

[0053] {

[0054] 2,1,0,1,0,1,0,1,0,1,0,1,0,1,0,0,0,0,0,0,0,0,0,0,1,0,1,0,1,0,1,0,1,0,1,0,1,0,1,0,1,0,1,0,1,2,2,2,2,2,2,2,2,1,0,1,0,1,0,1,0,1,0,1,0,1,0

[0055] };

[0056] g_aucTrSetHorz

[67] =

[0057] {

[0058] 2,1,0,1,0,1,0,1,0,1,0,1,0,1,2,2,2,2,2,2,2,2,2,1,0,1,0,1,0,1,0,1,0,1,0,1,0,1,0,1,0,1,0,1,0,1,0,1,0,1,0,1,0,1,0,1,0,1,0,0,0,0,0,0,0,1,0,1,0,1,0,1,0,1,0

[0059] };

[0060] This table provides the transformed subset indices in an array of 3 subsets:

[0061] g_aiTrSubsetIntra[3][2]={{DST7,DCT8},{DST7,DCT2},{DST7,DCT2}};

[0062] For example, for the first mode (0), both the horizontal and vertical mapping tables have a value of 2 (g_aucTrSetVert[0]=2, g_aucTrSetVert[0]=2). This means that both the horizontal and vertical subsets will be {DST7, DCT8}.

[0063] As can be seen, this is an example of a dependency between intra mode and transform selection. When using WAIP, the following solution can be used (constant value expansion):

[0064] IntraMode_WAIP=GetIntraModeWAIP(IntraMode,BlkWidth,BlkHeight)

[0065] IntraMode_WAIP = Maximum(Minimum(2,IntraMode_WAIP),66)

[0066] Where IntraMode is the current intra prediction mode. IntraMode_WAIP is the correction mode due to WAIP, which can contain values ​​above 66 and below zero due to WAIP. This value is obtained by the function GetIntraModeWAIP which gets the block width (BlkWidth) and height (BlkHeight). Then, IntraMode_WAIP is clipped between 2 and 66.

[0067] Recent contributions have proposed encoding the transform set index differently for modes beyond the diagonal. That is:

[0068]

[0069] When WAIP is applied, the only modification required is to obtain the actual prediction pattern in order to compare it with the diagonal pattern.

[0070] Therefore, the previous function should continue:

[0071] intraModeLuma=GetIntraModeWAIP(intraModeLuma,BlkWidth,BlkHeight)

[0072] Figure 5 One embodiment of a method 500 according to the general aspects described herein is shown, the method starting at start block 501, and control proceeds to block 510 for predicting samples of a rectangular video block using at least one of N reference samples from a row above the rectangular video block or at least one of M reference samples from a column to the left of the rectangular video block, wherein the number of wide angles increases in proportion to the aspect ratio of the rectangular video block, wherein if the prediction mode of the rectangular video block is set to exceed a maximum prediction angle, a prediction mode corresponding to the maximum prediction angle is used. Control proceeds from block 510 to block 520 to encode the rectangular video block using the prediction in intra coding mode.

[0073] Figure 6 One embodiment of a method 600 according to the general aspects described herein is shown, the method starting at start block 601, and control proceeds to block 610 for predicting samples of a rectangular video block using at least one of N reference samples from a row above the rectangular video block or at least one of M reference samples from a column to the left of the rectangular video block, wherein the number of wide angles increases in proportion to the aspect ratio of the rectangular video block, wherein if the prediction mode of the rectangular video block is set to exceed a maximum prediction angle, a prediction mode corresponding to the maximum prediction angle is used. Control proceeds from block 610 to block 620 to decode the rectangular video block using the prediction in intra coding mode.

[0074] Figure 7 One embodiment of a device 700 for compressing, encoding or decoding video using improved virtual temporal affine candidates is shown. The device includes a processor 710 and may be interconnected to a memory 720 through at least one port. Both the processor 710 and the memory 720 may also have one or more additional interconnections to external connections.

[0075] The processor 710 is also configured to insert or receive information in a bitstream and perform compression, encoding or decoding using any of the described aspects.

[0076] Various aspects are described herein, including tools, features, embodiments, models, methods, and the like. Many of these aspects are described as being specific, and are often described in a manner that enables sound limitations, at least to illustrate individual characteristics. However, this is for the purpose of clarity of description and does not limit the application or scope of those aspects. In fact, all of the different aspects can be combined and interchanged to provide additional aspects. In addition, these aspects can also be combined and interchanged with aspects described in earlier documents.

[0077] The embodiments described and contemplated in this document may be implemented in many different forms. Figure 2 , 3 4 provide some embodiments, but other embodiments are contemplated and are Figure 2 , 3 The discussion of 4 does not limit the breadth of implementation. At least one aspect generally relates to video encoding and decoding, and at least one other aspect generally relates to transmitting a generated or encoded bitstream. These and other aspects can be implemented as methods, apparatus, a computer-readable storage medium having stored thereon instructions for encoding or decoding video data according to any of the described methods, and / or a computer-readable storage medium having stored thereon a bitstream generated according to any of the described methods.

[0078] In this application, the terms "reconstruction" and "decoding" are used interchangeably, the terms "pixel" and "sample" are used interchangeably, and the terms "image", "picture" and "frame" are used interchangeably. Usually, but not necessarily, the term "reconstruction" is used on the encoder side, while "decoding" is used on the decoder side.

[0079] Various methods are described herein, and each method includes one or more steps or actions for implementing the described method. Unless a specific order of steps or actions is required for proper operation of the method, the order and / or use of specific steps and / or actions may be modified or combined.

[0080] Various methods and other aspects described in this document can be used to modify such Figure 2 and Figure 3 Modules of the video encoder 100 and decoder 200 shown in , such as intra prediction, entropy coding and / or decoding modules (160, 360, 145, 330). In addition, aspects of the present invention are not limited to VVC or HEVC, and can be applied to, for example, other standards and recommendations, whether pre-existing or developed in the future, and extensions of any such standards and recommendations (including VVC and HEVC). Unless otherwise specified or technically excluded, the aspects described in this document can be used alone or in combination.

[0081] Various numerical values ​​are used in this document, for example, {{1, 0}, {3, 1}, {1, 1}}. The specific values ​​are for example purposes, and the described aspects are not limited to these specific values.

[0082] Figure 2 An encoder 100 is shown. Variations of the encoder 100 are contemplated, but for the sake of clarity, the encoder 100 is described below without describing all contemplated variations.

[0083] Before being encoded, the video sequence may undergo pre-encoding processing (101), for example, applying a color transform to the input color picture (e.g., conversion from RGB 4:4:4 to YCbCr 4:2:0), or performing a remapping of the input picture components in order to obtain a signal distribution that is more resilient to compression (e.g., using histogram equalization of one color component of the color components). Metadata may be associated with the pre-processing and appended to the bitstream.

[0084] In encoder 100, a picture is encoded by encoder elements as described below. The picture to be encoded is divided (102) and processed in units such as CUs. Each unit is encoded using, for example, intra or inter mode. When the unit is encoded in intra mode, it performs intra prediction (160). In inter mode, motion estimation (175) and compensation (170) are performed. The encoder decides (105) which of intra mode or inter mode to use to encode the unit, and indicates the intra / inter decision by, for example, a prediction mode flag. For example, a prediction residual is calculated by subtracting (110) the predicted block from the original image block.

[0085] The prediction residual is then transformed (125) and quantized (130). The quantized transform coefficients, along with motion vectors and other syntax elements, are entropy coded (145) to output a bitstream. The encoder may skip the transform and apply quantization directly to the untransformed residual signal. The encoder may bypass the transform and quantization, i.e., encode the residual directly without applying a transform or quantization process.

[0086] The encoder decodes the coded block to provide a reference for further prediction. The quantized transform coefficients are dequantized (140) and inverse transformed (150) to decode the prediction residual. The decoded prediction residual and the prediction block are combined (155) to reconstruct the image block. An in-loop filter (165) is applied to the reconstructed picture to perform, for example, deblocking / SAO (sample adaptive offset) filtering to reduce coding artifacts. The filtered image is stored in a reference picture buffer (180).

[0087] Figure 3 2 shows a block diagram of a video decoder 200. In the decoder 200, a bitstream is decoded by decoder elements as described below. The video decoder 200 generally performs the same Figure 2 The encoding pass described in is a decoding pass that is the inverse of the encoding pass described in . The encoder 100 also typically performs video decoding as part of encoding the video data.

[0088] In particular, the input to the decoder includes a video bitstream, which may be generated by the video encoder 100. The bitstream is first entropy decoded (230) to obtain transform coefficients, motion vectors, and other decoding information. Picture segmentation information indicates how the picture is segmented. The decoder can therefore divide (235) the picture according to the decoded picture segmentation information. The transform coefficients are dequantized (240) and inverse transformed (250) to decode the prediction residual. The decoded prediction residual is combined (255) with the prediction block to reconstruct the image block. The prediction block can be obtained (270) according to intra-frame prediction (260) or motion compensated prediction (i.e., inter-frame prediction) (275). An in-loop filter (265) is applied to the reconstructed image. The filtered image is stored in a reference picture buffer (280).

[0089] The decoded picture may further undergo post-decoding processing (285), such as inverse color conversion (e.g., conversion from YCbCr 4:2:0 to RGB 4:4:4) or performing an inverse remapping of the remapping process performed in the pre-encoding process (101). The post-decoding process may use metadata derived in the pre-encoding process and signaled in the bitstream.

[0090] Figure 4 A block diagram of an example of a system in which various embodiments are implemented is shown. System 1000 can be implemented as a device including various components described below, and is configured to perform one or more aspects described herein. Examples of such devices include, but are not limited to, various electronic devices, such as personal computers, laptop computers, smart phones, tablet computers, digital multimedia set-top boxes, digital television receivers, personal video recording systems, connected household appliances and servers. The elements of system 1000 can be implemented individually or in combination in a single integrated circuit, multiple ICs and / or discrete components. For example, in at least one embodiment, the processing and encoder / decoder elements of system 1000 are distributed on multiple ICs and / or discrete components. In various embodiments, system 1000 is coupled to other similar systems or other electronic devices via, for example, a communication bus or by a dedicated input and / or output port communication. In various embodiments, system 1000 is configured to implement one or more aspects described herein.

[0091] System 1000 includes at least one processor 1010, which is configured to execute instructions loaded therein, for implementing various aspects described herein, for example. Processor 1010 may include embedded memory, input and output interfaces, and various other circuits known in the art. System 1000 includes at least one memory 1020 (e.g., volatile memory device and / or non-volatile memory device). System 1000 includes storage device 1040, which may include non-volatile memory and / or volatile memory, including but not limited to EEPROM, ROM, PROM, RAM, DRAM, SRAM, flash memory, disk drive, and / or optical drive. As a non-limiting example, storage device 1040 may include internal storage device, attached storage device, and / or network accessible storage device.

[0092] The system 1000 includes an encoder / decoder module 1030, which is configured to, for example, process data to provide encoded video or decoded video, and the encoder / decoder module 1030 may include its own processor and memory. The encoder / decoder module 1030 represents a module (one or more) that may be included in a device to perform encoding and / or decoding functions. As is known, a device may include one or both of the encoding and decoding modules. In addition, the encoder / decoder module 1030 may be implemented as a separate element of the system 1000 or may be incorporated into the processor 1010 as a combination of hardware and software as known to those skilled in the art.

[0093] Program code to be loaded onto the processor 1010 or the encoder / decoder module 1030 to perform the various embodiments described in this document may be stored in the storage device 1040 and subsequently loaded onto the memory 1020 for execution by the processor 1010. According to various embodiments, one or more of the processor 1010, the memory 1020, the storage device 1040, and the encoder / decoder module 1030 may store one or more of the various items during the execution of the processes described herein. These stored items may include, but are not limited to, input video, decoded video or portions of decoded video, bitstreams, matrices, variables, and intermediate or final results from the processing of equations, formulas, operations, and operational logic.

[0094] In several embodiments, memory within the processor 1010 and / or encoder / decoder module 1030 is used to store instructions and provide working memory for processing required during encoding or decoding. However, in other embodiments, memory external to the processing device (e.g., the processing device may be the processor 1010 or the encoder / decoder module 1030) is used for one or more of these functions. The external memory may be memory 1020 and / or a storage device 1040, such as a dynamic volatile memory and / or a non-volatile flash memory. In several embodiments, the external non-volatile flash memory is used to store the operating system of the television. In at least one embodiment, a fast external dynamic volatile memory such as RAM is used as working memory for video encoding and decoding operations, such as MPEG-2, HEVC, or VVC (Versatile Video Coding).

[0095] As shown in block 1130, input to the elements of system 1000 may be provided through various input devices. Such input devices include, but are not limited to, (i) an RF section that receives an RF signal transmitted over the air, for example, by a broadcaster, (ii) a composite input terminal, (iii) a USB input terminal, and / or (iv) an HDMI input terminal.

[0096] In various embodiments, the input device of block 1130 has associated corresponding input processing elements known in the art. For example, the RF section may be associated with elements for: (i) selecting a desired frequency (also referred to as selecting a signal, or band limiting a signal to a frequency band), (ii) down-converting the selected signal, (iii) again band-limiting to a narrower frequency band to select a signal frequency band that may be referred to as a channel in some embodiments, (iv) demodulating the down-converted and band-limited signal, (v) performing error correction, and (vi) demultiplexing to select a desired data packet stream. The RF section of various embodiments includes one or more elements to perform these functions, such as a frequency selector, a signal selector, a band limiter, a channel selector, a filter, a down-converter, a demodulator, an error corrector, and a demultiplexer. The RF section may include a tuner that performs various of these functions, including, for example, down-converting a received signal to a lower frequency (e.g., an intermediate frequency or a near-baseband frequency) or baseband. In a set-top box embodiment, the RF part and its relevant input processing element receive the RF signal that sends by wired (for example, cable) medium, and filter to the frequency band of expectation again by filtering, down-conversion and perform frequency selection.Various embodiments rearrange the order of above-mentioned (and other) element, remove some in these elements, and / or add other elements that perform similar or different functions.Adding element can include and insert element between existing element, for example, insert amplifier and analog-to-digital converter.In various embodiments, the RF part comprises antenna.

[0097] In addition, the USB and / or HDMI terminals may include corresponding interface processors for connecting the system 1000 to other electronic devices via USB and / or HDMI connections. It should be understood that various aspects of input processing, such as Reed-Solomon error correction, may be implemented, for example, within a separate input processing IC or processor 1010. Similarly, various aspects of USB or HDMI interface processing may be implemented within a separate interface IC or within processor 1010. The demodulated, error corrected, and demultiplexed streams are provided to various processing elements, including, for example, processor 1010 and encoder / decoder 1030, which operate in conjunction with memory and storage elements to process the data streams for presentation on output devices.

[0098] The components of system 1000 may be disposed within an integrated housing, within which the components may be interconnected and data may be transmitted therebetween using suitable connection means 1140, such as an internal bus known in the art, including an I2C bus, wiring, and a printed circuit board.

[0099] The system 1000 includes a communication interface 1050 that enables communication with other devices via a communication channel 1060. The communication interface 1050 may include, but is not limited to, a transceiver configured to send and receive data through the communication channel 1060. The communication interface 1050 may include, but is not limited to, a modem or a network card, and the communication channel 1060 may be implemented, for example, within a wired and / or wireless medium.

[0100] In various embodiments, a wireless network such as IEEE 802.11 is used to transmit the data stream to the system 1000. The wireless signals of these embodiments are received through a communication channel 1060 and a communication interface 1050, for example, suitable for Wi-Fi communication. The communication channel 1060 of these embodiments is typically connected to an access point or router, which provides access to external networks including the Internet to allow streaming applications and other over-the-top communications. Other embodiments use a set-top box that transmits data through an HDMI connection of an input box 1130 to provide streaming data to the system 1000. Still other embodiments use an RF connection of an input box 1130 to provide streaming data to the system 1000.

[0101] The system 1000 can provide output signals to various output devices, including a display 1100, a speaker 1110, and other peripherals 1120. In various examples of various embodiments, the other peripherals 1120 include one or more of the following: a stand-alone DVR, a disk player, a stereo system, a lighting system, and other devices that provide functions based on the output of the system 1000. In various embodiments, control signals are transmitted between the system 1000 and the display 1100, the speaker 1110, or other peripherals 1120 using signaling, such as AV. Link, CEC, or other communication protocols that enable device-to-device control with or without user intervention. The output devices can be communicatively coupled to the system 1000 via dedicated connections through the corresponding interfaces 1070, 1080, and 1090. Alternatively, the output devices can be connected to the system 1000 via the communication interface 1050 using the communication channel 1060. The display 1100 and the speaker 1110 can be integrated into a single unit in an electronic device (e.g., a television) along with other components of the system 1000. In various embodiments, the display interface 1070 includes a display driver, such as a timing controller (T Con) chip.

[0102] For example, if the RF portion of input 1130 is part of a separate set-top box, the display 1100 and speaker 1110 may alternatively be separate from one or more of the other components. In various embodiments where the display 1100 and speaker 1110 are external components, the output signal may be provided via a dedicated output connection, including, for example, an HDMI port, a USB port, or a COMP output.

[0103] The embodiments may be performed by computer software implemented by the processor 1010, or hardware, or a combination of hardware and software. As a non-limiting example, the embodiments may be implemented by one or more integrated circuits. The memory 1020 may be of any type suitable for the technical environment and may be implemented using any appropriate data storage technology, such as optical memory devices, magnetic memory devices, semiconductor-based memory devices, fixed memory, and removable memory, as non-limiting examples. The processor 1010 may be of any type suitable for the technical environment and may include one or more of the following as non-limiting examples: a microprocessor, a general-purpose computer, a special-purpose computer, and a processor based on a multi-core architecture.

[0104] Various implementations involve decoding. As used in the present application, "decoding" may include, for example, all or part of a process performed on a received coded sequence to produce a final output suitable for display. In various embodiments, such processes include one or more of the processes typically performed by a decoder, such as entropy decoding, inverse quantization, inverse transform, and differential decoding. In various embodiments, such processes also or alternatively include processes performed by a decoder of the various embodiments described in the present application, such as extracting indices of weights to be used for various intra-frame prediction reference arrays.

[0105] As a further example, in one embodiment, "decoding" refers only to entropy decoding, in another embodiment, "decoding" refers only to differential decoding, and in another embodiment, "decoding" refers to a combination of entropy decoding and differential decoding. Whether the phrase "decoding process" is intended to refer specifically to a subset of operations or generally to a broader decoding process will be clear based on the context of the specific description and is believed to be well understood by those skilled in the art.

[0106] Various implementations involve encoding. In a manner similar to the discussion above about "decoding", "encoding" as used in this application can include, for example, all or part of a process performed on an input video sequence to produce an encoded bitstream. In various embodiments, such processes include one or more processes typically performed by an encoder, such as segmentation, differential encoding, transforms, quantization, and entropy encoding. In various embodiments, such processes also or alternatively include processes performed by encoders of various implementations described in this application, such as weighting of an intra-frame prediction reference array.

[0107] As a further example, in one embodiment, "encoding" refers only to entropy encoding, in another embodiment, "encoding" refers only to differential encoding, and in another embodiment, "encoding" refers to a combination of differential encoding and entropy encoding. Whether the phrase "encoding process" is intended to refer specifically to a subset of operations or generally to a broader encoding process will become clear based on the context of the specific description and is believed to be well understood by those skilled in the art.

[0108] Note that the syntax elements used here are descriptive terms. Therefore, they do not exclude the use of other syntax element names.

[0109] When a figure is presented as a flow chart, it should be understood that it also provides a block diagram of the corresponding apparatus. Similarly, when a figure is presented as a block diagram, it should be understood that it also provides a flow chart of the corresponding method / process.

[0110] Various embodiments relate to rate-distortion calculation or rate-distortion optimization. In particular, during the encoding process, a balance or trade-off between code rate and distortion is usually considered, usually given a constraint on computational complexity. Rate-distortion optimization is usually formulated as minimizing a rate-distortion function, which is a weighted sum of code rate and distortion. There are different methods to solve the rate-distortion optimization problem. For example, these methods can be based on extensive testing of all coding options, including all considered modes or decoding parameter values, and a complete evaluation of their decoding costs and the associated distortion of the reconstructed signal after decoding and decoding. Faster methods can also be used to save coding complexity, in particular, to calculate approximate distortion based on prediction or prediction residual signals rather than reconstructed signals. A mixture of these two methods can also be used, for example, by using approximate distortion only for some possible coding options and full distortion for other coding options. Other methods only evaluate a subset of possible coding options. More generally, many methods use any of a variety of techniques to perform optimization, but optimization is not necessarily a complete evaluation of both coding costs and associated distortions.

[0111] The implementations and aspects described herein may be implemented in, for example, a method or process, a device, a software program, a data stream, or a signal. Even if discussed only in the context of a single form of implementation (e.g., discussed only as a method), the implementation of the features discussed may also be implemented in other forms (e.g., a device or program). For example, a device may be implemented with appropriate hardware, software, and firmware. The method may be implemented in, for example, a processor, which generally refers to a processing device, including, for example, a computer, a microprocessor, an integrated circuit, or a programmable logic device. The processor also includes a communication device, such as a computer, a cellular phone, a portable / personal digital assistant ("PDA"), and other devices that facilitate information communication between end users.

[0112] References to "one embodiment," or "an embodiment," or "implementation," or "an implementation," and other variations mean that a particular feature, structure, characteristic, etc. described in connection with the embodiment is included in at least one embodiment. Thus, the appearances of the phrases "in one embodiment," or "in an embodiment," or "in one implementation," or "in an implementation," and any other variations, in various places throughout this document are not necessarily all referring to the same embodiment.

[0113] Additionally, this document may refer to "determining" various pieces of information. Determining information may include, for example, one or more of: estimating the information, calculating the information, predicting the information, or retrieving the information from a memory.

[0114] Additionally, this document may refer to "accessing" various pieces of information. Accessing information may include, for example, one or more of: receiving information, retrieving information (e.g., from memory), storing information, moving information, copying information, calculating information, determining information, predicting information, or estimating information.

[0115] Additionally, this document may refer to "receiving" various information. Like "accessing," receiving is intended to be a broad term. Receiving information may include, for example, one or more of: accessing the information or retrieving the information (e.g., from a memory). Furthermore, during operations such as storing information, processing information, sending information, moving information, copying information, erasing information, calculating information, determining information, predicting information, or estimating information, "receiving" is often involved in one way or another.

[0116] It should be understood that, for example, in the case of "A / B," "A and / or B," and "at least one of A and B," use of any of the following " / ," "and / or," and "at least one of" is intended to encompass selection of only the first listed option (A), or only the second listed option (B), or both options (A and B). As a further example, in the case of "A, B, and / or C" and "at least one of A, B, and C," such wording is intended to include selection of only the first listed option (A), or only the second listed option (B), or only the third listed option (C), or only the first and second listed options (A and B), or only the first and third listed options (A and C), or only the second and third listed options (B and C), or all three options (A and B and C). This can be extended to multiple items listed, as will be apparent to one of ordinary skill in this and related arts.

[0117] In addition, as used herein, the word "signal / signaling" especially refers to a corresponding decoder indicating something. For example, in some embodiments, the encoder signals a specific one of a plurality of weights to be used for an intra-frame prediction reference array. In this way, in an embodiment, the same parameters are used on the encoder side and the decoder side. Therefore, for example, the encoder can send (explicit signaling) specific parameters to the decoder so that the decoder can use the same specific parameters. On the contrary, if the decoder already has specific parameters as well as other parameters, signaling can be used without sending (implicit signaling) to simply allow the decoder to know and select specific parameters. Bit savings are achieved in various embodiments by avoiding the transmission of any actual function. It should be understood that signaling can be implemented in various ways. For example, in various embodiments, one or more syntax elements, flags, etc. are used to signal information to the corresponding decoder. Although the foregoing relates to the verb form of the word "signal / signaling", the word "signal / signaling" can also be used as a noun in this article.

[0118] As will be apparent to one of ordinary skill in the art, implementations may generate various signals formatted to carry information that may be stored or transmitted, for example. The information may include, for example, instructions for executing a method, or data generated by one of the described implementations. For example, a signal may be formatted to carry a bitstream of the described embodiments. Such a signal may be formatted as, for example, an electromagnetic wave (e.g., using a radio frequency portion of a spectrum) or a baseband signal. Formatting may include, for example, encoding a data stream and modulating a carrier with the encoded data stream. The information carried by the signal may be, for example, analog or digital information. As is known, the signal may be transmitted over a variety of different wired or wireless links. The signal may be stored on a processor readable medium.

[0119] The foregoing description has described a number of embodiments. These and further embodiments include the following optional features, which, alone or in any combination, span a variety of claim categories and types:

[0120] - Use prediction directions during intra prediction in encoding and decoding above -135 degrees and 45 degrees

[0121] -Extended interaction between wide angle mode and PDPC

[0122] - Extend the prediction directions horizontally or vertically while removing some directions in the opposite direction to maintain the same number of total directions

[0123] - Expanded the number of directions to beyond -135 degrees and beyond 45 degrees

[0124] - Combine PDPC and wide-angle intra prediction to samples within a block

[0125] - Signaling from the encoder to the decoder which prediction directions are being used

[0126] -Use a subset of prediction directions

[0127] - The block is a CU with a rectangular shape

[0128] - The other block is an adjacent block

[0129] - a bitstream or signal comprising one or more of the described syntax elements or variations thereof.

[0130] - inserting in the signalling syntax elements enabling the decoder to process the bitstream in the opposite way to that performed by the encoder.

[0131] - creating and / or sending and / or receiving and / or decoding a bitstream or signal comprising one or more of the described syntax elements or variations thereof.

[0132] -TV, set-top box, cell phone, tablet computer, or other electronic device implementing any of the described embodiments.

[0133] - A TV, set-top box, cell phone, tablet computer, or other electronic device that executes any of the described embodiments and displays (eg, using a monitor, screen, or other type of display) the resulting images.

[0134] - A TV, set-top box, cellular phone, tablet computer, or other electronic device that tunes (eg, using a tuner) a channel to receive a signal including an encoded image, and performs any of the described embodiments.

[0135] - A TV, set-top box, cellular phone, tablet computer, or other electronic device that receives (e.g., using an antenna) a signal including an encoded image and performs any of the described embodiments.

[0136] - Various other generalized and specialized features are also supported and contemplated in the present disclosure.

Claims

1. A method comprising: predicting a rectangular video block based on a wide-angle prediction mode derived from a prediction mode, wherein the prediction mode is one of the prediction modes corresponding to the corresponding non-wide-angle; Based on the wide-angle prediction mode, determining an index to a table associating the prediction mode with a corresponding transform set; obtaining a block of coefficients from a residual block using a transform set selected from the table by the index; as well as The prediction mode is encoded into a bitstream.

2. An apparatus comprising a processor, wherein the processor is configured to: predicting a rectangular video block based on a wide-angle prediction block derived from a prediction mode, wherein the prediction mode is one of the prediction modes corresponding to the corresponding non-wide-angle; Based on the wide-angle prediction block, determining an index to a table associating the prediction modes with corresponding transform sets; obtaining a block of coefficients from a residual block using a transform set selected from the table by the index; as well as, The prediction mode is encoded into a bitstream.

3. A method comprising: decoding a prediction mode associated with a rectangular video block from a bitstream; predicting the rectangular video block based on a wide-angle prediction mode derived from the prediction mode, wherein the prediction mode is one of the prediction modes corresponding to corresponding non-wide-angle; Based on the wide-angle prediction mode, determining an index to a table associating the prediction mode with a corresponding transform set; as well as, A residual block is obtained from the coefficient block using a transform set selected from the table by the index.

4. An apparatus comprising a processor, the processor being configured to: decoding a prediction mode associated with a rectangular video block from a bitstream; predicting the rectangular video block based on a wide-angle prediction mode derived from the prediction mode, wherein the prediction mode is one of the prediction modes corresponding to corresponding non-wide-angle; Based on the wide-angle prediction mode, determining an index to a table associating the prediction mode with a corresponding transform set; as well as, A residual block is obtained from the coefficient block using a transform set selected from the table by the index.

5. The method according to claim 1 or 3, or the device according to claim 2 or 4, wherein: The wide angle corresponding to the wide-angle prediction mode exceeds −135 degrees or exceeds 45 degrees.

6. The method according to claim 1 or claim 3, or the apparatus according to claim 2 or claim 4, wherein: The forecast also includes: Position-dependent intra prediction combining (PDPC) is applied to samples within the rectangular video block.

7. The method according to claim 1 or 3, or the device according to claim 2 or 4, wherein: The prediction directions for the wide angles including the wide angle corresponding to the wide angle prediction mode are extended in the horizontal or vertical direction while removing a corresponding number of angles in the opposite direction to maintain the same number of total angles.

8. The method according to claim 1 or claim 3, or the apparatus according to claim 2 or claim 4, wherein: Determining the index includes: When the wide-angle prediction mode is greater than a maximum value for the encoded prediction mode, the index is set to a maximum value.

9. The method according to claim 1 or claim 3, or the apparatus according to claim 2 or claim 4, wherein: Determining the index includes: When the wide-angle prediction mode is less than a minimum value for the encoded prediction mode, the index is set to the minimum value.

10. The method according to claim 1 or claim 3, or the apparatus according to claim 2 or claim 4, wherein: Determining the index includes: The index is set to a prediction mode corresponding to a non-wide angle in an opposite direction of a wide angle corresponding to the wide angle prediction mode.

11. The method according to claim 10 or the apparatus according to claim 10, wherein: The transform set includes a horizontal transform and a vertical transform, and the horizontal transform and the vertical transform are interchanged.

12. The method according to claim 1 or claim 3, or the apparatus according to claim 2 or claim 4, wherein: Determining the index includes: The index is set to select a transform set from the table, wherein the selected transform set is predicted based on 1) a wide-angle prediction mode and 2) a learned dependency relationship between the transform sets.

13. An apparatus comprising: The apparatus according to any one of claims 4 to 12; as well as At least one of: (i) an antenna configured to receive a signal, the signal comprising the video block, (ii) a frequency band limiter configured to limit the received signal to a frequency band comprising the video block, and (iii) a display configured to display an output representing the video block.

14. A non-transitory computer-readable medium storing a computer program, the computer program causing a computer to execute the method of any one of claims 1, 3, and 5 to 12.

15. A computer program product comprising instructions which, when said program is executed by a computer, cause said computer to perform the method according to any one of claims 1, 3 and 5 to 12.

Images