Matching intra transform coding and wide-angle intra prediction

By predicting the maximum prediction angle using reference samples from rectangular blocks and employing enhanced transforms, the method addresses inefficiencies in video coding standards, enhancing compression efficiency and adaptability for rectangular blocks.

JP7578584B2Active Publication Date: 2024-11-06INTERDIGITAL VC HOLDINGS INC
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
JP2021509169
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Priority Date
2018-09-21
Filing Date
2019-09-19
Publication Date
2024-11-06
Estimated Expiration
2039-09-19

AI Technical Summary

Technical Problem

Existing video coding standards face inefficiencies in handling rectangular video blocks due to limitations in intra-prediction modes, particularly with the introduction of wide-angle prediction, which requires adjustments in transform and prediction processes to accommodate varying block shapes.

Method used

The method involves using reference samples from the top row and left column of rectangular video blocks to predict the maximum prediction angle, adjusting the prediction mode based on the aspect ratio, and incorporating enhanced transforms and non-separable secondary transforms to encode and decode video blocks effectively.

Benefits of technology

This approach enhances compression efficiency by adapting prediction modes to block shapes, optimizing transform sets, and maintaining consistent encoding and decoding processes for rectangular blocks, thereby improving video coding performance.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007578584000003
    Figure 0007578584000003
  • Figure 0007578584000004
    Figure 0007578584000004
  • Figure 0007578584000005
    Figure 0007578584000005
Patent Text Reader

Abstract

A method and apparatus for using wide-angle intra prediction with position-dependent intra prediction combination. Wide-angle intra prediction enables intra prediction direction angles higher than the traditional 45 degrees. Furthermore, position-dependent intra prediction combination (PDPC) is adopted in the specifications for next-generation video coding H.266 / VVC, enabling the use of more reference pixels along the edges of a block. In one embodiment, if a video block being coded or decoded is not square, an additional intra prediction direction is enabled along the long side of the block. An index is used to indicate the prediction direction and can accommodate the additional intra prediction in the long direction with correspondingly fewer prediction directions along the short side of the block. This maintains the number of prediction modes that need to be indexed, but allows the angle of the prediction mode to correspond to the shape of the block.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] Technical Field At least one embodiment herein generally relates to a method or apparatus for encoding or decoding, compressing or decompressing video. [Background technology]

[0002] background To achieve high compression efficiency, image and video coding schemes typically exploit spatial and temporal redundancy in video content using prediction and transformation, including motion vector prediction. In general, intra- or inter-prediction is used to exploit correlation within or between frames, and then the difference between the original and predicted image, often denoted as prediction error or prediction residual, is transformed, quantized, and entropy coded. To reconstruct the video, the compressed data is decoded by the inverse process corresponding to entropy coding, quantization, transformation, and prediction.

[0003] In the development of the Versatile Video Coding (VVC) standard, the shape of the blocks can be rectangular, which leads to wide-angle intra-prediction modes. Summary of the Invention

[0004] overview At least one embodiment of the present specification relates generally to a method or apparatus for encoding or decoding video, and more specifically to a method or apparatus for interaction between a maximum transform size and a transform coding tool in a video encoder or video decoder.

[0005] According to a first aspect, a method is provided, the method including: predicting a sample of a rectangular video block using at least one of N reference samples from a top row of the rectangular video block or at least one of M reference samples from a left column of the rectangular video block, where a number of wide angles increases in proportion to an aspect ratio of the rectangular block, and when a prediction mode for the rectangular video block is set to exceed a maximum prediction angle, a prediction mode corresponding to the maximum prediction angle is used, and encoding the rectangular video block using said prediction in an intra-coding mode.

[0006] According to a second aspect, a method is provided, the method including: predicting a sample of a rectangular video block using at least one of N reference samples from a top row of the rectangular video block or at least one of M reference samples from a left column of the rectangular video block, where a number of wide angles increases in proportion to an aspect ratio of the rectangular block, and when a prediction mode for the rectangular video block is set to exceed a maximum prediction angle, a prediction mode corresponding to the maximum prediction angle is used, and decoding the rectangular video block using said prediction in an intra-coded mode.

[0007] According to another aspect, there is provided an apparatus, the apparatus including a processor, the processor may be configured to encode a block of video or decode a bitstream by performing any of the methods described above.

[0008] According to another general aspect of at least one embodiment, an apparatus is provided that includes an apparatus according to any of the decoding embodiments and at least one of: (i) an antenna configured to receive a signal, the signal including a video block; (ii) a band limiter configured to limit the received signal to a frequency band including the video block; or (iii) a display configured to display an output representative of the video block.

[0009] According to another general aspect of at least one embodiment, a non-transitory computer-readable medium is provided that includes data content generated according to any of the described coding embodiments or variations thereof.

[0010] According to another general aspect of at least one embodiment, a signal including video data generated according to any of the described encoding embodiments or variations thereof is provided.

[0011] According to another general aspect of at least one embodiment, a bitstream is formatted to include data content generated according to any of the described encoding embodiments or variations thereof.

[0012] According to another general aspect of at least one embodiment, a computer program product is provided that includes instructions that, when executed by a computer, cause the computer to perform any of the described decoding embodiments or variations thereof.

[0013] These and other aspects, features and advantages of the general aspects will become apparent from the following detailed description of illustrative embodiments, which is to be read in connection with the accompanying drawings. [Brief description of the drawings]

[0014] BRIEF DESCRIPTION OF THE DRAWINGS [Figure 1] An example of intra-directional replacement for a landscape rectangle with width exceeding height is shown, where two modes (#2 and #3) are replaced by wide-angle modes (35 and 36). [Diagram 2] A standard, general-purpose video compression scheme. [Diagram 3] A standard general-purpose video decompression method is shown. [Figure 4] 1 illustrates an example of a processor-based subsystem for implementing the general aspects described. [Diagram 5] 1 illustrates one embodiment of a method according to the described aspects. [Figure 6] 1 illustrates another embodiment of a method according to the described aspects. [Figure 7] 1 illustrates an example of an apparatus according to the described aspects. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] Detailed Description At least one embodiment of the present specification generally relates to a method or apparatus for encoding or decoding and compressing video, and more particularly relates to a portion related to transform coding of intra-prediction residuals in which enhanced multiple transforms and / or secondary transforms are used in combination with wide-angle intra-prediction.

[0016] To achieve high compression efficiency, image and video coding schemes typically exploit spatial and temporal redundancy in video content using prediction and transformation, including motion vector prediction. In general, intra- or inter-prediction is used to exploit correlation within or between frames, and then the difference between the original and predicted image, often denoted as prediction error or prediction residual, is transformed, quantized, and entropy coded. To reconstruct the video, the compressed data is decoded by the inverse process corresponding to entropy coding, quantization, transformation, and prediction.

[0017] TECHNICAL FIELD The embodiments described herein are in the field of video compression and relate to video compression as well as video encoding and decoding.

[0018] The High Efficiency Video Coding (HEVC, ISO / IEC 23008-2, ITU-T H.265) video compression standard uses motion compensated temporal prediction to exploit the redundancy that exists between successive pictures of a video.

[0019] To achieve this, a motion vector is associated with each prediction unit (PU). Each coding tree unit (CTU) is represented by a coding tree in the compressed domain, which is a quadtree decomposition of the CTU, with each leaf called a coding unit (CU).

[0020] Then, each CU is given some intra- or inter-prediction parameters (prediction information). To achieve this, the CU is spatially divided into one or more prediction units (PUs), and some prediction information is assigned to each PU. The intra- or inter-coding mode is specified at the CU level.

[0021] The Joint Video Exploration Team (JVET) proposal for a new video compression standard known as the Joint Exploration Model (JEM) proposes to accept the quad-tree binary tree (QTBT) block partitioning structure due to its high compression performance. A block in binary tree (BT) can be divided into two equal-sized sub-blocks by splitting the block horizontally or vertically in the middle. As a result, unlike blocks in QT where blocks always have a square shape with equal height and width, blocks in BT can have a rectangular shape with unequal width and height. In HEVC, the direction of angular intra prediction is defined over 180 degrees from 45 degrees to -135 degrees, and this is also preserved in JEM, where the definition of the angular direction is independent of the shape of the target block.

[0022] To code these blocks, intra prediction is used, providing an estimated version of the block using previously reconstructed neighboring samples. The difference between the source block and the prediction is then coded. In the classical codecs mentioned above, a single line of reference samples to the left and above the current block is used.

[0023] Recent work has proposed wide-angle intra prediction, which allows for intra prediction direction angles higher than the traditional 45 degrees. Furthermore, position dependent intra prediction combination (PDPC) has been adopted in the current specification for the next generation video coding H.266 / VVC.

[0024] The Joint Video Exploration Team (JVET) proposal for a new video compression standard known as the Joint Exploration Model (JEM) proposes to accept the quad-tree binary tree (QTBT) block partitioning structure due to its high compression performance. A block in binary tree (BT) can be divided into two equal-sized sub-blocks by splitting the block horizontally or vertically in the middle. As a result, unlike blocks in QT where blocks always have a square shape with equal height and width, blocks in BT can have a rectangular shape with unequal width and height. In HEVC, the direction of angular intra prediction is defined over 180 degrees from 45 degrees to -135 degrees, and the direction of angular intra prediction is also kept in JEM, which defined the angle direction independently of the shape of the target block. However, since the idea of ​​dividing coding tree units (CTUs) into CUs is to capture objects or parts of objects, and the shape of a block is related to the orientation of the object, it makes sense to adapt the predefined prediction direction according to the shape of the block to obtain higher compression efficiency. In this context, the general aspect described proposes to redefine the intra-prediction direction for a rectangular target block.

[0025] In HEVC (High Efficiency Video Coding, H.265), the coding of frames of a video sequence is based on a quadtree (QT) block partitioning structure. A frame is partitioned into square coding tree units (CTUs), and all the CTUs are subjected to a quadtree-based partitioning into multiple coding units (CUs) based on a rate-distortion (RD) criterion. Each CU is either intra predicted, i.e., spatially predicted from causally neighboring CUs, or inter predicted, i.e., temporally predicted from an already decoded reference frame. In I slices, all CUs are intra predicted, whereas in P and B slices, CUs can be either intra or inter predicted. For intra prediction, HEVC defines 35 prediction modes, including one planar mode (indexed as mode 0), one DC mode (indexed as mode 1), and 33 angular modes (indexed as modes 2 to 34). The angular modes are associated with a prediction direction ranging from 45 degrees to -135 degrees clockwise. Since HEVC supports a quadtree (QT) block partition structure, all prediction units (PUs) have a square shape. Therefore, the definition of prediction angles from 45 degrees to -135 degrees is justified in terms of the shape of the PU. For a target prediction unit with a size of NxN pixels, the size of the top reference array and the left reference array is 2N+1 samples, respectively, which is the size required to cover the above angle range for all target pixels. Considering that the height and width of the PU are of equal length, the equality of the lengths of the two reference arrays also makes sense.

[0026] For the next video coding standard, the JVET effort, known as the Joint Exploration Model (JEM), proposes the use of 65 angular intra-prediction modes in addition to the planar and DC modes. However, the prediction directions are defined over the same angular range, i.e., clockwise from 45 degrees to -135 degrees. For a target block of size WXH pixels, the size of the top and left reference arrays is (W+H+1) pixels, respectively, which is the size required to cover the above angular range for all target pixels. This definition of angles in JEM was made for simplicity rather than any other special reason. However, doing so introduced some inefficiencies.

[0027] Figure 1 shows an example of how angular intra modes are replaced by wide angular modes for non-square blocks for the 35 intra directional modes. In this example, modes 2 and 3 are replaced by wide angular modes 35 and 36, with the direction of mode 35 pointing in the opposite direction to mode 3 and the direction of mode 36 pointing in the opposite direction to mode 4.

[0028] Figure 1 shows the intra-direction permutation for the case of a landscape rectangle (with > height). In this example, two modes (#2 and #3) are replaced by wide-angle modes (35 and 36).

[0029] With 65 intra-directional modes, wide-angle intra prediction can shift up to 10 modes. When the width of a block is greater than its height, for example, modes #2 through #11 are removed and modes #67 through #76 are added based on the general embodiment described herein.

[0030] The PDPC currently adopted in the draft for the future standard H.266 / VVC applies to several intra-modes, namely planar mode, DC mode, horizontal mode, vertical mode, diagonal mode, and so-called adjacent diagonal mode, i.e. in the direction close to the diagonal. In the example of Fig. 1, the diagonal modes correspond to modes 2 and 34. If two adjacent modes per diagonal direction are added, the adjacent modes can include, for example, modes 3, 4, 32, 33. The current design of the adopted PDPC considers eight modes per diagonal, i.e. 16 adjacent diagonal modes in total. The PDPC for the diagonal and adjacent diagonal modes is described in detail below.

[0031] Wide-angle intra prediction (WAIP) has recently been adopted in the current test model for Versatile Video Coding VVC (H.266), which is expected to be the successor to H.265 / HEVC. WAIP basically adapts the range of intra-directional modes to better fit the shape of a rectangular target block. For example, when WAIP is used for a landscape block, i.e., a block whose width exceeds its height, some horizontal modes are replaced by an additional vertical mode in the opposite direction beyond antidiagonal mode #34 (-135 degrees). Similarly, for a portrait block, i.e., a block whose height exceeds its width, some vertical modes are replaced by an additional horizontal mode in the opposite direction beyond mode #2 (45 degrees). Figure 1 shows an exemplary case where modes #2 and #3 are replaced by #35 and #36, which is not considered in classical intra prediction. To support additional prediction modes, the reference array on the long side of the block is extended to twice the length of the side. On the other hand, the reference array on the short side is shortened to twice the length of the side because some modes originating from that side are eliminated.

[0032] The newly introduced modes are called wide-angle modes. Modes beyond mode number #34 (-135 degrees) are numbered sequentially as #35, #36, etc. Similarly, the newly introduced modes beyond mode #2 (45 degrees) are numbered sequentially as #1, #2, etc. Modes #0 and #1 correspond to Planar and DC in HEVC, respectively. It should be noted that the number of intra prediction modes is now extended to 67 in VVC, where modes #0 and #1 correspond to PLANAR and DC modes, and the remaining 65 modes correspond to directional modes. In WAIP, the number of directions is extended to 85, where 10 more directions are added beyond mode #66 (-135 degrees) and mode #2 (45 degrees), respectively. In this case, the modes added beyond mode #66 (-135 degrees) are numbered sequentially as #67, #68...#76. Similarly, the modes added beyond mode #2 (45 degrees) are numbered sequentially as #-1, #-2...#-10. Of the 85 directional modes, only 65 are considered for any given block. If the target block is square, the directional modes remain unchanged; i.e., the modes range from #2 to #66. If the target block is landscape and the width is equal to twice the height, the directional modes range from #8 to #72. For all other landscape blocks, i.e., blocks with a width to height ratio of 4 or more, the directional modes range from #12 to #76. Similarly, if the target block is portrait and the height is equal to twice the width, the directional modes range from #-6 to #60. For all other portrait blocks, i.e., blocks with a height to width ratio of 4 or more, the directional modes range from #-10 to #56. Since the total number of directional modes is still 65, the encoding of the mode index remains unchanged; i.e., for encoding purposes, the wide-angle modes are indexed with the same index as the corresponding original mode in the opposite direction that is removed. In other words, the wide-angle modes are mapped to the index of the original modes. For a given target block, this mapping is one-to-one, so there is no discrepancy between the coding followed by the encoder and the decoder.

[0033] When WAIP is used, the actual coding intra prediction direction corresponds to the inverse of the coding intra prediction mode index, i.e. the coding mode index is not changed and the decoder derives the actual mode while knowing the block dimensions. This has implications for other coding tools that depend on the prediction mode. The general aspects described herein consider the impact on coding of the selection and index of both enhanced multiple transforms (EMT) and non-separable secondary transforms (NSST).

[0034] Both EMT and NSST rely on intra prediction modes. For example, in EMT, there is currently a table index that maps intra modes to the appropriate transform ser. This table has a size of the number of intra modes, i.e., 67 for current VVC. In each set of EMT, four pairs of horizontal and vertical transforms are predefined. For each prediction mode, the set of NSST includes three transforms that are learned offline in addition to the identity transform (i.e., NSST is not applied). When considering WAIP, the actual prediction mode can exceed the original maximum prediction mode index (#66) and can also have negative values. As mentioned above, up to 85 intra directions are considered in the current design. Therefore, for wide-angle prediction modes, the mapping table that relates prediction modes to transform sets cannot be used directly. The general aspects described herein propose three ways to solve this problem: 1) Constant value extension: whenever the prediction mode exceeds the maximum (#66), we use the maximum prediction mode value (#66) corresponding to the transform set. Similarly, when the prediction mode is negative, we use the transform set with the lowest value of angular prediction mode (#2). 2) Mirror Extension: For prediction modes that exceed the maximum or are negative, we use the corresponding set of transformations in the opposite direction, swapping horizontal and vertical pairs. 3) Augmentation with Offline Training Data: The dependency between EMT and prediction modes is learned by offline data. A similar procedure can be followed to learn the best set for new modes by using WAIP. In addition, NSST transformation matrices for those modes can be learned and added to the existing set.

[0035] It has been recently recognized that the coding of the EMT index can be optimized by considering the index of the prediction mode. For example, different CABAC contexts can be used for each prediction mode, or even for the modes above and below the diagonal mode. In addition, different strategies can be used to code the horizontal, vertical, and diagonal modes. When WAIP is used, the same problem as in the previous section occurs, because the actual prediction mode is not the same as the one to be coded.

[0036] The general aspects described herein solve this problem in a similar manner as in the previous section, i.e. there are two solutions: 1) Constant value extension: Whenever the prediction mode exceeds the maximum value (#66), the coding of the transform set index considers the maximum prediction mode value (#66), and if the prediction mode is negative, the coding of the transform set index considers that the lowest value of the angular prediction mode (#2) is used. 2) Extension with new values: Whenever the prediction mode exceeds the maximum value (#66) or becomes negative, the coding of the transform set index utilizes these new values ​​for the CABAC context. Furthermore, these new values ​​can be used to distinguish between horizontal, vertical, and diagonal modes.

[0037] In the JEM software, the mapping between intra prediction modes and transform sets is described as follows:

[0038] For each prediction mode (from 0 to 66), we define the horizontal (g_aucTrSetHorz) and vertical (g_aucTrSetVert) mapping tables as follows:

number

[0039] As can be seen, this is an example of a dependency between intra mode and transform selection. If WAIP is used, the following solution (constant value extension) can be used: IntraMode_WAIP = GetIntraModeWAIP(IntraMode, BlkWidth, BlkHeight) IntraMode_WAIP = maximum(minimum(2, IntraMode_WAIP),66) where IntraMode is the current intra prediction mode, IntraMode_WAIP is the corrected mode according to WAIP, and may include values ​​greater than 66 and less than zero according to WAIP. This value is obtained by the function GetIntraModeWAIP which takes the width (BlkWidth) and height (BlkHeight) of the block. IntraMode_WAIP is then clipped between 2 and 66. A recent contribution proposes to encode the transform set index differently for modes beyond the diagonal modes, namely:

number

[0040] When WAIP is applied, the only modification required is to get the actual prediction mode to compare with the diagonal mode.

[0041] So the previous function: intraModeLuma = GetIntraModeWAIP(intraModeLuma, BlkWidth, BlkHeight) This should be proceeded by.

[0042] One embodiment of a method 500 according to the general aspects described herein is illustrated in Figure 5. The method begins at start block 501, where control passes to block 510 for predicting a sample of a rectangular video block using at least one of the N reference samples from a row above the rectangular video block or at least one of the M reference samples from a left column of the rectangular video block, where the number of wide angles increases in proportion to the aspect ratio of the rectangular block, and where if a prediction mode for the rectangular video block is set to exceed a maximum prediction angle, a prediction mode corresponding to the maximum prediction angle is used. Control passes from block 510 to block 520 for encoding the rectangular video block using the aforementioned prediction in an intra-coded mode.

[0043] One embodiment of a method 600 according to the general aspects described herein is illustrated in Figure 6. The method begins at start block 601, where control passes to block 610 for predicting a sample of a rectangular video block using at least one of the N reference samples from a row above the rectangular video block or at least one of the M reference samples from a left column of the rectangular video block, where the number of wide angles increases in proportion to the aspect ratio of the rectangular block, and where if a prediction mode for the rectangular video block is set to exceed a maximum prediction angle, a prediction mode corresponding to the maximum prediction angle is used. Control passes from block 610 to block 620 for decoding the rectangular video block using the aforementioned prediction in an intra-coded mode.

[0044] 7 illustrates one embodiment of a device 700 for compressing, encoding, or decoding video using improved virtual temporal affine candidates. The device includes a processor 710, which may be interconnected to a memory 720 via at least one port. Both the processor 710 and the memory 720 may also have one or more additional interconnections to external connections.

[0045] Processor 710 is also configured to insert information into a bitstream or receive information in a bitstream, and to compress, encode, or decode using any of the described aspects.

[0046] This specification describes various aspects, including tools, features, embodiments, models, methods, and the like. Many of these aspects are specifically described and often described in ways that may appear limiting, at least to illustrate their individual characteristics. However, this is for purposes of clarity of description and not to limit the application or scope of the aspects. Indeed, all of the various aspects can be combined and interchanged to produce further aspects. Additionally, aspects can also be combined and interchanged with aspects described in prior applications.

[0047] The embodiments described and contemplated herein may be implemented in many different forms. Some embodiments are illustrated in Figures 2, 3, and 4 below, although other embodiments are contemplated and the discussion of Figures 2, 3, and 4 is not intended to limit the scope of implementations. At least one of the aspects relates generally to encoding and decoding video, and at least one other aspect relates generally to transmitting a generated or encoded bitstream. These and other aspects may be implemented as a method, an apparatus, a computer-readable storage medium having stored thereon instructions for encoding or decoding video data according to any of the described methods, and / or a computer-readable storage medium having stored thereon a bitstream generated according to any of the described methods.

[0048] In this application, the terms "reconstruct" and "decode" may be used interchangeably, the terms "pixel" and "sample" may be used interchangeably, and the terms "image," "picture," and "frame" may be used interchangeably. Usually, but not always, the term "reconstruct" is used on the encoder side, whereas "decode" is used on the decoder side.

[0049] Various methods are described herein, each of which includes one or more steps or actions for achieving the described method. Unless a specific order of steps or actions is required for a method to operate properly, the order and / or use of specific steps and / or actions can be modified or combined.

[0050] Various methods and other aspects described herein may be used to modify modules, such as the intra prediction, entropy coding, and / or decoding modules (160, 360, 145, 330) of the video encoder 100 and decoder 200 shown in Figures 2 and 3. Additionally, aspects herein are not limited to VVC or HEVC, but may be applied, for example, to other standards and recommendations, existing or developed in the future, and extensions of any such standards and recommendations, including VVC and HEVC. Unless otherwise specified or technically excluded, aspects described herein may be used individually or in combination.

[0051] Various numerical values ​​are used herein, for example, {{1,0}, {3,1}, {1,1}}, and the specific values ​​are for illustrative purposes and the described aspects are not limited to those specific values.

[0052] 2 shows an encoder 100. Variations of this encoder 100 are possible, but encoder 100 is described below for clarity without describing all anticipated variations.

[0053] Before being encoded, the video sequence may be subjected to pre-encoding processing (101), for example applying a color transformation to the input color picture (e.g. from RGB 4:4:4 to YCbCr 4:2:0) or remapping the input picture components to obtain a signal distribution that is more resilient to compression (e.g. using histogram equalization of one of the color components). Metadata may be associated with the pre-processing and may be added to the bitstream.

[0054] Within the encoder 100, a picture is encoded by the encoder elements as described below. The picture to be encoded is divided (102) into units of, for example, CUs, and processed. Each unit is encoded, for example, using intra mode or inter mode. If a unit is encoded in intra mode, the intra mode performs intra prediction (160). In inter mode, motion estimation (175) and motion compensation (170) are performed. The encoder decides (105) whether to use intra mode or inter mode to encode the unit, and indicates the intra / inter decision, for example, by a prediction mode flag. A prediction residual is calculated, for example, by subtracting (110) the predicted block from the original image block.

[0055] The prediction residual is then transformed (125) and quantized (130). The quantized transform coefficients, as well as motion vectors and other syntax elements, are entropy coded (145) to output a bitstream. The encoder can skip the transform and apply quantization directly to the untransformed residual signal. The encoder can bypass both the transform and quantization, i.e., the residual is coded directly without applying the transform or quantization processes.

[0056] The encoder decodes the coded block to provide a reference for further prediction. It dequantizes (140) and inverse transforms (150) the quantized transform coefficients to decode the prediction residual. It reconstructs an image block by combining (155) the decoded prediction residual with the predicted block. It applies an in-loop filter (165) to the reconstructed picture, for example to perform deblocking / sample adaptive offset (SAO) filtering to reduce coding artifacts. It stores the filtered image in a reference picture buffer (180).

[0057] Figure 3 shows a block diagram of a video decoder 200, in which the bitstream is decoded by elements of the decoder as described below. The video decoder 200 generally performs a decoding path that is the inverse of the encoding path described in Figure 2. The encoder 100 also generally performs video decoding as part of encoding the video data.

[0058] Specifically, the decoder input includes a video bitstream, such as may be generated by video encoder 100. The bitstream is first entropy decoded (230) to obtain transform coefficients, motion vectors, and other coding information. The picture partition information indicates how the picture is partitioned. The decoder can then partition the picture according to the decoded picture partition information (235). The transform coefficients are dequantized (240) and inverse transformed (250) to decode the prediction residual. The decoded prediction residual is combined (255) with a predicted block to reconstruct an image block. The predicted block may result from intra prediction (260) or motion compensated prediction (or inter prediction) (275) (270). An in-loop filter (265) is applied to the reconstructed image. The filtered image is stored in a reference picture buffer (280).

[0059] The decoded picture may further undergo post-decoding processing (285), such as an inverse color conversion (e.g., YCbCr 4:2:0 to RGB 4:4:4) or inverse remapping that reverses the remapping process performed in the pre-encoding processing (101). The post-decoding processing may use metadata derived in the pre-encoding processing and signaled in the bitstream.

[0060] FIG. 4 illustrates a block diagram of an example of a system in which various embodiments may be implemented. System 1000 may be implemented as a device including various components described below and configured to perform one or more of the aspects described herein. Examples of such devices include, but are not limited to, various electronic devices such as personal computers, laptop computers, smartphones, tablet computers, digital multimedia set-top boxes, digital television receivers, personal video recording systems, connected consumer electronics, and servers. Elements of system 1000 may be implemented alone or in combination within a single integrated circuit, multiple ICs, and / or separate components. For example, in at least one embodiment, the processing and encoder / decoder elements of system 1000 are distributed across multiple ICs and / or separate components. In various embodiments, system 1000 is communicatively coupled to other similar systems or other electronic devices, for example, via a communication bus or by dedicated input and / or output ports. In various embodiments, system 1000 is configured to implement one or more of the aspects described herein.

[0061] The system 1000 includes at least one processor 1010 configured to execute instructions loaded therein, for example to implement various aspects described herein. The processor 1010 may include embedded memory, input / output interfaces, and various other circuits known in the art. The system 1000 includes at least one memory 1020, such as a volatile memory device and / or a non-volatile memory device. The system 1000 includes a storage device 1040, which may include non-volatile and / or volatile memory, including, but not limited to, EEPROM, ROM, PROM, RAM, DRAM, SRAM, flash, magnetic disk drives, and / or optical disk drives. The storage device 1040 may include, by way of non-limiting example, an internal storage device, an additional storage device, and / or a network-accessible storage device.

[0062] The system 1000 includes an encoder / decoder module 1030 configured to process data to provide, for example, encoded or decoded video, which may include its own processor and memory. The encoder / decoder module 1030 represents a module that may be included within a device to perform encoding and / or decoding functions. As is known, a device may include one or both of an encoding module and a decoding module. Additionally, the encoder / decoder module 1030 may be implemented as a separate element of the system 1000 or may be incorporated within the processor 1010 as a combination of hardware and software, as is known to those skilled in the art.

[0063] Program code that is loaded onto the processor 1010 or the encoder / decoder 1030 to perform various embodiments described herein may be stored in the storage device 1040 and then loaded onto the memory 1020 for execution by the processor 1010. According to various embodiments, one or more of the processor 1010, the memory 1020, the storage device 1040, and the encoder / decoder module 1030 may store one or more of a variety of items during execution of the processes described herein. Such stored items may include, but are not limited to, input video, decoded video or portions of decoded video, bitstreams, matrices, variables, and intermediate or final results of processing of equations, formulas, operations, and computational logic.

[0064] In some embodiments, memory internal to the processor 1010 and / or the encoder / decoder module 1030 is used to store instructions and provide working memory for processing required during encoding or decoding. However, in other embodiments, memory external to the processing unit (e.g., the processing unit can be the processor 1010 or the encoder / decoder module 1030) is used for one or more of these functions. The external memory can be the memory 1020 and / or the storage 1040, such as dynamic volatile memory and / or non-volatile flash memory. In some embodiments, an external non-volatile flash memory is used to store the television's operating system. In at least one embodiment, a high-speed external dynamic volatile memory such as RAM is used as working memory for video coding and decoding operations such as MPEG-2, HEVC, or Versatile Video Coding (VVC).

[0065] Input to the elements of system 1000 may be provided by various input devices, as shown in block 1130. Such input devices include, but are not limited to, (i) an RF section for receiving an RF signal transmitted wirelessly, for example by a broadcaster, (ii) a composite input, (iii) a USB input, and / or (iv) an HDMI input.

[0066] In various embodiments, the input devices of block 1130 have associated individual input processing elements known in the art. For example, the RF section may be associated with elements for (i) selecting a desired frequency (also referred to as selecting a signal or band-limiting a signal to a frequency band), (ii) down-converting the selected signal, (iii) band-limiting again to a narrower frequency band to select a signal frequency band, which may be referred to as a channel in a particular embodiment (for example), (iv) demodulating the down-converted and band-limited signal, (v) performing error correction, and (vi) demultiplexing to select a desired stream of data packets. The RF section of various embodiments includes one or more elements for performing these functions, such as a frequency selector, a signal selector, a band-limiter, a channel selector, a filter, a down-converter, a demodulator, an error corrector, and a demultiplexer. The RF section may include a tuner to perform various of these functions, including, for example, down-converting a received signal to a lower frequency (e.g., an intermediate frequency or a frequency close to baseband) or to baseband. In some set-top box embodiments, the RF section and its associated input processing elements receive RF signals transmitted over a wired (e.g., cable) medium and perform frequency selection by filtering, downconverting, and filtering again to a desired frequency band. Various embodiments rearrange the order of the above-described (and other) elements, remove some of those elements, and / or add other elements that perform similar or different functions. Adding elements may include inserting elements between existing elements, such as inserting amplifiers and analog-to-digital converters. In various embodiments, the RF section includes an antenna.

[0067] Additionally, the USB and / or HDMI terminals may include individual interface processors for connecting the system 1000 to other electronic devices across the USB and / or HDMI connections. It should be understood that various aspects of the input processing, e.g., Reed-Solomon error correction, may be implemented, for example, in a separate input processing IC or in the processor 1010. Similarly, aspects of the USB or HDMI interface processing may be implemented in a separate interface IC or in the processor 1010. The modulated, error corrected, and demultiplexed stream is provided to various processing elements, including, for example, the processor 1010 and an encoder / decoder 1030 operating in combination with memory and storage elements, to process the data stream for presentation on an output device.

[0068] The various elements of the system 1000 may be provided within a unitary housing in which the various elements may be interconnected and data may be transmitted therebetween using suitable connection arrangements 1140, such as internal buses known in the art, including an I2C bus, hardwired, and printed circuit boards.

[0069] The system 1000 includes a communication interface 1050 that enables communication with other devices over a communication channel 1060. The communication interface 1050 may include, but is not limited to, a transceiver configured to transmit and receive data over the communication channel 1060. The communication interface 1050 may include, but is not limited to, a modem or a network card, and the communication channel 1060 may be implemented in a wired and / or wireless medium, for example.

[0070] In various embodiments, data is streamed to the system 1000 using a wireless network such as IEEE 802.11. The wireless signal in these embodiments is received over a communication channel 1060 and communication interface 1050 adapted for Wi-Fi communication, for example. The communication channel 1060 in these embodiments is typically connected to an access point or router that provides access to external networks, including the Internet, to enable streaming applications and other over-the-top communications. Other embodiments provide the stream data to the system 1000 using a set-top box that delivers data over an HDMI connection in the input block 1130. Still other embodiments provide the stream data to the system 1000 using an RF connection in the input block 1130.

[0071] The system 1000 can provide output signals to various output devices, including a display 1100, speakers 1110, and other peripherals 1120. In various example embodiments, the other peripherals 1120 include one or more of a standalone DVR, a disc player, a stereo system, a lighting system, and other devices that provide functionality based on the output of the system 1000. In various embodiments, control signals are communicated between the system 1000 and the display 1100, speakers 1110, or other peripherals 1120 using signaling such as AV.Link, CEC, or other communication protocols that allow inter-device control with or without user intervention. The output devices may be communicatively coupled to the system 1000 via dedicated connections through individual interfaces 1070, 1080, and 1090. Alternatively, the output devices may be connected to the system 1000 using a communication channel 1060 via the communication interface 1050. The display 1100 and speakers 1110 may be integrated into a single unit along with the other components of the system 1000 in an electronic device, such as a television. In various embodiments, the display interface 1070 includes a display driver, such as a timing controller (T Con) chip.

[0072] For example, if the RF portion of input 1130 is part of a separate set-top box, display 1100 and speakers 1110 may alternatively be separate from one or more of the other components. In various embodiments in which display 1100 and speakers 1110 are external components, the output signal may be provided by a dedicated output connection including, for example, an HDMI port, a USB port, or a COMP output.

[0073] The embodiments may be performed by computer software implemented by the processor 1010, or by hardware, or by a combination of hardware and software. As a non-limiting example, the embodiments may be implemented by one or more integrated circuits. The memory 1020 may be of any type suitable for the technical environment and may be implemented using any suitable data storage technology, such as, as non-limiting examples, optical memory devices, magnetic memory devices, semiconductor-based memory devices, fixed memories, and removable memories. The processor 1010 may be of any type suitable for the technical environment and may include, as non-limiting examples, one or more of a microprocessor, a general-purpose computer, a special-purpose computer, and a processor based on a multi-core architecture.

[0074] Various implementations include decoding. As used herein, "decoding" may encompass all or part of the processes performed on a received encoded sequence to result in a final output suitable for, e.g., display. In various embodiments, such processes include one or more of the processes typically performed by a decoder, e.g., entropy decoding, inverse quantization, inverse transform, and differential decoding. In various embodiments, such processes also or instead include processes performed by the decoders of various implementations described herein, e.g., extracting indices of weights used for various intra-prediction reference arrays.

[0075] As a further example, in some embodiments "decoding" refers to entropy decoding only, in other embodiments "decoding" refers to differential decoding only, and in other embodiments "decoding" refers to a combination of entropy decoding and differential decoding. Whether the phrase "decoding process" is intended to refer specifically to a subset of operations or to a broader decoding process generally will be clear based on the context of the specific description and will be well understood by one of ordinary skill in the art.

[0076] Various implementations include encoding. Similar to the discussion above regarding "decoding," "encoding" as used herein may encompass all or part of the processes performed on an input video sequence, for example to result in an encoded bitstream. In various embodiments, such processes include one or more of the processes typically performed by an encoder, such as partitioning, differential encoding, transforming, quantizing, and entropy encoding. In various embodiments, such processes also or instead include processes performed by the encoders of the various implementations described herein, such as weighting of intra-prediction reference arrays.

[0077] As a further example, in some embodiments "encoding" refers only to entropy encoding, in other embodiments "encoding" refers only to differential encoding, and in other embodiments "encoding" refers to a combination of differential and entropy encoding. Whether the phrase "encoding process" is intended to refer specifically to a subset of operations or to a broader encoding process generally will be clear based on the context of the specific description and will be well understood by one of ordinary skill in the art.

[0078] It should be noted that the syntax elements used herein are descriptive terms, so they do not preclude the use of other syntax element names.

[0079] Where a drawing is shown as a flow diagram, it should be understood that the drawing also provides a block diagram of the corresponding apparatus. Similarly, where a drawing is shown as a block diagram, it should be understood that the drawing also provides a flow diagram of the corresponding method / process.

[0080] Various embodiments refer to rate-distortion calculation or rate-distortion optimization. Specifically, during the encoding process, a balance or trade-off between rate and distortion is usually considered, often given a computational complexity constraint. Rate-distortion optimization is usually formulated as minimizing a rate-distortion function, which is a weighted sum of rate and distortion. There are various approaches to solving the rate-distortion optimization problem. For example, they may be based on a comprehensive test of all encoding options, including all modes or coding parameter values ​​considered, involving a full evaluation of the coding cost and the associated distortion of the reconstructed signal after coding and decoding. Faster approaches may also be used to reduce the coding complexity, particularly by calculating the approximate distortion based on a predicted or predicted residual signal, rather than the one that is reconstructed. A mixture of these two approaches may also be used, such as by using approximate distortion for only some of the possible encoding options and full distortion for other encoding options. Other approaches evaluate only a subset of the possible encoding options. More generally, many approaches use any of a variety of techniques to perform optimization, but the optimization is not necessarily a complete assessment of both the coding cost and the distortion involved.

[0081] The implementations and aspects described herein may be implemented, for example, by a method or process, an apparatus, a software program, a data stream, or a signal. Even if only discussed in the context of a single type of implementation (e.g., only discussed as a method), the implementation of the discussed features may also be implemented in other forms (e.g., an apparatus or a program). An apparatus may be implemented, for example, by appropriate hardware, software, and firmware. A method may be implemented, for example, by a processor, where a processor generally refers to processing devices including, for example, a computer, a microprocessor, an integrated circuit, or a programmable logic device. Processors also include communication devices, such as, for example, computers, mobile phones, portable / personal digital assistants ("PDAs"), and other devices that facilitate communication of information between end users.

[0082] Reference to "one embodiment," or "an embodiment," or "one implementation," or "an implementation," as well as other variants thereof, means that a particular feature, structure, characteristic, etc. described in connection with an embodiment is included in at least one embodiment. Thus, appearances of the phrases "in one embodiment," or "in an embodiment," or "in one implementation," or "in an implementation" in various places throughout this specification, as well as any other variants, do not necessarily all refer to the same embodiment.

[0083] Additionally, this specification may refer to "determining" various pieces of information. Determining information may include, for example, one or more of estimating information, calculating information, predicting information, or retrieving information from a memory.

[0084] Additionally, this specification may refer to "accessing" various pieces of information. Accessing information may include, for example, one or more of receiving information, retrieving information (e.g., from a memory), storing information, moving information, replicating information, computing information, determining information, predicting information, or estimating information.

[0085] Additionally, this specification may refer to "receiving" various pieces of information. Receiving is intended to be broad, similar to "accessing." Receiving information may include, for example, one or more of accessing information or retrieving information (e.g., from a memory). Furthermore, "receiving" typically involves some form of operation, such as, for example, storing information, processing information, transmitting information, moving information, copying information, erasing information, calculating information, determining information, predicting information, or estimating information.

[0086] For example, the use of either " / ", "and / or" and "at least one of" in the cases of "A / B", "A and / or B" and "at least one of A and B" should be understood to be intended to encompass selecting only the first listed option (A), or selecting only the second listed option (B), or selecting both options (A and B). As a further example, in the cases of "A, B, and / or C" and "at least one of A, B, and C", such language is intended to encompass selecting only the first listed option (A), or selecting only the second listed option (B), or selecting only the third listed option (C), or selecting only the first and second listed options (A and B), or selecting only the first and third listed options (A and C), or selecting only the second and third listed options (B and C), or selecting all three options (A and B and C). As will be apparent to one skilled in the art, this representation can be expanded to include as many items as are listed.

[0087] Further, as used herein, the word "signaling" refers to indicating something to a corresponding decoder, among other things. For example, in a particular embodiment, the encoder signals a particular one of a number of weights to be used for the intra-prediction reference array. In this way, the same parameters are used at both the encoder side and the decoder side in one embodiment. Thus, for example, the encoder can transmit a particular parameter to the decoder (explicit signaling), so that the decoder can use the same particular parameter. Conversely, if the decoder already has the particular parameter along with other parameters, the signaling can be used without transmission, simply to allow the decoder to know and select the particular parameter (implicit signaling). By avoiding transmitting any actual function, bit savings are realized in various embodiments. It should be understood that signaling can be realized in various ways. For example, one or more syntax elements, flags, etc. are used to signal information to a corresponding decoder in various embodiments. Although the above content relates to the verb form of the word "signal", the word "signal" can also be used as a noun in this specification.

[0088] As will be apparent to one of ordinary skill in the art, implementations can result in various signals formatted to carry information that can be, for example, stored or transmitted. Information can include, for example, instructions for performing a method or data produced by one of the described implementations. For example, a signal can be formatted to carry a bit stream of the described embodiments. Such a signal can be formatted, for example, as an electromagnetic wave (e.g., using a radio frequency portion of the spectrum) or as a baseband signal. Formatting can include, for example, encoding a data stream and modulating a carrier wave with the encoded data stream. The information carried by the signal can be, for example, analog information or digital information. The signal can be transmitted over a variety of different wired or wireless links as is known. The signal can be stored on a processor-readable medium.

[0089] The above description has set forth several embodiments. These and further embodiments include the following optional features, taken alone or in any combination, across a variety of different claim categories and types: - using prediction directions beyond -135 degrees and 45 degrees during intra prediction when encoding and decoding -Enhancing the interaction between the wide-angle mode and the PDPC - Expanding the predicted direction in the horizontal or vertical direction while removing some directions in the opposite direction to maintain the same total number of directions -- Extending both the number of directions greater than 135 degrees and the number of directions greater than 45 degrees; - Combining PDPC and wide-angle intra prediction for samples within a block - Signaling from the encoder to the decoder which prediction direction is being used -Use a subset of prediction directions -Blocks are CUs with rectangular shape -Other blocks are neighboring blocks - a bitstream or signal containing one or more of the syntax elements described or variations thereof - Inserting syntax elements in the signaling that allow the decoder to process the bitstream in the opposite way to what the encoder did. -creating and / or transmitting and / or receiving and / or decoding bitstreams or signals that include one or more of the described syntax elements or variations thereof; - A TV, set-top box, mobile phone, tablet, or other electronic device performing any of the described embodiments A TV, set-top box, mobile phone, tablet, or other electronic device that performs any of the described embodiments and displays the resulting images (e.g., using a monitor, screen, or other type of display). A TV, set-top box, mobile phone, tablet, or other electronic device that tunes a channel (e.g., using a tuner) to receive a signal that includes the encoded image and performs any of the described embodiments. A TV, set-top box, mobile phone, tablet, or other electronic device that receives (e.g., using an antenna) a signal containing the encoded image and performs any of the described embodiments. -Various other generalized and specialized features are also supported and contemplated throughout this disclosure.

Claims

1. predicting samples of a rectangular video block, each of the samples being predicted using at least one reference sample determined based on a wide angle, the wide angle being derived from a prediction angle corresponding to a prediction mode; determining an index into a table associating each transform set with a prediction mode based on the wide angle; transforming a residual block using a transform set selected from the table by the index, the residual block representing differences between the predicted samples and each sample of the rectangular video block; and encoding said prediction modes into a bitstream; A method comprising:

2. predicting samples of a rectangular video block, each of the samples being predicted using at least one reference sample determined based on a wide angle, the wide angle being derived from a prediction angle corresponding to a prediction mode; determining an index into a table associating each transform set with a prediction mode based on the wide angle; transforming a residual block using a transform set selected from the table by the index, the residual block representing differences between the predicted samples and each sample of the rectangular video block; and encoding said prediction modes into a bitstream; A processor configured to: Including, equipment.

3. Decoding, from the bitstream, a prediction mode associated with the rectangular video block; predicting samples of the rectangular video block, each of the samples being predicted using at least one reference sample determined based on a wide angle, the wide angle being derived from a prediction angle corresponding to a prediction mode; determining an index into a table associating each transform set with a prediction mode based on the wide angle; and inverse transforming a residual block using a transform set selected from the table by the index, the residual block representing differences between the predicted samples and each sample of the rectangular video block. A method comprising:

4. Decoding, from the bitstream, a prediction mode associated with the rectangular video block; predicting samples of the rectangular video block, each of the samples being predicted using at least one reference sample determined based on a wide angle, the wide angle being derived from a prediction angle corresponding to a prediction mode; determining an index into a table associating each transform set with a prediction mode based on the wide angle; and inverse transforming a residual block using a transform set selected from the table by the index, the residual block representing differences between the predicted samples and each sample of the rectangular video block. A processor configured to: Including, equipment.

5. The method of claim 3 , wherein the wide angle is greater than −135 degrees or greater than 45 degrees.

6. The method of claim 3 , wherein the predicting comprises applying position-dependent intra-prediction combining (PDPC) to samples within the rectangular video block.

7. The method of claim 3 , wherein a predicted direction for a wide angle is expanded horizontally or vertically to include the wide angle while removing some corresponding angles in the opposite direction to maintain the same total number of angles.

8. 4. The method of claim 3, wherein determining the index comprises setting the index to a maximum value when a prediction mode corresponding to the wide angle is greater than the maximum value for a prediction mode corresponding to the prediction angle.

9. 4. The method of claim 3, wherein determining the index comprises setting the index to a minimum value when a prediction mode corresponding to the wide angle is less than a minimum value for a prediction mode corresponding to the prediction angle.

10. The method of claim 3 , wherein determining the index includes setting a prediction mode corresponding to a prediction angle opposite to the wide angle to the index.

11. The method of claim 10 , wherein the set of transforms includes a vertical transform and a horizontal transform, and the vertical transform and the horizontal transform are interchanged.

12. 4. The method of claim 3, wherein determining the index includes setting the index to select a transform set from the table, the selected transform set being predicted according to a prediction mode corresponding to the wide angle, and the prediction of the transform set being based on a learned dependency between 1) the prediction mode corresponding to the wide angle and 2) the transform set.

13. An apparatus according to claim 4; at least one of: (i) an antenna configured to receive a signal, the signal including a video block; (ii) a band limiter configured to limit the received signal to a frequency band including the video block; and (iii) a display configured to display an output representative of the video block; An apparatus comprising:

14. A non-transitory computer-readable medium having recorded thereon a program for causing a computer to execute the method according to any one of claims 1, 3, and 5 to 12.

15. A computer program comprising instructions which, when executed by a computer, cause the computer to carry out the method according to any one of claims 1, 3 and 5 to 12.

Citation Information

Patent Citations

  • Video and image coding with wide-angle intra prediction

    JP2020503785A

  • Method and apparatus for processing a video signal using a reduced transform

    JP2021511752A

  • Method and apparatus for intra-prediction of non-square blocks in video compression

    JP2021524170A

  • Video and image coding with wide-angle intra prediction

    WO2018127624A1