Coding of intra-frame geometric partition modes with template matching

Abstract

Encoding of geometric prediction modes with template matching is implemented with embodiments of improved encoding and decoding. In one embodiment, the intra mode index and the partition mode are collectively encoded, allowing the uniquely generated index to be classified and the index with less cost to be encoded over a shorter code length. In another embodiment, classification of intra prediction modes of allowed intra geometric partition modes is based on template matching costs, so that the cheapest mode is encoded with fewer bits.

Description

[0001] Cross-references to related applications This application claims the benefit of European patent application serial number 23305509.4, filed on 7 April 2023, the entire contents of which are incorporated herein by reference. Technical Field

[0002] At least one embodiment in this section generally relates to a method or apparatus for video encoding or decoding, compression or decompression. Background Technology

[0003] To achieve high compression efficiency, image and video coding schemes typically employ prediction (including motion vector prediction) and transform to utilize spatial and temporal redundancy in the video content. Generally, intra-frame or inter-frame prediction is used to leverage intra-frame or inter-frame correlations, and then the differences between the original and predicted images (often expressed as prediction errors or prediction residuals) are transformed, quantized, and entropy-coded. To reconstruct the video, the compressed data is decoded through the inverse processes corresponding to entropy coding, quantization, transform, and prediction. Summary of the Invention

[0004] At least one embodiment in this discussion generally relates to a method or apparatus for video encoding or decoding, and more particularly, to a method or apparatus for encoding or decoding intra-frame geometric patterns (GPM) with template matching.

[0005] According to a first aspect, a method is provided. The method includes the steps of: assigning an index representing a combination of intra-frame modes and segmentation modes for a video block; encoding the index based on cost; setting a flag when the index is in a list of most probable modes; and encoding the video block using the list of most probable modes.

[0006] According to the second aspect, another method is provided. The method includes the steps of: determining when a tag representing a combination of intra-frame mode and segmentation mode for a video block is valid; parsing an encoded index to determine the combination of intra-frame mode and segmentation mode from a list of most probable modes; and decoding the video block using the most probable modes.

[0007] According to another aspect, an apparatus is provided. The apparatus includes a processor. The processor can be configured to encode video blocks or decode video data by performing any of the methods described above.

[0008] According to another general aspect of at least one embodiment, there is provided an apparatus comprising: a device according to any of the described encoding embodiments; and at least one of (i) an antenna configured to receive a signal, the signal comprising a video block, (ii) a band limiter configured to limit the received signal to a frequency band comprising the video block, and (iii) a display configured to display an output representative of the video block.

[0009] According to another general aspect of at least one embodiment, there is provided a non-transitory computer readable medium containing data content produced according to any of the described encoding embodiments or variants.

[0010] According to another general aspect of at least one embodiment, there is provided a signal comprising video data produced according to any of the described encoding embodiments or variants.

[0011] According to another general aspect of at least one embodiment, video data or a bitstream is formatted to include data content produced according to any of the described encoding embodiments or variants.

[0012] According to another general aspect of at least one embodiment, there is provided a computer program product comprising instructions which, when the program is executed by a computer, cause the computer to carry out any of the described decoding embodiments or variants.

[0013] These and other aspects, features, and advantages of the general aspects will become apparent to those of ordinary skill in the art from the following detailed description of exemplary embodiments taken in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0014] Figure 1 A standard general purpose video compression scheme is illustrated.

[0015] Figure 2 A standard general purpose video decompression scheme is illustrated.

[0016] Figure 3 A processor-based system for encoding / decoding under the general described aspects is illustrated.

[0017] Figure 4 Geometric partitioning is illustrated.

[0018] Figure 5 Geometric partitioning using angles 12 and distances 0 to 3 is illustrated.

[0019] Figure 6 Angles proposed for GEO with their corresponding width:height ratios are illustrated.

[0020] Figure 7 Figure illustrates geometric partition mode (GPM) with inter and intra prediction and available IPM candidates (a to c); (d) example of GPM with inter and intra prediction.

[0021] Figure 8 Figure illustrates one example of edges on a template.

[0022] Figure 9 illustrates spatial GPM candidates.

[0023] Figure 10 Figure illustrates spatial GPM template.

[0024] Figure 11 Figure illustrates GPM blending.

[0025] Figure 12 Figure illustrates the existing decoding order when one partition is intra GPM, which is compared to the proposed decoding order that combines GPM mode and intra index.

[0026] Figure 13 Figure illustrates an embodiment of the first method under the described aspects.

[0027] Figure 14 Figure illustrates an embodiment of the second method under the described aspects.

[0028] Figure 15 Figure illustrates one embodiment of the apparatus under the described aspects. DETAILED DESCRIPTION

[0029] The embodiments described herein are in the field of video compression and relate generally to video compression and video encoding and decoding, more particularly, to improve compression efficiency compared to existing video encoding systems.

[0030] To achieve high compression efficiency, image and video encoding schemes typically employ prediction (including motion vector prediction) and transform to exploit spatial and temporal redundancy in video content. Generally, intra or inter prediction is used to exploit intra or inter frame correlation, and then the difference between the original image and the predicted image (often denoted as prediction error or prediction residual) is transformed, quantized and entropy coded. To reconstruct the video, the compressed data is decoded by inverse processes corresponding to the entropy coding, quantization, transform and prediction.

[0031] In the HEVC (High Efficiency Video Coding) video compression standard, motion-compensated temporal prediction is employed to exploit the redundancy existing between successive pictures of a video.

[0032] To this end, a motion vector is associated to each prediction unit (PU), which is now introduced. In the compressed domain, each CTU (coding tree unit) is represented by a coding tree. This is a quad-tree partitioning of the CTU, where each leaf is called a coding unit (CU).

[0033] Each CU is then endowed with some intra- or inter-prediction parameters (prediction information). To achieve this, it is spatially partitioned into one or more prediction units (PU), each PU being assigned some prediction information. At the CU level, an intra- or inter- coding mode is assigned.

[0034] In HEVC, exactly one motion vector is assigned to each PU. This motion vector is used for the motion-compensated temporal prediction of the PU under consideration.

[0035] In the Versatile Video Coding (VVC) decoder developed by the JVET (Joint Video Exploration Team), CUs are no longer partitioned into PUs or TUs, and some motion data is directly assigned to each CU. In this new codec design, CUs can be partitioned into sub-CUs, for each of which a motion vector is computed.

[0036] Geometric merge mode (GEO), also called geometric partitioning mode (GPM) In this section and in the following, the acronyms GEO and GPM are both used interchangeably to refer to the geometric merge partitioning mode.

[0037] In VVC, the geometric merge mode (GEO) is supported, with 32 angles and 5 distances. For a total of 32 angles available for GEO, the angles are quantized between 0 and 360 degrees, with a step equal to 11.25 degrees. With the angle and the distance , the description of the geometric partitioning is depicted in Figure 4 .

[0038] From the maximum possible distance , with a fixed step, the distance is quantized, which indicates the distance with respect to the center of the block. For the distance , only the first half of the angles is available, because in this case the partitioning is symmetric. For example, the result of the geometric partitioning using the angle 12 and the distances 0 to 3 is depicted in Figure 5 .

[0039] For the distance Symmetric angles 16 to 31 are removed as they correspond to the same partition as 0-15. Angles 0 and 8 are also excluded as they are similar to binary partition of a CU, leaving only 14 angles for distance 0. Thus, at most 142 partition modes can be used by the geometric partitioning (14 + 32*4 = 142).

[0040] To simplify the GEO partitioning process, the angles in GEO are replaced by angles with a power of 2 as tangent. Since the tangent of the proposed angles is a power of 2 number, most of the multiplications can be replaced by bit-shifting. With the proposed angles, one row or column is needed to store each block size and each partition mode, as depicted in FIG. 1. Figure 6

[0041] Geometric partition mode (GPM) with template matching (TM) Template matching is applied to GPM. When GPM mode is enabled for a CU, a CU level flag is signaled to indicate whether TM is applied to both geometric partitions. With TM, the motion information of each geometric partition is refined. When TM is selected, a template is constructed using the left, top or both left and top neighboring samples according to the partition angle, as shown in Table 1. Using the same search pattern of merge mode, the motion is then refined by minimizing the difference between the current template and the template in the reference picture, with the half-pel interpolation filter disabled.

[0042] Table 1. Template for the first and second geometric partitions, where A stands for using the above sample, L stands for using the left sample, and L+A stands for using both the left and above samples. Figure 7 Figure 7 0 2 3 4 5 8 11 12 13 14 Table 2. Position of available neighboring blocks for IPM candidate derivation based on the angle of the GPM block boundary. A and L indicate the above and left of the prediction A A A A block. Angle of GPM First partition L+A A A L+A L+A L+A Second partition L L L L L+A L+A L+A L+A 16 18 19 20 21 24 27 28 29 30 L+A A A A A L+A Figure 7 Figure 7 Table 2. Position of available neighboring blocks for IPM candidate derivation based on the angle of the GPM block boundary. A and L indicate the above and left of the prediction A A block. Angle of GPM First partition L+A L L L L L+A L+A L+A

[0043] The GPM candidate list is constructed as follows: First, the interleaved List-0 (List 0) MV candidates and List-1 (List 1) MV candidates are derived directly from the regular merge candidate list, with the List-0 MV candidates having higher priority than the List-1 MV candidates. A pruning method with adaptive threshold based on the current CU size is applied to remove redundant MV candidates.

[0044] Second, the interleaved List-1 MV candidates and List-0 MV candidates are further derived directly from the regular merge candidate list, with the List-1 MV candidates having higher priority than the List-0 MV candidates. The same pruning method with adaptive threshold is also applied to remove redundant MV candidates.

[0045] Third, zero MV candidates are padded until the GPM candidate list is full.

[0046] ​For a GPM CU, GPM-MMVD and GPM-TM are exclusively enabled. This is achieved by first signaling the GPM-MMVD syntax. When both GPM-MMVD control flags are equal to false (i.e., GPM-MMVD is disabled for both GPM partitions), the GPM-TM flag is signaled to indicate whether template matching is applied to both GPM partitions. Otherwise (at least one GPM-MMVD flag is equal to true), the value of the GPM-TM flag is inferred to be false.

[0047] GPM with inter and intra prediction In GPM with inter and intra prediction, the final prediction samples are generated by weighting the inter prediction samples and the intra prediction samples for each GPM separate region. The inter prediction samples are derived by inter GPM, while the intra prediction samples are derived by an intra prediction mode (IPM) candidate list and an index signaled from the encoder. The IPM candidate list size is predefined to be 3. The available IPM candidates are respectively as Second partition The parallel angular mode (parallel mode) for GPM block boundary, the vertical angular mode (vertical mode) for GPM block boundary and the planar mode as shown in (a)-(c). In addition, the direct motion vector and IPM storage on GPM mixed region are introduced to further improve the coding performance. L+A GPM with inter and intra prediction as shown in d is constrained to reduce the signaling overhead of IPM and avoid the increase of the size of intra prediction circuit on hardware decoder. In addition, the direct motion vector and IPM storage on GPM mixed region are introduced to further improve the coding performance.

[0048] In DIMD (decoder-side intra mode derivation) and neighboring mode based IPM derivation, the parallel mode is registered first. Therefore, at most two IPM candidates derived from the decoder-side intra mode derivation (DIMD) method and / or neighboring blocks can be registered if there is no identical IPM candidate in the list. Regarding the neighboring mode derivation, there are at most five positions of available neighboring blocks, but they are subject to the angular constraint of GPM block boundary as shown in Table 2, which have already been used for GPM with template matching (GPM-TM).

[0049] L+A L+A L+A 0 2 3 4 5 8 11 12 13 14 L+A A A A A L+A Figure 8 Figure 10 Figure 11 A A Figure 12 Figure 13 Figure 14 Figure 15 L L L L Figure 1 Figure 1 Figure 1 Figure 2 16 18 19 20 21 24 27 28 29 30 Figure 1 A A A A Figure 2 Figure 1 Figure 3 Figure 3 A A ​ ​ ​ ​ L L L L ​ ​ ​

[0050] Intra GPM can be combined with GPM with merge with motion vector difference (GPM-MMVD). TIMD (template-based intra mode derivation) is used for the IPM candidates of intra GPM to further improve the coding performance. The parallel mode can be registered first, then the IPM candidates of TIMD, DIMD and neighboring blocks are registered.

[0051] Reordering of GPM partition modes based on template matching In the reordering of GPM partition modes based on template matching, given the motion information of the current GPM block, the respective TM cost value of the GPM partition modes is computed. Then, based on the TM cost values, all GPM partition modes are reordered in ascending order. Instead of sending the GPM partition modes, an index is signaled which indicates with a Golomb-Rice code where the exact GPM partition mode is located in the reordered list.

[0052] The reordering method of GPM partition modes is a two-step process executed after the respective reference templates of the two GPM partitions in the coding unit are generated as follows.

[0053] First, the GPM partition edges are extended into the reference templates of the two GPM partitions, resulting in 64 reference templates and computing the respective TM cost for each of the 64 reference templates.

[0054] Next, the GPM partition modes are reordered in ascending order based on their TM cost values and the best 32 partition modes are marked as available partition modes.

[0055] The edges on the template are extended from the edges of the current CU as shown in ​ but in the template area across the edges, no GPM blending process is used. After the ascending reordering using the TM cost, an index is signaled.

[0056] Spatial geometry partition mode (SGPM) The SGPM is an intra mode similar to the inter coding tool of GPM, where two prediction parts are generated from the intra prediction process. In this mode, a candidate list is built, each entry containing one partition split and two intra prediction modes, as shown in Fig. 9. 26 split modes and 3 intra prediction modes are used to form these combinations. The length of the candidate list is set to equal 16. The selected candidate index is signaled.

[0057] Using the template ​ , the list is reordered, where the SAD between the prediction and the reconstruction of the template is used for the ordering. The template size is fixed to 1.

[0058] For each split mode, an IPM list is derived, TIMD, horizontal mode and vertical mode. The IPM list size is set to 3. In this list, the TIMD derived mode is replaced by 2 derived modes with horizontal and vertical orientation.

[0059] The SGPM mode is applied with constrained block sizes: 4 <= width <= 64, 4 <= height <= 64, width < height * 8, height < width * 8, width * height >= 32.

[0060] Adaptive blending is also used for spatial GPM, where the blending depth T is derived as follows ​ • If min(width, height) == 4, then 1 / 2 T is selected • Else if min(width, height) == 8, then T is selected • Else if min(width, height) == 16, then 2 T is selected • Else if min(width, height) == 32, then 4 T is selected • Else, 8 T is selected In current ECM-8.0, when using geometric partition mode (GPM), template matching (TM) is used to classify split modes and inter merge candidates, but not intra candidates. Template matching is also not used to distinguish between using intra partition or inter partition.

[0061] It is proposed to use template matching to improve the selection and coding of intra GPM.

[0062] One embodiment proposes to jointly code the intra index and split mode to allow classification of uniquely produced indices and to code indices with smaller cost on a shorter code length.

[0063] One embodiment proposes to classify the allowed intra prediction modes in intra GPM based on their template matching cost, so that the cheapest mode is coded on fewer bits.

[0064] Improved signaling of intra index Jointly code split mode with intra candidates It is proposed to jointly code the split mode and intra index when one partition uses intra GPM, as described in ​

[0065] One unique index is assigned to each pair "intra index + GPM split mode" allowed on the current block. In some embodiments, the TM cost of each unique index is computed (in a variant, from the method described in another section, or in another variant, using all templates). Smaller code length is assigned to unique indices with smaller cost.

[0066] ​​In some embodiments, N (e.g., N = 16) modes are selected as the most probable modes (MPM). The coding of the unique index then uses a flag to indicate whether the selected unique index is part of the N MPMs. If it is part of the N MPMs, a truncated binary code of N symbols is used to decode which one of the N MPMs is selected. Otherwise, a truncated binary code of M-N symbols is used, M being the total number of "intra index + GPM split mode" pairs (in ECM-8.0, there are 3 allowed intra modes and 64 GPM modes, so M = 192).

[0067] In some embodiments, by not allowing combinations of intra index + GPM split mode with too high TM cost (i.e., not likely to be used), a fixed number N (e.g., N = 16) of unique indices are allowed to further save some bits.

[0068] In some embodiments, a single syntax element is used, which is used to signal the selected split mode, the prediction type (inter or intra) on each partition, and the two prediction modes. In this case, after each GPM partition (split + prediction index 0 + prediction index 1) is classified according to its cost, a single index is able to signal which mode is going to be used. This index can use, for example, a truncated unary code to favor lower indices. In some embodiments, a flag first indicates whether one of the partitions is intra. In this embodiment, according to whether the GPM partition has two inter partitions or whether the GPM partition uses intra / inter, the GPM partitions have been classified in two separate lists. A code that favors lower indices (e.g., truncated unary) is then used to signal which split and which predictors are used.

[0069] Reordering of intra candidates among intra candidates In ECM-8.0, there are three eligible intra prediction modes: planar, TIMD, DIMD. Instead of always having the same order, they can be classified according to their TM cost. Based on the embodiments described in the above previous sections, the template matching cost can be computed.

[0070] One embodiment of the method 1300 under the general aspects described herein is shown in ​The method begins, in an embodiment, at start box 1301 and control proceeds to box 1310 to assign an index representing an intra mode and partition mode combination for a video block. Control proceeds from box 1310 to box 1320 to encode the index based on cost. Control proceeds from box 1320 to box 1330 to set a flag when the index is in a most probable mode list. Control proceeds from box 1330 to box 1340 to encode the video block using the most probable mode list.

[0071] One embodiment of a method 1400 under the general aspects described herein is shown in ​ The method begins, in an embodiment, at start box 1401 and control proceeds to box 1410 to determine when a flag representing an intra mode and partition mode combination for a video block is valid. Control proceeds from box 1410 to box 1420 to parse an encoded index to determine the intra mode and partition mode combination from a most probable mode list. Control proceeds from box 1420 to box 1430 to decode the video block using the most probable mode.

[0072] ​ One embodiment of an apparatus 1500 for encoding, decoding, compressing, or decompressing video data using encoding with intra geometric partition mode with template matching is shown. The apparatus includes a processor 1510 and can be interconnected to a memory 1520 through at least one port. The processor 1510 and the memory 1520 can also each have one or more additional interconnections to external connections.

[0073] The processor 1710 is also configured to insert information in a bitstream or receive information in a bitstream and compress, encode, or decode using any of the aspects described.

[0074] The embodiments described herein include various aspects, including tools, features, embodiments, models, schemes, and so on. Many of these aspects are described specifically, and often in a manner that can sound limiting, at least to illustrate individual characteristics. However, this is for purposes of clarity of description, and not to limit the application or scope of those aspects. In fact, all of the different aspects can be combined and interchanged to provide further aspects. Moreover, aspects can also be combined with aspects described in earlier documents.

[0075] The aspects described and contemplated in this application can be implemented in many different forms. ​ 、 2 and 3 provide some embodiments, but other embodiments are contemplated, and ​ 、 2The discussion of FIGs. 3 does not limit the breadth of implementations. At least one of the aspects generally relate to video encoding and decoding, and at least one other aspect generally relates to communicating a generated or encoded bitstream. These and other aspects can be implemented as methods, devices, computer-readable storage media having stored thereon instructions for encoding or decoding video data according to any of the described methods, and / or computer-readable storage media having stored thereon a bitstream generated according to any of the described methods.

[0076] In this application, the terms “reconstruct” and “decode” can be used interchangeably, the terms “pixel” and “sample” can be used interchangeably, and the terms “image,” “picture,” and “frame” can be used interchangeably. Generally, but not necessarily, the term “reconstruct” is used on the encoder side, while “decode” is used on the decoder side.

[0077] Various methods are described herein, and each of the described methods includes one or more steps or actions for achieving the described method. Unless otherwise specified, the ordering of specific steps and / or actions of a method can be modified or combined, and / or the specific steps and / or actions can be combined with other steps and / or actions.

[0078] Various methods and other aspects described in this application can be used to modify modules of a video encoder 100 and decoder 200 as shown in ​ and ​ FIGs. 1 and 2. For example, the intra prediction, entropy encoding and / or decoding modules (160, 360, 145, 330) of the video encoder 100 and decoder 200. Moreover, the present aspects are not limited to VVC or HEVC, and can be applied, for example, to other standards and recommendations (whether existing or future developed) and extensions of any such standards and recommendations, including VVC and HEVC. Unless otherwise indicated or technically precluded, the aspects described in this application can be used individually or in combination.

[0079] Various numerical values are used in this application. The particular values are for exemplary purposes, and the described aspects are not limited to these particular values.

[0080] ​ An encoder 100 is illustrated. Variations of such an encoder 100 are contemplated, but the encoder 100 is described below for purposes of clarity without describing all desired variations.

[0081] Before being encoded, the video sequence can undergo pre-encoding processing (101), for example, applying a color transform to the input color pictures (e.g., from RGB 4:4:4 to YCbCr 4:2:0), or performing a remapping of the input picture components in order to obtain a more compression-resistant signal distribution (e.g., using a histogram equalization of one of the color components). Metadata can be associated with the pre-processing and attached to the bitstream.

[0082] In the encoder 100, the pictures are encoded by the encoder elements as described below. The pictures to be encoded are partitioned (102) and processed in units, for example, CUs. Each unit is encoded using, for example, intra or inter mode. When a unit is encoded in intra mode, it performs intra prediction (160). In inter mode, motion estimation (175) and compensation (170) are performed. The encoder decides (105) which of the intra or inter mode is to be used for encoding the unit, and indicates the intra / inter decision by, for example, a prediction mode flag. The prediction residual is calculated, for example, by subtracting (110) the prediction block from the original image block.

[0083] The prediction residual is then transformed (125) and quantized (130). The quantized transform coefficients, as well as the motion vectors and other syntax elements, are entropy coded (145) to output the bitstream. The encoder can skip the transform, and apply quantization directly to the non-transformed residual signal. The encoder can bypass both the transform and quantization, i.e., the residual is directly encoded without applying the transform or quantization process.

[0084] The encoder decodes the encoded blocks in order to provide references for further prediction. The quantized transform coefficients are dequantized (140) and inverse transformed (150) to decode the prediction residual. Combining (155) the decoded prediction residual and the prediction block, the image block is reconstructed. In-loop filters (165) are applied to the reconstructed pictures to perform, for example, deblocking / SAO (Sample Adaptive Offset) filtering to reduce coding artifacts. The filtered images are stored in the reference picture buffer (180).

[0085] ​ A block diagram of a video decoder 200 is shown. In the decoder 200, the bitstream is decoded by the decoder elements as described below. The video decoder 200 generally performs a decoding pass that is reciprocal to the encoding pass as described in ​ The encoder 100 also generally performs video decoding as part of encoding video data.

[0086] In particular, the input to the decoder includes a video bitstream, which can be produced by the video encoder 100. The bitstream is first entropy decoded (230) to obtain transform coefficients, motion vectors, and other coded information. Picture partitioning information indicates how the pictures are partitioned. From the decoded picture partitioning information, the decoder can therefore divide (235) the pictures. The transform coefficients are dequantized (240) and inverse transformed (250) to decode the prediction residuals. Combining (255) the decoded prediction residuals and the prediction blocks, the image blocks are reconstructed. From either intra prediction (260) or motion-compensated prediction (i.e., inter prediction) (275), the prediction blocks can be obtained (270). In-loop filters (265) are applied to the reconstructed images. The filtered images are stored in the reference picture buffer (280).

[0087] The decoded pictures can also go through post-decoding processing (285), e.g., inverse color transform (e.g., from YcbCr 4:2:0 to RGB 4:4:4) or inverse remapping, which performs the inverse of the remapping process performed in the pre-encoding processing (101). The post-decoding processing can use metadata that is derived in the pre-encoding processing and signaled in the bitstream.

[0088] ​ A block diagram illustrating an example of a system in which various aspects and embodiments can be implemented. The system 1000 can be embodied as a device including the various components described below and configured to perform one or more of the aspects described in this document. Examples of such devices include, but are not limited to, various electronic devices such as personal computers, laptop computers, smartphones, tablet computers, digital multimedia set top boxes, digital television receivers, personal video recording systems, connected home appliances, and servers. Independently or in combination, the elements of system 1000 can be embodied in a single integrated circuit (IC), multiple ICs, and / or discrete components. For example, in at least one embodiment, the processing and encoder / decoder elements of system 1000 are distributed across multiple ICs and / or discrete components. In various embodiments, the system 1000 is communicatively coupled to one or more other systems, or other electronic devices, e.g., via communication buses or through dedicated input and / or output ports. In various embodiments, the system 1000 is configured to implement one or more of the aspects described in this document.

[0089] The system 1000 includes at least one processor 1010 configured to execute instructions loaded therein for implementing the various aspects described in this document, for example. Processor 1010 can include embedded memory, input output interface, and various other circuitry as known in the art. The system 1000 includes at least one memory 1020 (e.g., a volatile memory device and / or a non-volatile memory device). The system 1000 includes a storage device 1040, which can include non-volatile memory and / or volatile memory including, but not limited to, Electrically Erasable Programmable Read-Only Memory (EEPROM), Read-Only Memory (ROM), Programmable Read-Only Memory (PROM), Random Access Memory (RAM), Dynamic Random Access Memory (DRAM), Static Random Access Memory (SRAM), flash, magnetic disk drive, and / or optical disk drive. The storage device 1040 can include an internal storage device, an attached storage device (including detachable and non-detachable storage devices), and / or a network accessible storage device, as non-limiting examples.

[0090] The system 1000 includes an encoder / decoder module 1030 configured, for example, to process data to provide encoded video or decoded video, and the encoder / decoder module 1030 can include its own processor and memory. The encoder / decoder module 1030 is representative of the module(s) that can be included in a device to perform the encoding and / or decoding functions. As is well known, a device can include one or both of the encoding and decoding modules. Additionally, the encoder / decoder module 1030 can be implemented as a separate element of the system 1000, or can be included within the processor 1010 as a combination of hardware and software, as is known to those skilled in the art.

[0091] Program code to be loaded onto the processor 1010 or the encoder / decoder 1030 to perform the various aspects described in this document can be stored in the storage device 1040 and then loaded onto the memory 1020 for execution by the processor 1010. In accordance with various embodiments, one or more of the processor 1010, the memory 1020, the storage device 1040, and the encoder / decoder module 1030 can store one or more of various items during the execution of the processes described in this document. Such stored items can include, but are not limited to, input video, decoded video or portions of decoded video, bitstreams, matrices, variables, and intermediate or final results from the processing of equations, formulas, operations, and operational logic.

[0092] In some embodiments, the processor 1010 and / or memory internal to the encoder / decoder module 1030 is used to store instructions and provide working memory for processing needed during encoding or decoding. However, in other embodiments, memory located external to the processing device, e.g., the processing device can be the processor 1010 or the encoder / decoder module 1030, is used for one or more of these functions. The external memory can be the memory 1020 and / or storage 1040, e.g., dynamic volatile memory and / or non-volatile flash memory. In several embodiments, external non-volatile flash memory is used to store, for example, the operating system of the television. In at least one embodiment, fast external dynamic volatile memory, such as RAM, is used as working memory for video encoding and decoding operations, such as for MPEG-2 (MPEG stands for Moving Picture Experts Group, MPEG-2 is also known as ISO / IEC 13818, and 13818-1 is also known as H.222, and 13818-2 is also known as H.262), HEVC (HEVC stands for High Efficiency Video Coding, also known as H.265 and MPEG-H Part 2), or VVC (Versatile Video Coding, a new standard developed by the JVET Joint Video Expert Team).

[0093] Input to the elements of the system 1000 can be provided through various input devices as indicated in block 1130. Such input devices include, but are not limited to: (i) a radio frequency (RF) portion that receives RF signals transmitted over the air, e.g., by a broadcaster, (ii) a component (COMP) input terminal (or set of COMP input terminals), (iii) a universal serial bus (USB) input terminal, and / or (iv) a high-definition multimedia interface (HDMI) input terminal. ​ Other examples, not shown in FIG. 11, include composite video.

[0094] In various embodiments, the input devices of block 1130 have associated respective input processing elements, as is known in the art. For example, the RF portion can be associated with elements adapted to: (i) select a desired frequency (also referred to as selecting a signal or band-limiting a signal to a band of frequencies), (ii) downconvert the selected signal, (iii) band-limit again to a narrower band of frequencies to select, for example, a signal frequency band which can be referred to in some embodiments as a channel, (iv) demodulate the downconverted and band-limited signal, (v) perform error correction, and (vi) demultiplex to select a desired stream of data packets. The RF portion of various embodiments includes one or more elements to perform these functions, for example, frequency selectors, signal selectors, band-limiters, channel selectors, filters, downconverters, demodulators, error correctors, and demultiplexers. The RF portion can include a tuner that performs various of these functions, including, for example, downconverting the received signal(s) to lower frequency(ies) (e.g., an intermediate frequency or a near-baseband frequency) or to baseband. In one set-top box embodiment, the RF portion and its associated input processing elements receive RF signals carried on a wired (e.g., cable) medium and perform frequency selection by filtering, downconverting, and filtering again to a desired frequency band. Various embodiments rearrange the order of the above-described (and other) elements, remove some of these elements, and / or add other elements performing similar or different functions. Adding elements can include inserting elements between existing elements, such as, for example, inserting amplifiers and an analog-to-digital converter. In various embodiments, the RF portion includes an antenna.

[0095] In addition, the USB and / or HDMI terminals can include respective interface processors for connecting the system 1000 to other electronic devices across USB and / or HDMI connections. It is to be understood that various aspects of input processing (e.g., Reed-Solomon error correction) can be implemented, for example, within separate input processing ICs or within the processor 1010, as desired. Similarly, aspects of USB or HDMI interface processing can be implemented within separate interface ICs or within the processor 1010, as desired. The demodulated, error corrected, and demultiplexed streams are provided to various processing elements, including, for example, the processor 1010 and the encoder / decoder 1030, which operate in conjunction with memory and storage elements to process the data streams as desired for presentation on output devices.

[0096] The various elements of the system 1000 can be provided within an integrated housing in which the various elements can be interconnected and communicate data therebetween using suitable connection arrangements, for example, internal buses (including Inter-IC (I2C) buses), wiring, and printed circuit boards, as is known in the art.

[0097] The system 1000 includes a communication interface 1050 that enables communication with other devices via a communication channel 1060. The communication interface 1050 can include, but is not limited to, a transceiver configured to transmit and to receive data via the communication channel 1060. The communication interface 1050 can include, but is not limited to, a modem or network card, and the communication channel 1060 can be implemented, for example, within a wired and / or a wireless medium.

[0098] In various embodiments, data is streamed or otherwise provided to the system 1000 using a wireless network, such as a Wi-Fi network, e.g., IEEE 802.11 (IEEE refers to the Institute of Electrical and Electronics Engineers). Wi-Fi signals of these embodiments are received via the communication channel 1060 and the communication interface 1050 adapted for Wi-Fi communication. The communication channel 1060 of these embodiments is typically connected to an access point or router that provides access to external networks, including the Internet, to allow streaming applications and other over-the-top communications. Other embodiments provide streamed data to the system 1000 using a set-top box that delivers the data via the HDMI connection of the input block 1130. Still other embodiments provide streamed data to the system 1000 using the RF connection of the input block 1130. As indicated above, various embodiments provide data in a non-streaming manner. In addition, various embodiments use wireless networks other than Wi-Fi, e.g., a cellular network or a Bluetooth network.

[0099] The system 1000 can provide output signals to various output devices, including a display 1100, speakers 1110, and other peripheral devices 1120. The display 1100 of various embodiments includes one or more of, for example, a touchscreen display, an organic light-emitting diode (OLED) display, a curved display, and / or a foldable display. The display 1100 can be used in a television, a tablet computer, a laptop computer, a cellular telephone (mobile phone), or another device. The display 1100 can also be integrated with other components (e.g., as in a smartphone), or separate (e.g., an external monitor for a laptop computer). In various examples of embodiments, the other peripheral devices 1120 include one or more of a digital video recorder (or digital versatile disk) (DVR, for both terms), a disc player, a stereo system, and / or a lighting system. Various embodiments use one or more of the peripheral devices 1120 to provide functionality based on the output of the system 1000. For example, a disc player performs the functionality of playing the output of the system 1000.

[0100] In various embodiments, control signals are transmitted to the system 1000 and display 1100, speaker 1110, or other peripheral devices 1120 using signaling such as AV.Link, Consumer Electronics Control (CEC), or other communications protocols that can enable control of devices by the system 1000 in the case of user intervention or without user intervention. The output devices can be communicatively coupled to the system 1000 through the respective interfaces 1070, 1080, and 1090 via dedicated connections, Alternatively, the output devices can be connected to the system 1000 through the communications interface 1050 using the communications channel 1060. In an electronic device such as, for example, a television, the display 1100 and speaker 1110 can be integrated in a single unit with the other components of the system 1000. In various embodiments, the display interface 1070 includes a display driver such as, for example, a timing controller (T Con) chip.

[0101] For example, if the RF portion of the input 1130 is part of a separate set-top box, the display 1100 and speaker 1110 can be separate from one or more of the other components. In various embodiments in which the display 1100 and speaker 1110 are external components, the output signals can be provided via dedicated output connections including, for example, HDMI ports, USB ports, or COMP outputs.

[0102] Embodiments can be implemented by computer software by the processor 1010, or by hardware, or by a combination of hardware and software. As a non-limiting example, embodiments can be implemented by one or more integrated circuits. The memory 1020 can have any of a variety of suitable architectures, and can be implemented using any suitable data storage technology, such as optical, magnetic or semiconductor storage technology, including fixed and removable storage media. As a non-limiting example, the processor 1010 can have any of a variety of suitable architectures, including one or more of a microprocessor, a general purpose computer, a special purpose computer, and a processor based on a multi-core architecture.

[0103] Various implementations relate to decoding. As used in this application, "decoding" can include all or a part of the processes performed, for example, on a received encoded sequence, in order to produce a final output suitable for display. In various embodiments, such processes include one or more of the processes typically performed by a decoder, for example, entropy decoding, inverse quantization, inverse transform, and differential decoding. In various embodiments, such processes also or instead include processes performed by the decoders of the various implementations described in this application.

[0104] As additional examples, in one embodiment "decoding" refers only to entropy decoding, in another embodiment "decoding" refers only to differential decoding, and in another embodiment "decoding" refers to a combination of entropy decoding and differential decoding. Based on the particular context of the description, whether the phrase "decoding process" is intended to refer specifically to a subset of operations or more generally to a broader decoding process will be clear, and is believed to be well understood by those skilled in the art.

[0105] Various implementations relate to encoding. In a manner analogous to the discussion above regarding "decoding," "encoding" as used in this application can include all or a portion of the processes performed, e.g., on an input video sequence, in order to produce an encoded bitstream. In various embodiments, such processes include one or more of the processes typically performed by an encoder, e.g., partitioning, differential encoding, transform, quantization, and entropy encoding. In various embodiments, such processes also include or instead include processes performed by the various implementations of encoders described in this application.

[0106] As additional examples, in one embodiment "encoding" refers only to entropy encoding, in another embodiment "encoding" refers only to differential encoding, and in another embodiment "encoding" refers to a combination of differential encoding and entropy encoding. Based on the particular context of the description, whether the phrase "encoding process" is intended to refer specifically to a subset of operations or more generally to a broader encoding process will be clear, and is believed to be well understood by those skilled in the art.

[0107] Note that syntax elements as used in this document are descriptive terms. As such, they do not preclude the use of other syntax element names.

[0108] When a drawing is presented as a flow diagram, it should be understood that, in alternate embodiments, additional or fewer steps can be utilized. Likewise, when a drawing is presented as a block diagram, it should be understood that, in alternate embodiments, additional or fewer blocks can be utilized. Similarly, when a drawing is presented as a block diagram, it should be understood that, in alternate embodiments, a corresponding flow diagram can be utilized. Likewise, when a drawing is presented as a flow diagram, it should be understood that, in alternate embodiments, a corresponding block diagram can be utilized.

[0109] Various embodiments can involve parametric models or rate-distortion optimization. In particular, during the encoding process, often given a constraint on computational complexity, a balance or tradeoff between code rate and distortion is typically considered. This can be measured by a rate-distortion optimization (RDO) metric, or by a least mean square (LMS), mean absolute error (MAE), or other such measure. Rate-distortion optimization is often expressed as minimizing a rate-distortion function, which is a weighted sum of code rate and distortion. There are different approaches to solving the rate-distortion optimization problem. For example, an approach can be based on extensive testing of all encoding options, including a full evaluation of the encoding cost and the associated distortion of the reconstructed signal after encoding and decoding, for all considered mode or encoding parameter values. To save encoding complexity, a faster approach can also be used, in particular, based on computing an approximate distortion based on a prediction or prediction residual signal rather than the reconstructed signal. A hybrid approach of the two can also be used, such as by using approximate distortion for only some of the possible encoding options and full distortion for the other encoding options. Other approaches evaluate only a subset of the possible encoding options. More generally, many approaches employ any of a variety of techniques to perform the optimization, but the optimization is not necessarily a full evaluation of both the encoding cost and the associated distortion.

[0110] The implementations and aspects described herein can be implemented in, for example, a method or a process, an apparatus, a software program, a data stream, or a signal. Even if only discussed in the context of a single form of implementation (for example, discussed only as a method), the implementation of features discussed can be implemented to be a

[0111] References to "one embodiment" or "an embodiment" or "one implementation" or "an implementation" and the like can mean that a particular feature, structure, characteristic, etc. being referred to is included in at least one embodiment. Thus, appearances of the phrases "in one embodiment" or "in an embodiment" or "in one implementation" or "in an implementation" or the like in various places in the specification are not necessarily all referring to the same embodiment.

[0112] Furthermore, the present application can involve "determining" various pieces of information. Determining the information can include one or more of, for example, estimating the information, calculating the information, predicting the information, or retrieving the information from memory.

[0113] Also, the application can relate to "accessing" various pieces of information. Accessing information can include one or more of, for example: receiving information, retrieving information (e.g., from storage), storing information, moving information, copying information, calculating information, determining information, predicting information, or estimating information.

[0114] Also, the application can relate to "receiving" various pieces of information. As with "accessing," receiving is intended to be a broad term. Receiving information can include one or more of, for example: accessing information or retrieving information (e.g., from storage). Also, in one or more ways, "receiving" is generally involved during operations such as, for example: storing information, processing information, transmitting information, moving information, copying information, erasing information, calculating information, determining information, predicting information, or estimating information.

[0115] It should be understood that, for example, in the case of "A / B," "A and / or B," and "at least one of A and B," the use of the " / "", "and / or", and "at least one of" is intended to be a broad term of inclusion. That is, the use of the " / "", "and / or", and "at least one of" is intended to be inclusive of a first listed option, of a second listed option, of both options, or of any number of the listed options. As another example, in the case of "A, B, and / or C" and "at least one of A, B, and C," the phrase is intended to be inclusive of a first listed option, of a second listed option, of a third listed option, of only the first and second listed options, of only the first and third listed options, of only the second and third listed options, of all three, or of any number of the listed options. This can be extended, as is clear to one of ordinary skill in the art, for any number of listed options, regardless of order.

[0116] Furthermore, as used herein, the word "signal" refers to, for example, indicating something to a corresponding decoder. For example, in certain embodiments, an encoder signals a particular transform, coding mode, or flag of a plurality of transforms, coding modes, or flags. In this way, in embodiments, the same transform, parameter, or mode is used on both the encoder side and the decoder side. Thus, for example, an encoder can communicate (explicitly signal) a particular parameter to a decoder so that the decoder can use the same particular parameter. Conversely, if the decoder already has the particular parameter along with other parameters, the signaling can be used without communication (implicitly signal) to simply allow the decoder to know and select the particular parameter. In various embodiments, by avoiding any actual communication of functionality, bit savings are realized. It should be understood that signaling can be done in various ways. For example, in various embodiments, one or more syntax elements, flags, etc. are used to signal information to a corresponding decoder. Although the foregoing refers to the verb form of the word "signal," the word "signal" can also be used as a noun herein.

[0117] It will be apparent to those skilled in the art that implementations can incorporate various signals. The signals can be formatted to carry information that can be, for example, stored or transmitted. The information can include, for example, instructions for performing a method or data generated by one of the described implementations. For example, a signal can be formatted to carry a bitstream that describes an embodiment. Such a signal can be formatted, for example, as an electromagnetic wave (e.g., using the radio frequency part of spectrum) or as a baseband signal. The formatting can include, for example, encoding a data stream and modulating a carrier with the encoded data stream. The information that the signal carries can be, for example, analog or digital information. The signal can be transmitted over a variety of different wired or wireless links, as is known. The signal can be stored on a processor-readable medium.

[0118] The previous section described many embodiments across various claim categories and types. Features of the embodiments can be provided individually or in any combination. In addition, across various claim categories and types, embodiments can include one or more of the following features, means, or aspects, individually or in any combination: At least one embodiment includes encoding or decoding a video block by assigning an index representative of a combination of an intra mode and a partition mode for the video block.

[0119] At least one embodiment includes encoding the above index based on a cost.

[0120] At least one embodiment also includes setting a flag when the index is in a most probable mode list.

[0121] At least one embodiment further includes using a truncated code to indicate which mode is to be used from the most probable mode list.

[0122] At least one embodiment further includes classifying intra prediction modes based on template matching cost.

[0123] At least one embodiment further includes a syntax element being used to signal a partition mode.

[0124] At least one embodiment includes any encoding or decoding operation based on the above.

[0125] At least one embodiment includes a bitstream or signal that includes one or more of the described syntax elements or variations thereof.

[0126] At least one embodiment includes a bitstream or signal that includes syntax that conveys information produced according to any of the described embodiments.

[0127] At least one embodiment includes creating and / or transmitting and / or receiving and / or decoding according to any of the described embodiments.

[0128] At least one embodiment includes a method, process, device, medium storing instructions, medium storing data, or signal according to any of the described embodiments.

[0129] At least one embodiment includes inserting a syntax element in a signal that enables a decoder to determine decoding information in a manner that corresponds to a manner used by an encoder.

[0130] At least one embodiment includes creating and / or transmitting and / or receiving and / or decoding a bitstream or signal that includes one or more of the described syntax elements or variations thereof.

[0131] At least one embodiment includes a TV, set-top box, cell phone, tablet or other electronic device that performs a transform method(s) according to any of the described embodiments.

[0132] At least one embodiment includes a TV, set-top box, cell phone, tablet or other electronic device that performs a transform method(s) according to any of the described embodiments, determines, and displays (e.g., using a monitor, screen or other type of display) the resulting image.

[0133] At least one embodiment includes a TV, set-top box, cell phone, tablet computer, or other electronic device that selects, band-limits, or tunes (e.g., using a tuner) a channel to receive a signal that includes an encoded image, and performs the transform method(s) according to any of the described embodiments.

[0134] At least one embodiment includes a TV, set-top box, cell phone, tablet computer, or other electronic device that receives a signal that includes an encoded image over the air (e.g., using an antenna), and performs the transform method(s).

Claims

1. A method comprising: assigning an index representative of a combination of an intra mode and a partition mode for a video block; encoding the index based on a cost; setting a flag when the index is in a most probable mode list; and encoding the video block using the most probable mode list.

2. An apparatus comprising: a processor configured to perform: assigning an index representative of a combination of an intra mode and a partition mode for a video block; encoding the index based on a cost; setting a flag when the index is in a most probable mode list; and encoding the video block using the most probable mode list.

3. A method comprising: determining when a flag representative of a combination of an intra mode and a partition mode for a video block is valid; parsing an encoded index to determine the combination of the intra mode and the partition mode from a most probable mode list; and decoding the video block using the most probable mode.

4. An apparatus comprising: a processor configured to perform: determining when a flag representative of a combination of an intra mode and a partition mode for a video block is valid; parsing an encoded index to determine the combination of the intra mode and the partition mode from a most probable mode list; and decoding the video block using the most probable mode.

5. The method of claim 1 or 3 or the apparatus of claim 2 or 4, wherein a truncated code is used to indicate which mode is to be used from the most probable mode list.

6. The method of any of claims 1, 3 or 5 or the apparatus of any of claims 2, 4 or 5, wherein intra prediction modes are sorted based on template matching cost.

7. The method of any of claims 1, 3, 5 or 6 or the apparatus of any of claims 2, 4, 5 or 6, wherein a syntax element is used to signal a partition mode.

8. The method of any of claims 1, 3, 5, 6 or 7 or the apparatus of any of claims 2, 4, 5, 6 or 7, wherein the combination of the intra mode and the partition mode is sorted by cost.

9. The method of any of claims 1, 3 or 5 to 8 or the apparatus of any of claims 2, 4 or 5 to 8, wherein a flag indicates whether at least one partition of the video block uses intra coding.

10. The method or apparatus of claim 9, wherein a sorted partitioning of the video block is sorted based on a number of partitions using inter coding.

11. The method or apparatus of claim 10, wherein a code indicating a combination of a coding mode and a partition mode of the video block is used based on a lower index.

12. An apparatus comprising: the apparatus of claim 4; and at least one of (i), (ii) and (iii), wherein (i) an antenna configured to receive a signal, the signal comprising the video block, (ii) a band limiter configured to limit the received signal to a frequency band comprising the video block, and (iii) a display configured to display an output representative of a video block.

13. A non-transitory computer readable medium containing data content generated according to the method of any one of claims 1 or 5 to 11 or by the apparatus of any one of claims 2 or 5 to 11, said data content for playback using a processor.

14. A signal comprising video data generated according to the method of any one of claims 1 or 5 to 11 or by the apparatus of any one of claims 2 or 5 to 11, said video data for playback using a processor.

15. A computer program product comprising instructions which, when the program is executed by a computer, cause the computer to carry out the method of any one of claims 1, 2 or 5 to 11.

Images