Advanced syntax simplification toolset for small chunks

Abstract

A decoding toolset activation scheme allows the use of one or more decoding tools in the encoding or decoding of video data based on multiple characteristics related to the video. In one embodiment, some decoding tools are enabled for block sizes below a certain threshold. In another embodiment, a set of decoding tools is enabled based on a flag of a block having specific characteristics. In yet another embodiment, these flags are included in the bitstream containing the video. In yet another embodiment, at least one of the flags is inferred to enable some decoding tools.

Description

Technical Field

[0001] At least one embodiment of the present invention relates to a method or apparatus for video encoding or decoding. Background Technology

[0002] To achieve high compression efficiency, image and video decoding schemes typically employ prediction (including spatial and / or motion vector prediction) and transform to utilize spatial and temporal redundancy in the video content. Generally, intra-frame or inter-frame prediction is used to leverage intra-frame or inter-frame correlations, followed by transform, quantization, and entropy decoding of the difference between the original and predicted images, typically represented as prediction error or prediction residuals. To reconstruct the video, the compressed data is decoded through inverse processing corresponding to entropy decoding, quantization, transform, and prediction. Various decoding tools can be used during the decoding and decoding processes. Summary of the Invention

[0003] The shortcomings and disadvantages of the prior art can be addressed by the main aspects described in this paper, which involve block shape adaptive intra-prediction direction in encoding and decoding.

[0004] According to a first aspect, a method is provided. The method includes the steps of: enabling one or more decoding tools to decode the at least one video block based on characteristics of at least one video block in a bitstream; including information in the bitstream indicating the use of the one or more decoding tools for a block having said characteristics; and storing, transmitting, or transmitting the bitstream.

[0005] According to a second aspect, a method is provided. The method includes the steps of: parsing a bitstream to obtain information indicating the use of the one or more decoding tools for decoding video blocks; using the information to enable one or more decoding tools for at least one video block in the bitstream; and decoding the bitstream.

[0006] According to another aspect, an apparatus is provided. The apparatus includes a processor. The processor can be configured to encode blocks of video or decode bitstreams by performing any of the methods described above.

[0007] According to another key aspect of at least one embodiment, an apparatus is provided comprising means according to any of the decoding embodiments; and at least one of: (i) an antenna configured to receive a signal including the video block, (ii) a band limiter configured to limit the received signal to a band including the video block, or (iii) a display configured to display an output representing the video block.

[0008] According to another key aspect of at least one embodiment, a non-transitory computer-readable medium is provided, which contains data content formed according to any of the described encoding embodiments or variations.

[0009] According to another key aspect of at least one embodiment, a signal comprising video data generated according to any of the described encoding embodiments or variations is provided.

[0010] According to another key aspect of at least one embodiment, the bitstream is formatted to include data content generated according to any of the described encoding embodiments or variations.

[0011] According to another key aspect of at least one embodiment, a computer program product including instructions is provided that, when a computer executes the program, cause the computer to perform any of the described decoding embodiments or variations.

[0012] These and other aspects, the features and advantages of the main aspects, will become apparent from the following detailed description of exemplary embodiments, taken in conjunction with the accompanying drawings. Attached Figure Description

[0013] Figure 1 This demonstrates a standard, general-purpose video compression scheme.

[0014] Figure 2 This demonstrates a standard, universal video compression scheme.

[0015] Figure 3 An example flowchart of layered image decoding depth is shown.

[0016] Figure 4 An example of layered image decoding depth is shown.

[0017] Figure 5 An embodiment of the encoding method according to the general aspects described is shown.

[0018] Figure 6 An embodiment of a decoding method based on the general aspects described is shown.

[0019] Figure 7 An embodiment of an apparatus for encoding or decoding using intra-frame prediction mode extension is shown.

[0020] Figure 8 A typical processor arrangement in which the described embodiments can be implemented is shown. Detailed Implementation

[0021] This invention belongs to the field of video compression, and relates to video compression, video encoding, and decoding. Its purpose is to improve compression efficiency compared to existing video compression systems based on hybrid architectures.

[0022] The Universal Video Decoding (VVC) Test Model (VTM) is a standardization effort aimed at improving video compression. In VVC, any target block in intra-frame prediction can have one of 67 prediction modes. Additionally, there are multiple inter-frame prediction modes and various decoding tools available.

[0023] In this text, a set of decoding tools used in modern codecs will be mentioned:

[0024] Long-tap and short-tap motion compensation filters: The filter size used for motion compensation (MC) prediction can be achieved using filters with different numbers of taps.

[0025] Two-way prediction: Two prediction blocks can be averaged to form a single prediction block.

[0026] Decoder-side motion vector refinement (also known as DMVR): In inter-frame bidirectional prediction, motion vectors can be refined using two motion prediction samples by minimizing the difference.

[0027] Bidirectional optical flow (also known as BDOF or BIO): BDOF allows offsets to be added to bidirectional prediction samples calculated using the bidirectional optical equation.

[0028] Affine Inter-Frame Prediction Mode: The motion vector for the MC of a sub-block is a function of the sub-block's position within the block. This function is typically constructed as a spatial linear interpolation of two or three reference motion vectors.

[0029] Local Illumination Compensation (LIC): MC predictions can be corrected for local illumination variations using a simple model. This correction can be linear, and the model parameters are typically derived from neighboring reconstructed and / or reference samples.

[0030] Affine Linear Weighted Intra-Prediction (ALWP): Some samples of an intra-prediction block are derived as a linear weighted sum of adjacent (potentially filtered) reconstructed samples.

[0031] Triangles: Non-rectangular prediction blocks (e.g., triangles) allow for greater flexibility when segmenting images into blocks.

[0032] Combined Inter-Frame and Intra-Frame Prediction (also known as CIIP): The main concept is to combine inter-frame prediction blocks with intra-frame prediction blocks. Inter-frame prediction is typically performed in merge mode. The weighting can be fixed or can vary depending on the sample position within the block.

[0033] Generalized bidirectional prediction (also known as GBI, BPWA, or BWA): In the case of inter-frame bidirectional prediction, two prediction blocks are computed and combined by a weighted sum. The weights to be used are decoded by the CU using indices referencing predefined weights.

[0034] In hybrid video codecs, decoding efficiency is typically measured using BD-rate (Bjontegaard incremental bit rate) gain. However, tool complexity, measured in terms of operations and memory bandwidth, is another important parameter to consider, as it can impact device manufacturing costs and power consumption. When designing (e.g., standardizing) or deploying decoding systems, a trade-off between decoding efficiency and tool complexity for each decoding tool to be implemented in the codec must be considered.

[0035] However, for some decoding tools, the trade-off between decoding efficiency and tool complexity varies depending on the block size. For example, for small blocks (e.g., 4×4), on the one hand, using long-tap filters (e.g., 8-tap) for motion compensation or bidirectional prediction significantly increases the number of operations per sample compared to using shorter-tap filters (e.g., 6-tap) or unidirectional prediction. On the other hand, for large images (e.g., 2K or 4K), 4×4 partitions are rarely chosen because larger block sizes generally offer a better trade-off, even though an exhaustive encoder must test many and expensive (in terms of operation count) 4×4 block size configurations.

[0036] This is why some codec design architects (e.g., standardization organizations) prefer to reduce the number of tools implementing 4×4 block size (e.g., not supporting bidirectional prediction for small blocks in VVC) or simplify tools supporting 4×4 block size (e.g., in existing proposals, replacing 8-tap filters with 6-tap filters for 4×4 blocks).

[0037] The solutions proposed to date (hard choices for simplification of small-piece tools) tend to favor certain decoding or architectures at the expense of others. For example, VVC has been chosen as a solution that offers a good trade-off between complexity and gain for large image sizes (e.g., ≥2K). If these choices are driven by the requirements of most applications based on large images, they negatively impact the use of lower image resolutions and jeopardize the versatility / adaptability of the final codec specification.

[0038] For example, another proposal is:

[0039] - Disable inter-frame prediction for 4x4 CUs

[0040] - Disable bidirectional prediction for 4x8 CU and 8x4 CU.

[0041] - Motion compensation for 4×4 CU and sub-CU using a 6-tap interpolation filter

[0042] Reports indicate that these simplified BD rate losses increase while image size decreases, as shown in Table 1. Table 2 also confirms that the BD rate loss is related to the amount of 4×4 CUs in the bitstream.

[0043] Table 1: BD rate loss for several simplified combinations of 4×4, 4×8 and 8×4 tools.

[0044]

[0045] Table 2: Ratio and BD rate loss of 4×4 blocks when combining several 4×4, 4×8 and 8×4 tools for simplification of four 416×240 sequences.

[0046] 416x240 % of 4x4 CU Y U V BasketballPass 7.29％ 0.27％ 0.78％ 0.85％ BQSquare 20.0％ 1.15％ 1.50％ 1.79％ BlowingBubbles 7.90％ 0.78％ 1.17％ 1.27％ RaceHorses 6.80％ 0.38％ 0.81％ 0.92％

[0047] The main aspects described in this paper present a High-Level Syntax (HLS) and associated processes that allow for control over the level of tool simplification associated with small chunks and better adapt the overall codec design to specific application requirements and a variety of use cases.

[0048] Example 1:

[0049] A flag is defined and decoded in the High-Level Syntax (HLS) that indicates whether a set of decoding tools is enabled (or disabled) for "small" blocks. This flag can be decoded, for example, in SPS, PPS, slice, or tile headers (310).

[0050] Examples of tools enabled / disabled in "small" blocks are shown in Table 3:

[0051] 1. Use long-tap filters (e.g., 8 taps) for motion compensation. In JVET software, small-tap filters are used for 4×4 blocks.

[0052] 2. Bidirectional prediction. In the JVET software, inter-frame prediction is disabled for block sizes smaller than 4x8 and 8x4.

[0053] 3. Inter-frame prediction. In the JVET software, inter-frame prediction is disabled for 4x4 blocks.

[0054] 4. Decoder-side motion vector refinement (also known as DMVR). In JVET software, DMVR is applied if the number of block samples is at least 64 and the height is at least 8.

[0055] 5. Bidirectional Optical Flow (also known as BDOF or BIO). In the JVET software, BDOF is disabled in 4x4 and 4x8 blocks [enabled if!(pu.Y().height == 4 || (pu.Y().width == 4 && pu.Y().height == 8))]

[0056] 6. Affine Inter-Frame Prediction Mode. In the JVET software, affine is allowed if both the width and height luminance block sizes are strictly greater than 8. (warning merge >= 8)

[0057] 7. Local Illumination Compensation (LIC). In the JVET software, LIC is disabled if the number of block samples is less than 64.

[0058] 8. Affine Linear Weighted Intra Prediction (ALWP). In JVET, ALWP is enabled regardless of the luminance block size. For small blocks, ALWP can be disabled

[0059] 9. Prediction Refinement with Optical Flow (also known as PROF). For small blocks, PROF can be disabled.

[0060] 10. Triangle. In JVET, if cu.lwidth() * cu.lheight() < TRIANGLE_MIN_SIZE = 64, then triangle is deactivated

[0061] 11. Combined Inter-Frame and Intra Prediction (also known as CIIP). In the JVET software, the triangle mode is disabled if the number of samples is strictly less than 64.

[0062] 12. Generalized Bidirectional Prediction (also known as GBI or BPWA or BWA). In the current JVET software, GBI is disabled only if cu.lwidth() * cu.lheight() < 256.

[0063] Table 3: Summary of the "small" block classification for each tool in JVET

[0064]

[0065] The same flags group the tool sets. Advantageously, one or more predefined thresholds associated with the block size are used to classify blocks as "small" (330). For example, if the number of (luminance) samples (width × height) is below the threshold, the block is classified as "small". In a variant, if both the width and height are below the predefined thresholds widthMin and heightMin, or if at least the width or height is below the predefined threshold = sizeMin, the block is classified as "small".

[0066] In one variation, the threshold can be defined for each tool. This means that the threshold is different for "tool i" and "tool j". For example, for tool 1, blocks with a width and height of less than 4 can be classified as "small" blocks, while for tool 2, blocks with a sample number of less than 32 can be classified as "small" blocks.

[0067] In one variant, if the CU (decoding unit) is considered "small" for "tool i" (330), then the associated parameter (e.g., "mode_flag(i)") is not present in the stream and is inferred. For example, "mode_flag(i)" is inferred to be zero ("tool i" does not apply). If the CU is considered "non-small", then the parameter associated with "tool i" may exist in the bit stream if other conditions of "tool i" are met (340).

[0068] For example, in the case of GBI (Generalized Bidirectional Prediction), if the CU is considered "small", then the syntax element "gbi_index" does not exist in the bitstream and is inferred as the default. Otherwise, if the CU is considered "non-small" and the CU is decoded in "inter-frame" and "merging" and "bidirectional prediction" modes (or, other conditions), then "gbi_index" is decoded (340).

[0069] Example 2:

[0070] In one variant, the threshold is a function of the minimum size of the luminance decoding block minBlkSize (e.g., log2_min_luma_coding_block_size_minus3 in HEVC) and / or the maximum luminance decoding block size (e.g., the CTU size in HEVC, log2_diff_max_min_luma_coding_block_size).

[0071] For example, a “small” block in Tool 1 could be a block with fewer than minBlkSize x minBlkSize samples.

[0072] Once the minimum or maximum block size is decoded, the threshold (320) is updated.

[0073] Example 3:

[0074] In one variant, the values ​​of the one or more thresholds are directly encoded in the HLS.

[0075] Example 4:

[0076] In one variant, the tag value is not decoded, but inferred from one or more other parameters. For example, in the case of layered decoding, the tag is inferred from the image size or image depth. Figure 4 For example, if the image size (number of brightness samples) is less than a threshold of 416 × 240, the flag is inferred to be true (meaning the toolset is enabled for "small" blocks). In another example, the flag is inferred to be true if the image size (number of brightness samples) is below a threshold and the layering depth of the current slice or tile (or group of tiles) is below another threshold (e.g., th = 3).

[0077] Example 5:

[0078] Based on the profile / tier / level, for some profiles / tiers / levels, the flags used for small blocks can be forced to predefined values. This means that it is inferred, or it may exist in the bitstream, but its value is constrained by the profile and / or tier and / or level, such as as specified in the specification.

[0079] For example, for low-level values, the flag value can be set to one (for "small" blocks, enable a set of decoding tools).

[0080] In the variant, the threshold for emb-1 can be defined by profile, layer, or level.

[0081] Figure 5 An embodiment of a method 500 using the general aspects described herein is illustrated, the method beginning at a start block 501 and control proceeding to a function block 510 to enable one or more decoding tools to decode the at least one video block in the bitstream based on characteristics of at least one video block. Control then proceeds from block 510 to block 520 to include information in the bitstream indicating the use of the one or more decoding tools for blocks having said characteristics. Control then proceeds from block 520 to block 530 for storing, transmitting, or transmitting the bitstream.

[0082] Figure 6 An embodiment of method 600 using the general aspects described herein is shown, which begins at start box 601 and control proceeds to function box 610 for parsing the bitstream to obtain information indicating the use of the one or more decoding tools to decode video blocks. Control then proceeds from box 610 to box 620 for using the information to enable the use of one or more decoding tools for at least one video block in the bitstream. Control then proceeds from box 620 to box 630 to decode the bitstream.

[0083] Figure 7An embodiment of an apparatus 700 for compressing, encoding, or decoding video using decoding or decoding tools is shown. The apparatus includes a processor 710 and can be interconnected with a memory 720 via at least one port. Both the processor 710 and the memory 720 may also have one or more additional interconnects to external connections.

[0084] The processor 710 is also configured to compress, encode, or decode information, or to insert or receive information in a bitstream using various decoding tools.

[0085] This application describes various aspects, including tools, features, embodiments, models, methods, etc. Many of these aspects are described as specific and are generally described in a manner that may sound restrictive, at least to illustrate individual characteristics. However, this is for the purpose of clarity and does not limit the application or scope of those aspects. In fact, all the different aspects can be combined and interchanged to provide other aspects. Furthermore, these aspects can also be combined and interchanged with those described in earlier documents.

[0086] The aspects described and anticipated in this application can be implemented in many different forms. The following... Figure 1 , 2 Sections 8 and 9 provide some embodiments, but other embodiments are conceivable, and... Figure 1 , 2 The discussion in section 8 does not limit the breadth of implementation. At least one of the aspects relates primarily to video encoding and decoding, and at least one other aspect relates primarily to transmitting the generated or encoded bitstream. These and other aspects can be implemented as methods, apparatus, computer-readable storage media having instructions stored thereon for encoding or decoding video data according to any of the described methods, and / or computer-readable storage media having bitstreams generated according to any of the described methods stored thereon.

[0087] In this application, the terms "reconstruction" and "decoding" are used interchangeably, as are the terms "pixel" and "sample," and the terms "image," "picture," and "frame." Typically, but not necessarily, the term "reconstruction" is used on the encoder side, while "decoding" is used on the decoder side.

[0088] This document describes various methods, and each method includes one or more steps or actions for implementing the described method. Unless the correct operation of the method requires a specific order of steps or actions, the order and / or use of specific steps and / or actions may be modified or combined.

[0089] The various methods and other aspects described in this application can be used to modify the module, for example, Figure 1 and Figure 2The intra-frame prediction, entropy decoding, and / or decoding modules (160, 360, 145, 330) of the video encoder 100 and decoder 200 are shown. Furthermore, the invention is not limited to VVC or HEVC and can be applied to, for example, other standards and proposals (whether pre-existing or developed in the future) and any extensions to such standards and proposals (including VVC and HEVC). Unless otherwise indicated or technically excluded, the aspects described in this application may be used individually or in combination.

[0090] Various numerical values ​​are used in this application. Specific values ​​are for illustrative purposes, and the aspects described are not limited to these specific values.

[0091] Figure 1 Encoder 100 is shown. Variations of encoder 100 can be envisioned, but for clarity, encoder 100 is described below without describing all anticipated variations.

[0092] Before being encoded, the video sequence may undergo pre-coding processing (101), such as applying a color transformation to the input color image (e.g., a conversion from RGB 4:4:4 to YCbCr 4:2:0), or performing a remapping of the input image components to obtain a signal distribution that is more resilient to compression (e.g., histogram equalization using one of the color components). Metadata may be associated with this pre-processing and appended to the bitstream.

[0093] In encoder 100, the image is encoded by encoder elements as described below. The image to be encoded is partitioned (102) in units such as CUs. Each unit is encoded using, for example, an intra-frame or inter-frame mode. When a unit is encoded in intra-frame mode, intra-frame prediction (160) is performed. In inter-frame mode, motion estimation (175) and compensation (170) are performed. The encoder determines (105) which of the intra-frame or inter-frame modes to use to encode the unit, and indicates the intra-frame / inter-frame decision by, for example, a prediction mode flag. For example, the prediction residual is calculated by subtracting (110) the prediction block from the original image block.

[0094] Then, the predicted residual is transformed (125) and quantized (130). The quantized transform coefficients, motion vectors, and other syntax elements are entropy decoded (145) to output a bitstream. The encoder may skip the transform and directly apply quantization to the untransformed residual signal. The encoder may bypass both the transform and quantization, i.e., directly decode the residual without applying the transform or quantization.

[0095] The encoder decodes the coded blocks to provide a reference for further prediction. The quantized transform coefficients are dequantized (140) and inversely transformed (150) to decode the prediction residuals. The decoded prediction residuals and prediction blocks are combined (155) to reconstruct an image block. An in-loop filter (165) is applied to the reconstructed image to perform, for example, unblocking / SAO (sample adaptive offset) filtering, thereby reducing coding artifacts. The filtered image is stored in a reference image buffer (180).

[0096] Figure 2 A block diagram of a video decoder 200 is shown. In decoder 200, the bitstream is decoded by decoder elements as described below. Video decoder 200 typically performs operations similar to... Figure 1 The encoding process described herein is the inverse of the decoding process. The encoder 100 typically also performs video decoding as part of the encoded video data.

[0097] Specifically, the input to the decoder includes a video bitstream, which can be generated by the video encoder 100. The bitstream is first entropy decoded (230) to obtain transform coefficients, motion vectors, and other decoding information. Image partitioning information indicates how the image is partitioned. The decoder can therefore partition (235) the image based on the decoded image partitioning information. The transform coefficients are dequantized (240) and inverse transformed (250) to decode the prediction residuals. The decoded prediction residuals are combined with prediction blocks (255) to reconstruct image blocks. The prediction blocks can be obtained from intra-frame prediction (260) or motion-compensated prediction (i.e., inter-frame prediction) (275) (270). An in-loop filter (265) is applied to the reconstructed image. The filtered image is stored in a reference image buffer (280).

[0098] The decoded image can undergo further post-decoding processing (285), such as inverse color transformation (e.g., conversion from YCbCr 4:2:0 to RGB 4:4:4) or inverse remapping of the remapping process performed in the pre-encoding process (101). The post-decoding processing may use metadata derived in the pre-encoding process and signaled in the bitstream.

[0099] Figure 8A block diagram illustrating an example system in which various aspects and embodiments are implemented is shown. System 1000 can be implemented as a device including the various components described below and configured to perform one or more aspects described herein. Examples of such devices include, but are not limited to, various electronic devices such as personal computers, laptop computers, smartphones, tablet computers, digital multimedia set-top boxes, digital television receivers, personal video recording systems, connected home appliances, and servers. Elements of system 1000 can be implemented individually or in combination in a single integrated circuit (IC), multiple ICs, and / or discrete components. For example, in at least one embodiment, the processing and encoder / decoder elements of system 1000 are distributed across multiple ICs and / or discrete components. In various embodiments, system 1000 is communicatively coupled to one or more other systems or other electronic devices via, for example, a communication bus or through dedicated input and / or output ports. In various embodiments, system 1000 is configured to implement one or more aspects described herein.

[0100] The system 1000 includes at least one processor 1010 configured to execute instructions loaded therein for implementing various aspects, such as those described herein. The processor 1010 may include embedded memory, input / output interfaces, and various other circuitry known in the art. The system 1000 includes at least one memory 1020 (e.g., a volatile memory device and / or a non-volatile memory device). The system 1000 includes a storage device 1040, which may include non-volatile memory and / or volatile memory, including but not limited to electrically erasable programmable read-only memory (EEPROM), read-only memory (ROM), programmable read-only memory (PROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, disk drives, and / or optical disk drives. As a non-limiting example, the storage device 1040 may include internal storage devices, attached storage devices (including removable and non-removable storage devices), and / or network-accessible storage devices.

[0101] System 1000 includes an encoder / decoder module 1030 configured to, for example, process data to provide encoded or decoded video, and the encoder / decoder module 1030 may include its own processor and memory. The encoder / decoder module 1030 represents one or more modules that may be included in a device to perform encoding and / or decoding functions. As is known, a device may include one or both of the encoding and decoding modules. Alternatively, the encoder / decoder module 1030 may be implemented as a separate element of system 1000 or may be incorporated into processor 1010 as a combination of hardware and software as known to those skilled in the art.

[0102] Program code to be loaded onto processor 1010 or encoder / decoder 1030 to execute the various aspects described herein may be stored in storage device 1040 and subsequently loaded onto memory 1020 for execution by processor 1010. According to various embodiments, one or more of processor 1010, memory 1020, storage device 1040, and encoder / decoder module 1030 may store one or more of various items during the execution of the processes described herein. These stored items may include, but are not limited to, input video, decoded video or portions of the decoded video, bitstreams, matrices, variables, and intermediate or final results from the processing of equations, formulas, operations, and operational logic.

[0103] In some embodiments, the memory within the processor 1010 and / or the encoder / decoder module 1030 is used to store instructions and provide working memory for processing required during encoding or decoding. However, in other embodiments, external memory (e.g., the processing device may be the processor 1010 or the encoder / decoder module 1030) is used for one or more of these functions. External memory may be memory 1020 and / or storage device 1040, such as volatile memory and / or non-volatile flash memory. In several embodiments, external non-volatile flash memory is used to store, for example, the operating system of a television. In at least one embodiment, a fast external dynamic volatile memory, such as RAM, is used as working memory for video decoding and decoding operations, such as working memory for MPEG-2 (MPEG stands for Moving Picture Experts Group; MPEG-2 is also known as ISO / IEC 13818, and 13818-1 is also known as H.222, and 13818-2 is also known as H.262), HEVC (HEVC stands for High Efficiency Video Decoding; also known as H.265 and MPEG-H Part 2), or VVC (Universal Video Decoding, a new standard developed by the Joint Video Team experts JVET).

[0104] As shown in box 1130, input to the components of system 1000 can be provided through various input devices. Such input devices include, but are not limited to: (i) an RF section that receives radio frequency (RF) signals transmitted over the air, for example by a broadcaster; (ii) component (COMP) input terminals (or a set of component input terminals); (iii) universal serial bus (USB) input terminals; and / or (iv) high-definition multimedia interface (HDMI) input terminals. Figure 8 Other examples not shown include composite videos.

[0105] In various embodiments, the input device of block 1130 has associated corresponding input processing elements known in the art. For example, the RF section may be associated with elements suitable for: (i) selecting a desired frequency (also known as selecting a signal, or limiting a signal band to a band), (ii) downconverting the selected signal, (iii) again limiting the band to a narrower band to select (e.g.,) a signal band that may be referred to as a channel in some embodiments), (iv) demodulating the downconverted and band-limited signal, (v) performing error correction, and (vi) demultiplexing to select a desired data packet stream. The RF section in various embodiments includes one or more elements to perform these functions, such as frequency selectors, signal selectors, band limiters, channel selectors, filters, downconverters, demodulators, error correctors, and demultiplexers. The RF section may include tuners that perform various of these functions, including, for example, downconverting received signals to a lower frequency (e.g., intermediate frequency or near-baseband frequency) or baseband. In one set-top box embodiment, the RF section and its associated input processing elements receive RF signals transmitted via a wired (e.g., cable) medium and perform frequency selection to a desired frequency band through filtering, down-conversion, and re-filtering. Various embodiments rearrange the order of the above (and other) elements, remove some of these elements, and / or add other elements that perform similar or different functions. Adding elements may include inserting elements between existing elements, such as inserting amplifiers and analog-to-digital converters. In various embodiments, the RF section includes an antenna.

[0106] Additionally, the USB and / or HDMI terminals may include corresponding interface processors for connecting the system 1000 to other electronic devices via USB and / or HDMI connections. It should be understood that various aspects of input processing (e.g., Solomon error correction) can be implemented as needed, for example, within a separate input processing IC or processor 1010. Similarly, various aspects of USB or HDMI interface processing can be implemented as needed within a separate interface IC or within processor 1010. The demodulated, error-corrected, and demultiplexed streams are provided to various processing elements, including, for example, processor 1010 and encoder / decoder 1030, which operate in conjunction with memory and storage elements to process the data streams as needed for presentation on the output device.

[0107] Various components of the system 1000 can be housed within an integrated housing. Within this integrated housing, the various components can be interconnected and transmit data therebetween using suitable connection arrangements (e.g., internal buses known in the art, including inter-IC (I2C) buses, wiring, and printed circuit boards).

[0108] The system 1000 includes a communication interface 1050 that enables communication with other devices via a communication channel 1060. The communication interface 1050 may include, but is not limited to, a transceiver configured to send and receive data via the communication channel 1060. The communication interface 1050 may include, but is not limited to, a modem or network interface card (NIC), and the communication channel 1060 may be implemented, for example, in a wired and / or wireless medium.

[0109] In various embodiments, a wireless network (e.g., a Wi-Fi network, such as IEEE 802.11 (IEEE refers to the Institute of Electrical and Electronics Engineers)) is used to stream or otherwise provide data to the system 1000. In these embodiments, the Wi-Fi signal is received via a communication channel 1060 and a communication interface 1050 suitable for Wi-Fi communication. The communication channel 1060 in these embodiments is typically connected to an access point or router that provides access to external networks, including the Internet, to allow streaming applications and other cloud-based communications. Other embodiments use a set-top box that transmits data via an HDMI connection to input box 1130 to provide streaming data to the system 1000. Still other embodiments use an RF connection to input box 1130 to provide streaming data to the system 1000. As described above, various embodiments provide data in a non-streaming manner. Additionally, various embodiments use wireless networks other than Wi-Fi, such as cellular networks or Bluetooth networks.

[0110] The system 1000 can provide output signals to various output devices, including a display 1100, a speaker 1110, and other peripheral devices 1120. The display 1100 in various embodiments includes one or more of the following: for example, a touchscreen display, an organic light-emitting diode (OLED) display, a flexible display, and / or a foldable display. The display 1100 can be used in televisions, tablets, laptops, cellular phones (mobile phones), or other devices. The display 1100 can also be integrated with other components (e.g., in a smartphone) or stand alone (e.g., an external monitor for a laptop computer). In various examples of embodiments, the other peripheral devices 1120 include one or more of the following: a standalone digital video disc (or digital multifunction disc) (DVR, for both), a disc player, a stereo system, and / or a lighting system. Various embodiments use one or more peripheral devices 1120 that provide functionality based on the output of the system 1000. For example, a disc player performs the function of playing the output of the system 1000.

[0111] In various embodiments, signaling (such as AV.Link, Consumer Electronics Control (CEC), or other communication protocols that enable device-to-device control with or without user intervention) is used to transmit control signals between system 1000 and display 1100, speaker 1110, or other peripheral devices 1120. The output devices may be communicatively coupled to system 1000 via dedicated connections through corresponding interfaces 1070, 1080, and 1090. Alternatively, the output devices may be connected to system 1000 via communication interface 1050 using communication channel 1060. The display 1100 and speaker 1110 may be integrated into a single unit within an electronic device (e.g., a television set) along with other components of system 1000. In various embodiments, display interface 1070 includes a display driver, such as a timing controller ((TCon) chip).

[0112] For example, if the RF portion of input 1130 is part of a separate set-top box, the display 1100 and speaker 1110 may alternatively be separated from one or more other components. In various embodiments where the display 1100 and speaker 1110 are external components, the output signal may be provided via a dedicated output connection, such as an HDMI port, a USB port, or a COMP output.

[0113] These embodiments can be implemented by processor 1010 or by computer software implemented in hardware or by a combination of hardware and software. As a non-limiting example, the embodiments can be implemented by one or more integrated circuits. The memory 1020 can be of any type suitable for the technical environment and can be implemented using any suitable data storage technology, such as optical memory devices, magnetic memory devices, semiconductor-based memory devices, fixed memory, and removable memory, as non-limiting examples. The processor 1010 can be of any type suitable for the technical environment and, as a non-limiting example, can include one or more of the following: microprocessors, general-purpose computers, special-purpose computers, and processors based on multi-core architectures.

[0114] Various implementations involve decoding. As used herein, "decoding" can include, for example, all or part of a process performed on a received encoded sequence to produce a final output suitable for display. In various embodiments, such processes include one or more of the processes typically performed by a decoder, such as entropy decoding, inverse quantization, inverse transform, and differential decoding. In various embodiments, such processes also or alternatively include processes performed by decoders of the various implementations described herein.

[0115] As a further example, in one embodiment, "decoding" refers only to entropy decoding; in another embodiment, "decoding" refers only to differential decoding; and in yet another embodiment, "decoding" refers to a combination of entropy decoding and differential decoding. Whether the phrase "decoding process" is intended to specifically refer to a subset of operations or generally to a broader decoding process will be clear based on the specific context of the description and is believed to be fully understood by those skilled in the art.

[0116] Various implementations involve encoding. In a manner similar to the above discussion of “decoding,” “encoding,” as used herein, can include all or part of a process performed, for example, on an input video sequence to produce an encoded bitstream. In various embodiments, such processes include one or more processes typically performed by an encoder, such as partitioning, differential coding, transform, quantization, and entropy decoding. In various embodiments, such processes also, or alternatively, include processes performed by encoders of the various implementations described herein.

[0117] As a further example, in one embodiment, "encoding" refers only to entropy decoding; in another embodiment, "encoding" refers only to differential encoding; and in yet another embodiment, "encoding" refers to a combination of differential encoding and entropy decoding. Whether the phrase "encoding process" is intended to specifically refer to a subset of operations or generally to a broader encoding process will become clear based on the specific context of the description and is believed to be fully understood by those skilled in the art.

[0118] Note that the grammatical elements used in this article are descriptive terms. Therefore, the use of other grammatical element names is not excluded.

[0119] When the accompanying drawings are presented as flowcharts, it should be understood that they also provide block diagrams of the corresponding apparatus. Similarly, when the accompanying drawings are presented as block diagrams, it should be understood that they also provide flowcharts of the corresponding methods / processes.

[0120] Various implementations may involve parametric models or rate distortion optimization. Specifically, during the encoding process, a balance or trade-off between rate and distortion is typically considered, usually with constraints on computational complexity. This can be measured by rate distortion optimization (RDO), or by least mean square (LMS), mean absolute error (MAE), or other such measures. RDO is typically formulated as minimizing a rate distortion function that is a weighted sum of rate and distortion. Different approaches exist to address the rate distortion optimization problem. For example, these approaches can be based on extensive testing of all encoding options, including all considered mode or decoding parameter values, and a complete evaluation of their decoding costs and the associated distortion of the reconstructed signal after decoding and decoding. Faster methods can also be used to save encoding complexity, particularly by calculating approximate distortion based on predicting or predicting the residual signal rather than the reconstructed signal. A hybrid of these two approaches can also be used, for example, by using approximate distortion only for some possible encoding options and full distortion for others. Other methods evaluate only a subset of possible encoding options. More generally, many methods employ any of a variety of techniques to perform the optimization, but the optimization is not necessarily a complete evaluation of both decoding cost and associated distortion.

[0121] The implementations and aspects described herein can be implemented, for example, in a method or process, apparatus, software program, data stream, or signal. Even if discussed only in the context of a single form of implementation (e.g., discussed only as a method), implementations of the discussed features can also be implemented in other forms (e.g., apparatus or program). For example, an apparatus can be implemented with appropriate hardware, software, and firmware. The method can be implemented, for example, in a processor, which generally refers to a processing device, including, for example, a computer, microprocessor, integrated circuit, or programmable logic device. Processors also include communication devices, such as computers, cellular phones, portable / personal digital assistants (“PDAs”), and other devices that facilitate information communication between end users.

[0122] The reference to "an embodiment," "an embodiment," "an implementation," or "an implementation," and other variations, means that a particular feature, structure, characteristic, etc., described in connection with that embodiment is included in at least one embodiment. Therefore, the phrases "in an embodiment," "in one embodiment," "in one implementation," or "in one implementation," and any other variations appearing in various places throughout this application, do not necessarily refer to the same embodiment.

[0123] Additionally, this application may involve "determining" various types of information. Determining the information may include, for example, one or more of the following: estimating the information, calculating the information, predicting the information, or retrieving the information from memory.

[0124] Furthermore, this application may relate to "accessing" various types of information. Accessing such information may include, for example, one or more of the following: receiving the information, retrieving the information (e.g., retrieving the information from a memory), storing the information, moving the information, copying the information, calculating the information, determining the information, predicting the information, or estimating the information.

[0125] Additionally, this application can refer to "receiving" various types of information. Like "accessing," receiving is intended to be a broad term. Receiving said information may include, for example, one or more of the following: accessing the information or (e.g., retrieving the information from memory). Furthermore, "receiving" is generally referred to in one or more ways during operations such as storing information, processing information, sending information, moving information, copying information, erasing information, calculating information, determining information, predicting information, or estimating information.

[0126] It should be understood that, for example, in the cases of “A / B,” “A and / or B,” and “at least one of A and B,” the use of any of the following “ / ,” “and / or,” and “at least one of” is intended to cover the selection of only the first listed option (A), or only the second listed option (B), or the selection of both options (A and B). As a further example, in the cases of “A, B, and / or C” and “at least one of A, B, and C,” such wording is intended to include selecting only the first listed option (A), or only the second listed option (B), or only the third listed option (C), or only the first and second listed options (A and B), or only the first and third listed options (A and C), or only the second and third listed options (B and C), or selecting all three options (A, B, and C). As will be apparent to those skilled in the art and related fields, this can be extended to multiple listed items.

[0127] Furthermore, as used herein, the term "signal" specifically refers to indicating something to the corresponding decoder. For example, in some embodiments, the encoder signals a particular one of multiple transforms, decoding modes, or flags. Thus, in one embodiment, the same transform, parameter, or mode is used on both the encoder and decoder sides. Therefore, for example, the encoder can send (explicitly signal) a specific parameter to the decoder so that the decoder can use that same specific parameter. Conversely, if the decoder already has the specific parameter as well as other parameters, signaling can be used without sending (implicitly signal) to simply allow the decoder to know and select the specific parameter. Bit savings are achieved in various embodiments by avoiding the transmission of any actual functionality. It should be understood that signaling can be implemented in various ways. For example, in various embodiments, one or more syntax elements, flags, etc., are used to signal information to the corresponding decoder. While the foregoing refers to the verb form of the term "signal," the term "signal" can also be used as a noun herein.

[0128] As will be apparent to those skilled in the art, implementations can generate various signals formatted to carry, for example, information that can be stored or transmitted. This information may include, for example, instructions for performing a method, or data generated by one of the described implementations. For example, the signal may be formatted to carry a bitstream of the described embodiments. Such a signal may be formatted as, for example, electromagnetic waves (e.g., using the radio frequency portion of the spectrum) or baseband signals. The formatting may include, for example, encoding the data stream and modulating a carrier wave using the encoded data stream. The information carried by the signal may be, for example, analog or digital information. As is known, the signal can be transmitted via various wired or wireless links. The signal may be stored on a processor-readable medium.

[0129] We have described several embodiments that can be provided across various claim classes and types. Features of these embodiments may be provided individually or in any combination. Furthermore, across various claim classes and types, embodiments may include one or more of the following features, devices, or aspects, individually or in any combination:

[0130] • A process or apparatus for transmitting information relating to encoding and decoding using one or more decoding tools.

[0131] • A process or apparatus for transmitting and performing information relating to encoding and decoding, wherein information inserted into a bit stream represents information for implementing one or more decoding tools.

[0132] • A process or apparatus for transmitting and performing information relating to encoding and decoding, wherein information inserted into a bit stream represents information for implementing one or more decoding tools.

[0133] • A process or apparatus for transmitting and performing information relating to encoding and decoding, wherein information inserted into a bit stream represents information for implementing one or more decoding tools.

[0134] • Includes a bitstream or signal of one or more of the described syntax elements or their variations.

[0135] • A bitstream or signal including a syntax that conveys information generated according to any of the described embodiments.

[0136] • Create and / or send and / or receive and / or decode according to any of the embodiments described.

[0137] • Methods, processes, apparatus, media for storing instructions, media for storing data, or signals according to any of the described embodiments.

[0138] • Inserting syntax elements into the signaling enables the decoder to determine the decoding mode in a manner corresponding to that used by the encoder.

[0139] • Creating and / or sending and / or receiving and / or decoding bitstreams or signals that include one or more described syntax elements or variations thereof.

[0140] TVs, set-top boxes, cellular phones, tablets, or other electronic devices that perform one or more transformation methods according to any of the described embodiments.

[0141] A TV, set-top box, cellular phone, tablet computer, or other electronic device that performs one or more transformation methods according to any of the described embodiments to determine and display (e.g., using a monitor, screen, or other type of display) the resulting image.

[0142] TVs, set-top boxes, cellular phones, tablets, or other electronic devices that select, band-limit, or tune (e.g., using a tuner) a channel to receive signals including encoded images, and perform transformation methods (one or more) according to any of the described embodiments.

[0143] • TVs, set-top boxes, cellular phones, tablets, or other electronic devices that receive signals including encoded images over the air (e.g., using an antenna) and perform transformation methods (one or more).

Claims

1. A method comprising: Based on 1) the characteristics of at least one video block in the bitstream and 2) an indicator flag included in the High-Level Syntax (HLS) indicating whether a decoding toolset is enabled for video blocks with a sample number below a predefined threshold, the decoding toolset is enabled to decode the at least one video block, wherein the decoding toolset includes one or more of the following: motion compensation filter, inter-frame prediction tool, intra-frame prediction tool, local illumination compensation tool, or bidirectional prediction tool. The bitstream includes information indicating the use of the decoding toolset for blocks having the aforementioned characteristics; and Store, transmit, or transfer the bit stream.

2. The method according to claim 1, wherein, The characteristic includes one or more thresholds for the video block size of the at least one video block.

3. The method of claim 1, wherein the information is included in the bitstream as a syntax.

4. The method of claim 1, wherein the information includes at least one flag indicating that at least one decoding tool within the decoding toolset is enabled or disabled.

5. The method of claim 4, wherein the at least one flag controls the enabling or disabling of one or more decoding tools in the decoding toolset.

6. The method according to claim 1, wherein, The corresponding threshold set is used for the corresponding decoding tools in the decoding toolset.

7. The method according to claim 1, wherein, At least one threshold is a function of the maximum or minimum size of the at least one video block included in the bitstream.

8. A method comprising: Parse the bitstream to obtain information indicating how the video blocks were decoded using a decoding toolset; The information is used to enable the decoding toolset for at least one video block in the bitstream, wherein the information indicates, via an indicator flag included in the High-Level Syntax (HLS), whether the decoding toolset is enabled for video blocks having a sample number below a predefined threshold, wherein the decoding toolset includes one or more of the following: motion compensation filters, inter-frame prediction tools, intra-frame prediction tools, local illumination compensation tools, or bidirectional prediction tools; and Decode the bit stream.

9. The method of claim 8, wherein the information is included in the bitstream as a syntax.

10. The method of claim 8, wherein the information includes at least one flag indicating that at least one decoding tool within the decoding toolset is enabled or disabled.

11. The method of claim 10, wherein the at least one flag controls the enabling or disabling of one or more decoding tools in the decoding toolset.

12. The method according to claim 8, wherein, The corresponding threshold set is used for the corresponding decoding tools in the decoding toolset.

13. The method according to claim 8, wherein, At least one threshold is a function of the maximum or minimum size of the at least one video block included in the bitstream.

14. An apparatus comprising: The processor, which is configured to execute: Parse the bitstream to obtain information indicating how the video blocks were decoded using a decoding toolset; The information is used to enable the decoding toolset for at least one video block in the bitstream, wherein the information indicates, via an indicator flag included in the High-Level Syntax (HLS), whether the decoding toolset is enabled for video blocks having a sample number below a predefined threshold, wherein the decoding toolset includes one or more of the following: motion compensation filters, inter-frame prediction tools, intra-frame prediction tools, local illumination compensation tools, or bidirectional prediction tools; and Decode the bit stream.

15. The apparatus of claim 14, wherein the information is included in the bitstream as a syntax.

16. The apparatus of claim 14, wherein the information includes at least one flag indicating that at least one decoding tool within the decoding toolset is enabled or disabled.

17. The apparatus of claim 16, wherein the at least one flag controls the enabling or disabling of one or more decoding tools in the decoding toolset.

18. The apparatus according to claim 14, wherein, The corresponding threshold set is used for the corresponding decoding tools in the decoding toolset.

19. The apparatus according to claim 14, wherein, At least one threshold is a function of the maximum or minimum size of the at least one video block included in the bitstream.

20. An apparatus comprising: The apparatus according to claim 14; as well as At least one of the following: (1) an antenna configured to receive a signal including the video block, (ii) a band limiter configured to limit the received signal to a band including the video block, and (iii) a display configured to display an output representing the video block.

Images