Method, apparatus and system for encoding and decoding trees or blocks of video samples

Abstract

To provide a system and method of decoding a transform block for a color channel of an image frame from a video bitstream.SOLUTION: The method comprises: determining a chroma format of the image frame, the chroma format having chroma channels of the image frame being subsampled relative to a luma channel of the image frame; determining a coefficient group size of the transform block, the coefficient group size being a largest area of the transform block of up to 16 samples, the coefficient group size being determined based only on the transform block size independently of both a color plane of the transform block and color plane subsampling due to the determined chroma format; and decoding the transform block using coefficient groups of the determined size from the video bitstream.SELECTED DRAWING: None

Description

[Technical Field]

[0001] REFERENCE TO RELATED APPLICATIONS This application claims the benefit under 35 U.S.C. § 119 of the filing date of Australian Patent Application No. 2019201653, filed March 11, 2019, the entire contents of which are incorporated herein by reference as if fully set forth herein.

[0002] The present invention relates generally to digital video signal processing, and more particularly to a method, apparatus and system for encoding and decoding trees or blocks of video samples. The present invention also relates to a computer program product including a computer readable medium having recorded thereon a computer program for encoding and decoding trees or blocks of video samples. [Background technology]

[0003] Many applications for video coding currently exist, including applications for the transmission and storage of video data. Many video coding standards have been developed, and others are currently under development. Recent developments in video coding standardization have led to the formation of a group called the "Joint Video Experts Team" (JVET). The Joint Video Experts Team (JVET) includes members of Study Group 16, Question 6 (SG16 / Q6) of the International Telecommunication Union's (ITU) Telecommunication Standardization Sector (ITU-T), also known as the "Video Coding Experts Group" (VCEG), and members of the International Organization for Standardization / International Electrotechnical Commission Joint Technical Committee 1 / Subcommittee 29 / Working Group 11 (ISO / IEC JTC1 / SC29 / WG11), also known as the "Moving Picture Experts group" (MPEG).

[0004] The Joint Video Experts Team (JVET) analyzed the responses at its 10th meeting in San Diego, USA, and issued a Call for Proposals (CfP). The submitted responses demonstrated video compression capabilities that significantly exceeded those of the current state-of-the-art video compression standard, i.e., "High Efficiency Video Coding" (HEVC). Based on this outperformance, it was decided to initiate a project to develop a new video compression standard, to be named "versatile video coding" (VVC). VVC is expected to address the continuing demand for ever-higher compression performance, particularly as video formats increase in capacity (e.g., at higher resolutions and higher frame rates) and address increasing market demand for service delivery over WANs, where bandwidth costs are relatively high. At the same time, VVC must be implementable on modern silicon processes and provide an acceptable tradeoff between achieved performance and implementation cost (e.g., in terms of silicon area, CPU processor load, memory usage, and bandwidth).

[0005] Video data includes a sequence of frames of image data, each containing one or more color channels. Generally, one primary color channel and two secondary color channels are required. The primary color channel is commonly called the "luma" channel, and the secondary color channels are commonly called the "chroma" channels. Video data is typically displayed in the RGB (red-green-blue) color space, which has a high degree of correlation between each of the three components. The video data representation viewed by an encoder or decoder often uses a color space such as YCbCr. YCbCr concentrates luminance, mapped to "luma" according to a transfer function, in the Y (primary) channel and chroma in the Cb and Cr (secondary) channels. Furthermore, the Cb and Cr channels may be spatially sampled (subsampled) at a lower rate compared to the luma channel, e.g., half horizontally and half vertically, known as the "4:2:0 chroma format." The 4:2:0 chroma format is commonly used for Internet video streaming, broadcast television, and Blu-ray. TMIt is commonly used in "consumer" applications, such as storage to disk. Subsampling the Cb and Cr channels at half the rate horizontally and not subsampling vertically is known as a "4:2:2 chroma format." The 4:2:2 chroma format is typically used in professional applications, including capturing footage for filmmaking and other applications. The higher sampling rate of the 4:2:2 chroma format makes the resulting video more resilient to editing operations such as color grading. Before distribution to consumers, 4:2:2 chroma format material is often converted to a 4:2:0 chroma format and then encoded for distribution to consumers. In addition to the chroma format, video is also characterized by its resolution and frame rate. Example resolutions are ultra-high definition (UHD) with a resolution of 3840x2160, or "8K" with a resolution of 7680x4320, with example frame rates of 60 or 120 Hz. The luma sample rate may range from approximately 500 megasamples / second to several gigasamples / second. For a 4:2:0 chroma format, the sample rate of each chroma channel is one-quarter of the luma sample rate, and for a 4:2:2 chroma format, the sample rate of each chroma channel is one-half of the luma sample rate.

[0006] The VVC standard is a "block-based" codec, where a frame is first divided into a square array of regions known as "coding tree units" (CTUs). CTUs typically occupy a relatively large area, such as 128 x 128 luma samples. However, the CTUs at the right and bottom of each frame may be smaller in area. Each CTU is associated with a "coding tree" for the luma channel and additional coding trees for the chroma channels. The coding trees define a decomposition of the CTU's region into a series of blocks, also called "coding blocks" (CBs). It is also possible for a single coding tree to specify blocks for both the luma and chroma channels; in that case, the collocated set of coding blocks is called a "coding unit" (CU); i.e., each CU has a coding block for each color channel. The CBs are processed for encoding or decoding in a specific order. As a result of the use of the 4:2:0 chroma format, a CTU having a luma coding tree for a 128x128 luma sample region has a corresponding chroma coding tree for a 64x64 chroma sample region co-located with the 128x128 luma sample region. When a single coding tree is used for the luma and chroma channels, the collection of co-located blocks for a given area is generally referred to as a "unit," e.g., the above-mentioned CU, as well as a "prediction unit" (PU) and a "transform unit" (TU). When separate coding trees are used for a given area, the above-mentioned CB, as well as a "prediction block" (PB) and a "transform block" (TB) are used.

[0007] Despite the above distinction between "unit" and "block", the term "block" may be used as a general term for an area or region of a frame where an operation is applied to all color channels.

[0008] For each CU, a prediction unit (PU) is generated for the contents (sample values) of the corresponding region of the frame data ("prediction unit"). Furthermore, a representation of the difference (or "residual" in the spatial domain) between the prediction and the contents of the region seen at the input to the encoder is formed. The differences for each color channel are transformed and coded as a sequence of residual coefficients, which can form one or more TUs for a given CU. The applied transform can be a discrete cosine transform (DCT) or other transform applied to each block of residual values. This transform is applied separably; i.e., a two-dimensional transform is performed in two passes. A block is first transformed by applying a one-dimensional transform to each row of samples in the block. Then, the partial results are transformed by applying a one-dimensional transform to each column of the partial results, producing a final block of transform coefficients that substantially decorrelates the residual samples. Transforms of various sizes are supported by the VVC standard, including transforms for rectangular-shaped blocks, where each side dimension is a power of two. The transform coefficients are quantized for entropy coding into the bitstream.

[0009] When spatial prediction ("intra prediction") is used to generate a PB, a set of reference samples is used to generate predicted samples for the current PB. The reference samples include samples adjacent to the PB that have already been "reconstructed" (adding residual samples to intra-predicted samples). These adjacent samples form rows above the PB and columns to the left of the PB. The rows and columns also extend beyond the PB boundary and include additional neighboring samples. Due to the traversal of the block in a Z-order scan, some of the reference samples have been reconstructed in the previous block. The use of samples from the previous block introduces a feedback dependency that limits the throughput of the block through a video encoder or decoder. Furthermore, when relatively small blocks are predicted from other frames ("inter prediction"), the memory bandwidth required to fetch the reference samples can become excessive, especially considering the additional samples required to accommodate subpixel interpolation filtering. Summary of the Invention

[0010] It is an object of the present invention to substantially overcome, or at least ameliorate, one or more disadvantages of existing arrangements.

[0011] One aspect of the present disclosure provides a method for decoding a transform block of a color channel of an image frame from a video bitstream, the method comprising: determining a chroma format of the image frame, the chroma format having chroma channels of the image frame subsampled relative to a luma channel of the image frame; determining a coefficient group size of the transform block, the coefficient group size being a maximum range of the transform block up to 16 samples, the coefficient group size being determined based only on the transform block size and being independent of both (i) the color planes of the transform block and (ii) the color plane subsampling due to the determined chroma format; and decoding the transform block from the video bitstream using coefficient groups of the determined size.

[0012] According to another aspect, a single table is used for transform blocks belonging to the luma and chroma color planes of an image frame of the bitstream.

[0013] According to another aspect, the coefficient group size is selected to have an aspect ratio closest to 1:1 within the constraints of the transform block width and height.

[0014] Another aspect of the present disclosure provides a non-transitory computer-readable medium having stored thereon a computer program implementing a method for decoding transform blocks of color channels of an image frame from a video bitstream, the program comprising: code for determining a chroma format of the image frame, the chroma format having chroma channels of the image frame subsampled relative to a luma channel of the image frame; code for determining a coefficient group size of the transform block; the coefficient group size being a maximum range of the transform block up to 16 samples, the coefficient group size being determined based only on the transform block size and being independent of both (i) the color planes of the transform block and (ii) the color plane subsampling due to the determined chroma format; and code for decoding the transform block from the video bitstream using coefficient groups of the determined size.

[0015] Another aspect of the present disclosure provides a video decoder configured to receive transform blocks of color channels of an image frame from a video bitstream; determine a chroma format for the image frame, the chroma format having the chroma channels of the image frame subsampled relative to the luma channel of the image frame; determine a coefficient group size for the transform block, the coefficient group size being a maximum range of the transform block up to 16 samples, the coefficient group size being determined based only on the transform block size and being independent of both (i) the color planes of the transform block and (ii) color plane subsampling due to the determined chroma format; and decode the transform blocks from the video bitstream using coefficient groups of the determined size.

[0016] Another aspect of the present disclosure provides a system comprising a memory and a processor, wherein the processor is configured to execute code stored in the memory for implementing a method for decoding transform blocks of color channels of an image frame from a video bitstream, the method comprising: determining a chroma format of the image frame, the chroma format having chroma channels of the image frame subsampled relative to a luma channel of the image frame; determining a coefficient group size of the transform block, the coefficient group size being a maximum range of the transform block up to 16 samples, the coefficient group size being determined based only on the transform block size and being independent of both (i) the color planes of the transform block and (ii) the color plane subsampling due to the determined chroma format; and decoding the transform block from the video bitstream using coefficient groups of the determined size.

[0017] Other aspects are also disclosed. [Brief explanation of the drawings]

[0018] At least one exemplary embodiment of the present invention will now be described with reference to the following drawings and appendices. [Figure 1] FIG. 1 is a schematic block diagram illustrating a video encoding and decoding system. [Figure 2A] FIG. 2A forms a schematic block diagram of a general-purpose computer system capable of implementing either or both of the video encoding and decoding systems of FIG. [Figure 2B] FIG. 2B forms a schematic block diagram of a general-purpose computer system capable of implementing either or both of the video encoding and decoding systems of FIG. [Figure 3] FIG. 3 is a schematic block diagram illustrating the functional modules of a video encoder. [Figure 4] FIG. 4 is a schematic block diagram showing the functional modules of a video decoder. [Figure 5]FIG. 5 is a schematic block diagram illustrating possible divisions of a block into one or more blocks in a generic video coding tree structure. [Figure 6] FIG. 6 is a schematic diagram of a data flow for achieving permitted division of a block into one or more blocks in a tree structure of generic video coding. [Figure 7A] FIG. 7A shows an example of dividing a coding tree unit (CTU) into several coding units (CUs). [Figure 7B] FIG. 7B shows an example of dividing a coding tree unit (CTU) into several coding units (CUs). [Figure 8A] FIG. 8A shows an example of dividing a coding tree unit (CTU) into several coding blocks (CB) in the luma channel and the chroma channel. [Figure 8B] FIG. 8B shows an example of dividing a coding tree unit (CTU) into several coding blocks (CB) in the luma channel and the chroma channel. [Figure 8C] FIG. 8C shows an example of dividing a coding tree unit (CTU) into several coding blocks (CB) in the luma channel and the chroma channel. [Figure 9] FIG. 9 shows a set of transform block sizes and associated scan patterns. [Figure 10] FIG. 10 shows a set of rules for generating a list of allowed splits in the luma and chroma coding trees. [Figure 11] FIG. 11 illustrates a method for encoding a coding tree of an image frame into a video bitstream. [Figure 12] FIG. 12 illustrates a method for decoding a coding tree of an image frame from a video bitstream. [Figure 13] FIG. 13 shows how a coding tree for an image frame is encoded into a video bitstream. [Figure 14] FIG. 14 illustrates a method for decoding a coding tree of an image frame from a video bitstream. [Figure 15] FIG. 15 shows a set of transform block partitions of an intra-prediction coding unit. [Figure 16] FIG. 16 illustrates a method for encoding coding units of an image frame into a video bitstream. [Figure 17] FIG. 17 illustrates a method for decoding a coding unit of an image frame from a video bitstream. DETAILED DESCRIPTION OF THE INVENTION

[0019] When referring to steps and / or features having the same reference number in any one or more of the accompanying drawings, those steps and / or features have the same function or operation for the purposes of this specification, unless a contrary intention appears.

[0020] As mentioned above, the use of samples from a previous block introduces feedback dependencies that can limit the throughput of a block in a video encoder or decoder. To ensure that high rates of processing blocks can be sustained, as required for typical real-time encoding and decoding applications, methods are desirable to mitigate the severity of the resulting feedback dependency loops. Feedback dependency loops are particularly problematic for the high sample rates of modern video formats, e.g., from 500-4000 samples per second, while ASIC (application-specific integrated circuit) clock frequencies are typically several hundred MHz.

[0021] FIG. 1 is a schematic block diagram illustrating functional modules of a video encoding and decoding system 100. System 100 can utilize different rules for allowable subdivision of regions in luma and chroma coding trees to reduce the worst-case block processing rate encountered. For example, system 100 can operate such that blocks are always sized as multiples of sixteen (16) samples, regardless of the block's aspect ratio. Furthermore, if the coding tree includes a partition indicating the presence of a small luma coding block, partitioning may be prohibited in the chroma channels, resulting in a single chroma CB co-located with multiple luma CBs. A chroma CB can use a single prediction mode, such as one intra prediction mode, independent of the prediction mode of each co-located luma CB (including when one or more luma CBs use inter prediction). Residual coefficient coding can also utilize multiples of 16 block sizes, including for blocks with a width or height of two samples.

[0022] System 100 includes source device 110 and destination device 130. Communication channel 120 is used to communicate encoded video information from source device 110 to destination device 130. In some configurations, source device 110 and destination device 130 may comprise either or both of a respective mobile phone handset or “smartphone,” in which case communication channel 120 is a wireless channel. In other configurations, source device 110 and destination device 130 may comprise video conferencing equipment, in which case communication channel 120 is typically a wired channel, such as an Internet connection. Furthermore, source device 110 and destination device 130 may comprise any of a wide range of devices, including devices that support over-the-air television broadcasts, cable television applications, Internet video applications (including streaming), and applications in which encoded video data is captured on some computer-readable storage medium, such as a hard disk drive in a file server.

[0023] As shown in FIG. 1 , source device 110 includes video source 112, video encoder 114, and transmitter 116. Video source 112 typically comprises a source of captured video frame data (shown as 113), such as an imaging sensor, a previously captured video sequence stored on a non-transitory recording medium, or video from a remote imaging sensor. Video source 112 may also be the output of a computer graphics card, e.g., displaying the video output of an operating system and various applications running on a computing device such as a tablet computer. Examples of source device 110 that may include an imaging sensor as video source 112 include smartphones, video cameras, professional video cameras, and network video cameras.

[0024] Video encoder 114 converts (or "encodes") imaged frame data (indicated by arrow 113) from video source 112 into a bitstream (indicated by arrow 115), as further described with reference to Figure 3. Bitstream 115 is transmitted by transmitter 116 over communication channel 120 as coded video data (or "coded video information"). Bitstream 115 may also be stored in non-transitory storage device 122, such as "flash" memory or a hard disk drive, until later transmitted over communication channel 120, or in lieu of transmission over communication channel 120.

[0025] Destination device 130 includes receiver 132, video decoder 134, and display device 136. Receiver 132 receives encoded video data from communication channel 120 and passes the received video data as a bitstream to video decoder 134 (indicated by arrow 133). Video decoder 134 then outputs decoded frame data (indicated by arrow 135) to display device 136. Decoded frame data 135 has the same chroma format as frame data 113. Examples of display device 136 include a cathode ray tube, a liquid crystal display such as a smartphone, a tablet computer, a computer monitor, or a standalone television set. It is also possible for the functionality of each of source device 110 and destination device 130 to be implemented in a single device, examples of which include a mobile phone handset and a tablet computer.

[0026] Notwithstanding the exemplary devices described above, each of source device 110 and destination device 130 may generally be configured within a general-purpose computing system via a combination of hardware and software components. Figure 2A illustrates such a computer system 200, including a computer module 201, input devices such as a keyboard 202, a mouse pointer device 203, a scanner 226, a camera 227 that may be configured as a video source 112, and a microphone 280, and output devices including a printer 215, a display device 214 that may be configured as a display device 136, and speakers 217. An external modulator-demodulator (modem) transceiver device 216 may be used by computer module 201 to communicate with a communications network 220 via connection 221. Communications network 220, which may represent communications channel 120, may be a wide area network (WAN) such as the Internet, a cellular telecommunications network, or a private WAN. If connection 221 is a telephone line, modem 216 may be a conventional "dial-up" modem. Alternatively, modem 216 may be a broadband modem if connection 221 is a high-capacity (e.g., cable or optical) connection. A wireless modem may also be used for wireless connection to communication network 220. Transceiver device 216 may provide the functionality of transmitter 116 and receiver 132, and communication channel 120 may be embodied within connection 221.

[0027] The computer module 201 typically includes at least one processor unit 205 and a memory unit 206. For example, the memory unit 206 can include semiconductor random access memory (RAM) and semiconductor read-only memory (ROM). The computer module 201 also includes several input / output (I / O) interfaces, including an audio / video interface 207 that couples to a video display 214, speakers 217, and microphone 280; an I / O interface 213 that couples to a keyboard 202, a mouse 203, a scanner 226, a camera 227, and optionally a joystick or other human interface device (not shown); and an interface 208 for an external modem 216 and a printer 215. The signal from the audio / video interface 207 to the computer monitor 214 is typically the output of a computer graphics card. In some implementations, the modem 216 may be incorporated into the computer module 201, for example, in the interface 208. The computer module 201 also has a local network interface 211, which enables coupling of the computer system 200 to a local area communications network 222, known as a local area network (LAN), via a connection 223. As shown in Figure 2A, the local communications network 222 can also be coupled to a wide network 220 via a connection 224, which typically includes a so-called "firewall" device or a device of similar functionality. The local network interface 211 can include an Ethernet™ circuit card, a Bluetooth™ wireless configuration, or an IEEE 802.11 wireless configuration, although many other types of interfaces may be implemented for the interface 211. The local network interface 211 can also provide the functionality of the transmitter 116, and the receiver 132 and the communication channel 120 can also be embodied in the local communications network 222.

[0028] I / O interfaces 208 and 213 can provide either or both serial and parallel connectivity, the former typically implemented according to the Universal Serial Bus (USB) standard and having a corresponding USB connector (not shown). Storage 209 is provided and typically includes a hard disk drive (HDD) 210. Other storage devices, such as floppy disk drives and magnetic tape drives (not shown), can also be used. An optical disk drive 212 is typically provided to serve as a non-volatile source of data. For example, portable memory devices, such as optical disks (e.g., CD-ROM, DVD, Blu-ray Disc™), USB-RAM, portable, external hard drives, and floppy disks, can be used as suitable sources of data for computer system 200. Typically, any of HDD 210, optical drive 212, and networks 220 and 222 may be configured to operate as video source 112 or as a destination for decoded video data to be stored for playback via display 214. The source device 110 and the destination device 130 of the system 100 may be embodied in a computer system 200 .

[0029] The components 205-213 of the computer module 201 typically communicate via an interconnected bus 204 and in a manner which results in a conventional mode of operation of the computer system 200 known to those skilled in the art. For example, the processor 205 is coupled to the system bus 204 using a connection 218. Similarly, the memory 206 and optical disk drive 212 are coupled to the system bus 204 by a connection 219. Examples of computers on which the above-described arrangements can be executed include IBM-PCs and compatibles, Sun SPARC stations, Apple Mac™ or alike computer systems.

[0030] Where appropriate or necessary, video encoder 114 and video decoder 134, as well as the methods described below, may be implemented using computer system 200. Specifically, video encoder 114, video decoder 134, and the methods described may be implemented as one or more software application programs 233 executable within computer system 200. Specifically, video encoder 114, video decoder 134, and the steps of the methods described are performed by instructions 231 (see FIG. 2B ) in software 233 executing within computer system 200. The software instructions 231 may be formed as one or more code modules, each for performing one or more specific tasks. The software may also be divided into two separate portions, where a first portion and corresponding code module performs the methods described and a second portion and corresponding code module manages a user interface between the first portion and a user.

[0031] The software may be stored on a computer-readable medium, including, for example, the storage devices described below. The software is loaded from the computer-readable medium into computer system 200 and then executed by computer system 200. Such software or a computer-readable medium having a computer program recorded thereon is a computer program product. Use of the computer program product in computer system 200 preferably results in an advantageous apparatus for implementing video encoder 114, video decoder 134, and the methods described.

[0032] The software 233 is typically stored on the HDD 210 or memory 206. The software is loaded into the computer system 200 from a computer-readable medium and executed by the computer system 200. Thus, for example, the software 233 may be stored on an optically readable disk storage medium (e.g., a CD-ROM) 225 that is read by the optical disk drive 212.

[0033] In some cases, the application programs 233 may be supplied to the user encoded on one or more CD-ROMs 225 and read via the corresponding drive 212 or read by the user from the network 220 or 222. Furthermore, software may also be loaded into the computer system 200 from other computer-readable media. A computer-readable storage medium refers to any non-transitory, tangible storage medium that provides recorded instructions and / or data to the computer system 200 for execution and / or processing. Examples of such storage media include floppy disks, magnetic tapes, CD-ROMs, DVDs, Blu-ray Discs™, hard disk drives, ROMs or integrated circuits, USB memories, magneto-optical disks, or computer-readable cards such as PCMCIA cards, whether such devices are internal or external to the computer module 201. Examples of transitory or non-tangible computer-readable transmission media that may also participate in providing software, application programs, instructions and / or video data or encoded video data to the computer module 401 include wireless or infrared transmission channels, as well as network connections to another computer or networked device, and the Internet or intranet, including email transmissions and information recorded on websites and the like.

[0034] A second portion of application program 233 and the corresponding code modules described above may be executed to implement one or more graphical user interfaces (GUIs) that are rendered or otherwise represented on display 214. Typically through manipulation of keyboard 202 and mouse 203, a user of application and computer system 200 can manipulate the interface in a functionally adaptable manner to provide control commands and / or input to the application associated with the GUI. Other forms of functionally adaptable user interfaces may also be implemented, such as an audio interface that utilizes speech prompts output via speaker 217 and user voice commands input via microphone 280.

[0035] Figure 2B is a detailed schematic block diagram of processor 205 and "memory" 234. Memory 234 represents a logical collection of all memory modules (including HDD 209 and semiconductor memory 206) accessible by computer module 201 of Figure 2A.

[0036] When the computer module 201 is first powered on, a power-on self-test (POST) program 250 is executed. The POST program 250 is typically stored in the ROM 249 of the semiconductor memory 206 of FIG. 2A. Hardware devices, such as the ROM 249, that store software are sometimes referred to as firmware. The POST program 250 inspects the hardware within the computer module 201 to ensure proper functioning and typically checks the processor 205, memory 234 (209, 206), and basic input / output system software (BIOS) module 251 (typically also stored in the ROM 249) for correct operation. If the POST program 250 runs successfully, the BIOS 251 boots the hard disk drive 210 of FIG. 2A. Booting the hard disk drive 210 executes a bootstrap loader program 252, resident on the hard disk drive 210, via the processor 205. This loads the operating system 253 into the RAM memory 206, upon which the operating system 253 begins operation. Operating system 253 is a system-level application executable by processor 205 and performs a variety of high-level functions, including processor management, memory management, device management, storage management, software application interface, and general-purpose user interface.

[0037] Operating system 253 manages memory 234 (209, 206) to ensure that each process or application running on computer module 201 has enough memory to execute without conflicting with memory allocated to another process. Furthermore, the different types of memory available in computer system 200 of FIG. 2A must be used appropriately so that each process can execute effectively. Thus, aggregate memory 234 is not intended to indicate how specific segments of memory are allocated (unless otherwise noted), but rather is intended to provide a general view of memory accessible by computer system 200 and how such segments are used.

[0038] As shown in FIG. 2B, processor 205 includes a number of functional modules, including a control unit 239, an arithmetic logic unit (ALU) 240, and a local or internal memory 248, sometimes referred to as a cache memory. Cache memory 248 typically includes a number of storage registers 244-246 within a register section. One or more internal buses 241 functionally interconnect these functional modules. Processor 205 also typically has one or more interfaces 242 for communicating with external devices via system bus 204 using connection 218. Memory 234 is coupled to bus 204 using connection 219.

[0039] Application program 233 includes a sequence of instructions 231, which may include conditional branch and loop instructions. Program 233 may also include data 232 used in the execution of program 233. Instructions 231 and data 232 are stored in memory locations 228, 229, 230 and 235, 236, 237, respectively. Depending on the relative sizes of instruction 231 and memory locations 228-230, a particular instruction may be stored in a single memory location, as indicated by the instruction shown in memory location 230. Alternatively, the instruction may be segmented into several parts, each stored in a separate memory location, as indicated by the instruction segments shown in memory locations 228 and 229.

[0040] Generally, the processor 205 is given a set of instructions to execute within it. The processor 205 waits for subsequent input, to which the processor 205 responds by executing another set of instructions. Each input may come from one or more of several sources, including data generated by one or more of the input devices 202, 203, data received from an external source via one of the networks 220, 202, data retrieved from one of the storage devices 206, 209, or data retrieved from a storage medium 225 inserted into a corresponding reader 212, all shown in FIG. 2A. Execution of the set of instructions may result in data output. Execution may also include storing data or variables to memory 234.

[0041] Video encoder 114, video decoder 134, and the described methods may use input variables 254 that are stored in corresponding memory locations 255, 256, 257 in memory 234. Video encoder 114, video decoder 134, and the described methods generate output variables 261, which are stored in corresponding memory locations 262, 263, 264 in memory 234. Intermediate variables 258 may be stored in memory locations 259, 260, 266, and 267.

[0042] Referring to processor 205 of FIG. 2B, registers 244, 245, 246, arithmetic logic unit (ALU) 240, and control unit 239 cooperate to perform the sequence of micro-operations required to perform a "fetch, decode, and execute" cycle for every instruction in the instruction set that makes up program 233. Each fetch, decode, and execute cycle is A fetch operation to fetch or read an instruction 231 from memory locations 228, 229, 230 A decode operation in which the control unit 239 determines which instructions have been fetched. The control unit 239 and / or the ALU 240 perform the operations to execute the instructions. It has.

[0043] Thereafter, a further fetch, decode, and execute cycle of the next instruction may be performed. Similarly, a store cycle may be performed in which control unit 239 stores or writes a value to memory location 232.

[0044] Each step or sub-process in the methods of Figures 10 and 11 described below is associated with one or more segments of program 233 and is typically performed by register sections 244, 245, 247, ALU 240, and control unit 239 within processor 205 working together to perform a fetch, decode, and execute cycle for all instructions in the instruction set for the noted segment of program 233.

[0045] Figure 3 is a schematic block diagram illustrating the functional modules of the video encoder 114. Figure 4 is a schematic block diagram illustrating the functional modules of the video decoder 134. Generally, data passes between the functional modules of the video decoder 134 and the video encoder 114 in groups of samples or coefficients, such as division of a block into fixed-size sub-blocks, or as an array. The video encoder 114 and the video decoder 134 can be implemented using a general-purpose computer system 200, as shown in Figures 2A and 2B, with the various functional modules being realized by software executable within the computer system 200, such as one or more software code modules of a software application program 233 resident on a hard disk drive 205 and controlled during its execution by the processor 205, or by dedicated hardware within the computer system 200. Alternatively, the video encoder 114 and the video decoder 134 may be implemented by a combination of software executable within the computer system 200 and dedicated hardware. The video encoder 114, the video decoder 134, and the described methods may alternatively be implemented by dedicated hardware, such as one or more integrated circuits that perform the functions or sub-functions of the described methods. Such dedicated hardware may include a graphics processing unit (GPU), a digital signal processor (DSP), an application specific standard product (ASSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or one or more microprocessors and associated memory. In particular, the video encoder 114 includes modules 310-386, and the video decoder 134 includes modules 420-496, which may each be implemented as one or more software code modules of the software application program 233.

[0046] The video encoder 114 of FIG. 3 is an example of a generic video coding (VVC) video encoding pipeline, although other video codecs may be used to perform the processing stages described herein. The video encoder 114 receives captured frame data 113, such as a series of frames, each of which includes one or more color channels. The frame data 113 may be in 4:2:0 chroma format or 4:2:2 chroma format. The block partitioner 310 first divides the frame data 113 into CTUs, generally square in shape, with a specific size for the CTUs configured to be used. The CTU size may be, for example, 64x64, 128x128, or 256x256 luma samples. The block partitioner 310 further divides each CTU into one or more CBs according to the luma coding tree and chroma coding tree. The CBs may have various sizes and include both square and non-square aspect ratios. The operation of the block partitioner 310 is further described with reference to FIG. 10. However, in the VVC standard, CBs, CUs, PUs, and TUs always have side lengths that are powers of 2. Thus, the current CB, denoted as 312, is output from block partitioner 310 and proceeds according to iterations over one or more blocks of the CTU according to the chroma coding tree and luma coding tree of the CTU. Options for partitioning a CTU into CBs are further described below with reference to Figures 5 and 6.

[0047] The CTUs resulting from the first partition of frame data 113 may be scanned in raster scan order and grouped into one or more "slices." A slice may be an "intra" (or "I") slice, where an intra slice (I slice) indicates that all CUs in the slice are intra-predicted. Alternatively, a slice may be uni- or bi-predictive ("P" or "B" slice, respectively), indicating the further availability of uni- and bi-prediction in the slice, respectively.

[0048] For each CTU, the video encoder 114 operates in two stages. In the first stage (called the "search" stage), the block partitioner 310 tests various potential configurations of the coding tree. Each potential configuration of the coding tree has an associated "candidate" CB. The first stage involves testing various candidate CBs to select the CB that provides high compression efficiency with low distortion. This testing typically involves Lagrangian optimization, whereby the candidate CBs are evaluated based on a weighted combination of rate (coding cost) and distortion (error with respect to the input frame data 113). The "best" candidate CB (the CB with the lowest evaluated rate / distortion) is selected for subsequent encoding into the bitstream 115. Evaluation of the candidate CBs includes the options of using the CB for a given area, further dividing the area according to various partitioning options and encoding each of the resulting smaller areas with additional CBs, or further dividing the area. Consequently, both the CB and the coding tree itself are selected in the search stage.

[0049] The video encoder 114 generates a predictive block (PB), indicated by arrow 320, for each CB, e.g., CB 312. PB 320 is a prediction of the content of the associated CB 312. A subtractor module 322 generates a difference, indicated as 324 (or "residual," referring to the difference being in the spatial domain), between PB 320 and CB 312. Difference 324 is the block-sized difference between corresponding samples in PB 320 and CB 312. Difference 324 is transformed and quantized, and represented as a transform block (TB), indicated by arrow 336. PB 320 and associated TB 336 are typically selected from one of many possible candidate CBs, for example, based on estimated cost or distortion.

[0050] A candidate coding block (CB) is a CB that results from one of the prediction modes available to the video encoder 114 for an associated PB and the resulting residual. Each candidate CB results in one or more corresponding TBs, as described below with reference to FIG. 8. The TBs 336 are a quantified, transformed representation of the difference 324. When combined with the predicted PB in the video decoder 114, the TBs 336 reduce the difference between the decoded CB and the original CB 312, at the expense of additional signal in the bitstream.

[0051] Thus, each candidate coding block (CB), i.e., a prediction block (PB) combined with a transform block (TB), has an associated coding cost (or "rate") and an associated differential (or "distortion"). The rate is typically measured in bits. The distortion of a CB is typically estimated as a sample-value differential, such as the sum of absolute differences (SAD) or the sum of squared differences (SSD). The estimate obtained from each candidate PB is used by a mode selector 386 to determine the intra-prediction mode (represented by arrow 388) using the differential 324. Estimating the coding cost associated with each candidate prediction mode and the corresponding residual coding can be performed at a significantly lower cost than entropy coding of the residual. Thus, a large number of candidate modes can be evaluated to determine the optimal mode for rate-distortion detection.

[0052] Determining the optimal mode from a rate-distortion perspective is typically accomplished using a variant of Lagrangian optimization. Selecting an intra-prediction mode 388 typically involves determining the coding cost for the residual data resulting from applying a particular intra-prediction mode. The coding cost can be approximated by using the "sum of absolute transform differences" (SATD), thereby using a relatively simple transform, such as the Hadamard transform, to obtain an estimated transform residual cost. In some embodiments using a relatively simple transform, the cost resulting from the simplified estimation method is monotonically related to the actual cost that would otherwise be determined from a full evaluation. In embodiments with monotonically related estimated costs, the simplified estimation method can be used to make the same decision (i.e., intra-prediction mode) while reducing the complexity of the video encoder 114. To allow for possible non-monotonicity in the relationship between the estimated cost and the actual cost, the simplified estimation method can be used to generate a list of best candidates. Non-monotonicity may arise, for example, from additional mode decisions available for encoding the residual data. The list of best candidates may be any number of candidates. Using the best candidates, a more thorough search can be performed to establish the optimal mode selection for encoding the residual data for each of the candidates, allowing for the final selection of the intra-prediction mode along with other mode decisions.

[0053] Other mode decisions include the ability to skip the forward transform, known as "transform skip." Skipping the transform is appropriate for residual data that lacks adequate correlation to reduce coding costs via representation as transform basis functions. Certain types of content, such as relatively simple computer-generated graphics, may exhibit similar behavior. In the case of a "skipped transform," the residual coefficients are still coded, even though the transform itself is not performed.

[0054] A Lagrangian or similar optimization process can be employed to select both the optimal partitioning of the CTU into CBs (by the block partitioner 310) and the selection of the best prediction mode from multiple possibilities. Through application of a Lagrangian optimization process of the candidate modes in the mode selection module 386, the intra-prediction mode with the lowest cost measure is selected as the “best” mode. The lowest-cost mode is the selected intra-prediction mode 388 and is encoded into the bitstream 115 by the entropy encoder 338. The selection of the intra-prediction mode 388 by the operation of the mode selection module 386 extends the operation of the block partitioner 310. For example, candidates for selection of the intra-prediction mode 388 can include modes applicable to a given block and also modes applicable to multiple smaller blocks collectively located with the given block. When including modes applicable to a given block and smaller co-located blocks, the candidate selection process is implicitly also a process of determining the best hierarchical decomposition of the CTU into CBs.

[0055] In a second phase of operation of the video encoder 114 (called the "encoding" stage), an iteration over the selected luma coding tree and the selected chroma coding tree, and therefore over each selected CB, is performed within the video encoder 114. In the iteration, the CB is encoded into a bitstream 115, as described further herein.

[0056] The entropy encoder 338 supports both variable-length coding of syntax elements and arithmetic coding of syntax elements. Arithmetic coding is supported using a context-adaptive binary arithmetic coding process. Arithmetically coded syntax elements consist of a sequence of one or more 'bins'. Bins, like bits, have a value of "0" or "1." However, bins are not coded as discrete bits in the bitstream 115. Bins have an associated predicted (or "likelihood" or "most likely") value and associated probability, known as a "context." When the actual bin to be coded matches the predicted value, a "most probable symbol" (MPS) is coded. Coding the most probable symbol is relatively inexpensive in terms of consumed bits. When the actual bin to be coded does not match the likely value, a "least probable symbol" (LPS) is coded. Coding the least probable symbol has a relatively high cost in terms of consumed bits. The bin coding technique allows for efficient coding of bins where the probabilities of "0" versus "1" are skewed. For syntax elements with two possible values ​​(i.e., flag), a single bin is sufficient. For syntax elements with many possible values, a range of bins is required.

[0057] The presence of a later bin in the sequence may be determined based on the value of a previous bin in the sequence. Furthermore, each bin may be associated with two or more contexts. The selection of a particular context may depend on the previous bin of the syntax element, the bin values ​​of adjacent syntax elements (i.e., from adjacent blocks), etc. Each time a context-encoded bin is encoded, the context (if any) selected for that bin is updated in a manner that reflects the new bin value. In this way, binary arithmetic coding schemes are said to be adaptive.

[0058] Also supported by the video encoder 114 are bins that lack context ("bypass bins"). Bypass bins are coded assuming an equal probability distribution between "0" and "1". Thus, each bin occupies one bit in the bitstream 115. The lack of context saves memory and reduces complexity. Thus, bypass bins are used when the distribution of values ​​for a particular bin is not biased. One example of an entropy coder that uses context and adaptation is known in the art as CABAC (Context-Adaptive Binary Arithmetic Coder), and many variations of this coder are used in video coding.

[0059] The entropy encoder 338 encodes the intra-prediction mode 388 using a combination of context coding bins and bypass coding bins. Typically, a list of "most probable modes" is generated in the video encoder 114. The list of most probable modes is typically a fixed length, such as three or six modes, and may include modes encountered in previous blocks. The context coding bin encodes a flag indicating whether the intra-prediction mode 388 is one of the most probable modes. If the intra-prediction mode 388 is one of the most probable modes, further signaling using bypass-coded bins is encoded. The further encoded signaling may indicate which most probable mode corresponds to the intra-prediction mode 388, for example, using a truncated unary bin string. Otherwise, the intra-prediction mode 388 is coded as a "remaining mode." The coding as a remaining mode uses an alternative syntax, such as a fixed-length code that is also coded using bypass-coded bins, to represent intra-prediction modes other than those present in the most probable mode list.

[0060] The multiplexer module 384 outputs the PB 320 according to the determined best intra-prediction mode 388 selected from the tested prediction modes for each candidate CB. The candidate prediction modes need not include all possible prediction modes supported by the video encoder 114.

[0061] Prediction modes are broadly classified into two categories. The first category is "intra-frame prediction" (also called "intra prediction"). In intra-frame prediction, a prediction for a block is generated, and the generation method may use other samples obtained from the current frame. For intra-predicted PBs, different intra-prediction modes can be used for luma and chroma, and therefore intra-prediction is mainly described with respect to operating on PBs.

[0062] The second category of prediction modes is "interframe prediction" (also called "inter prediction"). In interframe prediction, a prediction of a block is generated using samples from one or two frames that precede the current frame in the bitstream. Furthermore, for interframe prediction, a single coding tree is typically used for both the luma and chroma channels. The coding order of frames in the bitstream may differ from the order of frames at capture or display. If one frame is used for prediction, the block is said to be "uni-predictive" and has one associated motion vector. If two frames are used for prediction, the block is said to be "bi-predictive" and has two associated motion vectors. For P slices, each CU can be intra-predicted or uni-predicted. For B slices, each CU can be intra-predicted, uni-predicted, or bi-predicted. Frames are typically coded using a "group of pictures" structure that enables temporal hierarchy of frames. Temporal hierarchy of frames allows frames to reference previous and subsequent pictures in the order in which they are displayed. The images are coded in the order required to ensure that the dependencies for decoding each frame are met.

[0063] A subcategory of inter prediction is called "skip mode." Inter prediction and skip mode are described as two separate modes. However, both inter prediction and skip mode involve motion vectors that reference blocks of samples from a previous frame. Inter prediction involves a coded motion vector delta that specifies a motion vector relative to a motion vector predictor. The motion vector predictor is obtained from a list of one or more candidate motion vectors selected by a "merge index." The coded motion vector delta provides a spatial offset to the selected motion vector prediction. Inter prediction also uses coded residuals in the bitstream 133. Skip mode uses only an index (also called a "merge index") to select one of several motion vector candidates. The selected candidate is used without further signaling. Skip mode also does not support coding of residual coefficients. When skip mode is used, the lack of coded residual coefficients means that there is no need to perform a transform for skip mode. Therefore, skip mode typically does not cause pipelining issues. Pipelining issues may occur for both intra-predicted and inter-predicted CUs. Due to the limited signaling of skip mode, skip mode is useful for achieving very high compression performance when relatively high-quality reference frames are available. Bi-predictive CUs in higher temporal layers of a random access picture group structure typically have high-quality reference pictures and motion vector candidates that accurately reflect the underlying motion.

[0064] Samples are selected according to a motion vector and a reference picture index. The motion vector and reference picture index apply to all color channels; therefore, inter-prediction is primarily described in terms of operating on PUs, not PBs. Within each category (i.e., intra- and inter-frame prediction), different techniques can be applied to generate PUs. For example, intra-prediction can use values ​​from adjacent rows and columns of previously reconstructed samples in combination with a predetermined filtering and generation process to generate a PU. Alternatively, a PU may be described using a small number of parameters. Inter-prediction methods can vary in the number and precision of motion parameters. Motion parameters typically include a reference frame index indicating which reference frame from a list of reference frames should be used and a spatial transformation for each reference frame, but can also include more frames, special frames, or complex affine parameters such as scaling and rotation. Furthermore, a predetermined motion refinement process can be applied to generate a dense motion estimate based on the reference sample block.

[0065] Determining and selecting PB 320 and subtracting it from the original sample block in subtractor 322 results in a residual, represented by 324, with the lowest coding cost, which undergoes lossy compression. The lossy compression process includes the steps of transform, quantization, and entropy coding. A forward primary transform module 326 applies a forward transform to the difference 324, converting it from the spatial domain to the frequency domain and generating primary transform coefficients, represented by arrow 328. The primary transform coefficients 328 are passed to a forward secondary transform module 330, which performs a non-separable secondary transform (NSST) operation to generate transform coefficients, represented by arrow 332. The forward primary transform is typically separable, typically using a DCT-2 to transform the set of rows and then the set of columns of each block, although DCT-7 and DCT-8 may also be used, e.g., horizontally for block widths not exceeding 16 samples and vertically for block heights not exceeding 16 samples. The transform of each set of rows and columns is performed by first applying a one-dimensional transform to each row of the block to generate a partial result, and then applying a one-dimensional transform to each column of the partial result to generate a final result. The forward secondary transform is generally a non-separable transform, which is applied only to the residual of the intra-predicted CU and may nevertheless be bypassed. The forward secondary transform operates on either 16 samples (arranged as a top-left 4x4 sub-block of the primary transform coefficients 328) or 64 samples (arranged as a top-left 8x8 coefficient, arranged as four 4x4 sub-blocks of the primary transform coefficients 328). Furthermore, the matrix coefficients of the forward secondary transform are selected from multiple sets according to the intra-prediction mode of the CU, so that two sets of coefficients are available for use. The use of one of the sets of matrix coefficients, or the bypass of the forward secondary transform, is signaled with the "nsst_index" syntax element, which is coded using a truncated unary binarisation to represent the values ​​zero (no secondary transform applied), one (first set of selected matrix coefficients), or two (second set of selected matrix coefficients).

[0066] The transform coefficients 332 are passed to a quantizer module 334, where quantization by a quantization parameter is performed to generate residual coefficients, represented by arrow 336. The quantization parameter is constant for a given TB, thus resulting in uniform scaling for the generation of the residual coefficients for the TB. Non-uniform scaling is also possible by applying a quantization matrix, whereby the scaling factor applied to each residual coefficient is derived from a combination of the quantization parameter and a corresponding entry in a scaling matrix, typically having a size equal to the size of the TB. The residual coefficients 336 are provided to an entropy encoder 338 for encoding in the bitstream 115. Typically, the residual coefficients of each TB having at least one significant residual coefficient of a TU are scanned according to a scan pattern to generate an ordered list of values. The scan pattern generally scans the TB as a sequence of 4x4 sub-blocks, providing a regular scanning operation with a granularity of 4x4 sets of residual coefficients, with the arrangement of the sub-blocks depending on the size of the TB. Additionally, the prediction modes 388 and corresponding block partitions are also coded into the bitstream 115 .

[0067] As described above, the video encoder 114 needs access to a frame representation that corresponds to the frame representation seen by the video decoder 134. Accordingly, the residual coefficients 336 are also inverse quantized by an inverse quantizer module 340 to generate inverse transform coefficients, represented by arrow 342. The inverse transform coefficients 342 are passed through an inverse secondary transform module 344 to generate intermediate inverse transform coefficients, represented by arrow 346. The intermediate inverse transform coefficients 346 are passed to an inverse primary transform module 348 to generate residual samples, represented by arrow 350, for the CU. The type of inverse transform performed by the inverse secondary transform module 344 corresponds to the type of forward transform performed by the forward secondary transform module 330. The type of inverse transform performed by the inverse primary transform module 348 corresponds to the type of primary transform performed by the primary transform module 326. An addition module 352 adds the residual samples 350 and the PU 320 to generate reconstructed samples for the CU (indicated by arrow 354).

[0068] The reconstructed samples 354 are passed to a reference sample cache 356 and an in-loop filter module 368. The reference sample cache 356 is typically implemented using static RAM on the ASIC (thus avoiding costly off-chip memory accesses) and provides the minimum sample storage required to satisfy the dependencies for generating intra-frame PBs for subsequent CUs in the frame. The minimum dependencies typically include a "line buffer" of samples along the bottom of a row of CTUs for use by the CTU's next row and column buffering, the extent of which is set by the CTU's height. The reference sample cache 356 supplies reference samples (indicated by arrow 358) to a reference sample filter 360. The sample filter 360 applies a smoothing operation to generate filtered reference samples (indicated by arrow 362). The filtered reference samples 362 are used by an intra-frame prediction module 364 to generate an intra-predicted block of samples, represented by arrow 366. For each candidate intra-prediction mode, the intra-frame prediction module 364 generates a block of samples, i.e., 366.

[0069] The in-loop filter module 368 applies several filtering stages to the reconstructed samples 354. The filtering stages include a "deblocking filter" (DBF), which applies smoothing aligned to CU boundaries to reduce artifacts resulting from discontinuities. Another filtering stage present in the in-loop filter module 368 is an "adaptive loop filter" (ALF), which applies a Wiener-based adaptive filter to further reduce distortion. A further available filtering stage in the in-loop filter module 368 is a "sample adaptive offset" (SAO) filter. The SAO filter operates by first classifying the reconstructed samples into one or more categories and then applying an offset at the sample level according to the assigned category.

[0070] The filtered samples, represented by arrow 370, are output from the in-loop filter module 368. The filtered samples 370 are stored in a frame buffer 372. The frame buffer 372 typically has the capacity to store several pictures (e.g., up to 16) and is therefore stored in the memory 206. The frame buffer 372 is not typically stored using on-chip memory due to the large memory consumption required. Therefore, accessing the frame buffer 372 is costly in terms of memory bandwidth. The frame buffer 372 provides reference frames (represented by arrow 374) to a motion estimation module 376 and a motion compensation module 380.

[0071] The motion estimation module 376 estimates several “motion vectors” (denoted as 378), each a Cartesian spatial offset from the position of the current CB and referencing a block in one of the reference frames in the frame buffer 372. A filtered block of reference samples (denoted as 382) is generated for each motion vector. The filtered reference samples 382 form further candidate modes available for potential selection by the mode selector 386. Furthermore, for a given CU, the PU 320 may be formed using one reference block (“uni-prediction”) or two reference blocks (“bi-prediction”). For a selected motion vector, the motion compensation module 380 generates the PB 320 according to a filtering process that supports sub-pixel accuracy in motion vectors. Thus, the motion estimation module 376 (operating on many candidate motion vectors) can perform a simplified filtering process compared to that of the motion compensation module 380 (operating on only the selected candidate) to reduce computational complexity.

[0072] 3 is described with reference to generic video coding (VVC), other video coding standards or implementations may use the processing stages of modules 310-386. Frame data 113 (and bitstream 115) may also be read from (or written to) memory 206, hard disk drive 210, CD-ROM, Blu-ray Disc™, or other computer-readable storage medium. Additionally, frame data 113 (and bitstream 115) may be received from (or transmitted to) an external source, such as a server connected to communications network 220 or a radio frequency receiver.

[0073] The video decoder 134 is shown in Figure 4. The video decoder 134 of Figure 4 is an example of a generic video coding (VVC) video decoding pipeline, although other video codecs may be used to perform the processing steps described herein. As shown in Figure 4, a bitstream 133 is input to the video decoder 134. The bitstream 133 may be stored in memory 206, a hard disk drive 210, a CD-ROM, a Blu-ray disc, or any other suitable storage medium. TM or other non-transitory computer-readable storage medium. Alternatively, bitstream 133 may be received from an external source, such as a server connected to communications network 220 or a radio frequency receiver. Bitstream 133 includes encoded syntax elements representing imaging frame data to be decoded.

[0074] The bitstream 133 is input to the entropy decoder module 420. The entropy decoder module 420 extracts syntax elements from the bitstream 133 by decoding a sequence of "bins" and passes the values ​​of the syntax elements to other modules in the video decoder 134. The entropy decoder module 420 uses an arithmetic decoding engine to decode each syntax element as a sequence of one or more bins. Each bin can use one or more "contexts," with the context describing the probability levels used to encode the bin's "1" and "0" values. If multiple contexts are available for a given bin, a "context modeling" or "context selection" step is performed to select one of the available contexts for decoding the bin. The bin decoding process sequentially forms a feedback loop. The number of operations in the feedback loop is preferably minimized to enable the entropy decoder 420 to achieve a high throughput in bins per second. Context modeling relies on other properties of the bitstream known to the video decoder 134 when selecting a context, i.e., previous properties of the current bin. For example, the context may be selected based on the quadtree depth of the current CU in the coding tree. Preferably, the dependency is determined based on well-known characteristics or without requiring lengthy sequential processing before decoding the bin. The quadtree depth of the coding tree is an example of a dependency on context modeling that is easily known. The intra-prediction mode is an example of a dependency for context modeling that is relatively difficult or computationally intensive to determine. The intra-prediction mode is coded as either an index into a list of "most probable modes" (MPMs) or an index into a list of "remaining modes," and the selection between the MPMs and remaining modes is according to the decoded "intra_luma_mpm_flag."If MPM is used, the "intra_luma_mpm_idx" syntax element is decoded to select which of the most probable modes to use. Generally, there are six MPMs. If a remaining mode is used, the "intra_luma_remainder" syntax element is decoded to select which of the remaining (non-MPM) modes to use. Determining both the most probable mode and the remaining mode requires a significant number of operations and includes dependency on the intra-prediction mode of neighboring blocks. For example, the neighboring block may be the block to the upper left of the current block. Preferably, the bin context of each CU can be determined without knowing the signaled intra-prediction mode, allowing parsing by the arithmetic coding engine. Therefore, the feedback loop present in the arithmetic coding engine for sequential bin decoding avoids dependency on the intra-prediction mode. Due to the dependency of the MPM list configuration on the intra-prediction modes of neighboring blocks, the intra-prediction mode decision can be deferred to a subsequent processing stage using a separate feedback loop. Thus, the arithmetic decoding engine of the entropy decoder module 420 can parse intra_luma_mpm_flag, intra_luma_mpm_idx, and intra_luma_remainder without needing to know the intra-prediction modes of previous (e.g., neighboring) blocks. The entropy decoder module 420 applies an arithmetic coding algorithm, such as "context-adaptive binary arithmetic coding" (CABAC), to decode the syntax elements from the bitstream 133. The decoded syntax elements are used to reconstruct parameters within the video decoder 134. The parameters include residual coefficients (represented by arrow 424) and mode selection information, such as the intra-prediction mode (represented by arrow 458). The mode selection information also includes information such as motion vectors and the division of each CTU into one or more CBs. The parameters, typically in combination with sample data from previously decoded CBs, are used to generate PBs.

[0075] The residual coefficients 424 are input to the inverse quantization module 428. The inverse quantization module 428 performs inverse quantization (or “scaling”) on the residual coefficients 424 to generate reconstructed intermediate transform coefficients, represented by arrow 432, according to a quantization parameter. The reconstructed intermediate transform coefficients 432 are passed to the inverse secondary transform module 436, where a secondary transform is either applied or not operated (bypassed) according to a decoded “nsst_index” syntax element. The “nsst_index” is decoded from the bitstream 133 by the entropy decoder 420 under execution of the processor 205. As described with reference to FIG. 3 , the “nsst_index” is decoded from the bitstream 133 as a truncated unary syntax element having a value from zero to two. The inverse secondary transform module 436 generates reconstructed transform coefficients 440. If the use of a non-uniform inverse quantization matrix is ​​indicated in bitstream 133, video decoder 134 reads the quantization matrix from bitstream 133 as a sequence of scaling factors and arranges the scaling factors into a matrix. Inverse scaling uses the quantization matrix in combination with the quantization parameters to generate reconstructed intermediate transform coefficients 432.

[0076] The reconstructed transform coefficients 440 are passed to an inverse linear transform module 444, which converts the coefficients from the frequency domain back to the spatial domain. The TB is effectively based on the significant and insignificant residual coefficient values. The result of the operation of module 444 is a block of residual samples, represented by arrow 448. The residual samples 448 are equal in size to the corresponding CU. The residual samples 448 are provided to an addition module 450, where they are added to a decoded PB (represented as 452) to generate a block of reconstructed samples, represented by arrow 456. The reconstructed samples 456 are provided to a reconstructed sample cache 460 and an in-loop filtering module 488. The in-loop filtering module 488 generates a reconstructed block of frame samples, represented as 492. The frame samples 492 are written to a frame buffer 496.

[0077] The reconstructed sample cache 460 operates similarly to the reconstructed sample cache 356 of the video encoder 114. The reconstructed sample cache 460 provides storage for reconstructed samples needed to intra-predict subsequent CBs without going through the memory 206 (e.g., by substituting the data 232, which is typically on-chip memory). Reference samples, represented by arrow 464, are obtained from the reconstructed sample cache 460 and provided to a reference sample filter 468 to generate filtered reference samples, indicated by arrow 472. The filtered reference samples 472 are provided to an intra-frame prediction module 476. The module 476 generates blocks of intra-predicted samples, represented by arrow 480, according to the intra-prediction mode parameters 458 signaled in the bitstream 133 and decoded by the entropy decoder 420.

[0078] When the prediction mode of the CB is indicated as intra prediction in the bitstream 133, the intra prediction samples 480 form the decoded PB 452 via the multiplexer module 484. Intra prediction generates a predictive block (PB) of samples, i.e., a block in one color component derived using "neighboring samples" in the same color component. Neighboring samples are samples adjacent to the current block and have already been reconstructed by preceding it in block decoding order. When luma and chroma blocks are juxtaposed, the luma and chroma blocks can use different intra prediction modes. However, each of the two chroma channels shares the same intra prediction mode. Intra prediction is classified into three types: "DC intra prediction" involves populating the PB with a single value representing the average of neighboring samples; and "Planar intra prediction" involves populating the PB with samples according to a plane, with DC offsets and vertical and horizontal gradients derived from neighboring samples. "Angular intra prediction" involves populating a PB with neighboring samples that are filtered and propagated in a particular direction (or "angle") across the PB. In VVC65, angles are supported in rectangular blocks, which allows for additional angles not available in square blocks, resulting in a total of 87 angles. A fourth type of intra prediction is available for chroma PBs, whereby the PB is generated from collocated luma reconstructed samples according to a "cross-component linear model" (CCLM) mode. Three different CCLM modes are available, each of which uses a different model derived from neighboring luma and chroma samples. The derived model is then used to generate a block of samples for the chroma PB from the collocated luma samples.

[0079] When the prediction mode of the CB is indicated as inter prediction in the bitstream 133, the motion compensation module 434 uses the motion vector and the reference frame index to select and filter a block of samples from the frame buffer 496 to generate a block of inter predicted samples, represented as 438. The block of samples 498 is obtained from a previously decoded frame stored in the frame buffer 496. In the case of bi-prediction, two blocks of samples are generated and blended together to generate samples for the decoded PB 452. The frame buffer 496 is populated with filtered block data 492 from the in-loop filtering module 488. Similar to the in-loop filtering module 368 of the video encoder 114, the in-loop filtering module 488 applies any, at least, or all of the DBF, ALF, and SAO filtering operations. Generally, motion vectors are applied to both the luma and chroma channels, although the filtering process for the sub-sample interpolated luma and chroma channels is different. If the division in the coding tree results in a set of relatively small luma blocks and the corresponding chroma region is not divided into corresponding small chroma blocks, the block is coded and decoded as described with reference to Figures 13 and 14, respectively. In particular, if any small luma block is predicted using inter prediction, the inter prediction operation is performed only on the luma CB, and not on any portion of the corresponding chroma CB. The in-loop filtering module 368 generates filtered block data 492 from the reconstructed samples 456.

[0080] 5 is a schematic block diagram illustrating a set 500 of available divisions or splits of a region into one or more subregions within a generic video coding tree structure. The divisions shown in set 500 are available to block partitioner 310 of encoder 114 to divide each CTU into one or more CUs or CBs according to the coding tree, as determined by Lagrangian optimization, as described with reference to FIG.

[0081] Although set 500 shows only square regions being partitioned into other, possibly non-square sub-regions, it should be understood that while diagram 500 illustrates a potential partition, the containing region need not be square. If the containing region is non-square, the dimensions of the blocks resulting from the partition are scaled according to the aspect ratio of the containing block. Once a region is no longer partitioned, i.e., at a leaf node of the coding tree, a CU occupies the region. The particular sub-division of a CTU by block partitioner 310 into one or more CUs is referred to as the CTU's "coding tree."

[0082] The process of subdividing a region into subregions must terminate when the resulting subregion reaches a minimum CU size. In addition to constraining CUs to prohibit block regions smaller than a predetermined minimum size, e.g., 16 samples, CUs are constrained to have a minimum width or height of 4. Other minimum values ​​are possible for both width and height, or for width or height. The subdivision process can also terminate before the deepest level of decomposition, resulting in a CU larger than the minimum CU size. It is possible for no splitting to occur, resulting in a single CU occupying the entire CTU. A single CU occupying the entire CTU is the largest available coding unit size. A CU without splitting is also larger than the processing region size. Bipartitioning or tripartitioning at the highest level of the coding tree can result in CU sizes such as 64x128, 128x64, 32x128, and 128x32, each of which is larger than the processing region size. Examples of CUs larger than the processing region size are further described with reference to Figures 10A-10F. The use of a subsampled chroma format, such as 4:2:0, allows the video encoder 114 and video decoder 134 configuration to finish dividing the regions in the chroma channels earlier than in the luma channel.

[0083] At the leaf nodes of the coding tree, there are CUs with no further subdivisions. For example, leaf node 510 contains one CU. At the non-leaf nodes of the coding tree, there are two or more further divisions, each of which may contain a leaf node and thus one CU, or may contain further divisions into smaller regions. At each leaf node of the coding tree, there is one coding block for each color channel. A division that ends at the same depth for both luma and chroma results in three juxtaposed CBs. A division that ends at a luma depth deeper than chroma results in multiple luma CBs juxtaposed with the chroma channel CBs.

[0084] A quadtree partition 512 divides the inclusion region into four equally sized regions, as shown in Figure 5. Compared to HEVC, generic video coding (VVC) achieves further flexibility by adding horizontal halves 514 and vertical halves 516. Each of the partitions 514 and 516 divides the inclusion region into two equally sized regions. The partitions are along horizontal boundaries (514) or vertical boundaries (516) within the containing block.

[0085] Further flexibility in general-purpose video coding is achieved by adding horizontal 3-division 518 and vertical 3-division 520. 3-division 518 and 520 divide a block into three regions bounded either horizontally (518) or vertically (520) along ¼ and ¾ of the width or height of the containing region. The combination of quadtree, binary tree, and ternary tree is called a "QTBTTT." The root of the tree contains zero or more quadtree divisions (the "QT" section of the tree). Once the QT section ends, zero or more binary or ternary divisions occur (the "multi-tree" or "MT" section of the tree), ultimately terminating in a tree leaf node CB or CU. If the tree describes all color channels, the tree leaf node is a CU. If the tree describes the luma or chroma channels, the tree leaf node is a CB.

[0086] Compared to HEVC, which only supports quadtrees and therefore square blocks, QTBTTT offers many more possible CU sizes, especially when considering the possible recursive application of binary and / or ternary tree partitioning. The possibility of unusual (non-square) block sizes can be reduced by constraining partitioning options to eliminate partitions where the block width or height is either less than four samples or is not a multiple of four samples. Generally, this constraint applies when considering luma samples. However, in the described configuration, the constraint can be applied separately to blocks for chroma channels. Applying the constraint to partitioning options for chroma channels can result in different minimum block sizes for luma and chroma, such as when the frame data is in 4:2:0 or 4:2:2 chroma format. Each partition generates subregions whose side dimensions remain the same, are halved, or are quartered relative to the containing region. And because the CTU size is a power of two, the side dimensions of all CUs are also powers of two.

[0087] 6 is a schematic flow diagram illustrating a data flow 600 of a QTBTTT (or "coding tree") structure used in general-purpose video coding. The QTBTTT structure is used for each CTU to define the partitioning of the CTU into one or more CUs. The QTBTTT structure for each CTU is determined by a block partitioner 310 in the video encoder 114 and encoded into the bitstream 115 or decoded from the bitstream 133 by an entropy decoder 420 in the video decoder 134. The data flow 600 further characterizes the permissible combinations available to the block partitioner 310 for partitioning the CTU into one or more CUs according to the partitioning shown in FIG. 5.

[0088] Starting from the top level of the hierarchy, i.e., CTU, zero or more quadtree splits are first performed. Specifically, a quadtree (QT) split decision 610 is made by the block partitioner 310. A decision at 610 returning a "1" symbol indicates a decision to split the current node into four subnodes according to the quadtree split 512. This results in four new nodes, such as 620, being generated, and for each new node, a return is made to the QT split decision 610. Each new node is considered in raster (or Z-scan) order. Alternatively, if the QT split decision 610 indicates that no further splits should be performed (returning a "0" symbol), the quadtree split stops and a multi-tree (MT) split is then considered.

[0089] First, an MT split decision 612 is made by the block partitioner 310. At 612, a decision to perform MT splitting is indicated. Returning a "0" symbol at decision 612 indicates that no further splitting of the node into subnodes is performed. If no further splitting of the node is performed, the node is a leaf node of the coding tree and corresponds to a CU. The leaf node is output at 622. Alternatively, if MT split 612 indicates a decision to perform MT splitting (returning a "1" symbol), the block partitioner 310 proceeds to direction decision 614.

[0090] The direction decision 614 indicates the direction of the MT partition as either horizontal ("H" or "0") or vertical ("V" or "1"). The block partitioner 310 proceeds to decision 616 if decision 614 returns a "0" indicating a horizontal direction. The block partitioner 310 proceeds to decision 618 if decision 614 returns a "1" indicating a vertical direction.

[0091] In each of decisions 616 and 618, the number of partitions for the MT split is indicated as either two (bipartition or "BT" node) or three (tripartition or "TT") for the BT / TT split. That is, BT / TT split decision 616 is made by block partitioner 310 when the indicated direction from 614 is horizontal, and BT / TT split decision 618 is made by block partitioner 310 when the indicated direction from 614 is vertical.

[0092] The BT / TT split decision 616 indicates whether the horizontal split is a 2-way split 514, indicated by returning a "0," or a 3-way split 518, indicated by returning a "1." If the BT / TT split decision 616 indicates a 2-way split, then in an HBT CTU node generation step 625, two nodes are generated by the block partitioner 310 according to the horizontal 2-way split 514. If the BT / TT split 616 indicates a 3-way split, then in an HTT CTU node generation step 626, three nodes are generated by the block partitioner 310 according to the horizontal 3-way split 518.

[0093] The BT / TT split decision 618 indicates whether the vertical split is a 2-way split 516, indicated by returning a "0," or a 3-way split 520, indicated by returning a "1." If the BT / TT split 618 indicates a 2-way split, then in a VBT CTU node generation step 627, two nodes are generated by the block partitioner 310 according to the 2-way vertical split 516. If the BT / TT split 618 indicates a 3-way split, then in a VTT CTU node generation step 628, three nodes are generated by the block partitioner 310 according to the 3-way vertical split 520. For each node resulting from steps 625-628, the recursion of the data flow 600 back to the MT split decision 612 is applied in either a left-to-right or top-to-bottom order, depending on the direction 614. As a result, binary and ternary tree partitioning can be applied to generate CUs of various sizes.

[0094] The set of allowed and disallowed splits at each node of the coding tree is further described with reference to FIG.

[0095] 7A and 7B provide an example division 700 of a CTU 710 into several CUs or CBs. An example of a CU 712 is shown in FIG. 7A. FIG. 7A shows the spatial arrangement of CUs in the CTU 710. The example division 700 is also shown as a coding tree 720 in FIG. 7B.

[0096] At each non-leaf node in the CTU 710 in FIG. 7A, e.g., nodes 714, 716, and 718, the contained nodes (which may be further split or may be CUs) are scanned or traversed in "Z-order" to create a list of nodes, represented as columns in the coding tree 720. In the case of a quadtree split, the Z-order scan is from top-left to right, followed by bottom-left to right. In the case of horizontal and vertical splits, the Z-order scan (traversal) simplifies to a top-to-bottom and left-to-right scan, respectively. The coding tree 720 in FIG. 7B lists all nodes and CUs according to the applied scan order. Each split generates a list of two, three, or four new nodes at the next level of the tree until a leaf node (CU) is reached.

[0097] The image is decomposed into CTUs by block partitioner 310 and further decomposed into CUs, which are then forward transformed and quantized by video encoder 114, using the CUs to generate respective residual blocks (324), as described with reference to FIG. 3. The resulting TBs 336 are then scanned to form a sequential list of residual coefficients as part of the operation of entropy coding module 338. A comparable process is performed within video decoder 134 to obtain TBs from bitstream 133.

[0098] The examples in Figures 7A and 7B illustrate coding trees applicable to both luma and chroma channels. However, the examples in Figures 7A and 7B also illustrate behaviors related to traversing coding trees applicable only to luma channels or coding trees applicable only to chroma channels. For coding trees with many nested splits, the splitting options available at deeper levels are constrained by the limitations on the available block sizes of the corresponding small regions. The limitations on the available block sizes for small regions are imposed to prevent worst-case block processing rates that are so high as to impose an unreasonable burden on the implementation. In particular, the constraint that block sizes be multiples of sixteen (16) samples in chroma allows implementations to process samples at a granularity of sixteen (16) samples. The restriction on block sizes to multiples of sixteen (16) samples is particularly relevant to the "intra-reconstruction" feedback loop, i.e., the path within the video decoder 134 of Figure 4 including modules 450, 460, 468, 476, and 484, as well as the equivalent path within the video encoder 114. In particular, limiting block sizes to multiples of 16 (16) samples helps maintain throughput in intra-prediction modes. For example, "simultaneous data multiple instruction" (SIMD) microprocessor architectures typically operate on wide words that can contain 16 samples. The hardware architecture can also use a wide bus, such as a bus with a width of 16 samples, to transfer samples along the intra-reconstruction feedback loop. If a smaller block size, such as four samples, is used, the bus will be underutilized, for example, by one-quarter of the bus width containing sample data. While an underutilized bus can handle smaller blocks (i.e., fewer than 16 samples), in worst-case scenarios, such as many or all blocks of relatively small size, underutilization can result in hindering real-time operation of the encoder (114) or decoder (134). For inter-prediction, each block relies on reference samples obtained from a frame buffer (e.g., buffer 372 or 496).Because the frame buffer is populated with reference samples when processing the previous frame, there are no feedback dependency loops affecting the block-by-block operation for generating inter-predicted blocks. In addition to the feedback dependency loops related to intra-frame reconstruction, there is an additional concurrent feedback loop related to determining the intra-prediction mode 458. The intra-prediction mode 458 is determined by selecting a mode from the most probable mode list or by selecting a mode from the remaining mode list. The determination of the most probable mode list and the remaining mode list requires the intra-prediction modes of neighboring blocks. When relatively small block sizes are used, the most probable mode list and the remaining mode list need to be determined more frequently, i.e., at a frequency governed by the block size in samples and the channel sampling rate.

[0099] 8A, 8B, and 8C provide an example division of a CTU 800 (8A) by a coding tree 820 (FIG. 8B) with chroma division terminated before luma division and using a 4:2:0 chroma format. When chroma division is terminated, a pair of CBs is used, one for each chroma channel. For ease of explanation, the CTU 800 is assumed to be 64x64 luma samples in size. The CTU 800 is equivalent to a CTU size of 128x128 and a coding tree including one additional quadtree division. The quadtree division is applied to an 8x8 luma region 814. The 8x8 luma region 814 is divided into four 4x4 luma CBs, but no division occurs in the chroma channels. Instead, chroma CB pairs of a predetermined minimum size (16 in the example described) are used, one for each chroma channel. The chroma CB pair is typically of a minimum size corresponding to the minimum granularity for the number of samples desired to be processed simultaneously. For example, many implementations of video encoder 114 and video encoder 134 operate on sets of 16 samples, e.g., due to the use of correspondingly wide internal buses in hardware implementations. Furthermore, each luma CB resulting from the division at least partially overlaps with a pair of chroma CBs, and the aggregate luma CB completely overlaps with a pair of chroma CBs. In the example of region 814, a 4x4 pair of chroma CBs is generated. Figure 8C shows an example of how the resulting luma CBs and chroma CBs relate.

[0100] Referring again to 8A, a vertical bisection is applied to the 16x4 luma region 810. The 16x4 luma region 810 is split into two 8x4 luma CBs, but no split occurs in the chroma channels, resulting in a pair of 8x2 chroma CBs. A vertical trisection is applied to the 16x4 luma region 812. The 16x4 luma region 812 is split into 4x4, 4x8, and 4x4 luma CBs, but no split occurs in the chroma channels, resulting in a pair of 8x2 chroma CBs. A horizontal bisection is applied to the 8x16 luma region 816. The 8x16 luma region 816 is split into 8x4, 8x8, and 8x4 luma CBs, but no split occurs in the chroma channels, resulting in a pair of 4x8 chroma CBs. Thus, the chroma CBs are at least 16 samples in area.

[0101] FIG. 8C shows a portion of a CTU 800 with three color planes shown in an "exploded" (or separated) fashion to illustrate the different block structures within the different planes. Shown are a luma sample plane 850, a first chroma sample plane 852, and a second chroma sample plane 854. When the "YCbCr" color space is in use, the luma sample plane 850 contains the Y samples of the image frame, the first chroma sample plane 852 contains the Cb samples of the image frame, and the second chroma sample plane 854 contains the Cr samples of the image frame. Using a 4:2:0 chroma format, the first chroma sample plane 852 and the second chroma sample plane 854 will have half the sample density horizontally and vertically relative to the luma sample plane 850. As a result, the CB dimensions of the chroma blocks in a sample are typically half the dimensions of the corresponding luma CB. That is, for a 4:2:0 chroma format, the width and height of a chroma CB are half the width and height, respectively, of the co-located luma CB. For a 4:2:2 chroma format, the height of a chroma CB is half the height of the co-located luma CB, and the width is the same as the width of the co-located luma CB. For clarity, only the parent split in the coding tree of the 8x16 luma region 816 is shown, and the split is shown only in the luma sample plane 850. After the chroma split is completed, multiple luma CBs are juxtaposed with pairs of chroma CBs. For example, the coding tree of the CTU 800 includes a horizontal third split applied to the 8x16 luma region 816. The horizontal third split results in an 8x4 luma CB 860, an 8x8 luma CB 862, and an 8x4 luma CB 864, which reside in the luma sample plane 850. Because the 8x16 luma region 816 corresponds to an area of ​​4x8 chroma samples within the chroma sample planes (852 and 854), the three-way division of the coding tree does not apply to the chroma sample planes (852 and 854). Thus, the area of ​​4x8 chroma samples forms a leaf node for chroma, resulting in a pair of chroma CBs: chroma CB 866 for the first chroma sample plane 852 and chroma CB 868 for the second chroma sample plane 854.In the example of horizontal third division applied to the luma plane only, a minimum chroma CB size of 32 samples is achieved. Other example luma regions (810, 812, and 814) result in a minimum chroma CB size of 16, corresponding to the minimum luma block size and desired granularity of sample processing.

[0102] FIG. 9 shows a set 900 of transform block sizes and associated scan patterns for chroma channels resulting from use of a 4:2:0 chroma format. The set 900 can also be used for a 4:2:2 chroma format. The described configuration is suitable for use with image frames having a chroma format in which the chroma channels of the image frame are subsampled relative to the luma channel of the image frame, particularly for 4:2:0 and 4:2:2 formats. The set 900 does not include all possible chroma transform block sizes. Only chroma transform blocks having a width of 16 or less or a height of 8 or less are shown in FIG. 9. Chroma blocks with larger widths and heights are possible but are not shown in FIG. 9 for ease of reference.

[0103] The set of forbidden transform sizes 910 includes transform block sizes 2x2, 2x4, and 4x2, all of which have regions less than 16 samples. In other words, in the example of FIG. 9, a minimum transform size of 16 (16) chroma samples results from the operation of the described configuration, particularly for intra-prediction CB. Instances of forbidden transform sizes 910 are avoided by determining the partitioning options, as described with reference to FIG. 10. Residual coefficients in the transform are scanned in a two-layer approach, where the transform is divided into "sub-blocks" (or "coefficient groups"). The scan is performed along a scan path from the last significant (non-zero) coefficient toward the DC (top-left) coefficient. The scan path is defined as the progression within each sub-block ("lower layer") and from one sub-block to the next ("upper layer"). In the set 900, the 8x2TB 920 uses 8x2 sub-blocks, i.e., sub-blocks containing 16 residual coefficients. The 2x8TB922 uses 2x8 sub-blocks, i.e. it also contains 16 residual coefficients.

[0104] A TB whose width or height is 2 and whose other dimension is a multiple of 8 uses multiple 2x8 or 8x2 sub-blocks. Thus, in some examples, a chroma block having a width of 2 samples is encoded using sub-block division, each of size 2x8 samples, and a chroma block having a height of 2 samples is encoded in some examples using sub-block division, each of size 8x2 samples. For example, a 16x2 TB 916 has two 8x2 sub-blocks, each scanned as shown for TB 920. The scanning progresses from one sub-block to the next as shown in sub-block progression 917.

[0105] The 2x32TB (not shown in Figure 9) uses four 2x8 sub-blocks arranged as a 1x4 array. The residual coefficients within each sub-block are scanned as shown for the 2x8TB 922, proceeding from the lowest to the highest sub-block of the 1x4 array.

[0106] The larger the TB, the similar scanning progression is followed. For all TBs whose width and height are equal to or greater than 4, a 4x4 subblock scan is used. For example, a 4x8 TB 923 uses a 4x4 subblock scan 924, proceeding from the lower subblock to the upper subblock. A 4x4 TB 925 can be scanned in a similar manner. An 8x8 TB 929 uses a progression 930 for four 4x4 subblocks. In all cases, the scanning within a subblock and from subblock to subblock follows a backward diagonal scan, i.e., the scan proceeds backward from the "last" significant residual coefficient toward the top-left residual coefficient of the TB. Figure 9 also shows the scan order for, for example, an 8x4 TB 932, a 16x4 TB 934, and a 16x8 TB 936. Furthermore, depending on the location of the last significant coefficient along the scan path, only the portion of the subblock containing the last significant residual coefficient needs to be scanned, from the last significant coefficient position of the subblock back to the top-left residual coefficient. Further sub-blocks along the scan path in the forward direction (i.e., closer to the bottom right of the block) do not need to be scanned. The set 900, and in particular the forbidden transform sizes 910, imposes restrictions on the ability to divide regions (or nodes) of the chroma coding tree into sub-regions (or sub-nodes), as described with reference to FIG.

[0107] In a VVC system using 2x2, 2x4, and 4x2 TBs (set of TBs 910), 2x2 sub-blocks may be used for TBs with a width and / or height of two samples. As mentioned above, the use of TBs 910 increases the throughput constraints in the intra-reconstruction feedback dependency loop. Furthermore, the use of sub-blocks with only four coefficients increases the difficulty of parsing residual coefficients at higher throughputs. In particular, for each sub-block, a "significance map" indicates the significance of each residual coefficient contained therein. Encoding a one-value significance flag establishes the magnitude of the residual coefficient as at least one, while encoding a zero-value flag establishes the magnitude of the residual coefficient as zero. The magnitude and sign of the residual coefficient (from one forward) are coded only for "significant" residual coefficients. The significance bit is not coded; the magnitude (from zero) is always coded for the DC coefficient. High-throughput encoders and decoders may need to code or decode multiple significance map bins per clock cycle to maintain real-time operation. The difficulty of multi-bin encoding and decoding per cycle increases when there are more inter-bin dependencies, e.g., when smaller sub-block sizes are used. In system 100, the sub-block size is 16 (with the exception of the sub-block containing the last significant coefficient), regardless of the block size.

[0108] Figure 10 shows a set of rules 1000 for generating a list of allowed partitions in a chroma coding tree. Other frames may allow a mix of inter- and intra-predicted blocks. While the full set of available partitions of the coding tree was described with reference to Figure 6, limitations on available transform sizes constrain the specific partitioning options for a given region size. As described below, the partitioning options for each chroma channel are determined according to the dimensions of the region of the corresponding coding tree unit.

[0109] Rules 1020 for chroma regions indicate the allowed division of different regions. The allowed division of rules 1020 is expressed in units of luma samples, even though different chroma formats may be used, so the chroma channels are considered.

[0110] As nodes in the coding tree are traversed, a list of allowed splits for chroma is obtained by checking the availability of a set of split options with the region size of the coding tree. Split options that result in a region that can potentially be coded using CB are added to the list of allowed splits. For a region to be coded using CB, the region size must allow coding with an integer number of transforms of a specific size from set 900. The specific size is chosen to be the largest size that does not exceed the region size (considering both width and height). Thus, for smaller regions, a single transform is used. If the region size exceeds the size of the largest available transform, the largest available transform is tiled to occupy the entire region.

[0111] Considering a node in the coding tree with a given region (represented in luma samples), the ability to perform a given type of split is determined according to the split type and the chroma region area. As shown in Figure 10, split options are tested against the region size to determine whether the split option results in a subregion of a prohibited size. A split option that results in a subregion of a permitted size is considered a permitted chroma split 1070.

[0112] For example, in QT mode (corresponding to decision 610 in FIG. 6), as shown as rule 1021a for chroma regions, if the region is sized 8×8 in 4:2:0 format or 8×8 in 4:2:2 format, quadtree splitting is not allowed because the split would result in a 2×2 or 2×4 transform size for the chroma channels, respectively. The allowable region sizes are indicated by arrow 1021. Similarly, other allowable splits for chroma rule set 1020 are indicated by arrows 1022, 1023, 1024, 1025, and 1026 and are described below in connection with FIGS. 13 and 14. Arrows 1021, 1022, 1023, 1024, 1025, and 1026 each refer to allowed chroma split list 1070.

[0113] The region size of the chroma channels is described with respect to the luma sample grid. For example, an 8x4 region corresponds to a 4x2 transformation of the chroma channels when a 4:2:0 chroma format is used. When a 4:2:2 chroma format is used, an 8x4 region corresponds to a 4x4 transformation of the chroma. When a 4:4:4 chroma format is used, the chroma is not subsampled with respect to the luma, and therefore the transformation size in the chroma corresponds to the region size.

[0114] Allowable splitting options are further described in connection with Figures 13 and 14 below.

[0115] 11 shows a method 1100 for encoding a coding tree of an image frame into a video bitstream. The method 1100 may be implemented by an apparatus such as an configured FPGA, ASIC, or ASSP. Furthermore, the method 1100 may be performed by the video decoder 114 under execution by the processor 205. Accordingly, the method 1100 may be stored in a computer-readable storage medium and / or the memory 206. The method 1100 begins with step 1105 of determining a chroma format.

[0116] In a determine chroma format step 1105, the processor 205 determines the chroma format of the frame data 113 as one of a 4:2:0 chroma format or a 4:2:2 chroma format. The chroma format is a property of the frame data and does not change during operation of the method 1100. From step 1105, the method 1100 continues to a step 1110 of dividing the frame into CTUs under the control of the processor 205.

[0117] In a step 1110 of dividing a frame into CTUs, the block partitioner 310, under execution of the processor 205, divides the current frame of frame data 113 into an array of CTUs. The coding process begins across the CTUs resulting from the division. Control within the processor passes from step 1110 to a step 1120 of determining a coding tree.

[0118] In step 1120 of determining a coding tree, the video encoder 114, under the execution of the processor 205, combines and tests various prediction modes and partitioning options to arrive at a coding tree for the CTU. The video encoder 114 also derives a prediction mode and residual coefficients for each CU in the coding tree for the CTU. Generally, Lagrangian optimization is performed to select the optimal coding tree and CU for the CTU. When evaluating the use of inter prediction, a motion vector is selected from a set of candidate motion vectors. The candidate motion vectors are generated according to a search pattern. When evaluating the distortion test of the fetched reference block for the candidate motion vector, the application of prohibited chroma splitting in the coding tree is taken into account. If splitting is prohibited in chroma but allowed in luma, the resulting luma CB can use inter prediction. Because motion compensation is applied only to the luma channel, the distortion calculation takes luma distortion into account, not chroma distortion. If chroma splitting is prohibited, chroma distortion is not considered because motion compensation is not performed in the chroma channel. For chroma, the distortion resulting from the considered intra prediction mode and the coded chroma TB (if any) is taken into account. When both luma and chroma are considered, the inter prediction search may first select a motion vector based on the luma distortion and then "refine" the motion vector by also considering the chroma distortion. The refinement generally takes into account small variations in the motion vector values, such as sub-pixel displacements. If chroma splitting is prohibited and inter prediction evaluation is performed on small luma blocks, chroma refinement is not required. Control within processor 205 proceeds from step 1120 to step 1130, where the coding tree is coded.

[0119] In a coding tree encoding step 1130, video encoder 114, under execution of processor 205, performs a method 1300 described in relation to FIG. 13 to encode the coding tree of the current CTU into bitstream 115. Step 1130 is performed to encode the current CTU into the bitstream. Control in processor 205 passes from step 1130 to a last CTU test step 1140.

[0120] In last CTU test step 1140, processor 205 tests whether the current CTU is the last CTU in the slice or frame. If not ("NO" at step 1140), video encoder 114 proceeds to the next CTU in the frame, and control in processor 205 returns from step 1140 to step 1120 to continue processing the remaining CTUs in the frame. If the CTU is the last CTU in the frame or slice, step 1140 returns "YES" and method 1100 ends. Method 1100 results in the entire image frame being encoded into a bitstream as a sequence of CTUs.

[0121] 12 shows a method 1200 for decoding a coding tree of an image frame from a video bitstream. The method 1200 may be implemented by an apparatus such as an configured FPGA, ASIC, or ASSP. Furthermore, the method 1200 may be performed by the video decoder 134 under execution of the processor 205. Accordingly, the method 1200 may be stored in a computer-readable storage medium and / or the memory 206. The method 1200 begins with step 1205 of determining a chroma format.

[0122] In step 1205 of determining a chroma format, the processor 205 determines the chroma format of the frame data 113 as one of a 4:2:0 chroma format or a 4:2:2 chroma format. The chroma format is a property of the frame data and does not change during the operation of the method 1200. The video decoder 134 may determine the chroma format by a profile of the bitstream 133. The profile defines a set of encoding tools that can be used by a particular bitstream 133 and may constrain the chroma format to a particular value, such as 4:2:0. The profile is determined, for example, by decoding a “profile_idc” syntax element from the bitstream 133 or by decoding one or more constraint flags from the bitstream 133, each of which constrains the use of a particular tool in the bitstream 133. If the chroma format is not fully specified by the profile, further syntax, such as “chroma_format_idc,” may be decoded to determine the chroma format. The method 1200 continues under execution of the processor 205 from step 1205 to step 1210 where the frame is divided into CTUs.

[0123] In step 1210, video decoder 134, under execution of processor 205, determines a division of the current frame of frame data 133 to be decoded into an array of CTUs. Decoding progress begins across the CTUs resulting from the determined division. Control within the processor passes from step 1210 to step 1220, where coding tree decoding is performed.

[0124] In a decoding coding tree step 1220, video decoder 134, under execution of processor 205, performs method 1400 on the current CTU to decode the coding tree of the current CTU from bitstream 133. The current CTU is a selected one of the CTUs resulting from execution of step 1210. Control in processor 205 passes from step 1220 to a last CTU test step 1240.

[0125] In last CTU test step 1240, processor 205 tests whether the current CTU is the last CTU in the slice or frame. If not ("NO" at step 1240), video decoder 134 proceeds to the next CTU in the frame, and control in processor 205 returns from step 1240 to step 1220 to continue decoding CTUs from the bitstream. If the CTU is the last CTU in the frame or slice, step 1240 returns "YES" and method 1300 ends.

[0126] FIG. 13 illustrates a method 1300 for encoding a coding tree of an image frame into a video bitstream. The method 1300 may be implemented by a device such as a configured FPGA, ASIC, or ASSP. Furthermore, the method 1300 may be executed by the video encoder 114 under the execution of the processor 205. Accordingly, the method 1300 may be stored in a computer-readable storage medium and / or memory 206. The method 1300 encodes blocks into the bitstream 115 such that each block is minimally sized. The described configuration uses samples of a predetermined minimum size. The minimum size used in the described example is 16 samples, which is preferable from the perspective of some hardware and software implementations. However, different minimum sizes can nevertheless be used. For example, a processing granularity of 32 or 64, with a corresponding minimum block area of ​​32 or 64 samples, respectively, is possible. A coding block with a minimal area is advantageous for implementation feasibility in both hardware and software implementations. For software implementations, a minimum region of 16 samples aligns with typical single instruction multiple data (SIMD) instruction sets such as AVX-2 and SSE4. The method 1300, which is first invoked at the root node of the coding tree for the current CTU, begins with step 1310, which encodes the partitioning mode.

[0127] In a step 1310 of encoding a partition mode, the entropy encoder 338, under the execution of the processor 205, encodes the partition mode at the current node of the coding tree into the bitstream 115. The partition mode is one of the partitions described with reference to FIG. 5, and the step of encoding a partition mode allows for the encoding of only possible partitions. For example, quadtree partition 512 is only possible at the root node of the coding tree or under other quadtree partitions in the coding tree. As shown in connection with set 910, partitions that result in luma CBs with widths or heights of less than four samples are prohibited. Other constraints, for example, on the maximum depth of bipartitions and / or tripartitions, may also be valid, based on rule set 1010. Control in the processor 205 passes from step 1310 to a no-split test step 1320.

[0128] In no split test step 1320, processor 205 tests whether the current split is "no split" (i.e., 510). If the current split is no split 510 ("YES" at step 1320), processor 205 control passes from step 1320 to step 1330, where the CU is encoded. Otherwise, if the current split is not 510 ("NO" at step 1320), processor 205 control passes to chroma split inhibit test step 1340.

[0129] In a CU encoding step 1330, the entropy encoder 338, under execution of the processor 205, encodes the prediction mode of the CU and the residual of the CU into the bitstream 115. As step 1330 is reached for each leaf node of the coding tree, the method 1300 ends in a completion step 1330 and returns to the parent call in the coding tree traversal. Once all nodes of the coding tree have been traversed, the entire CTU is encoded into the bitstream 115 and control returns to the method 1100 to proceed to the next CTU in the image frame.

[0130] In chroma split prohibition test step 1340, processor 205 determines whether splitting for the current node in the coding tree is allowed to be applied to the chroma channels, as in step 1310, according to the chroma region 1020 splitting rule set of FIG. 10. If the current node in the coding tree covers a luma region of 128 luma samples (32x4 or 4x32 or 16x8 or 8x16), then a third split in the corresponding chroma region (16x2, 2x16, 8x4, or 4x8 chroma samples, respectively) is prohibited as indicated in rule set 1020. If a third split is allowed, the resulting block size will include a prohibited block size (e.g., 2x4 or 4x2). If the current node in the coding tree covers a luma region of 64 luma samples, then two-way splitting, three-way splitting, and quadtree splitting are prohibited as indicated in rule set 1020. Performing bisection, trisection, and quadtree division on a luma region of 64 luma samples results in prohibited chroma block sizes (2x2, 2x4, 4x2). If the division is not prohibited (i.e., the division is a permitted chroma division in list 1070), step 1340 returns "NO" and control of processor 205 passes from step 1340 to step 1350, where luma and chroma division is performed. Otherwise, if division is prohibited ("YES" at 1340), control of processor 205 passes to step 13100, where luma division is performed.

[0131] In step 1350 of performing luma and chroma splitting, processor 205 applies the splitting to divide the current region associated with the current node of the coding tree into sub-regions associated with sub-nodes of the coding tree. The splitting is applied according to the description of Figures 5 and 6. Control within processor 205 passes to step 1360 of selecting a region from step 1350.

[0132] In a select region step 1360, the processor selects one of the subregions resulting from step 1350. The subregion is selected according to a Z-order scan of the region. The selection proceeds through the subregions in subsequent iterations of step 1360. Control in processor 205 passes from step 1360 to a step 1370 of encoding the coding tree.

[0133] In a coding tree encoding step 1370, processor 205 recursively invokes method 1300 on the selected regions resulting from step 1360. Step 1370 further operates to encode the luma and chroma blocks, and associated prediction modes and residual coefficients, for each region in the bitstream. Control in processor 205 passes from step 1370 to a last region test step 1380.

[0134] In last region test step 1380, processor 205 tests whether the selected region selected in step 1360 is the last one of the regions resulting from the split mode partitioning as performed in step 1350. If the region is not the last region ("NO" at step 1380), control in processor 205 proceeds from step 1380 to step 1360 to continue stepping through the regions of the partition; otherwise, step 1380 returns "YES," method 1300 ends, and control in processor 205 proceeds to the parent call of method 1300.

[0135] In step 13100 of performing luma splitting, the splitting mode as coded in step 1310 is performed only on the luma channel by processor 205. As a result, the current node of the coding tree is split into multiple luma CBs according to the splitting mode. Pairs of chroma CBs are generated, i.e., only one chroma CB per chroma channel. Each resulting luma CB overlaps (is juxtaposed) with the pair of chroma CBs and the collectively resulting luma CB. The collective luma CB exactly covers the area of ​​the pair of chroma CBs. Also, the minimum area of ​​each luma CB and chroma CB is a minimum size, for example, 16 samples.

[0136] Steps 13100 and 1350 operate to determine the size of the chroma coding blocks for the chroma channels Cb and Cr, respectively. In step 1350, the chroma coding block size for the chroma channel is determined based on the split mode determined in step 1310. In step 13100, the chroma coding block size for the chroma channel is determined based on a predetermined minimum chroma block size. As described above, step 1350 is performed based on the chroma splits that are prohibited for the coding tree unit. As shown in rule set 1020 of FIG. 10, the allowable splits, and therefore the size of the chroma coding blocks, are determined based on the chroma format determined in step 1105.

[0137] Control within processor 205 passes from step 13100 to step 13110 where luma CB is selected.

[0138] In step 13110 of selecting a luma CB, the processor 205 selects the next luma CB of the CB obtained from step 13100. The method 13100 first selects the first CB, i.e., the top-left luma CB of the CBs resulting from the luma split. On subsequent invocations of step 13110, each "next" luma CB is selected according to a Z-order scan over the luma CB obtained from step 13100. Control in the processor 205 passes from step 13110 to step 13120 of encoding the luma CB.

[0139] In step 13120 of encoding the luma CB, the entropy encoder 338, under the execution of the processor 205, encodes the selected luma CB into the bitstream 115. Generally, a prediction mode and residual coefficients are encoded for the selected luma CB. The prediction mode encoded for the luma CB can use inter prediction or intra prediction. For example, "cu_skip_flag" is encoded to indicate the use of inter prediction without residual, otherwise, "pred_mode_flag" and optionally "pred_mode_ibc_flag" are encoded to indicate the use of intra prediction, inter prediction, or intra-block copying with optional residual coefficients, respectively. If residual may be present, the "cu_cbf" flag indicates the presence of at least one significant (non-zero) residual coefficient in any TB of the CB. If a CB is indicated to use inter prediction, the associated motion vector is applicable only to the luma CB. That is, the motion vector is not also applied to generate a PB associated with a partially collocated chroma CB. If the CB is instructed to use intra-block copying, the associated block vector is associated only with the luma CB and not with the partially collocated chroma CB. Control in processor 205 passes from step 13120 to last luma CB test step 13130.

[0140] In last luma CB test step 13130, processor 205 tests whether the luma CB selected in step 13110 is the last luma CB according to the Z-order iteration of luma CBs of the division performed in step 13100. If the selected luma CB is not the last one (“NO” in step 13130), control of processor 205 passes from step 13130 ​​to step 13120. Otherwise, step 13130 ​​returns “YES” and control of processor 205 passes to step 13140, which determines the chroma intra prediction mode.

[0141] In determine chroma intra-prediction mode 13140, the video encoder 114, under the execution of the processor 205, determines an intra-prediction mode for the pair of chroma CBs located together with the luma CB of step 13100. Step 13140 effectively determines that the chroma block is coded using intra-prediction. A determination is made whether the area occupied by the chroma CB is further divided into multiple luma CBs in the luma channel. The size of the chroma block for the channel is a predetermined minimum value (e.g., 16 samples) determined by the operation of step 1350. In step 13120, the intra-prediction mode for the pair of chroma CBs is determined, even if the corresponding luma CB was coded using inter-prediction. In one configuration, a single prediction mode, such as DC intra-prediction, is applied to each chroma CB. The use of a single prediction mode allows the mode to be determined by prohibiting chroma splitting (a "YES" result in step 1340) and does not require additional searching to determine which one of multiple possible modes should be used. Furthermore, bitstream 115 does not require additional signaling in this case, i.e., it does not need to encode an additional "intra_chroma_pred_mode" syntax element. However, when the configuration prohibits chroma splitting ("YES" in step 1340), higher compression performance can be achieved by signaling one of several possible intra-prediction modes by including the "intra_chroma_pred_mode" syntax element in bitstream 115. Video encoder 114 determines which intra-prediction mode to use. The intra-prediction mode is generally determined according to considerations of coding cost compared to distortion. However, higher compression performance is generally achieved compared to using a single intra-prediction mode for such chroma CB. Control in processor 205 proceeds from step 13140 to step 13150, where chroma CB is encoded.

[0142] In step 13150 of encoding the chroma CB, the entropy encoder 338, under execution of the processor 205, encodes the intra prediction mode of the chroma CB into the bitstream 115 using the “intra_chroma_pred_mode” syntax element when multiple intra prediction modes are available. When one intra prediction mode, for example, DC intra prediction, is available, “intra_chroma_pred_mode” is not coded into the bitstream 115. Available intra prediction modes for chroma intra prediction can include DC, planar, and the following angular prediction modes: horizontal, vertical, and diagonal-top-right. Available intra prediction modes can also include “direct mode” (DM_CHROMA), whereby the chroma intra prediction mode is obtained from the co-located luma CB, typically from the bottom and rightmost of the luma CB resulting from step 13100. If “cross-component linear model” intra prediction is available, the chroma CB can be predicted from samples from the luma CB. 14, the residual coefficients of chroma TB associated with chroma CB may also be encoded into bitstream 115. Once step 13150 is performed by processor 205, method 1300 ends and control within processor 205 returns to the parent invocation of method 1300.

[0143] FIG. 14 illustrates a method 1400 for decoding a coding tree of an image frame from a video bitstream, performed at step 1220 of method 1200. Method 1400 may be implemented by a device such as a configured FPGA, ASIC, or ASSP. Furthermore, method 1400 may be executed by video decoder 134 under execution of processor 205. As such, method 1400 may be stored in a computer-readable storage medium and / or memory 206. Method 1400 decodes blocks from bitstream 133 such that each block is no smaller than a minimum area, such as 16 samples, which is advantageous for implementation feasibility in both hardware and software. In the case of software, the 16-sample minimum area aligns with typical single instruction, multiple data (SIMD) instruction sets such as AVX-2 and SSE4. Method 1400, which is initially invoked at the root node of the coding tree of the current CTU, begins with step 1410, which decodes the partition mode.

[0144] In step 1410 of decoding the partition mode, the entropy decoder 420, under the execution of the processor 205, decodes the partition mode at the current node of the coding tree into the bitstream 133. The partition mode is one of the partitions described with reference to FIG. 5, and the method of encoding the partition mode is permitted even if partitions are prohibited in the chroma channels, i.e., only allows the encoding of partitions permitted in the luma channel. For example, quadtree partitioning 512 is only permitted at the root node of the coding tree or below other quadtree partitionings in the coding tree. Any partition that results in a luma CB with a width or height of less than four samples is prohibited. Therefore, the minimum luma CB size is 16 samples. Other constraints on the maximum depth of bipartitions and / or tripartitions may also be valid. Control in the processor 205 passes from step 1410 to a no-split test step 1420.

[0145] In no split test step 1420, processor 205 tests whether the current split is "no split" (i.e., 510). If the current split is no split 510 ("YES" at 1420), processor 205 control passes from step 1420 to step 1430, where the CU is decoded. Otherwise, step 1420 returns "NO" and processor 205 control passes to chroma split inhibit test step 1440.

[0146] In step 1430 of decoding a CU, the entropy decoder 420, under execution of the processor 205, decodes the prediction mode of the CU and the residual coefficients of the CU from the bitstream 115. Step 1430 operates to decode the coding unit using the residual coefficients and prediction mode determined from the bitstream by the entropy decoder 420. As step 1430 is reached at each leaf node of the coding tree, method 1400 terminates upon completion of step 1430 and returns to the parent call in the coding tree search. Once all nodes of the coding tree have been traversed, the entire CTU is decoded from the bitstream 133 and control returns to method 1200 to proceed to the next CTU in the image frame.

[0147] In chroma split prohibition test step 1440, processor 205 determines whether splitting for the current node in the coding tree is permitted to be applied to the chroma channels, as in step 1410, according to the chroma region 1020 splitting rule set of FIG. 10. Step 1440 determines whether split testing is prohibited, similar to step 1340 of method 1300. The operation of step 1440 prevents the occurrence of prohibited block sizes. If the chroma region is already at a minimum size, e.g., 16 chroma samples, further splitting of any type is not permitted because the resulting region is smaller than the allowed minimum. If the chroma region size is 32 samples and the corresponding split is a third (whether horizontally or vertically thirds), further splitting is also not permitted to avoid chroma blocks of region 8 chroma samples. If splitting is not prohibited (i.e., splitting is permitted), step 1450 returns "NO," and control of processor 205 passes from step 1440 to step 1450, which performs luma and chroma splitting. Otherwise, if splitting is prohibited (“YES” in step 1450), control of processor 205 proceeds to step 14100 where a chrominance intra-prediction mode is determined.

[0148] In step 1450 of performing luma and chroma splitting, processor 205 applies splitting to divide the current region associated with the current node of the coding tree into sub-regions associated with sub-nodes of the coding tree. The splitting is applied as described in connection with Figures 5 and 6.

[0149] Steps 14100 and 1450 operate to determine the size of the chroma coding blocks for the chroma channels Cb and Cr, respectively. In step 1450, the chroma coding block size of the chroma channel is determined based on the split mode decoded in step 1410. In step 14100, the chroma coding block size of the chroma channel is determined based on a predetermined minimum chroma block size. As described above, step 1450 is performed based on the prohibited chroma splits for the coding tree unit, which corresponds to a minimum chroma CB size of 16 (and 32 in the case of a three-way split of the luma region 128 samples). As shown in rule set 1020 of FIG. 10, the allowable splits, and therefore the size of the chroma coding blocks, are determined based on the chroma format determined in step 1205.

[0150] Control within processor 205 passes from step 1450 to a region selection step 1460 .

[0151] In a region selection step 1460, processor 205 selects one of the subregions resulting from step 1450, following a Z-order scan of the region. Step 1460 handles the progressive selection through the subregions in subsequent iterations. Control within processor 205 passes from step 1460 to step 1470, which decodes the coding tree.

[0152] In a decoding coding tree step 1470, processor 205 recursively invokes method 1400 on the selected regions resulting from the operation of step 1460. Step 1470 further operates to decode each region of the coding tree using the residual coefficients and prediction mode determined from the bitstream. Control in processor 205 passes from step 1470 to a last region test step 1480.

[0153] In a last region test step 1480, processor 205 tests whether the selected region is the last one of the regions resulting from the split mode partitioning performed in step 1450, as preselected in the last iteration of step 1460. If the region is not the last region ("NO" at step 1480), control of processor 205 transfers from step 1480 to step 1460 to continue stepping through the regions of the partition. Otherwise, step 1480 returns "YES," method 1400 ends, and control of processor 205 transfers to the parent call of method 1400.

[0154] In step 14100 of performing luma splitting, the split mode as coded in step 1410 is performed on the luma channel by processor 205 only. As a result, the current node of the coding tree is split into multiple luma CBs according to the split mode. Step 14100 operates to generate pairs of chroma CBs, i.e., only one chroma CB per chroma channel. Each resulting luma CB partially overlaps (is at least partially co-located with) its pair of chroma CBs, and collectively, the luma CBs completely overlap their pair of chroma CBs. Also, the minimum area of ​​each luma CB and chroma CB is 16 samples. Control in processor 205 proceeds from step 14100 to step 14110, where a luma CB is selected.

[0155] In step 14110 of selecting a luma CB, processor 205 selects the next luma CB of the CB obtained from step 14100. The selection of the next luma CB starts from the first CB, i.e., the top-left luma CB of the CB resulting from the luma split. On subsequent invocations of step 14110, each "next" luma CB is selected according to a Z-order scan over the luma CB obtained from step 14100. Control in processor 205 passes from step 14110 to step 14120 of decoding the luma CB.

[0156] In step 14120 of decoding the luma CB, the entropy decoder 420, under the execution of the processor 205, decodes the selected luma CB into the bitstream 115. Generally, a prediction mode and a residual are decoded for the selected luma CB. For example, "cu_skip_flag" is decoded to indicate the use of inter prediction without residual, otherwise "pred_mode_flag" and optionally "pred_mode_ibc_flag" are decoded to indicate the use of intra prediction, inter prediction, or intra-block copying with optional residual coefficients, respectively. If residual may be present, the "cu_cbf" flag indicates the presence of at least one significant (non-zero) residual coefficient in any TB of the CB. If a CB is indicated to use inter prediction, the associated motion vector is applicable only to the luma CB; i.e., the motion vector is not also applied to generate a PB associated with a partially collocated chroma CB. When a CB is instructed to use intra-block copying, the associated block vector is associated only with the luma CB and not with the partially collocated chroma CB. Control in processor 205 passes from step 14120 to step 14130, which tests the last luma CB.

[0157] In step 14130 of testing the last luma CB, the processor 205 tests whether the luma CB selected in step 14110 is the last luma CB according to the Z-order iteration of the luma CBs of the division performed in step 14100. If the selected luma CB is not the last one, control in the processor 205 passes from step 14130 to step 14110. Otherwise, control in the processor 205 passes to step 14140 of determining the chroma intra prediction mode.

[0158] In determine chroma intra-prediction mode 14140, video decoder 134, under the execution of processor 205, determines an intra-prediction mode for the pair of chroma CBs located together with the luma CB of step 14100. Step 14140 effectively determines that a chroma block has been coded using intra-prediction and should therefore be decoded using intra-prediction if the chroma block is the result of stopping splitting the coding tree for chroma while splitting the coding tree for luma is being performed, as determined by the operation of step 1440. In step 14120, an intra-prediction mode for the pair of chroma CBs is determined even if the corresponding luma CB was decoded using inter-prediction. In one configuration, a single prediction mode, such as DC intra-prediction, is applied to each chroma CB. The use of a single prediction mode allows the mode to be determined by prohibiting chroma splitting (a "YES" result in step 1440) and does not require additional searching to determine which one of multiple possible modes should be used. Furthermore, bitstream 134 does not require additional signaling in this case, i.e., it does not need to encode an additional "intra_chroma_pred_mode" syntax element. However, the configuration can achieve higher compression performance by signaling one intra-prediction mode out of several possible intra-prediction modes by including the "intra_chroma_pred_mode" syntax element in bitstream 134 when chroma split is prohibited ("YES" in step 1440). Video decoder 134 needs to determine the intra-prediction mode to be used in order to decode the "intra_chroma_pred_mode" syntax element from bitstream 134 using entropy decoder 420. Control in processor 205 proceeds from step 14140 to step 14150, where chroma CB is decoded.

[0159] In step 14150 of decoding the chroma CB, the entropy decoder 420, under the execution of the processor 205, determines the intra-prediction mode of the chroma CB from the bitstream 420, generally according to the decoded "intra_chroma_pred_mode" syntax element. The decoding of "intra_chroma_pred_mode" is performed when multiple intra-prediction modes are available. If only one intra-prediction mode, for example, DC intra-prediction, is available, the mode is inferred from the bitstream 133 without decoding additional syntax elements. The available intra-prediction modes for chroma intra-prediction may include DC, planar, and the following angular prediction modes: horizontal, vertical, and diagonal-top-right. The available intra-prediction modes may also include "direct mode" (DM_CHROMA), whereby the chroma intra-prediction mode is obtained from the luma CB generally aligned at the lowest and rightmost positions of the luma CB resulting from step 14100. If "cross-component linear model" intra prediction is available, the chroma CBs may be predicted from samples from the luma CBs. For a pair of chroma CBs, the "cu_cbf" flag signals the presence of at least one significant residual coefficient in any one of the pair of chroma CBs. If at least one significant residual coefficient exists in any one of the pair of chroma CBs, "tu_cbf_cb" and "tu_cbf_cr" signal the presence of at least one significant coefficient in the chroma CBs of the Cb and Cr channels, respectively. For chroma CBs with at least one significant residual coefficient, the "residual_coding" sequence of syntax elements is decoded to determine the residual coefficients of the respective chroma CBs. The residual coding syntax encodes the residual coefficients as a sequence of values ​​that populate the transform block from the last significant coefficient position to the top-left ("DC") coefficient position, following a backward diagonal scan. A back diagonal scan performs a scan of the transform block as a sequence of "sub-blocks" (or "coefficient groups"), typically of size 4x4, but also possible sizes 2x2, 2x4, 2x8, 8x2, and 4x2.The scanning within each coefficient group is diagonal backward, and the scanning from one sub-block to the next is also diagonal backward. Once step 14150 is performed by processor 205, method 1400 ends and control within processor 205 returns to the parent invocation of method 1400.

[0160] The coding tree approach of methods 1300 and 1400 maintains a minimum block area of ​​16 samples for 4:2:0 chroma format video data, facilitating high-throughput implementation in both software and hardware. Furthermore, restricting inter-prediction on luma CBs to small CB sizes reduces this worst-case memory bandwidth for motion compensation by avoiding the need to also fetch samples to generate motion-compensated chroma CBs. In particular, when the minimum chroma CB size is 2x2 and additional samples are required to provide filter support for subsampled interpolation of the chroma CB, a substantial increase in memory bandwidth is observed compared to only performing inter-prediction in the luma channel for small block sizes. The coding gain of motion compensation is substantially manifested in the luma channel, and therefore, omitting even small blocks from being motion-compensated achieves a memory bandwidth reduction with a relatively small coding performance impact. Furthermore, the memory bandwidth reduction contributes to the feasibility of performing motion compensation on 4x4 luma CBs and achieving the resulting coding gain.

[0161] In one configuration of the video encoder 114 and the video decoder 134, two or more luma splits can occur in the coding tree from the point where the chroma splitting of the coding tree ends. For example, an 8x16 luma region is not split in the chroma channels, resulting in a pair of 4x8 chroma CBs. In the luma channel, the 8x16 luma region is first split horizontally by three, and then one of the resulting luma CBs is further split. For example, the resulting 8x4 luma CB is bisected vertically into two 4x4 luma CBs. Configurations with two or more luma splits in the coding tree from the point where the chroma splitting of the coding tree ends restart the methods 1300 and 1400 in the video encoder 114 and the video decoder 134, respectively, in the chroma split prohibited region, with the modification that no more chroma CBs are required in subsequent invocations. In calls to methods 1300 and 1400 where a pair of chroma CBs is created, the entire chroma region is covered by the created chroma CBs, so recursive calls to methods 1300 and 1400 do not need to create additional chroma CBs.

[0162] Figure 15 shows a set 1500 of transform block partitions for an intra-prediction coding unit. A luma CB may be partitioned into one luma TB of the same size ("ISP_NO_SPLIT"). A luma CB of size 4x4 may have an area of ​​16 samples and not be further partitioned, resulting in one luma TB of size 4x4. A luma CB with an area of ​​32 samples may be partitioned into two partitions. For example, an 8x4 luma CB 1510 may be partitioned horizontally ("ISP_HOR_SPLIT") into two 8x2 luma TBs 1520 or vertically ("ISP_VER_SPLIT") into two 4x4 luma TBs 1530. If the luma CB 1510 is a 4x8 luma CB, the block may be partitioned horizontally into two 4x4 luma TBs at 1520 or vertically into two 2x8 luma TBs at 1530.

[0163] A luma CB of a region of 64 samples or more is divided into four regions. A luma CB 1550 having a larger region of 64 samples, with a width W and a height H, may be divided horizontally into four luma TBs 1560 of size Wx(H / 4) or vertically into four (W / 4)xH luma TBs. As shown in set 1500, dividing the luma CB into multiple regions results in smaller and smaller luma TBs. Intra prediction is performed for each luma TB to generate a PB, and an intra reconstruction process is performed within the luma CB from one region to the next.

[0164] FIG. 16 illustrates a method 1600 for encoding coding units of an image frame into a video bitstream 115. The method 1600 may be implemented by an apparatus such as a configured FPGA, ASIC, or ASSP. Furthermore, the method 1600 may be executed by the video encoder 114 under execution of the processor 205. As such, the method 1600 may be stored in a computer-readable storage medium and / or the memory 206. The method 1600 encodes blocks into the bitstream 115 such that coefficient group size is determined solely based on the transform block size, without further distinguishing between the luma and chroma channels. Because entropy coding is a critical feedback loop in the video encoder 114, it is advantageous to reduce the memory accesses or operations required for coefficient group size determination. The method 1600, which is invoked for each coding unit in the coding tree, i.e., invoked in step 1330 of FIG. 13, begins with step 1610, which encodes pred_mode. As described above, step 1330 is performed if step 1320 determines that the current split is no split 510 .

[0165] In a encode pred mode step 1610, the entropy encoder 338, under the execution of the processor 205, encodes the prediction mode of the CU into the bitstream 115. Control in the processor 205 passes from step 1610 to an intra prediction test step 1620.

[0166] In intra prediction test step 1620, processor 205 tests the prediction mode of the CU. If the prediction mode is intra prediction ("YES" at step 1620), processor 205 control passes from step 1620 to step 1650, where the processor encodes the intra sub-partition mode. Otherwise, if the prediction mode is not intra prediction ("No" at step 1620), processor 205 control passes from step 1620 to step 1630, where the processor encodes the merge flag and index.

[0167] In step 1630 of encoding a merge flag and index, the entropy encoder 338, under the execution of the processor 205, encodes a merge flag into the bitstream 115, which signals the use (or non-use) of a "merge mode" for inter prediction. The merge mode causes the motion vector of a CU to be obtained from a spatially (or temporally) neighboring block among a set of candidates of spatially (or temporally) neighboring blocks. When the merge mode is used, one candidate is selected with a corresponding "merge index." The merge index is coded in the bitstream 115 together with the merge flag. When "motion vector prediction" is used, a similar coding is performed, whereby one of several possible candidate motion vectors is signaled as a predictor using a flag. Control in the processor 205 passes from step 1630 to step 1640 of coding a motion vector delta.

[0168] In step 1640 of encoding a motion vector delta, the entropy encoder 338, under execution of the processor 205, encodes the motion vector delta into the bitstream 115. Step 1640 is performed if motion vector prediction is used for the CU. The motion vector delta specifies the delta between the motion vector predictor encoded in step 1630 and the motion vector used for motion compensation. Control in the processor 205 passes from step 1640 to a coded residual test step 1660. If motion vector prediction is not used for the CU, step 1640 is not performed and the method 1600 proceeds directly to step 1660.

[0169] In step 1650 of encoding the intra sub-partition mode, the entropy encoder 338, under execution of the processor 205, encodes a decision on whether to use intra sub-partitioning in the bitstream 115 using the context coding "Intra_subpartitions_mode_flag" syntax element. Intra sub-partitioning is available for the luma channel when the luma CB size is larger than the minimum luma transform block size, i.e., larger than 16 luma samples. Intra sub-partitioning splits a coding unit into multiple luma transform blocks, as shown in set 1500. If the luma CB is split into multiple TBs, "intra_subpartitions_split_flag" signals whether the splitting of the luma CB into multiple luma TBs occurs horizontally or vertically. Collectively, the "intra_subpartitions_mode_flag" and "intra_subpartitions_split_flag" encode three possible splits, enumerated as "ISP_NO_SPLIT", "ISP_HOR_SPLIT", and "ISP_VER_SPLIT". Control in processor 205 passes from step 1650 to a coded residual test step 1660.

[0170] In coding residual test step 1660, processor 205 determines whether at least one residual coefficient in any transform block of the coding block is significant. This determination includes all luma TBs resulting from the application of intra sub-division and pairs of chroma TBs associated with the two chroma channels. If at least one residual coefficient in either the luma TB or the chroma TB is significant, entropy encoder 338, under the execution of processor 205, arithmetically codes a "1" for the "cu_cbf" syntax element, step 1660 returns "YES," and processor 205 proceeds to step 1670, which determines the luma coefficient group size. If no significant residual coefficient exists in any TB of the CU, step 1660 returns "NO," a "0" is arithmetically coded for cu_cbf, method 1600 ends, and processor 205 proceeds to the next CU in the CTU.

[0171] In a determine luma coefficient group size step 1670, the processor determines the coefficient group size of one or more luma TBs (transform blocks) associated with a CU. If intra sub-partitioning is not used, there is one luma TB. If intra sub-partitioning is used, there are two or four luma TBs. The size of the luma TB depends on the intra sub-partitioning performed horizontally or vertically and the number of luma TBs, and therefore on the luma CU size, as shown in set 1500.

[0172] The coefficient group size is determined using the luma TB width and height, as shown in Table 1 below. Table 1 shows the transform block (TB) size to coefficient group mapping table for the luma and chroma channels, which has the same size coefficient group for the TB, regardless of whether the TB is for the luma or chroma channel. The TB width and height are powers of two, so Table 1 considers the log2 of the TB width and height, i.e., "log2TBwidth" and "log2TBheight," to form the first two indices into the third dimension of Table 1. The final dimensions of the table distinguish the width and height of the coefficient group. The dimensions of the coefficient group are stored as log2 width and log2 height. For example, a TB of size 16x16 results in an index (4,4) into Table 1, which returns (2,2), indicating a coefficient group size of 4x4. A TB of size (2x32) results in an index (1,5) into Table 1, which returns (1,3), indicating a coefficient group size of 2x8. Since the minimum area of ​​a luma TB is 16 samples, it is not accessed if log2width+log2height is less than 4 in Table 1. When intra sub-splitting is used for a CU, each luma TB has the same size, and therefore, the coefficient group size determination for the luma TB is performed once for the CU.

[0173] Table 2 below shows the mapping of transform block (TB) sizes to coefficient group sizes for luma and chroma channels where luma has a different coefficient group size for the same size TB compared to chroma. When Table 2 is used, an additional dimension, i.e., distinguishing luma from chroma, is required, and the table size is doubled compared to Table 1. The coefficient group sizes defined in Table 1 result in sizes that fit within the TB width and height but are the largest possible size whose area does not exceed 16 samples. Table 1 provides a set of coefficient group sizes from which the coefficient group size is selected. The selected coefficient group aspect ratio of width to height is kept as close to 1:1 as possible within the TB width and height constraints. Control in processor 205 proceeds from step 1670 to step 1680, which encodes the luma TB.

[0174] In step 1680 of encoding a luma TB, the entropy encoder 338, under the execution of the processor 205, encodes residual coefficients of one or more luma TBs of the CU into the bitstream 115. The coefficient group size determined in step 1670 is used for each luma TB. For each luma TB, a coded block flag indicating the presence of at least one significant coefficient in the luma TB is coded into the bitstream 115. If at least one significant coefficient exists in the luma TB, the last significant position is coded into the bitstream. The last significant position is defined as the last significant coefficient along a scan path that proceeds from the DC (top-left) coefficient of the TB to the bottom-right coefficient. The scan path is defined as a diagonal scan when dividing the TB into an array of non-overlapping sub-blocks, each sized as the coefficient group size and occupying the entire TB. The progression from one sub-block to the next in scan order also follows the diagonal scan. The entropy encoder 338 encodes a "coding sub-block flag" for each coefficient group other than the top-left coefficient group and the coefficient group containing the last significant coefficient. The coding sub-block flag indicates the presence of at least one significant residual coefficient in the sub-block. If there is no significant residual coefficient in a sub-block, the diagonal scan of the residual coefficients in the TB skips that sub-block. If there is at least one significant residual coefficient in a sub-block, all positions in that sub-block are scanned, the magnitude of each residual coefficient is coded, and the sign of each significant residual coefficient is coded. Control in the processor 205 passes from step 1680 to step 1690, where the chroma coefficient group size is determined.

[0175] In step 1690, the processor 205 determines the coefficient group size for a pair of chroma transform blocks associated with a CU. Regardless of whether the luma CB is split into multiple luma TBs, one chroma CB for each chroma channel is associated with a CU. The coefficient group size is determined using the chroma TB width and height, as shown in Table 1. TB width and height are powers of two; therefore, Table 1 considers that the log2 of the TB width and height, i.e., "log2TBwidth" and "log2TBheight," form the first two indices into the third dimension of Table 1. The final dimensions of the table distinguish the width and height of the coefficient group. The dimensions of the coefficient group are stored as log2 width and log2 height. For example, a TB of size 16x16 results in an index (4,4) in Table 1, which returns (2,2), indicating a coefficient group size of 4x4. A TB of size (2x32) yields an index (1,5) in Table 1, which returns (1,3) indicating a coefficient group size of 2x8. Since each chroma TB has the same size, the coefficient group size determination for a pair of chroma TBs is performed once per CU. When Table 2 is used, an additional dimension is required, namely the dimension distinguishing luma from chroma, and the table size is doubled compared to that of Table 1.

[0176] As described in connection with steps 1670 and 1690, the coefficient group size is determined based only on the transform block size, without further distinction between the luma and chroma channels. Therefore, the coefficient group size is determined regardless of whether the chroma format is 4:2:2 or 4:2:0. As described in connection with Table 1, the coefficient group size is based on the maximum range of the coefficient group, up to 16 samples. Step 1690 operates to determine the coefficient group size for the TB, regardless of the transform block in the color plane (applicable to the Cb and Cr channels) or the subsampling color plane (Y, Cb, or Cr) due to the chroma format. Table 1 is used in both steps 1670 and 1690. Therefore, a single table is used for the transform block belonging to the luma plane and each of the chroma color planes. Control in processor 205 passes from step 1690 to step 16100, which encodes the chroma TB.

[0177] In step 16100 of encoding a chroma TB, the entropy encoder 338, under the execution of the processor 205, encodes the residual coefficients of a pair of chroma TBs of a CU into the bitstream 115. The coefficient group size determined in step 1690 is used for the pair of chroma TBs. For each chroma TB, a coded block flag indicating the presence of at least one significant coefficient in the chroma TB is coded into the bitstream 115. The remainder of the coding steps for each chroma TB correspond to the coding process for a luma TB, as described with reference to step 1680. The method 1600 ends upon execution of step 16100, and control in the processor 205 proceeds to the next CU of the CTU.

[0178] FIG. 17 illustrates a method 1700 for decoding a coding unit of an image frame from a video bitstream 133. The method 1700 may be implemented by a device such as a configured FPGA, ASIC, or ASSP. Furthermore, the method 1700 may be executed by the video decoder 134 under the execution of the processor 205. As such, the method 1700 may be stored in a computer-readable storage medium and / or memory 206. The method 1700 decodes blocks from the bitstream 133 such that coefficient group size is determined based solely on the transform block size, without further distinguishing between the luma and chroma channels. Because entropy decoding is a critical feedback loop in the video encoder 134, it is advantageous to reduce the memory accesses or operations required for coefficient group size determination. The method 1700 is invoked for each coding unit in the coding tree, i.e., as invoked in step 1430 of FIG. 14. As described above, step 1430 is executed when the current partition is no partition 510. The method 1700 begins at step 1710 with decoding pred_mode.

[0179] In a decoding pred_mode step 1710, the entropy decoder 420, under the execution of the processor 205, decodes the prediction mode of the CU from the bitstream 133. Control in the processor 205 passes from step 1710 to an intra prediction test step 1720.

[0180] In intra prediction test step 1720, processor 205 tests the prediction mode of the CU decoded in step 1710. If the prediction mode is intra prediction, step 1720 returns "YES", and control in processor 205 passes from step 1720 to step 1750, where the intra sub-partition mode is decoded. Otherwise, if it is not intra prediction, step 1720 returns "NO", and control in processor 205 passes from step 1720 to step 1730, where the merge flag and index are decoded.

[0181] In step 1730, the entropy decoder 420, under the execution of the processor 205, decodes from the bitstream 133 a merge flag that signals whether a "merge mode" is used for inter prediction in the bitstream. The merge mode causes the motion vector of a CU to be obtained from a spatially (or temporally) adjacent block among a set of spatially or temporally adjacent candidate blocks. When the merge mode is used, one candidate is selected by a "merge index" and is also decoded from the bitstream 133. When "motion vector prediction" is used, a similar decoding is performed, whereby one of several possible candidate motion vectors is signaled as a predictor by a flag in the bitstream. Control in the processor 205 passes from step 1730 to step 1740, where the motion vector delta is decoded.

[0182] In a step 1740 of decoding motion vector delta, the entropy decoder 420, under the execution of the processor 205, decodes the motion vector delta from the bitstream 133. Step 1740 is performed if motion vector prediction is used for the CU. The motion vector delta specifies the delta between the motion vector predictor coded in step 1730 and the motion vector used for motion compensation. Control in the processor 205 passes from step 1740 to a coded residual test step 1760. If motion vector prediction is not used for the CU, step 1740 is not performed and control in the processor 205 passes directly to step 1760.

[0183] In step 1750 of decoding the intra sub-partition mode, the entropy decoder 420, under the execution of the processor 205, decodes the decision of whether to use intra sub-partitioning from the bitstream 133 using the context-coded "intra_subpartitions_mode_flag" syntax element. When the luma CB size is larger than the minimum luma transform block size, i.e., 16 luma samples, intra sub-partitioning is available for the luma channel. As shown in set 1500, intra sub-partitioning divides a coding unit into multiple luma transform blocks. If the luma CB is divided into multiple TBs, "intra_subpartitions_split_flag" signals whether the division of the luma CB into multiple luma TBs occurs horizontally or vertically. Collectively, "intra_subpartitions_mode_flag" and "intra_subpartitions_split_flag" encode three possible splits, enumerated as "ISP_NO_SPLIT", "ISP_HOR_SPLIT", and "ISP_VER_SPLIT". Control in processor 205 passes from step 1750 to a coded residual test step 1760.

[0184] In coding residual test step 1760, processor 205 determines whether at least one residual coefficient in any transform block of the coding block is significant. This determination includes all luma TBs resulting from the application of intra sub-division and pairs of chroma TBs associated with the two chroma channels. Under execution of processor 205, entropy encoder 420 arithmetically decodes the "cu_cbf" syntax element, and processor 205 determines whether at least one residual coefficient in any TB of the CU is significant. If at least one residual coefficient in either the luma TB or the chroma TB is significant, step 1760 returns "YES," and control in processor 205 passes to step 1770, which determines the luma coefficient group size. If there are no significant residual coefficients in any TB of the CU, as indicated by arithmetically decoded "zeros" for cu_cbf, step 1760 returns "no," the method 1700 ends, and processor 205 proceeds to the next CU in the CTU.

[0185] In determine luma coefficient group size step 1770, the processor 205 determines the coefficient group size of one or more luma transform blocks associated with the CU. The determination of step 1770 operates similarly to the determination of step 1670. Control within the processor 205 passes from step 1770 to step 1780, where the processor 205 decodes the luma TB.

[0186] In step 1780 of decoding a luma TB, the entropy decoder 420, under the execution of the processor 205, decodes residual coefficients of one or more luma TBs of a CU from the bitstream 133. The coefficient group size determined in step 1770 is used for each luma TB. For each luma TB, a coded block flag indicating the presence of at least one significant coefficient in the luma TB is decoded from the bitstream 133. If at least one significant coefficient is present in the luma TB, the last significant position is decoded from the bitstream. The last significant position is defined as the last significant coefficient along a scan path that proceeds from the DC (top-left) coefficient of the TB to the bottom-right coefficient. The scan path is defined as a diagonal scan when dividing the TB into an array of non-overlapping sub-blocks, each sized as the coefficient group size and occupying the entire TB. The progression from one sub-block to the next in scan order also follows the diagonal scan. The entropy encoder 338 encodes a "coded sub-block flag" for each coefficient group other than the top-left coefficient group and the coefficient group containing the last significant coefficient. The coded sub-block flag indicates the presence of at least one significant residual coefficient in the sub-block. If there is no significant residual coefficient in a sub-block, the diagonal scan of the residual coefficients in the TB skips that sub-block. If there is at least one significant residual coefficient in a sub-block, all positions in that sub-block are scanned, the magnitude of each residual coefficient is coded, and the sign of each significant residual coefficient is coded. Control in the processor 205 passes from step 1780 to step 1790, which determines the chroma coefficient group size.

[0187] In a determine chroma coefficient group size step 1790, the processor 205 determines the coefficient group size for a pair of chroma transform blocks associated with a CU. The determination made in step 1790 operates in the same manner as the determination made in step 1690.

[0188] Similar to step 1690, the coefficient group size is determined in step 1790 based on the transform block size, with no further distinction made between the luma and chroma channels. Thus, the coefficient group size is determined regardless of whether the chroma format is 4:2:2 or 4:2:0, or the corresponding subsampling in each color plane. As described in connection with Table 1, the coefficient group size is based on the maximum area of ​​the TB, which is up to 16 samples. Step 1690 operates to determine the coefficient group size of the TB regardless of the color plane (Cb or Cr) of the transform block. Control within processor 205 proceeds from step 1790 to step 17100, where the chroma TB is decoded.

[0189] In step 17100 of decoding chroma TBs, the entropy decoder 420, under the execution of the processor 205, decodes residual coefficients of a pair of chroma TBs of a CU from the bitstream 133. The coefficient group size determined in step 1790 is used for the pair of chroma TBs. For each chroma TB, a coded block flag is decoded from the bitstream 133, indicating the presence of at least one significant coefficient in the chroma TB. The remainder of the decoding process for each chroma TB operates in the same manner as for the luma TB, as described with reference to step 1780. The method 1700 ends upon execution of step 17100, and control in the processor 205 proceeds to the next CU of the CTU.

[0190] Table 3 shows the coding performance results obtained under the JVET "Common Test Conditions" (CTC) - "All Intra Main 10" configuration when using Table 1. The results in Table 3 were obtained using the "VVC Test Model" (VTM) software implementing methods 1600 and 1700, compared to the baseline VTM-4.0, which does not implement methods 1600 and 1700. Overall, there is no coding impact from the changes, and even a small gain is seen in the chroma channels, demonstrating that simplifying the mapping table from transform block size to coefficient group size is not detrimental to coding performance.

[0191] Video encoder 115 and video decoder 134 achieve memory reduction in the residual encoding / decoding process by harmonizing the coefficient group sizes of the luma TB and chroma TB using methods 1600 and 1700, respectively. As a result, the chroma TB can access coefficient group sizes such as 2x8; 4x2, 2x4, and 8x2, instead of only 2x2 and 4x4. For the luma TB, using intra subdivision allows sizes 16x1 and 1x16. While sizes 16x1 and 1x16 are available for chroma due to their presence in Table 1, the minimum width and height of a chroma block is two samples, and therefore sizes 16x1 and 1x16 are not used in the chroma TB. Because residual encoding and decoding are part of a feedback loop in the design, memory reduction corresponds, for example, to improved cache performance in a software implementation or critical path reduction in a hardware implementation.

[0192] uint32_t g_log2SbbSize[MAX_CU_DEPTH + 1][MAX_CU_DEPTH + 1][2] = / / ===== Luma / Chroma ===== { { { 0,0},{ 0,1},{ 0,2},{ 0,3},{ 0,4},{ 0,4},{ 0,4}}, { { 1,0},{ 1,1},{ 1,2},{ 1,3},{ 1,3},{ 1,3},{ 1,3}}, { { 2,0},{ 2,1},{ 2,2},{ 2,2},{ 2,2},{ 2,2},{ 2,2}}, { { 3,0},{ 3,1},{ 2,2},{ 2,2},{ 2,2},{ 2,2},{ 2,2}}, { { 4,0},{ 3,1},{ 2,2},{ 2,2},{ 2,2},{ 2,2},{ 2,2}}, { { 4,0},{ 3,1},{ 2,2},{ 2,2},{ 2,2},{ 2,2},{ 2,2}}, { { 4,0},{ 3,1},{ 2,2},{ 2,2},{ 2,2},{ 2,2},{ 2,2}}, { { 4,0},{ 3,1},{ 2,2},{ 2,2},{ 2,2},{ 2,2},{ 2,2}} }; Table 1: Conversion of block size to coefficient group mapping table for luma and chroma channels (with the same size coefficient group for TB whether the TB is for luma or chroma channel) uint32_t g_log2SbbSize[2][MAX_CU_DEPTH+1][MAX_CU_DEPTH+1][2] = { / / ===== Luma ===== { { {0,0}, {0,1}, {0,2}, {0,3}, {0,4}, {0,4}, {0,4}}, { {1,0}, {1,1}, {1,2}, {1,3}, {1,3}, {1,3}, {1,3}}, { {2,0}, {2,1}, {2,2}, {2,2}, {2,2}, {2,2}, {2,2}, {2,2}}, { {3,0}, {3,1}, {2,2}, {2,2}, {2,2}, {2,2}, {2,2}, {2,2}}, { {4,0}, {3,1}, {2,2}, {2,2}, {2,2}, {2,2}, {2,2}, {2,2}}, { {4,0}, {3,1}, {2,2}, {2,2}, {2,2}, {2,2}, {2,2}, {2,2}}, { {4,0}, {3,1}, {2,2}, {2,2}, {2,2}, {2,2}, {2,2}, {2,2}}, { {4,0}, {3,1}, {2,2}, {2,2}, {2,2}, {2,2}, {2,2}, {2,2}} }, / / ===== Chroma ===== { { {0,0}, {0,0}, {0,0}, {0,0}, {0,0}, {0,0}, {0,0}, {0,0}}, { {0,0}, {1,1}, {1,1}, {1,1}, {1,1}, {1,1}, {1,1}, {1,1}}, { {0,0}, {1,1}, {2,2}, {2,2}, {2,2}, {2,2}, {2,2}, {2,2}}, { {0,0}, {1,1}, {2,2}, {2,2}, {2,2}, {2,2}, {2,2}, {2,2}}, { {0,0}, {1,1}, {2,2}, {2,2}, {2,2}, {2,2}, {2,2}, {2,2}}, { {0,0}, {1,1}, {2,2}, {2,2}, {2,2}, {2,2}, {2,2}, {2,2}}, { {0,0}, {1,1}, {2,2}, {2,2}, {2,2}, {2,2}, {2,2}, {2,2}}, { {0,0}, {1,1}, {2,2}, {2,2}, {2,2}, {2,2}, {2,2}, {2,2}} }, }; Table 2: Conventional mapping of transform block sizes to coefficient group sizes for luma and chroma channels (with different coefficient group sizes for the same size TB for luma vs. chroma)

[0193] [Table 1]

[0194] Table 3: Coding performance resulting from having the same size coefficient group for the TB, regardless of whether the TB is for the luma or chroma channel.

[0195] Industrial Applicability The described arrangements are applicable to the computer and data processing industry, particularly in digital signal processing for encoding and decoding signals such as video and image signals, to achieve high compression efficiency.

[0196] In contrast to HEVC, the VVC system allows the use of separate coding trees for the luma and chroma channels to increase flexibility. However, as mentioned above, resulting problems can arise from the use of smaller chroma blocks, which impacts throughput. The configurations described herein determine appropriate rules when each coding tree unit is processed to help avoid throughput issues. Furthermore, as mentioned above, the configurations described above can help provide improved efficiency and accuracy of the arithmetic coding of the context coding bins used to describe each coding tree, given the rules for avoiding throughput issues.

[0197] The above describes only some embodiments of the present invention, modifications and / or variations can be made to the present invention without departing from the scope and spirit of the invention, and the embodiments are illustrative and not limiting.

Claims

1. 1. A method for decoding a transform block of an image frame from a bitstream, comprising: determining a chroma format of the image frame from a plurality of chroma formats including a 4:2:0 chroma format and a 4:2:2 chroma format; dividing the coding tree unit into one or more coding units, each having a luma coding block and a chroma coding block; determining a sub-block of the transform block, which is a luma transform block for a luma coded block or a chroma transform block for a chroma coded block; decoding the transform block from the bitstream using the sub-block; when a block size of a chroma coding block of a coding unit having a luma coding block and a chroma coding block in the coding tree unit of the 4:2:0 chroma format is 8x2, even if vertical ternary partitioning is performed on the luma coding block of the coding unit, vertical ternary partitioning on the chroma coding block of the coding unit is not permitted; when the luma coding block of the certain coding unit is divided by vertical ternary partitioning and the chroma coding block of the certain coding unit is not divided by vertical ternary partitioning, the chroma coding block of the certain coding unit is arranged at a position corresponding to three luma coding blocks obtained by vertical ternary partitioning of the luma coding block of the certain coding unit; determining that intra prediction will be used for the chroma coding block of the certain coding unit in the coding tree unit of the 4:2:0 chroma format if the block size of the chroma coding block of the certain coding unit is a predetermined size and a current partition mode for the certain coding unit is quadrant; the transform block is decoded based on (i) a size of the transform block and (ii) whether the transform block is a luma transform block or a chroma transform block; When determining the size of the sub-blocks of the transform block when the size of the transform block is 16x16, the size of the sub-blocks of the transform block is determined from the size of the transform block, regardless of whether the transform block is a luma transform block or a chroma transform block and the chroma format of the image frame. A method characterized by:

2. 1. A video decoder for decoding transform blocks of an image frame from a bitstream, comprising: means for determining a chroma format of the image frame from a plurality of chroma formats including a 4:2:0 chroma format and a 4:2:2 chroma format; means for dividing a coding tree unit into one or more coding units, each having a luma coding block and a chroma coding block; means for determining a sub-block of the transform block, the sub-block being a luma transform block for a luma coded block or a chroma transform block for a chroma coded block; means for decoding the transform block from the bitstream using the sub-block; when a block size of a chroma coding block of a coding unit having a luma coding block and a chroma coding block in the coding tree unit of the 4:2:0 chroma format is 8x2, even if vertical ternary partitioning is performed on the luma coding block of the coding unit, vertical ternary partitioning on the chroma coding block of the coding unit is not permitted; when the luma coding block of the certain coding unit is divided by vertical ternary partitioning and the chroma coding block of the certain coding unit is not divided by vertical ternary partitioning, the chroma coding block of the certain coding unit is arranged at a position corresponding to three luma coding blocks obtained by vertical ternary partitioning of the luma coding block of the certain coding unit; determining that intra prediction will be used for the chroma coding block of the certain coding unit in the coding tree unit of the 4:2:0 chroma format if the block size of the chroma coding block of the certain coding unit is a predetermined size and a current partition mode for the certain coding unit is quadrant; the transform block is decoded based on (i) a size of the transform block and (ii) whether the transform block is a luma transform block or a chroma transform block; When determining the size of the sub-blocks of the transform block when the size of the transform block is 16x16, the size of the sub-blocks of the transform block is determined from the size of the transform block, regardless of whether the transform block is a luma transform block or a chroma transform block and the chroma format of the image frame.

1. A video decoder comprising:

3. 1. A method for encoding a transform block of an image frame into a bitstream, comprising: determining a chroma format of the image frame from a plurality of chroma formats including a 4:2:0 chroma format and a 4:2:2 chroma format; dividing the coding tree unit into one or more coding units, each having a luma coding block and a chroma coding block; determining a sub-block of the transform block, the sub-block being a luma transform block for a luma coded block or a chroma transform block for a chroma coded block; encoding the transform block into the bitstream using the sub-blocks; when a block size of a chroma coding block of a coding unit having a luma coding block and a chroma coding block in the coding tree unit of the 4:2:0 chroma format is 8x2, even if vertical ternary partitioning is performed on the luma coding block of the coding unit, vertical ternary partitioning on the chroma coding block of the coding unit is not permitted; when the luma coding block of the certain coding unit is divided by vertical ternary partitioning and the chroma coding block of the certain coding unit is not divided by vertical ternary partitioning, the chroma coding block of the certain coding unit is arranged at a position corresponding to three luma coding blocks obtained by vertical ternary partitioning of the luma coding block of the certain coding unit; determining that intra prediction will be used for the chroma coding block of the certain coding unit in the coding tree unit of the 4:2:0 chroma format if the block size of the chroma coding block of the certain coding unit is a predetermined size and a current partition mode for the certain coding unit is quadrant; the transform block is coded based on (i) its size and (ii) whether the transform block is a luma transform block or a chroma transform block; When determining the size of the sub-blocks of the transform block when the size of the transform block is 16x16, the size of the sub-blocks of the transform block is determined from the size of the transform block, regardless of whether the transform block is a luma transform block or a chroma transform block and the chroma format of the image frame. A method characterized by:

4. 1. A video encoder for encoding transform blocks of an image frame into a bitstream, comprising: means for determining a chroma format of the image frame from a plurality of chroma formats including a 4:2:0 chroma format and a 4:2:2 chroma format; means for dividing a coding tree unit into one or more coding units, each having a luma coding block and a chroma coding block; means for determining a sub-block of the transform block, the sub-block being a luma transform block for a luma coded block or a chroma transform block for a chroma coded block; means for encoding the transform block into the bitstream using the sub-blocks; when a block size of a chroma coding block of a coding unit having a luma coding block and a chroma coding block in the coding tree unit of the 4:2:0 chroma format is 8x2, even if vertical ternary partitioning is performed on the luma coding block of the coding unit, vertical ternary partitioning on the chroma coding block of the coding unit is not permitted; when the luma coding block of the certain coding unit is divided by vertical ternary partitioning and the chroma coding block of the certain coding unit is not divided by vertical ternary partitioning, the chroma coding block of the certain coding unit is arranged at a position corresponding to three luma coding blocks obtained by vertical ternary partitioning of the luma coding block of the certain coding unit; determining that intra prediction will be used for the chroma coding block of the certain coding unit in the coding tree unit of the 4:2:0 chroma format if the block size of the chroma coding block of the certain coding unit is a predetermined size and a current partition mode for the certain coding unit is quadrant; the transform block is coded based on (i) its size and (ii) whether the transform block is a luma transform block or a chroma transform block; When determining the size of the sub-blocks of the transform block when the size of the transform block is 16x16, the size of the sub-blocks of the transform block is determined from the size of the transform block, regardless of whether the transform block is a luma transform block or a chroma transform block and the chroma format of the image frame.

1. A video encoder comprising:

Images