Method and apparatus for decoding transform block

Abstract

The present invention relates to a method and an apparatus for decoding a transform block. The invention also relates to a system and a method for decoding a transform block of a color channel of an image frame from a video bitstream. The method comprises: determining a chroma format of the image frame, the chroma format causing subsampling of a chroma channel of the image frame relative to a luminance channel of the image frame; determining a coefficient set size for the transform block, the coefficient set size being a maximum region of a transform block of up to 16 samples, the coefficient set size being based only on a transform block size and determined independently of both (i) a color plane of the transform block and (ii) color plane subsampling due to the determined chroma format; and decoding the transform block from the video bitstream using the coefficient group of the determined size.

Description

[0001] (This application is a divisional application of an application filed on January 20, 2020, with application number 2020800202929 and invention name “Method, device and system for encoding and decoding block tree of video samples”.) Technical Field

[0002] The present invention generally relates to digital video signal processing, and more particularly to a method, device and system for encoding and decoding a block tree of video samples. The present invention also relates to a computer program product comprising a computer readable medium having recorded thereon a computer program for encoding and decoding a block tree of video samples. Background Art

[0003] There are currently many applications for video coding, including applications for transmitting and storing video data. Many video coding standards have also been developed and other video coding standards are currently under development. Recent advances in video coding standardization have led to the formation of a group known as the "Joint Video Experts Group" (JVET). The Joint Video Experts Group (JVET) includes members of Study Group 16, Question 6 (SG16 / Q6) of the Telecommunication Standardization Sector (ITU-T) of the International Telecommunication Union (ITU), also known as the "Video Coding Experts Group" (VCEG); and members of ISO / IEC JTC1 / SC29 / WG11, also known as the "Moving Picture Experts Group" (MPEG).

[0004] The Joint Video Experts Team (JVET) issued a call for proposals (CfP) and analyzed the responses at its 10th meeting in San Diego, USA. The responses submitted showed that the video compression capabilities are significantly better than those of the current state-of-the-art video compression standard, namely "High Efficiency Video Coding" (HEVC). Based on this excellent performance, it was decided to start a project to develop a new video compression standard named "Versatile Video Coding" (VVC). It is expected that VVC will address the continued demand for even higher compression performance, especially as the capabilities of video formats increase (e.g., with higher resolutions and higher frame rates), and the growing market demand for service provision over WANs, where bandwidth costs are relatively high. At the same time, VVC must be implementable in contemporary silicon processes and provide an acceptable trade-off between the achieved performance and the implementation cost (e.g., in terms of silicon area, CPU processor load, memory utilization, and bandwidth).

[0005] Video data includes a sequence of frames of image data, each of which includes one or more color channels. Typically, one primary color channel and two secondary color channels are required. The primary color channel is often referred to as the "brightness" channel, and (one or more) secondary color channels are often referred to as the "chrominance" channel. Although video data is usually displayed in an RGB (red-green-blue) color space, this color space has a high degree of correlation between the three corresponding components. The video data representation seen by the encoder or decoder usually uses a color space such as YCbCr. YCbCr concentrates luminance (mapped to "brightness" according to the transformation equation) in the Y (primary) channel and concentrates chrominance in the Cb and Cr (secondary) channels. In addition, the Cb and Cr channels can be spatially sampled at a lower rate than the brightness channel (for example, half in the horizontal direction and half in the vertical direction (referred to as the "4:2:0 chroma format")). The 4:2:0 chroma format is commonly used in "consumer" applications such as Internet video streaming, broadcast television, and Blu-ray TM Storage on disk. Subsampling the Cb and Cr channels at half rate horizontally instead of vertically is referred to as the "4:2:2 chroma format". The 4:2:2 chroma format is commonly used in professional applications, including the capture of footage for filmmaking, etc. The higher sampling rate of the 4:2:2 chroma format makes the resulting video more resilient to editing operations (such as color grading, etc.). Before distribution to consumers, 4:2:2 chroma format material is often converted to 4:2:0 chroma format and then encoded for distribution to consumers. In addition to the chroma format, video is also characterized by resolution and frame rate. Example resolutions are ultra-high definition (UD) with a resolution of 3840×2160 or "8K" with a resolution of 7680×4320, and example frame rates are 60 Hz or 120 Hz. The luminance sample rate can range from about 500 megasamples / second to several gigasamples / second. For the 4:2:0 chroma format, the sampling rate of each chroma channel is one quarter of the luma sampling rate, and for the 4:2:2 chroma format, the sampling rate of each chroma channel is half of the luma sampling rate.

[0006] The VVC standard is a "block-based" codec in which a frame is first partitioned into an array of square areas called "coding tree units" (CTUs). A CTU typically occupies a relatively large area, such as 128×128 luma samples. However, the areas of the CTUs at the right and bottom edges of each frame may be smaller. Associated with each CTU is a "coding tree" for the luma channel and an additional coding tree for the chroma channel. The coding tree defines the decomposition of the area of ​​the CTU into a set of areas, also called "coding blocks" (CBs). Blocks for both luma and chroma channels can also be specified for a single coding tree, in which case a set of juxtaposed coding blocks is called a "coding unit" (CU), i.e., each CU has a coding block for each color channel. CBs are processed in a specific order for encoding or decoding. As a result of using the 4:2:0 chroma format, a CTU of a luma coding tree including a 128×128 luma sample area has a corresponding chroma coding tree of a 64×64 chroma sample area juxtaposed with a 128×128 luma sample area. When a single coding tree is used for the luma and chroma channels, a set of collocated blocks for a given region is often referred to as a "unit", such as the above-mentioned CU and "prediction unit" (PU) and "transform unit" (TU). When a separate coding tree is used for a given region, the above-mentioned CB and "prediction block" (PB) and "transform block" (TB) will be used.

[0007] Despite the above distinction between a “unit” and a “block,” the term “block” may be used as a general term for an area or region of a frame to which an operation is applied to all color channels.

[0008] For each CU, a prediction unit (PU) ("prediction unit") is generated for the content (sample values) of the corresponding region of the frame data. In addition, a representation of the difference (or "residual" in the spatial domain) between the prediction and the region content seen at the input of the encoder is formed. The difference of each color channel can be transformed and encoded as a sequence of residual coefficients, thereby forming one or more TUs for a given CU. The applied transform can be a discrete cosine transform (DCT) or other transform applied to each block of residual values. The transform is applied separately, that is, a two-dimensional transform is performed in two passes. The block is first transformed by applying a one-dimensional transform to each row of samples in the block. Then, the partial result is transformed by applying a one-dimensional transform to each column of the partial result to produce a final block of transform coefficients that essentially decorrelate the residual samples. The VVC standard supports transforms of various sizes, including transforms of rectangular blocks (each side size is a power of 2). The transform coefficients are quantized for entropy encoding in the bitstream.

[0009] When spatial prediction ("intra-frame prediction") is used to generate a PB, a set of reference samples is used to generate prediction samples for the current PB. The reference samples include samples adjacent to the PB that have been "reconstructed" (residual samples are added to the intra-frame prediction samples). These adjacent samples form rows above the PB and columns to the left of the PB. The rows and columns also extend beyond the PB boundaries to include additional nearby samples. Since blocks are scanned in a Z-order scan, some reference samples will be reconstructed in the immediately preceding block. Using samples from the immediately preceding block results in feedback dependencies, which may limit the throughput of blocks through a video encoder or decoder. In addition, in the case of predicting relatively small blocks from other frames ("inter-frame prediction"), the memory bandwidth used to extract reference samples may become too large, especially considering the additional samples required to accommodate sub-pixel interpolation filtering. Summary of the invention

[0010] It is an object of the present invention to substantially overcome or at least ameliorate one or more disadvantages of existing arrangements.

[0011] One aspect of the present invention provides a method for decoding a transform block of a color channel of an image frame from a video bitstream, the method comprising: determining a chroma format of the image frame, the chroma format causing the chroma channels of the image frame to be subsampled relative to the luma channel of the image frame; determining a coefficient group size of the transform block, the coefficient group size being a maximum area of ​​the transform block of up to 16 samples, the coefficient group size being determined based solely on the transform block size and being independent of both (i) color planes of the transform block and (ii) subsampling of the color planes due to the determined chroma format; and decoding the transform block from the video bitstream using coefficient groups of the determined size.

[0012] According to another aspect, a single table is used for transform blocks of luma color planes and chroma color planes of an image frame belonging to the bitstream.

[0013] According to another aspect, the coefficient group size is selected to have an aspect ratio closest to 1:1 within the constraints of the width and height of the transform block.

[0014] Another aspect of the present invention provides a non-transitory computer-readable medium having a computer program stored thereon for implementing a method for decoding a transform block of a color channel of an image frame from a video bitstream, the program comprising: code for determining a chroma format of the image frame, the chroma format causing the chroma channels of the image frame to be subsampled relative to the luma channel of the image frame; code for determining a coefficient group size of the transform block, the coefficient group size being a maximum area of ​​the transform block of up to 16 samples, the coefficient group size being determined based solely on the transform block size and being independent of both (i) the color planes of the transform block and (ii) the subsampling of the color planes due to the determined chroma format; and code for decoding the transform block from the video bitstream using the coefficient group of the determined size.

[0015] Another aspect of the present invention provides a video decoder configured to: receive a transform block of a color channel of an image frame from a video bitstream; determine a chroma format of the image frame, the chroma format causing the chroma channels of the image frame to be subsampled relative to the luma channel of the image frame; determine a coefficient group size for the transform block, the coefficient group size being a maximum area of ​​the transform block of up to 16 samples, the coefficient group size being determined based solely on the transform block size and being independent of both (i) color planes of the transform block and (ii) subsampling of the color planes due to the determined chroma format; and decode the transform block from the video bitstream using coefficient groups of the determined size.

[0016] Another aspect of the present invention provides a system, comprising: a memory; and a processor, wherein the processor is configured to execute code stored on the memory to implement a method for decoding a transform block of a color channel of an image frame from a video bitstream, the method comprising: determining a chroma format of the image frame, the chroma format causing the chroma channels of the image frame to be subsampled relative to the luma channel of the image frame; determining a coefficient group size of the transform block, the coefficient group size being a maximum area of ​​the transform block of up to 16 samples, the coefficient group size being determined based solely on the transform block size and being independent of both (i) the color planes of the transform block and (ii) the subsampling of the color planes due to the determined chroma format; and decoding the transform block from the video bitstream using coefficient groups of the determined size.

[0017] Other aspects are also disclosed. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] At least one example embodiment of the present invention will now be described with reference to the following drawings and appendices, in which:

[0019] Figure 1is a schematic block diagram illustrating a video encoding and decoding system;

[0020] Figure 2A and 2B Composition can be practiced Figure 1 A schematic block diagram of a general computer system for one or both of the video encoding and decoding systems;

[0021] Figure 3 is a schematic block diagram showing the functional modules of a video encoder;

[0022] Figure 4 is a schematic block diagram showing the functional modules of a video decoder;

[0023] Figure 5 is a schematic block diagram illustrating possible partitioning of a block into one or more blocks in a tree structure for general video coding;

[0024] Figure 6 A schematic diagram of a data flow for implementing a permissible partitioning of a block into one or more blocks in a tree structure of a general video coding;

[0025] Fig. 7A and 7B shows an example partitioning of a coding tree unit (CTU) into multiple coding units (CUs);

[0026] Fig. 8A , 8B and 8C show an example partitioning of a coding tree unit (CTU) in luma and chroma channels into multiple coding blocks (CBs);

[0027] Fig. 9 showing a set of transform block sizes and associated scan modes;

[0028] Fig.10 A set of rules for generating a list of allowed splits in a luma coding tree and a chroma coding tree is shown;

[0029] Fig.11 A method of encoding a coding tree of an image frame into a video bitstream is shown;

[0030] Fig.12 A method for decoding a coding tree of an image frame from a video bitstream is shown;

[0031] Fig.13 A method of encoding a coding tree of an image frame into a video bitstream is shown;

[0032] Fig.14 A method for decoding a coding tree of an image frame from a video bitstream is shown;

[0033] Fig.15 A set of transform block partitions showing an intra-prediction coding unit;

[0034] Fig.16 A method of encoding coding units of an image frame into a video bitstream is shown; and

[0035] Fig.17 A method of decoding coding units of an image frame from a video bitstream is shown. DETAILED DESCRIPTION

[0036] Where steps and / or features having the same reference numerals are referenced in any one or more of the figures, such steps and / or features have the same function(s) or operation(s) for purposes of this specification, unless otherwise intended.

[0037] As described above, using samples from the immediately previous block results in feedback dependencies, which can limit the throughput of blocks through a video encoder or decoder. Methods for mitigating the severity of the resulting feedback dependency loops are desired to ensure that high rates of processing blocks can be maintained, as required by typical real-time encoding and decoding applications. Feedback dependency loops are particularly problematic for the high sampling rates of modern video formats (e.g., 500-4000 samples / second), while ASIC (Application Specific Integrated Circuit) clock frequencies are typically hundreds of MHz.

[0038] Figure 1 is a schematic block diagram illustrating functional modules of the video encoding and decoding system 100. The system 100 can utilize different rules to allow subdivision of regions in the luma and chroma coding trees to reduce the worst-case block processing rate encountered. For example, the system 100 can operate such that the size of the block is always set to a multiple of 16 (sixteen) samples, regardless of the aspect ratio of the block. In addition, in the case where the coding tree includes a split indicating the presence of a small luma coding block, the split can be prohibited in the chroma channel so that a single chroma CB is juxtaposed with multiple luma CBs. Independent of the prediction mode of each juxtaposed luma CB (including the case where one or more luma CBs use inter-frame prediction), the chroma CBs can use a single prediction mode (such as an intra-frame prediction mode, etc.). Residual coefficient encoding can also utilize block sizes that are multiples of 16, including the case where the block has a width or height of two samples.

[0039] The system 100 includes a source device 110 and a destination device 130. A communication channel 120 is used to communicate encoded video information from the source device 110 to the destination device 130. In some configurations, one or both of the source device 110 and the destination device 130 may each include a mobile phone handset or "smart phone", in which case the communication channel 120 is a wireless channel. In other configurations, the source device 110 and the destination device 130 may include video conferencing equipment, in which case the communication channel 120 is typically a wired channel such as an Internet connection. In addition, the source device 110 and the destination device 130 may include any of a wide range of devices, including devices that support over-the-air television broadcasts, cable television applications, Internet video applications (including streaming), and applications that capture encoded video data on some computer-readable storage medium (such as a hard drive in a file server).

[0040] like Figure 1 As shown, source device 110 includes video source 112, video encoder 114, and transmitter 116. Video source 112 typically includes a source of captured video frame data (denoted as 113), such as a camera sensor, a previously captured video sequence stored on a non-transitory recording medium, or a video feed from a remote camera sensor. Video source 112 may also be the output of a computer graphics card (e.g., display operating system and video output of various applications executed on a computing device (e.g., tablet computer)). Examples of source device 110 that may include a camera sensor as video source 112 include smart phones, video camcorders, professional video cameras, and web video cameras.

[0041] The video encoder 114 converts (or "encodes") the captured frame data (indicated by arrow 113) from the video source 112 into a bitstream (indicated by arrow 115). The bitstream 115 is transmitted by a transmitter 116 as encoded video data (or "encoded video information") via a communication channel 120. The bitstream 115 may also be stored in a non-transitory storage device 122, such as a "flash" memory or a hard drive, until or in lieu of subsequent transmission over the communication channel 120.

[0042] The destination device 130 includes a receiver 132, a video decoder 134, and a display device 136. The receiver 132 receives the encoded video data from the communication channel 120 and passes the received video data to the video decoder 134 as a bit stream (indicated by arrow 133). The video decoder 134 then outputs the decoded frame data (indicated by arrow 135) to the display device 136. The decoded frame data 135 has the same chroma format as the frame data 113. Examples of the display device 136 include a cathode ray tube, a liquid crystal display (such as in a smart phone, a tablet computer, a computer monitor, or a stand-alone television). The functions of each of the source device 110 and the destination device 130 may also be embodied in a single device, examples of which include a mobile phone and a tablet computer.

[0043] Although example devices are described above, source device 110 and destination device 130 may each typically be configured within a general purpose computer system via a combination of hardware and software components. Figure 2A Such a computer system 200 is shown, which includes: a computer module 201; input devices such as a keyboard 202, a mouse pointer device 203, a scanner 226, a camera 227 that can be configured as a video source 112, and a microphone 280; and output devices including a printer 215, a display device 214 that can be configured as a display device 136, and a speaker 217. The computer module 201 can use an external modulator-demodulator (modem) transceiver device 216 to communicate with a communication network 220 via a connection 221. The communication network 220, which can represent a communication channel 120, can be a wide area network (WAN), such as the Internet, a cellular telecommunications network, or a private WAN. In the case where the connection 221 is a telephone line, the modem 216 can be a traditional "dial-up" modem. Alternatively, in the case where the connection 221 is a high-capacity (e.g., cable or optical) connection, the modem 216 can be a broadband modem. Wireless modems can also be used to make wireless connections to the communication network 220. The transceiver device 216 may provide the functionality of the transmitter 116 and the receiver 132 , and the communication channel 120 may be embodied in the wiring 221 .

[0044] The computer module 201 typically includes at least one processor unit 205 and a memory unit 206. For example, the memory unit 206 may have a semiconductor random access memory (RAM) and a semiconductor read-only memory (ROM). The computer module 201 also includes a plurality of input / output (I / O) interfaces, wherein the plurality of input / output (I / O) interfaces include: an audio-video interface 207 connected to a video display 214, a speaker 217, and a microphone 280; an I / O interface 213 connected to a keyboard 202, a mouse 203, a scanner 226, a camera 227, and an optional joystick or other human-machine interface device (not shown); and an interface 208 for an external modem 216 and a printer 215. The signal from the audio-video interface 207 to the computer monitor 214 is typically the output of a computer graphics card. In some implementations, the modem 216 may be built into the computer module 201, such as built into the interface 208. The computer module 201 also has a local network interface 211, which allows the computer system 200 to be connected to a local area communication network 222, known as a local area network (LAN), via a connection 223. Figure 2A As shown, the local communication network 222 can also be connected to the wide area network 220 via a connection 224, wherein the local communication network 222 typically includes a so-called "firewall" device or a device with similar functions. The local network interface 211 can include an Ethernet (EthernetTM) circuit card, a Bluetooth (BluetoothTM) wireless configuration, or an IEEE 802.11 wireless configuration; however, for the interface 211, a variety of other types of interfaces can be implemented. The local network interface 211 can also provide the functions of the transmitter 116 and the receiver 132, and the communication channel 120 can also be embodied in the local communication network 222.

[0045] I / O interfaces 208 and 213 may provide either or both of a serial connection and a parallel connection, wherein the former is typically implemented according to the Universal Serial Bus (USB) standard and has a corresponding USB connector (not shown). A storage device 209 is provided, and the storage device 209 typically includes a hard disk drive (HDD) 210. Other storage devices (not shown) such as floppy disk drives and tape drives may also be used. An optical disk drive 212 is typically provided to serve as a non-volatile source of data. Portable memory devices such as optical disks (e.g., CD-ROMs, DVDs, Blu-ray Discs), USB-RAMs, portable external hard disk drives, and floppy disks may be used as appropriate sources of data for the computer system 200. Typically, any of the HDD 210, optical disk drive 212, networks 220 and 222 may also be configured to operate as a video source 112, or as a destination for stored decoded video data to be reproduced via a display 214. The source device 110 and the destination device 130 of the system 100 may be embodied in the computer system 200.

[0046] The components 205-213 of the computer module 201 typically communicate via an interconnect bus 204 and in a manner that results in conventional modes of operation of the computer system 200 known to those skilled in the relevant art. For example, the processor 205 is connected to the system bus 204 using a connection 218. Likewise, the memory 206 and the optical drive 212 are connected to the system bus 204 via a connection 219. Examples of computers on which the described configuration may be practiced include IBM-PC and compatible machines, Sun SPARCstations, Apple Mac™, or similar computer systems.

[0047] Where appropriate or desired, the video encoder 114 and the video decoder 134 and the methods described below may be implemented using the computer system 200. In particular, the video encoder 114, the video decoder 134 and the methods to be described may be implemented as one or more software applications 233 executable within the computer system 200. In particular, instructions 231 (see FIG. 1 ) in the software 233 executed within the computer system 200 may be used to implement the video encoder 114 and the video decoder 134 and the methods to be described below. Figure 2B ) to implement the video encoder 114, the video decoder 134 and the steps of the method. The software instructions 231 may be formed into one or more code modules each for performing one or more specific tasks. The software may also be split into two separate parts, with a first part and corresponding code modules performing the method, and a second part and corresponding code modules managing a user interface between the first part and a user.

[0048] For example, the software may be stored in a computer readable medium including a storage device as described below. The software is loaded from the computer readable medium into the computer system 200 and then executed by the computer system 200. A computer readable medium with such software or a computer program recorded on the computer readable medium is a computer program product. The use of the computer program product in the computer system 200 preferably implements an advantageous device for implementing the video encoder 114, the video decoder 134 and the method.

[0049] The software 233 is typically stored in the HDD 210 or the memory 206. The software is loaded into the computer system 200 from a computer readable medium and executed by the computer system 200. Thus, for example, the software 233 may be stored on an optically readable disk storage medium (e.g., a CD-ROM) 225 read by the optical drive 212.

[0050] In some examples, the application 233 is provided to the user in a manner encoded on one or more CD-ROMs 225 and read via corresponding drives 212, or alternatively, the application 233 can be read by the user from the network 220 or 222. Further, the software can also be loaded into the computer system 200 from other computer-readable media. Computer-readable storage media refers to any non-transitory tangible storage medium that provides recorded instructions and / or data to the computer system 200 for execution and / or processing. Examples of such storage media include floppy disks, magnetic tapes, CD-ROMs, DVDs, Blu-ray Discs, hard drives, ROMs or integrated circuits, USB memories, magneto-optical disks, or computer-readable cards such as PCMCIA cards, etc., regardless of whether these devices are inside or outside the computer module 201. Examples of temporary or non-tangible computer-readable transmission media that may also be involved in providing software, applications, instructions and / or video data or encoded video data to the computer module 401 include: radio or infrared transmission channels and network connections to other computers or networked devices, and the Internet or Intranet including e-mail transmissions and information recorded on websites.

[0051] The second portion of the above-mentioned application 233 and the corresponding code modules can be executed to implement one or more graphical user interfaces (GUIs) to be drawn or otherwise presented on the display 214. Users and applications of the computer system 200 can operate the interface in a functionally applicable manner by typically operating the keyboard 202 and the mouse 203 to provide control commands and / or input to the applications associated with these (one or more) GUIs. Other forms of user interfaces that are functionally applicable can also be implemented, such as audio interfaces that utilize voice prompts output via the speaker 217 and user voice commands input via the microphone 280, etc.

[0052] Figure 2B is a detailed schematic block diagram of the processor 205 and the "memory" 234. The memory 234 represents Figure 2A A logical aggregation of all memory modules (including HDD 209 and semiconductor memory 206) that can be accessed by computer module 201 in.

[0053] When the computer module 201 is initially powered on, a power-on self-test (POST) program 250 is executed. The POST program 250 is usually stored in Figure 2A 249 of semiconductor memory 206. Hardware devices such as ROM 249 storing software are sometimes referred to as firmware. POST program 250 checks the hardware within computer module 201 to ensure proper operation, and typically checks processor 205, memory 234 (209, 206), and basic input-output system software (BIOS) module 251, which is also typically stored in ROM 249, for correct operation. Once POST program 250 runs successfully, BIOS 251 starts Figure 2A The hard disk drive 210 is started so that the boot loader 252 residing on the hard disk drive 210 is executed via the processor 205. This loads the operating system 253 into the RAM memory 206, where the operating system 253 starts to work on the RAM memory 206. The operating system 253 is a system-level application executable by the processor 205 to implement various high-level functions including processor management, memory management, device management, storage management, software application interface, and general user interface.

[0054] The operating system 253 manages the memory 234 (209, 206) to ensure that each process or application running on the computer module 201 has sufficient memory to execute without conflicting with memory allocated to other processes. In addition, the appropriate use of Figure 2AThe different types of memory available in the computer system 200 are described so that each process can run efficiently. Therefore, the aggregate memory 234 is not intended to illustrate how to allocate specific segments of memory (unless otherwise specified), but rather to provide an overview of the memory accessible to the computer system 200 and how to use it.

[0055] like Figure 2B As shown, the processor 205 includes a plurality of functional modules, wherein the plurality of functional modules include a control unit 239, an arithmetic logic unit (ALU) 240, and a local or internal memory 248 sometimes referred to as a cache memory. The cache memory 248 typically includes a plurality of storage registers 244-246 in a register section. One or more internal buses 241 functionally interconnect these functional modules. The processor 205 also typically has one or more interfaces 242 for communicating with external devices via the system bus 204 using a connection 218. The memory 234 is connected to the bus 204 using a connection 219.

[0056] The application program 233 includes an instruction sequence 231 that may include conditional branch instructions and loop instructions. The program 233 may also include data 232 used when executing the program 233. The instructions 231 and data 232 are stored in memory locations 228, 229, 230 and 235, 236, 237, respectively. Depending on the relative size of the instructions 231 and the memory locations 228-230, a particular instruction may be stored in a single memory location, as described by the instruction shown in memory location 230. Alternatively, the instruction may be split into multiple parts, each stored in a separate memory location, as described by the instruction segments shown in memory locations 228 and 229.

[0057] Typically, the processor 205 is given a set of instructions, which are executed within the processor 205. The processor 205 awaits a subsequent input, which the processor 205 reacts to by executing another set of instructions. Each input may be provided from one or more of a plurality of sources, including data generated by one or more of the input devices 202, 203, data received from an external source via one of the networks 220, 202, data retrieved from one of the storage devices 206, 209, or data retrieved from a storage medium 225 inserted into a corresponding reader 212 (all of which are described in detail in the accompanying drawings). Figure 2A Execution of a set of instructions may result in outputting data in some cases. Execution may also involve storing data or variables to memory 234.

[0058] The video encoder 114, the video decoder 134, and the method may use input variables 254 stored in corresponding memory locations 255, 256, 257 within the memory 234. The video encoder 114, the video decoder 134, and the method produce output variables 261 stored in corresponding memory locations 262, 263, 264 within the memory 234. Intermediate variables 258 may be stored in memory locations 259, 260, 266, and 267.

[0059] refer to Figure 2B The processor 205, registers 244, 245, 246, arithmetic logic unit (ALU) 240 and control unit 239 work together to perform micro-operation sequences required to perform a "fetch, decode and execute" cycle for each instruction in the instruction set that constitutes the program 233. Each fetch, decode and execute cycle includes:

[0060] A fetch operation for fetching or reading an instruction 231 from a memory location 228 , 229 , 230 ;

[0061] a decode operation in which the control unit 239 determines which instruction was fetched; and

[0062] An execution operation, wherein in the execution operation, the control unit 239 and / or the ALU 240 executes the instruction.

[0063] Thereafter, further fetch, decode and execute cycles for the next instruction may be performed. Likewise, a store cycle may be performed, whereby the control unit 239 stores or writes a value to the memory location 232 .

[0064] To be explained Fig.10 and Fig.11 Each step or sub-processing in the method is associated with one or more segments of the program 233, and is typically performed by the register units 244, 245, 247, ALU 240 and control unit 239 in the processor 205 working together to perform fetch, decode and execute cycles for each instruction in the instruction set of the segment of the program 233.

[0065] Figure 3 is a schematic block diagram illustrating the functional modules of the video encoder 114 . Figure 4 1 is a schematic block diagram showing the functional modules of the video decoder 134. Typically, data is transferred between the functional modules within the video encoder 114 and the video decoder 134 in groups of samples or coefficients (such as partitioning of a block into fixed-size sub-blocks, etc.) or as an array. Figure 2A and 2BAs shown, the video encoder 114 and the video decoder 134 can be implemented using a general-purpose computer system 200, wherein various functional modules can be implemented using dedicated hardware within the computer system 200, using software executable within the computer system 200 (such as one or more software code modules of a software application 233 residing on a hard disk drive 205 and controlled by a processor 205 for execution, etc.). Alternatively, the video encoder 114 and the video decoder 134 can be implemented using a combination of dedicated hardware and software executable within the computer system 200. The video encoder 114, the video decoder 134 and the method can alternatively be implemented in dedicated hardware such as one or more integrated circuits that perform the functions or sub-functions of the method. Such dedicated hardware may include a graphics processing unit (GPU), a digital signal processor (DSP), an application-specific standard product (ASSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or one or more microprocessors and associated memory. In particular, the video encoder 114 includes modules 310 - 386 , and the video decoder 134 includes modules 420 - 496 , where each of these modules may be implemented as one or more software code modules of the software application 233 .

[0066] although Figure 3 The video encoder 114 is an example of a Versatile Video Coding (VVC) video encoding pipeline, but other video codecs may also be used to perform the processing stages described herein. The video encoder 114 receives captured frame data 113, such as a series of frames (each frame including one or more color channels). The frame data 113 may be in a 4:2:0 chroma format or a 4:2:2 chroma format. The block partitioner 310 first partitions the frame data 113 into CTUs, which are generally square in shape and are configured such that a specific size of the CTU is used. For example, the size of the CTU may be 64×64, 128×128, or 256×256 luma samples. The block partitioner 310 further partitions each CTU into one or more CBs corresponding to a luma coding tree or a chroma coding tree. CBs have various sizes and may include both square and non-square aspect ratios. Reference Fig.10 The operation of the block partitioner 310 is further described. However, in the VVC standard, CB, CU, PU, ​​and TU always have a side length that is a power of 2. Thus, the current CB (denoted as 312) is output from the block partitioner 310, thereby advancing according to the iteration of one or more blocks of the CTU, according to the luma coding tree and the chroma coding tree of the CTU. Figure 5 and 6 To further illustrate the options for partitioning a CTU into CBs.

[0067] The CTUs resulting from the first partitioning of the frame data 113 may be scanned in raster scan order and may be grouped into one or more "slices". A slice may be an "intra" (or "I") slice. An intra slice (I slice) indicates that each CU in the slice is intra predicted. Optionally, a slice may be uni-predicted or bi-predicted ("P" or "B" slices, respectively), indicating the additional availability of uni-prediction and bi-prediction in the slice, respectively.

[0068] For each CTU, the video encoder 114 operates in two stages. In the first stage (referred to as the "search" stage), the block partitioner 310 tests various potential configurations of the coding tree. Each potential configuration of the coding tree has an associated "candidate" CB. The first stage involves testing various candidate CBs to select a CB that provides high compression efficiency and low distortion. This test typically involves Lagrangian optimization, whereby the candidate CBs are evaluated based on a weighted combination of rate (coding cost) and distortion (error with respect to the input frame data 113). The "best" candidate CB (the CB with the lowest evaluated rate / distortion) is selected for subsequent encoding in the bitstream 115. The evaluation of the candidate CBs includes the following options: use the CB for a given area, or split the area according to various splitting options and use other CBs to encode each smaller resulting area or further split the area. As a result, both the CB and the coding tree itself are selected in the search stage.

[0069] The video encoder 114 generates a prediction block (PB) indicated by arrow 320 for each CB (e.g., CB 312). PB 320 is a prediction of the content of the associated CB 312. Subtractor module 322 generates a difference (or "residual," which means that the difference is in the spatial domain) between PB 320 and CB 312, represented as 324. Difference 324 is the block size difference between corresponding samples in PB 320 and CB 312. Difference 324 is transformed, quantized, and represented as a transform block (TB) indicated by arrow 336. PB 320 and associated TB 336 are typically selected from one of a plurality of possible candidate CBs (e.g., based on an evaluated cost or distortion).

[0070] A candidate coding block (CB) is a CB derived from one of the prediction modes available to the video encoder 114 for the associated PB and the resulting residual. Each candidate CB results in one or more corresponding TBs, as described below with reference to FIG8. TB 336 is a quantized and transformed representation of difference 324. When combined with a predicted PB in the video decoder 114, TB 336 reduces the difference between the decoded CB and the original CB 312 at the expense of additional signaling in the bitstream.

[0071] Thus, each candidate coding block (CB) (i.e., a combination of a prediction block (PB) and a transform block (TB)) has an associated coding cost (or "rate") and an associated difference (or "distortion"). The rate is typically measured in bits. The distortion of the CB is typically estimated as a difference in sample values, such as the sum of absolute differences (SAD) or the sum of squared differences (SSD), etc. The mode selector 386 uses the difference 324 to determine the estimate obtained from each candidate PB to determine the intra-frame prediction mode (indicated by arrow 388). The estimation of the coding cost associated with each candidate prediction mode and the corresponding residual encoding can be performed at a significantly lower cost than the entropy encoding of the residual. Therefore, multiple candidate modes can be evaluated to determine the best mode in terms of rate distortion.

[0072] Determining the best mode in terms of rate distortion is typically implemented using a variation of Lagrangian optimization. Selecting an intra-prediction mode 388 typically involves determining the coding cost of the residual data resulting from applying a particular intra-prediction mode. The coding cost can be approximated by using the "sum of absolute transform differences" (SATD), thereby obtaining an estimated transform residual cost using a relatively simple transform such as a Hadamard transform. In some implementations using relatively simple transforms, the cost obtained from the simplified estimation method is monotonically related to the actual cost that would otherwise be determined from a complete evaluation. In implementations with monotonically related estimated costs, the simplified estimation method can be used to make the same decision (i.e., intra-prediction mode), where the complexity of the video encoder 114 is reduced. In order to allow for possible non-monotonicity in the relationship between the estimated cost and the actual cost, the simplified estimation method can be used to generate a list of best candidates. For example, non-monotonicity can be obtained from other mode decisions that can be used for encoding of the residual data. The list of best candidates can have any number. The best candidates can be used to conduct a more complete search to establish the best mode selection for encoding the residual data of each candidate, thereby allowing the final selection of the intra-prediction mode and other mode decisions.

[0073] Other mode decisions include the ability to skip the forward transform (referred to as "transform skip"). Skipping a transform is appropriate for residual data that lacks sufficient correlation to reduce the coding cost via an expression that is a transform basis function. Certain types of content, such as relatively simple computer generated graphics, may exhibit similar behavior. For a "skipped transform", the residual coefficients are still encoded even though the transform itself is not performed.

[0074] A Lagrangian or similar optimization process may be employed for both the selection of the best partition of a CTU to a CB (using the block partitioner 310) and the selection of the best prediction mode from multiple possibilities. By applying the Lagrangian optimization process of the candidate modes in the mode selector module 386, the intra-prediction mode with the lowest cost measure is selected as the best mode. The lowest cost mode is the selected intra-prediction mode 388 and is also encoded in the bitstream 115 by the entropy encoder 338. The selection of the intra-prediction mode 388 by the operation of the mode selector module 386 is extended to the operation of the block partitioner 310. For example, the candidates for the selection of the intra-prediction mode 388 may include a mode applicable to a given block and additional modes applicable to multiple smaller blocks that are collectively co-located with the given block. In the case of including modes applicable to a given block and smaller co-located blocks, the process of selecting the candidates is implicitly also a process for determining the best hierarchical decomposition of the CTU to the CB.

[0075] In the second stage of the operation of the video encoder 114 (referred to as the "encoding" stage), iterations of the selected luma coding tree and the selected chroma coding tree (and thus the respective selected CBs) are performed in the video encoder 114. In the iterations, the CBs are encoded in the bitstream 115 as further described herein.

[0076] The entropy encoder 338 supports both variable length encoding of syntactic elements and arithmetic encoding of syntactic elements. Arithmetic coding is supported using context-adaptive binary arithmetic coding processing. The syntactic elements of arithmetic coding are composed of a sequence of one or more "bins (binary files)". Like a bit, the value of a bin is "0" or "1". However, a bin is not encoded in the bit stream 115 as a discrete bit. A bin has an associated prediction (or "likely" or "maximum probability") value and an associated probability (referred to as "context"). When the actual bin to be encoded matches the predicted value, the "maximum probability symbol" (MPS) is encoded. Encoding the maximum probability symbol is relatively cheap in terms of consuming bits. When the actual bin to be encoded does not match the possible value, the "minimum probability symbol" (LPS) is encoded. Encoding the minimum probability symbol has a relatively high cost in terms of consuming bits. The bin encoding technique enables bins with probability skews of "0" vs "1" to be efficiently encoded. For a syntactic element with two possible values ​​(i.e., "flag"), a single bin is sufficient. For syntax elements with many possible values, a sequence of bins is needed.

[0077] The presence of a later bin in the sequence may be determined based on the value of an earlier bin in the sequence. In addition, each bin may be associated with more than one context. A particular context may be selected based on, for example, an earlier bin in a syntax element and the bin values ​​of adjacent syntax elements (i.e., bin values ​​from adjacent blocks). Each time a context coding bin is encoded, the context selected for that bin (if any) is updated to reflect the new bin value. In this way, the binary arithmetic coding scheme is considered to be adaptive.

[0078] The video encoder 114 also supports bins lacking context ("bypass bins"). Bypass bins are encoded assuming an equal probability distribution between "0" and "1". Thus, each bin occupies one bit in the bitstream 115. The absence of context saves memory and reduces complexity, so bypass bins where the distribution of values ​​for a particular bin is not skewed are used. An example of an entropy encoder that employs context and adaptation is known in the art as CABAC (Context Adaptive Binary Arithmetic Coder), and many variations of this encoder are employed in video encoding.

[0079] The entropy encoder 338 encodes the intra-frame prediction mode 388 using a combination of context-coded and bypass-coded bins. Typically, a list of "maximum probability modes" is generated in the video encoder 114. The list of maximum probability modes is typically a fixed length such as three or six modes, and may include modes encountered in early blocks. The context-coded bin encodes a flag indicating whether the intra-frame prediction mode is one of the maximum probability modes. If the intra-frame prediction mode 388 is one of the maximum probability modes, further signaling is encoded using the bypass-coded bin. For example, further signaling after encoding uses a truncated unary bin string to indicate which maximum probability mode corresponds to the intra-frame prediction mode 388. Otherwise, the intra-frame prediction mode 388 is encoded as a "remaining mode". Encoding as a remaining mode uses an alternative syntax such as a fixed-length code, and is also encoded using a bypass-coded bin to represent an intra-frame prediction mode other than the intra-frame prediction mode present in the maximum probability mode list.

[0080] The multiplexer module 384 outputs the PB 320 according to the determined best intra prediction mode 388 selected from the tested prediction modes of the candidate CBs. The candidate prediction modes need not include every conceivable prediction mode supported by the video encoder 114.

[0081] Prediction modes are roughly divided into two categories. The first category is "intra-frame prediction" (also called "intra prediction"). In intra-frame prediction, a prediction of a block is generated, and the generation method can use other samples obtained from the current frame. For intra-frame prediction PB, it is possible to use different intra-frame prediction modes for luma and chroma, so intra-frame prediction is mainly described in terms of operation on PB.

[0082] The second category of prediction modes is "inter-frame prediction" (also called "inter prediction"). In inter prediction, samples from one or two frames that precede the current frame in the order of the coded frames in the bitstream are used to generate a prediction of the block. In addition, for inter prediction, a single coding tree is typically used for both the luminance channel and the chrominance channel. The order of the coded frames in the bitstream may be different from the order of the frames when captured or displayed. When one frame is used for prediction, the block is called "uni-predicted" and has two associated motion vectors. When two frames are used for prediction, the block is called "bi-predicted" and has two associated motion vectors. For P slices, each CU can be intra-predicted or uni-predicted. For B slices, each CU can be intra-predicted, uni-predicted, or bi-predicted. Frames are typically encoded using a "group of pictures" structure, thereby achieving a temporal hierarchy of frames. The temporal hierarchy of frames allows frames to reference previous and subsequent pictures in the order in which the frames are displayed. Images are encoded in the order necessary to ensure that the dependencies for decoding each frame are met.

[0083] A subcategory of inter prediction is called "skip mode". Inter prediction and skip mode are described as two different modes. However, both inter prediction mode and skip mode involve motion vectors that refer to blocks of samples from previous frames. Inter prediction involves encoding motion vector deltas to specify motion vectors relative to motion vector predictors. Motion vector predictors are obtained from a list of one or more candidate motion vectors selected using a "merge index". The encoded motion vector delta provides a spatial offset relative to the selected motion vector prediction. Inter prediction also uses the encoded residual in the bitstream 133. Skip mode uses only an index (also called a "merge index") to select one from several motion vector candidates. The selected candidate is used without any further signaling. Moreover, skip mode does not support encoding of any residual coefficients. The absence of encoded residual coefficients when using skip mode means that no transformation of skip mode is required. Therefore, skip mode does not usually cause pipeline processing problems. Pipeline processing problems may be the case for intra-predicted CUs and inter-predicted CUs. Due to the limited signaling of skip mode, skip mode is useful for achieving very high compression performance when relatively high-quality reference frames are available. A bi-predicted CU in a higher temporal layer of a random access group of pictures structure typically has high-quality reference pictures and motion vector candidates that accurately reflect the underlying motion.

[0084] Samples are selected based on motion vectors and reference picture indices. Motion vectors and reference picture indices apply to all color channels, so inter prediction is described primarily in terms of operations on PUs rather than PBs. Within each category (i.e., intra and inter prediction), different techniques can be applied to generate PUs. For example, intra prediction can use the value combination direction of adjacent rows and columns from previously reconstructed samples to generate PUs according to a specified filtering and generation process. Alternatively, a small number of parameters can be used to describe PUs. Inter prediction methods may vary in the number of motion parameters and their accuracy. Motion parameters typically include a reference frame index (which indicates which reference frames from a reference frame list will be used plus the spatial translation of each reference frame), but may include more frames, special frames, or complex affine parameters such as scaling and rotation. In addition, a predetermined motion refinement process can be applied to generate a dense motion estimate based on a block of reference samples.

[0085] In the case where the PB 320 is determined and selected and subtracted from the original sample block at the subtractor 322, a residual (denoted as 324) with the lowest coding cost is obtained and the residual is lossily compressed. The lossy compression process includes the steps of transform, quantization and entropy coding. The forward main transform module 326 applies a forward transform to the difference 324, thereby converting the difference 324 from the spatial threshold to the frequency domain and generating the main transform coefficients represented by arrow 328. The main transform coefficients 328 are passed to the forward secondary transform module 330 to generate the transform coefficients represented by arrow 332 by performing a non-separable secondary transform (NSST) operation. The forward main transform is usually separable, so that a group of rows and then a group of columns of each block are transformed, for example, horizontally for a block width of no more than 16 samples and vertically for a block height of no more than 16 samples. The transformation of each group of rows and columns is performed by first applying a one-dimensional transform to each row of the block to produce a partial result and then applying a one-dimensional transform to each column of the partial result to produce a final result. The forward secondary transform is usually a non-separable transform that is only applicable to the residual of the intra-frame predicted CU, but it may also be bypassed. The forward secondary transform operates on 16 samples (arranged as the upper left 4×4 sub-block of the main transform coefficients 328) or 64 samples (arranged as the upper left 8×8 coefficients, arranged as four 4×4 sub-blocks of the main transform coefficients 328). In addition, the matrix coefficients of the forward secondary transform are selected from multiple groups according to the intra-frame prediction mode of the CU, so that two groups of coefficients are available for use. Using a group of matrix coefficients or bypassing the forward secondary transform is signaled by using a truncated unary binarization coding to express the value of zero (no secondary transform applied), one (the selected first group of matrix coefficients) or two (the selected second group of matrix coefficients).

[0086] The transform coefficients 332 are passed to a quantizer module 334. At module 334, quantization according to a "quantization parameter" is performed to produce residual coefficients represented by arrow 336. The quantization parameter is constant for a given TB, resulting in uniform scaling of the generated residual coefficients for the TB. Non-uniform scaling can also be achieved by applying a "quantization matrix", whereby the scaling factor applied to each residual coefficient is derived from a combination of the quantization parameter and the corresponding entry in the scaling matrix whose size is generally equal to the size of the TB. The residual coefficients 336 are supplied to an entropy encoder 338 for encoding in the bitstream 115. Typically, the residual coefficients of each TB and at least one significant residual coefficient of the TU are scanned according to a scanning pattern to generate an ordered list of values. The scanning pattern typically scans the TB as a sequence of 4×4 "sub-blocks", thereby providing a conventional scanning operation at a granularity of 4×4 sets of residual coefficients, where the configuration of the sub-blocks depends on the size of the TB. In addition, the prediction mode 388 and the corresponding block partitions are also encoded in the bitstream 115.

[0087] As described above, the video encoder 114 needs access to a frame representation that corresponds to the frame representation seen in the video decoder 134. Thus, the residual coefficients 336 are also inverse quantized by the dequantizer module 340 to produce inverse transform coefficients represented by arrow 342. The inverse transform coefficients 342 pass through the inverse secondary transform module 344 to produce intermediate inverse transform coefficients represented by arrow 346. The intermediate inverse transform coefficients 346 are passed to the inverse main transform module 348 to produce residual samples of the TU represented by arrow 350. The type of inverse transform performed by the inverse secondary transform module 344 corresponds to the type of forward transform performed by the forward secondary transform module 330. The type of inverse transform performed by the inverse main transform module 348 corresponds to the type of main transform performed by the main transform module 326. The summation module 352 adds the residual samples 350 and the PU 320 to produce the reconstructed samples of the CU (indicated by arrow 354).

[0088] The reconstructed samples 354 are passed to the reference sample cache 356 and the in-loop filter module 368. The reference sample cache 356, which is typically implemented using static RAM on an ASIC (thus avoiding expensive off-chip memory accesses), provides the minimum sample storage required to satisfy the dependencies used to generate intra PBs for subsequent CUs in the frame. The minimum dependencies typically include a "line buffer" of samples along the bottom of a row of CTUs for use by the next row of CTUs and a column buffer with a range set by the height of the CTU. The reference sample cache 356 supplies reference samples (represented by arrow 358) to the reference sample filter 360. The sample filter 360 applies a smoothing operation to produce filtered reference samples (indicated by arrow 362). The filtered reference samples 362 are used by the intra prediction module 364 to produce an intra prediction block of samples represented by arrow 366. For each candidate intra prediction mode, the intra prediction module 364 produces a sample block, namely 366.

[0089] The in-loop filter module 368 applies several filtering stages to the reconstructed samples 354. The filtering stages include a "deblocking filter" (DBF), which applies smoothing aligned with CU boundaries to reduce artifacts caused by discontinuities. Another filtering stage present in the in-loop filter module 368 is an "adaptive loop filter" (ALF), which applies a Wiener-based adaptive filter to further reduce distortion. Another available filtering stage in the in-loop filter module 368 is a "sample adaptive offset" (SAO) filter. The SAO filter works by first classifying the reconstructed samples into one or more categories and applying an offset at the sample level according to the assigned category.

[0090] Filtered samples, represented by arrow 370, are output from the in-loop filter module 368. The filtered samples 370 are stored in a frame buffer 372. The frame buffer 372 typically has the capacity to store several (e.g., up to 16) pictures and is therefore stored in the memory 206. Due to the large memory consumption required, the frame buffer 372 is typically not stored using on-chip memory. As such, access to the frame buffer 372 is expensive in terms of memory bandwidth. The frame buffer 372 provides a reference frame (represented by arrow 374) to the motion estimation module 376 and the motion compensation module 380.

[0091] The motion estimation module 376 estimates a plurality of "motion vectors" (denoted as 378), each of which is a Cartesian spatial offset relative to the position of the current CB, thereby referencing a block in one of the reference frames in the frame buffer 372. A filtered block of reference samples (denoted as 382) is generated for each motion vector. The filtered reference samples 382 form a further candidate mode for potential selection by the mode selector 386. In addition, for a given CU, the PB 320 may be formed using one reference block ("uni-prediction"), or may be formed using two reference blocks ("bi-prediction"). For a selected motion vector, the motion compensation module 380 generates the PU 320 according to a filtering process that supports sub-pixel precision in the motion vector. In this way, the motion estimation module 376 (which operates on many candidate motion vectors) can perform a simplified filtering process compared to the motion compensation module 380 (which operates only on the selected candidate) to achieve reduced computational complexity.

[0092] Although the reference to Versatile Video Coding (VVC) describes Figure 3 The video encoder 114 of the embodiment of the present invention may be used, but other video coding standards or implementations may also use the processing stages of modules 310-386. The frame data 113 (and the bit stream 115) may also be read from the memory 206, the hard disk drive 210, the CD-ROM, the blue disc (Blue-ray diskTM) or other computer-readable storage medium (or written to the memory 206, the hard disk drive 210, the CD-ROM, the blue disc or other computer-readable storage medium). In addition, the frame data 113 (and the bit stream 115) may be received from an external source (such as a server or a radio frequency receiver connected to the communication network 220) (or sent to the external source).

[0093] exist Figure 4 The video decoder 134 is shown in FIG. Figure 4 The video decoder 134 of FIG. 1 is an example of a Versatile Video Coding (VVC) video decoding pipeline, but other video codecs may also be used to perform the processing stages described herein. Figure 4As shown, the bitstream 133 is input to the video decoder 134. The bitstream 133 can be read from the memory 206, the hard disk drive 210, the CD-ROM, the Blu-ray disc or other non-transitory computer-readable storage medium. Alternatively, the bitstream 133 can be received from an external source (such as a server or a radio frequency receiver connected to the communication network 220). The bitstream 133 contains the encoding syntax elements representing the captured frame data to be decoded.

[0094] The bitstream 133 is input to the entropy decoder module 420. The entropy decoder module 420 extracts syntax elements from the bitstream 133 by decoding a sequence of "bins" and passes the values ​​of the syntax elements to other modules in the video decoder 134. The entropy decoder module 420 uses an arithmetic decoding engine to decode each syntax element into a sequence of one or more bins. Each bin can use one or more "contexts", where the context describes the probability level of encoding "one" and "zero" values ​​for the bin. In the case where multiple contexts are available for a given bin, a "context modeling" or "context selection" step is performed to select one of the available contexts to decode the bin. The process of decoding the bin forms a sequential feedback loop. The number of operations in the feedback loop is preferably minimized so that the entropy decoder 420 can achieve a high throughput in bins / second. Context modeling depends on other characteristics of the bitstream known to the video decoder 134 when selecting the context, that is, the characteristics before the current bin. For example, the context can be selected based on the quadtree depth of the current CU in the coding tree. The dependency is preferably based on characteristics that are well known before decoding the bin, or determined without long sequential processing. The quadtree depth of the coding tree is an example of a context modeling dependency that is easy to know. The intra-frame prediction mode is an example of a context modeling dependency that is relatively difficult or computationally intensive to determine. The intra-frame prediction mode is encoded as an index to a "maximum probability mode" (MPM) list or an index to a "remaining mode" list, where the MPM and the remaining mode are selected according to the decoded "intra_luma_mpm_flag". When using MPM, decode the "intra_luma_mpm_idx" syntax element to select which of the maximum probability modes to use. There are usually six MPMs. When using the remaining mode, decode the "intra_luma_remainder" syntax element to select which of the remaining (non-MPM) modes to use. Determining both the maximum probability mode and the remaining mode requires a lot of operations and includes dependencies on the intra-frame prediction mode of neighboring blocks. For example, the neighboring blocks can be blocks above and to the left of the current block. Desirably, the context of the bin of each CU can be determined so that it can be parsed by the arithmetic coding engine without knowing the intra-frame prediction mode notified by the signal. Therefore, the feedback loop present in the arithmetic coding engine for sequential bin decoding avoids dependence on intra prediction mode. With a separate feedback loop resulting from the dependence of the MPM list construction on the intra prediction mode of the neighboring blocks, intra prediction mode determination can be postponed to a subsequent processing stage. Therefore, the arithmetic decoding engine of the entropy decoder module 420 is able to parse intra_luma_mpm_flag, intra_luma_mpm_idx, intra_luma_remainder without knowing the intra prediction mode of any earlier (e.g., neighboring) blocks.The entropy decoder module 420 applies an arithmetic coding algorithm, such as "context adaptive binary arithmetic coding" (CABAC), to decode syntax elements from the bitstream 133. The decoded syntax elements are used to reconstruct parameters within the video decoder 134. The parameters include residual coefficients (represented by arrow 424) and mode selection information (represented by arrow 458) such as intra-frame prediction modes. The mode selection information also includes information such as motion vectors, and the partitioning of each CTU into one or more CBs. The parameters are used to generate PBs, usually in combination with sample data from previously decoded CBs.

[0095] The residual coefficients 424 are input to a dequantizer module 428. The dequantizer module 428 inverse quantizes (or "scales") the residual coefficients 424 to create reconstructed intermediate transform coefficients according to the quantization parameters (indicated by arrow 432). Based on the decoded "nsst_index" syntax element, the reconstructed intermediate transform coefficients 432 are passed to an inverse secondary transform module 436 where a secondary transform is applied or no operation (bypass). Under execution of the processor 205, the entropy decoder 420 decodes "nsst_index" from the bitstream 133. As shown in FIG. Figure 3 As described above, "nsst_index" is decoded from the bitstream 133 as a truncated unary syntax element with a value of zero to two. The inverse secondary transform module 436 produces the reconstructed transform coefficients 440. If the use of a non-uniform inverse quantization matrix is ​​indicated in the bitstream 133, the video decoder 134 reads the quantization matrix from the bitstream 133 as a sequence of scaling factors and arranges the scaling factors into a matrix according to the inverse scaling. The inverse scaling uses the quantization matrix in combination with the quantization parameter to create the reconstructed intermediate transform coefficients 432.

[0096] The reconstructed transform coefficients 440 are passed to an inverse main transform module 444. Module 444 transforms the coefficients from the frequency domain back to the spatial domain. TB is effectively based on significant residual coefficients and non-significant residual coefficient values. The result of the operation of module 444 is a block of residual samples represented by arrow 448. The residual samples 448 are equal in size to the corresponding CU. The residual samples 448 are supplied to a summing module 450. At the summing module 450, the residual samples 448 are added to the decoded PB represented by 452 to produce a block of reconstructed samples represented by arrow 456. The reconstructed samples 456 are supplied to a reconstructed sample cache 460 and an in-loop filtering module 488. The in-loop filtering module 488 produces a reconstructed block of frame samples represented by 492. The frame samples 492 are written to a frame buffer 496.

[0097] The reconstructed sample cache 460 operates in a manner similar to the reconstructed sample cache 356 of the video encoder 114. The reconstructed sample cache 460 provides storage for the reconstructed samples needed for intra prediction of subsequent CBs without the memory 206 (e.g., by using data 232, which is typically on-chip memory, instead). Reference samples, represented by arrows 464, are obtained from the reconstructed sample cache 460 and are supplied to a reference sample filter 468 to produce filtered reference samples, represented by arrows 472. The filtered reference samples 472 are supplied to an intra prediction module 476. The module 476 produces a block of intra prediction samples, represented by arrows 480, based on the intra prediction mode parameters 458 represented in the bitstream 133 and decoded by the entropy decoder 420.

[0098] When the prediction mode of the CB is indicated in the bitstream 133 as intra prediction, the intra prediction samples 480 form the decoded PB 452 via the multiplexer module 484. Intra prediction produces a prediction block (PB) of samples, that is, a block in one color component derived using "neighboring samples" in the same color component. Neighboring samples are samples adjacent to the current block and have been reconstructed because they are in front in the block decoding order. In the case of juxtaposition of luminance and chrominance blocks, different intra prediction modes can be used for luminance and chrominance blocks. However, the two chrominance channels each share the same intra prediction mode. Intra prediction is divided into three types. "DC intra prediction" involves filling the PB with a single value representing the average of neighboring samples. "Plane intra prediction" involves filling the PB with samples corresponding to a plane, where the DC offset and vertical and horizontal gradients are derived from neighboring samples. "Angle intra prediction" involves filling the PB with neighboring samples filtered and propagated on the PB in a specific direction (or "angle"). In VVC, 65 angles are supported, with rectangular blocks being able to take advantage of additional angles not available to square blocks to produce a total of 87 angles. A fourth type of intra prediction is available for chroma PBs, whereby PBs are generated from juxtaposed luma reconstruction samples according to a "cross component linear model" (CCLM) mode. Three different CCLM modes are available, each using a different model derived from adjacent luma and chroma samples. The derived model is then used to generate a sample block of the chroma PB from the juxtaposed luma samples.

[0099] When the prediction mode of the CB is indicated in the bitstream 133 as intra prediction, the motion compensation module 434 uses the motion vector and the reference frame index to select and filter a block of samples from the frame buffer 496 to produce a block of inter-frame predicted samples represented as 438. The sample block 498 is obtained from a previously decoded frame stored in the frame buffer 496. In order to perform bi-prediction, two blocks of samples are generated and blended together to produce samples of the decoded PB 452. The frame buffer 496 is populated with filter block data 492 from the in-loop filtering module 488. As with the in-loop filtering module 368 of the video encoder 114, the in-loop filtering module 488 applies any, at least or all of the DBF, ALF and SAO filtering operations. Typically, the motion vector is applied to both the luma and chroma channels, but the filtering process for sub-sample interpolated luma and chroma channels is different. When a split in the coding tree results in a collection of relatively small luma blocks and the corresponding chroma regions are not partitioned into corresponding small chroma blocks, the sub-sample interpolated luma and chroma channels are respectively separated as shown in the reference tree. Fig.13 and 14 The blocks are encoded and decoded as described. In particular, if inter-frame prediction is used to predict any of the small luminance blocks, the inter-frame prediction operation is performed only on the luminance CB and not on any part of the corresponding chrominance CB. The in-loop filtering module 368 generates filtered block data 492 from the reconstructed samples 456.

[0100] Figure 5 5 is a schematic block diagram showing a set 500 of possible partitions or splits of a region into one or more sub-regions in a tree structure for general video coding. Figure 3 As described, the partitioning shown in set 500 may be utilized by block partitioner 310 of encoder 114 to partition each CTU into one or more CUs or CBs according to the encoding number as determined by Lagrangian optimization.

[0101] Although set 500 only illustrates partitioning of a square region into other possible non-square sub-regions, it should be understood that diagram 500 illustrates potential partitioning and does not require that the containing region be square. If the containing region is non-square, the size of the blocks resulting from the partitioning is scaled according to the aspect ratio of the containing block. Once a region is not split further, i.e., at a leaf node of the coding tree, a CU occupies the region. A specific sub-partition of a CTU into one or more CUs using block partitioner 310 is referred to as a "coding tree" for the CTU.

[0102] The process of sub-partitioning a region into sub-regions must terminate when the resulting sub-regions reach the minimum CU size. In addition to constraining the CU to prohibit block regions from being smaller than a predetermined minimum size of, for example, 16 samples, the CU is constrained to have a minimum width or height of four. Other minimum values ​​are also possible for both width and height or for both width or height. The sub-partitioning process can also be terminated before the deepest level of the decomposition, resulting in a CU larger than the minimum CU size. It is possible that no splitting occurs, resulting in a single CU occupying the entire CTU. A single CU occupying the entire CTU is the maximum available coding unit size. In addition, the CU for which no splitting occurs is larger than the processing region size. As a result of a binary or ternary split at the highest level of the coding tree, CU sizes such as 64×128, 128×64, 32×128 and 128×32 are possible, each of which is also larger than the processing region size. Reference Fig. 7A and Figure 7B An example of a CU that is larger than the processing region size is further described.Due to the use of a subsampled chroma format such as 4:2:0, etc., the arrangement of the video encoder 114 and the video decoder 134 may terminate the splitting of regions in the chroma channels earlier than in the luma channel.

[0103] In the absence of further sub-division, there is a CU at the leaf node of the coding tree. For example, leaf node 510 contains a CU. At the non-leaf nodes of the coding tree, there is a split to two or more other nodes, each of which may contain a leaf node (and thus a CU), or contain further splits to smaller areas. At each leaf node of the coding tree, there is a coding block for each color channel. A split that terminates at the same depth for both brightness and chrominance results in three juxtaposed CBs. A split that terminates at a deeper depth for brightness than for chrominance results in multiple brightness CBs juxtaposed with the CBs of the chrominance channels.

[0104] like Figure 5 As shown, quadtree split 512 splits the containing area into four equal-sized areas. Compared to HEVC, Versatile Video Coding (VVC) achieves additional flexibility by adding horizontal binary split 514 and vertical binary split 516. Splits 514 and 516 each split the containing area into two equal-sized areas. The splits are along horizontal boundaries (514) or vertical boundaries (516) within the containing block.

[0105] Further flexibility is achieved in general video coding by adding ternary horizontal split 518 and ternary vertical split 520. Ternary split 518 and 520 divide the block into three regions that form boundaries in the horizontal direction (518) or vertical direction (520) along 1 / 4 and 3 / 4 of the width or height of the containing region. The combination of quadtree, binary tree and ternary tree is called "QTBTTT". The root of the tree includes zero or more quadtree splits (the "QT" part of the tree). Once the QT part is terminated, zero or more binary or ternary splits ("multi-tree" or "MT" part of the tree) may occur, eventually ending in a CB or CU at the leaf node of the tree. In the case where the tree describes all color channels, the leaf node of the tree is a CU. In the case where the tree describes a luminance channel or a chrominance channel, the leaf node of the tree is a CB.

[0106] Compared to HEVC, which only supports quadtrees and therefore only supports square blocks, QTBTTT obtains more possible CU sizes, especially considering the possible recursive application of binary and / or ternary tree splits. The possibility of abnormal (non-square) block sizes can be reduced by constraining the splitting options to eliminate block widths or heights that will be less than four samples or splits that will not be a multiple of four samples. Typically, the constraints will apply when considering luma samples. However, in the described arrangement, the constraints can be applied separately to blocks of chroma channels. The application of the constraints of the splitting options to the chroma channels may result in different minimum block sizes for luma vs chroma (for example, when the frame data adopts a 4:2:0 chroma format or a 4:2:2 chroma format). Each split produces a sub-region with an unchanged, bisected or quartered side size relative to the containing area. Then, since the CTU size is a power of 2, the side sizes of all CUs are also powers of 2.

[0107] Figure 6 6 is a schematic flow diagram showing a data flow 600 of a QTBTTT (or "coding tree") structure used in general video coding. The QTBTTT structure is used for each CTU to define the partitioning of the CTU into one or more CUs. The QTBTTT structure for each CTU is determined by the block partitioner 310 in the video encoder 114 and encoded into the bitstream 115 or decoded from the bitstream 133 by the entropy decoder 420 in the video decoder 134. Figure 5 As shown partitioned, data flow 600 further features permitted combinations that may be used by block partitioner 310 to partition a CTU into one or more CUs.

[0108] Starting from the top level of the hierarchy, i.e., at the CTU, zero or more quadtree partitions are first performed. Specifically, a quadtree (QT) split decision 610 is made by the block partitioner 310. The decision at 610 returns a "1" symbol, which indicates a decision to split the current node into four child nodes according to the quadtree split 512. The result is that four new nodes are generated, such as at 620, and for each new node, recursively return to the QT split decision 610. Each new node is considered in raster (or Z-scan) order. Alternatively, if the QT split decision 610 indicates that no further splitting is to be performed (returns a "0" symbol), the quadtree partitioning stops, and multi-tree (MT) splitting is then considered.

[0109] First, an MT split decision 612 is made by the block partitioner 310. At 612, a decision to perform an MT split is indicated. A "0" symbol is returned at the decision 612, indicating that no further splitting of the node into child nodes will be performed. If no further splitting of the node will be performed, the node is a leaf node of the coding tree and corresponds to a CU. The leaf node is output at 622. Alternatively, if the MT split 612 indicates a decision to perform an MT split (a "1" symbol is returned), the block partitioner 310 proceeds to a direction decision 614.

[0110] Direction decision 614 indicates the direction of the MT split as horizontal ("H" or "0") or vertical ("V" or "1"). If decision 614 returns "0" indicating a horizontal direction, block partitioner 310 proceeds to decision 616. If decision 614 returns "1" indicating a vertical direction, block partitioner 310 proceeds to decision 618.

[0111] In each of decisions 616 and 618, the number of partitions into which the MT is split is indicated as two (binary split or "BT" nodes) or three (ternary split or "TT") in the case of a BT / TT split. That is, the BT / TT split decision 616 is made by the block partitioner 310 when the direction indicated from 614 is horizontal, and the BT / TT split decision 618 is made by the block partitioner 310 when the direction indicated from 614 is vertical.

[0112] The BT / TT split decision 616 indicates whether the horizontal split is a binary split 514 indicated by returning “0” or a ternary split 518 indicated by returning “1”. When the BT / TT split decision 616 indicates a binary split, at a generate HBT CTU node step 625, the block partitioner 310 generates two nodes according to the binary horizontal split 514. When the BT / TT split 616 indicates a ternary split, at a generate HTT CTU node step 626, the block partitioner 310 generates three nodes according to the ternary horizontal split 518.

[0113] The BT / TT split decision 618 indicates whether the vertical split is a binary split 516 indicated by returning "0" or a ternary split 520 indicated by returning "1". When the BT / TT split 618 indicates a binary split, at step 627 of generating VBT CTU nodes, the block partitioner 310 generates two nodes according to the vertical binary split 516. When the BT / TT split 618 indicates a ternary split, at step 628 of generating VTT CTU nodes, the block partitioner 310 generates three nodes according to the vertical ternary split 520. For each node obtained from steps 625-628, the recursion of the data flow 600 back to the MT split decision 612 is applied in a left-to-right or top-to-bottom order according to the direction 614. As a result, binary and ternary tree splits can be applied to generate CUs of various sizes.

[0114] refer to Fig. 9 The set of allowed and disallowed splits at various nodes of the coding tree are further described.

[0115] Fig. 7A and 7B An example partition 700 of a CTU 710 into multiple CUs or CBs is provided. Fig. 7A An example CU 712 is shown in FIG. Fig. 7A 7 shows the spatial arrangement of CUs in a CTU 710. Example partitioning 700 in Figure 7B Also shown in FIG. 7 is a coding tree 720 .

[0116] exist Fig. 7A At each non-leaf node (e.g., nodes 714, 716, and 718) in the CTU 710 of the coding tree 720, the contained nodes (which may be further partitioned or may be CUs) are scanned or traversed in "Z order" to create a list of nodes represented as columns in the coding tree 720. For quadtree splits, the Z order scan results in an order from top left to right followed by a bottom left to right order. For horizontal and vertical splits, the Z order scan (traversal) is simplified to a scan from top to bottom and a scan from left to right, respectively. Figure 7B The coding tree 720 lists all nodes and CUs according to the scanning order applied. Each split generates a list of two, three or four new nodes at the next level of the tree until a leaf node (CU) is reached.

[0117] In reference Figure 3Where the image is decomposed into CTUs and further into CUs using block partitioner 310, and the CUs are used to generate respective residual blocks (324), the residual blocks are forward transformed and quantized using video encoder 114. The resulting TBs 336 are then scanned to form a sequential list of residual coefficients as part of the operation of entropy encoding module 338. Equivalent processing is performed in video decoder 134 to obtain the TBs from bitstream 133.

[0118] Fig. 7A and 7B The example description applies to coding trees for both luma and chroma channels. However, Fig. 7A and 7B The examples also illustrate the behavior with respect to traversing a coding tree that applies only to luma channels or to a coding tree that applies only to chroma channels. For coding trees with many nested splits, the available splitting options at deeper levels are constrained by restrictions on the available block sizes for the corresponding small regions. Restrictions on the available block sizes for small regions are imposed to prevent the worst case of block processing rate from being so high as to impose an unreasonable burden on the implementation. In particular, the constraint that the block size will be a multiple of 16 (sixteen) samples in chroma enables the implementation to process samples at a granularity of 16 (sixteen) samples. Constraining the block size to be a multiple of sixteen samples is particularly relevant to the "intra-frame reconstruction" feedback loop, i.e., Figure 4Paths in the video decoder 134 involving modules 450, 460, 468, 476, and 484 and equivalent paths in the video encoder 114. In particular, constraining the block size to a multiple of 16 (sixteen) samples helps maintain throughput in intra-frame prediction mode. For example, "simultaneous data multiple instructions" (SIMD) microprocessor architectures typically operate on wide words that can contain 16 samples. In addition, the hardware architecture can use a wide bus (such as a bus with a width of 16 samples) to transmit samples along the intra-frame reconstruction feedback loop. If a smaller block size (e.g., four samples) is used, the bus will be underutilized, for example, only one-quarter of the bus width contains sample data. Although an underutilized bus can handle smaller blocks (i.e., less than sixteen samples), in a worst-case scenario, such as many or all blocks have a relatively small size, underutilization may result in preventing real-time operation of the encoder (114) or decoder (134). For inter-frame prediction, each block depends on reference samples obtained from a frame buffer (such as buffer 372 or 496, etc.). Since the frame buffer is filled with reference samples when processing the previous frame, there is no feedback dependency loop that affects the block-by-block operation for generating the inter-frame predicted block. In addition to the feedback dependency loop related to intra-frame reconstruction, there is an additional and concurrent feedback loop related to the determination of the intra-frame prediction mode 458. The intra-frame prediction mode 458 is determined by selecting a mode from the maximum probability mode list or selecting a mode from the remaining mode list. The determination of the maximum probability mode list and the remaining mode list requires the intra-frame prediction mode of the adjacent block. When using a relatively small block size, the maximum probability mode list and the remaining mode list need to be determined more frequently, that is, at a frequency dominated by the block size in samples and the sampling rate of the channel.

[0119] Fig. 8A , 8B and 8C provide a coding tree 820 ( Figure 8B) in which the chroma split is terminated before the luma split and a 4:2:0 chroma format is used. In the case where the chroma split is terminated, CB pairs are used, one CB for each chroma channel. For ease of illustration, the size of CTU 800 is 64×64 luma samples. CTU 800 is equivalent to a CTU size of 128×128 and a coding tree including one additional quadtree split. Quadtree splitting is applied to the 8×8 luma region 814. The 8×8 luma region 814 is split into four 4×4 luma CBs, however no splitting occurs in the chroma channels. Instead, pairs of chroma CBs of a predetermined minimum size (16 in the example) are used, one pair for each chroma channel. Chroma CB pairs typically have a minimum size corresponding to the minimum granularity of the number of samples that can be expected to be processed simultaneously. For example, many implementations of the video encoder 114 and the video decoder 134 will operate on sets of 16 samples, for example due to the use of correspondingly wide internal buses in the hardware implementation. In addition, each luma CB obtained by the splitting at least partially overlaps with the chroma CB pair, and the overall luma CB completely overlaps with the chroma CB pair. In the example of region 814, 4×4 chroma CB pairs are generated. Figure 8C An example is shown of how the resulting luma CB and chroma CB are related.

[0120] Referring back to reference 8A, a vertical binary split is applied to the 16×4 luma region 810. The 16×4 luma region 810 is split into two 8×4 luma CBs, however, no split occurs in the chroma channels, resulting in 8×2 chroma CB pairs. A vertical ternary split is applied to the 16×4 luma region 812. The 16×4 luma region 812 is split into 4×4, 4×8, and 4×4 luma CBs, however, no split occurs in the chroma channels, resulting in 8×2 chroma CB pairs. A horizontal binary split is applied to the 8×16 luma region 816. The 8×16 luma region 816 is split into 8×4, 8×8, and 8×4 luma CBs, however, no split occurs in the chroma channels, resulting in 4×8 chroma CB pairs. Therefore, the chroma CBs are at least 16 samples in the region.

[0121] Figure 8CA portion of CTU 800 is shown with three color planes shown in a "decomposed" (or separated) manner to illustrate different block structures in different planes. A luma sample plane 850, a first chroma sample plane 852, and a second chroma sample plane 854 are shown. When the "YCbCr" color space is used, the luma sample plane 850 contains Y samples of the image frame, the first chroma sample plane 852 contains Cb samples of the image frame, and the second chroma sample plane 854 contains Cr samples of the image frame. Using the 4:2:0 chroma format results in the first chroma sample plane 852 and the second chroma sample plane 854 having half the sample density in the horizontal and vertical directions relative to the luma sample plane 850. Therefore, the CB size of the chroma block in the sample is generally half the size of the corresponding luma CB. That is, for the 4:2:0 chroma format, the chroma CB width and height are each half of the juxtaposed luma CB. For the 4:2:2 chroma format, the chroma CB height is half of the collocated luma CB, and the width is the same as the width of the collocated luma CB. For clarity, only the parent split in the coding tree of the 8×16 luma region 816 is shown, and the split is shown only in the luma sample plane 850. When the chroma split is terminated, multiple luma CBs are collocated with the chroma CB pairs. For example, the coding tree of CTU 800 includes a horizontal ternary split applied to the 8×16 luma region 816. The horizontal ternary split results in an 8×4 luma CB 860, an 8×8 luma CB 862, and an 8×4 luma CB 864 present in the luma sample plane 850. Since the 8×16 luma region 816 corresponds to a region of 4×8 chroma samples in the chroma sample planes (852 and 854), the ternary split of the coding tree is not applied to the chroma sample planes (852 and 854). Thus, the region of 4×8 chroma samples forms a leaf node for chroma, resulting in a chroma CB pair, i.e., a chroma CB 866 for the first chroma sample plane 852 and a chroma CB 868 for the second chroma sample plane 854. In the example where the horizontal ternary split is applied only in the luma plane, a minimum chroma CB size of 32 samples is achieved. Other example luma regions (810, 812, and 814) result in a minimum chroma CB size of 16, which corresponds to the minimum luma block size and the desired granularity of sample processing.

[0122] Fig. 9 A set 900 of transform block sizes and associated scan patterns for chroma channels resulting from using a 4:2:0 chroma format is shown. Set 900 may also be used for a 4:2:2 chroma format. The described arrangement is suitable for use with image frames having a chroma format in which the chroma channels of the image frame are subsampled relative to the luma channels of the image frame, particularly for 4:2:0 and 4:2:2 formats. Set 900 does not include all possible chroma transform block sizes. Fig. 9Only chroma transform blocks having a width less than or equal to sixteen or a height less than or equal to eight are shown in FIG. Chroma transform blocks having larger widths and heights may occur, but for ease of reference, Fig. 9 Not shown in FIG.

[0123] A set of prohibited transform sizes 910 includes transform block sizes 2×2, 2×4, and 4×2, all of which have an area of ​​less than sixteen samples. Fig. 9 In the example of , the minimum transform size of 16 (sixteen) chroma samples results from the operation of the described arrangement, in particular for intra prediction CB. Fig.10 The splitting options are determined as described above to avoid an example of prohibiting transform sizes 910. The residual coefficients in the transform are scanned in a two-layer manner, where the transform is divided into "sub-blocks" (or "coefficient groups"). The scan is performed along a scan path from the last significant (non-zero) coefficient back toward the DC (upper left) coefficient. The scan path is defined as the progression within each sub-block (the "lower layer") and the progression from one sub-block to the next sub-block (the "upper layer"). In set 900, 8×2 TB 920 uses 8×2 sub-blocks, i.e., sub-blocks containing sixteen residual coefficients. 2×8 TB 922 uses 2×8 sub-blocks, i.e., also contains sixteen residual coefficients.

[0124] TBs with a width or height of 2 and the other dimension being a multiple of 8 use multiple 2×8 or 8×2 subblocks. Thus, in some cases chroma blocks with a width of two samples are encoded using block-to-subblock partitioning, with subblocks each being 2×8 samples in size, and in some instances chroma blocks with a height of two samples are encoded using block-to-subblock partitioning, with subblocks each being 8×2 samples in size. For example, a 16×2 TB 916 has two 8×2 subblocks, each of which is scanned as shown for TB 920. The scan progresses from one subblock to the next as shown in subblock progression 917.

[0125] 2×32TB( Fig. 9 922) uses four 2x8 sub-blocks arranged in a one-by-four array. The residual coefficients in each sub-block are scanned as shown for the 2x8 TB 922, where the sub-blocks proceed from the lowest sub-block in the one-by-four array upwards to the highest sub-block.

[0126] Larger TBs follow a similar scanning progression. For all TBs whose width and height are each greater than or equal to four, a 4×4 subblock scan is used. For example, a 4×8 TB 923 uses a 4×4 subblock scan 924 with a progression from the lower subblock to the upper subblock. A 4×4 TB 925 can be scanned in a similar manner. An 8×8 TB 929 uses a progression 930 for four 4×4 subblocks. In all cases, the scan within a subblock and the progression from subblock to subblock follows a reverse diagonal scan, i.e., the scan proceeds from the "last" significant residual coefficient of the TB back toward the upper left residual coefficient. Fig. 9 Also shown is a scan order across, for example, 8×4 TB 932, 16×4 TB 934, and 16×8 TB 936. In addition, depending on the position of the last significant coefficient along the scan path, only the portion of the sub-block containing the last significant residual coefficient from the last significant coefficient position of the sub-block to the upper left residual coefficient need be scanned. Sub-blocks further along the scan path in the forward direction (i.e., closer to the lower right of the block) do not need to be scanned. Set 900 and specifically the prohibited transform size 910 imposes restrictions on the ability to split a region (or node) of a coding tree in chroma into sub-regions (or sub-nodes), as described with reference to FIG. Fig.10 described.

[0127] In a VVC system using 2×2, 2×4, and 4×2 TBs (a set of TB 910), a 2×2 sub-block may be used for a TB with a width and / or height of two samples. As described above, the use of TB 910 increases the throughput constraint in the intra-frame reconstruction feedback dependency loop. In addition, the use of sub-blocks with only four coefficients increases the difficulty of parsing residual coefficients at higher throughputs. In particular, for each sub-block, a "valid map" indicates the validity of each residual coefficient contained. A 1-value validity flag is encoded, which establishes the magnitude of the residual coefficient as at least one, and a zero-value flag is encoded, which establishes the magnitude of the residual coefficient as zero. The residual coefficient magnitude (from one forward) and sign are encoded only for "valid" residual coefficients. For the DC coefficient, no valid bits are encoded and the magnitude (from zero) is always encoded. High-throughput encoders and decoders may need to encode or decode multiple valid map bins per clock cycle to maintain real-time operation. When there are many dependencies between bins (eg, when using smaller subblock sizes), the difficulty of multi-bin encoding and decoding per cycle increases. In system 100, the subblock size is 16 (although the subblock containing the last significant coefficient is excluded), regardless of the block size.

[0128] Fig.10 A rule set 1000 is shown for generating a list of allowed splits in a chroma coding tree. Other frames may allow a mix of inter-predicted blocks and intra-predicted blocks. Although reference has been made to Figure 6The complete set of available splits of the coding tree is described, but the restrictions on the available transform sizes for a given region size impose constraints on the specific splitting options. As described below, the splitting options for the luma channel and the chroma channel are each determined according to the size of the region of the corresponding coding tree unit.

[0129] The rules 1020 for the chroma regions show the allowed splits for the different regions. Even though the chroma channels are under consideration, the allowed splits of the rules 1020 are expressed in units of luma samples, since different chroma formats may be used.

[0130] When traversing the nodes of the coding tree, a list of allowed splits for chroma is obtained by checking the availability of a set of split options using the region size of the coding tree. The split options for the regions that can be coded using CB are added to the list of allowed splits. For regions using CB coding, the region size must enable encoding using an integer number of transforms of a specific size from set 900. The specific size is selected to be the maximum size that does not exceed the region size (considering both width and height). In this way, for smaller regions, a single transform is used. In the case where the region size exceeds the size of the largest available transform, the largest available transform is spliced ​​to occupy the entire region.

[0131] When considering a node in the coding tree with a given area (expressed in luma samples), the ability to perform a given type of split is determined based on the split type and the area of ​​the chroma region. Fig.10 As shown, each split option is tested against the region size to determine whether the split option will result in a sub-region of a prohibited size. Split options that result in a sub-region of an allowed size are considered to allow chroma splitting 1070.

[0132] For example, as shown in rule 1021a of the chroma region, in the QT mode (corresponding to Figure 6 In the case of decision 610 of chroma rule set 1020, if the region has a size of 8×8 in 4:2:0 format or a size of 8×8 in 4:2:2 format, then a quadtree split is not allowed because the split will result in a transform size of 2×2 or 2×4 for the chroma channels, respectively. The allowable region sizes are indicated by arrow 1021. Similarly, other allowable splits of chroma rule set 1020 are indicated by arrows 1022, 1023, 1024, 1025, and 1026 and are described below with respect to Fig.13 and 14 Arrows 1021 , 1022 , 1023 , 1024 , 1025 , and 1026 each reference an allowed chroma split list 1070 .

[0133] Describes the region size for the chroma channels in terms of the luma sample grid. For example, when using the 4:2:0 chroma format, an 8×4 region corresponds to a 4×2 transform for the chroma channels. When using the 4:2:2 chroma format, an 8×4 region corresponds to a 4×4 transform in chroma. When using the 4:4:4 chroma format, chroma is not subsampled relative to luma, so the transform sizes in chroma correspond to the region sizes.

[0134] The permissible split options are described below in Fig.13 and 14 Further description.

[0135] Fig.11 A method 1100 for encoding a coding tree of an image frame in a video bitstream is shown. The method 1100 may be implemented by a device such as a configured FPGA, ASIC, or ASSP. In addition, the method 1100 may be performed by the video encoder 114 under execution of the processor 205. Thus, the method 1100 may be stored on a computer-readable storage medium and / or stored in the memory 206. The method 1100 begins at a determine chroma format step 1105.

[0136] At determine chroma format step 1105, processor 205 determines the chroma format of frame data 113 as one of a 4:2:0 chroma format or a 4:2:2 chroma format. The chroma format is a property of the frame data and does not change during operation of method 1100. Method 1100 continues from step 1105 to segment the frame into CTUs step 1110 under control of processor 205.

[0137] At step 1110 of partitioning the frame into CTUs, the block partitioner 310, under execution of the processor 205, partitions the current frame of the frame data 113 into an array of CTUs. Encoding of the CTUs resulting from the partitioning begins. Control in the processor proceeds from step 1110 to step 1120 of determining a coding tree.

[0138] At step 1120 of determining the coding tree, under the execution of the processor 205, the video encoder 114 combines and tests various prediction modes and splitting options to arrive at the coding tree of the CTU. The prediction modes and residual coefficients of the various CUs of the coding tree of the CTU are also derived. Typically, Lagrangian optimization is performed to select the best coding tree and CU of the CTU. When the evaluation uses inter-frame prediction, a motion vector is selected from a set of candidate motion vectors. Candidate motion vectors are generated according to a search pattern. When the extracted reference block of the candidate motion vector under evaluation is subjected to distortion testing, the application of prohibited chroma splitting in the coding tree is considered. When splitting is prohibited in chroma and splitting is allowed in luminance, the resulting luminance CB can use inter-frame prediction. Motion compensation is applied only to the luminance channel, so the distortion calculation takes into account the luminance distortion without considering the chroma distortion. Since motion compensation is not performed in the chroma channel when the chroma split is prohibited, the chroma distortion is not considered. For chroma, the distortion caused by the considered intra-frame prediction mode and the encoded chroma TB (if any) is considered. When considering both luma and chroma, the inter prediction search may first select a motion vector based on luma distortion and then "refine" the motion vector by also considering chroma distortion. Refinement typically takes into account small changes in motion vector values, such as sub-pixel displacements, etc. When chroma splitting is disabled and evaluation is made for inter prediction of small luma blocks, chroma refinement is not required. Control in the processor 205 proceeds from step 1120 to step 1130 where the coding tree is encoded.

[0139] At the encode coding tree step 1130, under execution of the processor 205, the video encoder 114 performs the method 1300 described with respect to Figure 1300 to encode the coding tree of the current CTU in the bitstream 115. Step 1130 is performed to encode the current CTU in the bitstream. Control in the processor 205 proceeds from step 1130 to the last CTU test step 1140.

[0140] At the last CTU test step 1140, the processor 205 tests whether the current CTU is the last CTU in the slice or frame. If not ("No" at step 1140), the video encoder 114 advances to the next CTU in the frame, and control in the processor 205 advances from step 1140 back to step 1120 to continue processing the remaining CTUs in the frame. If the CTU is the last one in the frame or slice, step 1140 returns "Yes" and the method 1100 terminates. As a result of the method 1100, the entire image frame is encoded in the bitstream as a sequence of CTUs.

[0141] Fig.12A method 1200 for decoding a coding tree of an image frame from a video bitstream is shown. The method 1200 may be implemented by a device such as a configured FPGA, ASIC, or ASSP. In addition, the method 1200 may be performed by the video decoder 134 under execution of the processor 205. Thus, the method 1200 may be stored on a computer-readable storage medium and / or stored in the memory 206. The method 1200 begins at a determine chroma format step 1205.

[0142] At the determine chroma format step 1205, the processor 205 determines the chroma format of the frame data 113 as one of a 4:2:0 chroma format or a 4:2:2 chroma format. The chroma format is a property of the frame data and does not change during the operation of the method 1200. The video decoder 134 can determine the chroma format with the help of a profile of the bitstream 133. The profile defines a set of coding tools that can be used by a specific bitstream 133, and can limit the chroma format to a specific value, such as 4:2:0, etc. The profile is determined by decoding, for example, a "profile_idc" syntax element from the bitstream 133, or by decoding one or more constraint flags from the bitstream 133, wherein each constraint flag constrains the use of a specific tool in the bitstream 133. In the case where the chroma format is not fully specified by the profile, other syntax such as "chroma_format_idc" can be decoded to determine the chroma format. The method 1200 continues from step 1205 to step 1210 of segmenting the frame into CTUs under the execution of the processor 205.

[0143] At the step 1210 of partitioning the frame into CTUs, the video decoder 134, under execution of the processor 205, determines the partitioning of the current frame of the frame data 133 to be decoded into a CTU array. Decoding of the CTUs resulting from the determined partitioning is started. Control in the processor proceeds from step 1210 to step 1220 of decoding the coding tree.

[0144] At a decode coding tree step 1220, under execution of the processor 205, the video decoder 134 performs the method 1400 for the current CTU to decode the coding tree of the current CTU from the bitstream 133. The current CTU is a CTU selected from the CTUs obtained by performing step 1210. Control in the processor 205 proceeds from step 1220 to a last CTU test step 1240.

[0145] At the last CTU test step 1240, the processor 205 tests whether the current CTU is the last one in the slice or frame. If not ("No" at step 1240), the video decoder 134 advances to the next CTU in the frame, and control in the processor 205 advances from step 1240 back to step 1220 to continue decoding CTUs from the bitstream. If the CTU is the last one in the frame or slice, step 1240 returns "Yes" and the method 1200 terminates.

[0146] Fig.13 A method 1300 for encoding a coding tree of an image frame in a video bitstream is shown. The method 1300 may be implemented by a device such as a configured FPGA, ASIC, or ASSP. In addition, the method 1300 may be performed by a video encoder 114 under the execution of a processor 205. Therefore, the method 1300 may be stored on a computer-readable storage medium and / or stored in a memory 206. The method 1300 causes the block to be encoded in the bitstream 115 so that each block is the smallest minimum area. The described arrangement uses samples of a predetermined minimum size. The minimum size used in the described example is 16 samples, which is preferred in some hardware and software implementations. However, different minimum sizes may still be used. For example, it is possible to process a corresponding minimum block area of ​​32 or 64 samples with a granularity of 32 or 64 and 32 or 64 samples, respectively. In both hardware and software implementations, encoding a block with a minimum area is advantageous for implementation feasibility. For software implementations, the minimum area of ​​16 samples is aligned with a typical single instruction multiple data (SIMD) instruction set (such as AVX-2 and SSE4, etc.). The method 1300 , initially invoked at the root node of the coding tree for the current CTU, begins at a coding split mode step 1310 .

[0147] At the encode split mode step 1310, the entropy encoder 338, under execution by the processor 205, encodes the split mode at the current node of the coding tree in the bitstream 115. The split mode is referenced Figure 5 One of the splitting modes described, and the step of encoding the splitting mode only allows possible splitting to be encoded. For example, quadtree splitting 512 is only possible at the root node of the coding tree or under other quadtree splitting in the coding tree. As shown in group 910, it is forbidden to obtain a splitting of the brightness CB with a width or height less than four samples. For example, based on rule set 1010, other constraints on the maximum depth of binary and / or ternary splitting may also be effective. Control in processor 205 advances from step 1310 to no split test step 1320.

[0148] At the no split test step 1320, the processor 205 tests whether the current split is "no split" (i.e., 510). If the current split is no split 510 ("yes" at step 1320), the control in the processor 205 proceeds from step 1320 to the encoding CU step 1330. Otherwise, if the current split is not 510 ("no" at step 1320), the control in the processor 205 proceeds to the chroma split prohibition test step 1340.

[0149] At the encode CU step 1330, under execution of the processor 205, the entropy encoder 338 encodes the prediction mode of the CU and the residual of the CU in the bitstream 115. When each leaf node of the coding tree is reached at step 1330, the method 1300 terminates upon completion of step 1330, returning to the parent call in the coding tree traversal. Once all nodes of the coding tree have been traversed, the entire CTU is encoded in the bitstream 115, and control returns to the method 1100 to advance to the next CTU in the image frame.

[0150] At the chroma split prohibition test step 1340, the processor 205 Fig.10 The chroma region 1020 split rule set of the coding tree is used to determine whether the split of the current node in the coding tree is allowed to be applied to the chroma channels according to step 1310. If the current node in the coding tree covers a luma region of 128 luma samples (32×4 or 4×32 or 16×8 or 8×16), then ternary splits in the corresponding chroma region (16×2, 2×16, 8×4, 4×8 chroma samples, respectively) are prohibited, as shown in the rule set 1020. If ternary splits are allowed, the resulting block sizes will include prohibited block sizes (e.g., 2×4 or 4×2). When the current node in the coding tree covers a luma region of 64 luma samples, binary, ternary, and quadtree splits are prohibited, as shown in the rule set 1020. Implementing binary, ternary, and quadtree splits for a luma region of 64 luma samples will result in prohibited chroma block sizes (2×2, 2×4, 4×2). If the split is not prohibited (i.e., the split is an allowed chroma split of list 1070), then step 1340 returns "no" and control in processor 205 proceeds from step 1340 to perform luma and chroma split step 1350. Otherwise, if the split is prohibited ("yes" at 1340), then control in processor 205 proceeds to perform luma split step 13100.

[0151] At the luma and chroma splitting step 1350, the processor 205 applies the splitting to divide the current region associated with the current node of the coding tree into sub-regions associated with the child nodes of the coding tree. Figure 5 and 6From step 1350 , control in processor 205 proceeds to select region step 1360 .

[0152] At a select region step 1360, the processor selects one of the sub-regions resulting from step 1350. The sub-regions are selected based on a Z-order scan of the region. In subsequent iterations of step 1360, the selection progresses through the sub-regions. Control in the processor 205 proceeds from step 1360 to a code tree step 1370.

[0153] At step 1370 of encoding the coding tree, the processor 205 recursively calls method 1300 for the selected region obtained in step 1360. Step 1370 further operates to encode the luma and chroma blocks and the associated prediction modes and residual coefficients into the bitstream for each region. Control in the processor 205 passes from step 1370 to a last region test step 1380.

[0154] At the last region test step 1380, the processor 205 tests whether the selected region as selected in step 1360 is the last of the regions resulting from the split mode segmentation as implemented in step 1350. If the region is not the last region ("No" at step 1380), control in the processor 205 proceeds from step 1380 to step 1360 to continue progressing through the split regions. Otherwise, step 1380 returns "Yes", the method 1300 terminates, and control in the processor 205 proceeds to the parent call of the method 1300.

[0155] At the luma splitting step 13100, the splitting pattern as encoded at step 1310 is performed by the processor 205 only in the luma channel. As a result, the current node of the coding tree is divided into multiple luma CBs according to the splitting pattern. Only chroma CB pairs are generated, i.e., one chroma CB for each chroma channel. Each resulting luma CB partially overlaps (juxtaposes) with the chroma CB pairs and the luma CBs obtained collectively. The overall luma CB accurately covers the area of ​​the chroma CB pairs, with the area of ​​the chroma CB pairs. In addition, the minimum area of ​​each luma CB and chroma CB is the minimum size, such as 16 samples.

[0156] Steps 13100 and 1350 each operate to determine the size of the chroma coding blocks for the chroma channels Cb and Cr. At step 1350, the chroma coding block size for the chroma channels is determined based on the split mode determined at step 1310. At step 13100, the chroma coding block size for the chroma channels is determined based on a predetermined minimum chroma block size. As described above, step 1350 is implemented based on the chroma splitting prohibited for the coding tree unit. Fig.10As shown in rule set 1020 , the allowed splits and corresponding chroma coding block sizes are determined based on the chroma format determined in step 1105 .

[0157] Control in processor 205 proceeds from step 13100 to select brightness CB step 13110.

[0158] At Select Luma CB Step 13110, the processor 205 selects the next luma CB of the CBs resulting from Step 13100. Method 13100 initially selects the first CB, the top left luma CB of the CBs resulting from the luma split. Upon subsequent calls to Step 13110, each "next" luma CB is selected based on a Z-order scan of the luma CBs resulting from Step 13100. Control in the processor 205 proceeds from Step 13110 to an Encode Luma CB Step 13120.

[0159] At the encoding luma CB step 13120, under the execution of the processor 205, the entropy encoder 338 encodes the selected luma CB in the bitstream 115. Typically, the prediction mode and residual coefficients are encoded for the selected luma CB. The prediction mode for luma CB encoding can use inter-frame prediction or intra-frame prediction. For example, "cu_skip_flag" is encoded to indicate the use of inter-frame prediction without any residual, otherwise "pred_mode_flag" and optional "pred_mode_ibc_flag" are encoded to indicate the use of intra-frame prediction, inter-frame prediction or intra-block copying, each with an optional residual coefficient. When residuals may exist, the "cu_cbf" flag signals that there is at least one valid (non-zero) residual coefficient in any TB of the CB. When the CB is indicated to use inter-frame prediction, the associated motion vector is only applicable to the luma CB. That is, the motion vector should not be used to generate any PB associated with any partially collocated chroma CB. When a CB is indicated to use intra-block copying, the associated block vector is associated only with the luma CB and not with any partially collocated chroma CB. Control in the processor 205 passes from step 13120 to a final luma CB test step 13130.

[0160] At a last luma CB test step 13130, the processor 205 tests whether the luma CB selected at step 13110 is the last luma CB based on the Z-order iteration of the split luma CBs performed at step 13100. If the selected luma CB is not the last one ("No" at step 13130), control in the processor 205 proceeds from step 13130 ​​to step 13120. Otherwise, step 13130 ​​returns "Yes" and control in the processor 205 proceeds to a determine chroma intra prediction mode step 13140.

[0161] At the determination of the chroma intra prediction mode 13140, under the execution of the processor 205, the video encoder 114 determines the intra prediction mode of the chroma CB pair juxtaposed with the luma CB of step 13100. Step 13140 efficiently determines to encode the chroma block using intra prediction. It is determined whether the area occupied by the chroma CB is further split into multiple luma CBs in the luma channel. The size of the chroma block of the channel is a predetermined minimum value (e.g., 16 samples) determined by the operation of step 1350. Even if the corresponding luma CB is encoded using inter prediction at step 13120, the intra prediction mode of the chroma CB pair is determined. In one configuration, a single prediction mode, such as DC intra prediction, is applied to each chroma CB. The use of a single prediction mode allows the mode to be determined by means of prohibition of splitting chroma (a "yes" result at step 1340), and does not involve additional searches to determine which mode of multiple possible modes to use. In addition, for this case, the bitstream 115 does not require additional signaling, that is, no additional "intra_chroma_pred_mode" syntax element needs to be encoded. However, when chroma splitting has been prohibited ("yes" at step 1340), higher compression performance is achieved by including the "intra_chroma_pred_mode" syntax element in the bitstream 115, signaling one of several possible intra-frame prediction modes. The video encoder 114 determines which intra-frame prediction mode to use. The intra-frame prediction mode is generally determined based on considerations of the encoding cost compared to the distortion. However, higher compression performance is generally obtained compared to using a single intra-frame prediction mode for such a chroma CB. Control in the processor 205 proceeds from step 13140 to the encoding chroma CB step 13150.

[0162] At the encode chroma CB step 13150, under execution of the processor 205, the entropy encoder 338 encodes the intra prediction mode of the chroma CB in the bitstream 115 using the "intra_chroma_pred_mode" syntax element when multiple intra prediction modes are available. When one intra prediction mode (e.g., DC intra prediction) is possible, the "intra_chroma_pred_mode" is not encoded in the bitstream 115. Available intra prediction modes for chroma intra prediction may include DC, planar, and the following angular prediction modes: horizontal, vertical, top right diagonal. The available intra prediction modes may also include a "direct mode" (DM_CHROMA), whereby the chroma intra prediction mode is obtained from the juxtaposed luma CB (typically the bottom and rightmost of the luma CB obtained by step 13100). When "cross component linear model" intra prediction is available, the chroma CB can be predicted from samples from the luma CB. The residual coefficients of the chroma TB associated with the chroma CB may also be encoded in the bitstream 115, as shown in reference to Fig.14 Once step 13150 is executed by processor 205, method 1300 terminates and control in processor 205 returns to the parent caller of method 1300.

[0163] Fig.14 A method 1400 of decoding a coding tree of an image frame from a video bitstream implemented in step 1220 of method 1200 is shown. Method 1400 may be implemented by a device such as a configured FPGA, ASIC, or ASSP. In addition, method 1400 may be performed by video decoder 134 under execution of processor 205. Therefore, method 1400 may be stored on a computer-readable storage medium and / or stored in memory 206. Method 1400 is obtained as follows: blocks are decoded from bitstream 133 so that each block is not less than a minimum area, such as 16 samples, which is advantageous for implementation feasibility in both hardware and software cases. For the software case, the minimum area of ​​16 samples is aligned with a typical single instruction multiple data (SIMD) instruction set (such as AVX-2 and SSE4, etc.). Method 1400, initially called at the root node of the coding tree of the current CTU, starts at decoding split mode step 1410.

[0164] At decode split mode step 1410, under execution of processor 205, entropy decoder 420 decodes the split mode at the current node of the coding tree from bitstream 133. The split mode is referenced Figure 5One of the splits described and the method for encoding the splitting mode only allows the allowed splitting to be encoded, that is, even when the split is prohibited in the chrominance channel, it is also permitted in the luminance channel. For example, quadtree splitting 512 is only possible at the root node of the coding tree or under other quadtree splits in the coding tree. The splitting of the luminance CB with a width or height less than four samples is prohibited. Therefore, the minimum luminance CB size is 16 samples. Other constraints on the maximum depth of binary and / or ternary splitting may also be effective. The control in the processor 205 advances to the unsplit test step 1420 from step 1410.

[0165] At the no split test step 1420, the processor 205 tests whether the current split is "no split" (i.e., 510). If the current split is no split 510 ("yes" at 1420), control in the processor 205 proceeds from step 1420 to a decode CU step 1430. Otherwise, step 1420 returns "no", and control in the processor 205 proceeds to a chroma split prohibit test step 1440.

[0166] At decode CU step 1430, under execution of processor 205, entropy decoder 420 decodes the prediction mode of the CU of bitstream 115 and the residual coefficient of the CU. Step 1430 operates to decode the coding unit using the residual coefficient and prediction mode determined from the bitstream by entropy decoder 420. When step 1430 reaches each leaf node of the coding tree, method 1400 terminates at step 1430 completion, returning to the parent call in the coding tree traversal. Once all nodes of the coding tree have been traversed, the entire CTU is decoded from the bitstream 133, and control returns to method 1200, advancing to the next CTU in the image frame.

[0167] At the chroma split prohibition test step 1440, the processor 205 Fig.10The chroma region 1020 splitting rule set is used to determine whether the splitting of the current node in the coding tree is allowed to be applied to the chroma channel according to step 1410. Step 1440 determines whether the split test is prohibited in a manner similar to step 1340 of method 1300. The operation of step 1440 prevents the occurrence of prohibited block sizes. When the chroma region is already at a minimum size (e.g., 16 chroma samples), no further splitting of any type is allowed because the resulting area will be less than the minimum allowed. When the chroma region size is 32 samples and the corresponding split is a ternary split (whether horizontal or vertical ternary split), further splitting is also not allowed to avoid chroma blocks of 8 chroma samples. If the split is not prohibited (i.e., the split is allowed), step 1450 returns "no" and control in the processor 205 proceeds from step 1440 to the step 1450 of performing luminance and chroma splitting. Otherwise, if splitting is prohibited ("YES" at step 1450), control in processor 205 proceeds to step 14100 where the chroma intra prediction mode is determined.

[0168] At the luma and chroma splitting step 1450, the processor 205 applies the splitting to split the current region associated with the current node of the coding tree into sub-regions associated with the sub-nodes of the coding tree. Figure 5 and 6 Apply the split as described.

[0169] Steps 14100 and 1450 each operate to determine the size of the chroma coding blocks for the chroma channels Cb and Cr. At step 1450, the chroma coding block sizes for the chroma channels are determined based on the split mode decoded at step 1410. At step 14100, the chroma coding block sizes for the chroma channels are determined based on a predetermined minimum chroma block size. As described above, step 1450 is implemented based on a chroma split that is prohibited for the coding tree unit, which corresponds to a minimum chroma CB size of 16 (and 32 in the case of a ternary split of 128 samples in the luma region). Fig.10 As shown in rule set 1020, based on the chroma format determined at step 1205, the allowable splits and the corresponding chroma coding block sizes are determined.

[0170] Control in processor 205 proceeds from step 1450 to select area step 1460 .

[0171] At select region step 1460, processor 205 selects one of the sub-regions obtained from step 1450 according to a Z-order scan of the region. Step 1460 operates to progress the selection through the sub-regions in subsequent iterations. Control in processor 205 passes from step 1460 to decode the coding tree step 1470.

[0172] At a decode coding tree step 1470, the processor 205 recursively calls method 1400 for the selected region resulting from the operation of step 1460. Step 1470 further operates to decode each region of the coding tree using the residual coefficients and prediction mode determined from the bitstream. Control in the processor 205 proceeds from step 1470 to a last region test step 1480.

[0173] At the last region test step 1480, the processor 205 tests whether the selected region preselected in the last iteration of step 1460 is the last of the regions resulting from the split mode segmentation implemented at step 1450. If the region is not the last region ("No" at step 1480), control in the processor 205 proceeds from step 1480 to step 1460 to continue to advance through the split regions. Otherwise, step 1480 returns "Yes", the method 1400 terminates, and control in the processor 205 proceeds to the parent call of the method 1400.

[0174] At the luma splitting step 14100, the splitting pattern decoded at step 1410 is performed only in the luma channel by the processor 205. As a result, the current node of the coding tree is divided into a plurality of luma CBs according to the splitting pattern. Step 14100 operates to generate only chroma CB pairs, i.e., one chroma CB for each chroma channel. Each of the obtained luma CBs partially overlaps (at least partially juxtaposed) with the chroma CB pairs, and collectively the luma CBs completely overlap with the chroma CB pairs. In addition, the minimum area of ​​each luma CB and chroma CB is 16 samples. Control in the processor 205 proceeds from step 14100 to select luma CB step 14110.

[0175] At Select Luma CB Step 14110, the processor 205 selects the next luma CB in the CBs obtained from Step 14100. The selection of the next luma CB begins with the first CB, which is the top left luma CB in the CBs obtained from the luma split. Upon subsequent calls to Step 14110, each "next" luma CB is selected based on a Z-order scan of the luma CBs obtained from Step 14100. Control in the processor 205 proceeds from Step 14110 to a Decode Luma CB Step 14120.

[0176] At the decode luma CB step 14120, under the execution of the processor 205, the entropy decoder 420 decodes the selected luma CB from the bitstream 115. Typically, the prediction mode and residual are decoded for the selected luma CB. For example, "cu_skip_flag" is decoded to indicate that inter-frame prediction is used without any residual, otherwise "pred_mode_flag" and optional "pred_mode_ibc_flag" are decoded to indicate the use of intra-frame prediction, inter-frame prediction or intra-block copying, each with an optional residual coefficient. When residuals may be present, the "cu_cbf" flag signals that there is at least one valid (non-zero) residual coefficient in any TB of the CB. When the CB is indicated to use inter-frame prediction, the associated motion vector is only applicable to the luma CB, that is, the motion vector is not applied to generate any PB associated with any partially collocated chroma CB. When the CB is indicated to use intra-block copying, the associated block vector is associated only with the luma CB, and not with any partially collocated chroma CB. Control in processor 205 proceeds from step 14120 to a final brightness CB test step 14130.

[0177] At a last luma CB test step 14130, the processor 205 tests whether the luma CB selected at step 14110 is the last luma CB based on the Z-order iteration of the split luma CBs performed at step 14100. If the selected luma CB is not the last, control in the processor 205 proceeds from step 14130 to step 14110. Otherwise, control in the processor 205 proceeds to a determine chroma intra prediction mode step 14140.

[0178] At determine chroma intra prediction mode 14140, under execution of the processor 205, the video decoder 134 determines the intra prediction mode of the chroma CB pair juxtaposed with the luma CB of step 14100. If, as determined by the operation of step 1440, the chroma block is the result of stopping the splitting of the chroma coding tree while the splitting of the luma coding tree occurs, then step 14140 effectively determines that the chroma block has been encoded using intra prediction and will therefore be decoded using intra prediction. The intra prediction mode of the chroma CB pair is determined even if the corresponding luma CB is decoded using inter prediction at step 14120. In one arrangement, a single prediction mode is applied to each chroma CB, such as DC intra prediction. The use of a single prediction mode allows the mode to be determined by means of the prohibition of splitting chroma (a "yes" result at step 1440), and does not involve additional searches to determine which mode of multiple possible modes to use. Furthermore, for this case, the bitstream 134 does not require additional signaling, i.e., no additional "intra_chroma_pred_mode" syntax element needs to be encoded. However, when chroma splitting has been disabled ("yes" at step 1440), higher compression performance is achieved by including the "intra_chroma_pred_mode" syntax element in the bitstream 134 to signal one of several possible intra-prediction modes. The video decoder 134 needs to use the entropy decoder 420 to determine which intra-prediction mode to use to decode the "intra_chroma_pred_mode" syntax element from the bitstream 134. Control in the processor 205 proceeds from step 14140 to a decode chroma CB step 14150.

[0179] At the decode chroma CB step 14150, under the execution of the processor 205, the entropy decoder 420 generally determines the intra prediction mode of the chroma CB from the bitstream 420 according to the decoded "intra_chroma_pred_mode" syntax element. When multiple intra prediction modes are available, "intra_chroma_pred_mode" is decoded. When only one intra prediction mode is available (e.g., DC intra prediction), the mode is inferred without decoding additional syntax elements from the bitstream 133. The available intra prediction modes for chroma intra prediction may include DC, plane, and the following angular prediction modes: horizontal, vertical, and upper right diagonal. The available intra prediction modes may also include a "direct mode" (DM_CHROMA), whereby the chroma intra prediction mode is obtained from the juxtaposed luma CB (usually the bottom and rightmost of the luma CB obtained from step 14100). When "cross component linear model" intra prediction is available, the chroma CB can be predicted from samples from the luma CB. For a chroma CB pair, the "cu_cbf" flag signals the presence of at least one significant residual coefficient in either of the chroma CB pairs. If there is at least one significant residual coefficient in either of the chroma CB pairs, "tu_cbf_cb" and "tu_cbf_cr" signal the presence of at least one significant coefficient in the chroma CBs for the Cb and Cr channels, respectively. For a chroma CB with at least one significant residual coefficient, the "residual_coding" sequence of syntax elements is decoded to determine the residual coefficient of the corresponding chroma CB. The residual coding syntax encodes the residual coefficients as a sequence of values ​​that fills the transform block from the last significant coefficient position to the upper left ("DC") coefficient position according to a backward diagonal scan. The backward diagonal scan performs the scanning of the transform block as a sequence of "sub-blocks" (usually of size 4×4, but sizes 2×2, 2×4, 2×8, 8×2, 4×2 are also possible) (or "coefficient groups"). The scanning within each coefficient group is in the backward diagonal direction, and the scanning from one sub-block to the next sub-block is also in the backward diagonal direction. Once step 14150 is performed by processor 205, method 1400 terminates and control in processor 205 returns to the parent call of method 1400.

[0180] The coding tree approach of method 1300 and method 1400, in which a minimum block area of ​​16 samples is maintained for video data in 4:2:0 chroma format, facilitates high throughput in both software and hardware. In addition, the restriction of inter-frame prediction of luma CBs to small CB sizes reduces this worst-case memory bandwidth of motion compensation memory bandwidth by avoiding the need to also extract samples for generating motion compensated chroma CBs. In particular, if the minimum chroma CB size is 2×2, and additional samples are required to provide filtering support for sub-sample interpolation of chroma CBs, a significant increase in memory bandwidth will be seen compared to inter-frame prediction in the luma channel only for small block sizes. The coding gain of motion compensation occurs substantially in the luma channel, so the omission of small blocks that are also motion compensated achieves a reduction in memory bandwidth, thereby having relatively little impact on coding performance. In addition, the reduction in memory bandwidth contributes to the feasibility of performing motion compensation on 4×4 luma CBs and achieving the resulting coding gain.

[0181] In one arrangement of the video encoder 114 and the video decoder 134, more than one luma split may occur in the coding tree starting from the point at which the chroma split of the coding tree terminates. For example, an 8×16 luma region is not split in the chroma channel, resulting in a 4×8 chroma CB pair. In the luma channel, the 8×16 luma region is first split using a horizontal ternary split, and then one of the resulting luma CBs is further split. For example, the resulting 8×4 luma CB is vertically binary split into two 4×4 luma CBs. An arrangement in which there is more than one luma split in the coding tree starting from the point at which the chroma split of the coding tree terminates re-calls methods 1300 and 1400 in the video encoder 114 and the video decoder 134, respectively, within the chroma split prohibited region, with the modification that no other chroma CBs are required in subsequent calls. In the calls of methods 1300 and 1400 that create chroma CB pairs, the entire chroma region is covered by the created chroma CBs, so the recursive calls of methods 1300 and 1400 do not require the creation of additional chroma CBs.

[0182] Fig.15A set 1500 of transform block partitions of an intra-predicted coding unit is shown. A luma CB may be partitioned into one luma TB of the same size (“ISP_NO_SPLIT”). A luma CB of size 4×4 has an area of ​​16 samples and is not further partitioned, resulting in one luma TB of size also 4×4. A luma CB with an area of ​​32 samples may be partitioned into two partitions. For example, an 8×4 luma CB 1510 may be partitioned horizontally (“ISP_HOR_SPLIT”) into two 8×2 luma TBs 1520 or vertically (“ISP_VER_SPLIT”) into two 4×4 luma TBs 1530. If the luma CB 1510 is a 4×8 luma CB, the block may be partitioned horizontally at 1520 into two 4×4 luma TBs or vertically at 1530 into two 2×8 luma TBs.

[0183] A luma CB of an area 64 or more samples is partitioned into one partition or into four partitions. A luma CB 1550 of width W and height H of an area of ​​64 or more samples may be partitioned horizontally into four luma TBs 1560 of size W×(H / 4), or may be partitioned vertically into four (W / 4)×H luma TBs 1570. As shown in set 1500, the luma CB is split into multiple partitions, which results in more and smaller luma TBs. Intra prediction is performed to generate PBs for each luma TB, and intra reconstruction processing is performed from one partition to the next within the luma CB.

[0184] Fig.16 A method 1600 is shown for encoding coding units of an image frame into a video bitstream 115. The method 1600 may be implemented by a device such as a configured FPGA, ASIC, or ASSP. Additionally, the method 1600 may be performed by the video encoder 114 under execution by the processor 205. Thus, the method 1600 may be stored on a computer-readable storage medium and / or in the memory 206. The method 1600 causes blocks to be encoded into the bitstream 115 such that the coefficient group size is determined based solely on the transform block size, and no further distinction is made between luma and chroma channels. Since entropy coding is a key feedback loop in the video encoder 114, it is advantageous to reduce the memory accesses or computations required for coefficient group size determination. The method 1600 is called for each coding unit in the coding tree (i.e., in Fig.13 The method 1600 of (called at step 1330 of ) starts at the encoding pred_mode step 1610. As described above, when step 1320 determines that the current split is no split 510, step 1330 is executed.

[0185] At an encode pred_mode step 1610, the entropy encoder 338, under execution by the processor 205, encodes the prediction mode of the CU into the bitstream 115. Control in the processor 205 passes from step 1610 to an intra prediction test step 1620.

[0186] At an intra prediction test step 1620, the processor 205 tests the prediction mode of the CU. If the prediction mode is intra prediction ("yes" at step 1620), control in the processor 205 proceeds from step 1620 to an encode intra sub-partition mode step 1650. Otherwise, if the prediction mode is not intra prediction ("no" at step 1620), control in the processor 205 proceeds from step 1620 to an encode merge flag and index step 1630.

[0187] At the encode merge flag and index step 1630, under the execution of the processor 205, the entropy encoder 338 encodes the merge flag into the bitstream 115 to signal that "merge mode" is used (or not used) for inter-frame prediction. The merge mode causes the motion vector of the CU to be obtained from the spatial (or temporal) neighboring blocks in the set of candidate spatial (or temporal) neighboring blocks. If the merge mode is used, a candidate is selected using the corresponding "merge index". The merge index is encoded in the bitstream 115 together with the merge flag. If "motion vector prediction" is used, similar encoding is performed, thereby using a flag to signal one of several possible candidate motion vectors as a predictor. Control in the processor 205 proceeds from step 1630 to the encode motion vector increment step 1640.

[0188] At an encode motion vector delta step 1640, the entropy encoder 338, under execution by the processor 205, encodes the motion vector delta into the bitstream 115. Step 1640 is performed when motion vector prediction is used for the CU. The motion vector delta specifies the delta between the motion vector predictor encoded at step 1630 and the motion vector to be used for motion compensation. Control in the processor 205 proceeds from step 1640 to an encoded residual test step 1660. If motion vector prediction is not used for the CU, step 1640 is not implemented and the method 1600 proceeds directly to step 1660.

[0189] At the encode intra subpartition mode step 1650, under execution of the processor 205, the entropy encoder 338 encodes the decision of whether to use intra subpartitioning into the bitstream 115 using the context-coded "intra_subpartitions_mode_flag" syntax element. When the luma CB size is larger than the minimum luma transform block size (i.e., larger than 16 luma samples), intra subpartitioning can be used for the luma channel. Intra subpartitioning splits the coding unit into multiple luma transform blocks, as shown in set 1500. If the luma CB is partitioned into multiple TBs, "intra_subpartitions_split_flag" signals whether the splitting of the luma CB into multiple luma TBs occurs horizontally or vertically. Together, "intra_subpartitions_mode_flag" and "intra_subpartitions_split_flag" encode three possible partitions enumerated as "ISP_NO_SPLIT", "ISP_HOR_SPLIT" and "ISP_VER_SPLIT". From step 1650 , control in processor 205 proceeds to an encoded residual test step 1660 .

[0190] At the coded residual test step 1660, the processor 205 determines whether at least one residual coefficient in any transform block of the coding block is valid. The determination includes all luma TBs resulting from the application of intra sub-partitioning and chroma TB pairs associated with two chroma channels. If at least one residual coefficient in any luma TB and chroma TB is valid, the entropy encoder 338, under the execution of the processor 205, arithmetic encodes "one" for the "cu_cbf" syntax element, step 1660 returns "yes" and the processor 205 proceeds to determine the luma coefficient group size step 1670. If there is no valid residual coefficient in any TB of the CU, step 1660 returns "no", arithmetic encodes "zero" for cu_cbf and the method 1600 terminates, where the processor 205 proceeds to the next CU in the CTU.

[0191] At a determine luma coefficient group size step 1670, the processor determines the coefficient group size of one or more luma TBs (transform blocks) associated with the CU. If intra subpartitioning is not in use, there is one luma TB. If intra subpartitioning is in use, there are two or four luma TBs. The size of the luma TB depends on whether the intra subpartitioning is done horizontally or vertically and the number of luma TBs, and accordingly depends on the luma CU size, as shown in set 1500.

[0192] The coefficient group size is determined using the luma TB width and height, as shown in Table 1 below. Table 1 shows a transform block (TB) size to coefficient group mapping table for luma and chroma channels (with coefficient groups of the same size for the TB regardless of whether the TB is for the luma or chroma channel). The TB width and height are powers of 2, and therefore Table 1 considers the log2 of the TB width and height, i.e., "log2TBwidth" and "log2TBheight" form the first two indices in the three dimensions of Table 1. The last dimension of the table distinguishes the width and height of the coefficient group. The coefficient group dimensions are stored as log2width and log2height. For example, a TB of size 16×16 results in indexing (4, 4) in Table 1, which returns (2, 2), indicating that the coefficient group size is 4×4. A TB of size (2×32) results in indexing (1, 5) in Table 1, which returns (1, 3), indicating that the coefficient group size is 2×8. The minimum area of ​​a luma TB is 16 samples, and thus the case where log2width+log2height is less than four in Table 1 is not accessed. In the case where intra sub-partitioning is used for a CU, each luma TB has the same size, and thus coefficient group size determination for the luma TB is performed once for the CU.

[0193] Table 2 below shows a mapping of transform block (TB) sizes to coefficient group sizes for luma and chroma channels (with different coefficient group sizes for the same size TB in luma compared to chroma). If Table 2 were to be used, an additional dimension would be required, i.e., distinguishing luma from chroma, and the table size would be doubled compared to Table 1. The coefficient group sizes defined as in Table 1 are such that the size is the largest possible size that fits the TB width and height while not exceeding 16 samples in a region. Table 1 provides a set of coefficient group sizes from which to select the coefficient group size. Within the constraints of the TB width and height, the selected coefficient group aspect ratio of width vs. height is kept as close to 1:1 as possible. Control in processor 205 proceeds from step 1670 to encode luma TB step 1680.

[0194] At the encode luma TB step 1680, under the execution of the processor 205, the entropy encoder 338 encodes the residual coefficients of one or more luma TBs of the CU into the bitstream 115. The determined coefficient group size of step 1670 is used for each luma TB. For each luma TB, the coded block flag is encoded into the bitstream 115, indicating that there is at least one significant coefficient in the luma TB. If there is at least one significant coefficient in the luma TB, the last significant position is encoded into the bitstream. The last significant position is defined as the last significant coefficient along the scan path from the DC (upper left) coefficient of the TB to the lower right coefficient. The scan path is defined as a diagonal scan within the array of non-overlapping sub-blocks into which the TB is divided, each sub-block is the size of the coefficient group and occupies the entire TB. The progress from one sub-block to the next sub-block in the scan order also follows the diagonal scan. For each coefficient group except the upper left coefficient group and the coefficient group containing the last significant coefficient, the entropy encoder 338 encodes the "coded sub-block flag". The coded sub-block flag indicates that there is at least one significant residual coefficient in the sub-block. If there are no significant residual coefficients in the sub-block, the diagonal scan of the residual coefficients in the TB skips the sub-block. If there is at least one significant residual coefficient in the sub-block, all positions within the sub-block are scanned and the magnitude of each residual coefficient is encoded and the sign of each significant residual coefficient is encoded. Control in processor 205 proceeds from step 1680 to determine chroma coefficient group size step 1690.

[0195] At a determine chroma coefficient group size step 1690, the processor 205 determines the coefficient group size for the chroma transform block pair associated with the CU. One chroma CB for each chroma channel is associated with the CU, regardless of whether the luma CB is split into multiple luma TBs. As shown in Table 1, the chroma TB width and height are used to determine the coefficient group size. The TB width and height are powers of 2, and therefore Table 1 considers the log2 of the TB width and height, that is, "log2TBwidth" and "log2TBheight" form the first two indexes in the three dimensions of Table 1. The last dimension of the table distinguishes the width and height of the coefficient group. The coefficient group dimensions are stored as log2width and log2height. For example, a TB of size 16×16 results in indexing (4, 4) in Table 1, which returns (2, 2), indicating that the coefficient group size is 4×4. A TB of size (2×32) results in indexing (1, 5) in Table 1, which returns (1, 3), indicating that the coefficient group size is 2×8. Each chroma TB has the same size, and thus the coefficient group sizing for the chroma TB pair is done once for the CU. If Table 2 were to be used, an additional dimension would be required, namely to distinguish between luma and chroma, and the table size would be twice that of Table 1.

[0196] As described with respect to steps 1670 and 1690, the coefficient group size is determined based solely on the transform block size, and no further distinction is made between the luma channel and the chroma channels. The coefficient group size is therefore determined independently of whether the chroma format is 4:2:2 or 4:2:0. As described with respect to Table 1, the coefficient group size is based on the maximum area of ​​the coefficient group of up to 16 samples. Step 1690 operates to determine the coefficient group size for the TB regardless of the subsampling in the color plane (Y or Cb or Cr) of the transform block or in the color plane (applicable to Cb and Cr channels) due to the chroma format. Table 1 is used in both steps 1670 and 1690. Thus, a single table is used for the transform block belonging to the luma plane and each chroma color plane. Control in the processor 205 proceeds from step 1690 to the encode chroma TB step 16100.

[0197] At the encode chroma TB step 16100, under execution by the processor 205, the entropy encoder 338 encodes the residual coefficients of the chroma TB pair of the CU into the bitstream 115. The determined coefficient group size of step 1690 is used for the chroma TB pair. For each chroma TB, a coded block flag is encoded into the bitstream 115, indicating that there is at least one significant coefficient in the chroma TB. As described with reference to step 1680, the remainder of the encoding steps for each chroma TB conforms to the encoding process for the luma TB. Upon execution of step 16100, the method 1600 terminates and control in the processor 205 proceeds to the next CU in the CTU.

[0198] Fig.17 A method 1700 is shown for decoding coding units of an image frame from a video bitstream 133. The method 1700 may be implemented by a device such as a configured FPGA, ASIC, or ASSP. Additionally, the method 1700 may be performed by the video decoder 134 under execution by the processor 205. Thus, the method 1700 may be stored on a computer-readable storage medium and / or in the memory 206. The method 1700 enables decoding of blocks from the bitstream 133 such that the coefficient group size is determined based solely on the transform block size and no further distinction is made between the luma channel and the chroma channels. Since entropy decoding is a key feedback loop in the video decoder 134, it is advantageous to reduce the memory accesses or computations required for coefficient group size determination. The method 1700 is called for each coding unit in the coding tree (i.e., in Fig.14 ) method 1700 is called at step 1430. As described above, if the current split is no split 510, step 1430 is performed. Method 1700 begins with a decoding pred_mode step 1710.

[0199] At a decode pred_mode step 1710, the entropy decoder 420, under execution by the processor 205, decodes the prediction mode of the CU from the bitstream 133. Control in the processor 205 passes from step 1710 to an intra prediction test step 1720.

[0200] At an intra prediction test step 1720, the processor 205 tests the prediction mode of the CU (as decoded at step 1710). If the prediction mode is intra prediction, step 1720 returns "yes" and control in the processor 205 proceeds from step 1720 to a decode intra sub-partition mode step 1750. Otherwise, if not intra prediction, step 1720 returns "no" and control in the processor 205 proceeds from step 1720 to a decode merge flag and index step 1730.

[0201] At decode merge flag and index step 1730, under execution of processor 205, entropy decoder 420 decodes the merge flag from bitstream 133, thereby signaling whether "merge mode" is used for inter-frame prediction in the bitstream. Merge mode causes the motion vector of the CU to be obtained from the spatial (or temporal) neighboring blocks in a set of candidate spatial or temporal neighboring blocks. If merge mode is used, a candidate is selected using the "merge index" (also decoded from bitstream 133). If "motion vector prediction" is used, similar decoding is performed, thereby signaling one of several possible candidate motion vectors as a predictor through a flag in the bitstream. Control in processor 205 proceeds from step 1730 to decode motion vector increment step 1740.

[0202] At a decode motion vector delta step 1740, the entropy decoder 420, under execution by the processor 205, decodes the motion vector delta from the bitstream 133. Step 1740 is performed when motion vector prediction is used for the CU. The motion vector delta specifies the delta between the motion vector predictor encoded at step 1730 and the motion vector to be used for motion compensation. Control in the processor 205 proceeds from step 1740 to an encoded residual test step 1760. If motion vector prediction is not used for the CU, step 1740 is not implemented and control in the processor 205 proceeds directly to step 1760.

[0203] At the decode intra subpartition mode step 1750, the entropy decoder 420, under execution by the processor 205, uses the context-coded "intra_subpartitions_mode_flag" syntax element to decode from the bitstream 133 the decision whether to use intra subpartitioning. Intra subpartitioning can be used for the luma channel when the luma CB size is larger than the minimum luma transform block size (i.e., larger than 16 luma samples). Intra subpartitioning splits the coding unit into multiple luma transform blocks, as shown in set 1500. If the luma CB is partitioned into multiple TBs, "intra_subpartitions_split_flag" signals whether the splitting of the luma CB into multiple luma TBs occurs horizontally or vertically. Together, "intra_subpartitions_mode_flag" and "intra_subpartitions_split_flag" encode three possible partitions enumerated as "ISP_NO_SPLIT", "ISP_HOR_SPLIT", and "ISP_VER_SPLIT". From step 1750 , control in processor 205 passes to an encoded residual test step 1760 .

[0204] At the coded residual test step 1760, the processor 205 determines whether at least one residual coefficient in any transform block of the coding block is valid. This determination includes all luma TBs resulting from the application of intra sub-partitioning and chroma TB pairs associated with two chroma channels. Under the execution of the processor 205, the entropy encoder 420 arithmetically decodes the "cu_cbf" syntax element, and the processor 205 determines whether at least one residual coefficient in any TB of the CU is valid. If at least one residual coefficient in any point luma TB and chroma TB is valid, step 1760 returns "yes" and control in the processor 205 proceeds to the determination of luma coefficient group size step 1770. If there is no valid residual coefficient in any TB of the CU (as indicated by "zero" arithmetically decoded for cu_cbf), step 1760 returns "no" and the method 1700 terminates, where the processor 205 proceeds to the next CU in the CTU.

[0205] At determine luma coefficient group size step 1770, processor 205 determines the coefficient group size for one or more luma transform blocks associated with the CU. The determination of step 1770 operates in the same manner as the determination of step 1670. Control in processor 205 proceeds from step 1770 to decode luma TB step 1780.

[0206] At the decode luma TB step 1780, under the execution of the processor 205, the entropy decoder 420 decodes the residual coefficients of one or more luma TBs of the CU from the bitstream 133. The determined coefficient group size of step 1770 is used for each luma TB. For each luma TB, a coded block flag is decoded from the bitstream 133, which indicates that there is at least one significant coefficient in the luma TB. If there is at least one significant coefficient in the luma TB, the last significant position is decoded from the bitstream. The last significant position is defined as the last significant coefficient along the scan path that advances from the DC (upper left) coefficient of the TB to the lower right coefficient. The scan path is defined as a diagonal scan within an array of non-overlapping sub-blocks into which the TB is partitioned, each sub-block is the size of the coefficient group and occupies the entire TB. The advancement from one sub-block to the next sub-block in the scan order also follows the diagonal scan. For each coefficient group except the upper left coefficient group and the coefficient group containing the last significant coefficient, the entropy encoder 338 encodes the "coded sub-block flag". The coded sub-block flag indicates that there is at least one significant residual coefficient in the sub-block. If there are no valid residual coefficients in the sub-block, the diagonal scan of the residual coefficients in the TB skips the sub-block. If there is at least one valid residual coefficient in the sub-block, all positions within the sub-block are scanned and the magnitude of each residual coefficient is encoded and the sign of each valid residual coefficient is encoded. Control in processor 205 proceeds from step 1780 to determine chroma coefficient group size step 1790.

[0207] At determine chroma coefficient group size step 1790, processor 205 determines the coefficient group size for the chroma transform block pair associated with the CU. The determination made in step 1790 operates in the same manner as the determination made in step 1690.

[0208] Similar to step 1690, at step 1790, the coefficient group size is determined based on the transform block size, and no further distinction is made between luma and chroma channels. The coefficient group size is therefore determined independently of whether the chroma format is 4:2:2 or 4:2:0, or the corresponding subsampling in the various color planes. As described with respect to Table 1, the coefficient group size is based on the maximum area of ​​the TB of up to 16 samples. Step 1690 operates to determine the coefficient group size for the TB regardless of the color plane (Cb or Cr) of the transform block. Control in the processor 205 proceeds from step 1790 to decode chroma TB step 17100.

[0209] At decode chroma TB step 17100, under execution of processor 205, entropy decoder 420 decodes the residual coefficients of the chroma TB pair of the CU from the bitstream 133. The determined coefficient group size of step 1790 is used for the chroma TB pair. For each chroma TB, a coded block flag is decoded from the bitstream 133, which indicates that there is at least one significant coefficient in the chroma TB. As described with reference to step 1780, the remainder of the decoding process for each chroma TB operates in the same manner as for the luma TB. Method 1700 terminates at execution of step 17100 and control in processor 205 proceeds to the next CU in the CTU.

[0210] Table 3 shows the coding performance results obtained when using Table 1 under the JVET "Common Test Conditions" (CTC) - "All Intra Main10" configuration. The results of Table 3 are obtained using the "VVC Test Model" (VTM) software implementing methods 1600 and 1700 compared to the baseline VTM-4.0 that does not implement methods 1600 and 1700. Overall, the change has no coding impact, and even some small gains are seen in the chroma channels, demonstrating that simplifying the transform block size to coefficient group size mapping table is not detrimental to coding performance.

[0211] The video encoder 115 and the video decoder 134 use methods 1600 and 1700, respectively, to achieve memory reduction in the residual encoding / decoding process by coordinating the coefficient group sizes of the luma TB and the chroma TB. Therefore, the chroma TB is entitled to use coefficient group sizes such as 2×8, 8×2, 2×4, 4×2 instead of just 2×2 and 4×4. For the luma TB, sizes of 16×1 and 1×16 are possible when intra sub-partitioning is used. Although sizes 16×1 and 1×16 can be used for chroma (as they are present in Table 1), the minimum width and height of the chroma block is two samples, so sizes 16×1 and 1×16 are not used for chroma TBs. Since residual encoding and decoding are part of the feedback loop in the design, the memory reduction corresponds to improvements in, for example, cache performance in software implementations or critical path reduction in hardware implementations.

[0212]

[0213] Table 1: Transform block size to coefficient group mapping table for luma and chroma channels (with the same size coefficient group for TB regardless of whether the TB is for luma or chroma channels)

[0214]

[0215]

[0216] Table 2: Conventional mapping of transform block size to coefficient group size for luma and chroma channels (with different coefficient group sizes for same size TB in luma vs chroma)

[0217]

[0218] Table 3: Coding performance obtained from having the same size coefficient groups for TB (regardless of whether the TB is for luma or chroma channels)

[0219] Industrial Applicability

[0220] The described arrangement is suitable for use in the computer and data processing industries and is particularly suitable for use in digital signal processing for encoding or decoding signals such as video and image signals, thereby achieving high compression efficiency.

[0221] In contrast to HEVC, the VVC system allows the use of separate coding trees for luma and chroma channels for increased flexibility. However, as described above, the use of smaller chroma blocks can affect throughput, so the resulting problems may occur. The arrangement described herein determines the appropriate rules for when each coding tree unit is processed to help avoid throughput problems. In addition, as described above, the described arrangement can help provide improved efficiency and accuracy of arithmetic coding of context coding bins for describing each coding tree, taking into account rules for avoiding throughput problems.

[0222] The foregoing describes only some embodiments of the present invention, and modifications and / or changes may be made to the present invention without departing from the scope and spirit of the present invention, wherein the embodiments are intended to be illustrative and not restrictive.

[0223] Citations of related patent applications

[0224] This application claims the benefit of priority under 35 U.S.C. §119 to Australian patent application No. 2019201653 filed on March 11, 2019, the entire contents of which are hereby incorporated by reference for all purposes.

Claims

1. A method for decoding a transform block of an image frame from a bitstream, the method comprising: Decoding a syntax element for determining a chroma format of the image frame from a plurality of chroma formats including a 4:2:0 chroma format and a 4:2:2 chroma format; determining, based on the syntax element, a chroma format of the image frame from the plurality of chroma formats including the 4:2:0 chroma format and the 4:2:2 chroma format; Splitting a coding tree unit into one or more coding units, each of the one or more coding units comprising a luma coding block and a chroma coding block; Determine a subblock of the transform block so that an area of ​​the subblock corresponds to 16 samples, the subblock being a luminance transform block of a luminance coding block or a chrominance transform block of a chrominance coding block; as well as decoding the transform block from the bitstream using the sub-block, Wherein, in a case where the block size of the chroma coding block in a given coding unit including the luma coding block and the chroma coding block in the coding tree unit having the 4:2:0 chroma format is 8×2, even if the luma coding block in the given coding unit is vertically ternary-split, the chroma coding block in the given coding unit is not allowed to be vertically ternary-split, Wherein, when the luma coding block in the given coding unit is split by vertical ternary splitting and the chroma coding block in the given coding unit is not split by vertical ternary splitting, the chroma coding block in the given coding unit is arranged at a position corresponding to three luma coding blocks obtained by vertical ternary splitting of the luma coding block in the given coding unit, wherein the transform block is decoded based on (i) a width and a height of the transform block, and (ii) whether the transform block is a luma transform block or a chroma transform block, and In which, when the log2 of the width of the transform block is 4 and the log2 of the height of the transform block is 4, when determining the log2 of the width of the subblock of the transform block and the log2 of the height of the subblock of the transform block, the log2 of the width of the subblock of the transform block and the log2 of the height of the subblock of the transform block are determined from (i) the log2 of the width of the transform block and the log2 of the height of the transform block, and regardless of (ii) whether the transform block is a luminance transform block or a chrominance transform block and (iii) the chrominance format of the image frame.

2. The method according to claim 1, in, In a case where the block size of the chroma coding block in the given coding unit in the coding tree unit having the 4:2:0 chroma format is a predetermined size, further splitting of the chroma coding block in the given coding unit is not allowed even if the luma coding block in the given coding unit is split, and Wherein, when the block size of the chroma coding block in the given coding unit is the predetermined size and the current split mode of the given coding unit is quad split, intra-frame prediction is determined as the chroma coding block for the given coding unit.

3. The method according to claim 1, wherein: In a case where log2 of the width of the transform block is 4 and log2 of the height of the transform block is 4, the determined log2 of the width of the subblock of the transform block is 2 and the determined log2 of the height of the subblock of the transform block is 2.

4. The method according to claim 1, wherein: The 8×2 indicates a width of 8 and a height of 2.

5. The method according to claim 1, wherein: In a case where the block size of the chroma coding block in the given coding unit including the luma coding block and the chroma coding block in the coding tree unit having the 4:2:0 chroma format is 8×2, the block size of the luma coding block in the given coding unit is 16×4, where 8×2 indicates a width of 8 and a height of 2, and 16×4 indicates a width of 16 and a height of 4.

6. A device for decoding a transform block of an image frame from a bitstream, the device comprising: a first decoding unit configured to decode a syntax element for determining a chroma format of the image frame from a plurality of chroma formats including a 4:2:0 chroma format and a 4:2:2 chroma format; A first determining unit configured to determine a chroma format of the image frame from the plurality of chroma formats including the 4:2:0 chroma format and the 4:2:2 chroma format based on the syntax element; a splitting unit configured to split the coding tree unit into one or more coding units, wherein the one or more coding units each include a luma coding block and a chroma coding block; A second determining unit is configured to determine a sub-block of the transform block, wherein the sub-block is a luminance transform block of a luminance coding block or a chrominance transform block of a chrominance coding block; as well as a second decoding unit configured to decode the transform block from the bitstream using the sub-block, Wherein, in a case where the block size of the chroma coding block in a given coding unit including the luma coding block and the chroma coding block in the coding tree unit having the 4:2:0 chroma format is 8×2, even if the luma coding block in the given coding unit is vertically ternary-split, the chroma coding block in the given coding unit is not allowed to be vertically ternary-split, Wherein, when the luma coding block in the given coding unit is split by vertical ternary splitting and the chroma coding block in the given coding unit is not split by vertical ternary splitting, the chroma coding block in the given coding unit is arranged at a position corresponding to three luma coding blocks obtained by vertical ternary splitting of the luma coding block in the given coding unit, wherein the transform block is decoded based on (i) a width and a height of the transform block, and (ii) whether the transform block is a luma transform block or a chroma transform block, and In which, when the log2 of the width of the transform block is 4 and the log2 of the height of the transform block is 4, when determining the log2 of the width of the subblock of the transform block and the log2 of the height of the subblock of the transform block, the log2 of the width of the subblock of the transform block and the log2 of the height of the subblock of the transform block are determined from (i) the log2 of the width of the transform block and the log2 of the height of the transform block, and regardless of (ii) whether the transform block is a luminance transform block or a chrominance transform block and (iii) the chrominance format of the image frame.

Images