Method, device, and program for coding and decoding block of video sample

Abstract

To provide a method and computer program for decoding a block of a video sample.SOLUTION: A method comprises the steps of: decoding a matrix intra-prediction flag for an object block in a coding tree unit; decoding a matrix prediction mode for the object block in the case where the matrix intra-prediction flag indicates that the matrix intra-prediction is used for the object block; selecting a matrix for the matrix intra-prediction for the object block in accordance with the matrix intra-prediction mode; and generating a prediction sample of the object block on the basis of the selected matrix. Therein, an area existing in the coding tree unit is divided into four blocks each having sizes of 16×8, and in the case where the object block is one of the four blocks, the decoding of the matrix intra-prediction flag for the object block depends on a situation of use of the matrix intra-prediction of two blocks that are adjacent to the object block.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] REFERENCE TO RELATED APPLICATIONS This invention claims the benefit under 35 U.S.C. §119 of Australian Patent Application No. 2019204437, filed June 24, 2019, as hereby incorporated by reference in its entirety as if fully set forth herein. Technical Field

[0002] The present invention relates generally to digital video signal processing, and in particular to a method, apparatus and computer program for encoding and decoding blocks of video samples. [Background technology]

[0003] Currently, there are many applications for video coding, including applications for the transmission and storage of video data. Many video coding standards have been developed and are currently under development. Recent progress in the standardization of video coding has led to the formation of a group called the "Joint Video Experts Team" (JVET). The Joint Video Experts Team (JVET) includes members of the International Telecommunication Union (ITU) Telecommunication Standardization Sector (ITU-T) Study Committee 16 Consultative Group 6 (SG16 / Q6), known as the "Video Coding Experts Group" (VCEG), and members of the International Organization for Standardization / International Electrotechnical Commission Joint Technical Committee 1 / Subcommittee 29 / Working Group 11 (ISO / IEC JTC1 / SC29 / WG11), known as the "Moving Picture Experts Group" (MPEG).

[0004] The Joint Video Experts Team (JVET) issued a Call for Proposals (CfP) and analyzed the submitted proposals at its 10th conference in San Diego, USA. The submitted proposals showed that the video compression capabilities significantly exceed the current state-of-the-art video compression standard, i.e., "High Efficiency Video Coding" (HEVC). Based on the results of this analysis, it was decided to initiate a project to develop a new video compression standard named "Versatile Video Coding" (VVC). VVC is expected to address the continuing demand for ever higher compression performance, especially as video formats improve in capabilities (e.g., higher resolutions and higher frame rates), as well as the growing market demand for service delivery over WANs, where bandwidth costs are relatively high. At the same time, VVC must be implementable on conventional silicon processes and must have an acceptable trade-off between the achieved performance and the implementation costs (e.g., silicon area, CPU processor load, memory utilization, bandwidth, etc.).

[0005] Video data includes image data for successive frames, and the image data for each frame includes one or more color channels. Generally, one primary color channel and two secondary color channels are required. The primary color channel is commonly called the "luma" channel, and the secondary color channel is commonly called the "chroma" channel. Video data is usually displayed in the RGB (Red-Green-Blue) color space, which has a high correlation between each of the three components. Encoders or decoders often represent video data using a color space such as YCbCr. YCbCr concentrates luminance, which is mapped to "luma" according to a transfer function, in the Y (primary) channel and chroma in the Cb and Cr (secondary) channels. Furthermore, the Cb and Cr channels may be sampled (subsampled) at a spatially lower rate compared to the luma channel, for example, half horizontally and half vertically, known as the "4:2:0 chroma format". The 4:2:0 chroma format is commonly used in "consumer" applications such as Internet video streaming, television broadcasting, and storage on Blu-Ray® discs. Formats that subsample the Cb, Cr channels at half the rate horizontally, but not vertically, are known as "4:2:2 chroma formats". 4:2:2 chroma formats are typically used in professional applications, including filming footage for film production. The higher the sampling rate of a 4:2:2 chroma format, the more flexibility the resulting footage has for editing, such as color grading. Before being delivered to consumers, 4:2:2 chroma format material is often converted to 4:2:0 chroma format before being encoded for delivery to consumers. In addition to chroma format, videos are also characterized by their resolution and frame rate. Examples of resolutions include ultra-high definition (UHD) with a resolution of 3840×2160, or "8K" with a resolution of 7680×4320, and examples of frame rates include 60Hz or 120Hz. Luma sample rates can range, for example, from about 500 megasamples per second to several gigasamples per second.For a 4:2:0 chroma format, the sample rate of each chroma channel is one-quarter of the luma sample rate, and for a 4:2:2 chroma format, the sample rate of each chroma channel is half the luma sample rate.

[0006] The VVC standard is a "block-based" encoding / decoding method, where a frame is first divided into square regions known as "Coding Tree Units" (CTUs). A CTU typically occupies a relatively large area, such as 128x128 luma samples. However, the CTU at the bottom right corner of each frame may be smaller in area. Each CTU is associated with a "coding tree" for the luma channel and additional coding trees for the chroma channels. The coding trees define the decomposition of the CTU's region into a set of blocks, also called "coding blocks" (CBs). A coding tree may specify blocks for both luma and chroma channels, in which case the co-located coding blocks are collectively referred to as a "coding unit" (CU). That is, each CU has a coding block for each color channel. The CBs are coded or decoded in a specific order. When using a 4:2:0 chroma format, a CTU with a luma code tree for a 128x128 luma sample region will have a corresponding chroma code tree for a 64x64 chroma sample region co-located with the 128x128 luma sample region. When a single code tree is used for the luma and chroma channels, the set of blocks co-located with a region is commonly referred to as a "unit", e.g., CU, "Prediction Unit" (PU), and "Transform Unit" (TU). When separate code trees are used for a region, CB, "Prediction Block" (PB), and "Transform Block" (TB) are used.

[0007] It should be noted that, despite the above distinction between "units" and "blocks", the term "block" is also used as a general term to describe an area of ​​a frame where processing is applied to all color channels.

[0008] For each CU, a prediction unit (PU) of the contents (sample values) of the corresponding region of the frame data is generated ("prediction unit"). In addition, a representation of the difference between the prediction and the contents of the region as seen at the input to the encoder (or "residual" in the spatial domain) is formed. The differences for each color channel are transformed and coded as a set of residual coefficients to form one or more TUs for a CU. The transformation applied is a discrete cosine transform (DCT) or other transformation, which is applied to the residual values ​​of each block. This transformation is applied separately; that is, the two-dimensional transformation is performed in two passes. The block is first transformed by applying a one-dimensional transform to the samples of each row in the block. The partial results are then transformed by applying a one-dimensional transform to each column of the partial results to produce a final block of transform coefficients that substantially decorrelates the residual samples. Transforms of various sizes are supported in the VVC standard, including the transformation of rectangular blocks with the dimensions of each side being a power of two. The transform coefficients are quantized for entropy coding into the bitstream.

[0009] VVC features intra-frame prediction and inter-frame prediction. Intra-frame prediction involves the use of previously processed samples in a frame that are used to generate a prediction of a sample of a current block in the frame. Inter-frame prediction involves using samples of a block taken from a previously decoded frame to generate a prediction of a sample of a current block in a frame. The previously decoded frame is offset from the spatial location of the current block according to a motion vector, and is often filtered. An intra-frame prediction block may be a uniform sample value ("DC intra-prediction"), a plane with an offset and horizontal and vertical gradients ("planar intra-prediction"), or a population of blocks with neighboring samples in a particular direction ("angular intra-prediction"), or the result of a matrix multiplication using neighboring samples and selected matrix coefficients. Because the neighboring samples include samples from previously processed blocks, the feedback loop of intra-frame prediction is highly limited and the computational complexity must be kept lower than that required to meet the highest supported resolution and frame rate. Summary of the Invention

[0010] It is an object of the present invention to substantially overcome, or at least ameliorate, one or more disadvantages of existing arrangements.

[0011] According to one aspect of the present disclosure, there is provided a method for decoding coding units of a code tree for an image frame from a video bitstream, the method comprising: dividing a region of the code tree into a plurality of coding blocks, each coding block including a predictive block; determining a matrix intra prediction flag for the predictive blocks of each coding block, each matrix intra prediction flag indicating whether matrix intra prediction was used for one of the predictive blocks of the coding blocks, the determination being based on (i) an area of ​​the region if the region meets a threshold, or (ii) an allocation of the region if the area of ​​the region does not meet the threshold; for each predictive block determined to use matrix intra prediction according to the determined flag, reading matrix coefficients from a memory; and decoding the coding unit using a reference sample of each predictive block and a predictive block generated for each coding unit in the region using the matrix coefficients.

[0012] According to another aspect, the threshold is a size greater than 512 luma samples.

[0013] According to another aspect, the threshold is a size greater than 64 luma samples.

[0014] According to another aspect, the budget allows for reading 40 words of a 4x4 block for the region.

[0015] According to another aspect, the matrix intra prediction flag of the CU is decoded only if matrix intra prediction is used.

[0016] According to another aspect, the matrix intra prediction flag of the CU is decoded regardless of whether matrix intra prediction was used or not.

[0017] According to another aspect of the present disclosure, there is provided a method for decoding coding units of a code tree for an image frame from a video bitstream, the method comprising: dividing a region of the code tree into a plurality of coding blocks, each of which includes a predictive block; determining a matrix intra prediction flag for predictive blocks of the coding blocks based on a size of each coding block, each matrix intra prediction flag indicating whether matrix intra prediction was used for the predictive block of the corresponding coding block; for each predictive block determined to use matrix intra prediction according to the determined flag, reading matrix coefficients from a memory; and decoding the coding units from reference samples of each predictive block and the predictive blocks of each coding unit in the region generated using the matrix coefficients.

[0018] According to another aspect, a matrix intra prediction flag is decoded if the size of the coding unit is not 4x4.

[0019] According to another aspect, a matrix intra prediction flag is decoded if the size of the coding unit is not one of 4x4, 8x4, or 4x8.

[0020] According to another aspect, a matrix intra prediction flag is decoded if the size of the coding unit is not one of 4x4, 8x4, 4x8, or 8x8.

[0021] According to another aspect, the matrix intra prediction flag is decoded if the size of the coding unit is not one of 4x4, 8x4, 4x8, 8x8, or 8x16.

[0022] According to another aspect of the present disclosure, there is provided a method for generating a predictive block of a code tree for an image frame from a video bitstream, the method comprising: determining a prediction mode of the coding unit by decoding a matrix intra-prediction mode flag from the bitstream, the prediction mode including a matrix intra-prediction mode; decoding a shortened binary codeword to determine the matrix intra-prediction mode; and generating the predictive block by performing matrix multiplication on reference samples adjacent to the predictive block and a matrix selected according to the decoded matrix intra-prediction mode.

[0023] According to another aspect of the present disclosure, there is provided a non-transitory computer-readable storage medium having stored thereon a computer program for implementing a method of decoding coding units of a code tree for an image frame from a video bitstream, the method comprising: dividing a region of the code tree into a plurality of coding blocks, each coding block including a predictive block; determining a matrix intra prediction flag for the predictive blocks of each coding block, each matrix intra prediction flag indicating whether matrix intra prediction was used for the predictive block of one of the coding blocks, the determination being based on (i) an area of ​​the region if the region meets a threshold, or (ii) an allocation of the region if the area of ​​the region does not meet the threshold; for each predictive block determined to use matrix intra prediction according to the determined flag, reading matrix coefficients from a memory; and decoding the coding unit using a reference sample of each predictive block and a predictive block generated for each coding unit in the region using the matrix coefficients.

[0024] According to another aspect of the present disclosure, a video decoder is provided, comprising: receiving coding units of a code tree for an image frame from a video bitstream; dividing a region of the code tree into a plurality of coding blocks, each coding block including a predictive block; determining a matrix intra prediction flag for the predictive blocks of each coding block, each matrix intra prediction flag indicating whether matrix intra prediction was used for the predictive block of one of the coding blocks, the determination being based on (i) an area of ​​the region if the region meets a threshold, or (ii) an allocation of the region if the area of ​​the region does not meet the threshold; for each predictive block determined to use matrix intra prediction according to the determined flag, reading matrix coefficients from a memory; and decoding the coding unit using a reference sample of each predictive block and a predictive block generated for each coding unit in the region using the matrix coefficients.

[0025] According to another aspect of the present disclosure, a system is provided, comprising: a memory; and a processor, the processor executing code stored in the memory for implementing a method for decoding coding units of a code tree for an image frame from a video bitstream, the method comprising: dividing a region of the code tree into a plurality of coding blocks, each coding block including a predictive block; determining a matrix intra prediction flag for the predictive blocks of each coding block, each matrix intra prediction flag indicating whether matrix intra prediction was used for the predictive block of one of the coding blocks, the determination being based on (i) an area of ​​the region if the region meets a threshold, or (ii) an allocation of the region if the area of ​​the region does not meet the threshold; for each predictive block determined to use matrix intra prediction according to the determined flag, reading matrix coefficients from a memory; and decoding the coding unit using a reference sample of each predictive block and a predictive block generated for each coding unit in the region using the matrix coefficients.

[0026] Other aspects are also disclosed.

[0027] At least one embodiment of the invention will now be described with reference to the following drawings and appendix. [Brief description of the drawings]

[0028] [Figure 1] FIG. 1 is a schematic block diagram illustrating a video encoding and decoding system. [Figure 2A] , [Figure 2B] 2A and 2B show a schematic block diagram of a general purpose computer system upon which one or both of the video encoding and decoding systems of FIG. 1 may be implemented. [Diagram 3] FIG. 3 is a schematic block diagram showing functional modules of a video encoder. [Figure 4] FIG. 4 is a schematic block diagram showing the functional modules of a video decoder. [Diagram 5] FIG. 5 is a schematic block diagram illustrating a possible division of a block into one or more blocks in a tree structure of Versatile Video Coding. [Figure 6] FIG. 6 is a schematic diagram of a data flow for achieving a possible division of one block into one or more blocks in a tree structure of Versatile Video Coding. [Figure 7A] , [Figure 7B] 7A and 7B show an example of dividing a coding tree unit (CTU) into several coding units (CUs). [Figure 8] FIG. 8 is a diagram illustrating an example of processing of the matrix intra prediction module. [Figure 9A] , [Figure 9B] 9A and 9B show an example CTU having regions where memory access bandwidth for matrix intra prediction modes is applied. [Figure 10] FIG. 10 shows the code tree corresponding to the example of FIG. [Figure 11]FIG. 11 illustrates a method for encoding coding units of an image frame, including coding units that use matrix intra prediction, into a video bitstream. [Figure 12] FIG. 12 illustrates a method for determining memory access budgets for matrix intra prediction across multiple coding units of a coding tree as used in FIG. [Figure 13] FIG. 13 illustrates a method for determining a coding unit for encoding a portion of an image frame into a video bitstream such as that used in FIG. [Figure 14] FIG. 14 shows how coding units of an image frame are encoded into a video bitstream such as that used in FIG. [Figure 15] FIG. 15 illustrates a method for decoding a code tree of an image frame from a video bitstream. [Figure 16] FIG. 16 illustrates a method for decoding an image frame coding unit from a video bitstream such as that used in FIG. [Figure 17] FIG. 17 illustrates another method for determining a coding unit for encoding a portion of an image frame into a video bitstream such as that used in FIG. [Figure 18] FIG. 18 shows another method for encoding coding units of an image frame into a video bitstream such as that used in FIG. [Figure 19] FIG. 19 shows another method for decoding an image frame coding unit from a video bitstream such as that used in FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0029] Detailed Description Including Best Mode When reference is made to steps and / or features having the same reference numbers in any one or more of the accompanying drawings, those steps and / or features have the same function or operation for the purposes of this specification, unless a contrary intention is indicated.

[0030] As mentioned above, the computational complexity of intraframe prediction is limited, especially in the case of matrix intraprediction (MIP). Although MIP can provide an effective solution in terms of minimizing errors, the computational complexity of applying MIP particularly impacts the worst-case block processing speed, for example for frames consisting of only 4x4 blocks. The block processing speed needs to be sufficient to support the resolution and frame rate of the intended application. Considering only the luma channel, a frame with "8K" resolution (7680x4320) at 120 frames per second needs to process 248.8x106 4x4 blocks per second. Even if the entire frame or video sequence does not reach the worst case, sub-regions that reach the worst case need to be processed without delaying the delivery of a fully decoded frame for display on the display. One reason for the complexity is due to the memory bandwidth required to retrieve the matrix coefficients selected according to the matrix intraprediction (MIP) mode, which can vary from block to block without constraint.

[0031] 1 is a schematic block diagram illustrating the functional modules of a video encoding and decoding system 100. To accommodate practical applications and / or to accommodate the coding advantages achieved by the MIP mode, the system 100 can utilize limitations associated with the application of the MIP mode to establish the worst-case memory bandwidth for selecting or reading matrix coefficients.

[0032] System 100 includes source device 110 and destination device 130. Communication channel 120 is used to communicate encoded video information from source device 110 to destination device 130. In some configurations, either or both of source device 110 and destination device 130 may be mobile phones or "smartphones," in which case communication channel 120 is a wireless channel. In other configurations, source device 110 and destination device 130 may be video conferencing equipment, in which case communication channel 120 is typically a wired channel, such as an Internet connection. Additionally, source device 110 and destination device 130 may be any of a wide range of devices, including devices supporting wireless television broadcasts, cable television applications, Internet video applications (including streaming), and applications in which encoded video data is captured to a computer-readable storage medium, such as a hard disk drive of a file server.

[0033] As shown in Figure 1, source device 110 includes video source 112, video encoder 114, and transmitter 116. Video source 112 typically includes a source of captured video frame data (shown as 113), such as an image capture sensor, a previously captured video sequence stored on a non-transitory recording medium, or footage from a remote image capture sensor. Video source 112 may also be, for example, the video output of an operating system or the output of a computer graphics card that displays various applications running on a computing device, such as a tablet computer. Source devices 110 that may include an image capture sensor as a video source 112 include smartphones, video camcorders, professional video cameras, and network video cameras.

[0034] As will be further described with reference to FIG. 3, video encoder 114 converts (or "encodes") captured frame data (indicated by arrow 113) from video source 112 into a bitstream (indicated by arrow 115). Bitstream 115 is transmitted as encoded video data (or "encoded video information") by transmitter 116 over communication channel 120. Bitstream 115 may also be stored in non-transitory storage device 122, such as "flash" memory or a hard disk drive, until later transmitted over communication channel 120 or in lieu of transmission over communication channel 120. For example, encoded video data may be provided on demand to customers over a wide area network (WAN) for video streaming applications.

[0035] The destination device 130 includes a receiver 132, a video decoder 134, and a display device 136. The receiver 132 receives encoded video data from the communication channel 120 and passes the received video data as a bitstream to the video decoder 134 (indicated by arrow 133). The video decoder 134 then outputs decoded frame data (indicated by arrow 135) to the display device 136. The decoded frame data 135 has the same chroma format as the frame data 113. The display device 136 may include, for example, a cathode ray tube or liquid crystal display such as a smartphone, a tablet computer, a computer monitor, or a standalone television device. It is also possible for the functions of the source device 110 and the destination device 130 to be embodied in a single device, such as a mobile phone or a tablet computer.

[0036] Without being limited to the above-mentioned exemplary devices, each of the source device 110 and the destination device 130 may be configured in a general-purpose computing system, typically through a combination of hardware and software components. FIG. 2A shows such a computer system 200, including a computer module 201, input devices such as a keyboard 202, a mouse pointer device 203, a scanner 226 and a camera 227, a microphone 280, which may be configured as a video source 112, and output devices including a printer 215, a display device 214, which may be configured as a display device 136, and a speaker 217. An external modulator-demodulator (modem) transceiver device 216 may be used for the computer module 201 to communicate bidirectionally with a communication network 220 via a connection 221. The communication network 220, which represents the communication channel 120, may be a (WAN), such as the Internet, a cellular communication network, or a private WAN. If the connection 221 is a telephone line, the modem 216 may be a conventional "dial-up" modem. Alternatively, the modem 216 may be a broadband modem where the connection 221 is a high-capacity (e.g., cable or optical) connection. A wireless modem may be used for wireless connection to the communications network 220. The transceiver device 216 may provide the functionality of the transmitter 116 and receiver 132, and the communications channel 120 may be embodied as the connection 221.

[0037] The computer module 201 typically includes at least one processor unit 205 and a memory unit 206. For example, the memory unit 206 may include a semiconductor random access memory (RAM) and a semiconductor read-only memory (ROM). The computer module 201 also includes a number of input / output (I / O) interfaces, including an audio-video interface 207 for coupling to a video display 214, a speaker 217 and a microphone 280, an I / O interface 213 for coupling to a keyboard 202, a mouse 203, a scanner 226, a camera 227 and optionally a joystick or other human interface device (not shown), an interface 208 for an external modem 216 and a printer 215. The signal from the audio-video interface 207 to the computer monitor 214 is typically an output of a computer graphics card. In some implementations, the modem 216 may be integrated into the computer module 201, for example in the interface 208. The computer module 201 also has a local network interface 211, which allows the computer system 200 to be connected to a local communication network 222, known as a local area network (LAN), via a connection 223. As shown in FIG. 2A, the local communication network 222 can also be connected to a wide network 220 via a connection 224, which typically includes a so-called "firewall" device or a device of similar functionality. The local network interface 211 may include an Ethernet circuit card, a Bluetooth wireless configuration, or an IEEE 802.11 wireless configuration. However, many other types of interfaces may be used in the interface 211. The local network interface 211 may also provide the functionality of the transmitter 116 and the receiver 132, and the communication channel 120 may be embodied in the local communication network 222.

[0038] The I / O interfaces 208 and 213 may provide either or both serial and parallel connections, the former typically implemented according to the Universal Serial Bus (USB) standard and having a corresponding USB connector (not shown). The storage device 209 typically includes a hard disk drive (HDD) 210. Other storage devices such as floppy disk drives and magnetic tape drives (not shown) may also be used. An optical disk drive 212 is typically provided to function as a non-volatile data source. Portable memory devices such as optical disks (CD-ROM, DVD, Blu-ray Disc®, etc.), USB-RAM, portable external hard drives, and floppy disks may be used as suitable data sources for the computer system 200. Typically, the HDD 210, the optical drive 212, and the networks 220 and 222 may all be configured to operate as a video source 112 or as a destination for decoded video data to be stored for playback on the display 214. The source device 110 and the destination device 130 of the system 100 may be implemented by the computer system 200.

[0039] The components 205 to 213 of the computer module 201 typically communicate via an interconnected bus 204 and in a manner which results in a conventional mode of operation of the computer system 200 known to those skilled in the art. For example, the processor 205 is coupled to the system bus 204 using a connection 218. Similarly, the memory 206 and the optical disk drive 212 are coupled to the system bus 204 by a connection 219. Computers on which the described arrangements can be practiced include, for example, IBM-PC's and compatibles, Sun SPARCstations, Apple Macs and similar computer systems.

[0040] Where appropriate or desired, the video encoder 114 and the video decoder 134, and the methods described below, may be implemented using the computer system 200. In particular, the video encoder 114, the video decoder 134, and the methods described may be implemented as one or more software application programs 233 executable within the computer system 200. In particular, the video encoder 114, the video decoder 134, and the steps of the methods described may be implemented by instructions 231 (see FIG. 2B) of the software 233 executed within the computer system 200. The software instructions 231 may be formed as one or more code modules, each for performing one or more specific tasks. The software may also be divided into two independent parts, where a first part and corresponding code modules perform the methods described, and a second part and corresponding code modules manage a user interface between the first part and a user.

[0041] The software may be stored in a computer readable medium, including, for example, the storage devices described below. The software is read from the computer readable medium into the computer system 200 and executed by the computer system 200. A computer readable medium having such software or a computer program recorded thereon is a computer program product. The use of the computer program product in the computer system 200 provides beneficial effects to the video encoder 114, the video decoder 134, and to advantageous apparatus for performing the methods described.

[0042] The software 233 is typically stored in the HDD 210 or in the memory 206. The software is read into the computer system 200 from a computer-readable medium and executed by the computer system 200. Thus, for example, the software 233 may be stored on an optically readable disk storage medium (e.g., a CD-ROM) 225 that is read by the optical disk drive 212.

[0043] In some cases, the application program 233 may be provided to the user encoded on one or more CD-ROMs 225 and read via the corresponding drive 212 or by the user from the network 220 or 222. Furthermore, the software may be read into the computer system 200 from other computer readable media. A computer readable storage medium refers to a non-transitory tangible storage medium that provides recorded instructions and / or data to the computer system 200 for execution and / or processing. Such storage media include, for example, floppy disks, magnetic tapes, CD-ROMs, DVDs, Blu-ray Discs, hard disk drives, ROMs or integrated circuits, USB memories, magneto-optical disks, or computer readable cards such as PCMCIA cards, and such devices may be either internal or external to the computer module 201. Transient or intangible computer-readable transmission media that can be used to provide software, application programs, instructions and / or video data or encoded video data to the computer module 401 include, for example, wireless or infrared transmission channels, network connections to other computers or network devices, and the Internet or intranets, including e-mail transmissions, information stored on websites, etc.

[0044] A second portion of the application programs 233 and the corresponding code modules described above may be executed to implement one or more graphical user interfaces (GUIs) that are rendered or otherwise represented on the display 214, and a user of the computer system 200 and applications may then manipulate the interfaces in a functionally adaptable manner to provide control commands and / or input to the applications associated with the GUIs, typically by operating a keyboard 202 and / or a mouse 203. Other forms of functionally adaptable user interfaces may also be implemented, such as a voice interface that utilizes speech prompts output via speaker 217 and user voice commands input via microphone 280.

[0045] Figure 2B is a detailed schematic block diagram of processor 205 and "memory" 234. Memory 234 represents the logical collection of all memory modules (including HDD 209 and semiconductor memory 206) accessible to computer module 201 of Figure 2A.

[0046] When the computer module 201 is first powered on, a power-on self-test (POST) program 250 is executed. The POST program 250 is typically stored in the ROM 249 of the semiconductor memory 206 of FIG. 2A. Hardware devices such as the ROM 249 that store software are sometimes called firmware. The POST program 250 checks the hardware in the computer module 201 to ensure that it is functioning properly, and checks whether the processor 205, the memory 234 (209, 206), and a basic input / output system software (BIOS) module 251, which is typically stored in the ROM 249, are operating correctly. If the POST program 250 is executed normally, the BIOS 251 starts the hard disk drive 210 of FIG. 2A. When the hard disk drive 210 starts up, a bootstrap loader program 252 resident on the hard disk drive 210 is executed via the processor 205. This causes the operating system 253 to be read into the RAM memory 206, and the operating system 253 starts operating. Operating system 253 is a system level application executable by processor 205 to implement a variety of high level functions, including processor management, memory management, device management, storage management, software application interface, and a general purpose user interface.

[0047] The operating system 253 manages the memory 234 (209, 206) so that each process and application running on the computer module 201 has enough memory to run without conflicting with memory allocated to another process. Additionally, the various types of memory available in the computer system 200 of FIG. 2A must be used appropriately so that each process can run effectively. Thus, the aggregate memory 234 is not intended to describe how specific segments of memory are allocated (unless otherwise noted), but rather is intended to illustrate the concept of memory accessible by the computer system 200 and how it is used.

[0048] As shown in FIG. 2B, the processor 205 includes a number of functional modules including a control unit 239, an arithmetic logic unit (ALU) 240, and a local or internal memory 248, sometimes referred to as a cache memory. The cache memory 248 typically includes a number of storage registers 244-246 in a register section. One or more internal buses 241 operatively interconnect these functional modules. The processor 205 also typically has one or more interfaces 242 for communicating with external devices via the system bus 204 using a connection 218. The memory 234 is coupled to the bus 204 using a connection 219.

[0049] Application program 233 includes a series of instructions 231, which may include conditional branch and loop instructions. Program 233 may also include data 232 used in the execution of program 233. Instructions 231 and data 232 are stored in memory locations 228, 229, 230 and 235, 236, 237, respectively. Depending on the relative sizes of instructions 231 and memory locations 228-230, an instruction may be stored in a single memory location, as represented by the instruction shown in memory location 230. Alternatively, an instruction may be split into multiple portions, each stored in a separate memory location, as represented by the segments of the instruction shown in memory locations 228 and 229.

[0050] In general, the processor 205 is given a set of instructions which it executes. The processor 205 waits for a next input for which the processor 205 executes another set of instructions. Each input may come from one or more of a number of sources, including data generated by one or more of the input devices 202, 203 shown in FIG. 2A, data received from an external source over one of the networks 220, 202, data retrieved from one of the storage devices 206, 209, or data retrieved from a storage medium 225 inserted into a corresponding reader 212. Execution of the set of instructions may result in output of data. Execution may also result in the storage of data or variables in the memory 234.

[0051] The video encoder 114, the video decoder 134, and the described methods may use input variables 254, which are stored in corresponding memory locations 255, 256, 257 of the memory 234. The video encoder 114, the video decoder 134, and the described methods may generate output variables 261, which are stored in corresponding memory locations 262, 263, 264 of the memory 234. Intermediate variables 258 may be stored in memory locations 259, 260, 266, and 267.

[0052] Returning to processor 205 of Figure 2B, registers 244, 245, 246, arithmetic logic unit (ALU) 240, and control unit 239 cooperate to perform the series of micro-operations required to perform a "fetch, decode, execute" cycle for every instruction in the instruction set that makes up program 233. Each fetch, decode, execute cycle includes the following: a fetch operation that fetches or reads instructions 231 from memory locations 228, 229, 230; a decode process in which the control unit 239 determines which instructions have been fetched; Execution process in which the control unit 239 and / or ALU 240 execute instructions.

[0053] A further fetch, decode, execute cycle is then performed for the next instruction. Similarly, a store cycle may be performed, which causes control unit 239 to store or write a value to memory location 232.

[0054] Each step or sub-process in the methods of Figures 11-19 is associated with one or more segments of program 233, as described below, and typically involves register sections 244, 245, 247, ALU 240, and control unit 239 within processor 205 working together to perform a fetch, decode, and execute cycle for all instructions in the instruction set for the noted segment of program 233.

[0055] FIG. 3 is a schematic block diagram showing the functional modules of the video encoder 114. FIG. 4 is a schematic block diagram showing the functional modules of the video decoder 134. In general, data is passed between the functional modules in the video encoder 114 and the video decoder 134 in groups of samples or coefficients, such as fixed-size sub-blocks or matrices of a block. The video encoder 114 and the video decoder 134 may be implemented using a general-purpose computer system 200, as shown in FIGS. 2A and 2B, and the various functional modules may be realized by dedicated hardware in the system 200 or by software executable in the computer system 200, such as one or more software code modules of a software application program 233 residing on a hard disk drive 205 and controlled by execution by the processor 205. Alternatively, the video encoder 114 and the video decoder 134 may be realized by a combination of dedicated hardware and software executable in the computer system 200. Alternatively, the video encoder 114, the video decoder 134, and the methods described may be realized by dedicated hardware, such as one or more integrated circuits performing the functions or sub-functions of the methods described. Such dedicated hardware may include a graphic processing unit (GPU), a digital signal processor (DSP), an application specific standard product (ASSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or one or more microprocessors and associated memories. In particular, the video encoder 114 includes modules 310-392, and the video decoder 134 includes modules 420-496, each of which may be implemented as one or more software code modules of the software application program 233.

[0056] The video encoder of FIG. 3 is an example of a video coding pipeline for Versatile Video Coding (VVC), although other video codecs may be used to perform the processing steps described herein. The video encoder 114 receives captured frame data 113, such as a sequence of frames, each frame including one or more color channels. The frame data 113 may be in any chroma format, for example, 4:0:0, 4:2:0, 4:2:2, or 4:4:4 chroma format. The block divider 310 first divides the frame data 113 into CTUs that are generally square in shape and configured such that a specific size for the CTU is used. The size of the CTU may be, for example, 64×64, 128×128, or 256×256 luma samples. The block divider 310 further divides each CTU into one or more CBs according to the luma code tree and the chroma code tree. The CBs may be of various sizes and include both square and non-square aspect ratios. The operation of the block divider 310 is further described with reference to Figures 11 to 19. However, in the VVC standard, the edge lengths of CB, CU, PU, ​​and TU are always powers of two. Thus, a current CB, denoted as 312, is output from the block divider 310, which proceeds by iterating over one or more blocks of the CTU according to the luma and chroma coding trees of the CTU. The option of dividing a CTU into CBs is further described below with reference to Figures 5 and 6. Although the operation is generally described in terms of CTUs, the video encoder 114 and the video decoder 134 can operate on smaller sized regions to reduce memory consumption. For example, each CTU can be divided into smaller regions called "virtual pipeline data units" (VPDUs) of size 64x64. The VPDUs form smaller data that is more suitable for pipeline processing in a hardware architecture, which reduces silicon area and therefore cost due to the reduced memory footprint compared to processing on an entire CTU.

[0057] The CTUs resulting from one division of the frame data 113 may be scanned in raster scan order and grouped into one or more "slices." A slice may be an "intra" (or "I") slice. Intra slices (I slices) indicate that all CUs within a slice are intra predicted. Alternatively, a slice may be unidirectionally predicted or bidirectionally predicted ("P" or "B" slices, respectively), further indicating the possibility of unidirectional and bidirectional prediction within a slice, respectively.

[0058] For each CTU, the video encoder 114 operates in two stages. In the first stage (called the "search" stage), the block partitioner 310 tests various potential configurations of the code tree. Each potential configuration of the code tree has an associated "candidate" CB. In the first stage, various candidate CBs are tested to select a CB that results in relatively high compression efficiency at relatively low distortion. The testing typically involves Lagrangian optimization, where the candidate CBs are evaluated based on a weighted combination of rate (coding cost) and distortion (error relative to the input frame data 113). The "best" candidate CB (the CB with the lowest evaluated rate / distortion) is selected for subsequent encoding into the bitstream 115. The evaluation of the candidate CBs includes options such as using the CB for a given area, or further dividing the region according to various partitioning options and further coding each resulting smaller area with the CB, or further dividing the region. As a result, both the CB and the code tree are selected in the search stage.

[0059] The video encoder 114 generates, for each CB, e.g., CB 312, a prediction block (PB) indicated by arrow 320. The PB 320 is a prediction of the content of the associated CB 312. A subtraction module 322 generates a difference between the PB 320 and the CB 312, indicated as 324 (or "residual", referring to the difference in the spatial domain). The difference 324 is the block-sized difference between corresponding samples in the PB 320 and the CB 312. The difference 324 is transformed and quantized, and represented as a transform block (TB) indicated by arrow 336. The PB 320 and the associated TB 336 are typically selected from one of many possible candidate CBs, e.g., based on an estimated cost or distortion.

[0060] A candidate coding block (CB) is a CB resulting from one of the prediction modes available to the video encoder 114 for the associated PB and the resulting residual. The TB 336 is a quantized and transformed representation of the difference 324. When combined with the predicted PB in the video decoder 114, the TB 336 reduces the difference between the decoded CB and the original CB 312 by adding signaling to the bitstream.

[0061] Thus, each candidate coding block (CB), i.e., combination of prediction block (PB) and transform block (TB), has an associated coding cost (or "rate") and an associated difference (or "distortion"). The rate is usually measured in bits. The distortion of a CB is usually estimated as a difference of sample values, such as sum of absolute differences (SAD) or sum of squared differences (SSD). The estimated values ​​resulting from each candidate PB may be determined by a mode selector 386 using the difference 324 to determine a prediction mode 387. The prediction mode 387 indicates a decision to use intra-frame prediction, inter-frame prediction, or matrix intra-prediction (MIP) for the current CB. The prediction mode 387 is determined as a mode selected from among possible modes including intra-prediction (including matrix intra-prediction, DC, planar, and angular intra-prediction) or inter-prediction, with associated motion vectors. The prediction mode 387 is usually selected by minimizing a distortion metric resulting from a Lagrangian optimization of the distortion resulting from each candidate mode summed with the result of scaling the associated rate by a lambda value. If matrix intra prediction is used, a matrix intra prediction mode (represented by arrow 388) is also determined to indicate which of several available matrix intra prediction modes is used for the current CB. The search to determine the use and selection of MIP modes for blocks, especially relatively small blocks, can be constrained to mitigate the worst-case memory bandwidth for retrieving matrix coefficients compared to an unconstrained search, as described with reference to Figures 11 to 14, and in another example Figures 17 and 18. The estimation of the coding cost associated with each candidate prediction mode and the coding of the corresponding residual can be performed at a significantly lower cost than the entropy coding of the residual. Thus, by evaluating several candidate modes, the optimal mode in terms of rate versus distortion can be determined.

[0062] The determination of the optimal mode in terms of rate versus distortion is typically performed using various Lagrangian optimizations. The selection of the matrix intra-prediction mode 388 typically involves a determination of the coding cost of the residual data resulting from applying the particular matrix intra-prediction mode. The coding cost can be approximated using the "sum of absolute transform differences" (SATD), which allows for an estimated transformed residual cost to be obtained using a relatively simple transform such as the Hadamard transform. In some cases using a relatively simple transform, the cost resulting from the simplified estimation method is monotonically related to the actual cost determined from the full evaluation. In cases involving monotonically related estimated costs, the simplified estimation method can be used to make the same decision (i.e., intra-prediction mode) while reducing the complexity in the video encoder 114. To accommodate possible non-monotonicity in the relationship between the estimated cost and the actual cost, the simplified estimation method can be used to generate a list of best candidates. The non-monotonicity can result, for example, from the determination of additional modes available for coding the residual data. The number of best candidates in the list can be any number. A more thorough search using the best candidates establishes the optimal mode choice for encoding the residual data for each candidate, which allows for the final selection of the intra-prediction mode along with other mode decisions.

[0063] Other mode decisions include the ability to skip the forward transform, known as "transform skip". Transform skipping is appropriate for residual data that is not sufficiently correlated to reduce the coding cost using expressions such as transform basis functions. Certain types of content, such as relatively simple computer-generated graphics, may exhibit similar properties. In the case of "skipped transform", the residual coefficients are still coded even though the transform itself is not performed.

[0064] A Lagrangian or similar optimization process may be used to both select the optimal partitioning of the CTU into CBs (by the block partitioner 310) as well as to select the best prediction mode from multiple possibilities. Through application of a Lagrangian optimization process of the candidate modes in the mode selection module 386, the intra-prediction mode with the least amount of cost is selected as the "best" mode. The least cost mode is the selected intra-prediction mode 388 and is encoded into the bitstream 115 by the entropy encoder 338. The selection of the intra-prediction mode 388 by the operation of the mode selector module 386 extends to the operation of the block partitioner 310. For example, candidates for the selection of the intra-prediction mode 388 may include a mode applicable to a given block and additional modes applicable to multiple smaller blocks co-located with the given block. In the case of including modes applicable to the given block and the co-located smaller blocks, the process of selecting candidates is also implicitly a process of determining the best hierarchical decomposition of the CTU into CBs.

[0065] In a second stage of operation of the video encoder 114 (called the "encoding" stage), an iterative process of the selected luma code tree and the selected chroma code tree, i.e., each selected CB, is performed in the video encoder 114. In the iterative process, the CBs are encoded into a bitstream 115, as described further herein.

[0066] The entropy encoder 338 supports both variable length coding of syntax elements and arithmetic coding of syntax elements. Arithmetic coding is supported using a context-adaptive binary arithmetic coding process. An arithmetically coded syntax element consists of a sequence of one or more "bins". Bins, like bits, have a value of "0" or "1". However, bins are not encoded into the bitstream 115 as discrete bits. Bins have an associated predicted (or "likely" or "most likely") value and an associated probability known as the "context". If the actual bin being coded matches the predicted value, then a "most likely symbol" (MPS) is coded. Coding the most likely symbol is relatively inexpensive in terms of bits consumed in the bitstream 115, including costs of less than one discrete bit. If the actual bin being coded does not match the expected value, then a "least likely symbol" (LPS) is coded. Coding the least likely symbol is relatively expensive in terms of bits consumed. The bin encoding technique allows for efficient encoding of bins with biased "0" vs. "1" probabilities. For syntax elements with two possible values ​​(i.e., "flags"), a single bin is sufficient. For syntax elements with many possible values, a set of bins is required.

[0067] The presence of a later bin in the sequence may be determined based on the values ​​of earlier bins in the sequence. Furthermore, each bin may be associated with multiple contexts. The selection of a particular context may depend on earlier bins in the syntax element, on bin values ​​of neighboring syntax elements (i.e., bin values ​​from neighboring blocks), etc. Each time a context-coded bin is coded, the context selected for that bin (if any) is updated to reflect the new bin value. In this way, binary arithmetic coding schemes are said to be adaptive.

[0068] Context-free bins ("bypass bins") are also supported by the video encoder 114. Bypass bins are coded assuming equal probability distribution of "0" and "1". Thus, each bin occupies one bit in the bitstream 115. Bypass bins are used when the distribution of values ​​for a particular bin is not biased, since the absence of context saves memory and reduces complexity. One example of an entropy coder that employs context and adaptation is known in the art as CABAC (context-adaptive binary arithmetic coder), and many variations of this coder have been employed in video coding.

[0069] The entropy encoder 338 encodes the prediction mode 387 and, if used for the current CB, encodes the matrix intra prediction mode 388 using a combination of context coding and bypass coded bins. A block of size 4×4 has 35 possible matrix intra prediction modes, and blocks of size not exceeding 8×8 (i.e., 4×8, 8×4, and 8×8) have 19 possible matrix intra prediction modes. For other block sizes, there are 11 possible matrix intra prediction modes. Typically, a list of "most likely matrix modes" (MPMs) is generated by the video encoder 114. The list of most likely modes is typically of fixed length, such as three modes. The list of most likely modes may include modes in previous blocks adjacent to the current block. For example, if the block above or to the left of the current block is used in MIP mode, the corresponding mode is present as the MPM of the current block. If the block above or to the left of the current block is used with angular intra prediction, the MPM list of the current block is filled with MIP modes derived via a lookup table that maps angular intra prediction modes to MIP modes. Additionally, the MPM list of a subsequent CU predicted using normal intra prediction (DC, planar, or angular) may include candidates for intra prediction modes derived from the CU coded using MIP modes, along with a mapping table from MIP modes to candidate normal intra prediction modes. The context coded bins code a flag indicating whether the intra prediction mode is one of the most likely modes. If the intra prediction mode 388 is one of the most likely modes, further signaling is coded using bypass coded bins. The coded further signaling indicates which of the most likely modes corresponds to the matrix intra prediction mode 388, for example using a shortened one-bin string. Otherwise, the intra prediction mode 388 is coded as a "remaining mode".Remaining mode coding uses an alternative syntax, such as a fixed-length code, coded using bypass coded bins, to represent intra-prediction modes other than those present in the most probable mode list.

[0070] In some configurations, it is possible to avoid the complexity of creating MPM lists for both the current CU (encoded using a MIP mode) and subsequent CUs, which are encoded using a normal intra prediction mode and may contain intra prediction mode candidates derived from neighboring blocks encoded in any MIP mode. When the creation of the MPM list is omitted, the binarization of the MIP modes is performed using shortened binary codes to represent the MIP modes. The shortened binary codes make the coding cost of each MIP mode relatively equal, whereas the MPM list makes the coding cost of each MIP added to the MPM list lower. Statistics obtained from the selection of MIP modes do not show a strong bias in the selection of MIP modes included in the MPM list compared to MIP modes not included in the MPM list, which indicates that omitting the generation of the MPM list does not reduce the compression efficiency. For a 4x4 block, 35 possible MIP modes can be encoded using 5-bit or 6-bit codes, with 5 bits for MIP modes 0 to 28 and 6 bits for MIP modes 29 to 34. For 4x8, 8x4, and 8x8 blocks, the 19 possible MIP modes can be encoded using a 4-bit or 5-bit code, with MIP modes 0 to 12 using 4 bits and MIP modes 13 to 18 using 5 bits. For blocks of other sizes, the 11 possible MIP modes can be encoded using a 3-bit or 4-bit code, with MIP modes 0 to 4 using 3 bits and MIP modes 5 to 10 using 4 bits.

[0071] A multiplexer module 384 selects from the tested prediction modes for each candidate CB according to the determined best prediction mode 387 to output PB 320. The candidate prediction modes need not include all possible prediction modes supported by the video encoder 114.

[0072] In inter-frame prediction, a prediction for a block is generated using samples from one or two frames preceding the current frame in the order of encoding the frames in the bitstream. Furthermore, in inter-frame prediction, a single code tree is typically used for both luma and chroma channels. The order of encoding the frames in the bitstream may differ from the order in which the frames are captured or displayed. If one frame is used for prediction, the block is called "unidirectionally predicted" and has one associated motion vector. If two frames are used for prediction, the block is called "bidirectionally predicted" and has two associated motion vectors. For P slices, each CU may be intra-predicted or unidirectionally predicted. For B slices, each CU may be intra-predicted, unidirectionally predicted, or bidirectionally predicted. Frames are typically coded using a "group of pictures" structure, which allows for a temporal hierarchy of frames. The temporal hierarchy of frames allows frames to reference previous and next pictures in the order of displaying the frames. Pictures are coded in the order required to ensure that dependencies for decoding each frame are met.

[0073] The samples are selected according to a motion vector and a reference image index. Inter prediction is mainly described in terms of processing on PUs, not PBs, since the motion vector and reference image index apply to all color channels. In each category (i.e., intraframe and interframe prediction), different techniques can be applied to generate PUs. For example, intra prediction uses values ​​from rows and columns adjacent to the previously reconstructed samples, in combination with an orientation, to generate PUs according to a predefined filtering and generation process. Alternatively, PUs may be described using a small number of parameters. Inter prediction methods may differ in the number of motion parameters and their precision. Motion parameters usually include a reference frame index, which indicates which reference frame to use from a list of reference frames, and a spatial transformation of each reference frame, but may also include more frames, special frames, or complex affine parameters such as scaling and rotation. Additionally, a predefined motion refinement process may be applied to generate dense motion estimates based on the referenced sample blocks.

[0074] By determining and selecting PB 320 and subtracting PB 320 from the original sample block in subtractor 322, a residual with the least coding cost, represented as 324, is obtained and subjected to lossy compression. The lossy compression process consists of a transformation step, a quantization step, and an entropy coding process. A forward primary transform module 326 applies a forward transform to the difference 324 to transform the difference 324 from the spatial domain to the frequency domain and generate primary transform coefficients represented by arrow 328. The primary transform coefficients 328 are passed to a forward secondary transform module 330, which performs a non-separable secondary transform (NSST) process to generate transform coefficients represented by arrow 332. The forward primary transform is usually separable and transforms a set of rows of each block and then a set of columns. The forward primary transform module 326 typically uses a type II discrete cosine transform (DCT-2), but may also use a type VII discrete sine transform (DST-7) and a type VIII discrete cosine transform (DCT-8), for example, horizontally for block widths not exceeding 16 samples, and vertically for block heights not exceeding 16 samples. The transformation of each set of rows and columns is performed by first applying a one-dimensional transform to each row of the block to generate an intermediate result, and then to each column of the intermediate result to generate a final result. The forward secondary transform of module 330 is generally a non-separable transform and is applied only to the residuals of intra-predicted CUs, but may be bypassed. The forward secondary transform operates on either 16 samples (arranged as the top left 4×4 sub-block of the primary transform coefficients 328) or 64 samples (arranged as the top left 8×8 coefficients, arranged as four 4×4 sub-blocks of the primary transform coefficients 328).

[0075] The transform coefficients 332 are passed to a quantization module 334. In the module 334, quantization is performed according to a "quantization parameter" to generate residual coefficients represented by arrow 336. The quantization parameter is constant for a given TB, resulting in uniform scaling to generate the residual coefficients of the TB. Non-uniform scaling may be achieved by applying a "quantization matrix", in which a scaling factor applied to each residual coefficient is derived from a combination of the quantization parameter and a corresponding entry in a scaling matrix, typically having the same size as the TB. The scaling matrix may be sized smaller than the size of the TB, and when applied to the TB, a nearest neighbor approach may be used to provide a scaling value for each residual coefficient from a scaling matrix smaller than the size of the TB. The residual coefficients 336 are provided to an entropy encoder 338 for encoding in the bitstream 115. Typically, the residual coefficients of each TB with at least one significant residual coefficient of the TU are scanned to generate a list of values ​​ordered according to a scan pattern. The scan pattern typically scans the TB as a sequence of 4x4 "sub-blocks", thereby providing a normal scanning operation at a granularity of 4x4 sets of residual coefficients, with the arrangement of the sub-blocks depending on the size of the TB. Additionally, the prediction mode 387, matrix intra prediction mode (if used) 388, and corresponding block partitioning are also coded in the bitstream 115.

[0076] As mentioned above, the video encoder 114 needs access to a frame representation that corresponds to the frame representation seen by the video decoder 134. Thus, the residual coefficients 336 are also inverse quantized by an inverse quantization module 340 to generate inverse transform coefficients represented by arrow 342. The inverse transform coefficients 342 are passed to an inverse secondary transform module 344 to generate intermediate inverse transform coefficients represented by arrow 346. The intermediate inverse transform coefficients 346 are passed to an inverse primary transform module 348 to generate residual samples of the TU represented by arrow 350. The type of inverse transform performed by the inverse secondary transform module 344 corresponds to the type of forward transform performed by the forward secondary transform module 330. The type of inverse transform performed by the inverse primary transform module 348 corresponds to the type of primary transform performed by the primary transform module 326. The summation module 352 sums the residual samples 350 and the PU 320 to generate reconstructed samples of the CU (indicated by arrow 354).

[0077] The reconstructed samples 354 are passed to a reference sample cache 356 and an in-loop filter module 368. The reference sample cache 356, typically implemented using static RAM on an ASIC (thus avoiding costly off-chip memory accesses), provides the minimum sample storage required to satisfy dependencies for generating the intra-frame PBs of subsequent CUs in the frame. The minimum dependencies typically include a "line buffer" of samples along the bottom of a row of CTUs for use in the next row of CTUs and a column buffering a range of columns set by the height of the CTU. The reference sample cache 356 provides reference samples (represented by arrow 358) to a reference sample filter 360. The sample filter 360 applies a smoothing operation to generate filtered reference samples (indicated by arrow 362). The filtered reference samples 362 are used by an intra-frame prediction module 364 to generate an intra-predicted block of samples, represented by arrow 366. For each candidate intra-prediction mode, the intra-frame prediction module 364 generates a block of samples 366. The block of samples 366 is generated by module 364 using techniques such as DC, planar or angular intra prediction, but not matrix intra prediction.

[0078] If the mode selector 386 selects matrix intra prediction for the current CB, a matrix intra prediction mode 388 is used to select (read) matrix coefficients 363 from the coefficient memory 392. The matrix coefficients 363 are passed to a matrix intra prediction module 390, which performs a matrix multiplication using the matrix coefficients 363 and the reference samples 358 to generate a matrix intra prediction block 393. The multiplexer 384 outputs the matrix intra prediction block 393 as the PB 320. The coefficient memory 392 has a limited bandwidth required to provide the matrix coefficients 393. In particular, different matrix intra prediction modes are used for each successive block to establish a worst-case memory bandwidth requirement. The mode selector 386 is operable to select a MIP mode for a block under a constraint that reduces the worst-case memory bandwidth of the coefficient memory 392, as described in connection with the operation of Figures 11 to 14. Reducing the worst-case memory bandwidth of the coefficient memory 392 reduces complexity, e.g., memory hardware area, without a proportionate loss in coding efficiency of the video encoder 114. The complexity can be reduced without a proportionate loss in coding efficiency because the statistics of MIP mode selection in an unconstrained search do not typically impose constraints on the MIP mode selection in the mode selector 386. Thus, the worst-case memory bandwidth is reduced without a commensurate loss in coding performance.

[0079] The in-loop filter module 368 applies several stages of filtering to the reconstructed samples 354. The filtering stages include a "deblocking filter" (DBF) that applies smoothing along CU boundaries to reduce artifacts due to discontinuities. Another filtering stage present in the in-loop filter module 368 is the "adaptive loop filter" (ALF), which applies a Wiener-based adaptive filter to further reduce distortion. A further filtering stage available in the in-loop filter module 368 is the "sample adaptive offset" (SAO) filter. The SAO filter works by first classifying the reconstructed samples into one or more categories and then applying an offset at the sample level according to the assigned category.

[0080] The filtered samples, represented by arrow 370, are output from the in-loop filter module 368. The filtered samples 370 are stored in a frame buffer 372. The frame buffer 372 typically has a capacity to store several images (e.g., up to 16) and is implemented in the memory 206. The frame buffer 372 is typically not implemented with on-chip memory due to the large memory consumption required. Access to the frame buffer 372 is therefore costly in terms of memory bandwidth. The frame buffer 372 provides reference frames (represented by arrow 374) to a motion estimation module 376 and a motion compensation module 380.

[0081] The motion estimation module 376 estimates several "motion vectors" (shown as 378), each of which is a Cartesian space offset from the position of the current CB and refers to a block in one of the reference frames in the frame buffer 372. A filtered block of reference samples (shown as 382) is generated for each motion vector. The filtered reference samples 382 form further candidate modes available for potential selection by the mode selector 386. Furthermore, for a given CU, the PU 320 may be formed using one reference block ("unidirectional prediction") or two reference blocks ("bidirectional prediction"). For a selected motion vector, the motion compensation module 380 generates the PB 320 according to a filtering process according to the sub-pixel accuracy of the motion vector. Thus, the motion estimation module 376 (operating with many candidate motion vectors) may perform a simplified filtering process than that of the motion compensation module 380 (operating with only the selected candidate), reducing computational complexity. If the video encoder 114 selects inter prediction for a CU, then the motion vector 378 is encoded into the bitstream 115.

[0082] 3 is described with reference to Versatile Video Coding (VVC), other video coding encoding standards or implementations may employ the processing stages of modules 310-386. The frame data 113 (and bitstream 115) may also be read from (or written to) memory 206, hard disk drive 210, a CD-ROM, a Blu-ray disc, or other computer-readable storage medium. Additionally, the frame data 113 (and bitstream 115) may be received from (or sent to) an external source, such as a server connected to a communications network 220 or a radio frequency receiver.

[0083] The video decoder 134 is shown in FIG. 4. The video decoder 134 of FIG. 4 is an example of a Versatile Video Coding (VVC) video decoding pipeline, although other video codecs may be used to perform the processing steps described herein. As shown in FIG. 4, a bitstream 133 is input to the video decoder 134. The bitstream 133 may be read from the memory 206, the hard disk drive 210, a CD-ROM, a Blu-ray disc, or other non-transitory computer-readable storage medium. Alternatively, the bitstream 133 may be received from an external source, such as a server connected to the communication network 220 or a radio frequency receiver. The bitstream 133 includes encoded syntax elements that represent captured frame data to be decoded.

[0084] The bitstream 133 is input to the entropy decoder module 420. The entropy decoder module 420 extracts syntax elements from the bitstream 133 by decoding a series of "bins" and passes the values ​​of the syntax elements to other modules in the video decoder 134. The entropy decoder module 420 uses an arithmetic decoding engine to decode each syntax element as a series of one or more bins. Each bin can use one or more "contexts", where a context represents a probability level used to code the "1" and "0" values ​​of the bin. If multiple contexts are available for a bin, a "context modeling" or "context selection" process is performed to select one of the available contexts to decode the bin. The process of decoding a bin forms a continuous feedback loop. The number of processes in the feedback loop is preferably minimized to allow the entropy decoder 420 to achieve a high throughput in bins / second. Context modeling relies on other attributes of the bitstream known to the video decoder 134 when selecting a context, i.e. attributes preceding the current bin. For example, the context may be selected based on the quadtree depth of the current CU in the coding tree. The dependencies are preferably based on attributes that are well known prior to decoding the bin or that can be determined without requiring long sequential processing.

[0085] The quadtree depth of the code tree is an example of a context modeling dependency that is easily known. Intra prediction modes, especially matrix intra prediction modes, are examples of context modeling and binarization dependencies that are relatively difficult or computationally expensive to determine. Matrix intra prediction modes are coded as an index into a list of "most likely modes" (MPMs) or as an index into a list of "remaining modes", and select between MPMs and remaining modes according to the decoded context coding flags. Other intra prediction modes are coded as an index into a list of "most likely modes" (MPMs) or as an index into a list of "remaining modes", and select between MPMs and remaining modes according to the decoded intraluma_MPM flags. If MPMs are used to code the matrix intra prediction modes, a shortened single bin string in the range 0 to 2 selects one of the MPMs from the MPM list. If remaining modes are used, a fixed length codeword is decoded to select which of the remaining (non-MPM) modes to use. The number of available MPM modes can be either 35, 19 or 11, depending on the block size. Thus, for an MPM list of length 3, the number of remaining modes is 32, 16, or 8, respectively. The remaining modes can be efficiently represented by fixed-length codewords of length 5, 4, or 3, respectively. Determining both the most likely mode and the remaining modes requires a significant amount of processing, including a dependency on the intra-prediction modes of the neighboring blocks. For example, the neighboring blocks may be blocks above and to the left of the current block. If the neighboring blocks use angular intra-prediction, a table lookup can be performed to map the angular intra-prediction modes to matrix intra-prediction modes for use in creating the MPM list. Alternatively, the configuration can use shortened value codewords to encode the MIP modes in each case, i.e., depending on whether the number of modes is 35, 19, or 11. The entropy decoder module 420 applies an arithmetic coding algorithm, for example, "context-adaptive binary arithmetic coding" (CABAC), to decode syntax elements from the bitstream 133.The decoded syntax elements are used to reconstruct parameters in video decoder 134. The parameters include residual coefficients (represented by arrow 424) and mode selection information such as intra prediction modes (represented by arrow 458). The mode selection information also includes information such as motion vectors and the division of each CTU into one or more CBs. The parameters, typically together with sample data from previously decoded CBs, are used to generate the PBs.

[0086] The residual coefficients 424 are input to an inverse quantization module 428. The inverse quantization module 428 performs inverse quantization (or "scaling") on the residual coefficients 424 to generate reconstructed intermediate transform coefficients, indicated by arrow 432, depending on the quantization parameters. The reconstructed intermediate transform coefficients 432 are passed to an inverse secondary transform module 436 to apply a secondary transform or are bypassed. The inverse secondary transform module 436 generates reconstructed transform coefficients 440. If the use of a non-uniform inverse quantization matrix is ​​indicated in the bitstream 133, the video decoder 134 reads the quantization matrix from the bitstream 133 as a series of scaling coefficients and arranges them as a matrix. The inverse scaling uses the quantization matrix in combination with the quantization parameters to generate the reconstructed intermediate transform coefficients 432.

[0087] The reconstructed transform coefficients 440 are passed to an inverse linear transform module 444, which converts the coefficients back from the frequency domain to the spatial domain. The operation by the module 444 results in a block of residual samples, indicated by arrow 448. The block of residual samples 448 is equal in size to the corresponding CU. The residual samples 448 are provided to a summing module 450, where they are summed with a decoded PB, indicated as 452, to generate a block of reconstructed samples, indicated as arrow 456. The reconstructed samples 456 are provided to a reconstructed sample cache 460 and an in-loop filtering module 488, which generates a block of reconstructed frame samples, indicated as 492. The frame samples 492 are written to a frame buffer 496.

[0088] The reconstructed sample cache 460 operates similarly to the reconstructed sample cache 356 of the video encoder 114. The reconstructed sample cache 460 provides storage for reconstructed samples necessary to intra-predict subsequent CBs without the memory 206 (e.g., by using the data 232 instead, typically an on-chip memory). Reference samples, represented by arrow 464, are retrieved from the reconstructed sample cache 460 and fed to a reference sample filter 468 to generate filtered reference samples, indicated by arrow 472. The filtered reference samples 472 are fed to an intraframe prediction module 476. The module 476 generates a block of intra-predicted samples, indicated by arrow 480, according to the intra-prediction mode parameter 458 signaled in the bitstream 133 and decoded by the entropy decoder 420. The block of samples 480 is generated using a mode such as DC, planar or angular intra-prediction, rather than matrix intra-prediction.

[0089] If the prediction mode of the CB is indicated in the bitstream 133 to use intra prediction (other than matrix intra prediction), the intra predicted samples 480 form the decoded PB 452 via the multiplexer module 484. Intra prediction generates a prediction block (PB) of samples, i.e., a block of one color component, derived using "neighboring samples" of the same color component. The neighboring samples are adjacent to the current block and precede it in the block decoding order, and therefore have already been reconstructed. If the luma and chroma blocks are at the same position, they may use different intra prediction modes. However, each of the two chroma channels shares the same intra prediction mode. Intra prediction is classified into three types: "DC intra prediction" involves filling the PB with a single value that represents the average of the neighboring samples. "Planar intra prediction" involves filling the PB with samples according to a plane, where the DC offset and vertical and horizontal gradients are derived from the neighboring samples. "Angular intra prediction" involves filling a PB with neighboring samples that have been filtered and propagated in a particular direction (or "angle") across the PB. In VVC, 65 angles are supported, with rectangular blocks having additional angles available that are not available in square blocks, yielding a total of 87 angles. A fourth type of intra prediction is available for chroma PBs, whereby the PB is generated from co-located luma reconstructed samples according to a "Cross-Component Linear Model" (CCLM) mode. Three different CCLM modes are available, each using a different model derived from neighboring luma and chroma samples. The derived model is then used to generate a block of samples for the chroma PB from the co-located luma samples.

[0090] When the bitstream 133 indicates that the prediction mode of CB is matrix intra prediction, the matrix intra prediction mode 458 is decoded and provided to the coefficient memory 486 and the matrix intra prediction module 482. The matrix coefficients 481 are read from the coefficient memory 486 of the selected matrix intra prediction mode and sent to the matrix intra prediction module 482. The selection of the matrix coefficients involves a memory read operation from the coefficient memory 486, and as explained with reference to Figures 8 and 9, the frequency of selecting MIP mode for a certain block size establishes the worst case memory bandwidth limit for memory accesses.

[0091] If the bitstream 133 indicates that the prediction mode of the CB is inter prediction, the motion compensation module 434 uses the motion vector and the reference frame index to select and filter a block of samples 498 from a frame buffer 496 to generate a block of inter prediction samples represented at 438. The block of samples 498 is obtained from a previously decoded frame stored in the frame buffer 496. In the case of bi-prediction, two blocks of samples are generated and mixed to generate samples for the decoded PB 452. The frame buffer 496 is filled with filtered block data 492 from the in-loop filtering module 488. Like the in-loop filtering module 368 of the video encoder 114, the in-loop filtering module 488 performs any of DBF, ALF and SAO filtering. Generally, the motion vector is applied to both the luma and chroma channels, although the filtering of sub-sampled interpolation of the luma and chroma channels is different.

[0092] 5 is a schematic block diagram illustrating an example partition 500 that can be obtained by partitioning or branching a region into one or more sub-regions in a Versatile Video Coding tree structure. The partition shown in the example partition 500 can be performed by the block partition unit 310 of the encoder 114, which partitions each CTU into one or more CUs or CBs according to a code tree, as determined by Lagrangian optimization, as described with reference to FIG.

[0093] Although the partition example 500 illustrates a square region being partitioned into other, possibly non-square, sub-regions, it should be understood that diagram 500 illustrates a possible partition and that the containing region need not be square. If the containing region is non-square, the dimensions of the resulting blocks are scaled according to the aspect ratio of the containing block. When a region is not partitioned further, i.e., at a leaf node of the coding tree, a CU occupies the region. The particular subdivision of a CTU by block partitioner 310 into one or more CUs is referred to as the "coding tree" of the CTU.

[0094] The process of subdividing a region into subregions must end when the resulting subregion reaches the minimum CU size. In addition to restricting CUs to prohibit block regions smaller than a certain minimum size, e.g., 16 samples, CUs are restricted to have a minimum width or height of 4. Other minimums are possible, either in terms of width and height or in terms of width or height. The process of subdivision may end before the deepest level of decomposition, resulting in a CU larger than the minimum CU size. It is also possible that no splitting occurs and a single CU occupies an entire CTU. A single CU occupying an entire CTU is the size of the largest available coding unit. Due to the use of subsampled chroma formats, such as 4:2:0, the configuration of the video encoder 114 and the video decoder 134 may end splitting regions for chroma channels earlier than for luma channels.

[0095] At the leaf nodes of the coding tree, there are CUs that cannot be further subdivided. For example, leaf node 510 contains one CU. At the non-leaf nodes of the coding tree, there is a further division into two or more nodes, each of which may be a leaf node forming one CU or a non-leaf node containing further division into smaller regions. At each leaf node of the coding tree, there is one coding block per color channel. If the division ends at the same depth for both luma and chroma, three CBs are generated that are in the same position. If the division ends at a depth where luma is deeper than chroma, multiple luma CBs are in the same position as the CBs of the chroma channels.

[0096] Quadtree partition 512 divides the containing region into four equally sized regions, as shown in Figure 5. Compared to HEVC, Versatile Video Coding (VVC) achieves more flexibility by including horizontal binary partition 514 and vertical binary partition 516. Partitions 514 and 516 each divide the containing region into two equally sized regions. The partitions are along either horizontal boundaries (514) or vertical boundaries (516) within the containing block.

[0097] Further flexibility is achieved in Versatile Video Coding by adding a horizontal third division 518 and a vertical third division 520. The third divisions 518 and 520 divide the block into three regions either horizontally (518) or vertically (520) along ¼ and ¾ of the width or height of the containing region. The combination of quadtrees, binary trees, and ternary trees is called a "QTBTTT". The root of the tree contains zero or more quadtree divisions (the "QT" section of the tree). Once the QT section is finished, zero or more bisections or third divisions occur (the "maybe tree" or "MT" section of the tree), eventually terminating with CBs or CUs at the leaf nodes of the tree structure. If the tree represents all color channels, the leaf nodes of the tree are CUs. If the tree represents luma or chroma channels, the leaf nodes of the tree are CBs.

[0098] Compared to HEVC, which only supports quad-trees and, as a result, square blocks, QTBTTT allows for more CU sizes, especially by considering possible recursive application of binary and / or ternary tree partitioning. By limiting the partitioning options, the possibility of abnormal block sizes (non-square) can be reduced, and partitioning into samples with block width or height less than 4 or not a multiple of 4 can be eliminated. In general, the restrictions are applied by considering luma samples. However, in the described configuration, the restrictions can be applied separately to blocks of chroma channels. Applying the restrictions of partitioning options to chroma channels, for example, when the frame data is in 4:2:0 chroma format or 4:2:2 chroma format, the minimum block size can be different for luma vs. chroma. Each partition generates a sub-region with the same, half, or quarter side dimensions relative to the containing region. And, because the CTU size is a power of 2, the side dimensions of all CUs are also a power of 2.

[0099] 6 is a schematic flow diagram illustrating a data flow 600 of the QTBTTT (or "code tree") structure used in Versatile Video Coding. The QTBTTT structure is used for each CTU and defines the partitioning of the CTU into one or more CUs. The QTBTTT structure of each CTU is determined by the block partitioner 310 of the video encoder 114 and encoded into the bitstream 115 or decoded from the bitstream 133 by the entropy decoder 420 of the video decoder 134. The data flow 600 further characterizes the permissible combinations available to the block partitioner 310 for partitioning the CTU into one or more CUs according to the partitioning states shown in FIG. 5.

[0100] Starting from the top level of the hierarchy, i.e., CTU, zero or more quadtree splits are performed first. Specifically, a quadtree (QT) split decision 610 is performed by the block splitter 310. A decision at 610 returning a "1" symbol indicates a decision to split the current node into four subnodes according to the quadtree split 512. As a result, four new nodes are generated, as at 620, and a recursion to the QT split decision 610 is performed for each new node. Each new node is considered in raster (or Z-scan) order. Alternatively, if the QT split decision 610 indicates that no further splits are to be performed (returning a "0" symbol), the quadtree split stops and a multi-tree (MT) split is subsequently considered.

[0101] First, an MT split decision 612 is made by the block splitter 310. At 612, a decision to perform MT splitting is indicated. Returning a "0" symbol at decision 612 indicates that no further splitting of the node into subnodes is to be performed. If no further splitting of the node is to be performed, the node is a leaf node of the coding tree and corresponds to a CU. The leaf node is output at 622. Alternatively, if the MT split 612 indicates a decision to perform MT splitting (returning a "1" symbol), the block splitter 310 proceeds to direction decision 614.

[0102] Orientation decision 614 indicates the orientation of the MT division as horizontal ("H" or "0") or vertical ("V" or "1"). If decision 614 returns a "0" indicating horizontal orientation, the block divider 310 proceeds to decision 616. If decision 614 returns a "1" indicating vertical orientation, the block divider 310 proceeds to decision 618.

[0103] In each of decisions 616 and 618, the number of divisions for the MT division is indicated as either 2 (bisection or "BT" node) or 3 (trisection or "TT") in the BT / TT division. That is, if the direction indicated by 614 is horizontal, then the BT / TT division decision 616 is made by the block divider 310, and if the direction indicated by 614 is vertical, then the BT / TT division decision 618 is made by the block divider 310.

[0104] The BT / TT split decision 616 indicates whether the horizontal split is bisection 514, indicated by returning a "0", or trisection 518, indicated by returning a "1". If the BT / TT split decision 616 indicates bisection, then in a HBT CTU node generation step 625, two nodes are generated by the block divider 310 according to the horizontal bisection 514. If the BT / TT split 616 indicates trisection, then in a HTT CTU node generation step 626, three nodes are generated by the block divider 310 according to the horizontal trisection 518.

[0105] BT / TT split decision 618 indicates whether the vertical split is bisection 516, indicated by returning "0", or trisection 520, indicated by returning "1". If BT / TT split 618 indicates bisection, then in VBT CTU node generation step 627, two nodes are generated by block splitter 310 according to vertical bisection 516. If BT / TT split 618 indicates trisection, then in VTT CTU node generation step 628, three nodes are generated by block splitter 310 according to vertical trisection 520. For each node resulting from steps 625-628, data flow 600 is repeated in the order from left to right and top to bottom according to direction 614, returning to MT split decision 612. As a result, binary tree splitting and trisection partitioning can be applied to generate CUs of various sizes.

[0106] 7A and 7B show an example partition 700 of a CTU 710 into multiple CUs or CBs. An example CU 712 is shown in Fig. 7A. Fig. 7A shows the spatial arrangement of CUs within the CTU 710. The example partition 700 is also shown as a code tree 720 in Fig. 7B.

[0107] For each non-leaf node of the CTU 710 in FIG. 7A, e.g., nodes 714, 716, 718, the contained nodes (further splits or CUs) are scanned or probed in "Z-order" to generate a list of nodes as shown in the columns of the coding tree 720. In a quadtree partition, the Z-order scan results from top-left to right, then bottom-left to right. For horizontal and vertical partitions, the Z-order scan is simplified to a top-down scan and a left-right scan, respectively. The coding tree 720 in FIG. 7B lists all nodes and CUs according to the applied scan order. Each partition generates a list of two, three, or four new nodes at the next tree level until a leaf node (CU) is reached.

[0108] Once the block divider 310 has decomposed the image, ultimately down to CUs, and used to generate respective residual blocks (324) as described with reference to Figure 3, the residual blocks are subject to forward transformation and quantization by the video encoder 114. The resulting TBs 336 are scanned in order to form a sequential list of residual coefficients as part of the operation of the entropy coding module 338. A similar process is performed by the video encoder 114 to obtain the TBs from the bitstream 133.

[0109] As a result of dividing the CPU into CUs, CUs of small size (e.g., 4x4, 4x8, or 8x4) appear spatially adjacent in the frame. Moreover, as an advantage of the hierarchical Z-order scan of the code tree, CUs of small size are processed consecutively in time. In particular, quadtree division of an 8x8 region results in a series of four 4x4 CUs, bisection of an 8x4 or 4x8 region results in a pair of 4x4 CUs, and bisection of an 8x8 region results in a pair of 4x8 regions or a pair of 8x4 regions. Each resulting region (of size 4x8 or 8x4) can form a CU or be further divided into 4x4 CUs. Applying trisection to a region of 64 samples produces three regions of size 16, 32, and 16, i.e., 4x4, 4x8 or 8x4, and 4x4. Applying trisection to a region of 128 samples produces three regions of size 32, 64, and 32, e.g., 4x8, 8x8, and 4x8. Small blocks (e.g. 4x4, 4x8, 8x4) are the result of dividing regions of size such as 64 or 128, so they are seen together both spatially and in the Z-order scan of the CTU.

[0110] FIG. 8 is a data flow 800 detailing the operation of matrix intra prediction modules 390 and 482 of video encoder 114 and video decoder 134, respectively, using 8×8 blocks. Matrix intra prediction is performed only on the luma channel in the example of FIG. 8, and there is no need to consider different block sizes resulting from using chroma formats such as 4:2:0 or 4:2:2. Modules 390 and 482 output blocks of matrix prediction samples such as 393 or 483, respectively. When mode selector 386 selects matrix intra prediction for a CU, block 393 is used as PB 320 in video encoder 114. When decoding the prediction mode of the CU from bitstream 133 indicates the use of matrix intra prediction, block 483 is used as PB 483.

[0111] The operation of modules 390 and 482 includes three steps: 1. averaging, 2. matrix multiplication and offset (bias) addition, and 3. bilinear interpolation. Bilinear interpolation is only performed if the size of the CU is greater than 4x4. The averaging step works as follows: A reference sample 802 (e.g., 358 or 464) is received in the data flow 800 and assigned as a top sample 822 and a left sample 820. If the width and height of the luma CB are greater than 4, the top sample 822 is divided into four sets (four pairs in the example of an 8x8 block in FIG. 8). The content values ​​of each of the four sets are averaged to generate four filtered top samples 826. Similarly, the left sample 820 is divided into four sets (four pairs in the example of an 8x8 block in FIG. 8). The content values ​​of each of the four sets are averaged to generate four filtered left samples 824. Thus, a total of eight filtered samples are input to the matrix multiplication module 828.

[0112] If the block width and height are equal to four (instead of eight as shown in FIG. 8), then the four top samples 822 are split into two pairs that are each averaged to produce two filtered top samples 826. Similarly, the four left samples 820 are split into two pairs that are each averaged to produce two filtered left samples 824, with a total of four filtered samples being input to the matrix multiplication module 828.

[0113] In the second step (matrix multiplication), the received matrix coefficients (363 or 481) are selected according to the matrix intra-prediction mode (388 or 458) and are also input to the matrix multiplication module 828. A set of offset or bias values ​​are also input to the matrix multiplication module. The offset or bias values ​​are added to the result of the matrix multiplication to include any DC shift. The matrix coefficients and bias values ​​are pre-determined. In other words, the matrix coefficients and bias values ​​are the result of an "offline" training process and are considered constant values ​​for the video encoder 114 and the video decoder 134.

[0114] For 4x4 CB, 35 MIP modes with 18 sets of matrix coefficients and bias values ​​(set A) are available. For 4x8, 8x4, and 8x8 CB, 19 MIP modes with 10 sets of matrix coefficients and bias values ​​(set B) are available. For other CB sizes, 11 MIP modes with 6 sets of matrix coefficients and bias values ​​(set C) are available. A given set of matrix coefficients and bias values ​​can be used for two MIP modes. One MIP mode uses the provided values, and the other MIP mode uses the transpose of the provided values. Furthermore, in some cases, a set of matrix coefficients and bias values ​​is dedicated to one MIP mode. Each of the three cases applies to each of sets A to C. The sizes of sets A to C are shown below, along with the number of words selected by 362 or 481 to be used for the generation of one PB (i.e., the application of the MIP mode to one CU) given each of sets A to C. Set A: 18 matrices of size 16x4, 18 offset vectors of size 16. The size of the values ​​chosen for CB is 16x4+16=80 words. Set B: 10 matrices of size 16x8, 10 offset vectors of size 16. The size of the values ​​chosen for CB is 16x8+16=144 words. Set C: 6 matrices of size 64x8, 10 offset vectors of size 64. The size of the values ​​chosen for CB is 64x8+64=576 words.

[0115] The video encoder 114 and the video decoder 134 process the video data at a pixel rate determined by the frame size and frame rate. Furthermore, the size of the luma CB is a multiple of 4 in width and height. Therefore, the memory bandwidth requirement for sets A-C can be expressed by accessing 4×4 luma samples. The worst-case access density for each set to sets A-C, respectively, i.e., each set of minimum block size, when there is no constraint on the use of MIP mode for each CU, is as follows: Set A: 4x4CB requires 80 words per 4x4 sample area. Set B: 4x8 and 8x4CB require 144÷2=72 words per 4x4 sample region, 8x8CB requires 144÷4=36 words per 4x4 sample region. Set C: 8x16 and 16x8 CBs require 576÷8=72 words per 4x4 sample region, 8x32, 16x16, and 32x8 CBs require 576÷16=36 words per 4x4 sample region, and larger size CBs require fewer words per 4x4 sample region.

[0116] As shown by the access density above, if there are no constraints on the use of MIP mode, in the worst case, all CBs may use MIP mode, and the code tree may decompose each CTU into a small CB size, which was used above to show the worst case coefficient memory (486, 392) bandwidth. The nominal word size of the matrix coefficients and bias values ​​is 16 bits, but fewer bits, e.g., 10 bits, may be sufficient. Fetching words in groups is one possible option. However, the memory bandwidth burden remains rather high.

[0117] Statistics on MIP mode selection show that typically in 20% of cases for a wide test set (as defined in JVET common test conditions document JVET-N1010), MIP mode is selected for adjacently located CBs when considering the top and left blocks. Thus, by limiting the frequency of MIP mode application, it is possible to reduce the worst-case memory bandwidth required for the coefficient memories 486 and 392 without incurring a commensurate reduction in compression efficiency gains in terms of MIP mode availability. Constraints on MIP mode selection are described with reference to Figures 11 to 16.

[0118] A matrix multiplication module 828 performs a matrix multiplication using a set of matrix coefficients (363 or 481) and the filtered reference samples (i.e., 824 and 826). A bias value is added to the output of the matrix multiplication to form a sparse block 830. The sparse block 830 is represented as the shaded samples that partially fill 393 or 483. The remaining samples of 393 or 483 are derived using bilinear interpolation with contributions from either the top reference sample 822 and the filtered left reference sample 824, or the left reference sample 820 and the filtered top sample 822 (as shown in FIG. 8).

[0119] FIG. 9A shows an example of a CTU 900. The CTU 900 has a region where the memory access bandwidth of MIP mode is applied. Matrix intra prediction is applied within a VPDU-sized region, i.e., a 64×64 quadrant of a 128×128 CPU 900. For example, the 512 luma sample region where the budget is applied includes (i) a 32×16 region 912, (ii) a 16×32 region 914, and (iii) a 64×8 region 916. The region 912 is further decomposed into various CUs, i.e., 912a-912h in FIG. 9B. In the example of FIG. 9B, CU 912a has a size of 16×4 and therefore belongs to set C, and CU 912d has a size of 8×8 and therefore belongs to set B, as well as 9112b, 912c, and 912e (all of size 8×4). CU912f and CU912g are each 4x4 and belong to set A. As described with reference to Figures 11 to 16, the area of ​​a region of 512 luma samples can be used in the described configuration as a threshold for determining whether to apply a constraint to a MIP mode. Since sets A to C above typically result from a division of regions, using the area of ​​a region of 512 samples is suitable as a constraint. However, the threshold may relate to a different area, as described below.

[0120] Figure 10 shows a code tree 10 corresponding to the example CTU 900 of Figure 9A. Region 912, as well as regions 914 and 916, correspond to nodes of the code tree where budgets for contained CUs are established for reading MIP matrix coefficients. The decomposition of a region into one or more CUs is not shown for simplicity. For each CU configured to use MIP mode, a budget according to the CU size for mapping to one of sets A-C is subtracted from the budget of the region (i.e., 912, 914, 916), as described below with reference to Figures 11 to 16.

[0121] FIG. 11 illustrates a method 1100 for encoding coding units of an image frame into a video bitstream 115. The method 1100 may be embodied by an apparatus such as an configured FPGA, ASIC, or ASSP. Furthermore, the method 1100 may be executed by the video encoder 114 under execution of the processor 205. Thus, the method 1100 may be stored in a computer-readable storage medium and / or memory 206. The operation of the method 1100 for encoding a sequence of coding units, an image frame, into a bitstream 115 places a limit on the use of matrix intra prediction. Thus, the operation of the method 1100 places a limit on the memory bandwidth required for matrix intra prediction compared to the worst-case potential memory bandwidth in the absence of the limit. The method 1100 begins with step 1110 of dividing a frame into CTUs.

[0122] In a step 1110 of dividing a frame into CTUs, the block divider 310 divides the current frame of frame data 113 into a matrix of CTUs under the execution of the processor 205. The coding process of the CTUs resulting from the division is started. Control in the processor passes from step 1110 to a code tree determination step 1120.

[0123] In a coding tree determination step 1120, under the execution of the processor 205, the video encoder 114 tests various partitioning options as described with reference to Figures 5 to 7, and performs the operations of a coding unit determination step 1130 to arrive at a coding tree for the CTU. The operations of step 1120 are described with reference to Figure 12. Control in the processor 205 passes from step 1120 to a coding unit prediction mode determination step 1130.

[0124] In a coding unit determination step 1130, the video encoder 114, under execution of the processor 205, determines a prediction mode to use in encoding the selected coding unit into the bitstream 115. The coding unit may be selected according to a scan pattern. Operation of step 1130 is further described with reference to FIG. 13. Once a prediction mode for a coding unit has been selected, control passes from step 1130 in the processor 205 to a coding unit encoding step 1140. In selecting a prediction mode for a coding unit, a particular combination of partitions to arrive at a given coding unit is also selected, since the coding unit itself results from a hierarchy of partitions of a coding tree, thereby determining the coding tree.

[0125] In encoding a coding unit, step 1140, the entropy encoder 338, under execution of the processor 205, encodes the coding unit determined in step 1130 into the bitstream 115. The determined code tree is efficiently encoded in step 1140 by the entropy encoder 338, under execution of the processor 205, into the bitstream 115 using "split flags" and other syntax elements to indicate the selected split, as shown in Figures 5 and 6.

[0126] Operations at step 1140 are further described with reference to Figure 14. Control in processor 205 passes from step 1140 to a final encoding unit test step 1150.

[0127] In a last coding unit test step 1150, the processor 205 tests whether the current coding unit is the last coding unit in the code tree of step 1120. If the current coding unit is the last one in the code tree of step 1120 ("YES" in step 1150), control in the processor 205 passes to a last CTU test step 1160. If the current coding unit is not the last one in the code tree of step 1120 ("NO" in step 1150), the next coding unit in the code tree of step 1120 is selected using the scan pattern for decision and encoding, and control in the processor 205 passes to step 1130. Thus, step 1130 is performed for each CU resulting from the code tree determined in step 1120.

[0128] In a last CTU test step 1160, the processor 205 tests whether the current CTU is the last CTU in the slice or frame. If not ("NO" at step 1160), the video encoder 114 proceeds to the next CTU in the frame and control in the processor 205 returns from step 1160 to step 1120 to continue processing the remaining CTUs in the frame. If the CTU is the last one in the frame or slice, step 1160 returns "YES" and the method 1100 ends. The operation of the method 1100 results in the entire image frame being encoded into a bitstream as a series of CTUs.

[0129] The method 1100 is performed for each image frame in a video sequence. The method 1100 can determine the CU for each CTU. In other words, the CU for a CTU is first determined in one pass or pipeline stage, followed by a second stage of encoding into a bitstream. The method 1100 may also determine the CU for each VPDU in a finer granularity, for example, to reduce memory consumption due to the smaller area of ​​the VPDU compared to the CTU.

[0130] FIG. 12 shows a method 1200 for determining a code tree for a CTU performed in step 1120. The method 1200 receives a CTU, generates candidate partitions and candidate coding units for evaluation, and finally selects the partitions and coding units to be coded into the bitstream 115. In particular, the method 1200 establishes a MIP mode memory bandwidth quota at a particular node or region in the code tree that limits the application of MIP mode in the coding unit under the node, i.e., in the spatial domain. The method 1200 may be embodied by an apparatus such as a configured FPGA, ASIC, or ASSP. Furthermore, the method 1200 may be executed by the video encoder 114 under the execution of the processor 205. Thus, the method 1200 may be stored in a computer-readable storage medium and / or memory 206. The method 1200 is performed for each node (region) in the candidate code tree. The method 1200 starts with a region area test step 1210.

[0131] In a region test step 1210, the video encoder 114, under execution of the processor 205, tests the area occupied by a candidate node of the coding tree according to the available partitioning options (using the partitioning options described in connection with FIG. 5). In the described configuration, the region relates to a node of the coding tree where a further partitioning and CUs may be formed before partitioning stops. The region does not have to correspond to the direct parent of the current coding unit. That is, the parent region for which the MIP mode memory access bandwidth budget is set may be multiple nodes above the current node of the coding tree. For example, a region of 512 luma samples can be partitioned into many smaller coding units, e.g., sizes 4×4, 4×8, 8×4, etc. As the encoder search proceeds, different candidate nodes resulting from the partitioning of the candidates are tested. Within each candidate node, different prediction modes of the resulting coding units are tested. If the area of ​​luma samples occupied by the region does not meet the threshold, the region area test evaluates to true and control in the processor 205 passes from step 1210 to a budget setting step 1220. If the area of ​​luma samples occupied by the region meets the threshold, the region area test evaluates to false and control in the processor 205 passes from step 1210 to a partition generation step 1230. In one embodiment, as shown in the examples of Figures 11 to 16, the threshold is 512 samples and is met when the region has an area greater than 512 samples. Other thresholds may be used, as described below. The threshold is typically predetermined based on the required performance of the encoder 114 and the decoder 134.

[0132] In a budget setting step 1220, the video encoder 114, under the execution of the processor 205, sets a budget for a region corresponding to the area of ​​the current node of the coding tree. This budget is available to all sub-nodes (sub-regions) in the current region resulting from the division of the current region into smaller regions. For all regions of 512 luma samples or less, the budget for the case of applying MIP mode can be taken into account when the coding unit prediction mode is being evaluated. The area of ​​512 luma samples may be subject to, for example, a trisection, resulting in coding units of area 128, 256, and 128 luma samples. The dimensions of the coding unit of area 128 luma samples may be 8x16 or 16x8, in which case the worst case memory bandwidth limit may be reached, resulting in the inability to use MIP mode for other coding units in the region, including the possibility of further subdivision of the region. If the 512 luma sample area is bisected, for example, into two 16×16 coding units with a luma sample area of ​​256, the worst case memory bandwidth limit is not reached and there is no restriction on the use of MIP mode for the resulting CUs. However, further splitting each of the 256 sample area regions into smaller CUs, i.e., CUs of size 16×8, 8×16, 8×8, 8×4, 4×8, and 4×4, may limit the use of MIP mode for later resulting CUs due to the earlier resulting CUs consuming budget.

[0133] Since 4x4 is the smallest luma coding unit size, and all other coding units are integer multiples of this size, and therefore all regions are multiples of this size, the budget can be expressed as a maximum word read per 4x4 block area. The budget is 40 word reads for a 4x4 block, which corresponds to (512÷(4x4)*40=1280 word reads) as the maximum number of read operations allowed in a 512 luma sample area for a luma sample area of ​​512. Each time MIP mode is used for a CU, the required number of reads is subtracted from the budget, and when the budget is insufficient for further use of MIP mode, no more CUs can use MIP mode, as will be described in connection with Figures 13 and 14. In this way, CUs of larger size can use MIP mode, for example a single CU of area 512 samples or a pair of CUs of area 256 samples can use MIP mode without being restricted by the constraint. If the code tree breaks down a sample of area 512 into many smaller CUs, the budget consumption may prohibit further CUs in the region from using MIP mode. For a 512 sample area with a budget of 40 words per 4×4, the total budget for the region is 512÷(4×4)*40=1280 words. If the 512 sample area is divided into four CUs of size 8×16 or 16×8, using MIP mode on two CUs consumes 576 words per CU, or 1152 words. The remaining budget of 128 word reads is insufficient for the other two CUs in the region to use MIP mode. Control in the processor 205 passes from step 1220 to a partition generation step 1230.

[0134] In a partition generation step 1230, the block partition unit 310, under the execution of the processor 205, generates a set of candidate partitions for the current node in the coding tree. The partitions are as shown in FIG. 5, and the relevant syntax elements are shown in FIG. 6 and illustrated in FIG. 7. In the case of "no partition", i.e. generating 510, a coding unit is generated later. In the case of generating other partition types, i.e. generating 512-520, additional nodes of the coding tree are generated according to each partition. The method 1200 is repeated to generate coding units for the additional nodes in the case of the generation with "no partition" when the method 1200 is performed later. The method 1200 is repeated for each node of the CTU in turn. Thus, all partitions of the CTU are determined. Also, to iterate the generation of partitions within a partition, all possible coding units in a given CTU are searched within the constraints of minimum CU size and depth recursion constraints that may limit the number of recursions using bipartitions, tripartites, and quadtree partitions. Control in the processor 205 passes from step 1230 to a CU generation step 1240.

[0135] In the CU generation step 1240, the block partition unit 310, under the execution of the processor 205, generates a candidate CU for each case in which "no partition" was generated in the partition generation step 1230. In step 1240, the prediction mode of the candidate CU has not yet been determined, and the final mode has not yet been selected, but the prediction mode of the preceding CU in the Z-order scan is known. Thus, adjacent (in terms of position within the CU) reference samples can be used for intra prediction from adjacent CUs from the same partition operation as the partition containing the current CU, or from adjacent regions resulting from different parent regions, or from an entirely different CTU. Step 1240 effectively partitions the region of the coding tree into coding blocks containing prediction blocks as described in connection with FIG. 3. The method 1200 ends in step 1240, and control in the processor 205 returns to the method 1100, where the prediction mode of the generated CU is determined.

[0136] FIG. 13 illustrates a method 1300 for determining the coding unit performed in step 1130. The method 1300 includes determining a prediction mode for the coding unit generated by performing the method 1200. The prediction modes include intra prediction, inter prediction, and MIP mode depending on a budget for using MIP mode in a region including the current coding unit. The budget established in the method 1200 is used to control whether to test MIP mode for the current CU. A portion of the budget is consumed (as described in connection with cases A-C above) each time a CU in the region is the size of a worst-case block depending on the number of codewords in the CU. Once the budget is consumed, no further coding units in the region that could potentially use MIP mode are searched for. That is, the method 1300 performs the test of MIP mode only if a previous performance of the method 1300 has not exhausted the configured memory access bandwidth budget that applies to a common parent node of the coding tree.

[0137] The region does not have to correspond to the direct parent of the current coding unit, i.e., the parent region for which the MIP mode memory access bandwidth budget is set may be multiple nodes above the current node in the coding tree. For example, a region of 512 luma samples may be divided into many smaller coding units of size, e.g., 4x4, 4x8, 8x4, etc. For each CU, the use of MIP mode is limited by the remainder of the budget for the region of 512 luma samples. The method 1300 may be implemented by the mode selector 386 or in part by the module 3960. The method 1300 may be embodied by an apparatus such as a configured FPGA, ASIC, or ASSP. Furthermore, the method 1300 may be executed by the video encoder 114 under execution of the processor 205. Thus, the method 1300 may be stored in a computer-readable storage medium and / or in the memory 206. The method 1300 starts with a step 1310 of testing an intra-prediction mode.

[0138] In a test intra prediction mode step 1310, under execution of the processor 205, the video encoder 114 tests the "normal" intra prediction modes, i.e., DC, planar, and angular intra prediction modes, that may be used in encoding the current coding unit. A Lagrangian optimization is typically performed to select the optimal intra prediction mode among the intra prediction modes available to the CU. The application of secondary transforms, i.e., 330 and 344, is also tested, as are different types of primary transforms (DCT-2, DCT-8, DST-7), including the transform skip case. Control in the processor 205 passes from step 1310 to a test inter prediction mode step 1320.

[0139] In an inter prediction mode test step 1320, under the execution of the processor 205, the video encoder 114 tests various motion vectors for generating an inter predicted PU. When evaluating the use of inter prediction, a motion vector is selected from among a set of candidate motion vectors. The candidate motion vectors are generated according to a search pattern. When testing the distortion of the reference block obtained for the candidate motion vector, the application of chroma splitting prohibited in the code tree is considered. If splitting is prohibited in chroma and allowed in luma, the resulting luma CB may use inter prediction. Since motion compensation is applied only to the luma channel, the distortion calculation considers luma distortion but not chroma distortion. If chroma splitting is prohibited, chroma distortion is not considered since no motion compensation is performed in the chroma channel. For chroma, the distortion due to the considered intra prediction mode and the coded chroma TB (if any) is considered. When considering both luma and chroma, the inter prediction search may first select a motion vector based on luma distortion and then "fine-tune" the motion vector by also considering chroma distortion. The fine-tuning typically considers small variations in the motion vector value, such as sub-pixel displacement. The particular motion vector may be one generated by a "merge mode", whereby the motion vector of the current CU is derived from motion vectors from neighboring CUs. The merge mode is more compactly expressed by the bitstream syntax compared to other motion vectors, which may require signaling of a "motion vector delta" applied in conjunction with a selected "motion vector predictor". The motion vector predictor is typically derived from a spatially or temporally neighboring CU. For an intra-coded slice, e.g., the first frame of a sequence of frames, step 1320 is not performed since inter prediction is not available. Control in the processor 205 passes from step 1320 to an within-budget test step 1330.

[0140] In an in-budget test step 1330, under the execution of the processor 205, the video encoder 114 tests whether the MIP mode memory access bandwidth budget is applicable to the current CU. The test performed in 1330 determines whether the current CU is contained within the 512 luma sample region for which the MIP mode budget was established in step 1220. If the current CU is larger than the 512 luma sample region, there is no budget constraint that applies. Thus, the CU is not subject to any further constraints on the use of MIP mode, and control in the processor 205 passes to a MIP mode test step 1340 (1330 "true"). If the size of the current CU is equal to or smaller than the size of 512 luma samples, the budget required to use MIP mode on the current CU is compared with the remaining budget for that region, as described with reference to FIG. 8. If there is not enough budget to apply MIP mode to the current CU, control in the processor 205 passes from step 1330 to a mode selection step 1350 (1330 "false"). If there is sufficient budget to apply the MIP mode to the current CU (step 1340 : true), control in the processor 205 passes from step 1330 to step 1340 .

[0141] In a MIP mode testing step 1340, the mode selector 386 tests various MIP modes to determine the best MIP mode to use for predicting the current CU from among the MIP modes available for the size of the CU. Similar to the testing of intra-prediction modes in step 1310, a Lagrangian optimization may be performed to counteract the coding costs of the tested MIP modes and their associated distortions to the residual. Control in the processor 205 passes from step 1340 to a mode selection step 1350.

[0142] In a mode selection step 1350, the mode selector 386, under execution of the processor 205, selects a final mode for the CU from the candidates obtained in steps 1310, 1320, and 1350. The method 1300 ends with control in the processor 205 returning to the method 1100.

[0143] 14 shows a method 1400 for encoding the coding units of a code tree of a CTU into a video bitstream 115, as performed in step 1140. The method 1400 may be embodied by an apparatus such as an configured FPGA, an ASIC, an ASSP, etc. Furthermore, the method 1400 may be executed by a video encoder 114, such as the entropy encoder 338, under execution of the processor 205. Thus, the method 1400 may be stored in a computer-readable storage medium and / or in the memory 206. The method 1400 starts with a prediction mode encoding step 1410.

[0144] In a prediction mode encoding step 1410, the entropy encoder 338, under the execution of the processor 205, encodes a flag using context coding bins indicating the use of either intra prediction (including both normal intra prediction mode use or MIP mode use) or inter prediction, as determined in step 1350 and indicated by the prediction mode 387. The encoding of the flag in step 1410 does not distinguish between normal intra prediction and MIP mode. The distinction between normal intra prediction and MIP intra prediction (if applicable) is encoded in a MIP mode flag encoding step 1430. Control in the processor 205 passes from step 1410 to a budget test step 1420.

[0145] In a budget test step 1420, under execution of the processor 205, the video encoder 114 tests whether a memory access bandwidth budget for MIP mode is applicable to the current CU. This test determines whether the current CU is contained within a 512 luma sample region for which a MIP mode budget was established in step 1220. If the current CU is contained within a region larger than 512 luma sample region, then there is no budget constraint to apply and the CU is not further restricted in its use of MIP mode. Control in the processor 205 passes to a MIP mode flag encoding step 1430 (1420 "TRUE").

[0146] If the current CU is in a region of 512 luma samples or less in size, the budget required to use MIP mode for the current CU is compared with the remaining budget for that region, as described with reference to FIG. 8. If there is not enough budget to apply MIP mode to the current CU, control in processor 205 passes from step 1420 to TB encoding step 1440 ("FALS" at step 1420). However, if there is enough budget to apply MIP mode to the current CU ("TRUE" at 1420), control in processor 205 passes from step 1420 to step 1430, even if the current CU is in a region of 512 samples or less. The operations of step 1420 correspond to those of step 1330, and thus only MIP flags are encoded in method 1400 for CUs for which MIP mode was found in method 1300.

[0147] In a MIP mode flag encoding step 1430, the entropy encoder 338, under execution of the processor 205, encodes into the bitstream 115 a context-encoded bin indicating the selection of MIP mode or not, as determined in step 1350. The context used to encode the bin is described with reference to step 1420. If the MIP mode is selected, the entropy encoder 338 also encodes the selection of which particular MIP mode was used for encoding into the bitstream 338. The MIP mode may be encoded using a shortened binary codeword instead of using a selection between the "most likely mode" and the remaining modes. Using a shortened binary codeword avoids the need to derive a list of most likely modes, including a potential table lookup where the most likely mode is derived from adjacent angular intra-predicted CUs. Control in the processor 205 passes from step 1430 to a TB encoding step 1440.

[0148] In a TB encoding step 1440, the entropy encoder 338, under the execution of the processor 205, encodes the residual coefficients of the TB associated with the current CU into a bitstream. In general, the flag of each TB indicates the presence of at least one significant coefficient, and the coefficients are encoded one by one according to a scan pattern that starts from the last significant coefficient position back to the DC (top left) coefficient position. This ends the method 1400, and control in the processor 205 returns to the method 1100.

[0149] 15 shows a method 1500 for decoding coding units and transform blocks of an image frame from a video bitstream 133. The method 1500 may be embodied by an apparatus such as an configured FPGA, ASIC, or ASSP. Furthermore, the method 1500 may be performed by the video decoder 134 under execution of the processor 205. Thus, the method 1500 may be stored in a computer-readable storage medium and / or in the memory 206. The method 1500 begins with step 1510 of dividing a frame into CTUs.

[0150] In a divide frame into CTUs step 1510, the video decoder 134, under the execution of the processor 205, divides the current frame of frame data 133 (to be decoded) into a matrix of CTUs. Decoding of the CTUs resulting from the division begins. Control in the processor 205 passes from step 1510 to a decoding coding unit step 1520.

[0151] In a decoding coding unit step 1520, the entropy decoder 420, under execution of the processor 205, decodes the split flags from the bitstream 133 according to the coding tree, as described with reference to Figures 5 to 7. By decoding the split flags, step 1520 can operate to determine the size and location of the CUs within the CTU, i.e. according to the coding tree of the CTU. Progression of the method 1300 includes repeating to step 1520, resulting in traversing the entire coding tree of the CTU and decoding each CU. Control in the processor 205 passes from step 1520 to a decoding coding unit step 1530.

[0152] In a decoding coding unit step 1530, the entropy decoder 420, under execution of the processor 205, decodes the coding unit from the bitstream 133. Step 1530 invokes a method 1600, described below in connection with Figure 16, to decode the CU. Control in the processor 205 passes from step 1530 to a test last coding unit step 1540.

[0153] In a last coding unit test step 1540, processor 205 tests whether the current coding unit is the last coding unit of the CTU, as determined from decoding the split flag in step 1520. If the current coding unit is the last coding unit of the CTU ("YES" in step 1540), control in the processor passes to a last CTU test step 1550. If the current coding unit is not the last coding unit in the code tree of step 1520 ("NO" in step 1540), the next coding unit in the code tree of step 1520 is selected for decoding, and control in processor 205 passes to step 1520.

[0154] In a last CTU test step 1550, processor 205 tests whether the current CTU is the last CTU in the slice or frame. If the current CTU is not the last CU ("NO" at step 1550), video decoder 134 proceeds to the next CTU in the frame or slice, and control in processor 205 returns from step 1550 to step 1520 to continue processing the remaining CTUs in the frame. If the CTU is the last CTU in the frame or slice, step 1550 returns "YES" and method 1500 ends. Method 1500 results in an entire image frame being decoded from the bitstream as a series of CTUs.

[0155] 16 shows a method 1600 of decoding a coding unit from a video bitstream 133, performed in step 1530. The method 1600 may be embodied by an apparatus such as a configured FPGA, ASIC, or ASSP. Furthermore, the method 1600 may be performed by the video decoder 134 under execution of the processor 205. Thus, the method 1600 may be stored on a computer-readable storage medium and / or in the memory 206. The method 1600 begins with a step 1602 of decoding a pred_mode flag.

[0156] In a step 1602 of decoding a pred_mode flag, the entropy decoder 420 decodes the context-coded bins to determine whether the current coding unit uses inter-prediction or intra-prediction (including MIP mode). Control in the processor 205 continues from step 1602 to an inter-prediction test step 1604. In step 1604, it is determined from the pred_mode flag whether inter-prediction was used. If the current coding unit uses inter-prediction (inter-prediction test step 1604 is "TRUE"), control in the processor 205 continues from step 1602 to an inter-prediction execution step 1606. In step 1606, inter-prediction is performed, and the resulting reference block (434) is fetched and filtered to generate a PU, followed by decoding the TB for each color channel and adding the TB (1690) to the PU to decode the CU. Control in the processor 205 continues from step 1606 to a TB addition step 1690.

[0157] If the current coding unit uses intra prediction or MIP mode (step 1604 "FALSE"), control in the processor 205 continues to a MIP mode budget test step 1610. In the MIP mode budget test step 1610, it is determined whether the MIP mode is available for the current CU depending on the budget. The budget corresponds to the budget described with reference to steps 1330 or 1420. In the described example, if the region containing the current CU is larger than the area threshold (512 luma samples), there is no applicable budget constraint and the MIP mode can be used (step 1610 TRUE). Control in the processor 205 passes from step 1610 to a MIP mode flag decoding step 1620.

[0158] At step 1610, if the area of ​​the region containing the current CU is equal to or less than the threshold (512 luma samples), the budget required to use MIP mode for the current CU is compared to the remaining budget for that region, as described with reference to Figure 8. If there is enough budget to apply MIP mode to the current CU, MIP mode is available ("TRUE" at step 1610) and method 1600 proceeds to step 1620. If there is insufficient budget, MIP mode is not available for the current CU ("FALSE" at MIP mode budget test step 1610). Control in processor 205 then proceeds to intra prediction mode decoding step 1612.

[0159] In step 1612, starting with decoding the intra prediction mode, normal intra prediction (DC, planar or angular) is used. Control in the processor 205 passes from step 1612 to an intra prediction execution step 1614. Then, in step 1614, intra prediction is performed. Control in the processor 205 passes from step 1614 to an add TB step 1690.

[0160] In performing step 1620, the MIP flag is decoded by the entropy decoder 420 using a single context coded bin. Because a bin uses one context, there is no need to select a context depending, for example, on the use of MIP mode of neighboring CUs, block size or other parameters that may be used to select one context from multiple possible contexts for decoding one bin. Control in the processor passes from step 1620 to step 1630 which tests whether MIP mode has been selected.

[0161] In step 1630, it is determined whether MIP mode is selected. If the decoded MIP mode flag indicates that MIP mode is not used ("FALSE" from step 1630 of testing if MIP mode is selected), control proceeds to step 1612 to decode the block using one of the normal intra-prediction modes. If the decoded MIP mode flag indicates that MIP mode is used for the CU ("TRUE" from step 1630), control in the processor 205 proceeds to MIP mode decoding step 1640.

[0162] The MIP mode is decoded from the bitstream 133 in performing step 1640. Control in the processor 205 passes from step 1640 to a read matrix coefficients step 1650. The decoded MIP mode is used in step 1650 to read a set of matrix coefficients 481 from the matrix coefficient memory 486. By limiting the use of the MIP mode in step 1610 according to the budgeted usage, the maximum memory bandwidth consumption required for the matrix coefficient memory 486 to supply the matrix coefficients 481 is reduced.

[0163] 8. From step 1650, the method 1600 transitions to a neighboring sample filtering step 1660. The neighboring reference samples 464 are filtered in step 1660. From step 1660, the method 1600 transitions to a matrix multiplication step 1670. Step 1600 can be performed by module 482, for example, using matrix coefficients 481 and the samples 464. The filtered reference samples and matrix coefficients 481 are multiplied in step 1670, similar to example 828 of FIG.

[0164] The method 1600 transitions from step 1670 to a PB interpolation step 1680. The sparse block (i.e., 830) determined by performing step 1670 is used to fill the PB (i.e., 483) by an interpolation process in step 1680. The method 1600 transitions from the PB interpolation step in step 1680 to a TB addition step 1690. The decoded residual is used to generate a TB, which is added to the PB 483 in step 1690 to decode the CU. The coding unit is decoded using the prediction block generated in step 1690. The method 1600 ends upon completion of step 1690.

[0165] In performing the method 1600, the video decoder 134 achieves support for matrix intra prediction with a constraint on the worst-case memory bandwidth for retrieving the matrix coefficients. The constraint described here does not overly restrict the selection of MIP mode in terms of impacting error compared to when the usage restriction is not enabled. The non-overly restrictive selection behavior is based on a determined statistical likelihood that typically only 20% of CUs coded in MIP mode are adjacent to a left or upper CU that also uses MIP mode. Boundaries between regions larger than the region used as the granularity for establishing memory access budgets, i.e., boundaries between 512 luma samples and CTUs included in the measurement of the 20% found ones.

[0166] In the embodiment of Fig. 11 to Fig. 16, whether MIP mode is used is shown based on a constraint. The constraint is implemented based on the area of ​​the region of the current CU and whether the matrix intra prediction flag is coded or decoded, which depends on whether the area of ​​the region of the current CU meets the threshold area, for example in Fig. 12. In the described implementation, as described in relation to steps 1330, 1420, 1610, whether the matrix intra prediction flag is coded depends on the area of ​​the region if the region meets the threshold area, and (ii) the allocation of the region if the area of ​​the region does not meet the threshold area. Thus, whether the matrix intra prediction flag is coded (or decoded) depends at least on whether the area of ​​the region meets the threshold area.

[0167] In the implementations described in Figures 11 to 14, the MIP mode flag is encoded into the bitstream 115 only if the budget test is satisfied, as described in connection with steps 1420 and 1430. Correspondingly, when decoding the bitstream in Figure 16, the MIP mode flag is decoded only if the MIP mode budget test step returns TRUE. In other words, the matrix intra prediction flag of a CU is decoded only if matrix intra prediction is used.

[0168] As described with reference to Figures 12 and 13, in another configuration of the video encoder 114 and the video decoder 134, the search for the MIP mode is limited. However, in this other configuration, the signaling of the MIP flag is included in the bitstream of each CU, regardless of the consumption state of the quota of each CU. That is, the matrix intra prediction flag of the CU is decoded regardless of whether matrix intra prediction is used. In this other configuration, as indicated by the dashed arrow 1411 in Figure 14, step 1420 is omitted and control proceeds to step 1430 in the video encoder 114 (step 1420 "TRUE"). In step 1430 of this other configuration, the MIP mode flag is included in all cases, but can be set to zero if the MIP mode was not selected in step 1350. Similarly, in the video decoder 134, step 1610 is omitted and control proceeds to step 1620 (effectively, step 1610 always returns "TRUE"). The MIP mode flag is decoded in step 1620 and is zero if MIP mode was not selected in step 1350 of encoding the bitstream. This avoids the additional burden on the entropy encoder 338 and the entropy decoder 420 of decoding whether to encode the MIP flag for each CU. Even though the bitstream includes a MIP mode flag for each CU, the memory bandwidth for reading the matrix coefficients 481 from the coefficient memory 486 of the video decoder 134 is still limited due to lookups as performed in methods 1200 and 1300, so the provision of resources required to handle this (limited) worst-case use of the MIP mode can be reduced.

[0169] The implementations described in connection with Figures 12 to 17 above apply restrictions on the use of MIP mode based on the area of ​​the region containing the CU. In yet another configuration of the video encoder 114 and video decoder 134, the constraint implemented in Figure 12 prohibits the use of matrix intra prediction for block sizes that correspond to worst-case memory bandwidth, rather than based on the area of ​​the region. Effectively, matrix intra prediction is used (and matrix intra prediction flags are coded or decoded) based on the size of each coding block. For block sizes 4x4, 4x8, 8x4, 8x8, 8x16, and 16x8, there is a maximum access density (e.g., measured for 4x4 luma sample blocks) of 80 words per 4x4 block. Of the previous set of block sizes, certain block sizes result in a worst-case of 72 words per 4x4 block, but for resource provisioning purposes, this can be treated in the same category as the 80 words per 4x4 block case. Forbidding the worst case block size results in a worst case of 40 words per 4x4 block (for a given block size, the worst case is 36 words per 4x4 block).

[0170] Fig. 17 shows a method 1700 for determining the coding unit implemented in step 1130. Method 1300 provides an alternative to method 1300, where the restriction is implemented with respect to prohibiting the use of matrix intra prediction for block sizes corresponding to the worst case memory bandwidth. Method 1700 includes determining a prediction mode for the coding unit generated by executing method 1200. The prediction modes include intra prediction, inter prediction, and MIP mode if MIP mode is available in the region including the current coding unit. Method 1700 starts with a step 1710 of testing the intra prediction mode.

[0171] In a test intra-prediction mode step 1310, the video encoder 114, under execution of the processor 205, tests the "normal" intra-prediction modes, i.e., DC, planar, and angular intra-prediction modes, for potential use in encoding the current coding unit, operating in the manner described for step 1310. Control in the processor 205 passes from step 1710 to a test inter-prediction mode step 1720.

[0172] In an inter prediction mode test step 1720, the video encoder 114, under the execution of the processor 205, tests various motion vectors for generating an inter predicted PU. When evaluating the use of inter prediction, a motion vector is selected from a set of motion vector candidates. Step 1720 operates in the same manner as step 1320. Control in the processor 205 passes from step 1720 to a MIP tolerance size test step 1730.

[0173] In a MIP allowed size test step 1730, the video encoder 114, under execution of the processor 205, tests whether the CU is of a size for which MIP mode is allowed. MIP mode is allowed for all sizes except for the worst case sizes 4×4, 4×8, 8×4, 8×8, 8×16, and 16×8 in some implementations. If the current CU is of an allowed size ("TRUE" at 1730), control in the processor 205 passes from step 1730 to a MIP mode test step 1740. If the current CU is of a prohibited size ("FALSE" at step 1730), control in the processor 205 passes from step 1730 to a mode selection step 1750.

[0174] In a MIP mode testing step 1740, the mode selector 386 tests various MIP modes to determine the best MIP mode available for the size of the CU to use to predict the current CU. Step 1740 operates similarly to step 1340. Processing in the processor 205 proceeds from step 1740 to a mode selection step 1750.

[0175] In a mode selection step 1750, the mode selector 386, under execution of the processor 205, selects a final mode for the CU from the candidates obtained in steps 1710, 1720, and 1750. The method 1700 ends when control in the processor 205 is returned to the method 1100.

[0176] Fig. 18 shows a method 1800 for encoding coding units of a code tree of a CTU into a video bitstream 115 in an implementation as performed in step 1140, in which the constraint relates to prohibiting the use of matrix intra prediction for block sizes corresponding to the worst case memory bandwidth. The method 1800 may be embodied by an apparatus such as a configured FPGA, ASIC, or ASSP. Furthermore, the method 1800 may be executed by the video encoder 114 under the execution of the processor 205. Thus, the method 1800 may be stored in a computer-readable storage medium and / or in the memory 206. The method 1800 starts with a prediction mode encoding step 1810.

[0177] In a prediction mode encoding step 1810, the entropy encoder 338, under execution of the processor 205, encodes a flag using a context coding bin indicating the use of either intra-prediction (including both the use of normal intra-prediction mode or the use of MIP mode), or intra-prediction, as determined in step 1750. Control in the processor 205 passes from step 1810 to a MIP tolerance size test step 1820.

[0178] In a MIP allowed size test step 1820, the video encoder 114, under execution of the processor 205, tests whether the current CU is of an allowed size. The allowed and prohibited sizes are the same as in step 1730 of the method 1700. If the current CU is of an allowed size, control in the processor 205 passes to a MIP mode flag encoding step 1830 (1820 "TRUE").

[0179] If the current CU is not of an acceptable size (step 1820: FALSE), control in the processor 205 passes from step 1820 to a step 1840 for encoding the TB.

[0180] In a MIP mode flag encoding step 1830, the entropy encoder 338, under execution of the processor 205, encodes into the bitstream 115 a context coding bin that indicates whether or not MIP mode was selected, as determined in step 1350. Step 1830 operates in the same manner as step 1430. Control in the processor 205 passes from step 1830 to a TB encoding step 1840.

[0181] In a TB encoding step 1840, the entropy encoder 338, under execution of the processor 205, encodes the residual coefficients of the TB associated with the current CU into a bitstream. Step 1840 operates in the same manner as step 1440. Then, the method 1800 ends, and control in the processor 205 returns to the method 1100.

[0182] 19 shows a method 1900 for decoding a coding unit from a video bitstream 133 in an implementation where a constraint is performed in step 1530 relating to prohibiting MIP mode for worst case blocks. The method 1900 may be embodied by an apparatus such as a configured FPGA, ASIC, or ASSP. Furthermore, the method 1900 may be performed by the video decoder 134 under execution of the processor 205. Thus, the method 1900 may be stored in a computer readable storage medium and / or in the memory 206. The method 1900 begins with a step 1902 of decoding a pred_mode flag.

[0183] In a pred_mode flag decoding step 1902, the entropy decoder 420 decodes the context coded bins to determine whether the current coding unit uses inter prediction or intra prediction (including MIP mode). If the current coding unit uses inter prediction (inter prediction test step 1904 "TRUE"), control in the processor passes to an inter prediction execution step 1906. In step 1606, inter prediction is performed, and the result is a reference block (434) that is fetched and filtered to generate a PU, followed by decoding the TB of each color channel and adding the TB (1690) to the PU to decode the CU. Similarly, control in the processor 205 passes from step 1906 to a TB addition step 1990.

[0184] If the current coding unit uses intra prediction or MIP mode (step 1904 "FALSE"), control in the processor 205 passes to a MIP allowed size test step 1910. In the MIP allowed size test step 1910, it is determined whether the MIP mode is allowed for the current CU size. The MIP allowed size is as described with reference to steps 1730 or 1820. If the current CU is of an allowed size (step 1910 "TRUE"), control in the processor 205 passes from step 1910 to a MIP mode flag decoding step 1920. If in step 1910 the current CU is not of an allowed size (step 1910 "FALSE"), control in the processor 205 passes to an intra prediction mode decoding step 1912.

[0185] In step 1912, a regular intra prediction (DC, planar or angular) is decoded and the method 1900 proceeds to a decoding intra prediction step 1914. Steps 1912 and 1914 operate as described in relation to steps 1612 and 1614. Control in the processor 205 continues from step 1914 to an add TB step 1990.

[0186] The MIP flag is decoded during step 1920. Control in the processor 205 transitions from step 1920 to step 1930 which tests whether MIP mode has been selected.

[0187] Step 1930 operates to determine whether MIP mode is selected. If the decoded MIP mode flag indicates that MIP mode is not used ("FALSE" from test MIP mode selected step 1930), control proceeds to step 1912 to decode the block using one of the normal intra-prediction modes. If the decoded MIP mode flag indicates that MIP mode is used for the CU ("TRUE" from step 1930), control proceeds in the processor to MIP mode decoding step 1940.

[0188] The MIP mode is decoded from the bitstream 133 in performing step 1940. Control in the processor 205 transitions from step 1940 to a read matrix coefficients step 1950. The decoded MIP mode is used in step 1950 to read a set of matrix coefficients 481 from the matrix coefficient memory 486. Matrix coefficients are read for each prediction block for which it was determined to use the MIP mode, as in step 1650.

[0189] 8. From step 1950, the method 1900 transitions to a neighboring sample filtering step 1660. The neighboring reference samples 464 are filtered in step 1960. From step 1960, the method 1900 proceeds to a matrix multiplication step 1970. The filtered reference samples and the matrix coefficients 481 are multiplied in step 1970, similar to example 828 of FIG.

[0190] From step 1970, the method 1900 transitions to a PB interpolation step 1980. The sparse block (i.e., 830) determined by performing step 1970 is used to interpolate PB (i.e., 483) by an interpolation process in step 1980. From the PB interpolation step 1980, the method 1900 transitions to a TB addition step 1990. The decoded residual is used to generate a TB, which is added to the PB 483 in step 1990, similar to step 1690, to decode the CU. The method 1900 ends upon completion of step 1990.

[0191] The variations described for methods 1300, 1400, and 1600 may also be applied to methods 1700, 1800, and 1900, respectively.

[0192] When block size is used as a criterion for the availability of MIP modes, there is no need to establish and update memory budgets as in method 1700, i.e., steps 1210 and 1220 are omitted. Removing MIP modes for certain block sizes is easier to implement than budgeting memory accesses, but the absence of MIP modes from many common block sizes reduces compression performance. The size of the coefficient memory (i.e., 392, 486) is reduced because there is no need to store the matrix coefficients of "set A" (18 sets of matrix coefficients and bias values) due to the absence of 4x4 blocks. Removing MIP modes from 4x4, 4x8, 8x4, 8x8, 8x16, and 16x8 ("small blocks") has the advantage that the feedback loop of the MIP modes, which is a relatively complex operation, does not need to support these small blocks.

[0193] In yet another configuration, the use of MIP mode is prohibited for a subset of small blocks (as listed above). For example, MIP mode is prohibited only for 4×4 blocks in steps 1730, 1820, and 1910, and allowed for all other block sizes, or MIP mode is prohibited for 4×4, 4×8, and 8×4 blocks, and allowed for all other block sizes. That is, matrix intra prediction is not used, and in some implementations, the matrix intra prediction flag is decoded if the size of the coding unit is one of the prohibited sizes. Although the worst-case memory bandwidth is not reduced compared to the complete absence of restrictions on the use of MIP mode, the exclusion of these very small block sizes reduces the stringency of the intra reconstruction feedback loop. The removal of 4×4 also eliminates the need to store the matrix coefficients associated with "set A" in the coefficient memory (i.e., 392, 486). Alternatively, the set of prohibited block sizes may be 4×4, 4×8, 8×4, and 8×8, in which case Set A and Set B do not exist in video encoder 114 or video decoder 134. Removing Set A and Set B reduces compression performance but memory consumption because matrix coefficients associated with Set A and Set B are unnecessary.

[0194] In yet another configuration, the memory budget may be even lower than 40 words per 4x4 luma sample area, for example 20 or 10 words per 4x4 luma sample area. The budget is set at larger region sizes, such as at nodes corresponding to region sizes of 1024 or 2048 luma samples. As with the above configuration, subsequent CUs within the limits are limited in the amount of memory budget available depending on the use of MIP modes by earlier CUs within those regions. Further reductions in memory bandwidth can be achieved at the expense of reduced compression efficiency.

[0195] In yet another configuration, memory budgets are established at steps 1220, 1330, 1420, and 1610 with a region size of 64 luma samples to apply to CUs of sizes 4×4, 4×8, and 8×4 ("small CU memory budget"). A separate memory budget is established with a region size of 512, but applies only to CUs with sizes greater than 8×8, particularly 8×16 and 16×8 ("larger CU memory budget"). Both budgets are set to 40 words per 4×4 luma sample area. Because the small CU memory budget applies only to CUs contained within a 64 luma sample area, whereas the larger CU memory budget applies only to CUs larger than 64 luma samples, the budgets form additional budgets to the total matrix memory bandwidth.

[0196] While the configurations disclosed herein describe memory bandwidth in terms of words per 4×4 memory region, it is understood that accesses to the memory will likely group words in some SIMD format to allow matrix coefficients to be read without requiring an excessively high clock frequency for the associated memory, however such wider memories are themselves costly and the matrix coefficients may be shared with other data in the same memory, resulting in contention for the reduced accesses due to the limitations of using MIP mode.

[0197] Restricting the use of MIP mode to limit the worst-case memory bandwidth per region can result in bias, where earlier CUs in each region use MIP mode and later CUs in the region cannot use MIP mode because the available allocation in that region has been consumed by earlier CUs. This bias does not usually occur because the distribution of CUs for which MIP mode is selected is generally somewhat sparse.

[0198] The described configuration has restrictions on when MIP mode may be used, thereby reducing computational complexity compared to allowing MIP mode to run without restrictions. The reduced complexity occurs without a proportional loss in coding efficiency, because the MIP mode selection statistics for an unrestricted search would not normally trigger the restrictions imposed on MIP mode selection in the mode selector 386. Thus, the worst-case memory bandwidth is reduced without a commensurate loss in coding efficiency. Industrial Applicability

[0199] The described arrangements are applicable in the computer and data processing industry, in particular in digital signal processing for encoding and decoding of signals such as video and image signals, to achieve high compression efficiency.

[0200] The arrangement described herein allows the use of a trellis-based state machine for residual encoding and decoding, which updates according to coefficient parity and selects the context and coefficient quantizers. The described arrangement allows the implementation of a trellis-based state machine without imposing excessive delays due to the sequential nature of the state updates.

[0201] The above describes only some embodiments of the present invention, which may be modified and / or altered without departing from the scope and spirit of the present invention, and the embodiments are illustrative and not limiting.

Claims

1. 1. A method for generating prediction samples for a current block in a coding tree unit for an image frame, comprising: decoding a matrix intra prediction flag for the current block, the matrix intra prediction flag indicating whether matrix intra prediction is used for the current block; if the matrix intra prediction flag indicates that matrix intra prediction is used for the current block, decoding a matrix intra prediction mode for the current block; selecting a matrix for matrix intra prediction for the current block according to the matrix intra prediction mode; generating the prediction samples by applying a multiplication of input samples based on samples adjacent to the current block with the matrix selected according to the matrix intra prediction mode; A shortened binary code may be used for the matrix intra prediction mode; The image frames may have a 4:2:0 chroma format; A method characterized in that, when an area in the coding tree unit is divided into four blocks, each having a size of 16x8, and the current block is one of the four blocks, the decoding of the matrix intra prediction flag for the current block depends on the status of use of matrix intra prediction for two blocks adjacent to the current block.

2. 2. The method of claim 1, wherein the decoding of the matrix intra prediction mode for the current block depends on a width of the current block and a height of the current block.

3. A plurality of block sizes are available, the plurality of block sizes being classified as first level, second level or third level; the first level includes at least a 4×4 size; 2. The method of claim 1, wherein a number of matrices available for matrix intra prediction for the first level is greater than a number of matrices available for matrix intra prediction for the second level.

4. 4. The method of claim 3, wherein the number of matrices available for matrix intra prediction for the first level is greater than the number of matrices available for matrix intra prediction for the third level.

5. 4. The method of claim 3, wherein the block sizes included in the second level are greater than 4x4 size.

6. 5. The method of claim 4, wherein the block sizes included in the third level are greater than 4x4 size.

7. 2. The method of claim 1, further comprising: determining a plurality of blocks including the current block in the coding tree unit by decoding a split flag for a block split in the coding tree unit, wherein a horizontal three-way split can be used as the block split.

8. 2. The method of claim 1, wherein the decoding of the matrix intra prediction flags depends on the width of the current block and the height of the current block.

9. 2. The method of claim 1 , wherein, when the size of the current block is one of an 8×16 size and a 16×8 size, the matrix intra prediction mode is one of all matrix intra prediction modes available for the 8×16 size and the 16×8 size, and a shortened binary code is used for the matrix intra prediction mode regardless of which matrix intra prediction mode is used.

10. 2. The method of claim 1, wherein, when the size of the current block is one of 8x16 and 16x8, (a) the matrix intra prediction mode is one of a plurality of matrix intra prediction modes available for the 8x16 and 16x8 sizes, (b) the number of a plurality of matrix intra prediction modes available for the 8x16 and 16x8 sizes is n, where n is an integer value greater than 0, (c) the matrix intra prediction mode is represented by an integer value greater than or equal to 0 and less than or equal to n-1, (d) an m-bit code (m is an integer value greater than 0) is used for the matrix intra prediction mode represented by an integer value greater than or equal to 0 and less than or equal to a predetermined integer value, and (e) an m+1-bit code is used for the matrix intra prediction mode represented by an integer value greater than the predetermined integer value and less than or equal to n-1.

11. 11. The method of claim 10, wherein n is an integer value greater than 2 to the mth power and less than 2 to the m+1th power.

12. 8. The method according to claim 7, wherein the block division can be a vertical division into thirds.

13. 1. A method for generating prediction samples for a current block in a coding tree unit for an image frame, comprising: selecting a matrix for matrix intra prediction for the current block; generating said predicted samples by applying a multiplication of said selected matrix with input samples based on neighboring samples of said current block; encoding a matrix intra prediction flag indicating whether matrix intra prediction is used for the current block; and if matrix intra prediction is used for the current block, encoding a matrix intra prediction mode used to identify the selected matrix; A shortened binary code may be used for the matrix intra prediction mode; The image frames may have a 4:2:0 chroma format; A method characterized in that, when an area in the coding tree unit is divided into four blocks each having a size of 16x8, and the current block is one of the four blocks, encoding of the matrix intra prediction flag for the current block depends on the status of use of matrix intra prediction for two blocks adjacent to the current block.

14. The method of claim 13 , wherein the encoding of the matrix intra prediction mode for the current block depends on a width of the current block and a height of the current block.

15. A plurality of block sizes are available, the plurality of block sizes being classified as first level, second level or third level; the first level includes at least a 4×4 size; 14. The method of claim 13, wherein a number of matrices available for matrix intra prediction for the first level is greater than a number of matrices available for matrix intra prediction for the second level.

16. 16. The method of claim 15, wherein a number of matrices available for matrix intra prediction for the first level is greater than a number of matrices available for matrix intra prediction for the third level.

17. 16. The method of claim 15, wherein the block sizes included in the second level are greater than 4x4 size.

18. 16. The method of claim 15, wherein the block sizes included in the third level are greater than 4x4 size.

19. 14. The method of claim 13, further comprising: determining a plurality of blocks including the current block in the coding tree unit by block partitioning, wherein a horizontal third partition can be used as the block partitioning.

20. 14. The method of claim 13, wherein the encoding of the matrix intra prediction flag depends on the width of the current block and the height of the block.

21. 14. The method of claim 13, wherein when the size of the block is one of an 8x16 size and a 16x8 size, the matrix intra prediction mode is one of all matrix intra prediction modes available for the 8x16 size and the 16x8 size, and a shortened binary code is used for the matrix intra prediction mode regardless of which matrix intra prediction mode is used.

22. 2. The method of claim 1, wherein, when the size of the current block is one of 8x16 and 16x8, (a) the matrix intra prediction mode is one of a plurality of matrix intra prediction modes available for the 8x16 and 16x8 sizes, (b) the number of a plurality of matrix intra prediction modes available for the 8x16 and 16x8 sizes is n, where n is an integer value greater than 0, (c) the matrix intra prediction mode is represented by an integer value greater than or equal to 0 and less than or equal to n-1, (d) an m-bit code (m is an integer value greater than 0) is used for the matrix intra prediction mode represented by an integer value greater than or equal to 0 and less than or equal to a predetermined integer value, and (e) an m+1-bit code is used for the matrix intra prediction mode represented by an integer value greater than the predetermined integer value and less than or equal to n-1.

23. 23. The method of claim 22, wherein n is an integer value greater than 2 to the mth power and less than 2 to the m+1th power.

24. 20. The method of claim 19, wherein the block division can be a vertical division into thirds.

25. 1. An apparatus for generating a prediction sample for a current block in a coding tree unit for an image frame, comprising: means for decoding a matrix intra prediction flag for the current block, the matrix intra prediction flag indicating whether matrix intra prediction is used for the current block; means for decoding a matrix intra-prediction mode for the current block if the matrix intra-prediction flag indicates that matrix intra-prediction is used for the current block; means for selecting a matrix for matrix intra prediction for the current block according to the matrix intra prediction mode; means for generating the prediction samples by applying a multiplication of input samples based on samples adjacent to the current block and the matrix selected according to the matrix intra prediction mode; A shortened binary code may be used for the matrix intra prediction mode; The image frames may have a 4:2:0 chroma format; An apparatus characterized in that, when an area in the coding tree unit is divided into four blocks each having a size of 16x8 and the current block is one of the four blocks, decoding of the matrix intra prediction flag for the current block depends on the status of use of matrix intra prediction for two blocks adjacent to the block.

26. 1. An apparatus for generating a prediction sample for a current block in a coding tree unit for an image frame, comprising: means for selecting a matrix for matrix intra prediction for the current block; means for generating the prediction samples by applying a multiplication of the selected matrix with input samples based on neighboring samples of the current block; means for encoding a matrix intra prediction flag indicating whether matrix intra prediction is used for the current block; and means for encoding a matrix intra-prediction mode used to identify the selected matrix if matrix intra-prediction is used for the current block; A shortened binary code may be used for the matrix intra prediction mode; The image frames may have a 4:2:0 chroma format; An apparatus characterized in that, when an area in the coding tree unit is divided into four blocks each having a size of 16x8 and the current block is one of the four blocks, encoding of the matrix intra prediction flag for the current block depends on the status of use of matrix intra prediction for two blocks adjacent to the current block.

27. A computer program product for causing a computer to carry out the method according to claim 1.

28. A computer program product for causing a computer to carry out the method according to claim 13.

Images