Method, apparatus and program for encoding and decoding blocks of video samples - Patents.com

Abstract

To provide a method and computer program for decoding a block of a video sample.SOLUTION: A method comprises the steps of: decoding a matrix intra-prediction flag for an object block in a coding tree unit; decoding a matrix prediction mode for the object block in the case where the matrix intra-prediction flag indicates that the matrix intra-prediction is used for the object block; selecting a matrix for the matrix intra-prediction for the object block in accordance with the matrix intra-prediction mode; and generating a prediction sample of the object block on the basis of the selected matrix. Therein, an area existing in the coding tree unit is divided into four blocks each having sizes of 16×8, and in the case where the object block is one of the four blocks, the decoding of the matrix intra-prediction flag for the object block depends on a situation of use of the matrix intra-prediction of two blocks that are adjacent to the object block.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] REFERENCE TO RELATED APPLICATIONS This invention claims the benefit under 35 U.S.C. §119 of Australian Patent Application No. 2019204437, filed June 24, 2019, as if fully described herein. Technical Field

[0002] The present invention relates generally to digital video signal processing, and more particularly to a method, apparatus, and computer program for encoding and decoding blocks of video samples. [Background technology]

[0003] Currently, there are many applications for video coding, including applications for the transmission and storage of video data. Many video coding standards have been developed and are currently under development. Recent progress in video coding standardization has led to the formation of a group called the "Joint Video Experts Team" (JVET). The Joint Video Experts Team (JVET) includes members of the International Telecommunication Union (ITU) Telecommunication Standardization Sector (ITU-T) Study Committee 16, Consultative Group 6 (SG16 / Q6), known as the "Video Coding Experts Group" (VCEG), and members of the International Organization for Standardization / International Electrotechnical Commission Joint Technical Committee 1 / Subcommittee 29 / Working Group 11 (ISO / IEC JTC1 / SC29 / WG11), known as the "Moving Picture Experts Group" (MPEG).

[0004] The Joint Video Experts Team (JVET) issued a call for proposals (CfP) and analyzed the proposals received at its 10th conference in San Diego, USA. The proposals demonstrated video compression capabilities that significantly exceeded the current state-of-the-art video compression standard, "High Efficiency Video Coding" (HEVC). Based on this analysis, it was decided to launch a project to develop a new video compression standard named "Versatile Video Coding" (VVC). VVC is expected to address the continuing demand for ever-higher compression performance, especially as video formats improve in capabilities (e.g., higher resolutions and higher frame rates), as well as the growing market demand for service delivery over wide area networks (WANs), where bandwidth costs are relatively high. At the same time, VVC must be implementable on conventional silicon processes and must offer an acceptable tradeoff between the achieved performance and implementation costs (e.g., silicon area, CPU processor load, memory utilization, bandwidth, etc.).

[0005] Video data includes image data for successive frames, each of which contains one or more color channels. Generally, one primary color channel and two secondary color channels are required. The primary color channel is commonly referred to as the "luma" channel, and the secondary color channels are commonly referred to as the "chroma" channels. Video data is typically displayed in the RGB (red-green-blue) color space, which has a high degree of correlation between each of the three components. Encoders or decoders often represent video data using a color space such as YCbCr. YCbCr concentrates luminance, mapped to "luma" according to a transfer function, in the Y (primary) channel and chroma in the Cb and Cr (secondary) channels. Furthermore, the Cb and Cr channels may be spatially subsampled at a lower rate than the luma channel—for example, half horizontally and half vertically, known as a "4:2:0 chroma format." The 4:2:0 chroma format is commonly used in consumer applications such as Internet video streaming, television broadcasting, and Blu-ray disc storage. Formats in which the Cb and Cr channels are subsampled horizontally at half the rate but not vertically are known as "4:2:2 chroma formats." 4:2:2 chroma formats are typically used in professional applications, including filmmaking. The higher sampling rate of 4:2:2 chroma formats allows greater flexibility for editing, such as color grading, of the resulting footage. Before being distributed to consumers, 4:2:2 chroma format material is often converted to 4:2:0 chroma format before being encoded for distribution to consumers. In addition to chroma format, video is also characterized by resolution and frame rate. Examples of resolutions include ultra-high definition (UHD) with a resolution of 3840 x 2160 or "8K" with a resolution of 7680 x 4320, and examples of frame rates include 60 Hz or 120 Hz. Luma sample rates range from approximately 500 megasamples per second to several gigasamples per second.For a 4:2:0 chroma format, the sample rate of each chroma channel is one-quarter of the luma sample rate, and for a 4:2:2 chroma format, the sample rate of each chroma channel is half the luma sample rate.

[0006] The VVC standard uses "block-based" encoding and decoding, where a frame is first divided into square regions known as "coding tree units" (CTUs). CTUs typically occupy a relatively large area, such as 128x128 luma samples. However, the CTUs at the bottom right of each frame may be smaller in area. Each CTU is associated with a "coding tree" for the luma channel and additional coding trees for the chroma channels. The coding trees define a decomposition of the CTU's region into a set of blocks, also known as "coding blocks" (CBs). A single coding tree may identify blocks for both the luma and chroma channels; in this case, the colocated coding blocks are collectively referred to as a "coding unit" (CU). That is, each CU contains coding blocks for each color channel. The CBs are encoded or decoded in a specific order. When using the 4:2:0 chroma format, a CTU with a luma code tree for a 128x128 luma sample region will have a corresponding chroma code tree for a 64x64 chroma sample region co-located with the 128x128 luma sample region. When a single code tree is used for the luma and chroma channels, the set of blocks co-located with a region is commonly referred to as a "unit," such as the CU, "Prediction Unit" (PU), and "Transform Unit" (TU). When separate code trees are used for a region, the CB, "Prediction Block" (PB), and "Transform Block" (TB) are used.

[0007] It should be noted that despite the above distinction between "unit" and "block," the term "block" is also used as a general term to describe an area of ​​a frame where processing is applied to all color channels.

[0008] For each CU, a prediction unit (PU) of the content (sample values) of the corresponding region of the frame data is generated ("prediction unit"). Furthermore, a representation of the difference between the prediction and the content of the region as seen at the input to the encoder (or "residual" in the spatial domain) is formed. The differences for each color channel are transformed and coded as a set of residual coefficients, forming one or more TUs for a given CU. The applied transform, a discrete cosine transform (DCT) or other transform, is applied to the residual values ​​of each block. This transform is applied independently; that is, the two-dimensional transform is performed in two passes. The block is first transformed by applying a one-dimensional transform to the samples in each row of the block. Then, the partial results are transformed by applying a one-dimensional transform to each column of the partial results, producing a final block of transform coefficients that substantially decorrelates the residual samples. Transforms of various sizes are supported in the VVC standard, including transforms of rectangular blocks whose side dimensions are powers of two. The transform coefficients are quantized for entropy coding into the bitstream.

[0009] VVC features intra-frame prediction and inter-frame prediction. Intra-frame prediction involves the use of previously processed samples within a frame to generate predicted values ​​for samples of a current block within the frame. Inter-frame prediction involves using samples of a block obtained from a previously decoded frame to generate predicted values ​​for samples of a current block within the frame. The previously decoded frame is offset from the spatial location of the current block according to a motion vector and is often filtered. An intra-frame predicted block can be a uniform sample value ("DC intra-prediction"), a plane with an offset and horizontal and vertical gradients ("planar intra-prediction"), a population of blocks applied with neighboring samples in a specific direction ("angular intra-prediction"), or the result of matrix multiplication using neighboring samples and selected matrix coefficients. Because neighboring samples include samples from previously processed blocks, the feedback loop for intra-frame prediction is highly limited, requiring computational complexity to be kept lower than that required to meet the highest supported resolution and frame rate. Summary of the Invention

[0010] It is an object of the present invention to substantially overcome, or at least ameliorate, one or more disadvantages of existing arrangements.

[0011] According to one aspect of the present disclosure, there is provided a method for decoding coding units of a code tree for an image frame from a video bitstream, the method comprising: dividing a region of the code tree into a plurality of coding blocks, each coding block including a predictive block; determining a matrix intra prediction flag for the predictive blocks of each of the coding blocks, each matrix intra prediction flag indicating whether matrix intra prediction was used for the predictive block of one of the coding blocks, the determination being based on (i) an area of ​​the region if the region meets a threshold, or (ii) an allocation of the region if the area of ​​the region does not meet the threshold; for each predictive block determined to use matrix intra prediction according to the determined flag, reading matrix coefficients from a memory; and decoding the coding unit using a reference sample of each predictive block and a predictive block generated for each coding unit in the region using the matrix coefficients.

[0012] According to another aspect, the threshold is a size greater than 512 luma samples.

[0013] According to another aspect, the threshold is a size greater than 64 luma samples.

[0014] According to another aspect, the budget allows for reading 40 words of a 4x4 block for the region.

[0015] According to another aspect, the matrix intra prediction flag of the CU is decoded only if matrix intra prediction is used.

[0016] According to another aspect, the matrix intra prediction flag of the CU is decoded regardless of whether matrix intra prediction was used.

[0017] According to another aspect of the present disclosure, there is provided a method for decoding coding units of a code tree for an image frame from a video bitstream, the method including: dividing a region of the code tree into a plurality of coding blocks, each including a predictive block; determining a matrix intra prediction flag for predictive blocks of the coding blocks based on a size of each coding block, wherein each matrix intra prediction flag indicates whether matrix intra prediction has been used for the predictive block of the corresponding coding block; for each predictive block determined to use matrix intra prediction according to the determined flag, reading matrix coefficients from a memory; and decoding the coding units from the predictive blocks of each coding unit in the region generated using reference samples of each predictive block and the matrix coefficients.

[0018] According to another aspect, a matrix intra prediction flag is decoded if the size of the coding unit is not 4x4.

[0019] According to another aspect, a matrix intra prediction flag is decoded if the size of the coding unit is not one of 4x4, 8x4, or 4x8.

[0020] According to another aspect, a matrix intra prediction flag is decoded if the size of the coding unit is not one of 4x4, 8x4, 4x8, or 8x8.

[0021] According to another aspect, the matrix intra prediction flag is decoded if the size of the coding unit is not one of 4x4, 8x4, 4x8, 8x8, or 8x16.

[0022] According to another aspect of the present disclosure, there is provided a method for generating a predictive block of a coding tree for an image frame from a video bitstream, the method comprising: determining a prediction mode of the coding unit by decoding a matrix intra-prediction mode flag from the bitstream, the prediction mode including a matrix intra-prediction mode; decoding a shortened binary codeword to determine the matrix intra-prediction mode; and generating the predictive block by performing matrix multiplication on reference samples adjacent to the predictive block and a matrix selected according to the decoded matrix intra-prediction mode.

[0023] According to another aspect of the present disclosure, there is provided a non-transitory computer-readable storage medium having stored thereon a computer program for implementing a method for decoding coding units of a code tree for an image frame from a video bitstream, the method including: dividing a region of the coding tree into a plurality of coding blocks, each coding block including a predictive block; determining a matrix intra-prediction flag for the predictive blocks of each of the coding blocks, each matrix intra-prediction flag indicating whether matrix intra-prediction was used for the predictive block of one of the coding blocks, the determination being based on (i) an area of ​​the region if the region meets a threshold, or (ii) an allocation of the region if the area of ​​the region does not meet the threshold; for each predictive block determined to use matrix intra-prediction according to the determined flag, reading matrix coefficients from a memory; and decoding the coding unit using a reference sample of each predictive block and a predictive block generated for each coding unit in the region using the matrix coefficients.

[0024] According to another aspect of the present disclosure, a video decoder is provided, comprising: receiving coding units of a code tree for an image frame from a video bitstream; dividing a region of the code tree into a plurality of coding blocks, each coding block including a predictive block; determining a matrix intra prediction flag for the predictive blocks of each coding block, each matrix intra prediction flag indicating whether matrix intra prediction was used for the predictive block of one of the coding blocks; determining the matrix intra prediction flag based on (i) an area of ​​the region if the region meets a threshold; or (ii) an allocation of the region if the area of ​​the region does not meet the threshold; for each predictive block determined to use matrix intra prediction according to the determined flag, reading matrix coefficients from a memory; and decoding the coding unit using a reference sample of each predictive block and a predictive block generated for each coding unit in the region using the matrix coefficients.

[0025] According to another aspect of the present disclosure, a system is provided, the system including a memory and a processor, the processor executing code stored in the memory for implementing a method for decoding coding units of a code tree for an image frame from a video bitstream, the method including: dividing a region of the code tree into a plurality of coding blocks, each coding block including a predictive block; determining a matrix intra prediction flag for the predictive blocks of each coding block, each matrix intra prediction flag indicating whether matrix intra prediction was used for the predictive block of one of the coding blocks, the determination being based on (i) an area of ​​the region if the region meets a threshold, or (ii) an allocation of the region if the area of ​​the region does not meet the threshold; for each predictive block determined to use matrix intra prediction according to the determined flag, reading matrix coefficients from a memory; and decoding the coding unit using a reference sample of each predictive block and a predictive block generated for each coding unit in the region using the matrix coefficients.

[0026] Other aspects are also disclosed.

[0027] At least one embodiment of the present invention will now be described with reference to the following drawings and appendices. [Brief explanation of the drawings]

[0028] [Figure 1] FIG. 1 is a schematic block diagram illustrating a video encoding and decoding system. [Figure 2A] , [Figure 2B] 2A and 2B show a schematic block diagram of a general-purpose computer system on which one or both of the video encoding and decoding systems of FIG. 1 may be implemented. [Figure 3] FIG. 3 is a schematic block diagram illustrating the functional modules of a video encoder. [Figure 4] FIG. 4 is a schematic block diagram showing the functional modules of a video decoder. [Figure 5] FIG. 5 is a schematic block diagram illustrating a possible division of a block into one or more blocks in a tree structure of Versatile Video Coding. [Figure 6] FIG. 6 is a schematic diagram of the data flow for achieving a possible division of one block into one or more blocks in a tree structure of Versatile Video Coding. [Figure 7A] , [Figure 7B] 7A and 7B show an example of dividing a coding tree unit (CTU) into several coding units (CUs). [Figure 8] FIG. 8 is a diagram illustrating an example of processing by the matrix intra prediction module. [Figure 9A] , [Figure 9B] 9A and 9B show an example CTU having regions where memory access bandwidth is applied for matrix intra prediction modes. [Figure 10] FIG. 10 shows the coding tree corresponding to the example of FIG. [Figure 11]FIG. 11 illustrates a method for encoding coding units of an image frame into a video bitstream, including coding units that use matrix intra prediction. [Figure 12] FIG. 12 illustrates a method for determining memory access budgets for matrix intra prediction across multiple coding units in a coding tree, such as that used in FIG. [Figure 13] FIG. 13 illustrates a method for determining a coding unit for encoding a portion of an image frame into a video bitstream such as that used in FIG. [Figure 14] FIG. 14 shows how coding units of an image frame are encoded into a video bitstream such as that used in FIG. [Figure 15] FIG. 15 shows a method for decoding a coding tree of an image frame from a video bitstream. [Figure 16] FIG. 16 illustrates a method for decoding an image frame coding unit from a video bitstream such as that used in FIG. [Figure 17] FIG. 17 illustrates another method for determining coding units for encoding a portion of an image frame into a video bitstream such as that used in FIG. [Figure 18] FIG. 18 shows another method for encoding coding units of an image frame into a video bitstream such as that used in FIG. [Figure 19] FIG. 19 shows another method for decoding an image frame coding unit from a video bitstream such as that used in FIG. DETAILED DESCRIPTION OF THE INVENTION

[0029] Detailed Description Including Best Mode When reference is made in any one or more of the accompanying drawings to steps and / or features having the same reference number, those steps and / or features have the same function or operation for the purposes of this specification, unless a contrary intention is indicated.

[0030] As mentioned above, the computational complexity of intraframe prediction is limited, especially in the case of matrix intraprediction (MIP). While MIP can provide an effective solution in terms of minimizing errors, the computational complexity of applying MIP particularly impacts the worst-case block processing speed, for example, for frames consisting of only 4x4 blocks. The block processing speed must be sufficient to support the target application resolution and frame rate. Considering only the luma channel, an "8K" resolution frame (7680x4320) at 120 frames per second requires processing 248.8x106 4x4 blocks per second. Even if the entire frame or video sequence does not reach the worst-case scenario, subregions that reach the worst-case scenario must be processed without delaying the delivery of a fully decoded frame for display. One reason for this complexity is the memory bandwidth required to retrieve the matrix coefficients selected according to the matrix intraprediction (MIP) mode, which can vary without constraint from block to block.

[0031] 1 is a schematic block diagram illustrating the functional modules of a video encoding and decoding system 100. To accommodate practical applications and / or to accommodate the coding advantages achieved by MIP mode, system 100 can utilize limitations associated with the application of MIP mode to establish the worst-case memory bandwidth for selecting or retrieving matrix coefficients.

[0032] System 100 includes source device 110 and destination device 130. Communication channel 120 is used to communicate encoded video information from source device 110 to destination device 130. In some configurations, either or both of source device 110 and destination device 130 may be mobile phones or “smartphones,” in which case communication channel 120 is a wireless channel. In other configurations, source device 110 and destination device 130 may be videoconferencing equipment, in which case communication channel 120 is typically a wired channel, such as an Internet connection. Furthermore, source device 110 and destination device 130 may be any of a wide range of devices, including devices supporting over-the-air television broadcasts, cable television applications, Internet video applications (including streaming), and applications in which encoded video data is captured to a computer-readable storage medium, such as a hard disk drive on a file server.

[0033] 1, source device 110 includes video source 112, video encoder 114, and transmitter 116. Video source 112 typically includes a source of captured video frame data (shown as 113), such as an image capture sensor, a previously captured video sequence stored on a non-transitory recording medium, or footage from a remote image capture sensor. Video source 112 may also be, for example, the video output of an operating system or the output of a computer graphics card that displays various applications running on a computing device, such as a tablet computer. Source devices 110 that may include an image capture sensor as the video source 112 include smartphones, video camcorders, professional video cameras, and network video cameras.

[0034] As further described with reference to FIG. 3, video encoder 114 converts (or "encodes") captured frame data (indicated by arrow 113) from video source 112 into a bitstream (indicated by arrow 115). Bitstream 115 is transmitted by transmitter 116 over communication channel 120 as coded video data (or "coded video information"). Bitstream 115 may also be stored in non-transitory storage device 122, such as "flash" memory or a hard disk drive, until later transmitted over communication channel 120, or in lieu of transmission over communication channel 120. For example, the coded video data may be provided on-demand to customers over a wide area network (WAN) for video streaming applications.

[0035] Destination device 130 includes receiver 132, video decoder 134, and display device 136. Receiver 132 receives encoded video data from communication channel 120 and passes the received video data as a bitstream to video decoder 134 (indicated by arrow 133). Video decoder 134 then outputs decoded frame data (indicated by arrow 135) to display device 136. Decoded frame data 135 has the same chroma format as frame data 113. Display device 136 may include, for example, a cathode ray tube or liquid crystal display such as a smartphone, tablet computer, computer monitor, or standalone television device. It is also possible for the functions of source device 110 and destination device 130 to be embodied in a single device, such as a mobile phone or tablet computer.

[0036] While not limited to the devices listed above, each of the source device 110 and the destination device 130 may be configured within a general-purpose computing system, typically through a combination of hardware and software components. Figure 2A illustrates such a computer system 200, including a computer module 201, input devices such as a keyboard 202 (which may be configured as a video source 112), a mouse pointer device 203, a scanner 226, and a camera 227, a microphone 280, and output devices including a printer 215, a display device 214 (which may be configured as a display device 136), and speakers 217. An external modulator-demodulator (modem) transceiver device 216 may be used by the computer module 201 to communicate bidirectionally with a communications network 220 via a connection 221. The communications network 220, representing the communications channel 120, may be a WAN (wide area network), such as the Internet, a cellular network, or a private wide area network (WAN). If the connection 221 is a telephone line, the modem 216 may be a conventional "dial-up" modem. Alternatively, the modem 216 may be a broadband modem if the connection 221 is a high-capacity (e.g., cable or optical) connection. A wireless modem may be used for wireless connection to the communications network 220. The transceiver device 216 may provide the functionality of the transmitter 116 and receiver 132, and the communications channel 120 may be embodied as the connection 221.

[0037] The computer module 201 typically includes at least one processor unit 205 and a memory unit 206. For example, the memory unit 206 may include semiconductor random access memory (RAM) and semiconductor read-only memory (ROM). The computer module 201 also includes a number of input / output (I / O) interfaces, including an audio-video interface 207 that couples to a video display 214, speakers 217, and a microphone 280; an I / O interface 213 that couples to a keyboard 202, a mouse 203, a scanner 226, a camera 227, and optionally a joystick or other human interface device (not shown); and an interface 208 for an external modem 216 and a printer 215. The signal from the audio-video interface 207 to the computer monitor 214 is typically an output of a computer graphics card. In some implementations, the modem 216 may be integrated into the computer module 201, e.g., within the interface 208. The computer module 201 also has a local network interface 211, which allows the computer system 200 to be connected to a local communications network 222, known as a local area network (LAN), via connection 223. As shown in FIG. 2A, the local communications network 222 can also be connected to a wide network 220 via connection 224, which typically includes a so-called "firewall" device or similar functionality. The local network interface 211 may include an Ethernet circuit card, a Bluetooth wireless configuration, or an IEEE 802.11 wireless configuration. However, many other types of interfaces may be used in the interface 211. The local network interface 211 may also provide the functionality of the transmitter 116 and receiver 132, and the communication channel 120 may be embodied in the local communications network 222.

[0038] The I / O interfaces 208 and 213 can provide either or both serial and parallel connections, the former typically implemented according to the Universal Serial Bus (USB) standard and having a corresponding USB connector (not shown). The storage device 209 typically includes a hard disk drive (HDD) 210. Other storage devices, such as floppy disk drives and magnetic tape drives (not shown), may also be used. An optical disk drive 212 is typically provided to function as a non-volatile data source. Portable memory devices, such as optical disks (CD-ROMs, DVDs, Blu-ray Discs, etc.), USB-RAM, portable external hard drives, and floppy disks, may be used as suitable data sources for the computer system 200. Typically, the HDD 210, the optical drive 212, and the networks 220 and 222 may all be configured to operate as a video source 112 or as a destination for decoded video data to be stored for playback on the display 214. The source device 110 and the destination device 130 of the system 100 may be implemented by the computer system 200.

[0039] The components 205 to 213 of the computer module 201 typically communicate via an interconnected bus 204 and in a manner which represents a conventional mode of operation of the computer system 200 known to those skilled in the art. For example, the processor 205 is coupled to the system bus 204 using a connection 218. Similarly, the memory 206 and optical disk drive 212 are coupled to the system bus 204 by a connection 219. Computers on which the described arrangements can be practiced include, for example, IBM-PCs and compatibles, Sun SPARCstations, Apple Mac® and similar computer systems.

[0040] Where appropriate or desired, video encoder 114 and video decoder 134, and the methods described below, may be implemented using computer system 200. In particular, video encoder 114, video decoder 134, and the methods described below may be implemented as one or more software application programs 233 executable within computer system 200. In particular, video encoder 114, video decoder 134, and the steps of the methods described may be implemented by instructions 231 (see FIG. 2B ) of software 233 executed within computer system 200. The software instructions 231 may be formed as one or more code modules, each for performing one or more specific tasks. The software may also be divided into two separate parts, with a first part and corresponding code modules performing the methods described and a second part and corresponding code modules managing a user interface between the first part and a user.

[0041] The software may be stored on a computer-readable medium, including, for example, the storage devices described below. The software is read from the computer-readable medium into and executed by computer system 200. A computer-readable medium having such software or a computer program recorded thereon is a computer program product. Use of the computer program product in computer system 200 provides beneficial effects to video encoder 114, video decoder 134, and other advantageous apparatus for performing the methods described.

[0042] The software 233 is typically stored on the HDD 210 or memory 206. The software is read into the computer system 200 from a computer-readable medium and executed by the computer system 200. Thus, for example, the software 233 may be stored on an optically readable disk storage medium (e.g., a CD-ROM) 225 that is read by the optical disk drive 212.

[0043] In some cases, the application program 233 may be supplied to the user encoded on one or more CD-ROMs 225 and read via the corresponding drive 212 or by the user from the network 220 or 222. Furthermore, software may be read into the computer system 200 from other computer-readable media. A computer-readable storage medium refers to a non-transitory, tangible storage medium that provides recorded instructions and / or data to the computer system 200 for execution and / or processing. Such storage media include, for example, floppy disks, magnetic tapes, CD-ROMs, DVDs, Blu-ray Discs, hard disk drives, ROMs or integrated circuits, USB memory sticks, magneto-optical disks, or computer-readable cards such as PCMCIA cards, and such devices may be internal or external to the computer module 201. Transient or intangible computer-readable transmission media that can be used to provide software, application programs, instructions and / or video data or encoded video data to the computer module 401 include, for example, wireless or infrared transmission channels, network connections to other computers or network devices, and the Internet or intranets, including email transmissions, information stored on websites, etc.

[0044] A second portion of application program 233 and the corresponding code modules described above may be executed to implement one or more graphical user interfaces (GUIs) that are rendered or otherwise represented on display 214. A user of computer system 200 and applications may then manipulate the interfaces in a functionally adaptable manner to provide control commands and / or input to the applications associated with the GUIs, typically by manipulating keyboard 202 and mouse 203. Other forms of functionally adaptable user interfaces may also be implemented, such as a voice interface that utilizes speech prompts output via speaker 217 and user voice commands input via microphone 280.

[0045] Figure 2B is a detailed schematic block diagram of processor 205 and "memory" 234. Memory 234 represents the logical collection of all memory modules (including HDD 209 and semiconductor memory 206) accessible by computer module 201 of Figure 2A.

[0046] When the computer module 201 is first powered on, a power-on self-test (POST) program 250 is executed. The POST program 250 is typically stored in the ROM 249 of the semiconductor memory 206 of FIG. 2A. Hardware devices, such as the ROM 249, that store software are sometimes referred to as firmware. The POST program 250 inspects the hardware within the computer module 201 to ensure that it is functioning properly and checks the correct operation of the processor 205, the memory 234 (209, 206), and the basic input / output system software (BIOS) module 251, which is typically stored in the ROM 249. If the POST program 250 runs successfully, the BIOS 251 boots the hard disk drive 210 of FIG. 2A. The booting of the hard disk drive 210 causes a bootstrap loader program 252, resident on the hard disk drive 210, to be executed via the processor 205. This loads the operating system 253 into the RAM memory 206, and the operating system 253 begins operation. Operating system 253 is a system-level application executable by processor 205 to implement various high-level functions, including processor management, memory management, device management, storage management, software application interface, and general-purpose user interface.

[0047] The operating system 253 manages the memory 234 (209, 206) so that each process and application running on the computer module 201 has enough memory to run without conflicting with memory allocated to another process. Furthermore, the various types of memory available in the computer system 200 of FIG. 2A must be used appropriately so that each process can run effectively. Thus, the aggregate memory 234 is not intended to describe how specific segments of memory are allocated (unless otherwise specified), but rather to illustrate the concept of memory accessible by the computer system 200 and how it is used.

[0048] As shown in FIG. 2B, the processor 205 includes a number of functional modules, including a control unit 239, an arithmetic logic unit (ALU) 240, and a local or internal memory 248, sometimes referred to as a cache memory. The cache memory 248 typically includes a number of storage registers 244-246 within a register section. One or more internal buses 241 operatively interconnect these functional modules. The processor 205 also typically has one or more interfaces 242 for communicating with external devices via the system bus 204 using connection 218. The memory 234 is coupled to the bus 204 using connection 219.

[0049] Application program 233 includes a series of instructions 231, which may include conditional branch and loop instructions. Program 233 may also include data 232 used in the execution of program 233. Instructions 231 and data 232 are stored in memory locations 228, 229, 230 and 235, 236, 237, respectively. Depending on the relative sizes of instruction 231 and memory locations 228-230, an instruction may be stored in a single memory location, as represented by the instruction shown in memory location 230. Alternatively, an instruction may be divided into multiple portions, each stored in a separate memory location, as represented by the segments of the instruction shown in memory locations 228 and 229.

[0050] Generally, the processor 205 is given a set of instructions to execute. The processor 205 waits for a next input, upon which the processor 205 executes another set of instructions. Each input may come from one or more of a number of sources, including data generated by one or more input devices 202, 203 shown in FIG. 2A, data received from an external source via one of the networks 220, 202, data retrieved from one of the storage devices 206, 209, or data retrieved from a storage medium 225 inserted into a corresponding reader 212. Execution of the set of instructions may, in some cases, result in the output of data. Execution may also result in the storage of data or variables in memory 234.

[0051] Video encoder 114, video decoder 134, and the described methods may use input variables 254, which are stored in corresponding memory locations 255, 256, and 257 of memory 234. Video encoder 114, video decoder 134, and the described methods generate output variables 261, which are stored in corresponding memory locations 262, 263, and 264 of memory 234. Intermediate variables 258 may be stored in memory locations 259, 260, 266, and 267.

[0052] Returning to processor 205 of Figure 2B, registers 244, 245, 246, arithmetic logic unit (ALU) 240, and control unit 239 collectively perform the series of micro-operations required to perform a "fetch, decode, execute" cycle for every instruction in the instruction set that makes up program 233. Each fetch, decode, execute cycle includes: a fetch operation that fetches or reads instructions 231 from memory locations 228, 229, 230; a decoding process in which control unit 239 determines which instructions have been fetched; Execution process in which the control unit 239 and / or ALU 240 execute instructions.

[0053] A further fetch, decode, execute cycle is then performed for the next instruction. Similarly, a store cycle may be performed, which causes control unit 239 to store or write a value to memory location 232.

[0054] Each step or sub-process in the methods of Figures 11 through 19 is associated with one or more segments of program 233, as described below, and typically involves register sections 244, 245, 247, ALU 240, and control unit 239 within processor 205 working together to perform a fetch, decode, and execute cycle for all instructions in the instruction set for the noted segment of program 233.

[0055] FIG. 3 is a schematic block diagram illustrating the functional modules of the video encoder 114. FIG. 4 is a schematic block diagram illustrating the functional modules of the video decoder 134. Generally, data is passed between functional modules within the video encoder 114 and the video decoder 134 in groups of samples or coefficients, such as fixed-size sub-blocks or matrices of a block. The video encoder 114 and the video decoder 134 can be implemented using a general-purpose computer system 200, as shown in FIGS. 2A and 2B, with the various functional modules being realized by dedicated hardware within the system 200 or by software executable within the computer system 200, such as one or more software code modules of a software application program 233 residing on a hard disk drive 205 and controlled by execution by the processor 205. Alternatively, the video encoder 114 and the video decoder 134 can be realized by a combination of dedicated hardware and software executable on the computer system 200. Alternatively, the video encoder 114, the video decoder 134, and the described methods can be realized by dedicated hardware, such as one or more integrated circuits that perform the functions or sub-functions of the described methods. Such dedicated hardware may include a graphics processing unit (GPU), a digital signal processor (DSP), an application specific standard product (ASSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or one or more microprocessors and associated memories. In particular, the video encoder 114 includes modules 310-392, and the video decoder 134 includes modules 420-496, each of which may be implemented as one or more software code modules of the software application program 233.

[0056] The video encoder of FIG. 3 is an example of a Versatile Video Coding (VVC) video coding pipeline, although other video codecs may be used to perform the processing steps described herein. The video encoder 114 receives captured frame data 113, such as a series of frames, each frame including one or more color channels. The frame data 113 may be in any chroma format, such as 4:0:0, 4:2:0, 4:2:2, or 4:4:4 chroma format. The block divider 310 first divides the frame data 113 into CTUs, which are generally square in shape and configured to use a specific size for the CTU. The CTU size may be, for example, 64x64, 128x128, or 256x256 luma samples. The block divider 310 further divides each CTU into one or more CBs according to the luma code tree and chroma code tree. The CBs come in a variety of sizes and may include both square and non-square aspect ratios. The operation of the block divider 310 is further described with reference to Figures 11 through 19. However, in the VVC standard, the edge lengths of the CBs, CUs, PUs, and TUs are always powers of two. Thus, a current CB, denoted as 312, is output from the block divider 310, which proceeds by iterating over one or more blocks of the CTU according to the CTU's luma and chroma coding trees. Options for dividing a CTU into CBs are further described below with reference to Figures 5 and 6. While the operation is generally described in terms of CTUs, the video encoder 114 and the video decoder 134 can operate on smaller sized regions to reduce memory consumption. For example, each CTU can be divided into smaller regions called "virtual pipeline data units" (VPDUs) of size 64x64. The VPDUs form smaller data that are more suitable for pipeline processing in a hardware architecture, thereby reducing silicon area and therefore cost due to the reduced memory footprint compared to processing entire CTUs.

[0057] The CTUs resulting from one division of the frame data 113 are scanned in raster scan order and may be grouped into one or more "slices." A slice may be an "intra" (or "I") slice. An intra slice (I slice) indicates that all CUs within the slice are intra predicted. Alternatively, a slice may be unidirectionally predicted or bidirectionally predicted ("P" or "B" slice, respectively), further indicating the possibility of unidirectional and bidirectional prediction within the slice, respectively.

[0058] For each CTU, the video encoder 114 operates in two stages. In the first stage (called the "search" stage), the block partitioner 310 tests various potential configurations of the coding tree. Each potential configuration of the coding tree is associated with a "candidate" CB. In the first stage, various candidate CBs are tested to select one that results in relatively high compression efficiency with relatively low distortion. The testing typically involves Lagrangian optimization, where the candidate CBs are evaluated based on a weighted combination of rate (coding cost) and distortion (error relative to the input frame data 113). The "best" candidate CB (the CB with the lowest evaluated rate / distortion) is selected for subsequent encoding into the bitstream 115. Evaluation of the candidate CBs includes options such as using the CB for a given area, or further dividing the region according to various partitioning options and further encoding each of the resulting smaller areas with the CB, or further dividing the region. As a result, both the CB and the coding tree are selected in the search stage.

[0059] The video encoder 114 generates a prediction block (PB), indicated by arrow 320, for each CB, e.g., CB 312. PB 320 is a prediction of the content of the associated CB 312. A subtraction module 322 generates a difference between PB 320 and CB 312, indicated as 324 (or "residual," which refers to the difference in the spatial domain). Difference 324 is the block-size difference between corresponding samples in PB 320 and CB 312. Difference 324 is transformed and quantized, and represented as a transform block (TB), indicated by arrow 336. PB 320 and the associated TB 336 are typically selected from one of many possible candidate CBs, based, for example, on estimated cost or distortion.

[0060] A candidate coding block (CB) is a CB that results from one of the prediction modes available to video encoder 114 for the associated PB and the resulting residual. TB 336 is a quantized and transformed representation of difference 324. When combined with the predicted PB in video decoder 114, TB 336 reduces the difference between the decoded CB and the original CB 312 by adding signaling to the bitstream.

[0061] Thus, each candidate coding block (CB), i.e., a combination of a prediction block (PB) and a transform block (TB), has an associated coding cost (or "rate") and an associated difference (or "distortion"). The rate is typically measured in bits. The distortion of a CB is typically estimated as a sample value difference, such as the sum of absolute differences (SAD) or the sum of squared differences (SSD). The estimated values ​​resulting from each candidate PB may be determined by a mode selector 386 using the difference 324 to determine a prediction mode 387. The prediction mode 387 indicates a decision to use intra-frame prediction, inter-frame prediction, or matrix intra-prediction (MIP) for the current CB. The prediction mode 387 is determined as a mode selected from among possible modes including intra-prediction (including matrix intra-prediction, DC, planar, and angular intra-prediction) or inter-prediction, with associated motion vectors. The prediction mode 387 is typically selected by minimizing a distortion metric resulting from a Lagrangian optimization of the distortion resulting from each candidate mode summed with the associated rate scaled by a lambda value. If matrix intra-prediction is used, a matrix intra-prediction mode (represented by arrow 388) is also determined to indicate which of several available matrix intra-prediction modes is being used for the current CB. The search to determine the use and selection of MIP modes for blocks, particularly relatively small blocks, may be constrained to mitigate the worst-case memory bandwidth for retrieving matrix coefficients compared to an unconstrained search, as described with reference to Figures 11 to 14 and, for another example, Figures 17 and 18. Estimating the coding cost associated with each candidate prediction mode and the corresponding coding of the residual can be performed at significantly less cost than entropy coding of the residual. Thus, by evaluating several candidate modes, the optimal mode in terms of rate versus distortion can be determined.

[0062] The determination of the optimal mode in terms of rate versus distortion is typically performed using various Lagrangian optimizations. The selection of the matrix intra-prediction mode 388 typically involves determining the coding cost of the residual data resulting from applying a particular matrix intra-prediction mode. The coding cost can be approximated using the "sum of absolute transform differences" (SATD), which allows for an estimated transform residual cost to be obtained using a relatively simple transform, such as the Hadamard transform. In some cases using a relatively simple transform, the cost resulting from the simplified estimation method is monotonically related to the actual cost determined from full evaluation. In cases involving monotonically related estimated costs, the simplified estimation method can be used to make the same decision (i.e., intra-prediction mode) while reducing complexity in the video encoder 114. To accommodate possible non-monotonicity in the relationship between the estimated cost and the actual cost, the simplified estimation method can be used to generate a list of best candidates. Non-monotonicity may arise, for example, from determining additional modes available for coding the residual data. The number of best candidates in the list may be any number. A more thorough search using the best candidates establishes the optimal mode choice for encoding the residual data for each candidate, which allows for the final selection of the intra-prediction mode along with other mode decisions.

[0063] Other mode decisions include the ability to skip the forward transform, known as "transform skip." Transform skip is appropriate for residual data that is not sufficiently correlated to reduce coding costs using formulas such as transform basis functions. Certain types of content, such as relatively simple computer-generated graphics, may exhibit similar properties. In the case of "skipped transform," the residual coefficients are still coded, even though the transform itself is not performed.

[0064] A Lagrangian or similar optimization process can be used to both select the optimal partitioning of the CTUs into CBs (by the block divider 310) and select the best prediction mode from multiple possibilities. Through application of a Lagrangian optimization process of candidate modes in the mode selector module 386, the intra-prediction mode with the lowest cost amount is selected as the “best” mode. The lowest-cost mode is the selected intra-prediction mode 388 and is encoded into the bitstream 115 by the entropy encoder 338. The selection of the intra-prediction mode 388 by the operation of the mode selector module 386 extends to the operation of the block divider 310. For example, the candidates for the selection of intra-prediction mode 388 may include a mode applicable to a given block and additional modes applicable to multiple smaller blocks co-located with the given block. In the case of including modes applicable to the given block and the co-located smaller blocks, the process of selecting candidates is also implicitly a process of determining the best hierarchical decomposition of the CTUs into CBs.

[0065] In a second stage of operation of the video encoder 114 (called the "encoding" stage), an iterative process of the selected luma code tree and the selected chroma code tree, i.e., each selected CB, is performed in the video encoder 114. In the iterative process, the CBs are encoded into a bitstream 115 as described further herein.

[0066] The entropy encoder 338 supports both variable-length coding of syntax elements and arithmetic coding of syntax elements. Arithmetic coding is supported using a context-adaptive binary arithmetic coding process. Arithmetically coded syntax elements consist of a sequence of one or more "bins." Bins, like bits, have a value of "0" or "1." However, bins are not encoded as discrete bits in the bitstream 115. Bins have an associated predicted (or "likely" or "most likely") value and an associated probability, known as the "context." If the actual bin being coded matches the predicted value, a "most likely symbol" (MPS) is coded. Coding a most likely symbol is relatively inexpensive in terms of bits consumed in the bitstream 115, including costs of less than one discrete bit. If the actual bin being coded does not match the expected value, a "least likely symbol" (LPS) is coded. Coding a least likely symbol is relatively expensive in terms of bits consumed. The bin encoding technique allows for efficient encoding of bins with biased "0" vs. "1" probabilities. For syntax elements with two possible values ​​(i.e., "flags"), a single bin is sufficient. For syntax elements with many possible values, a series of bins is required.

[0067] The presence of later bins in the sequence can be determined based on the values ​​of earlier bins in the sequence. Furthermore, each bin may be associated with multiple contexts. The selection of a particular context may depend on earlier bins in the syntax element, bin values ​​of adjacent syntax elements (i.e., bin values ​​from adjacent blocks), etc. Each time a context-coded bin is coded, the context selected for that bin (if any) is updated to reflect the new bin value. In this way, binary arithmetic coding schemes are said to be adaptive.

[0068] Context-free bins ("bypass bins") are also supported by the video encoder 114. Bypass bins are coded assuming equal probability distribution of "0" and "1". Thus, each bin occupies one bit in the bitstream 115. Because the absence of context saves memory and reduces complexity, bypass bins are used when the distribution of values ​​for a particular bin is not skewed. One example of an entropy coder that employs context and adaptation is known in the art as CABAC (Context-Adaptive Binary Arithmetic Coder), and many variations of this coder have been employed in video coding.

[0069] The entropy encoder 338 encodes the prediction mode 387 and, if used for the current CB, encodes the matrix intra-prediction mode 388 using a combination of context coding and bypass-coded bins. A block of size 4x4 has 35 possible matrix intra-prediction modes, and blocks of size not exceeding 8x8 (i.e., 4x8, 8x4, and 8x8) have 19 possible matrix intra-prediction modes. For other block sizes, there are 11 possible matrix intra-prediction modes. Typically, a list of "most likely matrix modes" (MPMs) is generated by the video encoder 114. The list of most likely modes is typically of a fixed length, such as three modes. The list of most likely modes may include modes in neighboring blocks of the current block. For example, if the block above or to the left of the current block is used in MIP mode, the corresponding mode is present as the MPM for the current block. If the block above or to the left of the current block uses angular intra prediction, the MPM list of the current block is filled with MIP modes derived via a lookup table that maps angular intra prediction modes to MIP modes. Furthermore, the MPM list of a subsequent CU predicted using regular intra prediction (DC, planar, or angular) may include candidate intra prediction modes derived from the CU coded using the MIP mode, along with a mapping table from MIP modes to candidate regular intra prediction modes. The context-coded bin encodes a flag indicating whether the intra prediction mode is one of the most likely modes. If the intra prediction mode 388 is one of the most likely modes, further signaling is coded using bypass-coded bins. The coded further signaling indicates which of the most likely modes corresponds to the matrix intra prediction mode 388, for example, using a shortened string of one bin. Otherwise, the intra prediction mode 388 is coded as a "remaining mode."Remaining mode coding uses alternative syntax, such as fixed-length codes coded using bypass-coded bins, to represent intra-prediction modes other than those present in the most likely mode list.

[0070] Some configurations avoid the complexity of creating MPM lists for both the current CU (encoded using an MIP mode) and subsequent CUs, which are encoded using a regular intra-prediction mode and may contain intra-prediction mode candidates derived from neighboring blocks encoded in any MIP mode. By omitting the creation of the MPM list, binarization of MIP modes is performed using shortened binary codes to represent the MIP modes. While shortened binary codes ensure that the coding cost of each MIP mode is relatively uniform, the MPM list reduces the coding cost of each MIP added to the MPM list. Statistics obtained from MIP mode selection show no strong bias in the selection of MIP modes included in the MPM list compared to MIP modes not included in the MPM list, indicating that omitting the creation of the MPM list does not reduce compression efficiency. For a 4x4 block, 35 possible MIP modes can be encoded using 5-bit or 6-bit codes: MIP modes 0 to 28 use 5 bits, and MIP modes 29 to 34 use 6 bits. For 4x8, 8x4, and 8x8 blocks, the 19 possible MIP modes can be encoded using 4-bit or 5-bit codes, with MIP modes 0-12 using 4 bits and MIP modes 13-18 using 5 bits. For blocks of other sizes, the 11 possible MIP modes can be encoded using 3-bit or 4-bit codes, with MIP modes 0-4 using 3 bits and MIP modes 5-10 using 4 bits.

[0071] A multiplexer module 384 selects from the tested prediction modes for each candidate CB according to the determined best prediction mode 387 and outputs PB 320. The candidate prediction modes need not include all possible prediction modes supported by the video encoder 114.

[0072] In inter-frame prediction, a block prediction is generated using samples from one or two frames preceding the current frame in the order in which the frames are coded in the bitstream. Furthermore, in inter-frame prediction, a single coding tree is typically used for both the luma and chroma channels. The order in which frames are coded in the bitstream may differ from the order in which the frames are captured or displayed. When one frame is used for prediction, the block is called "unidirectionally predicted" and has one associated motion vector. When two frames are used for prediction, the block is called "bidirectionally predicted" and has two associated motion vectors. For P slices, each CU can be intra-predicted or unidirectionally predicted. For B slices, each CU can be intra-predicted, unidirectionally predicted, or bidirectionally predicted. Frames are typically coded using a "group of pictures" structure, which allows for temporal hierarchy of frames. Temporal hierarchy of frames allows frames to reference previous and next pictures in the display order. Pictures are coded in the order required to ensure dependencies for decoding each frame are met.

[0073] Samples are selected according to a motion vector and a reference image index. Because the motion vector and reference image index apply to all color channels, inter-prediction is primarily described in terms of processing for PUs, not PBs. Within each category (i.e., intra-frame and inter-frame prediction), different techniques can be applied to generate PUs. For example, intra-prediction uses values ​​from rows and columns adjacent to a previously reconstructed sample, combined with orientation, to generate PUs according to a predefined filtering and generation process. Alternatively, PUs may be described using a small number of parameters. Inter-prediction methods may differ in the number and precision of motion parameters. Motion parameters typically include a reference frame index, which indicates which reference frame to use from a list of reference frames and the spatial transformation of each reference frame, but they can also include more frames, special frames, or complex affine parameters such as scaling and rotation. Furthermore, predefined motion refinement processes may be applied to generate dense motion estimates based on the referenced sample blocks.

[0074] By determining and selecting PB 320 and subtracting PB 320 from the original sample block in subtractor 322, a residual with the lowest coding cost, represented as 324, is obtained and subjected to lossy compression. The lossy compression process consists of a transform step, a quantization step, and an entropy coding process. A forward primary transform module 326 applies a forward transform to the difference 324 to convert the difference 324 from the spatial domain to the frequency domain, generating primary transform coefficients represented by arrow 328. The primary transform coefficients 328 are passed to a forward secondary transform module 330, which performs a non-separable secondary transform (NSST) operation to generate transform coefficients represented by arrow 332. The forward primary transform is typically separable, transforming a set of rows of each block before transforming a set of columns. The forward primary transform module 326 typically uses a Type II Discrete Cosine Transform (DCT-2), but may also use, for example, a Type VII Discrete Sine Transform (DCT-7) and a Type VIII Discrete Cosine Transform (DCT-8) horizontally for block widths not exceeding 16 samples and vertically for block heights not exceeding 16 samples. The transform of each set of rows and columns is performed by first applying a one-dimensional transform to each row of the block to generate an intermediate result, and then to each column of the intermediate result to generate a final result. The forward secondary transform of module 330 is generally a non-separable transform and is applied only to the residual of intra-predicted CUs, but may be bypassed. The forward secondary transform operates on either 16 samples (arranged as a 4x4 sub-block in the top left corner of the primary transform coefficients 328) or 64 samples (arranged as an 8x8 coefficient in the top left corner, arranged as four 4x4 sub-blocks of the primary transform coefficients 328).

[0075] The transform coefficients 332 are passed to a quantization module 334. Quantization is performed in the module 334 according to a quantization parameter to generate residual coefficients, represented by arrow 336. The quantization parameter is constant for a given TB, resulting in uniform scaling for generating the residual coefficients of the TB. Non-uniform scaling can also be achieved by applying a quantization matrix, in which a scaling factor applied to each residual coefficient is derived from a combination of the quantization parameter and a corresponding entry in a scaling matrix, typically the same size as the TB. The scaling matrix can be smaller than the size of the TB, and when applied to the TB, a nearest-neighbor approach can be used to provide a scaling value for each residual coefficient from a scaling matrix smaller than the size of the TB. The residual coefficients 336 are provided to an entropy encoder 338 for encoding in the bitstream 115. Typically, the residual coefficients of each TB with at least one significant residual coefficient of the TU are scanned to generate a list of values ​​ordered according to a scan pattern. The scan pattern typically scans the TB as a sequence of 4x4 "sub-blocks," thereby providing a normal scanning operation at a granularity of 4x4 sets of residual coefficients, with the sub-blocks being arranged according to the size of the TB. Additionally, the prediction mode 387, matrix intra prediction mode (if used) 388, and corresponding block partitioning are also coded in the bitstream 115.

[0076] As noted above, the video encoder 114 requires access to a frame representation that corresponds to the frame representation seen by the video decoder 134. Accordingly, the residual coefficients 336 are also inverse quantized by an inverse quantization module 340 to generate inverse transform coefficients, represented by arrow 342. The inverse transform coefficients 342 are passed to an inverse secondary transform module 344 to generate intermediate inverse transform coefficients, represented by arrow 346. The intermediate inverse transform coefficients 346 are passed to an inverse primary transform module 348 to generate residual samples for the TU, represented by arrow 350. The type of inverse transform performed by the inverse secondary transform module 344 corresponds to the type of forward transform performed by the forward secondary transform module 330. The type of inverse transform performed by the inverse primary transform module 348 corresponds to the type of primary transform performed by the primary transform module 326. A summation module 352 sums the residual samples 350 and the PU 320 to generate reconstructed samples for the CU (indicated by arrow 354).

[0077] The reconstructed samples 354 are passed to a reference sample cache 356 and an in-loop filter module 368. The reference sample cache 356, typically implemented using static RAM on an ASIC (thus avoiding costly off-chip memory accesses), provides the minimum sample storage necessary to satisfy dependencies for generating the intra-frame PBs of subsequent CUs in the frame. The minimal dependencies typically include a "line buffer" of samples along the bottom of a row of CTUs for use in the next row of CTUs and a column buffering range set by the height of the CTU. The reference sample cache 356 supplies reference samples (represented by arrow 358) to a reference sample filter 360. The sample filter 360 applies a smoothing operation to generate filtered reference samples (indicated by arrow 362). The filtered reference samples 362 are used by an intra-frame prediction module 364 to generate an intra-predicted block of samples, represented by arrow 366. For each candidate intra-prediction mode, the intra-frame prediction module 364 generates a block 366 of samples. The block of samples 366 is generated by module 364 using techniques such as DC, planar or angular intra prediction, but not matrix intra prediction.

[0078] If the mode selector 386 selects matrix intra-prediction for the current CB, a matrix intra-prediction mode 388 is used to select (read) matrix coefficients 363 from a coefficient memory 392. The matrix coefficients 363 are passed to a matrix intra-prediction module 390, which performs a matrix multiplication using the matrix coefficients 363 and the reference samples 358 to generate a matrix intra-prediction block 393. The multiplexer 384 outputs the matrix intra-prediction block 393 as the PB 320. The coefficient memory 392 has a limited bandwidth required to provide the matrix coefficients 393. In particular, different matrix intra-prediction modes are used for successive blocks to establish a worst-case memory bandwidth requirement. The mode selector 386 is operable to select a MIP mode for a block under constraints that reduce the worst-case memory bandwidth of the coefficient memory 392, as described in connection with the operation of Figures 11-14. Reducing the worst-case memory bandwidth of coefficient memory 392 reduces complexity, e.g., memory hardware area, without a proportional loss in coding efficiency of video encoder 114. Because the statistics of MIP mode selection in an unconstrained search do not impose constraints that are typically imposed on MIP mode selection in mode selector 386, complexity can be reduced without a proportional loss in coding efficiency. Thus, worst-case memory bandwidth is reduced without a commensurate loss in coding performance.

[0079] The in-loop filter module 368 performs several stages of filtering on the reconstructed samples 354. The filtering stages include a "deblocking filter" (DBF), which applies smoothing along CU boundaries to reduce artifacts due to discontinuities. Another filtering stage present in the in-loop filter module 368 is an "adaptive loop filter" (ALF), which applies a Wiener-based adaptive filter to further reduce distortion. A further filtering stage available in the in-loop filter module 368 is a "sample adaptive offset" (SAO) filter. The SAO filter operates by first classifying the reconstructed samples into one or more categories and then applying an offset at the sample level according to the assigned category.

[0080] The filtered samples, represented by arrow 370, are output from the in-loop filter module 368. The filtered samples 370 are stored in a frame buffer 372. The frame buffer 372 typically has a capacity to store several images (e.g., up to 16) and is implemented in the memory 206. The frame buffer 372 is typically not implemented with on-chip memory due to the large memory consumption required. Therefore, accessing the frame buffer 372 is costly in terms of memory bandwidth. The frame buffer 372 provides reference frames (represented by arrow 374) to a motion estimation module 376 and a motion compensation module 380.

[0081] The motion estimation module 376 estimates several “motion vectors” (shown as 378), each of which is a Cartesian spatial offset from the position of the current CB and references a block in one of the reference frames in the frame buffer 372. A filtered block of reference samples (shown as 382) is generated for each motion vector. The filtered reference samples 382 form further candidate modes available for potential selection by the mode selector 386. Furthermore, for a given CU, the PU 320 may be formed using one reference block (“unidirectional prediction”) or two reference blocks (“bidirectional prediction”). For a selected motion vector, the motion compensation module 380 generates the PB 320 according to a filtering process consistent with the sub-pixel accuracy of the motion vector. Thus, the motion estimation module 376 (operating with many candidate motion vectors) may perform a simplified filtering process than that of the motion compensation module 380 (operating with only the selected candidate), reducing computational complexity. When video encoder 114 selects inter prediction for a CU, motion vector 378 is coded into bitstream 115.

[0082] 3 is described with reference to Versatile Video Coding (VVC), other video coding encoding standards or implementations may employ the processing stages of modules 310-386. Frame data 113 (and bitstream 115) may also be read from (or written to) memory 206, hard disk drive 210, CD-ROM, Blu-ray disc, or other computer-readable storage medium. Additionally, frame data 113 (and bitstream 115) may be received from (or transmitted to) an external source, such as a server connected to communications network 220 or a radio frequency receiver.

[0083] The video decoder 134 is shown in FIG. 4. The video decoder 134 of FIG. 4 is an example of a Versatile Video Coding (VVC) video decoding pipeline, although other video codecs may be used to perform the processing steps described herein. As shown in FIG. 4, a bitstream 133 is input to the video decoder 134. The bitstream 133 may be read from memory 206, a hard disk drive 210, a CD-ROM, a Blu-ray disc, or other non-transitory computer-readable storage medium. Alternatively, the bitstream 133 may be received from an external source, such as a server connected to a communications network 220 or a radio frequency receiver. The bitstream 133 includes encoded syntax elements representing captured frame data to be decoded.

[0084] The bitstream 133 is input to the entropy decoder module 420. The entropy decoder module 420 extracts syntax elements from the bitstream 133 by decoding a series of "bins" and passes the values ​​of the syntax elements to other modules in the video decoder 134. The entropy decoder module 420 uses an arithmetic decoding engine to decode each syntax element as a series of one or more bins. Each bin can use one or more "contexts," which represent probability levels used to encode the bin's "1" and "0" values. If multiple contexts are available for a bin, a "context modeling" or "context selection" process is performed to select one of the available contexts to decode the bin. The bin decoding process forms a continuous feedback loop. The number of processes in the feedback loop is preferably minimized to enable the entropy decoder 420 to achieve a high throughput in bins per second. Context modeling relies on other attributes of the bitstream known to the video decoder 134 when selecting a context, i.e., attributes preceding the current bin. For example, the context may be selected based on the quadtree depth of the current CU in the coding tree. The dependencies are preferably based on attributes that are well known prior to decoding the bin or that can be determined without requiring long sequential processing.

[0085] The quadtree depth of a coding tree is a readily known example of a context modeling dependency. Intra-prediction modes, particularly matrix intra-prediction modes, are examples of context modeling and binarization dependencies that are relatively difficult or computationally intensive to determine. Matrix intra-prediction modes are coded as an index into a list of "most likely modes" (MPMs) or a list of "remaining modes," and one of these modes is selected according to the decoded context coding flag. Other intra-prediction modes are coded as an index into a list of "most likely modes" (MPMs) or a list of "remaining modes," and one of these modes is selected according to the decoded intraluma_MPM flag. If an MPM is used to encode a matrix intra-prediction mode, a shortened single-bin string ranging from 0 to 2 selects one of the MPMs from the MPM list. If a remaining mode is used, a fixed-length codeword is decoded to select which of the remaining (non-MPM) modes to use. The number of available MPM modes is either 35, 19, or 11, depending on the block size. Thus, for an MPM list of length 3, the number of remaining modes is 32, 16, or 8, respectively. The remaining modes can be efficiently represented by fixed-length codewords of length 5, 4, or 3, respectively. Determining both the most likely mode and the remaining modes requires a significant amount of processing, including dependence on the intra-prediction modes of neighboring blocks. For example, the neighboring blocks may be blocks above and to the left of the current block. If the neighboring blocks use angular intra-prediction, a table lookup can be performed to map the angular intra-prediction mode to a matrix intra-prediction mode for use in creating the MPM list. Alternatively, the configuration can encode the MIP modes using shortened value codewords depending on whether the number of modes is 35, 19, or 11 in each case. The entropy decoder module 420 applies an arithmetic coding algorithm, such as "context-adaptive binary arithmetic coding" (CABAC), to decode syntax elements from the bitstream 133.The decoded syntax elements are used to reconstruct parameters within video decoder 134. The parameters include residual coefficients (represented by arrow 424) and mode selection information such as intra-prediction modes (represented by arrow 458). The mode selection information also includes information such as motion vectors and the division of each CTU into one or more CBs. The parameters, typically along with sample data from previously decoded CBs, are used to generate PBs.

[0086] The residual coefficients 424 are input to an inverse quantization module 428. The inverse quantization module 428 performs inverse quantization (or "scaling") on the residual coefficients 424, generating reconstructed intermediate transform coefficients, indicated by arrow 432, depending on the quantization parameters. The reconstructed intermediate transform coefficients 432 are passed to an inverse secondary transform module 436, which applies a secondary transform or bypasses it. The inverse secondary transform module 436 generates reconstructed transform coefficients 440. If the use of a non-uniform inverse quantization matrix is ​​indicated in the bitstream 133, the video decoder 134 reads the quantization matrix from the bitstream 133 as a series of scaling coefficients and arranges them as a matrix. Inverse scaling uses the quantization matrix in combination with the quantization parameters to generate the reconstructed intermediate transform coefficients 432.

[0087] The reconstructed transform coefficients 440 are passed to an inverse linear transform module 444, which converts the coefficients from the frequency domain back to the spatial domain. The operation by module 444 results in a block of residual samples, indicated by arrow 448. The block of residual samples 448 is equal in size to the corresponding CU. The residual samples 448 are provided to a summation module 450, where they are summed with a decoded PB (denoted as 452) to generate a block of reconstructed samples, indicated by arrow 456. The reconstructed samples 456 are provided to a reconstructed sample cache 460 and an in-loop filtering module 488, which generates a block of reconstructed frame samples, denoted as 492. The frame samples 492 are written to a frame buffer 496.

[0088] The reconstructed sample cache 460 operates similarly to the reconstructed sample cache 356 of the video encoder 114. The reconstructed sample cache 460 provides storage for reconstructed samples needed to intra-predict subsequent CBs without the memory 206 (e.g., by using data 232 instead, typically on-chip memory). Reference samples, represented by arrow 464, are retrieved from the reconstructed sample cache 460 and provided to a reference sample filter 468 to generate filtered reference samples, represented by arrow 472. The filtered reference samples 472 are provided to an intra-frame prediction module 476. The module 476 generates blocks of intra-predicted samples, represented by arrow 480, according to the intra-prediction mode parameter 458 signaled in the bitstream 133 and decoded by the entropy decoder 420. The blocks of samples 480 are generated using modes such as DC, planar, or angular intra-prediction, rather than matrix intra-prediction.

[0089] If the prediction mode of the CB is indicated in the bitstream 133 to use intra prediction (other than matrix intra prediction), the intra predicted samples 480 form the decoded PB 452 via the multiplexer module 484. Intra prediction generates a predictive block of samples (PB), i.e., a block of one color component, derived using "neighboring samples" of the same color component. The neighboring samples are samples adjacent to the current block and precede it in block decoding order, and therefore have already been reconstructed. If the luma block and the chroma block are co-located, they may use different intra prediction modes. However, each of the two chroma channels shares the same intra prediction mode. Intra prediction is classified into three types: "DC intra prediction" involves filling the PB with a single value representing the average of neighboring samples; and "planar intra prediction" involves filling the PB with plane-dependent samples, where the DC offset and vertical and horizontal gradients are derived from the neighboring samples. "Angular intra prediction" involves filling a PB with neighboring samples that have been filtered and propagated in a specific direction (or "angle") across the PB. In VVC, 65 angles are supported, with rectangular blocks providing additional angles not available in square blocks, yielding a total of 87 angles. A fourth type of intra prediction is available for chroma PBs, whereby the PB is generated from co-located luma reconstructed samples according to a "cross-component linear model" (CCLM) mode. Three different CCLM modes are available, each using a different model derived from neighboring luma and chroma samples. The derived model is then used to generate a block of samples for the chroma PB from the co-located luma samples.

[0090] When the bitstream 133 indicates that the prediction mode for CB is matrix intra prediction, the matrix intra prediction mode 458 is decoded and provided to the coefficient memory 486 and the matrix intra prediction module 482. The matrix coefficients 481 are read from the coefficient memory 486 of the selected matrix intra prediction mode and sent to the matrix intra prediction module 482. The selection of the matrix coefficients involves a memory read operation from the coefficient memory 486, and as described with reference to Figures 8 and 9, the frequency of selecting MIP mode for a certain block size establishes the worst-case memory bandwidth limit for memory accesses.

[0091] If the bitstream 133 indicates that the prediction mode of the CB is inter prediction, the motion compensation module 434 uses the motion vector and the reference frame index to select and filter a block of samples 498 from a frame buffer 496 to generate a block of inter predicted samples, represented by 438. The block of samples 498 is obtained from a previously decoded frame stored in the frame buffer 496. In the case of bidirectional prediction, two blocks of samples are generated and mixed to generate samples for the decoded PB 452. The frame buffer 496 is filled with filtered block data 492 from the in-loop filtering module 488. Like the in-loop filtering module 368 of the video encoder 114, the in-loop filtering module 488 performs either DBF, ALF, or SAO filtering. Generally, the motion vector is applied to both the luma and chroma channels, although the filtering for sub-sample interpolation of the luma and chroma channels is different.

[0092] 5 is a schematic block diagram illustrating an example partition 500 that can be obtained by dividing or branching a region into one or more subregions in a Versatile Video Coding tree structure. The partition shown in example partition 500 can be performed by block partitioning unit 310 of encoder 114, which divides each CTU into one or more CUs or CBs according to a coding tree as determined by Lagrangian optimization, as described with reference to FIG. 3.

[0093] Although the example division 500 illustrates a square region being divided into other, possibly non-square, sub-regions, it should be understood that diagram 500 illustrates a possible division and that the containing region need not be square. If the containing region is non-square, the dimensions of the resulting blocks are scaled according to the aspect ratio of the containing block. When a region is no longer divided, i.e., at a leaf node of the coding tree, a CU occupies the region. The particular subdivision of a CTU into one or more CUs by the block divider 310 is referred to as the CTU's "coding tree."

[0094] The process of subdividing a region into subregions must terminate when the resulting subregion reaches the minimum CU size. In addition to restricting CUs to prohibit block regions smaller than a predetermined minimum size, e.g., 16 samples, CUs are restricted to have a minimum width or height of 4. Other minimum values ​​are possible, either in terms of width and height or width or height. The subdivision process may terminate before the deepest level of decomposition, resulting in a CU larger than the minimum CU size. It is also possible that no division occurs and a single CU occupies an entire CTU. A single CU occupying an entire CTU is the size of the largest available coding unit. Due to the use of subsampled chroma formats, such as 4:2:0, the configuration of the video encoder 114 and video decoder 134 may terminate the division of regions for chroma channels earlier than for luma channels.

[0095] Leaf nodes of the coding tree contain CUs that cannot be further subdivided. For example, leaf node 510 contains one CU. Non-leaf nodes of the coding tree may be further subdivided into two or more nodes, each of which may be a leaf node forming one CU or a non-leaf node containing further divisions into smaller regions. Each leaf node of the coding tree contains one coding block per color channel. If the division ends at the same depth for both luma and chroma, three co-located CBs are generated. If the division ends at a depth deeper for luma than for chroma, multiple luma CBs will be co-located with the CBs for the chroma channels.

[0096] Quadtree partitioning 512 divides the containing region into four equally sized regions, as shown in Figure 5. Compared to HEVC, Versatile Video Coding (VVC) achieves more flexibility by including horizontal binary partitioning 514 and vertical binary partitioning 516. Partitions 514 and 516 each divide the containing region into two equally sized regions. The partitions are along either horizontal boundaries (514) or vertical boundaries (516) within the containing block.

[0097] Further flexibility in Versatile Video Coding is achieved by adding horizontal trisection 518 and vertical trisection 520. Trisections 518 and 520 divide a block into three regions either horizontally (518) or vertically (520) along ¼ and ¾ of the width or height of the containing region. The combination of quadtree, binary tree, and ternary tree is called a "QTBTTT." The root of the tree contains zero or more quadtree divisions (the "QT" section of the tree). Once a QT section ends, zero or more bisections or trisections occur (the "multi-tree" or "MT" section of the tree), ultimately terminating in CBs or CUs at the leaf nodes of the tree structure. If the tree represents all color channels, the leaf nodes of the tree are CUs. If the tree represents the luma or chroma channels, the leaf nodes of the tree are CBs.

[0098] Compared to HEVC, which only supports quadtrees and, as a result, square blocks, QTBTTT allows for more CU sizes, particularly by considering the possible recursive application of binary and / or ternary tree partitioning. Restricting partitioning options can reduce the likelihood of abnormal (non-square) block sizes and eliminate partitioning into samples whose block width or height is less than four or is not a multiple of four. Generally, restrictions are applied by considering luma samples. However, in the described configuration, restrictions can be applied separately to chroma channel blocks. Applying restrictions to partitioning options to chroma channels—for example, if the frame data is in 4:2:0 chroma format or 4:2:2 chroma format—can result in different minimum block sizes for luma and chroma. Each partition generates a subregion with side dimensions that are the same, half, or one-quarter the size of the containing region. And because the CTU size is a power of two, the side dimensions of all CUs are also a power of two.

[0099] 6 is a schematic flow diagram illustrating a data flow 600 of the QTBTTT (or "coding tree") structure used in Versatile Video Coding. The QTBTTT structure is used for each CTU and defines the partitioning of the CTU into one or more CUs. The QTBTTT structure for each CTU is determined by the block partitioner 310 of the video encoder 114 and encoded into the bitstream 115 or decoded from the bitstream 133 by the entropy decoder 420 of the video decoder 134. The data flow 600 further characterizes the permissible combinations available to the block partitioner 310 for partitioning the CTU into one or more CUs according to the partitioning states shown in FIG. 5.

[0100] Starting from the top level of the hierarchy, i.e., CTU, zero or more quadtree splits are first performed. Specifically, a quadtree (QT) split decision 610 is made by the block splitter 310. A decision at 610 returning a "1" symbol indicates a decision to split the current node into four subnodes according to the quadtree split 512. As a result, four new nodes are created, as at 620, and a recursion to the QT split decision 610 is made for each new node. Each new node is considered in raster (or Z-scan) order. Alternatively, if the QT split decision 610 indicates that no further splits are to be performed (returning a "0" symbol), the quadtree split stops and a multi-tree (MT) split is subsequently considered.

[0101] First, an MT split decision 612 is made by the block splitter 310. At 612, a decision to perform MT splitting is indicated. Returning a "0" symbol at decision 612 indicates that no further splitting of the node into subnodes is to be performed. If no further splitting of the node is to be performed, the node is a leaf node of the coding tree and corresponds to a CU. The leaf node is output at 622. Alternatively, if the MT split 612 indicates a decision to perform MT splitting (returning a "1" symbol), the block splitter 310 proceeds to direction decision 614.

[0102] Direction decision 614 indicates the direction of the MT division as horizontal ("H" or "0") or vertical ("V" or "1"). If decision 614 returns "0", indicating a horizontal direction, the block divider 310 proceeds to decision 616. If decision 614 returns "1", indicating a vertical direction, the block divider 310 proceeds to decision 618.

[0103] In each of decisions 616 and 618, the number of divisions for MT division is indicated as either 2 (bisection or "BT" node) or 3 (trisection or "TT") in BT / TT division. That is, if the direction indicated by 614 is horizontal, BT / TT division decision 616 is made by block divider 310, and if the direction indicated by 614 is vertical, BT / TT division decision 618 is made by block divider 310.

[0104] The BT / TT split decision 616 indicates whether the horizontal split is bisection 514, indicated by returning a "0", or trisection 518, indicated by returning a "1". If the BT / TT split decision 616 indicates bisection, then in an HBT CTU node generation step 625, two nodes are generated by the block divider 310 according to the horizontal bisection 514. If the BT / TT split 616 indicates trisection, then in an HTT CTU node generation step 626, three nodes are generated by the block divider 310 according to the horizontal trisection 518.

[0105] The BT / TT split decision 618 indicates whether the vertical split is a bisection 516, indicated by returning a "0," or a trisection 520, indicated by returning a "1." If the BT / TT split 618 indicates a bisection, then in a VBT CTU node generation step 627, the block divider 310 generates two nodes according to the vertical bisection 516. If the BT / TT split 618 indicates a trisection, then in a VTT CTU node generation step 628, the block divider 310 generates three nodes according to the vertical trisection 520. For each node generated in steps 625-628, the data flow 600 is repeated, following the direction 614, from left to right and top to bottom, returning to the MT split decision 612. As a result, binary tree splitting and ternary tree splitting can be applied to generate CUs of various sizes.

[0106] 7A and 7B show an example partition 700 of a CTU 710 into multiple CUs or CBs. An example CU 712 is shown in FIG. 7A. FIG. 7A shows the spatial arrangement of CUs within the CTU 710. The example partition 700 is also shown as a coding tree 720 in FIG. 7B.

[0107] For each non-leaf node of the CTU 710 in FIG. 7A, e.g., nodes 714, 716, and 718, the contained nodes (further splits or CUs) are scanned or probed in "Z-order" to generate a list of nodes as shown in the columns of the coding tree 720. For quadtree splits, the Z-order scan results in an upper-left to right, followed by a lower-left to right. For horizontal and vertical splits, the Z-order scan (probing) simplifies to a top-to-bottom scan and a left-to-right scan, respectively. The coding tree 720 in FIG. 7B lists all nodes and CUs according to the scan order applied. Each split generates a list of two, three, or four new nodes at the next tree level until a leaf node (CU) is reached.

[0108] Once the block divider 310 has decomposed the image, ultimately down to CUs, and used the CUs to generate respective residual blocks (324) as described with reference to Figure 3, the residual blocks are subjected to forward transform and quantization by the video encoder 114. The resulting TBs 336 are scanned in order to form a sequential list of residual coefficients as part of the operation of the entropy coding module 338. A similar process is performed by the video encoder 114 to obtain the TBs from the bitstream 133.

[0109] As a result of dividing the CPU into CUs, smaller CUs (e.g., 4x4, 4x8, or 8x4) appear spatially adjacent within the frame. Furthermore, the hierarchical Z-order scan of the code tree benefits from smaller CUs being processed sequentially. Specifically, quadtree division of an 8x8 region results in a series of four 4x4 CUs; ​​bisection of an 8x4 or 4x8 region results in a pair of 4x4 CUs; ​​bisection of an 8x8 region results in a pair of 4x8 regions or a pair of 8x4 regions. Each resulting region (4x8 or 8x4 in size) can either form a CU or be further divided into 4x4 CUs. Applying trisection to a region of 64 samples produces three regions of sizes 16, 32, and 16, i.e., 4x4, 4x8, or 8x4, and 4x4. Applying trisection to a region of 128 samples produces three regions of sizes 32, 64, and 32, e.g., 4x8, 8x8, and 4x8. Small blocks (e.g., 4x4, 4x8, 8x4) are the result of dividing regions of size such as 64 or 128, and so are seen together both spatially and in the Z-order scan of the CTU.

[0110] Figure 8 is a dataflow 800 detailing the operation of matrix intra prediction modules 390 and 482 in video encoder 114 and video decoder 134, respectively, using an 8x8 block. Matrix intra prediction is performed only on the luma channel in the example of Figure 8, without needing to consider different block sizes resulting from using chroma formats such as 4:2:0 or 4:2:2. Modules 390 and 482 output blocks of matrix prediction samples, such as 393 or 483, respectively. When mode selector 386 selects matrix intra prediction for a CU, block 393 is used as PB 320 in video encoder 114. When decoding the prediction mode of the CU from bitstream 133 indicates the use of matrix intra prediction, block 483 is used as PB 483.

[0111] The operation of modules 390 and 482 includes three steps: 1. averaging, 2. matrix multiplication and offset (bias) addition, and 3. bilinear interpolation. Bilinear interpolation is only performed if the size of the CU is greater than 4x4. The averaging step operates as follows: A reference sample 802 (e.g., 358 or 464) is received in data flow 800 and assigned as a top sample 822 and a left sample 820. If the width and height of the luma CB are greater than 4, the top sample 822 is divided into four sets (four pairs in the example 8x8 block of Figure 8). The content values ​​of each of the four sets are averaged to generate four filtered top samples 826. Similarly, the left sample 820 is divided into four sets (four pairs in the example 8x8 block of Figure 8). The content values ​​of each of the four sets are averaged to generate four filtered left samples 824. Thus, a total of eight filtered samples are input to matrix multiplication module 828.

[0112] If the block width and height are equal to four (instead of eight as shown in FIG. 8), the four top samples 822 are split into two pairs that are each averaged to produce two filtered top samples 826. Similarly, the four left samples 820 are split into two pairs that are each averaged to produce two filtered left samples 824, for a total of four filtered samples that are input to the matrix multiplication module 828.

[0113] In the second step (matrix multiplication), the received matrix coefficients (363 or 481) are selected according to the matrix intra-prediction mode (388 or 458) and are also input to the matrix multiplication module 828. The matrix multiplication module also receives a set of offset or bias values. The offset or bias values ​​are added to the result of the matrix multiplication to include any DC shift. The matrix coefficients and bias values ​​are predetermined. In other words, the matrix coefficients and bias values ​​are the result of an “offline” training process and are considered constant values ​​by the video encoder 114 and the video decoder 134.

[0114] For a 4x4 CB, 35 MIP modes are available with 18 sets of matrix coefficients and bias values ​​(Set A). For 4x8, 8x4, and 8x8 CBs, 19 MIP modes are available with 10 sets of matrix coefficients and bias values ​​(Set B). For other CB sizes, 11 MIP modes are available with 6 sets of matrix coefficients and bias values ​​(Set C). A given set of matrix coefficients and bias values ​​can be used for two MIP modes. One MIP mode uses the provided values, and the other MIP mode uses the transpose of the provided values. Furthermore, in some cases, a set of matrix coefficients and bias values ​​is dedicated to one MIP mode. Each of the three cases applies to Sets A to C. The sizes of Sets A to C are shown below, along with the number of words selected by 362 or 481 to be used to generate one PB (i.e., apply the MIP mode to one CU) for each of Sets A to C. Set A: 18 matrices of size 16x4, 18 offset vectors of size 16. The size of the values ​​chosen for CB is 16x4+16=80 words. Set B: 10 matrices of size 16x8, 10 offset vectors of size 16. The size of the values ​​chosen for CB is 16x8+16=144 words. Set C: 6 matrices of size 64x8, 10 offset vectors of size 64. The size of the values ​​chosen for CB is 64x8+64=576 words.

[0115] The video encoder 114 and the video decoder 134 process video data at a pixel rate determined by the frame size and frame rate. Furthermore, the size of the luma CB is a multiple of 4 in width and height. Therefore, the memory bandwidth requirement for sets A to C can be expressed by accessing 4x4 luma samples. If there are no restrictions on the use of MIP mode for each CU, the worst-case access density for each set to sets A to C, i.e., each set with the smallest block size, is as follows: Set A: 4x4CB requires 80 words per 4x4 sample area. Set B: 4x8 and 8x4CB require 144÷2=72 words per 4x4 sample area, and 8x8CB requires 144÷4=36 words per 4x4 sample area. Set C: 8x16 and 16x8 CBs require 576 / 8 = 72 words per 4x4 sample region; 8x32, 16x16, and 32x8 CBs require 576 / 16 = 36 words per 4x4 sample region; larger CBs require fewer words per 4x4 sample region.

[0116] As indicated by the access density above, if there are no constraints on the use of MIP mode, in the worst case, all CBs may use MIP mode, and the code tree may decompose each CTU into a small CB size, which was used above to indicate the worst-case coefficient memory (486, 392) bandwidth. The nominal word size of the matrix coefficients and bias values ​​is 16 bits, but fewer bits, e.g., 10 bits, may be sufficient. Fetching words in groups is one viable option. However, the memory bandwidth burden remains somewhat high.

[0117] Statistics for MIP mode selection show that, typically, in 20% of cases for a wide test set (as defined in the JVET common test conditions document JVET-N1010), MIP mode is selected for adjacently located CBs when considering the top and left blocks. Thus, by limiting the frequency of MIP mode application, it is possible to reduce the worst-case memory bandwidth required for coefficient memories 486 and 392 without incurring a corresponding reduction in compression efficiency gains in terms of MIP mode availability. Constraints on MIP mode selection are described with reference to Figures 11-16.

[0118] A matrix multiplication module 828 performs a matrix multiplication using a set of matrix coefficients (363 or 481) and the filtered reference samples (i.e., 824 and 826). A bias value is added to the output of the matrix multiplication to form a sparse block 830. The sparse block 830 is represented as the shaded samples that partially fill 393 or 483. The remaining samples of 393 or 483 are derived using bilinear interpolation with contributions from either the top reference sample 822 and the filtered left reference sample 824, or the left reference sample 820 and the filtered top sample 822 (as shown in FIG. 8).

[0119] FIG. 9A shows an example of a CTU 900. The CTU 900 has a region where MIP mode memory access bandwidth is applied. Matrix intra prediction is applied within a VPDU-sized region, i.e., a 64×64 quadrant of a 128×128 CPU 900. For example, the 512 luma sample region to which the budget is applied includes (i) a 32×16 region 912, (ii) a 16×32 region 914, and (iii) a 64×8 region 916. Region 912 is further decomposed into various CUs, i.e., 912a-912h in FIG. 9B. In the example of FIG. 9B, CU 912a has a size of 16×4 and therefore belongs to set C, and CU 912d has a size of 8×8 and therefore belongs to set B, as do CUs 912b, 912c, and 912e (all of size 8×4). CU912f and CU912g are each 4x4 and belong to set A. As will be explained with reference to Figures 11 to 16, the area of ​​a region of 512 luma samples can be used in the described configurations as a threshold for determining whether to apply a constraint to a MIP mode. Because sets A to C above typically result from a division of a region, using the area of ​​a region of 512 samples is appropriate as a constraint. However, the threshold may relate to a different area, as will be explained below.

[0120] Figure 10 shows a coding tree 10 corresponding to the example CTU 900 of Figure 9A. Region 912, as well as regions 914 and 916, correspond to nodes in the coding tree where budgets for the contained CUs are established for reading MIP matrix coefficients. The decomposition of a region into one or more CUs is not shown for simplicity. For each CU configured to use MIP mode, a budget according to the CU size for mapping to one of sets A-C is deducted from the budget of the region (i.e., 912, 914, 916), as described below with reference to Figures 11-16.

[0121] 11 illustrates a method 1100 for encoding coding units of an image frame into a video bitstream 115. The method 1100 may be implemented by a device such as an configured FPGA, ASIC, or ASSP. Furthermore, the method 1100 may be executed by the video encoder 114 under execution of the processor 205. Accordingly, the method 1100 may be stored on a computer-readable storage medium and / or the memory 206. The operation of the method 1100 for encoding a series of coding units, or image frames, into the bitstream 115 imposes a limit on the use of matrix intra prediction. Therefore, the operation of the method 1100 imposes a limit on the memory bandwidth required for matrix intra prediction compared to the worst-case potential memory bandwidth without the limit. The method 1100 begins with step 1110, which divides a frame into CTUs.

[0122] In step 1110 of dividing a frame into CTUs, the block divider 310, under the execution of the processor 205, divides the current frame of frame data 113 into a matrix of CTUs. The encoding process of the resulting CTUs begins. Control in the processor passes from step 1110 to step 1120 of determining the code tree.

[0123] In a determine coding tree step 1120, under the execution of processor 205, video encoder 114 tests various partitioning options as described with reference to Figures 5-7, and performs the operations of determine coding unit step 1130 to arrive at a coding tree for the CTU. The operations of step 1120 are described with reference to Figure 12. Control in processor 205 passes from step 1120 to determine coding unit prediction mode step 1130.

[0124] In a coding unit determination step 1130, the video encoder 114, under execution of the processor 205, determines a prediction mode to use in encoding the selected coding unit into the bitstream 115. The coding unit may be selected according to a scan pattern. The operation of step 1130 is further described with reference to FIG. 13. Once a prediction mode for the coding unit has been selected, control in the processor 205 passes from step 1130 to a coding unit encoding step 1140. In selecting a prediction mode for a coding unit, a particular combination of partitions to arrive at a given coding unit is also selected, thereby determining the coding tree, since the coding unit itself results from a hierarchy of partitions of the coding tree.

[0125] In encoding coding units step 1140, entropy encoder 338, under execution of processor 205, encodes the coding units determined in step 1130 into bitstream 115. The determined coding trees are efficiently encoded into bitstream 115 in step 1140 by entropy encoder 338, under execution of processor 205, using "split flags" and other syntax elements to indicate the selected split, as shown in Figures 5 and 6.

[0126] The operations at step 1140 are further described with reference to Figure 14. Control in processor 205 passes from step 1140 to a final coding unit test step 1150.

[0127] In a last coding unit test step 1150, processor 205 tests whether the current coding unit is the last coding unit in the coding tree of step 1120. If the current coding unit is the last one in the coding tree of step 1120 (“YES” in step 1150), control in processor 205 passes to a last CTU test step 1160. If the current coding unit is not the last one in the coding tree of step 1120 (“NO” in step 1150), the next coding unit in the coding tree of step 1120 is selected using the scan pattern for determination and encoding, and control in processor 205 passes to step 1130. Thus, step 1130 is performed for each CU resulting from the coding tree determined in step 1120.

[0128] In last CTU test step 1160, processor 205 tests whether the current CTU is the last CTU in the slice or frame. If not ("NO" at step 1160), video encoder 114 proceeds to the next CTU in the frame, and control in processor 205 returns from step 1160 to step 1120 to continue processing the remaining CTUs in the frame. If the CTU is the last one in the frame or slice, step 1160 returns "YES" and method 1100 ends. The operation of method 1100 results in the entire image frame being encoded into a bitstream as a series of CTUs.

[0129] Method 1100 is performed for each image frame in a video sequence. Method 1100 can determine CUs for each CTU. In other words, the CUs for a CTU are first determined in one pass or pipeline stage, followed by a second stage of encoding into a bitstream. Method 1100 may also determine CUs more finely, for example, for each VPDU, since memory consumption is reduced due to the smaller area of ​​a VPDU compared to a CTU.

[0130] 12 illustrates a method 1200 for determining a coding tree for a CTU, performed in step 1120. The method 1200 receives a CTU, generates candidate partitions and candidate coding units for evaluation, and ultimately selects the partitions and coding units to be encoded into the bitstream 115. In particular, the method 1200 establishes a MIP mode memory bandwidth budget at a particular node or region in the coding tree, which limits the application of MIP mode in the coding unit under the node, i.e., in the spatial domain. The method 1200 may be embodied by a device such as a configured FPGA, ASIC, or ASSP. Furthermore, the method 1200 may be executed by the video encoder 114 under the execution of the processor 205. Accordingly, the method 1200 may be stored in a computer-readable storage medium and / or the memory 206. The method 1200 is performed for each node (region) in the candidate coding tree. The method 1200 begins with a region area test step 1210.

[0131] In region test step 1210, the video encoder 114, under the execution of the processor 205, tests the area occupied by a candidate node of the coding tree according to available partitioning options (using the partitioning options described in connection with FIG. 5). In the described configuration, the region relates to a node of the coding tree where further partitioning can occur to form a CU before partitioning stops. The region does not have to correspond to the immediate parent of the current coding unit. That is, the parent region for which the MIP mode memory access bandwidth budget is set can be multiple nodes above the current node in the coding tree. For example, a region of 512 luma samples can be partitioned into many smaller coding units, e.g., 4x4, 4x8, 8x4, etc. As the encoder search proceeds, various candidate nodes resulting from the candidate partitions are tested. Within each candidate node, various prediction modes for the resulting coding units are tested. If the area of ​​luma samples occupied by the region does not meet the threshold, the region area test evaluates to true, and control in the processor 205 passes from step 1210 to budget setting step 1220. If the area of ​​luma samples occupied by the region meets the threshold, the region area test evaluates to false and control in processor 205 passes from step 1210 to segment generation step 1230. In one embodiment, as shown in the examples of Figures 11 to 16, the threshold is 512 samples and is met when a region has an area greater than 512 samples. Other thresholds can also be used, as described below. The threshold is typically predetermined based on the required performance of encoder 114 and decoder 134.

[0132] In a budget setting step 1220, the video encoder 114, under the execution of the processor 205, sets a budget for a region corresponding to the area of ​​the current node of the coding tree. This budget is available to all sub-nodes (sub-regions) within the current region that result from dividing the current region into smaller regions. For all regions of 512 luma samples or less, the budget for applying MIP mode can be taken into account when evaluating coding unit prediction modes. An area of ​​512 luma samples may be subject to trisection, for example, resulting in coding units of areas 128, 256, and 128 luma samples. The dimensions of a coding unit of area 128 luma samples may be 8x16 or 16x8, which may reach worst-case memory bandwidth limitations and result in the inability to use MIP mode for other coding units within the region, including the possibility of further subdivision of the region. If an area of ​​512 luma samples is bisected, for example, into two 16x16 coding units of 256 luma sample area, the worst-case memory bandwidth limit is not reached and there is no restriction on the use of MIP mode for the resulting CUs. However, further dividing each of the 256 sample area regions into smaller CUs, i.e., CUs of size 16x8, 8x16, 8x8, 8x4, 4x8, and 4x4, may restrict the use of MIP mode for later resulting CUs due to the earlier resulting CUs consuming their budget.

[0133] Because 4x4 is the smallest luma coding unit size, and all other coding units are integer multiples of this size, and therefore all regions are multiples of this size, the budget can be expressed as a maximum word read per 4x4 block area. The budget is 40 word reads for a 4x4 block, which corresponds to (512÷(4x4)*40=1280 word reads) as the maximum number of read operations allowed in a 512 luma sample area for a luma sample area of ​​512. Each time MIP mode is used for a CU, the required number of reads is subtracted from the budget; if the budget becomes insufficient for further use of MIP mode, no more CUs can use MIP mode, as will be described in connection with Figures 13 and 14. In this way, CUs of larger sizes can use MIP mode; for example, a single CU of area 512 samples or a pair of CUs of area 256 samples can use MIP mode without being restricted by the constraint. If the coding tree decomposes a sample of area 512 into many smaller CUs, the consumed budget may prohibit further CUs in the region from using MIP mode. For a 512-sample area with a budget of 40 words per 4x4, the total budget for the region is 512 / (4x4)*40=1280 words. If the 512-sample area is partitioned into four CUs of size 8x16 or 16x8, using MIP mode on two CUs consumes 576 words per CU, or 1152 words. The remaining budget of 128 read words is insufficient to allow the other two CUs in the region to use MIP mode. Control in processor 205 passes from step 1220 to a partition generation step 1230.

[0134] In partition generation step 1230, the block partitioner 310, under the execution of the processor 205, generates a set of candidate partitions for the current node in the coding tree. The partitions are as shown in FIG. 5, and the associated syntax elements are shown in FIG. 6 and illustrated in FIG. 7. In the case of "no partition," i.e., generating 510, a coding unit is generated later. In the case of other partition types, i.e., generating 512-520, additional nodes in the coding tree are generated according to each partition. Method 1200 is repeated to generate coding units for the additional nodes if "no partition" generation is performed later when method 1200 is performed. Method 1200 is repeated for each node in the CTU in turn. Thus, all partitions of the CTU are determined. To iteratively generate partitions within a partition, all possible coding units in a given CTU are searched within the constraints of minimum CU size and depth recursion constraints, which may limit the number of recursions using bipartitions, tripartitions, and quadtree partitions. Control in the processor 205 passes from step 1230 to CU generation step 1240.

[0135] In CU generation step 1240, the block partition unit 310, under the execution of the processor 205, generates a candidate CU for each case in which "No Split" was generated in the partition generation step 1230. In step 1240, the prediction mode of the candidate CU has not yet been determined, and a final mode has not yet been selected, but the prediction mode of the preceding CU in the Z-order scan is known. Therefore, adjacent (in terms of position within the CU) reference samples can be used for intra prediction from neighboring CUs from the same partitioning operation as the partition containing the current CU, from neighboring regions resulting from different parent regions, or from an entirely different CTU. Step 1240 effectively partitions the coding tree region into coding blocks containing prediction blocks as described in connection with FIG. 3. Method 1200 ends in step 1240, and control in the processor 205 returns to method 1100, where the prediction mode of the generated CU is determined.

[0136] FIG. 13 illustrates a method 1300 for determining a coding unit performed in step 1130. Method 1300 includes determining a prediction mode for a coding unit generated by performing method 1200. The prediction modes include intra prediction, inter prediction, and MIP mode depending on a budget for using MIP mode within a region containing the current coding unit. The budget established by method 1200 is used to control whether to test MIP mode for the current CU. A portion of the budget is consumed (as described above in connection with cases A-C) each time a CU within the region reaches the worst-case block size, depending on the number of codewords in the CU. Once the budget is consumed, no further coding units within the region that could potentially use MIP mode are searched for. That is, method 1300 performs MIP mode testing only if a previous performance of method 1300 has not exhausted the configured memory access bandwidth budget that applies to a common parent node of the coding tree.

[0137] The region does not have to correspond to the direct parent of the current coding unit; i.e., the parent region for which the MIP mode memory access bandwidth budget is set may be multiple nodes above the current node in the coding tree. For example, a region of 512 luma samples may be divided into multiple smaller coding units, e.g., 4x4, 4x8, 8x4, etc. For each CU, MIP mode usage is limited by the remaining budget for the region of 512 luma samples. Method 1300 may be implemented by the mode selector 386 or in part by module 3960. Method 1300 may be embodied by a device such as a configured FPGA, ASIC, or ASSP. Furthermore, method 1300 may be performed by the video encoder 114 under execution of the processor 205. Thus, method 1300 may be stored on a computer-readable storage medium and / or memory 206. Method 1300 begins with a step 1310 of testing intra-prediction mode.

[0138] In intra-prediction mode test step 1310, under the execution of processor 205, video encoder 114 tests the "normal" intra-prediction modes, i.e., DC, planar, and angular intra-prediction modes, that may be used in encoding the current coding unit. A Lagrangian optimization is typically performed to select the optimal intra-prediction mode from among the intra-prediction modes available to the CU. The application of secondary transforms, i.e., 330 and 344, is also tested, as are different types of primary transforms (DCT-2, DCT-8, DST-7), including the transform skip case. Control in processor 205 passes from step 1310 to inter-prediction mode test step 1320.

[0139] In an inter prediction mode test step 1320, under the execution of the processor 205, the video encoder 114 tests various motion vectors for generating an inter predicted PU. When evaluating the use of inter prediction, a motion vector is selected from a set of candidate motion vectors. The candidate motion vectors are generated according to a search pattern. When testing the distortion of the reference block obtained for the candidate motion vector, the application of chroma splitting prohibited in the coding tree is taken into account. If splitting is prohibited for chroma but allowed for luma, the resulting luma CB may use inter prediction. Since motion compensation is applied only to the luma channel, the distortion calculation takes into account luma distortion but not chroma distortion. If chroma splitting is prohibited, chroma distortion is not taken into account because motion compensation is not performed on the chroma channel. For chroma, the distortion caused by the considered intra prediction mode and the coded chroma TB (if any) is taken into account. When both luma and chroma are considered, the inter-prediction search may first select a motion vector based on luma distortion and then "fine-tune" the motion vector by also considering chroma distortion. The fine-tuning typically considers small variations in motion vector values, such as subpixel displacements. The specific motion vector may be generated by a "merge mode," whereby the motion vector for the current CU is derived from motion vectors from neighboring CUs. Merge mode is more easily expressed in the bitstream syntax compared to other motion vectors, which may require signaling of a "motion vector delta" applied in conjunction with a selected "motion vector predictor." The motion vector predictor is typically derived from a spatially or temporally neighboring CU. For intra-coded slices, e.g., the first frame of a sequence of frames, inter prediction is not available, so step 1320 is not performed. Control in processor 205 passes from step 1320 to within-budget test step 1330.

[0140] In within-budget test step 1330, under execution of processor 205, video encoder 114 tests whether a MIP mode memory access bandwidth budget is applicable to the current CU. The test performed in 1330 determines whether the current CU is contained within the 512 luma sample region for which a MIP mode budget was established in step 1220. If the current CU is larger than 512 luma sample regions, no budget constraints apply. Therefore, the CU is not subject to any further constraints on the use of MIP mode, and control in processor 205 passes to MIP mode test step 1340 (1330 "true"). If the size of the current CU is equal to or smaller than 512 luma samples, the budget required to use MIP mode on the current CU is compared with the remaining budget for that region, as described with reference to FIG. 8. If there is not enough budget to apply MIP mode to the current CU, control in processor 205 passes from step 1330 to mode selection step 1350 (1330 "false"). If there is sufficient budget to apply MIP mode to the current CU (“true” in step 1340 ), control in processor 205 passes from step 1330 to step 1340 .

[0141] In a MIP mode testing step 1340, the mode selector 386 tests various MIP modes to determine the best MIP mode to use to predict the current CU from those available for the size of the CU. As with the testing of intra-prediction modes in step 1310, a Lagrangian optimization may be performed to counteract the coding costs of the tested MIP modes and their associated distortion to the residual. Control in the processor 205 passes from step 1340 to a mode selection step 1350.

[0142] In a mode selection step 1350, mode selector 386, under execution of processor 205, selects a final mode for the CU from the candidates obtained in steps 1310, 1320, and 1350. Method 1300 ends with control in processor 205 returning to method 1100.

[0143] 14 shows a method 1400 for encoding coding units of a coding tree of a CTU into a video bitstream 115, as performed in step 1140. The method 1400 may be embodied by an apparatus such as an configured FPGA, ASIC, ASSP, etc. Furthermore, the method 1400 may be performed by a video encoder 114, such as the entropy encoder 338, under execution by the processor 205. Thus, the method 1400 may be stored in a computer-readable storage medium and / or the memory 206. The method 1400 begins with a prediction mode encoding step 1410.

[0144] In a prediction mode encoding step 1410, the entropy encoder 338, under the execution of the processor 205, encodes a flag using context coding bins indicating the use of either intra-prediction (including both normal intra-prediction mode use or MIP mode use) or inter-prediction, as determined in step 1350 and indicated by the prediction mode 387. The encoding of the flag in step 1410 does not distinguish between normal intra-prediction and MIP mode. The distinction between normal intra-prediction and MIP intra-prediction (if applicable) is encoded in a MIP mode flag encoding step 1430. Control in the processor 205 passes from step 1410 to a budget test step 1420.

[0145] In budget test step 1420, under execution of processor 205, video encoder 114 tests whether a MIP mode memory access bandwidth budget is applicable to the current CU. This test determines whether the current CU is contained within a 512 luma sample region for which a MIP mode budget was established in step 1220. If the current CU is contained within a region larger than 512 luma samples, then no budget constraints apply and the CU is not further restricted in its use of MIP mode. Control in processor 205 passes to MIP mode flag encoding step 1430 (1420 "TRUE").

[0146] If the current CU is within a region of 512 luma samples or less in size, the budget required to use MIP mode for the current CU is compared with the remaining budget for that region, as described with reference to Figure 8. If there is not enough budget to apply MIP mode to the current CU, control in processor 205 proceeds from step 1420 to step 1440 to encode the TB ("FALS" at step 1420). However, if there is enough budget to apply MIP mode to the current CU ("TRUE" at 1420), control in processor 205 proceeds from step 1420 to step 1430, even if the current CU is within a region of 512 samples or less. The operations of step 1420 correspond to the operations of step 1330; therefore, only MIP flags are encoded in method 1400 for CUs for which MIP mode was found in method 1300.

[0147] In MIP mode flag encoding step 1430, the entropy encoder 338, under the execution of the processor 205, encodes context-encoded bins into the bitstream 115 indicating the selection of MIP mode or not, as determined in step 1350. The context used to encode the bins is described with reference to step 1420. If MIP mode is selected, the entropy encoder 338 also encodes the selection of which particular MIP mode was used for encoding into the bitstream 338. The MIP mode may be encoded using a shortened binary codeword instead of using a selection between the "most likely mode" and the remaining modes. Using a shortened binary codeword eliminates the need to derive a list of most likely modes, which involves a potential table lookup in which the most likely mode is derived from adjacent angular intra-predicted CUs. Control in the processor 205 passes from step 1430 to TB encoding step 1440.

[0148] In a TB encoding step 1440, the entropy encoder 338, under the execution of the processor 205, encodes the residual coefficients of the TB associated with the current CU into a bitstream. Generally, the flag for each TB indicates the presence of at least one significant coefficient, and the coefficients are encoded one by one according to a scan pattern that starts from the last significant coefficient position and returns to the DC (top left) coefficient position. This ends the method 1400, and control in the processor 205 returns to the method 1100.

[0149] 15 shows a method 1500 for decoding coding units and transform blocks of an image frame from a video bitstream 133. The method 1500 may be embodied by a device such as an configured FPGA, ASIC, or ASSP. Furthermore, the method 1500 may be performed by the video decoder 134 under execution of the processor 205. Thus, the method 1500 may be stored in a computer-readable storage medium and / or memory 206. The method 1500 begins with step 1510 of dividing a frame into CTUs.

[0150] In a divide frame into CTUs step 1510, video decoder 134, under the execution of processor 205, divides the current frame of frame data 133 (to be decoded) into a matrix of CTUs. Decoding of the resulting CTUs begins. Control in processor 205 passes from step 1510 to a decoding coding unit step 1520.

[0151] In a coding unit decoding step 1520, the entropy decoder 420, under execution of the processor 205, decodes split flags from the bitstream 133 according to the coding tree, as described with reference to Figures 5-7. By decoding the split flags, step 1520 can perform operations to determine the size and position of CUs within the CTU, i.e., according to the coding tree of the CTU. Progression of the method 1300 involves iterating to step 1520, thereby traversing the entire coding tree of the CTU and decoding each CU. Control in the processor 205 passes from step 1520 to a coding unit decoding step 1530.

[0152] In a decoding coding unit step 1530, the entropy decoder 420, under execution of the processor 205, decodes the coding unit from the bitstream 133. Step 1530 invokes a method 1600, described below in connection with Figure 16, to decode the CU. Control in the processor 205 passes from step 1530 to a test last coding unit step 1540.

[0153] In last coding unit test step 1540, processor 205 tests whether the current coding unit is the last coding unit of the CTU, as determined from decoding the split flag in step 1520. If the current coding unit is the last coding unit of the CTU (“YES” in step 1540), control in the processor passes to last CTU test step 1550. If the current coding unit is not the last coding unit in the coding tree of step 1520 (“NO” in step 1540), the next coding unit in the coding tree of step 1520 is selected for decoding, and control in processor 205 passes to step 1520.

[0154] In last CTU test step 1550, processor 205 tests whether the current CTU is the last CTU in the slice or frame. If the current CTU is not the last CU ("NO" at step 1550), video decoder 134 proceeds to the next CTU in the frame or slice, and control in processor 205 returns from step 1550 to step 1520 to continue processing the remaining CTUs in the frame. If the CTU is the last CTU in the frame or slice, step 1550 returns "YES" and method 1500 ends. Method 1500 results in the entire image frame being decoded from the bitstream as a series of CTUs.

[0155] 16 illustrates a method 1600 for decoding coding units from video bitstream 133, performed in step 1530. Method 1600 may be embodied by an apparatus such as an configured FPGA, ASIC, or ASSP. Furthermore, method 1600 may be performed by video decoder 134 under execution of processor 205. Thus, method 1600 may be stored on a computer-readable storage medium and / or in memory 206. Method 1600 begins with step 1602 of decoding a pred_mode flag.

[0156] In a pred_mode flag decoding step 1602, the entropy decoder 420 decodes the context-coded bins to determine whether the current coding unit uses inter-prediction or intra-prediction (including MIP mode). Control in the processor 205 continues from step 1602 to an inter-prediction test step 1604. In step 1604, it is determined from the pred_mode flag whether inter-prediction was used. If the current coding unit uses inter-prediction (inter-prediction test step 1604 is "TRUE"), control in the processor 205 continues from step 1602 to an inter-prediction execution step 1606. In step 1606, inter-prediction is performed, and the resulting reference block (434) is fetched and filtered to generate a PU, followed by decoding the TB for each color channel and adding the TB (1690) to the PU to decode the CU. Control in the processor 205 continues from step 1606 to a TB addition step 1690.

[0157] If the current coding unit uses intra prediction or MIP mode (step 1604: FALSE), control in processor 205 continues to MIP mode budget test step 1610. In MIP mode budget test step 1610, it is determined whether MIP mode is available for the current CU depending on the budget. The budget corresponds to the budget described with reference to steps 1330 or 1420. In the described example, if the region containing the current CU is larger than the area threshold (512 luma samples), there is no applicable budget constraint and MIP mode can be used (step 1610: TRUE). Control in processor 205 passes from step 1610 to MIP mode flag decoding step 1620.

[0158] At step 1610, if the area of ​​the region containing the current CU is less than or equal to the threshold (512 luma samples), the budget needed to use MIP mode for the current CU is compared to the remaining budget for that region, as described with reference to Figure 8. If there is enough budget to apply MIP mode to the current CU, MIP mode is available ("TRUE" at step 1610) and method 1600 proceeds to step 1620. If there is insufficient budget, MIP mode is not available for the current CU ("FALSE" at MIP mode budget test step 1610). Control in processor 205 then proceeds to intra-prediction mode decoding step 1612.

[0159] In step 1612, starting with decoding the intra prediction mode, normal intra prediction (DC, planar, or angular) is used. Control in processor 205 passes from step 1612 to perform intra prediction step 1614. Then, in step 1614, intra prediction is performed. Control in processor 205 passes from step 1614 to add TB step 1690.

[0160] In performing step 1620, the MIP flag is decoded by the entropy decoder 420 using a single context-coded bin. Because a bin uses one context, there is no need to select a context depending, for example, on the use of MIP modes in neighboring CUs, or on block size or other parameters that may be used to select one context from multiple possible contexts for decoding a bin. Control in the processor passes from step 1620 to step 1630, which tests whether MIP mode has been selected.

[0161] In step 1630, it is determined whether MIP mode is selected. If the decoded MIP mode flag indicates that MIP mode is not being used ("FALSE" in step 1630 of testing whether MIP mode is selected), control proceeds to step 1612 to decode the block using one of the regular intra-prediction modes. If the decoded MIP mode flag indicates that MIP mode is being used for the CU ("TRUE" in step 1630), control in processor 205 proceeds to MIP mode decoding step 1640.

[0162] The MIP modes are decoded from the bitstream 133 in performing step 1640. Control in the processor 205 passes from step 1640 to a read matrix coefficients step 1650. The decoded MIP modes are used in step 1650 to read a set of matrix coefficients 481 from the matrix coefficient memory 486. By limiting the use of the MIP modes in step 1610 according to the budgeted usage, the maximum memory bandwidth consumption required for the matrix coefficient memory 486 to supply the matrix coefficients 481 is reduced.

[0163] From step 1650, method 1600 transitions to a neighboring sample filtering step 1660. The neighboring reference samples 464 are filtered in step 1660. From step 1660, method 1600 transitions to a matrix multiplication step 1670. Step 1600 can be performed by module 482, for example, using matrix coefficients 481 and samples 464. The filtered reference samples and matrix coefficients 481 are multiplied in step 1670, similar to example 828 of FIG. 8 .

[0164] From step 1670, method 1600 transitions to a PB interpolation step 1680. The sparse block (i.e., 830) determined by performing step 1670 is used to fill the PB (i.e., 483) by an interpolation process in step 1680. Method 1600 transitions from the PB interpolation step of step 1680 to a TB addition step 1690. The decoded residual is used to generate a TB, which is added to the PB 483 in step 1690 to decode the CU. The coding unit is decoded using the prediction block generated in step 1690. Method 1600 ends upon completion of step 1690.

[0165] In performing method 1600, video decoder 134 achieves support for matrix intra prediction with a constraint on worst-case memory bandwidth for retrieving matrix coefficients. The constraints described herein do not overly restrict the selection of MIP modes in terms of impacting error compared to when usage restrictions are not enabled. The non-overly restrictive selection behavior is based on a determined statistical likelihood that typically no more than 20% of CUs coded in MIP mode are adjacent to a CU to the left or above that also uses MIP mode. Boundaries between regions larger than the region used as the granularity for establishing memory access budgets, i.e., boundaries between 512 luma samples and CTUs included in the 20% found measurement.

[0166] In the embodiments of Figures 11 to 16, whether the MIP mode is used is based on a constraint. The constraint is implemented, for example, in Figure 12, based on the area of ​​the region of the current CU and whether the matrix intra prediction flag is coded or decoded, which depends on whether the area of ​​the region of the current CU meets a threshold area. In the described implementation, as described in connection with steps 1330, 1420, and 1610, whether the matrix intra prediction flag is coded depends on the area of ​​the region if the region meets the threshold, and (ii) the allocation amount of the region if the area of ​​the region does not meet the threshold. Therefore, whether the matrix intra prediction flag is coded (or decoded) depends at least on whether the area of ​​the region meets the threshold area.

[0167] 11 through 14, the MIP mode flag is encoded into bitstream 115 only if the budget test is satisfied, as described in connection with steps 1420 and 1430. Correspondingly, when decoding the bitstream in Figure 16, the MIP mode flag is decoded only if the MIP mode budget test step returns TRUE. In other words, the matrix intra prediction flag of a CU is decoded only if matrix intra prediction is used.

[0168] As described with reference to Figures 12 and 13, in another configuration of video encoder 114 and video decoder 134, the search for MIP mode is limited. However, in this configuration, MIP flag signaling is included in the bitstream of each CU regardless of the budget consumption status of each CU. That is, the matrix intra prediction flag of a CU is decoded regardless of whether matrix intra prediction is used. In this configuration, as indicated by dashed arrow 1411 in Figure 14, step 1420 is omitted, and control proceeds to step 1430 in video encoder 114 ("TRUE" at step 1420). In this configuration, step 1430 includes the MIP mode flag in all cases, but can be set to zero if MIP mode was not selected at step 1350. Similarly, in video decoder 134, step 1610 is omitted, and control proceeds to step 1620 (effectively, step 1610 always returns "TRUE"). The MIP mode flag is decoded in step 1620 and is zero if MIP mode was not selected in bitstream encoding step 1350. This avoids the additional burden on entropy encoder 338 and entropy decoder 420 of decoding whether to encode a MIP flag for each CU. Even though the bitstream includes a MIP mode flag for each CU, the memory bandwidth for reading matrix coefficients 481 from coefficient memory 486 of video decoder 134 is still limited by lookups such as those performed in methods 1200 and 1300, thereby reducing the provision of resources required to handle this (limited) worst-case use of MIP mode.

[0169] The implementations described above in connection with Figures 12 through 17 apply restrictions on the use of MIP mode based on the area of ​​the region containing the CU. In yet another configuration of video encoder 114 and video decoder 134, the constraint implemented in Figure 12 prohibits the use of matrix intra prediction for block sizes corresponding to worst-case memory bandwidth, rather than based on the area of ​​the region. Effectively, matrix intra prediction is used (and the matrix intra prediction flag is coded or decoded) based on the size of each coding block. Block sizes of 4x4, 4x8, 8x4, 8x8, 8x16, and 16x8 result in a maximum access density (e.g., measured for 4x4 luma sample blocks) of 80 words per 4x4 block. Of the previous set of block sizes, certain block sizes result in a worst-case 72 words per 4x4 block, which for resource provisioning purposes can be treated in the same category as the 80 words per 4x4 block case. Forbidding the worst case block size results in a worst case of 40 words per 4x4 block (for a given block size, the worst case is 36 words per 4x4 block).

[0170] 17 shows a method 1700 for determining the coding unit performed in step 1130. Method 1300 provides an alternative to method 1300 in which the restriction is implemented with respect to prohibiting the use of matrix intra prediction for block sizes corresponding to the worst-case memory bandwidth. Method 1700 includes determining a prediction mode for the coding unit generated by performing method 1200. Prediction modes include intra prediction, inter prediction, and MIP mode if MIP mode is available in the region containing the current coding unit. Method 1700 begins with step 1710 of testing the intra prediction mode.

[0171] In intra-prediction mode test step 1310, video encoder 114, under the execution of processor 205, tests the "normal" intra-prediction modes, i.e., DC, planar, and angular intra-prediction modes, for potential use in encoding the current coding unit, operating in the manner described for step 1310. Control in processor 205 passes from step 1710 to inter-prediction mode test step 1720.

[0172] In inter prediction mode test step 1720, video encoder 114, under the execution of processor 205, tests various motion vectors for generating an inter predicted PU. When evaluating the use of inter prediction, a motion vector is selected from a set of motion vector candidates. Step 1720 operates in the same manner as step 1320. Control in processor 205 passes from step 1720 to MIP tolerance size test step 1730.

[0173] In MIP allowed size test step 1730, video encoder 114, under execution of processor 205, tests whether the CU is of a size for which MIP mode is allowed. MIP mode is allowed for all sizes except for the worst-case sizes of 4x4, 4x8, 8x4, 8x8, 8x16, and 16x8 in some implementations. If the current CU is of an allowed size (“TRUE” at 1730), control in processor 205 passes from step 1730 to MIP mode test step 1740. If the current CU is of a prohibited size (“FALSE” at step 1730), control in processor 205 passes from step 1730 to mode selection step 1750.

[0174] In MIP mode test step 1740, mode selector 386 tests various MIP modes to determine the best MIP mode, available for the size of the CU, to use to predict the current CU. Step 1740 operates similarly to step 1340. Processing in processor 205 proceeds from step 1740 to mode selection step 1750.

[0175] In a mode selection step 1750, the mode selector 386, under the execution of the processor 205, selects a final mode for the CU from the candidates obtained in steps 1710, 1720, and 1750. The method 1700 ends when control in the processor 205 returns to the method 1100.

[0176] 18 shows a method 1800 for encoding coding units of a coding tree of a CTU into a video bitstream 115 in an implementation as performed in step 1140, in which a restriction related to prohibiting the use of matrix intra prediction for block sizes corresponding to worst-case memory bandwidth is present. The method 1800 may be embodied by an apparatus such as an configured FPGA, ASIC, or ASSP. Furthermore, the method 1800 may be executed by the video encoder 114 under the execution of the processor 205. Thus, the method 1800 may be stored in a computer-readable storage medium and / or the memory 206. The method 1800 begins with a prediction mode encoding step 1810.

[0177] In a prediction mode encoding step 1810, the entropy encoder 338, under the execution of the processor 205, encodes a flag using a context coding bin that indicates the use of either intra-prediction (including both the use of normal intra-prediction mode or the use of MIP mode), or intra-prediction, as determined in step 1750. Control in the processor 205 passes from step 1810 to a MIP tolerance size test step 1820.

[0178] In MIP allowed size test step 1820, video encoder 114, under execution of processor 205, tests whether the current CU is of an allowed size. The allowed and prohibited sizes are the same as in step 1730 of method 1700. If the current CU is of an allowed size, control in processor 205 passes to MIP mode flag encoding step 1830 (1820 "TRUE").

[0179] If the current CU is not of an acceptable size (“FALSE” in step 1820), control in processor 205 passes from step 1820 to step 1840 for encoding the TB.

[0180] In a MIP mode flag encoding step 1830, entropy encoder 338, under execution of processor 205, encodes into bitstream 115 a context coding bin that indicates whether MIP mode was selected or not, as determined in step 1350. Step 1830 operates in the same manner as step 1430. Control in processor 205 passes from step 1830 to a TB encoding step 1840.

[0181] In a TB encoding step 1840, the entropy encoder 338, under the execution of the processor 205, encodes the residual coefficients of the TB associated with the current CU into a bitstream. Step 1840 operates in the same manner as step 1440. Then, the method 1800 ends, and control in the processor 205 returns to the method 1100.

[0182] 19 shows a method 1900 for decoding a coding unit from a video bitstream 133 in an implementation where a constraint is performed in step 1530 related to prohibiting MIP mode for worst-case blocks. The method 1900 may be embodied by a device such as an configured FPGA, ASIC, or ASSP. Furthermore, the method 1900 may be performed by the video decoder 134 under execution of the processor 205. Thus, the method 1900 may be stored in a computer-readable storage medium and / or memory 206. The method 1900 begins with step 1902 of decoding a pred_mode flag.

[0183] In a pred_mode flag decoding step 1902, the entropy decoder 420 decodes the context-coded bins to determine whether the current coding unit uses inter-prediction or intra-prediction (including MIP mode). If the current coding unit uses inter-prediction (inter-prediction test step 1904 is "TRUE"), control in the processor passes to an inter-prediction execution step 1906. In step 1606, inter-prediction is performed, resulting in a reference block (434) being fetched and filtered to generate a PU, followed by decoding the TB for each color channel and adding the TB (1690) to the PU to decode the CU. Similarly, control in the processor 205 passes from step 1906 to a TB addition step 1990.

[0184] If the current coding unit uses intra prediction or MIP mode (“FALSE” at step 1904), control in processor 205 passes to MIP allowed size test step 1910. In MIP allowed size test step 1910, it is determined whether MIP mode is allowed for the current CU size. The MIP allowed size is as described with reference to steps 1730 or 1820. If the current CU is of an allowed size (“TRUE” at step 1910), control in processor 205 passes from step 1910 to MIP mode flag decoding step 1920. If, at step 1910, the current CU is not of an allowed size (“FALSE” at step 1910), control in processor 205 transitions to intra prediction mode decoding step 1912.

[0185] In step 1912, the normal intra prediction (DC, planar or angular) is decoded and the method 1900 proceeds to a decode intra prediction step 1914. Steps 1912 and 1914 operate as described in connection with steps 1612 and 1614. Control in the processor 205 continues from step 1914 to an add TB step 1990.

[0186] The MIP flag is decoded during step 1920. Control in processor 205 transfers from step 1920 to step 1930, which tests whether MIP mode has been selected.

[0187] Step 1930 operates to determine whether MIP mode is selected. If the decoded MIP mode flag indicates that MIP mode is not being used ("FALSE" at step 1930 of testing whether MIP mode is selected), control proceeds to step 1912 to decode the block using one of the regular intra-prediction modes. If the decoded MIP mode flag indicates that MIP mode is being used for the CU ("TRUE" at step 1930), control in the processor proceeds to MIP mode decoding step 1940.

[0188] The MIP mode is decoded from the bitstream 133 in performing step 1940. Control in the processor 205 transitions from step 1940 to a read matrix coefficients step 1950. The decoded MIP mode is used in step 1950 to read a set of matrix coefficients 481 from the matrix coefficient memory 486. Matrix coefficients are read for each prediction block for which it was determined that MIP mode should be used, as in step 1650.

[0189] From step 1950, method 1900 transitions to a neighboring sample filtering step 1660. The neighboring reference samples 464 are filtered in step 1960. From step 1960, method 1900 proceeds to a matrix multiplication step 1970. The filtered reference samples and matrix coefficients 481 are multiplied in step 1970, similar to example 828 of FIG.

[0190] From step 1970, method 1900 transitions to a PB interpolation step 1980. The sparse block (i.e., 830) determined by performing step 1970 is used to interpolate PB (i.e., 483) through an interpolation process in step 1980. From PB interpolation step 1980, method 1900 transitions to a TB addition step 1990. The decoded residual is used to generate a TB, which is added to PB 483 in step 1990, similar to step 1690, to decode the CU. Method 1900 ends upon completion of step 1990.

[0191] The variations described for methods 1300, 1400, and 1600 can also be applied to methods 1700, 1800, and 1900, respectively.

[0192] When block size is used as the criterion for MIP mode availability, as in method 1700, there is no need to establish and update memory budgets, i.e., steps 1210 and 1220 are omitted. Eliminating MIP modes for specific block sizes is easier to implement than determining memory access budgets, but the absence of MIP modes from many common block sizes reduces compression performance. The size of coefficient memory (i.e., 392, 486) is reduced because the absence of 4x4 blocks eliminates the need to store "set A" matrix coefficients (18 sets of matrix coefficients and bias values). Eliminating MIP modes from 4x4, 4x8, 8x4, 8x8, 8x16, and 16x8 ("small blocks") has the advantage that the relatively complex MIP mode feedback loop does not need to support these small blocks.

[0193] In yet another configuration, the use of MIP mode is prohibited for a subset of small blocks (as listed above). For example, MIP mode is prohibited only for 4x4 blocks in steps 1730, 1820, and 1910, while being allowed for all other block sizes; alternatively, MIP mode is prohibited for 4x4, 4x8, and 8x4 blocks, while being allowed for all other block sizes. That is, matrix intra prediction is not used, and in some implementations, the matrix intra prediction flag is decoded if the coding unit size is one of the prohibited sizes. While worst-case memory bandwidth is not reduced compared to completely eliminating the restriction on MIP mode use, the exclusion of these very small block sizes reduces the stringency of the intra reconstruction feedback loop. Removing 4x4 also eliminates the need to store matrix coefficients associated with "Set A" in coefficient memory (i.e., 392, 486). Alternatively, the set of prohibited block sizes may be 4x4, 4x8, 8x4, and 8x8, in which case set A and set B do not exist in video encoder 114 or video decoder 134. Removing set A and set B reduces compression performance but memory consumption because matrix coefficients associated with set A and set B are unnecessary.

[0194] In yet another configuration, the memory budget may be even lower than 40 words per 4x4 luma sample area, for example, 20 or 10 words per 4x4 luma sample area. The budget is set at larger region sizes, such as nodes corresponding to region sizes of 1024 or 2048 luma samples. As with the configurations described above, later CUs within the limit are limited in their availability of memory budget depending on the MIP mode usage of earlier CUs within those regions. Further reductions in memory bandwidth can be achieved at the expense of reduced compression efficiency.

[0195] In yet another configuration, memory budgets are established at steps 1220, 1330, 1420, and 1610 with a region size of 64 luma samples to apply to CUs of sizes 4x4, 4x8, and 8x4 (the "small CU memory budget"). A separate memory budget is established with a region size of 512, but applies only to CUs of sizes greater than 8x8, particularly 8x16 and 16x8 (the "larger CU memory budget"). Both budgets are set at 40 words per 4x4 luma sample area. Because the small CU memory budget applies only to CUs contained within a 64 luma sample area, while the larger CU memory budget applies only to CUs greater than 64 luma samples, the budgets form additional budgets for the total matrix memory bandwidth.

[0196] While the configurations disclosed herein describe memory bandwidth in terms of words per 4x4 memory region, it is understood that memory accesses will likely group words in some SIMD fashion to allow matrix coefficients to be read without requiring excessively high clock frequencies for the associated memory. However, such wider memories are themselves costly, and matrix coefficients may be shared with other data in the same memory, resulting in contention for the reduced accesses due to the limitations of using MIP mode.

[0197] If we restrict the use of MIP mode to limit the worst-case memory bandwidth per region, it may result in a bias in which earlier CUs in each region use MIP mode and later CUs in that region cannot use MIP mode because the available allocation in that region is consumed by earlier CUs. This bias does not usually occur because the distribution of CUs that select MIP mode is generally somewhat sparse.

[0198] The described configuration has restrictions on when MIP mode can be used, thereby reducing computational complexity compared to allowing MIP mode to run without restrictions. The MIP mode selection statistics for an unrestricted search would not normally trigger the restrictions imposed on MIP mode selection in mode selector 386, thereby reducing complexity without a proportional loss in coding efficiency. Thus, worst-case memory bandwidth is reduced without a corresponding loss in coding efficiency. Industrial Applicability

[0199] The described arrangements are applicable to the computer and data processing industry, in particular digital signal processing for encoding and decoding of signals such as video and image signals, to achieve high compression efficiency.

[0200] The arrangements described herein allow for the use of trellis-based state machines for residual encoding and decoding, which update according to coefficient parity and select context and coefficient quantizers. The described arrangements allow for the implementation of trellis-based state machines without imposing excessive delays due to the sequential nature of state updates.

[0201] The above describes only some embodiments of the present invention, which may be modified and / or altered without departing from the scope and spirit of the present invention, and the embodiments are illustrative and not limiting.

Claims

1. 1. A method for generating prediction samples for a current block in a coding tree unit for an image frame, comprising: decoding a matrix intra prediction flag for the current block, the matrix intra prediction flag indicating whether matrix intra prediction is used for the current block; decoding a matrix intra-prediction mode for the current block if the matrix intra-prediction flag indicates that matrix intra-prediction is used for the current block; selecting a matrix for matrix intra prediction for the current block according to the matrix intra prediction mode; generating the predicted samples by applying a multiplication of input samples based on samples neighboring the current block and the matrix selected according to the matrix intra prediction mode; the samples adjacent to the current block include a sample adjacent to the left of the current block and a sample adjacent to the top of the current block; A shortened binary code may be used for the matrix intra prediction mode; The image frames may have a 4:2:0 chroma format; When an area in the coding tree unit is divided into four blocks, each having a size of 16×8, and the current block is one of the four blocks, decoding of the matrix intra prediction flag for the current block depends on a usage status of matrix intra prediction for two blocks adjacent to the current block; The two blocks adjacent to the target block are the left neighbor block and the upper neighbor block of the target block.

2. 2. The method of claim 1, wherein decoding of the matrix intra prediction mode for the current block depends on the width of the current block and the height of the current block.

3. A plurality of block sizes are available, and the plurality of block sizes are classified as first level, second level, or third level; the first level includes at least a 4x4 size; 2. The method of claim 1, wherein the number of matrices available for matrix intra prediction for the first level is greater than the number of matrices available for matrix intra prediction for the second level.

4. 4. The method of claim 3, wherein the number of matrices available for matrix intra prediction for the first level is greater than the number of matrices available for matrix intra prediction for the third level.

5. 4. The method of claim 3, wherein the block size included in the second level is greater than 4x4 size.

6. 5. The method of claim 4, wherein the block size included in the third level is greater than 4x4 size.

7. 2. The method of claim 1, further comprising: determining a plurality of blocks in the coding tree unit, including the current block, by decoding a split flag for a block split in the coding tree unit, wherein a horizontal triangulation can be used as the block split.

8. 2. The method of claim 1, wherein the decoding of the matrix intra prediction flag depends on the width of the current block and the height of the current block.

9. 2. The method of claim 1, wherein, when the size of the current block is one of an 8x16 size and a 16x8 size, the matrix intra prediction mode is one of all matrix intra prediction modes available for the 8x16 size and the 16x8 size, and a shortened binary code is used for the matrix intra prediction mode regardless of which matrix intra prediction mode is used.

10. 2. The method of claim 1, wherein, when the size of the current block is one of 8x16 and 16x8, (a) the matrix intra prediction mode is one of a plurality of matrix intra prediction modes available for the 8x16 and 16x8 sizes, (b) the number of a plurality of matrix intra prediction modes available for the 8x16 and 16x8 sizes is n, where n is an integer value greater than 0, (c) the matrix intra prediction mode is represented by an integer value greater than or equal to 0 and less than or equal to n-1, (d) an m-bit code (m is an integer value greater than 0) is used for the matrix intra prediction mode represented by an integer value greater than or equal to 0 and less than or equal to a predetermined integer value, and (e) an m+1-bit code is used for the matrix intra prediction mode represented by an integer value greater than the predetermined integer value and less than or equal to n-1.

11. 11. The method of claim 10, wherein n is an integer value greater than 2 to the mth power and less than 2 to the m+1th power.

12. 8. The method according to claim 7, wherein the block division can be a vertical division into thirds.

13. 1. A method for generating prediction samples for a current block in a coding tree unit for an image frame, comprising: selecting a matrix for matrix intra prediction for the current block; generating the predicted samples by applying multiplication of the selected matrix with input samples based on samples neighboring the current block; the samples adjacent to the current block include a sample adjacent to the left of the current block and a sample adjacent to the top of the current block; encoding a matrix intra prediction flag indicating whether matrix intra prediction is used for the current block; and if matrix intra prediction is used for the current block, encoding a matrix intra prediction mode used to identify the selected matrix; A shortened binary code may be used for the matrix intra prediction mode; The image frames may have a 4:2:0 chroma format; When an area in the coding tree unit is divided into four blocks each having a size of 16×8, and the current block is one of the four blocks, encoding of the matrix intra prediction flag for the current block depends on a usage status of matrix intra prediction for two blocks adjacent to the current block; The two blocks adjacent to the target block are the left neighbor block and the upper neighbor block of the target block.

14. The method of claim 13 , wherein the encoding of the matrix intra prediction mode for the current block depends on the width of the current block and the height of the current block.

15. A plurality of block sizes are available, and the plurality of block sizes are classified as first level, second level, or third level; the first level includes at least a 4x4 size; 14. The method of claim 13, wherein the number of matrices available for matrix intra prediction for the first level is greater than the number of matrices available for matrix intra prediction for the second level.

16. 16. The method of claim 15, wherein the number of matrices available for matrix intra prediction for the first level is greater than the number of matrices available for matrix intra prediction for the third level.

17. 16. The method of claim 15, wherein the block size included in the second level is greater than 4x4 size.

18. 16. The method of claim 15, wherein the block size included in the third level is greater than 4x4 size.

19. 14. The method of claim 13, further comprising: determining a plurality of blocks including the current block in the coding tree unit by block division, wherein the block division can be a horizontal division into three.

20. The method of claim 13 , wherein the encoding of the matrix intra prediction flag depends on the width of the current block and the height of the current block.

21. 14. The method of claim 13, wherein, when the size of the current block is one of an 8x16 size and a 16x8 size, the matrix intra prediction mode is one of all matrix intra prediction modes available for the 8x16 size and the 16x8 size, and a shortened binary code is used for the matrix intra prediction mode regardless of which matrix intra prediction mode is used.

22. 14. The method of claim 13, wherein, when the size of the current block is one of 8x16 and 16x8, (a) the matrix intra prediction mode is one of a plurality of matrix intra prediction modes available for the 8x16 and 16x8 sizes, (b) the number of a plurality of matrix intra prediction modes available for the 8x16 and 16x8 sizes is n, where n is an integer value greater than 0, (c) the matrix intra prediction mode is represented by an integer value greater than or equal to 0 and less than or equal to n-1, (d) an m-bit code (m is an integer value greater than 0) is used for the matrix intra prediction mode represented by an integer value greater than or equal to 0 and less than or equal to a predetermined integer value, and (e) an m+1-bit code is used for the matrix intra prediction mode represented by an integer value greater than the predetermined integer value and less than or equal to n-1.

23. 23. The method of claim 22, wherein n is an integer value greater than 2 to the mth power and less than 2 to the m+1th power.

24. 20. The method of claim 19, wherein the block division can be a vertical division into thirds.

25. 1. An apparatus for generating prediction samples for a current block in a coding tree unit for an image frame, comprising: means for decoding a matrix intra prediction flag for the current block, the matrix intra prediction flag indicating whether matrix intra prediction is used for the current block; means for decoding a matrix intra-prediction mode for the current block if the matrix intra-prediction flag indicates that matrix intra-prediction is used for the current block; means for selecting a matrix for matrix intra prediction for the current block according to the matrix intra prediction mode; means for generating the predicted samples by applying multiplication of input samples based on samples neighboring the current block and the matrix selected according to the matrix intra prediction mode; the samples adjacent to the current block include a sample adjacent to the left of the current block and a sample adjacent to the top of the current block; A shortened binary code may be used for the matrix intra prediction mode; The image frames may have a 4:2:0 chroma format; When an area in the coding tree unit is divided into four blocks each having a size of 16×8, and the current block is one of the four blocks, decoding of the matrix intra prediction flag for the current block depends on the usage status of matrix intra prediction of two blocks adjacent to the current block; The two blocks adjacent to the target block are a block adjacent to the left and a block adjacent above the target block.

26. 1. An apparatus for generating prediction samples for a current block in a coding tree unit for an image frame, comprising: means for selecting a matrix for matrix intra prediction for the current block; means for generating the predicted samples by applying multiplication of the selected matrix with input samples based on samples adjacent to the current block; the samples adjacent to the current block include a sample adjacent to the left of the current block and a sample adjacent to the top of the current block; means for encoding a matrix intra prediction flag indicating whether matrix intra prediction is used for the current block; means for encoding a matrix intra-prediction mode used to identify the selected matrix if matrix intra-prediction is used for the current block; A shortened binary code may be used for the matrix intra prediction mode; The image frames may have a 4:2:0 chroma format; When an area in the coding tree unit is divided into four blocks each having a size of 16×8, and the current block is one of the four blocks, encoding of the matrix intra prediction flag for the current block depends on a usage status of matrix intra prediction for two blocks adjacent to the current block; The two blocks adjacent to the target block are a block adjacent to the left and a block adjacent above the target block.

27. A computer program product for causing a computer to carry out the method of claim 1.

28. A computer program product for causing a computer to carry out the method according to claim 13.

Images