Method and device for generating prediction sample, storage medium and computer program product

Abstract

The invention discloses a method and device for generating prediction samples, a storage medium and a computer program product. A method of decoding a coding unit of a coding tree of an image frame from a video bitstream is also disclosed. The method comprises: splitting a region of a coding tree into a plurality of coding blocks, each coding block comprising a prediction block; and determining a matrix intra prediction flag for the prediction block of each coded block, each matrix intra prediction flag indicating whether to use matrix intra prediction for the prediction block of one of the coded blocks, the determination being based on an area of the region if the region satisfies a threshold, or based on a budget of the region if the area of the region does not satisfy the threshold. The method further includes reading matrix coefficients from a memory for each prediction block determined to use matrix intra prediction according to the determined flag; and decoding the coding units using prediction blocks generated for each coding unit in the region using the reference samples and the matrix coefficients of each prediction block.

Description

[0001] (This application is a divisional application of an application with a filing date of April 15, 2020, application number 2020800422492, and invention name “Method, device and system for encoding and decoding blocks of video samples”.) Technical Field

[0002] The present invention generally relates to digital video signal processing, and more particularly to methods, devices and systems for encoding and decoding blocks of video samples. The present invention also relates to a computer program product comprising a computer readable medium having recorded thereon a computer program for encoding and decoding blocks of video samples. Background Art

[0003] There are currently many applications for video encoding, including applications for transmitting and storing video data. Many video encoding standards have also been developed and other video encoding standards are currently under development. Recent advances in video coding standardization have led to the formation of a group known as the "Joint Video Experts Group" (JVET). The Joint Video Experts Group (JVET) includes members of Study Group 16, Question 6 (SG16 / Q6) of the Telecommunication Standardization Sector (ITU-T) of the International Telecommunication Union (ITU), also known as the "Video Coding Experts Group" (VCEG); and members of the International Organization for Standardization / International Electrotechnical Commission Joint Technical Committee 1 / Group Association 29 / Working Group 11 (ISO / IEC JTC1 / SC29 / WG11), also known as the "Moving Picture Experts Group" (MPEG).

[0004] The Joint Video Experts Team (JVET) issued a call for proposals (CfP) and analyzed the responses at its 10th meeting in San Diego, USA. The responses submitted showed that the video compression capabilities are significantly better than those of the current state-of-the-art video compression standard, namely "High Efficiency Video Coding" (HEVC). Based on this excellent performance, it was decided to start a project to develop a new video compression standard named "Versatile Video Coding" (VVC). It is expected that VVC will address the continued demand for even higher compression performance, especially as the capabilities of video formats increase (e.g., with higher resolutions and higher frame rates), and the growing market demand for service provision over WANs, where bandwidth costs are relatively high. At the same time, VVC must be implementable in contemporary silicon processes and provide an acceptable trade-off between the achieved performance and the implementation cost (e.g., in terms of silicon area, CPU processor load, memory utilization, and bandwidth).

[0005] Video data includes a sequence of frames of image data that each includes one or more color channels. Typically, a primary color channel and two secondary color channels are required. The primary color channel is commonly referred to as the "brightness" channel, and (one or more) secondary color channels are commonly referred to as the "chrominance" channel. Although video data is usually displayed in an RGB (red-green-blue) color space, the color space has a high degree of correlation between the three corresponding components. The video data representation seen by an encoder or decoder usually uses a color space such as YCbCr. YCbCr concentrates luminance (mapped to "brightness" according to the transformation equation) in the Y (primary) channel, and concentrates chrominance in the Cb and Cr (secondary) channels. In addition, the Cb and Cr channels can be spatially sampled at a lower rate than the brightness channel (e.g., half in the horizontal direction and half in the vertical direction (referred to as a "4:2:0 chroma format")). The 4:2:0 chroma format is commonly used in "consumer" applications, such as Internet video streaming, broadcast television, and storage on Blu-ray TM discs. Subsampling the Cb and Cr channels at half rate horizontally, but not vertically, is referred to as a "4:2:2 chroma format." The 4:2:2 chroma format is commonly used in professional applications, including the capture of footage for filmmaking, etc. The higher sampling rate of the 4:2:2 chroma format makes the resulting video more resilient to editing operations, such as color grading, etc. Before distribution to consumers, 4:2:2 chroma format material is often converted to a 4:2:0 chroma format and then encoded for distribution to consumers. In addition to the chroma format, video is also characterized by resolution and frame rate. Example resolutions are ultra-high definition (UD) with a resolution of 3840×2160 or "8K" with a resolution of 7680×4320, and example frame rates are 60 Hz or 120 Hz. Luma sample rates can range from approximately 500 megasamples / second to several gigasamples / second. For the 4:2:0 chroma format, the sampling rate of each chroma channel is one quarter of the luma sampling rate, and for the 4:2:2 chroma format, the sampling rate of each chroma channel is half of the luma sampling rate.

[0006] The VVC standard is a "block-based" codec in which a frame is first partitioned into an array of square areas called "coding tree units" (CTUs). A CTU typically occupies a relatively large area, such as 128×128 luma samples. However, the areas of the CTUs at the right and bottom edges of each frame may be smaller. Associated with each CTU is a "coding tree" for the luma channel and an additional coding tree for the chroma channel. The coding tree defines the decomposition of the area of ​​the CTU into a set of areas, also called "coding blocks" (CBs). Blocks for both luma and chroma channels can also be specified for a single coding tree, in which case a set of juxtaposed coding blocks is called a "coding unit" (CU), i.e., each CU has a coding block for each color channel. CBs are processed in a specific order for encoding or decoding. As a result of using the 4:2:0 chroma format, a CTU of a luma coding tree including a 128×128 luma sample area has a corresponding chroma coding tree of a 64×64 chroma sample area juxtaposed with a 128×128 luma sample area. When a single coding tree is used for the luma and chroma channels, a set of collocated blocks for a given region is often referred to as a "unit", such as the above-mentioned CU and "prediction unit" (PU) and "transform unit" (TU). When a separate coding tree is used for a given region, the above-mentioned CB and "prediction block" (PB) and "transform block" (TB) will be used.

[0007] Despite the above distinction between a “unit” and a “block,” the term “block” may be used as a general term for an area or region of a frame to which an operation is applied to all color channels.

[0008] For each CU, a prediction unit (PU) ("prediction unit") is generated for the content (sample values) of the corresponding region of the frame data. In addition, a representation of the difference (or "residual" in the spatial domain) between the prediction and the region content seen at the input of the encoder is formed. The difference of each color channel can be transformed and encoded as a sequence of residual coefficients, thereby forming one or more TUs for a given CU. The applied transform can be a discrete cosine transform (DCT) or other transform applied to each block of residual values. The transform is applied separately, that is, a two-dimensional transform is performed in two passes. The block is first transformed by applying a one-dimensional transform to each row of samples in the block. Then, the partial result is transformed by applying a one-dimensional transform to each column of the partial result to produce a final block of transform coefficients that essentially decorrelate the residual samples. The VVC standard supports transforms of various sizes, including transforms of rectangular blocks (each side size is a power of 2). The transform coefficients are quantized for entropy encoding in the bitstream.

[0009] VVC is characterized by intra-frame prediction and inter-frame prediction. Intra-frame prediction involves using previously processed samples in the frame being used to generate a prediction of the current block of samples in the frame. Inter-frame prediction involves using a block of samples obtained from a previously decoded frame to generate a prediction of the current block of samples in the frame. The previously decoded frame is offset from the spatial position of the current block according to a motion vector, which usually has filtering applied. The intra-frame prediction block can be a uniform sample value ("DC intra-frame prediction"), a plane with an offset and horizontal and vertical gradients ("plane intra-frame prediction"), or a group of blocks with neighboring samples applied in a specific direction ("angle intra-frame prediction") or the result of a matrix multiplication using neighboring samples and selected matrix coefficients. Since the neighboring samples include samples from the previously processed block, the feedback loop for intra-frame prediction is quite restrictive and the computational complexity needs to be kept below the level required to meet the highest supported resolution and frame rate. Summary of the invention

[0010] It is an object of the present invention to substantially overcome or at least ameliorate one or more disadvantages of existing arrangements.

[0011] One aspect of the present disclosure provides a method for decoding a coding unit of a coding tree of an image frame from a video bitstream, the method comprising: splitting a region of the coding tree into a plurality of coding blocks, each of the coding blocks comprising a prediction block; determining a matrix intra prediction flag of a prediction block of each of the coding blocks in the coding block, each matrix intra prediction flag indicating whether a prediction block for one of the coding blocks uses matrix intra prediction, the determination being (i) based on an area of ​​the region when the region satisfies a threshold, or (ii) based on a budget of the region when the area of ​​the region does not satisfy the threshold; reading a matrix coefficient from a memory for each prediction block determined to use matrix intra prediction according to the determined flag; and decoding the coding unit using the prediction block, the prediction block being generated for each coding unit in the region using reference samples of each prediction block and the matrix coefficient.

[0012] In another aspect, the threshold is a size greater than 512 luma samples.

[0013] In another aspect, the threshold is a size greater than 64 luma samples.

[0014] On the other hand, the budget enables reading of 40 words of a 4x4 block for the region.

[0015] On the other hand, the matrix intra prediction flag is decoded for the CU only if matrix intra prediction is used.

[0016] In another aspect, regardless of whether matrix intra prediction is used, a matrix intra prediction flag is decoded for the CU.

[0017] Another aspect of the present disclosure provides a method for decoding a coding unit of a coding tree of an image frame from a video bitstream, the method comprising: splitting a region of the coding tree into a plurality of coding blocks, each coding block comprising a prediction block; determining a matrix intra-prediction flag of a prediction block of the coding block based on a size of each coding block, each matrix intra-prediction flag indicating whether matrix intra-prediction is used for a prediction block of a corresponding coding block; reading matrix coefficients from a memory for each prediction block determined to use matrix intra-prediction according to the determined flag; and decoding the coding unit from the prediction block, the prediction block being a prediction block of each coding unit in the region generated using reference samples of each prediction block and the matrix coefficients.

[0018] On the other hand, when the size of the coding unit is not 4×4, the matrix intra prediction flag is decoded.

[0019] On the other hand, in case the size of the coding unit is not one of 4×4, 8×4 and 4×8, the matrix intra prediction flag is decoded.

[0020] On the other hand, in case the size of the coding unit is not one of 4×4, 8×4, 4×8 and 8×8, the matrix intra prediction flag is decoded.

[0021] On the other hand, in case the size of the coding unit is not one of 4×4, 8×4, 4×8, 8×8, 8×16 and 16×8, the matrix intra prediction flag is decoded.

[0022] Another aspect of the present disclosure provides a method for generating a prediction block of a coding tree of an image frame from a video bitstream, the method comprising: determining a prediction mode of a coding unit by decoding a matrix intra-frame prediction mode flag from the video bitstream; when the prediction mode indicates the use of the matrix intra-frame prediction mode, decoding a truncated binary codeword to determine the matrix intra-frame prediction mode; and generating the prediction block by applying matrix multiplication to reference samples adjacent to the prediction block and a matrix selected according to the decoded matrix intra-frame prediction mode.

[0023] Another aspect of the present disclosure provides a non-transitory computer-readable medium having a computer program stored thereon for implementing a method for decoding a coding unit of a coding tree of an image frame from a video bitstream, the method comprising: splitting a region of the coding tree into a plurality of coding blocks, each of the coding blocks comprising a prediction block; determining a matrix intra-prediction flag of a prediction block of each of the coding blocks in the coding blocks, each matrix intra-prediction flag indicating whether a prediction block for one of the coding blocks uses matrix intra-prediction, the determination being (i) based on an area of ​​the region when the region satisfies a threshold, or (ii) based on a budget of the region when the area of ​​the region does not satisfy the threshold; reading matrix coefficients from a memory for each prediction block determined to use matrix intra-prediction according to the determined flag; and decoding the coding unit using the prediction block, the prediction block being generated for each coding unit in the region using reference samples of each prediction block and the matrix coefficients.

[0024] Another aspect of the present disclosure provides a video decoder, which is configured to: receive coding units of a coding tree of an image frame from a video bitstream; split a region of the coding tree into multiple coding blocks, each of the coding blocks including a prediction block; determine a matrix intra-frame prediction flag of a prediction block of each of the coding blocks in the coding block, each matrix intra-frame prediction flag indicating whether a prediction block for one of the coding blocks uses matrix intra-frame prediction, the determination being (i) based on the area of ​​the region when the region satisfies a threshold, or (ii) based on a budget of the region when the area of ​​the region does not satisfy the threshold; for each prediction block determined to use matrix intra-frame prediction according to the determined flag, read matrix coefficients from a memory; and decode the coding unit using the prediction block, the prediction block being generated for each coding unit in the region using reference samples of each prediction block and the matrix coefficients.

[0025] Another aspect of the present disclosure provides a system, comprising: a memory; and a processor, wherein the processor is configured to execute code stored on the memory to implement a method for decoding a coding unit of a coding tree of an image frame from a video bitstream, the method comprising: splitting a region of the coding tree into a plurality of coding blocks, each of the coding blocks comprising a prediction block; determining a matrix intra prediction flag of a prediction block of each of the coding blocks in the coding blocks, each matrix intra prediction flag indicating whether a prediction block for one of the coding blocks uses matrix intra prediction, the determination being (i) based on an area of ​​the region when the region satisfies a threshold, or (ii) based on a budget of the region when the area of ​​the region does not satisfy the threshold; reading matrix coefficients from the memory for each prediction block determined to use matrix intra prediction according to the determined flag; and decoding the coding unit using the prediction block, the prediction block being generated for each coding unit in the region using reference samples of each prediction block and the matrix coefficients.

[0026] Other aspects are also disclosed. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] At least one embodiment of the present invention will now be described with reference to the following drawings and appendices, in which:

[0028] Figure 1 is a schematic block diagram illustrating a video encoding and decoding system;

[0029] Figure 2A and 2B Composition can be practiced Figure 1 A schematic block diagram of a general computer system for one or both of the video encoding and decoding systems;

[0030] Figure 3 is a schematic block diagram showing the functional modules of a video encoder;

[0031] Figure 4 is a schematic block diagram showing the functional modules of a video decoder;

[0032] Figure 5 is a schematic block diagram illustrating possible partitioning of a block into one or more blocks in a tree structure for general video coding;

[0033] Figure 6 A schematic diagram of a data flow for implementing a permissible partitioning of a block into one or more blocks in a tree structure of a general video coding;

[0034] Fig. 7A and 7B shows an example partitioning of a coding tree unit (CTU) into multiple coding units (CUs);

[0035] Figure 8 An example of the operation of a matrix intra prediction module is shown;

[0036] Fig.9A and 9B An example CTU is shown with a region where memory access bandwidth for matrix intra prediction mode is applied;

[0037] Fig.10 Shown with Fig.9A and 9B The encoding tree corresponding to the example of

[0038] Fig.11 A method for encoding coding units of an image frame into a video bitstream is shown, the coding units comprising coding units using matrix intra prediction;

[0039] Fig.12 Shown as Fig.11 Method for determining a budget for memory accesses for matrix intra prediction across a plurality of coding units of a coding tree unit for use in;

[0040] Fig.13 Shown as Fig.11 A method for determining a coding unit for encoding a portion of an image frame in a video bitstream for use in;

[0041] Fig.14 Shown as Fig.11 A method for encoding coding units of an image frame into a video bitstream for use in;

[0042] Fig.15 A method for decoding a coding tree of an image frame from a video bitstream is shown;

[0043] Fig.16 Shown as Fig.15 A method for decoding coding units of image frames from a video bitstream for use in;

[0044] Fig.17 Shown as Fig.11 An alternative method for determining a coding unit for encoding a portion of an image frame into a video bitstream for use in;

[0045] Fig.18 Shown as Fig.11 An alternative method for encoding coding units of an image frame into a video bitstream for use in; and

[0046] Fig.19 Shown as Fig.15 An alternative method used in for decoding coding units of image frames from a video bitstream. DETAILED DESCRIPTION

[0047] Where steps and / or features having the same reference numerals are referenced in any one or more of the figures, such steps and / or features have the same function(s) or operation(s) for purposes of this specification, unless otherwise intended.

[0048] As mentioned above, the computational complexity of intra prediction is restrictive, especially for matrix intra prediction (MIP). Although MIP can provide an effective solution in terms of minimizing errors, the computational complexity of applying MIP particularly affects the block processing rate under the worst case (e.g., frames consisting of only 4×4 blocks). The block processing rate needs to be sufficient to support the resolution and frame rate of the target application. Considering only the luma channel, an "8K" resolution frame (7680×4320) at 120 frames per second requires processing 248.8×10 blocks per second. 6 4×4 blocks. Even if the worst case is not achieved in the entire frame or video sequence, it is necessary to handle the localized area where the worst case is achieved without delaying the delivery of the fully decoded frame for presentation on the display. One aspect of complexity relates to the memory bandwidth required to extract the matrix coefficients selected according to the matrix intra prediction (MIP) mode, which may vary from block to block without constraints.

[0049] Figure 1 is a schematic block diagram illustrating functional modules of video encoding and decoding system 100. System 100 may utilize constraints on application of MIP mode to establish a worst-case memory bandwidth for selecting or reading matrix coefficients to allow practical implementation and / or commensurate with the encoding advantages achieved by MIP mode.

[0050] The system 100 includes a source device 110 and a destination device 130. A communication channel 120 is used to communicate encoded video information from the source device 110 to the destination device 130. In some configurations, one or both of the source device 110 and the destination device 130 may each include a mobile phone handset or "smart phone", in which case the communication channel 120 is a wireless channel. In other configurations, the source device 110 and the destination device 130 may include video conferencing equipment, in which case the communication channel 120 is typically a wired channel such as an Internet connection. In addition, the source device 110 and the destination device 130 may include any of a wide range of devices, including devices that support over-the-air television broadcasts, cable television applications, Internet video applications (including streaming), and applications that capture encoded video data on some computer-readable storage medium (such as a hard drive in a file server).

[0051] like Figure 1As shown, source device 110 includes video source 112, video encoder 114, and transmitter 116. Video source 112 typically includes a source of captured video frame data (denoted as 113), such as a camera sensor, a previously captured video sequence stored on a non-transitory recording medium, or a video feed from a remote camera sensor. Video source 112 may also be the output of a computer graphics card (e.g., display operating system and video output of various applications executed on a computing device (e.g., tablet computer)). Examples of source device 110 that may include a camera sensor as video source 112 include smart phones, video camcorders, professional video cameras, and web video cameras.

[0052] The video encoder 114 converts (or "encodes") the captured frame data (indicated by arrow 113) from the video source 112 into a bitstream (indicated by arrow 115). The bitstream 115 is transmitted by a transmitter 116 via a communication channel 120 as encoded video data (or "encoded video information"). The bitstream 115 may also be stored in a non-transitory storage device 122, such as a "flash" memory or hard drive, until or in lieu of subsequent transmission via the communication channel 120. For example, the encoded video data may be supplied to a customer via a wide area network (WAN) for video streaming applications when desired.

[0053] The destination device 130 includes a receiver 132, a video decoder 134, and a display device 136. The receiver 132 receives the encoded video data from the communication channel 120 and passes the received video data to the video decoder 134 as a bit stream (indicated by arrow 133). The video decoder 134 then outputs the decoded frame data (indicated by arrow 135) to the display device 136. The decoded frame data 135 has the same chroma format as the frame data 113. Examples of the display device 136 include a cathode ray tube, a liquid crystal display (such as in a smart phone, a tablet computer, a computer monitor, or a stand-alone television). The functions of each of the source device 110 and the destination device 130 may also be embodied in a single device, examples of which include a mobile phone and a tablet computer.

[0054] Although example devices are described above, source device 110 and destination device 130 may each typically be configured within a general purpose computer system via a combination of hardware and software components. Figure 2ASuch a computer system 200 is shown, which includes: a computer module 201; input devices such as a keyboard 202, a mouse pointer device 203, a scanner 226, a camera 227 that can be configured as a video source 112, and a microphone 280; and output devices including a printer 215, a display device 214 that can be configured as a display device 136, and a speaker 217. The computer module 201 can use an external modulator-demodulator (modem) transceiver device 216 to communicate with a communication network 220 via a connection 221. The communication network 220, which can represent a communication channel 120, can be a WAN, such as the Internet, a cellular telecommunications network, or a private WAN. In the case where the connection 221 is a telephone line, the modem 216 can be a traditional "dial-up" modem. Alternatively, in the case where the connection 221 is a high-capacity (e.g., cable or optical) connection, the modem 216 can be a broadband modem. Wireless modems can also be used to make wireless connections to the communication network 220. The transceiver device 216 may provide the functionality of the transmitter 116 and the receiver 132 , and the communication channel 120 may be embodied in the wiring 221 .

[0055] The computer module 201 typically includes at least one processor unit 205 and a memory unit 206. For example, the memory unit 206 may have a semiconductor random access memory (RAM) and a semiconductor read-only memory (ROM). The computer module 201 also includes a plurality of input / output (I / O) interfaces, wherein the plurality of input / output (I / O) interfaces include: an audio-video interface 207 connected to a video display 214, a speaker 217, and a microphone 280; an I / O interface 213 connected to a keyboard 202, a mouse 203, a scanner 226, a camera 227, and an optional joystick or other human-machine interface device (not shown); and an interface 208 for an external modem 216 and a printer 215. The signal from the audio-video interface 207 to the computer monitor 214 is typically the output of a computer graphics card. In some implementations, the modem 216 may be built into the computer module 201, such as built into the interface 208. The computer module 201 also has a local network interface 211, which allows the computer system 200 to be connected to a local area communication network 222, known as a local area network (LAN), via a connection 223. Figure 2AAs shown, the local communication network 222 can also be connected to the wide area network 220 via a connection 224, wherein the local communication network 222 typically includes a so-called "firewall" device or a device with similar functions. The local network interface 211 can include an Ethernet (EthernetTM) circuit card, a Bluetooth (BluetoothTM) wireless configuration, or an IEEE 802.11 wireless configuration; however, for the interface 211, a variety of other types of interfaces can be implemented. The local network interface 211 can also provide the functions of the transmitter 116 and the receiver 132, and the communication channel 120 can also be embodied in the local communication network 222.

[0056] I / O interfaces 208 and 213 may provide either or both of a serial connection and a parallel connection, wherein the former is typically implemented according to the Universal Serial Bus (USB) standard and has a corresponding USB connector (not shown). A storage device 209 is provided, and the storage device 209 typically includes a hard disk drive (HDD) 210. Other storage devices (not shown) such as floppy disk drives and tape drives may also be used. An optical disk drive 212 is typically provided to serve as a non-volatile source of data. Portable memory devices such as optical disks (e.g., CD-ROMs, DVDs, Blu-ray Discs), USB-RAMs, portable external hard disk drives, and floppy disks may be used as appropriate sources of data for the computer system 200. Typically, any of the HDD 210, optical disk drive 212, networks 220 and 222 may also be configured to operate as a video source 112, or as a destination for stored decoded video data to be reproduced via a display 214. The source device 110 and the destination device 130 of the system 100 may be embodied in the computer system 200.

[0057] The components 205-213 of the computer module 201 typically communicate via an interconnect bus 204 and in a manner that results in conventional modes of operation of the computer system 200 known to those skilled in the relevant art. For example, the processor 205 is connected to the system bus 204 using a connection 218. Likewise, the memory 206 and the optical drive 212 are connected to the system bus 204 via a connection 219. Examples of computers on which the described configuration may be practiced include IBM-PC and compatible machines, Sun SPARCstations, Apple Mac™, or similar computer systems.

[0058] Where appropriate or desired, the video encoder 114 and the video decoder 134 and the methods described below may be implemented using the computer system 200. In particular, the video encoder 114, the video decoder 134 and the methods to be described may be implemented as one or more software applications 233 executable within the computer system 200. In particular, instructions 231 (see FIG. 1 ) in the software 233 executed within the computer system 200 may be used to implement the video encoder 114 and the video decoder 134 and the methods to be described below. Figure 2B ) to implement the video encoder 114, the video decoder 134 and the steps of the method. The software instructions 231 may be formed into one or more code modules each for performing one or more specific tasks. The software may also be split into two separate parts, with a first part and corresponding code modules performing the method, and a second part and corresponding code modules managing a user interface between the first part and a user.

[0059] For example, the software may be stored in a computer readable medium including a storage device as described below. The software is loaded from the computer readable medium into the computer system 200 and then executed by the computer system 200. A computer readable medium with such software or a computer program recorded on the computer readable medium is a computer program product. The use of the computer program product in the computer system 200 preferably implements an advantageous device for implementing the video encoder 114, the video decoder 134 and the method.

[0060] The software 233 is typically stored in the HDD 210 or the memory 206. The software is loaded into the computer system 200 from a computer readable medium and executed by the computer system 200. Thus, for example, the software 233 may be stored on an optically readable disk storage medium (e.g., a CD-ROM) 225 read by the optical drive 212.

[0061] In some examples, the application 233 is provided to the user in a manner encoded on one or more CD-ROMs 225 and read via corresponding drives 212, or alternatively, the application 233 can be read by the user from the network 220 or 222. Further, the software can also be loaded into the computer system 200 from other computer-readable media. Computer-readable storage media refers to any non-transitory tangible storage medium that provides recorded instructions and / or data to the computer system 200 for execution and / or processing. Examples of such storage media include floppy disks, magnetic tapes, CD-ROMs, DVDs, Blu-ray Discs, hard drives, ROMs or integrated circuits, USB memories, magneto-optical disks, or computer-readable cards such as PCMCIA cards, etc., regardless of whether these devices are inside or outside the computer module 201. Examples of temporary or non-tangible computer-readable transmission media that may also be involved in providing software, applications, instructions and / or video data or encoded video data to the computer module 401 include: radio or infrared transmission channels and network connections to other computers or networked devices, and the Internet or Intranet including e-mail transmissions and information recorded on websites.

[0062] The second portion of the above-mentioned application 233 and the corresponding code modules can be executed to implement one or more graphical user interfaces (GUIs) to be drawn or otherwise presented on the display 214. Users and applications of the computer system 200 can operate the interface in a functionally applicable manner by typically operating the keyboard 202 and the mouse 203 to provide control commands and / or input to the applications associated with these (one or more) GUIs. Other forms of user interfaces that are functionally applicable can also be implemented, such as audio interfaces that utilize voice prompts output via the speaker 217 and user voice commands input via the microphone 280, etc.

[0063] Figure 2B is a detailed schematic block diagram of the processor 205 and the "memory" 234. The memory 234 represents Figure 2A A logical aggregation of all memory modules (including HDD 209 and semiconductor memory 206) that can be accessed by computer module 201 in.

[0064] When the computer module 201 is initially powered on, a power-on self-test (POST) program 250 is executed. The POST program 250 is usually stored in Figure 2A249 of semiconductor memory 206. Hardware devices such as ROM 249 storing software are sometimes referred to as firmware. POST program 250 checks the hardware within computer module 201 to ensure proper operation, and typically checks processor 205, memory 234 (209, 206), and basic input-output system software (BIOS) module 251, which is also typically stored in ROM 249, for correct operation. Once POST program 250 runs successfully, BIOS 251 starts Figure 2A The hard disk drive 210 is started so that the boot loader 252 residing on the hard disk drive 210 is executed via the processor 205. This loads the operating system 253 into the RAM memory 206, where the operating system 253 starts to work on the RAM memory 206. The operating system 253 is a system-level application executable by the processor 205 to implement various high-level functions including processor management, memory management, device management, storage management, software application interface, and general user interface.

[0065] The operating system 253 manages the memory 234 (209, 206) to ensure that each process or application running on the computer module 201 has sufficient memory to execute without conflicting with memory allocated to other processes. In addition, the appropriate use of Figure 2A The different types of memory available in the computer system 200 are described so that each process can run efficiently. Therefore, the aggregate memory 234 is not intended to illustrate how to allocate specific segments of memory (unless otherwise specified), but rather to provide an overview of the memory accessible to the computer system 200 and how to use it.

[0066] like Figure 2B As shown, the processor 205 includes a plurality of functional modules, wherein the plurality of functional modules include a control unit 239, an arithmetic logic unit (ALU) 240, and a local or internal memory 248 sometimes referred to as a cache memory. The cache memory 248 typically includes a plurality of storage registers 244-246 in a register section. One or more internal buses 241 functionally interconnect these functional modules. The processor 205 also typically has one or more interfaces 242 for communicating with external devices via the system bus 204 using a connection 218. The memory 234 is connected to the bus 204 using a connection 219.

[0067] The application program 233 includes an instruction sequence 231 that may include conditional branch instructions and loop instructions. The program 233 may also include data 232 used when executing the program 233. The instructions 231 and data 232 are stored in memory locations 228, 229, 230 and 235, 236, 237, respectively. Depending on the relative size of the instructions 231 and the memory locations 228-230, a particular instruction may be stored in a single memory location, as described by the instruction shown in memory location 230. Alternatively, the instruction may be split into multiple parts, each stored in a separate memory location, as described by the instruction segments shown in memory locations 228 and 229.

[0068] Typically, the processor 205 is given a set of instructions, which are executed within the processor 205. The processor 205 awaits a subsequent input, which the processor 205 reacts to by executing another set of instructions. Each input may be provided from one or more of a plurality of sources, including data generated by one or more of the input devices 202, 203, data received from an external source via one of the networks 220, 202, data retrieved from one of the storage devices 206, 209, or data retrieved from a storage medium 225 inserted into a corresponding reader 212 (all of which are described in detail in the accompanying drawings). Figure 2A Execution of a set of instructions may result in outputting data in some cases. Execution may also involve storing data or variables to memory 234.

[0069] The video encoder 114, the video decoder 134, and the method may use input variables 254 stored in corresponding memory locations 255, 256, 257 within the memory 234. The video encoder 114, the video decoder 134, and the method produce output variables 261 stored in corresponding memory locations 262, 263, 264 within the memory 234. Intermediate variables 258 may be stored in memory locations 259, 260, 266, and 267.

[0070] refer to Figure 2B The processor 205, registers 244, 245, 246, arithmetic logic unit (ALU) 240 and control unit 239 work together to perform micro-operation sequences required to perform a "fetch, decode and execute" cycle for each instruction in the instruction set that constitutes the program 233. Each fetch, decode and execute cycle includes:

[0071] A fetch operation for fetching or reading an instruction 231 from a memory location 228 , 229 , 230 ;

[0072] a decode operation in which the control unit 239 determines which instruction was fetched; and

[0073] An execution operation, wherein in the execution operation, the control unit 239 and / or the ALU 240 executes the instruction.

[0074] Thereafter, further fetch, decode and execute cycles for the next instruction may be performed. Likewise, a store cycle may be performed, whereby the control unit 239 stores or writes a value to the memory location 232 .

[0075] To be explained Figures 11 to 19 Each step or sub-processing in the method is associated with one or more segments of the program 233, and is typically performed by the register units 244, 245, 247, ALU 240 and control unit 239 in the processor 205 working together to perform fetch, decode and execute cycles for each instruction in the instruction set of the segment of the program 233.

[0076] Figure 3 is a schematic block diagram illustrating the functional modules of the video encoder 114 . Figure 4 1 is a schematic block diagram showing the functional modules of the video decoder 134. Typically, data is transferred between the functional modules within the video encoder 114 and the video decoder 134 in groups of samples or coefficients (such as partitioning of a block into fixed-size sub-blocks, etc.) or as an array. Figure 2A and 2B As shown, the video encoder 114 and the video decoder 134 can be implemented using a general-purpose computer system 200, wherein various functional modules can be implemented using dedicated hardware within the computer system 200, using software executable within the computer system 200 (such as one or more software code modules of a software application 233 residing on a hard disk drive 205 and controlled by a processor 205 for execution, etc.). Alternatively, the video encoder 114 and the video decoder 134 can be implemented using a combination of dedicated hardware and software executable within the computer system 200. The video encoder 114, the video decoder 134 and the method can alternatively be implemented in dedicated hardware such as one or more integrated circuits that perform the functions or sub-functions of the method. Such dedicated hardware may include a graphics processing unit (GPU), a digital signal processor (DSP), an application-specific standard product (ASSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or one or more microprocessors and associated memory. In particular, the video encoder 114 includes modules 310 - 392 , and the video decoder 134 includes modules 420 - 496 , where each of these modules may be implemented as one or more software code modules of the software application 233 .

[0077] although Figure 3 The video encoder 114 is an example of a Versatile Video Coding (VVC) video encoding pipeline, but other video codecs may also be used to perform the processing stages described herein. The video encoder 114 receives captured frame data 113, such as a series of frames (each frame including one or more color channels). The frame data 113 can be in any chroma format, such as a 4:0:0, 4:2:0, 4:2:2, or 4:4:4 chroma format. The block partitioner 310 first partitions the frame data 113 into CTUs, which are generally square in shape and are configured so that a specific size of the CTU is used. For example, the size of the CTU can be 64×64, 128×128, or 256×256 luma samples. The block partitioner 310 further partitions each CTU into one or more CBs corresponding to a luma coding tree or a chroma coding tree. CBs have various sizes and can include both square and non-square aspect ratios. Reference Figures 11 to 19 The operation of the block partitioner 310 is further described. However, in the VVC standard, CB, CU, PU, ​​and TU always have a side length that is a power of 2. Thus, the current CB (denoted as 312) is output from the block partitioner 310, thereby advancing according to the iteration of one or more blocks of the CTU, according to the luma coding tree and the chroma coding tree of the CTU. Figure 5 and 6 1 to further illustrate the option for partitioning a CTU into CBs. Although operations are generally described on a CTU-by-CTU basis, the video encoder 114 and the video decoder 134 may operate on smaller sized regions to reduce memory consumption. For example, each CTU may be divided into smaller regions, referred to as 'virtual pipeline data units' (VPDUs) of size 64×64. VPDUs form a data granularity that is more suitable for pipeline processing in a hardware architecture, where the reduction in memory footprint reduces silicon area and therefore reduces cost compared to operating on a full CTU.

[0078] The CTUs resulting from the first partitioning of the frame data 113 may be scanned in raster scan order and may be grouped into one or more "slices". A slice may be an "intra" (or "I") slice. An intra slice (I slice) indicates that each CU in the slice is intra predicted. Optionally, a slice may be uni-predicted or bi-predicted ("P" or "B" slices, respectively), indicating the additional availability of uni-prediction and bi-prediction in the slice, respectively.

[0079] For each CTU, the video encoder 114 operates in two stages. In the first stage (referred to as the "search" stage), the block partitioner 310 tests various potential configurations of the coding tree. Each potential configuration of the coding tree has an associated "candidate" CB. The first stage involves testing various candidate CBs to select a CB that provides relatively high compression efficiency and relatively low distortion. This test typically involves Lagrangian optimization, whereby the candidate CBs are evaluated based on a weighted combination of rate (coding cost) and distortion (error with respect to the input frame data 113). The "best" candidate CB (the CB with the lowest evaluated rate / distortion) is selected for subsequent encoding in the bitstream 115. The evaluation of the candidate CBs includes the following options: use the CB for a given region, or split the region according to various splitting options and use other CBs to encode each smaller resulting region or further split the region. As a result, both the CB and the coding tree itself are selected in the search stage.

[0080] The video encoder 114 generates a prediction block (PB) indicated by arrow 320 for each CB (e.g., CB 312). PB 320 is a prediction of the content of the associated CB 312. Subtractor module 322 generates a difference (or "residual," which means that the difference is in the spatial domain) between PB 320 and CB 312, represented as 324. Difference 324 is the block size difference between corresponding samples in PB 320 and CB 312. Difference 324 is transformed, quantized, and represented as a transform block (TB) indicated by arrow 336. PB 320 and associated TB 336 are typically selected from one of a plurality of possible candidate CBs (e.g., based on an evaluated cost or distortion).

[0081] A candidate coding block (CB) is a CB obtained from one of the prediction modes available to the video encoder 114 for the associated PB and the resulting residual. The TB 336 is a quantized and transformed representation of the difference 324. When combined with the predicted PB in the video decoder 114, the TB 336 reduces the difference between the decoded CB and the original CB 312 at the expense of additional signaling in the bitstream.

[0082] Thus, each candidate coding block (CB), i.e., a combination of a prediction block (PB) and a transform block (TB), has an associated coding cost (or "rate") and an associated difference (or "distortion"). The rate is typically measured in bits. The distortion of a CB is typically estimated as a difference in sample values, such as a sum of absolute differences (SAD) or a sum of squared differences (SSD), etc. The estimates obtained from each candidate PB can be determined by a mode selector 386 using the difference 324 to determine a prediction mode 387. The prediction mode 387 indicates a decision to use intra-frame prediction, inter-frame prediction, or matrix intra-frame prediction (MIP) for the current CB. The prediction mode 387 is determined as a selected mode among possible modes, including intra-frame prediction (including matrix intra-frame prediction, DC, planar and angular intra-frame prediction) or inter-frame prediction with an associated motion vector. The prediction mode 387 is typically selected by minimizing a distortion metric derived from a Lagrangian optimization of the distortions produced by each candidate mode summed with the result of scaling the associated rate using a Lambda value. When matrix intra prediction is in use, a matrix intra prediction mode (represented by arrow 388) is also determined to indicate which of several available matrix intra prediction modes is used for the current CB. Figures 11 to 14 As well as in alternative implementations Fig.17 and 18 As described in , the search for use and selection of MIP modes for deciding blocks (especially relatively small blocks) can be constrained to achieve a lower worst-case memory bandwidth for extracting matrix coefficients than an unconstrained search. The estimation of the coding cost associated with each candidate prediction mode and the corresponding residual coding can be performed at a significantly lower cost than the entropy coding of the residual. Therefore, multiple candidate modes can be evaluated to determine the best mode in terms of rate distortion.

[0083] Determining the best mode in terms of rate distortion is typically implemented using a variation of Lagrangian optimization. Selecting a matrix intra prediction mode 388 typically involves determining the coding cost of the residual data resulting from applying a particular matrix intra prediction mode. The coding cost can be approximated by using the "sum of absolute transform differences" (SATD), thereby obtaining an estimated transform residual cost using a relatively simple transform such as a Hadamard transform. In some implementations using relatively simple transforms, the cost obtained from the simplified estimation method is monotonically related to the actual cost that would otherwise be determined from a complete evaluation. In implementations with monotonically related estimated costs, the simplified estimation method can be used to make the same decision (i.e., intra prediction mode), where the complexity of the video encoder 114 is reduced. In order to allow for possible non-monotonicity in the relationship between the estimated cost and the actual cost, the simplified estimation method can be used to generate a list of best candidates. For example, non-monotonicity can be obtained from other mode decisions that can be used for encoding of the residual data. The list of best candidates can have any number. The best candidates can be used to conduct a more complete search to establish the best mode selection for encoding the residual data of each candidate, thereby allowing the final selection of the intra prediction mode and other mode decisions.

[0084] Other mode decisions include the ability to skip the forward transform (referred to as "transform skip"). Skipping a transform is appropriate for residual data that lacks sufficient correlation to reduce the coding cost via an expression that is a transform basis function. Certain types of content, such as relatively simple computer generated graphics, may exhibit similar behavior. For a "skipped transform", the residual coefficients are still encoded even though the transform itself is not performed.

[0085] A Lagrangian or similar optimization process may be employed for both the selection of the best partition of a CTU to a CB (using the block partitioner 310) and the selection of the best prediction mode from multiple possibilities. By applying the Lagrangian optimization process of the candidate modes in the mode selector module 386, the intra-prediction mode with the lowest cost measure is selected as the best mode. The lowest cost mode is the selected intra-prediction mode 388 and is also encoded in the bitstream 115 by the entropy encoder 338. The selection of the intra-prediction mode 388 by the operation of the mode selector module 386 is extended to the operation of the block partitioner 310. For example, the candidates for the selection of the intra-prediction mode 388 may include a mode applicable to a given block and additional modes applicable to multiple smaller blocks that are collectively co-located with the given block. In the case of including modes applicable to a given block and smaller co-located blocks, the process of selecting the candidates is implicitly also a process for determining the best hierarchical decomposition of the CTU to the CB.

[0086] In the second stage of the operation of the video encoder 114 (referred to as the "encoding" stage), iterations of the selected luma coding tree and the selected chroma coding tree (and thus the respective selected CBs) are performed in the video encoder 114. In the iterations, the CBs are encoded in the bitstream 115 as further described herein.

[0087] The entropy encoder 338 supports both variable length encoding of syntactic elements and arithmetic coding of syntactic elements. Arithmetic coding is supported using context-adaptive binary arithmetic coding processing. The syntactic elements of arithmetic coding consist of a sequence of one or more "bins (binary files)". Like a bit, the value of a bin is "0" or "1". However, bins are not encoded in the bit stream 115 as discrete bits. Bins have associated predicted (or "likely" or "maximum probability") values ​​and associated probabilities (referred to as "context"). When the actual bin to be encoded matches the predicted value, the "maximum probability symbol" (MPS) is encoded. Encoding the maximum probability symbol is relatively cheap in terms of consumed bits in the bit stream 115, including a total cost of less than one discrete bit. When the actual bin to be encoded does not match the possible value, the "minimum probability symbol" (LPS) is encoded. Encoding the minimum probability symbol has a relatively high cost in terms of consumed bits. The bin encoding technique enables bins with skewed probabilities of "0" vs "1" to be efficiently encoded. For syntax elements with two possible values ​​(ie, "flag"), a single bin is sufficient. For syntax elements with many possible values, a sequence of bins is required.

[0088] The presence of a later bin in the sequence may be determined based on the value of an earlier bin in the sequence. In addition, each bin may be associated with more than one context. A particular context may be selected based on, for example, an earlier bin in a syntax element and the bin values ​​of adjacent syntax elements (i.e., bin values ​​from adjacent blocks). Each time a context coding bin is encoded, the context selected for that bin (if any) is updated to reflect the new bin value. In this way, the binary arithmetic coding scheme is considered to be adaptive.

[0089] The video encoder 114 also supports bins lacking context ("bypass bins"). Bypass bins are encoded assuming an equal probability distribution between "0" and "1". Thus, each bin occupies one bit in the bitstream 115. The absence of context saves memory and reduces complexity, so bypass bins where the distribution of values ​​for a particular bin is not skewed are used. An example of an entropy encoder that employs context and adaptation is known in the art as CABAC (Context Adaptive Binary Arithmetic Coder), and many variations of this encoder are employed in video encoding.

[0090] The entropy encoder 338 encodes the prediction mode 387 (and, if used for the current CB, the matrix intra prediction mode 388) using a combination of context-encoded and bypass-encoded bins. Blocks of size 4×4 have 35 possible matrix intra prediction modes, otherwise blocks of size not exceeding 8×8 (i.e., 4×8, 8×4, and 8×8) have 19 possible matrix intra prediction modes. For other block sizes, there are 11 possible matrix intra prediction modes. Typically, a list of "maximum probability matrix modes" (MPMs) is generated in the video encoder 114. The list of maximum probability modes is typically a fixed length such as three modes. The list of maximum probability modes may include modes encountered in earlier blocks adjacent to the current block. For example, if the block above or to the left of the current block uses a MIP mode, the corresponding mode exists as the MPM of the current block. If the block above or to the left of the current block uses angular intra prediction, the MPM list of the current block may be filled with MIP modes derived via a lookup table that maps angular intra prediction modes to MIP modes. In addition, the MPM list of subsequent CUs predicted using conventional intra prediction (DC, planar, or angular) may include candidate intra prediction modes derived from a CU encoded using a MIP mode using a mapping table from MIP modes to candidate conventional intra prediction modes. The context-coded bin encodes a flag indicating whether the intra prediction mode is one of the maximum probability modes. If the intra prediction mode 388 is one of the maximum probability modes, further signaling is encoded using a bypass-coded bin. For example, the encoded further signaling uses a truncated unary bin string to indicate which maximum probability mode corresponds to the matrix intra prediction mode 388. Otherwise, the intra prediction mode 388 is encoded as a "residual mode". Encoding as a residual mode uses an alternative syntax such as a fixed-length code, and is also encoded using a bypass-coded bin to represent intra prediction modes other than the intra prediction modes present in the maximum probability mode list.

[0091] Some arrangements can avoid the complexity of MPM list construction for both the current CU (encoded using MIP mode) and the subsequent CU that can be encoded using conventional intra prediction modes, and can include candidate intra prediction modes derived from neighboring blocks encoded in any MIP mode. When MPM list construction is omitted, the MIP mode is binarized using a truncated binary code to represent the MIP mode. For each MIP mode, the truncated binary code results in a relatively uniform coding cost, while the MPM list has a low coding cost for each MIP added to the MPM list. Statistics obtained from MIP mode selection do not show a strong bias for selecting MIP modes included on the MPM list compared to MIP modes not included on the MPM list, indicating that omitting the generation of the MPM list does not reduce compression efficiency. For a 4×4 block, 35 possible MIP modes can be encoded using a 5-bit or 6-bit code, with 5 bits for MIP modes 0-28 and 6 bits for MIP modes 29-34. For 4×8, 8×4, and 8×8 blocks, 19 possible MIP modes can be encoded using 4-bit or 5-bit codes, with 4 bits for MIP modes 0-12 and 5 bits for MIP modes 13-18. For blocks of other sizes, 11 possible MIP modes can be encoded using 3-bit or 4-bit codes, with 3 bits for MIP modes 0-4 and 4 bits for MIP modes 5-10.

[0092] The multiplexer module 384 outputs the PB 320 according to the determined best prediction mode 387 selected from the tested prediction modes of the candidate CBs. The candidate prediction modes need not include every conceivable prediction mode supported by the video encoder 114.

[0093] In inter prediction, samples from one or two frames preceding the current frame in the order of coded frames in the bitstream are used to generate a prediction of the block. In addition, for inter prediction, a single coding tree is typically used for both the luminance channel and the chrominance channel. The order of coded frames in the bitstream may be different from the order of frames when captured or displayed. When one frame is used for prediction, the block is called "single prediction" and has two associated motion vectors. When two frames are used for prediction, the block is called "double prediction" and has two associated motion vectors. For P slices, each CU can be intra predicted or single predicted. For B slices, each CU can be intra predicted, single predicted, or double predicted. Frames are usually encoded using a "group of pictures" structure, thereby realizing a temporal hierarchical structure of frames. The temporal hierarchical structure of frames allows frames to refer to previous and subsequent pictures in the order in which frames are displayed. Images are encoded in the order necessary to ensure that the dependencies required to decode each frame are met.

[0094] Samples are selected based on motion vectors and reference picture indices. Motion vectors and reference picture indices apply to all color channels, so inter prediction is described primarily in terms of operations on PUs rather than PBs. Within each category (i.e., intra and inter prediction), different techniques can be applied to generate PUs. For example, intra prediction can use the value combination direction of adjacent rows and columns from previously reconstructed samples to generate PUs according to a specified filtering and generation process. Alternatively, a small number of parameters can be used to describe PUs. Inter prediction methods may vary in the number of motion parameters and their accuracy. Motion parameters typically include a reference frame index (which indicates which reference frames from a reference frame list will be used plus the spatial translation of each reference frame), but may include more frames, special frames, or complex affine parameters such as scaling and rotation. In addition, a predetermined motion refinement process can be applied to generate a dense motion estimate based on a block of reference samples.

[0095] In the case where the PB 320 is determined and selected and subtracted from the original sample block at the subtractor 322, a residual with the lowest coding cost (denoted as 324) is obtained and the residual is lossily compressed. The lossy compression process includes the steps of transform, quantization and entropy coding. The forward main transform module 326 applies a forward transform to the difference 324, thereby converting the difference 324 from the spatial threshold to the frequency domain and producing the main transform coefficients represented by arrow 328. The main transform coefficients 328 are passed to the forward secondary transform module 330 to produce the transform coefficients represented by arrow 332 by performing a non-separable secondary transform (NSST) operation. The forward main transform is generally separable, so that a group of rows and then a group of columns of each block are transformed. The forward main transform module 326 typically uses a type II discrete cosine transform (DCT-2), but type VII discrete cosine transform (DST-7) and type VIII discrete cosine transform (DCT-8) may also be available, for example for a block width of no more than 16 samples in the horizontal direction and a block height of no more than 16 samples in the vertical direction. The transformation of each group of rows and columns is performed by first applying a one-dimensional transform to each row of the block to produce an intermediate result and then applying a one-dimensional transform to each column of the intermediate result to produce a final result. The forward secondary transform of module 330 is typically a non-separable transform, which is only applicable to the residual of the intra-frame predicted CU, but it may also be bypassed. The forward secondary transform operates on 16 samples (arranged as the upper left 4×4 sub-block of the main transform coefficients 328) or 64 samples (arranged as the upper left 8×8 coefficients, arranged in four 4×4 sub-blocks of the main transform coefficients 328).

[0096] The transform coefficients 332 are passed to a quantizer module 334. At module 334, quantization according to a "quantization parameter" is performed to produce residual coefficients represented by arrow 336. The quantization parameter is constant for a given TB, resulting in uniform scaling of the generated residual coefficients for the TB. Non-uniform scaling can also be achieved by applying a "quantization matrix", whereby the scaling factor applied to each residual coefficient is derived from a combination of the quantization parameter and the corresponding entry in a scaling matrix whose size is generally equal to the size of the TB. The scaling matrix may have a size less than the size of the TB, and when applied to the TB, a nearest neighbor method is used to provide scaling values ​​for each residual coefficient from a scaling matrix whose size is less than the size of the TB. The residual coefficients 336 are supplied to an entropy encoder 338 for encoding in the bitstream 115. Typically, the residual coefficients of each TB and at least one significant residual coefficient of the TU are scanned according to a scan pattern to generate an ordered list of values. The scan pattern typically scans the TB into a sequence of 4×4 "sub-blocks", thereby providing a conventional scanning operation at a granularity of 4×4 sets of residual coefficients, where the configuration of the sub-blocks depends on the size of the TB. Additionally, the prediction mode 387 , matrix intra prediction mode (if used) 388 , and corresponding block partitions are also encoded in the bitstream 115 .

[0097] As described above, the video encoder 114 needs access to a frame representation that corresponds to the frame representation seen in the video decoder 134. Thus, the residual coefficients 336 are also inverse quantized by the dequantizer module 340 to produce inverse transform coefficients represented by arrow 342. The inverse transform coefficients 342 pass through the inverse secondary transform module 344 to produce intermediate inverse transform coefficients represented by arrow 346. The intermediate inverse transform coefficients 346 are passed to the inverse main transform module 348 to produce residual samples of the TU represented by arrow 350. The type of inverse transform performed by the inverse secondary transform module 344 corresponds to the type of forward transform performed by the forward secondary transform module 330. The type of inverse transform performed by the inverse main transform module 348 corresponds to the type of main transform performed by the main transform module 326. The summation module 352 adds the residual samples 350 and the PU 320 to produce the reconstructed samples of the CU (indicated by arrow 354).

[0098] The reconstructed samples 354 are passed to the reference sample cache 356 and the in-loop filter module 368. The reference sample cache 356, which is typically implemented using static RAM on an ASIC (thus avoiding expensive off-chip memory accesses), provides the minimum sample storage required to satisfy the dependencies used to generate intra PBs for subsequent CUs in the frame. The minimum dependencies typically include a "line buffer" of samples along the bottom of a row of CTUs for use by the next row of CTUs and a column buffer with a range set by the height of the CTU. The reference sample cache 356 supplies reference samples (represented by arrow 358) to the reference sample filter 360. The sample filter 360 applies a smoothing operation to produce filtered reference samples (indicated by arrow 362). The filtered reference samples 362 are used by the intra prediction module 364 to produce an intra prediction block of samples represented by arrow 366. For each candidate intra prediction mode, the intra prediction module 364 produces a sample block, namely 366. The sample block 366 is generated by module 364 using techniques such as DC, planar or angular intra prediction, but not matrix intra prediction.

[0099] If the mode selector 386 selects matrix intra prediction for the current CB, the matrix intra prediction mode 388 is used to select (read) the matrix coefficients 363 from the coefficient memory 392. The matrix coefficients 363 are passed to the matrix intra prediction module 390. The matrix intra prediction module 390 uses the matrix coefficients 363 and the reference samples 358 to perform matrix multiplication to produce a matrix intra prediction block 393. The multiplexer 384 outputs the matrix intra prediction block 393 as the PB 320. The coefficient memory 392 has a limited bandwidth that can be used to provide the matrix coefficients 393. In particular, different matrix intra prediction modes can be used for each consecutive block, thereby establishing a worst-case memory bandwidth requirement. As described with respect to Figures 11 to 14 As described above, the mode selector 386 is operable to select a MIP mode for a block under the constraint of reducing the worst-case memory bandwidth of the coefficient memory 392. Reducing the worst-case memory bandwidth of the coefficient memory 392 reduces complexity (e.g., hardware area of ​​the memory) without causing a proportional degradation in the encoding efficiency of the video encoder 114. Since the statistics of the MIP mode selection in the unconstrained search generally do not trigger the constraints applied to the MIP mode selection in the mode selector 386, the reduced complexity is achieved without a proportional degradation in encoding efficiency. Therefore, the worst-case memory bandwidth is reduced without a commensurate loss in encoding performance.

[0100] The in-loop filter module 368 applies several filtering stages to the reconstructed samples 354. The filtering stages include a "deblocking filter" (DBF), which applies smoothing aligned with CU boundaries to reduce artifacts caused by discontinuities. Another filtering stage present in the in-loop filter module 368 is an "adaptive loop filter" (ALF), which applies a Wiener-based adaptive filter to further reduce distortion. Another available filtering stage in the in-loop filter module 368 is a "sample adaptive offset" (SAO) filter. The SAO filter works by first classifying the reconstructed samples into one or more categories and applying an offset at the sample level according to the assigned category.

[0101] Filtered samples, represented by arrow 370, are output from the in-loop filter module 368. The filtered samples 370 are stored in a frame buffer 372. The frame buffer 372 typically has the capacity to store several (e.g., up to 16) pictures and is therefore stored in the memory 206. Due to the large memory consumption required, the frame buffer 372 is typically not stored using on-chip memory. As such, access to the frame buffer 372 is expensive in terms of memory bandwidth. The frame buffer 372 provides a reference frame (represented by arrow 374) to the motion estimation module 376 and the motion compensation module 380.

[0102] The motion estimation module 376 estimates a plurality of "motion vectors" (denoted as 378), each of which is a Cartesian spatial offset relative to the position of the current CB, thereby referencing a block in one of the reference frames in the frame buffer 372. A filtered block of reference samples (denoted as 382) is generated for each motion vector. The filtered reference samples 382 form a further candidate mode for potential selection by the mode selector 386. In addition, for a given CU, the PB 320 may be formed using one reference block ("uni-prediction"), or may be formed using two reference blocks ("bi-prediction"). For the selected motion vector, the motion compensation module 380 generates the PU 320 according to a filtering process that supports sub-pixel precision in the motion vector. In this way, the motion estimation module 376 (which operates on many candidate motion vectors) can perform a simplified filtering process compared to the motion compensation module 380 (which operates only on the selected candidate) to achieve reduced computational complexity. When the video encoder 114 selects inter-frame prediction for the CU, the motion vector 378 is encoded in the bitstream 115.

[0103] Although the reference to Versatile Video Coding (VVC) describes Figure 3The video encoder 114 of the embodiment of the present invention may be used, but other video coding standards or implementations may also use the processing stages of modules 310-386. The frame data 113 (and the bit stream 115) may also be read from the memory 206, the hard disk drive 210, the CD-ROM, the blue disc (Blue-ray diskTM) or other computer-readable storage medium (or written to the memory 206, the hard disk drive 210, the CD-ROM, the blue disc or other computer-readable storage medium). In addition, the frame data 113 (and the bit stream 115) may be received from an external source (such as a server or a radio frequency receiver connected to the communication network 220) (or sent to the external source).

[0104] exist Figure 4 The video decoder 134 is shown in FIG. Figure 4 The video decoder 134 of FIG. 1 is an example of a Versatile Video Coding (VVC) video decoding pipeline, but other video codecs may also be used to perform the processing stages described herein. Figure 4 As shown, the bitstream 133 is input to the video decoder 134. The bitstream 133 can be read from the memory 206, the hard disk drive 210, the CD-ROM, the Blu-ray disc or other non-transitory computer-readable storage medium. Alternatively, the bitstream 133 can be received from an external source (such as a server or a radio frequency receiver connected to the communication network 220). The bitstream 133 contains the encoding syntax elements representing the captured frame data to be decoded.

[0105] The bitstream 133 is input to the entropy decoder module 420. The entropy decoder module 420 extracts syntax elements from the bitstream 133 by decoding a sequence of "bins" and passes the values ​​of the syntax elements to other modules in the video decoder 134. The entropy decoder module 420 uses an arithmetic decoding engine to decode each syntax element into a sequence of one or more bins. Each bin can use one or more "contexts", where the context describes the probability level of encoding "one" and "zero" values ​​for the bin. In the case where multiple contexts are available for a given bin, a "context modeling" or "context selection" step is performed to select one of the available contexts to decode the bin. The process of decoding the bin forms a sequential feedback loop. The number of operations in the feedback loop is preferably minimized so that the entropy decoder 420 can achieve a high throughput in bins / second. Context modeling depends on other characteristics of the bitstream known to the video decoder 134 when selecting the context, that is, the characteristics before the current bin. For example, the context can be selected based on the quadtree depth of the current CU in the coding tree. The dependency is preferably based on characteristics that are well known before decoding the bin, or determined without long sequential processing.

[0106] The quadtree depth of the coding tree is an example of a dependency of context modeling that is easy to know. Intra-frame prediction modes (especially matrix intra-frame prediction modes) are examples of dependencies of context modeling and binarization that are relatively difficult to determine or computationally intensive. Matrix intra-frame prediction modes are encoded as an index to a list of "maximum probability modes" (MPM) or an index to a list of "remaining modes", where the MPM and the remaining modes are selected according to the decoded context coding flag. Other intra-frame prediction modes are encoded as an index to a list of 'maximum probability modes' (MPM) or an index to a list of 'remaining modes', where the MPM and the remaining modes are selected according to the decoded intra-frame brightness MPM flag. When MPM is used to encode matrix intra-frame prediction mode, a truncated unary bin string with a range of 0 to 2 selects one of the MPMs from the MPM list. When the remaining mode is used, a fixed-length codeword is decoded to select which of the remaining (non-MPM) modes to use. Depending on the block size, the number of available MIP modes is one of 35, 19 and 11. Thus, for an MPM list of length 3, the number of remaining modes is 32, 16, or 8, respectively. The remaining modes can be efficiently represented using fixed-length codewords of length 5, 4, or 3, respectively. Determining both the maximum probability mode and the remaining modes requires a large number of operations and includes a dependency on the intra prediction modes of neighboring blocks. For example, the neighboring blocks may be blocks above and to the left of the current block. If the neighboring blocks use angular intra prediction, a table lookup may be performed to map the angular intra prediction mode to a matrix intra prediction mode for use in populating the MPM list. Alternatively, the arrangement may use truncated binary codewords to encode the MIP mode for each case (i.e., whether the number of modes is 35, 19, or 11). The entropy decoder module 420 applies an arithmetic coding algorithm, such as "context adaptive binary arithmetic coding" (CABAC), to decode syntax elements from the bitstream 133. The decoded syntax elements are used to reconstruct parameters within the video decoder 134. The parameters include residual coefficients (represented by arrow 424) and mode selection information such as intra prediction mode (represented by arrow 458). The mode selection information also includes information such as motion vectors, and partitioning of each CTU into one or more CBs. The parameters are used to generate PBs, typically in combination with sample data from previously decoded CBs.

[0107] The residual coefficients 424 are input to a dequantizer module 428. The dequantizer module 428 inverse quantizes (or "scales") the residual coefficients 424 to create reconstructed intermediate transform coefficients (indicated by arrow 432) according to the quantization parameters. The reconstructed intermediate transform coefficients 432 are passed to an inverse secondary transform module 436 where a secondary transform is applied or no operation (bypass). The inverse secondary transform module 436 produces reconstructed transform coefficients 440. If the use of a non-uniform inverse quantization matrix is ​​indicated in the bitstream 133, the video decoder 134 reads the quantization matrix from the bitstream 133 as a sequence of scaling factors and arranges the scaling factors into a matrix according to the inverse scaling. The inverse scaling uses the quantization matrix in combination with the quantization parameters to create the reconstructed intermediate transform coefficients 432.

[0108] The reconstructed transform coefficients 440 are passed to an inverse main transform module 444. Module 444 transforms the coefficients from the frequency domain back to the spatial domain. The result of the operation of module 444 is a block of residual samples represented by arrow 448. The block of residual samples 448 is equal in size to the corresponding CU. The residual samples 448 are supplied to a summing module 450. At the summing module 450, the residual samples 448 are added to the decoded PB represented by 452 to produce a block of reconstructed samples represented by arrow 456. The reconstructed samples 456 are supplied to a reconstructed sample cache 460 and an in-loop filtering module 488. The in-loop filtering module 488 produces a reconstructed block of frame samples represented by 492. The frame samples 492 are written to a frame buffer 496.

[0109] The reconstructed sample cache 460 operates in a manner similar to the reconstructed sample cache 356 of the video encoder 114. The reconstructed sample cache 460 provides storage for the reconstructed samples needed for intra prediction of subsequent CBs without the memory 206 (e.g., by using data 232, which is typically on-chip memory, instead). Reference samples, represented by arrows 464, are obtained from the reconstructed sample cache 460 and are supplied to a reference sample filter 468 to produce filtered reference samples, represented by arrows 472. The filtered reference samples 472 are supplied to an intra prediction module 476. The module 476 produces a block of intra prediction samples, represented by arrows 480, based on the intra prediction mode parameters 458 represented in the bitstream 133 and decoded by the entropy decoder 420. The sample block 480 is generated using a mode other than matrix intra prediction, such as DC, planar or angular intra prediction.

[0110] When the prediction mode of the CB is indicated in the bitstream 133 to use intra prediction (rather than matrix intra prediction), the intra prediction samples 480 form the decoded PB 452 via the multiplexer module 484. Intra prediction produces a prediction block (PB) of samples, that is, a block in one color component derived using "neighboring samples" in the same color component. Neighboring samples are samples that are adjacent to the current block and have been reconstructed because they are in the front in the block decoding order. In the case of luminance and chrominance blocks being collocated, luminance and chrominance blocks can use different intra prediction modes. However, the two chrominance channels each share the same intra prediction mode. Intra prediction is divided into three types. "DC intra prediction" involves filling the PB with a single value representing the average of neighboring samples. "Plane intra prediction" involves filling the PB with samples corresponding to a plane, where the DC offset and vertical and horizontal gradients are derived from neighboring samples. "Angle intra prediction" involves filling the PB with neighboring samples filtered and propagated on the PB in a specific direction (or "angle"). In VVC, 65 angles are supported, with rectangular blocks being able to take advantage of additional angles not available to square blocks to produce a total of 87 angles. A fourth type of intra prediction is available for chroma PBs, whereby PBs are generated from juxtaposed luma reconstruction samples according to a "cross component linear model" (CCLM) mode. Three different CCLM modes are available, each using a different model derived from adjacent luma and chroma samples. The derived model is then used to generate a sample block of the chroma PB from the juxtaposed luma samples.

[0111] When the prediction mode of the CB is indicated in the bitstream 133 as matrix intra prediction, the matrix intra prediction mode 458 is decoded and supplied to the coefficient memory 486 and the matrix intra prediction module 482. The matrix coefficients 481 for the selected matrix intra prediction mode are read from the coefficient memory 486 and passed to the matrix intra prediction module 482. The selection of the matrix coefficients involves a memory read operation from the coefficient memory 486, where the worst case memory bandwidth limit for the memory access is established by the frequency of selecting the MIP mode for a given block size, as described in reference to FIG. Figure 8 and as described in 9.

[0112] When the prediction mode of the CB is indicated in the bitstream 133 as inter prediction, the motion compensation module 434 selects and filters a block of samples 498 from a frame buffer 496 using the motion vector and the reference frame index to produce a block of inter prediction samples indicated as 438. The block of samples 498 is obtained from a previously decoded frame stored in the frame buffer 496. To perform bi-prediction, two blocks of samples are generated and blended together to produce samples of the decoded PB 452. The frame buffer 496 is populated with filter block data 492 from the in-loop filtering module 488. The in-loop filtering module 488 applies any DBF, ALF, and SAO filtering operations as the in-loop filtering module 368 of the video encoder 114. Typically, motion vectors are applied to both the luma and chroma channels, but the filtering process for sub-sample interpolated luma and chroma channels is different.

[0113] Figure 5 5 is a schematic block diagram showing a set 500 of possible partitions or splits of a region into one or more sub-regions in a tree structure for general video coding. Figure 3 As described, the partitioning shown in set 500 may be utilized by block partitioner 310 of encoder 114 to partition each CTU into one or more CUs or CBs according to the encoding number as determined by Lagrangian optimization.

[0114] Although set 500 only illustrates partitioning of a square region into other possible non-square sub-regions, it should be understood that diagram 500 illustrates potential partitioning and does not require that the containing region be square. If the containing region is non-square, the size of the blocks resulting from the partitioning is scaled according to the aspect ratio of the containing block. Once a region is not split further, i.e., at a leaf node of the coding tree, a CU occupies the region. A specific sub-partition of a CTU into one or more CUs using block partitioner 310 is referred to as a "coding tree" for the CTU.

[0115] The process of sub-dividing a region into sub-regions must be terminated when the resulting sub-regions reach the minimum CU size. In addition to constraining the CU to prohibit block regions from being less than a predetermined minimum size of, for example, 16 samples, the CU is constrained to have a minimum width or height of four. Other minimum values ​​are also possible in terms of width and height or in terms of width or height. The sub-division process can also be terminated before the deepest level of decomposition, resulting in a CU larger than the minimum CU size. It is possible that no splitting occurs, resulting in a single CU occupying the entire CTU. A single CU occupying the entire CTU is the maximum available coding unit size. Due to the use of a sub-sampled chroma format (such as 4:2:0, etc.), the arrangement of the video encoder 114 and the video decoder 134 can terminate the splitting of regions in the chroma channel earlier than in the luminance channel.

[0116] In the absence of further sub-division, there is a CU at the leaf node of the coding tree. For example, leaf node 510 contains a CU. At the non-leaf node of the coding tree, there is a split to two or more other nodes, each of which is a leaf node that forms a CU, or a non-leaf node that contains further splits to smaller areas. At each leaf node of the coding tree, there is a coding block for each color channel. The split that terminates at the same depth for both brightness and chrominance obtains three juxtaposed CBs. The split that terminates at a deeper depth for brightness than for chrominance obtains multiple brightness CBs juxtaposed with the CBs of the chrominance channels.

[0117] like Figure 5 As shown, quadtree split 512 splits the containing area into four equal-sized areas. Compared to HEVC, Versatile Video Coding (VVC) achieves additional flexibility by adding horizontal binary split 514 and vertical binary split 516. Splits 514 and 516 each split the containing area into two equal-sized areas. The splits are along horizontal boundaries (514) or vertical boundaries (516) within the containing block.

[0118] Further flexibility is achieved in general video coding by adding ternary horizontal split 518 and ternary vertical split 520. Ternary split 518 and 520 divide the block into three regions that form boundaries in the horizontal direction (518) or vertical direction (520) along 1 / 4 and 3 / 4 of the width or height of the containing region. The combination of quadtree, binary tree and ternary tree is called "QTBTTT". The root of the tree includes zero or more quadtree splits (the "QT" part of the tree). Once the QT part is terminated, zero or more binary or ternary splits ("multi-tree" or "MT" part of the tree) may occur, eventually ending in a CB or CU at the leaf node of the tree. In the case where the tree describes all color channels, the leaf node of the tree is a CU. In the case where the tree describes a luminance channel or a chrominance channel, the leaf node of the tree is a CB.

[0119] Compared to HEVC, which only supports quadtrees and therefore only supports square blocks, QTBTTT obtains more possible CU sizes, especially considering the possible recursive application of binary and / or ternary tree splits. The possibility of abnormal (non-square) block sizes can be reduced by constraining the splitting options to eliminate block widths or heights that will be less than four samples or splits that will not be a multiple of four samples. Typically, the constraints will apply when considering luma samples. However, in the described arrangement, the constraints can be applied separately to blocks of chroma channels. The application of the constraints of the splitting options to the chroma channels may result in different minimum block sizes for luma vs chroma (for example, when the frame data adopts a 4:2:0 chroma format or a 4:2:2 chroma format). Each split produces a sub-region with an unchanged, bisected or quartered side size relative to the containing area. Then, since the CTU size is a power of 2, the side sizes of all CUs are also powers of 2.

[0120] Figure 6 6 is a schematic flow diagram showing a data flow 600 of a QTBTTT (or "coding tree") structure used in general video coding. The QTBTTT structure is used for each CTU to define the partitioning of the CTU into one or more CUs. The QTBTTT structure for each CTU is determined by the block partitioner 310 in the video encoder 114 and encoded into the bitstream 115 or decoded from the bitstream 133 by the entropy decoder 420 in the video decoder 134. Figure 5 As shown partitioned, data flow 600 further features permitted combinations that may be used by block partitioner 310 to partition a CTU into one or more CUs.

[0121] Starting from the top level of the hierarchy, i.e., at the CTU, zero or more quadtree partitions are first performed. Specifically, a quadtree (QT) split decision 610 is made by the block partitioner 310. The decision at 610 returns a "1" symbol, which indicates a decision to split the current node into four child nodes according to the quadtree split 512. The result is that four new nodes are generated, such as at 620, and for each new node, recursively return to the QT split decision 610. Each new node is considered in raster (or Z-scan) order. Alternatively, if the QT split decision 610 indicates that no further splitting is to be performed (returns a "0" symbol), the quadtree partitioning stops, and multi-tree (MT) splitting is then considered.

[0122] First, an MT split decision 612 is made by the block partitioner 310. At 612, a decision to perform an MT split is indicated. A "0" symbol is returned at the decision 612, indicating that no further splitting of the node into child nodes will be performed. If no further splitting of the node will be performed, the node is a leaf node of the coding tree and corresponds to a CU. The leaf node is output at 622. Alternatively, if the MT split 612 indicates a decision to perform an MT split (a "1" symbol is returned), the block partitioner 310 proceeds to a direction decision 614.

[0123] Direction decision 614 indicates the direction of the MT split as horizontal ("H" or "0") or vertical ("V" or "1"). If decision 614 returns "0" indicating a horizontal direction, block partitioner 310 proceeds to decision 616. If decision 614 returns "1" indicating a vertical direction, block partitioner 310 proceeds to decision 618.

[0124] In each of decisions 616 and 618, the number of partitions into which the MT is split is indicated as two (binary split or "BT" nodes) or three (ternary split or "TT") in the case of a BT / TT split. That is, the BT / TT split decision 616 is made by the block partitioner 310 when the direction indicated from 614 is horizontal, and the BT / TT split decision 618 is made by the block partitioner 310 when the direction indicated from 614 is vertical.

[0125] The BT / TT split decision 616 indicates whether the horizontal split is a binary split 514 indicated by returning “0” or a ternary split 518 indicated by returning “1”. When the BT / TT split decision 616 indicates a binary split, at a generate HBT CTU node step 625, the block partitioner 310 generates two nodes according to the binary horizontal split 514. When the BT / TT split 616 indicates a ternary split, at a generate HTT CTU node step 626, the block partitioner 310 generates three nodes according to the ternary horizontal split 518.

[0126] The BT / TT split decision 618 indicates whether the vertical split is a binary split 516 indicated by returning "0" or a ternary split 520 indicated by returning "1". When the BT / TT split 618 indicates a binary split, at step 627 of generating VBT CTU nodes, the block partitioner 310 generates two nodes according to the vertical binary split 516. When the BT / TT split 618 indicates a ternary split, at step 628 of generating VTT CTU nodes, the block partitioner 310 generates three nodes according to the vertical ternary split 520. For each node obtained from steps 625-628, the recursion of the data flow 600 back to the MT split decision 612 is applied in a left-to-right or top-to-bottom order according to the direction 614. As a result, binary and ternary tree splits can be applied to generate CUs of various sizes.

[0127] Fig. 7A and 7B An example partition 700 of a CTU 710 into multiple CUs or CBs is provided. Fig. 7A An example CU 712 is shown in FIG. Fig. 7A 7 shows the spatial arrangement of CUs in a CTU 710. Example partitioning 700 in Figure 7B Also shown in FIG. 7 is a coding tree 720 .

[0128] exist Fig. 7A At each non-leaf node (e.g., nodes 714, 716, and 718) in the CTU 710 of the coding tree 720, the contained nodes (which may be further partitioned or may be CUs) are scanned or traversed in "Z order" to create a list of nodes represented as columns in the coding tree 720. For quadtree splits, the Z order scan results in an order from top left to right followed by a bottom left to right order. For horizontal and vertical splits, the Z order scan (traversal) is simplified to a scan from top to bottom and a scan from left to right, respectively. Figure 7B The coding tree 720 lists all nodes and CUs according to the scanning order applied. Each split generates a list of two, three or four new nodes at the next level of the tree until a leaf node (CU) is reached.

[0129] In reference Figure 3 Where the image is decomposed into CTUs and further into CUs using block partitioner 310, and the CUs are used to generate respective residual blocks (324), the residual blocks are forward transformed and quantized using video encoder 114. The resulting TBs 336 are then scanned to form a sequential list of residual coefficients as part of the operation of entropy encoding module 338. Equivalent processing is performed in video decoder 134 to obtain the TBs from bitstream 133.

[0130] As a result of splitting the CTU into CUs, small-sized CUs (such as 4×4, 4×8, or 8×4) appear to be spatially adjacent in the frame. In addition, with the help of hierarchical Z-order scanning of the coding tree, small-sized CUs are processed temporally adjacent. In particular, the quadtree splitting of an 8×8 area results in a sequence of four 4×4 CUs, the binary splitting of an 8×4 or 4×8 area results in a pair of 4×4 CUs, and the binary splitting of an 8×8 area results in a pair of 4×8 areas or a pair of 8×4 areas. The resulting individual areas (of size 4×8 or 8×4) can form a CU or can be further split into 4×4 CUs. The ternary splitting applied in an area of ​​64 samples results in three areas of size 16, 32, and 16 (4×4, 4×8, or 8×4 and 4×4). The ternary splitting applied in an area of ​​128 samples results in three areas of size 32, 64, and 32 samples (e.g., 4×8, 8×8, and 4×8). Small blocks (eg, 4x4, 4x8, 8x4) are seen together both spatially and in the Z-order scan through the CTUs because they are the result of splitting regions of size such as 64 or 128.

[0131] Figure 8 A data flow 800 is shown that provides details of the operation of the matrix intra prediction modules 390 and 482 of the video encoder 114 and the video decoder 134, respectively, using 8×8 blocks. Figure 8 In the example of , matrix intra prediction is performed only in the luma channel, and therefore different block sizes resulting from using a chroma format such as 4:2:0 or 4:2:2 need not be considered. Modules 390 and 482 output matrix prediction sample blocks, such as 393 or 483, respectively. When the mode selector 386 selects matrix intra prediction for a CU, block 393 is used as PB 320 in the video encoder 114. When the use of matrix intra prediction is indicated by decoding the prediction mode of the CU from the bitstream 133, block 483 is used as PB 483.

[0132] The operation of modules 390 and 482 involves three steps: 1. averaging, 2. matrix multiplication and offset (bias) addition, and 3. bilinear interpolation. Bilinear interpolation is performed only when the CU size is greater than 4×4. The averaging step operates as follows. Reference samples 802 (e.g., 358 or 464) are received at data stream 800 and allocated as upper samples 822 and left samples 820. When the width and height of the luma CB are greater than 4, the upper samples 822 are divided into four sets ( Figure 8 The values ​​of the contents of each of the four sets are averaged to produce four filtered upper samples 826. Similarly, the left side samples 820 are divided into four sets ( Figure 8The values ​​of the contents of each of the four sets are averaged to produce four filtered left samples 824. Therefore, a total of eight filtered samples are used as input to the matrix multiplication module 828.

[0133] When the block width and height are equal to four (instead of Figure 8 8), the four upper samples 822 are divided into two pairs, and each pair is averaged to produce two filtered upper samples 826. Similarly, the four left samples 820 are divided into two pairs, and each pair is averaged to produce two filtered left samples 824, with a total of four filtered samples as input to the matrix multiplication module 828.

[0134] In the second step (matrix multiplication), the received matrix coefficients (363 or 481) are selected according to the matrix intra prediction mode (388 or 458) and are also input to the matrix multiplication module 828. In addition, the input to the matrix multiplication module is a set of offset or bias values. The offset or bias value is added to the result of the matrix multiplication to introduce any desired DC shift. The matrix coefficients and bias values ​​are predetermined. In other words, the matrix coefficients and bias values ​​are the result of the "offline" training process and are considered to be constant values ​​by the video encoder 114 and the video decoder 134.

[0135] For 4×4 CB, 35 MIP modes are available, with a set of 18 matrix coefficients and bias values ​​(set A). For 4×8, 8×4 and 8×8 CB, 19 MIP modes are available, with a set of 10 matrix coefficients and bias values ​​(set B). For other CB sizes, 11 MIP modes are available, with a set of 6 matrix coefficients and bias values ​​(set C). A given set of matrix coefficients and bias values ​​can be used for two MIP modes. One MIP mode uses the provided values, while the other MIP mode uses the transpose of the provided values. In addition, in one case, the set of matrix coefficients and bias values ​​is dedicated to a single MIP mode. Three cases are each applicable to each of the sets AC. The size of the set AC is as follows, and the number of words selected by 362 or 481 for generating a PB (i.e., applying the MIP mode to a CU) is also shown, which is given for each of the sets AC as follows:

[0136] Set A: 18 matrices of size 16×4, 18 offset vectors of size 16. The size of the selected values ​​of CB is 16×4+16=80 words.

[0137] Set B: 10 matrices of size 16×8, 10 offset vectors of size 16. The size of the selected values ​​of CB is 16×8+16=144 words.

[0138] Set C: 6 matrices of size 64×8, 10 offset vectors of size 64. The size of the selected values ​​of CB is 64×8+64=576 words.

[0139] The video encoder 114 and the video decoder 134 process video data at a pixel rate determined by the frame size and the frame rate. In addition, the luma CB size is a multiple of four in width and height. Therefore, the memory bandwidth requirements of the sets AC can be expressed in terms of accesses to 4×4 luma samples. When there are no constraints on the use of the MIP mode of each CU, the access density of the sets AC for the worst case (i.e., the minimum block size of the corresponding set) is as follows:

[0140] Set A: 4×4 CB requires 80 words per 4×4 sample region.

[0141] Set B: 4×8 and 8×4 CB require 144 ÷ 2 = 72 words per 4×4 sample area, and 8×8 CB requires 144 ÷ 4 = 36 words per 4×4 sample area.

[0142] Set C: 8×16 and 16×8 CBs require 576÷8=72 words per 4×4 sample area, 8×32, 16×16, 32×8 CBs require 576÷16=36 words per 4×4 sample area, and larger CB sizes require fewer words per 4×4 sample area.

[0143] As shown in the access density above, when there is no constraint on the use of MIP mode, in the worst case, all CBs may use MIP mode and the coding tree may decompose each CTU into the small CB sizes used above to illustrate the worst-case coefficient memory (486, 392) bandwidth. The nominal word size for matrix coefficients and bias values ​​is 16 bits, but fewer bits (e.g., 10 bits) may also be sufficient. Extracting words in groups is a possible implementation choice. However, the memory bandwidth burden is still somewhat high.

[0144] The statistics of MIP mode selection show that, typically, in 20% of cases over a wide test set (as defined in the JVET Common Test Conditions document JVET-N1010), a MIP mode is selected for a neighboring located CB when considering the above and left blocks. Thus, limiting the application frequency of the MIP mode can mitigate the worst-case memory bandwidth required for the coefficient memories 486 and 392 without resulting in a commensurate reduction in the compression efficiency gain from the availability of the MIP mode. Figures 11 to 16 Describes constraints on MIP mode selection.

[0145] Matrix multiplication module 828 performs a matrix multiplication using the matrix coefficients (363 or 481) and the set of filtered reference samples (i.e., 824 and 826). The bias value is added to the output of the matrix multiplication to form a sparse block 830. Sparse block 830 is shown as partially filled with shadow samples of 393 or 483. The remainder of the 393 or 483 samples is derived using bilinear interpolation, with contributions from the upper reference sample 822 and the left filtered reference sample 824 or the left reference sample 820 and the upper filtered sample 822 (e.g., Figure 8 shown).

[0146] Fig.9A An example CTU 900 is shown. CTU 900 has a region of memory access bandwidth to which MIP mode is applied. Matrix intra prediction is applied within a region of VPDU size (i.e., a 64×64 quadrant of a 128×128 CTU 900). For example, the 512 luma sample regions to which the budget applies include (i) a 32×16 region 912, (ii) a 16×32 region 914, and (iii) a 64×8 region 916. Region 912 is further decomposed into various CUs, i.e., Fig. 9B 912a-912h in. Fig. 9B In the example of , CU 912a is of size 16×4 and therefore belongs to set C, CU 912d is of size 8×8 and therefore belongs to set B as are CUs 912b, 912c, and 912e (all of size 8×4). CUs 912f and 912g are each 4×4 and therefore belong to set A. Figures 11 to 16 As described, the area of ​​a region of 512 luma samples can be used as a threshold in the described arrangement to determine whether to apply a constraint for a MIP mode. Using the area of ​​a region of 512 samples is suitable as a constraint because the above-mentioned sets A to C can generally be generated by splitting of a region. However, the threshold may involve different areas as described below.

[0147] Fig.10 Shown with Fig.9A 900. Region 912 corresponds to a node in the coding tree at which the budget of the contained CU is established for MIP matrix coefficient reading, as are regions 914 and 916. For ease of reference, the decomposition into one or more CUs in a region is not shown. Fig.11 As described in FIG16 , for each CU configured to use the MIP mode, a budget amount mapped to one of the sets AC according to the CU size is deducted from the region (ie, 912 , 914 , 916 ) budget.

[0148] Fig.11A method 1100 for encoding coding units of an image frame in a video bitstream 115 is shown. The method 1100 may be embodied by a device such as a configured FPGA, ASIC, or ASSP. In addition, the method 1100 may be performed by a video encoder 114 under execution by a processor 205. Thus, the method 1100 may be stored on a computer-readable storage medium and / or in a memory 206. The method 1100 is performed to encode an image frame as a sequence of coding units into a bitstream 115, which applies constraints to the use of matrix intra prediction. Thus, the operation of the method 1100 limits the memory bandwidth required for matrix intra prediction compared to the worst case potential memory bandwidth (as in the absence of constraints). The method 1100 begins at step 1110 of partitioning a frame into CTUs.

[0149] At step 1110 of partitioning the frame into CTUs, the block partitioner 310 partitions the current frame of the frame data 113 into a CTU array under the execution of the processor 205. Encoding of the CTUs resulting from the partitioning begins. Control in the processor proceeds from step 1110 to step 1120 of determining a coding tree.

[0150] At step 1120 of determining the coding tree, the video encoder 114, under execution of the processor 205, tests various splitting options (such as reference 1130) in conjunction with the operation of step 1130 of determining the coding unit. Figure 5 -7) to obtain the coding tree of CTU. Fig.12 The operation of step 1120 is described. From step 1120, control in the processor 205 proceeds to step 1130 where a prediction mode of a coding unit is determined.

[0151] At the determine coding unit step 1130, the video encoder 114, under execution of the processor 205, determines a prediction mode to be used when encoding the selected coding unit in the bitstream 115. The coding unit may be selected according to a scanning pattern. Fig.13 The operation of step 1130 is further described. Once a prediction mode is selected for a coding unit, control in the processor 205 proceeds from step 1130 to a step 1140 of encoding the coding unit. In selecting a prediction mode for a coding unit, a particular combination of splits to arrive at a given coding unit is also selected, as the coding unit itself is generated by a hierarchy of splits of the coding tree, thereby determining the coding tree.

[0152] At step 1140 of encoding the coding unit, the entropy encoder 338, under execution by the processor 205, encodes the coding unit determined at step 1130 in the bitstream 115. At step 1140, the entropy encoder 338, under execution by the processor 205, effectively encodes the determined coding tree in the bitstream 115 using the "split flag" and other syntax elements to indicate the selected split, such as Figure 5 and 6 shown.

[0153] refer to Fig.14 The operation at step 1140 is further described. From step 1140, control in processor 205 passes to a final encoding unit test step 1150.

[0154] At the last coding unit test step 1150, the processor 205 tests whether the current coding unit is the last coding unit in the coding tree of step 1120. If the current coding unit is the last one in the coding tree of step 1120 ("Yes" at step 1150), control in the processor 205 proceeds to a last CTU test step 1160. If the current coding unit is not the last one in the coding tree of step 1120 ("No" at step 1150), the next coding unit in the coding tree of step 1120 is selected using the scanning pattern for determination and encoding, and control in the processor 205 proceeds to step 1130. Therefore, for each CU obtained from the coding tree determined at step 1120, step 1130 is performed.

[0155] At the last CTU test step 1160, the processor 205 tests whether the current CTU is the last CTU in the slice or frame. If not ("No" at step 1160), the video encoder 114 advances to the next CTU in the frame, and control in the processor 205 advances from step 1160 back to step 1120 to continue processing the remaining CTUs in the frame. If the CTU is the last one in the frame or slice, step 1160 returns "Yes" and the method 1100 terminates. As a result of the operation of the method 1100, the entire image frame is encoded in the bitstream as a sequence of CTUs.

[0156] The method 1100 is performed for each image frame in the video sequence. The method 1100 may determine the CU on a CTU-by-CTU basis. In other words, the CU of the CTU may be determined first in a pass or pipeline stage, followed by a second stage for encoding in the bitstream 115. The method 1100 may also determine the CU at a finer granularity (e.g., on a VPDU-by-VPDU basis) to reduce memory consumption due to a smaller VPDU area compared to a CTU.

[0157] Fig.12A method 1200 for determining a coding tree for a CTU as implemented at step 1120 is shown. The method 1200 is executed to receive a CTU and generate candidate splits and candidate coding units for evaluation and final selection as splits and coding units to be encoded in the bitstream 115. In particular, the method 1200 establishes a budget for MIP mode memory bandwidth at a specific node or region in the coding tree, which constrains the application of the MIP mode in the coding unit below the node (i.e., within the spatial region). The method 1200 can be embodied by a device such as a configured FPGA, ASIC, or ASSP. In addition, the method 1200 can be performed by the video encoder 114 under the execution of the processor 205. Therefore, the method 1200 can be stored on a computer-readable storage medium and / or in the memory 206. The method 1200 is called for each node (region) in the candidate coding tree. The method 1200 begins at a region area test step 1210.

[0158] At the region area test step 1210, the video encoder 114, under the execution of the processor 205, determines the available split options (using the Figure 5 The area occupied by a candidate node in the coding tree is tested according to the splitting options described. In the context of the described arrangement, a region refers to a node in the coding tree that can be further split before the splitting stops, thereby forming a CU. A region need not correspond to the direct parent region of the current coding unit, that is, the parent region for setting the MIP mode memory access bandwidth budget can be multiple nodes above the current node in the coding tree. For example, a region of 512 luma samples can be split into many small coding units, for example with sizes such as 4×4, 4×8, 8×4, etc. As the encoder search proceeds, different candidate nodes resulting from the candidate split are tested. Within each candidate node, various prediction modes for the resulting coding unit are tested. When the area in luma samples occupied by the region does not meet the threshold, the region area test evaluates to true, and control in the processor 205 proceeds from step 1210 to the set budget step 1220. When the area in luma samples occupied by the region meets the threshold, the region area test evaluates to false, and control in the processor 205 proceeds from step 1210 to the generate split step 1230. In one implementation, as Figures 11 to 16 As described in the example of , the threshold is 512 samples, and the threshold is met when the region has an area greater than 512 samples. Other thresholds may be used as described below. The threshold is typically predetermined based on the required performance of the encoder 114 and the decoder 134.

[0159] At a set budget step 1220, the video encoder 114, under execution of the processor 205, sets a budget for a region corresponding to the area of ​​the current node in the coding tree. The budget can be used for any child nodes (sub-regions) within the current region, which are generated by splitting the current region into smaller regions. For all regions less than or equal to 512 luma samples, the budget can be considered in the application of the MIP mode when the coding unit prediction mode is being evaluated. The area of ​​512 luma samples can be subjected to, for example, a ternary split, resulting in coding units of areas 128, 256, and 128 luma samples. The coding unit of area 128 luma samples can have a size of 8×16 or 16×8, in which case the worst-case memory bandwidth limit may be reached, resulting in prohibiting the use of the MIP mode for other coding units in the region, including those resulting from further subdivision of the region. When the area of ​​512 luma samples is binary split into two coding units of 256 luma sample areas (e.g., 16×16), the worst-case memory bandwidth limitation is not reached, and there is no constraint on the use of MIP modes for the resulting CUs. However, further partitioning each 256 sample area region into smaller CUs (i.e., CUs of sizes 16×8, 8×16, 8×8, 8×4, 4×8, and 4×4) may result in constraining the use of MIP modes for later resulting CUs due to budget consumption by earlier resulting CUs.

[0160] The budget can be expressed as a maximum word read per 4×4 block area, since 4×4 is the size of the smallest luma coding unit, and all other coding units are integer multiples of this size, so all regions are also multiples of this size. The budget can be 40 word reads for a 4×4 block, which corresponds to (512÷(4×4))*40=1280 word reads for an area of ​​512 luma samples, as the maximum number of permitted read operations in a 512 luma sample area. Whenever MIP mode is used for a CU, the required number of reads is deducted from the budget, and when the budget is insufficient to further use the MIP mode, other CUs are prohibited from using the MIP mode, as described with respect to Fig.13 and 14As described. In this way, a larger sized CU can use the MIP mode, for example, a single CU of an area of ​​512 samples or a pair of CUs of an area of ​​256 samples can use the MIP mode without introducing any restrictions from the constraints. When the coding tree decomposes the area of ​​512 samples into a larger number of smaller CUs, the consumption of the budget may prohibit other CUs in the area from using the MIP mode. A 512 sample area with a budget of 40 words per 4×4 results in a total budget of 512÷(4×4)*40=1280 words for the area. When the 512 sample area is partitioned into four CUs of size 8×16 or 16×8, two CUs consume 576 words or 1152 words per CU using the MIP mode. The remaining budget for the 128-word read is not enough to enable the other two CUs in the area to use the MIP mode. Control in the processor 205 proceeds from step 1220 to step 1230 of generating the split.

[0161] In step 1230 of generating splits, the block partitioner 310 generates a set of candidate splits for the current node in the coding tree under the execution of the processor 205. Figure 5 , and the associated syntactic elements in Figure 6 is shown in and Fig. 7A and Figure 7B exemplified in . In the case where "no split" is generated (i.e., 510), coding units are generated later. In the case where other types of splits (i.e., 512-520) are generated, additional nodes in the coding tree are generated according to the respective splits. When the "no split" case is generated on a later traversal of method 1200, method 1200 is repeated to generate coding units for the additional nodes. Method 1200 is repeated sequentially for each node of the CTU. Thus, all splits of the CTU are determined. In addition, the recursive nature of this generation of splits within a split enables searching for all possible coding units in a given CTU within constraints on the minimum CU size and deep recursion constraints (which can limit the number of recursions using binary, ternary, and quadtree splits). Control in processor 205 proceeds from step 1230 to step 1240 where a CU is generated.

[0162] At step 1240 of generating CU, the block partitioner 310, under execution of the processor 205, generates a candidate CU for each case where "no split" is generated at step 1230 of generating split. At step 1240, the prediction mode of the candidate CU is yet to be determined, but the prediction mode of the preceding CU in the Z-order scan is known, although the final mode has not yet been selected. Therefore, neighboring (in terms of position in the CU) reference samples can be used for intra-frame prediction from a neighboring CU (possibly from the same split operation as the split containing the current CU), or from a neighboring region obtained from a different parent region, or from a completely different CTU. Step 1240 effectively splits the region of the coding tree into coding blocks, each coding block including information such as about Figure 3 The method 1200 terminates at step 1240, where control in the processor 205 returns to the method 1100, where the prediction mode of the generated CU is determined.

[0163] Fig.13 A method 1300 for determining a coding unit as implemented at step 1130 is shown. Method 1300 involves determining a prediction mode for a coding unit as generated by performing method 1200. Prediction modes include intra prediction, inter prediction, and MIP mode (based on a budget for using the MIP mode in a region containing the current coding unit). The budget established in method 1200 is used to control whether the MIP mode is tested for the current CU. Whenever a CU in a region has a worst-case block size (as described above with respect to cases AC), a portion of the budget is consumed, depending on the number of codewords in the CU. Once the budget is consumed, other coding units in the region are not searched for potential use of the MIP mode, i.e., a call to method 1300 only tests the MIP mode when an earlier call to method 1300 has not yet exhausted the memory access bandwidth budget set to apply to a common parent node in the coding tree.

[0164] The region does not need to correspond to the direct parent region of the current coding unit, that is, the parent region for which the MIP mode memory access bandwidth budget is set can be multiple nodes above the current node in the coding tree. For example, a region of 512 luma samples can be split into many small coding units, for example, with sizes such as 4×4, 4×8, 8×4, etc. For each CU, the use of the MIP mode is constrained by the remaining budget of the region of 512 luma samples. Method 1300 can be implemented by the mode selector 386, or partially by the module 3960. Method 1300 can be embodied by a device such as a configured FPGA, ASIC, or ASSP. In addition, method 1300 can be performed by the video encoder 114 under the execution of the processor 205. Therefore, method 1300 can be stored on a computer-readable storage medium and / or in the memory 206. Method 1300 starts at step 1310 of testing the intra-frame prediction mode.

[0165] At step 1310 of testing intra prediction modes, the video encoder 114, under execution of the processor 205, tests "conventional" intra prediction modes (i.e., DC, planar, and angular intra prediction modes) for potential use in encoding the current coding unit. Typically, a Lagrangian optimization is performed to select the best intra prediction mode among the available intra prediction modes for the CU. The application of secondary transforms (i.e., 330 and 344) is also tested, as are different types of primary transforms (DCT-2, DCT-8, DST-7) including transform skipping situations. Control in the processor 205 proceeds from step 1310 to step 1320 of testing inter prediction modes.

[0166] At step 1320 of testing the inter prediction mode, the video encoder 114, under the execution of the processor 205, tests various motion vectors to generate an inter prediction PU. When evaluating the use of inter prediction, a motion vector is selected from a set of candidate motion vectors. The candidate motion vector is generated according to a search pattern. When the extracted reference block of the candidate motion vector under evaluation is subjected to distortion testing, the application of prohibited chroma splitting in the coding tree is taken into account. If splitting is prohibited in chroma and splitting is allowed in luminance, the resulting luminance CB can use inter prediction. Motion compensation is applied only to the luminance channel, so the distortion calculation takes into account luminance distortion without considering chroma distortion. Since motion compensation is not performed in the chroma channel when chroma splitting is prohibited, chroma distortion is not considered. For chroma, the distortion caused by the considered intra prediction mode and the encoded chroma TB (if any) is taken into account. When considering both luminance and chroma, the inter prediction search can first select a motion vector based on luminance distortion and then "refine" the motion vector by also considering chroma distortion. Refinement typically takes into account small changes in motion vector values, such as sub-pixel displacements, etc. The particular motion vector may be a motion vector generated by a 'merge mode', whereby the motion vector of the current CU is derived from the motion vector from a neighboring CU. The merge mode is expressed more compactly in the bitstream syntax than other motion vectors that may require a 'motion vector delta' to be signaled relative to the selected 'motion vector predictor' application. The motion vector predictor is typically derived from spatially or temporally neighboring CUs. For intra-coded slices (e.g., the first frame of a sequence of frames), inter prediction is not available and step 1320 is not performed. Control in the processor 205 proceeds from step 1320 to step 1330 of an in-budget test.

[0167] At step 1330 of testing within budget, the video encoder 114, under execution by the processor 205, tests whether the MIP mode memory access bandwidth budget is applicable to the current CU. The test performed at 1330 determines whether the current CU is contained within the 512 luma sample region for which the MIP mode budget was established at step 1220. If the current CU is larger than the 512 luma sample region, then there are no applicable budget constraints. Therefore, the CU is not subject to further constraints when using the MIP mode, and control in the processor 205 proceeds to step 1340 of testing the MIP mode (1330 is "true"). If the size of the current CU is equal to or less than 512 luma samples, then the budget required for the MIP mode will be used for the current CU (as referenced). Figure 81340).

[0168] At step 1340 of testing MIP modes, the mode selector 386 tests various MIP modes to determine the best MIP mode for predicting the CU among the available MIP modes for the current CU size. As with the test intra prediction mode at step 1310, a Lagrangian optimization may be performed to weigh the distortion against the coding cost of the tested MIP modes and their associated residuals. Control in the processor 205 proceeds from step 1340 to step 1350 of selecting a mode.

[0169] At step 1350 of selecting a mode, the mode selector 386 selects the final mode of the CU from the candidates obtained by steps 1310, 1320 and 1350 under the execution of the processor 205. The method 1300 terminates when the control in the processor 205 returns to the state of the method 1100.

[0170] Fig.14 A method 1400 for encoding a coding unit of a coding tree of a CTU in a video bitstream 115 is shown as implemented at step 1140. The method 1400 may be embodied by a device such as a configured FPGA, ASIC, or ASSP. Additionally, the method 1400 may be performed by the video encoder 114 (e.g., by the entropy encoder 338) under execution by the processor 205. Thus, the method 1400 may be stored on a computer-readable storage medium and / or in the memory 206. The method 1400 begins at step 1410 of encoding a prediction mode.

[0171] At step 1410 of encoding prediction mode, the entropy encoder 338, under execution of the processor 205, encodes a flag using a context encoding bin indicating the use of intra prediction (including both conventional intra prediction mode use or MIP mode use) or inter prediction (as determined at step 1350 and represented by the prediction mode 387). The encoding of the flag at step 1410 does not distinguish between conventional intra prediction and MIP mode. The distinction between conventional intra prediction and MIP intra prediction (when applicable) is encoded at step 1430 of encoding MIP mode flag. Control in the processor 205 proceeds from step 1410 to step 1420 of budget test.

[0172] At a budget test step 1420, the video encoder 114, under execution by the processor 205, tests whether the MIP mode memory access bandwidth budget is applicable to the current CU. The test determines whether the current CU is contained within the 512 luma sample region for which the MIP mode budget was established at step 1220. If the current CU is contained within an area greater than 512 luma sample regions, then there are no applicable budget constraints and the CU is not further constrained when using the MIP mode. Control in the processor 205 proceeds to a step 1430 of encoding the MIP mode flag ("true" at 1420).

[0173] If the current CU is contained in a region whose size is equal to or less than 512 luma samples, it will be as in reference Figure 8 The budget required to use the MIP mode for the current CU described is compared with the remaining budget for the region. If there is not enough budget available to apply the MIP mode to the current CU, control in the processor 205 proceeds from step 1420 to step 1440 of encoding the TB (step 1420 is "false"). However, if there is enough budget available to apply the MIP mode to the current CU ("true" at 1420), control in the processor 205 proceeds from step 1420 to step 1430 even if the current CU is in a region less than or equal to 512 samples. The operation of step 1420 corresponds to the operation of step 1330, and therefore, for the CU for which the MIP mode is searched in method 1300, only the MIP flag is encoded in method 1400.

[0174] At step 1430 of encoding the MIP mode flag, the entropy encoder 338, under execution by the processor 205, encodes in the bitstream 115 a context encoding bin indicating whether the MIP mode as determined at step 1350 is selected. The context used to encode the bin is described with reference to step 1420. In the case where the MIP mode is selected, the entropy encoder 338 also encodes in the bitstream 338 the selection of which particular MIP mode to use. Instead of using a selection between a "maximum probability mode" and the remaining modes to encode the MIP mode, a truncated binary codeword may be used. Using a truncated binary codeword avoids the necessity of deriving a list of maximum probability modes (including a potential table lookup to derive the maximum probability mode from an adjacent angular intra-frame predicted CU). Control in the processor 205 proceeds from step 1430 to step 1440 of encoding the TB.

[0175] At step 1440 of encoding TB, the entropy encoder 338, under the execution of the processor 205, encodes the residual coefficients of the TB associated with the current CU in the bitstream. In general, the flag of each TB signals the presence of at least one significant coefficient, and the coefficients are encoded one by one according to a scanning pattern that advances from the last significant coefficient position back to the DC (upper left) coefficient position. Then, the method 1400 terminates and the control in the processor 205 returns to the method 1100.

[0176] Fig.15 A method 1500 is shown for decoding coding units and transform blocks of an image frame from a video bitstream 133. The method 1500 may be embodied by a device such as a configured FPGA, ASIC, or ASSP. Additionally, the method 1500 may be performed by the video decoder 134 under execution by the processor 205. Thus, the method 1500 may be stored on a computer-readable storage medium and / or in the memory 206. The method 1500 begins at step 1510 of segmenting a frame into CTUs.

[0177] At step 1510 of dividing the frame into CTUs, the video decoder 134 divides the current frame of the frame data 133 (to be decoded) into a CTU array under the execution of the processor 205. Decoding of the CTUs resulting from the division begins. Control in the processor 205 proceeds from step 1510 to step 1520 of decoding the coding unit.

[0178] At step 1520 of decoding the coding unit, the entropy decoder 420, under the execution of the processor 205, performs the following steps according to the reference Figure 5 -7 The coding tree described in step 1500 decodes the split flag from the bitstream 133. Decoding the split flag allows step 1520 to operate to determine the size and location of the CU within the CTU based on the coding tree of the CTU. Progress of method 1300 involves iteration on step 1520, resulting in a traversal on the coding tree of the CTU (each CU is decoded). Control in processor 205 passes from step 1520 to step 1530 of decoding the coding unit.

[0179] At step 1530 of decoding the coding unit, the entropy decoder 420 decodes the coding unit from the bitstream 133 under the execution of the processor 205. Step 1530 calls the following Fig.16 The method 1600 is described to decode a CU. Control in the processor 205 proceeds from step 1530 to step 1540 for a final coding unit test.

[0180] At the last coding unit test step 1540, the processor 205 tests whether the current coding unit is the last coding unit in the CTU as determined from the decoded split flag at step 1520. If the current coding unit is the last in the CTU ("yes" at step 1540), control in the processor proceeds to the last CTU test step 1550. If the current coding unit is not the last in the coding tree of step 1520 ("no" at step 1540), the next coding unit in the coding tree of step 1520 is selected for decoding, and control in the processor 205 proceeds to step 1520.

[0181] At the last CTU test step 1550, the processor 205 tests whether the current CTU is the last CTU in the slice or frame. If the current CU is not the last ("No" at step 1550), the video decoder 134 advances to the next CTU in the frame or slice, and control in the processor 205 advances from step 1550 back to step 1520 to continue processing the remaining CTUs in the frame. If the CTU is the last in the frame or slice, step 1550 returns "Yes" and the method 1500 terminates. As a result of the method 1500, the entire image frame is decoded into a sequence of CTUs from the bitstream.

[0182] Fig.16 A method 1600 for decoding coding units from a video bitstream 133 is shown as implemented at step 1530. The method 1600 may be embodied by a device such as a configured FPGA, ASIC, or ASSP. Additionally, the method 1600 may be performed by the video decoder 134 under execution by the processor 205. Thus, the method 1600 may be stored on a computer-readable storage medium and / or in the memory 206. The method 1600 begins at a decode pred_mode flag step 1602.

[0183] At the decode pred_mode flag step 1602, the entropy decoder 420 decodes the context coding bin to determine whether the current coding unit uses inter prediction or intra prediction (including MIP mode). Control in the processor 205 continues from step 1602 to an inter prediction test step 1604. Step 1604 is performed to determine whether inter prediction is used from the pred_mode flag. If the current coding unit uses inter prediction (the inter prediction test step 1604 is "true"), control in the processor 205 continues from step 1602 to a perform inter prediction step 1606. Inter prediction is performed at step 1606, so that the reference block (434) is extracted and filtered to generate a PU, and then the TBs of each color channel are decoded and the TBs (1690) are added to the PU to decode the CU. Control in the processor 205 continues from step 1606 to a step 1690 of adding TBs.

[0184] If the current coding unit uses intra prediction or MIP mode (step 1604 is "false"), control on the processor 205 continues to step 1610 of the MIP mode budget test. Step 1610 of the MIP mode budget test is performed to determine whether the MIP mode is available for the current CU based on the budget. The budget corresponds to the budget described with reference to step 1330 or 1420. In the described example, if the area containing the current CU is larger than the area threshold (512 luma samples), there is no applicable budget constraint and the MIP mode is available (step 1610 is "true"). Control in the processor 205 proceeds from step 1610 to step 1620 of decoding the MIP mode flag.

[0185] If at step 1610, the area of ​​the region containing the current CU is equal to or less than the threshold (512 luma samples), then Figure 8 The budget required to use the MIP mode for the current CU is compared to the remaining budget for the region. If sufficient budget is available to apply the MIP mode to the current CU, the MIP mode is available (step 1610 is "true"), and the method 1600 continues to step 1620. If sufficient budget is not available, the MIP mode is not available for the current CU (the MIP mode budget test step 1610 is "false"), and control in the processor 205 continues to step 1612 of decoding the intra prediction mode.

[0186] At step 1612, conventional intra prediction (DC, planar or angular) is used, starting from the decoded intra prediction mode. Control in the processor 205 continues from step 1612 to step 1614 where intra prediction is performed, and step 1614 operates to perform intra prediction. Control in the processor 205 continues from step 1614 to step 1690 where TB is added.

[0187] In the execution of step 1620, the MIP flag is decoded by the entropy decoder 420 using a single context encoding bin. The bin uses one context, i.e., no context selection is required, such as this depends on the use of MIP mode or block size for neighboring CUs or other parameters that can be used to select one context for decoding a single bin from multiple possible contexts. Control in the processor continues from step 1620 to a MIP mode selection test step 1630.

[0188] Step 1630 operates to determine whether MIP mode is selected. If the decoded MIP mode flag indicates that MIP mode is not used (MIP mode selection test step 1630 is "false"), control proceeds to step 1612 to decode the block using one of the conventional intra prediction modes. If the decoded MIP mode flag indicates that MIP mode is used for the CU (step 1630 is "true"), control in the processor 205 operates to continue to step 1640 to decode the MIP mode.

[0189] In the execution of step 1640, the MIP mode is decoded from the bitstream 133. Control in the processor 205 continues from step 1640 to a read matrix coefficients step 1650. At step 1650, the decoded MIP mode is used to read a set of matrix coefficients 481 from the matrix coefficient memory 486. By limiting the availability of the MIP mode for use at step 1610 using a budget, the matrix coefficient memory 486 provides a reduction in the maximum memory bandwidth consumption required for the matrix coefficients 481.

[0190] From step 1650, method 1600 continues to step 1660 of filtering neighboring samples. At step 1660, neighboring reference samples 464 are filtered. From step 1660, method 1600 continues to step 1670 of matrix multiplication. Step 1600 may be implemented by module 482, for example, using matrix coefficients 481 and samples 464. Similar to Figure 8 In example 828 , at step 1670 , the filtered reference samples and the matrix coefficients 481 are multiplied.

[0191] From step 1670, method 1600 proceeds to step 1680 of interpolating PBs. The sparse blocks (i.e., 830) determined from the execution of step 1670 are used to fill the PBs (i.e., 483) using the interpolation operation at step 1680. From step 1680 of interpolating PBs, method 1600 proceeds to step 1690 of adding TBs. The decoded residual is used to generate a TB, which is added to PB 483 at step 1690 to decode the CU. The coding unit is decoded using the prediction block generated at step 1690. Method 1600 terminates when step 1690 is completed.

[0192] In performing method 1600, the video decoder 134 implements support for matrix intra prediction with a constraint on the worst-case memory bandwidth for extracting matrix coefficients. The constraints described do not overly restrict the selection of MIP modes in terms of affecting errors compared to the case where no constraints are actually used. The operation of not overly restricting the selection is because typically only 20% of the CUs encoded with MIP modes have a determined statistical probability of having a left or upper neighbor that also uses the MIP mode. There are boundaries on areas larger than the area used as the granularity for establishing the memory access budget (i.e., 512 luma samples) and boundaries on CTUs included in the 20% of the measurements found.

[0193] exist Figures 11 to 16 In the example implementation of , whether to use MIP mode is based on constraints. For example, Fig.12 In the embodiment, the constraint is implemented based on the area of ​​the region of the current CU, and whether the matrix intra prediction flag is encoded or decoded depends on whether the area of ​​the region of the current CU meets the threshold area. In the described implementation, (i) if the region meets the threshold, whether the matrix intra prediction flag is encoded depends on the area of ​​the region, or (ii) as described with respect to steps 1330, 1420 and 1610, if the area of ​​the region does not meet the threshold, whether the matrix intra prediction flag is encoded depends on the budget of the region. Therefore, whether the matrix intra prediction flag is encoded (or decoded) depends at least on whether the area of ​​the region meets the threshold area.

[0194] exist Figures 11 to 14 In the implementation described in , as described with respect to steps 1420 and 1430, the MIP mode flag is encoded in the bitstream 115 only when the budget test is satisfied. Fig.16 When decoding the bitstream in , the MIP mode flag is decoded only when the MIP mode budget test step returns true. In other words, the matrix intra prediction flag is decoded for a CU only when matrix intra prediction is used.

[0195] In an alternative arrangement of the video encoder 114 and the video decoder 134, as shown in reference Fig.12 and 13 As described above, the search for MIP mode is limited. However, in an alternative arrangement, signaling of the MIP flag is included in the bitstream for each CU regardless of the state of budget consumption of each CU. In other words, the matrix intra prediction flag is decoded for the CU regardless of whether matrix intra prediction is used. In an alternative arrangement, in the video encoder 114, step 1420 is omitted and control proceeds to step 1430 ("true" at step 1420), such as using Fig.14Indicated by the dashed arrow 1411 in . In an alternative arrangement, the MIP mode flag is included in all cases of step 1430, but if the MIP mode is not selected at step 1350, it can be set to zero. Similarly, in the video decoder 134, step 1610 is omitted and control proceeds to step 1620 (in fact, step 1610 always returns "true"). The MIP mode flag is decoded at step 1620, and if the MIP mode is not selected at step 1350 of encoding the bitstream, the MIP mode flag is zero. The additional burden of decoding whether to encode the MIP flag of each CU in the entropy encoder 338 and the entropy decoder 420 is avoided. Although the bitstream now includes the MIP mode flag of each CU, the search as performed in methods 1200 and 1300 ensures that the memory bandwidth for reading the matrix coefficients 481 from the coefficient memory 486 in the video decoder 134 is still limited, thereby reducing the necessary supply of resources for dealing with this (constrained) worst-case use of the MIP mode.

[0196] The above article about Figures 12 to 17 The described implementation applies constraints on the use of MIP modes based on the area of ​​the region containing the CU. In yet another arrangement of the video encoder 114 and the video decoder 134, Fig.12 The constraints implemented at prohibit the use of matrix intra prediction for block sizes corresponding to the worst-case memory bandwidth rather than based on the area of ​​the region. In practice, matrix intra prediction (and the matrix intra prediction flag for encoding or decoding) is used based on the size of each coding block. Block sizes of 4×4, 4×8, 8×4, 8×8, 8×16, and 16×8 result in the highest access density of 80 words per 4×4 block (e.g., as measured relative to a 4×4 luma sample block). Some block sizes in the previous set of block sizes have a worst-case of 72 words per 4×4 block, however for resource provisioning purposes this can be treated in the same category as the case of 80 words per 4×4 block. Prohibiting the worst-case block size establishes a worst-case of 40 words per 4×4 block (some block sizes have a worst-case of 36 words per 4×4 block).

[0197] Fig.17 A method 1700 for determining a coding unit is shown as implemented in step 1130. Method 1700 provides an alternative to method 1300 for constraining implementations involving prohibiting the use of matrix intra prediction for block sizes corresponding to worst-case memory bandwidth. Method 1700 involves determining a prediction mode for a coding unit as generated by performing method 1200. Prediction modes include intra prediction, inter prediction, and MIP mode (according to the allowed use of MIP mode within a region containing the current coding unit). Method 1700 begins at step 1710 of testing an intra prediction mode.

[0198] At step 1710 of testing intra prediction modes, the video encoder 114, under execution by the processor 205, tests “normal” intra prediction modes (i.e., DC, planar, and angular intra prediction modes) for potential use in encoding the current coding unit, and operates in the manner described for step 1310. Control in the processor 205 passes from step 1710 to step 1720 of testing inter prediction modes.

[0199] At step 1720 of testing inter prediction mode, the video encoder 114, under execution of the processor 205, tests various motion vectors to generate an inter-predicted PU. When evaluating the use of inter prediction, a motion vector is selected from a set of candidate motion vectors. Step 1720 operates in the same manner as step 1320. Control in the processor 205 proceeds from step 1720 to step 1730 of MIP allowable size testing.

[0200] At step 1730 of the MIP allowable size test, the video encoder 114, under execution by the processor 205, tests whether the CU has a size that allows the MIP mode. In some implementations, the MIP mode is allowable for all sizes except the worst case sizes of 4×4, 4×8, 8×4, 8×8, 8×16, and 16×8. If the current CU is an allowable size (“true” at step 1730), control in the processor 205 proceeds from step 1730 to step 1740 of testing the MIP mode. If the current CU is a prohibited size (“false” at step 1730), control in the processor 205 proceeds from step 1730 to step 1750 of selecting a mode.

[0201] At step 1740 of testing the MIP mode, the mode selector 386 tests various MIP modes to determine the best MIP mode for predicting the current CU among the MIP modes available for the size of the CU. Step 1740 operates in the same manner as step 1340, and the processor 205 proceeds from step 1740 to step 1750 of selecting the mode.

[0202] At step 1750 of selecting a mode, the mode selector 386 selects the final mode of the CU from the candidates obtained by steps 1710, 1720 and 1750 under the execution of the processor 205. The method 1700 terminates when the control in the processor 205 returns to the state of the method 1100.

[0203] Fig.18A method 1800 for encoding a coding unit of a coding tree of a CTU in a video bitstream 115 as implemented at step 1140 in an implementation where a constraint involves prohibiting the use of matrix intra prediction for a block size corresponding to a worst-case memory bandwidth is shown. The method 1800 may be embodied by a device such as a configured FPGA, ASIC, or ASSP. Additionally, the method 1800 may be performed by the video encoder 114 under execution by the processor 205. Thus, the method 1800 may be stored on a computer-readable storage medium and / or in the memory 206. The method 1800 begins at step 1810 of encoding a prediction mode.

[0204] At step 1810 of encoding prediction mode, the entropy encoder 338, under execution of the processor 205, encodes a flag using a context encoding bin indicating the use of intra prediction (including both conventional intra prediction mode use or MIP mode use) or inter prediction as determined at step 1750. Control in the processor 205 proceeds from step 1810 to step 1820 of MIP allowable size test.

[0205] At step 1820 of the MIP allowable size test, the video encoder 114 tests whether the current CU is an allowable size under the execution of the processor 205. The allowable size and the prohibited size are the same as step 1730 of the method 1700. If the current CU is an allowable size, control in the processor 205 proceeds to step 1830 of encoding the MIP mode flag ("true" at 1820).

[0206] If the current CU is not of an allowable size (“false” at step 1820 ), control in the processor 205 proceeds from step 1820 to step 1840 where a TB is encoded.

[0207] At step 1830, encode MIP mode flag, the entropy encoder 338, under execution by the processor 205, encodes in the bitstream 115 a context coding bin indicating whether the MIP mode as determined at step 1350 is selected. Step 1830 operates in the same manner as step 1430. Control in the processor 205 passes from step 1830 to step 1840, which encodes TBs.

[0208] At step 1840 of encoding TB, the entropy encoder 338, under execution of the processor 205, encodes the residual coefficients of the TB associated with the current CU in the bitstream. Step 1840 operates in a similar manner to step 1440. Then, the method 1800 terminates and control in the processor 205 returns to the method 1100.

[0209] Fig.19A method 1900 for decoding coding units from a video bitstream 133 is shown as implemented at step 1530 for an implementation where the constraint involves a worst-case block-prohibited MIP mode. The method 1900 may be embodied by a device such as a configured FPGA, ASIC, or ASSP. Additionally, the method 1900 may be performed by the video decoder 134 under execution by the processor 205. Thus, the method 1900 may be stored on a computer-readable storage medium and / or in the memory 206. The method 1900 begins at step 1902 of decoding a pred_mode flag.

[0210] At step 1902 of decoding the pred_mode flag, the entropy decoder 420 decodes the context coding bin to determine whether the current coding unit uses inter prediction or intra prediction (including MIP mode). If the current coding unit uses inter prediction (step 1904 of the inter prediction test is "true"), control in the processor continues to step 1906 of inter prediction. Inter prediction is performed at step 1606, so that the reference block (434) is extracted and filtered to generate a PU, and then the TBs of each color channel are decoded and the TBs (1690) are added to the PU to decode the CU. Similarly, control in the processor 205 continues from step 1906 to step 1990 of adding TBs.

[0211] If the current coding unit uses intra prediction or MIP mode (step 1904 is "false"), control of the processor 205 continues to step 1910 of the MIP allowable size test. Step 1910 of the MIP allowable size test is performed to determine whether the MIP mode is allowable for the current CU size. The MIP allowable size is described with reference to step 1730 or 1820. If the current CU is an allowable size (step 1910 is "true"), control in the processor 205 proceeds from step 1910 to a decoding MIP mode flag step 1920. If the current CU is not an allowable size at step 1910 (step 1910 is "false"), control in the processor 205 continues to step 1912 of decoding the intra prediction mode.

[0212] At step 1912, a conventional intra prediction (DC, planar or angular) is decoded and the method 1900 continues to a decode intra prediction step 1914. Steps 1912 and 1914 operate in the manner described with respect to steps 1612 and 1614. Control in the processor 205 continues from step 1914 to step 1990 where a TB is added.

[0213] During execution of step 1920, the MIP flag is decoded. Control in processor 205 continues from step 1920 to step 1930 where a MIP mode selection test is performed.

[0214] Step 1930 operates to determine whether MIP mode is selected. If the decoded MIP mode flag indicates that MIP mode is not used (step 1930 of the MIP mode selection test is "false"), control proceeds to step 1912 to decode the block using one of the conventional intra prediction modes. If the decoded MIP mode flag indicates that MIP mode is used for the CU (step 1930 is "true"), control in the processor operates to continue to step 1940 of decoding MIP mode.

[0215] In the execution of step 1940, the MIP mode is decoded from the bitstream 133. Control in the processor 205 continues from step 1940 to a read matrix coefficient step 1950. At step 1950, the decoded MIP mode is used to read a set of matrix coefficients 481 from the matrix coefficient memory 486. As at step 1650, the matrix coefficients are read for each prediction block that is determined to use the MIP mode.

[0216] From step 1950, method 1900 continues to step 1960 of filtering neighboring samples. At step 1960, neighboring reference samples 464 are filtered. From step 1960, method 1900 continues to step 1970 of matrix multiplication. Similar to Figure 8 In example 828 , at step 1970 , the filtered reference samples and the matrix coefficients 481 are multiplied.

[0217] From step 1970, method 1900 continues to step 1980 of interpolating PBs. The sparse blocks (i.e., 830) determined from the execution of step 1970 are used to fill the PBs (i.e., 483) at step 1980 using interpolation operations. From step 1980 of interpolating PBs, method 1900 continues to step 1990 of adding TBs. Similar to step 1690, the decoded residual is used to generate a TB, which is added to PB 483 at step 1990 to decode the CU. Method 1900 terminates when step 1990 is completed.

[0218] The modifications described for methods 1300, 1400, and 1600 may also be applied to methods 1700, 1800, and 1900, respectively.

[0219] As in method 1700, when block size is used as a criterion for the availability of MIP modes, there is no need to establish and update the memory budget, i.e., steps 1210 and 1220 are omitted. For some block sizes, removing the MIP mode, while easier to implement than budgeting memory access, results in lower compression performance due to the absence of MIP modes for many popular block sizes. Since the absence of 4×4 blocks means that the matrix coefficients of "set A" (a set of 18 matrix coefficients and bias values) do not need to be stored, the size of the coefficient memory (i.e., 392, 486) is reduced. Removing the MIP mode from 4×4, 4×8, 8×4, 8×8, 8×16, and 16×8 ("small blocks") also has the advantage that the feedback loop for the MIP mode (which is a relatively complex operation) does not need to support these small blocks.

[0220] In yet another arrangement, the use of MIP mode is prohibited for a subset of small blocks (as listed above). For example, at steps 1730, 1820 and 1910, MIP mode is prohibited only for 4×4 blocks, but MIP mode is allowed for all other block sizes, or MIP mode is prohibited for 4×4, 4×8 and 8×4 blocks, but MIP mode is allowed for all other block sizes. In other words, matrix intra prediction is not used, and depending on the implementation, if the size of the coding unit is one of the prohibited sizes, the matrix intra prediction flag is decoded. The worst-case memory bandwidth is not reduced compared to the complete lack of restrictions on the use of MIP mode, but the severity of the intra-frame reconstruction feedback loop is reduced due to the exclusion of these very small block sizes. The removal of 4×4 also eliminates the need to store the matrix coefficients associated with "Set A" from the coefficient memory (i.e., 392, 486). Alternatively, the prohibited block size set may be 4×4, 4×8, 8×4, and 8×8, in which case sets A and B are not present in the video encoder 114 or the video decoder 134. Removing sets A and B reduces memory consumption because the matrix coefficients associated with sets A and B are not needed, at the expense of reduced compression performance.

[0221] In yet another arrangement, the memory budget is still lower than 40 words per 4×4 luma sample area, for example 20 or even 10 words per 4×4 luma sample area. The budget is established on larger region sizes, such as at nodes corresponding to region sizes of 1024 or 2048 luma samples, etc. As with the above arrangement, later CUs within the limits are constrained in the availability of memory budget according to the use of MIP modes of earlier CUs within these regions. Further reductions in memory bandwidth are achieved at the expense of lower compression efficiency.

[0222] In yet another arrangement, a memory budget is established at steps 1220, 1330, 1420, and 1610 with an area size of 64 luma samples to apply to CUs of size 4×4, 4×8, and 8×4 (“small CU memory budget”). A separate memory budget is established with an area size of 512, but only applies to CUs of size greater than 8×8, specifically 8×16 and 16×8 CUs (“larger CU memory budget”). Both budgets are set at 40 words per 4×4 luma sample area. The budgets form an additive budget for the total matrix memory bandwidth, since the small CU memory budget only applies to CUs contained within an area of ​​64 luma samples, while the larger CU memory budget only applies to CUs greater than 64 luma samples.

[0223] Although the arrangement disclosed herein describes memory bandwidth in words per 4×4 memory region, it should be understood that access to memory may group words in some SIMD form to allow reading matrix coefficients without requiring excessive clock frequency of the associated memory. However, such wider memory is inherently costly, and matrix coefficients may be shared with other data in the same memory, so that access contention is reduced by limiting the use of MIP mode.

[0224] Limiting the use of MIP modes on a per-region basis to limit the worst-case memory bandwidth may introduce a bias towards earlier (further up or to the left) CUs in respective regions to use MIP modes, while later CUs in a given region cannot use MIP modes because the available budget for the region is consumed by CUs encountered earlier. This bias is not usually encountered because the distribution of CUs that select MIP modes is usually somewhat sparse.

[0225] The described arrangement implements constraints on when MIP modes may be used, thereby reducing computational complexity compared to allowing MIP modes to be implemented without constraints. Since the statistics of MIP mode selection in the unconstrained search will not generally trigger constraints imposed on MIP mode selection in the mode selector 386, the reduced complexity is achieved without a proportional reduction in coding efficiency. Thus, the worst case memory bandwidth is reduced without a commensurate loss in coding performance.

[0226] Industrial Applicability

[0227] The described arrangement is suitable for use in the computer and data processing industries and is particularly suitable for use in digital signal processing for encoding or decoding signals such as video and image signals, thereby achieving high compression efficiency.

[0228] The arrangement described herein allows residual encoding and decoding to use a trellis-based state machine that updates according to coefficient parity and selects context and quantizer for coefficients. The described arrangement allows a trellis-based state machine to be implemented without imposing excessive delays due to the sequential nature of state updates.

[0229] The foregoing describes only some embodiments of the present invention, and modifications and / or changes may be made to the present invention without departing from the scope and spirit of the present invention, wherein the embodiments are intended to be illustrative and not restrictive.

[0230] Citations of Related Patent Applications

[0231] This application claims the benefit of priority under 35 U.S.C. §119 to Australian patent application No. 2019204437 filed on June 24, 2019, the entire contents of which are hereby incorporated by reference for all purposes.

Claims

1. A method for generating a prediction sample of a target block in a coding tree unit of an image frame, the image frame having a predetermined chroma format, the method comprising: determining a plurality of blocks in the coding tree unit including the target block by decoding a split flag of a block split in the coding tree unit, wherein a horizontal ternary split can be used as the target block split in the coding tree unit; Decoding a matrix intra prediction flag of the target block, the matrix intra prediction flag indicating whether matrix intra prediction is used for the target block; In a case where the matrix intra prediction flag indicates that matrix intra prediction is used for the target block, decoding a matrix intra prediction mode of the target block, wherein a truncated binary code can be used for the matrix intra prediction mode; selecting a matrix for matrix intra prediction of the target block according to the matrix intra prediction mode; and generating the prediction samples by applying a multiplication of input samples based on samples neighboring the target block and a matrix selected according to the matrix intra prediction mode, wherein a 4:2:0 chroma format or a 4:2:2 chroma format can be used as the predetermined chroma format, The decoding of the matrix intra prediction flag depends on the width of the target block and the height of the target block, and In which, when a given area in the coding tree unit is split into four blocks each having a size of 16×8 and the target block is one of the four blocks, decoding of the matrix intra-frame prediction flag of the target block depends on the usage status of the matrix intra-frame prediction of two blocks adjacent to the target block.

2. A method for generating a prediction sample of a target block in a coding tree unit of an image frame, the image frame having a predetermined chroma format, the method comprising: determining a plurality of blocks in the coding tree unit including the target block by block splitting in the coding tree unit, wherein a horizontal ternary split can be used as the target block splitting in the coding tree unit; Encoding a split flag for splitting the target block; selecting a matrix for matrix intra prediction of the target block; generating the prediction sample by applying a multiplication of an input sample based on samples adjacent to the target block and the selected matrix; Encoding a matrix intra prediction flag of the target block, wherein the matrix intra prediction flag indicates whether matrix intra prediction is used for the target block; as well as In case matrix intra prediction is used for the target block, encoding a matrix intra prediction mode indicating the selected matrix, wherein a 4:2:0 chroma format or a 4:2:2 chroma format can be used as the predetermined chroma format, Among them, truncated binary codes can be used in the matrix intra prediction mode, The encoding of the matrix intra prediction flag depends on the width of the target block and the height of the target block, and In which, when a given area in the coding tree unit is split into four blocks each having a size of 16×8 and the target block is one of the four blocks, the encoding of the matrix intra-frame prediction flag of the target block depends on the usage status of the matrix intra-frame prediction of two blocks adjacent to the target block.

3. A device for generating a prediction sample of a target block in a coding tree unit of an image frame, the image frame having a predetermined chroma format, the device comprising: a determining unit configured to determine a plurality of blocks including the target block in the coding tree unit by decoding a split flag of a block split in the coding tree unit, wherein a horizontal ternary split can be used as the block split in the coding tree unit; A first decoding unit, configured to decode a matrix intra prediction flag of the target block, the matrix intra prediction flag indicating whether matrix intra prediction is used for the target block; a second decoding unit configured to decode a matrix intra prediction mode of the target block when the matrix intra prediction flag indicates that matrix intra prediction is used for the target block, wherein a truncated binary code can be used for the matrix intra prediction mode; A selection unit configured to select a matrix for matrix intra prediction of the target block according to the matrix intra prediction mode; and a generating unit configured to generate the prediction sample by applying a multiplication of an input sample based on samples adjacent to the target block and a matrix selected according to the matrix intra prediction mode, wherein a 4:2:0 chroma format or a 4:2:2 chroma format can be used as the predetermined chroma format, The decoding of the matrix intra prediction flag depends on the width of the target block and the height of the target block, and In which, when a given area in the coding tree unit is split into four blocks each having a size of 16×8 and the target block is one of the four blocks, decoding of the matrix intra-frame prediction flag of the target block depends on the usage status of the matrix intra-frame prediction of two blocks adjacent to the target block.

4. A device for generating a prediction sample of a target block in a coding tree unit of an image frame, the image frame having a predetermined chroma format, the device comprising: a determining unit configured to determine a plurality of blocks in the coding tree unit including the target block by block splitting in the coding tree unit, wherein a horizontal ternary splitting can be used as the target block splitting in the coding tree unit; A first encoding unit, configured to encode a split flag of the target block split; A selection unit configured to select a matrix for matrix intra prediction of the target block; a generating unit configured to generate the prediction sample by applying a multiplication of an input sample based on samples adjacent to the target block and the selected matrix; A second encoding unit, configured to encode a matrix intra prediction flag of the target block, wherein the matrix intra prediction flag indicates whether matrix intra prediction is used for the target block; as well as a third encoding unit configured to encode a matrix intra prediction mode indicating the selected matrix when matrix intra prediction is used for the target block, wherein truncated binary codes can be used for the matrix intra prediction mode, wherein a 4:2:0 chroma format or a 4:2:2 chroma format can be used as the predetermined chroma format, The encoding of the matrix intra prediction flag depends on the width of the target block and the height of the target block, and In which, when a given area in the coding tree unit is split into four blocks each having a size of 16×8 and the target block is one of the four blocks, the encoding of the matrix intra-frame prediction flag of the target block depends on the usage status of the matrix intra-frame prediction of two blocks adjacent to the target block.

5. A non-transitory computer-readable storage medium storing a program, the program causing a computer to execute a method of generating a prediction sample of a target block in a coding tree unit of an image frame, the image frame having a predetermined chroma format, the method comprising: determining a plurality of blocks in the coding tree unit including the target block by decoding a split flag of a block split in the coding tree unit, wherein a horizontal ternary split can be used as the target block split in the coding tree unit; Decoding a matrix intra prediction flag of the target block, the matrix intra prediction flag indicating whether matrix intra prediction is used for the target block; In a case where the matrix intra prediction flag indicates that matrix intra prediction is used for the target block, decoding a matrix intra prediction mode of the target block, wherein a truncated binary code can be used for the matrix intra prediction mode; selecting a matrix for matrix intra prediction of the target block according to the matrix intra prediction mode; and generating the prediction samples by applying a multiplication of input samples based on samples neighboring the target block and a matrix selected according to the matrix intra prediction mode, wherein a 4:2:0 chroma format or a 4:2:2 chroma format can be used as the predetermined chroma format, The decoding of the matrix intra prediction flag depends on the width of the target block and the height of the target block, and In which, when a given area in the coding tree unit is split into four blocks each having a size of 16×8 and the target block is one of the four blocks, decoding of the matrix intra-frame prediction flag of the target block depends on the usage status of the matrix intra-frame prediction of two blocks adjacent to the target block.

6. A non-transitory computer-readable storage medium storing a program, the program causing a computer to execute a method of generating a prediction sample of a target block in a coding tree unit of an image frame, the image frame having a predetermined chroma format, the method comprising: determining a plurality of blocks in the coding tree unit including the target block by block splitting in the coding tree unit, wherein a horizontal ternary splitting can be used as the block splitting in the coding tree unit; encoding a split flag for the block split; selecting a matrix for matrix intra prediction of the target block; generating the prediction sample by applying a multiplication of an input sample based on samples adjacent to the target block and the selected matrix; Encoding a matrix intra prediction flag of the target block, wherein the matrix intra prediction flag indicates whether matrix intra prediction is used for the target block; as well as In case matrix intra prediction is used for the target block, encoding a matrix intra prediction mode indicating the selected matrix, wherein a 4:2:0 chroma format or a 4:2:2 chroma format can be used as the predetermined chroma format, Among them, truncated binary codes can be used in the matrix intra prediction mode, The encoding of the matrix intra prediction flag depends on the width of the target block and the height of the target block, and In which, when a given area in the coding tree unit is split into four blocks each having a size of 16×8 and the target block is one of the four blocks, the encoding of the matrix intra-frame prediction flag of the target block depends on the usage status of the matrix intra-frame prediction of two blocks adjacent to the target block.

7. A computer program product comprising a program which, when executed by a computer, causes the computer to perform the method according to claim 1.

8. A computer program product comprising a program which, when executed by a computer, causes the computer to perform the method according to claim 2.

Images