Method, apparatus and system for encoding and decoding transformed blocks of video samples - Patents.com

Abstract

To provide a method, an apparatus, a system and a program including a computer-readable medium for encoding and decoding a transformed block of a video sample in digital video signal processing.SOLUTION: A method 1200 comprises decoding a single truncated unary binarisation from a bitstream, the single truncated unary binarisation being used for a horizontal transform and a vertical transform of a transform block of the image frame. The method then determines a type of the horizontal and vertical transform based on the decoded single truncated unary binarisation and decodes the transform block in the image frame by applying the determined type of horizontal and vertical transform to the transform block of the image.SELECTED DRAWING: Figure 12

Description

[Technical field]

[0001] The present invention relates generally to digital video signal processing, and in particular to a method, apparatus and system for encoding and decoding transformed blocks of video samples. The present invention also relates to a computer program product including a computer readable medium having recorded thereon a computer program for encoding and decoding transformed blocks of video samples. [Background technology]

[0002] Many applications for video coding currently exist, including applications for the transmission and storage of video data. Many video coding standards have also been developed, and others are currently under development. Recent developments in video coding standardization have led to the formation of a group called the "Joint Video Experts Team" (JVET). The Joint Video Experts Team (JVET) includes members of Study Group 16, Question 6 (SG16 / Q6) of the International Telecommunication Union's (ITU) Telecommunication Standardization Sector (ITU-T), known as the "Video Coding Experts Group" (VCEG), and members of the International Organization for Standardization / International Electrotechnical Commission Joint Technical Committee 1 / Subcommittee 29 / Working Group 11 (ISO / IEC JTC1 / SC29 / WG11), also known as the "Moving Picture Experts group" (MPEG).

[0003] The Joint Video Experts Team (JVET) analyzed the responses and issued a Call for Proposals (CfP) at its 10th meeting held in San Diego, USA. The submitted responses demonstrated video compression capabilities that significantly outperformed those of the current state-of-the-art video compression standard, i.e., "high efficiency video coding" (HEVC). Based on this outperformance, it was decided to initiate a project to develop a new video compression standard, to be named "versatile video coding" (VVC). VVC is expected to address the continuing demand for ever-higher compression performance, especially as video formats increase in capabilities (e.g., at higher resolutions and higher frame rates) and address the increasing market demand for service delivery over WANs where bandwidth costs are relatively high. At the same time, VVC must be implementable in modern silicon processes and provide an acceptable trade-off between achieved performance versus implementation costs (e.g., in terms of silicon area, CPU processor load, memory usage, and bandwidth).

[0004] Video data includes a sequence of frames of image data, each of which includes one or more color channels. Generally, there is one primary color channel and two secondary color channels. The primary color channel is commonly called the "luminance" channel, and the secondary color channels are commonly called the "chroma" channels. Video data is typically represented in the RGB (red-green-blue) color space, which has a high degree of correlation between the three respective components. The video data representation seen by an encoder or decoder often uses a color space such as YCbCr, which concentrates luminance in the Y (primary) channel and chroma in the Cb and Cr (secondary) channels. Furthermore, the Cb and Cr channels may be sampled at a lower rate compared to the luminance channel, e.g., half horizontally and half vertically, known as "4:2:0 chroma format".

[0005] VVC is a "block-based" codec in which a frame is divided into blocks and the blocks are processed in a particular order. For each block, a prediction of the block's contents is generated and a representation is formed of the difference (or "residual" in the spatial domain) between the prediction and the actual block contents seen as input to the encoder. The difference may be coded as a sequence of residual coefficients resulting from applying a forward linear transform, such as the Discrete Cosine Transform (DCT) or other transform, to a block of residual values. This linear transform is applied separably to the block of residual samples, i.e., a two-dimensional transform is performed in two passes, first the rows are transformed by applying a one-dimensional transform to each row, and then this result is transformed by applying a one-dimensional transform to each column to produce a block of residual coefficients that substantially decorrelates the residual samples. Furthermore, multiple transforms are available for use in this row and column processing, requiring signaling to indicate which transform should be used. The option to use different transforms within a block gives a compression advantage, but this is slightly reduced by the need to code an index to select which transform should be used horizontally and vertically. Efficient coding of this signaling is beneficial to the impact of this signaling overhead. Summary of the Invention

[0006] It is an object of the present invention to substantially overcome, or at least ameliorate, one or more disadvantages of existing arrangements.

[0007] One aspect of the present disclosure provides a method for decoding a transform block in an image frame from a bitstream, the method including the steps of: decoding a single truncated mismatched binarization from the bitstream, the single truncated mismatched binarization being used for horizontal and vertical transforms of a transform block of the image frame; determining types of the horizontal and vertical transforms based on the decoded single truncated mismatched binarization; and decoding the transform block in the image frame by applying the determined type of horizontal and vertical transforms to the transform block of the image.

[0008] Another aspect of the present disclosure provides a non-transitive computer-readable medium including a computer application program executable by a computing device, the computer performing a method for decoding a transform block in an image frame from a bitstream, the method including the steps of: decoding a single truncated yet binary resulting from the bitstream, the single truncated yet binary being used for horizontal and vertical transformations of a transform block of the image frame; determining a type of the horizontal and vertical transformation based on the decoded single truncated yet binary; and decoding the transform block in the image frame by applying the determined type of horizontal and vertical transformation to the transform block of the image.

[0009] Other aspects are also disclosed. [Brief description of the drawings]

[0010] At least one embodiment of the invention will now be described with reference to the following drawings and appendix. [Figure 1] FIG. 1 is a schematic block diagram illustrating a video encoding and decoding system. [Figure 2A] FIG. 2A forms a schematic block diagram of a general purpose computer system in which one or both of the video encoding and decoding systems of FIG. 1 may be implemented. [Figure 2B] FIG. 2B forms a schematic block diagram of a general purpose computer system capable of implementing either or both of the video encoding and decoding systems of FIG. [Diagram 3] FIG. 3 is a schematic block diagram showing functional modules of a video encoder. [Figure 4] FIG. 4 is a schematic block diagram showing the functional modules of a video decoder. [Diagram 5] FIG. 5 is a schematic block diagram illustrating an available division of a block into one or more blocks in a generic video coding tree structure. [Figure 6] FIG. 6 is a schematic diagram of a data flow for achieving permitted division of a block into one or more blocks in a generic video coding tree structure. [Figure 7] FIG. 7 is an example of dividing a coding tree unit (CTU) into several coding units (CUs). [Figure 8A] FIG. 8A is a diagram showing intra prediction modes. [Figure 8B] FIG. 8B is a table showing the mapping from transform settings to specific transforms by transform index. [Figure 8C] FIG. 8C is a table showing the mapping from intra prediction modes to horizontal and vertical transform settings for transform blocks. [Figure 9] FIG. 9 is a schematic block diagram illustrating an inverse transform module of the video encoder of FIG. 3 or the video decoder of FIG. [Figure 10A] FIG. 10A is a diagram illustrating the relationship between truncated unary syntax elements and transformation indexes for horizontally and vertically indexing transformations. [Figure 10B] FIG. 10B illustrates alternative relationships between truncated unary syntax elements and transformation indexes for horizontally and vertically indexing transformations. [Figure 11]FIG. 11 is a flow chart diagram of a method for performing a primary transform in accordance with an explicit transform selection to encode a transform block of residual coefficients into a bitstream. [Figure 12] FIG. 12 is a flow chart diagram of a method for decoding a transform block of residual coefficients from a bitstream via a linear inverse transform in accordance with an explicit transform selection. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] When referring to steps and / or features having the same reference numbers in any one or more of the accompanying drawings, those steps and / or features have the same function or operation for purposes of this description, unless a contrary intention appears.

[0012] FIG. 1 is a schematic block diagram illustrating functional modules of a video encoding and decoding system 100. The system 100 may utilize a coefficient scanning method to improve compression efficiency and / or achieve reduced implementation costs. The system 100 includes a source device 110 and a destination device 130. A communication channel 120 is used to communicate encoded video information from the source device 110 to the destination device 130. In some configurations, the source device 110 and the destination device 130 may comprise either or both of a respective mobile phone handset or a "smartphone," in which case the communication channel 120 is a wireless channel. In other configurations, the source device 110 and the destination device 130 may comprise video conferencing equipment, in which case the communication channel 120 is typically a wired channel, such as an Internet connection. Furthermore, the source device 110 and the destination device 130 may comprise any of a wide range of devices, including devices supporting over-the-air television broadcasts, cable television applications, Internet video applications (including streaming), and applications in which encoded video data is captured on some computer-readable storage medium, such as a hard disk drive in a file server.

[0013] As shown in FIG. 1, source device 110 includes a video source 112, a video encoder 114, and a transmitter 116. Video source 112 typically has a source of captured video frame data (shown as 113), such as an imaging sensor, a previously captured video sequence stored on a non-transitory recording medium, or video from a remote imaging sensor. Video source 112 may also be the output of a computer graphics card, for example, displaying the video output of an operating system and various applications running on a computing device, such as a tablet computer. Examples of source device 110 that may include an imaging sensor as video source 112 include smartphones, video cameras, professional video cameras, and network video cameras. Video encoder 114 converts (or "encodes") the captured frame data (indicated by arrow 113) from video source 112 into a bit stream (indicated by arrow 115), as will be further described with reference to FIG. 3. Bit stream 115 is transmitted by transmitter 116 as encoded video data (or "encoded video information") over communication channel 120. The bitstream 115 may also be stored in a non-transitory storage device 122, such as “flash” memory or a hard disk drive, until later transmitted over the communication channel 120, or instead of being transmitted over the communication channel 120.

[0014] Destination device 130 includes a receiver 132, a video decoder 134, and a display device 136. Receiver 132 receives encoded video data from communication channel 120 and passes the received video data as a bitstream to video decoder 134 (indicated by arrow 133). Video decoder 134 then outputs decoded frame data, indicated by arrow 135, to display device 136. Examples of display device 136 include a liquid crystal display, such as a cathode ray tube, a smartphone, a tablet computer, a computer monitor, or a standalone television set. It is also possible that the functionality of each of source device 110 and destination device 130 is implemented in a single device, examples of which include a mobile phone handset and a tablet computer.

[0015] Notwithstanding the above exemplary devices, each of the source device 110 and the destination device 130 may generally be configured within a general-purpose computing system through a combination of hardware and software components. Figure 2A shows such a computer system 200, including a computer module 201, input devices such as a keyboard 202, a mouse pointer device 203, a scanner 226, a camera 227 that may be configured as a video source 112, and a microphone 280, and output devices including a printer 215, a display device 214 that may be configured as a display device 136, and a speaker 217. An external Modulator-Demodulator (Modem) transceiver device 216 may be used by the computer module 201 to communicate to and from a communication network 220 via a connection 221. The communication network 220, which may represent the communication channel 120, may be a wide area network (WAN) such as the Internet, a cellular telecommunications network, or a private WAN. If the connection 221 is a telephone line, the modem 216 may be a conventional "dial-up" modem. Alternatively, the modem 216 may be a broadband modem where the connection 221 is a high-capacity (e.g., cable or optical) connection. A wireless modem may also be used for wireless connection to the communications network 220. The transceiver device 216 may provide the functionality of the transmitter 116 and the receiver 132, and the communications channel 120 may be embodied within the connection 221.

[0016] The computer module 201 typically includes at least one processor unit 205 and a memory unit 206. For example, the memory unit 206 can include a semiconductor random access memory (RAM) and a semiconductor read-only memory (ROM). The computer module 201 also includes several input / output (I / O) interfaces, including an audio-video interface 207 that couples to a video display 214, a speaker 217, and a microphone 280, an I / O interface 213 that couples to a keyboard 202, a mouse 203, a scanner 226, a camera 227, and optionally a joystick or other human interface device (not shown), and an interface 208 for an external modem 216 and a printer 215. The signal from the audio-video interface 207 to the computer monitor 214 is typically the output of a computer graphics card. In some implementations, the modem 216 may be integrated into the computer module 201, for example in the interface 208. The computer module 201 also has a local network interface 211, which allows the computer system 200 to be coupled via a connection 223 to a local area communication network 222, known as a local area network (LAN). As shown in Fig. 2A, the local communication network 222 can also be coupled to a wide network 220 via a connection 224, which usually includes a so-called "firewall" device or a device of similar functionality. The local network interface 211 can be an Ethernet TM Circuit Card, Bluetooth TM This may include a wireless or IEEE 802.11 wireless configuration, although many other types of interfaces may be implemented for the interface 211. The local network interface 211 may also provide the functionality of the transmitter 116, and the receiver 132 and the communication channel 120 may also be embodied in a local communications network 222.

[0017] The I / O interfaces 208 and 213 may provide either or both serial and parallel connectivity, the former typically implemented according to the Universal Serial Bus (USB) standard and having a corresponding USB connector (not shown). Storage 209 is provided and typically includes a hard disk drive (HDD) 210. Other storage devices such as floppy disk drives and magnetic tape drives (not shown) may also be used. An optical disk drive 212 is typically provided to serve as a non-volatile source of data. For example, portable memory devices such as optical disks (e.g., CD-ROM, DVD, Blu ray Disc™), USB-RAM, portable, external hard drives, and floppy disks may be used as suitable sources of data for the computer system 200. Typically, any of the HDD 210, optical drive 212, network 220, and 222 may be configured to operate as a video source 112 or as a destination for decoded video data to be stored for playback via display 214. The source device 110 and the destination device 130 of the system 100, or the source device 110 and the destination device 130 of the system 100, may be embodied in a computer system 200.

[0018] The components 205-213 of the computer module 201 typically communicate via an interconnected bus 204 and in a manner which results in a conventional mode of operation of the computer system 200 known to those skilled in the art. For example, the processor 205 is coupled to the system bus 204 using a connection 218. Similarly, the memory 206 and optical disk drive 212 are coupled to the system bus 204 by a connection 219. Examples of computers on which the above-described arrangements can be executed include IBM-PC's and compatibles, Sun SPARC stations, Apple Macs, and similar. TM or a similar computer system.

[0019] Where appropriate or necessary, the video encoder 114 and the video decoder 134, as well as the methods described below, may be implemented using a computer system 200, and the video encoder 114, the video decoder 134, and the methods described may be implemented as one or more software application programs 233 executable within the computer system 200. In particular, the video encoder 114, the video decoder 134, and the steps of the methods described are performed by instructions 231 (see FIG. 2B) in the software 233 executed within the computer system 200. The software instructions 231 may be formed as one or more code modules, each for performing one or more specific tasks. The software may also be divided into two separate parts, where a first part and a corresponding code module perform the methods described, and a second part and a corresponding code module manage a user interface between the first part and a user.

[0020] The software may be stored on a computer readable medium, including, for example, the storage devices described below. The software is loaded from the computer readable medium into the computer system 200 and thereafter executed by the computer system 200. Such software or a computer readable medium having a computer program recorded thereon is a computer program product. Use of the computer program product in the computer system 200 preferably results in an advantageous apparatus for implementing the video encoder 114, the video decoder 134, and the methods described.

[0021] The software 233 is typically stored on the HDD 210 or memory 206. The software is loaded into the computer system 200 from a computer-readable medium and executed by the computer system 200. Thus, for example, the software 233 may be stored on an optically readable disk storage medium (e.g., a CD-ROM) 225 that is read by the optical disk drive 212.

[0022] In some cases, the application programs 233 may be provided to the user encoded on one or more CD-ROMs 225 and read via the corresponding drive 212 or read by the user from the network 220 or 222. Furthermore, the software may also be loaded into the computer system 200 from other computer readable media. A computer readable storage medium refers to any non-transitory tangible storage medium that provides recorded instructions and / or data to the computer system 200 for execution and / or processing. Examples of such storage media include floppy disks, magnetic tapes, CD-ROMs, DVDs, Blu-ray Discs™, hard disk drives, ROMs or integrated circuits, USB memories, magneto-optical disks, or computer readable cards such as PCMCIA cards, whether such devices are internal or external to the computer module 201. Examples of transitory or non-tangible computer-readable transmission media that may also participate in providing software, application programs, instructions and / or video data or encoded video data to the computer module 401 include wireless or infrared transmission channels, as well as network connections to another computer or networked device, and the Internet or intranets, including email transmissions and information recorded on websites and the like.

[0023] A second portion of the application programs 233 and corresponding code modules described above may be executed to implement one or more graphical user interfaces (GUIs) that are rendered or otherwise represented on the display 214. Typically through manipulation of a keyboard 202 and mouse 203, a user of the application and computer system 200 may manipulate the interface in a functionally adaptable manner to provide control commands and / or input to an application associated with the GUI. Other forms of functionally adaptable user interfaces may also be implemented, such as an audio interface that utilizes speech prompts output via speaker 217 and user voice commands input via microphone 280.

[0024] Figure 2B is a detailed schematic block diagram of processor 205 and "memory" 234. Memory 234 represents a logical collection of all memory modules (including HDD 209 and semiconductor memory 206) accessible by computer module 201 of Figure 2A.

[0025] When the computer module 201 is first powered on, a power-on self-test (POST) program 250 is executed. The POST program 250 is typically stored in the ROM 249 of the semiconductor memory 206 of FIG. 2A. Hardware devices such as the ROM 249 that store software are sometimes referred to as firmware. The POST program 250 inspects the hardware in the computer module 201 to ensure that it functions properly, and typically checks the processor 205, memory 234 (209, 206), and basic input / output system software (BIOS) module 251 (usually also stored in the ROM 249) for correct operation. If the POST program 250 runs successfully, the BIOS 251 boots the hard disk drive 210 of FIG. 2A. When the hard disk drive 210 boots, a bootstrap loader program 252 resident on the hard disk drive 210 is executed via the processor 205. This loads the operating system 253 into the RAM memory 206, on which the operating system 253 begins to operate. Operating system 253 is a system level application executable by processor 205 and performs a variety of high-level functions including processor management, memory management, device management, storage management, software application interface, and a general-purpose user interface.

[0026] The operating system 253 manages the memory 234 (209, 206) to ensure that each process or application executing on the computer module 201 has enough memory to execute without colliding with memory allocated to another process. Additionally, the different types of memory available in the computer system 200 of FIG. 2A must be used appropriately so that each process can execute effectively. Thus, the aggregate memory 234 is not intended to indicate how specific segments of memory are allocated (unless otherwise noted), but rather is intended to provide a general view of memory accessible by the computer system 200 and how such segments are used.

[0027] As shown in FIG. 2B, the processor 205 includes a number of functional modules, including a control unit 239, an arithmetic logic unit (ALU) 240, and a local or internal memory 248, sometimes referred to as a cache memory. The cache memory 248 typically includes a number of storage registers 244-246 in a register section. One or more internal buses 241 operatively interconnect these functional modules. The processor 205 also typically has one or more interfaces 242 for communicating with external devices via the system bus 204, using a connection 218. The memory 234 is coupled to the bus 204 using a connection 219.

[0028] The application program 233 includes a sequence of instructions 231, which may include conditional branch and loop instructions. The program 233 may also include data 232 used in the execution of the program 233. The instructions 231 and the data 232 are stored in memory locations 228, 229, 230 and 235, 236, 237, respectively. Depending on the relative sizes of the instructions 231 and the memory locations 228-230, a particular instruction may be stored in a single memory location, as indicated by the instruction shown in memory location 230. Alternatively, the instruction may be segmented into several portions, each stored in a separate memory location, as indicated by the instruction segments shown in memory locations 228 and 229.

[0029] In general, the processor 205 is given a set of instructions to execute therein. The processor 205 waits for a subsequent input, to which the processor 205 responds by executing another set of instructions. Each input may come from one or more of several sources, including data generated by one or more of the input devices 202, 203, data received from an external source via one of the networks 220, 202, data retrieved from one of the storage devices 206, 209, or data retrieved from a storage medium 225 inserted into a corresponding reader 212, all shown in FIG. 2A. Execution of the set of instructions may result in data being output. Execution may also include storing data or variables to memory 234.

[0030] The video encoder 114, the video decoder 134, and the described methods may use input variables 254 that are stored in corresponding memory locations 255, 256, 257 in memory 234. The video encoder 114, the video decoder 134, and the described methods generate output variables 261, which are stored in corresponding memory locations 262, 263, 264 in memory 234. Intermediate variables 258 may be stored in memory locations 259, 260, 266, and 267.

[0031] 2B, the registers 244, 245, 246, the arithmetic logic unit (ALU) 240, and the control unit 239 cooperate to execute the sequence of micro-operations required to perform a "fetch, decode, and execute" cycle for every instruction in the instruction set that constitutes the program 233. Each fetch, decode, and execute cycle is (a) a fetch operation to fetch or read an instruction 231 from memory locations 228, 229, 230 (b) A decode operation in which the control unit 239 determines which instruction has been fetched. (c) The control unit 239 and / or the ALU 240 execute the operation to execute the instruction. has.

[0032] Thereafter, further cycles of fetching, decoding, and executing the next instruction may be performed. Similarly, a store cycle may be performed in which control unit 239 stores or writes a value to memory location 232.

[0033] Each step or sub-process in the methods of Figures 12 and 13 described below is associated with one or more segments of program 233 and is typically performed by register sections 244, 245, 247, ALU 240, and control unit 239 within processor 205 working together to perform a fetch, decode, and execute cycle for all instructions in the instruction set for the noted segment of program 233.

[0034] FIG. 3 is a schematic block diagram showing the functional modules of the video encoder 114. FIG. 4 is a schematic block diagram showing the functional modules of the video decoder 134. In general, data passes between the functional modules in the video encoder 114 and the video decoder 134 in groups of samples or coefficients, such as division of a block into fixed-size sub-blocks, or as an array. The video encoder 114 and the video decoder 134 can be implemented using a general-purpose computer system 200, as shown in FIG. 2A and FIG. 2B, and the various functional modules can be realized by software executable within the computer system 200, such as one or more software code modules of a software application program 233 residing on a hard disk drive 205 and controlled during its execution by the processor 205, by dedicated hardware within the computer system 200. Alternatively, the video encoder 114 and the video decoder 134 may be implemented by a combination of software executable within the computer system 200 and dedicated hardware. The video encoder 114, the video decoder 134, and the methods described may alternatively be implemented with dedicated hardware, such as one or more integrated circuits performing the functions or sub-functions of the methods described. Such dedicated hardware may include a graphics processing unit (GPU), a digital signal processor (DSP), an application specific standard product (ASSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or one or more microprocessors and associated memories. In particular, the video encoder 114 includes modules 322-386, and the video decoder 134 includes modules 420-496, which may each be implemented as one or more software code modules of the software application program 233.

[0035] The video encoder 114 of FIG. 3 is an example of a generic video coding (VVC) video encoding pipeline, although other video codecs may be used to perform the processing stages described herein. The video encoder 114 receives imaged frame data 113, such as a sequence of frames, each frame including one or more color channels. The block partitioner 310 first divides the frame data 113 into regions, generally referred to as "coding tree units" (CTUs), which are generally square in shape, and a specific size for the CTUs is configured to be used. The size of the coding tree units may be, for example, 64×64, 128×128, or 256×256 luma samples. The block partitioner 310 further divides each CTU into one or more coding units (CUs), with the CUs having various sizes that can include both square and non-square aspect ratios. Thus, a current block 312, the "coding unit" (CU), is output from the block partitioner 310 and proceeds according to an iteration over one or more blocks of the CTU. However, the concept of a CU is not limited to the block partitioning resulting from the block partitioner 310. It can also be said that the video decoder 134 generates CUs that are typically approximations of the blocks from the block partitioner 310 due to the use of lossy compression techniques. Furthermore, the video encoder 114 generates CUs that have the same approximations as seen at the video decoder 134, allowing accurate knowledge of the sample data available to the block prediction methods at the video decoder 134. Options for partitioning CTUs into CUs are further described below with reference to Figures 5 and 6.

[0036] The coding tree units (CTUs) resulting from the initial partitioning of the frame data 113 are scanned in raster scan order and grouped into one or more "slices." Because the frame data 113 typically includes multiple color channels, the CTUs and CUs are associated with samples from all color channels that overlap with the block regions defined from the operation of the block partitioner 310. A CU can be said to include one or more code blocks (CBs), each CB occupying the same block region as a CU, but associated with each of the color channels of the frame data 113. Due to potentially different sampling rates of the chroma channels compared to the luma channel, the dimensions of the CBs of the chroma channels may differ from the dimensions of the CBs of the luma channels. When using a 4:2:0 chroma format, the CBs of the chroma channels of a CU have dimensions that are half the width and height of the CBs for the luma channels of the CU.

[0037] In iterating over all CUs resulting from the block partitioner 310, the video encoder 114 generates, for each block, e.g., block 312, a "prediction unit" (PU) indicated by arrow 320. The PU 320 is a prediction of the content of the associated CU 312. The subtractor module 322 generates a difference between the PU 320 and the CU 312, indicated as 324 (or "residual", referring to the difference being in the spatial domain). The difference 324 is the block-sized difference between corresponding samples in the PU 320 and the CU 312. The difference 324 is transformed and quantized, represented as a transform unit (TU) indicated by arrow 336. The PU 320 is typically selected as the "best" resulting one of many possible candidate PUs. A candidate PU is a PU resulting from one of the prediction modes available to the video encoder 114. Each candidate PU results in a corresponding transform unit. The transform unit 336 is a quantized and transformed representation of the difference 324. When combined with the predicted PU in the decoder 114, the transform unit 336 reduces the difference between the decoded CU and the original block 312 at the expense of additional signaling in the bitstream.

[0038] Therefore, each candidate PU has an associated coding cost (rate) and an associated difference (or "distortion"). The coding rate (cost) is typically measured in bits. The coding distortion of a block is typically estimated as a difference in sample values, such as sum of absolute differences (SAD) or sum of squared differences (SSD). The estimate obtained from each candidate PU is determined by a mode selector 386 using the difference 324 to determine an intra-prediction mode (represented by arrow 388). The estimation of the coding cost associated with each candidate prediction mode and the corresponding residual coding can be performed at a much lower cost than entropy coding of the residual. Therefore, a large number of candidate modes can be evaluated to determine the optimal mode for rate-distortion detection.

[0039] The determination of the optimal mode is typically accomplished using variations of Lagrangian optimization. The selection of an intra-prediction mode 388 typically involves determining an encoding cost for the residual data resulting from the application of a particular intra-prediction mode. The encoding cost can be approximated by using a "sum of transformed differences," whereby a relatively simple transform, such as a Hadamard transform, is used to obtain an estimated transformed residual cost. In implementations using relatively simple transforms, the simplified estimation method can be used to make the same decision (i.e., intra-prediction mode) while reducing the complexity of the video encoder 114 if the cost resulting from the simplified estimation method is monotonically related to the actual cost that would otherwise be determined from a full evaluation. To allow for possible non-monotonicity in the relationship between the estimated cost and the actual cost, possibly resulting from further mode decisions available for encoding the residual data, the simplified estimation method can be used to generate a list of best candidates. The list of best candidates can be any number. Using the best candidates, a more exhaustive search can be performed to establish an optimal mode selection for encoding the residual data for each of the candidates, allowing for the final selection of the intra-prediction mode along with other mode decisions.

[0040] Other mode decisions include the ability to skip primary and secondary transforms, known as "transform skip". Skipping transforms is appropriate for residual data that lacks adequate correlation for reduced coding costs via representation as transform basis functions. Certain types of content, such as relatively simple computer-generated graphics, may exhibit similar behavior.

[0041] Another mode decision associated with the mode selector module 386 is the selection of an explicit multiple transform index (represented by arrows 390). The explicit multiple transform index 390 has four possible values. These four possible values ​​are a combination of two indexes (shown in columns 1046 and 1048 of FIG. 10A), where each index ranges from 0 to 1. The first index is associated with a horizontal transform index H and the second index is associated with a vertical transform index V. The horizontal transform index H is also known as a horizontal transform selection and the vertical transform index V is known as a vertical transform selection. The selection and encoding of the explicit multiple transform index 390 is described with reference to FIG. 10.

[0042] A Lagrangian or similar optimization process may be employed to select both the optimal partitioning of the CTU into CUs (by the block partitioner 310) and the selection of the best prediction mode from multiple possibilities. Through application of the Lagrangian optimization process of the candidate modes in the mode selector module 386, the intra-prediction mode with the lowest cost measure is selected as the best mode. The best mode is the selected intra-prediction mode 388 and is encoded into the bitstream 115 by the entropy encoder 338. The selection of the intra-prediction mode 388 by the operation of the mode selector module 386 extends the operation of the block partitioner 310. For example, candidates for the selection of the intra-prediction mode 388 may include modes applicable to a given block and further modes applicable to multiple smaller blocks collectively located with the given block. In such a case, the process of selecting candidates may also be a process of determining the best hierarchical decomposition of the CTU into CUs.

[0043] The entropy encoder 338 supports both variable length coding of syntax elements and arithmetic coding of syntax elements. Arithmetic coding is supported using a context-adaptive binary arithmetic coding process. An arithmetically coded syntax element consists of a sequence of one or more 'bins'. A bin, like a bit, has a value of '0' or '1'. However, a bin is not coded as a discrete bit in the bitstream 115. A bin has an associated possible value and an associated probability, known as a 'context'. If the actual bin being coded matches a possible value, then a 'most probable symbol' (MPS) is coded. Coding the most probable symbol is relatively inexpensive in terms of bits consumed. If the actual bin being coded does not match a possible value, then a 'least probable symbol' (LPS) is coded. Coding the least probable symbol has a relatively high cost in terms of bits consumed. The bin coding technique allows for efficient coding of bins where the probability of '0' vs. '1' is skewed. For syntax elements with two possible values ​​(i.e., "flag"), a single bin is sufficient. For syntax elements with many possible values, a series of bins is required. The presence of later bins in the sequence may then be determined based on the values ​​of earlier bins in the sequence. Furthermore, each bin may be associated with multiple contexts, with the selection of a particular context depending on earlier bins in the syntax element, bin values ​​of adjacent syntax elements (i.e., from adjacent blocks), etc. Each time a bin is coded, the context is updated to adapt to the new bin value. In this way, binary arithmetic coding schemes are said to be adaptive.

[0044] The encoder 114 also supports bins that lack context ("bypass bins"). Bypass bins are coded assuming an equal probability distribution between "0" and "1". The lack of context saves memory and reduces complexity. Thus, bypass bins are used when the distribution of values ​​for a particular bin is not biased. One example of an entropy coder that uses context and adaptation is known in the art as CABAC (context-adaptive binary arithmetic coder), and many variations of this coder are used in video coding.

[0045] The entropy encoder 338 encodes the intra-prediction mode 388 using a combination of context coding bins and (optionally) bypass coding bins. Typically, a list of "most probable modes" is generated in the video encoder 114. The list of most probable modes is typically a fixed length, such as three or six modes, and may include modes that occurred in previous blocks. The context coding bin encodes a flag indicating whether the intra-prediction mode is one of the most probable modes. If the intra-prediction mode 388 is one of the most probable modes, further signaling indicating which most probable mode corresponds to the intra-prediction mode 388 is encoded, for example, using a truncated unary bin string. Otherwise, the intra-prediction mode 388 is encoded as a "remaining mode" using an alternative syntax, such as a fixed-length code, to represent intra-prediction modes other than those present in the most probable mode list.

[0046] The entropy encoder 338 also encodes an explicit multiple transform index 390 for a particular coding unit or transform block, as described with reference to Figure 10. In particular, the explicit multiple transform index 390 includes a vertical transform index V and a horizontal transform index H. Both the vertical transform index V and the horizontal transform index H are selected by the mode selector 386 (as described above). Based on a combination of the vertical transform index V and the horizontal transform index H (see Figure 10A), the entropy encoder 338 encodes a corresponding bin string (shown in column 1042 of Figure 10A) into the bitstream 115.

[0047] The multiplexer module 384 selects from the tested candidate prediction modes to output the PU 320 according to the determined best intra-prediction mode 388. The candidate prediction modes need not include all possible prediction modes supported by the video encoder 114. The prediction modes are broadly classified into two categories. The first category is "intra-frame prediction" (or "intra-prediction"). In intra-frame prediction, a prediction for a block is generated using other samples drawn from the current frame. The second category is "inter-frame prediction" (or "inter-prediction"). In inter-frame prediction, a prediction for a block is generated using samples from a frame preceding the current frame in the order of encoding the frames in the bitstream (which may differ from the order of the frames when imaged or displayed). Within each category (i.e., intra-prediction and inter-prediction), different techniques may be applied to generate the PU. For example, intra-prediction may use values ​​from adjacent rows and columns of previously reconstructed samples in combination with a direction to generate the PU according to a predetermined filtering process. Alternatively, the PU may be described using a small number of parameters. Inter prediction methods may differ in the number of motion parameters and their precision. Motion parameters typically include reference frame offsets and transformations relative to one or two reference frames, but may include more frames, dedicated frames, or complex affine parameters such as scaling and rotation. In addition, a certain motion refinement process may be applied to generate a dense motion estimate based on the reference sample blocks.

[0048] After determining and selecting the best PU 320 and subtracting it from the original sample block in a subtractor 322, a residual 324 with the lowest coding cost is obtained and subjected to lossy compression. The lossy compression process includes the steps of transformation, quantization, and entropy coding. A transform module 326 applies a first transform to the difference 324, transforming the difference 324 into the frequency domain and generating intermediate transform coefficients represented by arrows 328. The first transform is typically separable, transforming a set of rows of each block and then transforming a set of columns of each block. The transformation of each set of rows and columns is performed by applying a one-dimensional transform that is repeated for each row and column of the block. The transformation spans the entire block.

[0049] In particular, the intra prediction mode 388 and the explicit multiple transform index 390 are used to select a horizontal one-dimensional transform and a vertical one-dimensional transform that collectively form a primary transform performed by the transform module 326. Each of the horizontal one-dimensional transform and the vertical one-dimensional transform is a separable transform. The horizontal one-dimensional transform and the vertical one-dimensional transform are performed across the entire transform block to substantially decorrelate the residual samples.

[0050] The transform types available to the transform module 326 include DST-VII, DST-VIII, DCT-VIII, DST-I, and DCT-V. From the transform settings, three "transform sets" 842 are defined, as described with reference to FIG. 8B. For each direction, either horizontal or vertical, a transform set 842 is selected according to an intra-prediction mode 388, as described with reference to FIG. 8C. From each transform set 842, one transform type is selected and applied, as described with reference to FIG. 11. The intermediate transform coefficients 328 are passed to a secondary transform module 330.

[0051] The secondary transform module 330 operates on a subset of the intermediate transform coefficients 328, such as the intermediate transform coefficients occupying the top left 4×4 or 8×8 area of ​​the entire block. The other transform coefficients in the intermediate transform coefficients 328 pass through the module 330 unchanged. The secondary transform module 330 applies one of various transforms to the subset of the intermediate transform coefficients 328 to generate transform coefficients represented by arrows 332. The secondary transform module 330 applies a selected forward secondary transform in a manner similar to the inverse secondary transform module 344. The transforms available to the secondary transform module 330 are typically non-separable and therefore cannot be performed in two stages (i.e., rows and columns), as in the case of the transform module 326. Due to the higher complexity of the secondary transform, and the earlier decorrelation of the primary transform, it is sufficient to perform the secondary transform only on the top left region of the transform block. The transform performed by the secondary transform module 330 may be selected depending at least in part on the prediction mode. In addition, the video encoder 114 may consider the selection of the applied transform in module 330 as a further candidate test for selection based on a rate / distortion cost evaluation.

[0052] The transform coefficients 332 are passed to a quantizer module 334, where quantization by a "quantization parameter" is performed to generate residual coefficients represented by arrow 336. The quantization parameter is constant for a given transform block, thus resulting in uniform scaling for the generation of residual coefficients for the transform block. Non-uniform scaling is also possible by applying a "quantization matrix", whereby a scaling factor applied to each residual coefficient is derived from a combination of the quantization parameter and a corresponding entry in a scaling matrix, typically having a size equal to the size of the transform block. Quantization matrices are therefore costly to signal and are rarely, if ever, coded in the bitstream 115. Coding of the quantization matrix requires converting a two-dimensional matrix of scaling factors into a list of scaling factors that are entropy coded into the bitstream 115. The existing Z-order scan may be reused for this purpose, avoiding the overhead associated with supporting an additional scan pattern for the infrequently performed operation of coding the quantization matrix. The residual coefficients 336 are provided to an entropy encoder 338 for encoding into the bitstream 115. Typically, the residual coefficients of a transform block are scanned to generate an ordered list of values ​​according to a scan pattern. The scan pattern generally scans the transform block as a sequence of 4×4 “sub-blocks”, providing a regular scanning operation at the granularity of a 4×4 set of residual coefficients, the arrangement of the sub-blocks depending on the size of the transform block. In addition, the prediction mode and corresponding block partitioning are also coded into the bitstream 115. As mentioned above, the video encoder 114 needs access to a frame representation that corresponds to the frame representation seen by the video decoder 134. Thus, the residual coefficients 336 are also inverse quantized by an inverse quantizer module 340 to generate inverse transform coefficients, represented by arrow 342. The inverse transform coefficients 342 pass through an inverse secondary transform module 344.The inverse secondary transform module 344 applies the selected secondary transform as represented by arrow 346 to generate intermediate inverse transform coefficients. The intermediate inverse transform coefficients 346 are fed to the inverse transform module 348 to generate residual samples, represented by arrow 350, of the transform unit. The summation module 352 sums the residual samples 350 and the PU 320 to generate reconstructed samples (indicated by arrow 354) for the CU. The reconstructed samples 354 are passed to a reference sample cache 356 and an in-loop filter module 368. The reference sample cache 356 is typically implemented using static RAM on the ASIC (thus avoiding costly off-chip memory accesses) and provides the minimum sample storage required to satisfy dependencies for generating intra-frame predicted blocks for subsequent CUs in the frame. The minimum dependencies typically include a "line buffer" of samples along the bottom of the rows of the CTU to be used by the CTU's subsequent row and column buffering, the extent of which is set by the CTU's height. The reference sample cache 356 provides reference samples (indicated by arrow 358) to a reference sample filter 360. The sample filter 360 applies a smoothing operation to generate filtered reference samples (indicated by arrow 362). The filtered reference samples 362 are used by an intra-frame prediction module 364 to generate an intra-predicted block of samples represented by arrow 366. For each candidate intra-prediction mode, the intra-frame prediction module 364 generates a block of samples, i.e., 366.

[0053] The in-loop filter module 368 applies several filtering stages to the reconstructed samples 354. The filtering stages include a "deblocking filter" (DBF) that applies smoothing to CU boundaries to reduce artifacts resulting from discontinuities. Another filtering stage present in the in-loop filter module 368 is the "adaptive loop filter" (ALF), which applies a Wiener-based adaptive filter to further reduce distortion. A further available filtering stage in the in-loop filter module 368 is the "sample adaptive offset" (SAO) filter. The SAO filter operates by first classifying the reconstructed samples into one or more categories and applying an offset at the sample level according to the assigned category. The filtered samples 370 are output from the in-loop filter module 368. The filtered samples 370 are stored in a frame buffer 372. The frame buffer 372 typically has the capacity to store several (e.g., up to 16) pictures and is therefore stored in the memory 206. Access to the frame buffer 372 is therefore costly in terms of memory bandwidth. The frame buffer 372 provides a reference frame (represented by arrow 374 ) to a motion estimation module 376 and a motion compensation module 380 .

[0054] The motion estimation module 376 estimates several “motion vectors” (denoted as 378), each a Cartesian space offset from the position of the current CU, and referencing a block in one of the reference frames in the frame buffer 372. A filtered block of reference samples (denoted as 382) is generated for each motion vector. The filtered reference samples 382 form further candidate modes available for potential selection by the mode selector 386. Furthermore, for a given CU, the PU 320 may be formed using one reference block (“uni-predicted”) or may be formed using two reference blocks (“bi-predicted”). For a selected motion vector, the motion compensation module 380 generates the PU 320 according to a filtering process that supports sub-pixel accuracy in the motion vector. Thus, the motion estimation module 376 (operating on many candidate motion vectors) may possibly perform a simplified filtering process compared to the filtering process of the motion compensation module 380 (operating on only the selected candidate) to achieve reduced computational complexity.

[0055] 3 is described with reference to Generic Video Coding (VVC), other video coding standards or implementations may use the processing stages of modules 310-386. The frame data 113 (and bitstream 115) may also be read from (or written to) memory 206, hard disk drive 210, a CD-ROM, a Blu-ray Disc™, or other computer-readable storage medium. Additionally, the frame data 113 (and bitstream 115) may be received from (or transmitted to) an external source, such as a server connected to a communications network 220 or a radio frequency receiver.

[0056] The video decoder 134 is shown in FIG. 4. The video decoder 134 of FIG. 4 is an example of a generic video coding (VVC) video encoding pipeline, although other video codecs may be used to perform the processing stages described herein. As shown in FIG. 4, a bitstream 133 is input to the video decoder 134. The bitstream 133 may be read from the memory 206, the hard disk drive 210, a CD-ROM, a Blu-ray Disc™, or other non-transitory computer-readable storage medium. Alternatively, the bitstream 133 may be received from an external source, such as a server or a radio frequency receiver connected to the communication network 220. The bitstream 133 includes encoded syntax elements representing imaging frame data to be decoded.

[0057] The bitstream 133 is input to the entropy decoder module 420. The entropy decoder module 420 extracts syntax elements from the bitstream 133 and passes the values ​​of the syntax elements to other modules in the video decoder 134. The entropy decoder module 420 applies the CABAC algorithm to decode the syntax elements from the bitstream 133. The decoded syntax elements are used to reconstruct parameters in the video decoder 134. The parameters include residual coefficients (represented by arrows 424) and mode selection information such as intra prediction mode 458 and explicit multiple transform index 454. The mode selection information also includes information such as motion vectors and the division of each CTU into one or more CUs. The parameters are used to generate PUs, typically in combination with sample data from previously decoded CUs.

[0058] The bitstream 133 also includes bin strings from the bin string column 1042 (see FIG. 10A). The entropy decoder module 420 extracts the bin strings and generates a vertical transform setting index (from the vertical transform setting index column 1046) and a horizontal transform setting index (from the horizontal transform setting index column 1048).

[0059] The residual coefficients 424 are input to the inverse quantizer module 428. The inverse quantizer module 428 performs inverse scaling on the residual coefficients 424 to generate reconstructed intermediate transform coefficients (represented by arrow 432) according to the quantization parameter. If the use of a non-uniform inverse quantization matrix is ​​indicated in the bitstream 133, the video decoder 134 reads the quantization matrix from the bitstream 133 as a sequence of scaling factors and arranges the scaling factors in the matrix according to the Z-order scan used to encode the residual coefficients. The inverse scaling then uses the quantization matrix in combination with the quantization parameter to generate the reconstructed intermediate transform coefficients. The use of a Z-order scan for the residual coefficients to scan the quantization matrix scaling factors avoids the burden of the presence of an additional scan pattern and associated memory, as well as the complexity of a scan that is not frequently performed. The reconstructed intermediate transform coefficients 432 are passed to the inverse secondary transform module 436. The inverse secondary module 436 performs a "secondary inverse transform" to generate reconstructed transform coefficients, represented by arrow 440. A secondary transform is performed according to the determined transform block size. The reconstructed transform coefficients 440 are passed to an inverse transform module 444, as described with reference to FIG. 9. The module 444 transforms the coefficients from the frequency domain back to the spatial domain. The transform block is effectively based on the effective and ineffective residual coefficient values. The result of the operation of the module 444 is a block of residual samples represented by arrow 448. The residual samples 448 are equal in size to the corresponding CU. The residual samples 448 are provided to an addition module 450. In the addition module 450, the residual samples 448 are added to the decoded PU 452 to generate a block of reconstructed samples represented by arrow 456. The reconstructed samples 456 are provided to a reconstructed sample cache 460 and an in-loop filtering module 488. The in-loop filtering module 488 generates a reconstructed block of frame samples represented as 492. The frame samples 492 are written to a frame buffer 496.

[0060] The reconstructed sample cache 460 operates similarly to the reconstructed sample cache 356 of the video encoder 114. The reconstructed sample cache 460 provides storage for reconstructed samples needed to intra predict subsequent CUs without going through the memory 206 (e.g., by using the data 232 instead, which is typically an on-chip memory). Reference samples, represented by arrow 464, are obtained from the reconstructed sample cache 460 and provided to a reference sample filter 468 to generate filtered reference samples, indicated by arrow 472. The filtered reference samples 472 are provided to an intra-frame prediction module 476. The module 476 generates blocks of intra-predicted samples, represented by arrow 480, according to the intra-prediction mode parameters 458 signaled in the bitstream 133 and decoded by the entropy decoder 420.

[0061] When intra prediction is indicated in the bitstream 133 for the current CU, the intra prediction samples 480 form the decoded PU 452 via a multiplexer module 484.

[0062] When inter prediction is indicated in the bitstream 133 for the current CU, the motion compensation module 434 uses the motion vector and the reference frame index to select and filter a block of samples from the frame buffer 496 to generate a block of inter prediction samples 438. The block of samples 498 is obtained from a previously decoded frame stored in the frame buffer 496. In the case of bi-prediction, two blocks of samples are generated and blended together to generate samples for the decoded PU 452. The frame buffer 496 receives filtered block data 492 from the in-loop filtering module 488. Similar to the in-loop filtering module 368 of the video encoder 114, the in-loop filtering module 488 applies any, at least, or all of the DBF, ALF, and SAO filtering operations. The in-loop filtering module 368 generates the filtered block data 492 from the reconstructed samples 456.

[0063] 5 is a schematic block diagram illustrating a set 500 of available partitions or partitions of blocks into one or more blocks in a generic video coding tree structure. The partitions shown in set 500 are available to block partitioner 310 of encoder 114 to partition each CTU into one or more CUs according to Lagrangian optimization, as described with reference to FIG.

[0064] Although set 500 shows only square blocks being divided into other, possibly non-square blocks, it should be understood that diagram 500 illustrates potential divisions but does not constrain the containing block to be square. If the containing block is non-square, the dimensions of the blocks resulting from the division are scaled according to the aspect ratio of the containing block. A particular subdivision of a CTU into one or more CUs by block partitioner 310 is referred to as a "coding tree" of the CTU. In the context of this disclosure, a leaf node is a node at which the process of subdivision ends. The process of subdivision must end when the area corresponding to the leaf node is equal to the minimum coding unit size. The leaf node that results in the smallest sized coding unit is present at the deepest level of decomposition of the coding tree. The process of subdivision may also end before the deepest level of decomposition, resulting in coding units larger than the minimum coding unit size.

[0065] At the leaf nodes of the coding tree, there are CUs with no further subdivision. For example, leaf node 510 contains one CU. At the non-leaf nodes of the coding tree, there are divisions into two or more further nodes, each of which may contain one CU or may contain further divisions into smaller regions.

[0066] The quadtree partition 512 divides the inclusion region into four equally sized regions, as shown in Figure 5. Compared to HEVC, Generic Video Coding achieves further flexibility by adding a horizontal bisection 514 and a vertical bisection 516. Each of the partitions 514 and 516 divides the inclusion region into two equally sized regions. The partitions are along a horizontal boundary (514) or a vertical boundary (516) within the containing block.

[0067] Further flexibility in general purpose video coding is achieved by adding a horizontal 3-partition 518 and a vertical 3-partition 520. The 3-partitions 518 and 520 divide a block into three regions bounded either horizontally (518) or vertically (520) along ¼ and ¾ of the width or height of the containing region. The combination of quad-trees, binary trees, and tert-trees is called a “QTBTTT” or alternatively a multi-tree (MT).

[0068] Compared to HEVC, which only supports quad-trees and therefore square blocks, QTBTTT results in more possible CU sizes, especially considering possible recursive application of binary and / or ternary tree partitioning. The possibility of unusual (e.g., non-square) block sizes can be reduced by constraining the partitioning options to eliminate partitions that result in block widths or heights that are either less than 4 samples or that are not multiples of 4 samples. In general, this constraint applies when considering luma samples. However, this constraint can also be applied separately to blocks of chroma channels, which can result in different minimum block sizes for luma versus chroma, for example, when the frame data is in 4:2:0 chroma format.

[0069] 6 is a schematic flow diagram illustrating a data flow 600 of the QTBTTT (or "coding tree") structure used in generic video coding. The QTBTTT structure is used for each CTU to define the partitioning of the CTU into one or more CUs. The QTBTTT structure of each CTU is determined by the block partitioner 310 in the video encoder 114 and encoded into the bitstream 115 or decoded from the bitstream 133 by the entropy decoder 420 in the video decoder 134. The data flow 600 further characterizes the permissible combinations available to the block partitioner 310 for partitioning the CTU into one or more CUs according to the partitioning shown in FIG. 5.

[0070] Starting from the top level of the hierarchy, i.e., CTU, zero or more quadtree splits are first performed. Specifically, a quadtree (QT) split decision 610 is made by the block partitioner 310. A decision at 610 returning a '1' symbol indicates a decision to split the current node into four subnodes according to the quadtree split 512. This results in four new nodes, such as 620, being generated, and for each new node, a return is made to the QT split decision 610. Each new node is considered in raster (or Z-scan) order. Alternatively, if the QT split decision 610 indicates that no further splits should be performed (returning a '0' symbol), the quadtree split stops and a multi-tree (MT) split is then considered.

[0071] First, an MT split decision 612 is made by the block partitioner 310. At 612, a decision to perform MT splitting is indicated. Returning a "0" symbol at decision 612 indicates that no further splitting of the node into subnodes is performed. If no further splitting of the node is performed, the node is a leaf node of the coding tree and corresponds to a coding unit (CU). The leaf node is output at 622. Alternatively, if MT split 612 indicates a decision to perform MT splitting (returning a "1" symbol), the block partitioner 310 proceeds to direction decision 614.

[0072] Orientation decision 614 indicates the orientation of the MT partition as either horizontal ("H" or "0") or vertical ("V" or "1"). The block partitioner 310 proceeds to decision 616 if decision 614 returns a "0" indicating a horizontal orientation. The block partitioner 310 proceeds to decision 618 if decision 614 returns a "1" indicating a vertical orientation.

[0073] In each of decisions 616 and 618, the number of partitions for the MT split is indicated as either two (bipartite or 'BT' node) or three (tripartite or 'TT') in the BT / TT split. That is, the BT / TT split decision 616 is made by block partitioner 310 when the indicated direction from 614 is horizontal, and the BT / TT split decision 618 is made by block partitioner 310 when the indicated direction from 614 is vertical.

[0074] The BT / TT split decision 616 indicates whether the horizontal split is a 2-way split 514, indicated by returning a "0", or a 3-way split 518, indicated by returning a "1". If the BT / TT split decision 616 indicates a 2-way split, then in an HBT CTU node generation step 625, two nodes are generated by the block partitioner 310 according to the horizontal 2-way split 514. If the BT / TT split 616 indicates a 3-way split, then in an HTT CTU node generation step 626, three nodes are generated by the block partitioner 310 according to the horizontal 3-way split 518.

[0075] The BT / TT split decision 618 indicates whether the vertical split is a 2-way split 516, indicated by returning a "0", or a 3-way split 520, indicated by returning a "1". If the BT / TT split 618 indicates a 2-way split, then in a VBT CTU node generation step 627, two nodes are generated by the block partitioner 310 according to the vertical 2-way split 516. If the BT / TT split 618 indicates a 3-way split, then in a VTT CTU node generation step 628, three nodes are generated by the block partitioner 310 according to the vertical 3-way split 520. For each node resulting from steps 625-628, the recursion of the data flow 600 back to the MT split decision 612 is applied in a left-to-right or top-to-bottom order, depending on the direction 614. As a result, the 2-way and 3-way tree splits can be applied to generate CUs with various sizes.

[0076] Figures 7A and 7B provide an example division 700 of a CTU 710 into several coding units (CUs). An example of a CU 712 is shown in Figure 7A. Figure 7A shows the spatial arrangement of the CUs in the CTU 710. The example division 700 is also shown as a coding tree 720 in Figure 7B.

[0077] At each non-leaf node in the CTU 710 in FIG. 7A, e.g., nodes 714, 716, and 718, the contained nodes (which may be further split or may be CUs) are scanned or traversed in "Z-order" to create a list of nodes, represented as columns in the coding tree 720. In the case of a quadtree split, the Z-order scan is from top-left to right followed by bottom-left to right. In the case of horizontal and vertical splits, the Z-order scan (traversal) simplifies to a top-to-bottom and left-to-right scan, respectively. The coding tree 720 in FIG. 7B lists all nodes and CUs according to the applied scan order. Each split generates a list of 2, 3, or 4 new nodes at the next level of the tree until a leaf node (CU) is reached.

[0078] The image is decomposed into CTUs by the block partitioner 310 and further decomposed into CUs, which are forward transformed and quantized by the encoder 114, using the CUs to generate respective residual blocks (324), as described with reference to FIG. 3. The resulting transform blocks (TBs) 336 are then scanned to form a sequential list of residual coefficients as part of the operation of the entropy coding module 338. A comparable process is performed within the video decoder 134 to obtain the transform blocks from the bitstream 133.

[0079] FIG. 8A shows a set 800 of intra prediction modes for a transform block, which may be indicated using intra prediction modes 388 and 458. In FIG. 8A, 67 intra prediction modes are defined. Mode 0 is the "planar" intra prediction mode, Mode 1 is the "DC" intra prediction mode, and Modes 2-66 are "angular" intra prediction modes. The planar intra prediction mode (Mode 0) inputs plane-dependent samples, i.e., samples with offsets and gradients in the horizontal and vertical directions, into the prediction block. Planar parameters are obtained from neighboring reference samples, if available. Similarly, the DC intra prediction mode (Mode 1) also uses neighboring reference samples (if available) to input offsets into the prediction block.

[0080] The angular intra prediction modes (modes 2-66) set the block by generating a texture aligned to one of 65 directions or "angles". For clarity, a subset of the 65 angles is shown in FIG. 8A, modes 2, 18, 34, 50, and 66. For each mode, neighboring reference samples are used to generate a texture that sets the predicted block in the direction indicated by the arrow for the angular intra prediction mode. Additional angles, not explicitly shown in FIG. 8A, are in intermediate positions (i.e., modes 3-17, 19-33, 35-49, and 51-65). A first symmetry is evident from FIG. 8A along angular mode 34 and in the diagonal direction from the top left to the bottom right of the predicted block. The first symmetry shows that modes 2-34 correspond to modes 66-34, transposed along this diagonal axis.

[0081] FIG. 8B illustrates a mapping 840 from each transform setting 842 to a particular transform type by a transform index 844. Three transform settings 842 are defined and identified by respective transform setting indices of 0, 1, and 2. Each transform setting 842 includes two transform types selectable by the transform index 844. As a result, the mapping 840 allows a given transform type to be selected based on the transform setting index of the transform setting 842 and the transform index 844. The selected transform is a one-dimensional transform. Separate selections of transform types are made for the horizontal and vertical directions. The size of the transform (number of points) corresponds to the dimensions of the transform block. That is, the size of the transform corresponds to the height of the transform block when selecting a vertical transform, and the size of the transform corresponds to the width of the transform block when selecting a horizontal transform.

[0082] 8C is a table showing a mapping 880 from an intra-prediction mode to a transform setting index of a transform setting 842 for a transform block based on whether the transform block is in the horizontal orientation ("H") or vertical orientation ("V"). For each intra-prediction mode, one of the transform setting indexes associated with the three transform settings 842 is selected for the horizontal orientation H, and one of the transform setting indexes associated with the three transform settings 842 is selected for the vertical orientation V.

[0083] For example, intra prediction mode 14 is mapped to transform setting index 0 for vertical transform blocks, and intra prediction mode 14 is mapped to transform setting index 2 for horizontal transform blocks. As shown in the mapping 840 of FIG. 8B, transform setting index 0 is mapped to either DST-VII or DCT-VIII selectable according to the transform index 844. Also, as shown in the mapping 840 of FIG. 8B, transform setting index 2 is mapped to either DST-VII or DCT-V selectable according to the transform index 844. Thus, for vertical transform blocks, intra prediction mode 14 uses either DST-VII or DCT-VIII transform depending on the transform index 844. Thus, for horizontal transform blocks, intra prediction mode 14 uses either DST-VII or DCT-V transform depending on the transform index 844. The transform index 844 is based on the explicit multiple transform index 390 or 454. As described above, the explicit multiple transform indexes 390 are selected by the mode selector 306 and the explicit multiple transform indexes 454 are derived from the bitstream 133 by the entropy encoder 420 .

[0084] 9 is a schematic block diagram illustrating the inverse transform module 348 of the video encoder 114 or the inverse transform module 444 of the video decoder 134. The operation of both modules 348 and 444 is identical, reflecting the need for the video encoder 114 to model the inverse quantization and inverse transformation of residual coefficients performed in the video decoder 134.

[0085] Each of the inverse transform modules 348 and 444 includes a transform setting table module 910 , a transform index deriver module 920 , a horizontal transform selector module 940 , a vertical transform selector module 945 , a horizontal transform module 960 , and a vertical transform module 980 .

[0086] The conversion setting table module 910 receives the intra-prediction mode 388 or 458. If implemented as an inverse transform module 348, the conversion setting table module 910 receives the intra-prediction mode 388 from the mode selector 386 (see FIG. 3). If implemented as an inverse transform module 444, the conversion setting table module 910 receives the intra-prediction mode 458 from the entropy encoder 420 (see FIG. 4).

[0087] The transform setting table module 910 determines a horizontal transform setting index (represented as arrow 915) and a vertical transform setting index (represented as arrow 917) from the intra-prediction mode 388 or 458. The horizontal transform setting index 915 and the vertical transform setting index 917 are determined according to the mapping 880 (see FIG. 8C).

[0088] The transform index deriver 920 receives the explicit multiple transform indexes 390 or 454. If implemented as an inverse transform module 348, the transform index deriver 920 receives the explicit multiple transform indexes 390 from the mode selector 386 (see FIG. 3). If implemented as an inverse transform module 444, the transform index deriver 920 receives the explicit multiple transform indexes 454 from the entropy encoder 420 (see FIG. 4).

[0089] The transform index deriver 920 extracts horizontal transform indexes 920 and vertical transform indexes 925 from the explicit multiple transform indexes 390 or 454. The horizontal transform indexes 920 and vertical transform indexes 925 each range from 0 to 1.

[0090] The horizontal transform selector module 940 receives a horizontal transform setting index 915 (from the transform setting table module 910) and a horizontal transform index 920 (from the transform index deriver 920). The horizontal transform selector 940 generates a selected horizontal transform (represented as an arrow 950) from the horizontal transform setting index 915 and the horizontal transform index 920 according to the mapping 840 (see FIG. 8B). The horizontal transform setting index 915 is used to select one of the transform settings 842, and the horizontal transform index 920 is used to select one of the transform indexes 844.

[0091] The vertical transform selector module 945 receives the vertical transform setting index 917 (from the transform setting table module 910) and the vertical transform index 925 (from the transform index deriver 920). The vertical transform selector module 945 generates a selected vertical transform 955 from the vertical transform setting index 917 and the vertical transform index 925 according to the mapping 840 (see FIG. 8B). The vertical transform setting index 917 is used to select one of the transform settings 842, and the vertical transform index 925 is used to select one of the transform indexes 844.

[0092] The horizontal transform module 960 applies the selected horizontal transform 950 to the intermediate inverse transform coefficients 346 or the reconstructed transform coefficients 440. The intermediate inverse transform coefficients 346 are from the inverse secondary transform module 344 (see FIG. 3). The reconstructed transform coefficients 440 are from the inverse secondary transform module 436 (see FIG. 4).

[0093] The transform coefficients 346 or 440 are transformed according to a selected horizontal transform 950. That is, the selected horizontal transform 950 having a set of one-dimensional transforms is applied to each row of the array of transform coefficients 346 or 440. Each one-dimensional transform spans the width of a transform block. The horizontal transform module 960 outputs the horizontally transformed transform coefficients 970 as an array having a size equal to the current transform block. The transform coefficients 970 are provided to a vertical transform module 980. For ease of explanation, the transform coefficients 970 refer to both the horizontally transformed transform coefficients 346 and 440.

[0094] The vertical transform module 980 applies a selected vertical transform 955 having a set of one-dimensional transforms to each column of the horizontally transformed transform coefficients 970. The vertical transform module 980 generates a residual sample 350 or 448 from each horizontally transformed transform coefficient 346 or 440 using the vertical selection transform 955.

[0095] 10A shows a table 1000 illustrating the relationship between truncated unary syntax elements and transform indexes for transform selection. The transform indexes are a horizontal transform index H (shown in column 1048) and a vertical transform index V (shown in column 1046). Table 1000 includes a bin string column 1042, a binarization column 1044, a vertical transform index column 1046, and a horizontal transform index column 1048.

[0096] The bin string column 1042 has four possible bin strings, from truncated unary binarisation with a maximum value of 3 (i.e. ranging from 0 to 3). For the encoder 114, the bin strings shown in column 1042 are the bin strings that the entropy encoder 338 will encode into the bitstream 115. The bin strings that are encoded into the bitstream 115 are selected based on the horizontal transform index H and vertical transform index V selected by the mode selector 386.

[0097] For the decoder 134, the bin string shown in column 1042 is the bin string decoded by the entropy decoder 420 from the bitstream 133. The entropy decoder 420 then outputs the horizontal transform index H and the vertical transform index V based on the decoded bin string.

[0098] The binarization column 1044 contains the composition for each bin string in the bin string column 1042. In this nomenclature, "Ctx0" and "Ctx1" refer to two separate CABAC contents used to encode corresponding bins in the bin string column 1042. The term "Byp" refers to the bypass encoding bin used to encode corresponding bins in the bin string column 1042.

[0099] The vertical transform index column 1046 includes an index value (either 0 or 1) for selecting a transform from the transform set for the vertical 1D transform. The horizontal transform index column 1048 includes an index value (either 0 or 1) for selecting a transform from the transform set for the horizontal 1D transform. In the table 1000, the shortest bin string is assigned to the case of vertical index "0" and horizontal index "0". The shortest bin string corresponds to the case selected most frequently by the mode selector 386. And the next shortest bin string is assigned to the case of vertical index "1" and horizontal index "1", which corresponds to the case selected second most frequently by the mode selector 386. Then, the cases of index value combinations "01" or "10" for the vertical transform index and horizontal transform index combination are not equally likely, and the longest bin string in column 1044 is assigned. Furthermore, the distinction between these two cases is performed using bypass coding bins, reflecting the approximately equal occurrence probability between these two selections by the mode selector 386. The statistical properties described turn out to be roughly invariant to the transform size, and therefore the existence of these contexts, selected according to their position in the truncated unary bin string, is adequate to capture the asymmetric probability distribution of bin values.

[0100] FIG. 10B is a table 1050 illustrating alternative relationships between truncated unary syntax elements and transform indexes for horizontal and vertical transform indexing, where the third bin of the truncated unary binarization uses context encoding rather than bypass encoding.

[0101] 11 is a flow chart diagram of a method 1100 for performing a primary transform according to an explicit transform selection to encode a transform block of residual coefficients into a bitstream 115. The method 1100 may be implemented by an apparatus such as a configured FPGA, ASIC, or ASSP. Additionally, the method 1100 may be executed by the video encoder 114 under execution of the processor 205. As such, the method 1100 may be stored in a computer readable storage medium and / or memory 206. The method 1100 begins in the processor 205 at an intra-prediction mode decision step 1110.

[0102] The method 1100 begins with an intra prediction mode step 1110. In the intra prediction mode step 1110, the mode selector 386, under the execution of the processor 205, selects an intra prediction mode 388 for the prediction unit associated with the current coding unit. The selection is generally performed in two passes. In a first pass, all intra prediction modes of the luma prediction block are tested. For each mode, the residual cost is approximated using a "sum of absolute transform differences" method, such as the Hadamard transform. From this test, a list of "best" (lowest distortion) candidate prediction modes is derived. Then, a full test of the residual codes is performed on the list of candidate prediction modes. As a result of the full test, one intra prediction mode 388 is selected for the current coding unit. The intra prediction mode 388 is encoded by the entropy encoder 338 into the bitstream 115. Control in the processor 205 then passes from the intra prediction mode step 1110 to a transform setting determination step 1120.

[0103] In a transform setting determination step 1120, the inverse transform module 348, under execution of the processor 205, uses the determined intra-prediction mode 388 to determine a horizontal transform setting index 915 (as described above with respect to FIG. 9) and a vertical transform setting index 917 (as described above with respect to FIG. 9) according to the mapping 880 (see FIG. 8C). From the horizontal transform setting index 915, a setting for the transform for the horizontal transform is determined according to the mapping 840. From the vertical transform setting index 917, a setting for the transform for the vertical transform is determined according to the mapping 840.

[0104] As described in the above example, when the intra prediction mode 388 is mode 14, the possible transforms for the vertical transform are DST-VII or DCT-VIII, and the possible transforms for the horizontal transform are DST-VII or DCT-V.

[0105] Control within the processor 205 then passes from the determine transform settings step 1120 to a test transform step 1130 .

[0106] In testing the transform step 1130, the transform module 326 and the inverse transform module 348, under execution of the processor 205, test each combination of transforms from the transform settings for vertical transforms and the transform settings for horizontal transforms from step 1120. In an example where the intra-prediction mode 388 is mode 14, the DST-VII and DCT-VIII transforms are tested for the vertical transform, and the DST-VII or DCT-V transforms are tested for the horizontal transform.

[0107] As described above, the tests are performed by the transform module 326 and the inverse transform module 348. The particular transform in each configuration is referenced using a horizontal transform index 920 (for the horizontal transform) and a vertical transform index 925 (for the vertical transform). Each combination of vertical and horizontal transforms is tested to determine the distortion for encoding the residual.

[0108] During testing, the transform module 326 and the inverse transform module 348 perform the "normal operations" of the modules. Thus, for every inverse transform operation performed by the inverse transform module 348, the transform module 326 performs a corresponding forward transform operation. The test process may use approximation methods to perform tests with reduced complexity compared to performing a full transform.

[0109] Control within the processor 205 then passes from the test of the transform settings step 1130 to a transform selection step 1140 .

[0110] In a transform selection step 1140, a combination of vertical and horizontal transforms that results in the lowest distortion for coding the residual is selected. In an example where the intra prediction mode 388 is mode 14, the combination that provides the lowest distortion may be DCT-VIII for the vertical transform and DST-VII for the horizontal transform. Control in the processor 205 then passes from the transform selection step 1140 to apply horizontal transform 1150.

[0111] In a horizontal transform application step 1150, the inverse transform module 348 performs the horizontal transform. In particular, the horizontal transform module 960 performs the selected horizontal transform 950. In an example where the intra-prediction mode 388 is mode 14, DST-VII is used for the horizontal transform. Control in the processor 205 then passes from the horizontal transform application 1150 to the vertical transform application 1160.

[0112] In a vertical transform application step 1160, the inverse transform module 348 performs the vertical transform. In particular, the vertical transform module 980 performs the selected vertical transform 955. In an example where the intra prediction mode 388 is mode 14, DCT-VIII is used for the vertical transform. As a result of the horizontal transform application step 1150 and the vertical transform application step 1160, after optionally applying a secondary transform (i.e., the operation of module 330), the transform coefficients 332 are generated. Control in the processor 205 then passes from the vertical transform application step 1160 to a residual coefficient quantization step 1170.

[0113] In a residual coefficient quantization step 1170, the quantizer module 334, under execution of the processor 205, quantizes the transform coefficients 332 in accordance with the quantization parameter to generate residual coefficients 336. Control in the processor 205 then passes from the residual coefficient quantization step 1170 to a residual coefficient encoding step 1180.

[0114] In a residual coefficient coding step 1180, the entropy encoder 338, under execution of the processor 205, codes the residual coefficients 336 into the bitstream 115. The residual coefficients 336 for a given transform block are coded by scanning a two-dimensional array of coefficients into a list. The scan typically divides the transform block into one or more sub-blocks, each of size 4×4 (although other sizes are possible). The presence of significant coefficients in each sub-block is signaled with a “coded sub-block flag”, and one significance flag is signaled per residual coefficient in the sub-block if at least one significant coefficient is present. The coded sub-block flags and the significance flags use context coding bins. Additional indications of magnitude, such as “greater than 1 flag” and “greater than 2 flag”, are also coded using context coding bins. Finally, further indications of magnitude are typically coded using Golomb-Rice coding with bypass coding bins. The cost in bits of coding the residual coefficients 336 is heavily influenced by their magnitude. As a result of the selection of optimal EMT indices, their magnitude is slightly reduced, leading to improved compression efficiency. The binarization of these indices, performed in step 1190, further improves the compression efficiency. Control in the processor 205 then passes from the residual coefficients encoding step 1180 to an EMT bin string encoding step 1190.

[0115] In the EMT bin string encoding step 1190, the entropy encoder 338, under execution of the processor 205, encodes the horizontal transform index 920 and the vertical transform index 925 into the bitstream 115 using truncated unary binarization (max value of 3), context modeling, and mapping of bin strings to index values, as described with reference to FIG. 10A. The most frequently selected pair of indices ("0" horizontal and "0" vertical) is encoded using the shortest bin string ("0"), resulting in more efficient coding than the simple case of using one bin for each index. The second most frequently selected pair of indices ("1" horizontal and "1" vertical) is encoded using the second shortest bin string, i.e., "10", and the two approximately equally least likely combinations ("1" and "0", "0" and "1") are encoded using the longest bin string ("110" and "111"). The third bin uses bypass coding, since there is no statistical asymmetry available for exploitation using context coding. The method 1100 then ends upon completion of step 1190.

[0116] FIG. 12 is a flow chart diagram of a method 1200 for decoding a transform block of residual coefficients from the bitstream 133. The decoding of the method 1200 is performed by a linear inverse transform selected according to an explicit transform selection. The method 1200 may be implemented by an apparatus such as a configured FPGA, ASIC, or ASSP. Furthermore, the method 1200 may be executed by the video decoder 134 under the execution of the processor 205. As such, the method 1200 may be stored in a computer-readable storage medium and / or memory 206. The method 1200 starts with the processor 205 at an intra-prediction mode decoding step 1210.

[0117] In an intra-prediction mode decoding step 1210, the entropy decoder 420, under execution of the processor 205, decodes the intra-prediction modes 458 from the bitstream 133. The entropy decoder 420 generates a list of the most probable modes, including the modes considered most likely to have been selected by the mode selector 386. A context-coded bin is decoded to indicate the use of either the most probable mode or one of the other modes ("remaining modes") that are not in the most probable mode list. Further bins are then decoded to identify which most probable mode is in use or which remaining mode is in use. Control in the processor 205 then passes from the intra-prediction mode decoding step 1210 to a residual coefficient decoding step 1220.

[0118] In a residual coefficient decoding step 1220, the entropy decoder 420, under execution of the processor 205, decodes the list of residual coefficients and assembles the residual coefficients into an array for the transform block. The assembly follows a scan pattern, typically grouping the residual coefficients into 4x4 subblocks. The list of residual coefficients is typically scanned backwards, starting from the "last" significant residual coefficient back towards the top left residual coefficient (the "DC" residual coefficient, although this may not correspond to the block offset for transforms that are not based on cosine functions). Context coding bins are decoded to indicate the presence of significant residual coefficients in each subblock, and if indicated, additional context coding bins are decoded to indicate the significance of each residual coefficient in the subblock. The magnitude of each residual coefficient can be further characterized by decoding a "greater than 1" or "greater than 2" flag. Finally, for residual coefficients whose magnitude is not determined by the preceding context coding bins, the "residual magnitude" is decoded using Golomb-Rice coding with bypass coding bins. As a result of this decoding, the residual coefficients 424 of the transform block are decoded from the bitstream 133. Control then passes in the processor 205 from the residual coefficient decoding step 1220 to a residual coefficient dequantization step 1230.

[0119] In the residual coefficient inverse quantization step 1230, the inverse quantizer module 428, under the execution of the processor 205, generates intermediate transform coefficients 432 by scaling the residual coefficients 424 according to the quantization parameter. The intermediate transform coefficients 432 can be transformed by application of a non-separable secondary transform, typically applied only to the top-left region of the transform block, to generate the intermediate transform coefficients 432. As a result of this scaling operation, the intermediate transform coefficients 432 have approximately the same magnitude as their pre-quantization counterparts (i.e., the transform coefficients 332) in the video encoder. Through the optional application of an inverse non-separable secondary transform (i.e., the inverse secondary transform module 436), the reconstructed transform coefficients 440 are obtained from the intermediate transform coefficients 432. Control in the processor 205 then passes from the residual coefficient inverse quantization step 1230 to the EMT bin string decoding step 1240.

[0120] In an EMT bin string decoding step 1240, the entropy decoder 420, under execution of the processor 205, decodes a truncated unary bin string from the bitstream 133. The truncated unary bin string is one of the bin strings listed in column 1042 of FIG. 10A. As shown in FIG. 10A and described above, the unary bin string is limited in length to three bins (ranging from 0 to 3). The truncated unary bin string has a length of 1 to 3 bins, where the first bin is context coded, the second bin (if present) is context coded with a different context, and the third bin (if present) is bypass coded. This binarization is described with reference to FIG. 10. Control in the processor 205 then passes from the EMT bin string decoding step 1240 to an EMT index determination step 1250.

[0121] In an EMT index determination step 1250, a transform index deriver 920 in the video decoder 134, under execution of the processor 205, determines the horizontal transform index 920 and the vertical transform index 925 in accordance with table 1000 of Figure 10A. Control in the processor 205 then passes from the EMT index determination step 1250 to a transform setting determination step 1260.

[0122] In a transform setting determination step 1260, the inverse transform module 444, under execution of the processor 205, determines which transform settings in the mapping 840 should be used for the horizontal and vertical transforms of the primary inverse transform. In particular, using the intra prediction mode 458, the transform setting table 910 is used to determine the horizontal transform setting 915 and the vertical transform setting 917. Control in the processor 205 then passes from the transform setting determination step 1260 to a transform selection step 1270.

[0123] In a transform selection step 1270, the inverse transform module 444, under execution of the processor 205, determines a horizontal transform 950 and a vertical transform 955. The horizontal transform 950 and vertical transform 955 are determined by a horizontal transform selector 940 and a vertical transform selector 945, respectively. The modules take as inputs a horizontal transform set index 915, a vertical transform set index 917, a horizontal transform index 920, and a vertical transform index 925, as shown in Figure 9, and generate a selected horizontal transform 950 and a selected vertical transform 960 according to Figure 8B. Control in the processor 205 then passes from the transform selection step 1270 to an apply horizontal transform step 1280.

[0124] In a horizontal transform application step 1280, the inverse transform module 444, under execution of the processor 205, applies the selected horizontal transform 950 using the horizontal transform module 960. The selected horizontal transform 950 is applied to each row of the reconstructed transform coefficients 440 to generate horizontally transformed transform coefficients 970. The horizontally transformed transform coefficients 970 are provided to the vertical transform module 980. The vertical transform module 980 then generates the residual samples 448 by applying the selected vertical transform 955. The residual samples 950 are further used for summation with the intra-predicted PU to generate reconstructed samples, as described with reference to FIG. 4. The method 1200 then ends.

[0125] In another configuration of the methods 1100 and 1200, the binarisation of the truncated unary bin string is modified such that only one context coding bin is used. In particular, referring to FIG. 10A, the first bin of the binarisation 1044 remains context coded, but the second bin of the binarisation 1044 uses a bypass coding bin instead of context coding. The configuration using just one context coding bin saves implementation costs at the expense of a slight decrease in compression efficiency. However, the arrangement using only one context coding bin can achieve a slightly higher throughput. The higher throughput is because the CABAC processing in the entropy encoder 338 and the entropy decoder 420 can process the bypass coding bin at a higher rate than the context coding bin, at least due to the absence of context state updates.

[0126] As discussed with reference to FIG. 10A, the probability of selecting horizontal transform index H and vertical transform index V with values ​​of "0" and "1", respectively, or "1" and "0", respectively, is approximately equal. Therefore, another binarization than that shown in FIG. 10A reverses the use of the third bypass coding bin. A configuration that reverses the use of the third bin has a bin string of "111" mapped to "0", "1" for each vertical transform index and horizontal transform index, and a bin string of "110" mapped to "1", "0" for each vertical transform index and horizontal transform index. Because of the approximately equal probability between these two selections, a comparable compression advantage is achieved.

[0127] As a result of the binarization of explicit multiple transform selections for the horizontal and vertical directions, particularly as they are jointly coded according to their expected probability of selecting each index value, coding efficiency is improved compared to when separate syntax elements are used for the horizontal and vertical indices (e.g., one context coding bin each).

[0128] Industrial Applicability The described arrangements are applicable in the computer and data processing industry, and in particular in digital signal processing for encoding and decoding of signals such as video and image signals, achieving high compression efficiency without excessive cost in terms of memory bandwidth due to non-local scanning of residual coefficients.

[0129] The above describes only some embodiments of the present invention, which may be modified and / or altered without departing from the scope and spirit of the present invention, and the embodiments are illustrative and not limiting.

[0130] In the context of this specification, the term "comprising" means "including primarily, but not necessarily solely" or "having" or "including" and not "consisting only of." Variations of the term "comprising," such as "comprise" and "comprises," have correspondingly varying meanings. The claims which define the invention are as follows:

Claims

1. 1. A method for decoding a transform block in an image from a bitstream, comprising: a first decoding step of decoding, using arithmetic decoding, from the bitstream a truncated unary bin string having both a horizontal transform type and a vertical transform type associated therewith, the truncated unary bin string being used to determine both the horizontal transform type and the vertical transform type for a transform block being decoded; determining, using the one truncated unary bin string, both the horizontal transform type and the vertical transform type for the transform block to be decoded; having The one truncated unary bin string can specify different transform types for the horizontal transform type and the vertical transform type, The method further includes a second decoding step of decoding the transform block to be decoded by applying the determined horizontal transform type and the determined vertical transform type to the transform block to be decoded to perform a transform process; the one truncated unary bin string is a bin string of bins for which no bypass coding is used; the context values ​​of the bins used for arithmetic decoding of the bins are different from each other; The one truncated unary bin string corresponds to a number of values ​​for determining the horizontal transform type and the vertical transform type. A method comprising:

2. 2. The method of claim 1, wherein transform types selectable as at least one of the horizontal transform type and the vertical transform type include DST-VII and DST-VIII.

3. 1. A method for encoding transform blocks in an image to generate a bitstream that is decoded in a decoding method, comprising the steps of: a first determination step of determining a horizontal transform type for a transform block to be coded; a second determination step of determining a vertical transform type for the transform block to be coded; a first encoding step of encoding the transform block to be encoded by applying the determined horizontal transform type and the determined vertical transform type to the transform block to be encoded to perform a transform process; a second encoding step of encoding, into the bitstream using arithmetic coding, a single truncated unary bin string associated with both the horizontal transform type and the vertical transform type, the single truncated unary bin string being capable of specifying both the horizontal transform type and the vertical transform type for the transform block to be encoded; having The one truncated unary bin string can specify different transform types for the horizontal transform type and the vertical transform type, the one truncated unary bin string is a bin string of bins for which no bypass coding is used; the context values ​​of the bins used for arithmetic coding of the bins are different from each other, The single truncated unary bin string corresponds to a number of values ​​for specifying the horizontal transform type and the vertical transform type. A method comprising:

4. Conversion types selectable as at least one of the horizontal conversion type and the vertical conversion type include DST-VII and DST-VIII.

4. The method according to claim 3 .

5. 1. A decoding device for decoding a transform block in an image from a bitstream, comprising: a first decoding means for decoding, from the bitstream using arithmetic decoding, a truncated unary bin string having both a horizontal transform type and a vertical transform type associated therewith, the truncated unary bin string being used to determine both the horizontal transform type and the vertical transform type for a transform block being decoded; determining means for determining, using the one truncated unary bin string, both the horizontal transform type and the vertical transform type for the transform block to be decoded; having The one truncated unary bin string can specify different transform types for the horizontal transform type and the vertical transform type, The decoding device further includes a second decoding means for decoding the transform block to be decoded by applying the determined horizontal transform type and the determined vertical transform type to the transform block to be decoded to perform a transform process; the one truncated unary bin string is a bin string of bins for which no bypass coding is used; the context values ​​of the bins used for arithmetic decoding of the bins are different from each other; The one truncated unary bin string corresponds to a number of values ​​for determining the horizontal transform type and the vertical transform type. A decoding device comprising:

6. 1. An encoding device for encoding a transform block in an image to generate a bitstream to be decoded in a decoding device, comprising: a first determining means for determining a horizontal transform type for a transform block to be coded; second determining means for determining a vertical transform type for the transform block to be coded; a first encoding means for encoding the transform block to be encoded by applying the determined horizontal transform type and the determined vertical transform type to the transform block to be encoded and thereby encoding the transform block to be encoded; a second encoding means for encoding, into the bitstream using arithmetic encoding, a single truncated unary bin string associated with both the horizontal transform type and the vertical transform type, the single truncated unary bin string being capable of specifying both the horizontal transform type and the vertical transform type for the transform block to be encoded; having The one truncated unary bin string can specify different transform types for the horizontal transform type and the vertical transform type, the one truncated unary bin string is a bin string of bins for which no bypass coding is used; the context values ​​of the bins used for arithmetic coding of the bins are different from each other, The single truncated unary bin string corresponds to a number of values ​​for specifying the horizontal transform type and the vertical transform type.

13. An encoding device comprising:

7. A program for causing a computer to execute the method according to claim 1 or 2.

8. A program for causing a computer to execute the method according to claim 3 or 4.

Images