Method for decoding conversion block from bit stream, method for encoding conversion block into bit stream, decoding device, encoding device, and computer program

Abstract

To provide a system and a method for decoding a sub-block of a residual coefficient of a conversion block from a video bit stream.SOLUTION: A functional module of a video decoder includes: determining whether sign bit hiding is used for a sub-block, wherein the determining is based on the value of a conversion skip flag determined for the sub-block and the value of a sign bit hiding flag related to the sub-block; decoding sign bits in a number equal to the number of significant coefficients in the sub-block when the sign bit hiding is not used; and reconstructing a residual coefficient of the sub-block by using the decoded sign bits to decode the sub-block.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] REFERENCE TO RELATED APPLICATIONS This application claims the benefit under 35 USC § 119 of the filing date of Australian Patent Application No. 2020201753, filed March 10, 2020, the entirety of which is incorporated herein by reference as if fully set forth herein.

[0002] The present invention relates generally to digital video signal processing, and more particularly to a method, apparatus and system for encoding and decoding blocks of video samples. The present invention also relates to a computer program product including a computer-readable medium having recorded thereon a computer program for encoding and decoding blocks of video samples. [Background technology]

[0003] Many applications for video coding currently exist, including applications for the transmission and storage of video data. Many video coding standards have been developed, and others are currently under development. Recent developments in video coding standardization have led to the formation of a group called the "Joint Video Experts Team" (JVET). The Joint Video Experts Team (JVET) includes members of the International Telecommunication Union (ITU) Telecommunication Standardization Sector (ITU-T) Study Group 16, Question 6 (SG16 / Q6), also known as the "Video Coding Experts Group" (VCEG), and members of the International Organization for Standardization / International Electrotechnical Commission Joint Technical Committee 1 / Subcommittee 29 / Working Group 11 (ISO / IEC JTC1 / SC29 / WG11), also known as the "Moving Picture Experts group" (MPEG).

[0004] The Joint Video Experts Team (JVET) analyzed the responses at its 10th meeting in San Diego, USA, and issued a Call for Proposals (CfP). The submitted responses demonstrated video compression capabilities that significantly exceeded those of the current state-of-the-art video compression standard, i.e., "High Efficiency Video Coding" (HEVC). Based on this outperformance, it was decided to initiate a project to develop a new video compression standard, to be named "versatile video coding" (VVC). VVC is expected to address the continuing demand for ever-higher compression performance, particularly as video formats increase in capacity (e.g., at higher resolutions and higher frame rates) and address increasing market demand for service delivery over WANs, where bandwidth costs are relatively high. At the same time, VVC must be implementable on modern silicon processes and provide an acceptable tradeoff between achieved performance and implementation cost (e.g., in terms of silicon area, CPU processor load, memory usage, and bandwidth).

[0005] Video data includes a sequence of frames of image data, each containing one or more color channels. Generally, one primary color channel and two secondary color channels are required. The primary color channel is commonly referred to as the "luma" channel, and the secondary color channels are commonly referred to as the "chroma" channels. Video data is typically displayed in the RGB (red-green-blue) color space, which has a high degree of correlation between each of the three components. The video data representation viewed by an encoder or decoder often uses a color space such as YCbCr. YCbCr concentrates luminance, mapped to "luma" according to a transfer function, in the Y (primary) channel and chroma in the Cb and Cr (secondary) channels. Furthermore, the Cb and Cr channels may be spatially sampled (subsampled) at a lower rate compared to the luma channel, e.g., half horizontally and half vertically, known as the "4:2:0 chroma format." The 4:2:0 chroma format is commonly used for Internet video streaming, broadcast television, and Blu-ray. TMIt is commonly used in "consumer" applications, such as storage to disk. Subsampling the Cb and Cr channels at half the rate horizontally and not subsampling vertically is known as a "4:2:2 chroma format." The 4:2:2 chroma format is typically used in professional applications, including capturing footage for filmmaking and other applications. The higher sampling rate of the 4:2:2 chroma format makes the resulting video more resilient to editing operations such as color grading. Before distribution to consumers, 4:2:2 chroma format material is often converted to a 4:2:0 chroma format and then encoded for distribution to consumers. In addition to the chroma format, video is also characterized by its resolution and frame rate. Example resolutions are ultra-high definition (UHD) with a resolution of 3840x2160, or "8K" with a resolution of 7680x4320, with example frame rates of 60 or 120 Hz. The luma sample rate may range from approximately 500 megasamples / second to several gigasamples / second. For a 4:2:0 chroma format, the sample rate of each chroma channel is one-quarter of the luma sample rate, and for a 4:2:2 chroma format, the sample rate of each chroma channel is one-half of the luma sample rate.

[0006] The VVC standard is a "block-based" codec, where a frame is first divided into a square array of regions known as "coding tree units" (CTUs). CTUs typically occupy a relatively large area, such as 128 x 128 luma samples. However, the CTUs at the right and bottom of each frame may be smaller in area. Each CTU is associated with a "coding tree" for the luma channel and additional coding trees for the chroma channels. The coding trees define a decomposition of the CTU's region into a series of blocks, also called "coding blocks" (CBs). It is also possible for a single coding tree to specify blocks for both the luma and chroma channels; in that case, the set of juxtaposed coding blocks is called a "coding unit" (CU); i.e., each CU has a coding block for each color channel. The CBs are processed for encoding or decoding in a specific order. As a result of the use of the 4:2:0 chroma format, a CTU having a luma coding tree for a 128x128 luma sample region has a corresponding chroma coding tree for a 64x64 chroma sample region co-located with the 128x128 luma sample region. When a single coding tree is used for the luma and chroma channels, the collection of collocated blocks for a given area is generally referred to as a "unit," e.g., the above-mentioned CU, as well as a "prediction unit" (PU) and a "transform unit" (TU). When separate coding trees are used for a given area, the above-mentioned CB, as well as a "prediction block" (PB) and a "transform block" (TB) are used.

[0007] Despite the above distinction between "unit" and "block", the term "block" may be used as a general term for an area or region of a frame where an operation is applied to all color channels.

[0008] For each CU, a prediction unit (PU) is generated for the contents (sample values) of the corresponding region of frame data ("prediction unit"). If the PU is generated from sample values in a previously signaled frame, the prediction is called inter-prediction. If the PU is generated from previous samples in the same frame, the prediction is called intra-prediction. Furthermore, a representation of the difference (or "residual" in the spatial domain) between the prediction and the contents of the region seen at the input to the encoder is formed. The differences for each color channel are transformed and coded as a block of residual coefficients, which can form one or more TUs for a given CU. The residual coefficients can be transformed by a transform, such as a discrete cosine transform (DCT), a discrete sine transform (DST), or other transform, to generate a final block of transform coefficients that substantially decorrelate the residual samples. Substantial coding gain can be achieved by quantizing the transform coefficients. The quantized transform coefficients are then traversed in an order, such as a backward diagonal scan, and each coefficient is coded by an entropy encoder. Entropy coding consists of representing each coefficient with a syntax element, each of which is binarized. The binarized syntax elements are then either further coded by a context-adaptive binary arithmetic coder (CABAC) or passed through to the bitstream ("bypass coding").

[0009] For some classes of video content, such as screen content, it may be advantageous to avoid performing a transform. If a transform is to be avoided, the residual coefficients are quantized, traversed, and coded. Because the statistics of the residual coefficients are not the same as those of the transform coefficients, it is generally advantageous for the residual coefficients to be coded using a process different from the coding process for the transform coefficients. Typical methods used to code the residual coefficients include a "regular residual coding" (RRC) process and a "transform-skip residual coding" (TSRC) process, with a particular one of the processes being selected for a block depending on whether a transform has been performed.

[0010] In some use cases, it may be desirable to compress video data losslessly (i.e., without any coding loss). CUs may be losslessly encoded by skipping both the transform and quantization steps. In the TSRC process, quantization can be avoided by setting the "quantization parameter" to a value that does not indicate quantization. However, as noted above, the TSRC process is only suitable for a class of video content, such as screen content. Therefore, forcing lossless encoding of video data to use the TSRC process is suboptimal. It is desirable for lossless encoding to have more flexible options available depending on the statistics of the video data being encoded, while minimizing the amount of additional logic required to support the additional flexibility. Summary of the Invention

[0011] It is an object of the present invention to substantially overcome, or at least ameliorate, one or more disadvantages of existing arrangements.

[0012] One aspect of the present invention provides a method for decoding a sub-block of residual coefficients of a transform block from a video bitstream, the method including: determining whether sign bit hiding is used for the sub-block, the determination being based on a value of a transform skip flag determined for the sub-block and a value of a sign bit hiding flag associated with the sub-block; and if sign bit hiding is not used, decoding a number of sign bits equal to the number of significant coefficients in the sub-block; and decoding the sub-block by reconstructing the residual coefficients of the sub-block using the decoded sign bits.

[0013] According to another aspect, if the sign bit hiding flag has a value of TRUE, the transform skip flag has a value of FALSE, and the difference between the first significant position and the last significant position of the subblock is greater than 3, then sign bit hiding is used.

[0014] According to another aspect, if the sign bit hiding flag has a value of TRUE and the transform skip flag has a value of TRUE, then sign bit hiding is not used.

[0015] According to another aspect, if it is determined that sign bit hiding is used, the method further comprises decoding a number of sign bits equal to the number of significant coefficients in the sub-block minus one, and determining an additional sign bit from a sum of the parities of the significant coefficients of the sub-block.

[0016] Another aspect of the present invention provides a method for decoding a sub-block of residual coefficients of a transform block from a video bitstream, the method including: determining whether sign bit hiding is used for the sub-block, the determination being based on a value of a sign bit hiding flag and a value of a quantization parameter associated with the sub-block; and if sign bit hiding is not used, decoding a number of sign bits equal to the number of significant coefficients in the sub-block; and decoding the sub-block by reconstructing the residual coefficients of the sub-block using the decoded sign bits.

[0017] According to another aspect, if the sign bit hiding flag has a value of TRUE and the quantization parameter is equal to 4, then sign bit hiding is not used.

[0018] According to another aspect, if the sign bit hiding flag has a value of TRUE, the quantization parameter is not equal to 4, and the difference between the first significant position and the last significant position of the subblock is greater than 3, then sign bit hiding is used.

[0019] Another aspect of the present invention provides a method for decoding a sub-block of residual coefficients of a transform block from a video bitstream, the method including: determining whether sign bit hiding is used for the sub-block, the determination being based on a value of a sign bit hiding flag and a value of a TSRC disabled flag; if sign bit hiding is not used, decoding a number of sign bits equal to the number of significant coefficients in the sub-block; and decoding the sub-block by reconstructing the residual coefficients of the sub-block using the decoded sign bits.

[0020] According to another aspect, sign bit hiding is used when the sign bit hiding flag has a value of TRUE, the TSRC invalid flag has a value of FALSE, and the difference between the first significant position and the last significant position of the subblock is greater than 3.

[0021] According to another aspect, the sign bit hiding flag has a value of TRUE and the TSRC invalid flag has a value of TRUE.

[0022] Another aspect of the present invention provides a non-transitory computer-readable medium having stored thereon a computer program for implementing a method for decoding a sub-block of residual coefficients of a transform block from a video bitstream, the method including: determining whether sign bit hiding is used for the sub-block, the determination being based on a value of a transform skip flag determined for the sub-block and a value of a sign bit hiding flag associated with the sub-block; and if sign bit hiding is not used, decoding a number of sign bits equal to the number of significant coefficients in the sub-block; and decoding the sub-block by reconstructing the residual coefficients of the sub-block using the decoded sign bits.

[0023] Another aspect of the present invention provides a system having a memory and a processor, the processor configured to execute code stored in the memory to perform a method for decoding a sub-block of residual coefficients of a transform block from a video bitstream, the method including determining whether sign bit hiding is used for the sub-block, the determination being based on a value of a transform skip flag determined for the sub-block and a value of a sign bit hiding flag associated with the sub-block, and if sign bit hiding is not used, decoding a number of code bits equal to the number of significant coefficients in the sub-block, and decoding the sub-block by reconstructing the residual coefficients of the sub-block using the decoded code bits.

[0024] Another aspect of the present invention provides a video decoder configured to receive a sub-block of residual coefficients of a transform block from a video bitstream, determine whether sign bit hiding is used for the sub-block, the determination being based on the value of a transform skip flag determined for the sub-block and the value of a sign bit hiding flag associated with the sub-block, and if sign bit hiding is not used, decode the sub-block by decoding a number of code bits equal to the number of significant coefficients in the sub-block and using the decoded code bits to reconstruct the residual coefficients of the sub-block.

[0025] Other aspects are also described. [Brief explanation of the drawings]

[0026] At least one embodiment of the present invention will now be described with reference to the following drawings and appendices. [Figure 1] FIG. 1 is a schematic block diagram illustrating a video encoding and decoding system. [Figure 2A] FIG. 2A forms a schematic block diagram of a general-purpose computer system capable of implementing either or both of the video encoding and decoding systems of FIG. [Figure 2B]FIG. 2B forms a schematic block diagram of a general-purpose computer system capable of implementing either or both of the video encoding and decoding systems of FIG. [Figure 3] FIG. 3 is a schematic block diagram illustrating the functional modules of a video encoder. [Figure 4] FIG. 4 is a schematic block diagram showing the functional modules of a video decoder. [Figure 5] FIG. 5 is a schematic block diagram illustrating possible divisions of a block into one or more blocks in a generic video coding tree structure. [Figure 6] FIG. 6 is a schematic diagram of the data flow for achieving an accepted division of a block into one or more blocks in a generic video coding tree structure. [Figure 7A] FIG. 7A shows an example of division of a coding tree unit (CTU) into several coding units (CUs). [Figure 7B] FIG. 7B shows an example of division of a coding tree unit (CTU) into several coding units (CUs). [Figure 8A] FIG. 8A shows a two-level back diagonal scan. [Figure 8B] FIG. 8B shows a two-level forward diagonal scan. [Figure 9] FIG. 9 illustrates a method for encoding a transform block of residual coefficients. [Figure 10] FIG. 10 illustrates a method for decoding a transform block of residual coefficients. [Figure 11] FIG. 11 illustrates a method for quantizing a transform block of residual coefficients performed by the method of FIG. [Figure 12] FIG. 12 illustrates a method for scaling a transform block of quantized coefficients performed by the method of FIG. [Figure 13] FIG. 13 illustrates a method for encoding sub-blocks of quantized coefficients performed by the method of FIG. [Figure 14]FIG. 14 illustrates a method for decoding a sub-block of quantized coefficients performed by the method of FIG. DETAILED DESCRIPTION OF THE INVENTION

[0027] When referring to steps and / or features having the same reference number in one or more of the accompanying drawings, those steps and / or features have the same function or operation for the purposes of this specification, unless a contrary intention appears.

[0028] As mentioned above, it may be desirable for lossless encoding to be supported by the existing building blocks of a codec. However, using TSRC processing exclusively for lossless encoding can produce suboptimal encoding performance, because the diverse classes of video data encoded in a lossless manner cannot be guaranteed to exhibit the statistical properties for which the TSRC processing was designed. Therefore, greater flexibility in the selection of high-level building blocks that lossless encoding can use allows for superior encoding performance with minimal additional complexity to the overall design.

[0029] 1 is a schematic block diagram illustrating functional modules of a video encoding and decoding system 100. System 100 includes a source device 110 and a destination device 130. A communication channel 120 is used to communicate encoded video information from source device 110 to destination device 130. In some configurations, source device 110 and destination device 130 may comprise respective mobile phone handsets or “smartphones,” in which case communication channel 120 is a wireless channel. In other configurations, source device 110 and destination device 130 may comprise video conferencing equipment, in which case communication channel 120 is typically a wired channel, such as an Internet connection. Furthermore, source device 110 and destination device 130 may comprise any of a wide range of devices, including devices supporting over-the-air television broadcasts, cable television applications, Internet video applications (including streaming), and applications in which encoded video data is captured on some computer-readable storage medium, such as a hard disk drive in a file server.

[0030] 1, source device 110 includes video source 112, video encoder 114, and transmitter 116. Video source 112 typically comprises a source of captured video frame data (shown as 113), such as an imaging sensor, a previously captured video sequence stored on a non-transitory recording medium, or video sourced from a remote imaging sensor. Video source 112 may also be the output of a computer graphics card, e.g., displaying the video output of an operating system and various applications running on a computing device such as a tablet computer. Examples of source device 110 that may include an imaging sensor as video source 112 include smartphones, video cameras, professional video cameras, and network video cameras.

[0031] Video encoder 114 converts (or "encodes") imaged frame data (indicated by arrow 113) from video source 112 into a bitstream (indicated by arrow 115), as further described with reference to Figure 3. Bitstream 115 is transmitted by transmitter 116 over communication channel 120 as coded video data (or "coded video information"). Bitstream 115 may also be stored in non-transitory storage device 122, such as "flash" memory or a hard disk drive, until later transmitted over communication channel 120, or in lieu of transmission over communication channel 120.

[0032] Destination device 130 includes a receiver 132, a video decoder 134, and a display device 136. Receiver 132 receives encoded video data from communication channel 120 and passes the received video data as a bitstream to video decoder 134 (indicated by arrow 133). Video decoder 134 then outputs decoded frame data to display device 136 (indicated by arrow 135) to reproduce the video data. Decoded frame data 135 has the same chroma format as frame data 113. Examples of display device 136 include a cathode ray tube, a smartphone, a tablet computer, a computer monitor, or a liquid crystal display such as a standalone television set. It is also possible for the functionality of each of source device 110 and destination device 130 to be implemented in a single device, examples of which include a mobile phone handset and a tablet computer.

[0033] Notwithstanding the above exemplary devices, each of source device 110 and destination device 130 may be configured within a general-purpose computing system, typically via a combination of hardware and software components. Figure 2A illustrates such a computer system 200, including a computer module 201, input devices such as a keyboard 202, a mouse pointer device 203, a scanner 226, a camera 227 that may be configured as a video source 112, and a microphone 280, and output devices including a printer 215, a display device 214 that may be configured as a display device 136, and speakers 217. An external modulator-demodulator (modem) transceiver device 216 may be used by computer module 201 to communicate with a communications network 220 via connection 221. Communications network 220, which may represent communications channel 120, may be a wide area network (WAN) such as the Internet, a cellular telecommunications network, or a private WAN. If connection 221 is a telephone line, modem 216 may be a conventional "dial-up" modem. Alternatively, modem 216 may be a broadband modem if connection 221 is a high-capacity (e.g., cable or optical) connection. A wireless modem may also be used for wireless connection to communication network 220. Transceiver device 216 may provide the functionality of transmitter 116 and receiver 132, and communication channel 120 may be embodied within connection 221.

[0034] The computer module 201 typically includes at least one processor unit 205 and a memory unit 206. For example, the memory unit 206 can include semiconductor random access memory (RAM) and semiconductor read-only memory (ROM). The computer module 201 also includes several input / output (I / O) interfaces, including an audio / video interface 207 that couples to a video display 214, speakers 217, and a microphone 280; an I / O interface 213 that couples to a keyboard 202, a mouse 203, a scanner 226, a camera 227, and optionally a joystick or other human interface device (not shown); and an interface 208 for an external modem 216 and a printer 215. The signal from the audio / video interface 207 to the computer monitor 214 is typically an output of a computer graphics card. In some implementations, the modem 216 may be incorporated into the computer module 201, for example, in the interface 208. The computer module 201 also has a local network interface 211, which allows the computer system 200 to be coupled to a local area communications network 222, known as a local area network (LAN), via a connection 223. As shown in Figure 2A, the local communications network 222 can also be coupled to a wide network 220 via a connection 224, which typically includes a so-called "firewall" device or device of similar functionality. The local network interface 211 can be an Ethernet TM Circuit Card, Bluetooth TM This may include a wireless or IEEE 802.11 wireless configuration, although many other types of interfaces may be implemented for interface 211. Local network interface 211 may also provide the functionality of transmitter 116, and receiver 132 and communication channel 120 may also be embodied in local communications network 222.

[0035] The I / O interfaces 208 and 213 can provide either or both serial and parallel connectivity, the former typically being implemented according to the Universal Serial Bus (USB) standard and having a corresponding USB connector (not shown). A storage device 209 is provided and typically includes a hard disk drive (HDD) 210. Other storage devices, such as floppy disk drives and magnetic tape drives (not shown), may also be used. An optical disk drive 212 is typically provided to serve as a non-volatile source of data. Optical disks (e.g., CD-ROMs, DVDs, Blu-ray Discs) may be used. TM ), USB-RAM, portable, external hard drives, and portable memory devices such as floppy disks, for example, can be used as suitable sources of data for computer system 200. Typically, any of HDD 210, optical drive 212, and networks 220 and 222 may be configured to operate as video source 112 or as a destination for decoded video data to be stored for playback via display 214. Source device 110 and destination device 130 of system 100 may be embodied in computer system 200.

[0036] The components 205 to 213 of the computer module 201 typically communicate via an interconnected bus 204 and in a manner which results in a conventional mode of operation of the computer system 200 known to those skilled in the art. For example, the processor 205 is coupled to the system bus 204 using a connection 218. Similarly, the memory 206 and optical disk drive 212 are coupled to the system bus 204 by a connection 219. Examples of computers on which the above-described arrangements can be executed include IBM-PCs and compatibles, Sun SPARC stations, Apple Macs, TM or a similar computer system.

[0037] Where appropriate or necessary, video encoder 114 and video decoder 134, as well as the methods described below, may be implemented using computer system 200. Specifically, video encoder 114, video decoder 134, and the methods described may be implemented as one or more software application programs 233 executable within computer system 200. Specifically, video encoder 114, video decoder 134, and the steps of the methods described are performed by instructions 231 (see FIG. 2B ) in software 233 executing within computer system 200. The software instructions 231 may be formed as one or more code modules, each for performing one or more specific tasks. The software may also be divided into two separate portions, where a first portion and corresponding code module performs the methods described and a second portion and corresponding code module manages a user interface between the first portion and a user.

[0038] The software may be stored on a computer-readable medium, including, for example, the storage devices described below. The software is loaded from the computer-readable medium into computer system 200 and then executed by computer system 200. Such software or a computer-readable medium having a computer program recorded thereon is a computer program product. Use of the computer program product in computer system 200 preferably results in an advantageous apparatus for implementing video encoder 114, video decoder 134, and the methods described.

[0039] The software 233 is typically stored on the HDD 210 or in the memory 206. The software is loaded into the computer system 200 from a computer-readable medium and executed by the computer system 200. Thus, for example, the software 233 may be stored on an optically readable disk storage medium (e.g., a CD-ROM) 225 that is read by the optical disk drive 212.

[0040] In some cases, application program 233 may be encoded on one or more CD-ROMs 225 and supplied to the user and read via corresponding drive 212, or alternatively, may be read by the user from network 220 or 222. Additionally, software may also be loaded into computer system 200 from other computer-readable media. Computer-readable storage media refers to any non-transitory, tangible storage medium that provides recorded instructions and / or data to computer system 200 for execution and / or processing. Examples of such storage media are floppy disks, magnetic tape, CD-ROMs, DVDs, Blu-ray Discs, and the like. TM , hard disk drives, ROMs or integrated circuits, USB memory, magneto-optical disks, or computer-readable cards such as PCMCIA cards, etc., whether such devices are internal or external to the computer module 201. Examples of transitory or non-tangible computer-readable transmission media that may also participate in providing software, application programs, instructions and / or video data or encoded video data to the computer module 401 include wireless or infrared transmission channels, as well as network connections to another computer or networked device, and the Internet or intranets, including email transmissions and information recorded on websites and the like.

[0041] A second portion of application program 233 and the corresponding code modules described above may be executed to implement one or more graphical user interfaces (GUIs) that are rendered or otherwise represented on display 214. Typically through manipulation of keyboard 202 and mouse 203, a user of application and computer system 200 can manipulate the interface in a functionally adaptable manner to provide control commands and / or input to the application associated with the GUI. Other forms of functionally adaptable user interfaces may also be implemented, such as an audio interface that utilizes speech prompts output via speaker 217 and user voice commands input via microphone 280.

[0042] Figure 2B is a detailed schematic block diagram of processor 205 and "memory" 234. Memory 234 represents a logical collection of all memory modules (including HDD 209 and semiconductor memory 206) accessible by computer module 201 of Figure 2A.

[0043] When the computer module 201 is first powered on, a power-on self-test (POST) program 250 is executed. The POST program 250 is typically stored in the ROM 249 of the semiconductor memory 206 of FIG. 2A. Hardware devices, such as the ROM 249, that store software are sometimes referred to as firmware. The POST program 250 inspects the hardware within the computer module 201 to ensure proper functioning and typically checks the processor 205, memory 234 (209, 206), and a basic input / output system software (BIOS) module 251, which is typically also stored in the ROM 249, for correct operation. If the POST program 250 runs successfully, the BIOS 251 boots the hard disk drive 210 of FIG. 2A. Booting the hard disk drive 210 executes a bootstrap loader program 252, resident on the hard disk drive 210, via the processor 205. This loads the operating system 253 into the RAM memory 206, upon which the operating system 253 begins operation. Operating system 253 is a system-level application executable by processor 205 and performs a variety of high-level functions, including processor management, memory management, device management, storage management, software application interface, and general-purpose user interface.

[0044] Operating system 253 manages memory 234 (209, 206) to ensure that each process or application running on computer module 201 has enough memory to execute without conflicting with memory allocated to another process. Furthermore, the different types of memory available in computer system 200 of FIG. 2A must be used appropriately so that each process can execute effectively. Thus, aggregate memory 234 is not intended to indicate how specific segments of memory are allocated (unless otherwise noted), but rather is intended to provide a general view of memory accessible by computer system 200 and how such segments are used.

[0045] As shown in FIG. 2B, processor 205 includes a number of functional modules, including a control unit 239, an arithmetic logic unit (ALU) 240, and a local or internal memory 248, sometimes referred to as a cache memory. Cache memory 248 typically includes a number of storage registers 244-246 within a register section. One or more internal buses 241 functionally interconnect these functional modules. Processor 205 also typically has one or more interfaces 242 for communicating with external devices via system bus 204 using connection 218. Memory 234 is coupled to bus 204 using connection 219.

[0046] Application program 233 includes a sequence of instructions 231, which may include conditional branch and loop instructions. Program 233 may also include data 232 used in the execution of program 233. Instructions 231 and data 232 are stored in memory locations 228, 229, 230 and 235, 236, 237, respectively. Depending on the relative sizes of instruction 231 and memory locations 228-230, a particular instruction may be stored in a single memory location, as indicated by the instruction shown in memory location 230. Alternatively, the instruction may be segmented into several parts, each stored in a separate memory location, as indicated by the instruction segments shown in memory locations 228 and 229.

[0047] Generally, the processor 205 is given a set of instructions to execute within it. The processor 205 waits for a subsequent input, to which the processor 205 responds by executing another set of instructions. Each input may come from one or more of several sources, including data generated by one or more of the input devices 202, 203, data received from an external source via one of the networks 220, 202, data retrieved from one of the storage devices 206, 209, or data retrieved from a storage medium 225 inserted into a corresponding reader 212, all shown in FIG. 2A. Execution of the set of instructions may result in data output. Execution may also include storing data or variables to memory 234.

[0048] Video encoder 114, video decoder 134, and the described methods may use input variables 254 that are stored in corresponding memory locations 255, 256, 257 in memory 234. Video encoder 114, video decoder 134, and the described methods generate output variables 261, which are stored in corresponding memory locations 262, 263, 264 in memory 234. Intermediate variables 258 may be stored in memory locations 259, 260, 266, and 267.

[0049] Referring to processor 205 of FIG. 2B, registers 244, 245, 246, arithmetic logic unit (ALU) 240, and control unit 239 cooperate to perform the sequence of micro-operations required to perform a "fetch, decode, and execute" cycle for every instruction in the instruction set that makes up program 233. Each fetch, decode, and execute cycle is A fetch operation to fetch or read an instruction 231 from memory locations 228, 229, 230 A decode operation in which the control unit 239 determines which instructions have been fetched. The control unit 239 and / or the ALU 240 perform the operations to execute the instructions. It has.

[0050] Thereafter, a further fetch, decode, and execute cycle of the next instruction can be performed. Similarly, a store cycle can be performed in which control unit 239 stores or writes a value to memory location 232.

[0051] Each step or sub-process in the methods of Figures 19-14 described below is associated with one or more segments of program 233 and is typically performed by register sections 244, 245, 247, ALU 240, and control unit 239 within processor 205 working together to perform a fetch, decode, and execute cycle for all instructions in the instruction set for the noted segment of program 233.

[0052] Figure 3 is a schematic block diagram illustrating the functional modules of the video encoder 114. Figure 4 is a schematic block diagram illustrating the functional modules of the video decoder 134. Generally, data passes between the functional modules of the video decoder 134 and the video encoder 114 in groups of samples or coefficients, such as division of a block into fixed-size sub-blocks, or as an array. The video encoder 114 and the video decoder 134 can be implemented using a general-purpose computer system 200, as shown in Figures 2A and 2B, with the various functional modules being realized by software executable within the computer system 200, such as one or more software code modules of a software application program 233 resident on a hard disk drive 205 and controlled during its execution by the processor 205, or by dedicated hardware within the computer system 200. Alternatively, the video encoder 114 and the video decoder 134 may be implemented by a combination of software executable within the computer system 200 and dedicated hardware. The video encoder 114, the video decoder 134, and the described methods may alternatively be implemented by dedicated hardware, such as one or more integrated circuits that perform the functions or sub-functions of the described methods. Such dedicated hardware may include a graphics processing unit (GPU), a digital signal processor (DSP), an application specific standard product (ASSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or one or more microprocessors and associated memory. In particular, the video encoder 114 includes modules 310-386, and the video decoder 134 includes modules 420-496, which may each be implemented as one or more software code modules of the software application program 233.

[0053] The video encoder 114 of FIG. 3 is an example of a generic video coding (VVC) video encoding pipeline, although other video codecs may be used to perform the processing stages described herein. The video encoder 114 receives captured frame data 113, such as a series of frames, each of which includes one or more color channels. The frame data 113 may be in any chroma format, such as 4:0:0, 4:2:0, 4:2:2, or 4:4:4. The block partitioner 310 first divides the frame data 113 into CTUs, generally square in shape, with a specific size for the CTUs configured to be used. The CTU size may be, for example, 64x64, 128x128, or 256x256 luma samples. The block partitioner 310 further divides each CTU into one or more CBs according to the luma coding tree and chroma coding tree. The CBs may have various sizes and include both square and non-square aspect ratios. In the VVC standard, CBs, CUs, PUs, and TUs always have side lengths that are powers of 2. Thus, the current CB, denoted as 312, is output from the block partitioner 310 and proceeds according to iterations over one or more blocks of the CTU according to the chroma coding tree and luma coding tree of the CTU. Options for partitioning a CTU into CBs are further described below with reference to Figures 5 and 6.

[0054] The CTUs resulting from the initial partitioning of frame data 113 may be scanned in raster scan order and grouped into one or more "slices." A slice may be an "intra" (or "I") slice. An intra slice (I slice) indicates that all CUs within the slice are intra-predicted. Alternatively, a slice may be uni- or bi-predictive ("P" or "B" slice, respectively), indicating the further availability of uni- and bi-prediction in the slice, respectively.

[0055] For each CTU, the video encoder 114 operates in two stages. In the first stage (called the "search" stage), the block partitioner 310 tests various potential configurations of the coding tree. Each potential configuration of the coding tree has an associated "candidate" CB. The first stage involves testing various candidate CBs to select the CB that provides high compression efficiency with low distortion. This testing typically involves Lagrangian optimization, whereby the candidate CBs are evaluated based on a weighted combination of rate (coding cost) and distortion (error with respect to the input frame data 113). The "best" candidate CB (the CB with the lowest evaluated rate / distortion) is selected for subsequent encoding into the bitstream 115. Evaluation of the candidate CBs includes the options of using a CB for a given area, further dividing the area according to various partitioning options and encoding each of the resulting smaller areas with additional CBs, or further dividing the area. Consequently, both the CB and the coding tree itself are selected in the search stage.

[0056] For each CB, e.g., CB 312, the video encoder 114 generates a predictive block (PB), indicated by arrow 320. The PB 320 is a prediction of the content of the associated CB 312. A subtractor module 322 generates a difference, indicated as 324 (or "residual," referring to the difference being in the spatial domain), between the PB 320 and the CB 312. The residual 324 is the block-sized difference between corresponding samples in the PB 320 and the CB 312. The residual 324 is transformed and quantized, and represented as a transform block (TB), indicated by arrow 336. The PB 320 and associated TB 336 are typically selected from one of many possible candidate CBs, for example, based on estimated cost or distortion.

[0057] A candidate coding block (CB) is a CB that results from one of the prediction modes available to the video encoder 114 for an associated PB and the resulting residual. Each candidate CB results in one or more corresponding TBs. The TBs 336 are quantized, transformed representations of the residuals 324. When combined with the predicted PBs in the video decoder 114, the TBs 336 reduce the difference between the decoded CB and the original CBs 312, at the expense of additional signal in the bitstream.

[0058] Thus, each candidate coding block (CB), i.e., a prediction block (PB) combined with a transform block (TB), has an associated coding cost (or "rate") and an associated differential (or "distortion"). The rate is typically measured in bits. The distortion of a CB is typically estimated as a sample-value differential, such as the sum of absolute differences (SAD) or sum of squared differences (SSD). The estimate obtained from each candidate PB is determined by a mode selector 386 using the residual 324 to determine a prediction mode (represented by arrow 388). Estimating the coding cost associated with each candidate prediction mode and the corresponding residual coding can be performed at a significantly lower cost than entropy coding of the residual. Thus, a large number of candidate modes can be evaluated to determine the optimal mode for rate-distortion detection.

[0059] Determining the optimal mode from a rate-distortion perspective is typically accomplished using a variant of Lagrangian optimization. Selecting a prediction mode 388 typically involves determining the coding cost for the residual data resulting from applying a particular prediction mode. The coding cost can be approximated by using the "sum of absolute transform differences" (SATD), thereby using a relatively simple transform, such as the Hadamard transform, to obtain an estimated transform residual cost. In some embodiments using a relatively simple transform, the cost resulting from the simplified estimation method is monotonically related to the actual cost that would otherwise be determined from a full evaluation. In embodiments with monotonically related estimated costs, the simplified estimation method can be used to make the same decision (i.e., prediction mode) while reducing the complexity of the video encoder 114. To allow for possible non-monotonicity in the relationship between the estimated cost and the actual cost, the simplified estimation method can be used to generate a list of best candidates. Non-monotonicity may arise, for example, from additional mode decisions available for encoding the residual data. The list of best candidates may be any number of candidates. Using the best candidates, a more thorough search can be performed to establish the optimal mode selection for encoding the residual data for each of the candidates, allowing for the final selection of the prediction mode 388 along with other mode decisions.

[0060] Prediction modes are broadly classified into two categories. The first category is "intra-frame prediction" (also called "intra prediction"). In intra-frame prediction, a prediction for a block is generated, and the generation method may use other samples obtained from the current frame. Types of intra prediction include intra-planar, intra-DC, intra-angle, and matrix-weighted intra prediction (MIP). For intra-predicted PB, different intra-prediction modes can be used for luma and chroma, and therefore intra-prediction is mainly described with respect to operations on PB. Furthermore, chroma CB may be predicted from co-located luma samples by cross-component linear model prediction.

[0061] The second category of prediction modes is "interframe prediction" (also called "inter prediction"). In interframe prediction, a prediction of a block is generated using samples from one or two frames that precede the current frame in the bitstream in the order in which the frames are coded. Furthermore, for interframe prediction, a single coding tree is typically used for both the luma and chroma channels. The coding order of frames in the bitstream may differ from the order of frames at capture or display. If one frame is used for prediction, the block is said to be "uni-predictive" and has one associated motion vector. If two frames are used for prediction, the block is said to be "bi-predictive" and has two associated motion vectors. For P slices, each CU can be intra-predicted or uni-predicted. For B slices, each CU can be intra-predicted, uni-predicted, or bi-predicted. Frames are typically coded using a "group of pictures" structure that enables temporal hierarchy of frames. Temporal hierarchy of frames allows frames to reference previous and subsequent pictures in the order in which they are displayed. The images are coded in the order required to ensure that the dependencies for decoding each frame are met.

[0062] A subcategory of inter prediction is called "skip mode." Inter prediction mode and skip mode are described as two separate modes. However, both inter prediction mode and skip mode involve motion vectors that reference blocks of samples from a previous frame. Inter prediction involves a coded motion vector delta that specifies a motion vector relative to a motion vector predictor. The motion vector predictor is obtained from a list of one or more candidate motion vectors selected by a "merge index." The coded motion vector delta provides a spatial offset to the selected motion vector prediction. Inter prediction also uses coded residuals in the bitstream 133. Skip mode uses only an index (also called a "merge index") to select one of several motion vector candidates. The selected candidate is used without further signaling. Skip mode also does not support coding of residual coefficients. When skip mode is used, the lack of coded residual coefficients means that there is no need to perform a transform for skip mode. Therefore, skip mode typically does not cause pipelining issues. Pipelining issues may occur for both intra-predicted and inter-predicted CUs. Due to the limited signaling of skip mode, skip mode is useful for achieving very high compression performance when relatively high-quality reference frames are available. Bi-predicted CUs in higher temporal layers of a random access picture group structure typically have high-quality reference pictures and motion vector candidates that accurately reflect the underlying motion.

[0063] Samples are selected according to a motion vector and a reference picture index. The motion vector and reference picture index apply to all color channels; therefore, inter-prediction is primarily described in terms of operating on PUs, not PBs. Within each category (i.e., intra- and inter-frame prediction), different techniques can be applied to generate PUs. For example, intra-prediction can use values from adjacent rows and columns of previously reconstructed samples in combination with a direction to generate a PU according to a predetermined filtering and generation process. Alternatively, a PU may be described using a small number of parameters. Inter-prediction methods can vary in the number and precision of motion parameters. Motion parameters typically include a reference frame index indicating which reference frame should be used from a list of reference frames and a spatial transformation for each reference frame, but can also include more frames, special frames, or complex affine parameters such as scaling and rotation. Furthermore, a predetermined motion refinement process can be applied to generate a dense motion estimate based on the referenced sample block.

[0064] A Lagrangian or similar optimization process can be employed to select both the optimal partitioning of the CTU into CBs (by the block partitioner 310) and the selection of the best prediction mode from multiple possibilities. Through application of a Lagrangian optimization process of candidate modes in the mode selector module 386, the prediction mode with the lowest cost measure is selected as the “best” mode. The lowest-cost mode is the selected prediction mode 388 and is also encoded into the bitstream 115 by the entropy encoder 338. The selection of prediction mode 388 by the operation of the mode selector module 386 extends the operation of the block partitioner 310. For example, candidates for the selection of prediction mode 388 can include modes applicable to a given block and also modes applicable to multiple smaller blocks collectively located with the given block. When including modes applicable to a given block and smaller co-located blocks, the candidate selection process is implicitly also a process of determining the best hierarchical decomposition of the CTU into CBs.

[0065] In a second stage of operation of the video encoder 114 (called the “encoding” stage), iteration over the selected luma coding tree and the selected chroma coding tree, and thus over each selected CB, is performed in the video encoder 114. In the iteration, the CBs are encoded into a bitstream 115, as described further herein.

[0066] The entropy encoder 338 supports both variable-length coding of syntax elements and arithmetic coding of syntax elements. Arithmetic coding is supported using the context-adaptive binary arithmetic coding (CABAC) process. Arithmetically coded syntax elements consist of a sequence of one or more "bins." Bins, like bits, have a value of "0" or "1." Bins are not coded into the bitstream 115 as discrete bits. Bins have an associated predicted (or "likelihood" or "most likely") value and associated probability, known as a "context." When the actual bin to be coded matches the predicted value, a "most probable symbol" (MPS) is coded. Coding the most probable symbol is relatively inexpensive in terms of consumed bits. When the actual bin to be coded does not match the likely value, a "least probable symbol" (LPS) is coded. Coding the least probable symbol has a relatively high cost in terms of consumed bits. Bin coding techniques allow for efficient encoding of bins with biased "0" vs. "1" probabilities. For syntax elements with two possible values (i.e., "flags"), a single bin is sufficient. For syntax elements with many possible values, a series of bins is required.

[0067] The presence of a later bin in the sequence may be determined based on the value of a previous bin in the sequence. Furthermore, each bin can be associated with two or more contexts. The selection of a particular context can depend on the previous bin of the syntax element, the bin values of adjacent syntax elements (i.e., from adjacent blocks), etc. Each time a context-encoded bin is encoded, the context (if any) selected for that bin is updated in a manner that reflects the new bin value. In this way, binary arithmetic coding schemes are said to be adaptive.

[0068] Also supported by the video encoder 114 are bins that lack context ("bypass bins"). Bypass bins are coded assuming an equal probability distribution between "0" and "1". Thus, each bin occupies one bit in the bitstream 115. The absence of context saves memory and reduces complexity, and therefore bypass bins are used when the distribution of values for a particular bin is not skewed.

[0069] The entropy encoder 338 encodes the prediction mode 388 using a combination of context coding bins and bypass coding bins. For example, if the prediction mode 388 is an intra-prediction mode, a list of "most probable modes" is generated in the video encoder 114. The list of most probable modes is typically a fixed length, such as three or six modes, and may include modes encountered in previous blocks. The context coding bin encodes a flag indicating whether the prediction mode is one of the most probable modes. If the intra-prediction mode 388 is one of the most probable modes, further signaling using bypass coding bins is encoded. The encoded further signaling indicates which most probable mode corresponds to the intra-prediction mode 388, for example, using a truncated unary bin string. Otherwise, the intra-prediction mode 388 is coded as a "remaining mode." Coding as a remaining mode expresses intra-prediction modes other than those present in the most probable mode list using an alternative syntax, such as a fixed-length code that is also coded using bypass-coded bins.

[0070] The multiplexer module 384 outputs the PB 320 according to the determined best prediction mode 388 selected from the tested prediction modes for each candidate CB. The candidate prediction modes need not include all possible prediction modes supported by the video encoder 114.

[0071] After determining and selecting PB 320 and subtracting it from the original sample block in subtractor 322, a residual represented by 324 with the lowest coding cost is obtained and subjected to lossy compression. The lossy compression process includes the steps of transform, quantization, and entropy coding. A forward primary transform module 326 applies a forward transform to the residual 324, converting it from the spatial domain to the frequency domain and generating primary transform coefficients represented by arrow 328. The primary transform coefficients 328 are passed to a forward secondary transform module 330, which performs a non-separable secondary transform (NSST) operation to generate transform coefficients represented by arrow 332. The forward primary transform is typically separable, typically using a Type II Discrete Cosine Transform (DCT-2) to transform the set of rows and then the set of columns of each block, although Type VII Discrete Sine Transform (DCT-7) and Type VIII Discrete Cosine Transform (DCT-8) may also be used, e.g., horizontally for block widths not exceeding 16 samples and vertically for block heights not exceeding 16 samples. Transforming each set of rows and columns is performed by first applying a one-dimensional transform to each row of the block to generate an intermediate result, and then applying a one-dimensional transform to each column of the intermediate result to generate a final result. The forward secondary transform is generally a non-separable transform, which is applied only to the residual of intra-predicted CUs and may nevertheless be bypassed. The forward secondary transform operates on either 16 samples (arranged as a top-left 4x4 sub-block of the primary transform coefficients 328) or 64 samples (arranged as a top-left 8x8 coefficient, arranged as four 4x4 sub-blocks of the primary transform coefficients 328). Furthermore, the matrix coefficients of the forward secondary transform are selected from multiple sets according to the intra prediction mode of the CU, so that two sets of coefficients are available for use. The use of one of the sets of matrix coefficients, i.e., bypassing the forward secondary transform, is signaled with the "nsst_index" syntax element, which is coded using truncated unary binarization to represent the value zero (no secondary transform applied), one (the first set of selected matrix coefficients), or two (the second set of selected matrix coefficients).

[0072] The video encoder 114 may also choose to skip both the primary and secondary transforms, known as "transform skip" mode. Skipping the transform is appropriate for residual data that lacks adequate correlation to reduce coding costs via representation as transform basis functions. Certain types of content, such as relatively simple computer-generated graphics, may exhibit similar behavior. When transform skip mode is used, the transform coefficients 332 are the same as the residual coefficients 324.

[0073] The transform coefficients 332 are passed to a quantizer module 334. In the module 334, quantization by a “quantization parameter” is performed to generate quantized coefficients, represented by arrow 336. The quantization parameter is constant for a given TB, thus resulting in uniform scaling for the generation of residual coefficients for the TB. Non-uniform scaling is also possible by applying a “quantization matrix,” whereby the scaling factor applied to each residual coefficient is derived from a combination of the quantization parameter and a corresponding entry in a scaling matrix, typically having a size equal to the size of the TB. The scaling matrix may have a size smaller than the size of the TB, and when applied to the TB, a nearest neighbor approach is used to provide a scaling value for each residual coefficient from a scaling matrix smaller than the TB size. The quantized coefficients 336 are provided to an entropy encoder 338 for encoding in the bitstream 115. Typically, the quantized coefficients of each TB with at least one significant quantized coefficient are scanned according to a scan pattern to generate an ordered list of values. The scan pattern typically scans the TB as a sequence of 4x4 "sub-blocks," providing a regular scanning operation at the granularity of 4x4 sets of residual coefficients, with the placement of the sub-blocks depending on the size of the TB. Additionally, prediction modes 388 and corresponding block partitioning are also coded into the bitstream 115.

[0074] As mentioned above, the video encoder 114 requires access to a frame representation that corresponds to the frame representation seen by the video decoder 134. Accordingly, the quantized coefficients 336 are also inverse quantized by an inverse quantizer module 340 to generate reconstructed transform coefficients, represented by arrow 342. The reconstructed transform coefficients 342 pass through an inverse secondary transform module 344 to generate reconstructed primary transform coefficients, represented by arrow 346. The reconstructed primary transform coefficients 346 are passed to an inverse primary transform module 348 to generate reconstructed residual samples for the CU, represented by arrow 350. The type of inverse transform performed by the inverse secondary transform module 344 corresponds to the type of forward transform performed by the forward secondary transform module 330. The type of inverse transform performed by the inverse primary transform module 348 corresponds to the type of primary transform performed by the primary transform module 326. An addition module 352 adds the reconstructed residual samples 350 and the PU 320 to generate reconstructed samples for the CU (indicated by arrow 354).

[0075] The reconstructed samples 354 are passed to a reference sample cache 356 and an in-loop filter module 368. The reference sample cache 356 is typically implemented using static RAM on the ASIC (thus avoiding costly off-chip memory accesses) and provides the minimum sample storage necessary to satisfy the dependencies for generating intra-frame PBs for subsequent CUs in the frame. The minimum dependencies typically include a "line buffer" of samples along the bottom of a row of CTUs for use by the CTU's next row and column buffering, the extent of which is set by the CTU's height. The reference sample cache 356 supplies reference samples (indicated by arrow 358) to a reference sample filter 360. The sample filter 360 applies a smoothing operation to generate filtered reference samples (indicated by arrow 362). The filtered reference samples 362 are used by an intra-frame prediction module 364 to generate an intra-predicted block of samples, represented by arrow 366. For each candidate intra-prediction mode, the intra-frame prediction module 364 generates a block of samples, i.e., 366.

[0076] The in-loop filter module 368 applies several filtering stages to the reconstructed samples 354. The filtering stages include a "deblocking filter" (DBF), which applies smoothing aligned to CU boundaries to reduce artifacts resulting from discontinuities. Another filtering stage present in the in-loop filter module 368 is an "adaptive loop filter" (ALF), which applies a Wiener-based adaptive filter to further reduce distortion. A further available filtering stage in the in-loop filter module 368 is a "sample adaptive offset" (SAO) filter. The SAO filter operates by first classifying the reconstructed samples into one or more categories and then applying an offset at the sample level according to the assigned category.

[0077] The filtered samples, represented by arrow 370, are output from the in-loop filter module 368. The filtered samples 370 are stored in a frame buffer 372. The frame buffer 372 typically has the capacity to store several pictures (e.g., up to 16) and is therefore stored in the memory 206. The frame buffer 372 is typically not stored using on-chip memory due to the large memory consumption required. Therefore, accessing the frame buffer 372 is costly in terms of memory bandwidth. The frame buffer 372 provides reference frames (represented by arrow 374) to a motion estimation module 376 and a motion compensation module 380.

[0078] The motion estimation module 376 estimates several “motion vectors” (denoted as 378), each a Cartesian spatial offset from the position of the current CB and referencing a block in one of the reference frames in the frame buffer 372. A filtered block of reference samples (denoted as 382) is generated for each motion vector. The filtered reference samples 382 form further candidate modes available for potential selection by the mode selector 386. Furthermore, for a given CU, the PU 320 may be formed using one reference block (“uni-prediction”) or two reference blocks (“bi-prediction”). For a selected motion vector, the motion compensation module 380 generates the PB 320 according to a filtering process that supports sub-pixel accuracy in the motion vector. Thus, the motion estimation module 376 (operating on many candidate motion vectors) can perform a simplified filtering process compared to that of the motion compensation module 380 (operating on only the selected candidate) to achieve reduced computational complexity. When video encoder 114 selects inter prediction for a CU, motion vector 378 is coded into bitstream 115.

[0079] 3 is described with reference to generic video coding (VVC), other video coding standards or implementations may use the processing stages of modules 310-386. The frame data 113 (and bitstream 115) may be stored in memory 206, a hard disk drive 210, a CD-ROM, a Blu-ray disc TM or other computer-readable storage medium. Additionally, frame data 113 (and bitstream 115) may be received (or transmitted) from an external source, such as a server connected to communications network 220 or a radio frequency receiver.

[0080] The video decoder 134 is shown in Figure 4. The video decoder 134 of Figure 4 is an example of a generic video coding (VVC) video decoding pipeline, although other video codecs may be used to perform the processing stages described herein. As shown in Figure 4, a bitstream 133 is input to the video decoder 134. The bitstream 133 may be stored in memory 206, a hard disk drive 210, a CD-ROM, a Blu-ray disc TM or other non-transitory computer-readable storage medium. Alternatively, bitstream 133 may be received from an external source, such as a server connected to communications network 220 or a radio frequency receiver. Bitstream 133 includes encoded syntax elements representing imaging frame data to be decoded.

[0081] The bitstream 133 is input to the entropy decoder module 420. The entropy decoder module 420 extracts syntax elements from the bitstream 133 by decoding a sequence of "bins" and passes the values of the syntax elements to other modules in the video decoder 134. One example of a syntax element extracted from the bitstream 133 is the quantized coefficients 424. The entropy decoder module 420 uses an arithmetic decoding engine to decode each syntax element as a sequence of one or more bins. Each bin can use one or more "contexts," with the context describing the probability levels used to encode the bin's "1" and "0" values. If multiple contexts are available for a given bin, a "context modeling" or "context selection" step is performed to select one of the available contexts for decoding the bin. The process of decoding the bins sequentially forms a feedback loop. The number of operations in the feedback loop is preferably minimized to enable the entropy decoder 420 to achieve a high throughput in bins per second. Context modeling relies on other properties of the bitstream known to video decoder 134 when selecting a context, i.e., a previous property of the current bin. For example, the context may be selected based on the quadtree depth of the current CU in the coding tree. The dependency is preferably based on properties known prior to decoding the bin or determined without requiring lengthy sequential processing.

[0082] The quantized coefficients 424 are input to the inverse quantizer module 428. The inverse quantizer module 428 performs inverse quantization (or “scaling”) on the quantized coefficients 424 according to the quantization parameter to generate reconstructed intermediate transform coefficients, represented by arrow 432. If the use of a non-uniform inverse quantization matrix is indicated in the bitstream 133, the video decoder 134 reads the quantization matrix from the bitstream 133 as a sequence of scaling factors and places the scaling factors in the matrix. The inverse scaling uses the quantization matrix in combination with the quantization parameter to generate reconstructed intermediate transform coefficients 432. The reconstructed intermediate transform coefficients 432 are passed to the inverse secondary transform module 436, where a secondary transform may be applied according to the decoded “nsst_index” syntax element. The “nsst_index” is decoded from the bitstream 133 by the entropy decoder 420 under execution of the processor 205. The inverse secondary transform module 436 generates reconstructed transform coefficients 440.

[0083] The reconstructed transform coefficients 440 are passed to an inverse linear transform module 444. Module 444 converts the coefficients from the frequency domain back to the spatial domain. The result of the operation of module 444 is a block of residual samples, represented by arrow 448. The block of residual samples 448 is equal in size to the corresponding CU. The type of inverse linear transform can be a Type II Discrete Cosine Transform (DCT-2), a Type VII Discrete Sine Transform (DCT-7), a Type VIII Discrete Cosine Transform (DCT-8), or a "transform skip" mode. The use of transform skip mode is signaled by a transform skip flag decoded or otherwise inferred from the bitstream 133. When transform skip mode is used, the residual samples 448 are the same as the reconstructed transform coefficients 440.

[0084] The residual samples 448 are provided to a summing module 450, where they are added to a decoded PB (represented as 452) to generate a block of reconstructed samples represented by arrow 456. The reconstructed samples 456 are provided to a reconstructed sample cache 460 and an in-loop filtering module 488, which generates a reconstructed block of frame samples represented as 492. The frame samples 492 are written to a frame buffer 496.

[0085] The reconstructed sample cache 460 operates similarly to the reconstructed sample cache 356 of the video encoder 114. The reconstructed sample cache 460 provides storage for reconstructed samples needed to intra-predict subsequent CBs without the need for memory 206 (e.g., by substituting data 232, which is typically on-chip memory). Reference samples, represented by arrow 464, are obtained from the reconstructed sample cache 460 and provided to a reference sample filter 468 to generate filtered reference samples, indicated by arrow 472. The filtered reference samples 472 are provided to an intra-frame prediction module 476. The module 476 generates blocks of intra-predicted samples, represented by arrow 480, according to the intra-prediction mode parameters 458 signaled in the bitstream 133 and decoded by the entropy decoder 420.

[0086] When the prediction mode of the CB is indicated as intra prediction in the bitstream 133, the intra prediction samples 480 form the decoded PB 452 via the multiplexer module 484. Intra prediction generates a predictive block of samples (PB), i.e., a block within one color component derived using "neighboring samples" within the same color component. Neighboring samples are samples that are adjacent to the current block and precede it in block decoding order, and therefore have already been reconstructed. When luma and chroma blocks are juxtaposed, the luma and chroma blocks can use different intra prediction modes. However, each of the two chroma channels shares the same intra prediction mode.

[0087] There are four types of intra-prediction for luma blocks. "DC intra-prediction" involves populating the PB with a single value representing the average of neighboring samples. "Planar intra-prediction" involves populating the PB with samples according to a plane using vertical and horizontal gradients and DC offsets derived from neighboring samples. "Angular intra-prediction" involves populating the PB with neighboring samples that have been filtered and propagated in a specific direction (or "angle") across the PB. In VVC, the PB can select from up to 65 angles, allowing rectangular blocks to utilize different angles not available to square blocks. "Matrix intra-prediction" involves populating the PB by multiplying a reduced set of neighboring samples by one of many available matrices available to the video decoder 134. The reduced set of neighboring samples is generated by filtering and subsampling the neighboring samples. A reduced set of predicted samples is then generated by multiplying the reduced set of samples by a matrix and adding an offset vector. The matrix and associated offset vector are selected from a number of possible matrices depending on the size of the PB, and the specific selection of matrix and offset vector is indicated by the "MIP mode" syntax element. For example, for a PB with a size larger than 8x8, there are 11 MIP modes, while for a PB of 8x8 size, there are 19 MIP modes. Finally, the PB generated by matrix intra prediction is input from a reduced set of prediction samples by interpolation.

[0088] A fifth type of intra prediction is available for chroma PB, whereby the PB is generated from collocated luma reconstructed samples according to a "cross-component linear model" (CCLM) mode. Three different CCLM modes are available, each of which uses a different model derived from adjacent luma and chroma samples. The derived model is then used to generate blocks of samples for the chroma PB from the collocated luma samples.

[0089] When the prediction mode of the CB is indicated as inter-prediction in the bitstream 133, the motion compensation module 434 uses the motion vector and reference frame index to select and filter a block of samples 498 from a frame buffer 496 to generate a block of inter-predicted samples, represented as 438. The block of samples 498 is obtained from a previously decoded frame stored in the frame buffer 496. In the bi-predictive case, two sample blocks are generated and blended together to generate samples for the decoded PB 452. The frame buffer 496 receives filtered block data 492 from the in-loop filtering module 488. Similar to the in-loop filtering module 368 of the video encoder 114, the in-loop filtering module 488 applies one of DBF, ALF, and SAO filtering operations. Generally, motion vectors are applied to both the luma and chroma channels, although the filtering process for the sub-sampled interpolated luma and chroma channels is different. The frame buffer 496 outputs the decoded video samples 135.

[0090] 5 is a schematic block diagram illustrating a set 500 of available divisions or splits of a region into one or more subregions within a generic video coding tree structure. The divisions shown in set 500 are available to block partitioner 310 of encoder 114 to divide each CTU into one or more CUs or CBs according to the coding tree, as determined by Lagrangian optimization, as described with reference to FIG.

[0091] Although set 500 shows only square regions being divided into other, possibly non-square sub-regions, it should be understood that while diagram 500 illustrates potential divisions, the containing region need not be square. If the containing region is non-square, the dimensions of the blocks resulting from the division are scaled according to the aspect ratio of the containing block. Once a region is no longer divided, i.e., at a leaf node of the coding tree, a CU occupies the region. The particular subdivision of a CTU by block partitioner 310 into one or more CUs is referred to as the CTU's "coding tree."

[0092] The process of subdividing a region into subregions must end when the resulting subregion reaches the minimum CU size. In addition to constraining CUs to prohibit block regions smaller than a predetermined minimum size, e.g., 16 samples, CUs are constrained to have a minimum width or height of 4. Other minimum values are possible for both width and height, or for width or height. The subdivision process can also end before the deepest level of decomposition, resulting in a CU larger than the minimum CU size. It is possible for no division to occur, resulting in a single CU occupying the entire CTU. A single CU occupying the entire CTU is the largest available coding unit size. The use of a subsampled chroma format, such as 4:2:0, allows the video encoder 114 and video decoder 134 configurations to end the division of regions in the chroma channels earlier than in the luma channel.

[0093] At the leaf nodes of the coding tree, there are CUs with no further subdivisions. For example, leaf node 510 contains one CU. At the non-leaf nodes of the coding tree, there may be a division into two or more further nodes, each of which forms one CU, or a non-leaf node that contains further divisions into smaller regions. At each leaf node of the coding tree, there is one coding block for each color channel. A division that ends at the same depth for both luma and chroma results in three juxtaposed CBs. A division that ends at a luma depth deeper than chroma results in multiple luma CBs juxtaposed with chroma channel CBs.

[0094] A quadtree partition 512 divides the inclusion region into four equally sized regions, as shown in Figure 5. Compared to HEVC, generic video coding (VVC) achieves further flexibility by adding horizontal halves 514 and vertical halves 516. Each of the partitions 514 and 516 divides the inclusion region into two equally sized regions. The partitions are along horizontal boundaries (514) or vertical boundaries (516) within the containing block.

[0095] Further flexibility in general-purpose video coding is achieved by adding horizontal 3-division 518 and vertical 3-division 520. 3-division 518 and 520 divide a block into three regions bounded either horizontally (518) or vertically (520) along ¼ and ¾ of the width or height of the containing region. The combination of quadtree, binary tree, and ternary tree is called a "QTBTTT." The root of the tree contains zero or more quadtree divisions (the "QT" section of the tree). Once the QT section ends, zero or more binary or ternary divisions occur (the "multi-tree" or "MT" section of the tree), ultimately terminating in a tree leaf node CB or CU. If the tree describes all color channels, the tree leaf node is a CU. If the tree describes the luma or chroma channels, the tree leaf node is a CB.

[0096] Compared to HEVC, which only supports quadtrees and therefore square blocks, QTBTTT offers many more possible CU sizes, especially when considering the possible recursive application of binary and / or ternary tree partitioning. The possibility of unusual (non-square) block sizes can be reduced by constraining partitioning options to eliminate partitions in which the block width or height is either less than four samples or is not a multiple of four samples. Generally, this constraint applies when considering luma samples. However, in the described configuration, the constraint can be applied separately to blocks for chroma channels. Applying the constraint to partitioning options for chroma channels can result in different minimum block sizes for luma and chroma, for example, if the frame data is in 4:2:0 chroma format or 4:2:2 chroma format. Each partition generates subregions whose side dimensions remain the same, are halved, or are quartered with respect to the containing region. And because the CTU size is a power of two, the side dimensions of all CUs are also powers of two.

[0097] 6 is a schematic flow diagram illustrating a data flow 600 of a QTBTTT (or "coding tree") structure used in general-purpose video coding. The QTBTTT structure is used for each CTU to define the partitioning of the CTU into one or more CUs. The QTBTTT structure for each CTU is determined by a block partitioner 310 in video encoder 114 and encoded into bitstream 115 or decoded from bitstream 133 by an entropy decoder 420 in video decoder 134. Data flow 600 further characterizes the permissible combinations available to block partitioner 310 for partitioning the CTU into one or more CUs according to the partitioning shown in FIG. 5.

[0098] Starting from the top level of the hierarchy, i.e., CTU, zero or more quadtree splits are first performed. Specifically, a quadtree (QT) split decision 610 is made by the block partitioner 310. A decision at 610 returning a "1" symbol indicates a decision to split the current node into four subnodes according to the quadtree split 512. This results in four new nodes, such as 620, being generated, and for each new node, a return is made to the QT split decision 610. Each new node is considered in raster (or Z-scan) order. Alternatively, if the QT split decision 610 indicates that no further splits should be performed (returning a "0" symbol), the quadtree split stops and a multi-tree (MT) split is then considered.

[0099] First, an MT split decision 612 is made by the block partitioner 310. At 612, a decision to perform MT splitting is indicated. Returning a "0" symbol at decision 612 indicates that no further splitting of the node into subnodes is performed. If no further splitting of the node is performed, the node is a leaf node of the coding tree and corresponds to a CU. The leaf node is output at 622. Alternatively, if MT split 612 indicates a decision to perform MT splitting (returning a "1" symbol), the block partitioner 310 proceeds to direction decision 614.

[0100] The direction decision 614 indicates the direction of the MT partition as either horizontal ("H" or "0") or vertical ("V" or "1"). The block partitioner 310 proceeds to decision 616 if decision 614 returns a "0" indicating a horizontal direction. The block partitioner 310 proceeds to decision 618 if decision 614 returns a "1" indicating a vertical direction.

[0101] In each of decisions 616 and 618, the number of partitions for the MT split is indicated as either two (bipartition or "BT" node) or three (tripartition or "TT") for the BT / TT split. That is, BT / TT split decision 616 is made by block partitioner 310 when the indicated direction from 614 is horizontal, and BT / TT split decision 618 is made by block partitioner 310 when the indicated direction from 614 is vertical.

[0102] The BT / TT split decision 616 indicates whether the horizontal split is a 2-way split 514, indicated by returning a "0," or a 3-way split 518, indicated by returning a "1." If the BT / TT split decision 616 indicates a 2-way split, then in an HBT CTU node generation step 625, two nodes are generated by the block partitioner 310 according to the horizontal 2-way split 514. If the BT / TT split 616 indicates a 3-way split, then in an HTT CTU node generation step 626, three nodes are generated by the block partitioner 310 according to the horizontal 3-way split 518.

[0103] The BT / TT split decision 618 indicates whether the vertical split is a 2-way split 516, indicated by returning a "0," or a 3-way split 520, indicated by returning a "1." If the BT / TT split 618 indicates a 2-way split, then in a VBT CTU node generation step 627, two nodes are generated by the block partitioner 310 according to the 2-way vertical split 516. If the BT / TT split 618 indicates a 3-way split, then in a VTT CTU node generation step 628, three nodes are generated by the block partitioner 310 according to the 3-way vertical split 520. For each node resulting from steps 625-628, the recursion of the data flow 600 back to the MT split decision 612 is applied in either a left-to-right or top-to-bottom order, depending on the direction 614. As a result, binary and ternary tree partitioning can be applied to generate CUs of various sizes.

[0104] 7A and 7B provide an example division 700 of a CTU 710 into several CUs or CBs. An example of a CU 712 is shown in FIG. 7A. FIG. 7A shows the spatial arrangement of CUs in the CTU 710. The example division 700 is also shown as a coding tree 720 in FIG. 7B.

[0105] At each non-leaf node in CTU 710 in FIG. 7A , e.g., nodes 714, 716, and 718, the contained nodes (which may be further split or may be CUs) are scanned or traversed in “Z-order” to create a list of nodes, represented as columns in coding tree 720. In the case of quadtree splitting, the Z-order scan is from top-left to right, followed by bottom-left to right. In the case of horizontal and vertical splits, the Z-order scan (traversal) simplifies to a top-to-bottom scan and a left-to-right scan, respectively. Coding tree 720 in FIG. 7B lists all nodes and CUs according to the applied scan order. Each split generates a list of two, three, or four new nodes at the next level of the tree until a leaf node (CU) is reached.

[0106] After decomposing an image into CTUs and then further decomposing them into CUs by block partitioner 310, and using the CUs to generate each residual block (324), as described with reference to Figure 3, the residual blocks are forward transformed by video encoder 114. An equivalent inverse transform process is performed in video decoder 134 to obtain TBs from bitstream 133.

[0107] In the video encoder 114, the quantized coefficients 336 may be rearranged into a one-dimensional list by performing a two-level backward diagonal scan. Similarly, in the video decoder 134, the quantized coefficients 424 may be rearranged from a one-dimensional list into a two-dimensional collection of sub-blocks by the same two-level backward diagonal scan.

[0108] FIG. 8A illustrates a two-level back diagonal scan 810 of an exemplary 8×8 TB 800. The scan 810 is shown proceeding from the bottom-right residual coefficient position of the TB 800 back to the top-left (DC) residual coefficient position of the TB 800. The path of the scan 810 proceeds from one subblock to the next in 4×4 regions known as subblocks. For a TB with a width or height of 2, subblock sizes of 2×2, 2×8, or 8×2 are available. The scan within a particular subblock is performed according to a "coded subblock flag," or the subblock is skipped. When a subblock scan is skipped, all residual coefficients within the subblock are assumed to have a value of zero. Although the scan 810 is shown starting from the bottom-right residual coefficient position of the TB 800, for a given set of residual coefficients, the scan begins from the "last significant coefficient" position, which is the "last" coefficient when the coefficient order is considered to proceed from the DC coefficient instead of the scan order.

[0109] 8B illustrates an alternative two-level forward diagonal scan 860 of an exemplary 8×8 TB 850 used when TSRC processing is selected. When TSRC processing is used in video encoder 114, quantized coefficients 336 are rearranged into a one-dimensional list by scan 860. Similarly, when TSRC processing is used for the current TB in video decoder 134, quantized coefficients 424 are rearranged from a one-dimensional list into a two-dimensional collection of sub-blocks by scan 860. Scan 860 is shown proceeding from the top-left (DC) residual coefficient position of TB 850 to the bottom-right residual coefficient position of TB 850. Unlike scan 810, scan 860 does not terminate at the “last significant coefficient.”

[0110] 8A and 8B show scan patterns typically used in VVC. The example described herein uses scan pattern 810 to encode the residual coefficients transformed by module 326, and scan pattern 860 is used for transform-skipped transform blocks. However, in some implementations, other scan patterns may be used.

[0111] As mentioned above, the transform coefficients 332 are the same as the residual coefficients 324 when transform skip mode is used. Therefore, regardless of whether transform skip mode is selected, the transform coefficients 332 may be referred to as residual coefficients. If lossless coding is desired, the video encoder 114 selects transform skip for the current TB and signals a transform skip flag with a value of “TRUE” to the bitstream 133. The residual coefficients 332 associated with the current TB are coded into the bitstream 133. Two residual coding processes are available: a “regular residual coding” (RRC) process and a “transform skip residual coding” (TSRC) process. In normal operation of the video encoder 114, if transform skip is selected (the transform skip flag has a value of “TRUE”), the TSRC process is selected; otherwise (the transform skip flag has a value of “FALSE”), the RRC process is selected. However, it is typically not desirable for coding of the residual coefficients 332 to be handled exclusively by TSRC in the case of lossless coding.

[0112] In one configuration of the video encoder 114, the TSRC invalid flag is signaled within the bitstream 133. The TSRC invalid flag can be signaled at a relatively high level, such as once per sequence or once per picture, so that the relative cost of signaling the TSRC invalid flag is low. High-level syntax elements are typically grouped into parameter sets, such as a "sequence parameter set" (SPS) for sequence-level flags or a "picture parameter set" (PPS) for parameter-level flags. The TSRC invalid flag may be set to "TRUE" if the video data 113 belongs to a class that is deemed unsuitable for encoding with TSRC processing (in terms of coding loss and feature reproduction). An example of video data that is unsuitable for encoding with TSRC is natural scene content. The TSRC invalid flag may be set to "FALSE" when the video data 113 belongs to a class that is deemed to be successfully encoded with TSRC processing. Video data that is suitable for encoding with TSRC processing includes artificial screen content.

[0113] If video encoder 114 selects transform skip for the current TB and the TSRC disabled flag is set to "TRUE," residual coefficients 332 are encoded into bitstream 133 using the RRC process. Similarly, if video decoder 134 determines that transform skip is used for the current TB and the TSRC disabled flag is set to "TRUE," residual coefficients 432 are decoded from bitstream 133 using the RRC process.

[0114] 9 illustrates a method 900 for encoding a transform block of residual coefficients 332 using an RRC process. Method 900 may be implemented by an apparatus such as a configured FPGA, ASIC, or ASSP. Additionally, method 900 may be performed by video encoder 114 under execution of processor 205. As such, method 900 may be implemented as a module of software 233 stored on a computer-readable storage medium and / or in memory 206.

[0115] The method 900 is performed in some configurations by the video encoder 114 upon receiving the residual coefficients 332 at the quantizer 334 and then at the entropy encoder 338. The method 900 begins with a coefficient quantization step 910.

[0116] In the coefficient quantization step 910, step 910 invokes method 1100, which is described below in connection with FIG. 11 . Method 1100 may be implemented by an apparatus such as an configured FPGA, ASIC, or ASSP. Furthermore, method 1100 may be performed by video encoder 114 under execution by processor 205. As such, method 1100 may be implemented as a module of software 233 stored on a computer-readable storage medium and / or in memory 206. Method 1100 quantizes residual coefficients 332 to generate quantized coefficients 336. From step 910, method 900 proceeds to last position encoding step 920 under control of processor 205.

[0117] In a last position encoding step 920, the video encoder 114 finds the position of the last significant coefficient in the quantized coefficients 336 for a transform block of residual coefficients 332. The last significant coefficient is determined relative to the forward direction of an appropriate scan pattern, e.g., the direction of a two-level forward diagonal scan 860. A quantized coefficient is significant if the coefficient has any value other than zero. The position of the last significant coefficient is written to the bitstream 133. From step 920, the method 900 proceeds, under control of the processor 205, to a state initialization step 930.

[0118] In a state initialization step 930, the quantizer state Qstate is set to a value of zero. Additionally, the sub-block containing the last significant coefficient is selected in step 930. From step 930, the method 900 proceeds, under the control of the processor 205, to a coded sub-block flag determination step 940.

[0119] The descriptions here refer to some flags being "TRUE" or "FALSE." A setting of "TRUE" means that the flag value indicates that the requirement has been met or the mode has been selected. A setting of "FALSE" means that the flag value indicates that the requirement has not been met or the mode has not been selected.

[0120] In coded sub-block flag determination step 940, video encoder 114 determines and sets coded sub-block flags. If the currently selected sub-block is the first sub-block selected in state initialization step 930, the coded sub-block flag is set to "TRUE" but is not coded into bitstream 133. If the currently selected sub-block is identified as the last sub-block, as described below in connection with last sub-block test 970, the coded sub-block flag is set to "TRUE" but is not coded into bitstream 133.

[0121] Otherwise, video encoder 114 sets the coded sub-block flag to either (i) "TRUE" if there is at least one significant coefficient among the 4x4 quantized coefficients belonging to the selected sub-block, or (ii) "FALSE" if there are no significant coefficients, and encodes the coded sub-block flag into bitstream 133. Method 900 proceeds from step 940 to coded sub-block flag test step 950 under control of processor 205.

[0122] In a coded sub-block flag test step 950, the method 900 determines the value of the coded sub-block flag. If the coded sub-block flag is set to "TRUE", the method 900 proceeds to a sub-block coding step 960. Otherwise, if the coded sub-block flag is set to "FALSE", the method 900 proceeds to a last sub-block test step 970.

[0123] In a sub-block encoding step 960, the entropy encoder 338 encodes the quantized coefficients in the selected sub-block into the bitstream 133. Step 960 invokes a method 1300, described below in connection with Figure 13. From step 960, the method 900, under processor control, proceeds to a last sub-block test 970.

[0124] In last sub-block test 970, method 900 operates to determine whether the selected sub-block is the last sub-block in the current transform block. If the currently selected sub-block is the top left sub-block of the transform block, step 900 returns "YES" and method 900 ends. Otherwise, if the currently selected sub-block is not the top left sub-block of the transform block, step 970 returns "NO" and method 900 proceeds to step 980 to select the next sub-block.

[0125] In select next sub-block step 980, the next sub-block in the transform block is selected. The next sub-block in the backward diagonal scan order 810 is selected. From step 980, the method 900 proceeds to step 940, where the coded sub-block flag of the selected sub-block is determined.

[0126] 10 illustrates a method 1000 for decoding a transform block of residual coefficients 432 via an RRC process. Method 1000 may be implemented by an apparatus such as a configured FPGA, ASIC, or ASSP. Additionally, method 1000 may be performed by video decoder 134 under execution of processor 205. As such, method 1000 may be implemented as a module of software 233 stored on a computer-readable storage medium and / or in memory 206.

[0127] The method 1000 is implemented in some configurations by the video encoder 134 in the entropy decoder 420 upon receipt of the bitstream 133, and in the inverse quantizer module 428. The method 1000 begins with a last position decoding step 1010.

[0128] In a last position decoding step 1010, the last significant coefficient position of the transform block of residual coefficients 432 is decoded from the bitstream 133. From step 1010, the method 1000 proceeds under the control of the processor 205 to a state initialization step 1020.

[0129] In a state initialization step 1020, video decoder 134 initializes the quantizer state Qstate to a value of 0. Further, in step 1020, the sub-block containing the last significant coefficient position is selected. Method 1000 proceeds from step 1020 under the control of processor 205 to a coded sub-block flag determination step 1030.

[0130] In coded sub-block flag determination step 1030, video decoder 134 determines the coded sub-block flag. If the currently selected sub-block is the first sub-block selected in state initialization step 1020, the coded sub-block flag is set to "TRUE" (i.e., the coded sub-block flag is inferred to be "TRUE"). If the currently selected sub-block is identified as the last sub-block as described below in last sub-block test 1060, the coded sub-block flag is inferred to be "TRUE". Otherwise, video decoder 134 decodes the coded sub-block flag from bitstream 133. Method 1000 proceeds from step 1030 to coded sub-block flag test 1040 under the control of processor 205.

[0131] In coded sub-block flag test 1040, method 1000 tests the value of the coded sub-block flag determined in step 1030. If method 1000 determines in step 1040 that the coded sub-block flag has a value of "TRUE," method 1000 proceeds to sub-block decoding step 1050. Otherwise, if in step 1040 it is determined that the coded sub-block flag has a value of "FALSE," all of the quantized coefficients in the currently selected sub-block are assigned a value of zero, and method 1000 proceeds to last sub-block test 1060.

[0132] In a sub-block decoding step 1050, the entropy decoder 420 decodes the quantized coefficients of the selected sub-block from the bitstream 133. Step 1050 invokes a method 1400, described below in connection with Figure 14. The method 1000 proceeds to a last sub-block test 1060 under the control of the processor 205.

[0133] In the last sub-block test 1060, if the currently selected sub-block is the top left sub-block of the transform block, step 1060 returns "YES" and method 1000 proceeds to coefficient scale step 1080. Otherwise, step 1060 returns "NO" and method 1000 proceeds to step 1070 to select the next sub-block.

[0134] In select next sub-block step 1070, the next sub-block in the backward diagonal scan order 810 is selected. From step 1070, method 1000, under the control of processor 205, proceeds to coded sub-block flag determination step 1030.

[0135] In a coefficient scale step 1080, the inverse quantizer module 428 applies scaling to the quantized coefficients 424 to produce reconstructed residual coefficients 432. A sub-block is decoded by using the decoded code bits to reconstruct the residual coefficients of the sub-block. Step 1080 invokes method 1200, described below in connection with Figure 12. Method 1000 ends with the execution of step 1080.

[0136] 11 shows a method 1100 for quantizing the residual coefficients 332 of a transform block to generate quantized coefficients 336. The method 1100 is performed on the TB in step 910 of the method 900. The method 1100 begins with a DQ test 1110.

[0137] In DQ test 1110, video encoder 114 determines whether dependent quantization is used to quantize residual coefficients 332. Video encoder 114 checks the value of the enabled dependent quantization flag, which is signaled as high-level syntax in bitstream 133. The enabled dependent quantization flag determines whether dependent quantization is allowed within the flag. For example, the sequence-level dependent quantization flag determines whether dependent quantization is allowed when encoding an entire video sequence. The picture-level dependent quantization flag determines whether dependent quantization is allowed when encoding the current picture and takes precedence over the value of the sequence-level dependent quantization flag. If the enabled dependent quantization flag is "FALSE," step 1110 returns "NO," and method 1100 proceeds to scalar quantization step 1120.

[0138] In one configuration of DQ test 1110, if the valid dependent quantization flag is "TRUE," the video encoder 114 also checks the value of the transform skip flag for the current TB. If the valid dependent quantization flag is "TRUE" and the transform skip flag is "TRUE," step 1110 returns "NO" and method 1100 proceeds to scalar quantization step 1120. Otherwise, if the valid dependent quantization flag is "TRUE" and the transform skip flag is "FALSE," step 1110 returns "YES" and method 1100 proceeds to dependent quantization step 1130.

[0139] In another configuration of DQ test 1110, if the valid dependent quantization flag is "TRUE," the video encoder 114 also checks the value of the TSRC disabled flag. If the valid dependent quantization flag is "TRUE" and the TSRC disabled flag is "TRUE," step 1110 returns "NO" and method 1100 proceeds to scalar quantization step 1120. Otherwise, if the valid dependent quantization flag is "TRUE" and the TSRC disabled flag is "FALSE," step 1110 returns "YES" and method 1100 proceeds to dependent quantization step 1130.

[0140] In yet another configuration of the DQ test 1110, if the enabled dependent quantization flag is "TRUE", the video encoder 114 also checks the value of the quantization parameter (QP) of the current TB. The QP indicates the degree of quantization applied to the residual coefficients 332. The QP is the initial QP i and an offset that depends on the bit depth BD of the video encoder 114, QP=QP i +6*(BD-8). For example, QP i If is 4 and bit depth is 8, QP is determined to be 4. iIf the QP is -8 and the bit depth is 10, the QP is determined as 4. Typically, a QP of 4 indicates that the residual coefficients are not quantized, and thus, lossless operation is possible. However, higher values of QP can still achieve lossless operation. For example, if the video data 113 was originally captured at a bit depth of 8 but is supplied to the video encoder 114 at a higher bit depth, lossless operation is possible with a higher QP. For example, if the video data 113 was captured at a bit depth of 8 but is supplied to the video encoder 114 at a bit depth of 10, lossless operation is possible with a QP of 4, 10, or 16. The QP at which lossless operation is possible may be indicated by the minimum QP for the transform skip block, which is signaled in the high-level syntax parameter set. If the enable dependent quantization flag is "TRUE" and the QP is 4 (or any value indicating lossless operation), step 1110 returns "NO," and method 1100 proceeds to scalar quantization step 1120. Otherwise, if the enabled dependent quantization flag is 'TRUE' and QP is not 4 (or a similar value indicating lossless operation), then step 1110 returns 'YES' and method 1100 proceeds to dependent quantization step 1130 . In the scalar quantization step 1120, the residual coefficients 332 are represented as r[n]. The quantized coefficients q[n] are then generated by quantizing the residual coefficients r[n] according to equation (1) below. q[n]=(k*r[n]+offset)>>qbits (1) In equation (1), k is a scaling factor, qbits is a coarse quantization factor, and offset controls the placement of the quantization threshold. k, qbits, and offset are determined based on the value of the quantization parameter of the current TB. For example, if QP is 4, then k=1, qbits=0, and offset=0. Then, if QP is 4, then q[n]=r[n], and no loss occurs in the scalar quantization step. Method 1100 proceeds from step 1120 to SBH test 1140 under the control of processor 205.

[0141] In the dependent quantization step 1130, each residual coefficient r[n] may be quantized by one of several scalar quantizers. For the same QP, the scalar quantizers have the same quantization division number, but the quantization thresholds are offset relative to each other. The scalar quantizer for a particular residual coefficient r[n] depends on the current quantizer state Qstate, which is updated for each coefficient, and on the parity (least significant bit) of the resulting q[n]. Due to the dependency on the previous state, the optimal quantization result is not determined for each coefficient. One efficient way to determine the optimal quantization result is by building a "trellis" of possible quantization states at each coefficient position. The optimal quantization result can be found by equivalently finding the best path through the trellis. The optimal trellis path can be determined by applying the Viterbi algorithm. The method 1100 ends with the execution of step 1130.

[0142] In SBH test 1140, video encoder 114 determines whether sign bit hiding is used to modify quantized coefficients q[n] before encoding the coefficients of TB. Video encoder 114 checks the value of the significant sign bit hiding flag. The significant sign bit hiding flag is signaled as high-level syntax in bitstream 133. For example, the significant sign bit hiding flag can be signaled in the picture header. If the significant dependent quantization flag is "TRUE," the significant sign bit hiding flag is implicitly "FALSE." If the significant sign bit hiding flag is "FALSE," step 1140 returns "NO," and method 1100 ends.

[0143] In one configuration of SBH test 1140, the decision depends on the value of the significant code bit hiding flag and the value of the transform skip flag for the current TB. If the significant code bit hiding flag is "TRUE," the video encoder 114 also checks the value of the transform skip flag for the current TB. If the significant code bit hiding flag is "TRUE" and the transform skip flag is "TRUE," step 1140 returns "NO" and method 1100 ends. Otherwise, if the significant code bit hiding flag is "TRUE" and the transform skip flag is "FALSE," step 1140 returns "YES" and method 1100 proceeds to parity adjustment step 1150.

[0144] In another configuration of SBH test 1140, the decision depends on the value of the significant code bit hiding flag and the value of the TSRC invalid flag. If the significant code bit hiding flag is "TRUE," the video encoder 114 also checks the value of the TSRC invalid flag. If the significant code bit hiding flag is "TRUE" and the TSRC invalid flag is "TRUE," step 1140 returns "NO" and method 1100 ends. Otherwise, if the significant code bit hiding flag is "TRUE" and the TSRC invalid flag is "FALSE," step 1140 returns "YES" and method 1100 proceeds to parity adjustment step 1150.

[0145] In yet another configuration of SBH test 1140, the decision depends on the value of a significant sign bit hiding flag and the value of the quantization parameter (QP) for the current TB. If the significant sign bit hiding flag is "TRUE," the video encoder 114 also checks the value of QP for the current TB. If the significant sign bit hiding flag is "TRUE" and the QP is 4 (or any value indicating lossless operation), step 1140 returns "NO," and method 1100 ends. Otherwise, if the significant sign bit hiding flag is "TRUE" and the QP is not 4 (or a similar value indicating lossless operation), step 1140 returns "YES," and method 1100 proceeds to parity adjustment step 1150.

[0146] In a parity adjustment step 1150, the video encoder 114 checks the positions of the first and last significant coefficients for each sub-block in the current TB. If the difference between the first significant position and the last significant position of a sub-block is greater than a threshold (typically 3), sign bit hiding is used for that sub-block. For each sub-block in which sign bit hiding is used, the video encoder 114 checks the sign of the first significant coefficient in the sub-block and adjusts the parity of the coefficients in the sub-block accordingly. The parity of a coefficient is zero if the coefficient is even and one if the coefficient is odd. The sum of the parities of multiple coefficients is zero if the number of odd coefficients is odd and one if the number of odd coefficients is even. If the sign of the first significant coefficient in the sub-block is positive, the coefficients in the sub-block are adjusted so that the sum of their parities is zero. If the sign of the first significant coefficient in the sub-block is negative, the coefficients in the sub-block are adjusted so that the sum of their parities is one. The method 1100 ends after performing step 1150.

[0147] 12 shows a method 1200 for applying scaling to the quantized coefficients 424 to generate reconstructed residual coefficients 432. The method 1200 may be implemented by an apparatus such as an configured FPGA, ASIC, or ASSP. Furthermore, the method 1200 may be performed by the video decoder 134 under execution of the processor 205. As such, the method 1200 may be implemented as a module of software 233 stored on a computer-readable storage medium and / or in the memory 206. The method 1200 is implemented at step 1080 of the method 1000. The method 1200 begins with a DQ test 1210.

[0148] In DQ test 1210, video decoder 134 determines whether dependent quantization is used to dequantize quantized coefficients 424. Video decoder 134 checks the value of the enabled dependent quantization flag, which may be decoded from bitstream 133 or inferred based on the values of other high-level syntax flags. If the enabled dependent quantization flag is "FALSE," step 1210 returns "NO," and method 1200 proceeds to inverse scalar quantization step 1220.

[0149] In one configuration of DQ test 1210, if the valid dependent quantization flag is "TRUE," video decoder 134 also checks the value of the transform skip flag for the current TB. If the valid dependent quantization flag is "TRUE" and the transform skip flag is "TRUE," step 1210 returns "NO" and method 1100 proceeds to inverse scalar quantization step 1220. Otherwise, if the valid dependent quantization flag is "TRUE" and the transform skip flag is "FALSE," step 1210 returns "YES" and method 1100 proceeds to inverse dependent quantization step 1230.

[0150] In another configuration of DQ test 1210, if the valid dependent quantization flag is "TRUE," video decoder 134 also checks the value of the TSRC disabled flag. If the valid dependent quantization flag is "TRUE" and the TSRC disabled flag is "TRUE," step 1210 returns "NO" and method 1100 proceeds to inverse scalar quantization step 1220. Otherwise, if the valid dependent quantization flag is "TRUE" and the TSRC disabled flag is "FALSE," step 1210 returns "YES" and method 1100 proceeds to inverse dependent quantization step 1230.

[0151] In yet another configuration of DQ test 1210, if the enabled dependent quantization flag is "TRUE," video decoder 134 also checks the value of the quantization parameter (QP) for the current TB. If the enabled dependent quantization flag is "TRUE" and the QP is 4 (or any value indicating lossless operation), step 1210 returns "NO," and method 1100 proceeds to inverse scalar quantization step 1220. Otherwise, if the enabled dependent quantization flag is "TRUE" and the QP is not 4 (or a similar value indicating lossless operation), step 1210 returns "YES," and method 1100 proceeds to inverse dependent quantization step 1230.

[0152] In an inverse scalar quantization step 1220, the video decoder 134 scales the quantized coefficients 424 to generate reconstructed residual coefficients 432. The quantized coefficients 424 are denoted as q[n]. The reconstructed residual coefficients r[n] are generated by scaling the quantized coefficients q[n] according to equation (2) below.

[0153]

number

[0154] In equation (2), s is a scaling factor determined based on the value of QP for the current TB. For example, if QP is 4, then s=1 and r[n]=q[n]. Method 1200 ends with execution of step 1220.

[0155] In an inverse dependent quantization step 1230, the video decoder 134 applies inverse dependent quantization to the quantized coefficients 424 to generate reconstructed residual coefficients 432. The quantizer state Qstate is initially reset to zero. The quantized coefficients 424 are denoted by q[n]. Each coefficient position n is visited in backward diagonal scan order 810, and each reconstructed residual coefficient r[n] is calculated according to equation (3).

[0156]

number

[0157] In equation (3), s is a scaling factor determined based on the value of QP for the current TB.

[0158] After each reconstructed residual coefficient r[n] is calculated, the quantizer states are updated based on the parity of q[n] according to Table 1.

[0159] [Table 1]

[0160] The method 1200 concludes with the execution of step 1230 .

[0161] To take advantage of the statistical properties of the quantized coefficients 336, the quantized coefficients are binarized by the video encoder 114 (typically by the entropy encoder 338) into several syntax elements before encoding. For example, because the quantized coefficients 336 often have a value of zero, one syntax element is a significance flag, which is set to "FALSE" for quantized coefficients having a value of zero. When the significance flag is set to "FALSE," no further syntax elements for the associated quantized coefficient are signaled. The significance flag may be coded into the bitstream 133 by using a context-adaptive binary arithmetic coding (CABAC) entropy encoder.

[0162] Although CABAC encoders encode context-coded syntax elements relatively efficiently, limiting the number of context-coded syntax elements is generally desirable to minimize computational requirements and costs for hardware implementation. Therefore, after the quantized coefficients 336 are binarized into several syntax elements by the entropy encoder 338, some syntax elements are context-coded into the bitstream 133, while other syntax elements are bypass-coded into the bitstream 133. The total number of context-coded syntax element bins is limited per transform block. In the VVC standard, the limit is set at 1.75 bins per sample. For example, for an 8x8 transform block consisting of 64 samples, the context-coding bin budget is set at 112 bins. During the process of encoding a TB into the bitstream 133, whenever a syntax element is context-coded, the remaining context-coding bin budget is tracked and decremented. When the remaining context-coding bin budget is depleted, the remaining quantized coefficients and associated syntax elements must be bypass-coded.

[0163] 13 illustrates a method 1300 for encoding the quantized coefficients (336) of a currently selected sub-block into a bitstream 133. The method 1300 is performed at step 960 of the method 900. The method 1300 may be performed by an apparatus such as a configured FPGA, ASIC, or ASSP. Furthermore, the method 1300 may be performed by the video encoder 114 under execution of the processor 205. As such, the method 1300 may be implemented as a module of the software 233 stored on a computer-readable storage medium and / or in the memory 206. The method 1300 begins at step 1310 with selecting an initial coefficient.

[0164] In step 1310 of selecting a first coefficient, the method 1300 selects a quantized coefficient for the current sub-block. If the current sub-block contains the last significant coefficient position, the currently selected coefficient is set to the last significant coefficient. Otherwise, if the current sub-block does not contain the last significant coefficient position, the currently selected coefficient is set to the bottom right coefficient of the current sub-block. The method 1300 proceeds to check whether context coding is used 1320.

[0165] In context coding usage check 1320, video encoder 114 checks whether the remaining context coding bin budget is greater than or equal to 4. If the remaining context coding bin budget is greater than or equal to 4, step 1320 returns "YES" and method 1300 proceeds to context coding syntax element encoding step 1330. Otherwise, if the current context coding bin budget is less than 4, step 1320 returns "NO" and method 1300 proceeds to remainder pass encoding step 1370.

[0166] In a context-coding syntax element encoding step 1330, the video encoder 114 may encode multiple syntax elements into the bitstream 133 using a CABAC encoder, including a potentially significant flag, a greater-than-one flag, a parity flag, and a greater-than-three flag. Each bin associated with a syntax element is encoded by the CABAC encoder using a "context model." The context model for each bin may be selected depending on the current value of the quantizer state Qstate. Additionally, whenever a context-coding bin is encoded into the bitstream 133 by the CABAC encoder, the remaining context-coding bin budget is reduced by one in step 1330.

[0167] In step 1330, if the current coefficient is the last significant coefficient, the significance flag is set to "TRUE" but is not coded into the bitstream 133. If the current selected sub-block is not the first or last sub-block in the backward scan order 810, and the current selected coefficient is the last coefficient as described below in last coefficient check 1350, and all significance flags for previous coefficients in the current selected sub-block were "FALSE", the significance flag is set to "TRUE". The significance flag is not coded into the bitstream 133. If the current coefficient has a magnitude of zero, the significance flag is set to "FALSE" in step 1330 and is context coded into the bitstream 133. Otherwise, the significance flag is set to "TRUE" and is context coded into the bitstream 133 in step 1330.

[0168] If the current coefficient has a magnitude of 1, the greater than 1 flag is set to "FALSE" and is context coded into the bitstream 133 in step 1330. Otherwise, the greater than 1 flag is set to "TRUE" and is context coded into the bitstream 133.

[0169] If the current coefficient has a magnitude of at least 2, then the parity flag is set to "FALSE" if the current coefficient is even, and set to "TRUE" if the current coefficient is odd. The parity flag is context coded into bitstream 133 in step 1330. If the current coefficient has a magnitude greater than 3, then the greater than 3 flag is set to "TRUE" and context coded into bitstream 133 in step 1330. Otherwise, if the current coefficient has a magnitude of 2 or 3, then the greater than 3 flag is set to "FALSE" and context coded into bitstream 133.

[0170] Method 1300, under control of processor 205, proceeds from step 1330 to DQ test 1340. Depending on the coefficient selected in 1310, method 1300 sets (or possibly encodes) the significance flag before proceeding to step 1340. Otherwise, method 1300 encodes the last appropriate one of the greater than 1 flag, the parity flag, and the greater than 3 flag before proceeding to step 1340.

[0171] DQ test 1340 uses the same conditions checked in DQ test 1110 to determine whether step 1340 returns "YES" or "NO." If step 1340 returns "YES," method 1300 proceeds to update Qstate step 1345. Otherwise, if step 1340 returns "NO," method 1300 proceeds to last coefficient check 1350.

[0172] In an update Qstate step 1345, the quantizer state Qstate is updated based on the parity of the current coefficient according to Table 1. From step 1345, the method 1300 proceeds to check last coefficient 1350.

[0173] In last coefficient check 1350, video encoder 114 checks whether the currently selected coefficient is the top-left coefficient of the currently selected sub-block. If the currently selected coefficient is the top-left coefficient of the currently selected sub-block, step 1350 returns "YES" and method 1300 proceeds to remainder pass encoding step 1370. Otherwise, if the current coefficient is not the top-left coefficient, step 1350 returns "NO" and method 1300 proceeds to step 1360, where it selects the next coefficient.

[0174] A select next coefficient step 1360 selects the next coefficient of the currently selected sub-block in the backward diagonal scan order 810. From step 1360, the method 1300 proceeds to a context coding usage check 1320.

[0175] In remainder pass encoding step 1370, the residual magnitudes of the quantized coefficients of the currently selected sub-block are binarized and bypass coded into the bitstream 133, for example, by the entropy encoder 338. The quantized coefficients are coded, for example, in the backward diagonal scan order 810. If the quantized coefficients were context coded by the CABAC coder (i.e., the context coding usage check 1320 passed (returned "YES")), then if the flag greater than 3 is "TRUE", the quantized coefficient at scan position n has residual magnitude r[n]. The residual magnitude is determined using equation (4). r[n]=(x[n]-4)>>1, (4)

[0176] where, in equation (4), x[n] is the absolute magnitude of the quantized coefficient at scan position n. The magnitude r[n] is binarized and bypass coded into the bitstream 133. If the quantized coefficient was not context coded (the context coding usage check 1320 is not passed / returns "NO"), then the absolute magnitude x[n] is binarized and bypass coded into the bitstream 133. Method 1300 proceeds from step 1370 to SBH test 1380.

[0177] SBH test 1380 uses the same conditions checked in SBH test 1140 to determine whether step 1380 returns "YES" or "NO." If SBH test 1140 returns "NO," step 1380 returns "NO," and method 1300 proceeds to step 1390, where N symbols are encoded. Otherwise, video encoder 114 checks the positions of the first and last significant coefficients of the current sub-block. If the difference between the first significant position and the last significant position is greater than 3, step 1380 returns "YES," and method 1300 proceeds to step 1395, where N-1 symbols are encoded. Otherwise, step 1380 returns "NO," and method 1300 proceeds to step 1390, where N symbols are encoded.

[0178] As described in connection with step 1140, the sign bit hiding test can depend on a number of alternative flags or settings in different implementations. If the valid sign bit hiding flag is set (has a "TRUE" value), different implementations can make a decision based on whether the transform skip flag of the TB, the TSRC disabled flag, or the QP of the TB meets a threshold associated with lossless coding. Thus, step 1380 determines whether sign bit hiding is enabled or not depending on a value or flag associated with the transform block itself or a higher-level value of the TSRC disabled flag. Step 1380 provides a degree of flexibility for implementing lossless coding. An implementation that uses a value or flag associated with the transform block to determine whether sign bit hiding is enabled is particularly well-suited for allowing flexibility in implementing lossless coding using RRC.

[0179] In step 1390 of encoding N codes, the sign bits of any significant coefficients of the currently selected sub-block are bypass coded into bitstream 133. The sign bits are, for example, bypass coded into bitstream 133 based on the backward diagonal scan order 810. Method 1300 ends after performing step 1390.

[0180] In step 1395 of encoding N-1 signs, the sign bits of the significant coefficients of the currently selected sub-block are bypass coded into bitstream 133 based on the backward diagonal scan order 810. The sign bit associated with the first significant coefficient (the last visited in the backward diagonal scan order 810) is not coded into bitstream 133. In other words, if there are N significant coefficients in the currently selected sub-block, N-1 sign bits are bypass coded into bitstream 133. Method 1300 ends with the execution of step 1395.

[0181] 14 shows a method 1400 for decoding the quantized coefficients (424) of a currently selected sub-block from the bitstream 133. The method 1400 is performed at step 1050 of the method 1000. The method 1400 may be performed by an apparatus such as a configured FPGA, ASIC, or ASSP. Furthermore, the method 1400 may be performed by the video decoder 134 under execution of the processor 205. As such, the method 1400 may be implemented as a module of the software 233 stored on a computer-readable storage medium and / or in the memory 206. The method 1400 begins at step 1410 with selecting an initial coefficient.

[0182] In select first coefficient step 1410, method 1400 selects the first quantized coefficient of the current sub-block. If the current sub-block contains the last significant coefficient position, the currently selected coefficient is set to the last significant coefficient. Otherwise, the currently selected coefficient is set to the bottom right coefficient of the current sub-block. From step 1410, method 1400 proceeds to context coding usage check step 1420.

[0183] In context coding usage check 1420, video decoder 134 checks whether the remaining context coding bin budget meets a threshold, typically whether the remaining context coding bin budget for a transform block is 4 bins or more. If the remaining budget is 4 or more, step 1420 returns "YES" and method 1400 proceeds to context coding syntax element determination step 1430. Otherwise, if the remaining CABAC budget is less than the threshold (4 bins), step 1420 returns "NO" and method 1400 proceeds to remainder pass decoding step 1470.

[0184] In a context coding syntax element determination step 1430, video decoder 134 may decode a number of context coding syntax elements from bitstream 133 using a CABAC coder. Each bin associated with a syntax element is decoded by the CABAC coder using a “context model.” The context model for each bin may be selected depending on the current value of the quantizer state Qstate. Furthermore, whenever a context coding bin is decoded by the CABAC coder from bitstream 133, the remaining context coding bin budget is reduced by one.

[0185] If the current coefficient is the last significant coefficient, the significance flag is inferred to be "TRUE" rather than being decoded from the bitstream 133. If the currently selected sub-block is not the first or last sub-block in the backward scan order 810, the currently selected coefficient is the last coefficient as described below in last coefficient check 1450, and all significance flags for previous coefficients in the currently selected sub-block were "FALSE", the significance flag is inferred to be "TRUE". Otherwise, the significance flag is context decoded from the bitstream 133 in step 1430. If the significance flag is set to "FALSE", the currently selected coefficient is assigned a value of zero and method 1400 proceeds to DQ test 1440.

[0186] If the significance flag is set to "TRUE", then in step 1430, a flag greater than 1 is context decoded from the bitstream 133. If the flag greater than 1 is set to "FALSE", then the currently selected coefficient is assigned a magnitude of 1 and method 1400 proceeds to DQ test 1440.

[0187] If the greater than 1 flag is set to "TRUE," then the parity flag and the greater than 3 flag are context decoded from the bitstream 133. The method 1400 proceeds to DQ test 1440. The number of flags determined in step 1430 depends on the position and value of the coefficient selected in step 1410. Proceeding from step 1430 can occur after the significance flag is inferred or decoded, or after decoding the greater than 1 flag, parity flag, or greater than 3 flag, as appropriate.

[0188] DQ test 1440 uses the same conditions checked in DQ test 1210 to determine whether step 1440 returns "YES" or "NO." If step 1440 returns "YES," method 1400 proceeds to update Qstate step 1445. Otherwise, if step 1440 returns "NO," method 1400 proceeds to check last coefficient 1450.

[0189] In update Qstate step 1445, the quantizer state Qstate is updated based on the parity of the currently selected coefficient according to Table 1. If the currently selected coefficient has a value of zero, then parity is zero. Parity is one if the currently selected coefficient has a magnitude of one. Otherwise, if the parity flag is set to "FALSE," then parity is zero, and if the parity flag is set to "TRUE," then parity is one. Method 1400 proceeds from step 1445 to check last coefficient 1450.

[0190] In last coefficient check step 1450, video decoder 134 checks whether the currently selected coefficient is the top-left coefficient of the currently selected sub-block. If the currently selected coefficient is the top-left coefficient of the currently selected sub-block, step 1450 returns "YES" and method 1400 proceeds to remainder path decoding step 1470. Otherwise, if the currently selected coefficient is not the top-left coefficient, step 1450 returns "NO" and method 1400 proceeds to select the next coefficient step 1460.

[0191] A select next coefficient step 1460 selects the next coefficient of the currently selected sub-block in the backward diagonal scan order 810. From step 1460, the method 1400 proceeds to a context coding usage check 1420.

[0192] In remainder pass decoding step 1470, the remaining magnitudes of the quantized coefficients of the currently selected sub-block are bypass decoded from bitstream 133. The quantized coefficients are processed in backward diagonal scan order 810. If a quantized coefficient is context decoded (context coding usage check 1420 passes or returns "YES") and a flag greater than 3 is decoded with a value of "TRUE", the remaining magnitude r[n] is bypass decoded from bitstream 133, where n is the scan position of the quantized coefficient. The absolute magnitude x[n] of the quantized coefficient is determined as x[n]=4+p[n]+2*r[n], where p[n] has a value of zero if the parity flag is decoded as "FALSE" and p[n] has a value of one if the parity flag is decoded as "TRUE".

[0193] If the quantized coefficient is context decoded and a flag greater than 1 is decoded as "TRUE" but a flag greater than 3 is not decoded or is decoded as "FALSE", then the absolute magnitude is determined as x[n] = 2 + p[n]. If the quantized coefficient is not context decoded (the context coding usage check 1420 is not passed, returning "NO"), then the absolute magnitude x[n] is bypass decoded from the bitstream 133. Method 1400 proceeds from step 1470 to SBH test 1480.

[0194] In SBH test 1480, video decoder 134 determines whether sign bit hiding is being used, i.e., whether one sign bit for the currently selected sub-block has been inferred. The test used in step 1480 is related to the test used in step 1140 on the encoder side. Video decoder 134 checks the value of the valid sign bit hiding flag, which may be signaled as high-level syntax in bitstream 133. If the valid dependent quantization flag is "TRUE," the valid sign bit hiding flag is inferred to be "FALSE." If the valid sign bit hiding flag is "FALSE," step 1480 returns "NO," and method 1400 proceeds to code decoding step 1490.

[0195] Video decoder 134 checks the positions of the first and last significant coefficients of the currently selected sub-block. If the difference between the first significant position and the last significant position is less than or equal to 3, step 1480 returns "NO" and method 1400 proceeds to code decoding step 1490.

[0196] In one configuration of SBH test 1480, the decision depends on the value of the significant code bit hiding flag and the value of the transform skip flag for the current TB. If the significant code bit hiding flag is "TRUE" and the difference between the first significant position and the last significant position is greater than 3, the video decoder 134 also checks the value of the transform skip flag for the current TB. If the transform skip flag is "TRUE," step 1480 returns "NO," and method 1400 proceeds to code decoding step 1490. Otherwise, if the transform skip flag is "FALSE," step 1480 returns "YES," and method 1400 proceeds to code decoding and inferring step 1495.

[0197] In another configuration of SBH test 1480, the decision depends on the value of the valid code bit hiding flag and the value of the TSRC invalid flag. If the valid code bit hiding flag is "TRUE" and the difference between the first significant position and the last significant position is greater than 3, video decoder 134 also checks the value of the TSRC invalid flag. If the TSRC invalid flag is "TRUE," step 1480 returns "NO," and method 1400 proceeds to code decoding step 1490. Otherwise, if the TSRC invalid flag is "FALSE," step 1480 returns "YES," and method 1400 proceeds to code decoding and inferring step 1495.

[0198] In yet another configuration of the SBH test 1480, the decision depends on the value of the significant sign bit hiding flag, the value of the quantization parameter (QP) of the current TB, and the difference between the first and last significant positions of the quantization parameter QP of the transform block. If the significant sign bit hiding flag is "TRUE" and the difference between the first and last significant positions is greater than 3, the video decoder 134 also checks the value of the QP of the current TB. If QP is 4 (or any value indicating lossless operation), step 1480 returns "NO" and the method 1400 proceeds to code decoding step 1490. Otherwise, if QP is not 4 (or a similar value indicating lossless operation), step 1480 returns "YES" and the method 1400 proceeds to code decoding and inferring step 1495.

[0199] In a code pass decoding step 1490, the sign bits of any significant coefficients of the currently selected sub-block are bypass decoded from the bitstream 133. The sign bits are bypass decoded from the bitstream 133 in the backward diagonal scan order 810. The value of a quantized coefficient is set to −x[n] if its associated sign bit has a value of one. The value of a quantized coefficient is set to x[n] if its associated sign bit has a value of zero. The method 1400 ends with the execution of step 1490.

[0200] In a sign decoding and inferring step 1495, the sign bits of the significant coefficients of the currently selected sub-block are bypass decoded from the bitstream 133 in the backward diagonal scan order 810. The sign bit associated with the first significant coefficient (the last visited in the backward diagonal scan order 810) is not decoded from the bitstream 133. In other words, if there are N significant coefficients in the currently selected sub-block, N-1 code bits are bypass decoded from the bitstream 133. The sign bit associated with the first significant coefficient is inferred based on the parity sum of the significant coefficients. If the parity sum is zero, the sign bit associated with the first significant coefficient is inferred as zero. If the parity sum is one, the sign bit associated with the first significant coefficient is inferred as one. If the associated sign bit has a value of one, the value of the quantized coefficient is set to -x[n]. If the associated sign bit has a value of zero, the value of the quantized coefficient is set to x[n]. Method 1400 then ends.

[0201] The configurations described in methods 900 and 1000 allow lossless compression of video data to be performed while using the normal residual coding process. Dependent quantization and sign bit hiding are lossy coding tools that are flexibly disabled when lossless operations are desired, but may still be available to achieve improved coding performance in lossy coded blocks.

[0202] Industrial Applicability The described arrangements are applicable to the computer and data processing industry, particularly in digital signal processing for decoding and encoding signals such as video and image signals, to achieve high compression efficiency.

[0203] The above describes only some embodiments of the present invention, and modifications and / or variations can be made to the present invention without departing from the scope and spirit of the invention, and the embodiments are illustrative and not limiting.

Claims

1. 1. A method for decoding a transform block from a bitstream, comprising: decoding a first flag used to determine whether dependent quantization is used in the transform block; determining whether the transform block uses sign bit hiding, in which data indicating the sign of a significant coefficient at a certain position is not decoded from the bitstream; If the first flag is TRUE, the sign bit hiding is not used in the transform block; decoding the transform block using the dependent quantization when it is determined based on the first flag that the dependent quantization is to be used for the transform block; and if it is determined that the sign bit hiding is to be used, decoding the transform block using the sign bit hiding; After checking the first flag, checking a disable flag for transform skip residual coding, and if the disable flag is TRUE, the dependent quantization is not used; signaling the invalid flag depends on information indicated by the first flag; When the invalid flag is TRUE, the sign bit hiding is not used in the transform block even if the valid flag of the sign bit hiding is TRUE; The invalid flag indicates whether the first residual coding is applied instead of the second residual coding even when the transform process is skipped, and the first residual coding is a process for a block for which the transform process is not skipped, and the second residual coding is a process for a block for which the transform process is skipped. A method characterized by:

2. 1. A method for encoding a transform block into a bitstream, comprising: encoding a first flag used to determine whether dependent quantization is used in the transform block; determining whether the transform block uses sign bit hiding, in which data indicating the sign of a significant coefficient at a position is not coded into the bitstream; If the first flag is TRUE, the sign bit hiding is not used in the transform block; encoding the transform block using the dependent quantization when it is determined based on the first flag that the dependent quantization is to be used for the transform block; if it is determined that the sign bit hiding is to be used, encoding the transform block using the sign bit hiding; After checking the first flag, checking a disable flag for transform skip residual coding, and if the disable flag is TRUE, the dependent quantization is not used; signaling the invalid flag depends on information indicated by the first flag; When the invalid flag is TRUE, the sign bit hiding is not used in the transform block even if the valid flag of the sign bit hiding is TRUE; The invalid flag indicates whether the first residual coding is applied instead of the second residual coding even when the transform process is skipped, and the first residual coding is a process for a block for which the transform process is not skipped, and the second residual coding is a process for a block for which the transform process is skipped. A method characterized by:

3. 1. A decoding device for decoding a transform block from a bitstream, comprising: means for decoding a first flag used to determine whether dependent quantization is used in the transform block; means for determining whether the transform block uses sign bit hiding, in which data indicating the sign of a significant coefficient at a certain position is not decoded from the bitstream; If the first flag is TRUE, the sign bit hiding is not used in the transform block; means for decoding the transform block using the dependent quantization when it is determined based on the first flag that the dependent quantization is to be used for the transform block; means for decoding the transform block using the sign bit hiding when it is determined that the sign bit hiding is to be used; After checking the first flag, checking a disable flag for transform skip residual coding, and if the disable flag is TRUE, the dependent quantization is not used; signaling the invalid flag depends on information indicated by the first flag; When the invalid flag is TRUE, the sign bit hiding is not used in the transform block even if the valid flag of the sign bit hiding is TRUE; The invalid flag indicates whether the first residual coding is applied instead of the second residual coding even when the transform process is skipped, and the first residual coding is a process for a block for which the transform process is not skipped, and the second residual coding is a process for a block for which the transform process is skipped. A decoding device characterized by:

4. 1. A coding device for coding a transform block into a bitstream, comprising: means for encoding a first flag used to determine whether dependent quantization is used in the transform block; means for determining whether the transform block uses sign bit hiding, in which data indicating the sign of a significant coefficient at a certain position is not coded into the bitstream; If the first flag is TRUE, the sign bit hiding is not used in the transform block; means for encoding the transform block using the dependent quantization when it is determined based on the first flag that the dependent quantization is to be used for the transform block; means for encoding the transform block using the sign bit hiding when it is determined that the sign bit hiding is to be used; After checking the first flag, checking a disable flag for transform skip residual coding, and if the disable flag is TRUE, the dependent quantization is not used; signaling the invalid flag depends on information indicated by the first flag; When the invalid flag is TRUE, the sign bit hiding is not used in the transform block even if the valid flag of the sign bit hiding is TRUE; The invalid flag indicates whether the first residual coding is applied instead of the second residual coding even when the transform process is skipped, and the first residual coding is a process for a block for which the transform process is not skipped, and the second residual coding is a process for a block for which the transform process is skipped.

1. An encoding device comprising:

5. A computer program product for causing a computer to carry out the method of claim 1.

6. A computer program product for causing a computer to carry out the method according to claim 2.

Images