Method for decoding image, metho for encoding image into bit stream, decoder, encoder, and computer program

JP2024054338A5Active Publication Date: 2024-11-29CANON KK
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
JP2024020597
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Priority Date
2020-03-10
Filing Date
2024-02-14
Publication Date
2024-11-29
Estimated Expiration
2040-11-23

AI Technical Summary

Technical Problem

Existing video encoding methods for lossless compression of diverse video content types, such as screen content, are suboptimal due to the exclusive use of Transform Skip Residual Coding (TSRC) process, which may not align with the statistical properties of all video data, leading to inefficient encoding performance.

Method used

A method for decoding residual coefficients in video bitstreams that dynamically selects between sign bit hiding and code bit hiding based on flags and quantization parameters, allowing for more flexible lossless encoding without exclusive reliance on TSRC, thereby optimizing encoding performance for various video content types.

Benefits of technology

This approach enhances encoding flexibility and efficiency by adapting to the statistical properties of different video content, minimizing additional complexity and improving overall encoding performance.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 00000000_0000_ABST
    Figure 00000000_0000_ABST
Patent Text Reader

Abstract

To provide a method, an apparatus, and a system for encoding and decoding a block of a video sample.SOLUTION: A method includes: determining whether sign bit hiding is used for a subblock, wherein the determination is based on a value of a conversion skip flag determined for the subblock and a value of a sign bit hiding flag related to the subblock; when the sign bit hiding is not used, decoding sign bits in a number equal to the number of significant coefficients in the subblock; and decoding the subblock by reconstructing a residual coefficient of the subblock using the decoded sign bit.SELECTED DRAWING: Figure 4
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] REFERENCE TO RELATED APPLICATIONS This application claims the benefit under 35 USC § 119 of the filing date of Australian Patent Application No. 2020201753, filed March 10, 2020, the entirety of which is incorporated herein by reference as if fully set forth herein.

[0002] The present invention relates generally to digital video signal processing, and in particular to a method, apparatus and system for encoding and decoding blocks of video samples. The present invention also relates to a computer program product including a computer readable medium having recorded thereon a computer program for encoding and decoding blocks of video samples. [Background technology]

[0003] Many applications for video coding currently exist, including applications for the transmission and storage of video data. Many video coding standards have also been developed, and others are currently under development. Recent developments in video coding standardization have led to the formation of a group called the "Joint Video Experts Team" (JVET). The Joint Video Experts Team (JVET) includes members of Study Group 16, Question 6 (SG16 / Q6) of the International Telecommunication Union's (ITU) Telecommunication Standardization Sector (ITU-T), also known as the "Video Coding Experts Group" (VCEG), and members of the International Organization for Standardization / International Electrotechnical Commission Joint Technical Committee 1 / Subcommittee 29 / Working Group 11 (ISO / IEC JTC1 / SC29 / WG11), also known as the "Moving Picture Experts group" (MPEG).

[0004] The Joint Video Experts Team (JVET) analyzed the responses and issued a Call for Proposals (CfP) at its 10th meeting held in San Diego, USA. The submitted responses demonstrated video compression capabilities that significantly outperformed those of the current state-of-the-art video compression standard, i.e., "high efficiency video coding" (HEVC). Based on this outperformance, it was decided to initiate a project to develop a new video compression standard, to be named "versatile video coding" (VVC). VVC is expected to address the continuing demand for ever-higher compression performance, especially as video formats increase in capabilities (e.g., at higher resolutions and higher frame rates) and address the increasing market demand for service delivery over WANs where bandwidth costs are relatively high. At the same time, VVC must be implementable in modern silicon processes and provide an acceptable trade-off between achieved performance versus implementation costs (e.g., in terms of silicon area, CPU processor load, memory usage, and bandwidth).

[0005] Video data includes a sequence of frames of image data, each frame including one or more color channels. Generally, one primary color channel and two secondary color channels are required. The primary color channel is commonly called the "luma" channel, and the secondary color channels are commonly called the "chroma" channels. Video data is typically displayed in the RGB (red-green-blue) color space, which has a high degree of correlation between the three respective components. The video data representation seen by an encoder or decoder often uses a color space such as YCbCr. YCbCr concentrates luminance, mapped to "luma" according to a transfer function, in the Y (primary) channel, and chroma in the Cb and Cr (secondary) channels. Additionally, the Cb and Cr channels may be spatially sampled (subsampled) at a lower rate compared to the luma channel, e.g., half horizontally and half vertically, known as the "4:2:0 chroma format". The 4:2:0 chroma format is used widely in Internet video streaming, broadcast television, Blu-Ray, and other applications. TMIt is commonly used in "consumer" applications, such as storage to disk. Subsampling the Cb and Cr channels at half the rate horizontally, and not subsampling vertically, is known as a "4:2:2 chroma format". 4:2:2 chroma formats are typically used in professional applications, including capturing footage for film production and the like. The higher sampling rate of the 4:2:2 chroma format makes the resulting video more resilient to editing operations, such as color grading. Before distribution to consumers, 4:2:2 chroma format material is often converted to 4:2:0 chroma format, which is then encoded for distribution to consumers. In addition to the chroma format, videos are also characterized by their resolution and frame rate. Example resolutions are ultra-high definition (UHD) with a resolution of 3840x2160, or "8K" with a resolution of 7680x4320, with example frame rates of 60 or 120Hz. The luma sample rate may range from approximately 500 megasamples / second to several gigasamples / second. For a 4:2:0 chroma format, the sample rate of each chroma channel is one-quarter the luma sample rate, and for a 4:2:2 chroma format, the sample rate of each chroma channel is one-half the luma sample rate.

[0006] The VVC standard is a "block-based" codec, where a frame is first divided into a square array of regions known as "coding tree units" (CTUs). A CTU typically occupies a relatively large area, such as 128x128 luma samples. However, the right-most and bottom-most CTUs of each frame may be smaller in area. Associated with each CTU is a "coding tree" for the luma channel and an additional coding tree for the chroma channels. The coding trees define a decomposition of the CTU's region into a set of blocks, also called "coding blocks" (CBs). It is also possible for a single coding tree to specify blocks for both the luma and chroma channels, in which case the collection of juxtaposed coding blocks is called a "coding unit" (CU), i.e., each CU has a coding block for each color channel. The CBs are processed for encoding or decoding in a specific order. As a result of the use of the 4:2:0 chroma format, a CTU having a luma coding tree for a 128x128 luma sample region has a corresponding chroma coding tree for a 64x64 chroma sample region co-located with the 128x128 luma sample region. When a single coding tree is used for the luma and chroma channels, the collection of collocated blocks for a given area is generally referred to as a "unit", e.g., the CU mentioned above, as well as a "prediction unit" (PU) and a "transform unit" (TU). When separate coding trees are used for a given area, the CB mentioned above, as well as a "prediction block" (PB) and a "transform block" (TB) are used.

[0007] Notwithstanding the above distinction between "units" and "blocks", the term "block" may be used as a general term for an area or region of a frame where an operation is applied to all color channels.

[0008] For each CU, a prediction unit (PU) of the contents (sample values) of the corresponding region of frame data is generated ("Prediction Unit"). If the PU is generated from sample values ​​in a previously signaled frame, the prediction is called inter prediction. If the PU is generated from previous samples in the same frame, the prediction is called intra prediction. Furthermore, a representation of the difference (or "residual" in the spatial domain) between the prediction and the contents of the region seen at the input to the encoder is formed. The differences in each color channel are transformed and coded as a block of residual coefficients to form one or more TUs for a given CU. The residual coefficients may be transformed by a transform, such as a discrete cosine transform (DCT), a discrete sine transform (DST), or other transform, to generate a final block of transform coefficients that substantially decorrelates the residual samples. Substantial coding gain may be achieved by quantizing the transform coefficients. The quantized transform coefficients are then traversed in an order, such as a backward diagonal scan, and each coefficient is coded by an entropy encoder. Entropy coding consists of representing each coefficient with a syntax element, each of which is binarized. The binarized syntax elements are then either further coded by a context-adaptive binary arithmetic coder (CABAC) or passed through to the bitstream ("bypass coding").

[0009] In some classes of video content, such as screen content, it may be advantageous to avoid performing a transform. If a transform is to be avoided, the residual coefficients are quantized, traversed, and coded. Because the statistics of the residual coefficients are not the same as the statistics of the transform coefficients, it is generally advantageous for the residual coefficients to be coded using a process different from the coding process of the transform coefficients. Typical methods used to code the residual coefficients include a "regular residual coding" (RRC) process and a "transform skip residual coding" (TSRC) process, where a particular one of the processes is selected for a block depending on whether a transform has been performed or not.

[0010] In some use cases, it may be desirable to compress video data losslessly (i.e., without coding losses). A CU may be losslessly coded by skipping both the transform and quantization steps. In the TSRC process, quantization can be avoided by setting the "quantization parameter" to a value that does not indicate quantization. However, as mentioned above, the TSRC process is only suitable for a class of video content, such as screen content. Therefore, forcing lossless coding of video data to use the TSRC process is not optimal. It is desirable to have more flexible options available for lossless coding according to the statistics of the video data being coded, while minimizing the amount of additional logic required to support the additional flexibility. Summary of the Invention

[0011] It is an object of the present invention to substantially overcome, or at least ameliorate, one or more disadvantages of existing arrangements.

[0012] One aspect of the invention provides a method for decoding a sub-block of residual coefficients of a transform block from a video bitstream, the method including determining whether sign bit hiding is used for the sub-block, the determination being based on a value of a transform skip flag determined for the sub-block and a value of a sign bit hiding flag associated with the sub-block, and if sign bit hiding is not used, decoding a number of sign bits equal to a number of significant coefficients in the sub-block, and decoding the sub-block by reconstructing the residual coefficients of the sub-block using the decoded sign bits.

[0013] According to another aspect, if the sign bit hiding flag has a value of TRUE, the transform skip flag has a value of FALSE, and the difference between the first significant position and the last significant position of the subblock is greater than 3, then sign bit hiding is used.

[0014] According to another aspect, if the sign bit hiding flag has a value of TRUE and the transform skip flag has a value of TRUE, then sign bit hiding is not used.

[0015] According to another aspect, the method further comprises, if it is determined that sign bit hiding is used, decoding a number of sign bits equal to the number of significant coefficients in the sub-block minus one, and determining an additional sign bit from a sum of parity of the significant coefficients of the sub-block.

[0016] Another aspect of the invention provides a method for decoding a sub-block of residual coefficients of a transform block from a video bitstream, the method including determining whether sign bit hiding is used for the sub-block, the determination being based on a value of a sign bit hiding flag and a value of a quantization parameter associated with the sub-block, and if sign bit hiding is not used, decoding a number of sign bits equal to a number of significant coefficients in the sub-block, and decoding the sub-block by reconstructing the residual coefficients of the sub-block using the decoded sign bits.

[0017] According to another aspect, if the sign bit hiding flag has a value of TRUE and the quantization parameter is equal to four, then sign bit hiding is not used.

[0018] According to another aspect, if the sign bit hiding flag has a value of TRUE, the quantization parameter is not equal to 4, and the difference between the first and last significant positions of a subblock is greater than 3, then sign bit hiding is used.

[0019] Another aspect of the present invention provides a method for decoding a sub-block of residual coefficients of a transform block from a video bitstream, the method including determining whether sign bit hiding is used for the sub-block, the determination being based on a value of a sign bit hiding flag and a value of a TSRC disabled flag, and if sign bit hiding is not used, decoding a number of sign bits equal to a number of significant coefficients in the sub-block, and decoding the sub-block by reconstructing residual coefficients of the sub-block using the decoded sign bits.

[0020] According to another aspect, sign bit hiding is used when the sign bit hiding flag has a value of TRUE, the TSRC invalid flag has a value of FALSE, and the difference between the first significant position and the last significant position of the subblock is greater than three.

[0021] According to another aspect, the sign bit hiding flag has a value of TRUE and the TSRC invalid flag has a value of TRUE.

[0022] Another aspect of the invention provides a non-transitory computer-readable medium having a computer program stored thereon for implementing a method for decoding a sub-block of residual coefficients of a transform block from a video bitstream, the method including: determining whether sign bit hiding is used for the sub-block, the determination being based on a value of a transform skip flag determined for the sub-block and a value of a sign bit hiding flag associated with the sub-block; and if sign bit hiding is not used, decoding a number of sign bits equal to a number of significant coefficients in the sub-block; and decoding the sub-block by reconstructing the residual coefficients of the sub-block using the decoded sign bits.

[0023] Another aspect of the invention provides a system having a memory and a processor, the processor configured to execute code stored in the memory to perform a method for decoding a sub-block of residual coefficients of a transform block from a video bitstream, the method including determining whether sign bit hiding is used for the sub-block, the determination being based on a value of a transform skip flag determined for the sub-block and a value of a sign bit hiding flag associated with the sub-block, and if sign bit hiding is not used, decoding a number of sign bits equal to a number of significant coefficients in the sub-block, and decoding the sub-block by reconstructing residual coefficients of the sub-block using the decoded sign bits.

[0024] Another aspect of the invention provides a video decoder configured to receive a sub-block of residual coefficients of a transform block from a video bitstream, determine whether sign bit hiding is used for the sub-block, the determination being based on a value of a transform skip flag determined for the sub-block and a value of a sign bit hiding flag associated with the sub-block, and if sign bit hiding is not used, decode the sub-block by decoding a number of sign bits equal to the number of significant coefficients in the sub-block and using the decoded sign bits to reconstruct the residual coefficients of the sub-block.

[0025] Other aspects are also described. [Brief description of the drawings]

[0026] At least one embodiment of the invention will now be described with reference to the following drawings and appendix. [Figure 1] FIG. 1 is a schematic block diagram illustrating a video encoding and decoding system. [Figure 2A] FIG. 2A forms a schematic block diagram of a general purpose computer system in which one or both of the video encoding and decoding systems of FIG. 1 may be implemented. [Figure 2B]FIG. 2B forms a schematic block diagram of a general purpose computer system capable of implementing either or both of the video encoding and decoding systems of FIG. [Diagram 3] FIG. 3 is a schematic block diagram showing functional modules of a video encoder. [Figure 4] FIG. 4 is a schematic block diagram showing the functional modules of a video decoder. [Diagram 5] FIG. 5 is a schematic block diagram illustrating an available division of a block into one or more blocks in a generic video coding tree structure. [Figure 6] FIG. 6 is a schematic diagram of a data flow for achieving an accepted division of a block into one or more blocks in a generic video coding tree structure. [Figure 7A] FIG. 7A shows an example of division of a coding tree unit (CTU) into several coding units (CUs). [Figure 7B] FIG. 7B shows an example of division of a coding tree unit (CTU) into several coding units (CUs). [Figure 8A] FIG. 8A shows a two-level back diagonal scan. [Figure 8B] FIG. 8B shows a two-level forward diagonal scan. [Figure 9] FIG. 9 illustrates a method for coding a transform block of residual coefficients. [Figure 10] FIG. 10 illustrates a method for decoding a transform block of residual coefficients. [Figure 11] FIG. 11 illustrates a method for quantizing a transform block of residual coefficients performed by the method of FIG. [Figure 12] FIG. 12 illustrates a method for scaling a transform block of quantized coefficients performed by the method of FIG. [Figure 13] FIG. 13 illustrates a method for encoding sub-blocks of quantized coefficients performed by the method of FIG. [Figure 14]FIG. 14 illustrates a method for decoding a sub-block of quantized coefficients performed by the method of FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0027] When referring to steps and / or features having the same reference number in one or more of the accompanying drawings, those steps and / or features have the same function or operation for the purposes of this specification, unless a contrary intention appears.

[0028] As mentioned above, it may be desirable for lossless encoding to be supported by the existing building blocks of a codec. However, using TSRC processing exclusively for lossless encoding can produce suboptimal encoding performance, since the diverse classes of video data encoded in a lossless manner cannot be guaranteed to exhibit the statistical properties for which the TSRC processing was designed. Thus, greater flexibility in the selection of high-level building blocks that lossless encoding can use allows for superior encoding performance with minimal additional complexity to the overall design.

[0029] FIG. 1 is a schematic block diagram illustrating functional modules of a video encoding and decoding system 100. The system 100 includes a source device 110 and a destination device 130. A communication channel 120 is used to communicate encoded video information from the source device 110 to the destination device 130. In some configurations, the source device 110 and the destination device 130 may comprise either or both of a respective mobile phone handset or a "smartphone," in which case the communication channel 120 is a wireless channel. In other configurations, the source device 110 and the destination device 130 may comprise video conferencing equipment, in which case the communication channel 120 is typically a wired channel, such as an Internet connection. Furthermore, the source device 110 and the destination device 130 may comprise any of a wide range of devices, including devices supporting over-the-air television broadcasts, cable television applications, Internet video applications (including streaming), and applications in which encoded video data is captured on some computer-readable storage medium, such as a hard disk drive in a file server.

[0030] As shown in Figure 1, source device 110 includes video source 112, video encoder 114, and transmitter 116. Video source 112 typically comprises a source of captured video frame data (shown as 113), such as an imaging sensor, a previously captured video sequence stored on a non-transitory recording medium, or video sourced from a remote imaging sensor. Video source 112 may also be the output of a computer graphics card, for example, displaying the video output of an operating system and various applications running on a computing device, such as a tablet computer. Examples of source device 110 that may include an imaging sensor as video source 112 include smartphones, video cameras, professional video cameras, and network video cameras.

[0031] Video encoder 114 converts (or "encodes") imaged frame data (indicated by arrow 113) from video source 112 into a bitstream (indicated by arrow 115), as will be further described with reference to FIG. 3. Bitstream 115 is transmitted by transmitter 116 over communication channel 120 as coded video data (or "coded video information"). Bitstream 115 may also be stored in non-transitory storage device 122, such as "flash" memory or a hard disk drive, until later transmitted over communication channel 120, or in lieu of transmission over communication channel 120.

[0032] Destination device 130 includes a receiver 132, a video decoder 134, and a display device 136. Receiver 132 receives encoded video data from communication channel 120 and passes the received video data as a bitstream to video decoder 134 (indicated by arrow 133). Video decoder 134 then outputs decoded frame data to display device 136 (indicated by arrow 135) to reproduce the video data. Decoded frame data 135 has the same chroma format as frame data 113. Examples of display device 136 include liquid crystal displays such as cathode ray tubes, smartphones, tablet computers, computer monitors, or standalone television sets. It is also possible that the functionality of each of source device 110 and destination device 130 is implemented in a single device, examples of which include a mobile phone handset and a tablet computer.

[0033] Notwithstanding the above exemplary devices, each of the source device 110 and the destination device 130 may be configured within a general-purpose computing system, typically via a combination of hardware and software components. Figure 2A shows such a computer system 200, including a computer module 201, input devices such as a keyboard 202, a mouse pointer device 203, a scanner 226, a camera 227 that may be configured as a video source 112, and a microphone 280, and output devices including a printer 215, a display device 214 that may be configured as a display device 136, and a speaker 217. An external Modulator-Demodulator (Modem) transceiver device 216 may be used by the computer module 201 to communicate to and from a communication network 220 via a connection 221. The communication network 220, which may represent the communication channel 120, may be a wide area network (WAN), such as the Internet, a cellular telecommunications network, or a private WAN. If the connection 221 is a telephone line, the modem 216 may be a conventional "dial-up" modem. Alternatively, the modem 216 may be a broadband modem where the connection 221 is a high-capacity (e.g., cable or optical) connection. A wireless modem may also be used for wireless connection to the communications network 220. The transceiver device 216 may provide the functionality of the transmitter 116 and the receiver 132, and the communications channel 120 may be embodied within the connection 221.

[0034] The computer module 201 typically includes at least one processor unit 205 and a memory unit 206. For example, the memory unit 206 can include a semiconductor random access memory (RAM) and a semiconductor read-only memory (ROM). The computer module 201 also includes several input / output (I / O) interfaces, including an audio-video interface 207 that couples to a video display 214, a speaker 217, and a microphone 280, an I / O interface 213 that couples to a keyboard 202, a mouse 203, a scanner 226, a camera 227, and optionally a joystick or other human interface device (not shown), and an interface 208 for an external modem 216 and a printer 215. The signal from the audio-video interface 207 to the computer monitor 214 is typically the output of a computer graphics card. In some implementations, the modem 216 may be integrated into the computer module 201, for example in the interface 208. The computer module 201 also has a local network interface 211, which allows the computer system 200 to be coupled via a connection 223 to a local area communication network 222, known as a local area network (LAN). As shown in Fig. 2A, the local communication network 222 can also be coupled to a wide network 220 via a connection 224, which typically includes a so-called "firewall" device or a device of similar functionality. The local network interface 211 can be an Ethernet TM Circuit Card, Bluetooth TM This may include a wireless or IEEE 802.11 wireless configuration, although many other types of interfaces may be implemented for the interface 211. The local network interface 211 may also provide the functionality of the transmitter 116, and the receiver 132 and the communication channel 120 may also be embodied in a local communications network 222.

[0035] The I / O interfaces 208 and 213 may provide either or both serial and parallel connectivity, the former typically being implemented according to the Universal Serial Bus (USB) standard and having a corresponding USB connector (not shown). A storage device 209 is provided and typically includes a hard disk drive (HDD) 210. Other storage devices such as floppy disk drives and magnetic tape drives (not shown) may also be used. An optical disk drive 212 is typically provided to serve as a non-volatile source of data. Optical disks (e.g., CD-ROM, DVD, Blu ray Disc TM ), USB-RAM, portable, external hard drives, and portable memory devices such as floppy disks, for example, may be used as suitable sources of data for computer system 200. Typically, any of HDD 210, optical drive 212, networks 220 and 222 may be configured to operate as a video source 112 or as a destination for decoded video data to be stored for playback via display 214. Source device 110 and destination device 130 of system 100 may be embodied in computer system 200.

[0036] The components 205-213 of the computer module 201 typically communicate via an interconnected bus 204 and in a manner which results in a conventional mode of operation of the computer system 200 known to those skilled in the art. For example, the processor 205 is coupled to the system bus 204 using a connection 218. Similarly, the memory 206 and optical disk drive 212 are coupled to the system bus 204 by a connection 219. Examples of computers on which the above-described arrangements can be executed include IBM-PC's and compatibles, Sun SPARC stations, Apple Macs, and similar. TM or a similar computer system.

[0037] Where appropriate or necessary, the video encoder 114 and the video decoder 134, as well as the methods described below, may be implemented using the computer system 200. In particular, the video encoder 114, the video decoder 134, and the methods described may be implemented as one or more software application programs 233 executable within the computer system 200. In particular, the video encoder 114, the video decoder 134, and the steps of the methods described are performed by instructions 231 (see FIG. 2B) in the software 233 executed within the computer system 200. The software instructions 231 may be formed as one or more code modules, each for performing one or more specific tasks. The software may also be divided into two separate parts, where a first part and a corresponding code module perform the methods described, and a second part and a corresponding code module manage a user interface between the first part and a user.

[0038] The software may be stored on a computer readable medium, including, for example, the storage devices described below. The software is loaded from the computer readable medium into the computer system 200 and thereafter executed by the computer system 200. Such software or a computer readable medium having a computer program recorded thereon is a computer program product. Use of the computer program product in the computer system 200 preferably results in an advantageous apparatus for implementing the video encoder 114, the video decoder 134, and the methods described.

[0039] The software 233 is typically stored in the HDD 210 or in the memory 206. The software is loaded into the computer system 200 from a computer-readable medium and executed by the computer system 200. Thus, for example, the software 233 may be stored on an optically readable disk storage medium (e.g., a CD-ROM) 225 that is read by the optical disk drive 212.

[0040] In some cases, the application programs 233 may be encoded on one or more CD-ROMs 225 and provided to the user and read via a corresponding drive 212, or alternatively may be read by the user from the network 220 or 222. Additionally, the software may also be loaded into the computer system 200 from other computer readable media. A computer readable storage medium refers to any non-transitory, tangible storage medium that provides recorded instructions and / or data to the computer system 200 for execution and / or processing. Examples of such storage media are floppy disks, magnetic tapes, CD-ROMs, DVDs, Blu-ray Discs, and the like. TM , hard disk drives, ROMs or integrated circuits, USB memory, magneto-optical disks, or computer readable cards such as PCMCIA cards, etc., regardless of whether such devices are internal or external to the computer module 201. Examples of transitory or non-tangible computer readable transmission media that may also participate in providing software, application programs, instructions and / or video data or encoded video data to the computer module 401 include wireless or infrared transmission channels, as well as network connections to another computer or networked device, and the Internet or intranets, including email transmissions and information recorded on websites and the like.

[0041] A second portion of the application programs 233 and corresponding code modules described above may be executed to implement one or more graphical user interfaces (GUIs) that are rendered or otherwise represented on the display 214. Typically through manipulation of a keyboard 202 and mouse 203, a user of the application and computer system 200 may manipulate the interface in a functionally adaptable manner to provide control commands and / or input to an application associated with the GUI. Other forms of functionally adaptable user interfaces may also be implemented, such as an audio interface that utilizes speech prompts output via speaker 217 and user voice commands input via microphone 280.

[0042] Figure 2B is a detailed schematic block diagram of processor 205 and "memory" 234. Memory 234 represents a logical collection of all memory modules (including HDD 209 and semiconductor memory 206) accessible by computer module 201 of Figure 2A.

[0043] When the computer module 201 is first powered on, a power-on self-test (POST) program 250 is executed. The POST program 250 is typically stored in the ROM 249 of the semiconductor memory 206 of FIG. 2A. Hardware devices such as the ROM 249 that store software are sometimes referred to as firmware. The POST program 250 inspects the hardware in the computer module 201 to ensure that it functions properly, and typically checks the processor 205, the memory 234 (209, 206), and a basic input / output system software (BIOS) module 251, which is typically also stored in the ROM 249, for correct operation. If the POST program 250 runs successfully, the BIOS 251 boots the hard disk drive 210 of FIG. 2A. When the hard disk drive 210 boots, a bootstrap loader program 252 resident on the hard disk drive 210 is executed via the processor 205. This loads the operating system 253 into the RAM memory 206, on which the operating system 253 begins to operate. Operating system 253 is a system level application executable by processor 205 and performs a variety of high-level functions including processor management, memory management, device management, storage management, software application interface, and a general-purpose user interface.

[0044] The operating system 253 manages the memory 234 (209, 206) to ensure that each process or application executing on the computer module 201 has enough memory to execute without colliding with memory allocated to another process. Additionally, the different types of memory available in the computer system 200 of FIG. 2A must be used appropriately so that each process can execute effectively. Thus, the aggregate memory 234 is not intended to indicate how specific segments of memory are allocated (unless otherwise noted), but rather is intended to provide a general view of memory accessible by the computer system 200 and how such segments are used.

[0045] As shown in FIG. 2B, the processor 205 includes a number of functional modules, including a control unit 239, an arithmetic logic unit (ALU) 240, and a local or internal memory 248, sometimes referred to as a cache memory. The cache memory 248 typically includes a number of storage registers 244-246 in a register section. One or more internal buses 241 operatively interconnect these functional modules. The processor 205 also typically has one or more interfaces 242 for communicating with external devices via the system bus 204, using a connection 218. The memory 234 is coupled to the bus 204 using a connection 219.

[0046] The application program 233 includes a sequence of instructions 231, which may include conditional branch and loop instructions. The program 233 may also include data 232 used in the execution of the program 233. The instructions 231 and the data 232 are stored in memory locations 228, 229, 230 and 235, 236, 237, respectively. Depending on the relative sizes of the instructions 231 and the memory locations 228-230, a particular instruction may be stored in a single memory location, as indicated by the instruction shown in memory location 230. Alternatively, the instruction may be segmented into several portions, each stored in a separate memory location, as indicated by the instruction segments shown in memory locations 228 and 229.

[0047] In general, the processor 205 is given a set of instructions to execute therein. The processor 205 waits for a subsequent input, to which the processor 205 responds by executing another set of instructions. Each input may come from one or more of several sources, including data generated by one or more of the input devices 202, 203, data received from an external source via one of the networks 220, 202, data retrieved from one of the storage devices 206, 209, or data retrieved from a storage medium 225 inserted into a corresponding reader 212, all shown in FIG. 2A. Execution of the set of instructions may result in data being output. Execution may also include storing data or variables to memory 234.

[0048] The video encoder 114, the video decoder 134, and the described methods may use input variables 254 that are stored in corresponding memory locations 255, 256, 257 in memory 234. The video encoder 114, the video decoder 134, and the described methods generate output variables 261, which are stored in corresponding memory locations 262, 263, 264 in memory 234. Intermediate variables 258 may be stored in memory locations 259, 260, 266, and 267.

[0049] 2B, the registers 244, 245, 246, the arithmetic logic unit (ALU) 240, and the control unit 239 cooperate to execute the sequence of micro-operations required to perform a "fetch, decode, and execute" cycle for every instruction in the instruction set that constitutes the program 233. Each fetch, decode, and execute cycle is A fetch operation to fetch or read an instruction 231 from memory locations 228, 229, 230 A decode operation in which the control unit 239 determines which instruction has been fetched. The control unit 239 and / or the ALU 240 perform the operations to execute the instructions. has.

[0050] Thereafter, a further fetch, decode, and execute cycle of the next instruction can be performed. Similarly, a store cycle can be performed in which control unit 239 stores or writes a value to memory location 232.

[0051] Each step or sub-process in the methods of Figures 19-14 described below is associated with one or more segments of program 233 and is typically performed by register sections 244, 245, 247, ALU 240, and control unit 239 within processor 205 working together to perform a fetch, decode, and execute cycle for all instructions in the instruction set for the noted segment of program 233.

[0052] FIG. 3 is a schematic block diagram showing the functional modules of the video encoder 114. FIG. 4 is a schematic block diagram showing the functional modules of the video decoder 134. In general, data passes between the functional modules of the video encoder 134 and the video encoder 114 in groups of samples or coefficients, such as a division of a block into fixed-size sub-blocks, or as an array. The video encoder 114 and the video decoder 134 can be implemented using a general-purpose computer system 200, as shown in FIG. 2A and FIG. 2B, and the various functional modules can be realized by software executable within the computer system 200, such as one or more software code modules of a software application program 233 residing on a hard disk drive 205 and controlled during its execution by the processor 205, by dedicated hardware within the computer system 200. Alternatively, the video encoder 114 and the video decoder 134 may be implemented by a combination of software executable within the computer system 200 and dedicated hardware. The video encoder 114, the video decoder 134, and the methods described may alternatively be implemented with dedicated hardware, such as one or more integrated circuits performing the functions or sub-functions of the methods described. Such dedicated hardware may include a graphics processing unit (GPU), a digital signal processor (DSP), an application specific standard product (ASSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or one or more microprocessors and associated memories. In particular, the video encoder 114 includes modules 310-386, and the video decoder 134 includes modules 420-496, which may each be implemented as one or more software code modules of the software application program 233.

[0053] The video encoder 114 of FIG. 3 is an example of a generic video coding (VVC) video encoding pipeline, although other video codecs may be used to perform the processing stages described herein. The video encoder 114 receives imaged frame data 113, such as a sequence of frames, each frame including one or more color channels. The frame data 113 may be in any chroma format, for example, 4:0:0, 4:2:0, 4:2:2, or 4:4:4 chroma format. The block partitioner 310 first divides the frame data 113 into CTUs, generally square in shape, with a particular size for the CTUs configured to be used. The size of the CTUs may be, for example, 64×64, 128×128, or 256×256 luma samples. The block partitioner 310 further divides each CTU into one or more CBs according to the luma coding tree and the chroma coding tree. The CBs may have a variety of sizes and include both square and non-square aspect ratios. In the VVC standard, CBs, CUs, PUs, and TUs always have side lengths that are powers of 2. Thus, a current CB, denoted as 312, is output from the block partitioner 310 and proceeds according to iterations over one or more blocks of the CTU according to the chroma coding tree and luma coding tree of the CTU. Options for partitioning a CTU into CBs are further described below with reference to Figures 5 and 6.

[0054] The CTUs resulting from the initial partitioning of frame data 113 may be scanned in raster scan order and grouped into one or more "slices." A slice may be an "intra" (or "I") slice. Intra slices (I slices) indicate that all CUs in a slice are intra predicted. Alternatively, a slice may be uni- or bi-predictive ("P" or "B" slices, respectively), indicating further availability of uni- and bi-prediction in the slice, respectively.

[0055] For each CTU, the video encoder 114 operates in two stages. In the first stage (called the "search" stage), the block partitioner 310 tests various potential configurations of the coding tree. Each potential configuration of the coding tree has an associated "candidate" CB. The first stage involves testing various candidate CBs to select the CB that provides high compression efficiency with low distortion. This testing typically involves Lagrangian optimization, whereby the candidate CBs are evaluated based on a weighted combination of rate (coding cost) and distortion (error with respect to the input frame data 113). The "best" candidate CB (the CB with the lowest evaluated rate / distortion) is selected for subsequent encoding into the bitstream 115. The evaluation of the candidate CBs includes the options of using a CB for a given area, or further dividing the area according to various partitioning options and encoding each of the resulting smaller areas with further CBs, or further dividing the area. As a result, both the CB and the coding tree itself are selected in the search stage.

[0056] The video encoder 114 generates a prediction block (PB) for each CB, e.g., CB 312, indicated by arrow 320. The PB 320 is a prediction of the content of the associated CB 312. A subtractor module 322 generates a difference between the PB 320 and the CB 312, indicated as 324 (or "residual", referring to the difference being in the spatial domain). The residual 324 is a block-sized difference between corresponding samples in the PB 320 and the CB 312. The residual 324 is transformed and quantized, and represented as a transform block (TB) indicated by arrow 336. The PB 320 and associated TB 336 are typically selected from one of many possible candidate CBs, e.g., based on an estimated cost or distortion.

[0057] A candidate coding block (CB) is a CB resulting from one of the prediction modes available to the video encoder 114 for an associated PB and the resulting residual. Each candidate CB results in one or more corresponding TBs. The TB 336 is a quantized and transformed representation of the residual 324. When combined with the predicted PB in the video decoder 114, the TB 336 reduces the difference between the decoded CB and the original CB 312 at the expense of additional signal in the bitstream.

[0058] Thus, each candidate coding block (CB), i.e., a prediction block (PB) in combination with a transform block (TB), has an associated coding cost (or "rate") and an associated difference (or "distortion"). The rate is typically measured in bits. The distortion of a CB is typically estimated as a sample value difference, such as the sum of absolute differences (SAD) or sum of squared differences (SSD). The estimate obtained from each candidate PB is determined by a mode selector 386 using the residual 324 to determine a prediction mode (represented by arrow 388). The estimation of the coding cost associated with each candidate prediction mode and the corresponding residual coding can be performed at a significantly lower cost than entropy coding of the residual. Thus, a large number of candidate modes can be evaluated to determine the optimal mode for rate-distortion detection.

[0059] Determining the optimal mode from a rate-distortion perspective is typically accomplished using a variant of Lagrangian optimization. The selection of a prediction mode 388 typically involves determining an encoding cost for the residual data resulting from application of a particular prediction mode. The encoding cost may be approximated by using the "sum of absolute transform differences" (SATD), thereby using a relatively simple transform, such as the Hadamard transform, to obtain an estimated transformed residual cost. In some embodiments using a relatively simple transform, the cost resulting from the simplified estimation method is monotonically related to the actual cost that would otherwise be determined from a full evaluation. In embodiments having a monotonically related estimated cost, the simplified estimation method may be used to make the same decision (i.e., prediction mode) while reducing the complexity of the video encoder 114. To allow for possible non-monotonicity in the relationship between the estimated cost and the actual cost, the simplified estimation method may be used to generate a list of best candidates. The non-monotonicity may result, for example, from additional mode decisions available for encoding the residual data. The list of best candidates may be any number. Using the best candidates, a more thorough search can be performed to establish an optimal mode selection for encoding the residual data for each of the candidates, allowing for the final selection of the prediction mode 388 along with other mode decisions.

[0060] Prediction modes are broadly classified into two categories. The first category is "intra-frame prediction" (also called "intra prediction"). In intra-frame prediction, a prediction for a block is generated, and the generation method may use other samples obtained from the current frame. Types of intra prediction include intra-planar, intra-DC, intra-angle, and matrix weighted intra prediction (MIP). For intra-predicted PB, different intra prediction modes can be used for luma and chroma, and therefore intra prediction is mainly described with respect to the operation on PB. Furthermore, chroma CB may be predicted from co-located luma samples by cross-component linear model prediction.

[0061] The second category of prediction modes is "interframe prediction" (also called "inter prediction"). In interframe prediction, a prediction of a block is generated using samples from one or two frames that precede the current frame in the order of encoding the frames in the bitstream. Furthermore, for interframe prediction, a single coding tree is typically used for both luma and chroma channels. The encoding order of the frames in the bitstream may differ from the order of the frames at the time of capture or display. If one frame is used for prediction, the block is said to be "uni-predictive" and has one associated motion vector. If two frames are used for prediction, the block is said to be "bi-predictive" and has two associated motion vectors. For P slices, each CU may be intra-predicted or uni-predicted. For B slices, each CU may be intra-predicted, uni-predicted, or bi-predicted. Frames are typically coded using a "group of pictures" structure that allows for a temporal hierarchy of frames. The temporal hierarchy of frames allows frames to reference previous and following pictures in the order of displaying the frames. The images are coded in the order necessary to ensure that the dependencies for decoding each frame are met.

[0062] A subcategory of inter prediction is called "skip mode". Inter prediction mode and skip mode are described as two separate modes. However, both inter prediction mode and skip mode include a motion vector that references a block of samples from a previous frame. Inter prediction includes a coded motion vector delta that specifies a motion vector for a motion vector predictor. The motion vector predictor is obtained from a list of one or more candidate motion vectors selected with a "merge index". The coded motion vector delta provides a spatial offset to the selected motion vector prediction. Inter prediction also uses coded residuals in the bitstream 133. Skip mode uses only an index (also called a "merge index") to select one of several motion vector candidates. The selected candidate is used without further signaling. Skip mode also does not support coding of residual coefficients. When skip mode is used, the absence of coded residual coefficients means that there is no need to perform a transform for skip mode. Therefore, skip mode typically does not cause pipelining problems. Pipelining problems may be the case for intra-predicted CUs and inter-predicted CUs. Due to the limited signaling of skip mode, skip mode is useful for achieving very high compression performance when relatively high quality reference frames are available. Bi-predicted CUs in higher temporal layers of a random access picture group structure typically have high quality reference pictures and motion vector candidates that accurately reflect the underlying motion.

[0063] The samples are selected according to a motion vector and a reference picture index. The motion vector and the reference picture index apply to all color channels, and therefore inter prediction is mainly described in terms of operation on the PU, not the PB. Within each category (i.e., intra and inter frame prediction), different techniques can be applied to generate the PU. For example, intra prediction can use values ​​from adjacent rows and columns of previously reconstructed samples in combination with a direction to generate the PU according to a predetermined filtering and generation process. Alternatively, the PU may be described using a small number of parameters. Inter prediction methods can vary in the number of motion parameters and their accuracy. The motion parameters usually include a reference frame index indicating which reference frame should be used from a list of reference frames, and a spatial transformation for each of the reference frames, but can include more frames, special frames, or complex affine parameters such as scaling and rotation. Furthermore, a predetermined motion refinement process can be applied to generate a dense motion estimate based on the referenced sample block.

[0064] A Lagrangian or similar optimization process may be employed to select both the optimal partitioning of the CTU into CBs (by the block partitioner 310) and the selection of the best prediction mode from multiple possibilities. Through application of the Lagrangian optimization process of the candidate modes in the mode selector module 386, the prediction mode with the lowest cost measure is selected as the "best" mode. The lowest cost mode is the selected prediction mode 388, which is also encoded into the bitstream 115 by the entropy encoder 338. The selection of the prediction mode 388 by the operation of the mode selector module 386 extends the operation of the block partitioner 310. For example, the candidates for the selection of the prediction mode 388 may include a mode applicable to a given block and further a mode applicable to multiple smaller blocks collectively located with the given block. In the case of including a mode applicable to the given block and the smaller co-located blocks, the process of selecting the candidates is also a process of determining the best hierarchical decomposition of the CTU into CBs.

[0065] In a second stage of operation of the video encoder 114 (called the “encoding” stage), an iteration over the selected luma coding tree and the selected chroma coding tree, and thus each selected CB, is performed in the video encoder 114. In the iteration, the CBs are encoded into a bitstream 115, as described further herein.

[0066] The entropy encoder 338 supports both variable length coding of syntax elements and arithmetic coding of syntax elements. Arithmetic coding is supported using a context-adaptive binary arithmetic coding (CABAC) process. An arithmetically coded syntax element consists of a sequence of one or more "bins". A bin, like a bit, has a value of "0" or "1". Bins are not coded into the bitstream 115 as discrete bits. A bin has an associated predicted (or "likelihood" or "most likely") value and an associated probability, known as a "context". When the actual bin to be coded matches the predicted value, a "most probable symbol" (MPS) is coded. Coding the most probable symbol is relatively inexpensive in terms of bits consumed. When the actual bin to be coded does not match the likely value, a "least probable symbol" (LPS) is coded. Coding the least probable symbol has a relatively high cost in terms of bits consumed. Bin coding techniques allow for efficient encoding of bins with biased "0" vs. "1" probabilities. For syntax elements with two possible values ​​(i.e. "flags"), a single bin is sufficient. For syntax elements with many possible values, a series of bins is required.

[0067] The presence of a later bin in the sequence may be determined based on the value of a previous bin in the sequence. Furthermore, each bin may be associated with two or more contexts. The selection of a particular context may depend on the previous bin of the syntax element, the bin values ​​of adjacent syntax elements (i.e., from adjacent blocks), etc. Each time a context-encoded bin is encoded, the context (if any) selected for that bin is updated in a manner that reflects the new bin value. In this manner, the binary arithmetic coding scheme is said to be adaptive.

[0068] Also supported by the video encoder 114 are bins that lack context ("bypass bins"). Bypass bins are coded assuming an equal probability distribution between "0" and "1". Thus, each bin occupies one bit in the bitstream 115. The lack of context saves memory and reduces complexity, and therefore bypass bins are used when the distribution of values ​​for a particular bin is not skewed.

[0069] The entropy encoder 338 encodes the prediction mode 388 using a combination of context coding bins and bypass coding bins. For example, if the prediction mode 388 is an intra-prediction mode, a list of "most probable modes" is generated in the video encoder 114. The list of most probable modes is typically of a fixed length, such as three or six modes, and may include modes encountered in previous blocks. The context coding bins encode a flag indicating whether the prediction mode is one of the most probable modes. If the intra-prediction mode 388 is one of the most probable modes, further signaling using bypass coding bins is encoded. The encoded further signaling may indicate which most probable mode corresponds to the intra-prediction mode 388, for example, using a truncated unary bin string. Otherwise, the intra-prediction mode 388 is encoded as a "remaining mode". The encoding as a remaining mode may express intra-prediction modes other than those present in the most probable modes list using an alternative syntax, such as a fixed-length code that is also encoded using bypass coded bins.

[0070] The multiplexer module 384 outputs the PB 320 according to the determined best prediction mode 388 to select from the tested prediction modes for each candidate CB. The candidate prediction modes do not have to include all possible prediction modes supported by the video encoder 114.

[0071] After determining and selecting the PB 320 and subtracting it from the original sample block in a subtractor 322, a residual represented by 324 with the lowest coding cost is obtained and subjected to lossy compression. The lossy compression process includes the steps of transformation, quantization, and entropy coding. A forward primary transform module 326 applies a forward transform to the residual 324, transforming it from the spatial domain to the frequency domain and generating primary transform coefficients represented by arrow 328. The primary transform coefficients 328 are passed to a forward secondary transform module 330, which generates transform coefficients represented by arrow 332 by performing a non-separable secondary transform (NSST) operation. The forward primary transform is typically separable, typically using a type II discrete cosine transform (DCT-2) to transform the set of rows and then the set of columns of each block, although type VII discrete sine transform (DST-7) and type VIII discrete cosine transform (DCT-8) may also be utilized, for example, horizontally for block widths not exceeding 16 samples and vertically for block heights not exceeding 16 samples. The transformation of each set of rows and columns is performed by first applying a one-dimensional transform to each row of the block to generate an intermediate result, and then applying a one-dimensional transform to each column of the intermediate result to generate a final result. The forward secondary transform is generally a non-separable transform, which is only applied to the residuals of intra-predicted CUs, and may nevertheless be bypassed. The forward secondary transform operates on either 16 samples (arranged as the top-left 4x4 sub-block of the primary transform coefficients 328) or 64 samples (arranged as the top-left 8x8 coefficients, arranged as four 4x4 sub-blocks of the primary transform coefficients 328). Furthermore, the matrix coefficients of the forward secondary transform are selected from multiple sets according to the intra prediction mode of the CU, such that two sets of coefficients are available for use. The use of one of the sets of matrix coefficients, i.e., bypassing the forward secondary transform, is signaled with the "nsst_index" syntax element, which is coded using truncated unary binarization to represent the value zero (no secondary transform is applied), one (the first set of selected matrix coefficients), or two (the second set of selected matrix coefficients).

[0072] The video encoder 114 may also choose to skip both the primary and secondary transforms, known as a "transform skip" mode. Skipping the transform is appropriate for residual data that lacks adequate correlation to reduce coding costs via representation as transform basis functions. Certain types of content, such as relatively simple computer-generated graphics, may exhibit similar behavior. When transform skip mode is used, the transform coefficients 332 are the same as the residual coefficients 324.

[0073] The transform coefficients 332 are passed to a quantizer module 334. In the module 334, quantization by a "quantization parameter" is performed to generate quantized coefficients represented by arrow 336. The quantization parameter is constant for a given TB, thus resulting in uniform scaling for the generation of the residual coefficients for the TB. Non-uniform scaling is also possible by applying a "quantization matrix", whereby the scaling factor applied to each residual coefficient is derived from a combination of the quantization parameter and a corresponding entry in a scaling matrix, typically having a size equal to the size of the TB. The scaling matrix may have a size smaller than the size of the TB, and when applied to the TB, a nearest neighbor approach is used to provide a scaling value for each residual coefficient from a scaling matrix having a size smaller than the TB size. The quantized coefficients 336 are provided to an entropy encoder 338 for encoding in the bitstream 115. Typically, the quantized coefficients of each TB having at least one significant quantized coefficient are scanned according to a scan pattern to generate an ordered list of values. The scan pattern typically scans the TB as a sequence of 4x4 "sub-blocks", providing a regular scanning operation with a granularity of 4x4 sets of residual coefficients, where the arrangement of the sub-blocks depends on the size of the TB. Additionally, prediction modes 388 and corresponding block partitions are also coded into the bitstream 115.

[0074] As mentioned above, the video encoder 114 requires access to a frame representation that corresponds to the frame representation seen by the video decoder 134. Thus, the quantized coefficients 336 are also inverse quantized by the inverse quantizer module 340 to generate reconstructed transform coefficients represented by arrow 342. The reconstructed transform coefficients 342 pass through the inverse secondary transform module 344 to generate reconstructed primary transform coefficients represented by arrow 346. The reconstructed primary transform coefficients 346 are passed to the inverse primary transform module 348 to generate reconstructed residual samples of the TU represented by arrow 350. The type of inverse transform performed by the inverse secondary transform module 344 corresponds to the type of forward transform performed by the forward secondary transform module 330. The type of inverse transform performed by the inverse primary transform module 348 corresponds to the type of primary transform performed by the primary transform module 326. The summing module 352 sums the reconstructed residual samples 350 and the PU 320 to generate reconstructed samples of the CU (indicated by arrow 354).

[0075] The reconstructed samples 354 are passed to a reference sample cache 356 and an in-loop filter module 368. The reference sample cache 356 is typically implemented using static RAM on the ASIC (thus avoiding costly off-chip memory accesses) and provides the minimum sample storage required to satisfy the dependencies for generating intra-frame PBs for subsequent CUs in the frame. The minimum dependencies typically include a "line buffer" of samples along the bottom of a row of CTUs to be used by the next row and column buffering of the CTU, the extent of which is set by the height of the CTU. The reference sample cache 356 supplies reference samples (indicated by arrow 358) to a reference sample filter 360. The sample filter 360 applies a smoothing operation to generate filtered reference samples (indicated by arrow 362). The filtered reference samples 362 are used by an intra-frame prediction module 364 to generate an intra-predicted block of samples represented by arrow 366. For each candidate intra-prediction mode, the intra-frame prediction module 364 generates a block of samples, i.e., 366.

[0076] The in-loop filter module 368 applies several filtering stages to the reconstructed samples 354. The filtering stages include a "deblocking filter" (DBF) that applies smoothing aligned to CU boundaries to reduce artifacts resulting from discontinuities. Another filtering stage present in the in-loop filter module 368 is the "adaptive loop filter" (ALF), which applies a Wiener-based adaptive filter to further reduce distortion. A further available filtering stage in the in-loop filter module 368 is the "sample adaptive offset" (SAO) filter. The SAO filter operates by first classifying the reconstructed samples into one or more categories and applying an offset at the sample level according to the assigned category.

[0077] The filtered samples, represented by arrow 370, are output from the in-loop filter module 368. The filtered samples 370 are stored in a frame buffer 372. The frame buffer 372 typically has the capacity to store several pictures (e.g., up to 16) and is therefore stored in the memory 206. The frame buffer 372 is typically not stored using on-chip memory due to the large memory consumption required. Therefore, access to the frame buffer 372 is costly in terms of memory bandwidth. The frame buffer 372 provides reference frames (represented by arrow 374) to a motion estimation module 376 and a motion compensation module 380.

[0078] The motion estimation module 376 estimates several "motion vectors" (denoted as 378), each a Cartesian space offset from the position of the current CB, and referencing a block in one of the reference frames in the frame buffer 372. A filtered block of reference samples (denoted as 382) is generated for each motion vector. The filtered reference samples 382 form further candidate modes available for potential selection by the mode selector 386. Furthermore, for a given CU, the PU 320 may be formed using one reference block ("uni-predictive") or two reference blocks ("bi-predictive"). For a selected motion vector, the motion compensation module 380 generates the PB 320 according to a filtering process that supports sub-pixel accuracy in the motion vector. Thus, the motion estimation module 376 (operating on many candidate motion vectors) may perform a simplified filtering process compared to that of the motion compensation module 380 (operating on only the selected candidate) to achieve reduced computational complexity. When the video encoder 114 selects inter prediction for a CU, a motion vector 378 is encoded into the bitstream 115.

[0079] 3 is described with reference to generic video coding (VVC), other video coding standards or implementations may use the processing stages of modules 310-386. The frame data 113 (and bitstream 115) may be stored in memory 206, a hard disk drive 210, a CD-ROM, a Blu-ray disc TM or other computer-readable storage medium. Additionally, the frame data 113 (and bitstream 115) may be received (or transmitted) from an external source, such as a server connected to the communications network 220 or a radio frequency receiver.

[0080] The video decoder 134 is shown in Figure 4. The video decoder 134 of Figure 4 is an example of a generic video coding (VVC) video decoding pipeline, although other video codecs may be used to perform the processing stages described herein. As shown in Figure 4, a bitstream 133 is input to the video decoder 134. The bitstream 133 may be stored in a memory 206, a hard disk drive 210, a CD-ROM, a Blu-ray disc TM or other non-transitory computer readable storage medium. Alternatively, the bitstream 133 may be received from an external source, such as a server connected to the communications network 220 or a radio frequency receiver. The bitstream 133 includes encoded syntax elements representing the imaging frame data to be decoded.

[0081] The bitstream 133 is input to the entropy decoder module 420. The entropy decoder module 420 extracts syntax elements from the bitstream 133 by decoding a sequence of "bins" and passes the values ​​of the syntax elements to other modules in the video decoder 134. An example of a syntax element extracted from the bitstream 133 is the quantized coefficients 424. The entropy decoder module 420 uses an arithmetic decoding engine to decode each syntax element as a sequence of one or more bins. Each bin can use one or more "contexts", with the context describing the probability levels used to code the "1" and "0" values ​​of the bin. When multiple contexts are available for a given bin, a "context modeling" or "context selection" step is performed to select one of the available contexts to decode the bin. The process of decoding the bins sequentially forms a feedback loop. The number of operations in the feedback loop is preferably minimized to enable the entropy decoder 420 to achieve a high throughput in bins / second. Context modeling relies on other properties of the bitstream known to video decoder 134 when selecting a context, i.e., a property prior to the current bin. For example, a context may be selected based on the quadtree depth of the current CU in the coding tree. The dependency is preferably based on properties known prior to decoding the bin or determined without requiring lengthy sequential processing.

[0082] The quantized coefficients 424 are input to an inverse quantizer module 428. The inverse quantizer module 428 performs inverse quantization (or "scaling") on the quantized coefficients 424 according to the quantization parameter to generate reconstructed intermediate transform coefficients represented by arrow 432. If the use of a non-uniform inverse quantization matrix is ​​indicated in the bitstream 133, the video decoder 134 reads the quantization matrix from the bitstream 133 as a sequence of scaling factors and places the scaling factors in the matrix. The inverse scaling uses the quantization matrix in combination with the quantization parameter to generate reconstructed intermediate transform coefficients 432. The reconstructed intermediate transform coefficients 432 are passed to an inverse secondary transform module 436, where a secondary transform may be applied according to the decoded "nsst_index" syntax element. The "nsst_index" is decoded from the bitstream 133 by the entropy decoder 420 under execution of the processor 205. The inverse secondary transform module 436 generates reconstructed transform coefficients 440.

[0083] The reconstructed transform coefficients 440 are passed to an inverse primary transform module 444. The module 444 transforms the coefficients from the frequency domain back to the spatial domain. The result of the operation of the module 444 is a block of residual samples represented by an arrow 448. The block of residual samples 448 is equal in size to the corresponding CU. The type of inverse primary transform can be a type II discrete cosine transform (DCT-2), a type VII discrete sine transform (DST-7), a type VIII discrete cosine transform (DCT-8), or a "transform skip" mode. The use of transform skip mode is signaled by a transform skip flag that is decoded or otherwise inferred from the bitstream 133. When transform skip mode is used, the residual samples 448 are the same as the reconstructed transform coefficients 440.

[0084] The residual samples 448 are provided to a summing module 450. In the summing module 450, the residual samples 448 are added to a decoded PB (represented as 452) to generate a block of reconstructed samples represented by arrow 456. The reconstructed samples 456 are provided to a reconstructed sample cache 460 and to an in-loop filtering module 488. The in-loop filtering module 488 generates a reconstructed block of frame samples represented as 492. The frame samples 492 are written to a frame buffer 496.

[0085] The reconstructed sample cache 460 operates similarly to the reconstructed sample cache 356 of the video encoder 114. The reconstructed sample cache 460 provides storage for reconstructed samples necessary to intra-predict subsequent CBs without the memory 206 (e.g., by substituting the data 232, which is typically an on-chip memory). Reference samples, represented by arrow 464, are obtained from the reconstructed sample cache 460 and provided to a reference sample filter 468 to generate filtered reference samples, indicated by arrow 472. The filtered reference samples 472 are provided to an intra-frame prediction module 476. The module 476 generates blocks of intra-predicted samples, represented by arrow 480, according to the intra-prediction mode parameters 458 signaled in the bitstream 133 and decoded by the entropy decoder 420.

[0086] When the prediction mode of the CB is indicated as intra prediction in the bitstream 133, the intra prediction samples 480 form the decoded PB 452 via the multiplexer module 484. Intra prediction generates a predictive block (PB) of samples, i.e., a block in one color component derived using "neighboring samples" in the same color component. Neighboring samples are samples that are adjacent to the current block and are preceding it in the block decoding order, and therefore have already been reconstructed. When the luma and chroma blocks are juxtaposed, the luma and chroma blocks can use different intra prediction modes. However, each of the two chroma channels shares the same intra prediction mode.

[0087] Intra prediction for luma blocks consists of four types. "DC intra prediction" involves populating the PB with a single value that represents the average of neighboring samples. "Planar intra prediction" involves populating the PB with samples according to a plane, with vertical and horizontal gradients and DC offsets derived from neighboring samples. "Angular intra prediction" involves populating the PB with neighboring samples that have been filtered and propagated in a particular direction (or "angle") across the PB. In VVC, the PB can choose from up to 65 angles, and rectangular blocks can utilize different angles that are not available to square blocks. "Matrix intra prediction" involves populating the PB by multiplying a reduced set of neighboring samples with one of many available matrices available to the video decoder 134. The reduced set of neighboring samples is generated by filtering and subsampling the neighboring samples. A reduced set of predicted samples is then generated by multiplying the reduced set of samples with a matrix and adding an offset vector. The matrix and associated offset vector are selected from a number of possible matrices depending on the size of the PB, and the specific selection of the matrix and offset vector is indicated by the "MIP mode" syntax element. For example, for a PB having a size larger than 8x8, there are 11 MTP modes, while for a PB of 8x8 size, there are 19 MIP modes. Finally, the PB generated by matrix intra prediction is input from a reduced set of prediction samples by interpolation.

[0088] A fifth type of intra prediction is available for chroma PB, whereby the PB is generated from collocated luma reconstructed samples according to a "cross-component linear model" (CCLM) mode. Three different CCLM modes are available, each of which uses a different model derived from adjacent luma and chroma samples. The derived models are then used to generate blocks of samples for the chroma PB from the collocated luma samples.

[0089] When the prediction mode of the CB is indicated to be inter-prediction in the bitstream 133, the motion compensation module 434 uses the motion vector and the reference frame index to select and filter a block of samples 498 from the frame buffer 496 to generate a block of inter-predicted samples represented as 438. The block of samples 498 is obtained from a previously decoded frame stored in the frame buffer 496. In the case of bi-prediction, two sample blocks are generated and blended together to generate samples for the decoded PB 452. The frame buffer 496 receives filtered block data 492 from the in-loop filtering module 488. Similar to the in-loop filtering module 368 of the video encoder 114, the in-loop filtering module 488 applies any of the DBF, ALF, and SAO filtering operations. In general, the motion vector is applied to both the luma and chroma channels, but the filtering process for the sub-sampled interpolated luma and chroma channels is different. The frame buffer 496 outputs the decoded video samples 135.

[0090] 5 is a schematic block diagram illustrating a set 500 of available divisions or splits of a region into one or more sub-regions in a generic video coding tree structure. The divisions shown in set 500 are available to a block partitioner 310 of the encoder 114 to split each CTU into one or more CUs or CBs according to a coding tree, as determined by Lagrangian optimization, as described with reference to FIG.

[0091] Although set 500 shows only square regions being partitioned into other, possibly non-square sub-regions, it should be understood that while diagram 500 illustrates potential partitions, the containing region need not be square. If the containing region is non-square, the dimensions of the blocks resulting from the partition are scaled according to the aspect ratio of the containing block. Once a region is no longer partitioned, i.e., at a leaf node of the coding tree, a CU occupies the region. A particular sub-partition of a CTU by block partitioner 310 into one or more CUs is referred to as the "coding tree" of the CTU.

[0092] The process of subdividing a region into subregions must end when the resulting subregion reaches a minimum CU size. In addition to constraining the CU to prohibit block regions smaller than a certain minimum size, e.g., 16 samples, the CU is constrained to have a minimum width or height of 4. Other minimum values ​​are possible for both width and height, or for width or height. The process of subdivision can also end before the deepest level of decomposition, resulting in a CU larger than the minimum CU size. It is possible that no division occurs, resulting in a single CU occupying the entirety of a CTU. A single CU occupying the entirety of a CTU is the maximum available coding unit size. The use of a subsampled chroma format, such as 4:2:0, allows the configuration of the video encoder 114 and the video decoder 134 to end the division of regions in the chroma channels earlier than in the luma channel.

[0093] At the leaf nodes of the coding tree, there are CUs with no further subdivision. For example, leaf node 510 contains one CU. At the non-leaf nodes of the coding tree, there are divisions into two or more further nodes, each of which may be a leaf node forming one CU, or a non-leaf node containing further divisions into smaller regions. At each leaf node of the coding tree, there is one coding block for each color channel. A division that ends at the same depth for both luma and chroma results in three juxtaposed CBs. A division that ends at a luma depth deeper than chroma results in multiple luma CBs juxtaposed with the chroma channel CBs.

[0094] The quadtree partition 512 divides the inclusion region into four equally sized regions, as shown in Figure 5. Compared to HEVC, Generic Video Coding (VVC) achieves further flexibility by adding a horizontal bisection 514 and a vertical bisection 516. Each of the partitions 514 and 516 divides the inclusion region into two equally sized regions. The partitions are along a horizontal boundary (514) or a vertical boundary (516) within the containing block.

[0095] Further flexibility in generic video coding is achieved by adding a horizontal 3-partition 518 and a vertical 3-partition 520. The 3-partitions 518 and 520 divide the block into three regions bounded either horizontally (518) or vertically (520) along ¼ and ¾ of the width or height of the containing region. The combination of quad-trees, bin-trees, and ternary trees is called a "QTBTTT." The root of the tree contains zero or more quad-tree partitions (the "QT" section of the tree). Once the QT section ends, zero or more bins or ternary partitions occur (the "multi-tree" or "MT" section of the tree), eventually terminating in a leaf node CB or CU of the tree. If the tree describes all color channels, the tree leaf node is a CU. If the tree describes the luma or chroma channels, the tree leaf node is a CB.

[0096] Compared to HEVC, which supports only quad-trees and therefore only square blocks, QTBTTT results in many more possible CU sizes, especially considering possible recursive application of binary and / or ternary tree partitions. The likelihood of unusual (non-square) block sizes can be reduced by constraining the partitioning options to exclude partitions that result in block widths or heights that are either less than 4 samples or are not multiples of 4 samples. In general, this constraint applies when considering luma samples. However, in the described configuration, the constraint can be applied separately to blocks for chroma channels. Application of the constraint to partitioning options for chroma channels may result in different minimum block sizes for luma and chroma, for example, when the frame data is in 4:2:0 chroma format or 4:2:2 chroma format. Each partition generates sub-regions whose side dimensions remain unchanged, are halved, or are quartered with respect to this containing region. And since the CTU size is a power of 2, the side dimensions of all CUs are also powers of 2.

[0097] 6 is a schematic flow diagram illustrating a data flow 600 of the QTBTTT (or "coding tree") structure used in generic video coding. The QTBTTT structure is used for each CTU to define a partitioning of the CTU into one or more CUs. The QTBTTT structure for each CTU is determined by the block partitioner 310 in the video encoder 114 and encoded into the bitstream 115 or decoded from the bitstream 133 by the entropy decoder 420 in the video decoder 134. The data flow 600 further characterizes the permissible combinations available to the block partitioner 310 for partitioning the CTU into one or more CUs according to the partitioning shown in FIG. 5.

[0098] Starting from the top level of the hierarchy, i.e., CTU, zero or more quadtree splits are first performed. Specifically, a quadtree (QT) split decision 610 is made by the block partitioner 310. A decision at 610 returning a "1" symbol indicates a decision to split the current node into four subnodes according to the quadtree split 512. This results in four new nodes, such as 620, being generated, and for each new node, a return is made to the QT split decision 610. Each new node is considered in raster (or Z-scan) order. Alternatively, if the QT split decision 610 indicates that no further splits should be performed (returning a "0" symbol), the quadtree split stops and a multitree (MT) split is then considered.

[0099] First, an MT split decision 612 is made by the block partitioner 310. At 612, a decision to perform MT splitting is indicated. Returning a "0" symbol at decision 612 indicates that no further splitting of the node into subnodes is performed. If no further splitting of the node is performed, the node is a leaf node of the coding tree and corresponds to a CU. The leaf node is output at 622. Alternatively, if MT split 612 indicates a decision to perform MT splitting (returning a "1" symbol), the block partitioner 310 proceeds to direction decision 614.

[0100] The orientation decision 614 indicates the orientation of the MT partition as either horizontal ("H" or "0") or vertical ("V" or "1"). The block partitioner 310 proceeds to decision 616 if decision 614 returns a "0" indicating a horizontal orientation. The block partitioner 310 proceeds to decision 618 if decision 614 returns a "1" indicating a vertical orientation.

[0101] In each of decisions 616 and 618, the number of partitions for the MT split is indicated as either two (bipartite or "BT" node) or three (tripartite or "TT") in the BT / TT split. That is, the BT / TT split decision 616 is made by the block partitioner 310 when the indicated direction from 614 is horizontal, and the BT / TT split decision 618 is made by the block partitioner 310 when the indicated direction from 614 is vertical.

[0102] The BT / TT split decision 616 indicates whether the horizontal split is a 2-way split 514, indicated by returning a "0", or a 3-way split 518, indicated by returning a "1". If the BT / TT split decision 616 indicates a 2-way split, then in an HBT CTU node generation step 625, two nodes are generated by the block partitioner 310 according to the horizontal 2-way split 514. If the BT / TT split 616 indicates a 3-way split, then in an HTT CTU node generation step 626, three nodes are generated by the block partitioner 310 according to the horizontal 3-way split 518.

[0103] The BT / TT split decision 618 indicates whether the vertical split is a 2-way split 516, indicated by returning a "0", or a 3-way split 520, indicated by returning a "1". If the BT / TT split 618 indicates a 2-way split, then in a VBT CTU node generation step 627, two nodes are generated by the block partitioner 310 according to the vertical 2-way split 516. If the BT / TT split 618 indicates a 3-way split, then in a VTT CTU node generation step 628, three nodes are generated by the block partitioner 310 according to the vertical 3-way split 520. For each node resulting from steps 625-628, the recursion of the data flow 600 back to the MT split decision 612 is applied in a left-to-right or top-to-bottom order, depending on the direction 614. As a result, the 2-way and 3-way tree splits can be applied to generate CUs with various sizes.

[0104] 7A and 7B provide an example division 700 of a CTU 710 into several CUs or CBs. An example of a CU 712 is shown in FIG 7A. FIG 7A shows the spatial arrangement of CUs in the CTU 710. The example division 700 is also shown as a coding tree 720 in FIG 7B.

[0105] At each non-leaf node in the CTU 710 in FIG. 7A, e.g., nodes 714, 716, and 718, the contained nodes (which may be further split or may be CUs) are scanned or traversed in “Z-order” to create a list of nodes, represented as columns in the coding tree 720. In the case of a quadtree split, the Z-order scan is from top-left to right followed by bottom-left to right. In the case of horizontal and vertical splits, the Z-order scan (traversal) simplifies to a top-down scan and a left-to-right scan, respectively. The coding tree 720 in FIG. 7B lists all nodes and CUs according to the applied scan order. Each split generates a list of 2, 3, or 4 new nodes at the next level of the tree until a leaf node (CU) is reached.

[0106] Once an image has been decomposed into CTUs and then further decomposed into CUs by block partitioner 310, and the CUs are used to generate respective residual blocks (324), as described with reference to Figure 3, the residual blocks are forward transformed by video encoder 114. An equivalent inverse transform process is performed in video decoder 134 to obtain the TBs from bitstream 133.

[0107] In the video encoder 114, the quantized coefficients 336 may be rearranged into a one-dimensional list by performing a two-level diagonal backward scan. Similarly, in the video decoder 134, the quantized coefficients 424 may be rearranged from a one-dimensional list into a two-dimensional collection of sub-blocks by the same two-level diagonal backward scan.

[0108] FIG. 8A illustrates a two-level back diagonal scan 810 of an exemplary 8×8 TB 800. The scan 810 is shown proceeding from the bottom right residual coefficient position of the TB 800 back to the top left (DC) residual coefficient position of the TB 800. The path of the scan 810 proceeds from one subblock to the next in a 4×4 region known as a subblock. For a TB with a width or height of 2, subblock sizes of 2×2, 2×8, or 8×2 are available. The scan within a particular subblock is performed according to a “coded subblock flag” or the subblock is skipped. When a scan of a subblock is skipped, all residual coefficients within the subblock are inferred to have a value of zero. Although the scan 810 is shown starting from the bottom right residual coefficient position of the TB 800, for a given set of residual coefficients, the scan starts from the position of the “last significant coefficient”, which is the “last” coefficient when the order of the coefficients is considered to proceed from the DC coefficient instead of the scan order.

[0109] 8B illustrates an alternative two-level forward diagonal scan 860 of an exemplary 8×8 TB 850 used when TSRC processing is selected. When TSRC processing is used in the video encoder 114, the quantized coefficients 336 are rearranged by the scan 860 into a one-dimensional list. Similarly, when TSRC processing is used for the current TB in the video decoder 134, the quantized coefficients 424 are rearranged by the scan 860 from a one-dimensional list into a two-dimensional collection of sub-blocks. The scan 860 is shown proceeding from the top-left (DC) residual coefficient position of the TB 850 to the bottom-right residual coefficient position of the TB 850. Unlike the scan 810, the scan 860 does not end at the “last significant coefficient.”

[0110] 8A and 8B show scan patterns typically used in VVC. The example described herein uses scan pattern 810 to code the residual coefficients transformed by module 326, and scan pattern 860 is used for transform-skipped transform blocks. However, in some implementations, other scan patterns may be used.

[0111] As mentioned above, the transform coefficients 332 are the same as the residual coefficients 324 when the transform skip mode is used. Therefore, the transform coefficients 332 may be referred to as residual coefficients regardless of whether the transform skip mode is selected or not. If lossless coding is desired, the video encoder 114 selects the transform skip for the current TB and signals a transform skip flag with a value of "TRUE" to the bitstream 133. The residual coefficients 332 associated with the current TB are coded into the bitstream 133. Two residual coding processes are available: a "regular residual coding" (RRC) process, and a "transform skip residual coding" (TSRC) process. In normal operation of the video encoder 114, if the transform skip is selected (the transform skip flag has a value of "TRUE"), the TSRC process is selected, otherwise (the transform skip flag has a value of "FALSE"), the RRC process is selected. However, it is typically not desirable for the coding of the residual coefficients 332 to be exclusively handled by the TSRC in the case of lossless coding.

[0112] In one configuration of the video encoder 114, the TSRC invalid flag is signaled in the bitstream 133. The TSRC invalid flag may be signaled at a relatively high level, such as once per sequence or once per picture, so that the relative cost of signaling the TSRC invalid flag is low. High level syntax elements are typically grouped into parameter sets, such as a "sequence parameter set" (SPS) for sequence level flags and a "picture parameter set" (PPS) for parameter level flags. The TSRC invalid flag may be set to "TRUE" if the video data 113 belongs to a class that is deemed unsuitable (in terms of coding loss and feature reproduction) for encoding with TSRC processing. An example of video data that is unsuitable for encoding with TSRC is natural scene content. The TSRC invalid flag may be set to "FALSE" when the video data 113 belongs to a class that is deemed to be successfully encoded with TSRC processing. Video data that is suitable for encoding with TSRC processing includes artificial screen content.

[0113] If the video encoder 114 selects transform skip for the current TB and the TSRC disabled flag is set to "TRUE," the residual coefficients 332 are encoded into the bitstream 133 using the RRC process. Similarly, if the video decoder 134 determines that transform skip is used for the current TB and the TSRC disabled flag is set to "TRUE," the residual coefficients 432 are decoded from the bitstream 133 using the RRC process.

[0114] 9 illustrates a method 900 for encoding a transform block of residual coefficients 332 using an RRC process. The method 900 may be implemented by an apparatus such as a configured FPGA, ASIC, or ASSP. Additionally, the method 900 may be performed by the video encoder 114 under execution of the processor 205. As such, the method 900 may be implemented as a module of software 233 stored in a computer-readable storage medium and / or memory 206.

[0115] The method 900 is performed in some configurations by the video encoder 114 upon receiving the residual coefficients 332 at the quantizer 334 and then at the entropy encoder 338. The method 900 begins with a coefficient quantization step 910.

[0116] In the coefficient quantization step 910, the step 910 calls the method 1100 described below in relation to FIG. 11. The method 1100 may be implemented by an apparatus such as a configured FPGA, ASIC, or ASSP. Furthermore, the method 1100 may be executed by the video encoder 114 under execution of the processor 205. Thus, the method 1100 may be implemented as a module of the software 233 stored on a computer-readable storage medium and / or in the memory 206. The method 1100 quantizes the residual coefficients 332 to generate the quantized coefficients 336. The method 900 proceeds from the step 910 to a last position encoding step 920 under the control of the processor 205.

[0117] In a last position encoding step 920, the video encoder 114 finds the position of the last significant coefficient in the quantized coefficients 336 for a transform block of residual coefficients 332. The last significant coefficient is determined relative to the forward direction of an appropriate scan pattern, e.g., the direction of a two-level forward diagonal scan 860. A quantized coefficient is significant if the coefficient has any value other than zero. The position of the last significant coefficient is written to the bitstream 133. The method 900 proceeds from step 920 to a state initialization step 930 under the control of the processor 205.

[0118] In a state initialization step 930, the quantizer state Qstate is set to a value of zero. Further, the sub-block containing the last significant coefficient is selected in step 930. From step 930, the method 900 proceeds under the control of the processor 205 to a coded sub-block flag determination step 940.

[0119] The descriptions herein refer to some flags being "TRUE" or "FALSE." A setting of "TRUE" means that the flag value indicates that the requirement has been met or the mode has been selected. A setting of "FALSE" means that the flag value indicates that the requirement has not been met or the mode has not been selected.

[0120] In a coded sub-block flag determination step 940, video encoder 114 determines and sets a coded sub-block flag. If the currently selected sub-block was the first sub-block selected in state initialization step 930, the coded sub-block flag is set to "TRUE" but is not coded into bitstream 133. If the currently selected sub-block is identified as the last sub-block, as described below in connection with last sub-block test 970, the coded sub-block flag is set to "TRUE" but is not coded into bitstream 133.

[0121] Otherwise, video encoder 114 sets the coded sub-block flag to (i) "TRUE" if there is at least one significant coefficient among the 4x4 quantized coefficients belonging to the selected sub-block, or (ii) "FALSE" if there are no significant coefficients, and encodes the coded sub-block flag into bitstream 133. Method 900 proceeds from step 940, under control of processor 205, to a coded sub-block flag test step 950.

[0122] In a coded sub-block flag test step 950, the method 900 determines the value of the coded sub-block flag. If the coded sub-block flag is set to "TRUE", the method 900 proceeds to a sub-block coding step 960. Otherwise, if the coded sub-block flag is set to "FALSE", the method 900 proceeds to a last sub-block test step 970.

[0123] In a sub-block encoding step 960, the entropy encoder 338 encodes the quantized coefficients in the selected sub-block into the bitstream 133. Step 960 invokes a method 1300, described below in connection with Figure 13. From step 960, the method 900 proceeds under processor control to a last sub-block test 970.

[0124] In a last sub-block test 970, the method 900 operates to determine whether the selected sub-block is the last sub-block in the current transform block. If the currently selected sub-block is the top left sub-block of the transform block, step 900 returns "YES" and the method 900 ends. Otherwise, if the currently selected sub-block is not the top left sub-block of the transform block, step 970 returns "NO" and the method 900 proceeds to step 980 to select the next sub-block.

[0125] In select next sub-block step 980, the next sub-block in the transform block is selected. The next sub-block in the backward diagonal scan order 810 is selected. From step 980, the method 900 proceeds to step 940, where the coded sub-block flag of the selected sub-block is determined.

[0126] 10 illustrates a method 1000 for decoding a transform block of residual coefficients 432 by an RRC process. The method 1000 may be implemented by an apparatus such as a configured FPGA, ASIC, or ASSP. Additionally, the method 1000 may be executed by the video decoder 134 under execution of the processor 205. In this manner, the method 1000 may be implemented as a computer-readable storage medium and / or a module of software 233 stored in the memory 206.

[0127] The method 1000 is implemented in several configurations by the video encoder 134 in the entropy decoder 420 upon receipt of the bitstream 133, and in the inverse quantizer module 428. The method 1000 begins with a last position decoding step 1010.

[0128] In a last position decoding step 1010, the last significant coefficient position of the transform block of residual coefficients 432 is decoded from the bitstream 133. From step 1010, the method 1000 proceeds under the control of the processor 205 to a state initialization step 1020.

[0129] In a state initialization step 1020, the video decoder 134 initializes a quantizer state Qstate to a value of 0. Further, in step 1020, the sub-block containing the last significant coefficient position is selected. The method 1000 proceeds from step 1020 under the control of the processor 205 to a coded sub-block flag determination step 1030.

[0130] In coded sub-block flag determination step 1030, video decoder 134 determines the coded sub-block flag. If the currently selected sub-block is the first sub-block selected in state initialization step 1020, the coded sub-block flag is set to "TRUE" (i.e., the coded sub-block flag is inferred to be "TRUE"). If the currently selected sub-block is identified as the last sub-block as described below in last sub-block test 1060, the coded sub-block flag is inferred as "TRUE". Otherwise, video decoder 134 decodes the coded sub-block flag from bitstream 133. Method 1000 proceeds from step 1030 to coded sub-block flag test 1040 under control of processor 205.

[0131] In coded sub-block flag test 1040, method 1000 tests the value of the coded sub-block flag determined in step 1030. If the coded sub-block flag is determined in step 1040 to have a value of "TRUE", method 1000 proceeds to sub-block decoding step 1050. Otherwise, if the coded sub-block flag is determined in step 1040 to have a value of "FALSE", all of the quantized coefficients in the currently selected sub-block are assigned a value of zero and method 1000 proceeds to last sub-block test 1060.

[0132] In a sub-block decoding step 1050, the entropy decoder 420 decodes the quantized coefficients of the selected sub-block from the bitstream 133. Step 1050 invokes a method 1400, described below in connection with Figure 14. The method 1000 proceeds under control of the processor 205 to a last sub-block test 1060.

[0133] At last sub-block test 1060, if the currently selected sub-block is the top left sub-block of the transform block, step 1060 returns "YES" and method 1000 proceeds to coefficient scale step 1080. Otherwise, step 1060 returns "NO" and method 1000 proceeds to step 1070 to select the next sub-block.

[0134] In select next sub-block step 1070, the next sub-block in the backward diagonal scan order 810 is selected. From step 1070, the method 1000, under the control of the processor 205, proceeds to the coded sub-block flag determination step 1030.

[0135] In a coefficient scale step 1080, the inverse quantizer module 428 applies scaling to the quantized coefficients 424 to generate reconstructed residual coefficients 432. Sub-blocks are decoded by reconstructing the residual coefficients of the sub-blocks using the decoded sign bits. Step 1080 invokes a method 1200, described below in conjunction with FIG. 12. The method 1000 ends with the execution of step 1080.

[0136] 11 shows a method 1100 for quantizing the residual coefficients 332 of a transform block to generate quantized coefficients 336. The method 1100 is performed on the TB in step 910 of the method 900. The method 1100 begins with a DQ test 1110.

[0137] In the DQ test 1110, the video encoder 114 determines whether dependent quantization is used to quantize the residual coefficients 332. The video encoder 114 checks the value of the enabled dependent quantization flag, which is signaled as high-level syntax in the bitstream 133. The enabled dependent quantization flag determines whether dependent quantization is allowed within the flag. For example, the sequence level dependent quantization flag determines whether dependent quantization is allowed when encoding an entire video sequence. The picture level dependent quantization flag determines whether dependent quantization is allowed when encoding the current picture and takes precedence over the value of the sequence level dependent quantization flag. If the enabled dependent quantization flag is "FALSE", step 1110 returns "NO" and the method 1100 proceeds to the scalar quantization step 1120.

[0138] In one configuration of DQ test 1110, if the valid dependent quantization flag is "TRUE", the video encoder 114 also checks the value of the transform skip flag for the current TB. If the valid dependent quantization flag is "TRUE" and the transform skip flag is "TRUE", step 1110 returns "NO" and the method 1100 proceeds to the scalar quantization step 1120. Otherwise, if the valid dependent quantization flag is "TRUE" and the transform skip flag is "FALSE", step 1110 returns "YES" and the method 1100 proceeds to the dependent quantization step 1130.

[0139] In another configuration of DQ test 1110, if the valid dependent quantization flag is "TRUE", the video encoder 114 also checks the value of the TSRC disabled flag. If the valid dependent quantization flag is "TRUE" and the TSRC disabled flag is "TRUE", step 1110 returns "NO" and method 1100 proceeds to scalar quantization step 1120. Otherwise, if the valid dependent quantization flag is "TRUE" and the TSRC disabled flag is "FALSE", step 1110 returns "YES" and method 1100 proceeds to dependent quantization step 1130.

[0140] In yet another configuration of the DQ test 1110, if the enable dependent quantization flag is "TRUE", the video encoder 114 also checks the value of the quantization parameter (QP) of the current TB. The QP indicates the degree of quantization applied to the residual coefficients 332. The QP is the initial QP i and an offset that depends on the bit depth BD of the video encoder 114, QP=QP i +6*(BD-8). For example, QP i If is 4 and bit depth is 8, then QP is determined to be 4. iIf the QP is -8 and the bit depth is 10, the QP is determined as 4. Typically, a QP of 4 indicates that the residual coefficients are not quantized, and thus lossless operation is possible. However, higher values ​​of QP can still achieve lossless operation. For example, if the video data 113 was originally captured at a bit depth of 8, but is provided to the video encoder 114 at a higher bit depth, lossless operation is possible with a higher QP. For example, if the video data 113 was captured at a bit depth of 8, but is provided to the video encoder 114 at a bit depth of 10, lossless operation is possible with a QP of 4, 10, or 16. The QP at which lossless operation is possible may be indicated by a minimum QP for the transform skip block, which is signaled in the high-level syntax parameter set. If the enable dependent quantization flag is "TRUE" and the QP is 4 (or any value indicating lossless operation), step 1110 returns "NO" and the method 1100 proceeds to the scalar quantization step 1120. Otherwise, if the enable dependent quantization flag is “TRUE” and QP is not 4 (or a similar value indicating lossless operation), then step 1110 returns “YES” and method 1100 proceeds to dependent quantization step 1130 . In a scalar quantization step 1120, the residual coefficients 332 are represented as r[n]. The quantized coefficients q[n] are then generated by quantizing the residual coefficients r[n] according to the following equation (1): q[n]=(k*r[n]+offset)>>qbits (1) In equation (1), k is a scaling factor, qbits is a coarse quantization factor, and offset controls the placement of the quantization thresholds. k, qbits, and offset are determined based on the values ​​of the quantization parameters of the current TB. For example, if QP is 4, then k=1, qbits=0, and offset=0. Then, if QP is 4, then q[n]=r[n], and no loss occurs in the scalar quantization step. Method 1100 proceeds from step 1120 to SBH test 1140 under the control of processor 205.

[0141] In the dependent quantization step 1130, each of the residual coefficients r[n] may be quantized by one of a selection of multiple scalar quantizers. For the same QP, the scalar quantizers have the same quantization division number, but the quantization thresholds are offset relative to each other. The scalar quantizer for a particular residual coefficient r[n] depends on the current quantizer state Qstate, which is updated for each coefficient, and on the parity (least significant bit) of the resulting q[n]. Because of the dependency on the previous state, the optimal quantization result is not determined for each coefficient. One efficient way to determine the optimal quantization result is by building a "trellis" of possible quantization states at each coefficient position. The optimal quantization result can be found equivalently by finding the best path through the trellis. The optimal trellis path can be determined by applying the Viterbi algorithm. The method 1100 ends with the execution of step 1130.

[0142] In SBH test 1140, the video encoder 114 determines whether sign bit hiding is used to modify the quantized coefficients q[n] before encoding the coefficients of the TB. The video encoder 114 checks the value of the valid sign bit hiding flag. The valid sign bit hiding flag is signaled as a high level syntax in the bitstream 133. For example, the valid sign bit hiding flag can be signaled in the picture header. If the valid dependent quantization flag is "TRUE", the valid sign bit hiding flag is implicitly "FALSE". If the valid sign bit hiding flag is "FALSE", step 1140 returns "NO" and the method 1100 ends.

[0143] In one configuration of the SBH test 1140, the decision depends on the value of the valid sign bit hiding flag and the value of the transform skip flag for the current TB. If the valid sign bit hiding flag is "TRUE", the video encoder 114 also checks the value of the transform skip flag for the current TB. If the valid sign bit hiding flag is "TRUE" and the transform skip flag is "TRUE", step 1140 returns "NO" and the method 1100 ends. Otherwise, if the valid sign bit hiding flag is "TRUE" and the transform skip flag is "FALSE", step 1140 returns "YES" and the method 1100 proceeds to the parity adjustment step 1150.

[0144] In another configuration of the SBH test 1140, the decision depends on the value of the valid code bit hiding flag and the value of the TSRC invalid flag. If the valid code bit hiding flag is "TRUE", the video encoder 114 also checks the value of the TSRC invalid flag. If the valid code bit hiding flag is "TRUE" and the TSRC invalid flag is "TRUE", step 1140 returns "NO" and the method 1100 ends. Otherwise, if the valid code bit hiding flag is "TRUE" and the TSRC invalid flag is "FALSE", step 1140 returns "YES" and the method 1100 proceeds to the parity adjustment step 1150.

[0145] In yet another configuration of the SBH test 1140, the decision depends on the value of a significant sign bit hiding flag and the value of a quantization parameter (QP) for the current TB. If the significant sign bit hiding flag is "TRUE", the video encoder 114 also checks the value of QP for the current TB. If the significant sign bit hiding flag is "TRUE" and the QP is 4 (or any value indicating lossless operation), step 1140 returns "NO" and the method 1100 ends. Otherwise, if the significant sign bit hiding flag is "TRUE" and the QP is not 4 (or a similar value indicating lossless operation), step 1140 returns "YES" and the method 1100 proceeds to the parity adjustment step 1150.

[0146] In a parity adjustment step 1150, the video encoder 114 checks the positions of the first and last significant coefficient for each of the subblocks in the current TB. If the difference between the first significant position and the last significant position of the subblock is greater than a threshold (typically 3), then sign bit hiding is used for that subblock. For each subblock for which sign bit hiding is used, the video encoder 114 checks the sign of the first significant coefficient in the subblock and adjusts the parity of the coefficients in the subblock accordingly. The parity of the coefficient is zero if the coefficient is even and one if the coefficient is odd. The sum of the parity of the coefficients is zero if the number of odd coefficients is odd and one if the number of odd coefficients is even. If the sign of the first significant coefficient in the subblock is positive, then the coefficients in the subblock are adjusted to have a parity sum of zero. If the sign of the first significant coefficient in the subblock is negative, then the coefficients in the subblock are adjusted to have a parity sum of one. The method 1100 ends after performing step 1150.

[0147] 12 shows a method 1200 for applying scaling to the quantized coefficients 424 and generating reconstructed residual coefficients 432. The method 1200 may be implemented by an apparatus such as a configured FPGA, ASIC, or ASSP. Furthermore, the method 1200 may be executed by the video decoder 134 under execution of the processor 205. In this manner, the method 1200 may be implemented as a module of the software 233 stored in the computer-readable storage medium and / or memory 206. The method 1200 is implemented in step 1080 of the method 1000. The method 1200 begins with a DQ test 1210.

[0148] In DQ test 1210, video decoder 134 determines whether dependent quantization is used to dequantize quantized coefficients 424. Video decoder 134 checks the value of the enable dependent quantization flag, which may be decoded from bitstream 133 or inferred based on the values ​​of other high-level syntax flags. If the enable dependent quantization flag is "FALSE," step 1210 returns "NO" and method 1200 proceeds to inverse scalar quantization step 1220.

[0149] In one configuration of DQ test 1210, if the valid dependent quantization flag is "TRUE", the video decoder 134 also checks the value of the transform skip flag for the current TB. If the valid dependent quantization flag is "TRUE" and the transform skip flag is "TRUE", step 1210 returns "NO" and the method 1100 proceeds to the inverse scalar quantization step 1220. Otherwise, if the valid dependent quantization flag is "TRUE" and the transform skip flag is "FALSE", step 1210 returns "YES" and the method 1100 proceeds to the inverse dependent quantization step 1230.

[0150] In another configuration of DQ test 1210, if the valid dependent quantization flag is "TRUE", the video decoder 134 also checks the value of the TSRC disabled flag. If the valid dependent quantization flag is "TRUE" and the TSRC disabled flag is "TRUE", step 1210 returns "NO" and the method 1100 proceeds to the inverse scalar quantization step 1220. Otherwise, if the valid dependent quantization flag is "TRUE" and the TSRC disabled flag is "FALSE", step 1210 returns "YES" and the method 1100 proceeds to the inverse dependent quantization step 1230.

[0151] In yet another configuration of the DQ test 1210, if the enabled dependent quantization flag is "TRUE", the video decoder 134 also checks the value of the quantization parameter (QP) of the current TB. If the enabled dependent quantization flag is "TRUE" and the QP is 4 (or any value indicating lossless operation), step 1210 returns "NO" and the method 1100 proceeds to the inverse scalar quantization step 1220. Otherwise, if the enabled dependent quantization flag is "TRUE" and the QP is not 4 (or a similar value indicating lossless operation), step 1210 returns "YES" and the method 1100 proceeds to the inverse dependent quantization step 1230.

[0152] In an inverse scalar quantization step 1220, the video decoder 134 scales the quantized coefficients 424 to generate reconstructed residual coefficients 432. The quantized coefficients 424 are denoted as q[n]. The reconstructed residual coefficients r[n] are generated by scaling the quantized coefficients q[n] according to the following equation (2):

[0153]

number

[0154] In equation (2), s is a scaling factor that is determined based on the value of QP for the current TB. For example, if QP is 4, then s=1 and r[n]=q[n]. The method 1200 ends with execution of step 1220.

[0155] In an inverse dependent quantization step 1230, the video decoder 134 applies inverse dependent quantization to the quantized coefficients 424 to generate reconstructed residual coefficients 432. The quantizer state Qstate is initially reset to zero. The quantized coefficients 424 are denoted as q[n]. Each coefficient position n is visited in a backward diagonal scan order 810, and each reconstructed residual coefficient r[n] is calculated according to equation (3).

[0156]

number

[0157] In equation (3), s is a scaling factor determined based on the value of QP for the current TB.

[0158] After each reconstructed residual coefficient r[n] is calculated, the quantizer states are updated based on the parity of q[n] according to Table 1.

[0159] [Table 1]

[0160] The method 1200 concludes with execution of step 1230 .

[0161] To take advantage of the statistical properties of the quantized coefficients 336, the quantized coefficients are binarized by the video encoder 114 (typically by the entropy encoder 338) into several syntax elements before encoding. For example, since the quantized coefficients 336 often have a value of zero, one syntax element is a significance flag, which is set to "FALSE" for quantized coefficients having a value of zero. When the significance flag is set to "FALSE", no further syntax elements of the associated quantized coefficient are signaled. The significance flag may be coded into the bitstream 133 by using a context-adaptive binary arithmetic coding (CABAC) entropy coder.

[0162] Although the CABAC encoder encodes the context-coded syntax elements relatively efficiently, it is generally desirable to limit the number of context-coded syntax elements to minimize the computational requirements and costs for hardware implementation. Thus, after the quantized coefficients 336 are binarized by the entropy encoder 338 into several syntax elements, some syntax elements are context-coded into the bitstream 133, and other syntax elements are bypass-coded into the bitstream 133. The total number of context-coded syntax element bins is limited per transform block. In the VVC standard, the limit is set to 1.75 bins per sample. For example, for an 8×8 transform block of 64 samples, the context-coded bin budget is set to 112 bins. In the process of encoding the TB into the bitstream 133, whenever a syntax element is context-coded, the remaining context-coded bin budget is tracked and decremented. When the remaining context-coded bin budget is exhausted, the remaining quantized coefficients and associated syntax elements must be bypass-coded.

[0163] 13 shows a method 1300 for encoding the quantized coefficients (336) of a currently selected sub-block into the bitstream 133. The method 1300 is performed in step 960 of the method 900. The method 1300 may be performed by an apparatus such as a configured FPGA, ASIC, or ASSP. Furthermore, the method 1300 may be performed by the video encoder 114 under execution of the processor 205. Thus, the method 1300 may be implemented as a module of the software 233 stored in the computer-readable storage medium and / or the memory 206. The method 1300 begins with step 1310 of selecting an initial coefficient.

[0164] In the select first coefficient step 1310, the method 1300 selects a quantized coefficient for the current sub-block. If the current sub-block contains the last significant coefficient position, the current selected coefficient is set to the last significant coefficient. Otherwise, if the current sub-block does not contain the last significant coefficient position, the currently selected coefficient is set to the bottom right coefficient of the current sub-block. The method 1300 proceeds to a context coding usage check 1320.

[0165] In a context coding usage check 1320, the video encoder 114 checks whether the remaining context coding bin budget is greater than or equal to 4. If the remaining context coding bin budget is greater than or equal to 4, step 1320 returns "YES" and the method 1300 proceeds to a context coding syntax element encoding step 1330. Otherwise, if the current context coding bin budget is less than 4, step 1320 returns "NO" and the method 1300 proceeds to a remainder pass encoding step 1370.

[0166] In a context coding syntax element encoding step 1330, the video encoder 114 may encode a number of syntax elements into the bitstream 133 using a CABAC encoder, including a potentially significant flag, a flag greater than one, a parity flag, and a flag greater than three. Each bin associated with a syntax element is encoded by the CABAC encoder using a "context model." The context model for each bin may be selected depending on the current value of the quantizer state Qstate. Additionally, whenever a context coding bin is encoded into the bitstream 133 by the CABAC encoder, the remaining context coding bin budget is reduced by one in step 1330.

[0167] In step 1330, if the current coefficient is the last significant coefficient, the significance flag is set to "TRUE" but is not coded into the bitstream 133. If the current selected sub-block is not the first or last sub-block in the backward scan order 810, and the current selected coefficient is the last coefficient as described below in last coefficient check 1350, and all significance flags for previous coefficients in the current selected sub-block were "FALSE", the significance flag is set to "TRUE". The significance flag is not coded into the bitstream 133. If the current coefficient has a magnitude of zero, in step 1330, the significance flag is set to "FALSE" and is context coded into the bitstream 133. Otherwise, the significance flag is set to "TRUE" and is context coded into the bitstream 133 in step 1330.

[0168] If the current coefficient has a magnitude of 1, then the greater than 1 flag is set to "FALSE" and is context coded into the bitstream 133 in step 1330. Otherwise, the greater than 1 flag is set to "TRUE" and is context coded into the bitstream 133.

[0169] If the current coefficient has a magnitude of at least 2, then the parity flag is set to "FALSE" if the current coefficient is even and set to "TRUE" if the current coefficient is odd. The parity flag is context coded into the bitstream 133 in step 1330. If the current coefficient has a magnitude greater than 3, then the greater than 3 flag is set to "TRUE" and context coded into the bitstream 133 in step 1330. Otherwise, if the current coefficient has a magnitude of 2 or 3, then the greater than 3 flag is set to "FALSE" and context coded into the bitstream 133.

[0170] Method 1300 proceeds from step 1330 to a DQ test 1340 under control of processor 205. Depending on the coefficient selected in 1310, method 1300 proceeds to step 1340 after setting (or possibly encoding) the significance flag. Otherwise, method 1300 proceeds to step 1340 after encoding the last appropriate one of the greater than 1 flag, the parity flag, and the greater than 3 flag.

[0171] DQ test 1340 uses the same conditions checked in DQ test 1110 to determine whether step 1340 returns "YES" or "NO." If step 1340 returns "YES," method 1300 proceeds to update Qstate step 1345. Otherwise, if step 1340 returns "NO," method 1300 proceeds to last coefficient check 1350.

[0172] In an update Qstate step 1345, the quantizer state Qstate is updated based on the parity of the current coefficient according to Table 1. From step 1345, the method 1300 proceeds to check last coefficient 1350.

[0173] In last coefficient check 1350, the video encoder 114 checks whether the currently selected coefficient is the top-left coefficient of the currently selected sub-block. If the currently selected coefficient is the top-left coefficient of the currently selected sub-block, step 1350 returns "YES" and method 1300 proceeds to remainder pass encoding step 1370. Otherwise, if the current coefficient is not the top-left coefficient, step 1350 returns "NO" and method 1300 proceeds to step 1360 of selecting the next coefficient.

[0174] A select next coefficient step 1360 selects the next coefficient of the currently selected sub-block in the backward diagonal scan order 810. From step 1360, the method 1300 proceeds to a context coding usage check 1320.

[0175] In a remainder pass coding step 1370, the residual magnitudes of the quantized coefficients of the currently selected sub-block are binarized and bypass coded into the bitstream 133, for example by the entropy encoder 338. The quantized coefficients are coded, for example, in a backward diagonal scan order 810. If the quantized coefficients have been context coded by the CABAC coder (i.e., the context coding usage check 1320 is passed (returns "YES")), then if the flag greater than 3 is "TRUE", the quantized coefficient at scan position n has a residual magnitude r[n]. The residual magnitude is determined using equation (4). r[n]=(x[n]-4)>>1, (4)

[0176] where, equation (4), x[n] is the absolute magnitude of the quantized coefficient at scan position n. The magnitude r[n] is binarized and bypass coded into the bitstream 133. If the quantized coefficient was not context coded (context coding usage check 1320 is not passed / returns "NO"), then the absolute magnitude x[n] is binarized and bypass coded into the bitstream 133. Method 1300 proceeds from step 1370 to SBH test 1380.

[0177] SBH test 1380 uses the same conditions checked in SBH test 1140 to determine whether step 1380 returns "YES" or "NO". If SBH test 1140 returns "NO", step 1380 returns "NO" and method 1300 proceeds to step 1390 where N symbols are encoded. Otherwise, video encoder 114 checks the positions of the first and last significant coefficients of the current sub-block. If the difference between the first significant position and the last significant position is greater than 3, step 1380 returns "YES" and method 1300 proceeds to step 1395 where N-1 symbols are encoded. Otherwise, step 1380 returns "NO" and method 1300 proceeds to step 1390 where N symbols are encoded.

[0178] As described in relation to step 1140, the sign bit hiding test can depend on a number of alternative flags or settings in different implementations. If the enable sign bit hiding flag is set (has a "TRUE" value), different implementations can make a decision based on whether the transform skip flag of the TB, the TSRC disable flag, or the QP of the TB meets a threshold associated with lossless coding. Thus, step 1380 determines whether sign bit hiding is enabled or not depending on a value or flag associated with the transform block itself, or a higher level value of the TSRC disable flag. Step 1380 allows some flexibility for implementing lossless coding. An implementation that uses a value or flag associated with the transform block to determine whether sign hiding is enabled is particularly suitable for allowing flexibility in implementing lossless coding using RRC.

[0179] In a step 1390 of encoding N signs, the sign bits of any significant coefficients of the currently selected sub-block are bypass coded into the bitstream 133. The sign bits are, for example, bypass coded into the bitstream 133 based on the backward diagonal scan order 810. The method 1300 ends after performing step 1390.

[0180] In a step 1395 of encoding N-1 signs, the sign bits of the significant coefficients of the currently selected sub-block are bypass coded into the bitstream 133 based on the backward diagonal scan order 810. The sign bit associated with the first significant coefficient (the last visited in the backward diagonal scan order 810) is not coded into the bitstream 133. In other words, if there are N significant coefficients in the currently selected sub-block, then N-1 sign bits are bypass coded into the bitstream 133. The method 1300 ends with the execution of step 1395.

[0181] 14 shows a method 1400 for decoding the quantized coefficients (424) of a currently selected sub-block from the bitstream 133. The method 1400 is performed in step 1050 of the method 1000. The method 1400 may be performed by an apparatus such as a configured FPGA, ASIC, or ASSP. Furthermore, the method 1400 may be performed by the video decoder 134 under execution of the processor 205. Thus, the method 1400 may be implemented as a module of the software 233 stored in the computer-readable storage medium and / or memory 206. The method 1400 begins with step 1410 of selecting an initial coefficient.

[0182] In a select first coefficient step 1410, the method 1400 selects the first quantized coefficient of the current sub-block. If the current sub-block contains the last significant coefficient position, the currently selected coefficient is set to the last significant coefficient. Otherwise, the currently selected coefficient is set to the bottom right coefficient of the current sub-block. The method 1400 proceeds from step 1410 to a context coding usage check step 1420.

[0183] In context coding usage check 1420, video decoder 134 checks whether the remaining context coding bin budget meets a threshold, typically whether the remaining context coding bin budget for a transform block is equal to or greater than 4 bins. If the remaining budget is equal to or greater than 4, step 1420 returns "YES" and method 1400 proceeds to context coding syntax element determination step 1430. Otherwise, if the remaining CABAC budget is less than the threshold (4 bins), step 1420 returns "NO" and method 1400 proceeds to remainder path decoding step 1470.

[0184] In a context coding syntax element determination step 1430, the video decoder 134 may decode a number of context coding syntax elements from the bitstream 133 using a CABAC coder. Each bin associated with a syntax element is decoded by the CABAC coder using a "context model." The context model for each bin may be selected depending on the current value of the quantizer state Qstate. Additionally, whenever a context coding bin is decoded by the CABAC coder from the bitstream 133, the remaining context coding bin budget is decreased by one.

[0185] If the current coefficient is the last significant coefficient, the significance flag is inferred to be "TRUE" rather than being decoded from the bitstream 133. If the current selected sub-block is not the first or last sub-block in the backward scan order 810, the current selected coefficient is the last coefficient as described below in last coefficient check 1450, and all significance flags for previous coefficients in the current selected sub-block were "FALSE", the significance flag is inferred to be "TRUE". Otherwise, the significance flag is context decoded from the bitstream 133 in step 1430. If the significance flag is set to "FALSE", the currently selected coefficient is assigned a value of zero and method 1400 proceeds to DQ test 1440.

[0186] If the significance flag is set to "TRUE", then in step 1430, a flag greater than 1 is context decoded from the bitstream 133. If the flag greater than 1 is set to "FALSE", then the currently selected coefficient is assigned a magnitude of 1 and method 1400 proceeds to DQ test 1440.

[0187] If the greater than 1 flag is set to "TRUE", then the parity flag and the greater than 3 flag are context decoded from the bitstream 133. The method 1400 proceeds to a DQ test 1440. The number of flags determined in step 1430 depends on the position and value of the coefficient selected in step 1410. Proceeding from step 1430 can occur after a significance flag has been inferred or decoded, or after decoding the appropriate one of the greater than 1 flag, the parity flag, or the greater than 3 flag.

[0188] DQ test 1440 uses the same conditions checked in DQ test 1210 to determine whether step 1440 returns "YES" or "NO." If step 1440 returns "YES," method 1400 proceeds to update Qstate step 1445. Otherwise, if step 1440 returns "NO," method 1400 proceeds to last coefficient check 1450.

[0189] In update Qstate step 1445, the quantizer state Qstate is updated based on the parity of the currently selected coefficient according to Table 1. If the currently selected coefficient has a value of zero, then the parity is zero. The parity is one if the currently selected coefficient has a magnitude of one. Otherwise, if the parity flag is set to "FALSE", then the parity is zero, and if the parity flag is set to "TRUE", then the parity is one. Method 1400 proceeds from step 1445 to check last coefficient 1450.

[0190] In last coefficient check step 1450, video decoder 134 checks whether the currently selected coefficient is the top-left coefficient of the currently selected sub-block. If the currently selected coefficient is the top-left coefficient of the currently selected sub-block, step 1450 returns "YES" and method 1400 proceeds to remainder path decoding step 1470. Otherwise, if the currently selected coefficient is not the top-left coefficient, step 1450 returns "NO" and method 1400 proceeds to select the next coefficient step 1460.

[0191] A select next coefficient step 1460 selects the next coefficient of the currently selected sub-block in the backward diagonal scan order 810. From step 1460, the method 1400 proceeds to a context coding usage check 1420.

[0192] In remainder pass decoding step 1470, the remaining magnitudes of the quantized coefficients of the currently selected sub-block are bypass decoded from the bitstream 133. The quantized coefficients are processed in the backward diagonal scan order 810. If the quantized coefficient is context decoded (context coding usage check 1420 is passed or returns "YES") and a flag greater than 3 is decoded with a value of "TRUE", the remaining magnitude r[n] is bypass decoded from the bitstream 133, where n is the scan position of the quantized coefficient. The absolute magnitude x[n] of the quantized coefficient is determined as x[n]=4+p[n]+2*r[n], where p[n] has a value of zero if the parity flag is decoded as "FALSE" and p[n] has a value of one if the parity flag is decoded as "TRUE".

[0193] If the quantized coefficient is context decoded and a flag greater than 1 is decoded as "TRUE" but a flag greater than 3 is not decoded or is decoded as "FALSE", then the absolute magnitude is determined as x[n]=2+p[n]. If the quantized coefficient is not context decoded (context coding usage check 1420 is not passed, returning "NO"), then the absolute magnitude x[n] is bypass decoded from the bitstream 133. Method 1400 proceeds from step 1470 to SBH test 1480.

[0194] In SBH test 1480, the video decoder 134 determines whether sign bit hiding is used, i.e., whether one sign bit for the currently selected sub-block is inferred. The test used in step 1480 is related to the test used in step 1140 on the encoder side. The video decoder 134 checks the value of the valid sign bit hiding flag, which may be signaled as high level syntax in the bitstream 133. If the valid dependent quantization flag is "TRUE", the valid sign bit hiding flag is inferred to be "FALSE". If the valid sign bit hiding flag is "FALSE", step 1480 returns "NO" and the method 1400 proceeds to the code decoding step 1490.

[0195] The video decoder 134 checks the positions of the first and last significant coefficients of the currently selected sub-block. If the difference between the first significant position and the last significant position is less than or equal to 3, step 1480 returns "NO" and the method 1400 proceeds to the encoding / decoding step 1490.

[0196] In one configuration of the SBH test 1480, the decision depends on the value of the valid sign bit hiding flag and the value of the transform skip flag for the current TB. If the valid sign bit hiding flag is "TRUE" and the difference between the first significant position and the last significant position is greater than 3, the video decoder 134 also checks the value of the transform skip flag for the current TB. If the transform skip flag is "TRUE", step 1480 returns "NO" and the method 1400 proceeds to the code decoding step 1490. Otherwise, if the transform skip flag is "FALSE", step 1480 returns "YES" and the method 1400 proceeds to the code decoding and inferring step 1495.

[0197] In another configuration of the SBH test 1480, the decision depends on the value of the valid code bit hiding flag and the value of the TSRC invalid flag. If the valid code bit hiding flag is "TRUE" and the difference between the first significant position and the last significant position is greater than 3, the video decoder 134 also checks the value of the TSRC invalid flag. If the TSRC invalid flag is "TRUE", step 1480 returns "NO" and the method 1400 proceeds to the code decoding step 1490. Otherwise, if the TSRC invalid flag is "FALSE", step 1480 returns "YES" and the method 1400 proceeds to the code decoding and inferring step 1495.

[0198] In yet another configuration of the SBH test 1480, the decision depends on the value of the valid sign bit hiding flag, and the value of the quantization parameter (QP) of the current TB, the difference between the first significant position and the last significant position of the quantization parameter QP of the transform block. If the valid sign bit hiding flag is "TRUE" and the difference between the first significant position and the last significant position is greater than 3, the video decoder 134 also checks the value of the QP of the current TB. If the QP is 4 (or any value indicating lossless operation), the step 1480 returns "NO" and the method 1400 proceeds to the code decoding step 1490. Otherwise, if the QP is not 4 (or a similar value indicating lossless operation), the step 1480 returns "YES" and the method 1400 proceeds to the code decoding and inferring step 1495.

[0199] In a code pass decoding step 1490, the sign bits of any significant coefficients of the currently selected sub-block are bypass decoded from the bit stream 133. The sign bits are bypass decoded in the backward diagonal scan order 810 from the bit stream 133. The value of a quantized coefficient is set to -x[n] if its associated sign bit has a value of one. The value of a quantized coefficient is set to x[n] if its associated sign bit has a value of zero. The method 1400 ends with the execution of step 1490.

[0200] In a step 1495 of decoding and inferring signs, the sign bits of the significant coefficients of the currently selected sub-block are bypass decoded from the bit stream 133 in the backward diagonal scan order 810. The sign bit associated with the first significant coefficient (last visited in the backward diagonal scan order 810) is not decoded from the bit stream 133. In other words, if there are N significant coefficients in the currently selected sub-block, N-1 sign bits are bypass decoded from the bit stream 133. The sign bit associated with the first significant coefficient is inferred based on the sum of the parities of the significant coefficients. If the sum of the parities is zero, the sign bit associated with the first significant coefficient is inferred as zero. If the sum of the parities is one, the sign bit associated with the first significant coefficient is inferred as one. If the associated sign bit has a value of one, the value of the quantized coefficient is set to -x[n]. If the associated sign bit has a value of zero, the value of the quantized coefficient is set to x[n]. The method 1400 then ends.

[0201] The configurations described in methods 900 and 1000 allow lossless compression of video data to be performed while using the normal residual coding process. Dependent quantization and sign bit hiding are lossy coding tools that are flexibly disabled when lossless operations are desired, but may still be available to achieve improved coding performance in lossy coding blocks.

[0202] Industrial Applicability The described arrangements are applicable in the computer and data processing industry, particularly in digital signal processing for decoding, encoding signals such as video and image signals, to achieve high compression efficiency.

[0203] The above describes only some embodiments of the present invention, to which modifications and / or variations may be made without departing from the scope and spirit of the present invention, and the embodiments are illustrative and not limiting.

Claims

1. 1. A method for decoding a transform block from a bitstream, comprising: decoding a first flag used to determine whether to use dependent quantization in the transform block; determining whether the transform block uses sign bit hiding, in which data indicative of a sign of a significant coefficient at a position is not decoded from the bitstream; If the first flag is TRUE, the sign bit hiding is not used in the transform block; decoding the transform block using the dependent quantization when it is determined based on the first flag that the dependent quantization is used for the transform block; if it is determined that the sign bit hiding is to be used, decoding the transform block using the sign bit hiding; After checking the first flag, a disable flag for transform skip residual coding is checked, and if the disable flag is TRUE, the dependent quantization is not used; The invalid flag indicates whether the first residual coding is applied instead of the second residual coding even when the transform process is skipped, and the first residual coding is a process for a block for which the transform process is not skipped, and the second residual coding is a process for a block for which the transform process is skipped. A method comprising:

2. the first residual coding corresponds to a backward scan order starting at a bottom right position of a sub-block and ending at a top left position of the sub-block; 2. The method of claim 1, wherein the second residual coding corresponds to a forward scan order starting at a top-left location of a sub-block and ending at a bottom-right location of the sub-block.

3. 2. The method of claim 1, wherein if the sign bit hiding is not used, then as many pieces of sign data as there are significant coefficients in a subblock are decoded.

4. 2. The method of claim 1, wherein an enable flag for the sign bit hiding is at least TRUE if the sign bit hiding is to be used.

5. The method of claim 1, wherein the first flag is not a flag relating to a sequence parameter set.

6. 1. A method for encoding a transform block into a bitstream, comprising the steps of: encoding a first flag used to determine whether to use dependent quantization in the transform block; determining whether the transform block uses sign bit hiding, in which data indicating the sign of a significant coefficient at a position is not coded into the bitstream; If the first flag is TRUE, the sign bit hiding is not used in the transform block; encoding the transform block using the dependent quantization when it is determined based on the first flag that the dependent quantization is to be used for the transform block; if it is determined that the sign bit hiding is to be used, encoding the transform block using the sign bit hiding; After checking the first flag, a disable flag for transform skip residual coding is checked, and if the disable flag is TRUE, the dependent quantization is not used; The invalid flag indicates whether the first residual coding is applied instead of the second residual coding even when the transform process is skipped, and the first residual coding is a process for a block for which the transform process is not skipped, and the second residual coding is a process for a block for which the transform process is skipped. A method comprising:

7. the first residual coding corresponds to a backward scan order starting at a bottom right position of a sub-block and ending at a top left position of the sub-block; 7. The method of claim 6, wherein the second residual coding corresponds to a forward scan order starting at a top-left location of a sub-block and ending at a bottom-right location of the sub-block.

8. 7. The method of claim 6, wherein if the sign bit hiding is not used, then as many pieces of sign data as there are significant coefficients in a subblock are encoded.

9. 7. The method of claim 6, wherein an enable flag for the sign bit hiding is at least TRUE if the sign bit hiding is to be used.

10. The method of claim 6, wherein the first flag is not a flag relating to a sequence parameter set.

11. 1. A decoding device for decoding a transform block from a bitstream, comprising: means for decoding a first flag used to determine whether to use dependent quantization in the transform block; means for determining whether the transform block uses sign bit hiding, in which data indicative of the sign of a significant coefficient at a position is not decoded from the bitstream; If the first flag is TRUE, the sign bit hiding is not used in the transform block; means for decoding the transform block using the dependent quantization when it is determined based on the first flag that the dependent quantization is to be used for the transform block; means for decoding the transform block using the sign bit hiding if it is determined that the sign bit hiding is to be used; After checking the first flag, a disable flag for transform skip residual coding is checked, and if the disable flag is TRUE, the dependent quantization is not used; The invalid flag indicates whether the first residual coding is applied instead of the second residual coding even when the transform process is skipped, and the first residual coding is a process for a block for which the transform process is not skipped, and the second residual coding is a process for a block for which the transform process is skipped. A decoding device comprising:

12. 1. An encoding device for encoding a transform block into a bitstream, comprising: means for encoding a first flag used to determine whether dependent quantization is used in the transform block; means for determining whether the transform block uses sign bit hiding, in which data indicative of the sign of a significant coefficient at a position is not coded into the bitstream; If the first flag is TRUE, the sign bit hiding is not used in the transform block; means for encoding the transform block using the dependent quantization when it is determined based on the first flag that the dependent quantization is to be used for the transform block; means for encoding the transform block using the sign bit hiding if it is determined that the sign bit hiding is to be used; After checking the first flag, a disable flag for transform skip residual coding is checked, and if the disable flag is TRUE, the dependent quantization is not used; The invalid flag indicates whether the first residual coding is applied instead of the second residual coding even when the transform process is skipped, and the first residual coding is a process for a block for which the transform process is not skipped, and the second residual coding is a process for a block for which the transform process is skipped.

13. An encoding device comprising:

13. A computer program product for causing a computer to carry out the method according to any one of claims 1 to 5.

14. A computer program product for causing a computer to carry out the method according to any one of claims 6 to 10.