Method for decoding a conversion block from a bitstream, method for encoding a conversion block from a bitstream, decoding device, encoding device, computer program

Abstract

To provide a system and a method for decoding a sub-block of a residual coefficient of a conversion block from a video bit stream.SOLUTION: A functional module of a video decoder includes: determining whether sign bit hiding is used for a sub-block, wherein the determining is based on the value of a conversion skip flag determined for the sub-block and the value of a sign bit hiding flag related to the sub-block; decoding sign bits in a number equal to the number of significant coefficients in the sub-block when the sign bit hiding is not used; and reconstructing a residual coefficient of the sub-block by using the decoded sign bits to decode the sub-block.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] References to related applications This application claims the benefits under 35 U.S.C § 119 as of the filing date of Australian Patent Application No. 2020201753, filed on 10 March 2020, and is incorporated herein by reference as if it were fully described herein.

[0002] The present invention generally relates to digital video signal processing, and more particularly to methods, apparatus, and systems for encoding and decoding blocks of video samples. The present invention also relates to a computer program product including a computer-readable medium on which a computer program for encoding and decoding blocks of video samples is recorded. [Background technology]

[0003] Many applications for video coding currently exist, including applications for transmitting and storing video data. Many video coding standards have also been developed, and others are currently under development. Recent developments in video coding standardization have led to the formation of a group called the "Joint Video Experts Team" (JVET). The Joint Video Experts Team (JVET) includes members of Study Group 16, Question 6 (SG16 / Q6) of the International Telecommunication Union (ITU) Telecommunication Standardization Sector (ITU-T), also known as the "Video Coding Experts Group" (VCEG), and members of the International Organization for Standardization / International Electrotechnical Commission Joint Technical Committee 1 / Subcommittee 29 / Working Group 11 (ISO / IEC JTC1 / SC29 / WG11), also known as the "Moving Picture Experts group" (MPEG).

[0004] The Joint Video Experts Team (JVET) analyzed the responses at its 10th conference held in San Diego, USA, and issued a Call for Proposals (CfP). The submitted responses demonstrated video compression capabilities that significantly outperform those of the current state-of-the-art video compression standard, namely "High Efficiency Video Coding" (HEVC). Based on this outperformance, it was decided to launch a project to develop a new video compression standard named "versatile video coding" (VVC). VVC is expected to address the ongoing demand for ever-higher compression performance, particularly as video formats increase their capabilities (e.g., with higher resolution and higher frame rates) and address the growing market demand for service delivery over WANs where bandwidth costs are relatively high. At the same time, VVC must be feasible with modern silicon processes and must offer an acceptable trade-off between the performance achieved and the cost of implementation (e.g., in terms of silicon area, CPU processor load, memory usage, and bandwidth).

[0005] Video data consists of a sequence of frames of image data, each frame containing one or more color channels. Generally, one primary color channel and two secondary color channels are required. The primary color channel is commonly called the "luma" channel, and the secondary color channels are commonly called the "chroma" channels. Video data is typically displayed in the RGB (red-green-blue) color space, which has a high degree of correlation between each of the three elements. Video data representations seen by encoders or decoders often use color spaces such as YCbCr. YCbCr concentrates luminance, mapped to "luma" according to a transfer function, in the Y (primary) channel, and chroma in the Cb and Cr (secondary) channels. Furthermore, the Cb and Cr channels may be spatially sampled (subsampled) at a lower rate compared to the luma channel, for example, half horizontally and half vertically, in what is known as the "4:2:0 chroma format." The 4:2:0 chroma format is used in internet video streaming, broadcast television, and Blu-ray. TMIt is commonly used in "consumer" applications, such as saving to disk. Subsampling the Cb and Cr channels horizontally at half the rate and not subsampling vertically is known as the "4:2:2 chroma format." The 4:2:2 chroma format is typically used in professional applications, including capturing video for film production. The higher sampling rate of the 4:2:2 chroma format makes the resulting video more resilient to editing operations such as color grading. Before distribution to consumers, 4:2:2 chroma format material is often converted to the 4:2:0 chroma format and then encoded for distribution to consumers. In addition to the chroma format, video is also characterized by its resolution and frame rate. The example resolutions are Ultra High Definition (UHD) at 3840x2160 resolution, or "8K" at 7680x4320 resolution, with example frame rates of 60 or 120Hz. The luma sample rate may range from approximately 500 megasamples / second to several gigasamples / second. In a 4:2:0 chroma format, the sample rate for each chroma channel is one-quarter of the luma sample rate, and in a 4:2:2 chroma format, the sample rate for each chroma channel is half of the luma sample rate.

[0006] The VVC standard is a “block-based” codec, in which a frame is first divided into a square array of regions known as “coding tree units” (CTUs). CTUs generally occupy a relatively large area, such as 128 × 128 lumens. However, the CTUs at the right and bottom edges of each frame may be smaller in area. Each CTU is associated with a “coding tree” for the lumens channels and an additional coding tree for the chroma channels. The coding tree defines the decomposition of the CTU region into a set of blocks, also called “coding blocks” (CBs). It is also possible for a single coding tree to specify blocks for both the lumens and chroma channels, in which case the juxtaposed set of coding blocks is called a “coding unit” (CU), i.e., each CU has a coding block for each color channel. The CBs are processed to encode or decode in a specific order. As a result of using the 4:2:0 chroma format, a CTU having a luma coding tree for a 128×128 luma sample region has a corresponding chroma coding tree for a 64×64 chroma sample region arranged alongside the 128×128 luma sample region. When a single coding tree is used for the luma and chroma channels, the collated set of blocks for a given area is generally called a “unit,” e.g., the CU described above, as well as a “prediction unit” (PU) and a “transformation unit” (TU). When separate coding trees are used for a given area, the CB described above, as well as a “prediction block” (PB) and a “transformation block” (TB), are used.

[0007] Despite the above distinction between "unit" and "block," the term "block" may also be used as a general term for an area or region of a frame to which the behavior applies to all color channels.

[0008] For each CU, a prediction unit (PU) is generated of the content (sample value) of the corresponding region of the frame data ("prediction unit"). If the PU is generated from sample values ​​in a previously signaled frame, the prediction is called interpretation. If the PU is generated from previous samples in the same frame, the prediction is called intrapretation. Furthermore, a representation of the difference (or "residual" in the spatial domain) between the prediction and the content of the region seen in the input to the encoder is formed. The difference for each color channel can be transformed as a block of residual coefficients, encoded, and can form one or more TUs for a given CU. The residual coefficients can be transformed by transformations such as the discrete cosine transform (DCT), discrete sine transform (DST), or other transformations to generate a final block of transformation coefficients that substantially decorrelated the residual samples. Substantial coding gain can be achieved by quantizing the transformation coefficients. The quantized transformation coefficients are then traversed in an order such as a back diagonal scan, and each coefficient is encoded by an entropy encoder. Entropy coding involves representing each coefficient with a syntax element, each of which is binarized. The binarized syntax elements are then either further coded by a context-adaptive binary arithmetic encoder (CABAC) or passed to a bitstream ("bypass coding").

[0009] For some classes of video content, such as screen content, it may be advantageous to avoid performing transformations. If transformations should be avoided, residual coefficients are quantized, traversed, and encoded. Since the statistics of residual coefficients are not the same as those of transformed coefficients, it is generally advantageous to encode residual coefficients using a different process than the encoding process for transformed coefficients. Typical methods used to encode residual coefficients include the "normal residual coding" (RRC) process and the "transformation-skipped residual coding" (TSRC) process, with a particular one of these processes being selected for the block depending on whether a transformation was performed.

[0010] Depending on the use case, it may be desirable to compress video data losslessly (i.e., without encoding loss). CU can be encoded losslessly by skipping both the transformation and quantization steps. In the TSRC process, quantization can be avoided by setting the "quantization parameter" to a value that does not indicate quantization. However, as mentioned above, the TSRC process is only suitable for classes of video content such as screen content. Therefore, forcing lossless encoding of video data to use the TSRC process is not optimal. For lossless encoding, it is desirable to have more flexible options available depending on the statistics of the video data being encoded, while minimizing the amount of additional logic required to support that additional flexibility. [Overview of the project]

[0011] The object of the present invention is to substantially overcome, or at least improve upon, one or more drawbacks of existing configurations.

[0012] One aspect of the present invention provides a method for decoding a subblock of residual coefficients of a transform block from a video bitstream, the method comprising: determining whether sign bit hiding is used for the subblock, the determination being based on the value of a transform skip flag determined for the subblock and the value of a sign bit hiding flag associated with the subblock; if sign bit hiding is not used, decoding a number of sign bits equal to the number of significance coefficients in the subblock; and decoding the subblock by reconstructing the residual coefficients of the subblock using the decoded sign bits.

[0013] In another embodiment, sign bit hiding is used if the sign bit hiding flag has a value of TRUE, the conversion skip flag has a value of FALSE, and the difference between the first significant position and the last significant position of a subblock is greater than 3.

[0014] In another embodiment, sign bit hiding is not used if the sign bit hiding flag has a value of TRUE and the conversion skip flag has a value of TRUE.

[0015] In another embodiment, the method further comprises, if it is determined that sign bit hiding is to be used, decoding a number of sign bits equal to the number of effective coefficients in the subblock minus 1, and determining additional sign bits from the sum of the parities of the effective coefficients in the subblock.

[0016] Another aspect of the present invention provides a method for decoding a subblock of residual coefficients of a transformed block from a video bitstream, the method comprising: determining whether sign bit hiding is used for the subblock, the determination being based on the value of a sign bit hiding flag and the value of a quantization parameter associated with the subblock; if sign bit hiding is not used, decoding a number of sign bits equal to the number of significance coefficients in the subblock; and decoding the subblock by reconstructing the residual coefficients of the subblock using the decoded sign bits.

[0017] In another embodiment, if the sign bit hiding flag has a value of TRUE and the quantization parameter is equal to 4, sign bit hiding is not used.

[0018] In another embodiment, sign bit hiding is used if the sign bit hiding flag has a value of TRUE, the quantization parameter is not equal to 4, and the difference between the first significant position and the last significant position of a subblock is greater than 3.

[0019] Another aspect of the present invention provides a method for decoding a subblock of residual coefficients of a transformed block from a video bitstream, the method comprising: determining whether sign bit hiding is used for the subblock, the determination being based on the value of a sign bit hiding flag and a value of a TSRC disabled flag, and if sign bit hiding is not used, decoding a number of sign bits equal to the number of significance coefficients in the subblock; and decoding the subblock by reconstructing the residual coefficients of the subblock using the decoded sign bits.

[0020] In another embodiment, sign bit hiding is used when the sign bit hiding flag has a value of TRUE, the TSRC invalid flag has a value of FALSE, and the difference between the first significant position and the last significant position of a subblock is greater than 3.

[0021] In another embodiment, the sign bit hiding flag has a value of TRUE, and the TSRC invalid flag has a value of TRUE.

[0022] Another aspect of the present invention provides a non-temporary computer-readable medium containing a computer program for carrying out a method for decoding a subblock of residual coefficients of a conversion block from a video bitstream, the method comprising: determining whether sign bit hiding is used for the subblock, the determination being based on the value of a conversion skip flag determined for the subblock and the value of a sign bit hiding flag associated with the subblock; if sign bit hiding is not used, decoding a number of sign bits equal to the number of significance coefficients in the subblock; and decoding the subblock by reconstructing the residual coefficients of the subblock using the decoded sign bits.

[0023] Another aspect of the present invention provides a system having a memory and a processor, the processor configured to execute code stored in the memory to perform a method of decoding a sub-block of residual coefficients of a transform block from a video bitstream, the method comprising determining whether sign bit hiding is used for the sub-block, the determination being based on a value of a transform skip flag determined for the sub-block and a value of a sign bit hiding flag associated with the sub-block, and when sign bit hiding is not used, decoding a number of sign bits equal to the number of valid coefficients in the sub-block and decoding the sub-block by reconstructing the residual coefficients of the sub-block using the decoded sign bits.

[0024] Another aspect of the present invention provides a video decoder configured to receive a sub-block of residual coefficients of a transform block from a video bitstream, determine whether sign bit hiding is used for the sub-block, the determination being based on a value of a transform skip flag determined for the sub-block and a value of a sign bit hiding flag associated with the sub-block, and when sign bit hiding is not used, decode a number of sign bits equal to the number of significant coefficients in the sub-block and decode the sub-block by reconstructing the residual coefficients of the sub-block using the decoded sign bits.

[0025] Other aspects are described.

Brief Description of the Drawings

[0026] At least one embodiment of the present invention will be described with reference to the following drawings and appendices. [Figure 1] FIG. 1 is a schematic block diagram showing a video encoding and decoding system. [Figure 2A] FIG. 2A forms a schematic block diagram of a general-purpose computer system capable of implementing one or both of the video encoding and decoding systems of FIG. 1. [Figure 2B]Figure 2B forms a schematic block diagram of a general-purpose computer system capable of performing one or both of the video encoding and decoding systems of Figure 1. [Figure 3] Figure 3 is a schematic block diagram showing the functional module of the video encoder. [Figure 4] Figure 4 is a schematic block diagram showing the functional module of the video decoder. [Figure 5] Figure 5 is a schematic block diagram showing the available division of a block into one or more blocks in the tree structure of a general-purpose video encoding. [Figure 6] Figure 6 is a schematic diagram of the data flow for achieving accepted partitioning of blocks into one or more blocks in a tree structure of general-purpose video encoding. [Figure 7A] Figure 7A shows an example of dividing a coding tree unit (CTU) into several coding units (CUs). [Figure 7B] Figure 7B shows an example of dividing a coding tree unit (CTU) into several coding units (CUs). [Figure 8A] Figure 8A shows a two-level posterior diagonal scan. [Figure 8B] Figure 8B shows a two-level forward diagonal scan. [Figure 9] Figure 9 shows how to encode the transformation block of the residual coefficients. [Figure 10] Figure 10 shows how to decode the transformation block of residual coefficients. [Figure 11] Figure 11 shows a method for quantizing the transformation block of residual coefficients, which is performed by the method in Figure 9. [Figure 12] Figure 12 shows a method for scaling the transformation block of quantized coefficients, as performed by the method in Figure 10. [Figure 13] Figure 13 shows how to encode subblocks of quantized coefficients, as performed by the method in Figure 9. [Figure 14]Figure 14 shows how to decode subblocks of quantized coefficients, as performed by the method in Figure 10. [Modes for carrying out the invention]

[0027] Where one or more of the attached drawings refer to steps and / or features having the same reference numeral, those steps and / or features have the same function or operation for the purposes of this specification, unless otherwise intended.

[0028] As mentioned above, lossless coding is sometimes desirable to be supported by the existing building blocks of a codec. However, since the diverse classes of video data coded in a lossless manner cannot be guaranteed to exhibit the statistical properties for which TSRC processing is designed, exclusively using TSRC processing for lossless coding can produce suboptimal coding performance. Therefore, greater flexibility in the selection of high-level building blocks that lossless coding can use enables superior coding performance with minimal additional complexity to the overall design.

[0029] Figure 1 is a schematic block diagram showing the functional modules of a video encoding and decoding system 100. The system 100 includes a source device 110 and a destination device 130. A communication channel 120 is used to communicate encoded video information from the source device 110 to the destination device 130. In some configurations, the source device 110 and the destination device 130 may each consist of either or both a mobile phone handset or a “smartphone,” in which case the communication channel 120 is a wireless channel. In other configurations, the source device 110 and the destination device 130 may consist of video conferencing equipment, in which case the communication channel 120 is typically a wired channel, such as an internet connection. Furthermore, the source device 110 and the destination device 130 may consist of any of a wide range of devices, including devices that support wireless television broadcasting, cable television applications, internet video applications (including streaming), and applications in which encoded video data is captured onto some computer-readable storage medium, such as a hard disk drive in a file server.

[0030] As shown in Figure 1, the source device 110 includes a video source 112, a video encoder 114, and a transmitter 116. The video source 112 typically has a source of captured video frame data (shown as 113), such as an image sensor, a previously captured video sequence stored on a non-temporary recording medium, or video supplied from a remote image sensor. The video source 112 may also be the output of a computer graphics card, for example, displaying the video output of an operating system and various applications running on a computing device such as a tablet computer. Examples of source devices 110 that may include an image sensor as the video source 112 include smartphones, video cameras, professional video cameras, and network video cameras.

[0031] The video encoder 114 converts (or "encodes") captured frame data (indicated by arrow 113) from the video source 112 into a bitstream (indicated by arrow 115), as further explained with reference to Figure 3. The bitstream 115 is transmitted by the transmitter 116 over the communication channel 120 as encoded video data (or "encoded video information"). The bitstream 115 may also be stored in a non-temporary storage device 122, such as "flash" memory or a hard disk drive, until it is later transmitted over the communication channel 120, or instead of being transmitted over the communication channel 120.

[0032] The destination device 130 includes a receiver 132, a video decoder 134, and a display device 136. The receiver 132 receives encoded video data from the communication channel 120 and passes the received video data as a bitstream to the video decoder 134 (indicated by arrow 133). The video decoder 134 then outputs the decoded frame data to the display device 136 (indicated by arrow 135) and plays back the video data. The decoded frame data 135 has the same chroma format as the frame data 113. Examples of the display device 136 include liquid crystal displays such as cathode ray tubes, smartphones, tablet computers, computer monitors, or standalone television sets. It is also possible for the functionality of each of the source device 110 and the destination device 130 to be realized in a single device, such as a mobile phone handset and a tablet computer.

[0033] Despite the exemplary devices described above, each of the source device 110 and destination device 130 may be configured within a general-purpose computing system, typically through a combination of hardware and software components. Figure 2A shows such a computer system 200, which includes a computer module 201 and output devices including a keyboard 202, a mouse pointer device 203, a scanner 226, a camera 227 which can be configured as a video source 112, and a microphone 280, and a printer 215, a display device 214 which can be configured as a display device 136, and a speaker 217. An external modulator / demodulator (modem) transceiver device 216 may be used by the computer module 201 to communicate with a communication network 220 via connection 221. The communication network 220, which may represent a communication channel 120, may be the Internet, a cellular telecommunications network, or a wide area network (WAN) such as a private WAN. If connection 221 is a telephone line, the modem 216 may be a conventional “dial-up” modem. Alternatively, if connection 221 is a high-capacity (e.g., cable or fiber optic) connection, modem 216 may be a broadband modem. A wireless modem may also be used for a wireless connection to the communication network 220. The transceiver device 216 can provide the functionality of a transmitter 116 and a receiver 132, and the communication channel 120 can be embodied within connection 221.

[0034] The computer module 201 typically includes at least one processor unit 205 and a memory unit 206. For example, the memory unit 206 may have semiconductor random access memory (RAM) and semiconductor read-only memory (ROM). The computer module 201 also includes several input / output (I / O) interfaces, including an audio-video interface 207 coupled to a video display 214, a speaker 217, and a microphone 280; an I / O interface 213 coupled to a keyboard 202, a mouse 203, a scanner 226, a camera 227, and optionally a joystick or other human interface device (not shown); and an interface 208 for an external modem 216 and a printer 215. The signal from the audio-video interface 207 to the computer monitor 214 is generally the output of a computer graphics card. In some implementations, the modem 216 may be integrated into the computer module 201, for example, within interface 208. Computer module 201 also has a local network interface 211, which allows the computer system 200 to connect to a local area communication network 222, known as a local area network (LAN), via connection 223. As shown in Figure 2A, the local communication network 222 can also connect to a wide network 220 via connection 224, which typically includes a so-called "firewall" device or a device with similar functionality. The local network interface 211 is Ethernet TM Circuit card, Bluetooth TM This may include a wireless configuration or an IEEE 802.11 wireless configuration, but many other types of interfaces may be implemented for interface 211. The local network interface 211 can also provide the functionality of the transmitter 116, and the receiver 132 and communication channel 120 can also be implemented in the local communication network 222.

[0035] I / O interfaces 208 and 213 can provide either or both serial and parallel connectivity, the former typically implemented according to the Universal Serial Bus (USB) standard and having a corresponding USB connector (not shown). A storage device 209 is provided, typically including a hard disk drive (HDD) 210. Other storage devices such as floppy disk drives and magnetic tape drives (not shown) may also be used. An optical disk drive 212 is typically provided to function as a non-volatile source of data. Optical disks (e.g., CD-ROM, DVD, Blu-ray Disc) TM Portable memory devices such as USB-RAM, portable external hard drives, and floppy disks can be used, for example, as suitable sources of data for the computer system 200. Typically, one of the HDD 210, optical drive 212, or networks 220 and 222 may be configured to act as a video source 112 or as a destination for decoded video data to be stored for playback via the display 214. The source device 110 and destination device 130 of system 100 may be embodied in the computer system 200.

[0036] The components 205-213 of computer module 201 communicate, typically via interconnection bus 204, in a manner that brings about a conventional operating mode of computer system 200 known to those skilled in the art. For example, processor 205 is coupled to system bus 204 using connection 218. Similarly, memory 206 and optical disc drive 212 are coupled to system bus 204 by connection 219. Examples of computers in which the above configuration can be implemented include IBM-PC and compatible machines, Sun SPARC stations, and Apple Macs. TM Or it includes a similar computer system.

[0037] Where appropriate or necessary, the video encoder 114 and video decoder 134, and the method described below, can be implemented using a computer system 200. Specifically, the video encoder 114, video decoder 134, and the method described can be implemented as one or more software application programs 233 executable within the computer system 200. Specifically, the steps of the video encoder 114, video decoder 134, and the method described are performed by instructions 231 (see Figure 2B) in the software 233 executed within the computer system 200. The software instructions 231 may be formed as one or more code modules, each for performing one or more specific tasks. The software may also be divided into two separate parts, in which case the first part and its corresponding code modules perform the method described, and the second part and its corresponding code modules manage the user interface between the first part and the user.

[0038] The software can be stored, for example, on a computer-readable medium including the storage device described below. The software is loaded from the computer-readable medium into the computer system 200 and then executed by the computer system 200. Such software or a computer-readable medium having a computer program recorded on it is a computer program product. The use of the computer program product in the computer system 200 preferably brings together a video encoder 114, a video decoder 134, and other advantageous devices for carrying out the method described.

[0039] The software 233 is typically stored on the HDD 210 or in memory 206. The software is loaded from a computer-readable medium into the computer system 200 and executed by the computer system 200. For example, the software 233 can be stored on an optically readable disk storage medium (e.g., a CD-ROM) 225 that is read by an optical disk drive 212.

[0040] In some cases, the application program 233 may be encoded on one or more CD-ROMs 225 and supplied to the user, and read via the corresponding drive 212, or alternatively, read by the user from a network 220 or 222. Furthermore, the software may also be loaded into the computer system 200 from other computer-readable media. Computer-readable storage media refers to any non-temporary, tangible storage medium that provides the computer system 200 with recorded instructions and / or data for execution and / or processing. Examples of such storage media include floppy disks, magnetic tapes, CD-ROMs, DVDs, and Blu-ray Discs. TM This includes, but is not limited to, hard disk drives, ROMs or integrated circuits, USB memory sticks, magneto-optical disks, or computer-readable cards such as PCMCIA cards, whether such devices are inside or outside the computer module 201. Examples of temporary or intangible computer-readable transmission media that may also participate in providing software, application programs, instructions and / or video data or encoded video data to the computer module 401 include wireless or infrared transmission channels, as well as network connections to other computers or networked devices, and the Internet or intranets, including email transmissions and information recorded on websites, etc.

[0041] The second part of the application program 233 and the corresponding code modules described above may be executed to implement one or more graphical user interfaces (GUIs) that are rendered on the display 214 or otherwise represented. Typically, through the operation of the keyboard 202 and mouse 203, users of the application and computer system 200 can interact with the interface in a functionally adaptable manner and provide control commands and / or inputs to the GUI-related application. Other forms of functionally adaptable user interfaces can also be implemented, such as an audio interface that utilizes speech prompts output via the speaker 217 and user voice commands entered via the microphone 280.

[0042] Figure 2B is a detailed schematic block diagram of the processor 205 and “memory” 234. Memory 234 represents the logical collection of all memory modules (including HDD 209 and semiconductor memory 206) accessible by the computer module 201 in Figure 2A.

[0043] When the computer module 201 is first powered on, a power-on self-test (POST) program 250 is executed. The POST program 250 is typically stored in the ROM 249 of the semiconductor memory 206, as shown in Figure 2A. Hardware devices such as the ROM 249 that store software are sometimes called firmware. The POST program 250 checks the hardware within the computer module 201 to ensure it functions correctly, and typically checks the processor 205, memory 234 (209, 206), and the Basic Input / Output System Software (BIOS) module 251, which is also usually stored in the ROM 249, for correct operation. If the POST program 250 executes successfully, the BIOS 251 boots the hard disk drive 210, as shown in Figure 2A. Once the hard disk drive 210 is booted, a bootstrap loader program 252, which resides on the hard disk drive 210, is executed via the processor 205. This loads the operating system 253 into the RAM memory 206, and the operating system 253 then begins to operate. Operating System 253 is a system-level application executable by Processor 205, fulfilling various high-level functions including processor management, memory management, device management, storage management, software application interface, and general-purpose user interface.

[0044] The operating system 253 manages memory 234 (209, 206) to ensure that each process or application running on the computer module 201 has enough memory to run without conflicting with memory allocated to other processes. Furthermore, the different types of memory available in the computer system 200 in Figure 2A must be used appropriately so that each process can run effectively. Therefore, aggregated memory 234 is not intended to show how specific segments of memory are allocated (unless otherwise specified), but rather to provide a general view of the memory accessible by the computer system 200 and how such segments are used.

[0045] As shown in Figure 2B, the processor 205 includes a number of functional modules, including a control unit 239, an arithmetic logic unit (ALU) 240, and local or internal memory 248, sometimes called cache memory. The cache memory 248 typically contains a number of storage registers 244-246 within a register section. One or more internal buses 241 functionally interconnect these functional modules. The processor 205 also typically has one or more interfaces 242 for communicating with external devices via the system bus 204, using connections 218. Memory 234 is coupled to the bus 204 using connections 219.

[0046] The application program 233 includes a sequence of instructions 231, which may include conditional branch and loop instructions. The program 233 may also include data 232 used for the execution of the program 233. The instructions 231 and data 232 are stored in memory locations 228, 229, 230, and 235, 236, and 237, respectively. Depending on the relative size of the instructions 231 and memory locations 228-230, a particular instruction may be stored in a single memory location, as indicated by the instruction indicated at memory location 230. Alternatively, the instruction may be segmented into several parts, each stored in a separate memory location, as indicated by the instruction segments indicated at memory locations 228 and 229.

[0047] Generally, the processor 205 is given a set of instructions to be executed within it. The processor 205 waits for subsequent inputs and responds to these inputs by executing another set of instructions. Each input can be provided from one or more sources, including data generated by one or more of the input devices 202, 203, data received from an external source via one of the networks 220, 202, data retrieved from one of the storage devices 206, 209, or data retrieved from a storage medium 225 inserted into the corresponding reader 212, all of which are shown in Figure 2A. Execution of a set of instructions may result in the output of data. Execution may also include storing data or variables in memory 234.

[0048] The video encoder 114, video decoder 134, and the described method can use input variables 254 stored in corresponding memory locations 255, 256, and 257 in memory 234. The video encoder 114, video decoder 134, and the described method generate output variables 261, which are stored in corresponding memory locations 262, 263, and 264 in memory 234. Intermediate variables 258 may be stored in memory locations 259, 260, 266, and 267.

[0049] Referring to processor 205 in Figure 2B, registers 244, 245, 246, arithmetic logic unit (ALU) 240, and control unit 239 cooperate to execute the sequence of microoperations necessary to perform a “fetch, decode, and execute” cycle for all instructions in the instruction set that make up program 233. Each fetch, decode, and execute cycle is A fetch operation that fetches or reads instruction 231 from memory locations 228, 229, and 230. The control unit 239 performs a decoding operation to determine which instruction has been fetched. The control unit 239 and / or ALU 240 execute the operation to carry out the instruction. It has.

[0050] Subsequently, further fetch, decode, and execute cycles for the next instruction can be performed. Similarly, the control unit 239 can perform a storage cycle to store or write a value to memory location 232.

[0051] Each step or subprocess in the method shown in Figures 19 to 14, described later, is associated with one or more segments of program 233 and is typically executed by the cooperation of register sections 244, 245, 247, ALU 240, and control unit 239 in the processor 205 to perform a fetch, decode, and execute cycle for all instructions in the instruction set for the noted segment of program 233.

[0052] Figure 3 is a schematic block diagram showing the functional modules of the video encoder 114. Figure 4 is a schematic block diagram showing the functional modules of the video decoder 134. Generally, data passes between the functional modules of the video decoder 134 and the video encoder 114 in groups of samples or coefficients, such as by dividing blocks into fixed-size subblocks, or as arrays. The video encoder 114 and video decoder 134 can be implemented using a general-purpose computer system 200, as shown in Figures 2A and 2B, and the various functional modules can be implemented by dedicated hardware within the computer system 200, or by software executable within the computer system 200, such as one or more software code modules of a software application program 233 that resides on a hard disk drive 205 and is controlled by the processor 205 during its execution. Alternatively, the video encoder 114 and video decoder 134 may be implemented by a combination of software executable within the computer system 200 and dedicated hardware. The video encoder 114, video decoder 134, and the method described may, as an alternative, be implemented by dedicated hardware such as one or more integrated circuits that perform the functions or subfunctions of the method described. Such dedicated hardware may include a graphics processing unit (GPU), a digital signal processor (DSP), an application-specific standard product (ASSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or one or more microprocessors and associated memory. In particular, the video encoder 114 includes modules 310-386, and the video decoder 134 includes modules 420-496, each of which may be implemented as one or more software code modules of a software application program 233.

[0053] The video encoder 114 in Figure 3 is an example of a General-Purpose Video Coding (VVC) video coding pipeline, but other video codecs can also be used to perform the processing stages described herein. The video encoder 114 receives captured frame data 113, such as a series of frames, each frame containing one or more color channels. The frame data 113 may be in any chroma format, e.g., 4:0:0, 4:2:0, 4:2:2, or 4:4:4 chroma format. The block partitioner 310 first divides the frame data 113 into CTUs, which are generally square in shape, configured so that a specific size is used for the CTUs. The size of the CTUs can be, for example, 64×64, 128×128, or 256×256 luma samples. The block partitioner 310 further divides each CTU into one or more CBs according to the luma coding tree and the chroma coding tree. The CBs can be of various sizes and may include both square and non-square aspect ratios. In the VVC standard, CBs, CUs, PUs, and TUs always have side lengths that are powers of 2. Thus, the current CB, represented as 312, is output from block partitioner 310 and proceeds according to the chromacoding tree and lumacoding tree of the CTU, following iterations across one or more blocks of the CTU. Options for partitioning the CTU into CBs are further described below with reference to Figures 5 and 6.

[0054] The CTUs obtained from the initial split of frame data 113 may be scanned in raster scan order and grouped into one or more “slice.” The slices may be “intra” (or “I”) slices. An intra slice (I slice) indicates that all CUs in the slice are intra-predicted. Alternatively, the slices may be uni-predictive or bi-predictive (“P” or “B” slices, respectively), indicating further availability of uni- and bi-predictive capabilities in the slice, respectively.

[0055] For each CTU, the video encoder 114 operates in two stages. In the first stage (called the “search” stage), the block partitioner 310 tests various potential configurations of the coding tree. Each potential configuration of the coding tree has an associated “candidate” CB. The first stage involves testing various candidate CBs to select one that provides low distortion and high compression efficiency. This test generally involves Lagrangian optimization, thereby evaluating candidate CBs based on a weighted combination of rate (encoding cost) and distortion (error with respect to the input frame data 113). The “best” candidate CB (the CB with the lowest evaluated rate / distortion) is selected for subsequent encoding to the bitstream 115. The evaluation of candidate CBs includes the option of using the CB for a given area, or further dividing the area according to various partitioning options and encoding each of the resulting smaller areas with further CBs, or further dividing the area. As a result, both CBs and the coding tree itself are selected in the search stage.

[0056] The video encoder 114 generates a prediction block (PB) indicated by arrow 320 for each CB, e.g., CB312. PB320 is a prediction of the content of the associated CB312. The subtractor module 322 generates a difference between PB320 and CB312, indicated as 324 (or "residual," referring to the difference in the spatial domain). The residual 324 is the difference in block size between the corresponding samples in PB320 and CB312. The residual 324 is transformed, quantized, and represented as a transformed block (TB) indicated by arrow 336. PB320 and its associated TB336 are typically selected from one of many possible candidate CBs, for example, based on an evaluated cost or distortion.

[0057] A candidate coding block (CB) is a CB arising from one of the prediction modes available to the video encoder 114 for the associated PB and the resulting residual. Each candidate CB yields one or more corresponding TBs. TB336 is a quantized and transformed representation of the residual 324. When combined with the predicted PB in the video decoder 114, TB336 reduces the difference between the decoded CB and the original CB312 at the expense of additional signals in the bitstream.

[0058] Therefore, each candidate coding block (CB), i.e., the prediction block (PB) combined with the transform block (TB), has an associated coding cost (or "rate") and an associated difference (or "distortion"). The rate is typically measured in bits. The distortion of the CB is typically estimated as the difference of sample values, such as the sum of absolute differences (SAD) or the sum of squared differences (SSD). The estimate obtained from each candidate PB is determined by the mode selector 386 using the residual 324 to determine the prediction mode (represented by arrow 388). Estimating the coding cost associated with each candidate prediction mode and its corresponding residual coding can be performed at a considerably lower cost than entropy coding of the residuals. Therefore, a large number of candidate modes can be evaluated to determine the optimal mode in rate distortion detection.

[0059] Determining the optimal mode in terms of rate distortion is typically achieved using a variation of Lagrangian optimization. The selection of the prediction mode 388 typically involves determining the coding cost for the residual data resulting from the application of a particular prediction mode. The coding cost can be approximated by using the “Sum of Absolute Transform Differences” (SATD), thereby obtaining the estimated transform residual cost using a relatively simple transform such as the Hadamard transform. In some embodiments using relatively simple transforms, the cost obtained from the simplified estimation method is monotonically related to the actual cost that would otherwise be determined from a full evaluation. In embodiments with monotonically related estimated costs, the simplified estimation method can be used to make the same decision (i.e., prediction mode) while reducing the complexity of the video encoder 114. To allow for possible non-monotonicity in the relationship between the estimated cost and the actual cost, the simplified estimation method can be used to generate a list of best candidates. Non-monotonicity may arise, for example, from further mode determination available for coding the residual data. The list of best candidates may be any number. Using the best candidate, a more complete search can be performed to establish the optimal mode selection for encoding the residual data for each candidate, enabling the final selection of the prediction mode 388 along with other mode determinations.

[0060] Prediction modes are broadly classified into two categories. The first category is "intraframe prediction" (also called "intraprediction"). In intraframe prediction, predictions are generated for blocks, and the generation method may use other samples obtained from the current frame. Types of intraprediction include intraplanar, intraDC, intraangle, and matrix-weighted intraprediction (MIP). For intrapredicted PB, different intraprediction modes can be used for luma and chroma, and therefore intraprediction is mainly described in relation to its operation on PB. Furthermore, chroma CB may be predicted from luma samples placed in the same location by cross-component linear model prediction.

[0061] The second category of prediction modes is “interframe prediction” (also called “interpretation”). In interframe prediction, the prediction of a block is generated using samples from one or two frames that precede the current frame in the order in which the frames in the bitstream are encoded. Furthermore, for interframe prediction, a single coding tree is typically used for both the luma and chroma channels. The encoding order of the frames in the bitstream may differ from the order of the frames at capture or display. If one frame is used for prediction, the block is said to be “single-predicted” and has one associated motion vector. If two frames are used for prediction, the block is said to be “double-predicted” and has two associated motion vectors. For P slices, each CU can be intra-predicted or single-predicted. For B slices, each CU can be intra-predicted, single-predicted, or double-predicted. Frames are typically encoded using a “group of pictures” structure that allows for a temporal hierarchy of frames. The temporal hierarchy of frames allows frames to refer to preceding and succeeding pictures in the order in which they are displayed. The images are encoded in the order necessary to ensure that the dependencies for decoding each frame are met.

[0062] A subcategory of interpretation is called “skip mode”. Interpretation mode and skip mode are described as two separate modes. However, both interpretation mode and skip mode involve motion vectors that reference a block of samples from a preceding frame. Interpretation involves an encoded motion vector delta that specifies a motion vector for a motion vector predictor. The motion vector predictor is obtained from a list of one or more candidate motion vectors selected by a “merge index”. The encoded motion vector delta provides a spatial offset to the selected motion vector prediction. Interpretation also uses encoded residuals in bitstream 133. Skip mode selects one of several motion vector candidates using only an index (also called a “merge index”). The selected candidate is used without further signaling. Skip mode also does not support encoding residual coefficients. The absence of encoded residual coefficients when skip mode is used means that no transformation for skip mode needs to be performed. Therefore, skip mode typically does not result in pipeline problems. Pipelining problems can occur in the case of intrapredicted CUs and interpredicted CUs. Due to the limited signaling of skip mode, skip mode is useful for achieving very high compression performance when relatively high-quality reference frames are available. Bi-predicted CUs in higher time layers of a random access picture group structure typically have high-quality reference pictures and motion vector candidates that accurately reflect the underlying motion.

[0063] Samples are selected according to motion vectors and reference picture indices. Motion vectors and reference picture indices are applied to all color channels; therefore, inter-prediction is primarily described in terms of operation on the PU rather than the PB. Within each category (i.e., intra- and inter-frame prediction), different techniques can be applied to generate the PU. For example, intra-prediction can use values ​​from adjacent rows and columns of previously reconstructed samples, combined with a direction for generating the PU according to a predetermined filtering and generation process. Alternatively, the PU may be described using a small number of parameters. Inter-prediction methods can vary in the number and precision of motion parameters. Motion parameters typically include a reference frame index indicating which reference frame should be used from a list of reference frames, and a spatial transformation for each reference frame, but can include more frames, special frames, or complex affine parameters such as scaling and rotation. Furthermore, a predetermined motion refinement process can be applied to generate high-density motion estimates based on the referenced sample blocks.

[0064] By employing Lagrangian processing or a similar optimization process, both the optimal partitioning of the CTU into CBs (by the block partitioner 310) and the selection of the best prediction mode from multiple possibilities can be chosen. Through the application of the Lagrangian optimization process of candidate modes in the mode selector module 386, the prediction mode with the lowest cost measurement is selected as the "best" mode. The lowest cost mode is the selected prediction mode 388, which is also encoded in the bitstream 115 by the entropy encoder 338. The selection of the prediction mode 388 by the operation of the mode selector module 386 extends to the operation of the block partitioner 310. For example, candidates for the selection of the prediction mode 388 may include modes applicable to a given block, and further, modes applicable to multiple smaller blocks collectively arranged together with the given block. If modes applicable to a given block and smaller collated blocks are included, the process of implicitly selecting candidates is also the process of determining the best hierarchical decomposition of the CTU into CBs.

[0065] In the second stage of operation of the video encoder 114 (referred to as the "encoding" stage), iterations are performed in the video encoder 114 across the selected luma coding tree and the selected chroma coding tree, and thus across each selected CB. In each iteration, the CBs are encoded into the bitstream 115, as will be further described herein.

[0066] The entropy encoder 338 supports both variable-length coding and arithmetic coding of syntax elements. Arithmetic coding is supported using the context-adaptive binary arithmetic coding (CABAC) process. An arithmetic-encoded syntax element consists of a sequence of one or more "bins". A bin, like a bit, has a value of "0" or "1". A bin is not coded into the bitstream 115 as discrete bits. A bin has an associated predicted (or "probability" or "most likely") value and associated probability, known as the "context". When the actual bin to be coded matches the predicted value, the "most probable symbol" (MPS) is coded. Coded the most probable symbol is relatively inexpensive in terms of the bits consumed. When the actual bin to be coded does not match a likely value, the "least probable symbol" (LPS) is coded. Coded the least probable symbol is relatively expensive in terms of the bits consumed. Bin coding techniques enable efficient coding of bins with a biased probability of "0" versus "1". For syntax elements with two possible values ​​(i.e., "flags"), a single bin is sufficient. For syntax elements with many possible values, a series of bins is required.

[0067] The existence of a later bin in a sequence may be determined based on the value of the previous bin in the sequence. Furthermore, each bin can be associated with two or more contexts. The selection of a particular context may depend on the previous bin of the syntax element, the bin values ​​of adjacent syntax elements (i.e., those from adjacent blocks), and so on. Each time a context-encoded bin is encoded, the context selected for that bin (if any) is updated in a way that reflects the new bin value. In this way, binary arithmetic coding schemes are said to be adaptive.

[0068] Furthermore, the video encoder 114 supports bins without context ("bypass bins"). Bypass bins are encoded assuming an equal probability distribution between "0" and "1". Thus, each bin occupies one bit in the bitstream 115. Bypass bins are used when the distribution of values ​​for a particular bin is not biased, as the absence of context saves memory and reduces complexity.

[0069] The entropy encoder 338 encodes the prediction mode 388 using a combination of context coding bins and bypass coding bins. For example, if the prediction mode 388 is an intra-prediction mode, a list of "highest probability modes" is generated in the video encoder 114. The list of highest probability modes is typically a fixed length, such as three or six modes, and may include modes encountered in previous blocks. The context coding bin encodes a flag indicating whether the prediction mode is one of the highest probability modes. If the intra-prediction mode 388 is one of the highest probability modes, further signaling is encoded using the bypass coding bin. The encoded further signaling indicates which highest probability mode corresponds to the intra-prediction mode 388, for example, using a truncated unary bin string. Otherwise, the intra-prediction mode 388 is encoded as a "remaining mode". Encoding as a remaining mode represents intra-prediction modes other than those present in the list of highest probability modes, using an alternative syntax, such as a fixed-length code, which is also encoded using the bypass coding bin.

[0070] The multiplexer module 384 outputs PB320 according to the determined best prediction mode 388, selecting from the tested prediction modes for each candidate CB. The candidate prediction modes do not need to include all possible prediction modes supported by the video encoder 114.

[0071] PB320 is determined and selected, and subtracted from the original sample block by the subtractor 322, resulting in a residual represented by 324 with the lowest coding cost, which undergoes lossy compression. The lossy compression process includes the steps of transformation, quantization, and entropy coding. The forward linear transformation module 326 applies a forward transformation to the residual 324, transforming the residual 324 from the spatial domain to the frequency domain and generating linear transformation coefficients represented by arrow 328. The linear transformation coefficients 328 are passed to the forward quadratic transformation module 330, which performs an unseparable quadratic transformation (NSST) operation to generate transformation coefficients represented by arrow 332. Forward primary transforms are typically separable and usually use the Type II Discrete Cosine Transform (DCT-2) to transform a set of rows, and then a set of columns in each block, although Type VII Discrete Sine Transform (DST-7) and Type VIII Discrete Cosine Transform (DCT-8) may also be available, for example, horizontally for block widths not exceeding 16 samples and vertically for block heights not exceeding 16 samples. Transformations of each set of rows and columns are performed by first applying a one-dimensional transform to each row of the block to produce an intermediate result, and then applying a one-dimensional transform to each column of the intermediate result to produce the final result. Forward quadratic transforms are generally inseparable transforms and are applied only to the residuals of the intra-predicted CU, and may nevertheless be bypassed. Forward quadratic transforms operate on either 16 samples (arranged as upper-left 4x4 subblocks of 328 linear transform coefficients) or 64 samples (arranged as upper-left 8x8 coefficients, arranged as 4 4x4 subblocks of 328 linear transform coefficients). Furthermore, the matrix coefficients for the forward quadratic transformation are selected from multiple sets according to the CU's intra-prediction mode, so that two sets of coefficients are available for use. Using one of the sets of matrix coefficients, i.e., bypassing the forward quadratic transformation, is signaled by the syntax element of "nsst_index" and encoded using truncated unary binarization to represent a value of zero (no quadratic transformation applied), one (first set of selected matrix coefficients), or two (second set of selected matrix coefficients).

[0072] The video encoder 114 may also choose to skip both the primary and secondary transformations, known as the "transformation skip" mode. Skipping transformations is suitable for residual data that lacks sufficient correlation to reduce encoding costs through representations as transformation basis functions. Certain types of content, such as relatively simple computer-generated graphics, may exhibit similar behavior. When the transformation skip mode is used, the transformation coefficients 332 are the same as the residual coefficients 324.

[0073] The conversion coefficients 332 are passed to the quantizer module 334. In module 334, quantization is performed using the "quantization parameter" to produce quantization coefficients represented by arrows 336. The quantization parameter is constant for a given TB and therefore results in uniform scaling for generating residual coefficients for TB. Non-uniform scaling is also possible by applying a "quantization matrix," thereby deriving the scaling coefficient applied to each residual coefficient from a combination of the quantization parameter and the corresponding entry in a scaling matrix, which is typically equal in size to the size of TB. The scaling matrix can be smaller in size than the size of TB, and when applied to TB, the nearest neighbor approach is used to provide the scaling value for each residual coefficient from a scaling matrix smaller in size than the TB size. The quantization coefficients 336 are fed to the entropy encoder 338 for encoding in the bitstream 115. Typically, the quantization coefficients of each TB, which have at least one significant quantization coefficient, are scanned according to a scan pattern to generate an ordered list of values. The scan pattern generally scans the TB as a sequence of 4x4 "subblocks," providing a regular scan operation with a granularity of 4x4 sets of residual coefficients, where the arrangement of subblocks depends on the size of the TB. Furthermore, the prediction mode 388 and the corresponding block partitioning are also encoded in the bitstream 115.

[0074] As described above, the video encoder 114 requires access to a frame representation that corresponds to the frame representation seen by the video decoder 134. Therefore, the quantization coefficient 336 is also dequantized by the inverse quantizer module 340 to produce a reconstructed transformation coefficient represented by arrow 342. The reconstructed transformation coefficient 342 passes through the inverse quadratic transformation module 344 to produce a reconstructed linear transformation coefficient represented by arrow 346. The reconstructed linear transformation coefficient 346 is passed to the inverse linear transformation module 348 to produce a reconstructed residual sample of TU represented by arrow 350. The type of inverse transformation performed by the inverse quadratic transformation module 344 corresponds to the type of forward transformation performed by the forward quadratic transformation module 330. The type of inverse transformation performed by the inverse linear transformation module 348 corresponds to the type of linear transformation performed by the linear transformation module 326. The adder module 352 adds the reconstructed residual sample 350 and PU 320 to produce a reconstructed sample of CU (indicated by arrow 354).

[0075] The reconstructed sample 354 is passed to the reference sample cache 356 and the in-loop filter module 368. The reference sample cache 356 is typically implemented using static RAM on the ASIC (thus avoiding costly off-chip memory access) and provides the minimum sample storage necessary to satisfy dependencies for generating intra-frame PB for subsequent CUs in the frame. The minimum dependencies typically include a "line buffer" of samples along the bottom of the row of the CTU, used by the subsequent row and column buffering of the CTU, the range of which is set by the height of the CTU. The reference sample cache 356 supplies reference samples (indicated by arrow 358) to the reference sample filter 360. The sample filter 360 applies a smoothing operation to generate filtered reference samples (indicated by arrow 362). The filtered reference samples 362 are used by the intra-frame prediction module 364 to generate an intra-prediction block of samples represented by arrow 366. For each candidate intra-prediction mode, the intra-frame prediction module 364 generates a block of samples, i.e., 366.

[0076] The in-loop filter module 368 applies several filtering stages to the reconstructed sample 354. These filtering stages include a “deblocking filter” (DBF) that applies smoothing aligned to the CU boundary to reduce artifacts arising from discontinuities. Another filtering stage present in the in-loop filter module 368 is the “adaptive loop filter” (ALF), which applies a Wiener-based adaptive filter to further reduce distortion. A further available filtering stage in the in-loop filter module 368 is the “sample-adaptive offset” (SAO) filter. The SAO filter works by first classifying the reconstructed sample into one or more categories and then applying an offset at the sample level according to the assigned category.

[0077] The filtered sample, represented by arrow 370, is output from the in-loop filter module 368. The filtered sample 370 is stored in the frame buffer 372. The frame buffer 372 typically has enough capacity to store several pictures (e.g., up to 16) and is therefore stored in memory 206. Because the frame buffer 372 requires a large amount of memory, it is typically not stored using on-chip memory. Therefore, accessing the frame buffer 372 is costly in terms of memory bandwidth. The frame buffer 372 provides the reference frame (represented by arrow 374) to the motion estimation module 376 and the motion compensation module 380.

[0078] The motion estimation module 376 estimates several “motion vectors” (represented as 378), each being a Cartesian space offset from the current CB position and referencing a block in one of the reference frames in the frame buffer 372. A filtered block of reference samples (represented as 382) is generated for each motion vector. The filtered reference samples 382 form further candidate modes available for potential selection by the mode selector 386. Furthermore, for a given CU, the PU 320 may be formed using one reference block ("single prediction") or using two reference blocks ("double prediction"). For the selected motion vectors, the motion compensation module 380 generates the PB 320 according to a filtering process that supports sub-pixel precision within the motion vectors. Thus, the motion estimation module 376 (operating for many candidate motion vectors) can perform a simplified filtering process compared to that of the motion compensation module 380 (operating only for selected candidates) to achieve reduced computational complexity. When the video encoder 114 selects interpretation for CU, the motion vector 378 is encoded into the bitstream 115.

[0079] The video encoder 114 in Figure 3 is described with reference to General-Purpose Video Coding (VVC), but other video coding standards or implementations may also use the processing stages of modules 310-386. Frame data 113 (and bitstream 115) are stored in memory 206, hard disk drive 210, CD-ROM, and Blu-ray disc. TM Alternatively, it may be read from (or written to) other computer-readable storage media. Furthermore, frame data 113 (and bitstream 115) may be received from (or transmitted to) an external source such as a server connected to a communication network 220 or a radio frequency receiver.

[0080] The video decoder 134 is shown in Figure 4. The video decoder 134 in Figure 4 is an example of a general-purpose video coding (VVC) video decoding pipeline, but other video codecs can also be used to perform the processing stages described herein. As shown in Figure 4, the bitstream 133 is input to the video decoder 134. The bitstream 133 is input to memory 206, hard disk drive 210, CD-ROM, Blu-ray disc TM or it can be read from another non-temporary computer-readable storage medium. Alternatively, bitstream 133 may be received from an external source such as a server connected to a communication network 220 or a radio frequency receiver. Bitstream 133 includes encoded syntax elements representing the image frame data to be decoded.

[0081] Bitstream 133 is input to the entropy decoder module 420. The entropy decoder module 420 extracts syntax elements from bitstream 133 by decoding a sequence of "bins" and passes the values ​​of those syntax elements to other modules in the video decoder 134. An example of a syntax element extracted from bitstream 133 is the quantized coefficients 424. The entropy decoder module 420 uses an operational decoding engine to decode each syntax element as a sequence of one or more bins. Each bin can use one or more "contexts" along with a context that describes the probability level used to encode the "1" and "0" values ​​of the bin. If multiple contexts are available for a given bin, a "context modeling" or "context selection" step is performed to select one of the available contexts for decoding the bin. The process of decoding the bins forms a sequential feedback loop. The number of operations in the feedback loop is preferably minimized to enable the entropy decoder 420 to achieve high throughput in bins / second. Context modeling relies on other properties of the bitstream known to the video decoder 134 when selecting the context, i.e., the properties prior to the current bin. For example, the context may be selected based on the quadtree depth of the current CU in the coding tree. The dependency is preferably based on properties known before decoding the bin or determined without requiring lengthy sequential processing.

[0082] The quantized coefficients 424 are input to the inverse quantizer module 428. The inverse quantizer module 428 performs inverse quantization (or "scaling") on the quantized coefficients 424 according to the quantization parameters to produce reconstructed intermediate transformation coefficients represented by arrows 432. If the use of a non-uniform inverse quantization matrix is ​​indicated in bitstream 133, the video decoder 134 reads the quantization matrix from bitstream 133 as a sequence of scaling factors and places the scaling factors into the matrix. Inverse scaling uses the quantization matrix in combination with the quantization parameters to produce reconstructed intermediate transformation coefficients 432. The reconstructed intermediate transformation coefficients 432 are passed to the inverse quadratic transformation module 436, where a quadratic transformation may be applied according to the decoded "nsst_index" syntax element. "nsst_index" is decoded from bitstream 133 by the entropy decoder 420 under the execution of processor 205. The inverse quadratic transformation module 436 produces reconstructed transformation coefficients 440.

[0083] The reconstructed transformation coefficients 440 are passed to the inverse linear transformation module 444. Module 444 transforms the coefficients back from the frequency domain to the spatial domain. The result of the operation of module 444 is a block of residual samples represented by arrow 448. The block of residual samples 448 is equal in size to the corresponding CU. The type of inverse linear transformation can be Type II discrete cosine transform (DCT-2), Type VII discrete sine transform (DCT-7), Type VIII discrete cosine transform (DCT-8), or "transformation skip" mode. The use of transformation skip mode is signaled by a transformation skip flag that is decoded from bitstream 133 or otherwise inferred. When transformation skip mode is used, the residual samples 448 are the same as the reconstructed transformation coefficients 440.

[0084] The residual sample 448 is supplied to the adder module 450. In the adder module 450, the residual sample 448 is added to the decoded PB (represented as 452) to generate a reconstructed sample block represented by arrow 456. The reconstructed sample 456 is supplied to the reconstructed sample cache 460 and the in-loop filtering module 488. The in-loop filtering module 488 generates a reconstructed block of frame samples represented as 492. The frame sample 492 is written to the frame buffer 496.

[0085] The reconstructed sample cache 460 operates similarly to the reconstructed sample cache 356 of the video encoder 114. The reconstructed sample cache 460 provides storage for the reconstructed samples necessary to intra-predict subsequent CBs without memory 206 (for example, by using data 232, which is typically on-chip memory, instead). The reference sample represented by arrow 464 is obtained from the reconstructed sample cache 460 and fed to the reference sample filter 468 to generate the filtered reference sample indicated by arrow 472. The filtered reference sample 472 is fed to the intra-frame prediction module 476. The module 476 generates a block of intra-prediction samples represented by arrow 480 according to the intra-prediction mode parameter 458, which is signaled in the bitstream 133 and decoded by the entropy decoder 420.

[0086] If the prediction mode of the CB is shown to be intra-prediction in bitstream 133, the intra-prediction sample 480 forms the decoded PB 452 via the multiplexer module 484. Intra-prediction generates a predicted block (PB) of samples, i.e., a block in one color component derived using “adjacent samples” in the same color component. Adjacent samples are samples adjacent to the current block and have already been reconstructed because they precede it in the block decoding order. When luma and chroma blocks are juxtaposed, luma and chroma blocks can use different intra-prediction modes. However, two chroma channels each share the same intra-prediction mode.

[0087] Lumablock's intra-prediction consists of four types. "DC intra-prediction" involves inputting (populating) a single value representing the mean of adjacent samples into the PB. "Planar intra-prediction" involves inputting samples into the PB according to a plane, using vertical and horizontal slopes and a DC offset derived from adjacent samples. "Angular intra-prediction" involves inputting adjacent samples into the PB that have been filtered and propagated across the PB in a specific direction (or "angle"). In VVC, the PB can be selected from up to 65 angles, and rectangular blocks can utilize different angles that are not available to square blocks. "Matrix intra-prediction" involves inputting the PB by multiplying a reduced set of adjacent samples by one of the many available matrices available to the video decoder 134. The reduced set of adjacent samples is generated by filtering and subsampling adjacent samples. Then, the reduced set of predicted samples is generated by multiplying the reduced set of samples by a matrix and adding an offset vector. The matrix and associated offset vectors are selected from a number of possible matrices depending on the size of the PB, and the specific selection of the matrix and offset vectors is indicated by the “MIP mode” syntax element. For example, there are 11 MIP modes for PBs larger than 8x8, while there are 19 MIP modes for 8x8 sized PBs. Finally, the PB generated by matrix intraprediction is input from a reduced set of predicted samples obtained by interpolation.

[0088] A fifth type of intra-prediction is available for chroma PB, where the PB is generated from juxtaposed luma reconstituted samples according to a "cross-component linear model" (CCLM) mode. Three different CCLM modes are available, each using a different model derived from adjacent luma and chroma samples. The derived models are then used to generate blocks of chroma PB samples from the collated luma samples.

[0089] When the prediction mode of the CB is indicated to be inter-prediction in bitstream 133, the motion compensation module 434 generates a block of inter-predicted samples, represented as 438, using motion vectors and reference frame indices to select and filter a block of samples 498 from the frame buffer 496. The block of samples 498 is obtained from previously decoded frames stored in the frame buffer 496. In the case of bi-prediction, two sample blocks are generated and blended together to produce samples for the decoded PB 452. The frame buffer 496 is input with filtered block data 492 from the in-loop filtering module 488. Similar to the in-loop filtering module 368 of the video encoder 114, the in-loop filtering module 488 applies one of the DBF, ALF, or SAO filtering operations. Generally, motion vectors are applied to both the luma and chroma channels, but the filtering processes for subsample interpolation luma and chroma channels are different. The frame buffer 496 outputs the decoded video samples 135.

[0090] Figure 5 is a schematic block diagram showing a set 500 of available divisions or splits of a region into one or more subregions within the tree structure of a general-purpose video encoding. The divisions shown in set 500 are available to the block partitioner 310 of the encoder 114 to split each CTU into one or more CUs or CBs according to the coding tree, as determined by Lagrangian optimization as described with reference to Figure 3.

[0091] Figure 500 shows that only the square region is divided into other, possibly non-square, sub-regions, but it should be understood that Figure 500 shows potential divisions, but the containing region does not need to be square. If the containing region is non-square, the dimensions of the blocks resulting from the division are scaled according to the aspect ratio of the containing block. When a region can no longer be divided, i.e., at a leaf node of the coding tree, the CU occupies that region. A particular sub-division of a CTU into one or more CUs by the block partitioner 310 is called the "coding tree" of the CTU.

[0092] The process of subdividing a region into subregions must terminate when the resulting subregions reach a minimum CU size. In addition to restricting CUs to a predetermined minimum size, e.g., prohibiting block regions smaller than 16 samples, CUs are constrained to have a minimum width or height of 4. Other minimums are also possible with respect to both width and height, or width or height. The subdivision process can also terminate before the deepest level of decomposition, resulting in CUs larger than the minimum CU size. It is possible that no decomposition occurs, resulting in a single CU occupying the entire CTU. A single CU occupying the entire CTU is the largest available coding unit size. By using a subsampled chroma format such as 4:2:0, the configuration of the video encoder 114 and video decoder 134 can terminate the decomposition of regions in the chroma channel earlier than in the chroma channel.

[0093] Leaf nodes in a coding tree have CUs that do not have further subdivisions. For example, leaf node 510 contains one CU. Non-leaf nodes in a coding tree can be leaf nodes that have subdivisions into two or more further nodes, each forming one CU, or non-leaf nodes that have further subdivisions into smaller regions. In each leaf node of a coding tree, there is one coding block for each color channel. A subdivision that terminates at the same depth for both luma and chroma results in three juxtaposed CBs. A subdivision that terminates at a luma depth deeper than the chroma depth results in multiple luma CBs being juxtaposed with the chroma channel CBs.

[0094] The quadtree partition 512 divides the containment region into four equally sized regions, as shown in Figure 5. Compared to HEVC, General-Purpose Video Coding (VVC) achieves greater flexibility by adding horizontal 2-part partitions 514 and vertical 2-part partitions 516. Each of partitions 514 and 516 divides the containment region into two equally sized regions. The partitions are along the horizontal boundary (514) or vertical boundary (516) within the containment block.

[0095] Further flexibility in general-purpose video coding is achieved by adding horizontal 3-split 518 and vertical 3-split 520. 3-split 518 and 520 divide a block into three regions bounded either horizontally (518) or vertically (520) along 1 / 4 and 3 / 4 of the width or height of the containing region. The combination of quadtree, binary tree, and ternary tree is called "QTBTTT". The root of the tree contains zero or more quadtree splits (the "QT" sections of the tree). Once the QT sections end, zero or more binary or 3-splits (the "multitree" or "MT" sections of the tree) occur, finally ending with a CB or CU at the leaf node of the tree. If the tree describes all color channels, the tree leaf node is the CU. If the tree describes luma or chroma channels, the tree leaf node is the CB.

[0096] Compared to HEVC, which supports only quadtrees and therefore only square blocks, QTBTTT results in more possible CU sizes, especially when considering the possible recursive application of binary and / or ternary tree partitions. The possibility of abnormal (non-square) block sizes can be reduced by constraining partitioning options to eliminate partitions where the block width or height is less than 4 samples or not a multiple of 4 samples. Generally, this constraint is applied when considering luma samples. However, in the configuration described, the constraint can be applied separately to blocks for chroma channels. Applying constraints to partitioning options for chroma channels means that, for example, if the frame data is in a 4:2:0 chroma format or a 4:2:2 chroma format, the minimum block size may differ between luma and chroma. Each partition generates sub-regions with respect to this inclusion region where the side dimensions are unchanged, halved, or quartered. And since the CTU size is a power of 2, the side dimensions of all CUs are also powers of 2.

[0097] Figure 6 is a schematic flowchart showing the data flow 600 of a QTBTTT (or “coding tree”) structure used in general-purpose video coding. The QTBTTT structure is used for each CTU to define the partitioning of the CTU into one or more CUs. The QTBTTT structure for each CTU is determined by the block partitioner 310 in the video encoder 114 and encoded into the bitstream 115, or decoded from the bitstream 133 by the entropy decoder 420 in the video decoder 134. The data flow 600 further characterizes the acceptable combinations available to the block partitioner 310 for partitioning the CTU into one or more CUs according to the partitioning shown in Figure 5.

[0098] Starting from the top level of the hierarchy, i.e., the CTU, zero or more quadtree partitions are performed first. Specifically, a quadtree (QT) partition decision 610 is made by the block partitioner 310. A decision at 610 that returns a "1" symbol indicates a decision to split the current node into four subnodes according to quadtree partition 512. This results in four new nodes, such as 620, and for each new node, the process returns to the QT partition decision 610. Each new node is considered in raster (or Z-scan) order. Alternatively, if the QT partition decision 610 indicates that no further partitions should be performed (returning a "0" symbol), quadtree partitioning stops and multitree (MT) partitioning is then considered.

[0099] First, the block partitioner 310 performs an MT partition decision 612. 612 indicates a decision to perform an MT partition. Returning the symbol "0" in decision 612 indicates that no further partitioning of the node into subnodes will be performed. If no further partitioning of the node is performed, the node is a leaf node in the coding tree and corresponds to a CU. The leaf node is output in 622. Alternatively, if MT partition 612 indicates a decision to perform an MT partition (returning the symbol "1"), the block partitioner 310 proceeds to the direction decision 614.

[0100] Direction determination 614 indicates the direction of the MT partition as either horizontal ("H" or "0") or vertical ("V" or "1"). Block partitioner 310 proceeds to determination 616 if determination 614 returns "0", indicating the horizontal direction. Block partitioner 310 proceeds to determination 618 if determination 614 returns "1", indicating the vertical direction.

[0101] In decisions 616 and 618, the number of partitions in the MT partition is indicated as either two (2 partitions or "BT" nodes) or three (3 partitions or "TT") in the BT / TT partition. That is, the BT / TT partition decision 616 is performed by the block partitioner 310 when the indicated direction from 614 is horizontal, and the BT / TT partition decision 618 is performed by the block partitioner 310 when the indicated direction from 614 is vertical.

[0102] The BT / TT partitioning decision 616 indicates whether the horizontal partitioning is a 2-part partitioning 514, indicated by returning "0", or a 3-part partitioning 518, indicated by returning "1". If the BT / TT partitioning decision 616 indicates a 2-part partitioning, in the HBT CTU node generation step 625, two nodes are generated by the block partitioner 310 according to the horizontal 2-part partitioning 514. If the BT / TT partitioning 616 indicates a 3-part partitioning, in the HTT CTU node generation step 626, three nodes are generated by the block partitioner 310 according to the horizontal 3-part partitioning 518.

[0103] The BT / TT partition decision 618 indicates whether the vertical partition is a 2-part partition 516, indicated by returning "0", or a 3-part partition 520, indicated by returning "1". If the BT / TT partition 618 indicates a 2-part partition, the VBT CTU node generation step 627 generates two nodes by the block partitioner 310 according to the vertical 2-part partition 516. If the BT / TT partition 618 indicates a 3-part partition, the VTT CTU node generation step 628 generates three nodes by the block partitioner 310 according to the vertical 3-part partition 520. For each node resulting from steps 625-628, the recursion of the data flow 600, returning to the MT partition decision 612, is applied in a left-to-right or top-to-bottom order, depending on the direction 614. As a result, binary and ternary tree partitions can be applied to generate CUs of various sizes.

[0104] Figures 7A and 7B provide examples 700 of the division of CTU 710 into several CUs or CBs. An example of CU 712 is shown in Figure 7A. Figure 7A shows the spatial arrangement of CUs in CTU 710. The division example 700 is also shown in Figure 7B as a coding tree 720.

[0105] In Figure 7A, in each non-leaf node within the CTU 710, for example nodes 714, 716, and 718, the contained nodes (which may be further divided or CUs) are scanned or traversed in "Z-order" to create a list of nodes, which are represented as columns in the coding tree 720. In the case of a quadtree partition, the Z-order scan proceeds from top left to right, followed by bottom left to right. In the case of horizontal and vertical partitions, the Z-order scan (traversal) is simplified to a top-down scan and a left-to-right scan, respectively. The coding tree 720 in Figure 7B lists all nodes and CUs according to the applied scan order. Each partition generates a list of two, three, or four new nodes at the next level of the tree until a leaf node (CU) is reached.

[0106] The image is decomposed into CTUs, and further decomposed into CUs by the block partitioner 310. Using the CUs to generate each residual block (324) as described with reference to Figure 3, the residual blocks are forward-converted by the video encoder 114. An equivalent inverse conversion process is performed in the video decoder 134 to obtain TB from the bitstream 133.

[0107] In the video encoder 114, the quantized coefficients 336 can be rearranged into a one-dimensional list by performing a two-level backward diagonal scan. Similarly, in the video decoder 134, the quantized coefficients 424 can be rearranged from the one-dimensional list into a two-dimensional collection of subblocks by the same two-level backward diagonal scan.

[0108] Figure 8A shows a two-level backward diagonal scan 810 of an exemplary 8×8 TB800. Scan 810 is shown progressing from the lower right residual coefficient location of TB800 back to the upper left (DC) residual coefficient location of TB800. The path of scan 810 is in 4×4 regions known as subblocks, progressing from one subblock to the next. For TBs with a width or height of 2, 2×2, 2×8, or 8×2 subblock sizes are available. Scanning within a particular subblock is performed according to an "encoded subblock flag," or the subblock is skipped. If a scan of a subblock is skipped, it is assumed that all residual coefficients within the subblock have a value of zero. Although scan 810 is shown starting from the lower right residual coefficient location of TB800, for a given set of residual coefficients, the scan starts from the location of the "last significant coefficient," which is the "last" coefficient when the order of coefficients is considered to proceed from the DC coefficient instead of the scan order.

[0109] Figure 8B shows an exemplary 8x8 TB850 alternative two-level forward diagonal scan 860 used when TSRC processing is selected. When TSRC processing is used in the video encoder 114, the quantized coefficients 336 are rearranged by scan 860 into a one-dimensional list. Similarly, when TSRC processing is used for the current TB in the video decoder 134, the quantized coefficients 424 are rearranged by scan 860 from the one-dimensional list into a two-dimensional collection of subblocks. Scan 860 is shown progressing from the top-left (DC) residual coefficient position of TB850 to the bottom-right residual coefficient position of TB850. Unlike scan 810, scan 860 does not end at the “last significant coefficient”.

[0110] Figures 8A and 8B show scan patterns typically used in VVC. The example described herein uses scan pattern 810 to encode residual coefficients transformed by module 326, and scan pattern 860 is used for transformed blocks that were skipped. However, some implementations may use other scan patterns.

[0111] As described above, the conversion coefficient 332 is the same as the residual coefficient 324 when the conversion skip mode is used. Therefore, the conversion coefficient 332 may be referred to as the residual coefficient regardless of whether the conversion skip mode is selected or not. When lossless coding is desired, the video encoder 114 selects conversion skip for the current TB and signals a conversion skip flag with the value "TRUE" to the bitstream 133. The residual coefficient 332 associated with the current TB is coded into the bitstream 133. Two residual coding processes are available: the "Normal Residual Coding" (RRC) process and the "Conversion Skip Residual Coding" (TSRC) process. In the normal operation of the video encoder 114, if conversion skip is selected (the conversion skip flag has the value "TRUE"), the TSRC process is selected; otherwise (the conversion skip flag has the value "FALSE"), the RRC process is selected. However, it is typically undesirable that the coding of the residual coefficient 332 is handled exclusively by TSRC in the case of lossless coding.

[0112] In one configuration of the video encoder 114, a TSRC disable flag is signaled within the bitstream 133. The TSRC disable flag can be signaled at a relatively high level, such as once per sequence or once per picture, to reduce the relative cost of signaling the TSRC disable flag. High-level syntax elements are typically grouped into parameter sets, such as a “sequence parameter set” (SPS) for sequence-level flags and a “picture parameter set” (PPS) for parameter-level flags. The TSRC disable flag may be set to “TRUE” if the video data 113 belongs to a class that is not considered suitable for encoding by TSRC processing (in terms of encoding loss and feature reproduction). An example of video data not suitable for encoding by TSRC is natural scene content. The TSRC disable flag may be set to “FALSE” if the video data 113 belongs to a class that is considered suitable for encoding by TSRC processing. Video data suitable for encoding by TSRC processing includes artificial screen content.

[0113] If the video encoder 114 selects to skip conversion for the current TB and the TSRC disabled flag is set to "TRUE", the residual coefficient 332 is encoded into the bitstream 133 using the RRC process. Similarly, if the video decoder 134 determines that conversion skip is used for the current TB and the TSRC disabled flag is set to "TRUE", the residual coefficient 432 is decoded from the bitstream 133 using the RRC process.

[0114] Figure 9 shows a method 900 for encoding a transform block of residual coefficients 332 using an RRC process. Method 900 can be implemented by a configured FPGA, ASIC, or ASSP or other device. Furthermore, method 900 may be implemented by a video encoder 114 under the operation of a processor 205. Thus, method 900 can be implemented as a module of software 233 stored in a computer-readable storage medium and / or memory 206.

[0115] Method 900 is performed in several configurations by the video encoder 114, receiving the residual coefficient 332 and then the quantizer 334, followed by the entropy encoder 338. Method 900 begins with a coefficient quantization step 910.

[0116] In the coefficient quantization step 910, step 910 invokes method 1100, which is described below in relation to Figure 11. Method 1100 can be implemented by a device such as a configured FPGA, ASIC, or ASSP. Furthermore, method 1100 may be implemented by the video encoder 114 under the control of the processor 205. Thus, method 1100 can be implemented as a module of software 233 stored in a computer-readable storage medium and / or memory 206. Method 1100 quantizes the residual coefficient 332 to generate the quantized coefficient 336. Under the control of the processor 205, method 900 proceeds from step 910 to the final position coding step 920.

[0117] In the last position coding step 920, the video encoder 114 finds the position of the last significant coefficient within the quantized coefficients 336 for the transformed block of residual coefficients 332). The last significant coefficient is determined in relation to the forward direction of the appropriate scan pattern, for example, the direction of the two-level forward diagonal scan 860. A quantized coefficient is significant if the coefficient has any value other than zero. The position of the last significant coefficient is written to the bitstream 133. Under the control of the processor 205, method 900 proceeds from step 920 to the state initialization step 930.

[0118] In the state initialization step 930, the quantizer state Qstate is set to the value zero. Furthermore, in step 930, a subblock containing the last effective coefficient is selected. Under the control of the processor 205, method 900 proceeds from step 930 to the coding subblock flag determination step 940.

[0119] The explanation here refers to the status of some flags as "TRUE" or "FALSE". Setting a flag to "TRUE" means that the requirement is met or the mode is selected. Setting a flag to "FALSE" means that the requirement is not met or the mode is not selected.

[0120] In the encoding subblock flag determination step 940, the video encoder 114 determines and sets the encoding subblock flag. If the currently selected subblock is the first subblock selected in the state initialization step 930, the encoding subblock flag is set to "TRUE", but it is not encoded into the bitstream 133. If the currently selected subblock is identified as the last subblock, as described later in relation to the last subblock test 970, the encoded subblock flag is set to "TRUE", but it is not encoded into the bitstream 133.

[0121] Otherwise, the video encoder 114 encodes the encoded subblock flag into the bitstream 133 by setting the encoded subblock flag to "TRUE" if (i) there is at least one significance coefficient among the 4x4 quantization coefficients belonging to the selected subblock, or (ii) there is no significance coefficient. The method 900 proceeds from step 940 to the encoded subblock flag test step 950 under the control of the processor 205.

[0122] In the coded subblock flag test step 950, method 900 determines the value of the coded subblock flag. If the coded subblock flag is set to "TRUE", method 900 proceeds to the subblock coding step 960. Otherwise, if the coded subblock flag is set to "FALSE", method 900 proceeds to the final subblock test step 970.

[0123] In the subblock coding step 960, the entropy encoder 338 encodes the quantization coefficients in the selected subblock into the bitstream 133. Step 960 invokes method 1300, which is described below in relation to Figure 13. Method 900 proceeds from step 960 to the final subblock test 970 under the control of the processor.

[0124] In the final subblock test 970, method 900 operates to determine whether the selected subblock is the last subblock in the current transformation block. If the currently selected subblock is the top-left subblock of the transformation block, step 900 returns to "YES" and method 900 terminates. Otherwise, if the currently selected subblock is not the top-left subblock of the transformation block, step 970 returns to "NO" and method 900 proceeds to step 980 to select the next subblock.

[0125] In step 980, the next subblock is selected. The next subblock in the backward diagonal scan order 810 is selected. Method 900 proceeds from step 980 to step 940, which determines the encoded subblock flag of the selected subblock.

[0126] Figure 10 shows a method 1000 for decoding a transform block with residual coefficients 432 by an RRC process. Method 1000 can be implemented by a configured FPGA, ASIC, or ASSP or other device. Furthermore, method 1000 may be implemented by a video decoder 134 under the operation of a processor 205. Thus, method 1000 can be implemented as a module of software 233 stored in a computer-readable storage medium and / or memory 206.

[0127] Method 1000 is carried out in several configurations by the video encoder 134 in the entropy decoder 420 and in the inverse quantizer module 428 upon reception of the bitstream 133. Method 1000 begins with the last position decoding step 1010.

[0128] In the last position decoding step 1010, the last significant coefficient position of the transformed block of residual coefficients 432 is decoded from the bitstream 133. Under the control of the processor 205, method 1000 proceeds from step 1010 to the state initialization step 1020.

[0129] In the state initialization step 1020, the video decoder 134 initializes the quantizer state Qstate to the value 0. Furthermore, in step 1020, a subblock containing the last significance coefficient position is selected. Under the control of the processor 205, method 1000 proceeds from step 1020 to the coding subblock flag determination step 1030.

[0130] In the encoded subblock flag determination step 1030, the video decoder 134 determines the encoded subblock flag. If the currently selected subblock is the first subblock selected in the state initialization step 1020, the encoded subblock flag is set to "TRUE" (i.e., the encoded subblock flag is inferred to be "TRUE"). If the currently selected subblock is identified as the last subblock in the final subblock test 1060 as described below, the encoded subblock flag is inferred to be "TRUE". Otherwise, the video decoder 134 decodes the encoded subblock flag from the bitstream 133. Method 1000 proceeds from step 1030 to the encoded subblock flag test 1040 under the control of the processor 205.

[0131] In the coded subblock flag test 1040, method 1000 tests the value of the coded subblock flag determined in step 1030. If method 1000 determines in step 1040 that the coded subblock flag has a value of "TRUE", it proceeds to the subblock decoding step 1050. Otherwise, if it determines in step 1040 that the coded subblock flag has a value of "FALSE", all quantized coefficients in the currently selected subblock are assigned a value of zero, and method 1000 proceeds to the final subblock test 1060.

[0132] In the subblock decoding step 1050, the entropy decoder 420 decodes the quantization coefficients of the selected subblock from the bitstream 133. Step 1050 invokes method 1400, which is described below in relation to Figure 14. Method 1000 proceeds to the final subblock test 1060 under the control of processor 205.

[0133] In the final subblock test 1060, if the currently selected subblock is the top-left subblock of the transform block, step 1060 returns to "YES" and method 1000 proceeds to coefficient scaling step 1080. Otherwise, step 1060 returns to "NO" and method 1000 proceeds to step 1070 to select the next subblock.

[0134] In step 1070, where the next subblock is selected, the next subblock in the backward diagonal scan order 810 is selected. Method 1000 proceeds from step 1070 to the coding subblock flag determination step 1030 under the control of processor 205.

[0135] In coefficient scaling step 1080, the inverse quantizer module 428 applies scaling to the quantization coefficients 424 to generate reconstructed residual coefficients 432. The subblock is decoded by reconstructing the residual coefficients of the subblock using the decoded sign bits. Step 1080 invokes method 1200, which is described below in relation to Figure 12. Method 1000 ends with the execution of step 1080.

[0136] Figure 11 shows method 1100 for quantizing the residual coefficient 332 of the transformation block and generating the quantization coefficient 336. Method 1100 is performed on TB in step 910 of method 900. Method 1100 begins with DQ test 1110.

[0137] In the DQ test 1110, the video encoder 114 determines whether dependent quantization is used to quantize the residual coefficient 332. The video encoder 114 checks the value of the enabled dependent quantization flag, which is communicated as high-level syntax in the bitstream 133. The enabled dependent quantization flag determines whether dependent quantization is permitted within the range of the flag. For example, the sequence-level dependent quantization flag determines whether dependent quantization is permitted when encoding an entire video sequence. The picture-level dependent quantization flag determines whether dependent quantization is permitted when encoding the current picture and takes precedence over the value of the sequence-level dependent quantization flag. If the enabled dependent quantization flag is "FALSE", step 1110 returns "NO", and method 1100 proceeds to the scalar quantization step 1120.

[0138] In one configuration of DQ test 1110, when the valid dependent quantization flag is "TRUE", video encoder 114 also checks the value of the conversion skip flag for the current TB. When the valid dependent quantization flag is "TRUE" and the conversion skip flag is "TRUE", step 1110 returns "NO" and method 1100 proceeds to scalar quantization step 1120. Otherwise, when the valid dependent quantization flag is "TRUE" and the conversion skip flag is "FALSE", step 1110 returns "YES" and method 1100 proceeds to dependent quantization step 1130.

[0139] In another configuration of DQ test 1110, when the valid dependent quantization flag is "TRUE", video encoder 114 also checks the value of the TSRC invalid flag. When the valid dependent quantization flag is "TRUE" and the TSRC invalid flag is "TRUE", step 1110 returns "NO" and method 1100 proceeds to scalar quantization step 1120. Otherwise, when the valid dependent quantization flag is "TRUE" and the TSRC invalid flag is "FALSE", step 1110 returns "YES" and method 1100 proceeds to dependent quantization step 1130.

[0140] In yet another configuration of DQ test 1110, when the valid dependent quantization flag is "TRUE", video encoder 114 also checks the value of the quantization parameter (QP) of the current TB. QP indicates the degree of quantization applied to the residual coefficients 332. QP is determined as QP = QP i + 6*(BD - 8), from an initial QP i and an offset that depends on the bit depth BD of video encoder 114. For example, when QP i is 4 and the bit depth is 8, QP is determined to be 4. QP iIf QP is -8 and the bit depth is 10, then QP is determined to be 4. Typically, a QP of 4 indicates that the residual coefficients are not quantized and therefore reversible operation is possible. However, higher values ​​of QP can still achieve reversible operation. For example, if video data 113 was originally captured at a bit depth of 8 but is fed to the video encoder 114 at a higher bit depth, then a higher QP will allow for reversible operation. For example, if video data 113 was captured at a bit depth of 8 but is fed to the video encoder 114 at a bit depth of 10, then a QP of 4, 10, or 16 will allow for reversible operation. A QP at which reversible operation is possible may be indicated by the minimum QP for the transformation skip block, which is notified by the high-level syntax parameter set. If the effective dependent quantization flag is "TRUE" and QP is 4 (or any value indicating reversible operation), step 1110 returns "NO" and method 1100 proceeds to the scalar quantization step 1120. If not, and the effective dependent quantization flag is "TRUE" and QP is not 4 (or a similar value indicating a reversible operation), step 1110 returns "YES" and method 1100 proceeds to dependent quantization step 1130. In the scalar quantization step 1120, the residual coefficient 332 is expressed as r[n]. Then, the quantization coefficient q[n] is generated by quantizing the residual coefficient r[n] according to equation (1) below. q[n]=(k*r[n]+offset)>>qbits (1) In equation (1), k is the scaling factor, qbits is the coarse quantization factor, and offset controls the placement of the quantization threshold. k, qbits, and offset are determined based on the current quantization parameter values ​​of TB. For example, when QP is 4, k=1, qbits=0, and offset=0. Then, when QP is 4, q[n]=r[n], and no loss occurs in the scalar quantization step. Method 1100 proceeds from step 1120 to SBH test 1140 under the control of processor 205.

[0141] In dependent quantization step 1130, each residual coefficient r[n] can be quantized by one of several scalar quantizers. For the same QP, the scalar quantizers have the same number of quantization divisions, but their quantization thresholds are offset relative to each other. The scalar quantizer for a given residual coefficient r[n] depends on the current quantizer state Qstate, which is updated for each coefficient, and on the parity (least significant bit) of the resulting q[n]. Due to the dependence on previous states, the optimal quantization result is not determined on a coefficient-by-coefficient basis. One efficient way to determine the optimal quantization result is to construct a "trellis" of possible quantization states at each coefficient position. The optimal quantization result can be found by equivalently finding the best path through the trellis. The optimal trellis path can be determined by applying the Viterbi algorithm. Method 1100 ends with the execution of step 1130.

[0142] In the SBH test 1140, the video encoder 114 determines whether sign bit hiding is used to modify the quantization coefficient q[n] before encoding the TB coefficients. The video encoder 114 checks the value of the effective sign bit hiding flag. The effective sign bit hiding flag is notified as high-level syntax in the bitstream 133. For example, the effective sign bit hiding flag may be notified in the picture header. If the effective dependent quantization flag is "TRUE", the effective sign bit hiding flag is implicitly "FALSE". If the effective sign bit hiding flag is "FALSE", step 1140 returns "NO", and method 1100 terminates.

[0143] In one configuration of the SBH test 1140, the determination depends on the value of the valid sign bit hiding flag and the value of the conversion skip flag for the current TB. If the valid sign bit hiding flag is "TRUE", the video encoder 114 also checks the value of the conversion skip flag for the current TB. If the valid sign bit hiding flag is "TRUE" and the conversion skip flag is "TRUE", step 1140 returns "NO" and method 1100 terminates. Otherwise, if the valid sign bit hiding flag is "TRUE" and the conversion skip flag is "FALSE", step 1140 returns "YES" and method 1100 proceeds to parity adjustment step 1150.

[0144] In another configuration of the SBH test 1140, the determination depends on the value of the valid sign bit hiding flag and the value of the TSRC invalid flag. If the valid sign bit hiding flag is "TRUE", the video encoder 114 also checks the value of the TSRC invalid flag. If the valid sign bit hiding flag is "TRUE" and the TSRC invalid flag is "TRUE", step 1140 returns "NO" and method 1100 terminates. Otherwise, if the valid sign bit hiding flag is "TRUE" and the TSRC invalid flag is "FALSE", step 1140 returns "YES" and method 1100 proceeds to parity adjustment step 1150.

[0145] In yet another configuration of the SBH test 1140, the determination depends on the value of the effective sign bit hiding flag and the value of the quantization parameter (QP) for the current TB. If the effective sign bit hiding flag is "TRUE", the video encoder 114 also checks the value of QP for the current TB. If the effective sign bit hiding flag is "TRUE" and QP is 4 (or any value indicating a reversible operation), step 1140 returns "NO" and method 1100 terminates. Otherwise, if the effective sign bit hiding flag is "TRUE" and QP is not 4 (or a similar value indicating a reversible operation), step 1140 returns "YES" and method 1100 proceeds to parity adjustment step 1150.

[0146] In parity adjustment step 1150, the video encoder 114 checks the positions of the first and last significance coefficients for each subblock in the current TB. If the difference between the first and last significance positions of a subblock is greater than a threshold (typically 3), sign bit hiding is used for that subblock. For each subblock in which sign bit hiding is used, the video encoder 114 checks the sign of the first significance coefficient in the subblock and adjusts the parity of the coefficients in the subblock accordingly. The parity of a coefficient is zero if the coefficient is even and 1 if the coefficient is odd. The sum of the parity of multiple coefficients is zero if the number of odd coefficients is odd and 1 if the number of odd coefficients is even. If the sign of the first significance coefficient in a subblock is positive, the coefficients in the subblock are adjusted so that the sum of parity is zero. If the sign of the first significance coefficient in a subblock is negative, the coefficients in the subblock are adjusted so that the sum of parity is 1. Method 1100 terminates after step 1150 is performed.

[0147] Figure 12 shows Method 1200 for applying scaling to the quantization coefficient 424 to generate the reconstructed residual coefficient 432. Method 1200 can be implemented by a device such as a configured FPGA, ASIC, or ASSP. Furthermore, Method 1200 may be implemented by a video decoder 134 under the execution of a processor 205. Thus, Method 1200 can be implemented as a module of software 233 stored in a computer-readable storage medium and / or memory 206. Method 1200 is implemented in step 1080 of Method 1000. Method 1200 begins with a DQ test 1210.

[0148] In the DQ test 1210, the video decoder 134 determines whether dependent quantization is used to dequantize the quantization coefficient 424. The video decoder 134 checks the value of the effective dependent quantization flag, which may be decoded from the bitstream 133 or inferred based on the value of other high-level syntax flags. If the effective dependent quantization flag is "FALSE", step 1210 returns "NO", and method 1200 proceeds to the descalar quantization step 1220.

[0149] In one configuration of DQ test 1210, if the effective dependent quantization flag is "TRUE", the video decoder 134 also checks the value of the current TB conversion skip flag. If the effective dependent quantization flag is "TRUE" and the conversion skip flag is "TRUE", step 1210 returns "NO", and method 1100 proceeds to the inverse scalar quantization step 1220. Otherwise, if the effective dependent quantization flag is "TRUE" and the conversion skip flag is "FALSE", step 1210 returns "YES", and method 1100 proceeds to the inverse dependent quantization step 1230.

[0150] In an alternative configuration of DQ test 1210, if the enabled-dependent quantization flag is "TRUE", the video decoder 134 also checks the value of the TSRC disabled flag. If the enabled-dependent quantization flag is "TRUE" and the TSRC disabled flag is "TRUE", step 1210 returns "NO", and method 1100 proceeds to the inverse scalar quantization step 1220. Otherwise, if the enabled-dependent quantization flag is "TRUE" and the TSRC disabled flag is "FALSE", step 1210 returns "YES", and method 1100 proceeds to the inverse-dependent quantization step 1230.

[0151] In yet another configuration of DQ test 1210, if the effective dependent quantization flag is "TRUE", the video decoder 134 also checks the value of the quantization parameter (QP) of the current TB. If the effective dependent quantization flag is "TRUE" and QP is 4 (or any value indicating a reversible operation), step 1210 returns "NO", and method 1100 proceeds to the inverse scalar quantization step 1220. Otherwise, if the effective dependent quantization flag is "TRUE" and QP is not 4 (or a similar value indicating a reversible operation), step 1210 returns "YES", and method 1100 proceeds to the inverse dependent quantization step 1230.

[0152] In the inverse scalar quantization step 1220, the video decoder 134 scales the quantization coefficient 424 to produce the reconstructed residual coefficient 432. The quantization coefficient 424 is represented as q[n]. The reconstructed residual coefficient r[n] is produced by scaling the quantization coefficient q[n] according to equation (2) below.

[0153]

number

[0154] In equation (2), s is a scaling factor determined based on the value of QP for the current TB. For example, if QP is 4, then s=1 and r[n]=q[n]. Method 1200 ends with the execution of step 1220.

[0155] In the inverse-dependent quantization step 1230, the video decoder 134 applies inverse-dependent quantization to the quantization coefficients 424 to generate the reconstructed residual coefficients 432. The quantizer state Qstate is initially reset to zero. The quantization coefficients 424 are represented by q[n]. Each coefficient position n is visited in the backward diagonal scan order 810, and each reconstructed residual coefficient r[n] is calculated according to equation (3).

[0156]

number

[0157] In equation (3), s is the scaling factor determined based on the value of QP for the current TB.

[0158] After each reconstruction residual coefficient r[n] is calculated, the quantizer state is updated based on the parity of q[n] according to Table 1.

[0159] [Table 1]

[0160] Method 1200 ends with the execution of step 1230.

[0161] To take advantage of the statistical properties of the quantization coefficient 336, the quantization coefficient is binarized into several syntax elements by the video encoder 114 (typically by the entropy encoder 338) before encoding. For example, since the quantization coefficient 336 often has a value of zero, one syntax element is a significance flag, which is set to "FALSE" for quantization coefficients with a value of zero. When the significance flag is set to "FALSE", no further syntax elements of the associated quantization coefficient are communicated. The significance flag can be encoded into the bitstream 133 by using a context-adaptive binary arithmetic coding (CABAC) entropy encoder.

[0162] While the CABAC encoder encodes context-encoded syntax elements relatively efficiently, limiting the number of context-encoded syntax elements is generally desirable to minimize the computational requirements and costs for hardware implementation. Therefore, after the quantization coefficients 336 are binarized into several syntax elements by the entropy encoder 338, some syntax elements are context-encoded into the bitstream 133, while others are bypass-encoded into the bitstream 133. The total number of context-encoded syntax element bins is limited per transformation block. The VVC standard sets the limit at 1.75 bins per sample. For example, for an 8x8 transformation block consisting of 64 samples, the context-encoded bin budget is set at 112 bins. Whenever a syntax element is context-encoded during the process of encoding the TB into the bitstream 133, the remaining context-encoded bin budget is tracked and decremented. Whenever the remaining context-encoded bin budget is exhausted, the remaining quantization coefficients and associated syntax elements must be bypass-encoded.

[0163] Figure 13 shows a method 1300 for encoding the quantization coefficients (336) of the currently selected subblock into a bitstream 133. Method 1300 is performed in step 960 of Method 900. Method 1300 can be performed by a device such as a configured FPGA, ASIC, or ASSP. Furthermore, Method 1300 can be performed by a video encoder 114 under the execution of a processor 205. Thus, Method 1300 can be performed as a module of software 233 stored in a computer-readable storage medium and / or memory 206. Method 1300 begins with step 1310, in which the first coefficient is selected.

[0164] In step 1310, where the first coefficient is selected, method 1300 selects the quantization coefficient of the current subblock. If the current subblock contains the last significance coefficient position, the currently selected coefficient is set to the last significance coefficient. Otherwise, if the current subblock does not contain the last significance coefficient position, the currently selected coefficient is set to the coefficient of the lower right of the current subblock. Method 1300 then proceeds to context coding usage check 1320.

[0165] In the context coding usage check 1320, the video encoder 114 checks whether the remaining context coding bin budget is 4 or greater. If the remaining context coding bin budget is 4 or greater, step 1320 returns "YES", and method 1300 proceeds to the context coding syntax element coding step 1330. Otherwise, if the current context coding bin budget is less than 4, step 1320 returns "NO", and method 1300 proceeds to the reminder pass coding step 1370.

[0166] In the context coding syntax element coding step 1330, the video encoder 114 can encode a number of syntax elements into the bitstream 133 using a CABAC encoder that potentially includes significant flags, a greater than 1 flag, a parity flag, and a greater than 3 flag. Each bin associated with a syntax element is coded by the CABAC encoder using a "context model". The context model for each bin may be selected depending on the current value of the quantizer state Qstate. Furthermore, whenever a context coding bin is coded into the bitstream 133 by the CABAC encoder, the remaining context coding bin budget is reduced by 1 in step 1330.

[0167] In step 1330, if the current coefficient is the last significant coefficient, the significance flag is set to "TRUE" but is not encoded in bitstream 133. If the currently selected subblock is not the first or last subblock in the backward scan order 810, and the currently selected coefficient is the last coefficient as described below in the last coefficient check 1350, and all significance flags for previous coefficients in the currently selected subblock were "FALSE", then the significance flag is set to "TRUE". The significance flag is not encoded in bitstream 133. If the current coefficient has a magnitude of zero, in step 1330, the significance flag is set to "FALSE" and context-encoded in bitstream 133. Otherwise, the significance flag is set to "TRUE" and context-encoded in bitstream 1330.

[0168] If the current coefficient has a magnitude of 1, the flag greater than 1 is set to "FALSE" and context-encoded into bitstream 1330 in step 1330. Otherwise, the flag greater than 1 is set to "TRUE" and context-encoded into bitstream 133.

[0169] If the current coefficient has a magnitude of at least 2, the parity flag is set to "FALSE" if the current coefficient is even, and to "TRUE" if the current coefficient is odd. The parity flag is context-encoded into bitstream 1330 in step 1330. If the current coefficient has a magnitude greater than 3, the greater than 3 flag is set to "TRUE" and is context-encoded into bitstream 1330 in step 1330. Otherwise, if the current coefficient has a magnitude of 2 or 3, the greater than 3 flag is set to "FALSE" and is context-encoded into bitstream 133.

[0170] Method 1300 proceeds from step 1330 to DQ test 1340 under the control of processor 205. Depending on the coefficient selected in 1310, Method 1300 sets (or possibly encodes) a significance flag and then proceeds to step 1340. Otherwise, Method 1300 encodes the last appropriate one of the flags greater than 1, the parity flag, and the flag greater than 3 and then proceeds to step 1340.

[0171] In DQ test 1340, the same conditions checked in DQ test 1110 are used to determine whether step 1340 returns "YES" or "NO". If step 1340 returns "YES", method 1300 proceeds to Qstate update step 1345. Otherwise, if step 1340 returns "NO", method 1300 proceeds to final coefficient check 1350.

[0172] In step 1345, the Qstate update is performed, and the quantizer state Qstate is updated based on the parity of the current coefficients according to Table 1. Method 1300 proceeds from step 1345 to the final coefficient check 1350.

[0173] In the final coefficient check 1350, the video encoder 114 checks whether the currently selected coefficient is the top-left coefficient of the currently selected subblock. If the currently selected coefficient is the top-left coefficient of the currently selected subblock, step 1350 returns "YES", and method 1300 proceeds to reminder pass encoding step 1370. Otherwise, if the current coefficient is not the top-left coefficient, step 1350 returns "NO", and method 1300 proceeds to step 1360, which selects the next coefficient.

[0174] In step 1360, where the next coefficient is selected, the next coefficient of the currently selected subblock is selected in the backward diagonal scan order 810. Method 1300 proceeds from step 1360 to context coding usage check 1320.

[0175] In step 1370, which encodes the reminder path, the remaining magnitude of the quantization coefficients of the currently selected subblock is binarized and bypass-encoded to the bitstream 133 by, for example, the entropy encoder 338. The quantization coefficients are encoded, for example, in the backward diagonal scan order 810. If the quantization coefficients are context-encoded by the CABAC coder (i.e., the context encoding use check 1320 is passed (returned "YES")), and the flag greater than 3 is "TRUE", then the quantization coefficients at scan position n have a remaining magnitude r[n]. The remaining magnitude is determined using equation (4). r[n]=(x[n]-4)>>1, (4)

[0176] Here, equation (4), x[n] is the absolute magnitude of the quantization coefficient at scan position n. The magnitude r[n] is binarized and bypass-encoded into bitstream 133. If the quantization coefficient was not context-encoded (context-encoded use check 1320 did not pass / returned "NO"), the absolute magnitude x[n] is binarized and bypass-encoded into bitstream 133. Method 1300 proceeds from step 1370 to SBH test 1380.

[0177] The SBH test 1380 uses the same conditions checked in the SBH test 1140 to determine whether step 1380 returns "YES" or "NO". If the SBH test 1140 returns "NO", step 1380 returns "NO", and method 1300 proceeds to step 1390, which encodes N codes. Otherwise, the video encoder 114 checks the positions of the first and last significance coefficients in the current subblock. If the difference between the first and last significance positions is greater than 3, step 1380 returns "YES", and method 1300 proceeds to step 1395, which encodes N-1 codes. Otherwise, step 1380 returns "NO", and method 1300 proceeds to step 1390, which encodes N codes.

[0178] As described in relation to step 1140, the code bit hiding test can rely on a number of alternative flags or settings in different implementations. If the enabled code bit hiding flag is set (has a value of "TRUE"), different implementations may make a determination based on the TB's transform skip flag, the TSRC invalid flag, or whether the TB's QP meets the thresholds relevant to reversible coding. Thus, step 1380 determines whether code bit hiding is enabled depending on a value or flag related to the transform block itself, or a higher-level value of the TSRC invalid flag. Step 1380 provides some flexibility for implementing reversible coding. Implementations that use a value or flag related to the transform block to determine whether the code enables hiding are particularly well-suited to allowing flexibility when implementing reversible coding using RRC.

[0179] In step 1390, which encodes N codes, the code bits of any significance coefficients in the currently selected subblock are bypass-encoded into bitstream 133. The code bits are bypass-encoded into bitstream 133 based, for example, on the backward diagonal scan order 810. Method 1300 terminates after the execution of step 1390.

[0180] In step 1395, which encodes N-1 codes, the code bits of the significance coefficients of the currently selected subblock are bypass-encoded into bitstream 133 based on the backward diagonal scan order 810. The code bits associated with the first significance coefficient (the last to be encountered in the backward diagonal scan order 810) are not encoded into bitstream 133. In other words, if the currently selected subblock has N significance coefficients, then N-1 code bits are bypass-encoded into bitstream 133. Method 1300 ends with the execution of step 1395.

[0181] Figure 14 shows a method 1400 for decoding the quantization coefficient (424) of a currently selected subblock from the bitstream 133. Method 1400 is performed in step 1050 of Method 1000. Method 1400 can be performed by a device such as a configured FPGA, ASIC, or ASSP. Furthermore, Method 1400 may be performed by a video decoder 134 under the execution of a processor 205. Thus, Method 1400 can be performed as a module of software 233 stored in a computer-readable storage medium and / or memory 206. Method 1400 begins with step 1410 of selecting the first coefficient.

[0182] In step 1410, where the first coefficient is selected, method 1400 selects the first quantization coefficient of the current subblock. If the current subblock contains the last significance coefficient position, the currently selected coefficient is set to the last significance coefficient. Otherwise, the currently selected coefficient is set to the coefficient in the lower right of the current subblock. Method 1400 proceeds from step 1410 to step 1420, where context coding usage check is performed.

[0183] In the context coding usage check 1420, the video decoder 134 checks whether the remaining context coding bin budget meets a threshold, typically whether the remaining context coding bin budget for the transform block is 4 bins or more. If the remaining budget is 4 or more, step 1420 returns "YES", and method 1400 proceeds to the context coding syntax element determination step 1430. Otherwise, if the remaining CABAC budget is less than the threshold (4 bins), step 1420 returns "NO", and method 1400 proceeds to the reminder path decoding step 1470.

[0184] In the context coding syntax element determination step 1430, the video decoder 134 can decode a number of context coding syntax elements from the bitstream 133 using a CABAC coder. Each bin associated with a syntax element is decoded by the CABAC coder using a "context model". The context model for each bin may be selected depending on the current value of the quantizer state Qstate. Furthermore, whenever a context coding bin is decoded from the bitstream 133 by the CABAC coder, the remaining context coding bin budget is reduced by 1.

[0185] If the current coefficient is the last significant coefficient, the significance flag is inferred as "TRUE" instead of being decoded from bitstream 133. If the currently selected subblock is not the first or last subblock in the backward scan order 810, and the currently selected coefficient is the last coefficient as described below in the last coefficient check 1450, and all significance flags for previous coefficients in the currently selected subblock were "FALSE", then the significance flag is inferred as "TRUE". Otherwise, the significance flag is context-decoded from bitstream 133 in step 1430. If the significance flag is set to "FALSE", the currently selected coefficient is assigned a value of zero, and method 1400 proceeds to the DQ test 1440.

[0186] If the significance flag is set to "TRUE", in step 1430, a flag greater than 1 is context-decoded from bitstream 133. If the flag greater than 1 is set to "FALSE", the currently selected coefficient is assigned a magnitude of 1, and method 1400 proceeds to DQ test 1440.

[0187] If the flag greater than 1 is set to "TRUE", the parity flag and the flag greater than 3 are context-decoded from bitstream 133. Method 1400 proceeds to DQ test 1440. The number of flags determined in step 1430 depends on the position and value of the coefficients selected in step 1410. Proceeding from step 1430 may occur after the significance flag has been inferred or decoded, or after the appropriate flag greater than 1, parity flag, or flag greater than 3 has been decoded.

[0188] In DQ test 1440, the same conditions checked in DQ test 1210 are used to determine whether step 1440 returns "YES" or "NO". If step 1440 returns "YES", method 1400 proceeds to Qstate update step 1445. Otherwise, if step 1440 returns "NO", method 1400 proceeds to final coefficient check 1450.

[0189] In step 1445, the Qstate update is performed, and the quantizer state Qstate is updated based on the parity of the currently selected coefficients according to Table 1. If the currently selected coefficients have a value of zero, the parity is zero. The parity is 1 if the currently selected coefficients have a magnitude of 1. Otherwise, the parity is zero if the parity flag is set to "FALSE", and 1 if the parity flag is set to "TRUE". Method 1400 proceeds from step 1445 to the final coefficient check 1450.

[0190] In the final coefficient check step 1450, the video decoder 134 checks whether the currently selected coefficient is the top-left coefficient of the currently selected subblock. If the currently selected coefficient is the top-left coefficient of the currently selected subblock, step 1450 returns "YES", and method 1400 proceeds to reminder pass decoding step 1470. Otherwise, if the currently selected coefficient is not the top-left coefficient, step 1450 returns "NO", and method 1400 proceeds to step 1460 to select the next coefficient.

[0191] In step 1460, where the next coefficient is selected, the next coefficient of the currently selected subblock in the backward diagonal scan order 810 is selected. Method 1400 proceeds from step 1460 to context coding usage check 1420.

[0192] In reminder pass decoding step 1470, the remaining magnitude of the quantization coefficients of the currently selected subblock is bypass-decoded from bitstream 133. The quantization coefficients are processed in backward diagonal scan order 810. If the quantization coefficients are context-decoded (context coding use check 1420 is passed or returns "YES") and the flag greater than 3 is decoded with a value of "TRUE", the remaining magnitude r[n] is bypass-decoded from bitstream 133, where n is the scan position of the quantization coefficients. The absolute magnitude x[n] of the quantization coefficients is determined as x[n]=4+p[n]+2*r[n], where p[n] has a value of zero if the parity flag is decoded as "FALSE", and p[n] has a value of 1 if the parity flag is decoded as "TRUE".

[0193] If the quantization coefficients are context-decoded and flags greater than 1 are decoded as "TRUE" but flags greater than 3 are not decoded or are decoded as "FALSE", the absolute magnitude is determined as x[n] = 2 + p[n]. If the quantization coefficients are not context-decoded (context coding use check 1420 does not pass and returns "NO"), the absolute magnitude x[n] is bypass-decoded from bitstream 133. Method 1400 proceeds from step 1470 to SBH test 1480.

[0194] In the SBH test 1480, the video decoder 134 determines whether sign bit hiding is being used, i.e., whether one sign bit has been inferred for the currently selected subblock. The test used in step 1480 is the same as the test used in step 1140 on the encoder side. The video decoder 134 checks the value of the effective sign bit hiding flag, which may be communicated as high-level syntax within the bitstream 133. If the effective dependent quantization flag is "TRUE", the effective sign bit hiding flag is inferred to be "FALSE". If the effective sign bit hiding flag is "FALSE", step 1480 returns "NO", and method 1400 proceeds to the sign decoding step 1490.

[0195] The video decoder 134 checks the positions of the first and last significance coefficients of the currently selected subblock. If the difference between the first significance position and the last significance position is 3 or less, step 1480 returns "NO", and method 1400 proceeds to the coding / decoding step 1490.

[0196] In one configuration of the SBH test 1480, the determination depends on the value of the valid code bit hiding flag and the value of the conversion skip flag for the current TB. If the valid code bit hiding flag is "TRUE" and the difference between the first significant position and the last significant position is greater than 3, the video decoder 134 also checks the value of the conversion skip flag for the current TB. If the conversion skip flag is "TRUE", step 1480 returns "NO", and method 1400 proceeds to the code decoding step 1490. Otherwise, if the conversion skip flag is "FALSE", step 1480 returns "YES", and method 1400 proceeds to step 1495, which decodes and infers the code.

[0197] In another configuration of the SBH test 1480, the determination depends on the value of the valid code bit hiding flag and the value of the TSRC invalid flag. If the valid code bit hiding flag is "TRUE" and the difference between the first significant position and the last significant position is greater than 3, the video decoder 134 also checks the value of the TSRC invalid flag. If the TSRC invalid flag is "TRUE", step 1480 returns "NO", and method 1400 proceeds to the code decoding step 1490. Otherwise, if the TSRC invalid flag is "FALSE", step 1480 returns "YES", and method 1400 proceeds to step 1495, which decodes and infers the code.

[0198] In yet another configuration of the SBH test 1480, the determination depends on the value of the valid code bit hiding flag and the value of the quantization parameter (QP) of the current TB, and the difference between the first and last significant positions of the quantization parameter QP of the transform block. If the valid code bit hiding flag is "TRUE" and the difference between the first and last significant positions is greater than 3, the video decoder 134 also checks the value of the QP of the current TB. If QP is 4 (or any value indicating a reversible operation), step 1480 returns "NO" and method 1400 proceeds to the code decoding step 1490. Otherwise, if QP is not 4 (or a similar value indicating a reversible operation), step 1480 returns "YES" and method 1400 proceeds to step 1495, which decodes and infers the code.

[0199] In the code path decoding step 1490, the sign bit of any significance coefficient of the currently selected subblock is bypass-decoded from the bitstream 133. The sign bit is bypass-decoded from the bitstream 133 in the backward diagonal scan order 810. If the associated sign bit has a value of 1, the value of the quantization coefficient is set to -x[n]. If the associated sign bit has a value of zero, the value of the quantization coefficient is set to x[n]. Method 1400 ends with the execution of step 1490.

[0200] In step 1495, which decodes and infers the codes, the sign bits of the significance coefficients of the currently selected subblock are bypass-decoded from bitstream 133 in backward diagonal scan order 810. The sign bit associated with the first significance coefficient (the last one visited in backward diagonal scan order 810) is not decoded from bitstream 133. In other words, if the currently selected subblock has N significance coefficients, then N-1 sign bits are bypass-decoded from bitstream 133. The sign bit associated with the first significance coefficient is inferred based on the sum of the parity of the significance coefficients. If the sum of the parity is zero, the sign bit associated with the first significance coefficient is inferred to be zero. If the sum of the parity is 1, the sign bit associated with the first significance coefficient is inferred to be 1. If the associated sign bit has a value of 1, the value of the quantization coefficient is set to -x[n]. The value of the quantization coefficient is set to x[n] if the associated sign bit has a value of zero. Method 1400 then terminates.

[0201] The configurations described in Methods 900 and 1000 enable lossless compression of video data while using a standard residual coding process. Dependent quantization and code bit hiding are lossy coding tools that can be flexibly disabled when lossless operations are desired, but may still be available to achieve improved coding performance in lossy coding blocks.

[0202] Industrial applicability The described configuration is applicable to the computer and data processing industries, particularly to digital signal processing for decoding and encoding signals such as video and image signals, achieving high compression efficiency.

[0203] The foregoing describes only some embodiments of the present invention, and modifications and / or changes can be made to the present invention without departing from the scope and spirit of the invention. The embodiments are illustrative and not limiting.

Claims

1. A method for decoding a transform block from a bitstream, Decoding a first flag used to determine whether or not to use dependent quantization in the aforementioned transformation block, To determine whether to use sign bit hiding in the conversion block, where data indicating the sign of the significance coefficient at a certain position is not decoded from the bitstream, If the first flag is TRUE, the sign bit hiding is not used in the conversion block. If it is determined that the dependent quantization is to be used in the transformation block based on the first flag, the transformation block is decoded using the dependent quantization. If it is determined that the sign bit hiding should be used, the process includes decoding the conversion block using the sign bit hiding, After checking the first flag, the invalid flag for transform skip residual coding is checked, and if the invalid flag is TRUE, the dependent quantization is not used. The signaling of the invalid flag depends on the information indicated by the first flag. If the invalid flag is TRUE, the sign bit hiding will not be used in the conversion block, even if the sign bit hiding enable flag is TRUE. The invalid flag indicates whether the first residual coding is applied instead of the second residual coding even when the conversion process is skipped, the first residual coding is the process for blocks where the conversion process is not skipped, and the second residual coding is the process for blocks where the conversion process is skipped. A method characterized by the following:

2. A method for encoding a block to be converted into a bitstream, Encoding a first flag used to determine whether or not to use dependent quantization in the transformation block, To determine whether to use sign bit hiding in the conversion block, where data indicating the sign of the significance coefficient at a certain position is not encoded in the bitstream, If the first flag is TRUE, the sign bit hiding is not used in the conversion block. If it is determined based on the first flag that the dependent quantization should be used in the transformation block, the transformation block is encoded using the dependent quantization. If it is determined that the sign bit hiding should be used, the process includes encoding the conversion block using the sign bit hiding, After checking the first flag, the invalid flag for transform skip residual coding is checked, and if the invalid flag is TRUE, the dependent quantization is not used. The signaling of the invalid flag depends on the information indicated by the first flag. If the invalid flag is TRUE, the sign bit hiding will not be used in the conversion block, even if the sign bit hiding enable flag is TRUE. The invalid flag indicates whether the first residual coding is applied instead of the second residual coding even when the conversion process is skipped, the first residual coding is the process for blocks where the conversion process is not skipped, and the second residual coding is the process for blocks where the conversion process is skipped. A method characterized by the following:

3. A decoding device that decodes a conversion block from a bitstream, Means for decoding a first flag used to determine whether or not to use dependent quantization in the transformation block, Means for determining whether to use sign bit hiding in the conversion block, in which data indicating the sign of the significance coefficient at a certain position is not decoded from the bitstream, If the first flag is TRUE, the sign bit hiding is not used in the conversion block. If it is determined based on the first flag that the dependent quantization should be used in the transformation block, means for decoding the transformation block using the dependent quantization, If it is determined that the sign bit hiding should be used, the means for decoding the conversion block using the sign bit hiding is provided. After checking the first flag, the invalid flag for transform skip residual coding is checked, and if the invalid flag is TRUE, the dependent quantization is not used. The signaling of the invalid flag depends on the information indicated by the first flag. If the invalid flag is TRUE, the sign bit hiding will not be used in the conversion block, even if the sign bit hiding enable flag is TRUE. The invalid flag indicates whether the first residual coding is applied instead of the second residual coding even when the conversion process is skipped, the first residual coding is the process for blocks where the conversion process is not skipped, and the second residual coding is the process for blocks where the conversion process is skipped. A decoding device characterized by the following features.

4. An encoding device for encoding a block to be converted into a bitstream, Means for encoding a first flag used to determine whether or not to use dependent quantization in the transformation block, Means for determining whether to use sign bit hiding in the conversion block, where data indicating the sign of the significance coefficient at a certain position is not encoded in the bitstream, If the first flag is TRUE, the sign bit hiding is not used in the conversion block. If it is determined based on the first flag that the dependent quantization should be used in the transformation block, means for encoding the transformation block using the dependent quantization, If it is determined that the code bit hiding should be used, the system includes means for encoding the conversion block using the code bit hiding. After checking the first flag, the invalid flag for transform skip residual coding is checked, and if the invalid flag is TRUE, the dependent quantization is not used. The signaling of the invalid flag depends on the information indicated by the first flag. If the invalid flag is TRUE, the sign bit hiding will not be used in the conversion block, even if the sign bit hiding enable flag is TRUE. The invalid flag indicates whether the first residual coding is applied instead of the second residual coding even when the conversion process is skipped, the first residual coding is the process for blocks where the conversion process is not skipped, and the second residual coding is the process for blocks where the conversion process is skipped. An encoding device characterized by the following features.

5. A computer program for causing a computer to perform the method described in claim 1.

6. A computer program for causing a computer to perform the method described in claim 2.

Images